Skip to content

gh-161746: Use the Neri–Schneider algorithm for ordinal-to-date conversion in datetime - #161747

Open
mazalovmx wants to merge 9 commits into
python:mainfrom
mazalovmx:datetime-neri-schneider
Open

mazalovmx wants to merge 9 commits into
python:mainfrom
mazalovmx:datetime-neri-schneider

Conversation

@mazalovmx

@mazalovmx mazalovmx commented Nov 10, 2026 •

Copy link
Copy Markdown

gh-161746

Replace ordinal-to-date conversion with the Neri–Schneider algorithm
in both the C and pure-Python implementations. The C version follows
the same arithmetic steps, uses 32-bit intermediates for valid dates,
and preserves the year reported for out-of-range ordinals.

Verification of the review follow-up

  • Built the final working tree with MSVC 19.44.35230 x64 and Windows SDK
    10.0.26100.0, Release with LTCG, without PGO.
  • test_datetime, test_calendar, test_strptime, test_time,
    test_zoneinfo, test_plistlib, test_xmlrpc, and test___all__:
    SUCCESS; run=1,615 skipped=151. The build omits optional extensions including SSL
    and ctypes; skipped tests are recorded in the test log.
  • Exhaustively compared all 3,652,059 supported dates with the old conversion
    algorithm: zero mismatches for both C and pure Python.
  • Checked exact out-of-range error types and messages, subclass constructor
    arguments, and ISO upper bounds, including 20,008 large ordinals for both
    date and datetime.
  • Restored the deterministic first/last-day checks around the 4-, 100-, and
    400-year boundaries, as requested. The out-of-range year tests are retained.
    The PR no longer changes hypothesis_helper.py, and datetimetester
    imports with warnings treated as errors without loading Hypothesis.
  • Separately compiled the extracted C functions with MSVC and checked both
    against an independent Gregorian counter for all 3,652,061 internal
    ordinals through MAXORDINAL + 2.

Benchmarks

Previously reported Linux measurements at b3f74a0, before the Windows
ternary follow-up. These were not rerun as part of the Windows check.
pyperf, bench_ordinal.py, main (e891a91) vs. the patch; µs (C) or ms
(pure Python) per 1000 operations, mean of the pyperf runs:

Benchmark C main C patched C speedup Py main Py patched Py speedup
date.fromordinal, random dates 75.5 65.2 1.16x 2.64 2.01 1.31x
date.fromordinal, consecutive dates 63.2 — not significant 2.55 1.94 1.32x
date + timedelta(days=40) 95.3 82.0 1.16x 4.61 3.89 1.19x
datetime + timedelta(days=40) 90.5 81.2 1.11x 22.0 — not significant
date.fromisocalendar 170 146 1.16x 4.21 3.57 1.18x
date + timedelta(days=1), 1000 steps 75.9 78.2 1.03x slower 5.17 3.61 1.43x
Geometric mean 1.09x 1.24x

In C, date + timedelta(days=1) does not call ord_to_ymd()
(normalize_y_m_d() handles the step to the next month directly), so no gain
is expected there. The historical pyperf run reported about a 3% slowdown
in this control benchmark as significant; it was not rerun here.

Windows/MSVC review follow-up

The final C adjustment uses Cassio Neri's suggested conditional expressions.
MSVC 19.44.35230 x64 emits conditional moves instead of the final conditional
jump in a standalone /O2 build of the conversion function.

Compared commit 8ea445bffa8f with and without this change on Windows 11,
AMD Ryzen 5 3500U, x64 Release with LTCG, without PGO. Times are microseconds
per 1,000 operations:

Operation Before ternaries With ternaries Time change Paired speedup, 95% CI
date.fromordinal, random dates 142.9 146.7 +2.7% 0.932–1.021
date + timedelta(days=40) 160.3 161.1 +0.5% 0.955–1.039
date.fromisocalendar 302.9 299.9 -1.0% 0.976–1.042
datetime + timedelta(days=40) 169.0 169.6 +0.3% 0.956–1.035
date.fromordinal, consecutive dates 140.1 138.5 -1.1% 0.962–1.060
date + timedelta(days=1), 1000 steps 137.5 135.9 -1.2% 0.970–1.053

The six original benchmark loops were run through a stdlib runner, not
pyperf: 12 paired process runs, alternating A/B order, pinned to one logical
CPU, with three warmups and five samples per benchmark per process. The
95% paired-bootstrap intervals are for baseline time divided by time with
the change; all include 1. No consistent API speedup or regression was
detected. These measurements isolate the ternary change; they do not compare
the new calendar algorithm with the old algorithm on Windows. The C source
in the final follow-up is identical to the measured ternary version.

Earlier measurements (first version of the patch, 64-bit form; superseded)

pyperf, x86-64 VM, gcc 13.3, -O3, no PGO; C implementation, µs per 1000 operations:

Benchmark main patched Speedup
date.fromordinal, random dates 35.6 25.3 1.40x
date + timedelta(days=40) 42.9 35.7 1.20x
date.fromisocalendar 83.5 71.0 1.18x
datetime + timedelta(days=40) 37.9 34.1 1.11x
date.fromordinal, consecutive dates 26.3 23.8 1.10x
date + timedelta(days=1), 1000 steps 32.8 31.8 1.03x

With a PGO build (--enable-optimizations, gcc 13.3) of main (e3262d9) and of
that version, 1,000,000 operations, best of 7, builds alternated:

Operation main patched Speedup
date.fromordinal, random dates 62.6 ms 45.9 ms 1.36x
date + timedelta(days=90) 69.0 ms 60.8 ms 1.13x
date.fromisoformat('2024-W07-1') 88.6 ms 77.8 ms 1.14x
date + timedelta(days=10) 53.0 ms 52.5 ms 1.01x

pyperformance 1.14.0 (--fast, same two PGO builds, 119 benchmarks in common):
geometric mean 1.00x. No pyperformance benchmark exercises this code path, so
no change is expected there.

AI tools assisted with implementation, review, and local verification.

… conversion

Replace the 400/100/4/1-year cycle decomposition and the month
estimate-and-correct step in ord_to_ymd() (Modules/_datetimemodule.c)
and _ord2ymd() (Lib/_pydatetime.py) with Figure 12 of Neri and
Schneider, "Euclidean affine functions and their application to
calendar algorithms" (Softw Pract Exper. 2023;53(4):937-970), which
GCC's libstdc++, the Linux kernel, .NET and Go already use.

The C version computes the year and the month steps with one
multiplication each, and does the century step in 64 bits so that every
positive int ordinal gives the same result as before.  The pure-Python
version uses plain floor division.

Both versions return the same results as the old code for every
ordinal from 1 to 3,652,059; the C version also for every positive int,
and the pure-Python one for 2.3 million other integers, including
negative and huge ones.  In microbenchmarks on x86-64 (gcc 13, -O3),
date.fromordinal() on random dates is about 1.4x faster, date +
timedelta across a month boundary and date.fromisocalendar() about 1.2x
faster, and the pure-Python date.fromordinal() about 1.3x faster.

Add a test for the first and last days of every month in years at the
start and the end of the 4-, 100- and 400-year cycles.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@python-cla-bot

python-cla-bot Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

Comment thread Modules/_datetimemodule.c Outdated
* ceil(2**32 / DI4Y), so the high 32 bits are the quotient and the low
* 32 bits encode the remainder. */
const uint32_t year_scaled = 4 * day_of_century + 3;
const uint64_t year_product = (uint64_t)2939745 * year_scaled;

@StanFromIreland StanFromIreland Nov 10, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like these magic numbers, can you please benchmark the simpler approach (compilers are pretty smart, and should be able to deal with these), like you implemented in Python? If we accept this (I still have to think about this), they should be as close as possible for maintenance purposes.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. I rewrote it to follow _ord2ymd() step by step, with plain / and % by constants. gcc 14 and clang 19 turn them into multiplications anyway (no div instructions at -O3).

Results, C implementation, µs per 1000 operations (pyperf, gcc 14.2, -O3, no PGO):

Benchmark main magic numbers simple
date.fromordinal, random dates 101 60.8 (1.65x) 62.2 (1.62x)
date + timedelta(days=40) 104 82.0 (1.27x) 85.2 (1.22x)
date.fromisocalendar 184 146 (1.26x) 156 (1.18x)
datetime + timedelta(days=40) 102 78.7 (1.29x) 83.2 (1.22x)
date.fromordinal, consecutive dates 65.9 61.9 (1.06x) 59.8 (1.10x)
Geometric mean (6 benchmarks) 1.25x 1.21x

Simple vs. magic numbers: 4 of 6 benchmarks not significant, geometric mean 1.03x slower. On its own, the function takes about 6.5 ns instead of 5.5 ns (the old code: about 17 ns). That isn't worth the magic numbers, so I've pushed the simple version.

It still matches the old code for every positive int ordinal (gcc and clang) and for all dates in the supported range.

Speed up conversion of ordinals to dates in :mod:`datetime`, used by
:meth:`~datetime.date.fromordinal`, :meth:`~datetime.date.fromisocalendar`
and :class:`~datetime.timedelta` arithmetic, by switching to the
Neri–Schneider algorithm.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a link, and a rough % speedup.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Use plain division and modulo by constants instead of hand-derived
multiply-shift constants; compilers emit multiplications anyway.
Add a link to the paper and rough speedups to the NEWS entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bedevere-app bedevere-app Bot added the type-feature A feature request or enhancement label Nov 10, 2026
@StanFromIreland

Copy link
Copy Markdown
Member

CC @cassioneri this may be of interest.

Comment thread Lib/test/datetimetester.py Resolved
Comment thread Lib/_pydatetime.py Resolved
Comment thread Modules/_datetimemodule.c Outdated

*day = n + 1;
/* Days since 0000-03-01, the start of the computational calendar. */
const uint64_t days = (uint64_t)ordinal + 305;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can drop the 64-bit arithmetic here entirely. As far as I can tell, the only reason for it is so that date.fromordinal() with an ordinal past MAXORDINAL still gets to the same "year must be in 1..9999, not N" error that it raises on main, because fromordinal only checks ordinal < 1 before calling ord_to_ymd. The other two callers never pass anything larger than MAXORDINAL + 2: normalize_date already range-checks the ordinal, and iso_to_ymd can reach at most 10000-01-02, the last day of ISO year 9999, which its caller then rejects.

Since the calendar repeats exactly every 400 years, fromordinal can handle the error case by shifting the ordinal into the first 400-year cycle and adding the years back afterwards, which keeps the error message identical without needing a 64-bit path in the hot function:

    if (ordinal < 1) {
        PyErr_SetString(PyExc_ValueError, "ordinal must be >= 1");
        return NULL;
    }
    if (ordinal <= MAXORDINAL) {
        ord_to_ymd(ordinal, &year, &month, &day);
    }
    else {
        /* Out of range: find the year anyway, so new_date_subclass_ex()
         * raises the usual "year must be in 1..9999, not N" error.  The
         * calendar repeats every 400 years (DI400Y days), so convert the
         * equivalent date in the first 400-year cycle and add the years
         * back. */
        int cycles = (ordinal - 1) / DI400Y;
        ord_to_ymd(ordinal - cycles * DI400Y, &year, &month, &day);
        year += 400 * cycles;
    }
    return new_date_subclass_ex(year, month, day, type);

With that change, every intermediate value in ord_to_ymd fits in a uint32_t — 4 * days + 3 tops out around 14.6 million — so the function can assert ordinal <= MAXORDINAL + 2 and the comment explaining why the century step is done in 64 bits can go away. I checked this version against the ord_to_ymd on main for every positive int, and the year, month and day all match.

I did not see a measurable speed difference on x86-64 with either gcc or clang, so this is mostly an argument for simplicity. It may help somewhat on 32-bit platforms, where the 64-bit division can turn into a library call, though I have not measured that. The pure Python version does not need an equivalent change, since it uses arbitrary-precision integers and floor division.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, I used your version. ord_to_ymd() is now uint32_t only, with assert(ordinal <= MAXORDINAL + 2), and fromordinal() handles bigger ordinals with the 400-year shift. I checked it against main for all positive int, and the error message is the same. I also added a test that the error still has the correct year for some ordinals after MAXORDINAL.

I checked the changes twice.

mazalovmx and others added 3 commits November 10, 2026 22:01
* ord_to_ymd() uses only uint32_t; fromordinal() shifts larger
  ordinals by 400-year cycles, so the error message is unchanged.
* Replace the hand-picked loop with a Hypothesis round-trip test
  and add tests for the error year after MAXORDINAL.
* Explain each step in the comments of both implementations; cite
  the paper by DOI.
* Update the NEWS entry with new measurements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test.test___all__ imports test.datetimetester, which now imports
Hypothesis.  Hypothesis imports the deprecated typing.AnyStr, and
test___all__ fails on the unexpected DeprecationWarning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread Lib/test/support/hypothesis_helper.py Outdated

try:
import hypothesis
with warnings.catch_warnings():

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not really a fan of this, we're working around a Python 3.16 incompatibility in Hypothesis, and this is unrelated to the change here. The property test also doesn't cover much over the examples, also most Buildbots don't have Hypothesis installed so only the @examples run IIRC. I'd prefer reverting to the previous version of the test and dropping the Hypothesis changes. CC @pganssle

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted the property test to the previous deterministic checks of the first
and last days of every month around the 4-, 100-, and 400-year boundaries.
These checks use self.theclass and run without Hypothesis. The tests for the
year reported by out-of-range ordinals are retained.

Also removed the Hypothesis import from datetimetester and reverted all
changes to hypothesis_helper.py, so that file is no longer part of this PR.

Reduce duplication and make them a little more succinct.
@StanFromIreland

Copy link
Copy Markdown
Member

I pushed a little commit to tidy up the comments.

@cassioneri

Copy link
Copy Markdown

CC @cassioneri this may be of interest.

Hi all.

Lorenz and I wish to thank you for your interest in our work. We hope this goes well and our algorithms make their way into cpython.

My two cents follow.

  1. Kudos to @mazalovmx who did a great job. 👏🏻

  2. I'm sad that @StanFromIreland doesn't like the optimised versions (the one with magical numbers), but fair enough: he got a point and I won't argue about that. I just wish to mention that, as he said, "compilers are pretty smart" and apply an interesting technique to optimise divisions by constants. As a by product, % is optimised as well. However, our work goes further and apply special optimisations for % that no compiler implements. Can I kindly recommend you to watch my talk? It explains this and other relevant points: https://www.youtube.com/watch?v=J9KijLyP-yg (You might enjoy it 😃.)

  3. I looked at the assembly code generated by gcc 16.2, clang 23.1.0 and msvc latest (https://godbolt.org/z/fbnvn1nd6).

For gcc and clang, the assembly looks good to me. Unfortunately, msvc adds a branch for this part of the code:

    if (m > 12) {  /* January or February */
        *year = (int)(y + 1);
        *month = (int)(m - 12);
    }
    else {
        *year = (int)y;
        *month = (int)m;
    }

The above is equivalent to:

    const uint32_t jan_or_feb = m > 12;
    *year = (int) (jan_or_feb ? y + 1 : y);
    *month = (int) (jan_or_feb ? m - 12 : m);

(see https://godbolt.org/z/PGe83cPPx). This change has no impact whatsoever on gcc. For clang the assembly slightly changes but nothing that catch my attention. Now, for msvc the branch disappears and this is a good thing. Therefore, I do recommend this change. (Please rerun benchmarks.)

Finally, I have a question, out of curiosity: Why is there an implementation in Python and another one in C?

I hope this helps and, again, I thank you all for considering using our algorithms in cpython.

@StanFromIreland

Copy link
Copy Markdown
Member

Hello @cassioneri!

I just wish to mention that, as he said, "compilers are pretty smart" and apply an interesting technique to optimise divisions by constants. As a by product, % is optimised as well. However, our work goes further and apply special optimisations for % that no compiler implements. Can I kindly recommend you to watch my talk? It explains this and other relevant points: https://www.youtube.com/watch?v=J9KijLyP-yg (You might enjoy it 😃.)

Hmm, I'll think about this a bit more, but I'm still in favour of the current, clearer, logic.

I hadn't seen the talk previously, nice!

Now, for msvc the branch disappears and this is a good thing.

These are micro-optimisations, I haven't benchmarked but I presume it will have little impact in practice. @mazalovmx are you able to test this, I'm afraid I don't have a Windows machine.

Finally, I have a question, out of curiosity: Why is there an implementation in Python and another one in C?

Primarily, compatibility with other implementations (e.g. PyPy uses just the Python implementation), PEP 399 has more detailed information.

Restore deterministic ordinal boundary tests and revert the unrelated
Hypothesis warning workaround. Use Cassio Neri's suggested ternaries for
the January/February adjustment; MSVC emits conditional moves in the
standalone function, with no consistent API speedup in local measurements.

Copy link
Copy Markdown
Author

Applied the suggested ternary change and tested it on Windows 11, AMD Ryzen 5 3500U,
MSVC 19.44.35230 x64, Windows SDK 10.0.26100.0.

Your code-generation observation reproduces: in a standalone /O2 build of
the extracted function, the if/else version has a jbe, while the ternary
version uses two cmovbe instructions. Both compiled functions match an
independent Gregorian day counter for every ordinal from 1 to MAXORDINAL + 2.

I also built commit 8ea445bffa8 with and without exactly this change, using
x64 Release builds with LTCG, without PGO. Results below are microseconds per
1,000 operations:

Operation Before ternaries With ternaries Time change Paired speedup, 95% CI
date.fromordinal, random dates 142.9 146.7 +2.7% 0.932–1.021
date + timedelta(days=40) 160.3 161.1 +0.5% 0.955–1.039
date.fromisocalendar 302.9 299.9 -1.0% 0.976–1.042
datetime + timedelta(days=40) 169.0 169.6 +0.3% 0.956–1.035
date.fromordinal, consecutive dates 140.1 138.5 -1.1% 0.962–1.060
date + timedelta(days=1), 1000 steps 137.5 135.9 -1.2% 0.970–1.053

These reuse the six benchmark loops and inputs from my earlier script.
I used a stdlib runner, not pyperf: 12 paired process runs, alternating A/B
order, one logical CPU, three warmups and five samples per benchmark per
process. The intervals are paired bootstrap 95% intervals for the ratio of
baseline time to time with the change. All include 1; process-mean variation was about 3.5–6.8%.
I therefore cannot claim an API-level speedup or regression from this run.
The standalone assembly observation should not be read as an inspection of
inlined code in the final LTCG DLL.

Both builds passed test_datetime, test_calendar, test_strptime, and
test_time (1,453 tests, 135 skips each), matched the old conversion for all
3,652,059 supported dates, and passed the checks for exact out-of-range errors,
subclass constructor arguments, and ISO upper bounds. Hypothesis was absent;
the explicit examples ran. I have not tested PGO or another MSVC version.

After restoring the deterministic tests and reverting the Hypothesis helper
change, I rebuilt the final working tree and ran test_datetime,
test_calendar, test_strptime, test_time, test_zoneinfo, test_plistlib,
test_xmlrpc, and test___all__: SUCCESS (run=1,615 skipped=151). The all-dates
comparison and exact API checks also passed again. This build omits optional
extensions including SSL and ctypes; the skips are recorded in the test log.
The C source is identical to the ternary version measured above.

Comment thread Modules/_datetimemodule.c Resolved
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting merge type-feature A feature request or enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants