[mazalovmx]

gh-161746

Verification

• Same results as main for all 3,652,059 ordinals (0001-01-01 … 9999-12-31):
  SHA-256 over every date.fromordinal() result, plus fromisocalendar() and
  date + timedelta, for both the C and the pure-Python implementations.
• C: same results for every positive int ordinal (1 … 2**31-1), including
  the error messages past 9999-12-31; checked with gcc 13 and clang.
• Pure Python: same results for 2.3 million other integers, including negative
  and huge ones.
• test_datetime, test_calendar, test_strptime, test_time, test_zoneinfo,
  test_plistlib and test_xmlrpc pass. The PR adds a Hypothesis property test
  (fromordinal(d.toordinal()) == d, with explicit examples at the ends of
  the 4-, 100- and 400-year cycles) and tests for the year reported in the
  error past 9999-12-31.
• Lib/test/support/hypothesis_helper.py now silences the typing.AnyStr
DeprecationWarning that Hypothesis emits on import; otherwise
  test___all__ fails when it imports test.datetimetester.

Benchmarks

Current version (b3f74a0; the later commit only changes the test helper).
pyperf, bench_ordinal.py, main (e891a91) vs. the patch; µs (C) or ms
(pure Python) per 1000 operations, mean of the pyperf runs:

Benchmark	C main	C patched	C speedup	Py main	Py patched	Py speedup
date.fromordinal, random dates	75.5	65.2	1.16x	2.64	2.01	1.31x
date.fromordinal, consecutive dates	63.2	—	not significant	2.55	1.94	1.32x
date + timedelta(days=40)	95.3	82.0	1.16x	4.61	3.89	1.19x
datetime + timedelta(days=40)	90.5	81.2	1.11x	22.0	—	not significant
date.fromisocalendar	170	146	1.16x	4.21	3.57	1.18x
date + timedelta(days=1), 1000 steps	75.9	78.2	1.03x slower	5.17	3.61	1.43x
Geometric mean			1.09x			1.24x

In C, date + timedelta(days=1) does not call ord_to_ymd()
(normalize_y_m_d() handles the step to the next month directly), so no gain
is expected there. pyperf reports the 3% difference as significant, but it is
smaller than the standard deviation of the runs (about 5%).

Earlier measurements (first version of the patch, 64-bit form; superseded)

pyperf, x86-64 VM, gcc 13.3, -O3, no PGO; C implementation, µs per 1000 operations:

Benchmark	main	patched	Speedup
date.fromordinal, random dates	35.6	25.3	1.40x
date + timedelta(days=40)	42.9	35.7	1.20x
date.fromisocalendar	83.5	71.0	1.18x
datetime + timedelta(days=40)	37.9	34.1	1.11x
date.fromordinal, consecutive dates	26.3	23.8	1.10x
date + timedelta(days=1), 1000 steps	32.8	31.8	1.03x

With a PGO build (--enable-optimizations, gcc 13.3) of main (e3262d9) and of
that version, 1,000,000 operations, best of 7, builds alternated:

Operation	main	patched	Speedup
date.fromordinal, random dates	62.6 ms	45.9 ms	1.36x
date + timedelta(days=90)	69.0 ms	60.8 ms	1.13x
date.fromisoformat('2024-W07-1')	88.6 ms	77.8 ms	1.14x
date + timedelta(days=10)	53.0 ms	52.5 ms	1.01x

pyperformance 1.14.0 (--fast, same two PGO builds, 119 benchmarks in common):
geometric mean 1.00x. No pyperformance benchmark exercises this code path, so
no change is expected there.

🤖 Generated with Claude Code

[python-cla-bot]

All commit authors signed the Contributor License Agreement.

[StanFromIreland]

I don't like these magic numbers, can you please benchmark the simpler approach (compilers are pretty smart, and should be able to deal with these), like you implemented in Python? If we accept this (I still have to think about this), they should be as close as possible for maintenance purposes.

[mazalovmx]

Ok. I rewrote it to follow _ord2ymd() step by step, with plain / and % by constants. gcc 14 and clang 19 turn them into multiplications anyway (no div instructions at -O3).

Results, C implementation, µs per 1000 operations (pyperf, gcc 14.2, -O3, no PGO):

Benchmark	main	magic numbers	simple
date.fromordinal, random dates	101	60.8 (1.65x)	62.2 (1.62x)
date + timedelta(days=40)	104	82.0 (1.27x)	85.2 (1.22x)
date.fromisocalendar	184	146 (1.26x)	156 (1.18x)
datetime + timedelta(days=40)	102	78.7 (1.29x)	83.2 (1.22x)
date.fromordinal, consecutive dates	65.9	61.9 (1.06x)	59.8 (1.10x)
Geometric mean (6 benchmarks)		1.25x	1.21x

Simple vs. magic numbers: 4 of 6 benchmarks not significant, geometric mean 1.03x slower. On its own, the function takes about 6.5 ns instead of 5.5 ns (the old code: about 17 ns). That isn't worth the magic numbers, so I've pushed the simple version.

It still matches the old code for every positive int ordinal (gcc and clang) and for all dates in the supported range.

[StanFromIreland]

Add a link, and a rough % speedup.

[mazalovmx]

Done.

[StanFromIreland]

CC @cassioneri this may be of interest.

[pganssle]

This might actually be a good opportunity to use a property test for the property that theclass.fromordinal(d.toordinal()) == d (handwavy)

[mazalovmx]

Done. I replaced my loop with a property test: fromordinal(d.toordinal()) == d for strategies.dates(). The ends of the 4-, 100- and 400-year cycles, and min/max date, are added as @example, so they run also without Hypothesis. The date is created again with self.theclass, so the test checks the class under test.

[pganssle]

I think we can drop the 64-bit arithmetic here entirely. As far as I can tell, the only reason for it is so that date.fromordinal() with an ordinal past MAXORDINAL still gets to the same "year must be in 1..9999, not N" error that it raises on main, because fromordinal only checks ordinal < 1 before calling ord_to_ymd. The other two callers never pass anything larger than MAXORDINAL + 2: normalize_date already range-checks the ordinal, and iso_to_ymd can reach at most 10000-01-02, the last day of ISO year 9999, which its caller then rejects.

Since the calendar repeats exactly every 400 years, fromordinal can handle the error case by shifting the ordinal into the first 400-year cycle and adding the years back afterwards, which keeps the error message identical without needing a 64-bit path in the hot function:

    if (ordinal < 1) {
        PyErr_SetString(PyExc_ValueError, "ordinal must be >= 1");
        return NULL;
    }
    if (ordinal <= MAXORDINAL) {
        ord_to_ymd(ordinal, &year, &month, &day);
    }
    else {
        /* Out of range: find the year anyway, so new_date_subclass_ex()
         * raises the usual "year must be in 1..9999, not N" error.  The
         * calendar repeats every 400 years (DI400Y days), so convert the
         * equivalent date in the first 400-year cycle and add the years
         * back. */
        int cycles = (ordinal - 1) / DI400Y;
        ord_to_ymd(ordinal - cycles * DI400Y, &year, &month, &day);
        year += 400 * cycles;
    }
    return new_date_subclass_ex(year, month, day, type);

With that change, every intermediate value in ord_to_ymd fits in a uint32_t — 4 * days + 3 tops out around 14.6 million — so the function can assert ordinal <= MAXORDINAL + 2 and the comment explaining why the century step is done in 64 bits can go away. I checked this version against the ord_to_ymd on main for every positive int, and the year, month and day all match.

I did not see a measurable speed difference on x86-64 with either gcc or clang, so this is mostly an argument for simplicity. It may help somewhat on 32-bit platforms, where the 64-bit division can turn into a library call, though I have not measured that. The pure Python version does not need an equivalent change, since it uses arbitrary-precision integers and floor division.

[mazalovmx]

Done, I used your version. ord_to_ymd() is now uint32_t only, with assert(ordinal <= MAXORDINAL + 2), and fromordinal() handles bigger ordinals with the 400-year shift. I checked it against main for all positive int, and the error message is the same. I also added a test that the error still has the correct year for some ordinals after MAXORDINAL.

I checked the changes twice.

[StanFromIreland]

I'm not really a fan of this, we're working around a Python 3.16 incompatibility in Hypothesis, and this is unrelated to the change here. The property test also doesn't cover much over the examples, also most Buildbots don't have Hypothesis installed so only the @examples run IIRC. I'd prefer reverting to the previous version of the test and dropping the Hypothesis changes. CC @pganssle

[StanFromIreland]

I pushed a little commit to tidy up the comments.

[cassioneri]

CC @cassioneri this may be of interest.

Hi all.

Lorenz and I wish to thank you for your interest in our work. We hope this goes well and our algorithms make their way into cpython.

My two cents follow.

1. Kudos to @mazalovmx who did a great job. 👏🏻

2. I'm sad that @StanFromIreland doesn't like the optimised versions (the one with magical numbers), but fair enough: he got a point and I won't argue about that. I just wish to mention that, as he said, "compilers are pretty smart" and apply an interesting technique to optimise divisions by constants. As a by product, % is optimised as well. However, our work goes further and apply special optimisations for % that no compiler implements. Can I kindly recommend you to watch my talk? It explains this and other relevant points: https://www.youtube.com/watch?v=J9KijLyP-yg (You might enjoy it 😃.)

3. I looked at the assembly code generated by gcc 16.2, clang 23.1.0 and msvc latest (https://godbolt.org/z/fbnvn1nd6).

For gcc and clang, the assembly looks good to me. Unfortunately, msvc adds a branch for this part of the code:

    if (m > 12) {  /* January or February */
        *year = (int)(y + 1);
        *month = (int)(m - 12);
    }
    else {
        *year = (int)y;
        *month = (int)m;
    }

The above is equivalent to:

    const uint32_t jan_or_feb = m > 12;
    *year = (int) (jan_or_feb ? y + 1 : y);
    *month = (int) (jan_or_feb ? m - 12 : m);

(see https://godbolt.org/z/PGe83cPPx). This change has no impact whatsoever on gcc. For clang the assembly slightly changes but nothing that catch my attention. Now, for msvc the branch disappears and this is a good thing. Therefore, I do recommend this change. (Please rerun benchmarks.)

Finally, I have a question, out of curiosity: Why is there an implementation in Python and another one in C?

I hope this helps and, again, I thank you all for considering using our algorithms in cpython.

[StanFromIreland]

Hello @cassioneri!

I just wish to mention that, as he said, "compilers are pretty smart" and apply an interesting technique to optimise divisions by constants. As a by product, % is optimised as well. However, our work goes further and apply special optimisations for % that no compiler implements. Can I kindly recommend you to watch my talk? It explains this and other relevant points: https://www.youtube.com/watch?v=J9KijLyP-yg (You might enjoy it 😃.)

Hmm, I'll think about this a bit more, but I'm still in favour of the current, clearer, logic.

I hadn't seen the talk previously, nice!

Now, for msvc the branch disappears and this is a good thing.

These are micro-optimisations, I haven't benchmarked but I presume it will have little impact in practice. @mazalovmx are you able to test this, I'm afraid I don't have a Windows machine.

Finally, I have a question, out of curiosity: Why is there an implementation in Python and another one in C?

Primarily, compatibility with other implementations (e.g. PyPy uses just the Python implementation), PEP 399 has more detailed information.