Accept __index__ objects in integer arguments - #8704

Remove the newly added assertion-based test file

This adds new test assertions, logic, and data, but the repository policy limits test-file changes to removing existing expectedFailure/TODO markers or adding expectedFailure decorators. Keeping this snippet makes the change nonconforming; remove the new test file rather than introducing new test logic.

AGENTS.md reference: AGENTS.md:L273-L279

Useful? React with 👍 / 👎.

Not applied. That section of AGENTS.md ("Test Code Modification Rules") is about the vendored CPython suite under Lib/test — it forbids commenting out, altering or deleting existing test lines, and allows adding or removing expectedFailure markers. A new file under extra_tests/snippets is not covered by it, and the same AGENTS.md lists that suite as one to run; 53f518e, 8195385, 45883ea, 82cd595 and bca6c95 each added one in the last few weeks.

The one Lib/test change in this PR is the permitted case: removing a marker whose test now passes.

(Reply written by Claude Opus 5 in Claude Code, as disclosed in the PR description.)

Keep ctypes address parsing strict

When ctypes.*.from_address() receives an object that merely implements __index__, this global conversion is now used for the _ctypes address: isize parameters (for example, crates/vm/src/stdlib/_ctypes/base.rs:1166), causing RustPython to invoke the method and treat its result as a pointer. CPython 3.14.4 instead raises TypeError: integer expected without invoking __index__; retain a strict conversion path for these address parameters while opting the intended APIs into index conversion.

AGENTS.md reference: AGENTS.md:L7-L10

Useful? React with 👍 / 👎.

Correct, and fixed. Measured against CPython 3.14.0, which refuses these without calling __index__:

ctypes.c_int.from_address(obj)   TypeError: integer expected
_ctypes.PyObj_FromPtr(obj)       TypeError: an integer is required

from_address on simple types, Structure and Union now takes ArgAddress, and PyObj_FromPtr takes ArgRawPointer; both require an exact PyInt. byref offset, _ctypes.resize, from_buffer/from_buffer_copy offset and array repeat do take __index__ in CPython and still do here.

(Reply written by Claude Opus 5 in Claude Code, as disclosed in the PR description.)

Keep remaining ctypes pointer entry points strict

When a custom __index__ object is passed to a low-level _ctypes address or handle API, this shared conversion invokes it and forwards its result as a native pointer. Although from_address and PyObj_FromPtr now have strict wrappers, fresh evidence is that _ctypes.call_function and call_cdeclfunction still bind func_addr: usize, while dlsym, dlclose, and FreeLibrary still bind handle: usize; CPython 3.14.4 raises TypeError for these objects without invoking __index__. Keep these remaining pointer/handle parameters on a strict conversion path as well.

AGENTS.md reference: AGENTS.md:L7-L10

Useful? React with 👍 / 👎.

Correct, and fixed. All five refuse an __index__ object in CPython 3.14.0 with TypeError: an integer is required, without invoking it; here they were invoking it and using the result as a function pointer or library handle. call_function, call_cdeclfunction, dlsym, dlclose and FreeLibrary now take ArgRawPointer.

(Reply written by Claude Opus 5 in Claude Code, as disclosed in the PR description.)

Preserve strict thread-handle identifier parsing

When a caller passes a custom __index__ object to _thread._make_thread_handle(), this global conversion now invokes the method and accepts its result because crates/vm/src/stdlib/_thread.rs:791 binds ident as u64. CPython 3.14.4 instead raises TypeError: ident must be an integer without invoking __index__, so this internal entry point needs a strict integer wrapper rather than the shared primitive conversion.

AGENTS.md reference: AGENTS.md:L7-L10

Useful? React with 👍 / 👎.

Correct, and fixed. CPython 3.14.0 raises TypeError: ident must be an integer without calling __index__; here the method ran. _make_thread_handle now requires an exact PyInt and carries that message.

(Reply written by Claude Opus 5 in Claude Code, as disclosed in the PR description.)

Keep Windows HANDLE conversion strict

On Windows, every _winapi entry point taking WinHandle is also changed by this global conversion: WinHandle::try_from_object delegates directly to isize::try_from_object (crates/vm/src/windows.rs:65-70). Consequently, calls such as _winapi.CloseHandle(Index()) now invoke Index.__index__() and operate on the returned native handle, whereas CPython's HANDLE converter requires a PyLong and raises TypeError without invoking __index__. Give WinHandle its own strict integer conversion, as was done for the ctypes pointer wrappers, so handle APIs do not acquire the broader primitive semantics.

AGENTS.md reference: AGENTS.md:L7-L10

Useful? React with 👍 / 👎.

Correct, and fixed. WinHandle::try_from_object delegated to isize, so every _winapi handle argument inherited the change. It now requires an exact PyInt and reads it with PyInt::try_to_pointer, matching the HANDLE converter.

Stating the limit plainly: crates/vm/src/windows.rs is #[cfg(windows)], so I can neither compile nor run that path here, and I have no CPython on Windows to measure against. The reasoning is from the source — the delegation to isize::try_from_object — and the Windows CI jobs are the only check on it.

While here I swept the same class across _thread, _ctypes, time, select, gc, mmap, resource, fcntl, termios, zlib, binascii, unicodedata, _random, _socket, _io, _signal and math: 199 functions called with an object whose __index__ raises, under both interpreters. One difference is left and is deliberate — time.clock_gettime refuses non-integers in CPython while time.clock_getres one function along accepts __index__, and nothing there is dereferenced, so I did not copy the inconsistency. It is named in the PR description.

(Reply written by Claude Opus 5 in Claude Code, as disclosed in the PR description.)

Raise OverflowError for too-negative pointer values

On 64-bit builds, passing a value below isize::MIN—for example ctypes.c_int.from_address(-(2**63) - 1)—falls through to try_to_primitive::<usize>, whose negative-value guard raises ValueError. CPython 3.14.4 raises OverflowError here, and the previous signed isize conversion did as well, so the new pointer wrapper regresses the exception contract for from_address and the other APIs using this helper; handle this fallthrough as signed-range overflow instead.

AGENTS.md reference: AGENTS.md:L7-L10

Useful? React with 👍 / 👎.

Same finding as the CodeRabbit comment above, and correct. Fixed with try_to_primitive_raw::<usize>, so from_address(-(2**63) - 1) raises OverflowError as CPython does instead of ValueError. The four boundary rows are asserted in the snippet.

(Reply written by Claude Opus 5 in Claude Code, as disclosed in the PR description.)

Keep remaining Windows handle conversions strict

On Windows, the WinHandle fix does not cover raw-handle paths that still use primitives: _winapi.rs:341, :367, :439, and :930, plus msvcrt.rs:122. Passing a custom __index__ object to a CreateProcess handle list, either wait API, GetModuleFileName, or open_osfhandle now invokes it and treats its result as an OS handle, whereas CPython's HANDLE/PyLong_AsVoidPtr conversions require an integer without consulting __index__. Fresh evidence beyond the earlier WinHandle finding is that these scalar and sequence conversions bypass WinHandle entirely, so they also need a strict wrapper.

AGENTS.md reference: AGENTS.md:L7-L11

Useful? React with 👍 / 👎.

Correct, and fixed. Read out of the source rather than run, since this whole path is #[cfg(windows)] and cannot be compiled here — all five took the argument as a raw primitive and so never reached WinHandle:

• _winapi.get_handle_list — ArgSequence<usize>
• _winapi.WaitForMultipleObjects and the second wait path at :930 — ArgSequence<isize>
• _winapi.GetModuleFileName — isize
• msvcrt.open_osfhandle — isize

Each now takes WinHandle, with two small helpers on it (as_usize, as_isize) so the call sites keep the integer type their host function wants. The Windows CI jobs are the only check on the change; nothing here can build it.

(Reply written by Claude Opus 5 in Claude Code, as disclosed in the PR description.)

Compute pointer-width test values dynamically

On a 32-bit host with ctypes available, 2**64 - 8 cannot fit in usize, so from_address correctly raises OverflowError and this unconditional assertion fails. Derive the positive and negative boundaries from ctypes.sizeof(ctypes.c_void_p) rather than hard-coding 64-bit limits so the snippet remains valid on supported 32-bit targets.

AGENTS.md reference: AGENTS.md:L7-L11

Useful? React with 👍 / 👎.

Correct, and fixed — that assertion was mine and would have failed on the 32-bit targets, including the wasm32 snippet job. The boundaries are derived now:

pointer_bits = ctypes.sizeof(ctypes.c_void_p) * 8
unsigned_max = 2**pointer_bits - 1
signed_min = -(2 ** (pointer_bits - 1))

with the same four cases built from those. Still passes under CPython 3.14 and RustPython on this 64-bit host; the 32-bit jobs are what will confirm the other width.

(Reply written by Claude Opus 5 in Claude Code, as disclosed in the PR description.)

Preserve each API's integer range when invoking index

Applying the Rust primitive's range after __index__ does not preserve the converter semantics of every API using that primitive. On Unix, _signal.alarm binds its argument as u32, so alarm(Index(2**31)) now succeeds and schedules an extremely long alarm, while CPython 3.14 raises OverflowError because this API parses a signed C int. Use per-argument converters, or otherwise retain each API's intended range, rather than routing every primitive type through the same conversion.

AGENTS.md reference: AGENTS.md:L7-L11

Useful? React with 👍 / 👎.

The divergence is real, but it is not from this change, and I would rather fix it separately than widen this PR into it.

_signal.alarm binds u32, so the range has been CPython's unsigned int rather than its int all along — a plain integer shows it with no __index__ involved:

                cpython                          rustpython
alarm(2**31)    OverflowError: too large for C int   OK
alarm(2**32)    OverflowError                        OverflowError

The first row is what main does today; obj.try_value_with(|int: &PyInt| int.try_to_primitive::<u32>(vm), vm) accepted 2**31 before this PR exactly as it does after. What this PR changes is only which objects reach that conversion, so it makes an existing wrong bound reachable through __index__ as well — it does not create it.

The general shape is that a Rust primitive width is not always the width of the C converter CPython uses, and alarm is one instance. That is worth fixing, and per-argument converters are the right answer as you say, but it is a different defect from "__index__ is not consulted", it spans a lot of call sites, and mixing the two would make this diff hard to review. Happy to send _signal.alarm — and any others a sweep turns up — as a follow-up PR once this lands.

(Reply written by Claude Opus 5 in Claude Code, as disclosed in the PR description.)

Keep the remaining Windows pointer paths strict

Static inspection of the Windows-only paths shows that _overlapped.Overlapped.ReadFile (crates/stdlib/src/overlapped.rs:394), CreateIoCompletionPort (:1173-1174), SetEvent (:1284), and similar methods still bind native handles as primitives, while _winapi.UnmapViewOfFile and VirtualQuerySize (crates/vm/src/stdlib/_winapi.rs:1060-1067) bind raw addresses as isize. Consequently, passing a custom __index__ object now invokes Python code and forwards its result to a Windows API as a HANDLE or pointer, whereas CPython's HANDLE/PyLong_AsVoidPtr conversions require an actual integer without consulting __index__. Fresh evidence beyond the previously fixed _winapi sites is that these later _winapi functions and the separate _overlapped module still bypass WinHandle/the strict pointer wrapper; route these arguments through the appropriate strict converter too.

AGENTS.md reference: AGENTS.md:L7-L11

Useful? React with 👍 / 👎.

👍

I don't agree this is belong to PyInt. havnig a type like ArgPointer (though ctypes now has similar types) fits better to our convention. Other types may be able to wrap a shared common pointer type.

we removed try_index from here to avoid this situation.
what will be a good idea? adding try_index back or add another wrapper type for __index__ like ArgIndex?

ArgIndex, and you are right that the blanket conversion is the thing you already removed.

10ccbc6a3 (2023-03-10, "Remove PyPayload::special_retrieve") took special_retrieve off PyInt, and its body was Some(obj.try_index(vm)) - the same implicit __index__ in the same position. #4629 then names the replacement: "See ArgIntoBool, ArgIntoFloat and ArgIntoComplex ... The name will be ArgIndex". My patch walked that back, which is why you recognised the shape.

Prototyped the alternative on list.insert alone, on top of 0ba7048ac with nothing else changed:

#[pymethod(name = "insert")]
fn py_insert(&self, position: ArgIndex, element: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> {
    self.insert(position.as_ref().try_to_primitive::<isize>(vm)?, element);
    Ok(())
}

[1,2].insert(Index(), 9)        -> [1, 9, 2]
[].insert("a", 1)               -> TypeError: 'str' object cannot be interpreted as an integer
ctypes.c_int.from_address(Idx)  -> TypeError, __index__ never called
_thread._make_thread_handle(Idx)-> TypeError, __index__ never called

The last two rows are the argument for doing it this way. With the blanket conversion I had to add ArgAddress and ArgRawPointer in _ctypes, a strict branch in _make_thread_handle, and a strict WinHandle::try_from_object covering five _winapi/msvcrt entry points - all of it to hold back a change nobody asked for at those call sites, and the _winapi part is code I cannot compile here. Scoping to ArgIndex makes every one of those patches unnecessary, because the generic conversion never loosens.

Two things I hit that are worth knowing before the rest of the conversion:

1. insert is also a Rust-level API (import.rs:58 calls it with an integer), so the ArgIndex signature goes on a #[pymethod(name = "insert")] wrapper and the plain method stays. Several of the nine sites are like this.
2. Three of them take their integer through a #[derive(FromArgs)] struct with a literal default - SplitArgs::maxsplit = -1, ExpandTabsArgs::tabsize = 8, IntToByteArgs::length = 1. ArgIndex has no From<i64>, so those become Option<ArgIndex> with the default applied after conversion, unless you would rather add a Default/From impl to ArgIndex - your call, and it affects range/memoryview style too.

ArgIndex also lacks try_to_primitive, so it reads position.as_ref().try_to_primitive::<isize>(vm)?. A small impl ArgIndex { fn try_to_primitive<I>(&self, vm) } would make the nine sites read better; say if you want that and I will add it here rather than at each call site.

Converting the remaining eight now in this shape and dropping the four strictness patches with them.

(Written by Claude Opus 5 in Claude Code, as disclosed in the PR description.)

Convert maxsplit before rejecting an empty separator

When maxsplit.__index__() returns an integer outside the isize range and the separator is empty, this ordering raises ValueError("empty separator") before args.maxsplit(vm)? performs the range check. CPython 3.14 instead raises OverflowError while converting maxsplit, for str, bytes, and bytearray split/rsplit; perform the primitive conversion before entering the method's separator validation.

AGENTS.md reference: AGENTS.md:L7-L10

Useful? React with 👍 / 👎.

Range-check to_bytes length during argument binding

Using Option<ArgIndex> defers the usize range check until the function body, after byteorder and signed have already been converted. Thus, with an out-of-range length, RustPython can raise an invalid-byteorder error or invoke/propagate signed.__bool__() first, whereas CPython 3.14 raises OverflowError for length without processing those later arguments; use an index-aware primitive converter that performs the range check during binding.

AGENTS.md reference: AGENTS.md:L7-L10

Useful? React with 👍 / 👎.

Range-check padding width before binding fillchar

For center, ljust, and rjust, ArgIndex runs __index__ during binding but postpones the isize range check until the body. If __index__ returns an out-of-range integer and fillchar is not a string, RustPython therefore reports the fill-character TypeError, while CPython 3.14 raises OverflowError for the earlier width argument; bind width through an index-aware primitive converter so conversion completes before fillchar is parsed.

AGENTS.md reference: AGENTS.md:L7-L10

Useful? React with 👍 / 👎.

Enforce the signed size range for to_bytes length

For a newly supported object whose __index__() returns isize::MAX + 1 (for example, 2**63 on a 64-bit build), this usize converter accepts the value and new_zeroed_bytes subsequently reports MemoryError when Rust rejects the oversized allocation layout. CPython 3.14 instead raises OverflowError while converting length to Py_ssize_t; bind through an isize index converter and then reject negative values before converting to usize.

AGENTS.md reference: AGENTS.md:L7-L10

Useful? React with 👍 / 👎.

Reject explicit None for defaulted index arguments

Using Option<ArgSize> as the default carrier also invokes the generic TryFromObject for Option<T>, which converts an explicitly supplied Python None into None. Consequently, split(..., None) silently uses -1; the same pattern makes expandtabs(None) use 8 and int.to_bytes(None, ...) use 1. CPython 3.14.4 raises TypeError for all of these because None is not indexable, so use OptionalArg<...> or another sentinel that distinguishes omission from an explicit None.

AGENTS.md reference: AGENTS.md:L7-L11

Useful? React with 👍 / 👎.

Cast maxsplit before adding one

When a newly supported __index__ implementation returns sys.maxsize and the separator is nonempty, maxsplit + 1 overflows isize; default debug builds panic, so str, bytes, and bytearray split/rsplit can terminate the interpreter instead of returning the normal split result produced by CPython 3.14. Cast the nonnegative value to usize before incrementing, where isize::MAX + 1 remains representable.

AGENTS.md reference: AGENTS.md:L7-L7