Skip to content

ast: enforce integer string limits in AST-only parsing - #8454

Merged
youknowone merged 1 commit into
RustPython:mainfrom
widehyo1:fix-ast-int-max-str-digits
Aug 4, 2026
Merged

ast: enforce integer string limits in AST-only parsing#8454
youknowone merged 1 commit into
RustPython:mainfrom
widehyo1:fix-ast-int-max-str-digits

Conversation

@widehyo1

@widehyo1 widehyo1 commented Aug 4, 2026 •
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown
Contributor

Summary

Enforce int_max_str_digits for decimal integer literals parsed through
AST-only paths, including PyCF_ONLY_AST, ast.parse(), and
ast.literal_eval().

Details

Move the decimal-integer limit check from a manual source scan to the parser
token stream. Reuse it for normal compilation and _ast parsing, which keeps
the behavior consistent without duplicating lexer logic. The token-based check
excludes strings, comments, floats, complex literals, and non-decimal integer
literals.

The existing test_literal_eval_str_int_limit test now passes, so remove its
expectedFailure marker.

Testing

bash temp/ast/before-commit.sh completed successfully:

  • uv tool run prek run --all-files
  • cargo fmt --check
  • cargo run --release Lib/test/test_ast/test_ast.py
    • 227 tests run
    • 7 skipped
    • 18 expected failures
  • cargo run --release Lib/test/test_compile.py
    • 185 tests run
    • 50 skipped
  • cargo run --release -- -m test test_compile
  • cargo run --release -- -m test test_ast
  • cargo run -- extra_tests/snippets/stdlib_ast.py
  • cargo run -- extra_tests/snippets/builtin_compile.py
  • cargo clippy -p rustpython-compiler -p rustpython-vm -p rustpython-stdlib --all-targets
  • the configured workspace test suite
Workspace test command

rustpython-capi is tested separately in CI and is excluded from this local
workspace command.
rustpython-compiler-source is deprecated and is excluded from CI workspace
builds as well.

cargo test --workspace \
    --exclude rustpython-capi \
    --exclude rustpython_wasm \
    --exclude rustpython-compiler-source \
    --exclude rustpython-venvlauncher \
    --features threading \
    -- --quiet

AI assistance: Codex:gpt-5.6-sol

Summary by CodeRabbit

  • Bug Fixes
    • Decimal integer literals exceeding the configured digit limit are now rejected consistently during compilation and AST parsing.
    • Error reporting now correctly excludes non-decimal integer bases when validating integer length.

@widehyo1
ast: enforce integer string limits in AST-only parsing
404050e 
Validate decimal integer literals from the parser token stream before
lowering the parsed AST. Reuse the validator for normal compilation and
the `_ast` parsing path so `PyCF_ONLY_AST`, `ast.parse()`, and
`ast.literal_eval()` honor `int_max_str_digits` consistently.

Remove the now-passing `test_literal_eval_str_int_limit` expected-failure
marker. The token-based validator excludes non-decimal integers, floats,
complex literals, strings, and comments without duplicating lexer logic.

Assisted-by: Codex:gpt-5.6-sol
@coderabbitai

coderabbitai Bot commented Aug 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro Plus

Run ID: cd6b7af8-7ec1-4160-91d9-71e96e705ce2

📥 Commits

Reviewing files that changed from the base of the PR and between 184db11 and 404050e.

⛔ Files ignored due to path filters (1)
  • Lib/test/test_ast/test_ast.py is excluded by !Lib/**
📒 Files selected for processing (2)
  • crates/compiler/src/lib.rs
  • crates/vm/src/stdlib/_ast.rs
📝 Walkthrough

Walkthrough

The decimal integer literal length check moves from manual source-text scanning to token-based validation in the compiler crate, returning a CompileError and using parser tokens excluding non-decimal bases. Tokens is re-exported publicly. Compilation and symbol-table paths pass tokens to the validator. The VM's _ast.rs parse function enforces the same digit limit before AST conversion.

Changes

Decimal Integer Digit Limit Enforcement

Layer / File(s) Summary
Token type export
crates/compiler/src/lib.rs		 Tokens is re-exported publicly alongside TokenKind.
Token-based decimal digit validator
crates/compiler/src/lib.rs		 The digit limit check is rewritten to iterate tokens, skip hexadecimal, octal, and binary literals, count decimal digits, and return a CompileError when the configured limit is exceeded.
Post-parse validation wiring
crates/compiler/src/lib.rs		 post_parse_source_error accepts tokens and calls the validator. The main compile path, module symbol-table path, and expression symbol-table path each pass their parsed tokens into it.
VM AST parse enforcement
crates/vm/src/stdlib/_ast.rs		 The parse function checks source against int_max_str_digits and returns the resulting error before converting the parsed syntax into an AST.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Suggested reviewers: shaharnaveh, youknowone

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes enforcement of integer string limits in AST-only parsing.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

[x] lib: cpython/Lib/ast.py
[x] lib: cpython/Lib/_ast_unparse.py
[x] test: cpython/Lib/test/test_unparse.py
[x] test: cpython/Lib/test/test_type_comments.py

dependencies:

  • ast

dependent tests: (149 tests)

  • ast: test_ast test_builtin test_compile test_compiler_codegen test_dis test_fstring test_future_stmt test_peepholer test_peg_generator test_site test_ssl test_type_comments test_ucn test_unparse
    • annotationlib: test_annotationlib test_functools test_grammar test_inspect test_reprlib test_type_annotations test_type_params test_typing
      • dataclasses: test__colorize test_copy test_ctypes test_enum test_genericalias test_patma test_pprint test_pydoc test_regrtest test_zoneinfo
      • inspect: test_abc test_argparse test_asyncgen test_buffer test_clinic test_code test_collections test_coroutines test_decimal test_generators test_monitoring test_ntpath test_operator test_posixpath test_signal test_sqlite3 test_traceback test_turtle test_types test_unittest test_yield_from test_zipimport test_zipimport_support
    • dbm.dumb: test_dbm_dumb
    • inspect:
      • bdb: test_bdb test_pdb
      • cmd: test_cmd
      • importlib.metadata: test_importlib
      • pkgutil: test_pkgutil test_pyrepl test_runpy
      • rlcompleter: test_pyrepl test_rlcompleter
      • trace: test_trace
      • xmlrpc.server: test_docxmlrpc test_xmlrpc
    • pyclbr: test_pyclbr
    • traceback: test_asyncio test_code_module test_contextlib test_contextlib_async test_dictcomps test_exceptions test_http_cookiejar test_importlib test_iter test_listcomps test_pyexpat test_setcomps test_socket test_subprocess test_sys test_threadedtempfile test_threading test_unittest test_with
      • concurrent.futures.process: test_compileall test_concurrent_futures
      • http.cookiejar: test_urllib2
      • logging: test_asyncio test_hashlib test_logging test_support test_urllib2net
      • multiprocessing: test_asyncio test_concurrent_futures test_fcntl test_memoryview test_multiprocessing_main_handling test_re
      • py_compile: test_cmd_line_script test_importlib test_modulefinder test_py_compile
      • socketserver: test_imaplib test_socketserver test_wsgiref
      • threading: test_android test_asyncio test_bytes test_bz2 test_concurrent_futures test_context test_ctypes test_email test_external_inspection test_fork1 test_frame test_ftplib test_gc test_httplib test_httpservers test_importlib test_io test_ioctl test_itertools test_largefile test_linecache test_opcache test_pathlib test_poll test_poplib test_pyrepl test_queue test_robotparser test_sched test_smtplib test_super test_syslog test_termios test_threading_local test_time test_urllib2_localnet test_weakref test_winreg test_zstd
      • timeit: test_timeit

Legend:

  • [+] path exists in CPython
  • [x] up-to-date, [ ] outdated

ShaharNaveh
ShaharNaveh approved these changes Aug 4, 2026
View reviewed changes

@ShaharNaveh ShaharNaveh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome!

tysm:)

youknowone
youknowone approved these changes Aug 4, 2026
View reviewed changes

@youknowone youknowone left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, thank you!

@youknowone
youknowone merged commit 1fa76a3 into RustPython:main Aug 4, 2026
28 checks passed
@widehyo1
widehyo1 deleted the fix-ast-int-max-str-digits branch August 7, 2026 09:49
youknowone pushed a commit that referenced this pull request Sep 16, 2026
@widehyo1
ast: enforce integer string limits in AST-only parsing (#8454)
3b9c64b 
Validate decimal integer literals from the parser token stream before
lowering the parsed AST. Reuse the validator for normal compilation and
the `_ast` parsing path so `PyCF_ONLY_AST`, `ast.parse()`, and
`ast.literal_eval()` honor `int_max_str_digits` consistently.

Remove the now-passing `test_literal_eval_str_int_limit` expected-failure
marker. The token-based validator excludes non-decimal integers, floats,
complex literals, strings, and comments without duplicating lexer logic.

Assisted-by: Codex:gpt-5.6-sol
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@youknowone youknowone youknowone approved these changes
@ShaharNaveh ShaharNaveh ShaharNaveh approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@widehyo1 @youknowone @ShaharNaveh