Repository navigation
fix(ls): parse arguments through the tokenizer - #3999
Merged
Merged
Conversation
📊 Automated PR Analysis
SummaryRewrites the ls command's argument parsing to use a proper tokenizer/grammar instead of raw prefix scans, fixing cases where dash-prefixed paths (e.g. Review Checklist
Analyzed automatically by wshm · This is an automated analysis, not a human review. |
KuSh
force-pushed
the
feat/ls-arg-tokenizer
branch
2 times, most recently
from
September 20, 2026 23:41
2ae3912 to
7fa0fee
Compare
`rtk ls` classified arguments with raw prefix scans (`starts_with('-')`) and
never restored the `--` boundary, so it had no way to tell a flag from a path
that looks like one, or a flag from its own value.
- `rtk ls -- -al` listed the current directory instead of the file named `-al`:
`-al` was read as the flags `-a`/`-l`, and the path was dropped entirely.
- A flag's value was sorted into the path list, losing its position:
`rtk ls dir -I pattern` ran `ls -l -I dir pattern`, ignoring `dir` and listing
`pattern`.
- `-I -al` read the ignore pattern as flags, turning on all/long rendering.
- `--format long` (separate value) did not imply the long listing that
`--format=long` did.
`plan()` now tokenizes through `tokenize_grammar` with an ls grammar transcribed
from `ls --help`, and emits a `--` before the paths it hands the child. The
grammar resolves GNU long-option abbreviations (`--sor time` sorts by time),
which the new explicit boundary would otherwise break.
Co-Authored-By: Claude Opus 5 <[email protected]>
The option grammar was GNU's alone, and the long-flag table it added was only consulted for flag names. - `-I`/`-T`/`-w` take a value on GNU but are booleans on BSD (FreeBSD/macOS `ls(1)`; only `-D` takes one there). Reading a macOS operand as one of those values moved it ahead of the `--`, and BSD's non-permuting getopt then listed the current directory alongside it and failed on `--` itself: `rtk ls -lT /tmp` ran `ls -l -T /tmp -- .`. - `--all=x` was dropped as if it were a bare `--all`, turning ls's exit 2 into a successful listing of the current directory. - `--human-readable` and `--si` were forwarded where `-h` was dropped, so the child pre-formatted sizes RTK renders itself and `200K` was read as a link count: `rtk ls --human-readable big.bin` printed `1B`. - `--format=WORD` resolves an abbreviated *value* as well as an abbreviated name, so `--format=lon` is a long listing; RTK matched the full words only and dropped the octal permission column. - The `--` was appended unconditionally, so a flag still waiting for its value ate it: `rtk ls -a --indicator-style` reported an invalid argument `--` with exit 1 where ls reports the missing one with exit 2, and `rtk ls sub -I` silently took `--` as the ignore pattern and succeeded. Such a flag can only be the last argument, so no path before it needs the boundary; it now goes last instead. Co-Authored-By: Claude Opus 5 <[email protected]>
KuSh
force-pushed
the
feat/ls-arg-tokenizer
branch
from
October 6, 2026 23:07
7fa0fee to
93166b9
Compare
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
ls.rs::run()classified arguments with raw prefix scans —is_short_flagwasstarts_with('-') && !starts_with("--"), flags were.filter(|a| a.starts_with('-'))andpaths the complement — and never called
restore_double_dash. With no--awareness and nogrammar, it could not tell a flag from a path that looks like one, nor a flag from its own
value.
Reproducer (verified against GNU coreutils 9.10,
LC_ALL=C)-alwas read as the flags-a/-l, so the child ranls -la ..Three more from the same cause:
lsrtk ls dir -I patternls -l -I dir pattern— ignoreddir, listedpatternpattern, listsdirrtk ls -I -al-alis the patternrtk ls --format long--format=longwas recognised)What changed
plan()tokenizes throughtokenize_grammarwith anls_takes_valuegrammar transcribed fromls --helpand verified flag by flag against the real binary:value().claiming_dash_dash()--block-size --format --hide --ignore --indicator-style --quoting-style --sort --tabsize --time --time-style --width, and GNU's-I -T -wls --sort time alpha.txtconsumes the value;ls -I -- -aluses--as the pattern and then parses-alas flagsattached_only()--color --classify --hyperlinkls --color alwayslists a file namedalways;ls --hyperlink zzzreports "cannot access 'zzz'"-Fand the restls -F alwayslistsalways;ls -Falwayserrors on-w ays, so-Ftakes nothing even attachedshow_all/show_long/flags/paths now come fromTokenKindandToken::is_free_positional(),and
restore_double_dashruns first so the--the user typed is visible at all.The child command gains an explicit
--before its path list, which is what fixesrtk ls -- -aland any path that merely starts with a dash. That boundary would have broken GNU long-option
abbreviations (
ls --sor timereally does sort by time), so the grammar resolves them againstthe full option list, leaving ambiguous ones (
--ign) forlsitself to reject as it does today.Short-flag clustering is unchanged (
Dialect::Posix) and covered:-la,-lA,-lh1.The grammar is per-platform, and the value side gets the same treatment
-I,-Tand-wtake a value on GNU but are booleans on BSD —-Iprevents-Afrombeing auto-set for root,
-Tcompletes the time with-l,-wprints non-printables raw, andonly
-Dtakes a value there (FreeBSDls(1),same on macOS). Applying the GNU table on
macOS read an operand as a value and moved it ahead of the
--; BSD's getopt does not permute,so
rtk ls -lT /tmpranls -l -T /tmp -- .and listed the current directory as well as/tmpbefore failing on
--. AFlavorpicked fromtarget_osnow selects the short-option grammar,and the tests pin both flavours on any host.
Four more cases where RTK and real
lsdisagreed, all now matching on output and exit code:ls)rtk ls --all=x--alldropped as if bare; listed., exit 0option '--all' doesn't allow an argument, exit 2rtk ls --human-readable big.bin200Kparsed as a link count →big.bin 1Bbig.bin 200.0K(aliases dropped like-h)rtk ls --format=lon--format=longrtk ls -a --indicator-style--:invalid argument '--', exit 1requires an argument, exit 2A flag left waiting for its value can only be the last argument, so no path before it can have
been
---protected; it now goes last in the child argv instead of eating the boundary.Tests
Behavioural tests on the arg -> child-argv pipeline, each confirmed failing against the old
logic and passing after. The headline one:
55 tests in the module, full suite green, clippy clean,
rtk lsstartup 7.8 ms.