Skip to content

fix(ls): parse arguments through the tokenizer - #3999

Merged
KuSh merged 2 commits into
rtk-ai:developfrom
KuSh:feat/ls-arg-tokenizer
Oct 6, 2026
Merged

KuSh merged 2 commits into
rtk-ai:developfrom
KuSh:feat/ls-arg-tokenizer

Conversation

@KuSh

@KuSh KuSh commented Sep 12, 2026 •

Copy link
Copy Markdown
Collaborator

The bug

ls.rs::run() classified arguments with raw prefix scans — is_short_flag was
starts_with('-') && !starts_with("--"), flags were .filter(|a| a.starts_with('-')) and
paths the complement — and never called restore_double_dash. With no -- awareness and no
grammar, it could not tell a flag from a path that looks like one, nor a flag from its own
value.

Reproducer (verified against GNU coreutils 9.10, LC_ALL=C)

$ touch -- -al
$ ls -- -al          # real ls: lists the file
-al
$ rtk ls -- -al      # before: listed the whole directory, the path was dropped
alpha.txt
always
beta.log
-al

-al was read as the flags -a/-l, so the child ran ls -la ..

Three more from the same cause:

Input Before Real ls
rtk ls dir -I pattern ran ls -l -I dir pattern — ignored dir, listed pattern ignores pattern, lists dir
rtk ls -I -al read the ignore pattern as flags, switching on all/long rendering -al is the pattern
rtk ls --format long no long listing (only --format=long was recognised) long listing

What changed

plan() tokenizes through tokenize_grammar with an ls_takes_value grammar transcribed from
ls --help and verified flag by flag against the real binary:

Spec Flags Verified by
value().claiming_dash_dash() --block-size --format --hide --ignore --indicator-style --quoting-style --sort --tabsize --time --time-style --width, and GNU's -I -T -w ls --sort time alpha.txt consumes the value; ls -I -- -al uses -- as the pattern and then parses -al as flags
attached_only() --color --classify --hyperlink ls --color always lists a file named always; ls --hyperlink zzz reports "cannot access 'zzz'"
no value -F and the rest ls -F always lists always; ls -Falways errors on -w ays, so -F takes nothing even attached

show_all/show_long/flags/paths now come from TokenKind and Token::is_free_positional(),
and restore_double_dash runs first so the -- the user typed is visible at all.

The child command gains an explicit -- before its path list, which is what fixes rtk ls -- -al
and any path that merely starts with a dash. That boundary would have broken GNU long-option
abbreviations (ls --sor time really does sort by time), so the grammar resolves them against
the full option list, leaving ambiguous ones (--ign) for ls itself to reject as it does today.

Short-flag clustering is unchanged (Dialect::Posix) and covered: -la, -lA, -lh1.

The grammar is per-platform, and the value side gets the same treatment

-I, -T and -w take a value on GNU but are booleans on BSD — -I prevents -A from
being auto-set for root, -T completes the time with -l, -w prints non-printables raw, and
only -D takes a value there (FreeBSD ls(1),
same on macOS). Applying the GNU table on
macOS read an operand as a value and moved it ahead of the --; BSD's getopt does not permute,
so rtk ls -lT /tmp ran ls -l -T /tmp -- . and listed the current directory as well as /tmp
before failing on --. A Flavor picked from target_os now selects the short-option grammar,
and the tests pin both flavours on any host.

Four more cases where RTK and real ls disagreed, all now matching on output and exit code:

Input Before Now (= real ls)
rtk ls --all=x --all dropped as if bare; listed ., exit 0 option '--all' doesn't allow an argument, exit 2
rtk ls --human-readable big.bin 200K parsed as a link count → big.bin 1B big.bin 200.0K (aliases dropped like -h)
rtk ls --format=lon abbreviation unresolved, octal column dropped long listing, as --format=long
rtk ls -a --indicator-style the flag ate the --: invalid argument '--', exit 1 requires an argument, exit 2

A flag left waiting for its value can only be the last argument, so no path before it can have
been ---protected; it now goes last in the child argv instead of eating the boundary.

Tests

Behavioural tests on the arg -> child-argv pipeline, each confirmed failing against the old
logic and passing after. The headline one:

test_plan_double_dash_makes_following_arg_a_path
  left: ["-la", "--", "."]      # before
 right: ["-l", "--", "-al"]     # after

55 tests in the module, full suite green, clippy clean, rtk ls startup 7.8 ms.

@rtk-wshm-sync-bot rtk-wshm-sync-bot Bot added bug Something isn't working ls parsing labels Sep 12, 2026
@rtk-wshm-sync-bot

Copy link
Copy Markdown

wshm · Automated triage by AI

📊 Automated PR Analysis

🐛 Type bug-fix
🟡 Risk medium

Summary

Rewrites the ls command's argument parsing to use a proper tokenizer/grammar instead of raw prefix scans, fixing cases where dash-prefixed paths (e.g. -al after --) were misparsed as flags. It also correctly resolves GNU long-option abbreviations, value-taking flags, and inserts an explicit -- before the path list when invoking the child ls process.

Review Checklist

  • Tests present
  • Breaking change
  • Docs updated

Analyzed automatically by wshm · This is an automated analysis, not a human review.

@KuSh
KuSh force-pushed the feat/ls-arg-tokenizer branch 2 times, most recently from 2ae3912 to 7fa0fee Compare September 20, 2026 23:41

@aeppling aeppling left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

KuSh and others added 2 commits October 7, 2026 01:03
`rtk ls` classified arguments with raw prefix scans (`starts_with('-')`) and
never restored the `--` boundary, so it had no way to tell a flag from a path
that looks like one, or a flag from its own value.

- `rtk ls -- -al` listed the current directory instead of the file named `-al`:
  `-al` was read as the flags `-a`/`-l`, and the path was dropped entirely.
- A flag's value was sorted into the path list, losing its position:
  `rtk ls dir -I pattern` ran `ls -l -I dir pattern`, ignoring `dir` and listing
  `pattern`.
- `-I -al` read the ignore pattern as flags, turning on all/long rendering.
- `--format long` (separate value) did not imply the long listing that
  `--format=long` did.

`plan()` now tokenizes through `tokenize_grammar` with an ls grammar transcribed
from `ls --help`, and emits a `--` before the paths it hands the child. The
grammar resolves GNU long-option abbreviations (`--sor time` sorts by time),
which the new explicit boundary would otherwise break.

Co-Authored-By: Claude Opus 5 <[email protected]>
The option grammar was GNU's alone, and the long-flag table it added was only
consulted for flag names.

- `-I`/`-T`/`-w` take a value on GNU but are booleans on BSD (FreeBSD/macOS
  `ls(1)`; only `-D` takes one there). Reading a macOS operand as one of those
  values moved it ahead of the `--`, and BSD's non-permuting getopt then listed
  the current directory alongside it and failed on `--` itself: `rtk ls -lT /tmp`
  ran `ls -l -T /tmp -- .`.
- `--all=x` was dropped as if it were a bare `--all`, turning ls's exit 2 into a
  successful listing of the current directory.
- `--human-readable` and `--si` were forwarded where `-h` was dropped, so the
  child pre-formatted sizes RTK renders itself and `200K` was read as a link
  count: `rtk ls --human-readable big.bin` printed `1B`.
- `--format=WORD` resolves an abbreviated *value* as well as an abbreviated
  name, so `--format=lon` is a long listing; RTK matched the full words only and
  dropped the octal permission column.
- The `--` was appended unconditionally, so a flag still waiting for its value
  ate it: `rtk ls -a --indicator-style` reported an invalid argument `--` with
  exit 1 where ls reports the missing one with exit 2, and `rtk ls sub -I`
  silently took `--` as the ignore pattern and succeeded. Such a flag can only
  be the last argument, so no path before it needs the boundary; it now goes
  last instead.

Co-Authored-By: Claude Opus 5 <[email protected]>
@KuSh
KuSh force-pushed the feat/ls-arg-tokenizer branch from 7fa0fee to 93166b9 Compare October 6, 2026 23:07
@KuSh
KuSh merged commit e7a614c into rtk-ai:develop Oct 6, 2026
11 checks passed
@KuSh
KuSh deleted the feat/ls-arg-tokenizer branch October 6, 2026 23:16
@rtk-release-bot rtk-release-bot Bot mentioned this pull request Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ls parsing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants