Split: the editor's files over the API (PR-2.3) - #609

The third chunk of Phase 2 of the split-service spec (~/specs/collins/split-service-and-client.md §3.23 "Files over the API", the "Editor" paragraph): the editor panel opens no file of the project's and writes none. Every open is an fs.read, every save an fs.write the service refuses stale when the file moved underneath the buffer, and every file monitor a watch installed on the service pushing file-changed. GtkSource.File, FileLoader and FileSaver are gone from editor.py. Stacked on PR 610 (split/2.2-freshness-blobs at its final 2a472f4), itself on PR 608 (split/2.1-git-over-api at 2192055); the three commits were rebased onto it once it opened, with core.py, protocol.py and test_api_server.py merged to carry both chunks.

What the service serves (collins/service/files.py)

• fs.read {path, max}: the file's bytes decoded as UTF-8 or, when they are not UTF-8, as latin-1 with the encoding flagged (so the save writes them back the same way); binary (a NUL in the first 8 KiB) with an empty text; the mtime (microseconds, as git.info's index_mtime) the save that follows expects, and the size. Refused refused over max (5 MiB, protocol.FILE_TEXT_MAX, at most), for a path that is not a regular file, and outside every root; gone for a file that is not there.
• fs.write {path, text, expect_mtime, encoding}: the text written the way GtkSource.FileSaver (g_file_replace) wrote it: in place (open, compare off the descriptor, write, truncate, fsync) for a file that is one of several hard links, one in a directory the service cannot write, or one another user or group owns; otherwise by a temporary file beside it with the old mode and one os.replace, compared again right before the replace; a file that is not there is created 0o666 under the umask. A symlink is followed to the file it names (a link inside the project never turns into a copy). The compare is against expect_mtime: a file that moved is refused stale (PR 608's error code) and left exactly as it is; expect_mtime: null writes regardless (the Overwrite the user confirmed, a new file). The reply's mtime and size are read off the written descriptor. A text latin-1 cannot carry is written as UTF-8 and the reply says so.
• fs.watch {path, kind: file, handle, mtime} / fs.unwatch {handle}: one Gio.FileMonitor per client and handle (_FileWatch, the editor's monitor moved here with its 300 ms debounce; at most 512 per client), seeded by a first stat on a thread compared against the client's mtime (the file as it last read or wrote it): a file that already differs is one file-changed {handle, path, mtime, size, gone} at once (a change between the read and the watch, or while the client was disconnected), then one per burst whose stat moved. A handle watched again replaces its earlier watch; a client going away drops its watches. kind: dir is in the table for PR-2.4 and refused here.
• Confinement is PR 608's rule (a live session's cwd, a store session's cwd or project root, anything for a proven local client): the roots are computed on the main loop and the check runs on the worker (files.confined), on the path resolved through its symlinks right before the open, so a link swapped between the request and the read points nowhere outside. The read's open is non-blocking and the file's kind is read off the descriptor (a FIFO swapped in is refused, not waited on). Nothing blocks the main loop: every read, write and stat runs on a daemon thread behind a protocol.Deferred settled at PRIORITY_DEFAULT, and a worker that raises settles a failed refusal with the exception's words, so a client is never left waiting. ServiceCore routes the four types to core.files and shuts the watches down with the feed's. The service's refusal words name no file (the client puts the name in front).

The transport: chunking both ways

A file is at most 5 MiB and a frame 1 MiB, so fs.read's reply and fs.write's request can both exceed the cap. PR 608's chunked reply (stdout as TAG_BLOB frames ahead of a slim reply) is generalised: protocol.split_message / join_message chunk the first of CHUNKED_FIELDS (stdout, text) a message holds, as <field>_chunked / <field>_bytes; split_reply / join_reply keep their names, and split_request / join_request are the client's and the server's halves for a request. api/client.py sends a request's frames ahead of the slim request on the same connection (_send_request); api/server.py buffers TAG_BLOB frames per connection by stream (a budget of 4 × FILE_TEXT_MAX bytes per connection, the oldest transfer dropped past it) and joins a request before validating it, refusing one short of its bytes invalid. §3.2's rule ("never send a frame over 1 MiB; larger transfers are chunked", 0x03 either way) already said so; this is its request half.

The client (collins/remotefiles.py, collins/editor.py)

• remotefiles.read / write are the two blocking calls (the sync channel, from a worker thread); is_stale tells the refusal apart; refusal_words translates the service's msgid for the banner. remotefiles.Watcher mints a handle per watch, sends fs.watch (with the caller's mtime seed, kept current by update) / fs.unwatch by send, hands each file-changed for a handle to its listener (a bound method is held weakly, weakref.WeakMethod, and a dead one's watch is dropped at the next event) and reset() re-sends every live watch with its current seed after a reconnect (the service's are gone with it, and so would be anything written meanwhile: the seed catches it); install(link) in App._install_file_transport, reset() in App._on_connected.
• editor.py: _start_load reads on a thread (_off_main: a daemon thread, the answer landed at GLib.PRIORITY_DEFAULT) and _on_loaded fills the buffer outside the undo history (_fill), keeping mtime, size and encoding on the _OpenFile, stripping the file's final newline when the buffer's implicit trailing newline is on (what FileLoader did; _do_save puts it back, so no file shows an empty last line and a save adds a missing final newline) and marking the buffer filled: the view is not editable until then, and a save before it is refused ("still loading") rather than writing an empty buffer over the file. A refusal's words land in the banner, closing a fresh open's tab and keeping a reload's; load_id still makes a superseded read (a rename's) a no-op. The language guess reads the shebang off the text the service answered (editorfiles.first_line) and should_highlight takes the reply's size. _save (Ctrl+S) expects the last mtime and _on_saved raises the "changed on disk" dialog on stale, whose Overwrite saves with expect_mtime: null; save_all and the close flows' Save pass null from the start (the user's explicit consent, as before); saves are serialized (a second Ctrl+S during one waits and goes next with the mtime the first answered); a save written in another encoding than the read's is told. _watch_external_changes installs the watch seeded with opened.mtime through the bound method _on_file_changed (handle → _OpenFile in _watched); _check_external judges a file-changed against opened.mtime and size: a clean buffer reloads silently (cursor kept), a dirty one is told, gone marks it dirty and says so. An event that lands while a save or load is in flight waits (pending_change) for the reply's mtime, so the editor's own write never reads as a change. _retarget_open re-watches the new path (a rename keeps the mtime); _teardown_page unwatches one file and EditorPane.shutdown every one, called by TerminalTab.release_editor where the window closes a tab for good and when the window is destroyed.
• editorfiles.read_first_line, should_highlight(path) and load_guard (the editor's three disk reads) are gone: first_line(text) and should_highlight(size) are pure, and the load guard's three answers (a regular file, the size cap, the NUL sniff) are fs.read's. install(link) runs behind the files capability of the service's hello (App._install_file_transport, on every connect like _install_git_transport); without it no watch is sent, and the reads and writes an open or a save still sends come back refused into the banner.

The pathless allowlist

Six entries leave the PR-2.3 group (editor:EditorPane._watch_external_changes:…monitor_file, editorfiles:read_first_line:open, editorfiles:should_highlight:Path.stat, the three editorfiles:load_guard:* sites PR 608's wider walker found); request_root:Path.is_dir, set_agent_files:Path.is_file, image_guard's and gitsidebar._file_menu_items' stay, noted as fs.stat reads (PR-2.4), left where the list put them. Nothing was added.

Tests and checks

• tests/test_files.py: a read's fields; a watch seeded with an older mtime reports the change at once (a matching or null seed none, a vanished file gone); watches per client bounded; a write through a hard link updates both names, into a read-only directory writes in place (and stays stale-safe), a recreated file gets the umask's mode, the reply's mtime is the written file's own; confinement checked on the worker against the resolved file (a link out refused for read and write, a link in read); a FIFO refused, not waited on; latin-1 flagged; binary flagged with no text; a read over 5 MiB refused (and over the request's own max; at the cap exactly, read); a directory and a missing file refused; read, write and watch confined for a non-local client, admitted under a store root, a symlink out refused; a write round-trips with the mtime the next read has and leaves no temp file; a write with a moved mtime refused stale and the file untouched, null writing regardless; a vanished file with an expectation stale; the mode kept and a symlink followed; latin-1 written back and UTF-8 when it cannot carry the text; a failure failed with the reason; a read is a Deferred settled later on the dispatcher; a worker that raises settles failed; a watch reports a change once per debounce with the file's stat (two writes, one event; a burst that changed nothing, none), the real monitor fires, a deletion is gone, a handle re-watched replaces, unwatch and a client going away drop, a directory kind is refused.
• tests/test_remotefiles.py (the calls and their records, stale told apart, the watcher's handles, seeds and routing, update and reset re-sending the live watches with their current seed, a refused watch logged and kept, a listener that raises contained, no watch sent with no link installed, a bound-method listener held weakly and its watch dropped when it dies), tests/test_api_server.py (a fs.write over the frame cap arrives in chunks, is joined and written, and the fs.read of it comes back chunked; a chunked request short of its bytes is invalid and writes nothing), tests/test_protocol.py pins the five new types, the files cap and every field's bounds; tests/test_editorfiles.py follows the two pure helpers.
• scripts/check_editor_save.py (new, registered in run_e2e.py at an estimated weight): a real EditorPane against a scratch service through e2e_service.harness_link(): open (the text with no empty last line, the mtime, a watch, the language, the view editable once filled), edit, an external write over the dirty buffer raising the Reload banner with the edit kept, Reload bringing the disk's text and mtime, a save after it landing on disk with no banner and no reload from the editor's own write, a clean buffer following the disk silently with the cursor kept, a stale save asking (the dialog stubbed) with nothing written until Overwrite, a deleted file told and marked dirty, a change queued behind a failed reload still judged, a save before the first read lands refused with the file untouched, a change between the read and the watch reaching the buffer (the seed), the pane's shutdown dropping every watch, a closed tab's watch gone, a binary file refused with a banner and no tab. 57 checks.
• scripts/check_editor_narrow.py gains the same two lines (a scratch service and remotefiles.install) so its opens have somewhere to read from; no assertion, argument or expected value changed.

After the Opus review (the second push)

The review's three must-fixes and should-fixes 4 to 9 are in: a save of a never-filled buffer refused and the view non-editable until the first fill (1); EditorPane.shutdown from the tab's close and weak listeners in the watcher (2); fs.watch's mtime seed, compared on a thread, delivered at once, re-sent by reset (3); the saver's in-place / temp-file / new-file rules with the mode, owner and hard links kept (4); the compare right before the write and the reply's mtime off the written descriptor (5); serialized saves (6); the request chunk budget (7); the implicit trailing newline (8); confinement on the worker against roots from the main loop (9). Nits: the load-guard docstrings and the dead cap, the doubled error text (the service's msgids name no file), the non-blocking read open with the kind read off the descriptor, a watches-per-client bound, mtime and size compared, and the banner when a save's encoding differs from the read's. Not taken: Files._later stays beside GitFeed._later (their refusals differ, and one shared helper is a later tidy).

The verification pass added two: a read-only file of the user's own (0444 in a writable directory) is refused failed "Permission denied" rather than swapped out by the replace (what g_file_replace did; tested), and a read or write that lands after EditorPane.shutdown fills nothing and installs no watch (_shut; the e2e slows a read and shuts the pane under it). Plus: a chunked request's TAG_BLOB frames and its slim text go out in one I/O job, so two threads' large saves never interleave on the connection; Ctrl+S on a clean buffer mid silent reload is a no-op instead of a spurious stale dialog. A caveat worth knowing: the in-place branch (a hard link, a read-only directory, another owner's file) writes then truncates, so a crash in the middle of that write leaves the new head and the old tail — the saver this replaces was no safer on that branch, and the temp-and-replace branch is atomic.

Deviations from §3.23, each with its reason

• Requests are chunked too (split_request / join_request, the server's per-connection chunk buffer): §3.23 gives fs.read a 5 MiB max and fs.write the same text, and PR 608's chunking covered replies alone. Without it a file over 1 MiB would open and never save. The rule is §3.2's; only the request half is new.
• fs.write carries encoding and the reply says which was written: §3.23's read flags latin-1 "so the save writes them back the same way", which needs the save to be told; a text latin-1 cannot carry falls back to UTF-8 rather than failing the save (what the user typed is worth more than the file's old encoding).
• fs.read answers gone for a missing file rather than refused: gone is the code for "what the request names does not exist".
• A file-changed is pushed only when the stat moved since the last one (or the seed at the watch's start), not on every monitor burst: the editor compared mtimes itself before; now the service does, and the client still compares against its own mtime, so a save's own write is never a change.
• kind: dir is in fs.watch's table and refused: the wire shape is PR-2.4's to serve, pinned now so the field does not change under it.
• check_editor_narrow.py changed by its call path only (a scratch service and the module install), as §5 allows: an editor with no link to read from cannot open the two files the check opens.
• The files cap joins CAPABILITIES beside git, is advertised in hello and gates the client's install the way PR 608's git cap does.

What PR-2.4 and PR-2.5 need to know

• PR-2.4: files.Files is the home for fs.list / fs.walk / fs.stat and kind: dir watches (_FileWatch is a file's; a directory's wants monitor_directory and a dir-changed {handle, path} event type added to the table). remotefiles.Watcher keys by handle and takes any listener, so a tree's watch can share it. editor.request_root's Path.is_dir is the one editor site left on the allowlist, yours with fs.stat. editorfiles.load_guard / image_guard still exist and read the disk; nothing in the editor calls load_guard any more.
• PR-2.5: a rename (_retarget_open) re-watches the new path and keeps opened.mtime (a rename keeps the file's mtime); if fs.rename on the service lands the file at a different mtime, hand it back in the reply so the editor can take it. fs.trash in files.py is untouched by this PR (PR 608's review moves it onto a thread).
• Both: Files._later(message, name, work) is the Deferred pattern with the refusal on a raised worker; files.file_stat / files.mtime_us are the stat and the wire's mtime unit.

Verification

python3 -m pytest tests/ -q (9000 passed) and ruff check collins/ tests/ clean; the full headless e2e suite green except the known local-only check_notifications.py sound-row case ("the combo sits on Custom…"); check_editor_narrow.py 22/22 and check_editor_save.py 57/57 under the headless shell; by hand through the harness on scratch XDG_* dirs: open, edit, write the file externally, see the Reload banner, reload, save (what check_editor_save.py drives through the real widgets against the scratch service), a 2 MiB file written chunked and read back chunked (test_api_server.py).

🤖 Generated with Claude Code

https://claude.ai/code/session_011sbC3wuGjnM1AysiE7RjJ8