Repository navigation
Conversation
…s, and preview generation ## Summary - evaluation/unigenbench: handle empty streaming chunks, add exponential backoff on retries, clean up async client on close, record failed samples. - evaluation/paibench_g: support partial result persistence on worker failure so completed metrics are not discarded. - evaluation/paibench_c: clamp evaluation GPUs against CUDA_VISIBLE_DEVICES to prevent oversubscribing workers. - cookbooks/action: fail fast with actionable error when multi-node env vars are partially set. - cookbooks/transfer: make preview mp4 generation atomic to prevent partial file writes. - ci: guard notebook validation against empty file sets. Signed-off-by: umran666 <[email protected]>
sylvesterkaczmarek
left a comment
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
The partial-file cleanup currently covers the ffmpeg invocation, but not the final os.replace(partial, preview).
If the replace itself fails, for example because of a filesystem or permission error, _preview.partial.mp4 is left behind. Could the replace be included in the same try with partial.unlink(missing_ok=True) in a finally?
That would preserve the atomic final-file behaviour while also guaranteeing that failed encode or rename paths do not leave stale partial files.
…erage Address review feedback from @sylvesterkaczmarek: the atomic rename now lives inside the try block so a failed os.replace (permission error, filesystem fault) also triggers partial.unlink in the except handler, guaranteeing no stale .partial.mp4 files on any failure path. Signed-off-by: umran666 <[email protected]>
|
@sylvesterkaczmarek Yes!Good catch, moved os.replace inside the try so a failed rename also cleans up the partial file. |
sylvesterkaczmarek
left a comment
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
Re-reviewed the partial-file cleanup path. os.replace(partial, preview) is now inside the same try, and any encode or replace failure unlinks the partial before re-raising. My previous blocker is resolved.
|
Related evaluation-hardening findings are #361, #362, and #363. This PR improves worker failure and UniGenBench retry handling, but the PAIBench resume manifest still identifies the evaluator by path only, non-finite motion scores are checkpointed as completed, and UniGenBench still reuses any existing result without checking model or input identity. |
While running through the evaluation benchmarks and fine-tuning cookbooks, I ran into a few edge cases and stability issues where transient network hiccups, gateway idiosyncrasies, or missing env vars caused jobs to fail or abort before writing aggregate results.
choiceschunks emitted by some OpenAI-compatible VLM gateway endpoints (e.g., terminal usage-only frames) to avoid raisingIndexErrormid-stream.vlm.close()to prevent lingering connection warnings upon interpreter exit.--vlm_timeout(default 300s, up from hard-coded 100s) and--vlm_max_retry(default 3) so slower judge completions aren't prematurely dropped and re-prompted.ugb_scorer.pynow defensively parses both native python lists and legacy stringified list representations."failed_samples"key in the result JSON (alongsidesuccess_count) instead of being silently omitted.VIDEO_RETRIES = 1) so transient AMT scoring failures get a second attempt before failing the run."missing_videos"in the result JSON, rather than aborting prior to the merge step.CUDA_VISIBLE_DEVICESto clamp evaluation GPU workers to the actual number of visible devices, preventing oversubscription whenCOSMOS3_NUM_GPUSexceeds the physical count.run_paibench_c.shand added an explicit 1-sample smoke-test example.cookbooks/cosmos3/generator/action/finetune/*.shwhenNNODES,NODE_RANK, orMASTER_ADDRare partially set, surfacing a clear error before torchrun's rendezvous layer fails.preview_helpers.pynow encodes to a sibling.partial.mp4first and swaps atomically withos.replace, preventing corrupt/incomplete preview files if an interactive cell is interrupted..github/workflows/validate-notebooks.ymlsoruffexits cleanly when no notebooks match the glob.evaluation/cosmos3/generator/unigenbench/query_core_selfcheck.pyandevaluation/cosmos3/generator/paibench_g/merge_result_selfcheck.py(stdlib-only assert tests, requiring no GPU or network) to verify streaming chunk parsing and merge aggregation edge cases.