Repository navigation
fix(trainer): handle falsy values in get_args_from_peft_config - #328
Conversation
Signed-off-by: krishdef7 <[email protected]>
cb699c6 to
a0060b0
Compare
There was a problem hiding this comment.
Pull request overview
This PR fixes a bug where falsy configuration values (like lora_dropout=0.0 and apply_lora_to_mlp=False) were being silently dropped in the get_args_from_peft_config function. The fix ensures that explicit user configurations are respected regardless of their truthiness.
Changes:
- Updated condition in
get_args_from_peft_configfromif value:toif value is not None: - Added comprehensive unit tests for falsy value handling, including
lora_dropout=0.0,apply_lora_to_mlp=False, standard positive values, and error cases
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| kubeflow/trainer/backends/kubernetes/utils.py | Fixed condition to use is not None check instead of truthiness check to preserve falsy configuration values |
| kubeflow/trainer/backends/kubernetes/utils_test.py | Added unit tests to verify falsy values are correctly handled and included in generated arguments |
|
Hi @andreyvelich @astefanutti @szaher, Happy to make any changes if needed. This fixes an issue where valid configurations (e.g. lora_dropout=0.0) were silently ignored. Thanks! |
|
/ok-to-test |
astefanutti
left a comment
There was a problem hiding this comment.
@krishdef7 Thanks!
/lgtm
Signed-off-by: krishdef7 <[email protected]>
eec5975 to
6ad5f16
Compare
|
@astefanutti Thanks! All checks are passing now. Could you please re-add /lgtm when you have a moment? |
|
Hey @krishdef7, nice catch! I was going through this PR and noticed the same if value: pattern in get_args_from_dataset_preprocess_config could also cause issues: if dataset_preprocess_config.train_on_input: A user setting train_on_input=False (which the docstring describes as the default behavior) would have it silently dropped and the model then would train differently than intended with no error. Since you're already fixing this pattern, |
|
@ruskaruma Good catch — thanks for pointing this out! You're right, this follows the same falsy-value pattern and would drop an explicit I'll extend the fix here as well to ensure explicit boolean values are preserved, and add corresponding tests to keep the handling consistent across config paths. Appreciate the suggestion! |
Signed-off-by: krishdef7 <[email protected]>
629ccb6 to
c344fca
Compare
Signed-off-by: krishdef7 <[email protected]>
b074454 to
3c164b9
Compare
Signed-off-by: krishdef7 <[email protected]>
716af62 to
6d5191d
Compare
|
/ok-to-test |
|
@astefanutti All done — extended the fix to get_args_from_dataset_preprocess_config as well (ruskaruma's suggestion). Could you re-add /lgtm when the checks finish? |
|
@krishdef7 thanks! /ok-to-test |
andreyvelich
left a comment
There was a problem hiding this comment.
Thanks @krishdef7!
/lgtm
/approve
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: andreyvelich The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
* chore!: upgrade to Python 3.10 (kubeflow#282) This upgrades the minimum Python version for the project from 3.9 to 3.10. Python 3.9 is past end-of-life and dependencies will likely require a supported version soon. Signed-off-by: Jon Burdo <[email protected]> * chore: Confirm that a public ConfigMap exists to check version (kubeflow#250) * Confirm that a public ConfigMap exists to check version Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * python 3.9 fix Co-authored-by: Copilot <[email protected]> Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * Exceptiom handling better Co-authored-by: Copilot <[email protected]> Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * Addressing comments Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * Update kubeflow/trainer/backends/kubernetes/backend.py Co-authored-by: Andrey Velichkevich <[email protected]> Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * Refactored tests into a single function and followed agents.md Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * CI friendly edit Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * pre-commit format checked Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * Modified according to new updates Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * Ran pre-commit locally to fix formatting Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * unix2dos CLAUDE.md Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * Revert CLAUDE.md Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> --------- Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Andrey Velichkevich <[email protected]> * chore: added sdk docs website to readme (kubeflow#284) * docs: added sdk docs website to readme Signed-off-by: Akash Jaiswal <[email protected]> * format: order of sdk docs Signed-off-by: Akash Jaiswal <[email protected]> --------- Signed-off-by: Akash Jaiswal <[email protected]> * feat(trainer): add dataset and model initializer support to container backend (kubeflow#188) * feat(trainer): add dataset and model initializer support to container backend Add support for dataset and model initializers in the container backend to bring it to feature parity with the Kubernetes backend. Changes: - Add utility functions for building initializer commands and environment variables - Implement _run_initializers() and _run_single_initializer() methods in ContainerBackend - Run initializers sequentially before training containers start - Download datasets to /workspace/dataset and models to /workspace/model - Track initializer containers as separate steps in TrainJob - Support all initializer types: HuggingFace, S3, and DataCache - Add comprehensive unit tests for all initializer configurations - Handle initializer failures with proper cleanup and error messages Fixes kubeflow#171 Signed-off-by: HKanoje <[email protected]> * feat(trainer): address reviewer feedback for initializer support - Make initializer image configurable via ContainerBackendConfig - Make initializer timeout configurable (default 600 seconds) - Implement wait API in adapters instead of polling - Clean up successful initializer containers after completion - Clean up network on initializer failure - Raise ValueError for unsupported initializer types (no datacache fallback) All tests passing (173/173). Addresses all feedback from PR kubeflow#188. Signed-off-by: HKanoje <[email protected]> * chore(trainer): add cleanup helper to reduce duplication Add _cleanup_container_resources() helper method to consolidate duplicated cleanup logic for stopping/removing containers and deleting networks. Refactor 5 locations across train(), initializer handlers, and delete_job() to use this helper. Signed-off-by: HKanoje <[email protected]> * fix(trainer): use correct initializer images and working directory Address feedback for initializer support in container backend: - Use separate images for dataset/model initializers: - kubeflow/dataset-initializer:latest for datasets - kubeflow/model-initializer:latest for models (instead of kubeflow/training-operator:latest) - Update python commands to use pkg.initializers module: - python -m pkg.initializers.dataset (for dataset) - python -m pkg.initializers.model (for model) - Change initializer working_dir from /workspace to /app per Dockerfile convention Refs: https://github.com/kubeflow/trainer/tree/master/cmd/initializers Signed-off-by: HKanoje <[email protected]> * fix(container): address PR review comments for initializer support - Use GHCR images as default for dataset/model initializers - Replace suppress with try-except blocks - Refactor initializer utils with ContainerInitializer dataclass - Add get_dataset_initializer and get_model_initializer functions - Remove DataCache support (unsupported in container backend) - Merge initializer tests into test_train() and test_get_job_logs() - Remove duplicate test functions Signed-off-by: HKanoje <[email protected]> * fix(container): add name field to ContainerInitializer and remove init_type - Add name field to ContainerInitializer dataclass - Set name='dataset-initializer' and name='model-initializer' in utils - Remove init_type parameter from _run_single_initializer() - Use container_init.name for labels and log messages Signed-off-by: HKanoje <[email protected]> --------- Signed-off-by: HKanoje <[email protected]> * feat: add SparkClient API for SparkConnect session management (kubeflow#225) * feat(spark): add core types, dataclasses, and constants - Add SparkConnectInfo, SparkConnectState, Driver, Executor types - Add type tests for validation - Add Kubernetes backend constants (CRD group, version, defaults) Signed-off-by: Shekhar Rajak <[email protected]> * feat(spark): add backend base class and options pattern - Add RuntimeBackend abstract base class with session lifecycle methods - Add options pattern (Name, Image, Timeout, etc.) aligned with trainer SDK - Add validation utilities for connect parameters - Add comprehensive option tests Signed-off-by: Shekhar Rajak <[email protected]> * feat(spark): add KubernetesBackend for SparkConnect CRD operations - Implement KubernetesBackend with create/get/list/delete session methods - Add port-forward support for out-of-cluster connections - Add CRD builder utilities and URL validation - Add comprehensive backend and utils tests with parametrized patterns Signed-off-by: Shekhar Rajak <[email protected]> * feat(spark): add SparkClient API with KEP-107 compliant connect method - Implement SparkClient as main user interface for SparkConnect sessions - Support connect to existing server (base_url) or auto-create new session - Add public exports for SparkClient, Driver, Executor, options - Add SparkClient unit tests Signed-off-by: Shekhar Rajak <[email protected]> * chore(spark): add test infrastructure and package init files - Add test common utilities and fixtures - Add package __init__ files for test directories - Setup test/e2e/spark structure Signed-off-by: Shekhar Rajak <[email protected]> * feat(spark): add example scripts demonstrating SparkClient usage - Add spark_connect_simple.py with 3 usage levels (minimal, simple, advanced) - Add spark_advanced_options.py with full configuration examples - Add connect_existing_session.py for connecting to existing servers - Add demo and test scripts for local development Signed-off-by: Shekhar Rajak <[email protected]> * docs(spark): add documentation for SparkClient and E2E testing - Add examples/spark/README.md with usage guide - Add local Spark Connect testing documentation - Add E2E test README with CI/CD integration guide - Update KEP-107 proposal documentation Signed-off-by: Shekhar Rajak <[email protected]> * test(spark): add E2E test framework with cluster watcher - Add test_spark_examples.py with example validation tests - Add cluster_watcher.py for monitoring SparkConnect and pods during tests - Add run_in_cluster.py for executing examples as K8s Jobs Signed-off-by: Shekhar Rajak <[email protected]> * ci(spark): add GitHub Actions workflow and E2E cluster setup - Add test-spark-examples.yaml workflow for E2E validation - Add e2e-setup-cluster.sh for Kind cluster with Spark Operator - Add SparkConnect CRD, Kind config, and E2E runner Dockerfile - Update Makefile with E2E setup target - Update PR title check for spark prefix Signed-off-by: Shekhar Rajak <[email protected]> * chore(spark): add pyspark[connect] dependency and update lock file - Add spark extra with pyspark[connect]==3.4.1 for grpcio, pandas, pyarrow - Update uv.lock with resolved dependencies - Update .gitignore for spark-related files Signed-off-by: Shekhar Rajak <[email protected]> * Update kubeflow/spark/backends/base.py Co-authored-by: Andrey Velichkevich <[email protected]> Signed-off-by: Shekhar Prasad Rajak <[email protected]> * refactor(spark): rename backend.connect_session() to connect() Signed-off-by: Shekhar Rajak <[email protected]> * refactor: move session creation flow from SparkClient to backend.create_and_connect() Signed-off-by: Shekhar Rajak <[email protected]> --------- Signed-off-by: Shekhar Rajak <[email protected]> Signed-off-by: Shekhar Prasad Rajak <[email protected]> Co-authored-by: Andrey Velichkevich <[email protected]> * chore: bump minimum model-registry version to 0.3.6 (kubeflow#289) Signed-off-by: Jon Burdo <[email protected]> * fix: Improve CVE workflow (kubeflow#267) * fix: Improve CVE workflow Signed-off-by: Fiona-Waters <[email protected]> * fix: fix issue with bash compare Signed-off-by: Fiona-Waters <[email protected]> * feat: Add workflow to cleanup overrides in pyproject.toml Signed-off-by: Fiona-Waters <[email protected]> * fix: address review comments Signed-off-by: Fiona-Waters <[email protected]> * chore: refactor to reduce size of cve related workflows Signed-off-by: Fiona-Waters <[email protected]> --------- Signed-off-by: Fiona-Waters <[email protected]> * chore: upgrade code style for python3.10 (kubeflow#288) * chore: update code style for Python 3.10 This disables a couple ruff rules in pyproject.toml: ``` "UP007", # Use X | Y instead of Union[X, Y] (requires Python 3.10+) "UP045", # Use X | None instead of Optional[X] (requires Python 3.10+) ``` Then the code changes are made with: ``` uv run ruff check --fix uv run ruff format ``` Signed-off-by: Jon Burdo <[email protected]> * fix: handle unions, bools in convert_value The convert_value function didn't seems to be handling union types properly and also needs to handle `T | None` similarly to `Optional[None]` after the upgrade to Python 3.10. This fixes union types, an issue with bool conversion, and adds tests for this function. Signed-off-by: Jon Burdo <[email protected]> --------- Signed-off-by: Jon Burdo <[email protected]> * chore(ci): bump astral-sh/setup-uv from 5 to 7 (kubeflow#276) Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 5 to 7. - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@v5...v7) --- updated-dependencies: - dependency-name: astral-sh/setup-uv dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump the python-minor group across 1 directory with 4 updates (kubeflow#291) Bumps the python-minor group with 4 updates in the / directory: [coverage](https://github.com/coveragepy/coveragepy), [ruff](https://github.com/astral-sh/ruff), [pre-commit](https://github.com/pre-commit/pre-commit) and [ty](https://github.com/astral-sh/ty). Updates `coverage` from 7.10.7 to 7.13.4 - [Release notes](https://github.com/coveragepy/coveragepy/releases) - [Changelog](https://github.com/coveragepy/coveragepy/blob/main/CHANGES.rst) - [Commits](coveragepy/coveragepy@7.10.7...7.13.4) Updates `ruff` from 0.14.14 to 0.15.0 - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.14.14...0.15.0) Updates `pre-commit` from 4.3.0 to 4.5.1 - [Release notes](https://github.com/pre-commit/pre-commit/releases) - [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md) - [Commits](pre-commit/pre-commit@v4.3.0...v4.5.1) Updates `ty` from 0.0.14 to 0.0.16 - [Release notes](https://github.com/astral-sh/ty/releases) - [Changelog](https://github.com/astral-sh/ty/blob/main/CHANGELOG.md) - [Commits](astral-sh/ty@0.0.14...0.0.16) --- updated-dependencies: - dependency-name: coverage dependency-version: 7.13.4 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-minor - dependency-name: ruff dependency-version: 0.15.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-minor - dependency-name: pre-commit dependency-version: 4.5.1 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-minor - dependency-name: ty dependency-version: 0.0.16 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: Added examples to the documentation demonstrating different ways to handle ports (kubeflow#243) * update docs and add test cases. Signed-off-by: osamaahmed17 <[email protected]> * pre-commit error solved Signed-off-by: osamaahmed17 <[email protected]> * Update kubeflow/hub/api/model_registry_client.py Co-authored-by: Jon Burdo <[email protected]> Signed-off-by: Osama Tahir <[email protected]> * readme updated Signed-off-by: Osama Tahir <[email protected]> * Refactor model registry client test cases for clarity Signed-off-by: Osama Tahir <[email protected]> --------- Signed-off-by: osamaahmed17 <[email protected]> Signed-off-by: Osama Tahir <[email protected]> Co-authored-by: Jon Burdo <[email protected]> * chore(ci): bump peter-evans/create-pull-request from 6 to 8 (kubeflow#277) Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 6 to 8. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](peter-evans/create-pull-request@v6...v8) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-version: '8' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(ci): bump actions/checkout from 4 to 6 (kubeflow#278) Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: Adds a GitHub Actions workflow to check kubeflow/hub/OWNERS. (kubeflow#280) * Add OWNERS validation Signed-off-by: muhammadjunaid8047 <[email protected]> * Update .github/workflows/check-owners.yaml Co-authored-by: Jon Burdo <[email protected]> Signed-off-by: Muhammad Junaid <[email protected]> * Update OWNERS file check in workflow Signed-off-by: Muhammad Junaid <[email protected]> * Update paths in check-owners workflow Signed-off-by: Muhammad Junaid <[email protected]> --------- Signed-off-by: muhammadjunaid8047 <[email protected]> Signed-off-by: Muhammad Junaid <[email protected]> Co-authored-by: Jon Burdo <[email protected]> * fix: nightly security dependency updates (kubeflow#296) Co-authored-by: google-oss-prow <[email protected]> * chore(ci): bump aquasecurity/trivy-action from 0.33.1 to 0.34.0 in the actions group (kubeflow#297) Bumps the actions group with 1 update: [aquasecurity/trivy-action](https://github.com/aquasecurity/trivy-action). Updates `aquasecurity/trivy-action` from 0.33.1 to 0.34.0 - [Release notes](https://github.com/aquasecurity/trivy-action/releases) - [Commits](aquasecurity/trivy-action@0.33.1...0.34.0) --- updated-dependencies: - dependency-name: aquasecurity/trivy-action dependency-version: 0.34.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump pytest from 8.4.2 to 9.0.2 (kubeflow#301) Bumps [pytest](https://github.com/pytest-dev/pytest) from 8.4.2 to 9.0.2. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pytest@8.4.2...9.0.2) --- updated-dependencies: - dependency-name: pytest dependency-version: 9.0.2 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(trainer): Support namespaced TrainingRuntime in the SDK (#130) * feat(backend): Support namespaced TrainingRuntime in the SDK Signed-off-by: Moeed Shaik <[email protected]> * Fixed bugs and validated current test cases Signed-off-by: Moeed Shaik <[email protected]> * Fixed pre-commit test failure Signed-off-by: Moeed Shaik <[email protected]> * Addressed comments Signed-off-by: Moeed Shaik <[email protected]> * Fixed no attribute 'DEFAULT_TIMEOUT' error Signed-off-by: Moeed Shaik <[email protected]> * Added namespace-scoped runtime to test cases Signed-off-by: Moeed Shaik <[email protected]> * Addressed fallback logic bugs Signed-off-by: Moeed Shaik <[email protected]> * Added scope field to Runtime Signed-off-by: Moeed Shaik <[email protected]> * Improved code Signed-off-by: Moeed Shaik <[email protected]> * Fixed copilot's comments Signed-off-by: Moeed Shaik <[email protected]> * Shadow duplicate runtimes, priority to ns Signed-off-by: Moeed Shaik <[email protected]> * Fixed bug Signed-off-by: Moeed Shaik <[email protected]> * Fixed copilot comments Signed-off-by: Moeed Shaik <[email protected]> * Improved test cases to validate all possible cases Signed-off-by: Moeed Shaik <[email protected]> * small fix Signed-off-by: Moeed Shaik <[email protected]> * lint fix Signed-off-by: Moeed Shaik <[email protected]> * improved error message Co-authored-by: Copilot <[email protected]> Signed-off-by: Moeed <[email protected]> * refactored code Signed-off-by: Moeed Shaik <[email protected]> * improve code Signed-off-by: Moeed Shaik <[email protected]> * Removed RuntimeScope Signed-off-by: Moeed Shaik <[email protected]> * removed scope references and improved error handling as per kubeflow standards Signed-off-by: Moeed Shaik <[email protected]> --------- Signed-off-by: Moeed Shaik <[email protected]> Signed-off-by: Moeed <[email protected]> Co-authored-by: Copilot <[email protected]> * fix: Fix runtime lookup fallback and test local SDK in E2E (kubeflow#307) * fix: Install SDK locally in E2E workflow and improve error handling for runtime fetching in Kubernetes backend. Signed-off-by: XploY04 <[email protected]> * refactor: Explicitly return errors from and refine exception handling in . Signed-off-by: XploY04 <[email protected]> * docs: update comment to clarify Kubeflow SDK installation from source in e2e workflow. Signed-off-by: XploY04 <[email protected]> * feat: Enhance runtime retrieval tests to cover Kubernetes API 404/403 errors and partial success for list operations on timeout. Signed-off-by: XploY04 <[email protected]> * refactor: Update runtime listing to immediately raise exceptions on failure instead of collecting partial results. Signed-off-by: XploY04 <[email protected]> --------- Signed-off-by: XploY04 <[email protected]> * chore(ci): bump actions/setup-python from 5 to 6 (kubeflow#298) Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5 to 6. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v5...v6) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump the python-minor group with 2 updates (kubeflow#299) Bumps the python-minor group with 2 updates: [ruff](https://github.com/astral-sh/ruff) and [ty](https://github.com/astral-sh/ty). Updates `ruff` from 0.15.0 to 0.15.1 - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.15.0...0.15.1) Updates `ty` from 0.0.16 to 0.0.17 - [Release notes](https://github.com/astral-sh/ty/releases) - [Changelog](https://github.com/astral-sh/ty/blob/main/CHANGELOG.md) - [Commits](astral-sh/ty@0.0.16...0.0.17) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.15.1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-minor - dependency-name: ty dependency-version: 0.0.17 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: improve logging around packages_to_install (kubeflow#269) * improve logging around packages_to_install Signed-off-by: Brian Gallagher <[email protected]> * exit when pip install fails, append errors from both attempts Signed-off-by: Brian Gallagher <[email protected]> * Add shlex to address command injection vulnerabilities. Write pip install logfile to cwd Signed-off-by: Brian Gallagher <[email protected]> --------- Signed-off-by: Brian Gallagher <[email protected]> * feat: Add validate lockfile workflow to complement CVE scanning (kubeflow#306) * feat: Add validate lockfile workflow to complement CVE scanning Signed-off-by: Fiona-Waters <[email protected]> * fix: make cve fix pr branch static Signed-off-by: Fiona-Waters <[email protected]> --------- Signed-off-by: Fiona-Waters <[email protected]> * fix(trainer): handle falsy values in get_args_from_peft_config (kubeflow#328) * fix(trainer): handle falsy values in get_args_from_peft_config Signed-off-by: krishdef7 <[email protected]> * fix: apply pre-commit formatting Signed-off-by: krishdef7 <[email protected]> * fix: also handle falsy train_on_input in dataset_preprocess_config Signed-off-by: krishdef7 <[email protected]> * fix: add missing newline at end of utils_test.py Signed-off-by: krishdef7 <[email protected]> * fix: pre-commit formatting Signed-off-by: krishdef7 <[email protected]> --------- Signed-off-by: krishdef7 <[email protected]> * fix(optimizer): prevent input mutation in optimize() (kubeflow#322) * fix(optimizer): prevent input mutation in optimize() Signed-off-by: ruskaruma <[email protected]> * remove unnecessary things Signed-off-by: ruskaruma <[email protected]> * rename test Signed-off-by: ruskaruma <[email protected]> --------- Signed-off-by: ruskaruma <[email protected]> * feat: add TrainerClient examples for local PyTorch distributed training (kubeflow#312) * docs: add TrainerClient examples for local PyTorch distributed training - Add examples/trainer/pytorch_distributed_simple.py - Add examples/trainer/README.md - Demonstrates LocalProcessBackend usage without Kubernetes - Fixes kubeflow#218 Signed-off-by: Mansi Singh <[email protected]> * docs: add training examples table to SDK website Signed-off-by: Mansi Singh <[email protected]> * docs: expand examples table with PyTorch, MLX, DeepSpeed, and TorchTune examples grouped by framework Signed-off-by: Mansi Singh <[email protected]> --------- Signed-off-by: Mansi Singh <[email protected]> * chore: fix docstrings in TrainerClient (kubeflow#333) Signed-off-by: Transcendental-Programmer <[email protected]> * feat(spark): Refactor unit tests to sdk coding standards (kubeflow#293) * Refactored unit test Signed-off-by: digvijay-y <[email protected]> * Changes made Signed-off-by: digvijay-y <[email protected]> * Version Signed-off-by: digvijay-y <[email protected]> * Restructured clien_test Signed-off-by: digvijay-y <[email protected]> * reformated backend_test.py Signed-off-by: digvijay-y <[email protected]> * revert pyproject.toml and uv.lock changes Signed-off-by: digvijay-y <[email protected]> * Standarized spark backend tests Signed-off-by: digvijay-y <[email protected]> * backend_tests Signed-off-by: digvijay-y <[email protected]> --------- Signed-off-by: digvijay-y <[email protected]> * fix(optimizer): add missing get_job_events() to RuntimeBackend base c… (kubeflow#325) * fix(optimizer): add missing get_job_events() to RuntimeBackend base class Signed-off-by: ruskaruma <[email protected]> * Update kubeflow/optimizer/backends/base.py Co-authored-by: Copilot <[email protected]> Signed-off-by: Ruskaruma <[email protected]> * Update kubeflow/optimizer/backends/base.py Co-authored-by: Andrey Velichkevich <[email protected]> Signed-off-by: Ruskaruma <[email protected]> * fix: add abstractmethod, remove docstrings Signed-off-by: ruskaruma <[email protected]> * make get_job_events abstract in RuntimeBackend Signed-off-by: ruskaruma <[email protected]> * Update kubeflow/trainer/backends/localprocess/backend.py Co-authored-by: Andrey Velichkevich <[email protected]> Signed-off-by: Ruskaruma <[email protected]> * fix Signed-off-by: ruskaruma <[email protected]> --------- Signed-off-by: ruskaruma <[email protected]> Signed-off-by: Ruskaruma <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Andrey Velichkevich <[email protected]> * chore(spark): migrate SDK to kubeflow_spark_api Pydantic models (kubeflow#295) * chore(spark): add kubeflow-spark-api dependency Signed-off-by: tariq-hasan <[email protected]> * chore(spark): migrate options to typed Pydantic models Signed-off-by: tariq-hasan <[email protected]> * chore(spark): migrate utils to typed Pydantic models Signed-off-by: tariq-hasan <[email protected]> * chore(spark): migrate backend to typed Pydantic models Signed-off-by: tariq-hasan <[email protected]> * chore(spark): refactor tests to use typed models and cleanup Signed-off-by: tariq-hasan <[email protected]> * chore(spark): rename build_spark_connect_crd to build_spark_connect_cr Signed-off-by: tariq-hasan <[email protected]> * fix(spark): use typed model helpers in mock handlers Signed-off-by: tariq-hasan <[email protected]> * chore(spark): bump kubeflow-spark-api to 2.4.0 Signed-off-by: tariq-hasan <[email protected]> --------- Signed-off-by: tariq-hasan <[email protected]> * feat(docs): Update README with Spark Support (kubeflow#349) Signed-off-by: Andrey Velichkevich <[email protected]> * fix(trainer): return TRAINJOB_COMPLETE when all steps are done (kubeflow#340) * fix(local): return TRAINJOB_COMPLETE when all steps are done (kubeflow#338) Signed-off-by: priyank <[email protected]> * test(trainer): add test case for __get_job_status Signed-off-by: priyank <[email protected]> * fix(trainer): early return TRAINJOB_CREATED when job has no steps Signed-off-by: priyank <[email protected]> * test(trainer): refactor test_get_job_status with TestCase fixture Signed-off-by: priyank <[email protected]> --------- Signed-off-by: priyank <[email protected]> * fix(trainer): adapt SDK to removal of numProcPerNode from TorchMLPolicySource (kubeflow#360) Signed-off-by: tariq-hasan <[email protected]> * fix: Make validate-lockfile action non-blocking (kubeflow#361) Signed-off-by: Fiona-Waters <[email protected]> * chore(spark): change pyspark[connect] dependency (kubeflow#357) Change pyspark[connect] 3.4.1 dependency to pyspark-connect 4.0.1. This matches the version of Spark in the spark-operator container image (https://github.com/kubeflow/spark-operator/blob/master/Dockerfile#L17). Signed-off-by: Ali Maredia <[email protected]> * chore(spark): remove SDK-side validation from SparkClient (kubeflow#345) Remove all SDK-side input validation from the spark module. Validation will be handled server-side by the Spark Operator admission webhooks (spark-operator#2862). - Remove validation.py and validation_test.py - Remove isinstance checks from _create_session() - Remove ValidationError from public API Closes: kubeflow#272 Signed-off-by: Yassin Nouh <[email protected]> Signed-off-by: yassinnouh21 <[email protected]> * chore: Merge upstream/main (preserving downstream config) Signed-off-by: Brian Gallagher <[email protected]> * update workflow to skip requirements generation on merge conflict Signed-off-by: Brian Gallagher <[email protected]> * remove compatibility with python 3.9 and udpated tests Signed-off-by: Brian Gallagher <[email protected]> * fix tests Signed-off-by: Brian Gallagher <[email protected]> --------- Signed-off-by: Jon Burdo <[email protected]> Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> Signed-off-by: Akash Jaiswal <[email protected]> Signed-off-by: HKanoje <[email protected]> Signed-off-by: Shekhar Rajak <[email protected]> Signed-off-by: Shekhar Prasad Rajak <[email protected]> Signed-off-by: Fiona-Waters <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: osamaahmed17 <[email protected]> Signed-off-by: Osama Tahir <[email protected]> Signed-off-by: muhammadjunaid8047 <[email protected]> Signed-off-by: Muhammad Junaid <[email protected]> Signed-off-by: Moeed Shaik <[email protected]> Signed-off-by: Moeed <[email protected]> Signed-off-by: XploY04 <[email protected]> Signed-off-by: Brian Gallagher <[email protected]> Signed-off-by: krishdef7 <[email protected]> Signed-off-by: ruskaruma <[email protected]> Signed-off-by: Mansi Singh <[email protected]> Signed-off-by: Transcendental-Programmer <[email protected]> Signed-off-by: digvijay-y <[email protected]> Signed-off-by: Ruskaruma <[email protected]> Signed-off-by: tariq-hasan <[email protected]> Signed-off-by: Andrey Velichkevich <[email protected]> Signed-off-by: priyank <[email protected]> Signed-off-by: Ali Maredia <[email protected]> Signed-off-by: Yassin Nouh <[email protected]> Signed-off-by: yassinnouh21 <[email protected]> Co-authored-by: Jon Burdo <[email protected]> Co-authored-by: Surya Sameer Datta Vaddadi <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Andrey Velichkevich <[email protected]> Co-authored-by: Akash Jaiswal <[email protected]> Co-authored-by: Hrithik Kanoje <[email protected]> Co-authored-by: Shekhar Prasad Rajak <[email protected]> Co-authored-by: Fiona Waters <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Osama Tahir <[email protected]> Co-authored-by: Muhammad Junaid <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: google-oss-prow <[email protected]> Co-authored-by: Moeed <[email protected]> Co-authored-by: Yash Agarwal <[email protected]> Co-authored-by: krishdef7 <[email protected]> Co-authored-by: Ruskaruma <[email protected]> Co-authored-by: Mansi Singh <[email protected]> Co-authored-by: Priyansh Saxena <[email protected]> Co-authored-by: DIGVIJAY <[email protected]> Co-authored-by: Tariq Hasan <[email protected]> Co-authored-by: Priyank Patel <[email protected]> Co-authored-by: Ali Maredia <[email protected]> Co-authored-by: Yassin Nouh <[email protected]>
* chore!: upgrade to Python 3.10 (#282) This upgrades the minimum Python version for the project from 3.9 to 3.10. Python 3.9 is past end-of-life and dependencies will likely require a supported version soon. Signed-off-by: Jon Burdo <[email protected]> * chore: Confirm that a public ConfigMap exists to check version (#250) * Confirm that a public ConfigMap exists to check version Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * python 3.9 fix Co-authored-by: Copilot <[email protected]> Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * Exceptiom handling better Co-authored-by: Copilot <[email protected]> Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * Addressing comments Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * Update kubeflow/trainer/backends/kubernetes/backend.py Co-authored-by: Andrey Velichkevich <[email protected]> Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * Refactored tests into a single function and followed agents.md Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * CI friendly edit Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * pre-commit format checked Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * Modified according to new updates Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * Ran pre-commit locally to fix formatting Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * unix2dos CLAUDE.md Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> * Revert CLAUDE.md Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> --------- Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> Signed-off-by: Surya Sameer Datta Vaddadi <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Andrey Velichkevich <[email protected]> * chore: added sdk docs website to readme (#284) * docs: added sdk docs website to readme Signed-off-by: Akash Jaiswal <[email protected]> * format: order of sdk docs Signed-off-by: Akash Jaiswal <[email protected]> --------- Signed-off-by: Akash Jaiswal <[email protected]> * feat(trainer): add dataset and model initializer support to container backend (#188) * feat(trainer): add dataset and model initializer support to container backend Add support for dataset and model initializers in the container backend to bring it to feature parity with the Kubernetes backend. Changes: - Add utility functions for building initializer commands and environment variables - Implement _run_initializers() and _run_single_initializer() methods in ContainerBackend - Run initializers sequentially before training containers start - Download datasets to /workspace/dataset and models to /workspace/model - Track initializer containers as separate steps in TrainJob - Support all initializer types: HuggingFace, S3, and DataCache - Add comprehensive unit tests for all initializer configurations - Handle initializer failures with proper cleanup and error messages Fixes #171 Signed-off-by: HKanoje <[email protected]> * feat(trainer): address reviewer feedback for initializer support - Make initializer image configurable via ContainerBackendConfig - Make initializer timeout configurable (default 600 seconds) - Implement wait API in adapters instead of polling - Clean up successful initializer containers after completion - Clean up network on initializer failure - Raise ValueError for unsupported initializer types (no datacache fallback) All tests passing (173/173). Addresses all feedback from PR #188. Signed-off-by: HKanoje <[email protected]> * chore(trainer): add cleanup helper to reduce duplication Add _cleanup_container_resources() helper method to consolidate duplicated cleanup logic for stopping/removing containers and deleting networks. Refactor 5 locations across train(), initializer handlers, and delete_job() to use this helper. Signed-off-by: HKanoje <[email protected]> * fix(trainer): use correct initializer images and working directory Address feedback for initializer support in container backend: - Use separate images for dataset/model initializers: - kubeflow/dataset-initializer:latest for datasets - kubeflow/model-initializer:latest for models (instead of kubeflow/training-operator:latest) - Update python commands to use pkg.initializers module: - python -m pkg.initializers.dataset (for dataset) - python -m pkg.initializers.model (for model) - Change initializer working_dir from /workspace to /app per Dockerfile convention Refs: https://github.com/kubeflow/trainer/tree/master/cmd/initializers Signed-off-by: HKanoje <[email protected]> * fix(container): address PR review comments for initializer support - Use GHCR images as default for dataset/model initializers - Replace suppress with try-except blocks - Refactor initializer utils with ContainerInitializer dataclass - Add get_dataset_initializer and get_model_initializer functions - Remove DataCache support (unsupported in container backend) - Merge initializer tests into test_train() and test_get_job_logs() - Remove duplicate test functions Signed-off-by: HKanoje <[email protected]> * fix(container): add name field to ContainerInitializer and remove init_type - Add name field to ContainerInitializer dataclass - Set name='dataset-initializer' and name='model-initializer' in utils - Remove init_type parameter from _run_single_initializer() - Use container_init.name for labels and log messages Signed-off-by: HKanoje <[email protected]> --------- Signed-off-by: HKanoje <[email protected]> * feat: add SparkClient API for SparkConnect session management (#225) * feat(spark): add core types, dataclasses, and constants - Add SparkConnectInfo, SparkConnectState, Driver, Executor types - Add type tests for validation - Add Kubernetes backend constants (CRD group, version, defaults) Signed-off-by: Shekhar Rajak <[email protected]> * feat(spark): add backend base class and options pattern - Add RuntimeBackend abstract base class with session lifecycle methods - Add options pattern (Name, Image, Timeout, etc.) aligned with trainer SDK - Add validation utilities for connect parameters - Add comprehensive option tests Signed-off-by: Shekhar Rajak <[email protected]> * feat(spark): add KubernetesBackend for SparkConnect CRD operations - Implement KubernetesBackend with create/get/list/delete session methods - Add port-forward support for out-of-cluster connections - Add CRD builder utilities and URL validation - Add comprehensive backend and utils tests with parametrized patterns Signed-off-by: Shekhar Rajak <[email protected]> * feat(spark): add SparkClient API with KEP-107 compliant connect method - Implement SparkClient as main user interface for SparkConnect sessions - Support connect to existing server (base_url) or auto-create new session - Add public exports for SparkClient, Driver, Executor, options - Add SparkClient unit tests Signed-off-by: Shekhar Rajak <[email protected]> * chore(spark): add test infrastructure and package init files - Add test common utilities and fixtures - Add package __init__ files for test directories - Setup test/e2e/spark structure Signed-off-by: Shekhar Rajak <[email protected]> * feat(spark): add example scripts demonstrating SparkClient usage - Add spark_connect_simple.py with 3 usage levels (minimal, simple, advanced) - Add spark_advanced_options.py with full configuration examples - Add connect_existing_session.py for connecting to existing servers - Add demo and test scripts for local development Signed-off-by: Shekhar Rajak <[email protected]> * docs(spark): add documentation for SparkClient and E2E testing - Add examples/spark/README.md with usage guide - Add local Spark Connect testing documentation - Add E2E test README with CI/CD integration guide - Update KEP-107 proposal documentation Signed-off-by: Shekhar Rajak <[email protected]> * test(spark): add E2E test framework with cluster watcher - Add test_spark_examples.py with example validation tests - Add cluster_watcher.py for monitoring SparkConnect and pods during tests - Add run_in_cluster.py for executing examples as K8s Jobs Signed-off-by: Shekhar Rajak <[email protected]> * ci(spark): add GitHub Actions workflow and E2E cluster setup - Add test-spark-examples.yaml workflow for E2E validation - Add e2e-setup-cluster.sh for Kind cluster with Spark Operator - Add SparkConnect CRD, Kind config, and E2E runner Dockerfile - Update Makefile with E2E setup target - Update PR title check for spark prefix Signed-off-by: Shekhar Rajak <[email protected]> * chore(spark): add pyspark[connect] dependency and update lock file - Add spark extra with pyspark[connect]==3.4.1 for grpcio, pandas, pyarrow - Update uv.lock with resolved dependencies - Update .gitignore for spark-related files Signed-off-by: Shekhar Rajak <[email protected]> * Update kubeflow/spark/backends/base.py Co-authored-by: Andrey Velichkevich <[email protected]> Signed-off-by: Shekhar Prasad Rajak <[email protected]> * refactor(spark): rename backend.connect_session() to connect() Signed-off-by: Shekhar Rajak <[email protected]> * refactor: move session creation flow from SparkClient to backend.create_and_connect() Signed-off-by: Shekhar Rajak <[email protected]> --------- Signed-off-by: Shekhar Rajak <[email protected]> Signed-off-by: Shekhar Prasad Rajak <[email protected]> Co-authored-by: Andrey Velichkevich <[email protected]> * chore: bump minimum model-registry version to 0.3.6 (#289) Signed-off-by: Jon Burdo <[email protected]> * fix: Improve CVE workflow (#267) * fix: Improve CVE workflow Signed-off-by: Fiona-Waters <[email protected]> * fix: fix issue with bash compare Signed-off-by: Fiona-Waters <[email protected]> * feat: Add workflow to cleanup overrides in pyproject.toml Signed-off-by: Fiona-Waters <[email protected]> * fix: address review comments Signed-off-by: Fiona-Waters <[email protected]> * chore: refactor to reduce size of cve related workflows Signed-off-by: Fiona-Waters <[email protected]> --------- Signed-off-by: Fiona-Waters <[email protected]> * chore: upgrade code style for python3.10 (#288) * chore: update code style for Python 3.10 This disables a couple ruff rules in pyproject.toml: ``` "UP007", # Use X | Y instead of Union[X, Y] (requires Python 3.10+) "UP045", # Use X | None instead of Optional[X] (requires Python 3.10+) ``` Then the code changes are made with: ``` uv run ruff check --fix uv run ruff format ``` Signed-off-by: Jon Burdo <[email protected]> * fix: handle unions, bools in convert_value The convert_value function didn't seems to be handling union types properly and also needs to handle `T | None` similarly to `Optional[None]` after the upgrade to Python 3.10. This fixes union types, an issue with bool conversion, and adds tests for this function. Signed-off-by: Jon Burdo <[email protected]> --------- Signed-off-by: Jon Burdo <[email protected]> * chore(ci): bump astral-sh/setup-uv from 5 to 7 (#276) Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 5 to 7. - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](https://github.com/astral-sh/setup-uv/compare/v5...v7) --- updated-dependencies: - dependency-name: astral-sh/setup-uv dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump the python-minor group across 1 directory with 4 updates (#291) Bumps the python-minor group with 4 updates in the / directory: [coverage](https://github.com/coveragepy/coveragepy), [ruff](https://github.com/astral-sh/ruff), [pre-commit](https://github.com/pre-commit/pre-commit) and [ty](https://github.com/astral-sh/ty). Updates `coverage` from 7.10.7 to 7.13.4 - [Release notes](https://github.com/coveragepy/coveragepy/releases) - [Changelog](https://github.com/coveragepy/coveragepy/blob/main/CHANGES.rst) - [Commits](https://github.com/coveragepy/coveragepy/compare/7.10.7...7.13.4) Updates `ruff` from 0.14.14 to 0.15.0 - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/ruff/compare/0.14.14...0.15.0) Updates `pre-commit` from 4.3.0 to 4.5.1 - [Release notes](https://github.com/pre-commit/pre-commit/releases) - [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md) - [Commits](https://github.com/pre-commit/pre-commit/compare/v4.3.0...v4.5.1) Updates `ty` from 0.0.14 to 0.0.16 - [Release notes](https://github.com/astral-sh/ty/releases) - [Changelog](https://github.com/astral-sh/ty/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/ty/compare/0.0.14...0.0.16) --- updated-dependencies: - dependency-name: coverage dependency-version: 7.13.4 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-minor - dependency-name: ruff dependency-version: 0.15.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-minor - dependency-name: pre-commit dependency-version: 4.5.1 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-minor - dependency-name: ty dependency-version: 0.0.16 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: Added examples to the documentation demonstrating different ways to handle ports (#243) * update docs and add test cases. Signed-off-by: osamaahmed17 <[email protected]> * pre-commit error solved Signed-off-by: osamaahmed17 <[email protected]> * Update kubeflow/hub/api/model_registry_client.py Co-authored-by: Jon Burdo <[email protected]> Signed-off-by: Osama Tahir <[email protected]> * readme updated Signed-off-by: Osama Tahir <[email protected]> * Refactor model registry client test cases for clarity Signed-off-by: Osama Tahir <[email protected]> --------- Signed-off-by: osamaahmed17 <[email protected]> Signed-off-by: Osama Tahir <[email protected]> Co-authored-by: Jon Burdo <[email protected]> * chore(ci): bump peter-evans/create-pull-request from 6 to 8 (#277) Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 6 to 8. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](https://github.com/peter-evans/create-pull-request/compare/v6...v8) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-version: '8' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(ci): bump actions/checkout from 4 to 6 (#278) Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v4...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: Adds a GitHub Actions workflow to check kubeflow/hub/OWNERS. (#280) * Add OWNERS validation Signed-off-by: muhammadjunaid8047 <[email protected]> * Update .github/workflows/check-owners.yaml Co-authored-by: Jon Burdo <[email protected]> Signed-off-by: Muhammad Junaid <[email protected]> * Update OWNERS file check in workflow Signed-off-by: Muhammad Junaid <[email protected]> * Update paths in check-owners workflow Signed-off-by: Muhammad Junaid <[email protected]> --------- Signed-off-by: muhammadjunaid8047 <[email protected]> Signed-off-by: Muhammad Junaid <[email protected]> Co-authored-by: Jon Burdo <[email protected]> * fix: nightly security dependency updates (#296) Co-authored-by: google-oss-prow <[email protected]> * chore(ci): bump aquasecurity/trivy-action from 0.33.1 to 0.34.0 in the actions group (#297) Bumps the actions group with 1 update: [aquasecurity/trivy-action](https://github.com/aquasecurity/trivy-action). Updates `aquasecurity/trivy-action` from 0.33.1 to 0.34.0 - [Release notes](https://github.com/aquasecurity/trivy-action/releases) - [Commits](https://github.com/aquasecurity/trivy-action/compare/0.33.1...0.34.0) --- updated-dependencies: - dependency-name: aquasecurity/trivy-action dependency-version: 0.34.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump pytest from 8.4.2 to 9.0.2 (#301) Bumps [pytest](https://github.com/pytest-dev/pytest) from 8.4.2 to 9.0.2. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pytest-dev/pytest/compare/8.4.2...9.0.2) --- updated-dependencies: - dependency-name: pytest dependency-version: 9.0.2 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(trainer): Support namespaced TrainingRuntime in the SDK (#130) * feat(backend): Support namespaced TrainingRuntime in the SDK Signed-off-by: Moeed Shaik <[email protected]> * Fixed bugs and validated current test cases Signed-off-by: Moeed Shaik <[email protected]> * Fixed pre-commit test failure Signed-off-by: Moeed Shaik <[email protected]> * Addressed comments Signed-off-by: Moeed Shaik <[email protected]> * Fixed no attribute 'DEFAULT_TIMEOUT' error Signed-off-by: Moeed Shaik <[email protected]> * Added namespace-scoped runtime to test cases Signed-off-by: Moeed Shaik <[email protected]> * Addressed fallback logic bugs Signed-off-by: Moeed Shaik <[email protected]> * Added scope field to Runtime Signed-off-by: Moeed Shaik <[email protected]> * Improved code Signed-off-by: Moeed Shaik <[email protected]> * Fixed copilot's comments Signed-off-by: Moeed Shaik <[email protected]> * Shadow duplicate runtimes, priority to ns Signed-off-by: Moeed Shaik <[email protected]> * Fixed bug Signed-off-by: Moeed Shaik <[email protected]> * Fixed copilot comments Signed-off-by: Moeed Shaik <[email protected]> * Improved test cases to validate all possible cases Signed-off-by: Moeed Shaik <[email protected]> * small fix Signed-off-by: Moeed Shaik <[email protected]> * lint fix Signed-off-by: Moeed Shaik <[email protected]> * improved error message Co-authored-by: Copilot <[email protected]> Signed-off-by: Moeed <[email protected]> * refactored code Signed-off-by: Moeed Shaik <[email protected]> * improve code Signed-off-by: Moeed Shaik <[email protected]> * Removed RuntimeScope Signed-off-by: Moeed Shaik <[email protected]> * removed scope references and improved error handling as per kubeflow standards Signed-off-by: Moeed Shaik <[email protected]> --------- Signed-off-by: Moeed Shaik <[email protected]> Signed-off-by: Moeed <[email protected]> Co-authored-by: Copilot <[email protected]> * fix: Fix runtime lookup fallback and test local SDK in E2E (#307) * fix: Install SDK locally in E2E workflow and improve error handling for runtime fetching in Kubernetes backend. Signed-off-by: XploY04 <[email protected]> * refactor: Explicitly return errors from and refine exception handling in . Signed-off-by: XploY04 <[email protected]> * docs: update comment to clarify Kubeflow SDK installation from source in e2e workflow. Signed-off-by: XploY04 <[email protected]> * feat: Enhance runtime retrieval tests to cover Kubernetes API 404/403 errors and partial success for list operations on timeout. Signed-off-by: XploY04 <[email protected]> * refactor: Update runtime listing to immediately raise exceptions on failure instead of collecting partial results. Signed-off-by: XploY04 <[email protected]> --------- Signed-off-by: XploY04 <[email protected]> * chore(ci): bump actions/setup-python from 5 to 6 (#298) Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5 to 6. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v5...v6) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump the python-minor group with 2 updates (#299) Bumps the python-minor group with 2 updates: [ruff](https://github.com/astral-sh/ruff) and [ty](https://github.com/astral-sh/ty). Updates `ruff` from 0.15.0 to 0.15.1 - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/ruff/compare/0.15.0...0.15.1) Updates `ty` from 0.0.16 to 0.0.17 - [Release notes](https://github.com/astral-sh/ty/releases) - [Changelog](https://github.com/astral-sh/ty/blob/main/CHANGELOG.md) - [Commits](https://github.com/astral-sh/ty/compare/0.0.16...0.0.17) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.15.1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-minor - dependency-name: ty dependency-version: 0.0.17 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: improve logging around packages_to_install (#269) * improve logging around packages_to_install Signed-off-by: Brian Gallagher <[email protected]> * exit when pip install fails, append errors from both attempts Signed-off-by: Brian Gallagher <[email protected]> * Add shlex to address command injection vulnerabilities. Write pip install logfile to cwd Signed-off-by: Brian Gallagher <[email protected]> --------- Signed-off-by: Brian Gallagher <[email protected]> * feat: Add validate lockfile workflow to complement CVE scanning (#306) * feat: Add validate lockfile workflow to complement CVE scanning Signed-off-by: Fiona-Waters <[email protected]> * fix: make cve fix pr branch static Signed-off-by: Fiona-Waters <[email protected]> --------- Signed-off-by: Fiona-Waters <[email protected]> * fix(trainer): handle falsy values in get_args_from_peft_config (#328) * fix(trainer): handle falsy values in get_args_from_peft_config Signed-off-by: krishdef7 <[email protected]> * fix: apply pre-commit formatting Signed-off-by: krishdef7 <[email protected]> * fix: also handle falsy train_on_input in dataset_preprocess_config Signed-off-by: krishdef7 <[email protected]> * fix: add missing newline at end of utils_test.py Signed-off-by: krishdef7 <[email protected]> * fix: pre-commit formatting Signed-off-by: krishdef7 <[email protected]> --------- Signed-off-by: krishdef7 <[email protected]> * fix(optimizer): prevent input mutation in optimize() (#322) * fix(optimizer): prevent input mutation in optimize() Signed-off-by: ruskaruma <[email protected]> * remove unnecessary things Signed-off-by: ruskaruma <[email protected]> * rename test Signed-off-by: ruskaruma <[email protected]> --------- Signed-off-by: ruskaruma <[email protected]> * feat: add TrainerClient examples for local PyTorch distributed training (#312) * docs: add TrainerClient examples for local PyTorch distributed training - Add examples/trainer/pytorch_distributed_simple.py - Add examples/trainer/README.md - Demonstrates LocalProcessBackend usage without Kubernetes - Fixes #218 Signed-off-by: Mansi Singh <[email protected]> * docs: add training examples table to SDK website Signed-off-by: Mansi Singh <[email protected]> * docs: expand examples table with PyTorch, MLX, DeepSpeed, and TorchTune examples grouped by framework Signed-off-by: Mansi Singh <[email protected]> --------- Signed-off-by: Mansi Singh <[email protected]> * chore: fix docstrings in TrainerClient (#333) Signed-off-by: Transcendental-Programmer <[email protected]> * feat(spark): Refactor unit tests to sdk coding standards (#293) * Refactored unit test Signed-off-by: digvijay-y <[email protected]> * Changes made Signed-off-by: digvijay-y <[email protected]> * Version Signed-off-by: digvijay-y <[email protected]> * Restructured clien_test Signed-off-by: digvijay-y <[email protected]> * reformated backend_test.py Signed-off-by: digvijay-y <[email protected]> * revert pyproject.toml and uv.lock changes Signed-off-by: digvijay-y <[email protected]> * Standarized spark backend tests Signed-off-by: digvijay-y <[email protected]> * backend_tests Signed-off-by: digvijay-y <[email protected]> --------- Signed-off-by: digvijay-y <[email protected]> * fix(optimizer): add missing get_job_events() to RuntimeBackend base c… (#325) * fix(optimizer): add missing get_job_events() to RuntimeBackend base class Signed-off-by: ruskaruma <[email protected]> * Update kubeflow/optimizer/backends/base.py Co-authored-by: Copilot <[email protected]> Signed-off-by: Ruskaruma <[email protected]> * Update kubeflow/optimizer/backends/base.py Co-authored-by: Andrey Velichkevich <[email protected]> Signed-off-by: Ruskaruma <[email protected]> * fix: add abstractmethod, remove docstrings Signed-off-by: ruskaruma <[email protected]> * make get_job_events abstract in RuntimeBackend Signed-off-by: ruskaruma <[email protected]> * Update kubeflow/trainer/backends/localprocess/backend.py Co-authored-by: Andrey Velichkevich <[email protected]> Signed-off-by: Ruskaruma <[email protected]> * fix Signed-off-by: ruskaruma <[email protected]> --------- Signed-off-by: ruskaruma <[email protected]> Signed-off-by: Ruskaruma <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Andrey Velichkevich <[email protected]> * chore(spark): migrate SDK to kubeflow_spark_api Pydantic models (#295) * chore(spark): add kubeflow-spark-api dependency Signed-off-by: tariq-hasan <[email protected]> * chore(spark): migrate options to typed Pydantic models Signed-off-by: tariq-hasan <[email protected]> * chore(spark): migrate utils to typed Pydantic models Signed-off-by: tariq-hasan <[email protected]> * chore(spark): migrate backend to typed Pydantic models Signed-off-by: tariq-hasan <[email protected]> * chore(spark): refactor tests to use typed models and cleanup Signed-off-by: tariq-hasan <[email protected]> * chore(spark): rename build_spark_connect_crd to build_spark_connect_cr Signed-off-by: tariq-hasan <[email protected]> * fix(spark): use typed model helpers in mock handlers Signed-off-by: tariq-hasan <[email protected]> * chore(spark): bump kubeflow-spark-api to 2.4.0 Signed-off-by: tariq-hasan <[email protected]> --------- Signed-off-by: tariq-hasan <[email protected]> * feat(docs): Update README with Spark Support (#349) Signed-off-by: Andrey Velichkevich <[email protected]> * fix(trainer): return TRAINJOB_COMPLETE when all steps are done (#340) * fix(local): return TRAINJOB_COMPLETE when all steps are done (#338) Signed-off-by: priyank <[email protected]> * test(trainer): add test case for __get_job_status Signed-off-by: priyank <[email protected]> * fix(trainer): early return TRAINJOB_CREATED when job has no steps Signed-off-by: priyank <[email protected]> * test(trainer): refactor test_get_job_status with TestCase fixture Signed-off-by: priyank <[email protected]> --------- Signed-off-by: priyank <[email protected]> * fix(trainer): adapt SDK to removal of numProcPerNode from TorchMLPolicySource (#360) Signed-off-by: tariq-hasan <[email protected]> * fix: Make validate-lockfile action non-blocking (#361) Signed-off-by: Fiona-Waters <[email protected]> * chore(spark): change pyspark[connect] dependency (#357) Change pyspark[connect] 3.4.1 dependency to pyspark-connect 4.0.1. This matches the version of Spark in the spark-operator container image (https://github.com/kubeflow/spark-operator/blob/master/Dockerfile#L17). Signed-off-by: Ali Maredia <[email protected]> * chore(spark): remove SDK-side validation from SparkClient (#345) Remove all SDK-side input validation from the spark module. Validation will be handled server-side by the Spark Operator admission webhooks (spark-operator#2862). - Remove validation.py and validation_test.py - Remove isinstance checks from _create_session() - Remove ValidationError from public API Closes: #272 Signed-off-by: Yassin Nouh <[email protected]> Signed-off-by: yassinnouh21 <[email protected]> * chore(spark): add Spark documentation and API reference (#364) * docs(spark): add Spark documentation and API reference Signed-off-by: Amir380-A <[email protected]> * chore(spark): add spark installation section Signed-off-by: Amir380-A <[email protected]> * chore(docs): apply pre-commit formatting fixes Signed-off-by: Amir380-A <[email protected]> * chore(docs): apply changes Signed-off-by: Amir380-A <[email protected]> * Update docs/source/spark/index.rst Co-authored-by: Andrey Velichkevich <[email protected]> Signed-off-by: Amir Ibrahim <[email protected]> --------- Signed-off-by: Amir380-A <[email protected]> Signed-off-by: Amir Ibrahim <[email protected]> Co-authored-by: Andrey Velichkevich <[email protected]> * fix(trainer): ignore PEP 668 system python check (#384) Signed-off-by: Rob Bell <[email protected]> * feat(trainer): replace PodTemplateOverrides with RuntimePatches API (#381) * feat(trainer): replace PodTemplateOverrides with RuntimePatches API Signed-off-by: Fiona-Waters <[email protected]> * fix: address review comments Signed-off-by: Fiona-Waters <[email protected]> --------- Signed-off-by: Fiona-Waters <[email protected]> * fix(trainer): Fix packages installation with extra notation (#385) * fix(trainer): Fix packages installation with extra notation Signed-off-by: Andrey Velichkevich <[email protected]> * Update pip index URL format Signed-off-by: Andrey Velichkevich <[email protected]> --------- Signed-off-by: Andrey Velichkevich <[email protected]> * feat: Run dataset and model initializers in parallel (#313) * feat(trainer): Run dataset and model initializers in parallel (#290) Signed-off-by: priyank <[email protected]> * feat(container): refactor initializer dispatch into a loop Signed-off-by: priyank <[email protected]> * style(container): fix ruff formatting Signed-off-by: priyank <[email protected]> --------- Signed-off-by: priyank <[email protected]> * fix(trainer): add missing wildcard to .pt and .pth ignore patterns (#372) * fix(trainer): add missing wildcard to .pt and .pth ignore patterns Signed-off-by: Vahagn <[email protected]> * docs(trainer): fix wildcard patterns in S3ModelInitializer docstring Signed-off-by: ghazariann <[email protected]> --------- Signed-off-by: Vahagn <[email protected]> Signed-off-by: ghazariann <[email protected]> Co-authored-by: Vahagn <[email protected]> * chore(trainer): fix typos in TrainerClient docstrings (#394) Fix grammatical error in list_jobs returns description ("TrainJob exist" -> "TrainJobs exist") and add missing period in get_job docstring. Signed-off-by: andres75125 <[email protected]> * chore(trainer): Add API reference docs for kubeflow.trainer.options classes (#396) Signed-off-by: Fiona-Waters <[email protected]> * fix: support RC version format in Makefile release target (#398) Signed-off-by: Fiona-Waters <[email protected]> * Kubeflow SDK Official Release 0.4.0rc0 (#397) Signed-off-by: Fiona-Waters <[email protected]> * chore(ci): bump aquasecurity/trivy-action from 0.34.0 to 0.34.1 in the actions group (#319) Bumps the actions group with 1 update: [aquasecurity/trivy-action](https://github.com/aquasecurity/trivy-action). Updates `aquasecurity/trivy-action` from 0.34.0 to 0.34.1 - [Release notes](https://github.com/aquasecurity/trivy-action/releases) - [Commits](https://github.com/aquasecurity/trivy-action/compare/0.34.0...0.34.1) --- updated-dependencies: - dependency-name: aquasecurity/trivy-action dependency-version: 0.34.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): Bump Kubeflow Trainer API to 2.2.0 (#406) Signed-off-by: Andrey Velichkevich <[email protected]> * Kubeflow SDK Official Release 0.4.0 (#404) Signed-off-by: Andrey Velichkevich <[email protected]> * chore(docs): Add Apache Spark to the diagram (#425) Signed-off-by: Andrey Velichkevich <[email protected]> * fix: remove validate-lockfile workflow as it uses trivy (#427) Signed-off-by: Fiona-Waters <[email protected]> * chore(ci): Remove Trivy Action (#426) Signed-off-by: Andrey Velichkevich <[email protected]> * feat(docs): Add Kubeflow SDK 0.4 blog to the latest news (#428) Signed-off-by: Andrey Velichkevich <[email protected]> * fix: change documentation typo (#449) * fix: change documentation typo Signed-off-by: Slowlybomb <[email protected]> * fix: path in a comment Signed-off-by: Slowlybomb <[email protected]> --------- Signed-off-by: Slowlybomb <[email protected]> * chore(trainer): add data and model initializers guide (#414) * chore(trainer): add data and model initializers guide Add docs/source/train/initializers.rst covering dataset and model initializers for the container backend (added in #188, parallelised in #313). Includes per-type code examples, combined usage, ContainerBackendConfig options, and debugging via get_job_logs(). Signed-off-by: Ayush Petwal <[email protected]> * docs(trainer): address PR review comments on initializers guide - Fix model output path from /workspace/model-weights to /workspace/model to match the MODEL_PATH constant in constants.py - Clarify DataCacheInitializer is Kubernetes-only in the backend note and annotate the Available Initializers table row accordingly - Add DataCacheInitializer usage example with required fields (storage_uri, metadata_loc, num_data_nodes) and backend constraint note Signed-off-by: 1Ayush-Petwal <[email protected]> * docs(trainer): use latest tag for initializer images in guide Signed-off-by: 1Ayush-Petwal <[email protected]> --------- Signed-off-by: Ayush Petwal <[email protected]> Signed-off-by: 1Ayush-Petwal <[email protected]> * fix(trainer): improve error diagnostics and add retry logic (#451) * fix(trainer): improve error diagnostics and add retry logic Signed-off-by: Fiona-Waters <[email protected]> * chore: ensure consistent message, move test to util file Signed-off-by: Fiona-Waters <[email protected]> --------- Signed-off-by: Fiona-Waters <[email protected]> * fix(hub): add params to register_model() (#440) * fix(hub): add storage_key, storage_path, service_account_name to register_model Signed-off-by: 1Ayush-Petwal <[email protected]> * refactor(hub): group storage args into StorageConfig dataclass Replace the three flat kwargs (storage_key, storage_path, service_account_name) on register_model() with a single storage_config: StorageConfig | None = None parameter. Introduces kubeflow/hub/types/types.py with the StorageConfig dataclass (matching the trainer/optimizer/spark types/ pattern), re-exports it from kubeflow.hub, updates the docstring, and adds parametrized test cases verifying correct field forwarding. Signed-off-by: 1Ayush-Petwal <[email protected]> --------- Signed-off-by: 1Ayush-Petwal <[email protected]> * fix(trainer): validate polling_interval is strictly less than timeout (#402) * fix(trainer): validate polling_interval is strictly less than timeout In wait_for_job_status(), the guard used strict greater-than (>), allowing polling_interval == timeout to pass validation. This results in round(timeout/polling_interval) == 1, meaning the job is polled exactly once with no retry window — silently wrong behavior that contradicts the documented constraint 'must be less than timeout'. Fix: change > to >= in the validation guard. Fixes #400 Signed-off-by: Prabinder Singh <[email protected]> * fix(trainer): address review comments on polling_interval validation - Add guard for zero/negative polling_interval to prevent ZeroDivisionError at round(timeout / polling_interval); raise ValueError with clear message - Improve ValueError message for polling_interval >= timeout to include actual parameter values for easier debugging - Replace hardcoded "basic-job" string with BASIC_TRAIN_JOB_NAME constant in the new test case to match existing conventions - Add test cases for zero and negative polling_interval values Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: Prabinder Singh <[email protected]> * fix(trainer): extend polling_interval validation to all backends Apply same polling_interval validation to localprocess and optimizer backends as requested in PR review. - Add polling_interval <= 0 guard to prevent ZeroDivisionError - Change > to >= so equal values are also rejected (strictly less than) - Improve ValueError messages to include actual parameter values - Add parametrized test cases for all three validation scenarios (greater than, equal to, zero, negative) in both backend test files Co-Authored-By: Claude Sonnet 4.6 <[email protected]> Signed-off-by: Prabinder Singh <[email protected]> --------- Signed-off-by: Prabinder Singh <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]> * chore: Kubeflow SDK ROADMAP for 2026 (#326) Signed-off-by: kramaranya <[email protected]> * chore: update AGENTS.md with lint commands and current repo map (#484) - Add single-file lint/format/type-check commands for faster agent feedback loops - Add kubeflow/spark module to repo map (SparkClient, kubernetes backend) - Add kubeflow/hub/types and kubeflow/trainer/test directories to repo map - Condense development principles to concise rules (remove verbose code examples) - Fix typo in context awareness section Signed-off-by: kramaranya <[email protected]> Co-authored-by: Claude <[email protected]> * feat: add kfp-client wrapper proposal (#343) * feat: add kfp-client wrapper proposal Signed-off-by: MStokluska <[email protected]> * fix: address copilot comments Signed-off-by: MStokluska <[email protected]> * docs(proposals): refine KEP-125 PipelinesClient story split and run semantics Signed-off-by: MStokluska <[email protected]> * docs(proposals): remove trailing whitespace in KEP-125 README Signed-off-by: MStokluska <[email protected]> * docs(proposals): shorten the proposal and add kep.yaml Signed-off-by: MStokluska <[email protected]> * docs(proposal): address comments and upstream agreements Signed-off-by: MStokluska <[email protected]> * docs(proposal): address further comments Signed-off-by: MStokluska <[email protected]> * docs(proposal): address further comments regarding get_pipeline_version Signed-off-by: MStokluska <[email protected]> --------- Signed-off-by: MStokluska <[email protected]> * chore: rename lifecycle/needs-triage to needs-triage in issue templates (#487) Signed-off-by: abhijeet-dhumal <[email protected]> * feat(docs): add AI policy reference to contributing guide (#488) Direct contributors to review the Kubeflow AI Policy before using AI agents, aligning with the same note added to kubeflow/trainer. Signed-off-by: Andrey Velichkevich <[email protected]> Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> * chore(optimizer): add comprehensive unit tests for KubernetesBackend (#485) Cover all public methods of the optimizer's KubernetesBackend with parametrized unit tests mirroring the trainer backend test structure. Tests added for: optimize, get_job, list_jobs, get_job_logs, get_best_results, wait_for_job_status, delete_job, get_job_events. Each method tested with success paths and error scenarios (TimeoutError, RuntimeError, ValueError) using mock K8s APIs. Address review feedback: - Add payload verification for create_namespaced_custom_object - Add test_get_job_status_conditions for all status-mapping branches - Add get_job_logs branch coverage (trial_name, follow, empty, pending pod) - Restore polling_interval validation cases (zero, equal-to-timeout, negative) - Rework wait_for_job_status with CR-based mock and callback test - Fix docstrings, move TypeVar near imports, add positional-arg comments Signed-off-by: Slowlybomb <[email protected]> * feat(trainer): Support NPU labels in TrainJob device (#336) Implement support for NPU resource labels in resource limit validation, resolving the existing TODO to support additional accelerator types. Signed-off-by: Sujal Shah <[email protected]> * feat(ci): Replace Trivy CVE scanning with uv audit + OSV-Scanner (#495) Replaces the Trivy-based workflows removed in #427 with two new tools: - validate-lockfile.yaml: PR check using uv audit to diff-compare vulnerabilities between PR and base branch. Non-blocking (informational comments only). Note: uv audit is experimental; if its text output format changes, the fallback is switching to osv-scanner JSON output (both query the same OSV.dev database). - osv-scanner.yaml: Nightly scan using OSV-Scanner CLI (v2.3.8) with SARIF upload to the GitHub Security tab and auto-fix PRs. Integrates with existing .github/scripts/ utilities (update_overrides.py, compare_versions.py, extract_version.py) and the cleanup-overrides workflow for the full fix lifecycle. - osv-scanner.toml: Minimal config for suppressing false positives. Closes #478 Signed-off-by: Fiona-Waters <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> * fix(ci): fix --preview-features flag usage in uv audit (#509) uv audit requires `--preview-features audit` (with a value), not `--preview-features` as a bare flag. The bare flag causes exit code 2 (CLI usage error) on every run. Without the flag at all, uv audit works but emits a warning to stderr. Signed-off-by: Fiona-Waters <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> * fix(deps): exclude kubernetes 36.0.0 (multiple client regressions) (#507) Signed-off-by: tariq-hasan <[email protected]> * fix: nightly security dependency updates (#497) Co-authored-by: google-oss-prow <[email protected]> * fix(deps): pin below kubernetes 36.0.0 (multiple client regressions) (#511) Signed-off-by: tariq-hasan <[email protected]> * fix(docs): Fix light mode text visibility in custom CSS (#332) * fix(docs): Fix light mode text visibility in custom CSS Furo sets data-theme on body, not :root. Custom CSS used :root:not([data-theme="light"]) so dark overrides always applied. Switched to body-based selectors and added explicit body[data-theme="light"] block for correct light-mode colors. Fixes #331 Signed-off-by: Sridhar1030 <[email protected]> * docs: add --kf-accent to all theme blocks for consistency Signed-off-by: Sridhar1030 <[email protected]> * fix(docs): address Copilot review - remove redundant light block, add comments Remove duplicate body[data-theme="light"] block since :root values are inherited when the media-query selector stops matching. Add explanatory comment documenting why two dark blocks are needed (system-dark auto vs explicit dark selection). Signed-off-by: Sridhar1030 <[email protected]> --------- Signed-off-by: Sridhar1030 <[email protected]> * chore(ci): fix header comment loss in update_overrides on repeated runs (#503) * chore(ci): fix header comment loss in update_overrides on repeated runs Signed-off-by: Yogarajalakshmi S <[email protected]> * chore(ci): fix header reinsertion edge cases in update_overrides Signed-off-by: Yogarajalakshmi S <[email protected]> * chore(ci): handle CRLF line endings in override block insertion Signed-off-by: Yogarajalakshmi S <[email protected]> --------- Signed-off-by: Yogarajalakshmi S <[email protected]> * feat(hub): add upload-artifact utility (#514) * feat(hub): add upload-artifact utility for #475 Add standalone S3/OCI upload helper and types so artifacts can be uploaded without coupling to the model registry client. Signed-off-by: myesmin <[email protected]> Co-authored-by: Cursor <[email protected]> * fix(hub): align OCIUploadParams and _upload_to_oci with model-registry API Replace OCIUploadParams fields (model_uri, author, model_description, model_title) with base_image and oci_ref to match the actual save_to_oci_registry signature in the model-registry package. Add type: ignore[call-arg] on intentionally invalid Pydantic instantiations in tests that ty was flagging. Signed-off-by: myesmin <[email protected]> Co-Authored-By: Claude Sonnet 4.6 <[email protected]> --------- Signed-off-by: myesmin <[email protected]> Co-authored-by: Cursor <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]> * fix(ci): Fix git-cliff to generate changelog for patch releases (#527) * fix(ci): Fix git-cliff to generate changelog for patch releases Signed-off-by: Andrey Velichkevich <[email protected]> * Add final message Signed-off-by: Andrey Velichkevich <[email protected]> * Fetch only upstream tags Signed-off-by: Andrey Velichkevich <[email protected]> * Remove git-cliff from deps Signed-off-by: Andrey Velichkevich <[email protected]> * Fix commit msg Signed-off-by: Andrey Velichkevich <[email protected]> * Fix PR names and fetch upstream commit Signed-off-by: Andrey Velichkevich <[email protected]> * Update RELEASE docs to align with Trainer Signed-off-by: Andrey Velichkevich <[email protected]> * Update RELEASE.md Co-authored-by: Anya Kramar <[email protected]> Signed-off-by: Andrey Velichkevich <[email protected]> --------- Signed-off-by: Andrey Velichkevich <[email protected]> Co-authored-by: Anya Kramar <[email protected]> * Kubeflow SDK Official Release 0.4.1 (#532) Signed-off-by: Andrey Velichkevich <[email protected]> * chore(docs): update wait_for_run_status timeout default in KEP-125 (#535) Signed-off-by: MStokluska <[email protected]> * chore(trainer): Align list_runtimes docstring with namespace-scoped TrainingRuntime behavior (#305) * Align docstring with current behavior Signed-off-by: Moeed Shaik <[email protected]> * updated for get_runtime Signed-off-by: Moeed Shaik <[email protected]> --------- Signed-off-by: Moeed Shaik <[email protected]> Signed-off-by: Moeed <[email protected]> * chore(docs): Move KEPs to the proposals directory (#538) KEPs lived under docs/ even though they are project governance artifacts rather than user-facing documentation, and were never part of the Sphinx build. Move them to a top-level proposals/ directory to mirror kubeflow/trainer#3629, update the contributor guide and repository map, and fix a now-stale KEP link in the trainer backend. Signed-off-by: Andrey Velichkevich <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> * chore(docs): update wait_for_run_status to allow for indefinite wait, remove ERROR state and add system CA bundle support in KEP-125 (#539) Signed-off-by: MStokluska <[email protected]> * fix: nightly security dependency updates (#519) Co-authored-by: google-oss-prow <[email protected]> * chore(docs): Update release doc with milestone applier (#543) Signed-off-by: Andrey Velichkevich <[email protected]> * chore: Add ADOPTERS.md for Kubeflow SDK (#545) Signed-off-by: Eder Ignatowicz <[email protected]> Co-authored-by: Claude <[email protected]> * chore: rename SparkConnectInfo pod_name to driver_pod_name (#546) * refactor: rename SparkConnectInfo pod_name to driver_pod_name Signed-off-by: Sameer_yadav <[email protected]> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <[email protected]> Signed-off-by: Sameer_Yadav <[email protected]> * minor fix Signed-off-by: Sameer_yadav <[email protected]> --------- Signed-off-by: Sameer_yadav <[email protected]> Signed-off-by: Sameer_Yadav <[email protected]> Co-authored-by: Copilot Autofix powered by AI <[email protected]> * feat: [KEP-107] Add long-running job submission proposal for SparkClient (#524) * docs: update KEP-107 with review feedback Signed-off-by: Sameer_yadav <[email protected]> * revised Signed-off-by: Sameer_yadav <[email protected]> * updated Signed-off-by: Sameer_yadav <[email protected]> * docs: address final review feedback for SparkClient KEP Signed-off-by: Sameer_yadav <[email protected]> --------- Signed-off-by: Sameer_yadav <[email protected]> * chore: improve Spark SDK documentation with connection and session management (#537) * docs: improve Spark SDK documentation with connection and session management examples Signed-off-by: Sameer_yadav <[email protected]> * refinement Signed-off-by: Sameer_yadav <[email protected]> * chore: address review feedback Signed-off-by: Sameer_yadav <[email protected]> --------- Signed-off-by: Sameer_yadav <[email protected]> * fix(docs): use relative URLs for nav bar logo and examples links (#553) * fix(docs): use relative URLs for nav bar logo and examples links Signed-off-by: Dmytro Hryshchenko <[email protected]> Co-Authored-By: Claude Opus 4.6 <[email protected]> * chore: fix linter Signed-off-by: Dmytro Hryshchenko <[email protected]> --------- Signed-off-by: Dmytro Hryshchenko <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> * fix: improve Trainer control-plane ConfigMap warning message (#558) * Improve Trainer control-plane warning message Signed-off-by: Sumitha007 <[email protected]> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <[email protected]> Signed-off-by: Sumitha J <[email protected]> --------- Signed-off-by: Sumitha007 <[email protected]> Signed-off-by: Sumitha J <[email protected]> Co-authored-by: Copilot Autofix powered by AI <[email protected]> * fix(docs): correct broken tune cross-reference in Quickstart (#557) The "What's Next?" list in the Quickstart linked to `../tune/index`, but no `tune/` docs directory exists: the hyperparameter-tuning docs live under `optimize/` (referenced correctly by the top-level toctree and the Katib link in index.rst). The stale reference produced a Sphinx "unknown document: '../tune/index'" warning and rendered a dead link on the docs site. Point the reference at `../optimize/index` so the link resolves. The bullet's description ("Automatically tune hyperparameters") still matches, since that page is titled "Hyperparameter Tuning". Signed-off-by: Anas Khan <[email protected]> * fix(spark): correct stale spark_version default in build_spark_connect_cr docstring (#556) The build_spark_connect_cr docstring documented the spark_version default as 3.4.1, but the applied default is constants.DEFAULT_SPARK_VERSION, which was bumped to 4.0.1 to match the spark-operator image. Update the docstring to 4.0.1 so it reflects the actual default. Documentation-only; no behavior change. The default is already covered by test_minimal_cr. Signed-off-by: Anas Khan <[email protected]> * fix(trainer): serialize TorchTune dtype/loss enums by value, add unit tests for get_args_using_torchtune_config and get_trainer_cr_from_builtin_trainer (#416) * chore(trainer): add unit tests for get_args_using_torchtune_config and get_trainer_cr_from_builtin_trainer Add comprehensive unit tests for two previously untested utility functions in kubeflow/trainer/backends/kubernetes/utils.py: - test_get_args_using_torchtune_config: 8 parametrized test cases covering empty config, dtype only, all scalar fields (dtype/batch_size/epochs/loss), LoRA peft_config, dataset_preprocess_config, HuggingFace dataset URIs (directory and file paths via initializer), and invalid dtype error. - test_get_trainer_cr_from_builtin_trainer: 4 parametrized test cases covering num_nodes with batch_size, resources_per_node, empty config, and invalid config type error. These functions had zero test coverage. The new tests follow existing patterns in the file using TestCase dataclass and pytest.mark.parametrize. Signed-off-by: Prabhaharan Velu <[email protected]> * fix(trainer): serialize TorchTune dtype/loss enums by value; address review get_args_using_torchtune_config interpolated the DataType/Loss enum members directly (e.g. "dtype=DataType.BF16"), so torchtune received the enum repr instead of the intended value ("dtype=bf16", "loss=torchtune.modules.loss.CEWithChunkedOutputLoss"). Use .value for both. Tests: - Assert literal produced strings for dtype/loss instead of reusing the implementation's f-string, which made the assertions tautological. - Add a nested-directory dataset URI case covering the data_dir branch with a non-trivial relative path. - Add a num_nodes=0 case pinning the `if config.num_nodes:` boundary. - Add a case threading an initializer through get_trainer_cr_from_builtin_trainer and asserting the dataset arg lands in trainer_cr.args. Signed-off-by: Prabhaharan Velu <[email protected]> * test(trainer): update builtin trainer backend test for dtype/loss value serialization test_train's "valid flow with built in trainer" case asserted the old enum repr (loss=Loss.CEWithChunkedOutputLoss). Update it to the value-serialized form now produced by get_args_using_torchtune_config. Signed-off-by: Prabhaharan Velu <[email protected]> --------- Signed-off-by: Prabhaharan Velu <[email protected]> * fix(trainer): validate HuggingFaceModelInitializer storage_uri has a repo path (#564) HuggingFaceModelInitializer only checked the hf:// prefix, while its sibling HuggingFaceDatasetInitializer also rejects a storage_uri whose urlparse path is empty. A model storage_uri such as hf://model was silently accepted and only failed later at download time. Add the same repo path guard to the model initializer so an invalid storage_uri raises ValueError at construction with a clear message. Add parametrized tests for HuggingFaceModelInitializer, which had no prior coverage. Fixes #563 Signed-off-by: Sahil Kumar Singh <[email protected]> * fix(trainer): validate HuggingFace storage_uri has both user and repo (#575) The HuggingFace initializers only rejected a storage_uri with an empty path, so URIs missing either component, such as hf://user/ (no repo) or hf:///model (no user), were accepted and failed later at download time. Consolidate the validation into a shared _validate_hf_storage_uri helper used by both HuggingFaceModelInitializer and HuggingFaceDatasetInitializer, requiring a non-empty user and repo. Add tests for the dataset initializer, which had no prior coverage, and for the new rejected forms. Follow-up to #564. Signed-off-by: Sahil Kumar Singh <[email protected]> * feat: add update_trainjob_status utility for progress reporting (#368) * feat: add update_runtime_status utility for progress reporting Signed-off-by: abhijeet-dhumal <[email protected]> * test: add unit tests for update_runtime_status Signed-off-by: abhijeet-dhumal <[email protected]> * docs: add documentation for update_runtime_status utility Signed-off-by: abhijeet-dhumal <[email protected]> * fix: align env var names with kubeflow/trainer#3227 Signed-off-by: abhijeet-dhumal <[email protected]> * fix(trainer): harden update_runtime_status for thread safety and edge cases Signed-off-by: abhijeet-dhumal <[email protected]> * fix(trainer): fix token_path type hint and update progress reporting docs Signed-off-by: abhijeet-dhumal <[email protected]> * fix: address review feedback on update_runtime_status Signed-off-by: abhijeet-dhumal <[email protected]> * fix: pyproject file Signed-off-by: abhijeet-dhumal <[email protected]> * refactor: move status utility to kubernetes backend and rename Signed-off-by: abhijeet-dhumal <[email protected]> * refactor: rename estimated_time_remaining to estimated_remaining_seconds Signed-off-by: abhijeet-dhumal <[email protected]> * refactor: address review feedback on update_trainjob_status Signed-off-by: abhijeet-dhumal <[email protected]> --------- Signed-off-by: abhijeet-dhumal <[email protected]> * fix(trainer): don't report Complete when all container statuses are Unknown (#562) aggregate_status_from_containers() ran all() over statuses filtered to exclude UNKNOWN. When the list is empty or every status is UNKNOWN the filtered sequence is empty, so all([]) returns True and the job is reported Complete. This is reachable via aggregate_container_statuses() when a container cannot be inspected (get_container_status returns UNKNOWN), so an unreadable job wrongly reports success to get_job(), list_jobs(), wait_for_job_status() and get_job_logs(). Require at least one known status and all known statuses to be Complete before returning Complete; empty or all-UNKNOWN now aggregate to Unknown. Add unit test coverage for aggregate_status_from_containers. Fixes #561 Signed-off-by: Sahil Kumar Singh <[email protected]> * fix(trainer): Move TrainJob cleanup to finally block in get_runtime_packages (#582) * Move TrainJob cleanup to finally block in get_runtime_packages Signed-off-by: kingabdulsalamsas111-lgtm <[email protected]> * Apply ruff formatting fixes Signed-off-by: kingabdulsalamsas111-lgtm <[email protected]> * test: verify temporary TrainJob cleanup in get_runtime_packages Signed-off-by: kingabdulsalamsas111-lgtm <[email protected]> * Address review comments Signed-off-by: kingabdulsalamsas111-lgtm <[email protected]> --------- Signed-off-by: kingabdulsalamsas111-lgtm <[email protected]> * feat: add runtime kind support for TrainingRuntime and ClusterTrainingRuntime (#549) * Add runtime kind support for TrainingRuntime and ClusterTrainingRuntime Signed-off-by: Sumitha007 <[email protected]> * refactor(trainer): replace runtime kind constants with enum Signed-off-by: Sumitha007 <[email protected]> * refactor(trainer): replace runtime kind constants with enum Signed-off-by: Sumitha007 <[email protected]> * Add get_runtime test for namespaced runtime preference Signed-off-by: Sumitha007 <[email protected]> * Add test case for namespaced TrainingRuntime preference Signed-off-by: Sumitha007 <[email protected]> * refactor(tests): move namespaced TrainingRuntime test case to appropriate section Signed-off-by: Sumitha007 <[email protected]> --------- Signed-off-by: Sumitha007 <[email protected]> * feat: add PipelinesClient with kfp 2.17.0 (#591) * feat(pipelines): add PipelinesClient re-export from kfp (KEP-125) Signed-off-by: MStokluska <[email protected]> Signed-off-by: Esa Fazal <[email protected]> * docs(pipelines): add documentation and update status to Available Signed-off-by: MStokluska <[email protected]> Signed-off-by: Esa Fazal <[email protected]> * fix(pipelines): simplify re-export and fix ImportError handling Signed-off-by: MStokluska <[email protected]> Signed-off-by: Esa Fazal <[email protected]> * fix(pipelines): address coderabbit comments Signed-off-by: MStokluska <[email protected]> Signed-off-by: Esa Fazal <[email protected]> * fix(pipelines): address comments Signed-off-by: MStokluska <[email protected]> Signed-off-by: Esa Fazal <[email protected]> * fix(pipelines): fix pre-commit check Signed-off-by: MStokluska <[email protected]> Signed-off-by: Esa Fazal <[email protected]> * fix(pipelines): use released kfp 2.17.0 instead of git master Replace git-based kfp dependency with kfp[kubernetes]>=2.17.0 now that kfp.kubeflow_client has been published to PyPI. Update README version accordingly. Signed-off-by: Esa Fazal <[email protected]> * docs: update SDK diagram with Pipelines logo Signed-off-by: Esa Fazal <[email protected]> * fix: remove year from copyright boilerplate Signed-off-by: Esa Fazal <[email protected]> * docs: use correct KFP logo in SDK diagram Signed-off-by: Esa Fazal <[email protected]> --------- Signed-off-by: MStokluska <[email protected]> Signed-off-by: Esa Fazal <[email protected]> Co-authored-by: MStokluska <[email protected]> * fix(trainer): validate polling_interval in wait_for_job_status (#554) * fix(trainer): validate polling_interval in wait_for_job_status Adds a ValueError guard in TrainerClient.wait_for_job_status() to reject non-positive polling_interval values before delegating to the backend. Previously this could cause a CPU busy-loop (polling_interval=0) or a cryptic stdlib error (negative values). Fixes #550 Signed-off-by: HarshPopat23 <[email protected]> * Potential fix for pull request finding add proper messages Co-authored-by: Copilot Autofix powered by AI <[email protected]> Signed-off-by: HarshPopat23 <[email protected]> * test(trainer): update polling_interval test assertions to match new error messages Signed-off-by: HarshPopat23 <[email protected]> * refactor(common): extract polling_interval/timeout validation to shared utils Move polling_interval and timeout validation into a shared common_utils.validate_wait_intervals() function, used by both TrainerC…
Fixes #327
Summary
Replace
if value:withif value is not None:to ensure falsy valueslike
0.0andFalseare not silently dropped.Changes
get_args_from_peft_configlora_dropout=0.0apply_lora_to_mlp=FalseImpact
Ensures explicit user configurations are respected and not ignored.