|
| 1 | +# Idle Servable Management (Preview) {#ovms_docs_idle_servable_management} |
| 2 | + |
| 3 | +> **Preview feature:** Idle servable management is disabled by default. Its behavior and configuration can change in future releases. |
| 4 | +
|
| 5 | +Idle servable management keeps configured models and MediaPipe graphs visible to clients while unloading their heavy runtime resources when they are not needed. A request for an unloaded servable loads it again before inference. This lets one OVMS instance serve several large models with limited CPU or GPU memory. |
| 6 | + |
| 7 | +Optionally set an OpenVINO model cache directory with `--cache_dir` to reduce reload latency. This is recommended because reloading from cache avoids recompiling the model. The first request still bears the latency cost of waiting for the model to wake up. |
| 8 | + |
| 9 | + |
| 10 | +## Groups |
| 11 | + |
| 12 | +A group is a unit that OVMS loads and unloads together. Add `group_name` to a servable classic model or MediaPipe graph configuration. All servables in the same group load and unload together. Use groups for servables that must be available together, such as a graph and its dependent models. |
| 13 | + |
| 14 | +Without `group_name`, every classic model and MediaPipe graph is assigned to its own group. A group named `permanent` stays loaded and is never unloaded. |
| 15 | + |
| 16 | +For groups that contain only one non-permanent model or graph, setting `group_name` is optional. Keeping it explicit in configuration can improve readability. |
| 17 | + |
| 18 | +OVMS keeps one non-permanent group active. A request for servable from another group waits for current group to finish its requests, then loads requested group. First request after unload includes wake-up latency. |
| 19 | + |
| 20 | +You can create or update `config.json` with OVMS CLI: |
| 21 | + |
| 22 | +```txt |
| 23 | +set OVMS_MODEL_REPOSITORY_PATH=c:\models |
| 24 | +
|
| 25 | +ovms --pull --source_model OpenVINO/Qwen3-30B-A3B-Instruct-2507-int4-ov |
| 26 | +ovms --add_to_config --model_name OpenVINO/Qwen3-30B-A3B-Instruct-2507-int4-ov --group_name permanent |
| 27 | +
|
| 28 | +ovms --pull --source_model OpenVINO/bge-base-en-v1.5-int8-ov |
| 29 | +ovms --add_to_config --model_name OpenVINO/bge-base-en-v1.5-int8-ov --group_name rag |
| 30 | +
|
| 31 | +ovms --pull --source_model OpenVINO/bge-reranker-base-int8-ov |
| 32 | +ovms --add_to_config --model_name OpenVINO/bge-reranker-base-int8-ov --group_name rag |
| 33 | +
|
| 34 | +ovms --pull --source_model OpenVINO/FLUX.1-schnell-int4-ov |
| 35 | +ovms --add_to_config --model_name OpenVINO/FLUX.1-schnell-int4-ov |
| 36 | +``` |
| 37 | + |
| 38 | +The following resulting configuration keeps one large model loaded and groups retrieval models together. It loads image generation on demand. Download the referenced models before starting OVMS. |
| 39 | + |
| 40 | +```json |
| 41 | +{ |
| 42 | + "model_config_list": [ |
| 43 | + { |
| 44 | + "config": { |
| 45 | + "name": "OpenVINO/Qwen3-30B-A3B-Instruct-2507-int4-ov", |
| 46 | + "base_path": "./OpenVINO/Qwen3-30B-A3B-Instruct-2507-int4-ov", |
| 47 | + "group_name": "permanent" |
| 48 | + } |
| 49 | + }, |
| 50 | + { |
| 51 | + "config": { |
| 52 | + "name": "OpenVINO/bge-base-en-v1.5-int8-ov", |
| 53 | + "base_path": "./OpenVINO/bge-base-en-v1.5-int8-ov", |
| 54 | + "group_name": "rag" |
| 55 | + } |
| 56 | + }, |
| 57 | + { |
| 58 | + "config": { |
| 59 | + "name": "OpenVINO/bge-reranker-base-int8-ov", |
| 60 | + "base_path": "./OpenVINO/bge-reranker-base-int8-ov", |
| 61 | + "group_name": "rag" |
| 62 | + } |
| 63 | + }, |
| 64 | + { |
| 65 | + "config": { |
| 66 | + "name": "OpenVINO/FLUX.1-schnell-int4-ov", |
| 67 | + "base_path": "./OpenVINO/FLUX.1-schnell-int4-ov" |
| 68 | + } |
| 69 | + } |
| 70 | + ] |
| 71 | +} |
| 72 | +``` |
| 73 | + |
| 74 | +> **Important:** Concurrent requests targeting loaded and unloaded groups have no scheduling policy in this preview. While requests run on active group, incoming traffic continues to that group. Do not depend on fair routing or a bounded switch time between groups. Organize groups so that one active non-permanent group fits available host and device memory to avoid out-of-memory conditions during swaps. |
| 75 | +
|
| 76 | +## Enable idle management |
| 77 | + |
| 78 | +Set `--idle_unload_timeout_seconds` to a positive number when starting OVMS. `0`, the default, disables idle servable management. |
| 79 | +Start OVMS with a cache directory: |
| 80 | + |
| 81 | +```text |
| 82 | +ovms --config_path c:\models\config.json --cache_dir c:\models\cache --idle_unload_timeout_seconds 60 |
| 83 | +``` |
| 84 | + |
| 85 | +## Servable status |
| 86 | + |
| 87 | +Idle-unloaded servables remain `AVAILABLE` in readiness and status APIs. They wake on next inference request. Status and metrics requests do not reset idle timeout; only inference activity keeps a group loaded. |
| 88 | + |
| 89 | +## Metrics |
| 90 | + |
| 91 | +Enable metrics as described in [Metrics](./metrics.md). `ovms_graph_loaded` is default gauge for MediaPipe graphs. It has label `name` and reports `1` when graph resources are loaded or `0` when idle-unloaded. |
| 92 | + |
| 93 | +Classic models have no equivalent loaded-state metric in this preview. Monitor request latency to observe model wake-up time. |
0 commit comments