Repository navigation
Expand file tree
/
Copy path__init__.py
More file actions
93 lines (65 loc) · 3.95 KB
/
Copy path__init__.py
File metadata and controls
93 lines (65 loc) · 3.95 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
"""
The ray integration traces:
- Job lifetime (job submit, job run)
- Task submission and execution
- Actor method submission and execution
- Ray Serve proxy requests, deployment handle calls, and deployment execution
Enabling
~~~~~~~~
Ray instrumentation is experimental. It is deactivated by default. To enable it,
you have to follow one of the two methods below:
The recommended way to instrument Ray, is to instrument the Ray cluster using ddtrace-run::
DD_PATCH_MODULES="ray:true, aiohttp:false, grpc:false, requests:false" ddtrace-run ray start --head
DD_PATCH_MODULES will allow to reduce noise by sending only the jobs related spans.
You can also do it by starting Ray head with a tracing startup hook::
ray start --head --tracing-startup-hook=ddtrace.contrib.ray:setup_tracing
Note that this method does not provide full tracing capabilities if ``ray.init()`` is not called at the top
of your job scripts.
Ray Serve tracing is included in the Ray integration. There is no separate
``ray_serve`` patch flag. Once the Ray integration is enabled, importing
``ray.serve`` will automatically instrument Ray Serve components.
This traces:
- inbound Ray Serve HTTP and gRPC proxy requests
- ``DeploymentHandle.remote()`` calls
- request assignment and replica request handling
- functions and class methods defined on deployments created with
``serve.deployment``, including ``@serve.ingress`` deployments
Configuration
~~~~~~~~~~~~~
The Ray integration can be configured using environment variables:
- ``DD_TRACE_RAY_CORE_API``: Enable tracing of Ray's core API functions like ``ray.wait()``
(default: ``False``)
- ``DD_TRACE_RAY_ARGS_KWARGS``: Enable tracing of arguments and keyword arguments passed to
Ray tasks and actor methods (default: ``False``)
- ``DD_TRACE_RAY_SUBMISSION_SPANS_ENABLED``: Enable tracing of ``task.submit`` and
``actor_method.submit`` spans (default: ``False``). This is disabled by default to
reduce span volume in high-throughput Ray workloads when execution spans are sufficient.
Enable it when you need visibility into queuing/submission latency, fan-out patterns,
or when work is submitted but delayed before execution.
- ``DD_TRACE_RAY_IGNORED_ACTORS``: JSON object mapping actor class names to actor methods
to exclude from instrumentation, or ``"*"`` to exclude the full actor class (default:
empty). For example, ``{"ActorA": ["method1"], "ActorB": "*"}``. Matching is based
on class name only, not module.
- ``DD_TRACE_EXPERIMENTAL_LONG_RUNNING_FLUSH_INTERVAL``: Interval for resubmitting long-running
spans (default: ``120.0`` seconds)
- ``DD_TRACE_RAY_USE_ENTRYPOINT_AS_SERVICE_NAME``: Whether to use the job entrypoint as the
service name (default: ``False``). If ``True``, the entrypoint will be used as the service
name if DD_SERVICE is not set and a job name is not specified in the metadata.
- ``DD_TRACE_RAY_REDACT_ENTRYPOINT_PATHS``: Whether to redact file paths in the job entrypoint
(default: ``True``). If ``True``, file paths in the entrypoint will be redacted to avoid
leaking sensitive information.
Ray service name can be configured, in order of precedence by:
- specifying ``DD_SERVICE`` when initializing your Ray cluster.
- setting ``DD_TRACE_RAY_USE_ENTRYPOINT_AS_SERVICE_NAME=True``. In this case, the service
name will be the name of your entrypoint script.
- specifying in metadata during job submission::
ray job submit --metadata-json='{"job_name": "my_model"}' -- python entrypoint.py
By default, the service name will be ``unnamed.ray.job``.
Notes
~~~~~
- The integration disables Ray's built-in OpenTelemetry tracing to avoid duplicate telemetry.
- Actor methods like ``ping`` and ``_polling`` are excluded from tracing to reduce noise.
- Actors whose names start with an underscore (_) are not instrumented.
- For Ray Serve HTTP requests, resource names use the matched route pattern when
available (for example, ``/name/{name}``) to avoid high-cardinality spans.
"""