Repository navigation
feat(trace): add agent observability with Monocle #4024
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 5 commits
ba5caeb
60559dd
39b4d49
a97ff2c
745409c
803be48
c2910f2
c1f8bff
232fd03
93f968d
cb165f8
e0122c9
ec30867
6c36548
96fd6ec
3ac4310
2920b60
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -37,6 +37,7 @@ | |
| from app.gateway.trace_middleware import TraceMiddleware, resolve_trace_enabled | ||
| from deerflow.config import app_config as deerflow_app_config | ||
| from deerflow.logging_config import DEFAULT_LOG_DATE_FORMAT, DEFAULT_LOG_FORMAT, configure_logging | ||
| from deerflow.tracing.monocle import setup_monocle_tracing_if_enabled | ||
| from deerflow.uploads.manager import cleanup_stale_upload_staging_files | ||
|
|
||
| AppConfig = deerflow_app_config.AppConfig | ||
|
|
@@ -189,6 +190,14 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]: | |
| config = get_gateway_config() | ||
| logger.info(f"Starting API Gateway on {config.host}:{config.port}") | ||
|
|
||
| # Agent observability (Monocle). Off by default; enabled with | ||
| # MONOCLE_TRACING. Initialized here at startup — not at import time — so a | ||
| # plain `import deerflow.agents` never installs a process-global tracer. | ||
| try: | ||
| setup_monocle_tracing_if_enabled() | ||
|
WillemJiang marked this conversation as resolved.
Resolved
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Embedded / TUI / CLI path gets no Monocle tracing. LangSmith and Langfuse are attached at the graph invocation root for both That's a real asymmetry vs the other two providers — at minimum call it out in AGENTS.md (embedded/TUI users must call Also: this lifespan wiring — the one line that makes the feature work — has no test asserting it's invoked during startup. A focused test monkeypatching the helper and asserting it's called from the lifespan would close that gap. |
||
| except Exception: # pragma: no cover - observability must never break startup | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This Side note: a Monocle misconfiguration is silently ignored after this one startup log (per-run |
||
| logger.exception("Monocle tracing setup failed; continuing without it") | ||
|
Comment on lines
+198
to
+201
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Graceful-degrade vs. fail-fast asymmetry — worth documenting. This The behavior is internally consistent with "cannot break startup", but the asymmetry isn't called out in |
||
|
|
||
| # Pre-warm tiktoken encoding cache so the first memory-injection request | ||
| # never blocks on the BPE data download (which hits an OpenAI/Azure URL | ||
| # that may be unreachable in restricted networks — see issue #3402). | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -47,11 +47,27 @@ def validate(self) -> None: | |
| raise ValueError(f"Langfuse tracing is enabled but required settings are missing: {', '.join(missing)}") | ||
|
|
||
|
|
||
| class MonocleTracingConfig(BaseModel): | ||
| """Configuration for Monocle telemetry.""" | ||
|
|
||
| enabled: bool = Field(...) | ||
| exporters: str = Field(...) | ||
|
|
||
| @property | ||
| def is_configured(self) -> bool: | ||
| return self.enabled | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
LangSmith/Langfuse fold credential presence into Defensible as a composite-validation choice, but a one-line comment explaining why Monocle's |
||
|
|
||
| def validate(self) -> None: | ||
| # No external credentials are required; the "file" exporter writes locally. | ||
| return None | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Recommend an exporter allowlist ( |
||
|
|
||
|
|
||
| class TracingConfig(BaseModel): | ||
| """Tracing configuration for supported providers.""" | ||
|
|
||
| langsmith: LangSmithTracingConfig = Field(...) | ||
| langfuse: LangfuseTracingConfig = Field(...) | ||
| monocle: MonocleTracingConfig = Field(...) | ||
|
|
||
| @property | ||
| def is_configured(self) -> bool: | ||
|
|
@@ -125,6 +141,10 @@ def get_tracing_config() -> TracingConfig: | |
| secret_key=_first_env_value("LANGFUSE_SECRET_KEY"), | ||
| host=_first_env_value("LANGFUSE_BASE_URL") or "https://cloud.langfuse.com", | ||
| ), | ||
| monocle=MonocleTracingConfig( | ||
| enabled=_env_flag_preferred("MONOCLE_TRACING"), | ||
| exporters=_first_env_value("MONOCLE_EXPORTERS") or "file", | ||
| ), | ||
| ) | ||
| return _tracing_config | ||
|
|
||
|
|
@@ -149,6 +169,16 @@ def is_tracing_enabled() -> bool: | |
| return get_tracing_config().is_configured | ||
|
|
||
|
|
||
| def is_monocle_tracing_enabled() -> bool: | ||
| """Whether Monocle OTel observability is enabled (via ``MONOCLE_TRACING``). | ||
|
|
||
| Kept separate from :func:`get_enabled_tracing_providers` because Monocle is a | ||
| process-global instrumentor activated at startup, not a per-run LangChain | ||
| callback. | ||
| """ | ||
| return get_tracing_config().monocle.is_configured | ||
|
|
||
|
|
||
| def reset_tracing_config() -> None: | ||
| """Discard the cached :class:`TracingConfig` so the next call rebuilds it. | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| """Monocle telemetry: initialized once from the Gateway lifespan when ``MONOCLE_TRACING`` is set.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import logging | ||
|
|
||
| from deerflow.config import ( | ||
| get_enabled_tracing_providers, | ||
| get_tracing_config, | ||
| is_monocle_tracing_enabled, | ||
| ) | ||
|
|
||
| logger = logging.getLogger(__name__) | ||
|
|
||
|
|
||
| def setup_monocle_tracing_if_enabled() -> bool: | ||
| """Initialize Monocle telemetry when ``MONOCLE_TRACING`` is enabled; a no-op otherwise. | ||
|
|
||
| ``monocle_apptrace.setup_monocle_telemetry()`` is idempotent, so this stays a thin, | ||
| config-gated wrapper. Returns ``True`` when enabled. | ||
| """ | ||
| if not is_monocle_tracing_enabled(): | ||
| return False | ||
|
|
||
| # Only one OpenTelemetry provider can own the process. Monocle initializes here (at | ||
| # startup), before Langfuse's per-run handler, so enabling both means Langfuse loses | ||
| # its spans — warn so the operator turns one off. | ||
| if "langfuse" in get_enabled_tracing_providers(): | ||
| logger.warning("MONOCLE_TRACING is enabled alongside Langfuse; both need the global OpenTelemetry provider and only one can win. Enable only one of them.") | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This Langfuse-conflict warning is the key logic branch in the module and has no test. Add a
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Issue: the Langfuse/OTel conflict is asserted but not verified. In this repo, Langfuse is wired as a LangChain Two concerns:
Suggestion: verify against the installed |
||
|
|
||
| exporters = get_tracing_config().monocle.exporters | ||
| from monocle_apptrace import setup_monocle_telemetry | ||
|
|
||
| setup_monocle_telemetry(workflow_name="deer-flow", monocle_exporters_list=exporters) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Security: spans capture prompts + tool I/O verbatim, with no DeerFlow-controlled scrubbing layer. Auto-instrumentation records span inputs/outputs — prompts, tool arguments ( Two suggestions: (1) log a loud startup warning whenever a non-
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This call installs a process-global OTel The mitigation (off by default, optional extra) is sound - just flagging this as the one coupling to weigh against the |
||
| logger.info("Monocle telemetry enabled (exporters=%s)", exporters) | ||
| return True | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,6 +22,7 @@ dependencies = [ | |
| "pyjwt>=2.13.0", | ||
| "email-validator>=2.0.0", | ||
| "e2b-code-interpreter>=2.8.1", | ||
| "monocle_apptrace>=0.8.8", | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This should be an optional extra, not a hard runtime dependency. Monocle is off by default, yet |
||
| ] | ||
|
|
||
| [project.optional-dependencies] | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,76 @@ | ||
| """Tests for Monocle telemetry setup. | ||
|
|
||
| Covers the config gate (``MONOCLE_TRACING`` default off / toggle on), the setup | ||
| helper's behavior, and the regression that importing ``deerflow.agents`` no | ||
| longer sets up telemetry at import time. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import pytest | ||
|
|
||
| from deerflow.config import is_monocle_tracing_enabled | ||
| from deerflow.config.tracing_config import get_tracing_config, reset_tracing_config | ||
| from deerflow.tracing.monocle import setup_monocle_tracing_if_enabled | ||
|
|
||
|
|
||
| @pytest.fixture(autouse=True) | ||
| def clear_monocle_env(monkeypatch): | ||
| for name in ("MONOCLE_TRACING", "MONOCLE_EXPORTERS"): | ||
| monkeypatch.delenv(name, raising=False) | ||
| reset_tracing_config() | ||
| yield | ||
| reset_tracing_config() | ||
|
|
||
|
|
||
| def test_disabled_by_default(): | ||
| assert is_monocle_tracing_enabled() is False | ||
| assert get_tracing_config().monocle.enabled is False | ||
|
|
||
|
|
||
| def test_setup_noop_when_disabled(monkeypatch): | ||
| called = False | ||
|
|
||
| def _fail(*args, **kwargs): | ||
| nonlocal called | ||
| called = True | ||
|
|
||
| monkeypatch.setattr("monocle_apptrace.setup_monocle_telemetry", _fail) | ||
| assert setup_monocle_tracing_if_enabled() is False | ||
| assert called is False | ||
|
|
||
|
|
||
| def test_toggles_on_and_sets_up(monkeypatch): | ||
| monkeypatch.setenv("MONOCLE_TRACING", "true") | ||
| reset_tracing_config() | ||
|
|
||
| captured: dict = {} | ||
| monkeypatch.setattr("monocle_apptrace.setup_monocle_telemetry", lambda **kw: captured.update(kw)) | ||
|
|
||
| assert is_monocle_tracing_enabled() is True | ||
| assert setup_monocle_tracing_if_enabled() is True | ||
| assert captured == {"workflow_name": "deer-flow", "monocle_exporters_list": "file"} | ||
|
|
||
|
|
||
| def test_custom_exporters(monkeypatch): | ||
| monkeypatch.setenv("MONOCLE_TRACING", "true") | ||
| monkeypatch.setenv("MONOCLE_EXPORTERS", "file,console") | ||
| reset_tracing_config() | ||
|
|
||
| captured: dict = {} | ||
| monkeypatch.setattr("monocle_apptrace.setup_monocle_telemetry", lambda **kw: captured.update(kw)) | ||
|
|
||
| assert setup_monocle_tracing_if_enabled() is True | ||
| assert captured["monocle_exporters_list"] == "file,console" | ||
|
|
||
|
|
||
| def test_no_import_time_setup(): | ||
| """Regression: importing deerflow.agents must not install telemetry. | ||
|
|
||
| The setup call used to live at module import in ``deerflow/agents/__init__``. | ||
| It now happens only via the gateway lifespan, so the package namespace must | ||
| not even carry ``setup_monocle_telemetry``. | ||
| """ | ||
| import deerflow.agents as agents | ||
|
|
||
| assert not hasattr(agents, "setup_monocle_telemetry") | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This regression check could be stronger, and idempotency is untested. |
||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -30,6 +30,31 @@ logging: | |
| enabled: false | ||
| format: text | ||
|
|
||
| # ============================================================================ | ||
| # Monocle telemetry | ||
| # ============================================================================ | ||
| # Optional agent tracing via Monocle. Configured through environment variables | ||
| # (like LangSmith/Langfuse), not a config.yaml key, and OFF by default. When | ||
| # enabled, Monocle is initialized once at Gateway startup and auto-instruments | ||
| # the LLM/agent clients; a plain library import does not start tracing. | ||
| # | ||
| # MONOCLE_TRACING=true # enable (default: off) | ||
| # MONOCLE_EXPORTERS=file # file, console, okahu, s3, blob, gcs (default: file) | ||
| # OKAHU_API_KEY=okh_... # required only for the `okahu` exporter | ||
| # | ||
| # The `file` exporter writes one trace JSON per run to `.monocle/` under the | ||
| # process working directory (git-ignored). Each trace captures the workflow and | ||
| # per-agent/LLM/tool spans, including span inputs and outputs (prompts, tool | ||
| # arguments, and model responses), token usage, and timings. | ||
| # | ||
| # Data-exfiltration note: `file` persists prompts and model outputs to local | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Minor: this block sits under the |
||
| # disk; non-file exporters (e.g. `okahu`) send that same span data off-box to an | ||
| # external collector. Enable only exporters whose data handling you trust, and | ||
| # be aware that prompts/outputs may contain sensitive content. | ||
| # | ||
| # Note: Monocle and Langfuse both install a global OpenTelemetry TracerProvider; | ||
| # enable only one OTel-based tracer at a time (see backend/AGENTS.md → Tracing). | ||
|
|
||
| # ============================================================================ | ||
| # Token Usage | ||
| # ============================================================================ | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
User-facing version of the same Langfuse-conflict claim (see inline on
monocle.py)."...so it runs alongside LangSmith but not Langfuse" is stated as a flat rule, but rests on the same unverified assumption that Langfuse's callback path owns the global OTel provider. Worth softening ("may conflict; verify") or removing until the interaction is confirmed — otherwise Langfuse users are told they can't use Monocle when they likely can.