Repository navigation
feat(trace): add agent observability with Monocle #4024
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 7 commits
ba5caeb
60559dd
39b4d49
a97ff2c
745409c
803be48
c2910f2
c1f8bff
232fd03
93f968d
cb165f8
e0122c9
ec30867
6c36548
96fd6ec
3ac4310
2920b60
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -37,6 +37,7 @@ | |
| from app.gateway.trace_middleware import TraceMiddleware, resolve_trace_enabled | ||
| from deerflow.config import app_config as deerflow_app_config | ||
| from deerflow.logging_config import DEFAULT_LOG_DATE_FORMAT, DEFAULT_LOG_FORMAT, configure_logging | ||
| from deerflow.tracing.monocle import setup_monocle_tracing_if_enabled | ||
| from deerflow.uploads.manager import cleanup_stale_upload_staging_files | ||
|
|
||
| AppConfig = deerflow_app_config.AppConfig | ||
|
|
@@ -189,6 +190,14 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]: | |
| config = get_gateway_config() | ||
| logger.info(f"Starting API Gateway on {config.host}:{config.port}") | ||
|
|
||
| # Agent observability (Monocle). Off by default; enabled with | ||
| # MONOCLE_TRACING. Initialized here at startup — not at import time — so a | ||
| # plain `import deerflow.agents` never installs a process-global tracer. | ||
| try: | ||
| setup_monocle_tracing_if_enabled() | ||
|
WillemJiang marked this conversation as resolved.
Resolved
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Embedded / TUI / CLI path gets no Monocle tracing. LangSmith and Langfuse are attached at the graph invocation root for both That's a real asymmetry vs the other two providers — at minimum call it out in AGENTS.md (embedded/TUI users must call Also: this lifespan wiring — the one line that makes the feature work — has no test asserting it's invoked during startup. A focused test monkeypatching the helper and asserting it's called from the lifespan would close that gap. |
||
| except Exception: # pragma: no cover - observability must never break startup | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This Side note: a Monocle misconfiguration is silently ignored after this one startup log (per-run |
||
| logger.exception("Monocle tracing setup failed; continuing without it") | ||
|
Comment on lines
+198
to
+201
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Graceful-degrade vs. fail-fast asymmetry — worth documenting. This The behavior is internally consistent with "cannot break startup", but the asymmetry isn't called out in |
||
|
|
||
| # Pre-warm tiktoken encoding cache so the first memory-injection request | ||
| # never blocks on the BPE data download (which hits an OpenAI/Azure URL | ||
| # that may be unreachable in restricted networks — see issue #3402). | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -47,11 +47,37 @@ def validate(self) -> None: | |
| raise ValueError(f"Langfuse tracing is enabled but required settings are missing: {', '.join(missing)}") | ||
|
|
||
|
|
||
| _MONOCLE_EXPORTERS = ("file", "console", "okahu", "s3", "blob", "gcs") | ||
|
WillemJiang marked this conversation as resolved.
Resolved
|
||
|
|
||
|
|
||
| class MonocleTracingConfig(BaseModel): | ||
| """Configuration for Monocle telemetry.""" | ||
|
|
||
| enabled: bool = Field(...) | ||
| exporters: str = Field(...) | ||
| okahu_api_key: str | None = Field(...) | ||
|
|
||
| @property | ||
| def is_configured(self) -> bool: | ||
| return self.enabled | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
LangSmith/Langfuse fold credential presence into Defensible as a composite-validation choice, but a one-line comment explaining why Monocle's |
||
|
|
||
| def validate(self) -> None: | ||
| if not self.enabled: | ||
| return | ||
| selected = [e.strip() for e in self.exporters.split(",") if e.strip()] | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The |
||
| unknown = [e for e in selected if e not in _MONOCLE_EXPORTERS] | ||
| if unknown: | ||
| raise ValueError(f"MONOCLE_EXPORTERS has unknown exporter(s): {', '.join(unknown)}. Allowed: {', '.join(_MONOCLE_EXPORTERS)}.") | ||
| if "okahu" in selected and not self.okahu_api_key: | ||
| raise ValueError("Monocle 'okahu' exporter is selected but OKAHU_API_KEY is not set.") | ||
|
|
||
|
|
||
| class TracingConfig(BaseModel): | ||
| """Tracing configuration for supported providers.""" | ||
|
|
||
| langsmith: LangSmithTracingConfig = Field(...) | ||
| langfuse: LangfuseTracingConfig = Field(...) | ||
| monocle: MonocleTracingConfig = Field(...) | ||
|
|
||
| @property | ||
| def is_configured(self) -> bool: | ||
|
|
@@ -125,6 +151,11 @@ def get_tracing_config() -> TracingConfig: | |
| secret_key=_first_env_value("LANGFUSE_SECRET_KEY"), | ||
| host=_first_env_value("LANGFUSE_BASE_URL") or "https://cloud.langfuse.com", | ||
| ), | ||
| monocle=MonocleTracingConfig( | ||
| enabled=_env_flag_preferred("MONOCLE_TRACING"), | ||
| exporters=_first_env_value("MONOCLE_EXPORTERS") or "file", | ||
| okahu_api_key=_first_env_value("OKAHU_API_KEY"), | ||
| ), | ||
| ) | ||
| return _tracing_config | ||
|
|
||
|
|
@@ -149,6 +180,16 @@ def is_tracing_enabled() -> bool: | |
| return get_tracing_config().is_configured | ||
|
|
||
|
|
||
| def is_monocle_tracing_enabled() -> bool: | ||
| """Whether Monocle OTel observability is enabled (via ``MONOCLE_TRACING``). | ||
|
|
||
| Kept separate from :func:`get_enabled_tracing_providers` because Monocle is a | ||
| process-global instrumentor activated at startup, not a per-run LangChain | ||
| callback. | ||
| """ | ||
| return get_tracing_config().monocle.is_configured | ||
|
|
||
|
|
||
| def reset_tracing_config() -> None: | ||
| """Discard the cached :class:`TracingConfig` so the next call rebuilds it. | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| """Monocle telemetry: initialized once from the Gateway lifespan when ``MONOCLE_TRACING`` is set.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import logging | ||
|
|
||
| from deerflow.config import ( | ||
| get_enabled_tracing_providers, | ||
| get_tracing_config, | ||
| is_monocle_tracing_enabled, | ||
| ) | ||
|
|
||
| logger = logging.getLogger(__name__) | ||
|
|
||
|
|
||
| def setup_monocle_tracing_if_enabled() -> bool: | ||
| """Initialize Monocle telemetry when ``MONOCLE_TRACING`` is enabled; a no-op otherwise. | ||
|
|
||
| ``monocle_apptrace.setup_monocle_telemetry()`` is idempotent, so this stays a thin, | ||
| config-gated wrapper. Returns ``True`` when enabled. | ||
| """ | ||
| if not is_monocle_tracing_enabled(): | ||
| return False | ||
|
|
||
| monocle = get_tracing_config().monocle | ||
| # Fail fast on an unknown MONOCLE_EXPORTERS value or a missing OKAHU_API_KEY, | ||
| # with a clear message, before instrumenting. Validated here (not in the | ||
| # per-run callback path) so a config typo never breaks agent runs. | ||
| monocle.validate() | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Embedded/TUI path gets no signal that Monocle is a no-op. Because This matches the docs (embedded users call |
||
|
|
||
| # Only one OpenTelemetry provider can own the process. Monocle initializes here (at | ||
| # startup), before Langfuse's per-run handler, so enabling both means Langfuse loses | ||
| # its spans — warn so the operator turns one off. | ||
| if "langfuse" in get_enabled_tracing_providers(): | ||
| logger.warning("MONOCLE_TRACING is enabled alongside Langfuse; both need the global OpenTelemetry provider and only one can win. Enable only one of them.") | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This Langfuse-conflict warning is the key logic branch in the module and has no test. Add a
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Issue: the Langfuse/OTel conflict is asserted but not verified. In this repo, Langfuse is wired as a LangChain Two concerns:
Suggestion: verify against the installed |
||
|
|
||
| exporters = monocle.exporters | ||
|
|
||
| # Any exporter other than `file` moves trace data (prompts, tool inputs and | ||
| # outputs, completions) beyond the local .monocle/ directory, e.g. to an | ||
| # external collector (okahu, s3, ...). Warn loudly so it can't happen | ||
| # unnoticed. | ||
| non_file = [e.strip() for e in exporters.split(",") if e.strip() and e.strip() != "file"] | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Off-box warning fires for Either exclude |
||
| if non_file: | ||
| logger.warning( | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The README notes that "Monocle's exporters also see Langfuse's spans when both are enabled," but this runtime warning only mentions Monocle's own trace data. A user running Langfuse + |
||
| "Monocle is exporting trace data (prompts, tool inputs/outputs, completions) beyond the local .monocle/ file via: %s. Make sure that destination is trusted.", | ||
| ", ".join(non_file), | ||
| ) | ||
|
|
||
| try: | ||
| from monocle_apptrace import setup_monocle_telemetry | ||
| except ImportError as exc: | ||
| raise RuntimeError("MONOCLE_TRACING is enabled but monocle_apptrace is not installed. Install the 'monocle' extra: `uv sync --extra monocle` in backend/, or `pip install 'deerflow-harness[monocle]'`.") from exc | ||
|
|
||
| setup_monocle_telemetry(workflow_name="deer-flow", monocle_exporters_list=exporters) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Security: spans capture prompts + tool I/O verbatim, with no DeerFlow-controlled scrubbing layer. Auto-instrumentation records span inputs/outputs — prompts, tool arguments ( Two suggestions: (1) log a loud startup warning whenever a non-
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This call installs a process-global OTel The mitigation (off by default, optional extra) is sound - just flagging this as the one coupling to weigh against the |
||
| logger.info("Monocle telemetry enabled (exporters=%s)", exporters) | ||
| return True | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -65,6 +65,9 @@ postgres = [ | |
| redis = ["redis>=5.0.0"] | ||
| pymupdf = ["pymupdf4llm>=0.0.17"] | ||
| boxlite = ["boxlite>=0.9.7"] | ||
| # Agent observability (Monocle). Optional so a default install stays free of the | ||
| # OpenTelemetry stack; only pulled in when MONOCLE_TRACING is used. | ||
| monocle = ["monocle_apptrace>=0.8.8"] | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
|
|
||
| [build-system] | ||
| requires = ["hatchling"] | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
User-facing version of the same Langfuse-conflict claim (see inline on
monocle.py)."...so it runs alongside LangSmith but not Langfuse" is stated as a flat rule, but rests on the same unverified assumption that Langfuse's callback path owns the global OTel provider. Worth softening ("may conflict; verify") or removing until the interaction is confirmed — otherwise Langfuse users are told they can't use Monocle when they likely can.