Repository navigation
docs(observability): list agentest under evaluation - #9483
Houyun Sun (sunhouy) wants to merge 2 commits into
Conversation
Adds a page for agentest next to the Patronus evaluation page, plus a card on the observability overview and a nav entry in docs/docs.json. agentest runs deterministic checks over an agent (tool calls and their arguments, output shape, guardrail state) without calling a model, so it sits beside the tracing tools in this section rather than replacing them. The page keeps the setup short and points at the project README for the API, since that moves faster than a docs page can. Refs crewAIInc#9479
Preview approval requiredThis pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit |
Strix Security ReviewWarning This pull request has 1 commit after the last Strix review ( ✅ No open security findings. PR summary: Adds a documentation page for agentest under the Observability → Evaluation section, plus a card on the observability overview and a nav entry in docs/docs.json. Review summaryDocs-only change: a new observability page, one card on the overview, and one nav entry in docs/docs.json. No application code, no input handling, no authentication or query paths are touched, and the JSON still parses cleanly. Nothing in the added MDX is user-controlled — it is static prose compiled at build time — so there is no injection, XSS or deserialization surface here. One suspicion was investigated and rejected: the page instructs Ruled out (1)
Review 1 on this PR · Reviewed by Strix |
| <Steps> | ||
| <Step title="Install"> | ||
| ```shell | ||
| pip install "git+https://github.com/srpadmanabhuni/agentest" |
There was a problem hiding this comment.
🔒 Agentic Security Review
Severity: CRITICAL
Description: The new observability page tells readers to pip install "git+https://github.com/srpadmanabhuni/agentest" with no commit SHA, tag, or locked revision. That URL tracks the default branch main of a personal GitHub repo CrewAI does not own. pip builds whatever is on that branch at install time, and the same page tells readers to run the resulting checks beside their tests and in CI.
Impact: Anyone who can push to github.com/srpadmanabhuni/agentest, or who later takes over that account, can change the code that official CrewAI docs instruct customers and CI to fetch and execute. A later push needs no further docs edit. Install runs the repo build backend as the installing user, and the page sends follow-on test imports and CI runs at that same moving tree, so the follower's source and environment credentials are in the blast radius. Sibling observability pages install named PyPI distributions; this is the only git+https:// installer in the docs. The repo's own README already uses a tag pin (@v3.3.2), while main's pyproject is already 3.4.0, so the documented command is not a reviewed snapshot.
Remediation: Remove the floating VCS install from the published page until a reviewed artifact exists. A full 40-character commit SHA that CrewAI has reviewed is an acceptable interim pin if a GitHub install must stay; a git tag is not, because tags move. Prefer a versioned release under a controlled package name once it exists. Keep the warning not to install the unrelated PyPI project named agentest.
Related issue
Fixes #9479
Summary
Adds agentest to the Observability section under Evaluation & Quality Assurance, next to the Patronus page. Three files: the new page, one card on the overview, one nav entry in
docs/docs.json.The page is short on purpose. agentest's API is still moving, so instead of pasting a full integration guide I describe what the checks assert on (tool calls and their arguments, output shape, guardrail state, step order), then a minimal pytest example and a note on how it relates to the tracing pages in the same section: tracing tells you what a run did, agentest turns that into something that fails on the next commit.
The example uses
ScrapeWebsiteTooland asserts on its name, because that's the cheapest way to show a tool-level assertion without needing an LLM.Verification
Docs only, so no tests. What I did run:
lib/crewai+lib/crewai-toolsat 83d7407: returns['Read website content'], which is what the assertion expects.docs/docs.jsonparses as JSON after the edit; re-serialising the upstream blob withjson.dumps(indent=2)gives byte-identical output, so the diff is one added line (checked withdifflibagainst the blob sha).overview.mdxdiff is the card only;<CardGroup>nesting is unchanged.search/issues?q=repo:crewAIInc/crewAI+agentestreturns only [FEATURE] Docs: add agentest to observability evaluation listings #9479).Additional context
I did not add ar/ko/pt-BR copies. The observability pages in those locales are maintained by the docs team during the [docs-freeze] snapshots, and the page links to the project README for anything version-specific. Happy to add translations if you'd rather have them in the same PR.
The reporter asked for the
llm-generatedlabel on #9479; this PR was written by hand.