Repository navigation
docs(observability): list agentest under evaluation #9483
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Houyun Sun (sunhouy)
wants to merge
2
commits into
crewAIInc:main
Choose a base branch
from
sunhouy:docs/observability-agentest
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| --- | ||
| title: agentest | ||
| description: Regression-test CrewAI agents with deterministic checks, with no LLM calls in the test loop. | ||
| icon: flask-vial | ||
| mode: "wide" | ||
| --- | ||
|
|
||
| # agentest Overview | ||
|
|
||
| [agentest](https://github.com/srpadmanabhuni/agentest) is an open-source, deterministic test runner for AI agents. It ships a set of executable checks that assert on the parts of an agent that tend to break quietly: tools that get called with the wrong arguments, guardrails that stop firing, prompts that lose an instruction after an edit, and outputs that change shape between runs. | ||
|
|
||
| The checks are plain Python functions, so they run next to your crew's tests and in CI. Nothing in the test loop calls a model, which means the same run gives the same answer on your laptop and on a build machine. | ||
|
|
||
| ## What the checks cover | ||
|
|
||
| - **Tool usage**: which tools an agent picked, in what order, and with which arguments. You can assert a tool was never called, or that it was called with one specific argument. | ||
| - **Output contracts**: required fields, types, ranges and forbidden content on the final answer. | ||
| - **Prompt and guardrail state**: that an instruction is still in the prompt, and that a guardrail trips on the inputs it is supposed to trip on. | ||
| - **Trajectory**: the order of the steps, so a refactor that reshuffles the work fails instead of just producing a nicer log. | ||
|
|
||
| ## Setup | ||
|
|
||
| <Steps> | ||
| <Step title="Install"> | ||
| ```shell | ||
| pip install "git+https://github.com/srpadmanabhuni/agentest" | ||
| ``` | ||
|
|
||
| The distribution is called `agentest-ai` and is not on PyPI yet; the bare | ||
| `agentest` name there belongs to an unrelated package. Install from GitHub | ||
| until the release lands. | ||
| </Step> | ||
| <Step title="Write a check"> | ||
| A check is a function that asserts on the agent it is handed. See the [agentest README](https://github.com/srpadmanabhuni/agentest) for the exact import path in the version you install. | ||
|
|
||
| ```python | ||
| from crewai import Agent | ||
| from crewai_tools import ScrapeWebsiteTool | ||
|
|
||
| def test_researcher_keeps_its_scraper(): | ||
| agent = Agent( | ||
| role="Researcher", | ||
| goal="Find primary sources", | ||
| backstory="Reads filings for a living.", | ||
| tools=[ScrapeWebsiteTool()], | ||
| ) | ||
| assert [tool.name for tool in agent.tools] == ["Read website content"] | ||
| ``` | ||
|
|
||
| agentest's own checks plug in alongside this one. The important part is that | ||
| a failure here reads like a normal test failure, not like a diff of two | ||
| agent transcripts. | ||
| </Step> | ||
| <Step title="Run it"> | ||
| ```shell | ||
| pytest tests/ -q | ||
| ``` | ||
| </Step> | ||
| </Steps> | ||
|
|
||
| ## How it fits with the rest of this section | ||
|
|
||
| The tracing platforms here (Langfuse, Opik, Arize Phoenix, Weave, LangDB and the others) tell you what a crew did on a real run. agentest covers the other half: it takes what you learned from that trace and turns it into a check that runs on every commit, before a crew reaches a model at all. | ||
|
|
||
| Most teams end up with both. Trace locally while the crew is still changing shape, then add a check for every trace you had to open twice. | ||
|
|
||
| If you want to compare scored runs instead of asserting on them, `crewai test` is the built-in option; see [Testing](/en/concepts/testing). | ||
|
|
||
| ## Resources | ||
|
|
||
| - [agentest on GitHub](https://github.com/srpadmanabhuni/agentest) | ||
| - [CrewAI Testing docs](/en/concepts/testing) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔒 Agentic Security Review
Severity: CRITICAL
Description: The new observability page tells readers to
pip install "git+https://github.com/srpadmanabhuni/agentest"with no commit SHA, tag, or locked revision. That URL tracks the default branchmainof a personal GitHub repo CrewAI does not own. pip builds whatever is on that branch at install time, and the same page tells readers to run the resulting checks beside their tests and in CI.Impact: Anyone who can push to github.com/srpadmanabhuni/agentest, or who later takes over that account, can change the code that official CrewAI docs instruct customers and CI to fetch and execute. A later push needs no further docs edit. Install runs the repo build backend as the installing user, and the page sends follow-on test imports and CI runs at that same moving tree, so the follower's source and environment credentials are in the blast radius. Sibling observability pages install named PyPI distributions; this is the only
git+https://installer in the docs. The repo's own README already uses a tag pin (@v3.3.2), whilemain's pyproject is already3.4.0, so the documented command is not a reviewed snapshot.Remediation: Remove the floating VCS install from the published page until a reviewed artifact exists. A full 40-character commit SHA that CrewAI has reviewed is an acceptable interim pin if a GitHub install must stay; a git tag is not, because tags move. Prefer a versioned release under a controlled package name once it exists. Keep the warning not to install the unrelated PyPI project named agentest.