Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -362,6 +362,7 @@
"edge/en/observability/neatlogs",
"edge/en/observability/openlit",
"edge/en/observability/opik",
"edge/en/observability/agentest",
"edge/en/observability/patronus-evaluation",
"edge/en/observability/portkey",
"edge/en/observability/weave",
Expand Down
72 changes: 72 additions & 0 deletions docs/edge/en/observability/agentest.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
---
title: agentest
description: Regression-test CrewAI agents with deterministic checks, with no LLM calls in the test loop.
icon: flask-vial
mode: "wide"
---

# agentest Overview

[agentest](https://github.com/srpadmanabhuni/agentest) is an open-source, deterministic test runner for AI agents. It ships a set of executable checks that assert on the parts of an agent that tend to break quietly: tools that get called with the wrong arguments, guardrails that stop firing, prompts that lose an instruction after an edit, and outputs that change shape between runs.

The checks are plain Python functions, so they run next to your crew's tests and in CI. Nothing in the test loop calls a model, which means the same run gives the same answer on your laptop and on a build machine.

## What the checks cover

- **Tool usage**: which tools an agent picked, in what order, and with which arguments. You can assert a tool was never called, or that it was called with one specific argument.
- **Output contracts**: required fields, types, ranges and forbidden content on the final answer.
- **Prompt and guardrail state**: that an instruction is still in the prompt, and that a guardrail trips on the inputs it is supposed to trip on.
- **Trajectory**: the order of the steps, so a refactor that reshuffles the work fails instead of just producing a nicer log.

## Setup

<Steps>
<Step title="Install">
```shell
pip install "git+https://github.com/srpadmanabhuni/agentest"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Agentic Security Review

Severity: CRITICAL

Description: The new observability page tells readers to pip install "git+https://github.com/srpadmanabhuni/agentest" with no commit SHA, tag, or locked revision. That URL tracks the default branch main of a personal GitHub repo CrewAI does not own. pip builds whatever is on that branch at install time, and the same page tells readers to run the resulting checks beside their tests and in CI.

Impact: Anyone who can push to github.com/srpadmanabhuni/agentest, or who later takes over that account, can change the code that official CrewAI docs instruct customers and CI to fetch and execute. A later push needs no further docs edit. Install runs the repo build backend as the installing user, and the page sends follow-on test imports and CI runs at that same moving tree, so the follower's source and environment credentials are in the blast radius. Sibling observability pages install named PyPI distributions; this is the only git+https:// installer in the docs. The repo's own README already uses a tag pin (@v3.3.2), while main's pyproject is already 3.4.0, so the documented command is not a reviewed snapshot.

Remediation: Remove the floating VCS install from the published page until a reviewed artifact exists. A full 40-character commit SHA that CrewAI has reviewed is an acceptable interim pin if a GitHub install must stay; a git tag is not, because tags move. Prefer a versioned release under a controlled package name once it exists. Keep the warning not to install the unrelated PyPI project named agentest.

```

The distribution is called `agentest-ai` and is not on PyPI yet; the bare
`agentest` name there belongs to an unrelated package. Install from GitHub
until the release lands.
</Step>
<Step title="Write a check">
A check is a function that asserts on the agent it is handed. See the [agentest README](https://github.com/srpadmanabhuni/agentest) for the exact import path in the version you install.

```python
from crewai import Agent
from crewai_tools import ScrapeWebsiteTool

def test_researcher_keeps_its_scraper():
agent = Agent(
role="Researcher",
goal="Find primary sources",
backstory="Reads filings for a living.",
tools=[ScrapeWebsiteTool()],
)
assert [tool.name for tool in agent.tools] == ["Read website content"]
```

agentest's own checks plug in alongside this one. The important part is that
a failure here reads like a normal test failure, not like a diff of two
agent transcripts.
</Step>
<Step title="Run it">
```shell
pytest tests/ -q
```
</Step>
</Steps>

## How it fits with the rest of this section

The tracing platforms here (Langfuse, Opik, Arize Phoenix, Weave, LangDB and the others) tell you what a crew did on a real run. agentest covers the other half: it takes what you learned from that trace and turns it into a check that runs on every commit, before a crew reaches a model at all.

Most teams end up with both. Trace locally while the crew is still changing shape, then add a check for every trace you had to open twice.

If you want to compare scored runs instead of asserting on them, `crewai test` is the built-in option; see [Testing](/en/concepts/testing).

## Resources

- [agentest on GitHub](https://github.com/srpadmanabhuni/agentest)
- [CrewAI Testing docs](/en/concepts/testing)
4 changes: 4 additions & 0 deletions docs/edge/en/observability/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,10 @@ Observability is crucial for understanding how your CrewAI agents perform, ident
<Card title="Patronus AI" icon="shield-check" href="/en/observability/patronus-evaluation">
Comprehensive evaluation platform for LLM outputs and agent behaviors.
</Card>

<Card title="agentest" icon="flask-vial" href="/en/observability/agentest">
Deterministic regression checks for agent behaviour, run in your own test suite.
</Card>
</CardGroup>

## Key Observability Metrics
Expand Down