Zwep

A search engine for the sources you trust — and nothing else.

Zwep crawls a list of sites you choose, builds its own index, and answers your searches from it. No ads, no SEO spam, no tracking. It runs on your machine, and your queries never leave it.

What is this? · Is it for you? · Install · Usage · For AI agents · Docs

---

What is this?

Web search optimises for the average query across billions of people. That makes it very good at "weather tomorrow" and increasingly bad at finding the fifty sites you actually rely on — those are buried under content farms optimised to rank, not to be right.

Zwep flips the problem around. You pick the corpus. Add the sites that matter to you; Zwep crawls them, extracts clean text, scores each page for quality, and gives you a fast, faceted search over exactly that material. Every result comes from something you vouched for.

It is three things in one repository:

A crawler and indexer	Fetches your sources politely, extracts text and metadata, and indexes it in Meilisearch
A search API and web UI	A clean REST API with a fast browser interface on top
A CLI and MCP server	Search from your terminal, or give an AI agent your private corpus as a tool

Zwep respects robots.txt, rate-limits itself, and identifies as ZwepBot/1.0. It is not a general-purpose web scraper.

---

Is Zwep for you?

Yes, if you are:

• A researcher or analyst who keeps returning to the same 30 journals, agencies or company sites, and wants one search across all of them.
• A developer who wants to search a set of documentation sites without the results being polluted by tutorial spam and outdated Stack Overflow copies.
• Someone running a publication or a site who wants real search over their own archive, with ranking they control.
• Building with AI agents and tired of the agent citing a content farm. A curated index means every retrieved source already passed your judgement — see For AI agents.
• Privacy-minded. Your queries are answered locally. The UI loads no third-party scripts, analytics, or even web fonts.

Probably not, if you:

• Want to search the whole web. Zwep only knows what you tell it to crawl.
• Want a hosted service you sign up for. Zwep is self-hosted, single-tenant.
• Need multi-user accounts and per-user permissions. There is one admin key, not a user model — see SECURITY.md.
• Want to crawl sites that forbid it. Zwep honours robots.txt by design.

---

Install

There are two pieces, and you may only want the first.

1. The CLI and MCP server — from npm

Zero dependencies, works anywhere Node runs. This is what you install to use a Zwep, or to connect an AI agent to one.

npm install -g zwep

Or run it without installing:

npx zwep --help

It talks to a Zwep API over HTTP, so it works against a Zwep on your laptop, in a container, or on another machine:

export ZWEP_API=http://127.0.0.1:8080     # the default
zwep status

You usually do not need to start anything yourself. If a command needs the API and a Zwep is installed but not running, the CLI starts it in the background and waits for it:

$ zwep search "climate policy"
› Zwep is not running. Starting it from /home/you/zwep…
✓ Zwep is up. Run `zwep down` to stop it.

12 results for "climate policy" · 8 ms
…

Use zwep up / zwep down to control it explicitly, and --no-auto-start (or ZWEP_NO_AUTOSTART=1) to turn the behaviour off. Auto-start never fires when ZWEP_API points at a remote host — starting a local server would search a different index than you asked for.

2. The search engine itself — from source

This is the crawler, indexer, API and web UI. It needs Node 22.6+ and Meilisearch. Works on Windows, macOS and Linux.

Docker is the easiest way to get Meilisearch, but it is not required — a standalone binary works just as well. zwep doctor shows the steps for your platform either way.

# Get the code
git clone https://github.com/philppplik/zwep
cd zwep
npm install

# Configure — the defaults work for local development
cp .env.example .env

# Start Meilisearch (needs Docker; see below if you do not have it)
npm run infra:up

# Start the API (:8080) and the web UI (:5173)
npm run dev

Open http://localhost:5173. Then index something:

npx zwep crawl example          # the built-in smoke-test source
npx zwep search "example domain"

Platform notes

npm run dev is a Node script, so the same command works in PowerShell, cmd, Git Bash, zsh and bash. It health-checks the API before telling you the UI is ready, and forwards Ctrl-C to both processes — including on Windows, where it tears down the whole process tree.

Windows: start Docker Desktop before npm run infra:up. If better-sqlite3 fails to build, install the Visual Studio Build Tools with the "Desktop development with C++" workload, then run npm rebuild.

macOS (Apple Silicon): everything runs natively; Meilisearch's image is multi-arch.

Linux: if Playwright's Chromium fails to launch, run npx playwright install-deps chromium.

Without Docker

Zwep only needs Meilisearch reachable at MEILI_HOST. Install it natively and start it with a matching key:

meilisearch --master-key=zwep_dev_master_key_change_me

Redis is optional today — the crawler runs inline.

Something not working? Ask the tool

npx zwep doctor

It checks Node, the engine, .env, Docker, Meilisearch, the API and the ports — and for anything wrong, prints the exact command that fixes it. It also separates optional from required, so a missing Docker does not read like a broken install.

✗ Meilisearch    not reachable at http://127.0.0.1:7700

    Nothing is answering at http://127.0.0.1:7700. Pick whichever suits you:

      Docker is NOT installed. Either:

      a) Install Docker Desktop, then `npm run infra:up`
         https://docs.docker.com/get-docker/

      b) Run Meilisearch directly, no Docker needed:
         curl -L https://install.meilisearch.com | sh
         meilisearch --master-key=zwep_dev_master_key_change_me

Troubleshooting table

Symptom	Cause and fix
Search returns 500, or the Library shows ECONNREFUSED	The API is not running. The web UI proxies /v1/* to port 8080. Use npm run dev, which starts both.
index_unavailable	Meilisearch is unreachable. Run npm run infra:up, then check http://localhost:7700/health.
docker not found on npm run infra:up	Docker is optional. Meilisearch also ships a standalone binary — zwep doctor prints the steps for your platform.
EADDRINUSE, or the API exits at once	Something already has the port: API_PORT=8081 npm run dev. Or it is Zwep itself — check with zwep status.
Library says "Admin key required"	Set it under Settings → Admin access. In development it is zwep_admin_dev_key.
A crawl indexes 0 documents	The source's allowedDomains may not cover the seed's host, or robots.txt disallows ZwepBot. Run zwep crawl <source> --json to see skipped versus failed.
zwep: command not found	npm install -g zwep, or use npx zwep.
AI overview never appears	It is off by default. Enable it in Settings and pick a provider. For ollama, make sure ollama serve is running.

Still stuck? zwep doctor --json and zwep status --json print everything Zwep knows about its own state — include them when you open an issue.

---

Usage

Curating sources

A source is a set of seed URLs plus the domains the crawler may follow. Manage them in the Library UI, through the admin API, or from the terminal:

zwep sources add my-docs --seed https://example.com/docs --max-pages 200
zwep crawl my-docs
zwep sources list

Disabling a source keeps its configuration but drops it out of search results — the quickest way to see how a source affects your relevance.

Two kinds of source exist:

• web (default) — seeds plus allowedDomains. The crawler stays inside the allow-list, follows a sitemap if you give it one, and de-duplicates by canonical URL.
• google — a list of queries is run against Google and the result URLs are crawled. Off by default (GOOGLE_PROXY_ENABLED), because enabling it means your queries leave your machine. It is also fragile: Google serves a JavaScript-only shell to non-browser clients, so a plain fetch often returns nothing. It degrades gracefully rather than crashing.

Searching

From the browser at http://localhost:5173, or from the terminal:

zwep search "climate policy"
zwep search "climate policy" --source my-docs --type article --limit 5
zwep overview "climate policy"     # AI summary of the top results
zwep graph "climate"               # knowledge-graph neighbourhood
zwep repl                          # interactive session

Every command takes --json, and colour switches itself off when stdout is not a terminal — so piping into jq or a script always gives clean output:

zwep search "climate" --json | jq -r '.results[] | "\(.quality.score)\t\(.url)"'

All commands

SEARCH
  search <query>        Search the index
  suggest <partial>     Title autocompletions
  doc <id>              Print one document in full
  overview <query>      AI summary of the top results
  graph <term>          Knowledge-graph neighbourhood
  repl                  Interactive search session

SETUP
  doctor                Check this machine and explain every problem

SERVER
  up [--web]            Start a local Zwep in the background
  down                  Stop the Zwep this CLI started
  status                Index health and counts
  update                Check for a newer release

INDEX  (needs ZWEP_ADMIN_KEY)
  sources list          List curated sources
  sources add <name> --seed <url>[,<url>] [--domain d] [--max-pages N]
  sources enable|disable|rm <name>
  crawl <source>        Crawl one source (--all for every enabled one)
  index <url>           Crawl and index a single URL
  deindex --yes         Delete every indexed document

AGENTS
  mcp [--allow-write]   Run as an MCP server on stdio
  --json                Machine-readable output for any command

Staying up to date

zwep update                  # check, and print the right command for your install
npm install -g zwep@latest   # the CLI

The engine updates separately — git pull && npm install in your checkout. Settings → Updates shows both, and the web UI surfaces a notice when a release is out.

The check asks registry.npmjs.org for a version number and nothing else. It is made by the server, never the browser, so the registry never sees a referrer carrying your search. It runs at most once a day, and ZWEP_NO_UPDATE_CHECK=1 (or the toggle in Settings) switches it off entirely.

Plain HTTP

The REST API needs no client library:

curl "http://localhost:8080/v1/search?q=klimapolitik&limit=5&facets=true"

The full contract is in docs/api.md.

---

For AI agents

An agent's weakness on the open web is credulity — it cannot tell a primary source from a content farm. A curated index removes that problem by construction: every result already carries your judgement.

MCP server

zwep mcp speaks the Model Context Protocol over stdio. Register it once and any MCP-capable client — Claude Code, Claude Desktop, Cursor — can search your index as a first-class tool:

{
  "mcpServers": {
    "zwep": {
      "command": "npx",
      "args": ["-y", "zwep", "mcp"],
      "env": { "ZWEP_API": "http://127.0.0.1:8080" }
    }
  }
}

Tool	What it does
zwep_search	Ranked results with title, URL, excerpt, source and quality score
zwep_fetch_document	One document in full, truncated to protect the context window
zwep_list_sources	What the corpus actually covers
zwep_graph	Entities related to a term
zwep_stats	Index health

Read-only by default. The crawl tools (zwep_crawl_url, zwep_crawl_source) only appear when the server is started with --allow-write and an admin key is present — an agent that can index arbitrary URLs is a very different trust level from one that can only read what you curated:

{
  "command": "npx",
  "args": ["-y", "zwep", "mcp", "--allow-write"],
  "env": {
    "ZWEP_API": "http://127.0.0.1:8080",
    "ZWEP_ADMIN_KEY": "your-key"
  }
}

Tool failures come back inside the result with isError: true, not as transport errors, so an agent can read the message and recover instead of losing its turn.

---

How it works

 sources.yaml ─┐
               ├──▶ Crawler ──▶ Extractor ──▶ Indexer ──▶ Meilisearch
 Library UI  ──┤    robots.txt   Readability   quality       │
 CLI / API   ──┘    politeness   JSON-LD       scoring       │
                         │                                   │
                         └──▶ Knowledge graph (SQLite)       │
                                                             ▼
                                              Search API (Fastify, :8080)
                                                             │
                                          ┌──────────────────┼─────────────┐
                                          ▼                  ▼             ▼
                                      Web UI            zwep CLI      MCP agents

Stage	What it does
Crawler	Fetches allow-listed URLs. Honours robots.txt and crawl-delay, serializes requests per host, caps time and response size, and renders JavaScript-heavy pages with Playwright only when the raw HTML is too thin.
Extractor	Readability for the body, plus og:, JSON-LD and heading metadata. Normalizes language to ISO 639-1 and dates to ISO 8601.
Quality	Scores every document 0–100 from content depth, freshness, title clarity and structure. The breakdown is visible in the UI.
Indexer	Meilisearch behind an adapter interface, so the engine can be swapped. Typo tolerance, facets, optional hybrid vector search.
Graph	Extracts entities and co-mentions into SQLite. No extra container.
API	Fastify. /v1/search, /v1/suggest, /v1/document/:id, /v1/graph, /v1/overview, plus admin routes.

The whole web UI is about 18 kB gzipped, with no framework. Full detail in docs/architecture.md.

---

Configuration

Everything is environment variables; .env.example documents all of them. The ones that matter most:

Variable	Default	Notes
ZWEP_API	http://127.0.0.1:8080	Which Zwep the CLI and MCP server talk to.
ZWEP_ADMIN_KEY	zwep_admin_dev_key	Change this before exposing the API.
MEILI_MASTER_KEY	zwep_dev_master_key_change_me	Change this too.
API_HOST	127.0.0.1	Loopback by default. Widen only behind a proxy.
API_CORS_ORIGINS	*	Set to your real origin in production.
API_RATE_LIMIT	120	Requests per minute per IP. 0 disables.
GOOGLE_PROXY_ENABLED	false	Opt-in. Sends queries to Google.
EMBED_PROVIDER	none	ollama or openrouter enables semantic search.
LLM_PROVIDER	none	ollama (local) or openrouter (cloud) for AI Overview.

Read SECURITY.md before putting Zwep on a network — it states plainly what the security model does and does not cover.

Optional: semantic search and AI overview

Both are off by default, and both work fully locally with Ollama:

ollama pull nomic-embed-text   # embeddings
ollama pull llama3.1           # overview generation

Set EMBED_PROVIDER=ollama and LLM_PROVIDER=ollama, or pick them in Settings — the UI pushes them to the server without a restart. With openrouter, queries and result excerpts go to a third party; with ollama, nothing leaves your machine. If a provider is unreachable, the feature disables itself for a minute and search continues unaffected.

---

Development

npm run dev            # API + web UI
npm test               # 259 tests, no network needed
npm run test:watch     # re-run affected tests as you type
npm run verify         # lint + typecheck + test (what CI runs)
npm run build          # production web bundle

The suite runs in a few seconds with nothing installed beyond npm ci — Meilisearch, Ollama and OpenRouter are all faked. CI runs it on Ubuntu, Windows and macOS across Node 22 and 24, plus CodeQL and a dependency audit.

See CONTRIBUTING.md for project layout, code style and review expectations.

---

Documentation

Document	Contents
docs/concept.md	Why a private index, who it is for, what is out of scope
docs/architecture.md	Components, data flow, storage, deployment
docs/api.md	REST endpoints, parameters, response and document schemas
docs/design.md	Design system: tokens, type scale, components
docs/roadmap.md	What is built, what is next, and why
CONTRIBUTING.md	Setup, style, testing, pull requests
SECURITY.md	Security model, deployment checklist, reporting
CHANGELOG.md	What changed in each release

---

Status

Working, and in use. The crawl → extract → index → search path is complete, covered by 259 tests, and runs on all three desktop platforms. The knowledge graph, semantic search and AI overview are functional and optional.

Not there yet: a built-in scheduler, incremental re-crawling, and multi-user access control. The reasoning and priorities are in docs/roadmap.md.

Zwep is pre-1.0. The REST contract in docs/api.md is stable in practice, but it may still change before 1.0; breaking changes are listed in CHANGELOG.md.

Contributing

Contributions are welcome. CONTRIBUTING.md covers how to get set up, what the code is expected to look like, and how pull requests are reviewed. Bug reports and feature requests go through the issue templates; security reports go through a private advisory.

Everyone taking part is expected to follow the Code of Conduct.

Related

The cust*m Tab browser extension is one possible client — it can offer "search Zwep" from its dashboard. Neither project depends on the other's internals, only on the API contract.

License

MIT © Philipp Paulik

Free to use, modify, distribute and sell, including commercially. The only condition is that the copyright notice and licence text travel with the code. The software comes with no warranty.