Privacy-first, fully local document-to-Markdown extraction pipeline for macOS Apple Silicon and Linux, with a lightweight GLM-OCR-only mode for low-RAM Windows machines.
- Local-only processing (no cloud APIs)
- Parser-first routing with explainable route decisions
- LM Studio or Ollama localhost integration via
glm-ocrfor vision OCR and table repair - Outputs per file:
- Markdown (
.md) - structured audit JSON (
.json) - extraction log (
.log)
- Markdown (
- Resumable state (
job_state.json) - Confidence scoring (0-100) with automatic retries below threshold (
95by default) - Table-first validation and recovery logic
- CLI and local drag-and-drop GUI
pdfdocdocxcssxlsxlsxtxtpptpptxpng,jpg,jpeg,tiff,tif,bmp(via GLM-OCR mode)
./scripts/setup_macos.shThis installs local dependencies (tesseract, ocrmypdf, LibreOffice, Java) and Python packages.
uv venv .venv
source .venv/bin/activate
uv pip install -e .[full,dev]
cp config.sample.toml config.tomlInstall native dependencies with Homebrew:
brew install tesseract ocrmypdf poppler openjdk
brew install --cask libreofficeThe lite install runs the GLM-OCR pipeline only: PDFs and images go through a local vision model (glm-ocr, 1.1B parameters, ~2.2 GB), and txt/css are read directly. It needs no Docling/PyTorch, Tesseract, LibreOffice, or Java — every Python dependency ships as a plain wheel.
powershell -ExecutionPolicy Bypass -File scripts\setup_windows.ps1The script creates .venv, installs the [lite] extras, installs Ollama (via winget when available), pulls glm-ocr, and writes a lite config.toml. Manual equivalent:
python -m venv .venv
.venv\Scripts\pip install -e ".[lite]"
# install Ollama from https://ollama.com/download/windows, then:
ollama pull glm-ocr
.venv\Scripts\localdataextractor init-config --lite config.toml
.venv\Scripts\localdataextractor --config config.toml check-llmThe lite profile sets provider = "ollama", extraction_mode = "glm_ocr", and max_workers = 1 — on an 8 GB machine the loaded model (~3 GB) needs the headroom, so keep one worker and close heavy apps while processing. PDF pages stream through the model one at a time, so peak app memory stays low even for large documents. The first request after Ollama starts loads the model and is noticeably slower; the lite profile ships a 300 s GLM-OCR timeout for that reason.
Formats the lite install cannot parse (doc(x), ppt(x), xls(x)) are skipped from routing with a per-file diagnostic; install the [full] extras plus the native tools to process them.
The pipeline finds every native tool it shells out to (soffice, tesseract, ocrmypdf, java) through PATH, so the full feature set runs on Linux with no code changes — only the install commands differ from macOS.
# Debian / Ubuntu
sudo apt update
sudo apt install -y python3-venv tesseract-ocr tesseract-ocr-eng \
ocrmypdf poppler-utils libreoffice default-jre
# Fedora
sudo dnf install -y tesseract tesseract-langpack-eng ocrmypdf \
poppler-utils libreoffice java-latest-openjdk
# Arch
sudo pacman -S --needed tesseract tesseract-data-eng ocrmypdf \
poppler libreoffice-fresh jre-openjdkdefault-jre/jre-openjdk is only needed if you enable the Tika route ([tika].enabled = true, off by default). LibreOffice is only needed for legacy doc/ppt/xls conversion.
The GUI is Qt (PySide6) and needs the X11 platform libraries. If it exits with Could not load the Qt platform plugin "xcb", install them:
# Debian / Ubuntu
sudo apt install -y libxcb-cursor0 libxcb-icccm4 libxcb-keysyms1 \
libxcb-shape0 libxkbcommon-x11-0 libegl1Headless servers can skip that entirely and use the CLI, which never imports Qt.
Python 3.11 or newer is required. Same uv flow as macOS:
curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:$PATH" # uv installs here; add to your shell rc
uv venv .venv # or: uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install -e ".[full,dev]"
cp config.sample.toml config.tomluv venv --python 3.12 downloads and uses a managed interpreter when the distro's Python is too old, so uv alone is enough — the python3-venv package above is only needed for the stdlib fallback (python3 -m venv .venv + pip install -e ".[full,dev]").
On a machine without an NVIDIA GPU, install PyTorch from the CPU index first — Docling pulls it in transitively, and the default Linux wheel bundles the CUDA runtime (several GB):
uv pip install torch --index-url https://download.pytorch.org/whl/cpu
uv pip install -e ".[full,dev]"For low-RAM machines or servers, the [lite] profile described in the Windows section works identically on Linux (pure wheels, GLM-OCR only, no Docling/PyTorch, Tesseract, LibreOffice, or Java):
uv pip install -e ".[lite]"
localdataextractor init-config --lite config.tomlcurl -fsSL https://ollama.com/install.sh | sh
ollama pull glm-ocrThe installer registers a systemd service listening on 127.0.0.1:11434; without systemd, run ollama serve yourself. Then set provider = "ollama" under [llm] in config.toml (already set if you used init-config --lite) and verify:
localdataextractor --config config.toml check-llmLM Studio also ships a Linux AppImage if you prefer that backend on 127.0.0.1:1234.
source .venv/bin/activate
localdataextractor --config config.toml ingest ./input ./output --explain-route
localdataextractor-gui # or: python gui.pyWithout activating, prefix commands with uv run --no-sync (e.g. uv run --no-sync localdataextractor-gui). Keep --no-sync: a plain uv run syncs .venv to the lockfile's default dependency set, which has no extras, and would uninstall the [full]/[lite] packages you just installed. Use uv run --extra full instead if you want uv to manage the sync itself.
The macOS .app has no Linux equivalent in this repo, but a .desktop entry does the same job. Path= is what matters: it sets the working directory, which is how config.toml, .env, and the default output/ folder get resolved.
ROOT="$(pwd)"
mkdir -p ~/.local/share/applications
cat > ~/.local/share/applications/localdataextractor.desktop <<EOF
[Desktop Entry]
Type=Application
Name=localDataExtractor
Comment=Local document to Markdown extraction
Exec=$ROOT/.venv/bin/python $ROOT/gui.py
Path=$ROOT
Icon=$ROOT/src/localdataextractor/assets/logo_light.png
Terminal=false
Categories=Utility;Office;
EOF
update-desktop-database ~/.local/share/applications 2>/dev/null || trueIt then appears in the application menu. For an icon on the desktop itself, copy the file to ~/Desktop/, chmod +x it, and — on GNOME — right-click it and choose "Allow Launching".
config.tomlis local-only and gitignored; commitconfig.sample.tomlinstead.- Generated runtime folders (
output/,_out_smoke/) are gitignored. - If this directory is nested in another git repo, initialize and push from this folder:
cd /path/to/localDataExtractor
git init
git add .
git commit -m "Initial public release"The pipeline talks to one local OpenAI-compatible server. Two backends are supported, selected with [llm].provider in config.toml (or the GUI "LLM backend" dropdown):
| Provider | Default server URL |
|---|---|
lmstudio (default) |
http://127.0.0.1:1234/v1 |
ollama |
http://127.0.0.1:11434/v1 |
When base_url is omitted from config.toml, the selected provider's default URL is used automatically.
- Start LM Studio local server on
http://127.0.0.1:1234/v1. - Load the
glm-ocrmodel (used for both vision OCR and table repair). - Run:
localdataextractor --config config.toml check-llm- Install Ollama (
brew install ollamaor the macOS app) and make sure the server is running (ollama serve, or the menu-bar app). - Pull the vision OCR model:
ollama pull glm-ocr- Set the provider in
config.toml:
[llm]
provider = "ollama"- Run:
localdataextractor --config config.toml check-llmNotes:
- Ollama lists the model as
glm-ocr:latest; the configuredprimary_model = "glm-ocr"matches it automatically. - The first request after startup loads the model into memory and can take noticeably longer; raise
timeout_secondsif cold starts time out. - Ollama unloads idle models after a few minutes, so long jobs may pay the load cost again mid-run.
localdataextractor --config config.toml ingest ./input_docs ./output --explain-route --max-workers 2Options:
--include "**/*.pdf"(repeatable)--exclude "**/~$*"(repeatable)--dry-run--verbose--max-workers <int>--explain-route--mode standard|glm_ocr|highest_accuracy--pii/--no-pii(redact PII in extracted outputs; overrides[pii]config)--pii-mapping(also write a per-file placeholder→original mapping file; implies--pii)
localdataextractor --config config.toml resume ./output/job_state.json --explain-routelocaldataextractor verify ./outputlocaldataextractor init-config ./config.toml
# GLM-OCR-only lite profile for low-RAM machines:
localdataextractor init-config --lite ./config.tomllocaldataextractor-gui
# or
python gui.pyGUI supports:
- drag-and-drop files/folders
- output folder picker
- LLM backend selector (LM Studio / Ollama), seeded from
config.toml; the dropdown choice wins for the session, whileconfig.tomlis the persistent setting - API token field for servers that require auth
- extraction mode selector (Standard, GLM-OCR, Highest Accuracy)
- PII redaction checkbox (redacts extracted outputs, see "PII screening and redaction")
- GLM-OCR model availability indicator (checks the selected backend)
- start processing
- resume interrupted jobs
- per-file status panel with route, retries, confidence, outcome
- live log/error panel
./scripts/make_macos_app.sh # -> ~/Desktop/localDataExtractor.app
./scripts/make_macos_app.sh /ApplicationsThis writes a small .app bundle that cds into this checkout and starts the
GUI with .venv/bin/python gui.py, so config.toml, .env, and the default
output/ folder resolve exactly as they do from a terminal run. Double-click
the Desktop icon to launch it.
The bundle hardcodes the checkout and venv paths — re-run the script after
moving the project folder or recreating .venv. Launch output goes to
~/Library/Logs/localDataExtractor/gui.log.
For each source file:
output/<relative_path>/<basename>.mdoutput/<relative_path>/<basename>.jsonoutput/<relative_path>/<basename>.log- optional debug files under
output/<relative_path>/_debug/
Job-level resumable state:
output/job_state.json
With PII redaction enabled:
output/pii_report.jsonandoutput/pii_report.md(job-level summary)- optional
output/<relative_path>/<basename>.pii_map.json(only with--pii-mapping)
Three extraction modes are available via --mode (CLI) or the GUI dropdown:
| Mode | Behavior |
|---|---|
standard (default) |
Parser-first routing per file type; stops at first result meeting the confidence threshold |
glm_ocr |
GLM-OCR vision model as primary extractor (PDFs and images), with standard routes as fallback |
highest_accuracy |
Tries all available parsers including GLM-OCR and selects the highest confidence result |
GLM-OCR mode requires the glm-ocr model loaded in LM Studio or pulled in Ollama. It renders each PDF page to an image and sends it to the vision model for high-accuracy OCR of text, tables, and graphics.
txt,css:- deterministic direct read
- no LLM usage
- CSS wrapped in fenced
csscode block
pdf:- Docling first
- OCR-assisted Docling retry via OCRmyPDF
- Tika fallback when enabled
docx,pptx,xlsx,xls:- MarkItDown first
- Docling fallback
- Tika fallback when enabled
doc,ppt:- LibreOffice headless conversion to modern format
- then MarkItDown/Docling path
- Tika fallback when enabled
pdf,png,jpg,jpeg,tiff,tif,bmp:- GLM-OCR vision model first
- standard routes as fallback
- all other formats: standard routing
- all standard routes tried first
- GLM-OCR appended for PDFs and images
- no early stopping: every parser runs, best confidence wins
In every mode, routes whose parser dependencies are not installed (e.g. Docling on a lite install) are skipped at planning time; the first remaining route's detail notes what was skipped. If no route survives, the original plan is kept so the file fails with per-parser diagnostics.
Every decision is logged in route history and can be printed with --explain-route.
- Confidence range:
0-100 - Default threshold:
95 - Retries trigger when:
- overall confidence
< 95, or - any important table confidence
< 95
- overall confidence
- Each attempt stores:
- timestamp
- route and parser used
- model used (if any)
- confidence details
- retry reasons
- warnings and artifacts
- If all strategies fail to reach threshold, best result is emitted and marked
below_threshold=true.
Tables are first-class and include:
- table id
- source location metadata
- header/body rows
- row/column counts
- merged-cell metadata (when recoverable)
- confidence score
- validation warnings
Validation heuristics include:
- column consistency
- weak header detection
- duplicate-row ratio
- empty-cell anomaly ratio
- collapse risk
If table confidence is below [retry].confidence_threshold, the table is sent to the local model for normalization (disable with [llm] enable_vlm_repair = false).
Repair asks the model to echo every cell back, so request time scales with table size — roughly 4 s for a 4-row table and 90 s for a 30-row one on a small local model. Tables larger than [llm] repair_max_rows (default 40) are therefore skipped with a warning rather than run into the timeout.
Responses are structurally validated before they are applied. A repair is rejected — leaving the original table untouched — when the model omits body_rows, collapses the body into header_rows, drops or invents rows, or collapses a multi-column table into one column. Each rejection is recorded as a per-file warning naming the reason.
Structural validation cannot detect rewritten cell values. OCR-tuned models are prone to that: in local testing glm-ocr altered euro amounts while echoing a 30-row table back, where gemma3:4b returned them intact in comparable time. Point repair at a general text model when values matter:
[llm]
primary_model = "glm-ocr" # vision OCR
repair_model = "gemma3:4b" # text normalizationRepair also gives up per file after [llm] repair_failure_limit consecutive failures (default 3), instead of spending the timeout on every remaining table and again on every route attempt. Tables that were repaired successfully are not re-sent within the same file.
localdataextractor verify <output> checks:
- required output files exist (
.md,.json,.log) - JSON validity and schema keys
- confidence fields and below-threshold flag
- route history presence
- table presence in JSON and Markdown approximation when tables were detected
The [glm_ocr] section in config.toml controls vision-based OCR:
[llm]
provider = "lmstudio" # lmstudio | ollama
# base_url = "http://127.0.0.1:1234/v1" # optional; defaults per provider
primary_model = "glm-ocr"
fallback_model = "glm-ocr"
[glm_ocr]
enabled = true # enabled by default; glm-ocr serves as the single model
model_name = "glm-ocr" # model ID as served by the backend ("glm-ocr" matches Ollama's "glm-ocr:latest")
page_dpi = 300 # DPI for rendering PDF pages to images
max_pages = 50 # safety cap on pages sent to vision API
timeout_seconds = 180 # per-request timeout (vision is slower)The extraction mode can also be set in config:
[routing]
extraction_mode = "standard" # standard | glm_ocr | highest_accuracy- Docling/MarkItDown APIs may vary across versions; fallbacks are included.
- Legacy
doc/pptfidelity depends on local LibreOffice conversion quality. xlssupport depends on localxlrdavailability.- Tika fallback requires Java runtime and local
tika-app.jarpath in config. - Markdown tables may lose some advanced layout semantics; JSON table representation is authoritative.
- GLM-OCR mode processes pages one at a time; large PDFs will be slow. Use
max_pagesto cap. - Highest Accuracy mode runs all parsers and does not stop early, so it is slower than other modes.
Opt-in screening of extracted outputs for personally identifiable
information, with redaction and a findings summary. Enable via
[pii] enabled = true in config.toml, the --pii CLI flag, or the
"Redact PII in outputs" checkbox in the GUI. Off by default.
Detection is hybrid and fully local:
- Regex + checksum validators (stdlib only) for structured PII: emails, phone numbers (German + international), IBANs (mod-97), credit cards (Luhn), German tax IDs (IdNr checksum), Steuernummer/USt-IdNr, SSNs, German ID-card/passport numbers (keyword-gated, MRZ check digit), bank account numbers, and German street/PLZ addresses.
- The local LLM (same backend as extraction) finds person names and
free-form addresses. Disable with
use_llm = false; if the LLM server is down, regex results still apply and a warning is recorded.
Detected values are replaced with typed, numbered placeholders
([PERSON_1], [EMAIL_2]); the same value maps to the same placeholder
throughout a document. Dates are never redacted.
What gets redacted:
<basename>.mdand<basename>.json— all text surfaces including tables, metadata, and warnings. The JSON gains apiikey listing each placeholder with category, occurrence count, and page references (never the original values).pii_report.json/pii_report.mdat the output root summarize findings per file and per category.
Not redacted, by design: source documents (never touched), .log files,
_debug/ artifacts, and job_state.json. Delete _debug/ and .log
files before sharing outputs.
--pii-mapping (or write_mapping_file = true) additionally writes
<basename>.pii_map.json with the placeholder→original mapping for
de-redaction. This file contains the original PII values — handle it as
sensitive data.
- No cloud APIs are used by default.
- The LLM backend (LM Studio or Ollama) is enforced as localhost-only.
- Document processing remains on-device.
- Optional PII redaction of extracted outputs (see above).