Skip to content

About

Privacy-first, fully local document-to-Markdown extraction pipeline for macOS Apple Silicon.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

localDataExtractor

Privacy-first, fully local document-to-Markdown extraction pipeline for macOS Apple Silicon and Linux, with a lightweight GLM-OCR-only mode for low-RAM Windows machines.

Features

  • Local-only processing (no cloud APIs)
  • Parser-first routing with explainable route decisions
  • LM Studio or Ollama localhost integration via glm-ocr for vision OCR and table repair
  • Outputs per file:
    • Markdown (.md)
    • structured audit JSON (.json)
    • extraction log (.log)
  • Resumable state (job_state.json)
  • Confidence scoring (0-100) with automatic retries below threshold (95 by default)
  • Table-first validation and recovery logic
  • CLI and local drag-and-drop GUI

Supported input formats

  • pdf
  • doc
  • docx
  • css
  • xls
  • xlsx
  • txt
  • ppt
  • pptx
  • png, jpg, jpeg, tiff, tif, bmp (via GLM-OCR mode)

Recommended setup (macOS)

./scripts/setup_macos.sh

This installs local dependencies (tesseract, ocrmypdf, LibreOffice, Java) and Python packages.

Manual setup

uv venv .venv
source .venv/bin/activate
uv pip install -e .[full,dev]
cp config.sample.toml config.toml

Install native dependencies with Homebrew:

brew install tesseract ocrmypdf poppler openjdk
brew install --cask libreoffice

Lightweight Windows setup (8 GB RAM)

The lite install runs the GLM-OCR pipeline only: PDFs and images go through a local vision model (glm-ocr, 1.1B parameters, ~2.2 GB), and txt/css are read directly. It needs no Docling/PyTorch, Tesseract, LibreOffice, or Java — every Python dependency ships as a plain wheel.

powershell -ExecutionPolicy Bypass -File scripts\setup_windows.ps1

The script creates .venv, installs the [lite] extras, installs Ollama (via winget when available), pulls glm-ocr, and writes a lite config.toml. Manual equivalent:

python -m venv .venv
.venv\Scripts\pip install -e ".[lite]"
# install Ollama from https://ollama.com/download/windows, then:
ollama pull glm-ocr
.venv\Scripts\localdataextractor init-config --lite config.toml
.venv\Scripts\localdataextractor --config config.toml check-llm

The lite profile sets provider = "ollama", extraction_mode = "glm_ocr", and max_workers = 1 — on an 8 GB machine the loaded model (~3 GB) needs the headroom, so keep one worker and close heavy apps while processing. PDF pages stream through the model one at a time, so peak app memory stays low even for large documents. The first request after Ollama starts loads the model and is noticeably slower; the lite profile ships a 300 s GLM-OCR timeout for that reason.

Formats the lite install cannot parse (doc(x), ppt(x), xls(x)) are skipped from routing with a per-file diagnostic; install the [full] extras plus the native tools to process them.

Linux setup

The pipeline finds every native tool it shells out to (soffice, tesseract, ocrmypdf, java) through PATH, so the full feature set runs on Linux with no code changes — only the install commands differ from macOS.

1. Native dependencies

# Debian / Ubuntu
sudo apt update
sudo apt install -y python3-venv tesseract-ocr tesseract-ocr-eng \
  ocrmypdf poppler-utils libreoffice default-jre

# Fedora
sudo dnf install -y tesseract tesseract-langpack-eng ocrmypdf \
  poppler-utils libreoffice java-latest-openjdk

# Arch
sudo pacman -S --needed tesseract tesseract-data-eng ocrmypdf \
  poppler libreoffice-fresh jre-openjdk

default-jre/jre-openjdk is only needed if you enable the Tika route ([tika].enabled = true, off by default). LibreOffice is only needed for legacy doc/ppt/xls conversion.

The GUI is Qt (PySide6) and needs the X11 platform libraries. If it exits with Could not load the Qt platform plugin "xcb", install them:

# Debian / Ubuntu
sudo apt install -y libxcb-cursor0 libxcb-icccm4 libxcb-keysyms1 \
  libxcb-shape0 libxkbcommon-x11-0 libegl1

Headless servers can skip that entirely and use the CLI, which never imports Qt.

2. Python environment

Python 3.11 or newer is required. Same uv flow as macOS:

curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"     # uv installs here; add to your shell rc

uv venv .venv                            # or: uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install -e ".[full,dev]"
cp config.sample.toml config.toml

uv venv --python 3.12 downloads and uses a managed interpreter when the distro's Python is too old, so uv alone is enough — the python3-venv package above is only needed for the stdlib fallback (python3 -m venv .venv + pip install -e ".[full,dev]").

On a machine without an NVIDIA GPU, install PyTorch from the CPU index first — Docling pulls it in transitively, and the default Linux wheel bundles the CUDA runtime (several GB):

uv pip install torch --index-url https://download.pytorch.org/whl/cpu
uv pip install -e ".[full,dev]"

For low-RAM machines or servers, the [lite] profile described in the Windows section works identically on Linux (pure wheels, GLM-OCR only, no Docling/PyTorch, Tesseract, LibreOffice, or Java):

uv pip install -e ".[lite]"
localdataextractor init-config --lite config.toml

3. Local LLM backend

curl -fsSL https://ollama.com/install.sh | sh
ollama pull glm-ocr

The installer registers a systemd service listening on 127.0.0.1:11434; without systemd, run ollama serve yourself. Then set provider = "ollama" under [llm] in config.toml (already set if you used init-config --lite) and verify:

localdataextractor --config config.toml check-llm

LM Studio also ships a Linux AppImage if you prefer that backend on 127.0.0.1:1234.

4. Run

source .venv/bin/activate
localdataextractor --config config.toml ingest ./input ./output --explain-route
localdataextractor-gui          # or: python gui.py

Without activating, prefix commands with uv run --no-sync (e.g. uv run --no-sync localdataextractor-gui). Keep --no-sync: a plain uv run syncs .venv to the lockfile's default dependency set, which has no extras, and would uninstall the [full]/[lite] packages you just installed. Use uv run --extra full instead if you want uv to manage the sync itself.

Desktop launcher (Linux)

The macOS .app has no Linux equivalent in this repo, but a .desktop entry does the same job. Path= is what matters: it sets the working directory, which is how config.toml, .env, and the default output/ folder get resolved.

ROOT="$(pwd)"
mkdir -p ~/.local/share/applications
cat > ~/.local/share/applications/localdataextractor.desktop <<EOF
[Desktop Entry]
Type=Application
Name=localDataExtractor
Comment=Local document to Markdown extraction
Exec=$ROOT/.venv/bin/python $ROOT/gui.py
Path=$ROOT
Icon=$ROOT/src/localdataextractor/assets/logo_light.png
Terminal=false
Categories=Utility;Office;
EOF
update-desktop-database ~/.local/share/applications 2>/dev/null || true

It then appears in the application menu. For an icon on the desktop itself, copy the file to ~/Desktop/, chmod +x it, and — on GNOME — right-click it and choose "Allow Launching".

Public repo hygiene

  • config.toml is local-only and gitignored; commit config.sample.toml instead.
  • Generated runtime folders (output/, _out_smoke/) are gitignored.
  • If this directory is nested in another git repo, initialize and push from this folder:
cd /path/to/localDataExtractor
git init
git add .
git commit -m "Initial public release"

LLM backend setup

The pipeline talks to one local OpenAI-compatible server. Two backends are supported, selected with [llm].provider in config.toml (or the GUI "LLM backend" dropdown):

Provider Default server URL
lmstudio (default) http://127.0.0.1:1234/v1
ollama http://127.0.0.1:11434/v1

When base_url is omitted from config.toml, the selected provider's default URL is used automatically.

LM Studio

  1. Start LM Studio local server on http://127.0.0.1:1234/v1.
  2. Load the glm-ocr model (used for both vision OCR and table repair).
  3. Run:
localdataextractor --config config.toml check-llm

Ollama

  1. Install Ollama (brew install ollama or the macOS app) and make sure the server is running (ollama serve, or the menu-bar app).
  2. Pull the vision OCR model:
ollama pull glm-ocr
  1. Set the provider in config.toml:
[llm]
provider = "ollama"
  1. Run:
localdataextractor --config config.toml check-llm

Notes:

  • Ollama lists the model as glm-ocr:latest; the configured primary_model = "glm-ocr" matches it automatically.
  • The first request after startup loads the model into memory and can take noticeably longer; raise timeout_seconds if cold starts time out.
  • Ollama unloads idle models after a few minutes, so long jobs may pay the load cost again mid-run.

CLI usage

Ingest

localdataextractor --config config.toml ingest ./input_docs ./output --explain-route --max-workers 2

Options:

  • --include "**/*.pdf" (repeatable)
  • --exclude "**/~$*" (repeatable)
  • --dry-run
  • --verbose
  • --max-workers <int>
  • --explain-route
  • --mode standard|glm_ocr|highest_accuracy
  • --pii / --no-pii (redact PII in extracted outputs; overrides [pii] config)
  • --pii-mapping (also write a per-file placeholder→original mapping file; implies --pii)

Resume

localdataextractor --config config.toml resume ./output/job_state.json --explain-route

Verify

localdataextractor verify ./output

Create sample config

localdataextractor init-config ./config.toml
# GLM-OCR-only lite profile for low-RAM machines:
localdataextractor init-config --lite ./config.toml

GUI usage

localdataextractor-gui
# or
python gui.py

GUI supports:

  • drag-and-drop files/folders
  • output folder picker
  • LLM backend selector (LM Studio / Ollama), seeded from config.toml; the dropdown choice wins for the session, while config.toml is the persistent setting
  • API token field for servers that require auth
  • extraction mode selector (Standard, GLM-OCR, Highest Accuracy)
  • PII redaction checkbox (redacts extracted outputs, see "PII screening and redaction")
  • GLM-OCR model availability indicator (checks the selected backend)
  • start processing
  • resume interrupted jobs
  • per-file status panel with route, retries, confidence, outcome
  • live log/error panel

Desktop launcher (macOS)

./scripts/make_macos_app.sh              # -> ~/Desktop/localDataExtractor.app
./scripts/make_macos_app.sh /Applications

This writes a small .app bundle that cds into this checkout and starts the GUI with .venv/bin/python gui.py, so config.toml, .env, and the default output/ folder resolve exactly as they do from a terminal run. Double-click the Desktop icon to launch it.

The bundle hardcodes the checkout and venv paths — re-run the script after moving the project folder or recreating .venv. Launch output goes to ~/Library/Logs/localDataExtractor/gui.log.

Output layout

For each source file:

  • output/<relative_path>/<basename>.md
  • output/<relative_path>/<basename>.json
  • output/<relative_path>/<basename>.log
  • optional debug files under output/<relative_path>/_debug/

Job-level resumable state:

  • output/job_state.json

With PII redaction enabled:

  • output/pii_report.json and output/pii_report.md (job-level summary)
  • optional output/<relative_path>/<basename>.pii_map.json (only with --pii-mapping)

Extraction modes

Three extraction modes are available via --mode (CLI) or the GUI dropdown:

Mode Behavior
standard (default) Parser-first routing per file type; stops at first result meeting the confidence threshold
glm_ocr GLM-OCR vision model as primary extractor (PDFs and images), with standard routes as fallback
highest_accuracy Tries all available parsers including GLM-OCR and selects the highest confidence result

GLM-OCR mode requires the glm-ocr model loaded in LM Studio or pulled in Ollama. It renders each PDF page to an image and sends it to the vision model for high-accuracy OCR of text, tables, and graphics.

Routing behavior

Standard mode

  • txt, css:
    • deterministic direct read
    • no LLM usage
    • CSS wrapped in fenced css code block
  • pdf:
    • Docling first
    • OCR-assisted Docling retry via OCRmyPDF
    • Tika fallback when enabled
  • docx, pptx, xlsx, xls:
    • MarkItDown first
    • Docling fallback
    • Tika fallback when enabled
  • doc, ppt:
    • LibreOffice headless conversion to modern format
    • then MarkItDown/Docling path
    • Tika fallback when enabled

GLM-OCR mode

  • pdf, png, jpg, jpeg, tiff, tif, bmp:
    • GLM-OCR vision model first
    • standard routes as fallback
  • all other formats: standard routing

Highest Accuracy mode

  • all standard routes tried first
  • GLM-OCR appended for PDFs and images
  • no early stopping: every parser runs, best confidence wins

In every mode, routes whose parser dependencies are not installed (e.g. Docling on a lite install) are skipped at planning time; the first remaining route's detail notes what was skipped. If no route survives, the original plan is kept so the file fails with per-parser diagnostics.

Every decision is logged in route history and can be printed with --explain-route.

Confidence and retries

  • Confidence range: 0-100
  • Default threshold: 95
  • Retries trigger when:
    • overall confidence < 95, or
    • any important table confidence < 95
  • Each attempt stores:
    • timestamp
    • route and parser used
    • model used (if any)
    • confidence details
    • retry reasons
    • warnings and artifacts
  • If all strategies fail to reach threshold, best result is emitted and marked below_threshold=true.

Table handling

Tables are first-class and include:

  • table id
  • source location metadata
  • header/body rows
  • row/column counts
  • merged-cell metadata (when recoverable)
  • confidence score
  • validation warnings

Validation heuristics include:

  • column consistency
  • weak header detection
  • duplicate-row ratio
  • empty-cell anomaly ratio
  • collapse risk

LM table repair

If table confidence is below [retry].confidence_threshold, the table is sent to the local model for normalization (disable with [llm] enable_vlm_repair = false).

Repair asks the model to echo every cell back, so request time scales with table size — roughly 4 s for a 4-row table and 90 s for a 30-row one on a small local model. Tables larger than [llm] repair_max_rows (default 40) are therefore skipped with a warning rather than run into the timeout.

Responses are structurally validated before they are applied. A repair is rejected — leaving the original table untouched — when the model omits body_rows, collapses the body into header_rows, drops or invents rows, or collapses a multi-column table into one column. Each rejection is recorded as a per-file warning naming the reason.

Structural validation cannot detect rewritten cell values. OCR-tuned models are prone to that: in local testing glm-ocr altered euro amounts while echoing a 30-row table back, where gemma3:4b returned them intact in comparable time. Point repair at a general text model when values matter:

[llm]
primary_model = "glm-ocr"     # vision OCR
repair_model = "gemma3:4b"    # text normalization

Repair also gives up per file after [llm] repair_failure_limit consecutive failures (default 3), instead of spending the timeout on every remaining table and again on every route attempt. Tables that were repaired successfully are not re-sent within the same file.

Verification

localdataextractor verify <output> checks:

  • required output files exist (.md, .json, .log)
  • JSON validity and schema keys
  • confidence fields and below-threshold flag
  • route history presence
  • table presence in JSON and Markdown approximation when tables were detected

GLM-OCR configuration

The [glm_ocr] section in config.toml controls vision-based OCR:

[llm]
provider = "lmstudio"     # lmstudio | ollama
# base_url = "http://127.0.0.1:1234/v1"  # optional; defaults per provider
primary_model = "glm-ocr"
fallback_model = "glm-ocr"

[glm_ocr]
enabled = true           # enabled by default; glm-ocr serves as the single model
model_name = "glm-ocr"   # model ID as served by the backend ("glm-ocr" matches Ollama's "glm-ocr:latest")
page_dpi = 300            # DPI for rendering PDF pages to images
max_pages = 50            # safety cap on pages sent to vision API
timeout_seconds = 180     # per-request timeout (vision is slower)

The extraction mode can also be set in config:

[routing]
extraction_mode = "standard"  # standard | glm_ocr | highest_accuracy

Tradeoffs and known limitations

  • Docling/MarkItDown APIs may vary across versions; fallbacks are included.
  • Legacy doc/ppt fidelity depends on local LibreOffice conversion quality.
  • xls support depends on local xlrd availability.
  • Tika fallback requires Java runtime and local tika-app.jar path in config.
  • Markdown tables may lose some advanced layout semantics; JSON table representation is authoritative.
  • GLM-OCR mode processes pages one at a time; large PDFs will be slow. Use max_pages to cap.
  • Highest Accuracy mode runs all parsers and does not stop early, so it is slower than other modes.

PII screening and redaction

Opt-in screening of extracted outputs for personally identifiable information, with redaction and a findings summary. Enable via [pii] enabled = true in config.toml, the --pii CLI flag, or the "Redact PII in outputs" checkbox in the GUI. Off by default.

Detection is hybrid and fully local:

  • Regex + checksum validators (stdlib only) for structured PII: emails, phone numbers (German + international), IBANs (mod-97), credit cards (Luhn), German tax IDs (IdNr checksum), Steuernummer/USt-IdNr, SSNs, German ID-card/passport numbers (keyword-gated, MRZ check digit), bank account numbers, and German street/PLZ addresses.
  • The local LLM (same backend as extraction) finds person names and free-form addresses. Disable with use_llm = false; if the LLM server is down, regex results still apply and a warning is recorded.

Detected values are replaced with typed, numbered placeholders ([PERSON_1], [EMAIL_2]); the same value maps to the same placeholder throughout a document. Dates are never redacted.

What gets redacted:

  • <basename>.md and <basename>.json — all text surfaces including tables, metadata, and warnings. The JSON gains a pii key listing each placeholder with category, occurrence count, and page references (never the original values).
  • pii_report.json / pii_report.md at the output root summarize findings per file and per category.

Not redacted, by design: source documents (never touched), .log files, _debug/ artifacts, and job_state.json. Delete _debug/ and .log files before sharing outputs.

--pii-mapping (or write_mapping_file = true) additionally writes <basename>.pii_map.json with the placeholder→original mapping for de-redaction. This file contains the original PII values — handle it as sensitive data.

Privacy notes

  • No cloud APIs are used by default.
  • The LLM backend (LM Studio or Ollama) is enforced as localhost-only.
  • Document processing remains on-device.
  • Optional PII redaction of extracted outputs (see above).

About

Privacy-first, fully local document-to-Markdown extraction pipeline for macOS Apple Silicon.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages