The data import pipeline for Open Parliament TV. Fetches parliamentary proceedings and media feeds, parses them into a unified per-session JSON file, enriches with named-entity linking, sentence-level audio alignment, and named-entity recognition, then validates and publishes the result for the platform to ingest.
For the wider system context (repositories, data flow, the Stage 2 format) see the Architecture repo. The pipeline stages map to PIPELINE.md; the file format produced by the pipeline is specified in STAGE2-FORMAT.md; the structural differences across the implemented parliaments are surveyed in DATA-STRUCTURES.md.
Each implemented parliament lives under optv/parliaments/<CODE>/ and carries its own README documenting that pipeline's data model, merge strategy, runtime flags, and known limitations. New parliaments follow docs/ADDING-A-PARLIAMENT.md.
python3 -m pip install -r requirements.txt
# fetch + process the current period's data into <data_dir>
# (example shown for DE — every parliament has the same wrapper shape)
./optv/parliaments/DE/update <data_dir>
# or run the workflow manually with finer control:
./optv/parliaments/DE/workflow.py --period=21 <data_dir> \
--download-original --merge-speeches \
--link-entities --align-sentences --extract-entities<data_dir> is the per-parliament data directory, expected to be a sibling clone of the parliament's data repo (for example OpenParliamentTV-Data-DE for DE). Each --* flag is opt-in and idempotent; --force re-runs an already-completed stage. A lockfile (<data_dir>/optv.lock) blocks concurrent runs.
pip install -r requirements.txt installs the Python packages. Beyond that, the optional enrichment stages need prerequisites pip can't provide — and only for parliaments that run them (each declares its set in manifest.yaml as supported_stages):
- Align (
aeneas) — the system binariesffmpegandespeak. - NER — a spaCy model (per-parliament
locale.spacy_model, e.g.de_core_news_md, installed viapython -m spacy download <model>) and a runningentityfishingservice (passed as--ner-api-endpoint).
Parliaments that stop at download, parse, merge, nel need none of these (check each manifest.yaml).
optv/
├── parliaments/
│ └── <CODE>/ # one directory per implemented parliament
│ ├── manifest.yaml # per-parliament metadata read by Conductor (stages, periods, …)
│ ├── workflow.py # thin entry point — defines hooks, calls optv.shared.workflow
│ ├── common.py # Config class (paths + file naming); re-exports shared helpers
│ ├── scraper/ # fetch raw proceedings + media
│ ├── parsers/ # native format → intermediate JSON
│ ├── merger/ # join media + proceedings into Stage 2
│ ├── align_prep.py # optional — per-speech audio staging for align (adapter over optv.shared.audio_prep)
│ ├── update # shell wrapper with parliament-specific defaults
│ └── Makefile # download + merge targets driven by file mtimes
├── scripts/ # tracked maintenance / backfill / audit scripts, run via `python -m optv.scripts.<name>`
└── shared/ # cross-parliament infrastructure
├── schema/ # Stage 2 JSON schemas + reference doc
├── docs/EXAMPLES/ # example Stage 2 documents
├── validators/ # structural + semantic validators, CLI
├── pdf2tei/ # PDF proceedings → ParlaMint TEI → text (joined onto the spine by media-spine parliaments)
├── lang/ # language-specific text helpers, keyed by ISO 639-1 (de, …)
├── workflow.py # run_main + stage orchestrator + WorkflowHooks + shared argparser
├── config.py # BaseConfig (on-disk layout, status/mtime helpers)
├── publish.py # non-destructive publish helpers (demotion guard, carry-forward)
├── session_status.py # SessionStatus enum
├── speech_id.py # speech-id model normalizer (originID/originMediaID/originTextID)
├── merge_format.py # generic merge formatting helpers (slug, offsets, name split)
├── sequence_align.py # generic Needleman-Wunsch (shared by the PT/ES/DE-family mergers)
├── align.py # forced sentence alignment (aeneas)
├── audio_prep.py # per-speech audio staging for align: download-once/slice driver + align-hook factory
├── nel.py # named-entity linking (Wikidata)
├── ner.py # named-entity recognition (spaCy + entity-fishing)
├── agenda_types.py # cross-parliament agenda-type vocabulary + classifier registry
├── entity_dump_bootstrap.py # temporary pre-platform Wikidata entity-dump builder
├── whisper_diff.py # QC: faster-whisper transcription vs proceedings text (opt-in, not in the pipeline)
└── whisper_qc.py # QC: faster-whisper + Resemblyzer worker library (used by whisper_diff)
workflow.py is intentionally thin: orchestration (merge → NEL → align → NER → publish), the common argparser, lockfile handling and the publish helper all live in optv/shared/workflow.py. The per-parliament file only contains the four adapters that legitimately differ — download, parse, merge call shape, align call shape — plus parliament-specific CLI flags.
manifest.yaml is the per-parliament metadata file. The Conductor reads it to know which stages a parliament supports, which entity dump to use, and the retry defaults — see optv/parliaments/DE/manifest.yaml for the canonical example.
Each stage produces a side-by-side cache file per session (e.g. 21001-merged.json, 21001-aligned.json, 21001-ner.json) and runs only when its input is newer than its output. Paths below link to DE as the worked example; every parliament directory carries the same shape — see its README for parliament-specific module names and source formats. Which stages a parliament runs is declared in its manifest.yaml supported_stages (some stop after NEL, others run align/NER). Note: the German state parliaments' text/align stages are an experimental, unvalidated prototype — not yet cleared for platform integration (see each parliament's README).
| Stage | Module / script | Input | Output |
|---|---|---|---|
| Fetch | scraper/fetch_proceedings.py, scraper/fetch_media.py |
parliament APIs | original/{proceedings,media}/ |
| Parse | parsers/proceedings2json.py, parsers/media2json.py |
native formats (DE: TEI XML + RSS) | intermediate JSON |
| Merge | merger/merge_session.py |
proceedings + media JSON | cache/merged/*-merged.json |
| NEL | optv/shared/nel.py |
merged JSON + entity dump | people[].wid, faction normalisation |
| Align | optv/shared/align.py |
merged JSON + audio | cache/aligned/*-aligned.json (sentence timings) |
| NER | optv/shared/ner.py |
aligned JSON + entity-fishing API | cache/ner/*-ner.json (sentence entities) |
| Publish | _publish_as_processed() in optv/shared/workflow.py (uses helpers from optv/shared/publish.py) |
latest cache file | processed/*-session.json |
For the conceptual stage breakdown (parliament-agnostic), see Architecture/PIPELINE.md.
Stage 2 schemas and conventions: optv/shared/schema/README.md. Standalone CLI:
python -m optv.shared.validators.cli --dir <data_dir>/processed --schema full
python -m optv.shared.validators.cli --file session.json --no-semanticThe publish step also runs validation and logs findings; warnings do not block.
.venv/bin/pytest -q tests
# or
make -C optv/parliaments/DE testFlat layout under tests/, no config files. Covers pure helpers, the merger contract that catches originID/originTextID-class regressions, the agenda-type vocabulary, and an end-to-end smoke run on tiny synthetic fixtures. Stages that need external data or services (aeneas alignment, the NEL entity dump, entity-fishing NER) are not in the suite. CI runs the same command on every push and PR via .github/workflows/tests.yml.
An opt-in QC toolset cross-checks output without touching pipeline data. It has its own dependencies (pip install -r requirements-qc.txt — faster-whisper, Resemblyzer, torch) and is not part of the production pipeline:
optv/shared/whisper_diff.py(rank/transcribe/diff) — transcribes a session's audio with faster-whisper (+ optional Resemblyzer speaker-change detection, viawhisper_qc.py) and reports where it diverges from the proceedings text.optv/scripts/merge_audit.py— read-only sweep over merger output that flags text-accumulation anomalies (histograms + suspect lists).
DE (Deutscher Bundestag) is the reference implementation and the only parliament in production; the others are in development. For how their data structures differ, see DATA-STRUCTURES.md.
| Parliament | Name | Type | Data Repository | Status |
|---|---|---|---|---|
DE |
Deutscher Bundestag | national | Data-DE | production |
DE-BW |
Landtag von Baden-Württemberg | regional | Data-DE-BW | in development |
DE-BY |
Bayerischer Landtag | regional | Data-DE-BY | in development |
DE-HH |
Hamburgische Bürgerschaft | regional | Data-DE-HH | in development |
DE-NI |
Niedersächsischer Landtag | regional | Data-DE-NI | in development |
DE-NW |
Landtag Nordrhein-Westfalen | regional | Data-DE-NW | in development |
DE-RP |
Landtag Rheinland-Pfalz | regional | Data-DE-RP | in development |
DE-SH |
Landtag Schleswig-Holstein | regional | Data-DE-SH | in development |
DE-SN |
Sächsischer Landtag | regional | Data-DE-SN | in development |
DE-ST |
Landtag von Sachsen-Anhalt | regional | Data-DE-ST | in development |
ES |
Congreso de los Diputados | national | Data-ES | in development |
EU |
European Parliament | supranational | Data-EU | in development |
FI |
Eduskunta | national | Data-FI | in development |
FR |
Assemblée nationale | national | Data-FR | in development |
NO |
Stortinget | national | Data-NO | in development |
PT |
Assembleia da República | national | Data-PT | in development |
SE |
Sveriges Riksdag | national | Data-SE | in development |
TW |
Legislative Yuan (立法院) | national | Data-TW | in development |