Skip to content

About

OpenParliamentTV-Tools for parsing parliamentary data

Resources

Stars

4 stars

Watchers

4 watching

Forks

Repository files navigation

Open Parliament TV - Tools

The data import pipeline for Open Parliament TV. Fetches parliamentary proceedings and media feeds, parses them into a unified per-session JSON file, enriches with named-entity linking, sentence-level audio alignment, and named-entity recognition, then validates and publishes the result for the platform to ingest.

For the wider system context (repositories, data flow, the Stage 2 format) see the Architecture repo. The pipeline stages map to PIPELINE.md; the file format produced by the pipeline is specified in STAGE2-FORMAT.md; the structural differences across the implemented parliaments are surveyed in DATA-STRUCTURES.md.

Each implemented parliament lives under optv/parliaments/<CODE>/ and carries its own README documenting that pipeline's data model, merge strategy, runtime flags, and known limitations. New parliaments follow docs/ADDING-A-PARLIAMENT.md.

Quick start

python3 -m pip install -r requirements.txt

# fetch + process the current period's data into <data_dir>
# (example shown for DE — every parliament has the same wrapper shape)
./optv/parliaments/DE/update <data_dir>

# or run the workflow manually with finer control:
./optv/parliaments/DE/workflow.py --period=21 <data_dir> \
    --download-original --merge-speeches \
    --link-entities --align-sentences --extract-entities

<data_dir> is the per-parliament data directory, expected to be a sibling clone of the parliament's data repo (for example OpenParliamentTV-Data-DE for DE). Each --* flag is opt-in and idempotent; --force re-runs an already-completed stage. A lockfile (<data_dir>/optv.lock) blocks concurrent runs.

pip install -r requirements.txt installs the Python packages. Beyond that, the optional enrichment stages need prerequisites pip can't provide — and only for parliaments that run them (each declares its set in manifest.yaml as supported_stages):

  • Align (aeneas) — the system binaries ffmpeg and espeak.
  • NER — a spaCy model (per-parliament locale.spacy_model, e.g. de_core_news_md, installed via python -m spacy download <model>) and a running entityfishing service (passed as --ner-api-endpoint).

Parliaments that stop at download, parse, merge, nel need none of these (check each manifest.yaml).

Layout

optv/
├── parliaments/
│   └── <CODE>/              # one directory per implemented parliament
│       ├── manifest.yaml    # per-parliament metadata read by Conductor (stages, periods, …)
│       ├── workflow.py      # thin entry point — defines hooks, calls optv.shared.workflow
│       ├── common.py        # Config class (paths + file naming); re-exports shared helpers
│       ├── scraper/         # fetch raw proceedings + media
│       ├── parsers/         # native format → intermediate JSON
│       ├── merger/          # join media + proceedings into Stage 2
│       ├── align_prep.py    # optional — per-speech audio staging for align (adapter over optv.shared.audio_prep)
│       ├── update           # shell wrapper with parliament-specific defaults
│       └── Makefile         # download + merge targets driven by file mtimes
├── scripts/                # tracked maintenance / backfill / audit scripts, run via `python -m optv.scripts.<name>`
└── shared/                  # cross-parliament infrastructure
    ├── schema/              # Stage 2 JSON schemas + reference doc
    ├── docs/EXAMPLES/       # example Stage 2 documents
    ├── validators/          # structural + semantic validators, CLI
    ├── pdf2tei/             # PDF proceedings → ParlaMint TEI → text (joined onto the spine by media-spine parliaments)
    ├── lang/                # language-specific text helpers, keyed by ISO 639-1 (de, …)
    ├── workflow.py          # run_main + stage orchestrator + WorkflowHooks + shared argparser
    ├── config.py            # BaseConfig (on-disk layout, status/mtime helpers)
    ├── publish.py           # non-destructive publish helpers (demotion guard, carry-forward)
    ├── session_status.py    # SessionStatus enum
    ├── speech_id.py         # speech-id model normalizer (originID/originMediaID/originTextID)
    ├── merge_format.py      # generic merge formatting helpers (slug, offsets, name split)
    ├── sequence_align.py    # generic Needleman-Wunsch (shared by the PT/ES/DE-family mergers)
    ├── align.py             # forced sentence alignment (aeneas)
    ├── audio_prep.py        # per-speech audio staging for align: download-once/slice driver + align-hook factory
    ├── nel.py               # named-entity linking (Wikidata)
    ├── ner.py               # named-entity recognition (spaCy + entity-fishing)
    ├── agenda_types.py      # cross-parliament agenda-type vocabulary + classifier registry
    ├── entity_dump_bootstrap.py  # temporary pre-platform Wikidata entity-dump builder
    ├── whisper_diff.py      # QC: faster-whisper transcription vs proceedings text (opt-in, not in the pipeline)
    └── whisper_qc.py        # QC: faster-whisper + Resemblyzer worker library (used by whisper_diff)

workflow.py is intentionally thin: orchestration (merge → NEL → align → NER → publish), the common argparser, lockfile handling and the publish helper all live in optv/shared/workflow.py. The per-parliament file only contains the four adapters that legitimately differ — download, parse, merge call shape, align call shape — plus parliament-specific CLI flags.

manifest.yaml is the per-parliament metadata file. The Conductor reads it to know which stages a parliament supports, which entity dump to use, and the retry defaults — see optv/parliaments/DE/manifest.yaml for the canonical example.

Pipeline stages

Each stage produces a side-by-side cache file per session (e.g. 21001-merged.json, 21001-aligned.json, 21001-ner.json) and runs only when its input is newer than its output. Paths below link to DE as the worked example; every parliament directory carries the same shape — see its README for parliament-specific module names and source formats. Which stages a parliament runs is declared in its manifest.yaml supported_stages (some stop after NEL, others run align/NER). Note: the German state parliaments' text/align stages are an experimental, unvalidated prototype — not yet cleared for platform integration (see each parliament's README).

Stage Module / script Input Output
Fetch scraper/fetch_proceedings.py, scraper/fetch_media.py parliament APIs original/{proceedings,media}/
Parse parsers/proceedings2json.py, parsers/media2json.py native formats (DE: TEI XML + RSS) intermediate JSON
Merge merger/merge_session.py proceedings + media JSON cache/merged/*-merged.json
NEL optv/shared/nel.py merged JSON + entity dump people[].wid, faction normalisation
Align optv/shared/align.py merged JSON + audio cache/aligned/*-aligned.json (sentence timings)
NER optv/shared/ner.py aligned JSON + entity-fishing API cache/ner/*-ner.json (sentence entities)
Publish _publish_as_processed() in optv/shared/workflow.py (uses helpers from optv/shared/publish.py) latest cache file processed/*-session.json

For the conceptual stage breakdown (parliament-agnostic), see Architecture/PIPELINE.md.

Validation

Stage 2 schemas and conventions: optv/shared/schema/README.md. Standalone CLI:

python -m optv.shared.validators.cli --dir <data_dir>/processed --schema full
python -m optv.shared.validators.cli --file session.json --no-semantic

The publish step also runs validation and logs findings; warnings do not block.

Tests

.venv/bin/pytest -q tests
# or
make -C optv/parliaments/DE test

Flat layout under tests/, no config files. Covers pure helpers, the merger contract that catches originID/originTextID-class regressions, the agenda-type vocabulary, and an end-to-end smoke run on tiny synthetic fixtures. Stages that need external data or services (aeneas alignment, the NEL entity dump, entity-fishing NER) are not in the suite. CI runs the same command on every push and PR via .github/workflows/tests.yml.

Quality control

An opt-in QC toolset cross-checks output without touching pipeline data. It has its own dependencies (pip install -r requirements-qc.txt — faster-whisper, Resemblyzer, torch) and is not part of the production pipeline:

  • optv/shared/whisper_diff.py (rank / transcribe / diff) — transcribes a session's audio with faster-whisper (+ optional Resemblyzer speaker-change detection, via whisper_qc.py) and reports where it diverges from the proceedings text.
  • optv/scripts/merge_audit.py — read-only sweep over merger output that flags text-accumulation anomalies (histograms + suspect lists).

Implemented parliaments

DE (Deutscher Bundestag) is the reference implementation and the only parliament in production; the others are in development. For how their data structures differ, see DATA-STRUCTURES.md.

Parliament Name Type Data Repository Status
DE Deutscher Bundestag national Data-DE production
DE-BW Landtag von Baden-Württemberg regional Data-DE-BW in development
DE-BY Bayerischer Landtag regional Data-DE-BY in development
DE-HH Hamburgische Bürgerschaft regional Data-DE-HH in development
DE-NI Niedersächsischer Landtag regional Data-DE-NI in development
DE-NW Landtag Nordrhein-Westfalen regional Data-DE-NW in development
DE-RP Landtag Rheinland-Pfalz regional Data-DE-RP in development
DE-SH Landtag Schleswig-Holstein regional Data-DE-SH in development
DE-SN Sächsischer Landtag regional Data-DE-SN in development
DE-ST Landtag von Sachsen-Anhalt regional Data-DE-ST in development
ES Congreso de los Diputados national Data-ES in development
EU European Parliament supranational Data-EU in development
FI Eduskunta national Data-FI in development
FR Assemblée nationale national Data-FR in development
NO Stortinget national Data-NO in development
PT Assembleia da República national Data-PT in development
SE Sveriges Riksdag national Data-SE in development
TW Legislative Yuan (立法院) national Data-TW in development

Adding a new parliament

See docs/ADDING-A-PARLIAMENT.md.

About

OpenParliamentTV-Tools for parsing parliamentary data

Resources

Stars

4 stars

Watchers

4 watching

Forks

Releases

Packages

Used by

Contributors

Languages