Lhotse is a Python library for preparing multimodal datasets and building scalable PyTorch data pipelines. It provides manifests for audio, video, images, and text, on-the-fly augmentation, dynamic batching, efficient sharded storage, and indexed, resumable dataloading.
Documentation · Quick start · Tutorials · Contributing · Citation
Install the latest stable release from PyPI:
python -m pip install lhotseThis README and the latest documentation describe the development version on
master, which requires Python 3.11 or later. Some APIs and extras described
below may not yet be available in the stable release. The Python badge above
lists the versions tested on master; check PyPI for the stable release's requirements.
To use the development version:
python -m pip install git+https://github.com/lhotse-speech/lhotsePyTorch is required; torchaudio and TorchCodec 0.10 or later are optional audio
backends. TorchCodec also reads video (python -m pip install torchcodec) and
requires an FFmpeg version that it supports.
See optional dependencies for additional formats and
integrations.
Download the small public YesNo corpus, prepare its training split, and load a batch of audio and transcripts. This example uses waveforms directly, so it does not need precomputed features or worker processes.
from pathlib import Path
from torch.utils.data import DataLoader
from lhotse import CutSet
from lhotse.dataset import (
AudioSamples,
DynamicCutSampler,
K2SpeechRecognitionDataset,
)
from lhotse.recipes import download_yesno, prepare_yesno
corpus_dir = download_yesno(target_dir=Path("data"))
manifests = prepare_yesno(corpus_dir=corpus_dir)["train"]
cuts = CutSet.from_manifests(
recordings=manifests["recordings"],
supervisions=manifests["supervisions"],
output_path=Path("data") / "cuts.jsonl.gz",
lazy=True,
)
dataset = K2SpeechRecognitionDataset(input_strategy=AudioSamples())
sampler = DynamicCutSampler(cuts, max_duration=60, shuffle=False)
dataloader = DataLoader(dataset, sampler=sampler, batch_size=None)
batch = next(iter(dataloader))
print(batch["inputs"].shape) # Padded waveforms: (batch, samples).
print(batch["supervisions"]["text"])The cuts are saved as a JSONL manifest and opened lazily, without loading their
audio into memory. DynamicCutSampler streams the cuts into batches with at most
60 seconds of audio, and the dataset loads and pads their waveforms when requested.
Despite its name, K2SpeechRecognitionDataset does not require k2.
- Multimodal manifests: represent recordings, supervision segments, images, and audio/video cuts in human-readable manifests with Python and CLI APIs.
- Lazy processing: stream large JSONL manifests and compose filtering, mapping, shuffling, and dataset blending without loading everything into memory.
- On-the-fly transforms: truncate, pad, mix, and augment cuts while adjusting their supervision timing; compute features during loading or store them in advance.
- Dynamic batching: group examples by duration and use dynamic bucketing to reduce padding. Partition data across distributed ranks and dataloader workers to avoid overlapping samples.
- Efficient storage: use Lhotse Shar for modular, sharded sequential I/O from local or remote storage.
- Indexed and resumable dataloading: index JSONL and Shar data, pack shard
indexes into
.idxpackcatalogs, and checkpoint supported iterator pipelines. See indexed manifests for the requirements for exact restore, including gzip indexes and worker state. - Corpus recipes: prepare standard manifests for commonly used datasets.
The notebooks in examples
cover the following workflows. Open them on GitHub to read them or in Colab to run
them; installation and data requirements are described in each notebook.
| Tutorial | Run |
|---|---|
| Basic Lhotse workflow | Colab |
| Transforming data with cuts | Colab |
| WebDataset integration | Colab |
| Combining multiple datasets | Colab |
| Lhotse Shar: modular data format | Colab |
| Image and video loading | Colab |
| AIStore batch loading (requires an AIStore endpoint) | Colab |
Resumable, indexed dataloading (torchdata for the optional multi-worker example) |
Colab |
The video notebook reads video frames and audio tracks with TorchCodec 0.10 or later, which also requires FFmpeg. The AIStore notebook uses 100 training and 20 development recordings and creates and removes its own temporary bucket.
These conference materials provide additional background and may use older APIs:
- Interspeech 2023: tutorial notebook and slides (Microsoft sign-in required).
- Interspeech 2021: recorded lecture (3 hours).
- Icefall: speech recognition recipes combining Lhotse data pipelines with k2.
- NVIDIA NeMo: Lhotse dataloading with dynamic batching, dataset mixing, and Shar support.
- ESPnet example: a minimal integration notebook (Google sign-in may be required).
Lhotse shares its roots with Kaldi and supports importing and exporting Kaldi data.
RecordingSet describes recordings, SupervisionSet describes annotations such
as transcripts and speakers, and CutSet combines them into examples that can be
reshaped for training. Manifests store metadata and references to the underlying
data; audio transforms are applied when the data is loaded.
Cuts support mixing, truncation, padding, and augmentation. Feature extraction can run on the fly or ahead of training, with optional compressed storage. Lhotse also supports mixing features directly.
Install only the extras needed for your workflow. These extras describe master;
when using development-only features, include the extras in the source install:
python -m pip install 'lhotse[checkpoint,gzip] @ git+https://github.com/lhotse-speech/lhotse'For a published extra, use a quoted requirement such as
python -m pip install "lhotse[lilcom]".
| Extra | Enables |
|---|---|
checkpoint |
torchdata.stateful_dataloader.StatefulDataLoader for multi-worker checkpointing. |
gzip |
Indexed gzip JSONL with indexed_gzip; keep both .idx and .gzidx sidecars. |
lilcom |
Compressed features and arrays; lilcom_chunky is the preferred compressed feature backend. |
orjson |
Faster JSONL serialization and reading. |
webdataset |
WebDataset import/export. |
pillow |
Image loading with Pillow. |
aistore |
AIStore I/O and batch loading; configure AIS_ENDPOINT. |
kaldi |
Kaldi I/O and feature extraction via kaldi-native-io and kaldifeat. |
h5py |
HDF5 feature storage. |
dill |
Serialization of lazy transforms such as lambdas across processes. Also set LHOTSE_DILL_ENABLED=1 or call lhotse.set_dill_enabled(True). |
Additional audio backends, remote storage clients, feature extractors, and SPHERE support are covered in the installation reference.
See the configuration reference
for audio and I/O backend selection, timing tolerances, remote storage settings,
and serialization options. Dill is disabled by default even when installed;
enabling it can help with lazy transforms in a PyTorch DataLoader with
num_workers > 0.
Clone the repository, then install the locked development environment with uv:
git clone https://github.com/lhotse-speech/lhotse
cd lhotse
uv sync --locked
uv run --locked pre-commit installSee the contributor guide for the required uv version, system dependencies, tests, formatting, documentation builds, CPU/CUDA environments, dependency updates, and releases.
If you use Lhotse in research, please cite
Lhotse: a speech data representation library for the modern deep learning ecosystem
(NeurIPS Data-Centric AI Workshop, 2021). Citation metadata is available in
CITATION.cff.
Some contributions to this project were supported by National Science Foundation CCRI award 2120435.


