ROB: Scan the file once when resolving undefined references by SudiptoKumar · Pull Request #4431 · py-pdf/pypdf

SudiptoKumar · 2026-11-14T19:01:42Z

Bug

When PdfReader resolves an indirect reference to an object number that is not present in the cross-reference table, it falls back to a full-file regex rescan to try to locate a matching N G obj pattern. Failed lookups are cached per (generation, idnum), so repeated lookups of the same undefined object are cheap — but each distinct undefined reference is a fresh cache miss that triggers its own independent full-file materialization and regex scan. A PDF with N distinct undefined references at a fixed file size S therefore forces N full-file scans: O(N x S).

Root cause

PdfReader.get_object() in pypdf/_reader.py called get_stream_data(self.stream) followed by a re.search for that one object on every undefined-reference lookup, with no reuse across lookups.

Fix

Build a recovery index once, on the first undefined-reference lookup: a single regex pass over the file records the first "N G obj" marker for every object, and later lookups consult the index instead of rescanning. The index keys keep the digit strings as found in the file, so a lookup matches exactly what the old per-object re.search would find (e.g. "007 0 obj" does not match a lookup for 7 0 R). Behaviour is otherwise unchanged: objects present in the file body but missing from the xref table are still recovered the same way, and strict mode still raises PdfReadError for objects that are not defined.

Tests

Reproduced with the reproducer from the issue: 200 distinct undefined references in a ~50 KB file triggered 200 get_stream_data calls before the fix, 1 after it.
New regression test test_get_object__speed__distinct_invalid_lookups_scan_file_once (fails on unmodified main, passes with the fix).
New test test_get_object__recovery_index_finds_object_missing_from_xref locks in that an object present in the body but absent from the xref table is still recovered.
pytest tests/test_reader.py: 252 passed, 2 skipped; the 11 failures fail identically on unmodified main (missing sample files / modules in this environment).
ruff check and mypy clean; changed lines are ruff format clean.

Fixes #4428

Resolving an indirect reference to an object number that is absent from the xref table fell back to a full-file rescan for every distinct undefined reference, giving O(N x file size) behaviour. Build a recovery index of all 'N G obj' markers on the first such lookup and consult it for later ones, so the file is scanned once per reader. The index keys keep the digit strings as found in the file, so a lookup matches exactly what a per-object re.search would find. Fixes py-pdf#4428

codecov · 2026-11-14T19:11:47Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.87%. Comparing base (1e3e7e3) to head (943340c).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4431   +/-   ##
=======================================
  Coverage   98.87%   98.87%           
=======================================
  Files          59       59           
  Lines       12299    12309   +10     
  Branches     2342     2344    +2     
=======================================
+ Hits        12161    12171   +10     
  Misses         68       68           
  Partials       70       70

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

stefan6419846

Besides the code-level comments: How does this affect performance in the different cases, for example when only looking for a few objects, but having to parse a huge string?

stefan6419846 · 2026-11-14T22:23:58Z

+        #: digit strings from the file — to the (start, end) offsets of
+        #: the first "N G obj" marker, so resolving many undefined
+        #: references does not rescan the whole file per lookup.
+        self._recovery_index: dict[tuple[bytes, bytes], tuple[int, int]] | None = None


Cannot we use a more specific name? Recovery index is rather generic and I would not know at first sight what it refers to.

stefan6419846 · 2026-11-14T22:24:20Z

        self.xref = {}
        self.xref_free_entry: dict[int, dict[Any, Any]] = {}
        self.xref_objStm: dict[int, tuple[Any, Any]] = {}
+        #: Index of indirect-object headers, built lazily on the first


Suggested change

#: Index of indirect-object headers, built lazily on the first

#: Index of indirect object headers, built lazily on the first

stefan6419846 · 2026-11-14T22:24:49Z

        self.xref_free_entry: dict[int, dict[Any, Any]] = {}
        self.xref_objStm: dict[int, tuple[Any, Any]] = {}
+        #: Index of indirect-object headers, built lazily on the first
+        #: undefined-reference lookup; maps (generation, idnum) — as the


Suggested change

#: undefined-reference lookup; maps (generation, idnum) — as the

#: undefined reference lookup; maps (generation, idnum) - as the

stefan6419846 · 2026-11-14T22:24:58Z

        self.xref_objStm: dict[int, tuple[Any, Any]] = {}
+        #: Index of indirect-object headers, built lazily on the first
+        #: undefined-reference lookup; maps (generation, idnum) — as the
+        #: digit strings from the file — to the (start, end) offsets of


Suggested change

#: digit strings from the file — to the (start, end) offsets of

#: digit strings from the file - to the (start, end) offsets of

stefan6419846 · 2026-11-14T22:26:00Z

        finally:
            self._object_stream_resolution_stack.pop()

+    def _build_recovery_index(self) -> dict[tuple[bytes, bytes], tuple[int, int]]:


Please rename accordingly as well.

stefan6419846 · 2026-11-14T22:29:33Z

-                    m.start(0) + 1
-                )
-                self.stream.seek(m.end(0) + 1)
+                self.xref[indirect_reference.generation][indirect_reference.idnum] = offsets[0] + 1


Consider unpacking offsets before using them for more clarity on what these values refer to. Alternatively, we might even consider named tuples or similar here, although this might be overkill.

stefan6419846 · 2026-11-14T22:30:41Z

+
+def test_get_object__speed__distinct_invalid_lookups_scan_file_once() -> None:
+    # https://github.com/py-pdf/pypdf/issues/4428
+    # Each distinct undefined reference used to trigger its own full-file


We are only interested in the correct behavior, not the wrong one.

stefan6419846 · 2026-11-14T22:31:11Z

+
+
+def test_get_object__speed__distinct_invalid_lookups_scan_file_once() -> None:
+    # https://github.com/py-pdf/pypdf/issues/4428


Please align with other tests - we usually do not use the full issue link here. Additionally, we prefer docstrings here over such comments, if required.

stefan6419846 · 2026-11-14T22:31:34Z

+
+
+def test_get_object__recovery_index_finds_object_missing_from_xref() -> None:
+    # https://github.com/py-pdf/pypdf/issues/4428


stefan6419846 · 2026-11-14T22:31:49Z

+    # A second lookup comes from the cache, without another scan.
+    with mock.patch("pypdf._reader.get_stream_data", wraps=get_stream_data) as get_stream_data_mock:
+        assert reader.get_object(IndirectObject(4, 0, reader)) == 42
+    assert get_stream_data_mock.call_count == 0


Suggested change

assert get_stream_data_mock.call_count == 0

get_stream_data_mock.assert_not_called()

stefan6419846 requested changes Nov 14, 2026

View reviewed changes

Navigation Menu

Repository navigation

ROB: Scan the file once when resolving undefined references - #4431

ROB: Scan the file once when resolving undefined references#4431
SudiptoKumar wants to merge 1 commit into
py-pdf:mainfrom
SudiptoKumar:fix-undefined-ref-recovery-index

SudiptoKumar commented Nov 14, 2026

Uh oh!

codecov Bot commented Nov 14, 2026

Uh oh!

stefan6419846 left a comment

Uh oh!

stefan6419846 Nov 14, 2026

Uh oh!

stefan6419846 Nov 14, 2026

Uh oh!

stefan6419846 Nov 14, 2026

Uh oh!

stefan6419846 Nov 14, 2026

Uh oh!

stefan6419846 Nov 14, 2026

Uh oh!

stefan6419846 Nov 14, 2026

Uh oh!

stefan6419846 Nov 14, 2026

Uh oh!

stefan6419846 Nov 14, 2026

Uh oh!

stefan6419846 Nov 14, 2026

Uh oh!

stefan6419846 Nov 14, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	#: Index of indirect-object headers, built lazily on the first
	#: Index of indirect object headers, built lazily on the first

	#: undefined-reference lookup; maps (generation, idnum) — as the
	#: undefined reference lookup; maps (generation, idnum) - as the

	#: digit strings from the file — to the (start, end) offsets of
	#: digit strings from the file - to the (start, end) offsets of



		def test_get_object__speed__distinct_invalid_lookups_scan_file_once() -> None:
		# https://github.com/py-pdf/pypdf/issues/4428



		def test_get_object__recovery_index_finds_object_missing_from_xref() -> None:
		# https://github.com/py-pdf/pypdf/issues/4428

	assert get_stream_data_mock.call_count == 0
	get_stream_data_mock.assert_not_called()

Conversation

SudiptoKumar commented Nov 14, 2026

Bug

Root cause

Fix

Tests

Uh oh!

codecov Bot commented Nov 14, 2026

Codecov Report

Uh oh!

stefan6419846 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants