Skip to content

Support PyArrow 26 in the Python SDK - #40927

Open
Anurag-S1ngh wants to merge 1 commit into
apache:masterfrom
Anurag-S1ngh:feat/40919-pyarrow-26
Open

Anurag-S1ngh wants to merge 1 commit into
apache:masterfrom
Anurag-S1ngh:feat/40919-pyarrow-26

Conversation

@Anurag-S1ngh

Copy link
Copy Markdown

Summary

Adds PyArrow 26 to the supported range of the Python SDK and to the dependency postcommit matrix.

PyArrow 26 requires NumPy 2 at runtime, so the new test environment installs NumPy 2 while the existing PyArrow 17 to 25 environments keep using NumPy 1.26.4.

Changes

  • sdks/python/setup.py: raise the pyarrow upper bound from <26.0.0 to <27.0.0
  • sdks/python/tox.ini: add 26 to the py311-pyarrow-... environment, add 26: pyarrow>=26,<27, and make NumPy conditional so 17 to 25 stay on 1.26.4 while 26 uses numpy>=2.0.0,<2.5.0
  • sdks/python/test-suites/tox/py311/build.gradle: add testPy311pyarrow-26 and wire it into test and postCommitPyDep

Testing

Installed the SDK from source with Python 3.11, pyarrow==26.0.0 and numpy==2.4.6:

  • pytest apache_beam/io/parquetio_test.py -m uses_pyarrow: 35 passed, 1 skipped, 5 deselected
  • pytest apache_beam/typehints/arrow_type_compatibility_test.py apache_beam/typehints/schemas_test.py: 821 passed, 5 skipped
  • tox c -e py311-pyarrow-26 -k deps resolves pyarrow>=26,<27 and numpy>=2.0.0,<2.5.0
  • tox c -e py311-pyarrow-25 -k deps still resolves pyarrow>=25,<26 and numpy==1.26.4

The new full matrix entry runs as testPy311pyarrow-26 for reviewers who want the complete uses_pyarrow suite.

fixes #40919

Raise the pyarrow upper bound to <27.0.0 and add the 26 tox environment and Gradle task so the dependency postcommit covers it. PyArrow 26 requires NumPy 2, so that environment installs NumPy 2 while the older environments stay on 1.26.4.
@github-actions github-actions Bot added python size/s S: 10-49 lines changed labels Oct 31, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR adds support for pyarrow 26 by raising the upper bound to <27.0.0 in sdks/python/setup.py and adding the py311-pyarrow-26 test environment in sdks/python/tox.ini and sdks/python/test-suites/tox/py311/build.gradle. Because pyarrow 26 dropped support for numpy 1, the tox configuration appropriately pins numpy>=2.0.0,<2.5.0 for pyarrow 26 while preserving numpy==1.26.4 for earlier pyarrow versions.

🔍 General Feedback

  • The changes across setup.py, tox.ini, and build.gradle are clean, well-aligned, and follow the established patterns used for previous dependency updates (such as the recent pandas 2.3 / NumPy 2 split).
  • Consider incrementing .github/trigger_files/beam_PostCommit_Python_Dependency.json if you would like to trigger the PostCommit Python Dependency workflow directly on this PR to verify the new testPy311pyarrow-26 Gradle task in CI before merging.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @tvalentyn for label python.
Backup reviewer: claudevdm (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request]: Support PyArrow 26 in the Python SDK

1 participant