Skip to content

feat: add pybend Python binding with S3 connection and stage support - #18704

Merged
bohutang merged 6 commits into
databendlabs:mainfrom
bohutang:pybend-r2
Sep 9, 2025
Merged

bohutang merged 6 commits into
databendlabs:mainfrom
bohutang:pybend-r2

Conversation

@bohutang

@bohutang bohutang commented Sep 9, 2025 •

Copy link
Copy Markdown
Member

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

feat: add pybend Python binding with S3 connection and stage support

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Other

This change is Reviewable

- Add new Python binding examples for SQL with Parquet on S3 and S3 stages
- Implement connection-based table registration with S3/Azure/GCS/OSS/COS support
- Add comprehensive stage management functions (create, list, describe, drop)
- Update context API to support connection parameter in register_table methods
- Enhance embedded configuration with improved logging and directory structure
- Add connection tests and update existing test suites for compatibility
@github-actions github-actions Bot added the pr-feature this PR introduces a new feature to the codebase label Sep 9, 2025
@bohutang
bohutang marked this pull request as draft September 9, 2025 10:11
@github-actions

github-actions Bot commented Sep 9, 2025 •

Copy link
Copy Markdown
Contributor

🤖 Smart Auto-retry Analysis

Workflow: 17588325453

📊 Summary

  • Total Jobs: 83
  • Failed Jobs: 1
  • Retryable: 0
  • Code Issues: 1

❌ NO RETRY NEEDED

All failures appear to be code/test issues requiring manual fixes.

🔍 Job Details

  • ❌ linux / test_private_tasks: Not retryable (Code/Test)

🤖 About

Automated analysis using job annotations to distinguish infrastructure issues (auto-retried) from code/test issues (manual fixes needed).

- Add create_python_binding_telemetry_payload function to collect system info
- Report telemetry data during init_embedded initialization
- Include system specs, cache config, memory management, and storage engine info
- Use dedicated event_type 'python_binding_init' and source 'databend-python-binding'
- Mandatory reporting for Python binding usage analytics
Ensure license headers match the expected format for license checker.
- Enable intelligent memory management (80% system memory limit, 60% spill threshold)
- Simplify README with focused examples for local/remote files and tables
- Add comprehensive telemetry for memory and spill configuration
- Remove redundant API documentation in favor of practical examples
@bohutang
bohutang marked this pull request as ready for review September 9, 2025 23:11
@bohutang
bohutang merged commit 5d5a193 into databendlabs:main Sep 9, 2025
91 of 93 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature this PR introduces a new feature to the codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant