Skip to content

Add reqstorm - #127

Open
melihcolpan wants to merge 1 commit into
timofurrer:mainfrom
melihcolpan:add-reqstorm
Open

melihcolpan wants to merge 1 commit into
timofurrer:mainfrom
melihcolpan:add-reqstorm

Conversation

@melihcolpan

Copy link
Copy Markdown

Adds reqstorm to Networking.

httpx and aiohttp send requests; reqstorm handles the work around sending thousands of them to real APIs, which otherwise ends up as hand-written plumbing in every project:

  • per-host rate limits in any unit ("100/min"), or "auto" to follow the server's 429 responses and rate-limit headers
  • retries with capped, jittered exponential backoff and Retry-After, plus retry rounds at the end of the batch
  • pagination (next links, Link headers, cursors, page numbers), token refresh on 401, ETag caching, HTTP and SOCKS proxies
  • one result per request with the full attempt history, so a failure never stops the batch
  • results streamed to JSON Lines, CSV, SQLite, PostgreSQL or MySQL, with resume after an interruption
  • a live progress line with ETA, and per-run logging

It is built on aiohttp, has both an asyncio and a blocking API, a reqstorm command, typed code and tests on Python 3.9 to 3.13. Documentation: https://reqstorm.github.io

I'm the author.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant