All pull requests

Pull requests

• Add developer documentation for building from source and creating extensions

  Stale
  #59·P4X-ng with CopilotAI opened Nov 18, 2025·

  ·
  1

  1

  

• Add path-based filtering to restrict crawling to URL subtrees

  Stale
  #60·P4X-ng with CopilotAI opened Nov 18, 2025·

  ·
  1

  1

  

• Fix scrapy.xlib import for Scrapy 2.0+ compatibility

  Stale
  #58·P4X-ng with CopilotAI opened Nov 18, 2025·

  ·
  1

  1

  

• Fix Windows compatibility: replace platform-specific num_fds() with runtime check

  #61·P4X-ng with CopilotAI opened Nov 18, 2025·

  ·
  1

  

• Bump version to 0.3 with maintenance update

  Stale
  #56·P4X-ng with CopilotAI opened Nov 18, 2025·

  ·
  1

  1

  

• Fix WebSocket RPC message format mismatch breaking "Known sites" feature

  Stale
  #57·P4X-ng with CopilotAI opened Nov 18, 2025·

  ·
  1

  1

  

• Persist paused crawl state across restarts

  Stale
  #49·P4X-ng with CopilotAI opened Nov 18, 2025·

  ·
  1

  1

  

• Add resume_on_start option to control crawl resumption on restart

  Stale
  #48·P4X-ng with CopilotAI opened Nov 18, 2025·

  ·
  1

  1

  

• Add default MongoDB URIs to settings for MongoExportPipeline

  Stale
  #50·P4X-ng with CopilotAI opened Nov 18, 2025·

  ·
  1

  1

  

• Fix log_count/DEBUG continuing to increment after spider closes

  Stale
  #52·P4X-ng with CopilotAI opened Nov 18, 2025·

  ·
  1

  1

  

• Throttle WebSocket updates and optimize React rendering for transfer-heavy crawls

  Stale
  #51·P4X-ng with CopilotAI opened Nov 18, 2025·

  ·
  1

  1

  

• Fix SafeConfigParser import for Python 3.12+ compatibility

  Stale
  #53·P4X-ng with CopilotAI opened Nov 18, 2025·

  ·
  1

  1

  

• Add documentation and examples for custom spider usage

  Stale
  #54·P4X-ng with CopilotAI opened Nov 18, 2025·

  ·
  1

  1

  

• Add project upload endpoint for running custom Scrapy spiders

  Stale
  #55·P4X-ng with CopilotAI opened Nov 18, 2025·

  ·
  1

  4

  

• Bump twisted from 16.2.0 to 22.10.0

  dependenciesStale
  #47·dependabot[bot] opened Nov 4, 2022Bot·

  ·
  1

• Bump scrapy from 1.1.0 to 2.6.2

  dependenciesStale
  #46·dependabot[bot] opened Jul 29, 2022Bot·

  ·
  1

• Bump psutil from 4.2.0 to 5.6.6

  dependencies
  #37·dependabot[bot] opened Mar 12, 2020Bot·

  ·

• updated correct URL for tox's documentation in README

  #28·j-gao opened Nov 2, 2016·

  ·

• Custom spider argument support

  #14·sardok opened Mar 31, 2016·

  ·