Skip to content

fix(rag): treat a docs path on github.com as a repository - #9547

Merged
Vidit Ostwal (Vidit-Ostwal) merged 1 commit into
crewAIInc:mainfrom
Kiko0301:fix/data-types-github-docs-url
Nov 19, 2026
Merged

Vidit Ostwal (Vidit-Ostwal) merged 1 commit into
crewAIInc:mainfrom
Kiko0301:fix/data-types-github-docs-url

Conversation

@Kiko0301

Copy link
Copy Markdown
Contributor

Fixes #9542

Summary

DataTypes.from_content checked for "docs" in the URL path before it checked whether the host is GitHub, so a repository such as https://github.com/crewAIInc/docs was classified as DOCS_SITE and never reached GithubLoader.

This change resolves the GitHub hostname first and skips the path-based docs check on GitHub hosts. A docs subdomain such as docs.github.com and docs paths on other hosts still classify as DOCS_SITE.

Verification

  • New test test_repository_named_docs_is_github fails on main (DataType.DOCS_SITE vs DataType.GITHUB) and passes with the change.
  • New guards confirm https://docs.github.com/en/rest, https://example.com/docs/intro and https://docs.example.com/guide still return DOCS_SITE.
  • The full tests/rag/test_data_types.py module passes (20 passed).

AI assistance

I used an AI coding agent to investigate, implement, and test this change, and reviewed its results. Per CONTRIBUTING.md, please apply the llm-generated label; external contributors cannot add it with my current permissions.

@strix-security

strix-security Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Strix Security Review

Warning

This pull request has 22 commits after the last Strix review (6ea6331). Strix has not reviewed these changes.
Automatic review on push is off for this repository. To review the latest changes, tag @strix-security in a comment, or turn on re-review on push.

✅ No open security findings.

PR summary: This PR reorders the checks in `DataTypes.from_content` so the GitHub hostname is resolved before the path-based "docs" heuristic: `https:​//github.com/<org>/docs`-style URLs now classify as `GITHUB` and reach `GithubLoader`, while `docs.github.com` and docs paths on other hosts still classify as `DOCS_SITE`. It also adds tests for the new and preserved classifications.

Review summary

No security issues found. The change only alters which loader a URL is dispatched to, and only for URLs whose parsed hostname is github.com or a *.github.com subdomain and whose path contains "docs" — those move from the docs-site loader to the GitHub loader.

The new destination is strictly more constrained than the old one: GithubLoader rejects anything not literally prefixed https://github.com/, takes only the first two path segments as owner/repo, and issues PyGithub calls against the fixed API host, so no attacker-chosen host, port or URL is ever fetched and the configured token cannot be steered to a third party. The "docs" in url.netloc clause that catches docs.github.com is unchanged and still runs first, the hostname match cannot be spoofed through userinfo, unicode, percent-encoding, trailing dots or backslash forms, and the extension check that precedes the reordered block is untouched, so the docs-site fetch path is only ever narrowed, never widened.

One non-security consequence for the author: host forms the classifier now labels GITHUB but the loader's literal prefix check rejects — http://github.com/.../docs, https://www.github.com/..., https://github.com:8443/..., https://api.github.com/..., https://gist.github.com/... — now raise Invalid GitHub URL instead of loading, and the new tests do not cover those forms.

Ruled out (5)
  • Can a crafted URL make the classifier pick GITHUB while the loader contacts (or sends the configured token to) a non-GitHub host?
    Userinfo, backslash, port, uppercase, IDNA, unicode-dot and percent-encoded variants either fail the hostname match outright or are rejected by the loader's literal `https:​//github.com/` prefix check before any request is made, and `repo_name` only ever becomes a path segment on the fixed API host.
  • Does routing these URLs to the GitHub loader drop a security control the docs-site loader applied, such as SSRF validation or size limits?
    The GitHub loader never fetches the supplied URL at all — it derives `owner/repo` and calls the GitHub API with a token that in this repository is only plumbed by a tool that already forced `DataType.GITHUB` — so no untrusted destination becomes reachable.
  • Does the change increase reachability of the unbounded fetch and whole-body read paths in the text, webpage and docs-site loaders, or of the character-level ch…
    The extension check still runs first, the added `not is_github` clause is a conjunct that can only remove URLs from the docs branch, and both `GITHUB` and `DOCS_SITE` map to the same text chunker, so no input newly reaches those sinks.
  • Are the directory, MySQL/Postgres, XML, DOCX or YouTube-channel loader paths newly reachable through this reclassification?
    Those loaders are selected either by the untouched extension check or by an explicit `data_type`, and `from_content` never returns those types for URL input, so the diff changes nothing on those paths.
  • Can a GitHub token reach the model or a remote host by way of the newly reachable loader?
    The loader never puts the token into content, result metadata or exception text, and the only in-tree caller that supplies one forces `DataType.GITHUB` explicitly, so this reclassification does not open a new token path.

Review 1 on this PR · 6ea6331 · requested Nov 18 09:57 UTC when the PR was opened


Reviewed by Strix
Re-run review · Configure security review settings · Try a deeper review with @strix-security /deep

`DataTypes.from_content` checked for "docs" in the URL path before it
checked whether the host is GitHub. Any GitHub repository whose name or
path contains "docs" (for example https://github.com/crewAIInc/docs) was
therefore classified as DOCS_SITE and sent to the documentation crawler
instead of GithubLoader.

Run the hostname-based GitHub check first, but keep a docs host such as
docs.github.com and docs paths on other hosts classified as DOCS_SITE.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is a clean, minimal fix: resolving the GitHub host first and skipping only the path-based docs check keeps docs.github.com and non-GitHub docs paths as DOCS_SITE. I verified test_repository_named_docs_is_github fails on main (docs_site vs github) and passes with this change, and the full test_data_types.py module passes (22 passed with the issue test). Optional: add a case for a deeper path like github.com/org/repo/tree/main/docs to pin the intended behavior.

@Vidit-Ostwal
Vidit Ostwal (Vidit-Ostwal) merged commit 86b9b09 into crewAIInc:main Nov 19, 2026
55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] DataTypes.from_content classifies GitHub repositories named docs as DOCS_SITE

2 participants