Repository navigation
fix(rag): treat a docs path on github.com as a repository - #9547
Vidit Ostwal (Vidit-Ostwal) merged 1 commit into
Conversation
Strix Security ReviewWarning This pull request has 22 commits after the last Strix review ( ✅ No open security findings. PR summary: This PR reorders the checks in `DataTypes.from_content` so the GitHub hostname is resolved before the path-based "docs" heuristic: `https://github.com/<org>/docs`-style URLs now classify as `GITHUB` and reach `GithubLoader`, while `docs.github.com` and docs paths on other hosts still classify as `DOCS_SITE`. It also adds tests for the new and preserved classifications. Review summaryNo security issues found. The change only alters which loader a URL is dispatched to, and only for URLs whose parsed hostname is The new destination is strictly more constrained than the old one: One non-security consequence for the author: host forms the classifier now labels Ruled out (5)
Review 1 on this PR · Reviewed by Strix |
`DataTypes.from_content` checked for "docs" in the URL path before it checked whether the host is GitHub. Any GitHub repository whose name or path contains "docs" (for example https://github.com/crewAIInc/docs) was therefore classified as DOCS_SITE and sent to the documentation crawler instead of GithubLoader. Run the hostname-based GitHub check first, but keep a docs host such as docs.github.com and docs paths on other hosts classified as DOCS_SITE.
6ea6331 to
a4ab8b5
Compare
Vidit Ostwal (Vidit-Ostwal)
left a comment
There was a problem hiding this comment.
Thanks, this is a clean, minimal fix: resolving the GitHub host first and skipping only the path-based docs check keeps docs.github.com and non-GitHub docs paths as DOCS_SITE. I verified test_repository_named_docs_is_github fails on main (docs_site vs github) and passes with this change, and the full test_data_types.py module passes (22 passed with the issue test). Optional: add a case for a deeper path like github.com/org/repo/tree/main/docs to pin the intended behavior.
Fixes #9542
Summary
DataTypes.from_contentchecked for"docs"in the URL path before it checked whether the host is GitHub, so a repository such ashttps://github.com/crewAIInc/docswas classified asDOCS_SITEand never reachedGithubLoader.This change resolves the GitHub hostname first and skips the path-based docs check on GitHub hosts. A docs subdomain such as
docs.github.comand docs paths on other hosts still classify asDOCS_SITE.Verification
test_repository_named_docs_is_githubfails on main (DataType.DOCS_SITEvsDataType.GITHUB) and passes with the change.https://docs.github.com/en/rest,https://example.com/docs/introandhttps://docs.example.com/guidestill returnDOCS_SITE.tests/rag/test_data_types.pymodule passes (20 passed).AI assistance
I used an AI coding agent to investigate, implement, and test this change, and reviewed its results. Per CONTRIBUTING.md, please apply the
llm-generatedlabel; external contributors cannot add it with my current permissions.