Skip to content

Plan direct URL dependencies as installed by their direct_url.json - #70

Open
morisil wants to merge 6 commits into
mainfrom
issue-25-direct-url-installed
Open

morisil wants to merge 6 commits into
mainfrom
issue-25-direct-url-installed

Conversation

@morisil

@morisil morisil commented Nov 11, 2026

Copy link
Copy Markdown
Member

Fixes #25

The PLAN script no longer treats every direct URL requirement (pkg @ file:///…, pkg @ git+…, pkg @ https://…) as missing:
it compares the requirement with the direct_url.json (PEP 610) of the installed distribution.

  • VCS: same repository and subdirectory, and same commit (full hash) or same requested revision.
  • Archives: same URL and matching recorded hashes (pip), or the hash fragment of the recorded URL / the change time of a local archive in uv_cache.json (uv, which records no archive hashes); a local archive is rehashed.
  • Always missing (reinstalled, as before): local directories, URL requirements with extras, a name with more than one dist-info in the dependency directory (stale one left by pip --target --upgrade, pip --target leaves the dist-info of a replaced version behind, which falsely satisfies requirements #59), a requested hash of an algorithm the installer did not record.
  • When anything else is missing, the snippet's satisfied URL requirements are passed to the installer again — otherwise pip would replace them by the index copy.
  • README documents the trade-offs (a branch stays on the first-installed commit; uv trusts a remote archive by URL; the always-reinstalled cases).

Review

  • Earlier run (2026-11-09): 3 review rounds, blocked on two remaining false "satisfied" cases under pip.
  • This run: both fixed (stale dist-info, hash of another algorithm), main merged in; 1 review round — no blocking findings, verified live against real pip install --target directories and a local git repo. Polish applied (recorded hashes win over URL fragment, no re-parsing, lazy url2pathname).

Follow-ups

Pre-existing on main, commented on the existing issues rather than filed anew:

The next fix, #6, will be stacked on this branch — merge this PR first.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG

morisil and others added 6 commits November 9, 2026 22:36
Every requirement with a URL was planned as missing, so it was installed
again on each use of a snippet declaring it - also when already imported,
overwriting the files of a loaded module (a PermissionError on Windows),
and with a download or a build each time.

PLAN now compares the requirement with the PEP 610 direct_url.json of the
installed distribution, without hitting the network:
- VCS: the same repository URL (without credentials) and subdirectory, and
  the same commit for a full commit hash, or otherwise the same requested
  revision - a branch keeps the commit installed for it first, as the
  latest one cannot be known offline, while another branch of the same
  version is still installed.
- Archive: the same URL (without credentials and fragment) and
  subdirectory, and each hash of the URL fragment matching the recorded
  hashes; a local file:// archive is hashed again, as it can be rebuilt
  under the same name.
- Local directory: always installed again, as its content can change.
A distribution installed from an index never satisfies a URL requirement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
uv records "archive_info": {} without hashes, so a local archive and a
hash-pinned archive URL were still installed again on every use with uv,
which is used whenever it is on the PATH:
- a recorded hash also comes from the hash fragment, which uv keeps in
  the URL of a local archive;
- a remote archive with the same URL is satisfied unless a recorded hash
  contradicts the requested one, as uv drops the fragment of a remote URL
  and records no hash of it;
- a local archive without a recorded hash is compared by its modification
  time with the one uv records in uv_cache.json, as uv does itself.
A local archive is now hashed in one pass for all the recorded algorithms.

A URL dependency with extras stays missing, as the dependencies of the
extras are not checked (#47) - otherwise they would never be installed
when the base distribution came from the same URL.

The README tells how to update a branch or a remote archive whose content
changes under the same URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
…and shake hashes (#25)

pip installs into --target without looking at it, so when another
dependency of the same code was missing and required a distribution
installed from a URL, pip installed that distribution from the index
over the URL one, in the same version, which a constraint cannot
prevent. Each direct URL dependency is now passed to the installer
again whenever anything else is missing, as before #25, so the
installation is avoided only when nothing else is installed.

A shake_128 or shake_256 hash fragment failed every plan with a
TypeError, as its digest has no fixed length; such a local archive is
now planned as missing.

uv records the change time of a local archive in uv_cache.json on Unix
(the last write time on Windows), not its modification time, so that is
what is compared now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
…hm as missing (#25)

pip install --target --upgrade leaves the dist-info of the previous version,
maybe with its direct_url.json, so a URL requirement is missing
when the dependency directory holds more than one dist-info of the distribution.
A requested hash of another algorithm than the ones pip recorded
for a remote archive cannot be verified, so it is missing too,
keeping the trust in the URL when uv recorded no hashes.
url2pathname is imported only for a local archive,
as it imports http.client, email and ssl.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
…L dependencies (#25)

The hashes in archive_info are the ones verified on installation,
so a hash in the fragment of the recorded URL only adds an algorithm.
The URL requirements are collected by is_missing instead of parsed again.
README lists the cases of a direct URL dependency installed again on each use:
a stale dist-info of another version, and a hash of an algorithm pip does not record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FJmHQ3jHRbVdpwUiNUkApG
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Direct URL dependencies are reinstalled on every use, even when imported

1 participant