Repository navigation
Conversation
blob.Reader issues a single GET per object, so Bucket.Download streams large objects on one connection and can't saturate available bandwidth. Add an optional driver.DownloaderAt interface. When the download destination is an io.WriterAt and the driver implements DownloaderAt, Bucket.Download routes to it and skips opening a Reader, avoiding the eager GET that NewRangeReader would issue and discard. Drivers that don't implement it keep the existing single-stream behavior, and the public API and driver.Bucket interface are unchanged. Implement DownloaderAt in s3blob using the AWS transfer manager (feature/s3/transfermanager), which this driver already depends on for uploads. It fetches the object with concurrent ranged GETs written directly to the destination offsets. Issue: google#3804
Add TestDownloaderAt covering the three cases in Bucket.Download: an io.WriterAt destination with a driver that implements DownloaderAt uses DownloadAt; a non-io.WriterAt destination falls back to the reader even when DownloaderAt is available; and an io.WriterAt destination with a driver that lacks DownloaderAt falls back to the reader.
Download and Upload were added in 2023 but never folded into the blob howto, which documents only NewReader/NewRangeReader for reading. Add an ExampleBucket_Download that downloads into an *os.File (an io.WriterAt, so the parallel path is used on S3) and reference it from the "Reading data" section, alongside regenerating the examples.json entry.
Implement DownloadAt using the azblob DownloadFile helper, a method on the same *blob.Client the driver already uses, so this adds no new import or dependency. When the destination is an *os.File it downloads the blob in parallel blocks; for any other io.WriterAt it falls back to a single sequential stream, since azblob has no parallel download into a generic io.WriterAt. With this, blob.Bucket.Download into an *os.File fetches Azure blobs in parallel instead of on a single stream. Part of google#3804
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3805 (which adds
driver.DownloaderAtand theBucket.Downloadrouting); please review/merge that first. This PR's diff includes #3805's commit until it merges.What
Implements
driver.DownloaderAtin azureblob, soBucket.Downloadinto an*os.Filefetches Azure blobs with parallel blocks instead of a single stream.How
Uses the azblob
DownloadFilehelper, a method on the same*blob.Clientthe driver already uses forDownloadStream, so this adds no new import or dependency. It reuses the driver's existing client.Azure's parallel download helpers only accept
*os.File(DownloadFile) or[]byte(DownloadBuffer); there is no parallel download into a genericio.WriterAt. So:*os.Filedestination ->DownloadFile(parallel blocks).io.WriterAt-> single sequentialDownloadStream+ copy, which is correct but not parallel.The
*os.Filecase is the common one (downloading to disk).BeforeReadexposes*blob.DownloadFileOptions/*blob.Clienton the parallel path, and*blob.DownloadStreamOptionson the fallback.Block size and concurrency use the azblob defaults; a follow-up can thread tunables through once #3806 (the
ReaderOptionsknobs) lands.Testing
go vet,gofmtclean. Compile-timevar _ driver.DownloaderAt = (*bucket)(nil)assertion added.Bucket.Download->DownloadAt) is unit-tested in blob: add optional parallel download via io.WriterAt #3805 with a fake driver.