Repository navigation
Let buffered files defer the info() call so a whole-file read is one request - #2432
Open
bhaskargurram-ai wants to merge 2 commits into
Open
bhaskargurram-ai wants to merge 2 commits into
bhaskargurram-ai wants to merge 2 commits into
Conversation
…request Opening an AbstractBufferedFile in "rb" mode calls fs.info() to learn the file size, so the common ``open(path).read()`` pattern costs two round trips: the info() call and then a ranged fetch of the whole file, which most backends can serve without knowing the size. Add a class attribute, ``lazy_size`` (default False), that defers the info() call. The size and the read buffer are created on first use, so read(n), readinto, seek(..., 2), tell, len, pickling and an explicit size= behave as before. A read() or read(-1) from position 0, before anything needed the size, is served by a single fs.cat_file(path) call; the length of the result becomes the size, so later seeks and reads still do not call info(). Backends opt in on their file class. It requires that fs.cat_file does not itself go through open(), and that fetches do not depend on details such as an ETag or a version id, which is why the default keeps every existing file class on the eager path. The HTTP backend looks the size up in _open and is unchanged here. Tests use a request-counting filesystem to check the single call for the whole-file read, that partial reads, seek/tell, readinto, an explicit size, info() before read, pickling and close keep today's call pattern, and that the default file class still calls info() at open time. Fixes fsspec#1392
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AbstractBufferedFilecallsfs.info()as soon as it is opened in "rb" mode, to learn the file size. For the pattern in #1392,with fsspec.open(url) as f: process(f.read()), that is two round trips where one would do: theinfo()call, then a ranged fetch of the whole file, which most backends can serve without knowing the size. On a local HTTP server I seeHEADplusGET bytes=0-199999for a 200 kB file wherecat_filedoes a singleGET.This adds a class attribute
lazy_size(default False) onAbstractBufferedFile. When a file class sets it, opening in "rb" mode without an explicitsize=does not callinfo(). The size and the read buffer are created the first time something needs them, soread(n),readinto,seek(..., 2),tell(),len, pickling and an explicitsize=keep today's behaviour and call pattern. Aread()orread(-1)from position 0, before anything needed the size, is served by a singlefs.cat_file(path); the length of the result becomes the size, so later seeks and reads still never callinfo().I kept it opt-in rather than changing the default. The fast path needs
fs.cat_filenot to go throughopen()(the base implementation does, which would recurse), and backends such as s3fs and gcsfs readdetailsat open time for ETag, version or generation pinning, so their fetches should stay on that path until they route a whole-file read through the same keyed request. It also moves the "file not found" error fromopen()to the first read, which each backend should decide on for itself. The HTTP backend looks the size up in_opento choose betweenHTTPFileandHTTPStreamFile, so it is unchanged here; I can look at that separately if you want the same shortcut there.The tests use a small request-counting filesystem. The whole-file read makes exactly one
cat_filecall (on master it makesinfoplus a rangedcat_file); partial reads,seek/tell,readinto, explicitsize=,info()followed byread(), pickling andclose()keep the existing pattern; and the default file class still callsinfo()at open.Fixes #1392