Skip to content

Let buffered files defer the info() call so a whole-file read is one request - #2432

Open
bhaskargurram-ai wants to merge 2 commits into
fsspec:masterfrom
bhaskargurram-ai:perf/lazy-size-sequential-read
Open

bhaskargurram-ai wants to merge 2 commits into
fsspec:masterfrom
bhaskargurram-ai:perf/lazy-size-sequential-read

Conversation

@bhaskargurram-ai

Copy link
Copy Markdown
Contributor

AbstractBufferedFile calls fs.info() as soon as it is opened in "rb" mode, to learn the file size. For the pattern in #1392, with fsspec.open(url) as f: process(f.read()), that is two round trips where one would do: the info() call, then a ranged fetch of the whole file, which most backends can serve without knowing the size. On a local HTTP server I see HEAD plus GET bytes=0-199999 for a 200 kB file where cat_file does a single GET.

This adds a class attribute lazy_size (default False) on AbstractBufferedFile. When a file class sets it, opening in "rb" mode without an explicit size= does not call info(). The size and the read buffer are created the first time something needs them, so read(n), readinto, seek(..., 2), tell(), len, pickling and an explicit size= keep today's behaviour and call pattern. A read() or read(-1) from position 0, before anything needed the size, is served by a single fs.cat_file(path); the length of the result becomes the size, so later seeks and reads still never call info().

I kept it opt-in rather than changing the default. The fast path needs fs.cat_file not to go through open() (the base implementation does, which would recurse), and backends such as s3fs and gcsfs read details at open time for ETag, version or generation pinning, so their fetches should stay on that path until they route a whole-file read through the same keyed request. It also moves the "file not found" error from open() to the first read, which each backend should decide on for itself. The HTTP backend looks the size up in _open to choose between HTTPFile and HTTPStreamFile, so it is unchanged here; I can look at that separately if you want the same shortcut there.

The tests use a small request-counting filesystem. The whole-file read makes exactly one cat_file call (on master it makes info plus a ranged cat_file); partial reads, seek/tell, readinto, explicit size=, info() followed by read(), pickling and close() keep the existing pattern; and the default file class still calls info() at open.

Fixes #1392

…request

Opening an AbstractBufferedFile in "rb" mode calls fs.info() to learn the
file size, so the common ``open(path).read()`` pattern costs two round
trips: the info() call and then a ranged fetch of the whole file, which
most backends can serve without knowing the size.

Add a class attribute, ``lazy_size`` (default False), that defers the
info() call. The size and the read buffer are created on first use, so
read(n), readinto, seek(..., 2), tell, len, pickling and an explicit size=
behave as before. A read() or read(-1) from position 0, before anything
needed the size, is served by a single fs.cat_file(path) call; the length
of the result becomes the size, so later seeks and reads still do not call
info().

Backends opt in on their file class. It requires that fs.cat_file does not
itself go through open(), and that fetches do not depend on details such
as an ETag or a version id, which is why the default keeps every existing
file class on the eager path. The HTTP backend looks the size up in _open
and is unchanged here.

Tests use a request-counting filesystem to check the single call for the
whole-file read, that partial reads, seek/tell, readinto, an explicit size,
info() before read, pickling and close keep today's call pattern, and that
the default file class still calls info() at open time.

Fixes fsspec#1392
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shortcut to allow sequential read?

1 participant