Repository navigation
Give each MemoryFileSystem reader its own read-only file handle - #2480
Open
zhixiangli wants to merge 4 commits into
Open
zhixiangli wants to merge 4 commits into
zhixiangli wants to merge 4 commits into
Conversation
zhixiangli
force-pushed
the
fix/memory-independent-read-handles
branch
from
November 18, 2026 16:14
648720c to
d32e7b6
Compare
|
Additional platform/interpreter validation on d32e7b6: the full test_memory.py suite passes on native Windows with CPython 3.12.14: 178 passed. In particular, both CPython identity-based checks pass here: the reader/info/cat path retains the stored bytes object, and a closed-but-still-referenced reader does not force a copy on the subsequent update write. The snapshot, read-only buffer, independent-position and gzip cases pass as well. This extends the behavioral checks beyond the reported Linux/3.13 environment; it is not a memory/RSS benchmark or performance claim, and I did not test FUSE. No source changes. Codex assisted with running and reviewing the tests. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MemoryFileSystem.open(path, "rb")returns the storedMemoryFileitself, so every reader of a path is the same object:No other filesystem behaves like this, and the memory filesystem is what code built on fsspec is usually tested against, so code that reads one file through several handles fails there and nowhere else. I ran into it with parallel ranged reads of one file.
What this changes
open(path, "rb")returns a new read-only handle with its own position. The stored object is left alone, so an openr+bhandle keeps its position too.getvalue(), which shares the stored bytes object, so opening a reader copies nothing.write,writelinesandtruncateon the handle raiseio.UnsupportedOperation, which is whatLocalFileSystemgives for a file openedrb.getbuffer()returns a read-only view.MemoryFile.sizeandcat_fileusegetvalue()instead ofgetbuffer().getbuffer()makes a private copy of a buffer that is shared with a reader, soinfo(),ls()orcat_file()after anrbopen would otherwise copy the whole file.r+b,ab,a+b, the write modes and transactions are unchanged.Behaviour changes
rbhandle used to modify the stored file. It now raises.r+borabwrite is seen by readers opened after it, not by ones already open.closedand raises if it is used again.r+b,ab) while a reader is open copies the file once, because the bytes are shared until then. Overwriting withwbis unaffected.fsspec.fuseopens a read-write request (O_RDWR) as"rb"and writes through that handle. On a memory-backed mount this worked only because the handle was the stored object; it now raises, as it already does on other backends. I have leftfuse.pyalone and have not tested a real mount. I can mapO_RDWRto an update mode here or in a follow-up if you would like that.Measurements
CPython 3.13.15, Linux x86-64, against
masterat 2289743.rbhandle on a 256 MiB filecat_fileof a 64 MiB fileThe last row is about 100 ms for the first loop in a fresh process on this branch, and 44–48 ms after that.
Tests
17 new tests in
test_memory.py. Two of them assert that nothing was copied by comparing object identity, which relies on CPython sharing the bytes.