Skip to content

Give each MemoryFileSystem reader its own read-only file handle - #2480

Open
zhixiangli wants to merge 4 commits into
fsspec:masterfrom
zhixiangli:fix/memory-independent-read-handles
Open

zhixiangli wants to merge 4 commits into
fsspec:masterfrom
zhixiangli:fix/memory-independent-read-handles

Conversation

@zhixiangli

Copy link
Copy Markdown
Contributor

MemoryFileSystem.open(path, "rb") returns the stored MemoryFile itself, so every reader of a path is the same object:

import fsspec

fs = fsspec.filesystem("memory")
fs.pipe_file("/a", b"0123456789")

first = fs.open("/a", "rb")
first.seek(6)
second = fs.open("/a", "rb")

first is second   # True
first.tell()      # 0: opening the path again rewound the first reader
second.read(3)
first.read(2)     # b"34": the two readers share one cursor
first.write(b"XY")
fs.cat_file("/a") # b"01234XY789": a write through an "rb" handle changed the file

No other filesystem behaves like this, and the memory filesystem is what code built on fsspec is usually tested against, so code that reads one file through several handles fails there and nowhere else. I ran into it with parallel ranged reads of one file.

What this changes

  • open(path, "rb") returns a new read-only handle with its own position. The stored object is left alone, so an open r+b handle keeps its position too.
  • The handle is built from getvalue(), which shares the stored bytes object, so opening a reader copies nothing.
  • write, writelines and truncate on the handle raise io.UnsupportedOperation, which is what LocalFileSystem gives for a file opened rb. getbuffer() returns a read-only view.
  • Closing the handle really closes it. Nothing else uses it, and this releases its share of the bytes.
  • MemoryFile.size and cat_file use getvalue() instead of getbuffer(). getbuffer() makes a private copy of a buffer that is shared with a reader, so info(), ls() or cat_file() after an rb open would otherwise copy the whole file.
  • r+b, ab, a+b, the write modes and transactions are unchanged.

Behaviour changes

  • A write through an rb handle used to modify the stored file. It now raises.
  • A reader is a snapshot taken when it is opened. A later r+b or ab write is seen by readers opened after it, not by ones already open.
  • A closed read handle reports closed and raises if it is used again.
  • The first in-place write (r+b, ab) while a reader is open copies the file once, because the bytes are shared until then. Overwriting with wb is unaffected.
  • fsspec.fuse opens a read-write request (O_RDWR) as "rb" and writes through that handle. On a memory-backed mount this worked only because the handle was the stored object; it now raises, as it already does on other backends. I have left fuse.py alone and have not tested a real mount. I can map O_RDWR to an update mode here or in a follow-up if you would like that.

Measurements

CPython 3.13.15, Linux x86-64, against master at 2289743.

master this PR
Open and close an rb handle on a 256 MiB file 3.3 µs 4.8 µs
50 × whole-file cat_file of a 64 MiB file 363–400 ms under 1 ms
5000 × (read 16 bytes, then append 1 KiB) 37–44 ms 44–48 ms

The last row is about 100 ms for the first loop in a fresh process on this branch, and 44–48 ms after that.

Tests

17 new tests in test_memory.py. Two of them assert that nothing was copied by comparing object identity, which relies on CPython sharing the bytes.

@JonnyFaeth

Copy link
Copy Markdown

Additional platform/interpreter validation on d32e7b6: the full test_memory.py suite passes on native Windows with CPython 3.12.14: 178 passed. In particular, both CPython identity-based checks pass here: the reader/info/cat path retains the stored bytes object, and a closed-but-still-referenced reader does not force a copy on the subsequent update write. The snapshot, read-only buffer, independent-position and gzip cases pass as well.

This extends the behavioral checks beyond the reported Linux/3.13 environment; it is not a memory/RSS benchmark or performance claim, and I did not test FUSE. No source changes. Codex assisted with running and reviewing the tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants