Skip to content

Handle PathLike sources in the async bulk methods like the sync ones - #2477

Merged
martindurant merged 3 commits into
fsspec:masterfrom
prawalpokharel:fix/async-pathlike-sources
Nov 18, 2026
Merged

martindurant merged 3 commits into
fsspec:masterfrom
prawalpokharel:fix/async-pathlike-sources

Conversation

@prawalpokharel

Copy link
Copy Markdown
Contributor

#2310 made the synchronous copy, get and put run a single PathLike source through stringify_path before deciding whether it is one path or a list. The async versions were not changed, so AsyncFileSystem._copy, _get and _put still treat a pathlib.Path source as a list and flatten it into the target. _expand_path only accepts str as a single path, while the sync expand_path has taken os.PathLike since #1475.

With s3fs 2026.9.0 against moto, and a source directory containing a/file and b/file:

put(str source)              {'a/file': 'a', 'b/file': 'b'}
put(Path source)             {'file': 'b'}
get(PurePosixPath source)    raises FileNotFoundError: The specified key does not exist.
copy(PurePosixPath source)   raises TypeError: 'PurePosixPath' object is not iterable

put is the bad one: the source is a local path, the call succeeds, and files with the same name in different subdirectories overwrite each other in the bucket.

This adds the same stringify_path line to _copy, _get and _put, and makes _expand_path accept os.PathLike like the sync method. The _expand_path part is needed for mv, which copies and then calls rm: without it mv(Path, ..., recursive=True) copied the tree and then raised TypeError, leaving both copies. Lists of sources and explicit source/target pairs behave as before.

The tests are in test_async.py and use a small AsyncFileSystem subclass that only implements the primitives. 16 of the 33 new cases fail on master. With the change, test_async.py, test_spec.py, test_utils.py and the local, memory and reference tests pass on Python 3.13, and the s3fs test suite gives the same result as on master (462 passed). Nothing was run on Windows.

#2382 edits the lines just above these in the same three methods, so whichever is merged second will need a small rebase.

AsyncFileSystem._copy, _get and _put decided whether the source was a single
path with isinstance(source, str) before normalizing it, so a pathlib.Path or
other os.PathLike source was treated like a list of sources: _get and _put
flattened the directory tree, letting same-named files from different
subdirectories overwrite each other. _expand_path iterated over a PathLike
source, so _copy, _rm, _cat and mv raised TypeError.

Normalize the source with stringify_path in _copy, _get and _put, as the
synchronous methods have done since fsspec#2310, and accept os.PathLike in
_expand_path as the synchronous expand_path does.
@JonnyFaeth

Copy link
Copy Markdown

Checked the Windows gap noted in the description at c257522: all 33 existing PathLike cases pass on native Windows / Python 3.12.14. Replacing only fsspec/asyn.py with c257522^ gives 16 failures and 17 passes; restoring head returns 33 passes.

This includes real Windows Path sources/local file reads for put and local destination writes for get, including repeated basenames in different directories and existing/non-existing targets. The remote side remains your _TreeFS fixture; I did not run S3/moto, live cloud storage, or the full suite. The tests are unchanged and yours. Codex assisted with execution and reporting.

@prawalpokharel

Copy link
Copy Markdown
Contributor Author

@JonnyFaeth, Thanks for checking this on Windows, that was the one platform I couldn't run.

The 16 failing / 17 passing split against the parent asyn.py is the same one I got on Linux, so the tests are exercising the same code paths there. CI on this PR is still waiting for approval, so yours is the only Windows result so far.

@martindurant

Copy link
Copy Markdown
Member

The change is fine, but I suggest we need maybe 1 in 10 of those tests.

@prawalpokharel

Copy link
Copy Markdown
Contributor Author

Cut the tests down to four cases, one per method (copy, mv, get, put). I also merged master to fix the changelog conflict with #2478.

@martindurant
martindurant merged commit f906a50 into fsspec:master Nov 18, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants