Repository navigation
Handle PathLike sources in the async bulk methods like the sync ones - #2477
Conversation
AsyncFileSystem._copy, _get and _put decided whether the source was a single path with isinstance(source, str) before normalizing it, so a pathlib.Path or other os.PathLike source was treated like a list of sources: _get and _put flattened the directory tree, letting same-named files from different subdirectories overwrite each other. _expand_path iterated over a PathLike source, so _copy, _rm, _cat and mv raised TypeError. Normalize the source with stringify_path in _copy, _get and _put, as the synchronous methods have done since fsspec#2310, and accept os.PathLike in _expand_path as the synchronous expand_path does.
|
Checked the Windows gap noted in the description at This includes real Windows |
|
@JonnyFaeth, Thanks for checking this on Windows, that was the one platform I couldn't run. The 16 failing / 17 passing split against the parent asyn.py is the same one I got on Linux, so the tests are exercising the same code paths there. CI on this PR is still waiting for approval, so yours is the only Windows result so far. |
|
The change is fine, but I suggest we need maybe 1 in 10 of those tests. |
…e-sources # Conflicts: # docs/source/changelog.rst
|
Cut the tests down to four cases, one per method ( |
#2310 made the synchronous
copy,getandputrun a single PathLike source throughstringify_pathbefore deciding whether it is one path or a list. The async versions were not changed, soAsyncFileSystem._copy,_getand_putstill treat apathlib.Pathsource as a list and flatten it into the target._expand_pathonly acceptsstras a single path, while the syncexpand_pathhas takenos.PathLikesince #1475.With s3fs 2026.9.0 against moto, and a source directory containing
a/fileandb/file:putis the bad one: the source is a local path, the call succeeds, and files with the same name in different subdirectories overwrite each other in the bucket.This adds the same
stringify_pathline to_copy,_getand_put, and makes_expand_pathacceptos.PathLikelike the sync method. The_expand_pathpart is needed formv, which copies and then callsrm: without itmv(Path, ..., recursive=True)copied the tree and then raisedTypeError, leaving both copies. Lists of sources and explicit source/target pairs behave as before.The tests are in
test_async.pyand use a smallAsyncFileSystemsubclass that only implements the primitives. 16 of the 33 new cases fail on master. With the change,test_async.py,test_spec.py,test_utils.pyand the local, memory and reference tests pass on Python 3.13, and the s3fs test suite gives the same result as on master (462 passed). Nothing was run on Windows.#2382 edits the lines just above these in the same three methods, so whichever is merged second will need a small rebase.