Merge branch 'main' into dynamic=chunksize

Author: skshetry

.pre-commit-config.yaml

@@ -11,7 +11,7 @@ repos:
       - id: black
         exclude: ^docs/
   - repo: https://github.com/pycqa/flake8
-    rev: '4.0.1'
+    rev: '7.3.0'
     hooks:
       - id: flake8
         exclude: tests/|^docs/|__init__.py

README.md

@@ -0,0 +1,18 @@
+s3fs
+====
+
+[|Build Status|](https://github.com/fsspec/s3fs/actions)
+[|Documentation|](https://s3fs.readthedocs.io/en/latest/?badge=latest)
+
+S3FS builds on [aiobotocore](https://aiobotocore.readthedocs.io/en/latest/)
+to provide a convenient Python filesystem interface for S3.
+
+
+Support
+-------
+
+Work on this repository is supported in part by:
+
+"Anaconda, Inc. - Advancing AI through open source."
+
+<a href="https://anaconda.com/"><img src="https://camo.githubusercontent.com/b8555ef2222598ed37ce38ac86955febbd25de7619931bb7dd3c58432181d3b6/68747470733a2f2f626565776172652e6f72672f636f6d6d756e6974792f6d656d626572732f616e61636f6e64612f616e61636f6e64612d6c617267652e706e67" alt="anaconda logo" width="40%"/></a>

README.rst

@@ -1,6 +1,18 @@
 Changelog
 =========
 
+2025.9.0
+--------
+
+- update README for distribution compliance
+
+2025.7.0
+--------
+
+- fix exclusive write for small files (#974)
+- acknowledge Anaconda support (#972)
+- fix test typo (#970)
+
 2025.5.1
 --------
 

docs/source/changelog.rst

@@ -0,0 +1,126 @@
+Code of Conduct
+===============
+
+All participants in the fsspec community are expected to adhere to a Code of Conduct.
+
+As contributors and maintainers of this project, and in the interest of
+fostering an open and welcoming community, we pledge to respect all people who
+contribute through reporting issues, posting feature requests, updating
+documentation, submitting pull requests or patches, and other activities.
+
+We are committed to making participation in this project a harassment-free
+experience for everyone, treating everyone as unique humans deserving of
+respect.
+
+Examples of unacceptable behaviour by participants include:
+
+- The use of sexualized language or imagery
+- Personal attacks
+- Trolling or insulting/derogatory comments
+- Public or private harassment
+- Publishing other's private information, such as physical or electronic
+  addresses, without explicit permission
+- Other unethical or unprofessional conduct
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviours that they deem inappropriate,
+threatening, offensive, or harmful.
+
+By adopting this Code of Conduct, project maintainers commit themselves
+to fairly and consistently applying these principles to every aspect of
+managing this project. Project maintainers who do not follow or enforce
+the Code of Conduct may be permanently removed from the project team.
+
+This code of conduct applies both within project spaces and in public
+spaces when an individual is representing the project or its community.
+
+If you feel the code of conduct has been violated, please report the
+incident to the fsspec core team.
+
+Reporting
+---------
+
+If you believe someone is violating theCode of Conduct we ask that you report it
+to the  Project by emailing [email protected]. All reports will be kept
+confidential. In some cases we may determine that a public statement will need
+to be made. If that's the case, the identities of all victims and reporters
+will remain confidential unless those individuals instruct us otherwise.
+If you believe anyone is in physical danger, please notify appropriate law
+enforcement first.
+
+In your report please include:
+
+- Your contact info
+- Names (real, nicknames, or pseudonyms) of any individuals involved.
+  If there were other witnesses besides you, please try to include them as well.
+- When and where the incident occurred. Please be as specific as possible.
+- Your account of what occurred. If there is a publicly available record
+  please include a link.
+- Any extra context you believe existed for the incident.
+- If you believe this incident is ongoing.
+- If you believe any member of the core team has a conflict of interest
+  in adjudicating the incident.
+- What, if any, corrective response you believe would be appropriate.
+- Any other information you believe we should have.
+
+Core team members are obligated to maintain confidentiality with regard
+to the reporter and details of an incident.
+
+What happens next?
+~~~~~~~~~~~~~~~~~~
+
+You will receive an email acknowledging receipt of your complaint.
+The core team will immediately meet to review the incident and determine:
+
+- What happened.
+- Whether this event constitutes a code of conduct violation.
+- Who the bad actor was.
+- Whether this is an ongoing situation, or if there is a threat to anyone's
+  physical safety.
+- If this is determined to be an ongoing incident or a threat to physical safety,
+  the working groups' immediate priority will be to protect everyone involved.
+
+If a member of the core team is one of the named parties, they will not be
+included in any discussions, and will not be provided with any confidential
+details from the reporter.
+
+If anyone on the core team believes they have a conflict of interest in
+adjudicating on a reported issue, they will inform the other core team
+members, and exempt themselves from any discussion about the issue.
+Following this declaration, they will not be provided with any confidential
+details from the reporter.
+
+Once the working group has a complete account of the events they will make a
+decision as to how to response. Responses may include:
+
+- Nothing (if we determine no violation occurred).
+- A private reprimand from the working group to the individual(s) involved.
+- A public reprimand.
+- An imposed vacation
+- A permanent or temporary ban from some or all spaces (GitHub repositories, etc.)
+- A request for a public or private apology.
+
+We'll respond within one week to the person who filed the report with either a
+resolution or an explanation of why the situation is not yet resolved.
+
+Once we've determined our final action, we'll contact the original reporter
+to let them know what action (if any) we'll be taking. We'll take into account
+feedback from the reporter on the appropriateness of our response, but we
+don't guarantee we'll act on it.
+
+Acknowledgement
+---------------
+
+This CoC is modified from the one by `BeeWare`_, which in turn refers to
+the `Contributor Covenant`_ and the `Django`_ project.
+
+.. _BeeWare: https://beeware.org/community/behavior/code-of-conduct/
+.. _Contributor Covenant: https://www.contributor-covenant.org/version/1/3/0/code-of-conduct/
+.. _Django: https://www.djangoproject.com/conduct/reporting/
+
+.. raw:: html
+
+    <script data-goatcounter="https://projspec.goatcounter.com/count"
+        async src="//gc.zgo.at/count.js"></script>

docs/source/code-of-conduct.rst

@@ -340,6 +340,7 @@ Contents
    development
    api
    changelog
+   code-of-conduct
    :maxdepth: 2
 
 
@@ -351,3 +352,12 @@ Indices and tables
 * :ref:`genindex`
 * :ref:`modindex`
 * :ref:`search`
+
+
+These docs pages collect anonymous tracking data using goatcounter, and the
+dashboard is available to the public: https://s3fs.goatcounter.com/ .
+
+.. raw:: html
+
+    <script data-goatcounter="https://s3fs.goatcounter.com/count"
+        async src="//gc.zgo.at/count.js"></script>

docs/source/index.rst

@@ -1,3 +1,3 @@
 aiobotocore>=2.5.4,<3.0.0
-fsspec==2025.5.1
+fsspec==2025.9.0
 aiohttp!=4.0.0a0, !=4.0.0a1

requirements.txt

@@ -1478,6 +1478,18 @@ async def _info(self, path, bucket=None, key=None, refresh=False, version_id=Non
                 pass
             except ClientError as e:
                 raise translate_boto_error(e, set_cause=False)
+        else:
+            try:
+                out = await self._call_s3("head_bucket", Bucket=bucket, **self.req_kw)
+                return {
+                    "name": bucket,
+                    "type": "directory",
+                    "size": 0,
+                    "StorageClass": "DIRECTORY",
+                    "VersionId": out.get("VersionId"),
+                }
+            except ClientError as e:
+                raise translate_boto_error(e, set_cause=False)
 
         try:
             # We check to see if the path is a directory by attempting to list its
@@ -2472,6 +2484,7 @@ def n_bytes_left() -> int:
 
     def commit(self):
         logger.debug("Commit %s" % self)
+        match = {"IfNoneMatch": "*"} if "x" in self.mode else {}
         if self.tell() == 0:
             if self.buffer is not None:
                 logger.debug("Empty file committed %s" % self)
@@ -2485,15 +2498,11 @@ def commit(self):
                 kw = dict(Key=self.key, Bucket=self.bucket, Body=data, **self.kwargs)
                 if self.acl:
                     kw["ACL"] = self.acl
-                write_result = self._call_s3("put_object", **kw)
+                write_result = self._call_s3("put_object", **kw, **match)
             else:
                 raise RuntimeError
         else:
             logger.debug("Complete multi-part upload for %s " % self)
-            if "x" in self.mode:
-                match = {"IfNoneMatch": "*"}
-            else:
-                match = {}
             part_info = {"Parts": self.parts}
             write_result = self._call_s3(
                 "complete_multipart_upload",

s3fs/core.py

@@ -33,7 +33,6 @@ class TestS3fsPipe(abstract.AbstractPipeTests, S3fsFixtures):
 
 
 class TestS3fsOpen(abstract.AbstractOpenTests, S3fsFixtures):
-
     test_open_exclusive = pytest.mark.xfail(
         reason="complete_multipart_upload doesn't implement condition in moto"
     )(abstract.AbstractOpenTests.test_open_exclusive)

s3fs/tests/derived/s3fs_test.py

@@ -18,6 +18,7 @@
 import fsspec.core
 from dateutil.tz import tzutc
 
+import botocore
 import s3fs.core
 from s3fs.core import S3FileSystem
 from s3fs.utils import ignoring, SSEParams
@@ -2888,7 +2889,14 @@ def test_exist_after_delete(s3):
     assert not s3.exists(test_dir)
 
 
-@pytest.mark.xfail(reason="moto doesn't support conditional MPU")
+# condition: True if running on botocore < 1.36.0
+# The below tests for exclusive writes will fail on older versions of botocore.
+old_botocore = version.parse(botocore.__version__) < version.parse("1.36.0")
+
+
+@pytest.mark.xfail(
+    reason="moto doesn't support IfNoneMatch for MPU when object created via MPU"
+)
 def test_pipe_exclusive_big(s3):
     chunksize = 5 * 2**20  # minimum allowed
     data = b"x" * chunksize * 3
@@ -2899,17 +2907,89 @@ def test_pipe_exclusive_big(s3):
     assert not s3.list_multipart_uploads(test_bucket_name)
 
 
-@pytest.mark.xfail(reason="moto doesn't support conditional MPU")
-def test_put_exclusive_big(s3, tempdir):
+@pytest.mark.xfail(
+    old_botocore, reason="botocore<1.33.0 lacks IfNoneMatch support", strict=True
+)
+def test_pipe_exclusive_big_after_small(s3):
+    """Test conditional MPU after creating object via put_object
+
+    This test is required because moto's implementation of IfNoneMatch for MPU
+    only works when the object is initially created via put_object and not via
+    MPU.
+    """
     chunksize = 5 * 2**20  # minimum allowed
-    data = b"x" * chunksize * 3
-    fn = f"{tempdir}/afile"
-    with open(fn, "wb") as f:
-        f.write(fn)
-    s3.put(fn, f"{test_bucket_name}/afile", data, mode="overwrite", chunksize=chunksize)
-    s3.put(fn, f"{test_bucket_name}/afile", data, mode="overwrite", chunksize=chunksize)
+
+    # First, create object via put_object (small upload)
+    s3.pipe(f"{test_bucket_name}/afile", b"small", mode="overwrite")
+
+    # Now try multipart upload with mode="create" (should fail)
     with pytest.raises(FileExistsError):
-        s3.put(
-            fn, f"{test_bucket_name}/afile", data, mode="create", chunksize=chunksize
+        s3.pipe(
+            f"{test_bucket_name}/afile",
+            b"c" * chunksize * 3,
+            mode="create",
+            chunksize=chunksize,
         )
+
     assert not s3.list_multipart_uploads(test_bucket_name)
+
+
+@pytest.mark.xfail(
+    reason="moto doesn't support IfNoneMatch for MPU when object created via MPU"
+)
+def test_put_exclusive_big(s3, tmpdir):
+    chunksize = 5 * 2**20  # minimum allowed
+    fn = f"{tmpdir}/afile"
+    with open(fn, "wb") as f:
+        f.write(b"x" * chunksize * 3)
+    s3.put(fn, f"{test_bucket_name}/afile", mode="overwrite", chunksize=chunksize)
+    s3.put(fn, f"{test_bucket_name}/afile", mode="overwrite", chunksize=chunksize)
+    with pytest.raises(FileExistsError):
+        s3.put(fn, f"{test_bucket_name}/afile", mode="create", chunksize=chunksize)
+    assert not s3.list_multipart_uploads(test_bucket_name)
+
+
+@pytest.mark.xfail(
+    old_botocore, reason="botocore<1.33.0 lacks IfNoneMatch support", strict=True
+)
+def test_put_exclusive_big_after_small(s3, tmpdir):
+    """Test conditional MPU after creating object via put_object.
+
+    This test is required because moto's implementation of IfNoneMatch for MPU
+    only works when the object is initially created via put_object and not via
+    MPU.
+    """
+    chunksize = 5 * 2**20  # minimum allowed
+    fn = str(tmpdir.join("afile"))
+    with open(fn, "wb") as f:
+        f.write(b"x" * chunksize * 3)
+
+    # First, create object via put_object (small upload)
+    s3.pipe(f"{test_bucket_name}/afile", b"small", mode="overwrite")
+
+    # Now try multipart upload with mode="create" (should fail)
+    with pytest.raises(FileExistsError):
+        s3.put(fn, f"{test_bucket_name}/afile", mode="create", chunksize=chunksize)
+
+    assert not s3.list_multipart_uploads(test_bucket_name)
+
+
+@pytest.mark.xfail(
+    old_botocore, reason="botocore<1.33.0 lacks IfNoneMatch support", strict=True
+)
+def test_put_exclusive_small(s3, tmpdir):
+    fn = f"{tmpdir}/afile"
+    with open(fn, "wb") as f:
+        f.write(b"x")
+    s3.put(fn, f"{test_bucket_name}/afile", mode="overwrite")
+    s3.put(fn, f"{test_bucket_name}/afile", mode="overwrite")
+    with pytest.raises(FileExistsError):
+        s3.put(fn, f"{test_bucket_name}/afile", mode="create")
+    assert not s3.list_multipart_uploads(test_bucket_name)
+
+
+def test_bucket_info(s3):
+    info = s3.info(test_bucket_name)
+    assert "VersionId" in info
+    assert info["type"] == "directory"
+    assert info["name"] == test_bucket_name