Skip to main content

Configure s3fs (Python library) to work with Flexify.IO

Using s3fs with Azure Blob Storage​

By default, s3fs uses maximal multipart limits allowed by AWS S3 for copy operation within cloud. Azure has lower limits, so additional adjustments are needed:

  • s3fs.core.MANAGED_COPY_THRESHOLD variable should be set to 5000 MiB (or less)

  • each call of s3fs.S3FileSystem.copy() operation should explicitly include non-default block size value 4000 MiB (or less)

  • set connection timeouts for higher values, like 1500 seconds

s3fs code example​

import s3fs

source_path = "data/5001MiB.bin"
destination_path = "data/5001MiB_2.bin"
s3fs.core.MANAGED_COPY_THRESHOLD = 5000 * 2**20

def main():
cargs = {
"endpoint_url": "http://<proxy_ip>",
"aws_access_key_id": "<my_key_id>",
"aws_secret_access_key": "<my_key_id>",
"verify": True
}
s3 = s3fs.S3FileSystem(anon=False, client_kwargs=cargs,
config_kwargs={"connect_timeout": 1500, "read_timeout": 1500})
s3.copy(source_path, destination_path, block=4000 * 2**20)

if __name__ == "__main__":
main()