Configure s3fs (Python library) to work with Flexify.IO
Using s3fs with Azure Blob Storage
By default, s3fs uses maximal multipart limits allowed by AWS S3 for copy operation within cloud. Azure has lower limits, so additional adjustments are needed:
s3fs.core.MANAGED_COPY_THRESHOLDvariable should be set to 5000 MiB (or less)each call of
s3fs.S3FileSystem.copy()operation should explicitly include non-default block size value 4000 MiB (or less)set connection timeouts for higher values, like 1500 seconds
s3fs code example
import s3fs
source_path = "data/5001MiB.bin"
destination_path = "data/5001MiB_2.bin"
s3fs.core.MANAGED_COPY_THRESHOLD = 5000 * 2**20
def main():
cargs = {
"endpoint_url": "http://<proxy_ip>",
"aws_access_key_id": "<my_key_id>",
"aws_secret_access_key": "<my_key_id>",
"verify": True
}
s3 = s3fs.S3FileSystem(anon=False, client_kwargs=cargs,
config_kwargs={"connect_timeout": 1500, "read_timeout": 1500})
s3.copy(source_path, destination_path, block=4000 * 2**20)
if __name__ == "__main__":
main()