Multi-cloud storage with Yandex.Cloud and Amazon S3

multi cloud architecture is the new must have flexify io allows transparently distributing data between yandex cloud object storage and amazon s3 the following scenarios are supported storing a portion of the data in yandex cloud, while the rest of the data is stored in amazon s3 cashing/offloading a portion of amazon s3 data to yandex cloud replicating data between yandex cloud and amazon s3 migrating data between amazon s3 and yandex cloud in all scenarios, both yandex cloud and amazon web service virtual machines, as well as other services, have access to all the data, transparently and in a single namespace deploying flexify io for yandex cloud customers, we recommend deploying flexify io from yandex cloud marketplace in the yandex cloud console choose create vm in the compute cloud section in the image/boot disk selection select cloud marketplace and click show more search for flexify and select the latest version of flexify io as the base image specify vm cpu and ram depending on your projected usage (we recommend at least 2 cpu and 4 gb ram), login, your public ssh key, and click create vm once the vm is up and running, find vm's public ip address and open it in a browser it may take a few minutes for the vm to start please, be patient and wait for the create account dialog to apperar if you choose to use https to connect to the vm, note that the initial ssl certificate will be self signed and not automatically accepted by most browsers create an account with any username or email address and password you will later use to sign in to this flexify io installation adding storage accounts to flexify io once signed in to the flexify io console, you can add one or more cloud storage accounts to add a storage account, switch to the data tab and click add storage the add storage account dialog will open in this dialog, you can add a storage account from a supported cloud or on premises object storage, including yandex cloud, amazon s3, azure blob storage, dell emc ecs, and others adding yandex cloud to add yandex cloud object storage, you would first need to generate yandex cloud access keys in the yandex cloud console select a cloud and choose service accounts from the left menu create a new service account with the storage admin role select the newly created account, click create new key , and then create static access key copy the key id and the secret key to the clipboard in the flexify io console, choose yandex cloud from the storage provider drop down list and paste the access keys click add storage this will add yandex cloud storage account to flexify io and start refreshing buckets to display statistics adding amazon s3 to add an amazon s3 storage account to flexify io, you would need to have amazon s3 keys with permission to access amazon s3 storage in the amazon web services console, choose iam (available at https //console aws amazon com/iam/home https //console aws amazon com/iam/home ) on the users tab click add user specify the user name and choose to grant programmatic access assign the amazons3fullaccess policy to the newly created account while it is the easiest way, flexify io does not need the full amazon s3 access rights for the example of iam policy compatible with flexify io, click policy example in the add storage account dialog once the user is created, copy paste the access key id and the secret access key in the flexify io console, choose yandex cloud from the storage provider drop down list and paste newly generated access keys click add storage to add the storage account migrating data from amazon s3 to yandex cloud once you have storage accounts added to flexify io, you can copy or move data between them on the data tab click transfer data (or click new migration on the migrations tab) in the from field select one or more buckets to migrate data from in the to field choose yandex cloud object storage you can select any existing bucket, or let flexify io create a new bucket about bucket names you can type any name of the new bucket however, make sure that the bucket name is globally unique even if someone else already has a bucket with the same name, you will be to be able to use it optionally , click advanced settings and fine tune the migration setting click start migration and monitor the migration progress combining data from amazon s3 and yandex cloud the unique feature of flexify io is the ability to combine data from two or more clouds into single virtual storage and make it accessible through a unified s3 compatible endpoint to configure the flexify io virtual endpoint in the flexify io console switch to the endpoint tab if no endpoint created yet, click new endpoint click the (+) sign to attach yandex cloud object storage to the virtual endpoint click (+) again to attach amazon s3 storage to the virtual endpoint now all objects from yandex cloud and amazon s3 are combined and available via the s3 compatible endpoint on your flexify io virtual machine you would need to direct your application to use your machine's ip address (such as 178 154 254 21 in this example) with the access key and the secret key displayed in the endpoint settings flexify io will take s3 requests and forward them to all attached storages, combining the results for example, if you have plane jpg and ship jpg in your amazon s3 account, and ship jpg and train jpg in your yandex cloud account, the combined view via the flexify io virtual endpoint will have all three objects plane jpg , ship jpg , and train jpg read policy imagine that in the example above amazon s3 and yandex cloud has different versions of ship jpg what version will the client get if requesting ship jpg via the virtual endpoint? this is controlled by the read policy in the endpoint settings by default, with the newest policy, flexify io will compare timestamps of objects in amazon s3 and yandex cloud and choose the latest version this works best for the migration scenario where you write new data to yandex cloud only, but need data yet in amazon s3 to be readily available with the fastest policy, flexify io will deliver the version of the object from the cloud that responds first this works best for the scenario when you have replicas of identical objects in both clouds and would like to optimize for the best performance write policy by default, flexify io will replicate all data written via the virtual endpoint between all attached storages both yandex cloud and amazon s3 in this case you may want new data to be saved to yandex cloud only, while having data still in amazon s3 visible and accessible to achieve this, you can disable writing data to amazon s3 by clicking the amazon s3 account on the storage bar and removing the put new data here checkbox now new data will be saved to yandex cloud object storage only virtual buckets bucket names need to be unique and you may not be able to create a bucket with the name you need in one or more clouds flexify io allows mapping bucket names with the help of virtual buckets to create a new virtual bucket, just click new virtual bucket on the endpoint card and specify the virtual bucket name then you can attach multiple buckets to the virtual bucket, and make all objects from the attached buckets available via the virtual bucket with virtual buckets, you can also combine data from multiple buckets of the same cloud provider scaling out the flexify io virtual machine image from yandex cloud marketplace is a powerful solution for multi cloud storage and migrating data between clouds but it's limited to a single virtual machine this limits options for horizontal scalability and fault tolerance if you'd like to deploy flexify io in the true cloud native style with several horizontally scalable, stateless, and centrally controlled flexify io engines, please, contact us for installation instructions