How Flexify.IO object storage virtualization works
This article explains how Flexify.IO can be used to convert protocols, migrate or distribute data between object storages.
Overview
Flexify.IO sits between your application and your object storages. Your application sends S3 requests to a Flexify.IO virtual endpoint, and Flexify.IO forwards them to one or more storages.
With a virtual endpoint, Flexify.IO can:
Translate storage APIs, for example S3 to Azure Blob Storage.
Combine data from several storages into a single virtual bucket.
Distribute new data between storages while your application keeps seeing one bucket.
Migrate data between storages with no downtime and no changes to your application.
Improve performance by reading each object from the storage that answers best
Flexify.IO does not store or modify your data. If you stop using Flexify.IO, point your application back at your storage directly. You lose Flexify.IO features, but your data stays where it is.
Single storage (pass-through)
In the simplest configuration, the endpoint has one S3-compatible storage attached. Flexify.IO relays every request to that storage.
This configuration adds no features, but it is a good way to test your application with Flexify.IO before moving to the scenarios below.
API translation
Instead of an S3-compatible storage, you can attach a storage that uses a different API, such as Microsoft Azure Blob Storage.
Flexify.IO converts S3 requests to Azure Blob requests on the fly. Your application keeps using S3, while the data is stored in Azure as blobs. S3 buckets map to Azure containers and S3 objects map to blobs.
This is useful when you move workloads from Amazon Web Services to Microsoft Azure, because you don't need to change application code.
Multiple storages
You can attach several storages to the same endpoint. Each storage keeps its own set of objects, but your application sees them combined in a single bucket.
Flexify.IO handles requests as follows:
Read | Sent to the attached storages. Which copy is returned depends on the endpoint's Selection policy (see below). |
List | Sent to the attached storages. The results are merged into one listing. |
Write | Sent to the storage(s) where Store data uploaded via Virtual Endpoint here is turned on. The storage bar marks them as (store). |
Delete | Applied to the storages where Delete data deleted via Virtual Endpoint from here is turned on. |
The Selection policy in the endpoint's Settings decides which copy a read returns when an object exists in more than one storage:
Newest: returns the object with the latest timestamp. This is the default.
Fastest: returns the object from the storage that responds first.
Priority: returns the object from the first storage that has it, in the order shown on the storage bar. Drag the storages to change the order.
You can change which storage receives new writes at any time. All new data then goes to the storage you choose, without any change to your application.
You can also move data between storages in the background. Your application still sees all storages combined while the data moves.
Is performance affected?
We paid special attention to ensuring no significant effect on performance. You can expect no more than a few milliseconds of added delay. For example, Flexify.IO streams data: it starts relaying an object while it is still receiving it, instead of waiting for the whole object to arrive.
In many cases, performance actually improves, for example when you use the Fastest selection policy or store new data in the fastest storage.
It's all about cloud
Flexify.IO is designed and tested for object storage. Most cloud storages, such as Amazon S3 and Azure Blob Storage, are object storages.
Data kept on a local disk or NAS is usually block or file storage, which doesn't fit the cloud well. When you move data to the cloud, switch to object storage.
You can still mount object storage as a local disk or file share. If you do, check how your application accesses the data, because this can hurt performance badly.
See Adding file or block storage to Flexify.IO with MinIO for a step-by-step guide on how to add legacy block or file storage to Flexify.IO.
Does Flexify.IO store data?
No. Flexify.IO relays your requests to one or more third-party storages, such as Amazon S3 or Microsoft Azure Blob Storage. Your data stays in your storage accounts.
Getting started
See How to distribute data across multiple cloud providers with Flexify.IO for the step-by-step Virtual Endpoint configuration guide.