Adding File or Block Storage to Flexify.IO with MinIO

Flexify.IO works exclusively with object storage. However, for scenarios such as on-premises-to-cloud migration, you may want to add a legacy directory or a file share to Flexify.IO. While Flexify.IO cannot access data from your file system or a network share directly, it can do so with the help of MinIO.

Solution architecture
Solution architecture

Configuring and Starting MinIO

First, start MinIO Server on a physical or virtual machine where your file or block storage is mounted. The storage can be mounted as a volume or a network share.

We recommend at least 4 vCPU and 16 GB RAM to run MinIO. Otherwise, you may not be able to achieve the maximum migration throughput.

  • Download MinIO, if not already installed.
Linux
  • Create a temporary directory that will be used as MinIO data "root".
Linux

Use temporary directory as MinIO data root

MinIO Server allows exposing a file system directory via S3-compatible API. When doing so, the directories just inside Minio's data "root" directory become buckets. It imposes certain limitations on their names and may cause issues. That's why it's not recommended to point the MinIO server to an existing data directory of your file system.

  • Start the MinIO server pointing to this directory.
Linux

Start MinIO before mapping data

Start MinIO before creating any mount points in the data directory. Otherwise MinIO would fail on the startup check.

Secure Password

Make sure to choose a secure password.

Open port 9000

MinIO uses port 9000 by default. You may need to open port 9000 on a firewall to enable outside connections to the MinIO Server on your machine.

  • Create a directory that will represent an S3 bucket and mount your data path to it. In most cases, you'd like all your data to be in a single bucket. The data path can be an entire volume (such as /mnt/volume_ams3_01) or any directory on your local or network file system.
Linux

Read-only Access

Using the -r option of the mount command would protect your data from accidental deletion and is recommended for the on-prem-to-cloud migration scenario. Make sure to umount the data directory before your delete minio-root.

Now the content of the /path/to/data directory is accessible via the S3 API and can be found in the mybucket bucket. You can see it by opening http://<your-server-ip>:9000/minio in your browser or by configuring any S3 client to use http://<your-server-ip>:9000 endpoint with minio/MySecurePassword as the S3 keys.

Adding MinIO to Flexify.IO

From Flexify.IO's standpoint, MinIO is no different from any other S3-compatible storage. To add MinIO to Flexify.IO, simply click Add Storage Account on the Data tab, select Minio from the drop-down list, and specify your MinIO server's IP:port and S3 keys.

Adding MinIO to Flexify.IO
Adding MinIO to Flexify.IO

SSL/TLS

By default MinIO uses HTTP. Make sure to remove the "Encrypt transfer with SSL/TLS" checkbox or configure SSL certificates on the MinIO server.

MinIO added to Flexify.IO as a storage account
MinIO added to Flexify.IO as a storage account

Now your file/disk storage is added to Flexify.IO, and you can migrate data to Amazon S3 or another cloud storage, or even attach it to the Flexify.IO multi-cloud virtual endpoint.

Performance Considerations

Disk storage might be slow. Especially in the case of rotational media, such as HDD-based RAID or Google Cloud Storage's standard persistent disks, storage systems may also suffer from high latency. Response times might as high as 20 ms, limiting sequential I/O to around 50 IOPS. But most storage systems allow many requests to be served in parallel, increasing the cumulative IOPS to thousands.

Unfortunately, MinIO is not taking advantage of parallelism when listing objects (each object requires a separate stat system call), and on a high latency rotational media would not be able to list more than ~50 objects/second. This could limit the total throughput of the migration in case your objects are relatively small in size.

IOPS and wait time for rotational (sdb) vs. solid state (sbc) media when listing objects via MinIO
IOPS and wait time for rotational (sdb) vs. solid state (sbc) media when listing objects via MinIO

There are two alternative solutions to this issue:

  • Convert the disk to a low-latency type, such as Standard to Balanced in Google Cloud Platform.
  • Use the special MinIO build prepared by Flexify.IO that avoids those unnecessary sequential stat calls.