Adding File or Block Storage to Flexify.IO with MinIO
Flexify.IO works exclusively with object storage. However, for scenarios such as on-premises-to-cloud migration, you may want to add a legacy directory or a file share to Flexify.IO. While Flexify.IO cannot access data from your file system or a network share directly, it can do so with the help of MinIO.
Configuring and Starting MinIO
First, start MinIO Server on a physical or virtual machine where your file or block storage is mounted. The storage can be mounted as a volume or a network share.
We recommend at least 4 vCPU and 16 GB RAM to run MinIO. Otherwise, you may not be able to achieve the maximum migration throughput.
- Download MinIO, if not already installed.
- Create a temporary directory that will be used as MinIO data "root".
Use temporary directory as MinIO data root
MinIO Server allows exposing a file system directory via S3-compatible API. When doing so, the directories just inside Minio's data "root" directory become buckets. It imposes certain limitations on their names and may cause issues. That's why it's not recommended to point the MinIO server to an existing data directory of your file system.
- Start the MinIO server pointing to this directory.
Start MinIO before mapping data
Start MinIO before creating any mount points in the data directory. Otherwise MinIO would fail on the startup check.
Secure Password
Make sure to choose a secure password.
Open port 9000
MinIO uses port 9000 by default. You may need to open port 9000 on a firewall to enable outside connections to the MinIO Server on your machine.
- Create a directory that will represent an S3 bucket and mount your data path to it. In most cases, you'd like all your data to be in a single bucket. The data path can be an entire volume (such as /mnt/volume_ams3_01) or any directory on your local or network file system.
Read-only Access
Using the -r option of the mount command would protect your data from accidental deletion and is recommended for the on-prem-to-cloud migration scenario. Make sure to umount the data directory before your delete minio-root.
Now the content of the /path/to/data directory is accessible via the S3 API and can be found in the mybucket bucket. You can see it by opening http://<your-server-ip>:9000/minio in your browser or by configuring any S3 client to use http://<your-server-ip>:9000 endpoint with minio/MySecurePassword as the S3 keys.
Adding MinIO to Flexify.IO
From Flexify.IO's standpoint, MinIO is no different from any other S3-compatible storage. To add MinIO to Flexify.IO, simply click Add Storage Account on the Data tab, select Minio from the drop-down list, and specify your MinIO server's IP:port and S3 keys.
SSL/TLS
By default MinIO uses HTTP. Make sure to remove the "Encrypt transfer with SSL/TLS" checkbox or configure SSL certificates on the MinIO server.
Now your file/disk storage is added to Flexify.IO, and you can migrate data to Amazon S3 or another cloud storage, or even attach it to the Flexify.IO multi-cloud virtual endpoint.
Performance Considerations
Disk storage might be slow. Especially in the case of rotational media, such as HDD-based RAID or Google Cloud Storage's standard persistent disks, storage systems may also suffer from high latency. Response times might as high as 20 ms, limiting sequential I/O to around 50 IOPS. But most storage systems allow many requests to be served in parallel, increasing the cumulative IOPS to thousands.
Unfortunately, MinIO is not taking advantage of parallelism when listing objects (each object requires a separate stat system call), and on a high latency rotational media would not be able to list more than ~50 objects/second. This could limit the total throughput of the migration in case your objects are relatively small in size.
There are two alternative solutions to this issue:
- Convert the disk to a low-latency type, such as Standard to Balanced in Google Cloud Platform.
- Use the special MinIO build prepared by Flexify.IO that avoids those unnecessary sequential stat calls.