Migrating objects metadata
Object metadata consists of additional properties that are also migrated with the object’s data. Header values like Content-Type, Content-Encoding, etc., and custom metadata such as x-amz-meta-headers are migrated along with objects.
In some cases, metadata must be dropped because the destination clouds do not support it.
Limitation Examples
B2 only allows a maximum of 10 headers. Other headers are dropped.
Content-Dispositionheader value can be any in Amazon S3 but is checked for RFC compliance in B2.Some providers may silently ignore system metadata headers.
Flexify.IO drops metadata in the following order:
Flexify-specific metadata
Customer’s custom metadata
“System” metadata (Content-Type, etc.)
A warning is saved to the migration log when metadata is dropped by Flexify.IO.
Flexify.IO headers
Flexify.IO may add metadata headers during the migration as illustrated in the table below:
| The last modification date of the source object. |
| MD5 hash of the source object. |
| ETag of the source object (in cloud’s original format). |
| Dropbox hash of the source objects. |
Those headers are only written if their value cannot be stored natively (for example, B2 allows to specify modification date, Azure allows to specify MD5 hash for multi-part uploads, ETags is the same as md5, etc.), and the values cannot be known otherwise (for example, if the cloud would calculate and show MD5 for the object, we will not write fl-original-md5).
Flexify.IO uses header values for comparing last modified dates, hashes, or ETags, enabling incremental migration to find unmodified or identical objects and skip them.
The values are “inherited”. If object has fl-original-last-modified value and it is migrated to another cloud, the value of the header in the destination will be the same.
Encoding
Cloud storage providers differ in which characters they allow in object keys, bucket names and custom metadata names. During migration, Flexify.IO replaces unsupported characters with sequences the destination accepts. Applications that read through Flexify.IO see the original names. Applications that read directly from the destination see the encoded names.
Object keys
In some cases, Flexify.IO must encode characters in object keys.
Cloud provider | Encoded characters | Encoding Format |
|---|---|---|
Azure Blob Storage |
|
|
Backblaze B2 (Native API) |
|
|
Bucket names
Some clouds don’t support dots . in bucket names. For such providers, we replace dots with -fldot-. The replacement string can be configured in the storage account settings.
Metadata names
In some cases, Flexify.IO must encode custom metadata names. Azure Blob requires metadata names to follow C# naming convention which does not support special characters like dash - in custom metadata names, so they encode dash as _ when migrating.
Metadata character | Azure Blob substitute |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
If first symbol of metadata name is digit, it will be encoded with __<digit>__ .
Limitations and known issues
Because the fl-original-etag value is not carried over, if a user migrates from A to B, and then again from B to C, we cannot assure correct incremental migration directly from A to C.