Skip to main content

Migrating objects metadata

Object metadata consists of additional properties that are also migrated with the object’s data. Header values like Content-Type, Content-Encoding, etc., and custom metadata such as x-amz-meta-headers are migrated along with objects.

In some cases, metadata must be dropped because the destination clouds do not support it.

Limitation Examples​

  • B2 only allows a maximum of 10 headers. Other headers are dropped.

  • Content-Disposition header value can be any in Amazon S3 but is checked for RFC compliance in B2.

  • Some providers may silently ignore system metadata headers.

Flexify.IO drops metadata in the following order:

  1. Flexify-specific metadata

  2. Customer’s custom metadata

  3. “System” metadata (Content-Type, etc.)

A warning is saved to the migration log when metadata is dropped by Flexify.IO.

Flexify.IO headers​

Flexify.IO may add metadata headers during the migration as illustrated in the table below:

fl-original-last-modified

The last modification date of the source object.

fl-original-md5

MD5 hash of the source object.

fl-original-etag

ETag of the source object (in cloud’s original format).

fl-original-dbx-hash

Dropbox hash of the source objects.

Those headers are only written if their value cannot be stored natively (for example, B2 allows to specify modification date, Azure allows to specify MD5 hash for multi-part uploads, ETags is the same as md5, etc.), and the values cannot be known otherwise (for example, if the cloud would calculate and show MD5 for the object, we will not write fl-original-md5).

Flexify.IO uses header values for comparing last modified dates, hashes, or ETags, enabling incremental migration to find unmodified or identical objects and skip them.

The values are “inherited”. If object has fl-original-last-modified value and it is migrated to another cloud, the value of the header in the destination will be the same.

Encoding​

Cloud storage providers differ in which characters they allow in object keys, bucket names and custom metadata names. During migration, Flexify.IO replaces unsupported characters with sequences the destination accepts. Applications that read through Flexify.IO see the original names. Applications that read directly from the destination see the encoded names.

Object keys​

In some cases, Flexify.IO must encode characters in object keys.

Cloud provider

Encoded characters

Encoding Format

Azure Blob Storage

  • 0x1f

  • 0x7f

  • From 0x81 to 0x9f

?u1234

Backblaze B2 (Native API)

  • <32

  • 127 (DEL)

?u1234

Bucket names​

Some clouds don’t support dots . in bucket names. For such providers, we replace dots with -fldot-. The replacement string can be configured in the storage account settings.

Metadata names​

In some cases, Flexify.IO must encode custom metadata names. Azure Blob requires metadata names to follow C# naming convention which does not support special characters like dash - in custom metadata names, so they encode dash as _ when migrating.

Metadata character

Azure Blob substitute

-

___

+

__plus__

!

__excl__

#

__hash__

$

__dollar__

'

__apo__

*

__star__

.

__dot__

^

__caret__

`

__acc__

|

__bar__

~

__tilde__

__

__und__

If first symbol of metadata name is digit, it will be encoded with __<digit>__ .

Limitations and known issues​

Because the fl-original-etag value is not carried over, if a user migrates from A to B, and then again from B to C, we cannot assure correct incremental migration directly from A to C.