Skip to main content

Management Server Metrics

Flexify.IO Management Server implements Prometheus exposer to export multiple metrics in addition to metrics that Azure native resources it runs on provide.

Prometheus

Configuring Exposer​

Prometheus exposer is disabled by default. To enable the exposer set the environment variable:

PROMETHEUS_ENABLED=true

This will enable /metrics path on the management server. By default it will be available without authentication.

For CE and Compose options where Management Server and Web Console are placed behind nginx, the path is /flexify-io/manage/api/metrics.

To enable basic authentication, set:

PROMETHEUS_USERNAME=prometheus
PROMETHEUS_PASSWORD=<your_password>

Exposed Metrics​

Engines​

Most metrics include labels that allow more granular monitoring per organization or engine configuration:

  • org - ID of organization the entity belongs to

  • config - ID of engine configuration of entity’s assigned engine or engine itself

  • state - state of the engine. Can be:

    • ALIVE_OPERATING

    • ALIVE_WARMING_UP

    • ALIVE_WINDING_DOWN

    • DEAD

    • DEPLOYING

    • NEW

    • UNDEPLOYING

Metric

Labels

Description

Observation

fl_mgmt_eng_count

  • org

  • config

  • state

Gauge. The number of engines per state excluding deleted and unregistered engines.

⚠️ Should be no engines in the DEAD state.

The number of engines in ALIVE_OPERATING state should be at least as much as expected. May temporary increase during upgrade.

fl_mgmt_eng_registration_total

none

Counter. Number of successful engine registrations

Too many registrations during a time interval may indicate unstable engines.

fl_mgmt_eng_hard_reboot_total

  • org

  • config

Counter. Number of engine hard reboots detected by registration without un-registration

⚠️ Should not increase.

fl_mgmt_eng_registration_failed_total

none

Counter. Number of failed engine registrations.

⚠️ Should not increase.

fl_mgmt_eng_die_total

  • org

  • config

Counter. Number of engine deaths (communication timeouts) detected

⚠️ Should not increase, but occasional events due to network connectivity issues are expected.

fl_mgmt_eng_tasks_request_process_duration_seconds

  • org

  • config

Histogram. Time to process engine request for the list of tasks.

Should not increase dramatically and generally stay withing seconds interval.

fl_mgmt_eng_tasks_request_failed_total

  • org

  • config

Counter. Failed engine task requests.

⚠️ Should not increase fast, occasional errors are normal.

fl_mgmt_eng_report_process_duration_seconds

  • org

  • config

Histogram. Time to process successful engine requests.

Should not increase dramatically and generally stay withing seconds interval.

fl_mgmt_eng_report_failed_total

  • org

  • config

Counter. Failed engine reports.

⚠️ Should not increase fast, occasional errors are normal.

fl_mgmt_eng_report_interval_seconds

  • org

  • config

Histogram. Interval between successful engines reports.

Should not increase dramatically and generally stay around configured report interval (5 seconds by default).

fl_mgmt_eng_reported_migration_tasks

  • org

  • config

Gauge. Number of migration tasks reported by all engines.

Should stay reasonably low.

fl_mgmt_eng_reported_cleanup_tasks

  • org

  • config

Gauge. Number of cleanup tasks reported by all engines.

Should stay reasonably low.

fl_mgmt_eng_reported_storage_info_tasks

  • org

  • config

Gauge. Number of storage info (refresh) tasks reported by all engines.

Should stay reasonably low.

fl_mgmt_eng_reported_forwarding_tasks

  • org

  • config

Gauge. Number of forwarding (proxy) tasks reported by all engines (including dead).

Should stay reasonably low.

fl_mgmt_eng_reported_forwarding_task_errors

  • org

  • config

Gauge. Number of errors in active forwarding tasks on all engines.

⚠️ Should be zero.

fl_mgmt_eng_proxy_client_connections

  • org

  • config

Gauge. Total number of connections on all engines.

Should not grow over time and stay below environment-specific threshold.

Endpoints​

  • org - organization ID this endpoint belongs to

  • state - state of the endpoint, one of ENABLED, DISABLED, or DELETING. Note that DELETED endpoints are not included in statistics.

  • config - engine configuration ID

Metric

Labels

Description

Observation

fl_mgmt_endpoints_count

  • org

  • state

Gauge. The number of endpoints configured.

The number of ENABLED endpoints should be what you’d expect.

fl_mgmt_endpoint_errors_total

  • org

  • config

Counter. The number of forwarding tasks errors reports by engines.

⚠️ Should not grow at all.

State update​

type - type of state update, one of migration, endpoint, storage, or storage_refresh.

Metric

Labels

Description

Observation

fl_mgmt_state_update_requests_total

  • type

The number of times state update is requested.

Should not grow at rate much above average.

fl_mgmt_state_update_entities_count

  • type

The number of entities selected for state update.

Should not be too high.

fl_mgmt_state_update_selection_duration_seconds

  • type

Time it took to select entities for state update.

Should not be too high.

fl_mgmt_state_update_duration_seconds

  • type

Time to finish state update, including time to select entities for state update

Should not be too high.

fl_mgmt_state_update_retried_entities_total

  • type

The number of entities retried in a state update.

Should not grow too fast.

fl_mgmt_state_update_failed_entities_total

  • type

The number of entities that failed state update.

⚠️ Should not grow at all.

Cardinality​

For most metrics with labels, metrics cardinality equals to the total number of all engine configurations in all organizations.

For `fl_mgmt_eng_count cardinality is 7 times larger then that.

Exemplars​

Most metrics have exemplars attached allowing to identify problematic entities for diagnostic purposes. It’s essential that exemplars are stored and accessible together with metrics values.

Azure Native Metrics

In addition to metrics exposed by the Management Server via Prometheus we also recommend monitoring native Azure metrics.

App Service Plan​

Azure Metric

Criteria

CPU Percentage

< 80%

Memory Percentage

< 90%. Note that normally most of consumed memory is discardable cache.

App Service​

Azure App Service that runs the Management Server (API) container, not App Service that runs the Web Console container.

Azure Metric

Criteria

Health check status

Should be 100 (healthy)

Requests

Should not be much above average value.

Should not drop to near-zero values.

Response Time

Should not be above acceptable threshold (usually < 500 ms).

Data In

Should not drop much below average value.

Data Out

Should not drop much below average value.

Http 2xx
Http 3xx

Should not drop to near-zero values.

Http 4xx

Should not be too high (occasional 4xx responses are normal)

Http Server Errors

Should not be too high (occasional 5xx responses are normal)

SQL Database​

Azure Metric

Criteria

Availability

100%

DTU percentage

< 50-80% depending on required response time.

If you update engine by scaling up VMSS to double the size, keep DTU below 50%.

Data space used percent

< 50%

Workers percentage

< 80%. Higher values may indicate management server configuration mismatch to the SQL Database SKU.