Management Server Metrics
Flexify.IO Management Server implements Prometheus exposer to export multiple metrics in addition to metrics that Azure native resources it runs on provide.
Prometheus
Configuring Exposer
Prometheus exposer is disabled by default. To enable the exposer set the environment variable:
PROMETHEUS_ENABLED=true
This will enable /metrics path on the management server. By default it will be available without authentication.
For CE and Compose options where Management Server and Web Console are placed behind nginx, the path is /flexify-io/manage/api/metrics.
To enable basic authentication, set:
PROMETHEUS_USERNAME=prometheus
PROMETHEUS_PASSWORD=<your_password>
Exposed Metrics
Engines
Most metrics include labels that allow more granular monitoring per organization or engine configuration:
org- ID of organization the entity belongs toconfig- ID of engine configuration of entity’s assigned engine or engine itselfstate- state of the engine. Can be:ALIVE_OPERATINGALIVE_WARMING_UPALIVE_WINDING_DOWNDEADDEPLOYINGNEWUNDEPLOYING
Metric | Labels | Description | Observation |
|---|---|---|---|
|
| Gauge. The number of engines per state excluding deleted and unregistered engines. | ⚠️ Should be no engines in the The number of engines in |
| none | Counter. Number of successful engine registrations | Too many registrations during a time interval may indicate unstable engines. |
|
| Counter. Number of engine hard reboots detected by registration without un-registration | ⚠️ Should not increase. |
| none | Counter. Number of failed engine registrations. | ⚠️ Should not increase. |
|
| Counter. Number of engine deaths (communication timeouts) detected | ⚠️ Should not increase, but occasional events due to network connectivity issues are expected. |
|
| Histogram. Time to process engine request for the list of tasks. | Should not increase dramatically and generally stay withing seconds interval. |
|
| Counter. Failed engine task requests. | ⚠️ Should not increase fast, occasional errors are normal. |
|
| Histogram. Time to process successful engine requests. | Should not increase dramatically and generally stay withing seconds interval. |
|
| Counter. Failed engine reports. | ⚠️ Should not increase fast, occasional errors are normal. |
|
| Histogram. Interval between successful engines reports. | Should not increase dramatically and generally stay around configured report interval (5 seconds by default). |
|
| Gauge. Number of migration tasks reported by all engines. | Should stay reasonably low. |
|
| Gauge. Number of cleanup tasks reported by all engines. | Should stay reasonably low. |
|
| Gauge. Number of storage info (refresh) tasks reported by all engines. | Should stay reasonably low. |
|
| Gauge. Number of forwarding (proxy) tasks reported by all engines (including dead). | Should stay reasonably low. |
|
| Gauge. Number of errors in active forwarding tasks on all engines. | ⚠️ Should be zero. |
|
| Gauge. Total number of connections on all engines. | Should not grow over time and stay below environment-specific threshold. |
Endpoints
org- organization ID this endpoint belongs tostate- state of the endpoint, one ofENABLED,DISABLED, orDELETING. Note thatDELETEDendpoints are not included in statistics.config- engine configuration ID
Metric | Labels | Description | Observation |
|---|---|---|---|
|
| Gauge. The number of endpoints configured. | The number of |
|
| Counter. The number of forwarding tasks errors reports by engines. | ⚠️ Should not grow at all. |
State update
type - type of state update, one of migration, endpoint, storage, or storage_refresh.
Metric | Labels | Description | Observation |
|---|---|---|---|
|
| The number of times state update is requested. | Should not grow at rate much above average. |
|
| The number of entities selected for state update. | Should not be too high. |
|
| Time it took to select entities for state update. | Should not be too high. |
|
| Time to finish state update, including time to select entities for state update | Should not be too high. |
|
| The number of entities retried in a state update. | Should not grow too fast. |
|
| The number of entities that failed state update. | ⚠️ Should not grow at all. |
Cardinality
For most metrics with labels, metrics cardinality equals to the total number of all engine configurations in all organizations.
For `fl_mgmt_eng_count cardinality is 7 times larger then that.
Exemplars
Most metrics have exemplars attached allowing to identify problematic entities for diagnostic purposes. It’s essential that exemplars are stored and accessible together with metrics values.
Azure Native Metrics
In addition to metrics exposed by the Management Server via Prometheus we also recommend monitoring native Azure metrics.
App Service Plan
Azure Metric | Criteria |
|---|---|
CPU Percentage | < 80% |
Memory Percentage | < 90%. Note that normally most of consumed memory is discardable cache. |
App Service
Azure App Service that runs the Management Server (API) container, not App Service that runs the Web Console container.
Azure Metric | Criteria |
|---|---|
Health check status | Should be 100 (healthy) |
Requests | Should not be much above average value. Should not drop to near-zero values. |
Response Time | Should not be above acceptable threshold (usually < 500 ms). |
Data In | Should not drop much below average value. |
Data Out | Should not drop much below average value. |
Http 2xx | Should not drop to near-zero values. |
Http 4xx | Should not be too high (occasional 4xx responses are normal) |
Http Server Errors | Should not be too high (occasional 5xx responses are normal) |
SQL Database
Azure Metric | Criteria |
|---|---|
Availability | 100% |
DTU percentage | < 50-80% depending on required response time. If you update engine by scaling up VMSS to double the size, keep DTU below 50%. |
Data space used percent | < 50% |
Workers percentage | < 80%. Higher values may indicate management server configuration mismatch to the SQL Database SKU. |