Observability Requirements
To ensure reliable operation of your Flexify IO deployment and timely troubleshooting it is essential to be prepared.
The checklist below must be completed for each production-ready self-hosted Flexify IO instance.
Access
You’re able to sign in to Flexify IO Management Console as administrator
You’re able to locate and access all Flexify IO resource in Azure portal:
Management Server (API) App Service
Web Console App Service
SQL Database
Engines VMSS
Load balancer
You’re able to access App Service Diagnostic site (AKA Advanced Tools or SCM)
You’re able to download Management Server logs from the SCM site
You’re able to establish SSH session to the Management Server container via the SCM site
You’re able to connect to the SQL Database and run custom read-write queries
You’re able to send S3 requests to each engine and to the load balancer
You’re able to SSH to each engine, locate and download diagnostic information (logs, crash dumps, etc.) if needed
You’re able to copy and send/share files with diagnostic information (such as logs) to Flexify for analysis
You’re able to run basic network diagnostic tools (
ping,traceroute,nslookup,nc,curl,ss) from the SSH sessions on the Management Server and Engines
Monitoring and Alerts
Management Server monitoring and alerts are configured according to Management Server Metrics
Management Server email alerts are configured, tested, and monitored
Engine monitoring and alerts are configured for each engine according to Engine Metrics
Monitoring tools allows sorting and viewing exemplars associated with metrics
Health checks
The load balancer is configured to preform health checks on each engine
Automated end-to-end health checks are configured. The health check should include uploading and/or downloading an object from an attached storage account via the load balancer
[Recommended] The health check includes HTTP GET request (rather then TCP connection only) to a public object in a virtual bucket
You will receive an alert if any of those health checks fail and this is periodically tested/verified
You will receive an alert if health checks are not performed anymore
Diagnostic and audit logs
Management Server is configured to send HTTP logs and App Service Application Logs to Azure Log Analytics workspace, Azure Storage Account, and/or another similar service.
You’re able to access those logs for the period of 90 days or more
Engine diagnostic logs (Engine logs) with at least
INFOlevel are configured to be automatically uploaded to Azure Storage AccountEngine poxy reports and migration reports (if you run migrations) with
ALLare configured to be automatically uploaded to Azure Storage AccountYou can access both engine diagnostic logs and proxy/migration reports for a period of at 90 days or more
You can access both engine diagnostic logs and proxy/migration reports directly on each engine (if they are not uploaded to the storage account yet)