End-to-end monitoring for a rebuilt hypervisor, storage, and network platform

The challenge

With our client, we had migrated its hypervisor environment, along with a new storage solution and a new network overlay. The legacy monitoring systems were still covering the platform that had been replaced. Newly deployed servers, hypervisors and their VMs, storage, and the network devices carrying the EVPN data plane all needed coverage. New hypervisor logs sat locally on each node, with retention of roughly one week.

What was at risk:

• Until monitoring was adapted, the whole transformation could not be completed.

What we did

• Extended the monitoring platform the client already ran, after confirming it supported both the new hypervisor environment and the new storage.
• Instrumented each layer through the interface it exposes: out-of-band hardware APIs for the servers, SNMP for the network devices, platform APIs and native agents for the operating systems, hypervisors and VMs.
• Extended coverage to the newly purchased switches carrying the EVPN data plane and to the appliances connected to the lab environments, as well as the compute tier.
• Proposed centralized log collection over syslog to replace local-only hypervisor logging, specified around long-term retention, fast event lookup, centralized access across components, and alerting on specific events and log patterns.
• Assessed candidate technologies against the client’s own constraints: licensing compatibility, maturity and viability, interface usability, integration with their centralized enterprise authentication, Ceph support, and EVPN data plane support.
• Prepared working proof-of-concept stacks, configured from settings drawn from our earlier engagements, so the client could test each option in practice before deciding.
• Recommended a dedicated dashboarding layer once the built-in dashboards proved too limited for the level of visualization and analysis required.
• Ran requirements gathering, proof-of-concept preparation and UX design as parallel tracks, with a UX engineer working across all three solution areas, to fit the client’s limited team availability and a tight schedule.

How we built it

The solution covered three areas: resource monitoring and alerting, centralized log management, and advanced dashboards. Because the schedule was tight, the work ran as three parallel tracks: requirements gathering, proof-of-concept preparation, and UX design.

1. Resource monitoring and alerting Requirements gathering confirmed Zabbix as the client’s primary monitoring platform. Because Zabbix supported both the newly deployed hypervisor environment and the Ceph-based storage, the existing setup was extended with additional metrics and alerting thresholds. The implemented monitoring covers customer hardware appliances, the Juniper QFX switches providing the EVPN overlay and the Juniper EX switches providing administrative network access, all via SNMP; the health and hardware status of the Dell servers hosting the hypervisors via the iDRAC API; and Linux operating systems, hypervisors and hosted virtual machines via hypervisor APIs and native Zabbix agents. sFlow/NetFlow integration was postponed until a future Zabbix release provides it.
2. Centralized log management We proposed a centralized log management solution based on the syslog protocol, collecting from all hypervisor nodes, selected virtual machines, network devices, and other relevant infrastructure components. OpenSearch, Graylog, VictoriaLogs, and Grafana Loki were considered against four requirements: long-term log retention, fast event lookup, centralized access to logs from multiple components, and custom alerting based on specific events and log patterns.
3. Advanced dashboards Zabbix’s built-in dashboarding was too limited for the level of visualization and analysis needed, so we recommended a separate platform. In a short proof-of-concept phase we presented dashboards built with Prometheus and Grafana, with mature integrations for hypervisors, bare-metal servers, and Ceph-based storage: a centralized view of resource utilization, infrastructure health, and historical trends. With VictoriaLogs for logging, VictoriaMetrics would keep metrics and logs in one ecosystem and one operational model.
4. UX across all three A UX engineer supported all three areas, so key infrastructure metrics are easy to locate and moving from a high-level overview to a detailed investigation of a specific issue is intuitive. The dashboards and user flows were designed and refined on that basis.

The proof-of-concept stacks were configured using predefined settings derived from our own experience. The client’s limited hours went into testing working systems, and their engineers could get familiar with each interface and operational model before committing.