From a failure alert to ranked root causes in minutes

The challenge

Syslog is high-volume and repetitive, and when something breaks the raw events grow fast and overwhelm the people reading them. Alerting on top of a data center monitoring platform has to detect anomalies accurately while keeping false positives low. And accurate root-cause analysis often needs more than logs: the current config and runtime state of the devices involved, and what fixed similar problems before.

What was at risk:

• Slow root-cause analysis and longer downtime, operators buried in repeated events and false alerts, and hypotheses built from logs alone

What we did

• Built an alerting system on AI-driven multi-factor analysis of metrics and logs from network devices, IT hardware and hypervisors
• Trained an ML model on historical trouble-ticket data to point to the most likely root causes of alarms, and added dedicated views to the monitoring platform’s frontend
• Compressed syslog before any LLM sees it: events grouped by device, process and error code, and similar messages normalized into one
• Split complex failures into issue buckets with embeddings and semantic clustering, and analyzed each bucket separately
• Added a coordination agent that plans which extra evidence to fetch, from live config and state to past resolved tickets, and folds it into the analysis
• Delivered the result as an alert with severity, description, ranked root causes and suggested next steps

How it works

The PoC runs on a live Containerlab infrastructure of 12 routers connecting four sites, with servers running virtual machines and all syslog collected on one central server. A failure goes through five stages.

1. Detection The platform counts errors in the syslog of each device. Once they cross a threshold, set to 20 in the demo, it raises an alert with a severity and a description drawn from the error messages, and starts the root-cause analysis automatically.
2. Aggregation Repeated errors are grouped by device, process and error code, counted, and stamped with their first and last occurrence. Messages that differ only slightly are normalized into one by an LLM step. The aggregated log is then sent to the LLM with a prompt that asks for likely root causes and resolution actions.
3. Clustered analysis For complex failures, such as two concurrent faults or problems across protocols and layers, each normalized message gets an embedding. Clustering separates semantically different errors into buckets, each bucket is analyzed on its own, and the results are synthesized.
4. Augmented analysis A coordination agent reads the symptoms and builds a query plan for additional data: how the affected nodes are connected, the status of the interfaces between them, BGP peer status on both sides, BFD state and past resolved tickets with similar symptoms, retrieved through RAG. That context is added to the aggregated log before the LLM reasons over it.
5. Output The engineer gets the alert together with several potential root causes, each with a probability, the evidence the agent retrieved, and suggested next steps. The analyses can run in sequence or in parallel.

A second PoC applies the same idea to an existing data center monitoring platform: multi-factor analysis over metrics and logs drives the alerting, and an ML model trained on historical trouble tickets ranks the most likely root causes of each alarm, shown in dedicated views in the platform’s frontend.