Building an automation module for a network observability solution

The challenge

Integrating a customer's virtual private cloud with the platform was a manual, error-prone process. Every cloud provider exposes flow logs differently, so AWS, GCP and Azure each needed their own access, storage and log configuration before the platform could consume them, and the resulting configuration had to be reproducible and trackable.

What was at risk:

• Errors in a manual integration process, and time and labor spent on every new cloud deployment

What we did

• Provided a team of developers and DevOps engineers with expertise in network monitoring and observability, cloud and automation.
• Designed and built the entire solution from the ground up: architecture, technology stack, execution and testing.
• Enabled configuration of the customer's observability platform with an Infrastructure-as-Code approach, using widely adopted technologies.
• Prepared AWS, GCP and Azure configurations to be consumed by the client's platform.
• Implemented a Terraform provider in Go, a Go SDK for the platform API, Terraform/Ansible modules and unit tests.
• Worked in close cooperation with the client and a fast feedback loop, with regular, recorded demos of new features.

How it works

Each cloud gets the configuration the platform needs to read its flow logs, and Terraform applies it automatically. Four parts make that work.

1. AWS IAM roles and policies give the platform access to EC2 and S3 resources, an S3 bucket stores the flow logs, and the Flow Log configuration is set.
2. GCP Flow logs are enabled for the required subnets, a flow log sink filters them to a Pub/Sub topic with a subscription, and roles allow the platform to pull the logs.
3. Azure The appropriate providers are turned on for flow logs and network watchers, a network watcher gets logs from the resource group, a storage account is configured, and Azure roles grant the platform access.
4. Terraform automation A Terraform provider written in Go uses the observability platform's API to configure flow log export from a cloud deployment, and Terraform/Ansible modules integrate new virtual private clouds automatically, with automated export from AWS and GCP. A Go SDK for the platform API and unit tests complete it.