Nutanix Xi Epoch is the full-stack monitoring and alerting service for every enterprise application at any scale across public clouds and private infrastructure. It generates terabytes of metrics and integrates with over 200 common applications to holistically understand application performance. The system is operations-ready from first launch.
The challenge was to build a continuous integration system that would be easy to use and simple to develop for multiple engineers around the world.
The first stage of the project involved migrating CI workflows from Jenkins to CircleCI. This was achieved by methodically gathering requirements and creating reusable resources to improve build reproducibility and reliability. Changes to the toolchain brought corresponding changes to the build system’s structure and flows. Multiple stages have been separated to parallelize the work and speed up the process and make it as close to Agile and DevOps standards as possible. Ultimately, faster feedback means the work is done more effectively. Using CircleCI empowered us to make the builds highly reproducible and improve the way we conducted them.
The second stage involved preparing a scalable test automation framework that would be easy to integrate with a continuous integration system and enable developers to launch it locally against any built part of a system. This was achieved by mocking up every part of the system and using a containerized environment (docker and docker-compose) for the data aggregators and test execution.
Diagram 1 presents the test architecture in CI for testing the Collector. All components in this architecture, except Agent, run within separated docker containers connected together using docker-compose.
Diagram 2 presents the scheme for Periodic builds that are executed by CircleCI in constant time intervals (currently every 6 hours). This process performs full builds of every component in the system, runs them using Kubernetes and finally performs all the tests. Additionally, the test results are delivered to the user via Slack.