CI/CD and test automation for Nutanix Xi Epoch

The challenge

Xi Epoch is developed by multiple engineers around the world. The challenge was to build a continuous integration system that would be easy to use and simple to develop for all of them, together with a test automation framework that fits into it.

The CI/CD system had to run periodic, scheduled builds; run full system builds or selected components only; run partial builds triggered by pull requests; and stay highly scalable and reliable. The tests had to cover a complex, cloud-based architecture with a multi-platform framework: smoke tests, single component tests, end-to-end tests and UI tests.

What made it hard:

• A complex build system.
• Multiple cross-dependencies with multiple management systems.
• A complex, cloud-based architecture to test on several platforms.

What we did

• Migrated CI workflows from Jenkins to CircleCI, gathering requirements methodically.
• Created reusable resources that improve build reproducibility and reliability.
• Restructured the build system’s flow, separating stages so they run in parallel, as close to Agile and DevOps standards as possible.
• Set up periodic builds, executed by CircleCI every 6 hours, that build every component, run them on Kubernetes and perform all the tests, with results delivered via Slack.
• Prepared a scalable test automation framework that integrates with the continuous integration system.
• Made it possible for developers to launch the tests locally against any built part of the system.
• Mocked up every part of the system and used a containerized environment, Docker and Docker Compose, for the data aggregators and test execution.

How we built it

The project was split into two phases: designing the new CI/CD system, then building the test automation framework that plugs into it.

1. Migration to CircleCI CI workflows moved from Jenkins to CircleCI. We gathered requirements methodically and created reusable resources, which made builds highly reproducible and reliable.
2. Parallel stages The new toolchain brought a new structure and flow to the build system. Multiple stages were separated so the work runs in parallel and the process speeds up. Faster feedback means the work is done more effectively.
3. Periodic builds CircleCI runs a periodic workflow from the master branch at constant intervals, currently every 6 hours. It performs full builds of every component in the system, pushes container images to a registry and test reports to cloud storage, deploys to a development cluster on Kubernetes Engine, runs all the tests, and sends a Slack notification with links to the reports.
4. Test automation framework We prepared a scalable test automation framework that integrates easily with the continuous integration system and lets developers launch it locally against any built part of the system.
5. Testing the collector In the test architecture for the collector, a test plan script prepares the device under test and launches test execution. The collector publishes data through a listener to a message broker, and the collector tests subscribe to it and run the test suites. Every component except the agent runs in its own Docker container, connected together with Docker Compose.

The overall architecture is constructed from separate blocks with minimal dependencies. With separated components and clean CI workflows the build system is more efficient, and the team discovers and fixes issues immediately.

In the client’s words

CodiLime's DevOps teams helped us reduce our development cycle time by 50%, and its software engineering helped us improve the quality of our product by increasing our test coverage by more than 80% in crucial parts of our system. This allows us to deliver new features and capabilities seamlessly without compromising on simplicity to our global customer base. We're looking forward to further collaboration with CodiLime as the Epoch product evolves with time.

Harjot Gill, General Manager, Nutanix Xi Epoch