SD-WAN testbed bring-up cut from 2.5h to minutes

The challenge

Every test cycle starts with a testbed: SD-WAN routers, virtual and physical, deployed across several hosts and brought up into a working topology. Bringing up one testbed took around 2.5 hours, and more than 100 users needed them. The client needed a deployment tool that is fast, scalable and reliable across multiple hosts, works with various testing frameworks and stays open to any hypervisor and any type of end device.

What was at risk:

• Hours lost to setup before every test cycle, multiplied across more than 100 users, and lab operating costs that grew with every new testbed

What we did

• Started with a high-level software design, documented with UML diagrams and design patterns, to keep future maintenance costs down
• Kept the design open to any hypervisor and any type of end device, with a modular architecture open for future extensions
• Built a custom Python deployment tool on heavy multitasking, multithreading and cloud-init, focused on reliability and speed
• Made deployment granular down to one VM per host, across hosts running different hypervisors such as ESXi and KVM
• Covered deployment and bring-up of both virtual and physical routers
• Gave it a user-friendly CLI presentation layer and an easy API for automation scripts
• Integrated it with open-source and proprietary test frameworks

How it works

The tool takes a testbed from a topology to running, brought-up routers that tests can use, in four stages.

1. Request A developer or QA engineer starts a deployment from the CLI, or an automation script or test framework calls the tool’s API.
2. Placement The tool spreads the topology across hosts, down to one VM per host, on whichever hypervisor each host runs, such as ESXi or KVM.
3. Deployment and bring-up Virtual machines are deployed in parallel through multitasking and multithreading, and cloud-init configures them as they start. Physical routers in the topology are brought up by the same tool. A topology of 10 virtual machines deploys in minutes.
4. Testing The finished testbed is handed to the open-source or proprietary test framework that requested it, or to the engineer for manual work.