10,000+ VMs moved off VMware onto an open platform

The challenge

The environments ran on VMware ESXi over VLANs, and the 802.1Q standard caps a network at around 4,094 VLANs. With one isolated network per support case, the company had consumed that namespace. The way forward was an overlay model, EVPN/VXLAN, with roughly 16 million segments. NSX was not part of the environment, and adding it would have stacked more cost and complexity on licensing fees that were already hard to justify. Underneath, the switching ran on end-of-life hardware and storage was NFS-based.

What was at risk:

• Growth of the support environments, blocked by the VLAN ceiling, and virtualization licensing costs that kept climbing at scale

What we did

• Evaluated the alternatives against the client’s technical and commercial requirements, recommended Proxmox, and proved it in a proof of concept before committing
• Migrated virtualization from VMware ESXi to Proxmox VE 9.x across multiple clusters, the largest running 32 nodes, and replaced NFS with hyperconverged Ceph
• Kept the network topology as it was during the move, and ran extensive scale and stress testing, so the migration stayed low-risk
• Integrated Proxmox into the client’s existing automation portal, so the end users’ workflow stayed the same
• Implemented EVPN/BGP as the control plane for VXLAN, giving each customer case its own full pool of up to 4,094 VLANs, and moved switching to SONiC
• Extended Proxmox at code level where it fell short: cluster tuning for stable quorum, faster network restarts, selective restart, custom SDN extensions and a netlink-based integration layer
• Automated cluster and node deployment with Ansible, and drove network configuration through GitOps and a source of truth

How it works

The migration ran as a phased programme, from assessment to an open platform that CodiLime keeps supporting.

1. Assessment The team evaluated the available alternatives against the client’s networking, scalability and commercial requirements, and recommended Proxmox.
2. Proof of concept Before committing, CodiLime validated Proxmox at enterprise scale and showed that the requirements Proxmox does not support natively could be met through platform-level engineering.
3. Phase 1: off VMware Virtualization moved from VMware ESXi to Proxmox VE 9.x, and NFS storage to hyperconverged Ceph, which Proxmox supports natively and which makes VM creation faster. Cluster and node deployment is automated with Ansible. Proxmox plugs into the client’s automation portal, and the network topology was kept unchanged to hold the risk down.
4. Code-level extensions With 32 nodes and thousands of network objects per host, Corosync heartbeats and the PMXCFS cluster filesystem were tuned for stable quorum. The team cut per-node network reconfiguration from ~90 to ~15 seconds, restarts only the nodes a change affects, and extended Proxmox SDN so each client-created network has its own complete VLAN pool. A netlink-based integration layer, transparent to Proxmox, talks to the client’s automation portal in real time.
5. Phase 2: network foundation EVPN/BGP now runs as the control plane for a VXLAN overlay built on FRRouting, raising capacity from ~4,094 VLANs to ~16 million VNIs. SONiC replaced the end-of-life switching with all features kept, and network configuration is generated dynamically from GitOps and a source of truth.
6. Support CodiLime provides ongoing support and managed services for the custom-extended platform.

When the scope had to change mid-delivery, the team limited Phase 1 to reduce risk and restructured the plan around the same end goal.