Every network team running serious infrastructure eventually hits the same fork in the road: the automation you have no longer matches the network you're running. From there, the question becomes Shakespearean:
To build or not to build?
The build option has a reputation problem, and it's mostly deserved. In-house network automation projects stall, sprawl, or quietly become one person's unmaintainable side project. So when a build succeeds at scale, it's worth understanding exactly why.
At a recent AutoCon session, Aldrin Isaac, head of site network engineering at eBay, and Katarzyna Kurowska, lead developer on the project from CodiLime, walked through eBay's decision to build Spectron, its data center fabric automation platform: the reasons, the results, and the parts they'd do differently.
The tool that stopped fitting the network
eBay has been running SONiC, the open-source network operating system, across its data center fabrics for a few years now. In its early days, before the network had settled into its current shape, eBay handled fabric build automation with an in-house Python tool. In Aldrin's words, it was a good tool that became clunky, and its deeper problems are ones many teams will recognize.
The first was drift. The tool was build automation: it could construct a fabric, but it had no ongoing relationship with it. Meanwhile, eBay's cloud controller continuously updated top-of-rack switches with new subnet information as workloads came and went. Within a couple of months of a fabric going live, the build tool's view of the network no longer matched the network. The team fell back on configuration backups for events like RMAs, which worked… until it didn't.
The failure case Aldrin described is instructive. Suppose a 32×400G top-of-rack switch from one manufacturer fails, and the replacement on hand comes from another. Subtle differences between the platforms, down to the qualified software release, meant the old tool simply couldn't handle the swap. A backup of the failed device's configuration couldn't be applied to the new one. What should be routine hardware replacement became manual work.
The second problem was that the tool was configuration-centric. Every new fabric, every new behavior, every new flavor of compute attached to the network (storage, GPUs, whatever came next) meant writing more code. And new flavors keep coming over the life of a fabric.
Requirements first, tooling second
What makes eBay's story useful to teams facing the same decision is that they didn't start by choosing between products and code. They started by writing down what the automation had to be, and that list is worth reading as a template.
It had to be intent-oriented: declare the desired state of the network, not the steps to reach it. It had to be spec-driven, with network designs expressed as specifications that generate everything downstream, such as abstract topology, cabling matrix, and device configuration. It had to treat hardware as fungible: replace any device, from any vendor, at any point in the network lifecycle, without touching the rest of the system. It had to be composable, so new topologies and behaviors could be assembled from existing building blocks rather than coded from scratch. It had to be lightweight enough to run on a laptop in a disaster-recovery scenario, yet powerful enough to rebuild massive networks from specs alone. And it had to make testing a first-class citizen: every design decision verifiable against an emulated digital twin before anything touched expensive physical infrastructure.
Two further requirements are easy to miss but did real work in the decision. eBay wanted control over licensing. They'd watched open-source projects change terms midstream, MongoDB being the example Aldrin cited, and they wanted the option to open-source or share the result themselves someday. And they wanted network engineers, not just developers, to be able to change the system's behavior.
Hold that list against the commercial market, and the build decision starts to explain itself. Any one of those requirements might be met by an off-the-shelf platform. The full set described a system nobody was selling.
What they built
Spectron, now running in production at eBay, is the embodiment of that list. Network designs live in specifications. A fabric spec defines topology as an abstract graph before a single IP address is assigned. Layers are added one at a time: addressing, then routing, then platform specifications that bind abstract nodes to actual hardware, with the system validating that each binding satisfies its constraints. At every layer, the design can be emulated on GNS3 and tested before the next decision is made. When physical devices come up through zero-touch provisioning, they request their configuration from Spectron, which generates it dynamically from the device's role, placement, and hardware details.
The hardware-fungibility requirement is met through a unified configuration model: every device's configuration is represented once, abstractly, and rendered out per platform. Supporting new hardware means providing a renderer, not redesigning the topology. Under the application sits a generic toolkit that lets engineers add business logic as microservices without downtime, schema migrations, or restarts.
Notably, network engineers cross into development here by design. Models are expressed as Go structures rather than YAML, and eBay's network engineers write them. Asked about the user-adoption struggles that sink so many automation rollouts, Aldrin's answer was simple: "We're the user." The people operating the network helped build the system that operates it.
What building actually cost
Here is where the story earns its keep, because the team was unusually honest about the issues.
The headline number first: the Spectron application was built in under a year, by three or four developers. Three of those developers came from CodiLime, not eBay. "We didn't have the ability to expand our development staff," Aldrin explained, so eBay brought in outside software engineers and embedded them with its own network engineers. Building in-house did not mean staffing in-house.
The toolkit underneath took longer, and Aldrin admitted he wishes they hadn't invested so much effort there. Solving orchestration generically, before reality had a chance to correct the design, consumed time the team would later want back. That's the cost of building: you pay for your own detours, and there's no vendor roadmap to blame.
Katarzyna's lessons learned fill out the cost side further. Abstractions, the mechanism behind all that adaptability, are genuinely hard to get right. Prototypes have a way of becoming the final product. And testing only happens if it's easy, because if you make verification cumbersome, engineers will skip it, no matter how disciplined the culture.
Her answer to what made the project succeed where so many fail wasn't technical at all: network engineers embedded in the development cycle, a short feedback loop, and communication that let engineers express intent and developers understand requirements. Orchestration for network operators, as she put it, not orchestration for orchestration’s sake.
What this story says about the build decision
Spectron is what a build looks like when it works, in production, at eBay scale, first version delivered in under a year, and confident enough in its own architecture that the team is showcasing it on conference stages.
Part of that success was knowing exactly where to spend the engineering. The effort went where eBay's network is unlike anyone else's in areas like the specs, the abstractions, the fabric lifecycle logic that encodes how eBay specifically runs networks, while commodity layers like open-source SONiC were adopted rather than rewritten. A focused build, decided component by component.
The team behind it tells a similar story. eBay didn’t choose to expand its own development staff; instead, it brought in developers from outside and embedded them with the network engineers who would live with the result.
And the build isn't finished, because it never really is. The team is now working on a day-2 change-and-test framework, EVPN overlay support, and high availability. The kind of things that, on the buy side of the argument, would be a vendor's problem. Something to keep in mind.
If your requirements list looks like eBay's in that it’s specific enough that no product describes your network, strategic enough to be worth owning, this story shows the shape a successful build takes: adopt the commodity layers, own the differentiating ones, get the development capacity wherever it lives, and never let the people who run the network drift out of the room where it's automated.
