From telemetry to action when an SLO breaks, for FluxNinja’s Aperture

The challenge

FluxNinja was building Aperture to observe traffic continuously and act on it when a service level objective was breached. Before any policy could decide, every request had to be classified, labeled and processed by many Aperture components, all of it on live traffic. CPU usage had to stay flat and the host system light, and the architecture had to stay scalable, modular and maintainable.

What was at risk:

• A reliability layer that itself loads the host it protects.

What we did

• Optimized the telemetry pipelines that carry logs, metrics and traces into the policy engine.
• Built the request classifiers that turn default request metadata into the labels a policy can decide on.
• Made the ingestion pipelines configurable, so a high volume of traffic moves through quickly to each policy.
• Gave Aperture several ways to plug in, so it fits an existing setup.
• Designed and built the user interface for the paid dashboard product.

How we built it

Aperture works on policies. A policy is a small reliability application: it observes traffic continuously, decides when a service level objective is breached, and acts, for example by shedding load or limiting concurrency. Policies are written declaratively as a signal processing graph and can be reconfigured while running, which works once the data reaching a policy is already classified, labeled and processed.

1. Dynamic ingestion A policy has to be able to act on any label or property of a request. The team chose the OpenTelemetry Collector and prepared its receivers, processors and exporters as a configurable list, so the platform digests a large amount of traffic quickly and still hands each policy the specific data it needs.
2. Flat CPU usage Every incoming request passes through many Aperture components before a decision is made, and all of that had to stay invisible in CPU usage and light on the host system. That was the hardest part of the work.
3. Classification For classification we worked on default request metadata, using the OPA Rego language to express the rules. Labels come out of the metadata a request already carries.
4. Three ways in We built all three ways to use Aperture: as an Envoy filter for service-mesh deployments; as an SDK in Go, Java, JavaScript and TypeScript, and Python; and as a sidecar for virtual machine and local deployments.
5. ARC On top of the open-source project sits FluxNinja ARC, the commercial system where users see dashboards for their own services and the policies applied to them. We designed and developed that interface.

Throughout, the brief was a modern architecture that stays scalable, modular and maintainable, with room to explore new algorithms, data structures and technologies for the client’s flagship product.

In the client’s words

In the startup world, founders’ primary focus is on gathering requirements from users and researching the latest technologies for architecture, etc. However, you need a trusted and skilled engineering team to get all these things implemented. The FluxNinja team has partnered with CodiLime to hire that engineering talent. We have been super impressed with the quality of the work and the dedication of the CodiLime team. It has been a very cost-effective process and helped us scale our team to over 15 team members in 3 months. Thank you, team CodiLime!

Jai Desai, Co-founder & COO, FluxNinja