Orchestration

Building a cloud-native application for Neptune.ai’s data science lab

Who the client is

Neptune.ai started out as a cloud-native data science lab. Individuals and teams of data scientists could run multiple experiments simultaneously, shortening the time needed to reach results. Neptune took on the complexity of infrastructure management in the experimentation process, while making it easier to collaborate, share results and manage the building of an ML model. Today, Neptune’s mission is to give production ML teams the same level of control while building models as software developers have while shipping applications.

The challenge

Both the CodiLime and Neptune.ai teams knew from the start that building machine learning models requires massive computing power, so the selection of technologies and devices was crucial. At the beginning there was a spike in demand for computing power; by the end it had dropped sharply. The platform had to meet that demand and stay easy to manage and cost-effective for users.

What the platform had to handle:

Massive computing power for building machine learning models.
A spike in demand at the start of the project, followed by a sharp drop.

What we did

Delivered the initial implementation of the service, including a custom Kubernetes autoscaler.
Designed and implemented the infrastructure, with Neptune running on Kubernetes so the whole process of establishing and closing VMs is seamless and smooth.
Used Helm templates and a custom scaling service to reduce the time needed to run new machines and launch an experiment.
Gave all VMs shared access to the training dataset in use.
Used Kubernetes’ granular access control so that only authorized workloads reach the data.
Provided IT support and IT advice.

How it works

Neptune runs on Google Cloud Platform in two Kubernetes clusters: a core cluster with the platform’s services and a worker cluster where experiments run.

Core cluster A data scientist’s requests pass through load balancing to the backend services, a set of microservices with MySQL and Kafka. Keycloak handles identity, and a distributed file storage holds the data.
Worker cluster Each experiment runs the user’s machine learning code in the worker cluster, which talks to the backend services and reads from the distributed file storage. All VMs share access to the training dataset in use, which makes experimentation faster and brings non-trivial cost optimizations.
Custom autoscaling Autoscaling services in the core cluster scale the worker cluster on demand. With Helm templates and the custom scaling service, Neptune reduced the time needed to run new machines and launch an experiment, and GPUs are provisioned on demand only.
Access control and portability Kubernetes’ granular access control guarantees that only authorized workloads have access to the data. Kubernetes also makes Neptune infrastructure-agnostic, so it can be established in a private or public cloud.

In the client’s words

CodiLime's expertise in software engineering was indispensable at the beginning of Neptune's journey. CodiLime helped us build and release the first version of Neptune and convince investors that our product is a real game changer for data scientists. Without any hesitation, I can recommend CodiLime as a reliable technology partner for every tech startup.

Piotr Niedźwiedź, CEO, Neptune Labs

Outcomes

Technical
GPUs on demand only
A custom autoscaler scales the worker cluster on demand and provisions GPUs only when an experiment needs them.
Business
Non-trivial cost optimizations
All VMs share access to the training dataset in use, making experimentation faster and cheaper.
Made possible
Computing-heavy experiments
Computing-heavy data experiments accommodated through a spike in demand, on a cloud-native design that runs in a private or public cloud.