Observability5 min read
A guide to using OpenTelemetry Operator for Kubernetes
In this article
Observability is a key part of modern cloud computing. OpenTelemetry helps by automatically keeping track of how different parts of your system communicate. Think of it like a dashboard for all your microservices. One standout feature? Auto-instrumentation. It does a lot of the setup work for you. But how do you use OpenTelemetry in Kubernetes? That's where the OpenTelemetry Operator comes in. It makes setting everything up in Kubernetes easy. In this post, we’ll introduce the OpenTelemetry Operator and show you a practical example of using its auto-instrumentation feature.
How it works: a glimpse into the OpenTelemetry Operator and OpenTelemetry Collector
At its core, the OpenTelemetry Operator streamlines the process of integrating observability into your Kubernetes applications. Here's a breakdown of what it does and how:
- Management of OpenTelemetry Collector Deployments: The operator manages the deployment of OpenTelemetry Collectors, ensuring they run efficiently in your cluster.
- Auto-Instrumentation: This is where the magic happens. For applications running in the cluster, the operator automates the instrumentation process. How? By injecting sidecars or init containers into deployments. This means less manual work for you and seamless integration for popular services like .NET, Java, Node.js, Go, nginx, and Python.
- Extending Kubernetes API with custom resource definitions (CRDs):
- Instrumentation CRD: Handles the task of instrumenting applications within the cluster.
- OpenTelemetry Collector CRD: Oversees the various OpenTelemetry deployments in the ecosystem.
For a visual representation, take a peek at the following diagram:
Setting up OpenTelemetry Operator and managed Collector
In this section, we'll go step-by-step through the deployment of the OpenTelemetry Operator and the OpenTelemetry Collector it manages.
Create K8s cluster
We will need a K8s cluster. For our tutorial kind cluster will be good enough. To create one:
1kind create cluster -n opentelemetry-operator
Deploying Cert-manager
The OpenTelemetry operator requires Cert-manager. Deploy it with:
1kubectl apply -f2https://github.com/cert-manager/cert-manager/releases/download/v1.12.0/cert-manager.yaml3kubectl rollout status --watch --timeout=5m -n cert-manager \4 deployments/cert-manager \5 deployments/cert-manager-cainjector \6 deployments/cert-manager-webhook
Installing the OpenTelemetry Operator
With Cert-manager in place, let's deploy the OpenTelemetry Operator:
1kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml2kubectl rollout status --watch --timeout=5m \3 -n opentelemetry-operator-system \4 deployments/opentelemetry-operator-controller-manager
Configuring and deploying the Collector
Now, it's time to define and deploy our OpenTelemetry Collector.
1kubectl apply -f - <<EOF2apiVersion: opentelemetry.io/v1alpha13kind: OpenTelemetryCollector4metadata:5 name: demo6spec:7 config: |8 receivers:9 otlp:10 protocols:11 grpc:12 http:13 processors:14 memory_limiter:15 check_interval: 1s16 limit_percentage: 7517 spike_limit_percentage: 1518 batch:19 send_batch_size: 1000020 timeout: 10s21 exporters:22 otlp:23 endpoint: jaeger-collector:431724 tls:25 insecure: true26 logging:27 service:28 pipelines:29 traces:30 receivers: [otlp]31 processors: [memory_limiter, batch]32 exporters: [logging, otlp]33 metrics:34 receivers: [otlp]35 processors: [memory_limiter, batch]36 exporters: [logging]37 logs:38 receivers: [otlp]39 processors: [memory_limiter, batch]40 exporters: [logging]41EOF
Our OpenTelemetry Collector will collect traces in OTLP format and send them to the Jaeger platform. For testing purposes, it will be enough to deploy it from Jaeger all in one image.
1kubectl apply -f - <<EOF2apiVersion: apps/v13kind: Deployment4metadata:5 name: jaeger6 labels:7 app: jaeger8 app.kubernetes.io/name: jaeger9 app.kubernetes.io/component: all-in-one10spec:11 replicas: 112 strategy:13 type: Recreate14 selector:15 matchLabels:16 app: jaeger17 app.kubernetes.io/name: jaeger18 app.kubernetes.io/component: all-in-one19 template:20 metadata:21 labels:22 app: jaeger23 app.kubernetes.io/name: jaeger24 app.kubernetes.io/component: all-in-one25 annotations:26 prometheus.io/scrape: "true"27 prometheus.io/port: "16686"28 spec:29 containers:30 - env:31 - name: COLLECTOR_OTLP_ENABLED32 value: "true"33 image: jaegertracing/all-in-one34 name: jaeger35 ports:36 - containerPort: 431737 protocol: TCP38 - containerPort: 1668639 protocol: TCP40 readinessProbe:41 httpGet:42 path: "/"43 port: 1426944 initialDelaySeconds: 545---46apiVersion: v147kind: Service48metadata:49 name: jaeger-query50 labels:51 app: jaeger52 app.kubernetes.io/name: jaeger53 app.kubernetes.io/component: query54spec:55 ports:56 - name: query-http57 port: 8058 protocol: TCP59 targetPort: 1668660 selector:61 app.kubernetes.io/name: jaeger62 app.kubernetes.io/component: all-in-one63 type: LoadBalancer64---65apiVersion: v166kind: Service67metadata:68 name: jaeger-collector69 labels:70 app: jaeger71 app.kubernetes.io/name: jaeger72 app.kubernetes.io/component: collector73spec:74 ports:75 - name: jaeger-collector-otlp-grpc76 port: 431777 protocol: TCP78 targetPort: 431779 selector:80 app.kubernetes.io/name: jaeger81 app.kubernetes.io/component: all-in-one82 type: ClusterIP
Setting up auto-instrumentation resources
With our platform configured to collect traces, the next step is to instrument our applications using the instrumentation objects. These objects enable our apps to automatically collect and send trace telemetry data without manual instrumentation.
For demonstration purposes, we'll set up instrumentation for two applications: a Python application and a Node.js application.
Instrumentation for the Python application
1kubectl apply f - <<EOF2apiVersion: opentelemetry.io/v1alpha13kind: Instrumentation4metadata:5 name: python-demo-instrumentation6spec:7 exporter:8 endpoint: http://demo-collector:43189 propagators:10 - tracecontext11 - baggage12 sampler:13 type: parentbased_traceidratio14 argument: "1"15EOF
Instrumentation for the Node.js application
1kubectl apply f - <<EOF2apiVersion: opentelemetry.io/v1alpha13kind: Instrumentation4metadata:5 name: node-demo-instrumentation6spec:7 exporter:8 endpoint: http://demo-collector:43179 propagators:10 - tracecontext11 - baggage12 sampler:13 type: parentbased_traceidratio14 argument: "1"15EOF
These configurations ensure that both Python and Node.js applications are equipped to gather trace telemetry data and communicate it to our previously set up OpenTelemetry Collector.
Building and deploying instrumented applications
To demonstrate how auto-instrumentation works, we'll set up a simple application stack:
- Python App: Acts as our backend.
- Node.js App: Serves as the frontend, making requests to the Python backend. This will help us understand how metrics are collected and how spans are associated.
Python backend
Here's a minimal Flask app:
1from flask import Flask2app = Flask(__name__)34@app.route('/')5def hello():6 return "Hello World!"78if __name__ == '__main__':9 app.run(host='0.0.0.0', port=8000)
Node.js frontend
This Node.js app fetches telemetry data from the Flask backend:
1const express = require('express');2const fetch = require('isomorphic-fetch');34const app = express();5const port = 3000;67app.get('/', async (req, res) => {8 try {9 const response = await fetch('http://flask-app-service:8000/');10 const data = await response.text();11 res.send(data);12 } catch (error) {13 console.error('Error:', error);14 res.status(500).send('Internal Server Error');15 }16});1718app.listen(port, () => {19 console.log(`Server is running on port ${port}`);20});
Note: Both apps are devoid of OpenTelemetry library imports or in-code instrumentation. This highlights the power of auto-instrumentation; we don't manually set up our apps to send traces to the OpenTelemetry Collector.
Dockerizing apps
After writing the application code, you'll need to containerize these apps. Once containerized, you can load them into the kind cluster:
1kind load docker-image -n python-demo-app2kind load docker-image -n node-demo-app
Deploy applications with instrumentation enabled
Before we deploy our applications, we need to prepare the specifications. Annotations for instrumentation.opentelemetry.io are used to indicate which instrumentation should be injected into the deployments.
Deploying the backend
1kubectl apply -f - <<EOF2apiVersion: apps/v13kind: Deployment4metadata:5 name: flask-app6spec:7 replicas: 18 selector:9 matchLabels:10 app: flask-app11 template:12 metadata:13 labels:14 app: flask-app15 annotations:16 instrumentation.opentelemetry.io/inject-python: python-demo-instrumentation17 spec:18 containers:19 - name: python-test-app20 image: python-demo-app21 imagePullPolicy: IfNotPresent22 resources:23 limits:24 memory: "128Mi"25 cpu: "500m"26 ports:27 - containerPort: 800028---29apiVersion: v130kind: Service31metadata:32 name: flask-app-service33spec:34 selector:35 app: flask-app36 ports:37 - port: 800038 targetPort: 800039EOF
Deploying the frontend
1kubectl apply -f - <<EOF2apiVersion: apps/v13kind: Deployment4metadata:5 name: node-app6spec:7 replicas: 18 selector:9 matchLabels:10 app: node-app11 template:12 metadata:13 labels:14 app: node-app15 annotations:16 instrumentation.opentelemetry.io/inject-nodejs: node-demo-instrumentation17 spec:18 containers:19 - name: node-test-app20 image: node-demo-app21 imagePullPolicy: IfNotPresent22 resources:23 limits:24 memory: "128Mi"25 cpu: "500m"26 ports:27 - containerPort: 300028---29apiVersion: v130kind: Service31metadata:32 name: node-app-service33spec:34 selector:35 app: node-app36 ports:37 - port: 300038 targetPort: 300039EOF
After executing these commands, our deployments will be initiated. Once they are ready, it's interesting to observe how the deployments have been modified by the operator to enable metric collection. Let's inspect one of them:
1kubectl describe pod flask-app-6879886cdb-2rjwt
Upon examination, we can observe several modifications:
- New volume creation
1Volumes:2 ...3 opentelemetry-auto-instrumentation:4 Type: EmptyDir (a temporary directory that shares a pod's lifetime)5 Medium:6 SizeLimit: <unset>
- Injection of an initialization container: This container copies OpenTelemetry instrumentation libraries to the newly added volume.
1Init Containers:2 opentelemetry-auto-instrumentation:3 Container ID: containerd://2119db597d29bd714a0314848df113135d945c9e69b9887f6c15088cbea8685b4 Image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.40b05 Image ID: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python@sha256:c3816761a4b739aa5f46cca3500b1e4c7926fec7e02871c2117dfb4bb45eb8876 Port: <none>7 Host Port: <none>8 Command:9 cp10 -a11 /autoinstrumentation/.12 /otel-auto-instrumentation/13 State: Terminated14 Reason: Completed15 Exit Code: 016 Started: Fri, 01 Sep 2023 12:36:28 +020017 Finished: Fri, 01 Sep 2023 12:36:28 +020018 Ready: True19 Restart Count: 020 Limits:21 cpu: 500m22 memory: 32Mi23 Requests:24 cpu: 50m25 memory: 32Mi26 Environment: <none>27 Mounts:28 /otel-auto-instrumentation from opentelemetry-auto-instrumentation (rw)29 /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bjggg (ro)
- Addition of environment variables: These variables are set for the main container.
1Containers:2 python-test-app:3 ...4 Environment:5 PYTHONPATH: /otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation:/otel-auto-instrumentation6 OTEL_TRACES_EXPORTER: otlp7 OTEL_EXPORTER_OTLP_TRACES_PROTOCOL: http/protobuf8 OTEL_METRICS_EXPORTER: otlp9 OTEL_EXPORTER_OTLP_METRICS_PROTOCOL: http/protobuf10 OTEL_SERVICE_NAME: flask-app11 OTEL_EXPORTER_OTLP_ENDPOINT: http://demo-collector:431812 OTEL_RESOURCE_ATTRIBUTES_POD_NAME: flask-app-6879886cdb-2rjwt (v1:metadata.name)13 OTEL_RESOURCE_ATTRIBUTES_NODE_NAME: (v1:spec.nodeName)14 OTEL_PROPAGATORS: tracecontext,baggage15 OTEL_TRACES_SAMPLER: parentbased_traceidratio16 OTEL_TRACES_SAMPLER_ARG: 117 OTEL_RESOURCE_ATTRIBUTES: k8s.container.name=python-test-app,k8s.deployment.name=flask-app,k8s.namespace.name=default,k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.name=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME),k8s.replicaset.name=flask-app-6879886cdb,service.version=python-demo-app
- Verification of directories: The PYTHONPATH environment variable points to directories that have numerous OpenTelemetry Python modules and their dependencies.
1root@flask-app-6879886cdb-2rjwt:/app# ls -w 1 /otel-auto-instrumentation/2Deprecated-1.2.14.dist-info3__pycache__4_distutils_hack5asgiref6asgiref-3.7.2.dist-info7backoff8backoff-2.2.1.dist-info9bin10certifi11certifi-2023.5.7.dist-info12charset_normalizer13charset_normalizer-3.2.0.dist-info14deprecated15distutils-precedence.pth16google17googleapis_common_protos-1.59.1-py3.9-nspkg.pth18googleapis_common_protos-1.59.1.dist-info19idna20idna-3.4.dist-info21importlib_metadata22importlib_metadata-6.8.0.dist-info23opentelemetry24opentelemetry_api-1.19.0.dist-info25opentelemetry_distro-0.40b0.dist-info26opentelemetry_exporter_otlp_proto_common-1.19.0.dist-info27opentelemetry_exporter_otlp_proto_http-1.19.0.dist-info28opentelemetry_instrumentation-0.40b0.dist-info29opentelemetry_instrumentation_aio_pika-0.40b0.dist-info30opentelemetry_instrumentation_aiohttp_client-0.40b0.dist-info31opentelemetry_instrumentation_aiopg-0.40b0.dist-info32opentelemetry_instrumentation_asgi-0.40b0.dist-info33opentelemetry_instrumentation_asyncpg-0.40b0.dist-info34opentelemetry_instrumentation_boto-0.40b0.dist-info35opentelemetry_instrumentation_boto3sqs-0.40b0.dist-info36opentelemetry_instrumentation_botocore-0.40b0.dist-info37opentelemetry_instrumentation_celery-0.40b0.dist-info38opentelemetry_instrumentation_confluent_kafka-0.40b0.dist-info39opentelemetry_instrumentation_dbapi-0.40b0.dist-info40opentelemetry_instrumentation_django-0.40b0.dist-info41opentelemetry_instrumentation_elasticsearch-0.40b0.dist-info42opentelemetry_instrumentation_falcon-0.40b0.dist-info43opentelemetry_instrumentation_fastapi-0.40b0.dist-info44opentelemetry_instrumentation_flask-0.40b0.dist-info45opentelemetry_instrumentation_grpc-0.40b0.dist-info46opentelemetry_instrumentation_httpx-0.40b0.dist-info47opentelemetry_instrumentation_jinja2-0.40b0.dist-info48opentelemetry_instrumentation_kafka_python-0.40b0.dist-info49opentelemetry_instrumentation_logging-0.40b0.dist-info50opentelemetry_instrumentation_mysql-0.40b0.dist-info51opentelemetry_instrumentation_mysqlclient-0.40b0.dist-info52opentelemetry_instrumentation_pika-0.40b0.dist-info53opentelemetry_instrumentation_psycopg2-0.40b0.dist-info54opentelemetry_instrumentation_pymemcache-0.40b0.dist-info55opentelemetry_instrumentation_pymongo-0.40b0.dist-info56opentelemetry_instrumentation_pymysql-0.40b0.dist-info57opentelemetry_instrumentation_pyramid-0.40b0.dist-info58opentelemetry_instrumentation_redis-0.40b0.dist-info59opentelemetry_instrumentation_remoulade-0.40b0.dist-info60opentelemetry_instrumentation_requests-0.40b0.dist-info61opentelemetry_instrumentation_sklearn-0.40b0.dist-info62opentelemetry_instrumentation_sqlalchemy-0.40b0.dist-info63opentelemetry_instrumentation_sqlite3-0.40b0.dist-info64opentelemetry_instrumentation_starlette-0.40b0.dist-info65opentelemetry_instrumentation_system_metrics-0.40b0.dist-info66opentelemetry_instrumentation_tornado-0.40b0.dist-info67opentelemetry_instrumentation_tortoiseorm-0.40b0.dist-info68opentelemetry_instrumentation_urllib-0.40b0.dist-info69opentelemetry_instrumentation_urllib3-0.40b0.dist-info70opentelemetry_instrumentation_wsgi-0.40b0.dist-info71opentelemetry_propagator_aws_xray-1.0.1.dist-info72opentelemetry_propagator_b3-1.19.0.dist-info73opentelemetry_propagator_jaeger-1.19.0.dist-info74opentelemetry_propagator_ot_trace-0.40b0.dist-info75opentelemetry_proto-1.19.0.dist-info76opentelemetry_sdk-1.19.0.dist-info77opentelemetry_semantic_conventions-0.40b0.dist-info78opentelemetry_util_http-0.40b0.dist-info79packaging80packaging-23.1.dist-info81pkg_resources82protobuf-4.23.4.dist-info83psutil84psutil-5.9.5.dist-info85requests86requests-2.31.0.dist-info87setuptools88setuptools-68.0.0.dist-info89typing_extensions-4.7.1.dist-info90typing_extensions.py91urllib392urllib3-2.0.3.dist-info93wrapt94wrapt-1.15.0.dist-info95zipp96zipp-3.16.2.dist-info
- Inside the /otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation directory, we find code responsible for injecting these modules into our application.
For Node.js applications, the process of injecting OpenTelemetry instrumentation libraries is similar.
Generating traffic
To verify that our setup is working as expected, we can expose our Node.js application and generate some sample traffic.
1kubectl port-forward svc/node-app-service 3000:3000 &2curl localhost:30003Hello World!⏎
We should receive a "Hello World!" response. But, were the traces collected? Let's find out.
Expose Jaeger web UI:
1kubectl port-forward svc/jaeger-query 9080:80 &
Then, navigate to http://localhost:9080 in your browser. If you search and expand the span traces for the node-app application, you should notice that traces have been successfully collected and are interlinked appropriately with our backend.
Summary
In conclusion, the Opentelemetry Operator stands as a groundbreaking tool in the modern telemetry landscape. It seamlessly facilitates the tracing of intricate applications on Kubernetes platforms. The major highlight? You can attain comprehensive application insights without altering a single line of your current application's code. This is especially invaluable in scenarios where an environment relies heavily on third-party applications. It streamlines the task of tracing, ensuring efficiency, accuracy, and depth of understanding, all while minimizing the manual overhead often associated with such endeavors. As we progress into an era of ever-evolving digital complexity, tools like OpenTelemetry Operator will undoubtedly be indispensable for businesses aiming to maintain transparency, performance, and user satisfaction.
Observability is a key part of modern cloud computing. OpenTelemetry helps by automatically keeping track of how different parts of your system communicate. Think of it like a dashboard for all your microservices. One standout feature? Auto-instrumentation. It does a lot of the setup work for you. But how do you use OpenTelemetry in Kubernetes? That's where the OpenTelemetry Operator comes in. It makes setting everything up in Kubernetes easy. In this post, we’ll introduce the OpenTelemetry Operator and show you a practical example of using its auto-instrumentation feature.
Here, you can find more information about our environment services, where we describe how we can support you in terms of monitoring and observability.
