Kubernetes Auto-instrumentation

Introduction

Middleware's auto-instrumentation solution, built on top of the The OpenTelemetry (OTel) Kubernetes Operator, provides automatic detection and instrumentation for .NET, Java, Node.js, Python, and Go services.

The auto-instrumentation feature allows you to:

1. Automatically detect application languages across your cluster.
2. Enable/disable instrumentation per namespace or application.
3. Manage instrumentation through a user-friendly UI.
4. Apply changes without manual configuration.

Middleware Application Monitoring leverages the OpenTelemetry Operator for Kubernetes and can be used to auto-instrument your applications implemented in supported programming languages.

Prerequisites

1 Kubernetes Version

Kubernetes version 1.21.0 or above. Check with the following command:

Shell

shell

kubectl version

2 Middleware Kubernetes Agent

Install Middleware Kubernetes agent using these instructions. Check that the agent is installed with the following command:

Shell

shell

kubectl get pods -n mw-agent-ns

Setup Middleware Auto-instrumentation

Installation Methods

helm repo add jetstack https://charts.jetstack.io --force-update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.14.5 --set installCRDs=true
helm repo add middleware-labs https://helm.middleware.io
helm install mw-auto middleware-labs/mw-autoinstrumentation \
--set mw.apiKey=<MW_API_KEY> \
--set mw.target=https://deofs.stage.env.middleware.io:443 \
--set clusterMetadata.name=<YOUR_CLUSTER_NAME> \
--namespace mw-autoinstrumentation \
--create-namespace

helm repo add middleware-labs https://helm.middleware.io
helm install mw-auto middleware-labs/mw-autoinstrumentation \
--set mw.apiKey=<MW_API_KEY> \
--set mw.target=https://deofs.stage.env.middleware.io:443 \
--set clusterMetadata.name=<YOUR_CLUSTER_NAME> \
--namespace mw-autoinstrumentation \
--create-namespace

--set webhook.includedNamespaces="{namespace1,namespace2}"

--set webhook.userExcludedNamespaces="{namespace1,namespace2}"

MW_INSTALL_CERT_MANAGER=true MW_API_KEY="<MW_API_KEY>" bash -c "$(curl -L https://install.middleware.io/scripts/mw-kube-auto-instrumentation-install.sh)"

MW_API_KEY="<MW_API_KEY>" bash -c "$(curl -L https://install.middleware.io/scripts/mw-kube-auto-instrumentation-install.sh)"

MW_EXCLUDED_NAMESPACES="namespace1,namespace2"

MW_INCLUDED_NAMESPACES="namespace1,namespace2 "

kubectl apply -f https://install.middleware.io/manifests/autoinstrumentation/mw-otel-auto-instrumentation.yaml

Configuring Auto-instrumentation

Application Selection

1. Go to the "Choose Target Application" section in the Kubernetes agent installation screen:

   • View automatically detected applications and their languages.
   • Enable or disable instrumentation for specific applications.
   • Use "Select All" to toggle instrumentation for all detected applications.
   

2. For each application, you can see:

   • Application name.
   • Workload type (Deployment, StatefulSet or Daemonset).
   • Programming language (automatically detected).
   • Current instrumentation status.

Applying Changes

After making changes to your instrumentation configuration:

1. Click "Save Changes" to apply your configuration.

2. Restart the affected applications for the changes to take effect. Example:-

   kubectl rollout restart deployment <deployment-name> -n <namespace>

   

Core Components

1. OpenTelemetry Operator: The foundation of the auto-instrumentation system

   • Manages the lifecycle of OpenTelemetry instrumentation.
   • Handles the injection of instrumentation agents.
   • Integrated as a dependency in the Middleware Helm chart.

2. Middleware-Specific Components:

   • Language Detector: Automatically identifies the programming language of your applications.
   • Auto-injector Webhook: Mutates pod specifications to add necessary instrumentation annotations.

These components work together to:

• Detect application languages automatically.
• Apply appropriate instrumentation configurations.
• Manage the lifecycle of instrumentation agents.
• Provide visibility through the Middleware UI.

Verification

To verify that auto-instrumentation is working:

1. Check the auto-instrumentation pods:

kubectl get pods -n mw-autoinstrumentation

2. Check the detected applications in the Middleware UI. Verify that they are enabled and have been restarted.
3. Check application logs for instrumentation messages.
4. Monitor your applications in the Middleware APM section.

Manual Custom Configuration

When to Use Manual Configuration

Use manual configuration in these scenarios:

• Language detection isn't working for your application.
• You need custom instrumentation settings.
• Your application has special requirements.
• You want fine-grained control over the instrumentation.

In order to auto-instrument your applications, the OTel Kubernetes Operator needs to know which Kubernetes Pods to instrument and which automatic instrumentation configuration, called Instrumentation Custom Resource (CR), to use for those Pods.

The OTel Kubernetes Operator installation steps described in the Installation section also installs the default Instrumentation CR called mw-autoinstrumentation in the mw-agent-ns namespace.

To confirm that Instrumentation CR is installed, issue the folowing command

kubectl get otelinst mw-autoinstrumentation -n mw-agent-ns

The Instrumentation CR looks like below

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: mw-autoinstrumentation
  namespace: mw-agent-ns
spec:
  exporter:
    endpoint: http://mw-service.mw-agent-ns:9319
  propagators:
    - tracecontext

This Instrumentation CR will be used to annotate Pod specifications. The annotations will depend on the programming language used to implement process running inside the Pod.

Below is the list of supported programming languages and the annotations used for auto-instrumentation.

Java

To enable auto-instrumentation for a Java application, add the following annotation to your Pod specification:

instrumentation.opentelemetry.io/inject-java: "mw-agent-ns/mw-autoinstrumentation"

Node.js

To enable auto-instrumentation for a Node.js application, add the following annotation to your Pod specification:

instrumentation.opentelemetry.io/inject-nodejs: "mw-agent-ns/mw-autoinstrumentation"

Python

To enable auto-instrumentation for a Python application, add the following annotation to your Pod specification:

instrumentation.opentelemetry.io/inject-python: "mw-agent-ns/mw-autoinstrumentation"

.NET

To enable auto-instrumentation for a .NET application and specify the runtime identifier (RID), use the following annotations:

For Linux glibc based images (default):

instrumentation.opentelemetry.io/inject-dotnet: "mw-agent-ns/mw-autoinstrumentation"
instrumentation.opentelemetry.io/otel-dotnet-auto-runtime: "linux-x64"  # Optional as it's the default value

For Linux musl based images:

instrumentation.opentelemetry.io/inject-dotnet: "mw-agent-ns/mw-autoinstrumentation"
instrumentation.opentelemetry.io/otel-dotnet-auto-runtime: "linux-musl-x64"

Go

To enable auto-instrumentation for a Go application, you need to set the path to the executable and ensure the container has elevated permissions. Use the following annotations in the Pod specifications:

instrumentation.opentelemetry.io/inject-go: "mw-agent-ns/mw-autoinstrumentation"
instrumentation.opentelemetry.io/otel-go-auto-target-exe: "/path/to/container/executable"

Additionally, set the required security context for the container in the Pod specification (spec.containers) as shown below:

securityContext:
  privileged: true
  runAsUser: 0

Resource Attributes

If you want to add resource attributes to the traces generated by auto-instrumentation, you can add annotations in the following format to your Pod specification.

resource.opentelemetry.io/your-key: "your-value"

your-key and "your-value" are the keys and values for the resource attributes you want to add. You can add multiple resource attributes by having multiple such annotations.

Advanced Configuration

The annotations described above for all the supported programming languages used mw-agent-ns/mw-autoinstrumentation as the value for the annotation. This is because the OTel Kubernetes Operator installed using Middleware installation instructions creates mw-autoinstrumentation Instrumentation CR in mw-agent-ns namespace.

You can also create your own Instrumentation CR in any namespace and use them in the auto-instrumentation annotations.

Below are the all possible values for the auto-instrumentation annotation:

• "true": Inject an Instrumentation CR from the current namespace. It is expected that you will have only one Instrumentation CR defined in the current namespace. The behavior could be unpredictable if you have multiple Instrumentation CRs defined in the current namespace.
• "my-instrumentation": Inject a specific Instrumentation CR with name my-instrumentation from the current namespace.
• "my-other-namespace/my-instrumentation": Inject a specific Instrumentation CR with name my-instrumentation from namespace my-other-namespace. This is the option used in all our examples above with Instrumentation CR name “mw-otel-auto-instrumentation” and namespace mw-agent.
• "false": Do not inject an Instrumentation CR. This is useful for temporarily stopping instrumentation.

Explore your data on Middleware

Once the auto-instrumentation setup is complete and the relevant Pods are restarted, the data should start flowing to Middleware for any new requests made to your applications. You can explore APM traces data by going to the APM section of your Middleware account.

Uninstall

helm uninstall mw-auto --namespace mw-autoinstrumentation

helm uninstall cert-manager --namespace cert-manager

helm uninstall mw-auto --namespace mw-autoinstrumentation

MW_UNINSTALL_CERT_MANAGER=true MW_API_KEY="<MW_API_KEY>"
bash -c "$(curl -L https://install.middleware.io/scripts/mw-kube-auto-instrumentation-uninstall.sh)"

MW_API_KEY="<MW_API_KEY>"
bash -c "$(curl -L https://install.middleware.io/scripts/mw-kube-auto-instrumentation-uninstall.sh)"

Troubleshooting

Google Kubernetes Engine

If you are using GKE private cluster, you will need to add a firewall rule that allows your GKE control plane CIDR block access to port 9443/tcp on worker nodes.

Use the command below to find out GKE control plane CIDR block

gcloud container clusters describe <CLUSTER_NAME> \
  --region <REGION> \
  --format="value(privateClusterConfig.masterIpv4CidrBlock)"

• Replace <CLUSTER_NAME> with the name of your cluster.
• Replace <REGION> with the region of your cluster.

For example,

gcloud container clusters describe demo-cluster --region us-central1-c --format="value(privateClusterConfig.masterIpv4CidrBlock)"

# Example output: 172.16.0.0/28

Then you can add a firewall rule to allow ingress from this IP range and TCP port 9443 using command below

gcloud compute firewall-rules create cert-manager-9443 \
  --source-ranges <GKE_CONTROL_PLANE_CIDR> \
  --target-tags ${GKE_CONTROL_PLANE_TAG}  \
  --allow TCP:9443

<GKE_CONTROL_PLANE_CIDR> and <GKE_CONTROL_PLANE_TAG> can be found by following the steps in the GKE firewall docs.

More information can be found in the Official GCP Documentation. See the GKE documentation on adding rules and the Kubernetes issue for more detail.

Applications not detected:

• Check if your application uses a supported programming language.
• For multi-container pods, ensure main application container is first in the pod spec.
• Ensure applications are running and healthy.
• Check language detector logs for potential issues.
• Restart the language detector if detection problems persist.

Instrumentation not working:

• Ensure the namespace is enabled for auto-instrumentation.
• Confirm the application was restarted after enabling instrumentation.
• Confirm required annotations are present in object YAML.
• Examine webhook admission errors and mw-autoinstrumentation namespace component logs.
• Verify network connectivity to Middleware endpoints.
• Check opentelemtery operator logs.