Skip to main content

⎈ Cilium Tetragon - eBPF-based Security Observability and Runtime Enforcement

πŸ™Œ Overview​

The containers and their infrastructure are immutable. It means it's very difficult to detect certain attacks, vulnerabilities, and detections using traditional tools and technologies. In this scenario, we will see how we can leverage the popular open-source tool like Cilium Tetragon to detect and perform runtime security monitoring using the tracingpolicy in action.

Scenario-21 Tetragon Overview

By the end of the scenario, we will understand and learn the following

  1. Deploying the helm chart into the Kubernetes cluster
  2. Performing the log analysis and detection of security events in the Kubernetes Cluster
  3. Use, analyze and detect security issues in near real-time using Cilium Tetragon

⚑️ The story​

This scenario is to deploy runtime security monitoring & detection for containers and Kubernetes resources. Also, explore and see how we can detect certain issues and perform detections using tracingpolicy based on attacker triggers.


Make sure you run the following deployment using Helm with v3. Refer to helm installation

  • To deploy the Cilium Tetragon helm chart run the following commands
helm repo add cilium
helm repo update
helm install tetragon cilium/tetragon -n kube-system
  • You can verify that the Tetragon pods are in running state by running the following command
kubectl rollout status -n kube-system ds/tetragon -w

Scenario 21 helm tetragon setup

🎯 Goal​


Deploy the Cilium Tetragon and detect security events in near real-time using the tracingpolicy and events.

πŸͺ„ Hints & Spoilers​

✨ Want to know more?
I think it's best to refer to the official Cilium Tetragon documentation here πŸ™Œ

πŸŽ‰ Solution & Walkthrough​

🎲 Method 1​


Cilium’s new Tetragon component enables powerful realtime, eBPF-based Security Observability and Runtime Enforcement.

Tetragon detects and is able to react to security-significant events, such as

  • Process execution events
  • System call activity
  • I/O activity including network & file access

When used in a Kubernetes environment, Tetragon is Kubernetes-aware - that is, it understands Kubernetes identities such as namespaces, pods and so-on - so that security event detection can be configured in relation to individual workloads.

  • Get more details about the Tetragon deployment by running the following command
kubectl get pods -n kube-system --selector
  • Manually obtaining the logs from the Tetragon using the following command
kubectl logs -n kube-system -l -c export-stdout -f 
  • Now, let's use the system-monitor pod and try privileges escalation to the host system and see if that detects by Tetragon
export POD_NAME=$(kubectl get pods -l "app=system-monitor" -o jsonpath="{.items[0]}")
kubectl exec -it $POD_NAME bash
  • Let's read the sensitive file /etc/shadow
cat /etc/shadow

Scenario 21 tetragon detecting /etc/shadow

  • You can also see these events in a nicer way using the official tetra cli client in your local system. Refer to the documentation for specific binary as per your OS and Architecture
tar -xvzf tetra-linux-amd64.tar.gz
  • Now you can run the following command to pass the output of the Tetragon events to tetra cli locally to see in a nicer way
kubectl logs -n kube-system -l -c export-stdout -f | ./tetra getevents -o compact --namespace default --pod system-monitor-deployment-*

Scenario 21 tetragon detecting /etc/shadow using tetra cli


tetra cli provides the context awareness of the Kubernetes, namespaces and other details like processes, etc. When querying you can even limit them to processes, namespace, pod, even regex supported.

  • Let's take a spin for detecting the privilege escalation attacks using the Tetragon. You can perform the container escape using the system-monitor pod to gain host system access by running the following commands
# Get into the system-monitor pod
export POD_NAME=$(kubectl get pods -l "app=system-monitor" -o jsonpath="{.items[0]}")
kubectl exec -it $POD_NAME bash

# Exploit to gain the host system access using nsenter
nsenter -t 1 -m --uts --ipc --pid

Scenario 21 performing container escape using nsenter

  • As you can see following, Tetragon in action detecting these attacks in near real-time
kubectl logs -n kube-system -l -c export-stdout -f | ./tetra getevents -o compact --namespace default --pod system-monitor-deployment-*

Scenario 21 tetragon detecting container escape

  • Hooray πŸ₯³ , now we can see that Tetragon detected the attack and notified using the events

πŸ”– References​