Generating flame graphs on containerized workloads requires a bit of fiddling around with reasons (and solutions) discussed thoroughly here and here. As the title suggests, we’ll try to apply the methods above but modify it slightly to make it work in the context of workloads (scoped to Java-based applications) running inside a Kubernetes cluster.
The entire idea is illustrated below:
The way it works is as follows:
Via the “profiler” pod, run a container (using the
docker.sockwe mounted earlier) and explicitly set the PID Namespace to join the target pod’s namespace via the
docker run -it --pid=container:<container_id> <perf-container-image>
It is also possible to let Kubernetes handle the PID namespace gymnastics but it will involve modification of the deployment resource.
Exec into the container you just created and perform the actual profiling:
perfon PID 1 [**] .
perf record -F 99 -a -g -p 1 -- sleep 60
java -cp attach-main.jar:tools.jar net.virtualvoid.perf.AttachOnce PID 1
3.3 Generate the Flame Graph.
perf script -f | FlameGraph/stackcollapse-perf.pl | FlameGraph/flamegraph.pl --color=java --hash > flamegraph.svg
3.4 You may need to do a combination of
kubectl cpto get the generated flame graph out of the container and into your machine.
To make sure that everything is working accordingly, I created a sample Spring Boot application and then used Christopher’s sample code for comparison. Sure enough I got the following graph (interactive version):