Weeknote 7: Memray + k8s

I wanted to track a memory leak in a Python program. The program was leaking only in production and so I had to figure out how to use Memray to attach to a process in Kubernetes. There were a few hurdles on the way, so here’s what I did.

Add Memray to your container image. We use Poetry for our Python projects, so I added Memray as a dependency.
```
poetry add memray
```
If you’re using the official Docker Python images as a base, be sure to use the non-slim variant. The debug symbols have been stripped from the slim variant.
Memray relies gdb (or alternatively lldb), so install that. We’ll also need the setcap binary, so install that too. On Debian-based images:
```
apt-get install gdb libcap2-bin
```
Unless you’re running a privileged container, gdb needs CAP_SYS_TRACE capability to work. As explained in py-spy docs, add it to Deployment.spec.template.spec.containers in your k8s spec
```
securityContext:
  capabilities:
    add:
      - SYS_PTRACE
```
Use setcap to add CAP_SYS_PTRACE to the permissible and effective capability sets of the gdb binary.
```
setcap cap_sys_ptrace+pe /usr/bin/gdb
```
Use kubectl exec to attach Memray to your Python process. Typically it’s PID is 1.
```
kubectl exec -it <your pod> -- memray attach 1
```

This starts memray’s live TUI. In practice you’ll want to generate a flamegraph, but I’ll let you to figure that out.

Note that CAP_SYS_PTRACE can be used for privilege escalation.

Something you should not do is to use setcap to set cap_sys_ptrace on the Python binary. The trouble is that it makes the actual process you want to inspect non-dumpable. As explained by PR_SET_DUMPABLE man page, a process’s dumpable attribute can get set to 0 when it executes a program that has capabilities:

The process executes (execve(2)) a program that has file capabilities (see capabilities(7)), but only if the permitted capabilities gained exceed those already permitted for the process.

I did this mistake because I thought that Memray itself needs the capability. That’s not the case since it relies on gdb.

Debugging notes

/proc/PID/status is your friend - see the lines starting with Cap for capability sets. You can use capsh --decode to make sense of the numebrs.
If /proc/PID/mem is owned by root even though the uid of the process is something else, that means the process is non-dumpable

Recommendation: Making music with C64

I bumped into Linus Åkesson’s video Making 8-bit Music From Scratch at the Commodore 64 BASIC Prompt and I recommend watching it. He shows how to code a small sequencer on Commodore 64 and makes some music with it.

Photo: Fog over reed in a sea shore on an autumn morning. Sunrise colors the clouds pink.