Disk full, no files
df says 100%, du finds nothing.
You learn: a deleted file keeps its space while something holds it open.
Open the lab →Prod is down.
Your terminal.
Fix it.
Hands-on troubleshooting labs for DevOps and SRE engineers. We hand you a broken production system; you SSH in from your own terminal and bring it back.
Start a lab
The first labs are free.
Firing
02:14 UTC
checkout is returning 503
~ $ fixmyprod start disk-full --attach
creating session… variant 2… ready in 41s
root@web-01:~# df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p1 20G 20G 0 100% /
root@web-01:~# du -xsh / 2>/dev/null
4.1G /
root@web-01:~# exit
~ $ fixmyprod check
solve probe.http FAIL got 503, want 200 stable 60s
~ $
01
Your terminal, not a browser tab
Plain ssh from the shell you already use: your keys, your config, your muscle memory.
02
Failures from real incidents
More than one cause, a first lead that is wrong, noise that looks like the problem. Like the pages you actually get.
03
Load and scale
Make nginx hold the traffic we send at it. Untangle scheduling on a 200-node cluster. Not just puzzles on one host.
Labs
The binary was never installed
The service never started because apt update failed at boot. You can curl the mirror just fine.
You learn: how a half-working network lies to you.
Needs 10k rps, holds 2k
The box sits at 12% CPU and refuses connections. Every fix raises the ceiling, until you hit the next one.
You learn: where web servers really run out.
Half the pods are Pending
200 nodes, plenty of room, and the scheduler still says no. kubectl only: there is no host to SSH into.
You learn: to read the scheduler's reasons instead of guessing them.
Start a lab
The first labs are free. A subscription comes later.
-
01
Install the CLI
brew install fixmyprod/tap/fixmyprod -
02
Sign in with GitHub or Google
fixmyprod login -
03
Start a lab and SSH in
fixmyprod start disk-full