How computers work, lesson 2, about 22 minutes
Processes and the kernel
A program is a file. A process is that file brought to life, with an ID, memory, open files and a parent. This lesson takes one apart and finds the program that runs all the others.
By the end, you can
- Read any process's vital signs straight from /proc.
- Explain fork and exec, and why zombies exist.
- Fix the container that takes 10 seconds to stop.
You type node server.js and press Enter. What exists now that didn't exist a second ago?
Not the output. The things the machine had to create, track and eventually clean up.
Before you pressed Enter, node was a file on disk: a few dozen megabytes of machine code and data, sitting still. Nothing about a file runs. After you press Enter, there is something new on the machine: a process.
The difference matters. You can run the same program ten times and get ten processes, each with its own memory, its own open files and its own ID. Kill one and the other nine don't notice. The file never changed.
Reading a process from the inside
On Linux, the kernel publishes what it knows about every process as files under /proc. Nothing there is stored on disk. When you read /proc/7/status, the kernel writes the answer on the spot. Open a Linux shell and look at one.
inside: docker run -it --rm alpine shsleep 300 & echo $!7
1. Start a process in the backgroundThe & runs it in the background. $! is the PID the kernel just gave it.inside: docker run -it --rm alpine shgrep -E '^(Name|State|Pid|PPid|Uid|VmRSS)' /proc/7/statusName: sleep State: S (sleeping) Pid: 7 PPid: 1 Uid: 0 0 0 0 VmRSS: 876 kB
2. Ask the kernel about itIt's asleep, waiting on a timer. Its parent is PID 1, the shell. It runs as root (UID 0) and uses about 876 KB of RAM.inside: docker run -it --rm alpine shls -l /proc/7/fd0 -> /dev/pts/0 1 -> /dev/pts/0 2 -> /dev/pts/0
3. List its open filesThree file descriptors: stdin, stdout and stderr, all pointing at your terminal. Every process starts with these three.
That's three of its parts. Here is the whole thing, taken apart.
Identity
Its PID, its parent's PID and the user it runs as. Permissions are checked against this user on every file it opens.
grep -E '^(Pid|PPid|Uid)' /proc/7/statusPid: 7 PPid: 1 Uid: 0 0 0 0
Every command reads from /proc/7, the sleep process from Trace 1. Run them in your own container and swap in your PID.
The kernel keeps all of this in one structure per process, called task_struct. /proc is a window onto it.
Check 1
You start the same program twice. Which of these do the two processes share?
Every process has a parent
A new process is never created from nothing. An existing process makes a copy of itself with a system call traditionally named fork (on modern Linux, clone). The copy then replaces its own program with a new one using exec. That's how your shell runs ls: it forks, the child execs ls, and the parent waits.
You can watch it happen. strace prints every system call a process makes. Give the container permission to trace, install it, and watch a shell run two commands.
on your Macdocker run -it --rm --cap-add SYS_PTRACE alpine sh
1. Start a shell that's allowed to traceTracing another process is a privilege. SYS_PTRACE grants it to this container only.inside: docker run -it --rm alpine shapk add -q stracestrace -f -e trace=clone,execve,wait4 sh -c 'ls / > /dev/null; echo done'execve("/bin/sh", ["sh", "-c", ...]) = 0 clone(child_stack=NULL, flags=SIGCHLD) = 14 [pid 13] wait4(-1 <unfinished ...> [pid 14] execve("/bin/ls", ["ls", "/"]) = 0 [pid 14] +++ exited with 0 +++ <... wait4 resumed> = 14 done
2. Read the system callsThe shell (PID 13) clones itself. The child, PID 14, execs ls. The parent waits until ls exits, then carries on and prints done.
The dead child nobody collected
That last step, the parent calling wait, is not optional. When a process exits, the kernel keeps a small record of it, mostly its exit code, until the parent collects it. Until then the dead process is a zombie. It uses no CPU and almost no memory, but it holds on to its PID.
inside: docker run -it --rm alpine shsh -c 'sleep 1 & exec sleep 30' &
1. A parent that never waitsThe inner shell starts a short sleep as its child, then execs into sleep 30. sleep 30 never calls wait, so nobody will collect the child.inside: docker run -it --rm alpine shsleep 2; ps -o pid,ppid,stat,commPID PPID STAT COMMAND 1 0 S sh 7 1 S sleep 9 7 Z sleep
2. Look a second after the child diesPID 9 has exited, but its state is Z. Its parent, PID 7, is busy sleeping and will never collect it.
Why your container takes ages to stop
Inside a container, your program is usually PID 1, and PID 1 is special. Linux won't apply the default action of a signal to PID 1 unless that process installed a handler for it. So when docker stop sends SIGTERM to a program that never planned for it, nothing happens. Docker waits out its grace period, then sends SIGKILL, which can't be ignored.
on your Macdocker run -d --name t1 alpine sleep 300time docker stop -t 10 t1real 10.10
1. Stop a plain containersleep is PID 1 and has no SIGTERM handler, so the signal is ignored. Docker gives up after the 10 second grace period and kills it.on your Macdocker inspect t1 --format '{{.State.ExitCode}}'137
2. Check how it died137 is 128 plus 9. It was killed by signal 9, SIGKILL. It never got the chance to shut down cleanly.on your Macdocker run -d --name t2 --init alpine sleep 300time docker stop -t 10 t2real 0.09
3. Add --init and try again--init makes a tiny init program PID 1. It forwards SIGTERM to sleep, which is no longer PID 1, so the default action applies and it exits at once with code 143 (128 plus 15).
Check 2
A Node container takes about 10 seconds to stop and exits with code 137. What's the most likely cause?
The program that runs the others
The kernel is a program too, but it doesn't show up in ps as one process. It's the code that runs when a process asks for something only the machine can do: read a file, send a packet, wait for a timer. The CPU has two modes. In user mode, your program runs and can only touch its own memory. When it makes a system call, the CPU switches to kernel mode, runs kernel code with full access to the hardware, then switches back.
Our sleeping process from Trace 1 is, right now, stopped in the middle of a system call. The kernel will tell you exactly where.
inside: docker run -it --rm alpine shcat /proc/7/wchanhrtimer_nanosleep
1. Where is it waiting?wchan names the kernel function the process is blocked in. sleep asked the kernel to wake it later, and the kernel parked it on a high resolution timer.inside: docker run -it --rm alpine shcut -d' ' -f1 /proc/7/syscall101
2. Which system call is it in?101 is nanosleep on 64 bit ARM. On an Intel machine the same call is number 35. Numbers differ by CPU, names don't.
Time spent in the kernel is real time your program pays for. time splits it into user time, running your own code, and sys time, running the kernel on your behalf. Watch what happens when a program makes millions of tiny system calls.
inside: docker run -it --rm alpine shtime sh -c 'i=0; while [ $i -lt 300000 ]; do i=$((i+1)); done'real 0m 0.28s user 0m 0.28s sys 0m 0.00s
1. Pure computationCounting needs no help from the kernel. All the time is user time.inside: docker run -it --rm alpine shtime dd if=/dev/zero of=/dev/null bs=1 count=1000000real 0m 0.30s user 0m 0.08s sys 0m 0.21s
2. Copy 1 MB, one byte at a timeEach byte is one read and one write system call: two million trips into the kernel. Most of the time is now sys time.inside: docker run -it --rm alpine shtime dd if=/dev/zero of=/dev/null bs=1M count=1real 0m 0.00s user 0m 0.00s sys 0m 0.00s
3. Copy the same 1 MB in one goTwo system calls instead of two million. The work didn't change, only the number of times we crossed into the kernel.
Check 3
A service reads a large file one byte at a time and is slow. time shows most of its time as sys. What's the cheapest fix?
One core, many processes
A CPU core runs one thing at a time. Your laptop has hundreds of processes, so the kernel's scheduler takes turns: it runs a process for a few milliseconds, saves its registers, loads another, and repeats, thousands of times a second. Each swap is a context switch. Thevoluntary_ctxt_switches line you saw in the teardown counts the times a process gave up the CPU itself, usually to wait for something. Involuntary switches are the times the scheduler took it away.
That's the bottom of this lesson's stack: a file becomes a process, the process asks the kernel for everything, and the kernel shares out the CPU. The next lesson looks closely at the door between them, the system call.
Keep these
- A program is a file. A process is a running instance with its own PID, memory, open files, parent and state.
- On Linux, /proc/<pid> shows everything the kernel knows about a process, generated live.
- New processes come from fork (clone) and exec. A parent must wait for its children, or they linger as zombies.
- PID 1 ignores signals it hasn't asked to handle. Use --init, or handle SIGTERM, so containers stop cleanly.
- System calls cross into the kernel, and each crossing costs time. Fewer, bigger calls are faster.
Sources
Primary references only: the documentation the systems are built from.
- Operating Systems: Three Easy Pieces, chapter 5: Process APIArpaci-Dusseau, University of WisconsinThe textbook account of fork, exec and wait.
- Operating Systems: Three Easy Pieces, chapter 6: Limited Direct ExecutionArpaci-Dusseau, University of WisconsinUser mode, kernel mode and context switches.
- proc(5): the /proc filesystemLinux man-pages
- fork(2)Linux man-pages
- execve(2)Linux man-pages
- wait(2)Linux man-pagesIncludes the definition of a zombie.
- signal(7)Linux man-pages
- pid_namespaces(7)Linux man-pagesWhy PID 1 in a namespace ignores signals it has no handler for.
- docker container run: --initDocker documentation
- strace(1)Linux man-pages