Understanding User and Kernel Mode

The green line is total CPU time; the red line is Kernel time. The gap between the two is User time.

Not exactly.

Total CPU time means wall clock time, which is the total time the program took to run. CPU time, as you said, is the time that the CPU was running in level 0 (kernel mode) and User time is the time that the process was executing (ie. using the CPU) in level 3 (user mode).

There is a huge difference of User mode and total time because the process can be waiting for a resource and not use the CPU at all. To clarify, check the example below:

  1. Create a FIFO:
    $ mkfifo foo

  2. ‘cat’ the fifo (ie. read from it)
    $ time cat foo

It’ll block, waiting from data or EOF, but there’s nothing writing to the FIFO so it’ll wait forever, without a single User mode CPU utilization.

  1. In another terminal write something to the FIFO:
    $ echo “foo” foo

The first terminal will then write “foo” and exit (echo sends EOF). The time reported will be something like this:

real 0m12.239s
user 0m0.000s
sys 0m0.008s

Which means it spent 0.008 seconds to send the output throughout the FIFO, 0 seconds in user mode and 12.24 seconds waiting.