Back to blog
Jan 27, 2026
5 min read

OSTEP Study Notes: Limited Direct Execution

Notes on limited direct execution: Running processes effectively while retaining control over the CPU

Limited Direct Execution

Run the program directly on the CPU. Direct execution has the advantage of being fast; the program runs natively on the hardware CPU. The process should be able to perform I/O and some other restricted operations without giving the process complete control over the system.

Protected Control Transfer

The hardware assists the OS by providing different modes of execution. In user mode, applications don’t have full access to the hardware resources. In kernel mode, the OS has access to the full resources of the machine. Special instruction to trap into the kernel andreturn-from-trap back to user-mode programs are also provided, as well instruction that allow the OS to tell the hardware where the trap table resides in memory.

When a user process wants to perform a privileged operation it should perform a system call. To execute a system call, a program must execute a special trap instruction. This instruction simultaneously jumps into the kernel and raises the privilege level to kernel mode; once in the kernel, the system can now perform whatever privileged operations are needed (if allowed)., and do the required work for the calling process. When finished, the OS calls a special return-from-trap instruction, which returns into the calling program while simultaneously reducing the privilege level back to user mode. The hardware need to be a bit careful when executing a trap, in that it must make sure to save enough of the caller’s registers in order to be able to return correctly when the OS issues the return-from-trap instruction.

How does the trap know which code to run inside the OS? The kernel sets a trap table at boot time. One of the first things the OS does is to tell the hardware what code to run when certain exceptional events occur. The Os informs the hardware of the locations of these trap handlers. Once the hardware is informed, it remembers the location of these handlers until the machine is next rebooted. Being able to execute the instruction to tell the hardware where the trap tables are is a very powerful capability - It is also a privileged operation.

OS (Kernel mode)HardwareProgram (User mode)
Create entry for process list
Allocate memory for the program
Load the program into memory
Setup user stack
Fill kernel stack with registers/PC
Return-from-trap
Restore registers from kernel
Stack
Move to user mode
Jump to main
Run main
Call system call
Trap into OS
Save registers to kernel stack
Move to kernel mode
Jump to trap handler
Handle trap
Do work of the system call
Return-from-trap
Restore registers from kernel stack
Move to user mode
Jump to PC after trap
Return from main
Trap (via exit())
Free memory of process
Remove from process list

Switching Between Processes

Cooperative Approach - Wait for system calls

The OS trusts the processes of the systems to behave reasonably. Processes that run for too long are assumed to periodically give up the CPU so that the OS can decide to run some other task. Most processes transfer control of the CPU to the OS quite frequently by making system calls e.g open a file and subsequently read it, or create a new process.

OSs often have to deal with misbehaving processes, those that either through design or accident attempt to do something they shouldn’t. In modern systems, the way the OS tries to handle such malfeasance is to simply terminate the process.

Non-Cooperative Approach - The OS takes control

The addition of a timer interrupt gives the OS the ability to run again on a CPU even if processes act in a non-cooperative fashion. This hardware feature is essential in helping the OS maintain control of the machine.