Building an Operating System in Rust: Part 1 - Zero-Cost Bare-Metal Freestanding Binary

Building an Operating System in Rust: Part 1 — Zero-Cost Bare-Metal Freestanding Binary

Building an operating system is the ultimate proving ground for software engineering. Historically, operating system kernels were written almost exclusively in C and Assembly language because low-level systems programming required direct hardware manipulation, manual pointer arithmetic, and absolute control over hardware memory layouts.

However, writing an operating system kernel in C introduces persistent security vulnerabilities. The vast majority of critical vulnerabilities in modern enterprise operating systems—including Linux, Windows, and macOS—stem from memory safety bugs: buffer overflows, dangling pointers, double-free faults, and data races.

Rust changes this dynamic forever. By enforcing strict ownership, borrowing, and lifetime rules at compile time, Rust delivers the bare-metal execution speed of C with compile-time memory safety.

This guide marks Part 1 of our comprehensive series: Building an Operating System in Rust from Scratch. In this article, you will learn how to decouple Rust from the standard operating system environment, strip away the runtime, and build a bare-metal #![no_std] freestanding executable that boots directly on bare computer hardware without an underlying OS.

1. What is a Freestanding (Bare-Metal) Binary?

When you write a conventional Rust application, Cargo links your code to the Rust standard library (std). The standard library provides foundational abstractions:

• Dynamic heap allocation (Vec, String, Box, HashMap)
• Threading and synchronization primitives (std::thread, std::sync::Mutex)
• File system I/O (std::fs::File)
• Network sockets (std::net::TcpStream)
• Standard input/output streams (println!, eprintln!)

All of these abstractions rely on underlying system calls provided by the host operating system (POSIX libc on Linux/macOS or kernel32.dll on Windows).

Architectural Transition: In a hosted runtime, your application makes calls to std, which delegates to the host OS kernel via system calls (libc, kernel32.dll). In our bare-metal #![no_std] kernel, execution links strictly against core and issues instructions directly to physical CPU registers with zero runtime layers.

When building an operating system kernel, there is no host operating system to provide system calls. We must create a freestanding binary (also known as a bare-metal #![no_std] program) that runs directly on the CPU without assuming the existence of an operating system runtime or C runtime (crt0).

2. Setting Up the Rust Bare-Metal Project

To begin, initialize a new Rust binary package using Cargo:

cargo new --bin adoreka_os_kernel
cd adoreka_os_kernel

By default, Cargo creates a standard binary with a main.rs file. Let's inspect the initial Cargo.toml:

[package]
name = "adoreka_os_kernel"
version = "0.1.0"
edition = "2024"

[profile.dev]
panic = "abort"

[profile.release]
panic = "abort"

Notice the panic = "abort" configuration. In standard Rust, when a thread panics, the runtime unwinds the stack to run destructors. Stack unwinding requires complex personality functions and landing pads provided by OS-level libraries. In a bare-metal kernel, we cannot unwind the stack, so we configure Rust to immediately abort on panic.

3. Disabling the Standard Library: The #![no_std] Attribute

Open src/main.rs. To instruct the Rust compiler (rustc) not to link the standard library, add the #![no_std] crate-level attribute at the very top:

#![no_std]

fn main() {
    // Will fail to compile initially
}

If you execute cargo build right now, the Rust compiler will report two fundamental errors:

1. error: #[panic_handler] function required, but not found: The standard library normally provides a panic handler that prints error messages to the terminal console. Without std, we must define our own custom panic handler.
2. error: requires 'start' lang_item: A standard C runtime initialization routine calls main. Without the C runtime, execution must begin at a customized entry point.

4. Implementing a Custom Panic Handler

In Rust, the core library is a subset of the standard library that does not depend on any operating system features. It contains essential primitives such as Result, Option, iterators, mathematical operations, and low-level memory utilities.

We can define a custom panic handler using core::panic::PanicInfo:

use core::panic::PanicInfo;

/// This function is invoked by the Rust runtime whenever a panic occurs.
#[panic_handler]
fn panic(_info: &PanicInfo) -> ! {
    // Loop indefinitely to halt the CPU execution
    loop {}
}

The exclamation mark (!) signifies a diverging function—a function that never returns to its caller. Since our kernel cannot exit to an operating system when an unrecoverable error occurs, halting the CPU inside an infinite loop is the safest course of action.

5. Overwriting the Entry Point: Eliminating crt0

In standard computer architectures, execution does not begin directly at fn main(). Instead, the operating system loader jumps to a C runtime initialization entry point typically named _start. This routine configures the stack pointer, initializes registers, calls global constructors, and finally calls your application's main() function.

Because our bare-metal kernel does not possess a C runtime loader, we must disable main and declare our own bare-metal entry point:

#![no_std]
#![no_main]

use core::panic::PanicInfo;

/// Our custom bare-metal entry point.
/// We use `extern "C"` to enforce the standard C calling convention.
#[no_mangle]
pub extern "C" fn _start() -> ! {
    // The kernel code begins execution here

    loop {}
}

#[panic_handler]
fn panic(_info: &PanicInfo) -> ! {
    loop {}
}

Let's dissect the attributes used:

• #![no_main]: Tells the Rust compiler that the application does not follow the standard main entry point convention.
• #[no_mangle]: Disables Rust symbol name mangling. By default, the Rust compiler scrambles function names (e.g., _ZN17adoreka_os_kernel6_start17h9b2a...) to enforce uniqueness. Disabling name mangling ensures the linker can locate the exact symbol name _start.
• extern "C": Instructs the compiler to follow the standard C Application Binary Interface (ABI) calling convention, allowing bootloaders or assembly routines to call this function reliably.

6. Target Specifications: Compiling for a Bare-Metal Target

If you attempt to compile this code using cargo build on Windows, Linux, or macOS, the default host linker will still try to link against host system libraries. To compile for bare-metal hardware, we must target a target architecture that has no underlying OS.

Rust provides first-class support for bare-metal compilation targets. For x86_64 computer architectures, we can use the x86_64-unknown-none target:

rustup target add x86_64-unknown-none

Now, build your bare-metal freestanding kernel:

cargo build --target x86_64-unknown-none

The build will complete with zero errors! Rust will produce a pure ELF executable in target/x86_64-unknown-none/debug/adoreka_os_kernel.

7. Inspecting the Bare-Metal Binary

To verify that our binary contains zero operating system dependencies, we can inspect its ELF section headers using rust-objdump:

cargo install cargo-binutils
rustup component add llvm-tools

cargo objdump --bin adoreka_os_kernel -- -h

Output:

adoreka_os_kernel:	file format elf64-x86-64

Sections:
Idx Name          Size     VMA              Type
  0               00000000 0000000000000000 
  1 .text         0000000a 0000000000201000 TEXT
  2 .comment      00000024 0000000000000000 

The resulting binary contains only the bare .text code section containing our _start loop instructions. There are no shared library dependencies, no dynamic linkers, and zero external runtime dependencies.

Summary & What Comes Next

In this initial chapter, we established the fundamental building block of our Rust operating system:

In Part 2: VGA Text Mode Architecture & Volatile Framebuffers, we will connect our freestanding binary to physical hardware by building a type-safe VGA buffer driver to print colored text directly to the computer screen.