Lecture 5: Testing, tooling and development practices in Rust

About me

I am a doctoral researcher in scientific machine learning at Chair of scientific computing (Chair 05 at TUM School of CIT).

focusing on novel neural network architectures for scientific tasks using randomized algorithms
used Rust in implementing and training neural networks, web (backend) and Solana programs (blockchain)
links:

homepage at www.cs.cit.tum.de/sccs/personen/atamert-rahma
github at github.com/alphagergedan
linkedin at www.linkedin.com/in/atamert-rahma-6888a419b/
hobby blog at alphagergedan.github.io

Why testing matters?

"Testing leads to failure, and failure leads to understanding." — Burt Rutan

Testing ensures code correctness and reliability.
Rust encourages testing through its ecosystem and tooling.
Use your program, try to break it, and write tests for it.

Writing tests

Directory structure in Rust

.
├── Cargo.lock
├── Cargo.toml
├── src/
│   ├── lib.rs
│   ├── main.rs
│   ├── module_1/
│   │   ├── mod.rs
│   │   └── [...]
│   └── [...]
└── [...]

Unit tests are usually placed inside the modules they are testing:

Inline in the same file as the module (e.g., src/module_1/mod.rs)
Or in a separate file within the module directory (e.g., src/module_1/tests.rs)

Writing tests

Anatomy of a Test function

#[test]
fn test_name() {
    // Setup: Prepare variables or context
    [...]

    // Assertion: Ensure a condition is true
    assert!(...);

    // Assertion: Check equality of values
    assert_eq!(...);

    // Assertion: Check inequality of values
    assert_ne!(...);
}

⇒ Run your tests using `$ cargo test` on the console

Writing tests

Example (inline unit tests): `src/add/mod.rs`

pub fn add(a: i32, b: i32) -> i32 {
    a + b
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_add() {
        let result = add(2, 3);
        assert_eq!(result, 5);
    }
}

Writing tests

Example (separate unit tests): `src/add/mod.rs`

pub fn add(a: i32, b: i32) -> i32 {
    a + b
}

#[cfg(test)]
mod tests;

`src/add/tests.rs`

use super::*;

#[test]
fn test_add() {
    let result = add(2, 3);
    assert_eq!(result, 5);
}

Writing tests

Running the tests of a module

$ cargo test add

[...]
running 1 test
test add::tests::test_add ... ok
[...]

⇒ You can also run only a specific test with `$ cargo test add::test_add`

⇒ What about failing tests?

Writing tests

Failing test example

$ cargo test some_other_module

[...]
test some_other_module::tests::test_some_function ... FAILED

failures:

---- some_other_module::tests::test_some_function stdout ----
thread 'some_other_module::tests::test_some_function' panicked:
assertion `left == right` failed
  left: 0
 right: 1
[...]

assertion `left == right` failed may not give us enough information

⇒ Add custom failure messages

Writing tests

Adding custom failure messages: `src/some_other_module/tests.rs`

use super::*;

#[test]
fn test_some_function() {
    let result = some_function(); // 0
    assert_eq!(
        some_function(), 1,
        "{} must equal 1", result
    );
}

assertion `left == right` failed: 0 must equal 1

⇒ Useful for giving context about what an assertion means

⇒ Also works with `assert!`, `assert_ne!`

Writing tests

Checking for panics with `should_panic`

#[test]
#[should_panic]
fn test_divide_by_zero() {
    div(1, 0);
}

Specifying the expected message

#[test]
#[should_panic(expected = "Cannot divide by zero")]
fn test_divide_by_zero() {
    div(1, 0);
}

Writing tests

What if the function we want to test returns a `Result` type?

fn add_strings(a: &str, b: &str) -> Result<i32, String> {
    let num_a: i32 = a.parse()
        .map_err(|_| "Failed to parse left operand")?;
    let num_b: i32 = b.parse()
        .map_err(|_| "Failed to parse right operand")?;
    Ok(num_a + num_b)
}

You can use `Result` type in tests

#[test]
fn test_add_strings() -> Result<(), String> {
    let result = add_strings("2", "3")?;
    assert_eq!(result, 5, "Two plus three must equal five");
    Ok(())
}

Writing tests

Running tests parallel or consecutively

In Rust, tests are run in parallel by default

If your tests need to share some state, e.g. consecutively read, process and write to a file, then they cannot be run in parallel

$ cargo test -- --test-threads=1

--test-threads specifies the number of threads you want to use to the test binary

⇒ Specifying `--test-threads=1` means running the tests consecutively

Writing tests

Ignoring some tests unless specifically requested

#[test]
#[ignore]
fn expensive_test() {
    // code that takes an hour to run
}

⇒ `$ cargo test -- --ignored` to only run the ignored tests

⇒ `$ cargo test -- --include-ignored` to run all tests

Writing tests

Documentation tests

/// Adds two numbers.
///
/// # Examples
///
/// ```
/// use my_crate::add;
///
/// assert_eq!(add(2,2), 4);
/// ```
pub fn add(a: i32, b: i32) -> i32 {
    a + b
}

⇒ Doc tests are also run with $ cargo test, Rust ecosystem encourages testing through documentation!

⇒ This makes sure that examples within your documentation are up to date and working.

Writing tests

In Rust, Integration tests are entirely external to your code

.
├── Cargo.lock
├── Cargo.toml
├── src/
│   └── [...]
└── tests/
    └── integration_test.rs

⇒ Integration tests can only call functions that are part of your public API

Each file in the `tests/` directory is a separate crate

⇒ We need to bring our library into each test crate's scope

We do not need to annotate any code in `tests/` with `#[cfg(test)]`

`$ cargo test` runs first unit tests, then integration tests, and then the doc tests

Goal of integration tests: To test whether many parts of your library work together correctly

Writing tests

Example CLI application (add_cli)

`src/main.rs`

use add_cli::{add, parse};
use std::env;

fn main() -> Result<(), String> {
    let args: Vec<String> = env::args().collect();
    let left: i32 = parse(&args[1])?;
    let right: i32 = parse(&args[2])?;
    print!("Sum: {}\n", add(left, right));
    Ok(())
}

`$ cargo run -- 1 2` outputs `Sum: 3` with a newline

Writing tests

Example integration test for `parse` and `add`

`tests/test_parse_and_add.rs`

use add_cli::{add, parse};

#[test]
fn test_successful_parse_and_add() -> Result<(), String> {
    let num_str_left = "1";
    let num_str_right = "2";

    let left: i32 = parse(num_str_left)?;
    let right: i32 = parse(num_str_right)?;

    let sum = add(left, right);
    assert_eq!(sum, 3);

    Ok(())
}

Writing tests

Example end-to-end test for the app

`tests/test_add_cli.rs`

use std::process::Command;

#[test]
fn test_valid_input() {
    let output = Command::new(env!("CARGO_BIN_EXE_add_cli"))
        .arg("1").arg("2")
        .output()
        .expect("Failed to execute command");
    assert!(output.status.success());

    let stdout = String::from_utf8(output.stdout)
        .expect("Failed to parse stdout");
    assert_eq!(stdout, "Sum: 3\n");
}

Writing tests

Outlook: Unofficial testing tools

nextest: Faster and prettier test runner (doctests are not supported yet).

Very easy to get started with cargo-nextest

tarpaulin: Code coverage tool.

It can show how many lines of your code has been covered with tests, and which lines are uncovered:
22.58% coverage, 7/31 lines covered

proptest: Property-based testing.

It can automatically generate a wide range of complex test inputs to verify properties of your code.
If a test fails, it minimizes the failing input to the simplest case that it still reproduces the bug.

Rust development practices

Test-driven development (TDD)

As a concept: Write tests before you write the code.

Red, Green, Refactor cycle:

Write a failing test (Red).
Write the minimal code to pass the test (Green).
Refactor the code while keeping tests passing.

⇒ Forces you to think about what you want the code to do before you write it.

Rust development practices

How to start out with a skeleton in Rust?

`src/main.rs`

// The main executable will consist of many methods
// We can usually make many decision-mistakes here
// Example: Training a model to approximate a complex function.
//    1) Generate a dataset
//    2) Configure a model
//    3) Fit the model
//    4) Inspect the error and visualize the solution
fn main() {}

⇒ Create your skeleton using `todo!(message)`

⇒ TDD can provide safe and maintainable divide & conquer for the implementation of your application logic

⇒ Makes implementing complex applications more managable

Rust development practices

Starting with a skeleton using `todo!`

`src/main.rs`

fn main() {
    let input = 5;
    let result = square(input);
    println!("The square of 5 is {}", result);
}

fn square(x: i32) -> i32 {
    todo!("Implement square function");
}

$ cargo run gives

not yet implemented: Implement square function

Rust development practices

Adding debugging checks with `debug_assert!`, `debug_assert_eq!`

fn square(x: i32) -> i32 {
    debug_assert!(x.abs() < 1000,
        "Input is too large for this example");
    todo!("Implement square function");
}

⇒ Use `todo!` to make your code compile for your skeleton

⇒ `debug_assert!` is like assert but only runs in debug mode

⇒ `assert!` can be used too if you must abort in an unrecoverable state

You can also use the unofficial `debug_print` crate to have print macros that are not compiled in release builds

Rust development practices

Writing unit tests for your skeleton

#[test]
fn test_square() {
    assert_eq!(square(2), 4);
    assert_eq!(square(-3), 9);
    assert_eq!(square(0), 0);
}

Implement the function

fn square(x: i32) -> i32 {
    debug_assert!(x.abs() < 1000,
        "Input is too large for this example");
    x * x
}

⇒ Keep your functions small and focused

Rust development practices

Thinking in expressions

let status = if logged_in {
    "Active"
} else {
    "Inactive"
};

let state = loop {

    // game loop

    if game_over {
        break 1
    }
};

Rust development practices

Thinking in expressions

let grade = match score {
    90..=100 => "A",
    80..=89 => "B",
    70..=79 => "C",
    _ => "D",
};

⇒ Rust's expression-oriented design enables code that is modular and testable in isolation

Rust development practices

Use `mod` to organize your code into modules

`src/lib.rs`

pub mod math {
    pub fn square(x: i32) -> i32 {...}
}

⇒ If your module gets large, separate it into its own file using `src/math.rs`

⇒ You can also separate it into its own directory using `src/math/mod.rs`

⇒ Outlook: Workspaces / Having crates in your crates that share the same `Cargo.lock`

Rust development practices

Documentation practices: Use `//!` for module docs

`src/lib.rs`

// [Usually some License information]

//! This is the root module documentation for the crate.
//! It describes the overall purpose and functionality.

// Module definition, this is a normal comment
pub mod math;

Rust development practices

Documentation practices: Use `///` for function docs

`src/math.rs`

//! This is the documentation for the `math` module.

/// Docs for the square function
/// Calculates the square of a number.
pub fn square(x: i32) -> i32 {
    x * x
}

Rust development practices

More clean code practices: Unnesting match arms for readability

Nested match

match result {
    Ok(value) => process(value),
    Err(err) => match err {
        Error::NotFound => handle_not_found(),
        Error::PermissionDenied => handle_permission(),
    },
}

Unnested match

match result {
    Ok(value) => process(value),
    Err(Error::NotFound) => handle_not_found(),
    Err(Error::PermissionDenied) => handle_permission(),
}

Rust development practices

Nested match

let msg = match a {
    Some(x) => match b {
        Some(y) => format!("Both: {}, {}", x, y),
        None => format!("Only a: {}", x),
    },
    None => match b {
        Some(y) => format!("Only b: {}", y),
        None => String::from("Neither"),
    }
};

Rust development practices

Unnested match

let msg = match (a, b) {
    (Some(x), Some(y)) => format!("Both: {}, {}", x, y),
    (Some(x), None) => format!("Only a: {}", x),
    (None, Some(y)) => format!("Only b: {}", y),
    (None, None) => String::from("Neither"),
};

Formatting and linting

Code formatting with `rustfmt`

Before running `$ cargo fmt`

fn main() { let x=5;let y=10;println!("Sum: {}",x+y); }

After running `$ cargo fmt`

fn main() {
    let x = 5;
    let y = 10;
    println!("Sum: {}", x + y);
}

⇒ Outlook: You can customize the style through `rustfmt.toml`

Formatting and linting

`rustc` gives you warnings when you run `$ cargo build` to build your code

warning: variable `someNumber` should have a snake case name
  --> src/main.rs:23:9
   |
23 |     let someNumber = 12;
   |         ^^^^^^^^^^ help: convert the identifier to
                              snake case: `some_number`

Rust-analyzer

An implementation of Language Server Protocol for Rust
IDE support for Rust (code completion, definitions from docs etc.)
E.g., you can directly go to documentations and see example usage
Using Rust-analyzer: Install it and set up in your code editor (neovim, vscode etc.)

Formatting and linting

Linting with `clippy`

More than 500 lints
It can catch common mistakes and improve code quality
Great for idiomatic Rust and also sometimes can suggest optimizations

⇒ Leads to fewer bugs and better performance

Formatting and linting

Linting with `clippy`

let vec = vec![1, 2, 3];
if vec.len() > 0 {
    println!("Vector is not empty");
}

After running `$ cargo clippy`

warning: length comparison to zero
--> src/main.rs:27:8
 |
27 |     if vec.len() > 0 {
 |        ^^^^^^^^^^^^^ help: using `!is_empty` is clearer
                              and more explicit:
                              `!vec.is_empty()`

Benchmarking

Benchmarking provides a feedback mechanism for your refactored code in TDD!

First, change the performance of a Rust program without changing its code

⇒ by changing its build configuration at `Cargo.toml`

Common pitfall: First ensure that you are using the release build via

$ cargo build --release

Common trade-off: compile time vs. runtime speed

Prefer $ cargo check instead of $ cargo build to quickly ensure your code still compiles during development

Omit the --release flag during development for builds: Debug builds are usually faster than release builds

⇒ Depending on the metric we would like to optimize there are different build strategies

Benchmarking

⇒ Minimizing runtime speed

Disabling codegen units (used to parallelize compilation)

[profile.release]
codegen-units = 1

Increasing Link-Time optimization (LTO)

[profile.release]
lto = true    # or "thin". This is default for release
# lto = "fat" # Uncomment for more aggresive LTO

Benchmarking

CPU specific instructions (e.g. AVX SIMD for x86-64 CPUs) at the cost of compatibility

$ RUSTFLAGS="-C target-cpu=native" cargo build --release

⇒ compare outputs of

`$ rustc --print cfg` and `$ rustc --print cfg -C target-cpu=native`

to see if the CPU features are being detected correctly

Outlook: using an alternative heap allocator (e.g. jemalloc, mimalloc), profile-guided optimization

Benchmarking

⇒ Minimizing binary size

Optimization level

[profile.release]
opt-level = "z"

# Uncomment for vectorization of loops
# Targets minimal binary size little less aggressively
# opt-level = "s"

Abort on panic! (disable unwinding the stack)

[profile.release]
panic = "abort"

Goal of unwinding is to safely exit the program by e.g. cleaning up memory

Benchmarking

⇒ Minimizing compile times

Linking

A big part of compile time is actually linking time (particularly when rebuilding a program after a small change)
⇒ you can try different linkers like lld or mold which are often faster

$ RUSTFLAGS="-C link-arg=-fuse-ld=lld" cargo build --release

Tip for slow running applications: Partially enable compiler optimizations in debug/dev mode (from Bevy)

[profile.dev]
opt-level = 1   # Enable minimal optimization for your code
[profile.dev.package."*"]
opt-level = 3   # Enable optimization for dependencies

Goal: Speeding up builds in semi-optimized (debug) builds
⇒ This slows down clean builds

Benchmarking

Disabling Link-Time optimization to reduce compile times

[profile.release]
lto = "off"

Outlook: Alternative code generator (e.g. cranelift)

Benchmarking

Usually we are interested in minimizing runtime

⇒ How to start out with benchmarking?

Rust has built-in benchmark tests but they are currently only available in nightly Rust

Wall clock benchmarking using std::time::Instant

Outlook: Unofficial crates for fine-graiened measurements with warm-ups to minimize variance

e.g. criterion

Benchmarking

Wall clock benchmarking using `std::time::Instant`

let start = std::time::Instant::now();
expensive_function();
let secs = start.elapsed().as_secs_f64();

⇒ Great for manually benchmarking long running functions

⇒ May suffer from high variance: Use multiple runs and take the average

⇒ Rely on external crates like `criterion` when necessary

Benchmarking

Rust enum size = Largest variant size + Discriminator (8 bytes)

enum E {
    A,                    // holds 0  bytes
    B(i32),               // holds 4  bytes
    C(u64, u8, u64, u8),  // holds 18 bytes
    D(Vec<u32>)           // holds 24 bytes
}

Consider boxing large variants in enums

enum E {
    A,                              // holds 0 bytes
    B(i32),                         // holds 4 bytes
    C(Box<(u64, u8, u64, u8)>),     // holds 8 bytes
    D(Box<Vec<u32>>)                // holds 8 bytes
}

The latter enum type has size 8 + 8, while the former had size 24 + 8.

Benchmarking

If you have a `Vec` that is unlikely to be changed in the future

let v: Vec<u32> = vec![1, 2, 3];

You can convert it to a boxed slice

let boxed_slice: Box<[u32]> = Box::new([1, 2, 3]);
let v: Vec<u32> = boxed_slice.into_vec(); // back to Vec

⇒ This decreases size from 8 (len) + 8 (capacity) + 8 (pointer) bytes to 8 (len) + 8 (capacity) bytes because slices don't hold capacity

⇒ A boxed slice can be converted into a Vec type without cloning or reallocation

Wrap-Up

Unit tests: Test small, isolated pieces of functionality with assertion macros.
Integration tests: Focus on how different parts of your application interact (public API).
Documentation tests: Keep documentation examples up-to-date by using inline doc tests.
Fail fast: Use #[should_panic], assertions and debug assertions to catch errors early.
TDD approach: Use it for maintainable code in Rust.
Clean code: Clean your code using best practices and Rust's official tools like rustfmt and clippy
Benchmarking: Validate performance of what matters, detect hot spots and bottlenecks in your code. Critical to ensure your refactored code enhances/maintains performance.

Resources

https://doc.rust-lang.org/book/
https://doc.rust-lang.org/cargo/guide/project-layout.html
https://proptest-rs.github.io/proptest/intro.html
https://nexte.st/
https://github.com/xd009642/tarpaulin
https://rust-classes.com/chapter_8_1
https://fettblog.eu/slides/idiomatic-rust/
https://docs.rs/debug_print/latest/debug_print/
https://rust-lang.github.io/rustfmt/?version=v1.6.0&search=
https://doc.rust-lang.org/clippy/
https://nnethercote.github.io/perf-book/general-tips.html
https://doc.rust-lang.org/nightly/unstable-book/library-features/test.html
https://benw.is/posts/how-i-improved-my-rust-compile-times-by-seventy-five-percent#cranelift
https://bheisler.github.io/criterion.rs/book/getting_started.html

Testing Go to github.com/alphagergedan/getting-rusty-bootcamp

About me

Why testing matters?

Today's agenda

Writing tests

Writing tests

Directory structure in Rust

Writing tests

Anatomy of a Test function

⇒ Run your tests using $ cargo test on the console

Writing tests

Example (inline unit tests): src/add/mod.rs

Writing tests

Example (separate unit tests): src/add/mod.rs

src/add/tests.rs

Writing tests

Running the tests of a module

⇒ You can also run only a specific test with $ cargo test add::test_add

⇒ What about failing tests?

Writing tests

Failing test example

assertion `left == right` failed may not give us enough information

⇒ Add custom failure messages

Writing tests

Adding custom failure messages: src/some_other_module/tests.rs

assertion `left == right` failed: 0 must equal 1

⇒ Useful for giving context about what an assertion means

⇒ Also works with assert!, assert_ne!

Writing tests

Checking for panics with should_panic

Specifying the expected message

Writing tests

What if the function we want to test returns a Result type?

You can use Result type in tests

Writing tests

Running tests parallel or consecutively

If your tests need to share some state, e.g. consecutively read, process and write to a file, then they cannot be run in parallel

⇒ Specifying --test-threads=1 means running the tests consecutively

Writing tests

Ignoring some tests unless specifically requested

⇒ $ cargo test -- --ignored to only run the ignored tests

⇒ $ cargo test -- --include-ignored to run all tests

Writing tests

Documentation tests

Writing tests

In Rust, Integration tests are entirely external to your code

⇒ Integration tests can only call functions that are part of your public API

Each file in the tests/ directory is a separate crate

⇒ We need to bring our library into each test crate's scope

We do not need to annotate any code in tests/ with #[cfg(test)]

$ cargo test runs first unit tests, then integration tests, and then the doc tests

Writing tests

Example CLI application (add_cli)

src/main.rs

$ cargo run -- 1 2 outputs Sum: 3 with a newline

Writing tests

Example integration test for parse and add

tests/test_parse_and_add.rs

Writing tests

Example end-to-end test for the app

tests/test_add_cli.rs

Writing tests

Outlook: Unofficial testing tools

Rust development practices

Rust development practices

Test-driven development (TDD)

Red, Green, Refactor cycle:

⇒ Forces you to think about what you want the code to do before you write it.

Rust development practices

How to start out with a skeleton in Rust?

src/main.rs

⇒ Create your skeleton using todo!(message)

⇒ TDD can provide safe and maintainable divide & conquer for the implementation of your application logic

⇒ Makes implementing complex applications more managable

Rust development practices

Starting with a skeleton using todo!

src/main.rs

not yet implemented: Implement square function

Rust development practices

Adding debugging checks with debug_assert!, debug_assert_eq!

Testing
Go to github.com/alphagergedan/getting-rusty-bootcamp

⇒ Run your tests using `$ cargo test` on the console

Example (inline unit tests): `src/add/mod.rs`

Example (separate unit tests): `src/add/mod.rs`

`src/add/tests.rs`

⇒ You can also run only a specific test with `$ cargo test add::test_add`

Adding custom failure messages: `src/some_other_module/tests.rs`

⇒ Also works with `assert!`, `assert_ne!`

Checking for panics with `should_panic`

What if the function we want to test returns a `Result` type?

You can use `Result` type in tests

⇒ Specifying `--test-threads=1` means running the tests consecutively

⇒ `$ cargo test -- --ignored` to only run the ignored tests

⇒ `$ cargo test -- --include-ignored` to run all tests

Each file in the `tests/` directory is a separate crate

We do not need to annotate any code in `tests/` with `#[cfg(test)]`

`$ cargo test` runs first unit tests, then integration tests, and then the doc tests

`src/main.rs`

`$ cargo run -- 1 2` outputs `Sum: 3` with a newline

Example integration test for `parse` and `add`

`tests/test_parse_and_add.rs`

`tests/test_add_cli.rs`

`src/main.rs`

⇒ Create your skeleton using `todo!(message)`

Starting with a skeleton using `todo!`

`src/main.rs`

Adding debugging checks with `debug_assert!`, `debug_assert_eq!`

⇒ Use `todo!` to make your code compile for your skeleton

⇒ `debug_assert!` is like assert but only runs in debug mode

⇒ `assert!` can be used too if you must abort in an unrecoverable state

You can also use the unofficial `debug_print` crate to have print macros that are not compiled in release builds

Use `mod` to organize your code into modules

`src/lib.rs`

⇒ If your module gets large, separate it into its own file using `src/math.rs`

⇒ You can also separate it into its own directory using `src/math/mod.rs`

⇒ Outlook: Workspaces / Having crates in your crates that share the same `Cargo.lock`

Documentation practices: Use `//!` for module docs

`src/lib.rs`

Documentation practices: Use `///` for function docs

`src/math.rs`

Code formatting with `rustfmt`

Before running `$ cargo fmt`

After running `$ cargo fmt`

⇒ Outlook: You can customize the style through `rustfmt.toml`

`rustc` gives you warnings when you run `$ cargo build` to build your code

Linting with `clippy`

Linting with `clippy`

After running `$ cargo clippy`

⇒ by changing its build configuration at `Cargo.toml`

⇒ compare outputs of

`$ rustc --print cfg` and `$ rustc --print cfg -C target-cpu=native`

to see if the CPU features are being detected correctly