Testing
Go to github.com/alphagergedan/getting-rusty-bootcamp

About me

I am a doctoral researcher in scientific machine learning at Chair of scientific computing (Chair 05 at TUM School of CIT).

Why testing matters?

"Testing leads to failure, and failure leads to understanding." — Burt Rutan

  • Testing ensures code correctness and reliability.
  • Rust encourages testing through its ecosystem and tooling.
  • Use your program, try to break it, and write tests for it.

Today's agenda

  1. Writing tests
  2. Rust development practices
  3. Formatting and linting
  4. Benchmarking

Writing tests

Writing tests

Directory structure in Rust

.
├── Cargo.lock
├── Cargo.toml
├── src/
│   ├── lib.rs
│   ├── main.rs
│   ├── module_1/
│   │   ├── mod.rs
│   │   └── [...]
│   └── [...]
└── [...]

Unit tests are usually placed inside the modules they are testing:

Writing tests

Anatomy of a Test function

#[test]
fn test_name() {
    // Setup: Prepare variables or context
    [...]

    // Assertion: Ensure a condition is true
    assert!(...);

    // Assertion: Check equality of values
    assert_eq!(...);

    // Assertion: Check inequality of values
    assert_ne!(...);
}

⇒ Run your tests using $ cargo test on the console

Writing tests

Example (inline unit tests): src/add/mod.rs

pub fn add(a: i32, b: i32) -> i32 {
    a + b
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_add() {
        let result = add(2, 3);
        assert_eq!(result, 5);
    }
}

Writing tests

Example (separate unit tests): src/add/mod.rs

pub fn add(a: i32, b: i32) -> i32 {
    a + b
}

#[cfg(test)]
mod tests;

src/add/tests.rs

use super::*;

#[test]
fn test_add() {
    let result = add(2, 3);
    assert_eq!(result, 5);
}

Writing tests

Running the tests of a module

$ cargo test add

[...]
running 1 test
test add::tests::test_add ... ok
[...]

⇒ You can also run only a specific test with $ cargo test add::test_add

⇒ What about failing tests?

Writing tests

Failing test example

$ cargo test some_other_module

[...]
test some_other_module::tests::test_some_function ... FAILED

failures:

---- some_other_module::tests::test_some_function stdout ----
thread 'some_other_module::tests::test_some_function' panicked:
assertion `left == right` failed
  left: 0
 right: 1
[...]

assertion `left == right` failed may not give us enough information


⇒ Add custom failure messages

Writing tests

Adding custom failure messages: src/some_other_module/tests.rs

use super::*;

#[test]
fn test_some_function() {
    let result = some_function(); // 0
    assert_eq!(
        some_function(), 1,
        "{} must equal 1", result
    );
}

assertion `left == right` failed: 0 must equal 1


⇒ Useful for giving context about what an assertion means


⇒ Also works with assert!, assert_ne!

Writing tests

Checking for panics with should_panic

#[test]
#[should_panic]
fn test_divide_by_zero() {
    div(1, 0);
}

Specifying the expected message

#[test]
#[should_panic(expected = "Cannot divide by zero")]
fn test_divide_by_zero() {
    div(1, 0);
}

Writing tests

What if the function we want to test returns a Result type?

fn add_strings(a: &str, b: &str) -> Result<i32, String> {
    let num_a: i32 = a.parse()
        .map_err(|_| "Failed to parse left operand")?;
    let num_b: i32 = b.parse()
        .map_err(|_| "Failed to parse right operand")?;
    Ok(num_a + num_b)
}

You can use Result type in tests

#[test]
fn test_add_strings() -> Result<(), String> {
    let result = add_strings("2", "3")?;
    assert_eq!(result, 5, "Two plus three must equal five");
    Ok(())
}

Writing tests

Running tests parallel or consecutively


In Rust, tests are run in parallel by default

If your tests need to share some state, e.g. consecutively read, process and write to a file, then they cannot be run in parallel

$ cargo test -- --test-threads=1
--test-threads specifies the number of threads you want to use to the test binary

⇒ Specifying --test-threads=1 means running the tests consecutively

Writing tests

Ignoring some tests unless specifically requested

#[test]
#[ignore]
fn expensive_test() {
    // code that takes an hour to run
}

⇒ $ cargo test -- --ignored to only run the ignored tests


⇒ $ cargo test -- --include-ignored to run all tests

Writing tests

Documentation tests

/// Adds two numbers.
///
/// # Examples
///
/// ```
/// use my_crate::add;
///
/// assert_eq!(add(2,2), 4);
/// ```
pub fn add(a: i32, b: i32) -> i32 {
    a + b
}
⇒ Doc tests are also run with $ cargo test, Rust ecosystem encourages testing through documentation!
⇒ This makes sure that examples within your documentation are up to date and working.

Writing tests

In Rust, Integration tests are entirely external to your code

.
├── Cargo.lock
├── Cargo.toml
├── src/
│   └── [...]
└── tests/
    └── integration_test.rs

⇒ Integration tests can only call functions that are part of your public API


Each file in the tests/ directory is a separate crate

⇒ We need to bring our library into each test crate's scope


We do not need to annotate any code in tests/ with #[cfg(test)]

$ cargo test runs first unit tests, then integration tests, and then the doc tests


Goal of integration tests: To test whether many parts of your library work together correctly

Writing tests

Example CLI application (add_cli)


src/main.rs

use add_cli::{add, parse};
use std::env;

fn main() -> Result<(), String> {
    let args: Vec<String> = env::args().collect();
    let left: i32 = parse(&args[1])?;
    let right: i32 = parse(&args[2])?;
    print!("Sum: {}\n", add(left, right));
    Ok(())
}

$ cargo run -- 1 2 outputs Sum: 3 with a newline

Writing tests

Example integration test for parse and add


tests/test_parse_and_add.rs

use add_cli::{add, parse};

#[test]
fn test_successful_parse_and_add() -> Result<(), String> {
    let num_str_left = "1";
    let num_str_right = "2";

    let left: i32 = parse(num_str_left)?;
    let right: i32 = parse(num_str_right)?;

    let sum = add(left, right);
    assert_eq!(sum, 3);

    Ok(())
}

Writing tests

Example end-to-end test for the app


tests/test_add_cli.rs

use std::process::Command;

#[test]
fn test_valid_input() {
    let output = Command::new(env!("CARGO_BIN_EXE_add_cli"))
        .arg("1").arg("2")
        .output()
        .expect("Failed to execute command");
    assert!(output.status.success());

    let stdout = String::from_utf8(output.stdout)
        .expect("Failed to parse stdout");
    assert_eq!(stdout, "Sum: 3\n");
}

Writing tests

Outlook: Unofficial testing tools


Rust development practices

Rust development practices

Test-driven development (TDD)

As a concept: Write tests before you write the code.

Red, Green, Refactor cycle:

  1. Write a failing test (Red).
  2. Write the minimal code to pass the test (Green).
  3. Refactor the code while keeping tests passing.

⇒ Forces you to think about what you want the code to do before you write it.

Rust development practices

How to start out with a skeleton in Rust?


src/main.rs

// The main executable will consist of many methods
// We can usually make many decision-mistakes here
// Example: Training a model to approximate a complex function.
//    1) Generate a dataset
//    2) Configure a model
//    3) Fit the model
//    4) Inspect the error and visualize the solution
fn main() {}

⇒ Create your skeleton using todo!(message)


⇒ TDD can provide safe and maintainable divide & conquer for the implementation of your application logic


⇒ Makes implementing complex applications more managable

Rust development practices

Starting with a skeleton using todo!


src/main.rs

fn main() {
    let input = 5;
    let result = square(input);
    println!("The square of 5 is {}", result);
}

fn square(x: i32) -> i32 {
    todo!("Implement square function");
}
$ cargo run gives

not yet implemented: Implement square function

Rust development practices

Adding debugging checks with debug_assert!, debug_assert_eq!

fn square(x: i32) -> i32 {
    debug_assert!(x.abs() < 1000,
        "Input is too large for this example");
    todo!("Implement square function");
}

⇒ Use todo! to make your code compile for your skeleton


⇒ debug_assert! is like assert but only runs in debug mode


⇒ assert! can be used too if you must abort in an unrecoverable state


You can also use the unofficial debug_print crate to have print macros that are not compiled in release builds

Rust development practices

Writing unit tests for your skeleton

#[test]
fn test_square() {
    assert_eq!(square(2), 4);
    assert_eq!(square(-3), 9);
    assert_eq!(square(0), 0);
}

Implement the function

fn square(x: i32) -> i32 {
    debug_assert!(x.abs() < 1000,
        "Input is too large for this example");
    x * x
}

⇒ Keep your functions small and focused

Rust development practices

Thinking in expressions

let status = if logged_in {
    "Active"
} else {
    "Inactive"
};
let state = loop {

    // game loop

    if game_over {
        break 1
    }
};

Rust development practices

Thinking in expressions

let grade = match score {
    90..=100 => "A",
    80..=89 => "B",
    70..=79 => "C",
    _ => "D",
};
⇒ Rust's expression-oriented design enables code that is modular and testable in isolation

Rust development practices

Use mod to organize your code into modules


src/lib.rs

pub mod math {
    pub fn square(x: i32) -> i32 {...}
}

⇒ If your module gets large, separate it into its own file using src/math.rs


⇒ You can also separate it into its own directory using src/math/mod.rs


⇒ Outlook: Workspaces / Having crates in your crates that share the same Cargo.lock

Rust development practices

Documentation practices: Use //! for module docs


src/lib.rs

// [Usually some License information]

//! This is the root module documentation for the crate.
//! It describes the overall purpose and functionality.

// Module definition, this is a normal comment
pub mod math;

Rust development practices

Documentation practices: Use /// for function docs


src/math.rs

//! This is the documentation for the `math` module.

/// Docs for the square function
/// Calculates the square of a number.
pub fn square(x: i32) -> i32 {
    x * x
}

Rust development practices

More clean code practices: Unnesting match arms for readability


Nested match

match result {
    Ok(value) => process(value),
    Err(err) => match err {
        Error::NotFound => handle_not_found(),
        Error::PermissionDenied => handle_permission(),
    },
}

Unnested match

match result {
    Ok(value) => process(value),
    Err(Error::NotFound) => handle_not_found(),
    Err(Error::PermissionDenied) => handle_permission(),
}

Rust development practices

Nested match

let msg = match a {
    Some(x) => match b {
        Some(y) => format!("Both: {}, {}", x, y),
        None => format!("Only a: {}", x),
    },
    None => match b {
        Some(y) => format!("Only b: {}", y),
        None => String::from("Neither"),
    }
};

Rust development practices

Unnested match

let msg = match (a, b) {
    (Some(x), Some(y)) => format!("Both: {}, {}", x, y),
    (Some(x), None) => format!("Only a: {}", x),
    (None, Some(y)) => format!("Only b: {}", y),
    (None, None) => String::from("Neither"),
};

Formatting and linting

Formatting and linting

Code formatting with rustfmt



Before running $ cargo fmt

fn main() { let x=5;let y=10;println!("Sum: {}",x+y); }

After running $ cargo fmt

fn main() {
    let x = 5;
    let y = 10;
    println!("Sum: {}", x + y);
}

⇒ Outlook: You can customize the style through rustfmt.toml

Formatting and linting

rustc gives you warnings when you run $ cargo build to build your code

warning: variable `someNumber` should have a snake case name
  --> src/main.rs:23:9
   |
23 |     let someNumber = 12;
   |         ^^^^^^^^^^ help: convert the identifier to
                              snake case: `some_number`

Rust-analyzer

  • An implementation of Language Server Protocol for Rust
  • IDE support for Rust (code completion, definitions from docs etc.)
  • E.g., you can directly go to documentations and see example usage
  • Using Rust-analyzer: Install it and set up in your code editor (neovim, vscode etc.)

Formatting and linting

Linting with clippy


Formatting and linting

Linting with clippy

let vec = vec![1, 2, 3];
if vec.len() > 0 {
    println!("Vector is not empty");
}

After running $ cargo clippy

warning: length comparison to zero
--> src/main.rs:27:8
 |
27 |     if vec.len() > 0 {
 |        ^^^^^^^^^^^^^ help: using `!is_empty` is clearer
                              and more explicit:
                              `!vec.is_empty()`

Benchmarking

Benchmarking

Benchmarking provides a feedback mechanism for your refactored code in TDD!



First, change the performance of a Rust program without changing its code


⇒ by changing its build configuration at Cargo.toml


  • Common pitfall: First ensure that you are using the release build via

    $ cargo build --release

  • Common trade-off: compile time vs. runtime speed

  • Prefer $ cargo check instead of $ cargo build to quickly ensure your code still compiles during development

  • Omit the --release flag during development for incremental builds: Debug builds are usually faster than release builds

    • ⇒ Depending on the metric we would like to optimize there are different build strategies

Benchmarking

⇒ Minimizing runtime speed

Disabling codegen units (used to parallelize compilation)

[profile.release]
codegen-units = 1

Increasing Link-Time optimization (LTO)

[profile.release]
lto = true    # or "thin". This is default for release
# lto = "fat" # Uncomment for more aggresive LTO

Benchmarking

CPU specific instructions (e.g. AVX SIMD for x86-64 CPUs) at the cost of compatibility

$ RUSTFLAGS="-C target-cpu=native" cargo build --release

⇒ compare outputs of

$ rustc --print cfg and $ rustc --print cfg -C target-cpu=native

to see if the CPU features are being detected correctly


Outlook: using an alternative heap allocator (e.g. jemalloc, mimalloc), profile-guided optimization

Benchmarking

⇒ Minimizing binary size

Optimization level

[profile.release]
opt-level = "z"

# Uncomment for vectorization of loops
# Targets minimal binary size little less aggressively
# opt-level = "s"

Abort on panic! (disable unwinding the stack)

[profile.release]
panic = "abort"
Goal of unwinding is to safely exit the program by e.g. cleaning up memory

Benchmarking

⇒ Minimizing compile times

Linking

A big part of compile time is actually linking time (particularly when rebuilding a program after a small change)
⇒ you can try different linkers like lld or mold which are often faster 
$ RUSTFLAGS="-C link-arg=-fuse-ld=lld" cargo build --release
Tip for slow running applications: Partially enable compiler optimizations in debug/dev mode (from Bevy) 
[profile.dev]
opt-level = 1   # Enable minimal optimization for your code
[profile.dev.package."*"]
opt-level = 3   # Enable optimization for dependencies
Goal: Speeding up incremental builds in semi-optimized (debug) builds
⇒ This slows down clean builds

Benchmarking

Disabling Link-Time optimization to reduce compile times

[profile.release]
lto = "off"

Outlook: Alternative code generator (e.g. cranelift)

Benchmarking

Usually we are interested in minimizing runtime

⇒ How to start out with benchmarking?


Benchmarking

Wall clock benchmarking using std::time::Instant

let start = std::time::Instant::now();
expensive_function();
let secs = start.elapsed().as_secs_f64();

⇒ Great for manually benchmarking long running functions


⇒ May suffer from high variance: Use multiple runs and take the average


⇒ Rely on external crates like criterion when necessary

Benchmarking

Rust enum size = Largest variant size + Discriminator (8 bytes)

enum E {
    A,                    // holds 0  bytes
    B(i32),               // holds 4  bytes
    C(u64, u8, u64, u8),  // holds 18 bytes
    D(Vec<u32>)           // holds 24 bytes
}

Consider boxing large variants in enums

enum E {
    A,                              // holds 0 bytes
    B(i32),                         // holds 4 bytes
    C(Box<(u64, u8, u64, u8)>),     // holds 8 bytes
    D(Box<Vec<u32>>)                // holds 8 bytes
}
The latter enum type has size 8 + 8, while the former had size 24 + 8.

Benchmarking

If you have a Vec that is unlikely to be changed in the future

let v: Vec<u32> = vec![1, 2, 3];

You can convert it to a boxed slice

let boxed_slice: Box<[u32]> = Box::new([1, 2, 3]);
let v: Vec<u32> = boxed_slice.into_vec(); // back to Vec
⇒ This decreases size from 8 (len) + 8 (capacity) + 8 (pointer) bytes to 8 (len) + 8 (capacity) bytes because slices don't hold capacity

⇒ A boxed slice can be converted into a Vec type without cloning or reallocation

Wrap-Up

Resources

Next steps