Macros

By now you’ve learned about many of the tools Rust provides for abstracting and reusing code. These units of code reuse have a rich semantic structure. For example, functions have a type signature, type parameters have trait bounds, and overloaded functions must belong to a particular trait.

This structure means that Rust’s core abstractions have powerful compile-time correctness checking. But this comes at the price of reduced flexibility. If you visually identify a pattern of repeated code, you may find it’s difficult or cumbersome to express that pattern as a generic function, a trait, or anything else within Rust’s semantics.

Macros allow us to abstract at a syntactic level. A macro invocation is shorthand for an "expanded" syntactic form. This expansion happens early in compilation, before any static checking. As a result, macros can capture many patterns of code reuse that Rust’s core abstractions cannot.

The drawback is that macro-based code can be harder to understand, because fewer of the built-in rules apply. Like an ordinary function, a well-behaved macro can be used without understanding its implementation. However, it can be difficult to design a well-behaved macro! Additionally, compiler errors in macro code are harder to interpret, because they describe problems in the expanded code, not the source-level form that developers use.

These drawbacks make macros something of a "feature of last resort". That’s not to say that macros are bad; they are part of Rust because sometimes they’re needed for truly concise, well-abstracted code. Just keep this tradeoff in mind.

Defining a macro

You may have seen the vec! macro, used to initialize a vector with any number of elements.

let x: Vec<u32> = vec![1, 2, 3];

This can’t be an ordinary function, because it takes any number of arguments. But we can imagine it as syntactic shorthand for

let x: Vec<u32> = {
    let mut temp_vec = Vec::new();
    temp_vec.push(1);
    temp_vec.push(2);
    temp_vec.push(3);
    temp_vec
};

We can implement this shorthand, using a macro: ¹

macro_rules! vec {
    ( $( $x:expr ),* ) => {
        {
            let mut temp_vec = Vec::new();
            $(
                temp_vec.push($x);
            )*
            temp_vec
        }
    };
}

Whoa, that’s a lot of new syntax! Let’s break it down.

macro_rules! vec { ... }

This says we’re defining a macro named vec, much as fn vec would define a function named vec. In prose, we informally write a macro’s name with an exclamation point, e.g. vec!. The exclamation point is part of the invocation syntax and serves to distinguish a macro from an ordinary function.

Matching

The macro is defined through a series of rules, which are pattern-matching cases. Above, we had

( $( $x:expr ),* ) => { ... };

This is like a match expression arm, but the matching happens on Rust syntax trees, at compile time. The semicolon is optional on the last (here, only) case. The "pattern" on the left-hand side of => is known as a ‘matcher’. These have their own little grammar within the language.

The matcher $x:expr will match any Rust expression, binding that syntax tree to the ‘metavariable’ $x. The identifier expr is a ‘fragment specifier’; the full possibilities are enumerated later in this chapter. Surrounding the matcher with $(...),* will match zero or more expressions, separated by commas.

Aside from the special matcher syntax, any Rust tokens that appear in a matcher must match exactly. For example,

macro_rules! foo {
    (x => $e:expr) => (println!("mode X: {}", $e));
    (y => $e:expr) => (println!("mode Y: {}", $e));
}

fn main() {
    foo!(y => 3);
}

will print

mode Y: 3

With

foo!(z => 3);

we get the compiler error

error: no rules expected the token `z`

Expansion

The right-hand side of a macro rule is ordinary Rust syntax, for the most part. But we can splice in bits of syntax captured by the matcher. From the original example:

$(
    temp_vec.push($x);
)*

Each matched expression $x will produce a single push statement in the macro expansion. The repetition in the expansion proceeds in "lockstep" with repetition in the matcher (more on this in a moment).

Because $x was already declared as matching an expression, we don’t repeat :expr on the right-hand side. Also, we don’t include a separating comma as part of the repetition operator. Instead, we have a terminating semicolon within the repeated block.

Another detail: the vec! macro has two pairs of braces on the right-hand side. They are often combined like so:

macro_rules! foo {
    () => {{
        ...
    }}
}

The outer braces are part of the syntax of macro_rules!. In fact, you can use () or [] instead. They simply delimit the right-hand side as a whole.

The inner braces are part of the expanded syntax. Remember, the vec! macro is used in an expression context. To write an expression with multiple statements, including let-bindings, we use a block. If your macro expands to a single expression, you don’t need this extra layer of braces.

Note that we never declared that the macro produces an expression. In fact, this is not determined until we use the macro as an expression. With care, you can write a macro whose expansion works in several contexts. For example, shorthand for a data type could be valid as either an expression or a pattern.

Repetition

The repetition operator follows two principal rules:

1. $(...)* walks through one "layer" of repetitions, for all of the $names it contains, in lockstep, and
2. each $name must be under at least as many $(...)*s as it was matched against. If it is under more, it’ll be duplicated, as appropriate.

This baroque macro illustrates the duplication of variables from outer repetition levels.

macro_rules! o_O {
    (
        $(
            $x:expr; [ $( $y:expr ),* ]
        );*
    ) => {
        &[ $($( $x + $y ),*),* ]
    }
}

fn main() {
    let a: &[i32]
        = o_O!(10; [1, 2, 3];
               20; [4, 5, 6]);

    assert_eq!(a, [11, 12, 13, 24, 25, 26]);
}

That’s most of the matcher syntax. These examples use $(...)*, which is a "zero or more" match. Alternatively you can write $(...)+ for a "one or more" match. Both forms optionally include a separator, which can be any token except + or *.

This system is based on "Macro-by-Example" (PDF link).

Hygiene

Some languages implement macros using simple text substitution, which leads to various problems. For example, this C program prints 13 instead of the expected 25.

#define FIVE_TIMES(x) 5 * x

int main() {
    printf("%d\n", FIVE_TIMES(2 + 3));
    return 0;
}

After expansion we have 5 * 2 + 3, and multiplication has greater precedence than addition. If you’ve used C macros a lot, you probably know the standard idioms for avoiding this problem, as well as five or six others. In Rust, we don’t have to worry about it.

macro_rules! five_times {
    ($x:expr) => (5 * $x);
}

fn main() {
    assert_eq!(25, five_times!(2 + 3));
}

The metavariable $x is parsed as a single expression node, and keeps its place in the syntax tree even after substitution.

Another common problem in macro systems is ‘variable capture’. Here’s a C macro, using a GNU C extension to emulate Rust’s expression blocks.

#define LOG(msg) ({ \
    int state = get_log_state(); \
    if (state > 0) { \
        printf("log(%d): %s\n", state, msg); \
    } \
})

Here’s a simple use case that goes terribly wrong:

const char *state = "reticulating splines";
LOG(state)

This expands to

const char *state = "reticulating splines";
{
    int state = get_log_state();
    if (state > 0) {
        printf("log(%d): %s\n", state, state);
    }
}

The second variable named state shadows the first one. This is a problem because the print statement should refer to both of them.

The equivalent Rust macro has the desired behavior.

macro_rules! log {
    ($msg:expr) => {{
        let state: i32 = get_log_state();
        if state > 0 {
            println!("log({}): {}", state, $msg);
        }
    }};
}

fn main() {
    let state: &str = "reticulating splines";
    log!(state);
}

This works because Rust has a hygienic macro system. Each macro expansion happens in a distinct ‘syntax context’, and each variable is tagged with the syntax context where it was introduced. It’s as though the variable state inside main is painted a different "color" from the variable state inside the macro, and therefore they don’t conflict.

This also restricts the ability of macros to introduce new bindings at the invocation site. Code such as the following will not work:

macro_rules! foo {
    () => (let x = 3;);
}

fn main() {
    foo!();
    println!("{}", x);
}

Instead you need to pass the variable name into the invocation, so that it’s tagged with the right syntax context.

macro_rules! foo {
    ($v:ident) => (let $v = 3;);
}

fn main() {
    foo!(x);
    println!("{}", x);
}

This holds for let bindings and loop labels, but not for items. So the following code does compile:

macro_rules! foo {
    () => (fn x() { });
}

fn main() {
    foo!();
    x();
}

Recursive macros

A macro’s expansion can include more macro invocations, including invocations of the very same macro being expanded. These recursive macros are useful for processing tree-structured input, as illustrated by this (simplistic) HTML shorthand:

macro_rules! write_html {
    ($w:expr, ) => (());

    ($w:expr, $e:tt) => (write!($w, "{}", $e));

    ($w:expr, $tag:ident [ $($inner:tt)* ] $($rest:tt)*) => {{
        write!($w, "<{}>", stringify!($tag));
        write_html!($w, $($inner)*);
        write!($w, "</{}>", stringify!($tag));
        write_html!($w, $($rest)*);
    }};
}

fn main() {
    use std::fmt::Write;
    let mut out = String::new();

    write_html!(&mut out,
        html[
            head[title["Macros guide"]]
            body[h1["Macros are the best!"]]
        ]);

    assert_eq!(out,
        "<html><head><title>Macros guide</title></head>\
         <body><h1>Macros are the best!</h1></body></html>");
}

Debugging macro code

To see the results of expanding macros, run rustc --pretty expanded. The output represents a whole crate, so you can also feed it back in to rustc, which will sometimes produce better error messages than the original compilation. Note that the --pretty expanded output may have a different meaning if multiple variables of the same name (but different syntax contexts) are in play in the same scope. In this case --pretty expanded,hygiene will tell you about the syntax contexts.

rustc provides two syntax extensions that help with macro debugging. For now, they are unstable and require feature gates.

• log_syntax!(...) will print its arguments to standard output, at compile time, and "expand" to nothing.

• trace_macros!(true) will enable a compiler message every time a macro is expanded. Use trace_macros!(false) later in expansion to turn it off.

Syntactic requirements

Even when Rust code contains un-expanded macros, it can be parsed as a full syntax tree. This property can be very useful for editors and other tools that process code. It also has a few consequences for the design of Rust’s macro system.

One consequence is that Rust must determine, when it parses a macro invocation, whether the macro stands in for

• zero or more items,
• zero or more methods,
• an expression,
• a statement, or
• a pattern.

A macro invocation within a block could stand for some items, or for an expression / statement. Rust uses a simple rule to resolve this ambiguity. A macro invocation that stands for items must be either

• delimited by curly braces, e.g. foo! { ... }, or
• terminated by a semicolon, e.g. foo!(...);

Another consequence of pre-expansion parsing is that the macro invocation must consist of valid Rust tokens. Furthermore, parentheses, brackets, and braces must be balanced within a macro invocation. For example, foo!([) is forbidden. This allows Rust to know where the macro invocation ends.

More formally, the macro invocation body must be a sequence of ‘token trees’. A token tree is defined recursively as either

• a sequence of token trees surrounded by matching (), [], or {}, or
• any other single token.

Within a matcher, each metavariable has a ‘fragment specifier’, identifying which syntactic form it matches.

• ident: an identifier. Examples: x; foo.
• path: a qualified name. Example: T::SpecialA.
• expr: an expression. Examples: 2 + 2; if true { 1 } else { 2 }; f(42).
• ty: a type. Examples: i32; Vec<(char, String)>; &T.
• pat: a pattern. Examples: Some(t); (17, 'a'); _.
• stmt: a single statement. Example: let x = 3.
• block: a brace-delimited sequence of statements and optionally an expression. Example: { log(error, "hi"); return 12; }.
• item: an item. Examples: fn foo() { }; struct Bar;.
• meta: a "meta item", as found in attributes. Example: cfg(target_os = "windows").
• tt: a single token tree.

There are additional rules regarding the next token after a metavariable:

• expr and stmt variables may only be followed by one of: => , ;
• ty and path variables may only be followed by one of: => , = | ; : > [ { as where
• pat variables may only be followed by one of: => , = | if in
• Other variables may be followed by any token.

These rules provide some flexibility for Rust’s syntax to evolve without breaking existing macros.

The macro system does not deal with parse ambiguity at all. For example, the grammar $($i:ident)* $e:expr will always fail to parse, because the parser would be forced to choose between parsing $i and parsing $e. Changing the invocation syntax to put a distinctive token in front can solve the problem. In this case, you can write $(I $i:ident)* E $e:expr.

Scoping and macro import/export

Macros are expanded at an early stage in compilation, before name resolution. One downside is that scoping works differently for macros, compared to other constructs in the language.

Definition and expansion of macros both happen in a single depth-first, lexical-order traversal of a crate’s source. So a macro defined at module scope is visible to any subsequent code in the same module, which includes the body of any subsequent child mod items.

A macro defined within the body of a single fn, or anywhere else not at module scope, is visible only within that item.

If a module has the macro_use attribute, its macros are also visible in its parent module after the child’s mod item. If the parent also has macro_use then the macros will be visible in the grandparent after the parent’s mod item, and so forth.

The macro_use attribute can also appear on extern crate. In this context it controls which macros are loaded from the external crate, e.g.

#[macro_use(foo, bar)]
extern crate baz;

If the attribute is given simply as #[macro_use], all macros are loaded. If there is no #[macro_use] attribute then no macros are loaded. Only macros defined with the #[macro_export] attribute may be loaded.

To load a crate’s macros without linking it into the output, use #[no_link] as well.

An example:

macro_rules! m1 { () => (()) }

// Visible here: `m1`.

mod foo {
    // Visible here: `m1`.

    #[macro_export]
    macro_rules! m2 { () => (()) }

    // Visible here: `m1`, `m2`.
}

// Visible here: `m1`.

macro_rules! m3 { () => (()) }

// Visible here: `m1`, `m3`.

#[macro_use]
mod bar {
    // Visible here: `m1`, `m3`.

    macro_rules! m4 { () => (()) }

    // Visible here: `m1`, `m3`, `m4`.
}

// Visible here: `m1`, `m3`, `m4`.

When this library is loaded with #[macro_use] extern crate, only m2 will be imported.

The Rust Reference has a listing of macro-related attributes.

The variable $crate

A further difficulty occurs when a macro is used in multiple crates. Say that mylib defines

pub fn increment(x: u32) -> u32 {
    x + 1
}

#[macro_export]
macro_rules! inc_a {
    ($x:expr) => ( ::increment($x) )
}

#[macro_export]
macro_rules! inc_b {
    ($x:expr) => ( ::mylib::increment($x) )
}

inc_a only works within mylib, while inc_b only works outside the library. Furthermore, inc_b will break if the user imports mylib under another name.

Rust does not (yet) have a hygiene system for crate references, but it does provide a simple workaround for this problem. Within a macro imported from a crate named foo, the special macro variable $crate will expand to ::foo. By contrast, when a macro is defined and then used in the same crate, $crate will expand to nothing. This means we can write

#[macro_export]
macro_rules! inc {
    ($x:expr) => ( $crate::increment($x) )
}

to define a single macro that works both inside and outside our library. The function name will expand to either ::increment or ::mylib::increment.

To keep this system simple and correct, #[macro_use] extern crate ... may only appear at the root of your crate, not inside mod.

The deep end

The introductory chapter mentioned recursive macros, but it did not give the full story. Recursive macros are useful for another reason: Each recursive invocation gives you another opportunity to pattern-match the macro’s arguments.

As an extreme example, it is possible, though hardly advisable, to implement the Bitwise Cyclic Tag automaton within Rust’s macro system.

macro_rules! bct {
    // cmd 0:  d ... => ...
    (0, $($ps:tt),* ; $_d:tt)
        => (bct!($($ps),*, 0 ; ));
    (0, $($ps:tt),* ; $_d:tt, $($ds:tt),*)
        => (bct!($($ps),*, 0 ; $($ds),*));

    // cmd 1p:  1 ... => 1 ... p
    (1, $p:tt, $($ps:tt),* ; 1)
        => (bct!($($ps),*, 1, $p ; 1, $p));
    (1, $p:tt, $($ps:tt),* ; 1, $($ds:tt),*)
        => (bct!($($ps),*, 1, $p ; 1, $($ds),*, $p));

    // cmd 1p:  0 ... => 0 ...
    (1, $p:tt, $($ps:tt),* ; $($ds:tt),*)
        => (bct!($($ps),*, 1, $p ; $($ds),*));

    // Halt on empty data string:
    ( $($ps:tt),* ; )
        => (());
}

Exercise: use macros to reduce duplication in the above definition of the bct! macro.

Common macros

Here are some common macros you’ll see in Rust code.

panic!

This macro causes the current thread to panic. You can give it a message to panic with:

panic!("oh no!");

vec!

The vec! macro is used throughout the book, so you’ve probably seen it already. It creates Vec<T>s with ease:

let v = vec![1, 2, 3, 4, 5];

It also lets you make vectors with repeating values. For example, a hundred zeroes:

let v = vec![0; 100];

assert! and assert_eq!

These two macros are used in tests. assert! takes a boolean. assert_eq! takes two values and checks them for equality. true passes, false panic!s. Like this:

// A-ok!

assert!(true);
assert_eq!(5, 3 + 2);

// Nope :(

assert!(5 < 3);
assert_eq!(5, 3);

try!

try! is used for error handling. It takes something that can return a Result<T, E>, and gives T if it’s a Ok<T>, and returns with the Err(E) if it’s that. Like this:

use std::fs::File;

fn foo() -> std::io::Result<()> {
    let f = try!(File::create("foo.txt"));

    Ok(())
}

This is cleaner than doing this:

use std::fs::File;

fn foo() -> std::io::Result<()> {
    let f = File::create("foo.txt");

    let f = match f {
        Ok(t) => t,
        Err(e) => return Err(e),
    };

    Ok(())
}

unreachable!

This macro is used when you think some code should never execute:

if false {
    unreachable!();
}

Sometimes, the compiler may make you have a different branch that you know will never, ever run. In these cases, use this macro, so that if you end up wrong, you’ll get a panic! about it.

let x: Option<i32> = None;

match x {
    Some(_) => unreachable!(),
    None => println!("I know x is None!"),
}

unimplemented!

The unimplemented! macro can be used when you’re trying to get your functions to typecheck, and don’t want to worry about writing out the body of the function. One example of this situation is implementing a trait with multiple required methods, where you want to tackle one at a time. Define the others as unimplemented! until you’re ready to write them.

Procedural macros

If Rust’s macro system can’t do what you need, you may want to write a compiler plugin instead. Compared to macro_rules! macros, this is significantly more work, the interfaces are much less stable, and bugs can be much harder to track down. In exchange you get the flexibility of running arbitrary Rust code within the compiler. Syntax extension plugins are sometimes called ‘procedural macros’ for this reason.