Architecture

Overview

The compiler is a Cuarzo program in compiler/. A hand-written front end feeds a C source generator, and GCC builds the native program.

main.cz and its imports
   |
   v
modules.cz: load files -> tokens per file   (lexer.cz)
   |
   v
parser.cz: tokens -> one list of AST nodes  (ast.cz, conditions.cz)
   |
   v
resolver.cz: names across modules -> one namespace
   |
   v
codegen.cz, statements.cz, expressions.cz, builtins.cz: checks + C
   |
   v
GCC (`-lm`) -> native program               (driver.cz)

There is no bytecode, interpreter, package manager, or optimizer of its own; --optimize asks GCC to optimize the C.

Modules

Each module imports what it uses. Imports may form cycles, so modules may depend on each other in both directions.

Components

Command line and driver

main.cz implements run, build, c, check, and help. A bare .cz argument means run; check only compiles, writing nothing. driver.cz checks that the source exists, writes the C beside the program (build/<name>.c beside the source by default), runs gcc -fwrapv -fno-strict-aliasing -ffp-contract=off [options] <c> -o <program> -lm [unit flags] through the run built-in, and runs the program with the remaining arguments, returning its exit status. A path that starts with - is passed as ./-... so it is never read as an option. The unit flags are the linker and compiler flags, and the pkg-config packages, that the program's runtime units ask for on this platform (see C runtime). On Windows the linker also gets --no-insert-timestamp, so that the same source and options give the same program; GCC is otherwise deterministic.

Options that shape the program itself go to target.cz: -D NAME=value into TARGET.defines, --platform and --arch into the target that when conditions test (for check and c only), --build-version, and --check-overflow, which code generation copies to Gen.check_overflow. They are also kept in RECHECK_OPTIONS, so the runs that report further errors see the same program. --release adds -O2 and the flags that strip symbols (-s, or -Wl,-x,-S for the macOS linker); --gui adds -mwindows on Windows and, on macOS, moves the program into name.app/Contents/MacOS with an Info.plist after it is built. --resources and --universal set RESOURCES and UNIVERSAL in driver.cz: every program built is linked with the object windres makes of the resource script, or built with -arch arm64 -arch x86_64.

cuarzo bundle reads [bundle] (project.cz) and writes what it asks for as JSON. The work is std/bundle's, which needs zlib for PNG and gzip, so it runs in a program of its own, built in the project's build/.cuarzo once for each standard library: the compiler stays dependent on nothing but GCC. That program builds the application with the compiler, --resources and --universal among its options.

Loading modules

modules.cz tokenizes the main file, finds import "name" pairs in its tokens, and loads each new file relative to the importing file. Imports in a branch of a when that is left out are skipped: untaken_tokens in conditions.cz finds those branches from the tokens alone, counting blocks by their do, and by each if that does not continue an else, against their end. A name given with as in such a branch maps to no module. Paths are normalized so a file is loaded once however it is spelled. After every file is tokenized, the names of all structs and enums are collected, and only then is each file parsed. Types can therefore be used before their declaration and across files. The prelude, prelude.cz, is embedded in the compiler and parsed first, so every program has Option and Result; then statements come out in load order, the main file first.

While loading, each file's imports are recorded as plain, as pub import, or as as name; a name that is a folder with an index.cz loads that file. resolver.cz then works out what each file sees: itself, the prelude, and the public declarations (the parser's public, from pub) of the files it imports and of those they pass on with pub import (passed_on); name.x reaches the public declarations of the file imported as name and of those it passes on. It reports two declarations of one name that a file sees together, a private name used from another file, and a name used where its declaration cannot be seen. Declarations that never meet keep their name for the first and get name__module for the others, and every use, lib.name included, is rewritten to the one it means. Code generation therefore still sees a single namespace. shown in diagnostics.cz turns area__square back into square.area in messages.

An import that starts with std/ names a standard library module. Its path becomes <std>/name.cz, which no file on disk can have, and its text comes from STD_MODULES in stdlib.cz, a constant map filled with embed when the compiler is built. The standard library is therefore ordinary Cuarzo in std/, and programs that use it need nothing beside the compiler.

Lexer

The lexer scans bytes without Unicode tables. Identifiers and digits are ASCII, string and comment contents may hold any UTF-8 text, and columns count UTF-8 bytes. It recognizes LF, CRLF, and bare CR newlines, decodes the simple escapes in ESCAPES, \xHH, and \u{...}, and rejects any other escape. Integer literals, decimal, hexadecimal, or binary, become decimal digits without leading zeroes; float literals keep their spelling without _. An f-string becomes FSTRING_START, then STRING tokens for its text and the tokens of each {expression} between INTERP_OPEN and INTERP_CLOSE, then FSTRING_END. Every token records its file, line, and column.

Reserved tokens are broader than the implemented grammar, so token presence alone does not define language support.

Parser and AST

The parser is recursive descent with one function per precedence level for binary expressions. Blocks end at end, else, or end of file according to their surrounding construct. One Node struct represents every construct; the comment above it in ast.cz lists which fields each kind uses.

The parser rejects nested declarations and imports, return outside functions, break and continue outside loops, non-call expression statements, invalid assignment targets, and top-level statements in imported files.

When errors are collected, the parser goes on after a syntax error. parse_guarded parses each statement, declaration, and member of a type under catch_panic; an error in one puts back what parsing it changed of the parser (ParserState), leaves an ERROR node in its place (members are left out), and resynchronize moves past it, reading on from the token with the error: to the end of that line, past the end of the blocks the statement opens, or to the end, else, or when of the block around it. Line breaks inside brackets are not tokens, so a bracket left open would hide the rest of the file; a line that starts no further right than the statement, with anything but a closing bracket, starts the next statement too. A block whose first line has the error before its do or then still has a body, the lines further right, and an end as far right as its first line, which are passed too. An end that closes no block is an error of its own. load_program stops once every file is parsed, if there was a syntax error, so the checks that follow never see an ERROR node.

Every token and node knows where it is. A token's line and col are its first character, end_line and end_col the place just after its last, and offset and end_offset the same places in bytes; the lexer gives a token its end when it has read it (add_token, or end_token for one added first). A node's line and col are where errors about it point, such as at an operator, and start_* and end_* are the code it covers: spanned gives a node the tokens its parse function read, so a statement covers its end and an expression in brackets its brackets, and spanned_from covers an operator or a call from its left operand on. fill_spans gives each node the parser made without a place of its own the place of its parts; derived methods, which no code stands for, have none. The lexer keeps comments apart (lex gives tokens and comments), and attach_comments in comments.cz gives each one to a node, with blank lines counted, when a tool asks: the compiler itself never does. positions.cz converts byte columns to the UTF-16 columns of the Language Server Protocol and back.

A when whose condition ends in do, rather than then, is a compile-time condition. The parser evaluates each condition with eval_condition against TARGET, parses every branch, and splices the statements of the branch taken into the enclosing list in place of the when, so no node for it reaches later phases. While it parses a branch left out, untaken is nonzero, and imports there are neither looked for nor counted. A constant named by -D gets a literal of its type in place of the value written (build_constant). Struct and enum names are collected from every branch, so a branch left out can be parsed; check_type reports a type that was only declared there.

Code generator

Code generation is a single pass that type-checks while it emits. Each expression returns its C text and records its Cuarzo type on the node, so the parent can check operands without walking the subtree again. Lexical scopes are a stack of bindings; inner bindings disappear when their block ends.

Before emitting definitions, the generator registers every top-level function, enum, struct, and constant. The output order is:

1. the C runtime: the units the program needs;
2. enum name tables;
3. struct typedefs, struct definitions, and constructors;
4. constants, as file-scope variables;
5. function prototypes, then function definitions: the program's functions, each generic function once for each set of type arguments it is used with, and the C functions of functions written with => and of named functions used as values;
6. cz_program, which sets the constants and then runs the top-level statements of the main file in order, and main, which calls it.

Constants are set in an order that puts each after the ones it uses. When a constant's value calls a function, the generator first walks that function and the functions it calls, setting the constants they read; a constant reached again along that walk is a cycle, reported with the chain of constants and functions between.

User names are prefixed by kind so they cannot collide with C keywords, the C library, or the runtime's cz_ helpers:

An enum whose members carry values is emitted differently: each value is a CzVariant, a heap object with the member's index and one CzSlot per value. A slot is a union of a 64-bit integer, an unsigned integer, a double, and a pointer, so one C type serves every member and every type argument of a generic enum: the type of a value, worked out by the compiler, picks the slot member it is written to and read from. Type arguments therefore need no C code of their own. A match stores its value in a temporary and becomes an if-else chain on the index or the value, so break and continue inside it still reach the enclosing loop. A pattern becomes a condition (pattern_condition, member_condition, struct_condition) and the declarations, in the branch's setup, of the names it binds; a pattern inside another tests its part only after the outer test holds. A branch with a guard tests it after its setup and, when it is false, jumps with goto to a label at the start of its else, which holds the rest of the chain.

A method is a FUNC node whose name is its type, as written inside a struct or enum or before the method's name; the parser places the methods of a type after it in the program, so every later phase sees ordinary functions. They are kept out of the resolver's names and out of g.funcs, in g.methods, since a method belongs to its type everywhere. A call value.name(...) generates the value first, then finds the method by its type (find_method), filling the type parameters the receiver gives, and passes the value as the first argument; Type.name(...) is a STATIC node (find_static), and Kind.name(...) on an enum becomes one when no member has the name. A method used as a value gets a small wrapper that holds the value (bound_method). Generic methods are instances like generic functions.

An interface is satisfied structurally (unsatisfied in interfaces.cz says why a type does not). A value of an interface type is a CzIface: the value in a slot and a table of C functions, one per method in the order the interface lists them, written once per pair of interface and type (interface_table); each takes the slot and calls the type's method, or for ToText on a built-in type, writes its text. Conversion happens in gen_expr: a value generated where an interface it satisfies is expected is wrapped with cz_iface_new, so every place that passes an expected type, from declarations to arguments and array elements, converts. A call through an interface stores the value in a temporary and calls cz_iface_method(value, i) with cz_iface_value(value) first. The bounds of type parameters are checked where type arguments are settled: generic calls, methods, and generic types written in code. The prelude's standard interfaces also let operators call methods (operator_method): == calls equals through a null-safe cz_equal__T, the orderings compare, print to_text, and +, -, *, /, %, unary -, and their compound assignments the methods of Add and its kin (arithmetic_method), on two values of the same type of the program's own.

#[derive(...)] is read by the parser: after a struct or an enum, derived_methods asks derive.cz for the Cuarzo text of each method from the fields' written types, tokenizes it with every token at the derive that asked for it, and parses it with parse_function as if it were written inside the type, so the methods are ordinary ones from then on. The text calls generic helpers of the prelude (prelude_arrays_equal_by, prelude_show_array, ...) and of std/json (json_of_array, json_to_array, ...). Maps whose keys are of the program's own types carry a CzKeyOps table (key_ops) with C functions that call the type's hash and equals.

Deferred code is a function written with do, made where the defer is and pushed on the runtime's stack of deferred code (cz_defer_push in core.c), which the collector scans. A block that holds a defer records the stack's depth when it starts and runs the code above it when it ends (cz_defer_unwind); a function with deferred code anywhere in it records its depth in cz_defer_base, which each return and each ? that returns unwind to; a loop whose body has some records the depth at the start of each pass, which break and continue unwind to (loop_marks). A runtime error prints its message and trace, then runs all the deferred code, unless it happened while doing so. with is a block that declares each value and defers its close.

A tuple is a CzVariant too, with tag 0 and one slot per item, so its type (int,str) needs no C type of its own; pair.0 reads a slot. Tuples are never null. Equality and the text of each tuple type are small C functions written once per type when first needed (tuple_equality and tuple_text, recorded in helpers). A declaration that takes a tuple apart (UNPACK), and a loop whose element is written (a, b), store the tuple in a temporary and declare a variable per item. A match on a tuple tests a condition per item pattern (pattern_condition), binding names in the branch's setup.

A function value is a CzFunc: a pointer to a C function and the values the function captured, in slots. A function written with => or do becomes a C function czl_N whose first parameter is the CzFunc itself, so its body reads what it captured from there; the copies are made where the function is written. A named function used as a value gets a wrapper czw_f with the same first parameter, so every function value is called the same way.

A function written with do is generated like a named function, with its own scopes, temporaries, result type, and hoisted ? checks, all saved and restored around it. Without a result type written or expected, ret_type is "" and the first return with a value sets it. Its statements are not entered one by one for error recovery: an error inside it leaves out the statement that holds it, since a return left out could change its type (the resolver does the same through lambda_depth).

A shared variable is a cell: T *czv_x points to a cz_alloc block, and every use reads (*czv_x). A function that uses it captures the pointer in a slot's p (cell_pointer), and reads and writes (*((T *)cz_env->slots[k].p)), so all of them, and the code around them, work on the same value. The collector sees the cell through the stack and the slots like any other object. A Binding and a LambdaContext record which variables are cells (in_cell).

The lexer tracks the open brackets and blocks: a line break inside brackets is not a statement end, unless a do or an if was opened inside them since, as in the body of a function written with do passed to a call.

A generic function is generated anew for each set of type arguments it is used with, which come from the arguments of each call. Its body is generated with each type parameter replaced, and an error inside it says which types and which use led there.

A generic struct gets a C struct and a constructor for each set of type arguments: map_type records each such type the first time C is written for it (struct_instances), and the struct section of the output is written last, once every use is known, in the place before the constants. The types of a field come from substituting the type arguments into the declared type (field_types), so Node[int] holding a Node[int] asks for no new instance. A constructor takes its type arguments from Box[int](...) (the CALL node's decl), from the type expected, or from the values, as a generic enum member does.

value? cannot return from the middle of a C expression, so the check is hoisted: the value goes into a temporary before the statement that holds it, an if returns it when it is an ERROR or NONE, and the expression reads what the OK or SOME holds. That is why ? is allowed only where a statement can run its checks first. With a message, the if returns a new ERROR made by the prelude's prelude_error_context; with another error type, one made by that type's from; in the program body (in_program), it writes the error's text to standard error and exits instead.

The prelude's Error is the one struct the runtime knows: core.c declares its type and the prototype of its constructor, czn_Error, so built-ins that fail can make one (cz_result_error_kind), with the place of the calling function's frame. Its fields must therefore stay in the order the prototype lists. Result[T] is turned into Result[T,Error] as the type is parsed, and shown writes it back as Result[T] in messages. Code generation converts a str to an Error where one is expected, and back with to_text. Each function numbers its temporaries from 1.

An f-string becomes one call to cz_str_build with its pieces, so it builds the text in one allocation. A map is a CzMap, a hash table that also keeps its entries in insertion order. A struct constructor takes every field: a call that leaves fields out or names them is reordered by the generator, which fills in the defaults where the struct is made.

C leaves the order of a call's arguments and of most operands open, but Cuarzo evaluates them from left to right. When an operand list holds a call that may change state (any call but those to the built-ins listed as pure in PURE_BUILTINS), each earlier operand that is not a literal or a variable is first stored in a temporary declared at the top of the function, and the stores are joined to the expression with the comma operator, which C evaluates in order: (cz_t1 = czf_a(), cz_t2 = czf_b(), czf_f(cz_t1, cz_t2, czf_c())).

print and eprint become printf and fprintf(stderr, ...) with a format chosen from each argument's type. Exponentiation becomes pow, which is why programs are linked with -lm.

Safety of the generated code

The generator keeps every Cuarzo program memory-safe and free of undefined behavior in C:

• an array element is read with cz_array_get, which copies it into a temporary, and pop copies the removed element out the same way, so no pointer into an array is held while other code runs;
• bytes (CzBytes, in bytes.c) are a length and a pointer to data that never changes: a literal is a constant object of the program, and a slice points into the bytes it is cut from, which the collector keeps alive through that pointer;
• a view (cz_array_view) holds its base array and an offset rather than a pointer into the base's buffer, and cz_array_elements finds the elements on each use, so a view stays right when its base grows into a new buffer and stops the program when the base became shorter than it;
• an assignment to an element or field becomes a small C block that evaluates the target's parts, then the value into cz_value, and only then takes the address cz_slot it stores through;
• integer / and % call cz_div_* and cz_mod_*, which reject division by zero and never trap;
• a float becomes an integer only through a conversion, which calls cz_float_to_signed or cz_float_to_unsigned to reject values that do not fit;
• strings passed to printf go through cz_str_arg, which rejects null;
• string and character literals escape \, quotes, line breaks, and a ? that follows another ?, so no trigraph can form;
• a non-void function must end with a return on every path, so no caller receives an indeterminate value;
• the parser limits nesting to 200 levels and the generator limits expression depth to 1000, so deep input is an error rather than a stack overflow.

Programs are built with -fwrapv, so signed overflow wraps around. With --check-overflow, +, -, *, and unary - on integers become GCC's __builtin_add_overflow and its kin, which work out the exact result into a temporary of the result type and say whether it fit; cz_overflowed stops the program when it did not. fit_code checks the same way an integer stored as a narrower type: gen_expr applies it where an integer type is expected, as it applies the other implicit conversions. The wrapping, checked, and saturating built-ins use the same GCC built-ins in every build.

C runtime

The dependency-free C runtime in compiler/runtime/ is emitted, unit by unit, into the programs that need it. Arrays use one generic header plus statically typed casts at each access; capacity grows geometrically. The runtime checks indexes, slice bounds, null references, empty pops, missing map keys, allocation failure, and size overflow. It reads files with UTF-8 validation, and it works with directories, temporary files, the environment, terminals, and other programs on both POSIX systems and Windows. On Windows, arguments, the console, the environment, the working directory, and temporary files go through the UTF-16 interfaces, so their text stays UTF-8; reading and writing files uses the C library's functions, so a file path there should be ASCII. run never uses a shell: POSIX systems start programs with posix_spawnp, and Windows resolves the program itself, searching PATH but not the current directory and starting only .exe and .com files, then quotes each argument for the C runtime.

The event loop (loop.c) keeps its callbacks, which are function values, in an array that is a root of the collector, and waits on its sources: with poll on POSIX and an I/O completion port on Windows. Each source is a handler in C that turns what the system reports into calls of the program's callbacks. Signals reach poll through a pipe their handler writes to, and the console's handler posts them to the port. Processes (spawn.c) are pipes on POSIX and overlapped named pipes on Windows, and those on a pseudo-terminal (pty.c) read and write its one descriptor on POSIX, and on Windows the pipes of a pseudo-console, which is closed on a thread of the system's pool when the process ends, since closing it waits for its last output to be read. Watched files (watch.c) have a backend for each system, inotify, FSEvents, ReadDirectoryChangesExW, and kqueue, which all hand changes to one function that applies the patterns to ignore and calls the callback. FSEvents calls back on a thread of its own, which only copies the changes with malloc and wakes poll through a pipe, so no other thread touches the collector's heap.

Desktop applications make the system's window loop the event loop's wait. desktop.c lists the built-ins and needs desktop_core.c, which chooses the window system and gives the program its events, and the backends: desktop_windows.c (Win32), desktop_macos.c (Cocoa, through the Objective-C runtime's C functions, with classes made when the application starts), and desktop_gtk.c (GTK 3, loaded with dlopen and declared by the unit itself, checked against GTK's headers by a test). desktop_start sets cz_loop_gui_wait: on Windows it waits with MsgWaitForMultipleObjectsEx for window messages and for an event that a thread blocked on the completion port sets when completions come; on macOS a thread polls the loop's descriptors while the main thread waits for Cocoa's next event, and posts an event when they are ready; with GTK the loop's descriptors and timers are a GSource of GLib's main context. While the system runs a loop of its own, as when a window is resized or a dialog is open, a timer runs the loop's sources and due timers. Events reach the program as text with fields between tabs, which std/desktop reads into typed events; an open window holds the loop, as a process does.

Web views (webview.c, with webview_windows.c, webview_gtk.c, and webview_macos.c) put the system's engine in a window. WebView2's interfaces are vtables of structs whose methods the unit calls are typed and the rest are pointers, written by a script from WebView2.h; its handlers are objects of the unit's own, a vtable for each kind. Engine callbacks become desktop events that start with webview, which std/desktop gives to std/webview through desktop_listen: messages of the page, navigation, titles, downloads, and requests for the application's files, which std/webview answers with webview_respond while the event runs. Units that put things in windows follow them through cz_desktop_resize_hook and cz_desktop_closed_hook.

C functions are declared with extern func (foreign.cz). The C numbers are names the parser gives the Cuarzo number of the same size on the target platform, so the rest of the compiler knows only pointers (ptr[T], Handle, and c_func(...), which are void * in C) as new types. Each extern function is declared to C under a name of its own, czc_name, whose assembler label is the C symbol, with the underscore some platforms put before C names (CZ_SYMBOL in runtime/ffi.c); the program's own declaration never clashes with one in a header the runtime includes, and two extern declarations may mean one symbol with different signatures. A call converts each argument for C: text is given as it is for a cstr, and as a UTF-16 copy that is freed after the call for a wstr; out arguments go through temporaries written back after the call. The files marked #![unsafe] reach code generation as attributes of the program, and every use of C checks that its file is one of them.

cuarzo bindgen (bindgen.cz) has GCC preprocess a header with -E -dD, which keeps the #define lines and the line markers that say which file each part comes from. It tokenizes the text, parses the declarations with a declarator parser that builds C types from the inside out (a hole that (*name)(...) fills), and only then works out the #define values, so they can use the enums and typedefs. A type becomes the Cuarzo type of a C signature; a struct is written when the header defines it or a written declaration uses it, until no more are used.

Async functions run on coroutines (task.c). The parser turns an async function into the function its callers call, which makes a Task, starts the other on a coroutine with task_start, and returns the task, and the function that does the work, named with __async, whose returns give Result.OK so that ? fails the task (async.cz). A coroutine is a stack of its own: a fiber on Windows, and on POSIX memory that a few instructions of assembly switch the stack pointer and the registers a function must keep to, for x86-64 and AArch64. Switching also swaps the runtime's state of the stack: the chain of frames for traces, the deferred code, and the bounds the stack guard checks, so defer, ?, and traces work in a task as anywhere else; a coroutine starts with a copy of the frames that started it. await is prelude_await: on a coroutine, task_wait adds a callback that resumes it to the task and switches back to what resumed it; in the program's body it runs the loop until the task settles. Settling a task queues its callbacks to run soon on the loop. The collector scans, besides the stack that runs, the stack of each coroutine that waits from where it stopped, with the registers it saved there, and the program's own stack while a coroutine runs. Async functions were first planned as state machines in C; stacks of their own leave the rest of the code generator as it is instead: locals stay C locals, and await can be anywhere in an expression.

Strings are immutable char* values, so the runtime keeps a small cache from string pointer to length. Repeatedly indexing one string, as a lexer does, stays linear instead of rescanning it; the collector clears the cache so it never pins a freed string.

Memory management

Every Cuarzo value carries a small header and lives in one linked list. When the live heap passes a threshold, a conservative mark-and-sweep collector runs: it marks every object reachable from the CPU registers (flushed with setjmp), the native stack between the current frame and the program body's frame, and the global slots registered with cz_add_root (the constants that hold references), then frees the rest. Because the roots are ordinary C variables and nothing moves, no value the program still holds is ever freed, and pointers stay valid. The collector reclaims reference cycles, which reference counting could not.

This relies on how the generated code is built. Live values are on the stack or in callee-saved registers at each call, and main records a stack anchor above cz_program (the function holding the program body) to bound the scan; cz_program is CZ_NOINLINE, so it stays below the anchor when GCC optimizes. Optimized code may keep only a pointer into the middle of an object, such as to an array element, while it works on it, so any pointer into an object's data, or just past its end, keeps the object alive. With --optimize the driver passes -O2; -fno-strict-aliasing is always passed, since the runtime reads slots and array elements through more than one pointer type. Array element reads copy out through cz_array_get, pop clears the vacated slot, and growth leaves spare capacity zeroed, so no stale pointer keeps a dead value alive.

cz_start also measures the thread's stack (GetCurrentThreadStackLimits or getrlimit); each function's CZ_ENTER then reports deep recursion as a runtime error before the stack overflows.

For the trace of a runtime error, every generated function, cz_program included, starts with CZ_FRAME(name, file). It declares a CzFrame on the C stack that links to the caller's, and points cz_frame at it. Each statement starts with CZ_LINE(n), and while and else if conditions set it again as they run. Before each return the value is stored, then CZ_LEAVE points cz_frame back at the caller. Constants are set at the start of cz_program with CZ_AT(file, n), since they come from any file. cz_runtime_errorf puts the terminal back, prints the message, then walks the frames from cz_frame, showing a call that repeats itself once and at most CZ_TRACE_FRAMES frames before the program body.

The cuarzo command chooses runtime behavior with GCC defines: --no-gc (-DCZ_NO_GC) frees nothing until exit, --max-memory (-DCZ_MAX_MEMORY) caps the heap, --mem-stats (-DCZ_MEM_STATS) prints usage at exit, and --gc-stress (-DCZ_GC_STRESS) collects before every allocation for testing. The emitted C is identical regardless; only the define changes.

The runtime is split by concern into units in compiler/runtime/: core.c (headers, options, shared types, errors), memory.c, arrays.c, enums.c (members that carry values, Result, and function values), maps.c (hash maps that keep insertion order), arithmetic.c, text.c, and console.c (the console's settings, the command line, and writing) are part of every program; input.c (reading standard input), terminal.c (its size, raw mode, keys, and text written at exit), system.c (random numbers, the environment, the working directory, time, and the platform), files.c, and process.c are part of the programs that use them. runtime.cz lists them in order, and each is included with embed, which reads the file when the compiler itself is compiled. The compiler therefore carries its runtime and needs no files beside it, while the runtime stays ordinary C that editors and GCC can check on its own.

Each file starts with a comment that declares its unit:

/* @unit process
   @needs files
   @builtins run start_process poll_process wait_process
   @link windows -lws2_32
   @cflags unix -pthread
   @pkg linux gtk+-3.0

   What the unit is for. */

@always marks a unit every program has. Code generation records each built-in a program calls in used_builtins; units_for keeps the @always units, the units that list one of those built-ins, and the units they @need, in file order. A standard library module brings the units of the built-ins its functions call. Before the units, the C defines CZ_UNIT_<NAME> for each one, so a unit every program has can call into an optional one only when it is there, as cz_terminal_settle in console.c does with terminal.c. @link, @cflags, and @pkg (a pkg-config package) may start with a platform name, or unix for every platform but Windows; compile_program returns the units, and unit_build_flags in the driver turns them into GCC flags. No unit needs C++ or Objective-C yet; the first that does will add a @language line and its compilation in the driver.

Error boundaries

• Lexical, grammar, name, type, and arity problems stop compilation with [ERROR E0042] file:line:col: message on standard error and status 1, before any C is written, with a line for each related place. Up to ten are reported; see below. check --json prints them all as JSON instead.
• Problems with the command line or source file, and GCC failures, are reported as [ERROR] message.
• Runtime faults belong to the generated program and are reported as Cuarzo runtime error: ... on standard error with a nonzero status, followed by the trace of where the program was.

When improving diagnostics, preserve the phase boundary and add a test for the earliest phase that can reliably detect the error.

More than one error

The cuarzo command collects errors (collect_errors in diagnostics.cz): fail keeps the error and panics, and what is being checked catches it, with catch_panic, which unwinds to where it was called.

Code generation, which is also where types are checked, checks each expression under guard_expression (in gen_expr). An error in an expression gives it the error type, ERROR_TYPE (<error>), and checking goes on with what holds it, so each error of a statement is reported. The error type fits wherever a type is expected (assignable), a condition or a loop over a value of it checks its blocks as usual, and the variable of such a loop has it too. error_at does not report an error about a node with a part of the error type (has_error_part): that error is the first one's doing. A generic function is not made for a type that has the error type in it. Past GUARDED_DEPTH levels of an expression, an error leaves the shallower expression around it, so long chains keep to the stack.

The resolver and code generation check each statement with guard_resolve and guard_gen, which put the resolver or the generator back as they were and leave the statement out, as is_skipped then says; constants, field defaults, and the "can end without returning" check are statements too. Checking goes on with the next statement. A statement left out reports nothing more, so no error appears twice, and a variable declaration left out still declares its variable with the type written, so the code after it does not cascade into errors about that variable. Syntax errors are collected by the parser (see Parser and AST), and end the check after every file is parsed. Errors outside any statement, such as duplicate declarations and unknown types in signatures, end the check too. Name errors come before type errors, since the resolver checks the whole program before code generation. The program's C is not used when there was an error. Up to ERROR_LIMIT errors, ten for the command, are reported.

The canonical layout

format.cz lays code out for cuarzo fmt and editors from the tokens and comments, so it keeps the author's line breaks, literals as written, and comments where they were. It walks the lines with a stack of what is open, as the lexer does: brackets, and blocks with the indentation of the line that opened them and their kind (if, match, a match's branch, or any other). A line in a block is indented four spaces more than the line that opened it; end, else, and a match's when close what they end. A line inside brackets continues a statement and keeps how far right of it it was written; the bracket remembers that statement. laid_out spaces a line's tokens (spaced decides between two tokens), with f-strings and comments as written. assemble keeps one blank line at most, and none just inside a block. format_range gives lines one by one, for an editor's range. Code whose blocks or brackets do not close is not laid out. The syntax tests lay out every file of the repository and check that doing it again changes nothing and that the tokens stay the same; the CLI tests check that the repository is laid out.

The language server

lsp.cz reads messages of the Language Server Protocol (headers, then a body of Content-Length bytes) from standard input or a socket and answers them in order, building and reading JSON with std/json. Open documents are SOURCE_OVERLAYS. The first one opened starts a workspace (workspace.cz) for the program's main file; each change goes to change_file, and the diagnostics of the open documents, with lint's warnings, are sent after it. A document outside that program is analyzed on its own, without the parse cache. STOP_CHECKING asks input_waiting(), so a check stops when the editor sends something new and the next one takes it in. Each request maps to a query of queries.cz, or to format.cz or lint.cz, with positions converted between the compiler's byte columns and UTF-16 (or kept in bytes when the editor offers UTF-8). A request that fails answers with an error and the server goes on. The runtime unit lspio gives input_waiting and a TCP socket on the loopback address, loading Winsock at run time on Windows so the seed still links with -lm alone. The lsp tests drive scripted sessions, from a file and over a socket, and check the answers against the queries.

Debugging

With DEBUG_INFO (--debug), code generation puts a #line directive before each statement of a file on disk, and before each function, whose opening lines all take the function's line, so a debugger shows the .cz file and stops at its lines. After each function it writes #line cz-c, which the driver turns into the directive that gives the following code the C file's own lines again; the runtime's code and that of files not on disk, as the prelude's, keep them. bind_variable notes each variable of the function being generated, with the C that reads it (czv_x, or (*czv_x) for a shared one), and a function written as an expression notes what it took, read from its slots. debuginfo.cz writes the table of those functions, the constants, and every type they use, with the kind of each, its C type, and its parts.

dap.cz sits between the editor and the debugger's adapter. It forwards requests, and answers those about the program's code itself: stack traces are filtered to the frames of .cz files, the variables of a frame are read with C expressions the table gives, and walked by type, conditions and watches are translated from a small Cuarzo expression syntax to C, and hit counts and log messages are kept by continuing the program without telling the editor. Runtime errors stop at the runtime's cz_debug_error, which every runtime error passes, with its message.

Projects

project.cz finds the cuarzo.toml of a file or folder by looking in its folder and the folders above it, reads it with std/toml's read_toml, which gives the line of each key, and checks every part, collecting what is wrong as diagnostics with codes, so they are reported as the program's errors are. main.cz applies the profile chosen: it parses the profile's options, sets its constants, and parses the command line's options again, so they win. cuarzo check checks each program and library in turn with check_program, setting TARGET.platform for each platform it is for, and reports an error once, saying the platforms it was found on when they are not all those its file was checked for.

Tests

testrun.cz finds the test_, bench_, setup, and teardown functions of each *_test.cz file by its tokens, and makes its runner: the file's own text with an import of std/test and a call for each test at its end, which stands for the file (SOURCE_OVERLAYS) while it is checked and built, so errors point at the file's own lines. Each runner reports to an events file, a JSON line for each test as it starts and ends; a test that starts and never ends took its program down. With --coverage, code generation (COVERAGE) counts each line with a statement, and the program writes the counts as it ends to the file CUARZO_COVERAGE names.

Warnings

lint.cz walks each file of an analyzed program, but the standard library's, with a stack of scopes of variables, counting how each is read and set: a variable nothing reads is unused (W0001, W0002) or set and never read (W0007), and one declared with the name of an outer one hides it (W0006). Code after what ends a block never runs (W0005); a private function whose name its file never uses again is unused (W0004); an import is unused when its name, with as, is never written, or else when no name the file uses means a declaration of the module or of one it passes on (W0003, found with symbol_at). x = x and comparisons of a value with itself are mistakes (W0008, W0009). Warnings are diagnostics with severity WARNING, with fixes where a change is safe; comments with lint: allow or lint: allow-file silence them (silenced).

Questions about code

queries.cz answers what editors ask, for std/compiler and the language server. analyze runs check_program and keeps each file's text, tokens, and top-level nodes, with comments attached, and an index of the program's declarations and of each type's methods. The checker does not record what each name means; the queries find it again from the typed tree: a name's token, the nodes around it (nodes_at), and the variables visible there (locals_at, which walks the blocks before the place in order), or else the program's declarations by name, or, after a ., the members of the type checking gave the value before it. So a question about code with errors still finds what was read: in a statement that could not be parsed, a name means what a name used there would, and a variable's type is the one written in its declaration. References are the name tokens with the same text that mean the same declaration; a rename is their edits, with conflicts where the new name already means something.

Checking again after a change

workspace.cz checks a program again at each change, as an editor needs at each keystroke. open_workspace turns on PARSE_CACHING in modules.cz: each file's text, tokens, comments, imports, type names, and statements are kept while its text does not change, and parsing the program again only parses the files that changed. The trees are used again as they are. The resolver renames names in place, so it remembers each node before it changes it (remember), and parse_program puts the names back (restore_names) before resolving again; the checker only sets each expression's type, which gen_expr clears before it checks one. An editor's text that is not saved stands for the file (SOURCE_OVERLAYS), and a file is read again only when it is said to have changed (TRUST_CACHE, CHANGED_FILES).

A change checks the code of the files it can matter to: the changed file, and, when what it declares changed (declaration_text: signatures, fields, members, and constants, without the code of functions), every file that imports it, directly or not. CHECK_ONLY in codegen.cz makes code generation check only the functions and statements of those files; declarations, names, and syntax are checked in every file each time. The errors that checking the code of each file found are kept (BODY_ERRORS_START tells them from the others), so the other files keep theirs; an error in a generic function made for a use in a file checked again is dropped with that file's. STOP_CHECKING in diagnostics.cz is asked before each statement; when it says to stop, the check panics up to check_workspace, which gives NONE and takes the change in with the next. Memory follows the program: files that leave it are dropped from every cache.

Code generation finds declarations by name in Gen.declared (find_declared), not by searching lists, so checking a large program does not slow down with its number of declarations. tools/incremental_bench.cz times it all on a program of 50,000 lines; CI runs it with a limit of 100 ms for a change to a function's code.

std/compiler is std/compiler.cz, which passes on the compiler's own modules. The compiler carries them as standard library modules, compiler/<name>, and embed in them reads what the compiler carries (std_file_text in modules.cz): the prelude, the runtime units, and the standard library itself.

Diagnostics

A Diagnostic has, besides its place and message, a code, a severity, the range of code it covers, related places, and fixes. fail makes it: the code comes from code_of in codes.cz, which finds the kind of error whose message, with * for what changes, fits; when several do, the one with the most written text. The rest is set just before the error is reported, and taken by the next fail: error_at sets the range from the node's place and the parser's fail_at from the token's; relate adds a related place (a name declared twice relates to its first declaration); and suggest adds a fix, a title and text edits: closest_name for a misspelled variable, conversion_fix for a value that needs a conversion, the branch a match lacks, and the import of a module a name is declared in (import_name in resolver.cz). An error that is not reported, as one another causes, forgets what was set for it (forget_extras). diagnostic_text is how the command line shows one, and diagnostics_json how check --json does.

check_program in compiler.cz is checking for tools: it collects every error, without a limit, and gives them with the program's tree, whose expressions have their types in type; with syntax errors, the tree is as parsed, with ERROR nodes. It writes no C, reports nothing, and leaves the compiler as it found it, so a tool can check program after program in one process. compile_file, as the tests use it, stops at the first error.

Extension workflow

A language change normally travels through this dependency chain:

spelling -> token -> AST shape -> parser -> code generator -> test/example -> docs

Not every feature needs a new token or AST node, but every implemented feature must have parser, code-generation, and test evidence. A built-in with fixed parameter and result types needs only its runtime function and an entry in SIGNATURES in builtins.cz; its name is then reserved. After changing the compiler, rebuild the seed with tools/bootstrap.cz; see SELF_HOSTING.md.

Testing architecture

tests/main.cz builds a fresh compiler from the current sources into build/tests and runs these suites:

Unit checks of valid code run inside the test program, which imports the compiler modules. Anything that may fail runs the fresh compiler as a separate process, so one failure does not stop the run.