Testing

aux4 includes a markdown-based testing framework for writing and running tests against your commands.

Installation

> aux4 aux4 pkger install aux4/test

Usage

Run Tests

> aux4 test run [directory|file] [options]

Options:

--configFile <path> — Path to a config.yaml file for variable substitution
--config <section> — Config section name to load variables from
--aiConfig <section> — Config section for AI-powered validation (expect:ai, expect:ai:score)
--group <name> — Run only a specific test group (can be repeated for multiple groups)
--output <path> — Write test results as JSON to a file

Examples:

# Run all tests in current directory
> aux4 test run

# Run a specific test file
> aux4 test run test/greet.test.md

# Run with config variables
> aux4 test run --configFile config.yaml --config dev

# Run a specific group
> aux4 test run --group smoke

# Save results to JSON
> aux4 test run --output results.json

Add Tests

Programmatically add new tests to existing .test.md files:

> aux4 test add --testFile <file> --level <heading_level> --name <name> [--file <filename>] [--execute <command>]

Inspect Tests

Extract all files from a test scenario and save them to a directory. Useful for debugging test fixtures:

> aux4 test inspect --testFile <file> --name <scenario_name> --dir <output_directory>

Writing Tests with .test.md Files

aux4 tests are written in Markdown files with the .test.md extension. These files use a structured format with headings to define test scopes and code blocks for test execution and expectations.

Test Structure and Scope

Tests are organized hierarchically using Markdown headings:

# (H1): Defines the main test scenario/suite
## (H2-H6): Defines nested test scenarios within the parent scope

Each heading level creates a new test scope that can contain:

Setup/teardown commands
Test files
Execute/expect test pairs

Code Blocks

Execute Block

Defines the command to run:

```execute
echo "Hello World"
```

Expect Block

Defines the expected output with optional modifiers:

```expect
Hello World
```

Expect Modifiers

You can modify expect behavior using colon-separated modifiers:

partial: Match substring instead of exact match
ignoreCase: Case-insensitive matching
regex: Treat expected value as regular expression

Modifiers can be combined:

```expect:partial
Hello
```

```expect:ignoreCase
hello world
```

```expect:regex
Hello.*World
```

```expect:partial:ignoreCase
hello
```

```expect:regex:ignoreCase
error code: [a-z]+\d+
```

JSON Modifier

Use expect:json to compare JSON output. Both the actual output and expected value are parsed as JSON before comparison, so formatting differences don't matter:

```execute
aux4 config get dev
```

```expect:json
{
  "host": "localhost",
  "port": 3000
}
```

Can be combined with other modifiers:

```expect:json:partial
{
  "host": "localhost"
}
```

AI Validation

Use expect:ai to validate output using an LLM. The expect block contains a natural language description of what the output should be. Requires the aux4/ai-agent package installed and --aiConfig flag:

```execute
aux4 greet --name Alice
```

```expect:ai
The output should be a friendly greeting that includes the name Alice
```

> aux4 test run --configFile config.yaml --aiConfig ai

AI Scoring

Use expect:ai:score to score output on a numeric scale. Define the evaluation criteria and pass threshold:

```execute
aux4 greet --name Alice
```

```expect:ai:score
name: friendliness
eval: How friendly is this greeting?
range: 1-5
pass: 3
```

Fields:

name — Score label shown in output
eval — Evaluation prompt for the LLM
range — Numeric scale (default: 1-5)
pass — Minimum passing score (default: 3)

Text Similarity

Use expect:similar to compare output using deterministic text similarity metrics without AI:

```execute
aux4 greet --name Alice
```

```expect:similar
metric: fuzzy
pass: 0.9
---
Hello Alice! Welcome aboard.
```

Fields:

metric — Similarity algorithm: fuzzy (Levenshtein, default), cosine (word vectors), jaccard (word set overlap)
pass — Minimum similarity score from 0 to 1 (default: 0.8)
After ---, the reference text to compare against
Can also use file: path/to/reference.txt to load reference from a file

Can be combined with :ignoreCase:

Wildcard Patterns in Partial Matching

When using expect:partial, you can use special wildcard patterns:

*?: Match any characters on a single line
**: Match any characters across multiple lines (multiline wildcard)

```execute
echo "File: test.txt Size: 1024 bytes"
```

```expect:partial
File: *? Size: *? bytes
```

```execute
echo -e "Header: Processing\n\nMultiple lines\nof output\n\nFooter: Complete"
```

```expect:partial
Header:**Footer: Complete
```

You can combine wildcards:

```execute
echo -e "File: test.txt\nSize: 1024 bytes\nModified: today"
```

```expect:partial
File: *?**Modified: *?
```

Multiple Expect Blocks

A single execute command can have multiple expect blocks with different modifiers:

```execute
echo "[Line 1](lines_1): Hello World
[Line 2](lines_2): Testing 123
[Line 3](lines_3): Final result"
```

```expect:partial
Hello World
```

```expect:partial
Testing 123
```

```expect:regex
Line \d+: Hello World
```

Error Block

Expect specific error output:

```error
Command not found
```

Error blocks support the same modifiers as expect blocks (partial, ignoreCase, regex).

Timeout Block

Configure a custom timeout for tests in milliseconds. This is useful for long-running commands:

```timeout
5000
```

```execute
sleep 2 && echo "Command completed"
```

```expect
Command completed
```

If no timeout is specified, the default timeout is used.

Setup and Teardown Blocks

beforeAll: Run once before all tests in the scope
afterAll: Run once after all tests in the scope
beforeEach: Run before each test in the scope
afterEach: Run after each test in the scope

```beforeAll
mkdir temp
```

```afterAll
rm -rf temp
```

File Block

Create temporary files for testing:

```file:test.txt
This is test content
that spans multiple lines
```

File Scope

Files are scoped to their parent heading level and all nested tests within it. Files created at one level are:

Available to all nested tests (deeper heading levels)
Not available to sibling tests (same heading level)
Not available to tests in other parent sections

## Parent Scenario

```file:parent-file.txt
This file is created at parent level
```

### Nested Test - File Available

```execute
cat parent-file.txt
```

```expect
This file is created at parent level
```

## Sibling Scenario - File NOT Available

```execute
cat parent-file.txt 2>/dev/null || echo "File not found"
```

```expect
File not found
```

Flexible Backticks

When your test content contains markdown code blocks, you can use more than 3 backticks for the outer block. The test parser will correctly match the same number of backticks:

````execute
printf "Code block:\n\`\`\`javascript\nconsole.log(\"hello\");\n\`\`\`\n"
````

````expect
Code block:
```javascript
console.log("hello");
```
````

This works for all block types: execute, expect, error, file, timeout, and hooks.

Dataset Block

Datasets allow you to parameterize tests with JSON data. Each entry in the dataset generates a separate test run:

```dataset
file: users.json
root: $.items
key: name
```

```execute
aux4 greet --name {{name}}
```

```expect:partial
Hello {{name}}
```

Fields:

file — Path to a JSON file
root — JSONPath expression to extract an array from nested JSON (optional)
key — Field name used for test labeling in output (optional)

Shorthand syntax for simple cases:

```dataset:users.json
```

Given users.json:

[
  { "name": "Alice", "age": 30 },
  { "name": "Bob", "age": 25 }
]

This generates two tests: one for Alice and one for Bob. Each entry's fields become available as {{fieldName}} variables.

Variable Substitution

Test files support {{variableName}} substitution in execute, expect, error, and file blocks. Variables come from:

Config files — Pass --configFile config.yaml --config section to load variables
Datasets — Each dataset entry's fields become variables

> aux4 test run --configFile config.yaml --config dev

Given config.yaml:

config:
  dev:
    host: localhost
    port: 3000

You can use {{host}} and {{port}} in your tests:

```execute
curl http://{{host}}:{{port}}/health
```

```expect
ok
```

Test Suites

Group test files into suites using test.suite.md files. Suites let you organize tests into named groups and run specific groups with the --group flag:

## smoke
- health.test.md
- auth.test.md

## integration
- api.test.md
- database.test.md

## performance (optional)
- load.test.md

Groups marked (optional) only run when explicitly requested:

# Run all non-optional groups
> aux4 test run

# Run only smoke tests
> aux4 test run --group smoke

# Run multiple groups
> aux4 test run --group smoke --group integration

# Run optional group
> aux4 test run --group performance

Example Test File

# Echo Tests

## Basic functionality

```beforeAll
mkdir -p test-output
```

```afterAll
rm -rf test-output
```

### Exact match

```execute
echo "Hello, World!"
```

```expect
Hello, World!
```

### Case insensitive matching

```execute
echo "Hello, World!"
```

```expect:ignoreCase
hello, world!
```

### Partial matching

```execute
echo "The quick brown fox jumps"
```

```expect:partial
quick brown
```

### Regular expression matching

```execute
echo "Version 1.2.3"
```

```expect:regex
Version \d+\.\d+\.\d+
```

## Error handling

```execute
nonexistent-command
```

```error:partial
command not found
```

Adding Tests with the Add Command

The aux4 test add command allows you to programmatically add new tests to existing .test.md files. This is useful for automation or when you want to quickly scaffold new test cases.

When you run aux4 test add, it executes the command and automatically includes an expect block with the current output.

Command Syntax

> aux4 test add --testFile <test_file> --level <heading_level> --name <test_name> [options]

Parameters

--testFile: Path to the .test.md file where the test should be added
--level: Heading level (1-6) that determines the test scope hierarchy
--name: Name of the test (will become the heading text)
--file <filename>: Add file blocks to the test (can be used multiple times). Reads the content of existing files.
--execute <command>: Add an execute block with the specified command

Examples

The add command generates markdown content that follows the test file structure. Here are examples of what gets generated:

Simple test with execution

> aux4 test add --testFile my.test.md --level 2 --name "Test echo command" --execute "echo 'Hello World'"

## Test echo command

```execute
echo 'Hello World'
```

```expect
Hello World
```

Test with file setup

> aux4 test add --testFile api.test.md --level 2 --name "Test API endpoint" --file config.json --file data.txt --execute "curl -X POST localhost:3000/api"

The --file option reads the content of existing files and adds them to the test as file blocks:

## Test API endpoint

```file:config.json
{
  "apiUrl": "http://localhost:3000",
  "timeout": 5000
}
```

```file:data.txt
sample data content
```

```execute
curl -X POST localhost:3000/api
```

```expect
{"status": "ok"}
```

Nested test

> aux4 test add --testFile integration.test.md --level 3 --name "Error handling" --execute "invalid-command"

### Error handling

```execute
invalid-command
```

```expect
command not found
```

The add command will insert the new test at the appropriate location in the file based on the heading level, maintaining the hierarchical structure of your test file.

Testing aux4 Commands

You can test aux4 commands themselves using .test.md files. This is particularly useful for testing custom aux4 plugins or validating aux4 behavior.

Setting up aux4 Commands for Testing

To test aux4 commands, you typically need a .aux4 file that defines the command behavior. Here's an example of testing a simple greeting command:

# aux4 Hello Command Test

## Setup aux4 command

```file:.aux4
{
  "profiles": [
    {
      "name": "main",
      "commands": [
        {
          "name": "hello",
          "execute": [
            "echo \"Hello, ${name}\""
          ],
          "help": {
            "text": "Simple greeting command",
            "variables": [
              {
                "name": "name",
                "text": "Name to greet"
              }
            ]
          }
        }
      ]
    }
  ]
}
```

### Test greeting command

```execute
aux4 hello --name John
```

```expect
Hello, John
```

### Test with different name

```execute
aux4 hello --name "Alice Smith"
```

```expect
Hello, Alice Smith
```

Table of Contents

Table of Contents

Table of Contents

Table of Contents

Table of Contents

Table of Contents

Table of Contents

Table of Contents

Table of Contents

Table of Contents

Table of Contents

Table of Contents

Table of Contents

Table of Contents

Table of Contents

Table of Contents

Table of Contents

Testing

Installation

Usage

Run Tests

Add Tests

Inspect Tests

Writing Tests with .test.md Files

Test Structure and Scope

Code Blocks

Execute Block

Expect Block

Expect Modifiers

JSON Modifier

AI Validation

AI Scoring

Text Similarity

Wildcard Patterns in Partial Matching

Multiple Expect Blocks

Error Block

Timeout Block

Setup and Teardown Blocks

File Block

File Scope

Flexible Backticks

Dataset Block

Variable Substitution

Test Suites

Example Test File

Adding Tests with the Add Command

Command Syntax

Parameters

Examples

Simple test with execution

Test with file setup

Nested test

Testing aux4 Commands

Setting up aux4 Commands for Testing