# Testing

aux4 includes a markdown-based testing framework for writing and running tests against your commands.

## Installation

Install [aux4/test](https://hub.aux4.io/r/public/packages/aux4/test):

```bash
> aux4 aux4 pkger install aux4/test
```

## Usage

### Run Tests

```bash
> aux4 test run [directory|file] [options]
```

**Options:**

- `--configFile <path>` — Path to a config.yaml file for variable substitution
- `--config <section>` — Config section name to load variables from
- `--aiConfig <section>` — Config section for AI-powered validation (`expect:ai`, `expect:ai:score`)
- `--group <name>` — Run only a specific test group (can be repeated for multiple groups)
- `--output <path>` — Write test results as JSON to a file

**Examples:**

```bash
# Run all tests in current directory
> aux4 test run

# Run a specific test file
> aux4 test run test/greet.test.md

# Run with config variables
> aux4 test run --configFile config.yaml --config dev

# Run a specific group
> aux4 test run --group smoke

# Save results to JSON
> aux4 test run --output results.json
```

### Add Tests

Programmatically add new tests to existing `.test.md` files:

```bash
> aux4 test add --testFile <file> --level <heading_level> --name <name> [--file <filename>] [--execute <command>]
```

### Inspect Tests

Extract all files from a test scenario and save them to a directory. Useful for debugging test fixtures:

```bash
> aux4 test inspect --testFile <file> --name <scenario_name> --dir <output_directory>
```

## Writing Tests with .test.md Files

aux4 tests are written in Markdown files with the `.test.md` extension. These files use a structured format with headings to define test scopes and code blocks for test execution and expectations.

### Test Structure and Scope

Tests are organized hierarchically using Markdown headings:

- **# (H1)**: Defines the main test scenario/suite
- **## (H2-H6)**: Defines nested test scenarios within the parent scope

Each heading level creates a new test scope that can contain:

- Setup/teardown commands
- Test files
- Execute/expect test pairs

### Code Blocks

#### Execute Block

Defines the command to run:

````markdown
```execute
echo "Hello World"
```
````

#### Expect Block

Defines the expected output with optional modifiers:

````markdown
```expect
Hello World
```
````

#### Expect Modifiers

You can modify expect behavior using colon-separated modifiers:

- **`partial`**: Match substring instead of exact match
- **`ignoreCase`**: Case-insensitive matching
- **`regex`**: Treat expected value as regular expression

Modifiers can be combined:

````markdown
```expect:partial
Hello
```

```expect:ignoreCase
hello world
```

```expect:regex
Hello.*World
```

```expect:partial:ignoreCase
hello
```

```expect:regex:ignoreCase
error code: [a-z]+\d+
```
````

#### JSON Modifier

Use `expect:json` to compare JSON output. Both the actual output and expected value are parsed as JSON before comparison, so formatting differences don't matter:

````markdown
```execute
aux4 config get dev
```

```expect:json
{
  "host": "localhost",
  "port": 3000
}
```
````

Can be combined with other modifiers:

````markdown
```expect:json:partial
{
  "host": "localhost"
}
```
````

#### AI Validation

Use `expect:ai` to validate output using an LLM. The expect block contains a natural language description of what the output should be. Requires the [aux4/ai-agent](https://hub.aux4.io/r/public/packages/aux4/ai-agent) package installed and `--aiConfig` flag:

````markdown
```execute
aux4 greet --name Alice
```

```expect:ai
The output should be a friendly greeting that includes the name Alice
```
````

```bash
> aux4 test run --configFile config.yaml --aiConfig ai
```

#### AI Scoring

Use `expect:ai:score` to score output on a numeric scale. Define the evaluation criteria and pass threshold:

````markdown
```execute
aux4 greet --name Alice
```

```expect:ai:score
name: friendliness
eval: How friendly is this greeting?
range: 1-5
pass: 3
```
````

Fields:
- `name` — Score label shown in output
- `eval` — Evaluation prompt for the LLM
- `range` — Numeric scale (default: `1-5`)
- `pass` — Minimum passing score (default: `3`)

#### Text Similarity

Use `expect:similar` to compare output using deterministic text similarity metrics without AI:

````markdown
```execute
aux4 greet --name Alice
```

```expect:similar
metric: fuzzy
pass: 0.9
---
Hello Alice! Welcome aboard.
```
````

Fields:
- `metric` — Similarity algorithm: `fuzzy` (Levenshtein, default), `cosine` (word vectors), `jaccard` (word set overlap)
- `pass` — Minimum similarity score from 0 to 1 (default: `0.8`)
- After `---`, the reference text to compare against
- Can also use `file: path/to/reference.txt` to load reference from a file

Can be combined with `:ignoreCase`:

#### Wildcard Patterns in Partial Matching

When using `expect:partial`, you can use special wildcard patterns:

- **`*?`**: Match any characters on a single line
- **`**`**: Match any characters across multiple lines (multiline wildcard)

````markdown
```execute
echo "File: test.txt Size: 1024 bytes"
```

```expect:partial
File: *? Size: *? bytes
```

```execute
echo -e "Header: Processing\n\nMultiple lines\nof output\n\nFooter: Complete"
```

```expect:partial
Header:**Footer: Complete
```
````

You can combine wildcards:

````markdown
```execute
echo -e "File: test.txt\nSize: 1024 bytes\nModified: today"
```

```expect:partial
File: *?**Modified: *?
```
````

#### Multiple Expect Blocks

A single execute command can have multiple expect blocks with different modifiers:

````markdown
```execute
echo "Line 1: Hello World
Line 2: Testing 123
Line 3: Final result"
```

```expect:partial
Hello World
```

```expect:partial
Testing 123
```

```expect:regex
Line \d+: Hello World
```
````

#### Error Block

Expect specific error output:

````markdown
```error
Command not found
```
````

Error blocks support the same modifiers as expect blocks (`partial`, `ignoreCase`, `regex`).

#### Timeout Block

Configure a custom timeout for tests in milliseconds. This is useful for long-running commands:

````markdown
```timeout
5000
```

```execute
sleep 2 && echo "Command completed"
```

```expect
Command completed
```
````

If no timeout is specified, the default timeout is used.

#### Setup and Teardown Blocks

- **`beforeAll`**: Run once before all tests in the scope
- **`afterAll`**: Run once after all tests in the scope
- **`beforeEach`**: Run before each test in the scope
- **`afterEach`**: Run after each test in the scope

````markdown
```beforeAll
mkdir temp
```

```afterAll
rm -rf temp
```
````

#### File Block

Create temporary files for testing:

````markdown
```file:test.txt
This is test content
that spans multiple lines
```
````

##### File Scope

Files are scoped to their parent heading level and all nested tests within it. Files created at one level are:
- Available to all nested tests (deeper heading levels)
- **Not** available to sibling tests (same heading level)
- **Not** available to tests in other parent sections

````markdown
## Parent Scenario

```file:parent-file.txt
This file is created at parent level
```

### Nested Test - File Available

```execute
cat parent-file.txt
```

```expect
This file is created at parent level
```

## Sibling Scenario - File NOT Available

```execute
cat parent-file.txt 2>/dev/null || echo "File not found"
```

```expect
File not found
```
````

#### Flexible Backticks

When your test content contains markdown code blocks, you can use more than 3 backticks for the outer block. The test parser will correctly match the same number of backticks:

`````markdown
````execute
printf "Code block:\n\`\`\`javascript\nconsole.log(\"hello\");\n\`\`\`\n"
````

````expect
Code block:
```javascript
console.log("hello");
```
````
`````

This works for all block types: `execute`, `expect`, `error`, `file`, `timeout`, and hooks.

#### Dataset Block

Datasets allow you to parameterize tests with JSON data. Each entry in the dataset generates a separate test run:

````markdown
```dataset
file: users.json
root: $.items
key: name
```

```execute
aux4 greet --name {{name}}
```

```expect:partial
Hello {{name}}
```
````

Fields:
- `file` — Path to a JSON file
- `root` — JSONPath expression to extract an array from nested JSON (optional)
- `key` — Field name used for test labeling in output (optional)

Shorthand syntax for simple cases:

````markdown
```dataset:users.json
```
````

Given `users.json`:

```json
[
  { "name": "Alice", "age": 30 },
  { "name": "Bob", "age": 25 }
]
```

This generates two tests: one for Alice and one for Bob. Each entry's fields become available as `{{fieldName}}` variables.

#### Variable Substitution

Test files support `{{variableName}}` substitution in execute, expect, error, and file blocks. Variables come from:

1. **[Config files](/config)** — Pass `--configFile config.yaml --config section` to load variables
2. **Datasets** — Each dataset entry's fields become variables

```bash
> aux4 test run --configFile config.yaml --config dev
```

Given `config.yaml`:

```yaml
config:
  dev:
    host: localhost
    port: 3000
```

You can use `{{host}}` and `{{port}}` in your tests:

````markdown
```execute
curl http://{{host}}:{{port}}/health
```

```expect
ok
```
````

#### Test Suites

Group test files into suites using `test.suite.md` files. Suites let you organize tests into named groups and run specific groups with the `--group` flag:

````markdown
## smoke
- health.test.md
- auth.test.md

## integration
- api.test.md
- database.test.md

## performance (optional)
- load.test.md
````

Groups marked `(optional)` only run when explicitly requested:

```bash
# Run all non-optional groups
> aux4 test run

# Run only smoke tests
> aux4 test run --group smoke

# Run multiple groups
> aux4 test run --group smoke --group integration

# Run optional group
> aux4 test run --group performance
```

### Example Test File

````markdown
# Echo Tests

## Basic functionality

```beforeAll
mkdir -p test-output
```

```afterAll
rm -rf test-output
```

### Exact match

```execute
echo "Hello, World!"
```

```expect
Hello, World!
```

### Case insensitive matching

```execute
echo "Hello, World!"
```

```expect:ignoreCase
hello, world!
```

### Partial matching

```execute
echo "The quick brown fox jumps"
```

```expect:partial
quick brown
```

### Regular expression matching

```execute
echo "Version 1.2.3"
```

```expect:regex
Version \d+\.\d+\.\d+
```

## Error handling

```execute
nonexistent-command
```

```error:partial
command not found
```
````

## Adding Tests with the Add Command

The `aux4 test add` command allows you to programmatically add new tests to existing `.test.md` files. This is useful for automation or when you want to quickly scaffold new test cases.

When you run `aux4 test add`, it executes the command and automatically includes an `expect` block with the current output.

### Command Syntax

```bash
> aux4 test add --testFile <test_file> --level <heading_level> --name <test_name> [options]
```

### Parameters

- **`--testFile`**: Path to the `.test.md` file where the test should be added
- **`--level`**: Heading level (1-6) that determines the test scope hierarchy
- **`--name`**: Name of the test (will become the heading text)
- **`--file <filename>`**: Add file blocks to the test (can be used multiple times). Reads the content of existing files.
- **`--execute <command>`**: Add an execute block with the specified command

### Examples

The add command generates markdown content that follows the test file structure. Here are examples of what gets generated:

#### Simple test with execution

```bash
> aux4 test add --testFile my.test.md --level 2 --name "Test echo command" --execute "echo 'Hello World'"
```

````markdown
## Test echo command

```execute
echo 'Hello World'
```

```expect
Hello World
```
````

#### Test with file setup

```bash
> aux4 test add --testFile api.test.md --level 2 --name "Test API endpoint" --file config.json --file data.txt --execute "curl -X POST localhost:3000/api"
```

The `--file` option reads the content of existing files and adds them to the test as file blocks:

````markdown
## Test API endpoint

```file:config.json
{
  "apiUrl": "http://localhost:3000",
  "timeout": 5000
}
```

```file:data.txt
sample data content
```

```execute
curl -X POST localhost:3000/api
```

```expect
{"status": "ok"}
```
````

#### Nested test

```bash
> aux4 test add --testFile integration.test.md --level 3 --name "Error handling" --execute "invalid-command"
```

````markdown
### Error handling

```execute
invalid-command
```

```expect
command not found
```
````

The add command will insert the new test at the appropriate location in the file based on the heading level, maintaining the hierarchical structure of your test file.

## Testing aux4 Commands

You can test aux4 commands themselves using `.test.md` files. This is particularly useful for testing custom aux4 plugins or validating aux4 behavior.

### Setting up aux4 Commands for Testing

To test aux4 commands, you typically need a `.aux4` file that defines the command behavior. Here's an example of testing a simple greeting command:

````markdown
# aux4 Hello Command Test

## Setup aux4 command

```file:.aux4
{
  "profiles": [
    {
      "name": "main",
      "commands": [
        {
          "name": "hello",
          "execute": [
            "echo \"Hello, ${name}\""
          ],
          "help": {
            "text": "Simple greeting command",
            "variables": [
              {
                "name": "name",
                "text": "Name to greet"
              }
            ]
          }
        }
      ]
    }
  ]
}
```

### Test greeting command

```execute
aux4 hello --name John
```

```expect
Hello, John
```

### Test with different name

```execute
aux4 hello --name "Alice Smith"
```

```expect
Hello, Alice Smith
```
````