Evaluation Deep Dive

How SigmaShake evaluates a tool call against governance rules.

Algorithm

evaluate(rules: Rule[], call: ToolCall): EvalResult

1. Sort rules by priority descending (100 before 50)
2. Filter to enabled rules only
3. For each rule in sorted order: a. Check targetMatches(rule.target, call.tool) — does the rule apply to this tool type? b. Check groupMatches(rule.groups, call) — do any condition groups match? c. If both match → return immediately (short-circuit, first-match wins)
4. Default: return {decision: "allow"} if no rule matches

Decision vocabulary: allow, block (DSL DENY), ask, force, log, and shadow. A force result refuses the original risky action and returns a safe substitute or required safer path for the agent to use next.

Target matching

A rule's target is compared against the tool's capability:

DENY execution  →  only matches tools classified as "execute" (Bash, shell)
DENY any        →  matches all tools

The capability is determined by the active client adapter, which maps tool names to the 6-capability taxonomy (execute, read, write, search, agent, network).

Condition evaluation

Groups (OR logic)

Conditions separated by OR form separate groups. Any group matching is sufficient:

IF command CONTAINS "rm -rf"     ← Group 1
OR command CONTAINS "rm -r"      ← Group 2

Group 1 matches OR Group 2 matches → rule fires.

Within a group (AND logic)

Conditions joined by AND (or on consecutive lines after IF) must all match:

IF path ENDS_WITH ".env"         ← Group 1, condition 1
AND content CONTAINS "API_KEY"   ← Group 1, condition 2

Both must match for the group to match.

Field resolution

Fields are extracted from the ToolCall input:

function getField(field: Field, call: ToolCall): string {
  if (field.startsWith('input.')) {
    const key = field.slice(6);
    const val = call.input[key];
    if (val === undefined || val === null) return '';
    return typeof val === 'string' ? val : JSON.stringify(val);
  }
  switch (field) {
    case 'command': return call.input.command ?? '';
    case 'path':    return call.input.file_path ?? call.input.path ?? '';
    case 'content': return call.input.content ?? call.input.new_string ?? '';
    case 'tool':    return call.tool;
    default:        return '';
  }
}

Performance

Metric	Typical value
Evaluation latency	Design target under 2 ms; measured p50 0.39 ms, p99 1.91 ms (2026-06-10 daemon-socket bench run, ssg 0.29.156, n=500)
Rules loaded	20-50
Glob cache size	Up to 1000 patterns
Regex cache size	Up to 1000 patterns

Safety features

Fail-secure regex

If a regex pattern is invalid or dangerous (nested quantifiers), it matches rather than silently failing. This triggers the rule, erring on the side of caution.

Loop guard

Detects when the same call is repeated consecutively and blocks it to prevent infinite loops. The engine loop guard blocks at 5 identical calls. ssg eval blocks the 3rd consecutive identical Bash command.

No circuit breaker

The shipped Go engine has no circuit breaker. It never auto-allows a call after repeated DENY decisions, and it ignores circuit_breaker in the [hook] block and SSG_HOOK_CIRCUIT_BREAKER. A circuit breaker existed in the legacy TypeScript engine only. 2026-10-11 note: earlier versions of this page described it as an option. It is not shipped.

ASK approval mode

When a rule returns an ASK decision, the engine needs a human to approve or deny before the tool call proceeds. In the Claude Code hook, the hook posts the approval request to the ssg web dashboard (ssg serve must be running) and waits for the answer. The wait is 20 seconds by default and 45 seconds at most. If no answer arrives in that time, the hook returns the ASK decision to the host harness, and the harness applies its own ASK behavior.

The ask_mode setting below is a legacy setting from the TypeScript engine only. The Go hook ignores ask_mode and always uses the dashboard wait described above. There is no inline terminal (tty) prompt in the Go hook.

Legacy mode (TypeScript only)	Behavior
tty	Printed an inline prompt to the terminal via /dev/tty. It was the TypeScript default. The Go hook does not use it.
dashboard	Posted the approval request to the ssg web dashboard. The Go hook always does this, with no setting needed.

Legacy TypeScript .sigmashake/config.toml example (the Go hook does not read it):

[hook]
ask_mode = "tty"          # legacy TypeScript default; ignored by the Go hook
# ask_mode = "dashboard"  # legacy TypeScript opt-in; ignored by the Go hook

Do not use the --ask_mode flag of ssg eval to choose the approval channel of the hook. The Go hook does not read ask_mode.

Note 2026-10-13: an earlier version of this page presented ask_mode = "tty" as the default and as a setting you can change. The Go hook has no such setting.

If /dev/tty is unavailable (headless CI), the legacy TTY mode falls back with a block decision and a message on stderr.

Governed terminal panes (the dashboard's terminal) resolve ASK inline: the pane prints [SSG] GOVERNANCE: <reason> followed by Allow anyway? [y/N], and only y runs the command (Enter = no). No dashboard approval is involved. The pane's guard achieves this with ssg hook eval --json --defer_ask, which returns the unresolved verdict as {"decision":"ask"} with exit code 3 instead of waiting on the dashboard or applying ask_fallback. --defer_ask is flag-only — it cannot be enabled from config.toml or the environment — so every other --json caller keeps the fail-closed ask_fallback behaviour.

Opting a terminal pane out of governance entirely is an operator-owned config switch. It affects only dashboard terminal panes (new and resumed); agent hook evaluation is unchanged, and any absent or unrecognised value keeps panes governed:

[terminal]
govern = false   # open dashboard terminal panes as plain shells — nothing typed there is evaluated

An ungoverned pane starts with the banner [SSG] terminal governance is OFF ([terminal].govern = false in config.toml).