Detail Bug Report

app.detail.dev/org_0f605249-55f3-46eb-880c-20ae3e45ae99/…/bug_b963f2a2-17d2-4cb9-b4db-a53ebe769dd2

Introduced in #10 by @ripgrim on Jul 22, 2026

Summary

• Context: packages/sdk/src/stored-rule.ts (storedRuleIssue) is the author-time gate that runs on every custom-rule save (packages/db/src/services/custom-rules.ts → customRuleDefinitionIssue, called by the web server fn, the MCP save_custom_rule tool, and the CLI). Its stated contract is to "mirror what the evaluator enforces at run time" so bad rules are "reject[ed] … before they store" (packages/sdk/src/stored-rule.ts doc comment).
• Bug: The verb-against-signal-kind check is applied only to the top-level comparison; the nested comparison inside a not wrapper is validated by argIssue recursively, which checks arg types but never checks the inner verb's applicability to the signal kind.
• Actual vs. expected: A rule like { kind: "not", args: [{ kind: "under", args: ["main"] }] } on the text signal pr.title returns null (accepted, storable) from storedRuleIssue, while evaluateSignalRule throws SignalEvaluationError: comparison under does not apply to text signals on every evaluation — the write path accepts a rule the evaluator unconditionally rejects.
• Impact: Any saved rule with a cross-kind verb inside not never enforces: every evaluation of it is caught by the worker and recorded as skipped, degrading the workflow to the "A rule could not run" / needs-human-review path (or silently passing where the fail-closed floor is off). Reachable through the MCP tool and CLI, which accept arbitrary definition JSON and bypass the UI builder.

Code with Bug

packages/sdk/src/stored-rule.ts

const verbs = VERBS_FOR_KIND[comparedKind];
if (!verbs?.includes(rule.comparison.kind)) {
    return `${rule.comparison.kind} does not apply to this signal`;
}
return argIssue(comparedKind, rule.comparison);

if (verb === "not") {
    const inner = args[0];
    if (
        typeof inner !== "object" ||
        inner === null ||
        typeof (inner as { kind?: unknown }).kind !== "string" ||
        !Array.isArray((inner as { args?: unknown }).args)
    ) {
        return "this rule is missing the comparison it negates";
    }
    return argIssue(kind, inner as { kind: string; args: readonly unknown[] }); // <-- BUG 🔴 inner verb's kind-applicability is never checked; only its arg types are
}

Explanation

storedRuleIssue enforces a “verb allowed for this signal kind” gate only at the top level (VERBS_FOR_KIND[comparedKind]). When the comparison is not, argIssue unwraps the inner comparison and recurses directly back into argIssue without re-checking inner.kind against VERBS_FOR_KIND.

As a result, invalid cross-kind comparisons (e.g. text signal wrapped around a list-only verb like under) pass validation and can be stored, but the evaluator’s per-kind switch rejects them at runtime and throws SignalEvaluationError.

Codebase Inconsistency

• The evaluator is strict for nested comparisons: in packages/sdk/src/evaluate.ts, not evaluates by dispatching the inner comparison through the same per-kind logic and fails with comparison ${comparison.kind} does not apply to text signals for invalid kinds.
• Existing tests in packages/sdk/src/stored-rule.test.ts already assert cross-kind verbs are rejected at the top level (e.g., anyIn on text, containsAny on list), but there is no equivalent test for those same verbs when nested under not.

Recommended Fix

In the not branch of argIssue in packages/sdk/src/stored-rule.ts, apply the same VERBS_FOR_KIND gate to inner.kind before recursing, and add tests that wrap the existing cross-kind verb cases in not to ensure the validator mirrors runtime behavior.

History

This bug was introduced in commit 67bd7145 (feat(custom-rules): runtime catalog, stored rules, generic evaluation).