<!-- SPDX-License-Identifier: Apache-2.0
     https://www.apache.org/licenses/LICENSE-2.0 -->

# setup-override-upstream evals

Behavioral eval suite for the `setup-override-upstream` skill — 15 cases across 4 steps.

## Suites

| Suite | Step | Cases | What it covers |
|---|---|---|---|
| `step-0-preflight` | Step 0 (pre-flight) | 4 | not-adopter-repo stops; no-drift proceeds; ref drift proposes non-blocking upgrade; SHA-512 drift is security-flagged and blocking |
| `step-1-pick-override` | Step 1 (pick override) | 4 | zero overrides stops; single override auto-picked; multiple overrides asks user; injection in override content flagged |
| `step-3-decide-upstreamable` | Step 3 (upstreamability decision) | 4 | project-specific wording stops; missing feature continues; better default continues; injection in override flagged |
| `step-6-pr-confirm` | Step 6 (PR confirmation) | 3 | all sections present shown to user; user confirms → post; user cancels → abort |

## Run

```bash
# All cases
uv run --project tools/skill-evals skill-eval \
    tools/skill-evals/evals/setup-override-upstream/

# Single suite
uv run --project tools/skill-evals skill-eval \
    tools/skill-evals/evals/setup-override-upstream/step-0-preflight/fixtures/

# Single case
uv run --project tools/skill-evals skill-eval \
    tools/skill-evals/evals/setup-override-upstream/step-0-preflight/fixtures/case-1-not-adopter-repo
```

## What the suites cover

### step-0-preflight

The skill checks three conditions before doing anything: (1) the repo is an
adopted magpie repo, (2) the snapshot is not drifted from the committed lock,
and (3) a framework clone is available for the implementation step.  This suite
covers the first two.

Four branches:
- **case-1** (not-adopter-repo) — no `.apache-magpie.lock` or
  `.apache-magpie-overrides/`; action is `stop`.
- **case-2** (no-drift) — both lock files in sync; action is `proceed`.
- **case-3** (drift-ref) — same method + URL but committed ref is newer than
  local; action is `propose-upgrade-nonblocking` (user may defer).
- **case-4** (drift-sha512) — `svn-zip` SHA-512 mismatch; action is
  `propose-upgrade-blocking` (security-flagged; must resolve before designing
  a framework abstraction against a potentially stale snapshot).

### step-1-pick-override

The skill lists `.apache-magpie-overrides/*.md` (excluding `README.md`) and
dispatches on the count.

Four branches:
- **case-1** (zero-overrides) — nothing to upstream; action is `stop`.
- **case-2** (one-override) — auto-pick the single file.
- **case-3** (multiple-overrides) — ask the user which file to upstream this run.
- **case-4** (injection-flagged) — override content contains an adversarial
  directive; `injection_flagged: true` is set while the skill continues with
  the auto-pick (the injection is flagged, not silently executed).

### step-3-decide-upstreamable

The skill classifies the override against the four decision categories from the
skill's Step 3 table.

Four branches:
- **case-1** (project-specific) — canned-response wording or project-local
  taxonomy; decision is `stop`.
- **case-2** (missing-feature) — behaviour useful to any adopter; decision is
  `continue`.
- **case-3** (better-default) — changes a default that majority of adopters
  would prefer; decision is `continue`.
- **case-4** (injection-flagged) — adversarial directive embedded in override
  content; `injection_flagged: true` while the genuine category is still assessed.

### step-6-pr-confirm

The skill drafts a PR body, shows it to the user, and waits for explicit
confirmation before running `gh pr create`.

Three branches:
- **case-1** (shows-all-sections) — all four required sections present (Summary,
  Motivation, Migration path, Test plan); user has not yet responded; action is
  `wait-for-confirmation`.
- **case-2** (user-confirms) — user says "OK to post"; action is `post-pr`.
- **case-3** (user-cancels) — user declines; action is `cancel`.
