{
  "batch": "deep-56",
  "setup": "Installed Hermes from source with uv sync --locked --python 3.14, checkout fdde27e7dae174cb4ba842589e92508852b7dd9f, Python 3.14.10; plugins --help confirmed install/validate/doctor. Fetched each field.json repository at its full pinned SHA and subdirectory. Ran the supplied RUN.md tryit.py unchanged with empty and 3-5-call plans, using real AIAgent and Nous tests/fakes/fake_llm_provider.py, fresh scratch Hermes homes, no live provider accounts. Added pytest, pytest-asyncio, PyYAML, JSON Schema, plugin vision dependencies and Playwright/Chromium for tests. Later isolated test runs use scratch TMPDIR, GIT_CEILING_DIRECTORIES to prevent enclosing-workspace discovery, and multiprocessing fork because forkserver Unix sockets are prohibited here. Runtime/test outcomes below are direct local observations; linked pinned sources in sources.json support the code and documentation assessments. Scripted-model success does not establish real-model reasoning, live camera accuracy, or delivery in actual Desktop/messaging apps. No other seat scores used; ranks compare only these six plugins.",
  "plugins": [
    {
      "pr": 152159,
      "name": "genui",
      "validate": "Passed catalog admission with dependency installation, enable and doctor: 3 tools and 1 hook; manifest/declarations, security scan, no-core-override and Desktop surface all passed, no warnings.",
      "inside": "Loaded plugin/ at pinned 9465bec650ec4ad78e517575822858e5c133033b. Adds show_component, update_component, ask_user and a Desktop-only pre_llm_call hint; Desktop directive renderer, backend route and four schema-defined card types are present in code. Empty-plan chat completed; tools deferred behind tool search. Four planned calls inside Hermes: showed a task card, updated its steps/status successfully, returned an actionable schema error for status=bogus, and ask_user returned a nonblocking next-turn instruction. All responses recorded; conversation completed. Actual Hermes Desktop was not launched; browser tests used SDK/React stand-ins.",
      "tests": "Passed: 53 tests, including Playwright Chromium rendering, live updates, click-to-user-message behavior, schemas/store, plugin wiring and benchmark tests. Initial collection needed the development-only Playwright dependency; after installation the complete suite passed. Read agent, storage, schemas, Desktop channel, runtime/themes and backend code.",
      "useful": 4,
      "reliable": 5,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 2,
      "best": "A well-tested live-card state and interaction layer makes long Hermes Desktop tasks and approval choices substantially easier to follow.",
      "worst": "Its benefit is concentrated in Desktop and four predefined card types, and the model must still emit the directive and accurately update the card after each result.",
      "blocked": null
    },
    {
      "pr": 153596,
      "name": "sentinel-vision",
      "validate": "Passed catalog admission with dependency installation, enable and doctor: 9 tools, 0 hooks; capability declarations, dependency check, security scan, no-core-override and Desktop surface passed, no admission warnings. Isolation audit explicitly requires in-process operation because its post_message handle belongs to the gateway event loop.",
      "inside": "Loaded sentinel-vision/ at pinned fb48bf5c835fa2e8acb74dfc05c52458b3c1194d. Adds nine source/watch/rule/look/event/feedback/teaching/setup tools, /sentinel, auxiliary vision/rule tasks, watcher threads and a Desktop pane. Empty chat completed. Five planned calls: listed empty rules/budget, queried today's empty event history, rejected an unsupported source, reported a missing video without crashing, and returned setup diagnostics for absent camera/display and unverified model/delivery configuration. Supplemental real-Hermes calls created and listed a package-at-door recipe on a synthetic local image folder, queried events, and rejected a bad source; vision_calls_per_hour=0 deliberately paused judging. Real visual detection and outbound photo delivery were not demonstrated.",
      "tests": "Full suite after installing development dependencies: 1678 passed, 1 failed, 2 skipped; failure was test_delete_keeps_events, whose snapshot temporary file disappeared before os.replace. That test passed alone, but a separate deterministic scratch probe confirmed the underlying race: invoking normal prune between flush and replace deleted the active .tmp file and save_snapshot raised StoreError. Two optional Hermes-checkout comparison tests skipped. Read manifest, README, registration, tools, recipes, source/secret handling, storage, retention, engine and tests; synthetic change-gate scene checks passed, not a claim of real-camera accuracy.",
      "useful": 4,
      "reliable": 3,
      "polish": 4,
      "creative": 5,
      "beyond": 5,
      "total": 26,
      "rank": 4,
      "best": "A local change gate, bounded vision judging, persistent event memory and feedback turn Hermes into an ongoing observer of the user's own cameras and screen.",
      "worst": "Snapshot retention can delete an in-progress image write, and actual vision accuracy and visible alert delivery still depend on unverified device, model and gateway setup.",
      "blocked": null
    },
    {
      "pr": 152596,
      "name": "coach",
      "validate": "Passed catalog check with dependencies, enable and doctor: 2 tools and 7 hooks; security scan safe, declarations and Desktop surface passed. One non-failing catalog warning: private agent.usage_pricing.estimate_usage_cost import in coach/advisor.py:270.",
      "inside": "Loaded pinned fb3e1d931dc87e441ad425d2830ea783e901bc85. Adds coach_ask/coach_decide, /coach, per-turn evidence, completion-review and subagent/session hooks plus a Desktop settings page. Empty-plan chat completed; tools deferred behind tool search. Three planned calls returned cleanly: a rollout-plan consultation said no coach model configured, an empty question returned question_required, and a two-option decision returned decide_disabled. No external provider was configured, so actual second-model advice and paid Jev decisions were not demonstrated. A supplementary configured local fixture returned scripted advice through ctx.llm; this required the documented skip_same_model=false setting because the fake server identifies every response as fake-model. With the default guard, the fallback to that primary model was correctly rejected as same_model_fallback. This is routing/error-path evidence, not evidence of a real independent opinion.",
      "tests": "Passed final isolated full suite: 448 passed, 3 skipped; Node Desktop check passed for 6 views and 40 settings. After adding missing PyYAML, the first complete run had 446 passed/2 failed/3 skipped: one failure was forbidden forkserver Unix sockets, the other enclosing-workspace Git discovery; both passed with fork and a Git ceiling, and the complete suite then passed. Read hooks, advisor, evidence, settings, secrets and review-gate tests.",
      "useful": 4,
      "reliable": 4,
      "polish": 4,
      "creative": 3,
      "beyond": 4,
      "total": 23,
      "rank": 6,
      "best": "Bounded evidence-aware completion reviews and session budgets can give the agent a visible second opinion without handing the reviewer arbitrary tools.",
      "worst": "A useful second opinion still requires selecting and paying for another model, and its private-core pricing import adds compatibility risk; no real second-model quality was measured.",
      "blocked": null
    },
    {
      "pr": 152450,
      "name": "heats",
      "validate": "Passed catalog admission with dependency installation, enable and doctor: 3 tools and 3 hooks, tool-request middleware, security scan safe and no admission warnings. The chat loader emitted a non-fatal unknown provides_middleware manifest-field warning; registration still succeeded.",
      "inside": "Loaded pinned 2bcee10e9ca6489b7fd23bb0362a36eacdae269f. Adds heats_run/wait/apply, /heats and CLI commands, subagent worktree middleware, a pre_verify test referee and Desktop race/referee views. Empty chat completed. Four planned calls: an untrusted race was immediately blocked pending human approval, n=99 rejected by the schema, unknown wait/apply run IDs returned clear errors. Supplemental real-Hermes calls in a trusted scratch repo: missing pytest in the isolated runtime was reported clearly; selecting the built-in unittest runner then measured an already-green baseline and correctly refused to launch an unnecessary race. No live model-driven repair race was demonstrated; runner tests exercise races with fake launchers.",
      "tests": "Passed isolated full suite: 211 passed, covering snapshot/apply roundtrips, dirty-checkout preservation, worktree isolation, lap restarts, test protection, timeouts, approvals and referee behavior. Initial 206 passed/5 failed were enclosing-workspace Git discovery effects; all five disappeared after scratch TMPDIR and GIT_CEILING_DIRECTORIES isolation. Read manifest, wiring, runner, checks, git operations, referee, isolation and tests.",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 27,
      "rank": 3,
      "best": "The default completion referee directly improves the user's coding agent by turning real failing tests into another repair turn, with optional isolated attempts and protected-test judging.",
      "worst": "The referee misses shell-made edits, race dependencies are shared with the real checkout, and the available runtime demonstration stopped at approval or baseline checks rather than a successful model-driven repair.",
      "blocked": null
    },
    {
      "pr": 151376,
      "name": "joust",
      "validate": "Passed catalog admission with dependencies, enable and doctor: 8 tools and 4 hooks; declarations, security scan, no-core-override and Desktop surface passed without admission warnings.",
      "inside": "Loaded pinned b88242eb72f50c6dfa284d1f4335e8b59b30afbf. Adds launch/evaluate/diff/steer/merge/status/crossfire/report tools, lifecycle hooks, worktree middleware, /joust, CLI and Desktop arena. Four planned calls in Hermes: the built-in three-contender demo completed with hash-bucket-index winning; status listed its persisted bout; an empty title/contender launch and missing-bout diff returned clean errors. The demo writes predetermined contender implementations, so it verifies real Git/testing/ranking, not model repair ability. A separate scratch-only symlink probe confirmed the middleware returns an escaping symlink path but pre_tool_call blocks that call before execution.",
      "tests": "Full Python suite: 93 passed, 3 failed, 1 skipped, plus 3 passing subtests. Failures: sh printf does not interpret the test's hex escapes here; an alleged nonrepo inherited the enclosing Git repo and triggered recursive copy fallback; symlink path-clamp assertion failed. A Git-boundary wrapper made the nonrepo test pass; the symlink assertion still failed. Node Desktop smoke: 55 checks passed, 0 failed, 2 warnings (extra list fetch on selection and malformed directive stuck loading). Read engine, schemas, handlers, hooks, dashboard, Desktop and tests.",
      "useful": 4,
      "reliable": 3,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 25,
      "rank": 5,
      "best": "Crossfire tests rival patches against each other's new tests, adding a substantive correctness check to the parallel-worktree and approval-gated merge workflow.",
      "worst": "Symlink clamping fails its own assertion, and the unchecked directory-copy fallback recursed into its destination during the enclosing-repository test.",
      "blocked": null
    },
    {
      "pr": 149738,
      "name": "trust-ladder",
      "validate": "Passed: hermes plugins validate --install-deps --json exited 0; manifest, capability declarations, dependency, security and no-core-override checks passed, no warnings. Enable and doctor both passed: 2 tools and 5 hooks.",
      "inside": "Loaded pinned fae4e9df58767e5e3832c8f65936b97868bc98fc. Adds trust_ladder_learn/task, /teach, /trust, /verify and enforcement/reminder/completion hooks. Empty-plan chat completed; plugin tools are deferred behind Hermes tool search. Five planned calls: learned a command guard from bad/good examples; the actual terminal echo forbidden-marker call was blocked before execution; created a verifier task; its echo verified-marker verifier ran through Hermes terminal and passed; bogus task action was rejected by schema validation. All five responses were recorded and conversation completed without crashing.",
      "tests": "Passed final isolated full suite: 96 passed. First concurrent run was 95 passed/1 failed on a timing-based regex-window assertion; that test passed alone and the complete suite then passed with test isolation. Read compiler, runtime, storage, schemas and tests; formal-model claims were not independently model-checked.",
      "useful": 5,
      "reliable": 5,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 1,
      "best": "Persistent, example-verified correction guards demonstrably changed what Hermes was allowed to execute, with a useful catch ledger and objective completion verification.",
      "worst": "Pattern guards remain best-effort and fail open on hook errors, so indirect shell commands can escape a correction and model-added rules can over-block.",
      "blocked": null
    }
  ]
}