{
  "batch": "deep-22",
  "setup": "Installed Hermes source once with uv sync --locked --python 3.14; Python 3.14.10, Hermes git fdde27e7dae174cb4ba842589e92508852b7dd9f. Used the supplied tryit.py and Nous FakeLLMServer in isolated scratch profiles, empty plan first and then scripted AIAgent conversations. Command/hook plugins additionally exercised through real Hermes CLI or registered command handlers. Added pytest and PyYAML only to the scratch environment; tests ran without Hermes on PYTHONPATH. Runtime findings and test counts are my local observations; linked pinned source supports code claims. Desktop GUI was not launched and paid helper calls were not attempted. Scores and ranks are this seat only. The tied 28-point entries are ordered by expected daily retention: persistent personal apps first, proven learned deterministic routines second, and context/cost controls with disabled automatic replay third.",
  "plugins": [
    {
      "pr": 153809,
      "name": "living-spaces",
      "validate": "Passed: catalog validate --install-deps --json, enable and doctor --ci all exited 0. Admission warnings: private hermes_cli.plugin_session.session_key, cron.jobs and cron.executions imports (rule 9), plus raw window.hermesDesktop bridge (rule 8). A passing gate is not maintainer admission.",
      "inside": "Loaded 15 living_spaces tools, build-space skill and Desktop route/directive code. Five scripted AIAgent calls: tool_search discovered the tools; spaces_create built Reading Desk with a real HTML file and revision 1; spaces_list and spaces_get retrieved it; spaces_write_files rejected ../escape.txt with a structured invalid-path error. Creation explicitly warned that native maintaining-session context was absent, so canvas bot actions were unavailable in this CLI test. No GUI rendering or scheduled agent work claimed. Pinned source: https://github.com/akinduroifedayo/living-spaces/tree/e6145ccbd028edf55417822020426f4830343eb6/src/living-spaces",
      "tests": "Passed: 395 Python tests plus 1,017 subtests; 261 JavaScript tests (sandbox, reference apps, UI/session, permissions and recovery). Added PyYAML and JS test dependencies in scratch; supplied a test-only dependency resolver for hardcoded developer package locations and built the renderer metafile required by tests. No plugin production source was changed. Full Desktop application and real scheduled agent work were not run.",
      "useful": 5,
      "reliable": 3,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "total": 28,
      "rank": 1,
      "best": "Persistent agent-built apps with shared records, revision recovery, sandboxed Desktop canvases and native upkeep turn Hermes into a personal application workspace.",
      "worst": "Catalog validation warns about private cron/session imports and a raw Desktop bridge; Desktop-only bot context and the live GUI/upkeep path remain unverified here.",
      "blocked": null
    },
    {
      "pr": 152544,
      "name": "crystallize",
      "validate": "Passed: supplied tryit.py catalog validate --install-deps --json exited 0; enable and doctor --ci also exited 0.",
      "inside": "Loaded four crystal tools, eight hooks, two middleware registrations, commands, auxiliary slot, skill and Desktop code; validation passed with no catalog warnings (runtime separately warned about the provides_middleware manifest field). Five scripted AIAgent calls: tool discovery and routine listing worked; absent routine and invalid result IDs failed cleanly; invalid boolean was refused by Hermes schema validation before invocation. Supplementary real-Hermes chat sent an initial prompt, opted in through /crystallize chat on, and ran terminal cat twice on a pretty-printed 50-record JSON table: the model received compact exact JSON with a full_result_id, then a duplicate-result note, with zero helper calls in recorded events. Original text was retained under chat-owned storage. No measured real-provider monetary saving claimed. Pinned source: https://github.com/itsflownium/hermes-crystallize/tree/fc34138782796e94bca9131dd12c71f2dd843031/ Exact full-result recovery through its real Hermes-registered handler matched the stored original byte-for-byte; another chat and a traversal-shaped ID were refused cleanly.",
      "tests": "Passed: 1,455 pytest tests, 2 skipped; covers SDK boundaries, unsafe replay refusal, cost metadata, original recovery, cron drift, model routing, dashboard and lifecycle controls. Supplemental context-cleaning run passed. Recovery ownership and byte equality passed locally.",
      "useful": 5,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 3,
      "best": "Working context cleaning, duplicate-result suppression and chat-owned exact recovery improve Hermes itself, with unusually careful cost accounting and explicit unsupported-operation fallbacks.",
      "worst": "Automatic nested routine replay is deliberately disabled at this pin because guarded dispatch is unverified, and paid helper quality, billing savings and the Desktop GUI were not independently established.",
      "blocked": null
    },
    {
      "pr": 151804,
      "name": "dress-rehearsal",
      "validate": "Passed: supplied tryit.py catalog validate --install-deps --json exited 0; enable and doctor --ci also exited 0.",
      "inside": "Loaded two hooks and /rehearse; no new model tools. Supplied runner with native command calls around AIAgent: armed rehearsal, read fixture, intercepted valid and missing-path writes, rendered diff, rejected malformed drop syntax, dropped the bad step, and replayed the kept write successfully. Added adversarial three-call Hermes run: terminal executed a harmless scratch program named cat that wrote a sentinel while armed; read_file confirmed the write, and a missing-path write was still recorded. This is a reproduced safety defect, not a model refusal or missing dependency. See rehearsal.py is_read_only_command/should_record (basename allowlist). Pinned source: https://github.com/kawsthingz/hermes-dress-rehearsal/tree/392fc16e7b7057e71f8dfd42b90eefc9cb1b3b28/",
      "tests": "Passed: 84 pytest cases using the documented tests-directory root. An initial root-level invocation hit package-relative-import collection errors; corrected invocation passed. Supplementary real-Hermes basename test exposed an uncovered safety failure.",
      "useful": 5,
      "reliable": 2,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 25,
      "rank": 5,
      "best": "Exact recorded-call replay, editable steps, diffs and file-drift checks make a broadly useful approval workflow that a planning prompt cannot enforce.",
      "worst": "While armed it trusts command basenames: a scratch executable named cat performed a write before approval, yet the rehearsal report still said nothing ran.",
      "blocked": null
    },
    {
      "pr": 150840,
      "name": "kata",
      "validate": "Passed: supplied tryit.py catalog validate --install-deps --json exited 0; enable and doctor --ci also exited 0.",
      "inside": "Loaded kata_list, six hooks, /kata, CLI, prompt section and dashboard backend. Four scripted AIAgent calls: tool_search discovered kata_list; list returned empty history; review and approve of nonexistent candidates returned clean errors. Supplementary real-Hermes integration ran read_file then terminal wc in three distinct conversations; mining proposed kata_read_terminal with support 3 and confidence 0.78; review showed all 12 checks, explicit approval registered the tool, and its real registry handler completed both underlying Hermes tools. Unexpected arguments returned ok:false with both steps skipped. The compiled execution was invoked through the registered handler, not a fourth model conversation. Desktop inspected, not launched. Pinned source: https://github.com/opeblow/kata/tree/cc59b08f1bbeeb02524c060faac04edb34de0564/kata",
      "tests": "Passed: 61 pytest tests (units, admission, mining, execution, schema/workflow drift, redaction, commands and dashboard); added local real-Hermes learning/approval/execution probe also passed.",
      "useful": 5,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 2,
      "best": "It actually turns repeated real Hermes calls into a reviewed, registered deterministic tool, with useful evidence, traceability and drift checks.",
      "worst": "Its shell effect classifier has an incomplete dangerous-flag allowlist, and timed-out worker threads cannot be cancelled, limiting the safety and resource guarantees of learned routines.",
      "blocked": null
    },
    {
      "pr": 149968,
      "name": "lahja",
      "validate": "Passed: supplied tryit.py catalog validate --install-deps --json exited 0; enable and doctor --ci also exited 0.",
      "inside": "Loaded: registered Lahja transcription provider, /stt-fix and native lahja CLI; no model tools. Four scripted terminal calls inside AIAgent: models listed pinned downloadable models; status precisely reported missing whisper-cli, ffmpeg, VAD and dialect weights; test missing.wav returned a controlled missing-whisper-cli error; setup --model invalid-model returned an explicit unknown-model error/exit 2. Missing dependencies are expected setup failures, not crashes. No real audio decoded. Pinned source: https://github.com/Tarekkharsa/hermes-lahja/tree/a7a43b1389a1ae169ff5c7401e9ec2ec9a53b51d/lahja",
      "tests": "Passed: 99 tests; 6 real-engine cases skipped because whisper.cpp/model fixtures were unavailable. Tests cover provider envelopes, routing, corrections, download checksums/limits and secret-free subprocesses.",
      "useful": 3,
      "reliable": 4,
      "polish": 4,
      "creative": 3,
      "beyond": 5,
      "total": 24,
      "rank": 6,
      "best": "A native, local Arabic-dialect transcription provider improves Hermes voice notes with language routing, VAD, corrections and scrubbed subprocess environments.",
      "worst": "It serves a language-specific audience and requires external whisper.cpp, ffmpeg and large model downloads; actual transcription and the advertised accuracy gains were not independently reproduced here.",
      "blocked": null
    },
    {
      "pr": 149414,
      "name": "xray",
      "validate": "Passed: supplied tryit.py catalog validate --install-deps --json exited 0; enable and doctor --ci also exited 0.",
      "inside": "Loaded: no model tools, two hooks, /xray, native xray CLI and Desktop/backend code. Four scripted terminal calls inside AIAgent: detonate clean-calc returned GREEN; detonate sunny-weather returned RED with bait reads and exfiltration destination appearing at +30/+90 days; show sunny-weather recovered the report; show nonexistent name returned a clean error/exit 1. No bait was planted in a real home. Desktop and live enforcement were inspected, not exercised. Pinned source: https://github.com/cipoklean/hermes-xray/tree/21ca5f6fb5e46fb6d1a0ec2e179d055b96c69f52/xray",
      "tests": "Passed: 13/13 unittest tests, including clock-warp, egress/spawn blocking, secret-read tripwire and bait safety.",
      "useful": 4,
      "reliable": 4,
      "polish": 4,
      "creative": 5,
      "beyond": 5,
      "total": 27,
      "rank": 4,
      "best": "Time-warped honeytoken detonation and attributed runtime traps give Hermes a distinctive, working plugin supply-chain diagnostic.",
      "worst": "Its Python audit hooks are bypassable by native code, and opt-in bait occupies real credential paths, so GREEN is diagnostic evidence rather than a trust guarantee.",
      "blocked": null
    }
  ]
}