{
  "batch": "trial-1",
  "setup": "Hermes cloned from https://github.com/NousResearch/hermes-agent at ddc5d27eb38ec60f098b2168e48e7356cb4735a0 into test/scratch/hermes; uv sync --locked --python 3.14 succeeded (Python 3.14.10). CLI help confirmed install, validate and doctor. Used the supplied RUN.md and unmodified tryit.py with real AIAgent and Nous tests/fakes/fake_llm_provider.py, isolated scratch HERMES_HOME directories, empty plans followed by 2–5 scripted calls per entry. Installed pytest, pytest-asyncio, python-telegram-bot and aiohttp in that scratch environment for tests. All run results below are local observations, not independently published evidence; source URLs identify the pinned material reviewed. No real Home Assistant, Telegram account or Hermes Desktop was connected. The initial offered-tool diff is empty even for plugins whose registered tools execute successfully (Hermes defers tool discovery); doctor and actual calls establish registration. No other seats or scores consulted. Sources: https://gist.githubusercontent.com/gtmcknight/594c47336546c8763e641b46cf9af4ec/raw/field.json and https://gist.githubusercontent.com/gtmcknight/594c47336546c8763e641b46cf9af4ec/raw/RUN.md; catalog policy at https://raw.githubusercontent.com/NousResearch/hermes-agent/ddc5d27eb38ec60f098b2168e48e7356cb4735a0/plugin-catalog/README.md.",
  "plugins": [
    {
      "pr": 0,
      "name": "done-bell",
      "validate": "Passed: hermes plugins validate --install-deps --json returned exit 0 in both runs; security scan safe, declared capabilities match, desktop surface and no-core-override checks passed; no warnings.",
      "inside": "Loaded: enable and doctor exit 0; doctor recorded 1 tool and 4 hooks. Empty-plan conversation completed. Four planned done_bell_status calls returned settings, muted successfully, accepted/coerced the deliberately wrong-type mute='false' to false, and returned unmuted JSON without Markdown. Conversation completed without a crash. A completed-turn event was checked separately in persisted bell state. OS notification delivery and the Desktop pane were not exercised; DONE_BELL_NO_OS=1 suppressed OS effects. Pinned source: https://github.com/aydnOktay/hermes-done-bell-/tree/3fa519a291d91b0e6c44bd42762190363ce80c39 ; reviewed README.md, plugin.yaml, __init__.py, bell_tools.py, bell_store.py, bell_hooks.py, schemas/context, dashboard backend, Desktop code and tests.",
      "tests": "Passed: 9 pytest tests; cover defaults, events, approval cycle, mute, hook kwargs and notification platform gate. No OS or Desktop end-to-end test.",
      "useful": 4,
      "reliable": 4,
      "polish": 3,
      "creative": 2,
      "beyond": 4,
      "total": 21,
      "best": "A small, credential-free extension that makes turn completion and approval waits visible through real lifecycle hooks, notifications and a Desktop status chip.",
      "worst": "The README's installation path is personal and Windows-specific, with a manual Desktop copy step and no demonstrated cross-platform notification delivery.",
      "blocked": null
    },
    {
      "pr": 0,
      "name": "homeassistant",
      "validate": "Passed: validation exit 0, doctor exit 0, registration matches 4 tools and 0 hooks, security scan safe and no core overrides. Admission warns that adapter.py imports the private gateway.restart.is_supervised_gateway_launch helper; trusted_inbound is disclosed.",
      "inside": "Loaded and enabled; doctor confirms four tools plus code registers a homeassistant gateway platform, inbound state-change handling and standalone cron delivery. Empty-plan chat completed. Without HASS_TOKEN the availability check intentionally hides ha_* tools: the planned entity list and light.desk read each returned tool unavailable, and the third, malformed service-domain request was not dispatched; Hermes ended with invalid tool call, not a plugin exception. This does not demonstrate device control. Supplemental direct handler probes rejected a traversal entity ID and blocked shell_command before networking; valid list/get returned structured connection errors against the unavailable default HA host. Pinned source: https://github.com/NousResearch/hermes-official-plugins/tree/e7a218f26db88426ab3e7376f0e000b6e356f5e7/homeassistant ; reviewed README, manifest, pyproject, registration, tools.py, adapter.py and all test modules.",
      "tests": "Passed: 212 pytest tests, 4 skipped (platform/compatibility/optional-dependency conditions). Mocks exercise REST, entity filters, unsafe service refusal, adapter events/reconnect/heartbeat, registration and standalone delivery; no live Home Assistant instance was tested.",
      "useful": 4,
      "reliable": 4,
      "polish": 4,
      "creative": 2,
      "beyond": 5,
      "total": 24,
      "best": "A full gateway platform turns the user's own home events into agent input and supports constrained device tools and cron notifications through Hermes secret handling.",
      "worst": "Real device and reconnect behavior remains unverified here, and the catalog check flags reliance on a private gateway helper despite otherwise extensive mocked coverage.",
      "blocked": null
    },
    {
      "pr": 152819,
      "name": "idm-radio",
      "validate": "Passed: both catalog validations exit 0, security scan safe, desktop SDK surface and no-core-override checks pass, declarations match registration; no warnings.",
      "inside": "Loaded and enabled; doctor exit 0 reports 1 tool and 2 hooks, plus /idm in code. Empty-plan chat completed. Four planned idm_radio requests exercised status, play with night/60, invalid action dance, and stop. Status worked; play/stop explicitly reported desktop_received=false and that nothing changed without Desktop; Hermes rejected dance by enum validation before invoking the handler, then continued to completion. Code reports tool outcomes without arguments/results and implements event-driven local Web Audio synthesis, visuals and track WAV export. Audio/visual behavior in real Hermes Desktop was not tested. Pinned source: https://github.com/discodirector/idm-session-radio/tree/3c3e6dfc71f61ca9533353486f0cc858e0279c19 ; reviewed README, manifest, Python registration/hooks, package/build files, Desktop TypeScript and shipped bundle, receiver bookkeeping, audio/visual engine and render scripts.",
      "tests": "No automated assertion suite supplied; test/ contains media render scripts. npm run typecheck failed with 2 TS2345 errors in desktop-src/promo.ts (panel-sky and bg-tunnel are not valid visual modes). Building the supplied demo bundle with build/build.mjs --test-only passed. The supplied 170-second scripted demo rendered successfully in headless Chromium using its already available executable (the package-default browser revision was absent): a valid non-silent stereo 44.1 kHz 16-bit WAV, 2 track summaries with 10/9 tools and 2/1 failures. A shortened 4-second render also completed. Browser warnings reported clamped compressor release and negative filter-frequency values. These render checks do not establish real Desktop playback/export.",
      "useful": 3,
      "reliable": 3,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 25,
      "best": "It makes the agent's actual work audible and visible through Desktop events, synthesis and saveable session tracks, with a complete visual identity and no remote audio service.",
      "worst": "The TypeScript check fails in promotional code and there is no assertion suite; successful CLI control responses cannot establish that Desktop playback and WAV export work.",
      "blocked": null
    },
    {
      "pr": 150003,
      "name": "pow-tracker",
      "validate": "Passed: both validation runs exit 0; enable and doctor exit 0, with 0 tools/hooks as declared, safe security scan and no core overrides. Runtime warns that the top-level dashboard manifest field is unknown and ignored; dashboard discovery instead uses dashboard/manifest.json.",
      "inside": "Loaded; the empty-plan chat and a two-call chat reading its README then a missing scratch file completed, with the missing-file error handled by Hermes. These built-in calls do not test billing, since register(ctx) intentionally adds no agent tools or hooks. Supplemental FastAPI TestClient calls exercised the pinned router with real Hermes configuration and scratch-only SQLite storage: client creation, task creation, one-hour time entry, invoice total 100 USD and CSV export returned 200; reversed timestamps returned 422. Listing clients after creation exposed active equal to the client's folder-path string, despite create_client returning active=0; code inserts literal 0 in folder's position, places the folder string in active and repairs only folder. The UI tests active by truthiness, so this is a real client-selection defect. No full web-dashboard UI, live auto-capture or PDF export was tested. Pinned source: https://github.com/BigBossRabbit/pow-tracker/tree/d57be8fcbed69f39bd735de41540ff04a4bbc30f ; reviewed README, manifest, pass-only registration, dashboard manifest, backend, frontend and both test modules.",
      "tests": "Passed: 30 unittest tests (settings, SQLite seeding/migration, opt-in capture defaults, source policy checks and WebSocket credential checks). Supplemental API workflow confirmed billing/CSV and bad-span rejection but uncovered the client active-field corruption; these are additional probes, not 30 billing tests.",
      "useful": 3,
      "reliable": 2,
      "polish": 3,
      "creative": 2,
      "beyond": 3,
      "total": 16,
      "best": "Local per-client SQLite records, opt-in file-activity capture and authenticated invoice exports make a useful dashboard for freelancers.",
      "worst": "Creating a client stores its folder path in active, making an inactive client truthy in the UI, and the passing tests miss this basic multi-client workflow bug.",
      "blocked": null
    },
    {
      "pr": 151395,
      "name": "ratcatcher",
      "validate": "Passed: both catalog validations exit 0; safe security scan, no core overrides and all tool/hook declarations match. Enable and doctor exit 0. Telegram handler registration correctly requires in-process operation rather than the isolated plugin host.",
      "inside": "Loaded; doctor reports 4 tools and 1 pre_tool_call hook; registration also adds a Telegram handler, slash command and skill. Empty-plan chat completed. Five planned calls: a wallet/seed-phrase scam scored 1.0 and would be removed; a benign 'Never share your seed phrase' warning scored 0.34 and would get a second look; a 24-hour report returned an empty log and connected_to_telegram=false; adding github.com was refused as too common; empty message text returned a clear error. Conversation completed. Code and mocked tests cover live handler interception, strikes, moderation/report buttons, release and approval gating; no actual Telegram deletion, ban, release or live LLM second look was attempted. Pinned source: https://github.com/Truthgeng/ratcatcher/tree/04b5a548bb38791d559a451a56dda723ea1b9f94 ; reviewed README, manifest, registration/approval hook, detector, moderator, store, tools, skill and tests/corpus.",
      "tests": "Passed: 226 pytest tests including scam/clean corpora, storage, Telegram handlers with mocked bot actions, admin clone handling, callbacks and mutation approval rules. Live moderation and model-backed borderline review remain unverified.",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 27,
      "best": "A real pre-agent Telegram bouncer combines deterministic scam rules, model-assisted review and human-approved administrative changes with reversible moderation and strong local tests.",
      "worst": "Default enforce mode guards every group and can ban on sight, while the benign warning probe still triggered second-look review, so mistakes and model cost need careful operator tuning.",
      "blocked": null
    },
    {
      "pr": 153194,
      "name": "stageglass",
      "validate": "Failed in both supplied-runner catalog checks: capability probe reported 'capability probe produced no result (exit 0)' even though its detail contains HERMES_VALIDATE_JSON with the 7 registered hooks and onair command. Security scan was safe, desktop surface and no-core-override checks passed. Enable and doctor nevertheless returned 0. Reliable is capped at 2 under the assignment's failed-catalog-check rule; no unsupported attribution of the gate failure to a specific source bug.",
      "inside": "Loaded; doctor confirms 0 tools and 7 hooks, plus /onair in registration. Empty plan completed. Planned read_file calls for its README and a missing scratch file both returned through real Hermes. The runner's chat.result is null because TRYIT completion text was on stderr; that text confirms completed=true and process exit 0. A supplemental copy of the supplied runner invoked the actual Hermes-registered onair handler: on yielded ON AIR; HTTP /snapshot contained the current ask and both tool steps; invalid banana returned usage; hold cleared ask, tool, args and steps in the next snapshot. This verifies the loopback feed through real agent hooks, not OBS integration. Pinned source: https://github.com/Benito3262/stageglass/tree/9d208c532e7a6262646608d73089f1b95a1700cd ; reviewed README, manifest, registration, feed, redactor/wordlist, HTTP/SSE server, static page, Desktop chip and tests.",
      "tests": "Passed after isolation: 22 unittest tests, covering redaction fixtures, safety holds, command parsing/registration, loopback restriction and actual HTML/snapshot/SSE secret exclusion. Initial run with Hermes checkout on PYTHONPATH had 10 passing tests and 2 test-module import errors because its tests package shadowed this plugin's namespace; the standalone Python 3.14 interpreter resolved this without changing plugin source. Supplemental real-Hermes on/hold/invalid-command and snapshot probes passed.",
      "useful": 4,
      "reliable": 2,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 24,
      "best": "An opt-in audience feed observes real agent and approval hooks while keeping raw session text out of the overlay through layered redaction and fail-closed holds.",
      "worst": "Catalog admission fails at the capability probe in this environment despite successful doctor and live overlay probes, and novel secret formats still depend on heuristic detection.",
      "blocked": null
    }
  ]
}