{
  "batch": "deep-33",
  "setup": "Installed uv and CPython 3.14.10 under test/scratch, cloned NousResearch/hermes-agent at fdde27e7dae174cb4ba842589e92508852b7dd9f, and ran uv sync --locked --python 3.14. Confirmed plugins help lists install, validate and doctor. Used the supplied RUN.md/tryit.py and Hermes's own FakeLLMServer with fresh isolated Hermes homes, first empty plans and then 3-4 planned tool calls per plugin, including bad inputs. All six pinned SHAs and folder mappings came from field.json. Installed plugin/test dependencies locally; did not supply any real model, credential or account. Catalog --install-deps was attempted by tryit; some concurrent dependency preparations retried/failed publication, so also ran standalone full JSON catalog validation after dependencies were present: all passed. Latest Hermes uses lazy tool advertisement: empty added_by_plugin arrays did not mean absent registrations, and the named tools dispatched successfully. Hardware-dependent voice, camera and real LoRA training/model quality were not established by scripted-model runs. Public source URLs are in sources.json; test counts and call outcomes below are direct local observations, not entrant benchmark claims.",
  "plugins": [
    {
      "pr": 152458,
      "name": "housemate",
      "validate": "Pinned https://github.com/RaYYeR220/housemate/tree/09c1a1d0a88bfa8762cee6b3ccf94271b0f172c5. Passed: catalog validation exit 0, no warnings; enable and runtime doctor exit 0; doctor registered 7 tools and 1 hook.",
      "inside": "Loaded inside real Hermes; seven room/announcement/timer/alarm/screen tools, Housemate platform, pre_llm_call spoken-answer hook, commands, CLI, blueprints and SDK Desktop Rooms page. Empty-plan chat completed. Four-call chat completed: housemate_rooms, housemate_timer_set(seconds=120,label=tea,room=Kitchen), and housemate_timer_list all returned the actionable hub-not-running error; negative seconds=-3 was rejected by Hermes schema before invocation. Thus graceful missing-gateway behavior was verified, not actual room speech. Hub/control/WebSocket/timer/approval behavior was additionally exercised by its tests.",
      "tests": "Passed: 569 passed, 1 skipped. Ran with Hermes installed, plugin tests on PYTHONPATH and importlib collection from outside plugin root; initial root-directory run had 23 failures caused by tools.py shadowing Hermes tools, resolved by runner import-path correction without changing plugin code.",
      "useful": 4,
      "reliable": 5,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "rank": 2,
      "best": "A substantial new Hermes voice platform reuses ordinary browser devices, preserves room context and approval handling, and ships a finished Rooms interface with extensive passing tests.",
      "worst": "It needs an always-running gateway plus STT/TTS and continuous microphone streaming, and real-device speech latency and approvals were not reproduced here.",
      "blocked": null,
      "total": 28
    },
    {
      "pr": 153113,
      "name": "standing-orders",
      "validate": "Pinned https://github.com/jaxsonsprinkles/standing-orders/tree/1959097a915e53624262c235e9eb102babece01f. Passed: catalog validation exit 0, no warnings; enable and doctor exit 0; doctor registered 8 tools and the standing_orders platform.",
      "inside": "Loaded inside real Hermes; eight order-management/backtest/correction tools, camera gateway platform, CLI and Desktop pane. Empty-plan chat completed. Four-call chat completed: so_status reported camera engine not started and zero counters; so_list_orders returned []; so_create_order for a ten-second mug alert returned a clear missing-reference-frame error; so_arm(order_id=does-not-exist) returned no-such-order with known=[]. Read the local frame-difference/CLIP/vision cascade, time-rule engine, budget handling and correction replay; live capture/firing was not reproduced on this host.",
      "tests": "Passed: 170 passed, 2 skipped, after installing its declared numpy/OpenCV/ONNX/tokenizer/PyYAML dependencies. The passing suite exercises compiler contracts, budgets, correction replay, backtests, gates, camera sources and tools; it is not a fresh live-camera benchmark.",
      "useful": 3,
      "reliable": 4,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "rank": 6,
      "best": "Plain-language camera watches become budgeted, backtestable event programs with evidence clips and correction replay that can wake Hermes through a real platform adapter.",
      "worst": "The pinned implementation explicitly lacks Linux and Windows camera support, and neither a live watch nor the vision-model path was demonstrated here.",
      "blocked": null,
      "total": 27
    },
    {
      "pr": 150800,
      "name": "hypnos",
      "validate": "Pinned https://github.com/0xNoramiya/hypnos/tree/0a809b9085fdeaed63fd909b927afcc41de512de. Passed: catalog validation exit 0, no warnings; enable and doctor exit 0; runtime registered 9 hooks and llm_execution middleware, with no model-facing plugin tools.",
      "inside": "Loaded inside real Hermes; capture/approval/outcome/finalization hooks, model-routing middleware, five slash commands, hypnos CLI and Desktop sleep/brain UI. Empty-plan conversation passed through normally. Four planned Hermes terminal calls exercised its actual CLI: hypnos status showed capture on, one captured turn and no brain; hypnos habits returned no habits yet; malformed brain diff 1 returned usage and exit 2 without traceback; hypnos doctor identified the missing trainer interpreter and Docker with installation/fallback guidance. The real LoRA SFT/DPO backend, exam/promotion, shadow routing and unlearning code were read; real weight training was not run.",
      "tests": "Passed: 229 passed, 2 skipped. Its fixture trainer initially exited 127 because python3 was absent from PATH; rerun with a scratch-local alias to the installed interpreter passed. Night/promotion/unlearning tests use the committed fake trainer; optional real trainer/torch tests were skipped.",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "rank": 3,
      "best": "Hypnos changes Hermes itself through personal LoRA training, held-out promotion checks, shadow-to-live routing, rollback and lineage-aware forgetting rather than merely rewriting a memory prompt.",
      "worst": "Its strongest benefit remains unverified on real weights here, while supported habit extraction and the separate trainer/model setup narrow the audience for daily use.",
      "blocked": null,
      "total": 28
    },
    {
      "pr": 149159,
      "name": "understudy",
      "validate": "Pinned https://github.com/stffinfcti/understudy/tree/e0920f9e3641f0e24afe6ee01b21322c070920fd. Passed: catalog validation exit 0, no warnings; enable and doctor exit 0; 2 tools registered; noninteractive enabling correctly withheld model/provider-override capability grants.",
      "inside": "Loaded inside real Hermes; understudy_audition and understudy_report, commands/CLI, report artifacts and Desktop run browser. Empty-plan chat completed. Three-call chat completed: dry-run audition(sample=4) on a fresh profile reported zero eligible decisions and no calls/spend; report returned no-audition-found; candidates=[] was rejected by Hermes minItems validation before invocation. Read the read-only history corpus, redaction, stratified replay, blinded judge, baseline-noise comparison, pricing and retry/budget code. No live candidate quality or paid-provider audition was measured.",
      "tests": "Passed: 94 passed, including audition, corpus/history, sampling, judgment/metrics, retries, budget/trust handling, reports and dashboard routes.",
      "useful": 5,
      "reliable": 5,
      "polish": 5,
      "creative": 4,
      "beyond": 4,
      "rank": 4,
      "best": "A polished, consent-aware replay harness gives Hermes users evidence from their own history before changing models, with baseline noise, uncertainty and cost reporting.",
      "worst": "A one-step replay cannot prove a cheaper model will finish an entire task, and useful conclusions require sufficient history plus additional model calls.",
      "blocked": null,
      "total": 27
    },
    {
      "pr": 151704,
      "name": "guild",
      "validate": "Pinned https://github.com/Nnadijoshuac/hermes-guild/tree/bf5d32c7b02b1cb3b4a7fe6fdd029f09c3a956d6. Passed: catalog validation exit 0, no warnings; enable and doctor exit 0; 4 tools and 4 hooks registered.",
      "inside": "Loaded inside real Hermes; status/sync/propose/review tools, lifecycle and write-guard hooks, human review commands, CLI and SDK Desktop page. Empty-plan chat completed. Four-call chat completed: guild_status gave init/join guidance, guild_sync and guild_review cleanly reported unconfigured membership, and guild_propose(skill=../bad,note=invalid name) also stopped at the membership guard. Thus the bad name did not reach name validation in that fresh profile. The plugin test suite additionally ran two local members against a local bare git repository through propose/review/approve/sync/reject/revert; no external team service was used.",
      "tests": "Passed: 10 passed when run from tests/ as its README specifies, including the two-member git workflow, redaction, self-approval refusal and write guards. Initial execution from repository root had 8 failures because its tools.py shadowed Hermes tools; the documented working directory resolved them without code changes.",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "rank": 5,
      "best": "Guild turns independently learned skills into a reviewed team library with secret scrubbing, version history and hooks that redirect normal edits into proposals.",
      "worst": "Its read-only protection inspects ordinary tool arguments rather than arbitrary executed code, and reviewer policy still needs remote branch protection for server-side enforcement.",
      "blocked": null,
      "total": 27
    },
    {
      "pr": 153851,
      "name": "foreman",
      "validate": "Pinned https://github.com/PedValme/foreman/tree/c15453cb2fc07fa674d6f7b548a553b3ce9e6f5d/plugin. Passed: catalog validation exit 0; warning for private import tools.terminal_tool.get_session_cwd in foreman/gate.py:661 (catalog rule 9 compatibility concern), not a failing check; enable and doctor exit 0; 2 tools and 2 hooks registered.",
      "inside": "Loaded inside real Hermes; foreman_check/foreman_review, pre_tool_call and pre_verify guards, auxiliary review model registration, commands and SDK Desktop panel. Empty-plan chat completed. Four-call chat completed: foreman_check on a clean local fixture returned CLEAN; on a fixture adding @pytest.mark.skip it returned BLOCKED/test-skipped with file, line and fix; an absolute nonexistent child path unexpectedly resolved to its enclosing repository and returned CLEAN rather than rejecting the path; foreman_review on the clean fixture returned NOTHING TO REVIEW without a model call. All repositories and changes were disposable local fixtures; no PR or merge was sent.",
      "tests": "Final rerun: 384 tests run, 381 passed, 2 failed, 1 skipped. Set GIT_CEILING_DIRECTORIES at scratch to remove three enclosing-repository artifacts from the first run (378 passed, 4 failed, 1 error, 1 skipped). The two remaining failures concern long SARIF path diagnostics truncating away the filename; verdicts still correctly remain INCOMPLETE. No plugin code was changed.",
      "useful": 5,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "rank": 1,
      "best": "Broadly useful deterministic hooks stop visible test sabotage and publishing shortcuts while retaining human-owned exceptions and merges, backed by extensive detector and workflow coverage.",
      "worst": "A nonexistent supplied path can silently check the enclosing repository, long SARIF error paths lose the filename, and one private Hermes import weakens compatibility.",
      "blocked": null,
      "total": 28
    }
  ]
}