{"batch": "deep-50", "setup": "Installed Hermes from source (git clone --depth 1, uv sync --locked --python 3.14, Python kept inside test/scratch because the sandbox hid the default uv dir); tryit.py ran each plugin at its pinned commit with an empty plan and then 1 to 3 planned calls. pytest, pyyaml added to the venv for tests. The sandbox blocks some sockets and file ops, which hit local-hands tests and a few others.", "plugins": [{"pr": 154078, "name": "harness-story", "validate": "passed (hermes plugins validate exit 0, every check ok)", "inside": "Loaded (5 tools, 12 hooks; stderr warns unknown capability 'filesystem:write'). harness_story_list and harness_story_generate(use_llm=false) worked and built a story from the session; inspect of a missing story gave a clean error.", "tests": "passed: 5 passed, 1 skipped (pytest)", "useful": 3, "reliable": 3, "polish": 3, "creative": 3, "beyond": 3, "total": 18, "rank": 6, "best": "Turns a session's hook telemetry and state.db into an animated HTML story with real data, working offline.", "worst": "Default narrative endpoint is a hard-coded LAN IP (192.168.1.12), the manifest uses an unknown capability and an external runtime, and the tests are thin.", "blocked": null}, {"pr": 153443, "name": "local-hands", "validate": "passed (hermes plugins validate exit 0, every check ok)", "inside": "Loaded as a backend with 1 tool ('hands', 1 hook). Bad action was rejected by schema validation; status/use calls returned 'Operation not permitted' because this sandbox blocks the hub's socket, so the hub could not be exercised.", "tests": "partial: 59 of 78 tests ran under unittest, 38 errors all PermissionError from the sandbox, so not a clean pass; pytest collection also fails on the root __init__ unless run as unittest", "useful": 4, "reliable": 4, "polish": 5, "creative": 4, "beyond": 5, "total": 27, "rank": 2, "best": "Hermes stays on your server while its terminal and file tools run on your own computer over pinned TLS with per-action approval, which only a plugin backend can do.", "worst": "Could not run the hub here, and the 0.0.0.0 default bind and the larger attack surface need careful setup.", "blocked": null}, {"pr": 150497, "name": "thoth", "validate": "passed (hermes plugins validate exit 0, every check ok)", "inside": "Loaded: 19 tools, 3 hooks. thoth_status without a brain returned a clear 'run thoth_init' error; thoth_assign and thoth_plan with missing args were rejected without invoking.", "tests": "passed: 77 passed, 6 skipped (pytest)", "useful": 4, "reliable": 4, "polish": 5, "creative": 5, "beyond": 5, "total": 28, "rank": 1, "best": "Hermes plans on a shared board and hands tasks to the real Claude Code and Codex CLIs in git worktrees, with cross-review and merge, on the user's own logins.", "worst": "Needs Claude Code or Codex CLIs and git set up, so the full loop was not run here, and the 19-tool surface is large.", "blocked": null}, {"pr": 150251, "name": "gemini-live", "validate": "passed (hermes plugins validate exit 0, every check ok)", "inside": "Loaded (no Hermes tools, no hooks; works via dashboard and Desktop, voice tools live in the call engine). Nothing to call as a tool, so only load, enable and doctor were checked; reads the key through the secret scope, not os.environ.", "tests": "mostly passed: pytest 126 passed, 1 failed (test_shell cat /etc/os-release, likely sandbox read limits); node engine test 1 passed", "useful": 4, "reliable": 4, "polish": 5, "creative": 4, "beyond": 5, "total": 27, "rank": 3, "best": "Full-duplex Gemini voice calls inside Hermes Desktop that delegate to the real agent, steer threads and recall memory, with clear docs and screenshots.", "worst": "Needs a Gemini key and a microphone, so the live call could not be run, and it is an interface for one provider.", "blocked": null}, {"pr": 152596, "name": "coach", "validate": "passed (hermes plugins validate exit 0, every check ok)", "inside": "Loaded: 2 tools (coach_ask, coach_decide), 7 hooks. coach_ask with no coach model returned a clean 'no coach model configured'; empty question and a one-option decide gave clean errors.", "tests": "mostly passed: 450 passed, 1 failed (test_lockfile PermissionError from the sandbox)", "useful": 3, "reliable": 4, "polish": 4, "creative": 3, "beyond": 3, "total": 20, "rank": 5, "best": "Four clear modes (consult, active review, review-before-done, decide), secrets read through Hermes's store, and a large test suite.", "worst": "A second-opinion model is something a skill or sub-agent prompt could largely do, and it does nothing until a coach model is configured.", "blocked": null}, {"pr": 149272, "name": "no-cap", "validate": "passed (hermes plugins validate exit 0, every check ok)", "inside": "Loaded: 0 tools, 4 hooks (post_tool_call, transform_llm_output, pre_verify, pre_llm_call). A terminal call ran normally with the hooks on; receipts are stamped on the final reply, which the scripted model could not exercise.", "tests": "passed: 5378 passed, 1 skipped, 17 xfailed, 1 failed (test_doctor exit code, likely sandbox) in about 7 minutes", "useful": 4, "reliable": 4, "polish": 5, "creative": 4, "beyond": 4, "total": 25, "rank": 4, "best": "Checks claims such as 'tests pass' or 'pushed' in the agent's reply against that turn's actual tool calls with no extra model, a leaderboard and dashboard; a prompt cannot verify itself.", "worst": "Claim matching is heuristic so it can miss or misflag, and it adds no new capability, only an audit of replies.", "blocked": null}]}