{ "batch": "deep-51", "setup": "Installed real NousResearch/hermes-agent at b89e04eb7021a549db13d910b7a29483ead4f043 from source using uv sync --locked --python 3.14 (CPython 3.14.10); plugins CLI exposes install/validate/doctor. Fetched all six exact field.json SHAs. Used RUN.md tryit.py and Hermes tests/fakes/fake_llm_provider.py, empty plans first, then scripted real AIAgent conversations. All work and dependencies are disposable test/scratch fixtures; no real model/API account/device credentials used. Hermes defers plugin tools behind tool_search by default, so raw added_by_plugin lists were misleading (the initial bare-tools baseline was empty); registration/doctor and actual call results, not that list, establish plugin additions. Time Machine additionally used a scratch-only runner configuration selecting a separate protected workspace and disabling tool deferral. Scores and ranking are judgments; execution/test outcomes below are local observations, not upstream claims. Close-score ordering favors the correction guard over recovery tooling, general coding utility over the DAW niche, and personally owned hardware reach over paid telephony; these are retention judgments, not measured adoption.", "plugins": [ { "pr": 149738, "name": "trust-ladder", "validate": "Pinned https://github.com/merdoks/trust-ladder@fae4e9df58767e5e3832c8f65936b97868bc98fc repository root. Passed: all 15 catalog checks, no warnings; enabled and doctor passed (2 tools, 5 hooks).", "inside": "Loaded. Adds correction guards, post-write checks, standing-rule injection, human-review escalation, verified tasks, and /teach /trust /verify. Five-step real-Hermes plan: learned a guard for printf forbidden-demo; a subsequent terminal request was blocked before execution; created demo-check with verifier printf verified; verification returned exit 0 and passed=true; empty rule returned a clean error. Guard blocking and verifier dispatch both worked in the actual agent loop.", "tests": "Passed: 96 pytest tests (0 failures), including compiler, runtime, reliability and statechart replays.", "useful": 5, "reliable": 5, "polish": 4, "creative": 4, "beyond": 5, "total": 28, "rank": 1, "best": "Turns repeat user corrections into tested, persistent tool-boundary enforcement, with a real verifier rather than another promise in memory.", "worst": "Pattern guards are best effort and fail open on hook errors, so aliases and script indirection can bypass the reassuring \u201cimpossible\u201d wording.", "blocked": null }, { "pr": 149553, "name": "hermes-time-machine", "validate": "Pinned https://github.com/ddaikodaiko/hermes-time-machine@1ac9c78e539dd518d6c8d8ffd64395b72bab30d4 repository root. Passed: all 15 catalog checks, no warnings; enabled and doctor passed (5 tools, 5 hooks).", "inside": "Loaded. Adds atm_checkpoint/rewind/fork/list/effects and checkpoint, approval and effect-memory hooks. Initial five calls cleanly refused checkpointing a workspace overlapping plugin data, listed no checkpoints, rejected a nonexistent id and returned an empty effect ledger. With an explicitly configured separate scratch workspace, checkpoint/list/preview/effects succeeded and bad id failed cleanly. A second five-call conversation confirmed automatic checkpointing and unattended restore approval denial, but also exposed a mismatch: write_file inside the configured protected workspace was labeled an irreversible outside-workspace write because effect classification uses the session cwd. No remote push/publish was performed; effect-memory behavior was covered by unit tests and code review.", "tests": "Passed: documented unittest discovery ran 122 tests, all passed after placing temporary fixtures more than 12 ancestors below the enclosing task Git repository; the initial nested-repository run had 121 passes and one false failure in test_missing_git_dir_is_safe because _find_git walked into the ancestor repository. Pytest was unsuitable here (relative-import setup errors).", "useful": 5, "reliable": 4, "polish": 4, "creative": 5, "beyond": 5, "total": 28, "rank": 2, "best": "Remembers irreversible external effects after conversation rewind and compaction while giving the agent incremental checkpoints and human-gated restores.", "worst": "When configured workspace and session cwd differ, the effect gate can falsely call a protected file write irreversible and block it pending approval.", "blocked": null }, { "pr": 152258, "name": "kumi", "validate": "Pinned https://github.com/user1303836/kumi@7f8b0de9f52d4bca526914c469e084a554388486 folder hermes-plugin. Passed: catalog validation including capability, security, Desktop surface and core-override checks; enabled and doctor passed (30 tools, 8 hooks). One non-fatal warning: private core import tools.approval.is_current_session_yolo_enabled (tools.py:597).", "inside": "Loaded. Registers DAW/audio tools, context references, an auxiliary listening task, lifecycle hooks, skills and Desktop/dashboard surfaces. Four-call real-Hermes plan: kumi_status reported needs_runtime with pinned version 1.24.81 and size; kumi_setup plan for REAPER returned the exact runtime step and digest without installing anything; kumi_connect returned setup_needed cleanly; kumi_read with nonexistent step was unavailable before setup rather than executed. The actual DAW/audio paths were not exercised: no installed runtime or DAW here. The invalid-input path also has extensive fixture-runtime unit coverage.", "tests": "Documented unittest runner: 496 tests, 492 passed, 2 skipped, 1 failure, 1 error. Failure: test_scripts expected an absolute Full code path but policy abbreviates a path under home with ~. Error: test_setup constructs a long fixture path that exceeds setup._MAX_PATH=1024 under this long scratch prefix and is rejected as bad_runtime_reply. These are path-sensitive test limitations, not evidence that real DAW setup or approval was bypassed. Initial pytest run had package-relative-import setup errors and was superseded by unittest.", "useful": 3, "reliable": 4, "polish": 5, "creative": 5, "beyond": 5, "total": 27, "rank": 4, "best": "A detailed native music-production workspace combines DAW control, local audio measurements, undo/history and a sample-map Desktop interface with carefully scoped approvals.", "worst": "Its substantial runtime-and-DAW setup limits day-one reach, and the main music workflow remains unverified here despite extensive fixture tests.", "blocked": null }, { "pr": 151376, "name": "joust", "validate": "Pinned https://github.com/mrnetwork0001/Joust@b88242eb72f50c6dfa284d1f4335e8b59b30afbf repository root. Passed: catalog validation including declared tools/hooks/middleware, security scan, no-core-override and Desktop SDK surface; no warnings. Enabled and doctor passed (8 tools, 4 hooks plus tool_request middleware).", "inside": "Loaded. Adds worktree-based coding tournaments, subagent lifecycle binding, path middleware, benchmark/Crossfire tools, approval-gated merge tools, slash/CLI/skill and Desktop/dashboard arena. Three real-Hermes calls: status returned an empty ledger; run_demo completed a real three-worktree tournament and selected hash-bucket-index; nonexistent bout id returned a clean error. The demo writes its contenders itself, so it does not prove live model-driven subagents; the bundled Hermes-chain tests separately exercised discovery, middleware, lifecycle hooks, terminal dispatch and the no-parent subagent path.", "tests": "Documented unittest runner with Hermes importable: 97 tests, 94 passed, 2 failed, 1 skipped (Desktop smoke prerequisites). Failures: non-UTF8 test expected replacement characters but this /bin/sh printed literal hex escapes; symlink-clamping test returned a lexically inside path still resolving to /etc/hosts. The latter is a real helper regression, although middleware records a violation for the pre-tool veto. Earlier runs were stopped when a supposed non-repository fixture discovered the enclosing task repository; final run used a scratch-only invalid Git boundary. No plugin code was changed.", "useful": 4, "reliable": 4, "polish": 5, "creative": 4, "beyond": 5, "total": 27, "rank": 3, "best": "Makes parallel coding alternatives tangible and comparable through isolated worktrees, rival-test Crossfire, measured ranking and a polished merge-review arena.", "worst": "Its symlink-clamping regression fails locally and shell text is not sandboxed, so worktree scoping should not be treated as full execution isolation.", "blocked": null }, { "pr": 152976, "name": "agentline", "validate": "Pinned https://github.com/prakhar-raj-031104/hermes-agentline@6e4c34e61084e5db154a0159bc57440b6ffbceeb repository root. Passed: catalog validation, including dependency resolution, capability matching, security scan and no-core-override; enabled and doctor passed (9 tools, 1 hook plus platform and skill).", "inside": "Loaded and registered the AgentLine gateway platform and phone tools. In tryit.py without an account key, status/messages tools were not offered and direct scripted calls returned tool-unavailable; the emergency-call step stopped as an invalid model tool call. This reflects has_api_key availability gating, not a plugin import crash. Then ran its bundled integration_hermes.py in real Hermes with local REST/WebSocket stand-ins: status and transcript retrieval worked; 911 was blocked; unattended call/SMS requests failed closed without dialing; an explicitly approval-disabled test configuration exercised a completed mocked outbound call. Registered platform adapter delivered allowed SMS and replies, routed a live caller question and answer, ignored strangers, handled late answers by SMS, and reconnected the relay. No real AgentLine account, phone call or billed action was used.", "tests": "Passed: 38 pytest unit tests and 26/26 bundled real-Hermes integration assertions; live external service not tested.", "useful": 4, "reliable": 5, "polish": 3, "creative": 4, "beyond": 5, "total": 26, "rank": 6, "best": "Gives Hermes an actual phone/SMS platform with live caller questions, scoped secrets, allowlists and approvals proven through real gateway plumbing against local stand-ins.", "worst": "It depends on a paid US/Canada-only hosted service and its key-gated tools disappear before setup, with less visible product polish than this batch\u2019s leaders.", "blocked": null }, { "pr": 153443, "name": "local-hands", "validate": "Pinned https://github.com/noegng/local-hands@62327c4174582e3e8ea2d0b1e8fc57d4f4a24842 repository root. Passed: catalog admission and capability/security checks; enabled and doctor passed (1 tool, 1 hook, terminal backend and commands). One non-fatal warning: private core import tools.environments.base.EnvironmentConnectionError (provider.py:375).", "inside": "Loaded and registered hands plus the terminal-environment provider, /hands, CLI commands and target-context hook. Three-call real-Hermes plan attempted status, use of an unknown machine, and switching to brain; every handler returned a caught [Errno 1] Operation not permitted error. The sandbox denies socket(AF_UNIX), which the hub control RPC requires, so pairing and actual terminal/file routing could not run here. This was a clean tool error rather than an uncaught handler crash; no remote-device success is claimed. Read the provider, TLS hub/daemon and approval code as well as protocol tests.", "tests": "Documented unittest discovery reported Ran 59 tests, FAILED (errors=38): 22 successful method results; 37 method errors plus an Exec class-setup error, principally sandbox PermissionError at AF_UNIX construction. The 19 Exec methods were not reached after class setup failed. Initial pytest collection had package-relative-import errors and was superseded by unittest. No claim that its 78-test upstream suite passed locally.", "useful": 5, "reliable": 3, "polish": 4, "creative": 4, "beyond": 5, "total": 26, "rank": 5, "best": "Extends Hermes\u2019s existing terminal and file tools onto personally owned computers through an outbound pinned-TLS daemon with local human approval.", "worst": "The active computer is shared across gateway conversations rather than scoped per chat, and end-to-end operation could not be verified under this sandbox\u2019s Unix-socket restriction.", "blocked": null } ] }