{ "batch": "deep-47", "setup": "Installed the real NousResearch/hermes-agent source at fdde27e7dae174cb4ba842589e92508852b7dd9f using uv sync --locked --python 3.14 (Python 3.14.10); added its locked dev group and Hypothesis for plugin tests. hermes plugins --help exposes install, validate and doctor. Used RUN.md/tryit.py with Nous's FakeLLMServer and AIAgent, disposable profile homes, no external credentials. Scratch-only harness adjustments: bypass the proxy for localhost; retain complete validation JSON; accommodate current validate --install-deps staging the plugin before enable, and for Teleoperator separate admission and runtime homes to avoid a broken managed source path; apply explicitly stated plugin settings to test profiles. Entrant source is unchanged at each field.json pin. Tests and calls below are local observations, not entrant claims. Deferred tool discovery can hide plugin schemas from the first model offer; doctor registrations and executed calls establish loading. No Desktop browser, live service account or external SSH host was used. Rankings prioritize likely everyday retention and observed reliability when totals are close. In particular, Project Command Center outranks Syrinx despite its creativity/polish points because Syrinx's outbound-action guarantee is too broad; Triage's manual rule-8 violation lowers its reliability to 2. Sources use immutable pinned raw URLs; observed test results are distinguished from README claims.", "plugins": [ { "pr": 152899, "name": "project-command-center", "validate": "Passed: all 16 catalog checks, security scan safe, no warnings; enable and doctor passed (3 tools, 2 hooks). Pin eaa3e2d3a046eb677f4c67aae98b78f1dcec4043, NyllRE/project-command-center repository root.", "inside": "Loaded in real Hermes. Empty plan completed; pcc_board read an empty board, pcc_set_status created an active local test card, pcc_work_item linked PR 152899, and pcc_board read back the persisted lane and link. Invalid lane was rejected by Hermes schema validation before invocation; no crash. Desktop board and authenticated GitHub operations were assessed from source and tests, not a live Desktop/account.", "tests": "All 8 own Python script suites passed (445 printed checks); Git fixtures were isolated from the enclosing checkout before rerunning the 3 affected suites. Node: 50 passed, 6 failed, all six requiring a full-history Hermes checkout at a hard-coded home path; supplied graph fixtures and other Desktop tests passed. No live GitHub credentials or Desktop UI session.", "useful": 4, "reliable": 4, "polish": 5, "creative": 3, "beyond": 5, "total": 26, "rank": 3, "best": "A polished Desktop project board shares durable session lanes and issue/PR links with the agent, closing a useful gap between coding conversations and project state.", "worst": "Its audience is mainly Desktop GitHub developers, and the graph tests assume a particular local Hermes checkout instead of shipping self-contained real-history fixtures.", "blocked": null }, { "pr": 153221, "name": "capalis", "validate": "Passed: all catalog checks, security scan safe, no warnings; enable and doctor passed (7 tools, 1 gateway hook). Pin 78843af2e15812fc391a74fe1cbcf2ac82a03de1, Femtech-web/Capalis/plugin.", "inside": "Loaded. Empty plan completed. All five planned requests returned: status exposed unavailable account-backed adapters and available HTTPS GET; setup named missing tokens and configuration routes; capabilities listed schemas and evidence limits; recipe list was empty; compiling an empty draft returned E_INTENT/E_TRIGGER/E_STEPS without crashing. No external accounts were supplied, so live cross-service writes/read-back and phone approvals were not exercised.", "tests": "pytest: 304 passed, 1 failed, 1 skipped (306 collected). test_closed_without_a_fix_says_so remained WAITING instead of VERIFIED; isolated repetitions failed once then passed twice, indicating a timing-sensitive test/runtime path. Live account test skipped. Property, storage, approval, deduplication, drift and mocked-adapter tests otherwise passed.", "useful": 3, "reliable": 4, "polish": 4, "creative": 4, "beyond": 5, "total": 25, "rank": 5, "best": "It adds durable, owner-approved automation with pinned recipes, independent read-back, deduplication and dependency-drift quarantine instead of trusting an agent to report success.", "worst": "Several service tokens and channel settings are needed for its main workflows, and the closure-notification test intermittently fails before verification completes.", "blocked": null }, { "pr": 154048, "name": "triage", "validate": "Automated check passed all 16 checks (security scan safe), with private-core-import and Desktop-selector warnings; enable and doctor passed. Manual review finds a catalog rule 8 conflict: desktop/plugin.js:839-845 changes core jump-button inline styles, and line 1440 overrides .thread-jump-button bottom with !important CSS. Pin 913a61d62b0795926531ecdfbf8980a1e9c0b9c9, Saderius/hermes-triage repository root.", "inside": "Loaded. Empty plan completed. With its bundled demo skill folder configured, a review request automatically attached pr-review. A second 5-call run with auto_accept_threshold=0 and interactive=false demonstrated search without attachment, explicit skill_load returning the exact body, repeat-load deduplication, unknown-name reporting and an empty-query error. No crash; no Desktop renderer was launched.", "tests": "pytest: 139 passed, 2 skipped (141 collected); both skips are Windows-only file-change notification tests. Skills, indexing, loading, hiding/restoration, Desktop source/backend, context, compaction and guard tests passed.", "useful": 4, "reliable": 2, "polish": 4, "creative": 4, "beyond": 5, "total": 24, "rank": 6, "best": "It replaces a large always-present skill index with searchable, selectively loaded skills, with real request middleware and compaction-aware session bookkeeping.", "worst": "Despite passing the automated gate, its DOM queries, inline style writes and !important CSS reposition Hermes's own jump button, contrary to catalog rule 8.", "blocked": null }, { "pr": 149685, "name": "hermes-teleoperator", "validate": "Passed: all 16 catalog checks. Security scan caution for SSH-directory references; private-import warnings for the two tools.approval gates. Same-profile tryit enable failed on a missing managed package-source path; using a separate admission profile allowed ordinary native-directory enable and doctor to pass unchanged (1 tool, 2 hooks, execution middleware). Pin 4dfb08eb0bba00f7c2036613290a9e6da5650d8d, Tsurgcom/hermes-teleoperator repository root.", "inside": "Loaded after isolating catalog validation from runtime installation. All 5 calls returned in real Hermes: status reported local, switch local succeeded, switch to an unconfigured machine returned an explicit error with fallback=false, remote search without a selected remote refused cleanly, and final status remained local. No external SSH host was available. Separately exercised the official v1.2.0 receiver whose SHA256 exactly matches version.py at the pin; protocol, file/patch/search, persistent shell, process cleanup, skill and transfer tests passed.", "tests": "Own unittest runner: 153 tests, 2 failures, 5 error reports, 16 skipped before supplying the receiver; deployment failures/errors all trace to the sandbox lacking /proc shell-ELF probe data. With the checksum-pinned receiver, standalone suite: 8 passed, 1 skipped (root/chroot proof). Node Desktop suite: 34 passed. Actual SSH/deployment and chroot tests unavailable; Rust build not completed. The initial pytest command could not collect the hyphenated package, so its documented unittest runner was used.", "useful": 4, "reliable": 4, "polish": 4, "creative": 5, "beyond": 5, "total": 27, "rank": 2, "best": "Conversation-scoped SSH routing makes one Hermes agent operate native tools on other machines through a small standalone receiver without installing Hermes or a model there.", "worst": "Remote execution bypasses other plugins' pre_tool_call guards and full native-tool parity, and Unix-only receiver setup still requires careful SSH provisioning.", "blocked": null }, { "pr": 153938, "name": "syrinx", "validate": "Passed: all 16 catalog checks, security scan safe; warning for private import agent.pet.store.register_local_pet; enable and doctor passed (4 tools, 6 hooks). Pin 444db83b1a4fe902916cec25b2de27c389075538, cstljuan/syrinx repository root.", "inside": "Loaded. At the actual test time its default night policy queued the empty-plan request before the model, with a clear /late override. With documented night_enabled=false in the disposable profile, all 5 planned requests worked: start a 30-minute block, park a question with Markdown default, block git push before execution, reject an empty question, and report focus active with one parked question. No waiting for the focus timer. Static classifier probes (no network execution) showed requests.post in execute_code and browser_exec POST code receive no outward-action hold.", "tests": "pytest: 64 passed, 0 failed. Tests cover core focus/night/standing-answer logic, hooks, backend, privacy/presence and cards. Extra source-level classifier probes exposed unheld generic Python/browser network writes. macOS auto-focus and Desktop rendering were not run.", "useful": 4, "reliable": 3, "polish": 5, "creative": 5, "beyond": 5, "total": 27, "rank": 4, "best": "It turns attention management into real agent behavior by parking questions with defaults, batching their return and keeping a code-counted account of work done while the user is away.", "worst": "The claim that nothing leaves the machine during focus is broader than its regex-based hold logic, which does not catch generic Python HTTP writes or browser execution.", "blocked": null }, { "pr": 151929, "name": "promise-keeper", "validate": "Passed: all 16 catalog checks, security scan safe; warnings for private core imports in the working-directory helper and live test script. Enable and doctor passed (0 tools, 3 hooks; execution middleware is registered and was exercised). Pin ab91fee21f607d5515a8e007cf615fc285718dd0, MokiMeow/promise-keeper repository root.", "inside": "Loaded and completed the empty plan. Five native-tool requests read a fixture, wrote and read an allowed note, returned a clean missing-file error, and blocked an attempted rules.db overwrite under built-in R0. The reply proposed the user's prod-folder rule for confirmation rather than activating it silently. The plugin's live_e2e.py then exercised real Hermes discovery, user slash-command confirmation, protected writes/deletes/patches, middleware-only blocking, allowed work, self-protection and revocation: 19/19 checks passed.", "tests": "pytest after isolating Git discovery from the enclosing checkout: 664 passed, 11 failed, 8 skipped, 2 xfailed (685 collected). All remaining failures concern Windows command/path forms evaluated on Linux; Windows behavior was not verified. Four apparent Git fallback failures in the first run disappeared with fixture isolation. Real Hermes live_e2e.py: 19 passed, 0 failed. Desktop browser click harness not run.", "useful": 5, "reliable": 4, "polish": 5, "creative": 4, "beyond": 5, "total": 28, "rank": 1, "best": "It makes a user's persistent rules enforceable outside the model, with explicit activation, quoted explanations, self-protection and a clear Desktop control panel.", "worst": "Its shell analysis remains best effort, with eleven Windows-form assertions failing on this Linux run and documented gaps for opaque third-party tools and later argument rewrites.", "blocked": null } ] }