{
  "batch": "deep-15",
  "setup": "Hermes installed once from https://github.com/NousResearch/hermes-agent at fdde27e7dae174cb4ba842589e92508852b7dd9f using uv sync --locked --python 3.14; Python 3.14.10. install/validate/doctor CLI commands were present. Used the supplied RUN.md and tryit.py with Nous FakeLLMServer and real AIAgent.run_conversation, without a real provider or account. Initial loopback HTTP 502s were fixed by excluding localhost from the proxy. The stock runner enable stage timed out preparing Hermes's separate package-manager runtime. Supplemental copies of tryit.py therefore pre-enabled each plugin in an isolated test profile, installed declared dependencies into the Hermes venv, disabled tool deferral and title generation, and retained the real catalog validator, doctor, loader and agent/tool dispatch. --install-deps preparation was refused with lazy installs disabled; catalog checks still ran against installed dependencies, so this is not a verified fresh catalog installation. Plugin source was not patched. Housemate's planned run uses its real SQLite store, TimerService and authenticated loopback ControlServer without speakers; Scout is enabled through its operator CLI; agent-si selects its memory provider with synthetic principal and session indexing disabled. Runtime/test counts below are local observations; source URLs identify pinned implementations, not independent certification of those observations.",
  "plugins": [
    {
      "pr": 153357,
      "name": "twring",
      "validate": "Passed all 15 catalog checks with no warnings; runtime doctor passed registration as a platform. The standard tryit.py enable stage timed out at 120 seconds in Hermes PM preparation; the pre-enabled test-profile run passed enable and doctor. Pin/folder: https://github.com/saasagelabs/twring/tree/5a7083e3c9d1cd656f1fd85da771d6eee3d2267a/integrations/hermes.",
      "inside": "Loaded the twring platform and its CLI, with no agent tools or hooks added, as expected for this adapter. Three terminal calls executed inside real Hermes: twring status returned an actionable not-linked message; twring --help exposed link/status/unlink; link with an ftp server failed immediately with the HTTPS requirement. The unit suite separately exercised the actual Hermes adapter over local fake Twring WebSockets, including streaming and approval responses. No real Twring account, mobile app or speech server was exercised. Sources: pinned adapter.py and cli.py in sources.json.",
      "tests": "84 passed, 0 failed, including tests/test_adapter.py against real Hermes and a fake local WebSocket server; linking/CLI/protocol tests also passed.",
      "useful": 5,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 3,
      "best": "It makes the user's existing Hermes reachable by phone voice and typed chat, with native approval responses and scheduled-result ringing rather than a separate replacement agent (pinned README.md and adapter.py in sources.json).",
      "worst": "The central mobile speech and ringing experience still depends on a linked Twring account and server, so local adapter tests cannot establish live phone reliability (pinned README.md in sources.json).",
      "blocked": null
    },
    {
      "pr": 153853,
      "name": "keryx",
      "validate": "Passed: all 15 catalog checks, including dependency imports, capability declarations, security scan and no core override; runtime doctor passed with one tool and two hooks. --install-deps could not prepare the separate PM runtime, as explained in setup. Pin: https://github.com/hsyvy/keryx/tree/6c5cd7fc468a314eef0b524a29b1f12c68c0e569/keryx.",
      "inside": "Loaded place_call, pre_tool_call and agent_loop_stopped hooks, /calls and keryx CLI. Five scripted Hermes calls ran: 112 was blocked as emergency; a malformed number was blocked; missing goal was rejected; CLI doctor reported missing PipeWire programs/phone service; CLI simulate completed a fake disclosed call with achieved outcome and transcript. No real call was dialed and no live voice-provider connection was tested. Source: https://raw.githubusercontent.com/hsyvy/keryx/6c5cd7fc468a314eef0b524a29b1f12c68c0e569/keryx/tools.py.",
      "tests": "75 passed, 3 failed in 78 tests after fixing the scratch temp directory; all three failures are Bluetooth tests assuming pw-record/pw-play/pw-dump are installed, including DTMF attempted after that preflight failed. Call-flow, guardrail and fake voice tests passed. These failures limit hardware evidence; they were not unhandled Hermes tool crashes.",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 5,
      "beyond": 4,
      "total": 26,
      "rank": 6,
      "best": "A real own-SIM outbound voice workflow combines single-use Hermes approval, deterministic dial restrictions, disclosure, transcript and mid-call questions, well beyond a prompt (pinned tools.py and guardrails.py in sources.json).",
      "worst": "Linux/PipeWire/Bluetooth and a paid realtime voice provider narrow adoption, and the three hardware-assumption test failures leave the actual phone path unverified (pinned README.md in sources.json).",
      "blocked": null
    },
    {
      "pr": 152458,
      "name": "housemate",
      "validate": "Passed all 16 catalog checks without warnings, including Desktop surface and security scan; runtime doctor passed seven tools and one hook. The stock enable timeout and dependency preparation caveat are in setup. Pin: https://github.com/RaYYeR220/housemate/tree/09c1a1d0a88bfa8762cee6b3ccf94271b0f172c5.",
      "inside": "Loaded the housemate platform, seven room/timer/screen tools, spoken-answer hook and streamed TTS provider. Five scripted Hermes tool calls reached the real authenticated ControlServer and SQLite TimerService: listed Kitchen/Office, created a 600-second kitchen tea timer, listed it, rejected -1 seconds with a clear range error, then cancelled it successfully. This was a headless hub with offline rooms, not a microphone, speaker, wake-word or STT/TTS demonstration. Sources: pinned core/control.py, core/models.py and README.md in sources.json.",
      "tests": "569 passed, 1 skipped, 0 failed with real Hermes importable, including platform contracts, approval/clarify behavior, pairing, control API, timers, persistence and Desktop contracts. An initial run from the plugin root had 23 import-shadowing failures; running from a neutral scratch directory resolved all of them without changing plugin source.",
      "useful": 5,
      "reliable": 5,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 29,
      "rank": 1,
      "best": "It turns spare browser devices into a coherent room-aware Hermes platform with persistent timers, wake arbitration, native approvals and a finished Rooms interface (pinned README.md and control.py in sources.json).",
      "worst": "The full hands-free experience still requires paired browsers, working STT/TTS and downloaded wake/VAD models, and physical audio behavior was not verified here (pinned README.md and models.py in sources.json).",
      "blocked": null
    },
    {
      "pr": 154025,
      "name": "omniscience",
      "validate": "Passed all 16 catalog checks, including Desktop surface, with a review warning for raw window.hermesDesktop bridge calls at desktop/plugin.js lines 47, 59 and 179. Runtime doctor passed 14 observer hooks and no tools. Dependency preparation caveat is in setup. Pin: https://github.com/KeyArgo/hermes-omniscience/tree/38ea8c249698235836a0aa3f35c480f2d87c93fa.",
      "inside": "Loaded all 14 observer hooks, /omni command and Desktop integration, with no added agent tools expected. Three scripted read_file calls ran inside Hermes: valid fixture, missing file, valid one-line range. Inspection of the actual events.db found 15 session/turn/model/tool events, with the three tool statuses ok/error/ok. Stored event rows contained neither the synthetic email/card nor the fixture text; tool records carried argument names, duration and result length. The live Desktop renderer was not visually exercised. Sources: pinned sanitize.py, desktop/plugin.js and README.md in sources.json.",
      "tests": "Python: 793 passed, 12 skipped, 0 failed. Frontend: 595 tests passed across 56 files. npm run build passed TypeScript checking and the Vite production build, with only a large-chunk advisory. These are observed counts, not the README badges.",
      "useful": 5,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 2,
      "best": "It makes real Hermes execution visible through metadata-only lifecycle tracing, a live graph and deterministic replay without a model or external tracing account (pinned README.md and sanitize.py in sources.json).",
      "worst": "Its Desktop page still reads its viewer key through raw Desktop bridge APIs flagged by catalog review, creating SDK-compatibility debt despite the passing gate (pinned desktop/plugin.js in sources.json).",
      "blocked": null
    },
    {
      "pr": 153872,
      "name": "local-scout",
      "validate": "Passed all 15 catalog checks with no warnings; doctor passed nine tools, nine hook registrations (pre_auxiliary_call has two callbacks) and both middleware registrations. Dependency preparation caveat is in setup. Pin: https://github.com/Angerosteles/local-scout/tree/c1e732ba4e9d431a711ea141c0095352a95bd7a2.",
      "inside": "Loaded all nine scout tools plus masking/observation hooks and request/execution middleware. Enabled report/observe/insights/mask consent with the actual operator CLI. All five scripted calls ran: read_file returned [EMAIL]/[CARD] with a prompt-injection alert; scout_report recorded those categories; scout_calculate returned a cost estimate; ../escape dataset ID returned INVALID_INPUT; scout_handle without the necessary grant returned CONSENT_REQUIRED. Recorded fake-provider messages contained neither synthetic email nor card, including the initial user message. No optional basal server or calibrated in-turn triage was tested. Sources: pinned scout/core.py, scout/prompt_guard.py and README.md in sources.json.",
      "tests": "unittest discovery ran 287 cases: 282 passed, 1 failed, 4 skipped. Diagnostic replay isolated the failure to test_artifacts_reload_and_decision_measured_outcome expecting own_cnn training with optional torch absent; the plugin correctly returned PLAN_ONLY and an unavailable message. logreg, knn and embed_search training succeeded in that replay. Initial pytest collection/package-name errors were harness layout issues; the source remained unchanged.",
      "useful": 4,
      "reliable": 4,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "rank": 5,
      "best": "Consent-scoped middleware demonstrably removed detected personal data from real Hermes provider payloads and tool output while keeping local audit counts instead of raw content (pinned core.py and prompt_guard.py in sources.json).",
      "worst": "One-way masking prevents the model from using original addresses, and pattern-based detection and optional calibrated triage have explicit coverage limits (pinned README.md in sources.json).",
      "blocked": null
    },
    {
      "pr": 149242,
      "name": "agent-si",
      "validate": "Passed all 15 catalog checks, with a review warning for five imports from plugins.memory.config_schema; runtime doctor passed the exclusive memory provider. Dependency preparation caveat is in setup. Pin: https://github.com/joelbrilliant/hermes-agent-si/tree/adf7abc1d89b1e1ea686ac2fd29e2b95b454f4fe.",
      "inside": "Loaded agent-si as the selected Hermes memory provider and exposed brain_search, brain_get and brain_remember. With a synthetic principal and indexing disabled, five real Hermes calls stored a quoted Friday fact, added a conflicting quoted Saturday fact, retrieved both as disputed, refused an invented Sunday quotation, and found the fact and its source through search. Cross-agent filesystem indexing was intentionally not run; its two Hermes schema variants are covered by the plugin tests. Sources: pinned __init__.py and README.md in sources.json.",
      "tests": "19 passed, 0 failed with HERMES_AGENT_SRC pointing to the real Hermes checkout; tests cover provider lifecycle, memory-manager integration, quote/evidence refusal, disputes, settings and both Hermes history schemas.",
      "useful": 5,
      "reliable": 4,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 27,
      "rank": 4,
      "best": "It gives Hermes persistent, source-citing memory that preserves contradictory claims and enforces quotation provenance in code rather than trusting a prompt (pinned __init__.py and README.md in sources.json).",
      "worst": "Recall is keyword-based, and the documented shared upstream module names plus private config-schema imports leave compatibility and plugin-collision risk (pinned README.md and the observed catalog warning).",
      "blocked": null
    }
  ]
}