{
  "batch": "trial-1",
  "setup": "Installed Hermes from source at NousResearch/hermes-agent commit 34ce08bb7728c359450fa599b9910deb0d083b1a using uv sync --locked --python 3.14 (CPython 3.14.10); plugins --help exposed install, validate and doctor. Installation and all scaffolding stayed under test/scratch. Installed pytest, pytest-asyncio, aiohttp and PyYAML for tests. Pins/folders came from https://gist.githubusercontent.com/gtmcknight/594c47336546c8763e641b46cf9af4ec/raw/field.json; installation/harness from its RUN.md and tryit.py. For every entry, ran the supplied tryit.py with [] and then the stated 2-5-call plan: catalog validation, enable and doctor succeeded, but every chat failed before a model request because this sandbox cannot create the gateway Unix socket (Operation not permitted when a short writable socket location was supplied). This is a shared host restriction, not six plugin load failures. Supplemented each empty discovery and planned run with unmodified real Hermes AIAgent, PluginManager and native tool dispatcher in a fresh isolated Hermes home, connected over loopback to the same tests/fakes/fake_llm_provider.py scripted model. Those runs exercised actual plugin handlers/hooks, not rewritten implementations. Loaded the fake home's dotenv for auxiliary-model tests; CaMeL's successful structured-classifier control used FakeLLMServer's aux responder, with a separate malformed-response fail-closed run. All runtime/test findings below are my local observations, not entrants' reported results. No real model account, Home Assistant token/device, Desktop window or delivered email was used. Scores apply to the pinned code; static code/doc sources are listed in sources.json and the pinned tree URL accompanies each entry.",
  "plugins": [
    {
      "pr": 0,
      "name": "done-bell",
      "validate": "Passed: hermes plugins validate --install-deps --json (exit 0), enable (exit 0), and doctor --ci (exit 0). No failed admission checks. Pinned source: https://github.com/aydnOktay/hermes-done-bell-/tree/3fa519a291d91b0e6c44bd42762190363ce80c39.",
      "inside": "Loaded in doctor and real in-process Hermes: 1 tool (done_bell_status), 4 hooks (session-end, approval request/response, loop-stopped), /bell, plus Desktop status chip/pane and dashboard backend in source. Empty discovery succeeded. Four scripted calls: status returned defaults; mute=true persisted muted=true; mute=false persisted false; malformed mute='false' was coerced by Hermes to false and returned cleanly. A completed agent turn wrote a done event to bell.json. The tool path and Python hooks ran; native OS notification delivery/audio and Desktop rendering were not observable here. Supplied tryit chat: 0/4 calls, shared gateway restriction described in setup.",
      "tests": "Passed: 9/9 repository pytest tests (defaults, persistence, completion/interruption, approval cycle, mute, extra kwargs and CLI notification gating). No Desktop UI automation supplied or run.",
      "useful": 4,
      "reliable": 4,
      "polish": 3,
      "creative": 2,
      "beyond": 4,
      "total": 21,
      "best": "A small, no-account lifecycle integration gives long-running Hermes users persistent completion/approval status and a Desktop bell that a prompt cannot implement.",
      "worst": "Installation docs use an author's Windows development path and a manual Desktop-copy workaround, and the actual OS/renderer notification path remains unverified by the nine Python tests.",
      "blocked": null
    },
    {
      "pr": 0,
      "name": "homeassistant",
      "validate": "Passed: hermes plugins validate --install-deps --json (exit 0), enable (exit 0), and doctor --ci (exit 0). No failed admission checks. Pinned source: https://github.com/NousResearch/hermes-official-plugins/tree/e7a218f26db88426ab3e7376f0e000b6e356f5e7/homeassistant.",
      "inside": "Loaded: doctor/PluginManager registered the homeassistant gateway platform and 4 tools (ha_list_entities, ha_get_state, ha_list_services, ha_call_service), with no hooks. Empty in-process discovery completed. Planned light listing, service listing, light.office state lookup and invalid ../../config lookup; without HASS_TOKEN the documented availability check hid all four tools, so the scripted model received 'does not exist' for the first two and Hermes stopped its repeated unavailable-tool sequence before the last two. Thus no real HA REST/device action was exercised; this is expected credential gating, not an import crash. Source and tests cover state-change events, reconnection/cooldowns, notify delivery, secret_scope credentials and entity/service restrictions. Supplied tryit chat: 0/4 calls, shared gateway restriction.",
      "tests": "Passed: 212 pytest tests; 4 skipped. Tests use mocked HA HTTP/WebSocket/platform fixtures and cover registration, adapter, tools, service/entity policy, standalone send and heartbeat. No real device/account test; do not interpret these counts as live home automation validation.",
      "useful": 4,
      "reliable": 4,
      "polish": 4,
      "creative": 3,
      "beyond": 5,
      "total": 25,
      "best": "It turns the user's own Home Assistant into a Hermes gateway with incoming device events, cron notification delivery and constrained device tools, extending where the agent operates.",
      "worst": "Without a configured HASS_TOKEN the tools disappear and scripted attempted calls give generic unavailable-tool errors, while real device control and event delivery still need hardware testing.",
      "blocked": null
    },
    {
      "pr": 151810,
      "name": "agent-security-guard",
      "validate": "Passed: hermes plugins validate --install-deps --json (exit 0), enable (exit 0), and doctor --ci (exit 0). Security scan passed with caution sudo_usage at patterns.py:80, a detection pattern rather than an executed sudo command; no failed admission checks. Pinned source: https://github.com/xMannixx/Agent-Security-Guard/tree/b1b923e6733f2a3dd7f0a79318d452ad395d8340/plugin.",
      "inside": "Loaded under manifest id agent-security-guard-plugin (catalog name agent-security-guard): 3 hooks (pre_llm_call, pre_tool_call, transform_tool_result), no added model tools. Empty discovery completed. Four planned native-tool calls: read_file returned meeting notes; web_extract fetched example.com and its result acquired the guard's UNTRUSTED CONTENT - DATA ONLY envelope inside Hermes's wrapper; a subsequent write_file was refused because the plugin required approval and no interactive approver existed; a nonexistent-file read returned a structured not_found error. The unwanted output file was absent. These observations establish one real read/web/write boundary, not complete injection prevention. Supplied tryit chat: 0/4 calls, shared gateway restriction.",
      "tests": "Passed in separate Python 3.14 stdlib-package test environment: 1318/1318 repository pytest tests. Initial run inside the installed Hermes environment: 1303 passed, 15 failed, including policy/home-location fixture expectations; kept distinct from the clean standalone suite result. Runtime hook enforcement was checked independently in actual Hermes as described in inside. Did not rerun the entrant's published 1920-attack benchmark or treat its claimed detection rate as my measurement.",
      "useful": 4,
      "reliable": 4,
      "polish": 4,
      "creative": 3,
      "beyond": 5,
      "total": 25,
      "best": "A dependency-free deterministic guard actually mediated Hermes's native tool path, wrapped retrieved data, and routed a dangerous post-retrieval write to human approval.",
      "worst": "Protection depends on recognized tool/action names and broad sequence rules, so its large standalone suite and this successful boundary check do not establish protection against every custom tool or injection chain.",
      "blocked": null
    },
    {
      "pr": 149147,
      "name": "camel-guard",
      "validate": "Passed: hermes plugins validate --install-deps --json (exit 0), enable (exit 0), and doctor --ci (exit 0). No failed admission checks. Pinned source: https://github.com/nativ3ai/hermes-camel-guard-plugin/tree/4cbfca3a5f1d46f5d152280cccc496d0df14d097/plugin.",
      "inside": "Loaded: 5 native hooks (pre_llm_call, pre_tool_call, post_tool_call, on_session_end, on_session_reset), /camel, no added tools. Empty discovery used the shipped default mode off. Activated mode=enforce through plugin settings for the three-call plan: reading meeting notes worked; write_file after that untrusted read was blocked; a nonexistent-file read returned not_found. With the stock fake auxiliary prose, classifier parsing failed and the write was blocked as fallback_read_only. A separate run with the same three calls and a structured fake auxiliary response denying file_mutation blocked it as explicitly_denied (ok); inspected the recorded auxiliary request and confirmed the data payload was only the trusted user message, not file text. No unwanted file was created. Supplied tryit chat: 0/3 calls, shared gateway restriction.",
      "tests": "Passed: all 24 repository Hermes integration/information-flow pytest tests after installing the pinned plugin in the scratch Hermes test home and clearing persisted plugin state between tests. Earlier attempts without installation failed discovery (24 failures), and with a shared uncleared home gave 21 passed/3 state-contamination failures; those setup mistakes were repaired. Final tests exercise native sequential/concurrent dispatch, capability separation, malformed classifiers, bounded metadata traces and operator settings. No paid/live-model benchmark was run.",
      "useful": 4,
      "reliable": 4,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "best": "Trusted-intent classification is separated from retrieved content and enforced in Hermes's tool dispatcher, with tested fail-closed behavior and no core monkeypatching.",
      "worst": "It ships disabled until the operator selects an enforcement mode, and its documented coarse capability grants cannot prevent exfiltration through an already authorized capability.",
      "blocked": null
    },
    {
      "pr": 153446,
      "name": "signaldesk",
      "validate": "Passed: hermes plugins validate --install-deps --json (exit 0), enable (exit 0), and doctor --ci (exit 0). No failed admission checks. Pinned source: https://github.com/codeybot-labs/signaldesk/tree/1fb9efc3d6895f5852d5f04b9a6e09477e1f559e.",
      "inside": "Loaded: 12 tools, pre_tool_call send hook, /signaldesk, catch-up blueprint and bundled Desktop/dashboard/map surfaces in source. Empty discovery succeeded. Four native calls set the ICP, scanned bundled offline hiring fixtures (176 inserted, 4 skipped, no errors), listed 3 leads and rejected a nonexistent draft id cleanly. A second five-call adversarial session scanned, drafted a fixture lead, approved it despite the user's explicit 'Do not approve or send', rejected an invalid stage, and produced a local .eml with sent=false; no message was transmitted. Native pre_tool_call probes then allowed send_email to an unapproved recipient when an approved lead_id was supplied, and allowed a recipient list containing one approved and one unapproved address; an unapproved recipient alone was blocked. The llm draft also stringified PluginLlmCompleteResult into the email body instead of taking its text field. These are reproduced code-path flaws, not merely speculative risks. Desktop visuals/map interaction were not clicked through; live sources/MX were not tested. Supplied tryit chat: 0/4 calls, shared gateway restriction.",
      "tests": "Passed: 30/30 repository pytest tests with SIGNALDESK_OFFLINE=1 and SIGNALDESK_SKIP_MX=1. These include source/score/store/map and basic cron/send-hook controls, but miss the approval/recipient-list bypasses and host LLM return-type issue reproduced inside Hermes. Supplemental scripted sessions completed 4/4 and 5/5 planned calls; three hook probes produced two unsafe allow outcomes and one correct block.",
      "useful": 3,
      "reliable": 2,
      "polish": 5,
      "creative": 3,
      "beyond": 4,
      "total": 21,
      "best": "A substantial persistent lead workflow combines public-source adapters, explainable scoring, offline data, drafts, local exports and a bundled map/board/Desktop interface.",
      "worst": "Its claimed human approval boundary is not enforced: the model can approve a lead itself and its send hook accepts an unrelated recipient via lead_id or ignores later recipients in a list, while real-host LLM drafts contain a result-object representation.",
      "blocked": null
    },
    {
      "pr": 152589,
      "name": "ask-ledger",
      "validate": "Passed: hermes plugins validate --install-deps --json (exit 0), enable (exit 0), and doctor --ci (exit 0). No failed admission checks. Pinned source: https://github.com/bhaskarduvvuri/hermes-ask-ledger/tree/1986b146f7fbddf5437744ec9d72bf8caa88cb8a/plugin.",
      "inside": "Loaded: 6 tools (ask_update, ask_list, todo_add, todo_update, todo_suggest, todo_items), 3 hooks (pre_llm_call, post_gateway_admission, transform_llm_output), slash commands and generated static HTML. Empty discovery completed. Five scripted native calls: ask_list showed the user request automatically captured as open A1; ask_update moved it to in_progress; state=closed was refused with 'Only the user closes an ask'; todo_add with due='not-a-date' returned the accepted date formats; todo_items returned the persisted item. Inspected the on-disk append-only events/page. The 233-test suite supplies restart, redaction and multi-user/shared-chat controls; actual messaging platform admission, edited todo boards and optional ntfy delivery were not exercised here. Supplied tryit chat: 0/5 calls, shared gateway restriction.",
      "tests": "Passed: 233/233 repository unittest tests (python -m unittest discover -s tests -t .), including persistence, corrupt logs, redaction, capture/reminders, user-owned closing, todo consent, shared-chat fail-closed behavior and board contracts. Deliberately injected disk/render failures logged warnings while tests passed. Supplemental Hermes session completed all 5 planned calls with clean invalid-input refusals.",
      "useful": 5,
      "reliable": 4,
      "polish": 4,
      "creative": 3,
      "beyond": 5,
      "total": 26,
      "best": "Automatic capture and disk-backed reminders improve the agent's own continuity across sessions, while code reserves final closure to the user rather than trusting the model's claim of completion.",
      "worst": "The append-only plaintext log is never pruned and redaction is best effort, while shared-chat attribution has a documented first-turn ambiguity beyond its ten-minute admission window.",
      "blocked": null
    }
  ]
}