{
  "batch": "deep-06",
  "setup": "Installed Hermes source at https://github.com/NousResearch/hermes-agent commit 77d34519df69cc3c4ece7e7b260f5c2de6898fbe using uv sync --locked --python 3.14; plugins help confirmed install/validate/doctor. Fetched every plugin repository at its field.json SHA; all six use the repository root. Ran the supplied tryit.py empty-plan discovery, then planned calls through real AIAgent with Nous tests/fakes/fake_llm_provider.py: no real model, credential or account. For final planned runs, a scratch copy of tryit.py retained full validation output and recorded requests; setup preludes called the real /fluent adopt handler and /confidential handlers, with vision explicitly routed to the same loopback test model for the on-host lock. Hermes and plugin handlers were not replaced. Corrected plans use Hermes tool_search/tool_describe/tool_call for deferred plugin tools. Early scratch-home collisions and disabled-lazy-install PM timeouts were repaired; final planned runs for all six passed validation, enable and doctor and completed their conversations. Installed pytest/PyYAML and confidential-mode declared dependencies in scratch. Test counts and runtime outcomes below are local observations, not published entrant measurements; overlapping test runs are not additive. Read pinned READMEs, manifests, implementation and tests. No live messaging gateway, attested vendor inference or interactive Desktop UI was tested; Lineage Desktop rendering was tested with Node. Rank breaks the top score tie in favor of broader everyday usefulness and less setup: Fluent, Lineage, Habits.",
  "plugins": [
    {
      "pr": 150807,
      "name": "manyhands",
      "validate": "Passed: supplied tryit.py catalog validate exit 0, including isolated registration, declared tools/hooks/middleware and no core override; warning destructive_root_rm points to a README example. Initial enable and doctor both exit 0 (6 tools, 3 hooks).",
      "inside": "Loaded at bc5ec1cb57bb26a133f7967bd06dee6441409544. Adds six person/memory/audit tools, identity/role hooks, memory-filtering middleware, /many and an owner approval transport. Final five-call Hermes plan searched/described deferred tools, dispatched whoami and people, rejected empty remember text cleanly, then dispatched recall and audit. It returned local operator/owner, an empty roster and recall, and an audit trail of tool decisions and scripted-model usage. Initial tool listing omitted the six deferred tools; discovery found all six. Gateway delivery of the owner DM was not exercised.",
      "tests": "Passed: 68 stdlib self-tests (0 failures/errors) and repository live_integration.py against real Hermes PluginManager (LIVE INTEGRATION: PASS), covering human command authorization, role blocking, one-shot grants, private facts, identity spoofing and shared-session attribution. Final planned conversation completed all 5 calls.",
      "useful": 4,
      "reliable": 4,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "rank": 4,
      "best": "Human identity, per-person memory and owner-directed approvals make a shared Hermes gateway substantially more useful than a prompt could.",
      "worst": "A live gateway is needed for owner DM approvals, and calls nested inside execute_code can lose identity and fail closed.",
      "total": 26,
      "blocked": null
    },
    {
      "pr": 151543,
      "name": "seedguard",
      "validate": "Passed: catalog validation, enable and doctor all exit 0 in supplied empty-plan tryit.py; declared hooks/middleware and registration checks passed; no new tools.",
      "inside": "Loaded at 4809791d06471235fe3243fd1888c66eefb129e1. Adds request/execution/tool middleware, output and gateway transforms, plus /seedguard. In real AIAgent with three calls, a synthetic DEMO_API_KEY assignment in user text and terminal output became an ENV_SECRET placeholder; an unknown placeholder was blocked with an explanation, and ordinary text passed. Follow-up real-provider-request inspection found a boundary gap: the same synthetic assignment immediately after a single quote in a shell snippet passed unchanged in the first user request (printf followed by a single-quoted DEMO_API_KEY assignment); literal quoted command arguments in assistant history also survived. No real secret was used. These are observed misses, not hypothetical leakage. Armed recovery was covered by repository integration tests, not by a live human chat.",
      "tests": "Passed: 157 tests, 1 skipped, no failures in a clean run with HERMES_DISABLE_LAZY_INSTALLS=1, including the repository real-Hermes integration test (8.00s). Final planned harness conversation completed all 3 calls. Additional recorded-request probe reproduced the quoted-assignment detection miss.",
      "useful": 5,
      "reliable": 3,
      "polish": 4,
      "creative": 3,
      "beyond": 5,
      "rank": 6,
      "best": "Session-scoped placeholders and human-armed tool rehydration let Hermes use supported secret formats without normally showing their values to the main model.",
      "worst": "A quoted shell assignment leaked the synthetic value to the recorded model request, and auxiliary model routes are explicitly outside the middleware protection.",
      "total": 25,
      "blocked": null
    },
    {
      "pr": 149244,
      "name": "lineage",
      "validate": "Passed: supplied empty-plan and final planned catalog validation, enable and doctor all exit 0; no validation warnings. Ten observer hooks, no tools/middleware, slash/CLI commands and Desktop integration.",
      "inside": "Loaded at 61de70980f4322838a9b53f4802623146f1046c2. Three planned real-Hermes calls read a scratch CSV, wrote a new Markdown artifact and cleanly reported a nonexistent input. hermes lineage show --json produced a lineage.receipt.v1 receipt with the write_file producer, fake-model call, artifact hash and evidenced CSV link. CLI receipt --write and check confirmed the artifact/receipt hashes and signature; verify reported fresh, then stale after the CSV changed. No new tools, as expected for observer hooks. Desktop rendering passed Node tests; no interactive Desktop session.",
      "tests": "Passed: 19 Python tests and 2 Node Desktop tests, no failures. Actual CLI receipt --write and check returned valid artifact/receipt hashes and a valid signature; verify returned fresh, then stale (exit 2, expected) after modifying the source CSV.",
      "useful": 4,
      "reliable": 5,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "rank": 2,
      "best": "Automatic, inspectable receipts connect an agent-produced file to actual tool calls and labelled sources without depending on the model to remember its provenance.",
      "worst": "Inferred links are not proof of derivation, terminal attribution uses a bounded time window, and receipt signatures authenticate only the issuing install.",
      "total": 28,
      "blocked": null
    },
    {
      "pr": 151301,
      "name": "confidential-mode",
      "validate": "Passed: empty-plan and final planned catalog validation, enable and doctor all exit 0. Hooks, execution middleware, provider profiles and commands registered; validator warns about documented private Hermes imports.",
      "inside": "Loaded at e407486a2227be8cafe7b4350586f3016ebff573. Adds confidential provider profiles, preflight/locking, model/tool gates, metadata ledger, receipts and slash/CLI commands; no new tools. Real-Hermes preflight correctly refused on-host-only while vision was auto/off-tier. Explicitly routing vision to the scripted loopback server made check pass and on-host-only lock succeed. A three-call real-AIAgent conversation then allowed a local printf, blocked web_search because no human approval was available, and cleanly reported a nonexistent file. No vendor credential was supplied: real Tinfoil/NEAR attestation, encrypted inference and live re-verification were not tested.",
      "tests": "Focused suite: 325 passed, 2 failed in 179.97s across model/tool gates, registration/preflight, NEAR encryption, offline replay and lock generation. Both failures are registry-baseline assertions for absent Yuanbao yb_* tools from older Hermes 84a0301, not a failed live lock/call. Initial full-suite attempt hit a 900s limit after 663 passes, 25 failures, 9 skips and 4 xfails; this began before declared vendor dependencies were installed. A dependency-complete full rerun was stopped after 747 passes and 6 xfails; it was not a full pass. Final planned conversation completed all 3 calls, including expected unattended web-search refusal.",
      "useful": 3,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "rank": 5,
      "best": "Verified provider transports plus route preflight and per-conversation tool consent give privacy-conscious Hermes users capabilities a prompt cannot enforce.",
      "worst": "The lock leaves shell/code network access uninspected and can only record an auxiliary call that strays mid-turn; many private Hermes imports also increase compatibility risk.",
      "total": 26,
      "blocked": null
    },
    {
      "pr": 150353,
      "name": "fluent",
      "validate": "Passed: supplied empty-plan catalog validation, enable and doctor all exit 0; registration matches 3 hooks, command and context reference; no added model-facing tools.",
      "inside": "Loaded at 182c4301f2437e784e9022f1f38576e1d68c3f5f. In real Hermes, the /fluent handler adopted installed git 2.43.0 (149 commands and 2,895 options), showed a high-confidence git status card and cleanly reported a nonexistent program. A three-call AIAgent plan allowed git status --short, blocked git status --shrot before execution with the --short correction, then allowed the corrected command. Hooks add option validation, risk approvals and output notices; @cli cards and a dashboard are implemented. No destructive command was executed and no interactive approval/UI was tested.",
      "tests": "Passed: 402 Python tests, 140 skipped because the corresponding programs are not installed; no failures. Reviewed adoption/parser/guard/store tests and command corpus. Final planned harness validation, enable, doctor and conversation all succeeded; all 3 terminal calls returned, including the expected invalid-option refusal.",
      "useful": 5,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "rank": 1,
      "best": "Catching fabricated flags against the actual installed binary is a practical improvement to everyday Hermes terminal work, demonstrated live without a model account.",
      "worst": "The guard is intentionally silent for unadopted programs, uncertain parses and remote backends, and a session-specific PATH can differ from the binary it learned.",
      "total": 28,
      "blocked": null
    },
    {
      "pr": 153606,
      "name": "habits",
      "validate": "Passed: isolated empty-plan and final planned catalog validation, enable and doctor all exit 0. Registers 2 tools, 7 hooks, skills, slash/CLI commands and HTML run/gallery pages. Validator warns about disclosed private Hermes imports and pattern-scan cautions; no failed admission checks.",
      "inside": "Loaded at c2e32395e0e3461fe56c7d68b7bc993ee0954277. Five-call real-AIAgent plan discovered/described the tools then ran a small reflex: one real terminal printf and a final templated answer, ending done with zero decision questions. A nonexistent habit returned a clean refusal and available names; a cyclic graph with an undefined variable was rejected with both concrete rule violations and nothing ran. The hooks for learning from history, scheduling and answering before the main model are implemented but were not end-to-end timed or driven against a real decision model. No benchmark speed claim was assumed true.",
      "tests": "Focused graph/runner/shell/guard/dreaming/ledger/replay suite: 1,949 passed in 86.02s. Full suite (4,938 collected) reached 3,084 passes and 2 failures before a 900s limit. Both failures reproduced individually: attachment test paths have this workspace hidden ancestor, which attachments() intentionally rejects; another assertion hard-codes 0.0s but observed 0.2s. These are not clean full-suite passes. Final 5-call Hermes conversation completed normally, with clean refusals for nonexistent habit and cyclic/undefined-variable graph.",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "rank": 3,
      "best": "Turning repeated Hermes actions into checked executable reflexes, with practice, human promotion and a hand-back path, changes how the agent works rather than just what it is told.",
      "worst": "Message-triggered habits can add a decision-model call before ordinary replies, and the ledger/run pages can retain raw arguments or output, including secrets.",
      "total": 28,
      "blocked": null
    }
  ]
}