{
  "batch": "deep-21",
  "setup": "Installed real NousResearch/hermes-agent source at fdde27e7dae174cb4ba842589e92508852b7dd9f with uv sync --locked --python 3.14 (CPython 3.14.10); plugins --help exposed install, validate and doctor. Used the supplied RUN.md/tryit.py and Nous tests/fakes/fake_llm_provider.py: empty plan first, then scripted AIAgent conversations for all six pinned plugins. Added pytest and pytest-asyncio only for testing. Separate scratch Hermes homes, no real model, account, credential or external delivery. Supplemental copy of tryit.py selected memory.provider=samskara with embedded://seed/demo.ttl and extraction disabled, and set Say-Do Gate to annotate with a deliberately unsupported final claim. Hermes deferred ordinary plugin tools behind tool_search/tool_call, so added_by_plugin=[] in the initial model offer does not mean registration failed; doctor and actual dispatch confirmed them. Desktop code and Node suites were inspected/exercised, but no graphical Desktop or live GitHub/SMTP/SPARQL account was exercised. Runtime observations below are local test evidence, not entrant claims; public sources pin the inspected implementation. Ranking breaks close totals by expected everyday retention across Hermes users.",
  "plugins": [
    {
      "pr": 149611,
      "name": "hermes-security",
      "validate": "Passed: hermes plugins validate --install-deps --json, enable --no-allow-tool-override and doctor --ci all exited 0. Pinned https://github.com/NousResearch/hermes-official-plugins@f211940dd81f32b85ec26e2e07de8aa5e24df9e0/hermes-security. Manifest, dependency, isolated registration, declarations, collision, security and core-override checks passed; Desktop surface passed where applicable.",
      "inside": "Loaded; doctor registered nine scan lifecycle tools and three hooks, with Security Desktop page, skills and CLI/slash integration in code. Three scripted calls: security_scan_start on a scratch plugin repository with dependency_check=false created a snapshot-bound scan and review packets; security_scan_get for an unknown ID returned not_found; start on a nonexistent directory returned target_error without crashing. This verifies planning/storage and errors, not model vulnerability discovery, a completed report, dependency downloads or active validation.",
      "tests": "Python suite: 654 passed, 1 failed, 4 skipped, 1 xfailed. The failing test_process_writers hit sandbox PermissionError creating a multiprocessing forkserver AF_UNIX socket; not evidence of a store assertion failure. Four skipped scenarios and expected failure remain unverified. No browser E2E run. Reviewed report, policy, citation and hostile-repository tests; EVALUATION.md expressly distinguishes offline boundaries from model quality and records failing development model gates.",
      "useful": 5,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 2,
      "best": "A substantial native security-review system preserves source evidence, coverage gaps, validation authorization and resumable reports instead of merely supplying an audit prompt.",
      "worst": "A successful scan start does not establish detection quality: the pinned evaluation records failed development discovery/agreement gates and unrecoverable historical bundles, while full model-led audits were not tested here.",
      "blocked": null
    },
    {
      "pr": 153144,
      "name": "signet",
      "validate": "Passed: hermes plugins validate --install-deps --json, enable --no-allow-tool-override and doctor --ci all exited 0. Pinned https://github.com/Wecko-ai/hermes-signet@ceaab2a705907392711658af5707f2b27bd405d8. Manifest, dependency, isolated registration, declarations, collision, security and core-override checks passed; Desktop surface passed where applicable.",
      "inside": "Loaded two deferred tools, three approval/dispatch hooks and /signet. Four scripted calls: valid signet_send_mail was blocked without an interactive approver and preserved as pending; signet_outbox showed that draft and its reason; CRLF recipient injection returned refused; terminal himalaya message send was blocked by the hook before execution. Its own real-Hermes integration additionally tested an approved .eml matching recipients/subject/body, denial, YOLO/off holding, explicit two-step slash sending and a fresh prompt after always-allow. No mail was delivered externally.",
      "tests": "108 Python tests plus 140 subtests passed. tests/hermes_integration.py passed all 7/7 scenarios against real Hermes, including approval once, denial, YOLO, approvals off, direct-mailer guard, missing interactive user and one-showing always-allow. SMTP and command delivery to a real service were not exercised.",
      "useful": 4,
      "reliable": 5,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 4,
      "best": "Single-use approval seals bind outgoing mail to the exact canonical draft and attachments, with tested fail-closed behavior even when Hermes bypasses ordinary approvals.",
      "worst": "Its explicitly heuristic direct-mailer guard is not a sandbox, and owner-listed trusted recipients can bypass prompting; retention is strongest for users who already delegate email.",
      "blocked": null
    },
    {
      "pr": 149365,
      "name": "say-do-gate",
      "validate": "Passed: hermes plugins validate --install-deps --json, enable --no-allow-tool-override and doctor --ci all exited 0. Pinned https://github.com/jpaulupshaw/hermes-say-do-gate@03402ba924f9db0dbdcfd807df7cd7f3cd131664. Manifest, dependency, isolated registration, declarations, collision, security and core-override checks passed; Desktop surface passed where applicable.",
      "inside": "Loaded all three hooks (post_tool_call, pre_llm_call and transform_llm_output), no tools or service. Three planned existing-tool calls returned a successful printf, exit 1 and a clean missing-file error. In a supplemental real AIAgent conversation with mode=annotate, the scripted final All tests pass. became All tests pass. followed by [say-do] Claimed: tests pass. Ledger: 0 test commands run.; printing 3 passed did not count as running a test runner. Default observe mode preserved Done. unchanged. Tests also exercised genuine Hermes lifecycle hooks/finalizer, contradicted claims and prior-turn receipts.",
      "tests": "393/393 pytest tests passed after setting documented SAYDO_UPSTREAM to the installed scratch Hermes source. Initial run was 392 passed/1 failed solely because the isolation guard expected a sibling upstream path; no implementation changes. Includes real upstream discovery, held/unheld finalizer, claim exclusions, ledger, failure-open behavior and bounded session state.",
      "useful": 5,
      "reliable": 5,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 1,
      "best": "A tiny dependency-free runtime extension catches unsupported completion claims against actual tool receipts for essentially every Hermes user, and its real reply transformation worked.",
      "worst": "It recognizes a bounded deterministic claim/command vocabulary, checks only final replies, and default observe mode defers correction until a later turn while annotate sacrifices streaming.",
      "blocked": null
    },
    {
      "pr": 152899,
      "name": "project-command-center",
      "validate": "Passed: hermes plugins validate --install-deps --json, enable --no-allow-tool-override and doctor --ci all exited 0. Pinned https://github.com/NyllRE/project-command-center@eaa3e2d3a046eb677f4c67aae98b78f1dcec4043. Manifest, dependency, isolated registration, declarations, collision, security and core-override checks passed; Desktop surface passed where applicable.",
      "inside": "Loaded three deferred tools, two hooks and the Desktop board/backend. Five scripted calls: pcc_board read an empty board; pcc_set_status created an active local evaluation project; another board read returned the persisted lane/note; invalid status nonsense was rejected by Hermes schema validation without invocation; pcc_work_item attached issue 7 to the actual test session. Separate hook tests checked context injection and gh pr create receipts. No authenticated GitHub writes or graphical board were attempted.",
      "tests": "Pytest collected no tests because the Python suites are script harnesses. Ran all eight documented Python scripts separately: 8/8 exited 0, covering hooks, session lanes, git routes, branch switching, worktrees, icons, avatars and GitHub state contracts; optional developer-local real repositories were skipped. Node tests: 50 passed, 6 failed; all six require the hardcoded developer Hermes checkout with real merge history, unavailable here. Synthetic graph fixtures and other UI contract suites passed; this is not an all-green Node run.",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 3,
      "beyond": 5,
      "total": 26,
      "rank": 5,
      "best": "A polished native Desktop workbench connects projects, sessions, GitHub work and per-turn agent context through one shared local board with conservative repository operations.",
      "worst": "Its value concentrates on Desktop/GitHub development workflows, live authenticated behavior was not verified, and six graph tests depend on an unavailable hardcoded real checkout.",
      "blocked": null
    },
    {
      "pr": 152421,
      "name": "sheets",
      "validate": "Passed: hermes plugins validate --install-deps --json, enable --no-allow-tool-override and doctor --ci all exited 0. Pinned https://github.com/Adolanium/hermes-sheets@65888b7c6a0304d61ac5f1d060a5d4bd95ee73eb. Manifest, dependency, isolated registration, declarations, collision, security and core-override checks passed; Desktop surface passed where applicable.",
      "inside": "Loaded four deferred tools, the grids skill and a substantial Desktop spreadsheet/backend. Final five-call plan: sheets_list gave actionable no-grid/setup guidance; sheets_create made a one-row Evaluation grid; sheets_read returned the row and untrusted-data notice; sheets_propose queued a prompt column and explicitly reported nothing applied; a read of an unknown column returned a useful error. Earlier malformed schema inputs were cleanly rejected. A fresh home allowed create/read before the sharing setting existed: code blocks only known-and-off sharing, a narrower consent guarantee than the README suggests. Desktop rendering, paid model fills and real research were not exercised.",
      "tests": "14/14 Python tests passed; 161/161 Node workspace tests passed, including typed values, stale proposals, review/restore, chat sharing, model-session workflow contracts, imports, formula precedence/dependency cycles, totals and Excel export. The Node harness uses mocked host requests and does not demonstrate a live graphical Desktop or real model research.",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 27,
      "rank": 3,
      "best": "A product-like agentic spreadsheet makes research and classification inspectable through typed cells, sources, proposals, review receipts and undo inside Hermes Desktop.",
      "worst": "Missing sharing.json is treated as permission to create and read chat grids despite the starts-off promise, and a stale README paragraph still denies formulas that the tested implementation supports.",
      "blocked": null
    },
    {
      "pr": 149823,
      "name": "samskara",
      "validate": "Passed: hermes plugins validate --install-deps --json, enable --no-allow-tool-override and doctor --ci all exited 0. Pinned https://github.com/SebaWag/hermes-samskara@d807e875da3176301db9955e0352c9c8adb6cb01. Manifest, dependency, isolated registration, declarations, collision, security and core-override checks passed; Desktop surface passed where applicable.",
      "inside": "Loaded as an exclusive memory provider, not ordinary registered tools/hooks. Initial default tryit conversation offered no samskara tools until memory.provider was selected; that is configuration, not a load failure. With the documented provider selected and embedded demo corpus, five tools appeared and all five planned calls returned: status healthy, PostgreSQL search produced multi-entity provenance-bearing relationships, DELETE query was rejected as only SELECT/ASK allowed, explicit write appended one journal triple pending reindex, and conflicts reported none. Provider prefetch had a hit. No external graph service, full reindex, live extraction or cross-session persistence-to-recall cycle was demonstrated.",
      "tests": "150/150 offline pytest tests passed, including graph failure modes, trust boundaries, ontology cycles, conflicts, weights, journal, extraction/reconciliation and embedded query handling. The live graph integration script was not run because no SPARQL service was configured. Inspected host-LLM adapter: unavailable ctx.llm skips extraction rather than borrowing another provider credential.",
      "useful": 4,
      "reliable": 4,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "rank": 6,
      "best": "A real Hermes memory-provider extension adds graph-path recall, provenance, ontology inference and explicit contradictory facts instead of another similarity-search prompt.",
      "worst": "Embedded recall is deliberately read-only, journaled facts need external reindexing to become graph recall, and automatic semantic extraction is unavailable on the documented host path without ctx.llm.",
      "blocked": null
    }
  ]
}