{
  "batch": "deep-05",
  "setup": "Hermes source fdde27e7dae174cb4ba842589e92508852b7dd9f, installed once with uv sync --locked --python 3.14 (Python 3.14.10); CLI exposes install/validate/doctor. Used supplied RUN.md and tryit.py with Nous FakeLLMServer. Localhost bypass was needed for the scripted model. Admission checks additionally captured through the unmodified validate_plugin_dir API because tryit truncates stdout. Clean tryit empty-plan reruns passed validation, doctor and conversation for all six; CLI enable for clearvoice/hallway-test exceeded its 120-second dependency-preparation budget. Supplemental real AIAgent runs used separate scratch profiles, in_process loading and the same fake provider; Lethe was enabled through its registered slash-command handler. Installed test/browser dependencies in scratch. Desktop app, microphone, real speech models and production messaging accounts were not available. Tests and runtime statements below are local observations, not entrant claims; pinned source URLs are recorded in sources.json. Ranks compare only these six round-3 entries.",
  "plugins": [
    {
      "pr": 152908,
      "name": "contract-necromancer",
      "validate": "Passed catalog admission: manifest, declared registrations, dependency metadata, security scan and no core overrides; doctor passed with two tools and four hooks.",
      "inside": "Loaded necro_status, necro_check, /necro and verification/output/file guards. Initial calls correctly reported an unfrozen project. In a supplemental real Hermes session, the registered user command froze four bundled acceptance contracts; five planned calls found two failing checks, blocked rewriting a frozen test, read the source, wrote the genuine fix and then passed all four checks with a receipt. A separate unsuccessful conversation was stamped Not done rather than accepting Done. Independent standalone receipt verification with --run returned CONFIRMED. Pin: https://github.com/vijaygopalbalasa/hermes-necromancer/tree/0ff7bec40aba10091a84f94025d4471984fd7386.",
      "tests": "385 bundled tests under Python 3.12: 381 passed, 3 failed, 1 skipped. Two exact-metadata assertions encounter an extra ignored=true flag because scratch is ignored by the ancestor Git repository; the third measured a 5.71s guard call against a 5s threshold and passed alone in 1.94s. Earlier Hermes-Python run hit a 900s limit after 172 passes/2 failures. Real-Hermes frozen-project probes and independent receipt verification passed.",
      "useful": 5,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 1,
      "best": "Independent test execution, human-frozen contracts and recheckable receipts make the user’s own coding agent more trustworthy in a way a prompt cannot enforce.",
      "worst": "It is a guardrail rather than a sandbox: same-user processes can evade the early guard, and the documented shell-only path stamps failure without keeping the agent working.",
      "blocked": null
    },
    {
      "pr": 153302,
      "name": "lethe",
      "validate": "Passed: manifest, hook registration, security scan, core override and Desktop SDK checks; no warnings; isolation audit identifies a need for in_process execution.",
      "inside": "Loaded five hooks, six slash commands and no tools. In a real AIAgent session, invoking registered /lethe on blocked memory add, an invalid user-memory action and skill creation; /lethe-where counted three blocked writes. Unknown command option returned help. /lethe-end deleted the actual SessionDB row, verified absent immediately afterward. A second five-call real-Hermes probe confirmed write_file can save directly to memories/USER.md while Lethe is on; that file remained after /lethe-end deleted the transcript. Desktop/TUI views were reviewed, not launched. Pin: https://github.com/itsflownium/hermes-lethe/tree/8b93e2dce9402648c7e238c1d8c83dc649f2f204.",
      "tests": "67 passed (including parameterized cases); no failures. Tests cover guards, state, commands, session deletion, arm window and Desktop-facing API behavior; Desktop JavaScript syntax passed. The additional direct-write probe exposed a gap outside that suite.",
      "useful": 5,
      "reliable": 2,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "rank": 4,
      "best": "A per-chat switch that actually vetoes built-in memory writes and erases the transcript is a broadly useful change to Hermes itself.",
      "worst": "Direct file writes bypass the memory veto and survive transcript deletion, while the status view still says nothing saved; external provider sync is also explicitly excluded.",
      "blocked": null
    },
    {
      "pr": 152533,
      "name": "vellum",
      "validate": "Passed: manifest, capability registration, security scan, core override and Desktop SDK checks; no warnings.",
      "inside": "Loaded: four deferred tools, five hooks and paper-canvas skill; no extra eagerly offered tools. All four planned calls reached Hermes: read, draw, look with a traversal-like canvas id, and clear were safely refused outside an authorized paper turn. A supplemental paper-turn probe registered synthetic cached text/card/state through the actual backend API: vellum_read returned it, a wrong-canvas look was refused, and draw returned a bounded no-Desktop-answer error after 30 seconds. Full Desktop rendering/audio synchronization was not exercised. Pin: https://github.com/0xsaltylollipop/vellum/tree/286b44d587500bd4792decdc73370791262aec0b.",
      "tests": "73 collected: 72 passed, 1 failed (streamed-text performance must finish below 2 seconds); isolated rerun also failed. Seven additional headless Chromium checks passed against the unmodified sandbox runtime: D3 drawing, persistent names, network refusal, bounded infinite loop, continued drawing, removed WebRTC and clearing. Desktop JavaScript syntax passed; no live Desktop end-to-end test.",
      "useful": 4,
      "reliable": 3,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "total": 27,
      "rank": 2,
      "best": "A sandboxed, session-bound canvas inside the paper gives Hermes a distinctive research capability that a prompt cannot supply.",
      "worst": "The principal Desktop experience remains unverified here, and the streaming-performance regression test failed twice.",
      "blocked": null
    },
    {
      "pr": 152475,
      "name": "hallway-test",
      "validate": "Passed: admission validator including declared tool, dependency metadata, security scan and Desktop SDK surface.",
      "inside": "Loaded hallway_test, /hallway and an auxiliary-model slot. Three calls ran in real Hermes: file URL and metadata IP were refused; demo:fixed with one persona and four steps wrote an HTML replay and summary card. The generic scripted auxiliary reply supplied no meaningful decisions, so 0/1 finishing is not evidence against the fixed demo. Bundled browser tests separately exercised real actions. Pin: https://github.com/lexipej/hallway-test/tree/7e5381a56f411114e52e0a61d73664142dba7803.",
      "tests": "35 passed after installing Chromium and missing runtime libraries in scratch; initial browser failures were environmental. Includes perception limits, keyboard enforcement, before/after demo, reporting and URL guards; Desktop JavaScript syntax passed.",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 4,
      "total": 25,
      "rank": 5,
      "best": "Code-enforced perception and action limits turn Hermes into a practical accessibility testing lab with replayable evidence.",
      "worst": "Its multi-browser, many-model-call workflow is substantial overhead, and simulated users remain an approximation of disabled people.",
      "blocked": null
    },
    {
      "pr": 149978,
      "name": "clearvoice",
      "validate": "Passed direct catalog admission check, including all three tool registrations, security scan and Desktop SDK surface; audio dependency installation assessed separately.",
      "inside": "Loaded three deferred tools and /clearvoice. All five calls ran: speak queued exact text awaiting the user; empty text, unapproved file output, relative transcription path and missing recording returned clear errors. No audio was played, transcribed or synthesized; speech engines/models were unavailable in the test environment. Pin: https://github.com/thebangarang/clearvoice-hermes-plugin/tree/d4bc366ec72d4f2af1a1772b3a94f48bd0e0b156.",
      "tests": "163 collected: 158 passed, 4 failed, 1 skipped. Three failures expose the missing bundled clearvoice_engine/data/piper_voices.json; the fourth expects relative output-path validation before the absent local synthesis engine. Desktop JavaScript syntax passed.",
      "useful": 3,
      "reliable": 2,
      "polish": 5,
      "creative": 4,
      "beyond": 4,
      "total": 22,
      "rank": 6,
      "best": "User-controlled speech queuing and local processing make a thoughtful accessibility bridge with a substantial Desktop interface.",
      "worst": "The pinned checkout omits its advertised Piper voice index, breaking non-English voice selection despite polished seventeen-language claims.",
      "blocked": null
    },
    {
      "pr": 151411,
      "name": "salts-secrets",
      "validate": "Passed admission validation and clean CLI enable/doctor/conversation; warnings: public-role tools/hooks not registered in owner-role probe, and ssh_dir_access matched a privacy-rule definition, not a demonstrated credential read.",
      "inside": "Owner role loaded salt_propose_public, two hooks and /salt. Three real Hermes calls queued an innocuous note without publishing, refused empty text and refused a password-bearing note. No owners configured produced an explicit all-guests warning. A further five-call real-Hermes public-role run loaded six hooks and public_notes_lookup; it refused terminal, memory and local-image vision access and returned an empty public-document set. Actual messaging gateway routing and revocation were unit-tested/read, not exercised against live accounts. Pin: https://github.com/dotv-7/salts-secrets-plugin/tree/09c1b426b08e41380a47233c4f3cead36a017ecf/salts_secrets.",
      "tests": "378 collected: 372 passed, 5 failed, 1 skipped. Three audit assertions fail conservatively because scratch public directories sit below a Git root. Two classification assertions were caused by mode-600 fixtures triggering owner_only_file; both passed when rerun with ordinary fixture permissions. No failure demonstrated a leak. Owner and public-role Hermes probes passed.",
      "useful": 4,
      "reliable": 4,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "rank": 3,
      "best": "Native profile routing, owner approval and guest tool restrictions provide meaningful isolation for sharing a personally trained agent.",
      "worst": "Setup and revocation span two profiles plus a live gateway, so safe operation depends on completing its isolation setup; the production messaging path was not exercised here.",
      "blocked": null
    }
  ]
}