{
  "batch": "deep-x-07",
  "setup": "Installed real NousResearch/hermes-agent source at a84025ade22d2a4085f575d4353072e5b702635d with uv sync --locked --python 3.14 (Python 3.14.4). CLI exposes install, validate and doctor. Used RUN.md's tryit.py first with empty plans, then 2-4 scripted calls per readable plugin; an instrumented copy additionally retained FakeLLMServer request bodies, without changing plugin code. Each run used a fresh isolated Hermes home. Installed pytest, pytest-asyncio and aiohttp in the scratch virtualenv for entrant tests; no model account or paid service used. All code, clones and runtime files stayed in test/scratch. Scores judge these exact plugin pins, not current upstream branches; test observations below are local execution evidence. Field/pins: https://gist.githubusercontent.com/gtmcknight/594c47336546c8763e641b46cf9af4ec/raw/field.json ; runner: https://gist.githubusercontent.com/gtmcknight/594c47336546c8763e641b46cf9af4ec/raw/RUN.md . No graphical Hermes Desktop, telco account, whisper.cpp engine or speech models were available, so those end-to-end paths are explicitly unverified.",
  "plugins": [
    {
      "pr": 149968,
      "name": "lahja",
      "validate": "Passed: catalog validation exit 0, all 15 checks true, no warnings; enable and doctor exit 0.",
      "inside": "Loaded in real Hermes: registers the lahja transcription provider, /stt-fix and the lahja CLI; no model-facing tools/hooks expected. Three planned terminal calls inside AIAgent: lahja status reported missing whisper-cli, ffmpeg and model assets; lahja models successfully listed the two Whisper models and Silero VAD; lahja test with a nonexistent audio filename returned a clean missing-whisper-cli setup error, without an exception. Real speech decoding and the published WER improvement were not reproduced. Pin: a7a43b1389a1ae169ff5c7401e9ec2ec9a53b51d; folder lahja. Code: https://raw.githubusercontent.com/Tarekkharsa/hermes-lahja/a7a43b1389a1ae169ff5c7401e9ec2ec9a53b51d/lahja/__init__.py",
      "tests": "99 passed, 6 skipped; skipped tests require real whisper.cpp, model, VAD and audio fixtures. Provider/routing, transcript filtering, corrections, manifest and security/error regressions ran. Test source: https://raw.githubusercontent.com/Tarekkharsa/hermes-lahja/a7a43b1389a1ae169ff5c7401e9ec2ec9a53b51d/tests/test_real_engine.py",
      "useful": 3,
      "reliable": 4,
      "polish": 5,
      "creative": 3,
      "beyond": 5,
      "rank": 5,
      "best": "A genuine local Hermes STT provider preserves Arabic dialect speech and adds persistent taught corrections, with careful download verification and bounded subprocess handling.",
      "worst": "The main speech path could not run without whisper.cpp and large model downloads, and the dialect improvement remains an entrant benchmark rather than an independently reproduced result.",
      "blocked": null,
      "total": 25
    },
    {
      "pr": 153640,
      "name": "patient-research",
      "validate": "Not run: pinned repository could not be fetched.",
      "inside": "Not loaded: git fetch of https://github.com/solo386/patient-research at 8642b35bfc6a169ad20cc2f343f8c6dd34d82681 failed with a GitHub authentication prompt; the pinned raw plugin/plugin.yaml and codeload archive both returned HTTP 404. GitHub API fallback returned 403 rate-limit exceeded. This does not establish whether the repository is private, deleted or renamed. Pin and folder plugin supplied by https://gist.githubusercontent.com/gtmcknight/594c47336546c8763e641b46cf9af4ec/raw/field.json",
      "tests": "Not run: no pinned code could be read; 0 tests executed.",
      "useful": 0,
      "reliable": 0,
      "polish": 0,
      "creative": 0,
      "beyond": 0,
      "rank": 6,
      "best": "No substantiated strength can be assigned without the pinned code.",
      "worst": "The submitted repository and commit were inaccessible through Git, raw-file and archive retrieval.",
      "blocked": "Could not read the pinned patient-research repository: Git fetch required authentication and pinned raw and archive URLs returned HTTP 404.",
      "total": 0
    },
    {
      "pr": 152533,
      "name": "vellum",
      "validate": "Passed: catalog validation exit 0, all 16 checks true including desktop surface and no core override; enable and doctor exit 0.",
      "inside": "Loaded: Hermes doctor registered four tools (vellum_draw/look/clear/read), five hooks and the paper-canvas skill. Tools are discoverable/deferred rather than added to the initial 19-tool model menu. All four planned direct tool requests ran through Hermes; read was refused outside its owning paper chat, and draw/look/clear were refused outside a user-opened paper turn, including a malformed canvas id. These were controlled hook refusals, not crashes. No real Desktop paper, rendered drawing or synchronized TTS was exercised. Pin: 286b44d587500bd4792decdc73370791262aec0b; repository root. Code: https://raw.githubusercontent.com/0xsaltylollipop/vellum/286b44d587500bd4792decdc73370791262aec0b/paper_anim/canvas.py",
      "tests": "73 passed, 0 failed, 0 skipped. Covers card ownership, fake draft rejection, cross-chat/cron/subagent isolation, bounded voice work, paper chunks, safe relay writes, iframe/CSP and backend route checks; browser rendering itself is not covered by this run. https://raw.githubusercontent.com/0xsaltylollipop/vellum/286b44d587500bd4792decdc73370791262aec0b/tests/test_vellum.py",
      "useful": 3,
      "reliable": 4,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "rank": 2,
      "best": "An unusually original extension gives the user's own Hermes an interactive figure inside the paper, with per-chat ownership and voice/drawing coordination that a skill cannot supply.",
      "worst": "Its valuable path depends on Hermes Desktop, arXiv HTML and working voice services, while this run could verify only headless guards and tests, not the live reading experience.",
      "blocked": null,
      "total": 27
    },
    {
      "pr": 153524,
      "name": "shadow-army",
      "validate": "Passed: catalog validation exit 0, all 16 checks true including desktop surface and no core override; enable and doctor exit 0.",
      "inside": "Loaded: registers army_gate, army_verdict, six hooks, a delegation provider, context and verify-loop skill; doctor confirms two tools and six hooks. Three planned tool calls: missing-card gate query returned an actionable error; an unbound PASS was refused; INVALID verdict was rejected by Hermes argument validation before invocation. No call crashed. Nested real-Hermes host tests demonstrated observed terminal-check evidence, PASS-bound close, edit-to-STALE refusal and publish approval hooks, using synthetic route/child observations rather than actual two-provider inference. Pin: aa340f06ffe10dce5c63cfabc0e0c6de3e9d1e41; folder shadow-army. Code: https://raw.githubusercontent.com/SilentKnight87/shadow-army/aa340f06ffe10dce5c63cfabc0e0c6de3e9d1e41/shadow-army/bridge.py",
      "tests": "Failed: isolated rerun executed 157 outer unittest tests, 154 passed and 3 failed; the failed integration wrapper separately ran 30 host-probe tests, 28 passed and 2 failed. Remaining failures concern setup/context compatibility (extra setup row and missing expected context names) and terminal backend scope/nonlocal refusal; initial ancestor-Git-related failures disappeared after setting a Git discovery ceiling. The probe also warns about its short synthetic dashboard token. No actual provider hop tested. Sources: https://raw.githubusercontent.com/SilentKnight87/shadow-army/aa340f06ffe10dce5c63cfabc0e0c6de3e9d1e41/host_probe.py and https://raw.githubusercontent.com/SilentKnight87/shadow-army/aa340f06ffe10dce5c63cfabc0e0c6de3e9d1e41/shadow-army/setup.py",
      "useful": 4,
      "reliable": 3,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "rank": 4,
      "best": "Code-bound independent verification can actually refuse a premature Kanban close and require approval before an unreviewed push, with a polished trial and ledger.",
      "worst": "Compatibility and terminal-scope regressions remain on the tested Hermes source, and its documented publish guard deliberately fails open on parsing errors or unsupported shell forms.",
      "blocked": null,
      "total": 26
    },
    {
      "pr": 152850,
      "name": "ussd-platform",
      "validate": "Passed: catalog validation exit 0, all 15 checks true, no warnings; declared aiohttp dependency installed; enable and doctor exit 0.",
      "inside": "Loaded as Hermes's in-process ussd gateway platform, with no model-facing tools or hooks. After an empty-plan run and CLI inspection, three planned terminal calls inside AIAgent exercised the installed adapter obtained from Hermes's real platform registry: a local simulator HTTP dial/message round trip produced the expected CON reply; a malformed phone callback returned HTTP 400 with zero agent dispatches; missing webhook secret refused startup cleanly. The round-trip response handler was a local echo, not live inference; no telco/SMS service was contacted. Pin: f5f69e215dad54c06d2977320a080ab1c68f10ca; repository root. Code: https://raw.githubusercontent.com/GodlyDeveloper1/hermes/f5f69e215dad54c06d2977320a080ab1c68f10ca/adapter.py",
      "tests": "23 passed, 0 failed, 0 skipped after installing declared aiohttp in the shared test environment; includes 17 engine tests and 6 live adapter tests. HTTP callback authentication, allowlists, weak-secret refusal, local simulator isolation, real Hermes approval settlement, paging, PIN lockout and late-answer behavior ran. Actual Africa's Talking SMS/telco delivery untested. https://raw.githubusercontent.com/GodlyDeveloper1/hermes/f5f69e215dad54c06d2977320a080ab1c68f10ca/tests/test_adapter_live.py",
      "useful": 3,
      "reliable": 5,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "rank": 1,
      "best": "A complete gateway turns an ordinary keypad phone into access to the user's own Hermes, including paged replies and real human approval menus, rather than merely wrapping an API.",
      "worst": "Real handset access requires an aggregator and short-code setup, USSD/SMS expose conversation text to the operator, and in-memory inboxes disappear on restart.",
      "blocked": null,
      "total": 28
    },
    {
      "pr": 152764,
      "name": "redactgate",
      "validate": "Passed: catalog validation exit 0, all 15 checks true, no warnings; enable and doctor exit 0 (runtime emitted a nonfatal unknown provides_middleware manifest-field warning).",
      "inside": "Loaded: two middleware registrations, five hooks and /redact, with no added tools. Planned write_file, read_file and nonexistent-file read all completed inside Hermes; the bad path produced a structured not-found error. In all four recorded main-model requests the sample name/email were replaced with placeholders, while the local file contained the original values; read-back tool output was redacted again. This first conversation used literal sample values in scripted write arguments, so placeholder-to-tool restoration is established by the separate real-PluginManager integration test, not that write alone. Pin: d4bb87d78381d0514cd8dd74525fea98a5b46d62; repository root. Code: https://raw.githubusercontent.com/ctmakc/hermes-redactgate/d4bb87d78381d0514cd8dd74525fea98a5b46d62/vault.py",
      "tests": "63 passed, 0 failed, 0 skipped, including 4 real-Hermes integration tests for middleware, stream restoration, slash command and hardening. Additional local stress probe of SessionVault.token found a collision at 7,908 distinct EMAIL values with a fixed synthetic test key: two addresses shared one token and restore returned the first for the second; this is not an entrant-test failure count. https://raw.githubusercontent.com/ctmakc/hermes-redactgate/d4bb87d78381d0514cd8dd74525fea98a5b46d62/tests/test_hermes_integration.py",
      "useful": 5,
      "reliable": 3,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "rank": 3,
      "best": "Automatic redaction of main-model history and tool output, paired with local-tool restoration, adds broadly useful privacy at a boundary that a prompt cannot enforce.",
      "worst": "Six-hex placeholders have no collision disambiguation and reproduced wrong-email restoration, while auxiliary model calls remain outside rewriting and require separate hardening or local routing.",
      "blocked": null,
      "total": 26
    }
  ]
}