{ "batch": "deep-12", "setup": "Installed real NousResearch/hermes-agent source at 20d111eaaaf9561dff69ba581d059ce44fa1eec0 with uv sync --locked --python 3.14 (CPython 3.14.10); verified plugins install/validate/doctor commands. Used supplied tryit.py and Hermes tests/fakes/fake_llm_provider.py, with isolated synthetic Hermes homes, no real model/account/key. Set HERMES_DISABLE_LAZY_INSTALLS=1 after initial runtime-provisioning interference. Plugin tools are deferred behind Hermes tool_call, so added_by_plugin=[] is not evidence of missing tools. Supplemental configured runner only adds documented provider config and records fake-model request messages; plugin code remains at the field.json pins. Counts below are local observations, not entrant claims; no physical Desktop/microphone or paid inference was tested.", "plugins": [ { "pr": 152408, "name": "rich-paste", "validate": "Passed all 16 catalog checks and runtime doctor (one llm_request middleware, no agent tools). Warning: private core imports in the replay benchmark and transcription integration (agent.image_routing, tools.transcription_tools, is_stt_enabled); these are compatibility risks, not admission failures.", "inside": "Loaded. No added tools is expected: this is request middleware plus Desktop composer/backend UI. Three planned terminal calls ran in real AIAgent, exercising the installed middleware and its FastAPI routes: two image blocks were interleaved between their text markers without mutating the original request; fetching a loopback image returned blocked_address; malformed urls returned HTTP 400. Backend probes used FastAPI TestClient, not a running Desktop. Pinned implementation: https://raw.githubusercontent.com/ZhenningLang/rich-paste/1e3ec4bb45ce016aee1ce3e4d07d1d19fcc46e16/interleave.py", "tests": "Local Python suite: 246 passed, 2 failed; failures were fake-DNS checked-address and hotlink-retry socket timeouts (sandbox/proxy interaction possible, unresolved). Local Node 24: npm test directory invocation failed; explicit node --test tests/desktop/*.test.ts with HERMES_AGENT_DIR pointing at real source gave 170 passed, 1 skipped (no built macOS Desktop release). No live macOS clipboard/UI or paid benchmark was run.", "useful": 5, "reliable": 4, "polish": 5, "creative": 3, "beyond": 5, "total": 27, "rank": 2, "best": "Fixes image-to-paragraph association at the actual model-request boundary while adding a finished rich-paste composer workflow.", "worst": "macOS-only clipboard testing and two unresolved network-security test timeouts leave platform and transport gaps despite excellent conversion and middleware coverage.", "blocked": null }, { "pr": 151704, "name": "guild", "validate": "Passed all 16 catalog checks, no warnings; runtime doctor passed (4 tools, 4 hooks). Admission did not detect the secret-packaging issue found by a separate local probe.", "inside": "Loaded four deferred guild tools plus skill-protection/sharing hooks, CLI/slash commands and Desktop RPC UI. Initial four AIAgent calls returned clean not-in-a-guild setup errors. For a configured run, a local bare remote and two synthetic member profiles exercised propose -> human-review API approval -> sync; AIAgent status then showed deploy-preview v1 approved by the other member, sync and review succeeded, and a bad ../bad proposal was blocked by the unattended human-approval gate without hanging. Pinned sharing code: https://raw.githubusercontent.com/Nnadijoshuac/hermes-guild/bf5d32c7b02b1cb3b4a7fe6fdd029f09c3a956d6/core.py ; text classification: https://raw.githubusercontent.com/Nnadijoshuac/hermes-guild/bf5d32c7b02b1cb3b4a7fe6fdd029f09c3a956d6/scrub.py", "tests": "Entrant suite passed locally using documented cd tests && python -m pytest -q: 10 passed, covering two-member sharing/review/revert, scrubbing and guards. Initial run from repository root had 8 failures/2 passes; documented import configuration resolved those. Additional local security probe at the same pin confirmed that .env is classified nontext and its synthetic API_KEY remained byte-for-byte in the remote proposal, while notes.md was redacted; a linked.txt symlink also copied an outside-skill fixture into that proposal. All probes used owned temporary files and a local bare remote.", "useful": 4, "reliable": 1, "polish": 5, "creative": 4, "beyond": 5, "total": 24, "rank": 6, "best": "A reviewed, versioned team skill library integrates deeply with Hermes and worked through two local member profiles and the real agent tool loop.", "worst": "Sharing can push .env secrets unchanged and follow symlinks outside the skill, undermining the promise that proposed files are scrubbed before they leave the machine.", "blocked": null }, { "pr": 149750, "name": "neural-mesh", "validate": "Passed all 15 catalog checks; caution only for rm commands in recorded demo evidence. Runtime doctor FAILED: unknown on_pre_compress and on_memory_write hooks, plus declared hooks not registered. This did not prevent the documented memory-provider loading path from working.", "inside": "Empty-plan install enabled successfully but added no tools until memory.provider was explicitly set to neural-mesh as documented. With that setting, real AIAgent exposed mesh_remember, mesh_recall, mesh_asof and mesh_stats. Four calls stored a 30-day rotation fact, recalled it, rejected an invalid as_of date and reported the pure-Python store. A second three-call conversation, with old/new facts prepared through the pinned provider API, returned current=30 days and March 2026=90 days, then cleanly rejected another bad date. Initial unconfigured mesh tool calls were unknown-tool errors, not plugin crashes. Pinned implementation: https://raw.githubusercontent.com/D0xedDevi0/hermes-neural-mesh/26abb610f3cbdd4215fbb898d4a2478317dfc5e5/__init__.py", "tests": "Passed locally: python -m pytest tests -q, 23 passed. Covers temporal supersession, precision floor, prefetch, configuration, thread ownership and provider lifecycle; the broader vendored engine has no separately shipped test suite here.", "useful": 5, "reliable": 3, "polish": 4, "creative": 4, "beyond": 5, "total": 26, "rank": 3, "best": "Real Hermes retrieved different current and historical truths from the same local store, with no external service or embedding account.", "worst": "The manifest misdeclares provider lifecycle names as runtime hooks and fails doctor, while its durable recalled-memory store is plaintext sensitive data.", "blocked": null }, { "pr": 149738, "name": "trust-ladder", "validate": "Passed: all 15 catalog checks, no warnings; runtime doctor passed (2 tools, 5 hooks).", "inside": "Loaded. Adds trust_ladder_learn/task, correction capture, persistent pre-tool guards, post-write checks, per-turn reminders and /teach /trust /verify. Five scripted calls inside AIAgent: learned an echo-forbidden guard at rung 1; terminal echo forbidden was blocked before execution; created smoke task; verifier printf verified passed with exit 0; empty correction returned empty rule without crashing. Tools worked through tool_call despite the empty added_by_plugin list. Pinned implementation: https://raw.githubusercontent.com/merdoks/trust-ladder/fae4e9df58767e5e3832c8f65936b97868bc98fc/ladder.py", "tests": "Passed locally: python -m pytest -q from tests/: 96 passed. Includes compiler, runtime, reliability and statechart regression replays; TLA+ proofs were read, not rerun.", "useful": 5, "reliable": 5, "polish": 4, "creative": 4, "beyond": 5, "total": 28, "rank": 1, "best": "Turns user corrections into persistent, example-checked enforcement at Hermes tool boundaries, and the guard plus verifier both worked in the real agent loop.", "worst": "Pattern guards can be bypassed by scripts or aliases and fail open on hook errors; the README install command still uses an owner placeholder and there is no Desktop management pane.", "blocked": null }, { "pr": 151371, "name": "smart-dictation", "validate": "Passed all 16 catalog checks, no warnings, including declared transcription provider and Python dependency admission; runtime doctor passed.", "inside": "Loaded smart_dictation transcription provider and smart_dictation_correction auxiliary task, with no agent tools by design; Desktop Native/Contextual UI is shipped. With stt.provider set in an isolated home, three planned terminal probes inside real AIAgent resolved the registered provider, passed a short generated WAV and passed malformed audio. Provider registration/availability succeeded; the valid WAV returned stt_unavailable because no model weights were cached, and malformed audio returned transcription_failed with empty transcript, without a crash or download. These probes called the real registered provider in terminal child processes, not a physical Desktop microphone or a live correction service. Pinned provider: https://raw.githubusercontent.com/metamindedu/hermes-smart-dictation/e62b98787d4d056b8d7c333e05a80c6779f9ba2a/native_provider.py", "tests": "Passed after declared Python dependencies were installed: 88 Python tests. Initial run without numpy/av had 3 duration-boundary failures and 85 passes, resolved by installing the pinned dependencies. npm test initially passed 87 with 6 skipped; with real Hermes source and its required Desktop test dependencies, all 93 Node tests passed with no skips, including native composer, query client and launcher contracts. UI tests still use simulated media/provider primitives; no successful physical-speech decode or paid correction was tested.", "useful": 4, "reliable": 4, "polish": 5, "creative": 3, "beyond": 5, "total": 26, "rank": 4, "best": "Integrates local dictation into Hermes’s native speech provider with unusually thorough consent, profile ownership, stale-draft protection and no-tool correction tests.", "worst": "Users must provision cached model weights and opt into profile-wide STT/correction settings, and a successful real speech-to-corrected-draft path remains unverified here.", "blocked": null }, { "pr": 153524, "name": "shadow-army", "validate": "Passed all 16 catalog checks, no warnings; runtime doctor passed (2 tools, 6 hooks and delegation-provider registration).", "inside": "Loaded army_gate/army_verdict, six runtime hooks, a delegation backend, verify-loop skill and Desktop gate/trial UI. Three direct AIAgent calls cleanly rejected a nonexistent card and unbound PASS/FAIL verdicts. A second three-call AIAgent plan used terminal child probes with the real Hermes manager/dispatcher: created a native Kanban card in an owned committed Git fixture, read UNVERIFIED/No receipt for this tree, confirmed kanban_complete was blocked, and rejected a forged unbound PASS. No live independent model-family review was possible with the scripted fake model; host tests use synthetic route identities. Pinned enforcement: https://raw.githubusercontent.com/SilentKnight87/shadow-army/aa340f06ffe10dce5c63cfabc0e0c6de3e9d1e41/shadow-army/service.py", "tests": "Local unittest discovery: 157 tests, 154 passed and 3 failed after setting GIT_CEILING_DIRECTORIES to prevent temporary nonrepositories from discovering the enclosing checkout. Failures: setup-row count, compatibility-warning expectation, and the nested real-Hermes host suite. Nested host suite: 30 tests, 28 passed/2 failed, specifically nonlocal-terminal refusal and effective terminal-scope handling. The main host proof did run a real terminal check, accept PASS, close a real Kanban card, then make edited code STALE and refuse close; route/lifecycle events were synthetic. One Node pane test passed, plus both JavaScript syntax checks. Initial discovery without Git ceiling had 6 outer failures; final count above isolates remaining defects.", "useful": 4, "reliable": 2, "polish": 5, "creative": 4, "beyond": 5, "total": 25, "rank": 5, "best": "Enforces code-bound independent review at the completion boundary, with stale-receipt detection, assertion-removal evidence and a useful account-free trial.", "worst": "Its copied Hermes context-variable names break terminal-scope safety checks on the tested host, and two model families plus an extra review impose substantial setup and inference cost.", "blocked": null } ] }