{ "batch": "deep-08", "setup": "Installed real Hermes from NousResearch/hermes-agent source at fdde27e7dae174cb4ba842589e92508852b7dd9f using uv sync --locked --python 3.14 in test/scratch/hermes; install, validate and doctor CLI commands verified. Used the provided tryit.py and upstream tests/fakes/fake_llm_provider.py, with localhost excluded from the proxy, fresh scratch Hermes homes, empty plans then realistic plans including invalid input. Additional direct real-Hermes lifecycle/platform probes are identified per entry. Pytest/pytest-asyncio/PyYAML and JS test dependencies were installed only in scratch. No live model, X account, Apple Photos library or Desktop/GPU client was supplied; external-service and client behavior is distinguished from local/fake-service evidence. Scores judge the pinned source; reliability caps follow observed admission/load/crash failures. Rank tie-breaks are judgment calls about likely continued installation: general zero-token receipt checking first, coding-diff governance second, outbound cancellation third; the private Desktop creation workflow edges the account-dependent X transport.", "plugins": [ { "pr": 153851, "name": "foreman", "validate": "Passed: tryit catalog validate --install-deps, enable and doctor exited 0; all 16 admission checks passed. Warning: private core import foreman/gate.py:661 tools.terminal_tool.get_session_cwd. Optional LLM override capabilities were correctly withheld noninteractively while the plugin still enabled.", "inside": "Pinned source https://github.com/PedValme/foreman/tree/c15453cb2fc07fa674d6f7b548a553b3ce9e6f5d/plugin. Loaded 2 tools, pre_tool_call/pre_verify hooks, /foreman and a reviewer auxiliary task; plugin tools were deferred behind Hermes tool_call in the empty model offer. All 5 planned calls returned, automatically routed through tool_call: clean fixture CLEAN, skipped-test fixture BLOCKED with file/line/fix, nonexistent path resolved the enclosing repository as CLEAN (surprising bad-input behavior), --no-verify commit was blocked before execution, and review returned COULD NOT RUN honestly because only a generated binary file differed. Repeated the 5 calls with eager schemas and saw foreman_check/foreman_review offered explicitly. Desktop panel and a real second-model critique were not exercised.", "tests": "Actual tests/ suite with git discovery isolated: 380 passed, 3 failed, 1 skipped, plus 37 passing subtests. Two failures truncate SARIF filenames in long-path diagnostics; one 300 KB adversarial shell-input timing bound failed while runtime setup was running. Initial root-wide run also collected the deliberately failing cart demo and inherited-repository assumptions; those are excluded from the corrected unit-suite count. Focused recheck also failed: the quoted-command case took 3.64 seconds against a 3-second limit; the initial broad run had passed that timing test.", "useful": 5, "reliable": 4, "polish": 5, "creative": 4, "beyond": 5, "total": 28, "rank": 2, "best": "It guards the real diff and agent tool execution against skipped tests, weakened checks, secrets and hook bypasses, while reserving exceptions and merging for the person.", "worst": "Long-path SARIF errors lose the filename, adversarial shell parsing exceeded its 3-second test bound, and a private terminal-core import leaves a compatibility seam.", "blocked": null }, { "pr": 149365, "name": "say-do-gate", "validate": "Passed: tryit catalog validate --install-deps, enable and doctor all exited 0; all 15 admission checks passed, no warnings.", "inside": "Pinned source https://github.com/jpaulupshaw/hermes-say-do-gate/tree/03402ba924f9db0dbdcfd807df7cd7f3cd131664. Loaded 3 hooks and no tools, as declared. Empty and 3-call conversations completed. A real unittest command passed, reading the test file succeeded, and reading a missing file returned a structured not_found error. Supplementary real AIAgent run in annotate mode executed one passing and one failing unittest run; the scripted final claim All tests pass received a [say-do] footer identifying the exit-1 receipt. Upstream lifecycle/finalizer integration tests also passed.", "tests": "393 passed (pytest), including real upstream Hermes discovery, lifecycle hooks, finalizer, malformed receipts, session/turn isolation and fail-open cases.", "useful": 5, "reliable": 5, "polish": 4, "creative": 4, "beyond": 5, "total": 28, "rank": 1, "best": "A deterministic receipt ledger can flag unsupported claims across ordinary Hermes work without another model call, using native reply and tool hooks.", "worst": "The three-turn ledger and supported command/claim grammar limit coverage, while default observe mode only nudges on the next turn.", "blocked": null }, { "pr": 153365, "name": "lumina", "validate": "Passed: catalog validate --install-deps exited 0 and all 16 static/registration admission checks passed; doctor exited 0 with 10 tools and 4 hooks but warned that osxphotos, photoscript and Vision were not installed. Enable exited 1: PyObjC build requires macOS / sw_vers.", "inside": "Pinned source https://github.com/samirsawarkar/lumina/tree/aaced5beb7bdb93cf7267060bb81fb49d4762c67. Not enabled in the real Linux Hermes conversation because its declared Apple-only dependencies could not build. The empty plan added no tools; planned library-info/search received unavailable-tool errors and the conversation stopped before invalid-export/expiry could execute. These are unavailable calls, not successful plugin executions. Source and portable tests cover the ten tool handlers, Privacy Check, verified redaction, outbound Leak Guard and Desktop integration; actual Photos/OCR/video/UI behavior was not verified.", "tests": "84 passed, 3 failed, 65 skipped (pytest). Failures were test_no_library_and_no_access, test_errors_never_leak_paths and test_status_no_library: osxphotos was absent on Linux. Library/macOS-dependent cases skipped; no Photos library or Apple Vision was available.", "useful": 3, "reliable": 2, "polish": 5, "creative": 4, "beyond": 5, "total": 24, "rank": 6, "best": "On supported Macs, the code combines photo-library access with verified redacted copies and hooks that can stop raw-ID exports across other tools.", "worst": "Apple-only dependencies prevented enablement here, so the reliability cap is 2 and the central Photos/OCR privacy workflow remains unverified; Leak Guard also explicitly fails open on OCR errors.", "blocked": null }, { "pr": 152063, "name": "undo-send", "validate": "Passed: tryit catalog validate --install-deps, enable and doctor all exited 0; 16 checks passed. Runtime warned that provides_middleware is an unknown manifest field, but the public tool_execution middleware actually registered and operated.", "inside": "Pinned source https://github.com/helenn-szn/undo-send/tree/10d388c951245ceeabc1101bb8ff65e75f4f6737. Loaded tool_execution middleware, pre_gateway_dispatch, /unsend, /sendnow and /undo-send; no new model-facing tools, as declared. The original 3-call plan completed: ordinary terminal read ran, git push --dry-run was held for the default 20 seconds then returned the normal no-destination git error, and a missing file returned structured not_found. Additional real AIAgent run with a 2-second window made 4 calls: read passed immediately, a pending dry-run push was cancelled via the plugin shared Store.cancel API and returned cancelled_by_user without executing, the next was released after its grace period, and missing-file handling remained sound. No external message or push was sent. Actual Desktop clicks and live chat concurrency were not exercised.", "tests": "64 passed (pytest from tests/), covering classification, grace-period release, cancellation/send-now races, chat origin and sender restrictions, redacted summaries and dashboard API.", "useful": 5, "reliable": 4, "polish": 4, "creative": 4, "beyond": 5, "total": 27, "rank": 3, "best": "It adds a real cancellation window across Hermes tool execution, with shared atomic state so the gateway and Desktop can stop an action before it runs.", "worst": "Outbound detection is heuristic: the shell patterns miss forms such as git -C repo push and curl --json without -X, and default holds also delay unattended runs before automatically releasing them.", "blocked": null }, { "pr": 154067, "name": "hermes-design-studio", "validate": "Passed: tryit catalog validate --install-deps, enable and doctor all exited 0; all 16 checks passed, including Desktop SDK and no-core-override checks. Security-scan caution referred to evidence files: exec_string in p1-static-security.json and system_passwd_access text in recorded install reports; these were reviewed as quoted scanner/test artifacts, not executable credential access.", "inside": "Pinned source https://github.com/1lpredest1nato/hermes-design-studio/tree/e317ae8e8594a68e7a41ee45a888048d069a44f0. Loaded 7 tools plus the usage skill, with tools initially deferred behind Hermes tool_call; doctor saw all seven registrations. All 5 planned calls returned in a completed real AIAgent conversation: catalog query found Ordered Dither, imported a real 16x16 PNG into content-addressed project storage, read back the matching source/revision, preview returned an honest bounded no-renderer error, and an invalid project action was rejected by Hermes schema validation before handler invocation. No actual Desktop/WebGL renderer or native Save as was exercised. Backend tests exercise revision conflicts, atomic commands, shader-draft approval and image/bridge validation. An additional 5-call real-Hermes conversation with eager tool schemas also completed and explicitly offered all seven Design Studio tools.", "tests": "355 Python tests passed (8 multiprocessing warnings). Original 55-test JavaScript suite: 54 passed, 1 failed in the upload UI test expecting two chunks after a fixed 60 ms. The unchanged focused test also failed; an isolated diagnostic copy adding a 500 ms wait then passed that entire upload/download test, indicating a timing-sensitive assertion rather than proving a broken upload. Original tests hardcode the author’s desktop dependency path; scratch-installed react/react-dom/esbuild/jsdom plus NODE_PATH supplied those dependencies without changing production source. Real GPU/headless-client scripts were not run.", "useful": 4, "reliable": 4, "polish": 5, "creative": 4, "beyond": 5, "total": 27, "rank": 4, "best": "A substantial native image workspace gives Hermes revision-safe projects, shader effects and human-approved new tools, with multimodal previews and useful metrics for models without vision.", "worst": "Preview and PNG export need a connected Desktop renderer, UI verification is timing-sensitive and tied to an author-specific test path, and user-authored GLSL is explicitly not a GPU sandbox.", "blocked": null }, { "pr": 153861, "name": "summon", "validate": "Passed: tryit catalog validate --install-deps, enable and doctor all exited 0; all 15 checks passed without warnings. Isolation audit correctly identifies the gateway platform adapter as in-process rather than host-isolated.", "inside": "Pinned source https://github.com/gtmcknight/hermes-summon/tree/ccfc8c4163fa736785b4f0cd7322742f39c6d435. Loaded a genuine X platform adapter through Hermes platform_registry, not additional tools. Empty and 2-call scripted-model conversations completed; the generic terminal/missing-file calls were only smoke tests, not X tests. Separately discovered the pinned plugin in real Hermes and made 4 platform calls: check_requirements=false, validate_config(empty)=false, connect() returned false with x_config and the gateway-setup instruction, and send(invalid-target) safely refused because no post had summoned it. Verified it subclasses real gateway.platforms.base. No live X posting, owner account, bot key or trial account was supplied; X request/reply behavior was exercised only by its fake-X unit tests.", "tests": "32 passed (unittest), using real Hermes gateway classes and fake X responses for signing, owner filtering, thread handling, secret/contact suppression, deduplication, long-post fallback and daily budgets. Relay end-to-end runner attempted after installing Wrangler, but its local worker could not start: uv_interface_addresses returned Unknown system error 1; relay assertions therefore were not verified.", "useful": 4, "reliable": 4, "polish": 5, "creative": 3, "beyond": 5, "total": 26, "rank": 5, "best": "It turns the owner’s Hermes into an X reply platform with account-restricted polling, thread context and approval-gated skill learning instead of a standalone stateless bot.", "worst": "The public bot remains dependent on X credentials, pricing and an optional author-run relay, and neither live posting nor the relay end-to-end path could be verified here.", "blocked": null } ] }