{
  "batch": "deep-49",
  "setup": "Installed real NousResearch/hermes-agent from source at fdde27e7dae174cb4ba842589e92508852b7dd9f with uv sync --locked --python 3.14, entirely under test/scratch; plugins --help exposed install, validate and doctor. Used the supplied tryit.py and Hermes tests/fakes/fake_llm_provider.py, with no external model credentials (S2). Added pytest, pytest-asyncio and PyYAML for local tests. Each repository was fetched at field.json's exact SHA, root folder (S1); no entrant implementation was changed. First ran an empty plan, then 2-4 calls including bad input for each. Additional runner copy only merged documented configuration before chat to select the memory provider, turn on recording or configure Coach's scripted advisor; commands were invoked through Hermes's registered handlers. Coach's same-model guard was explicitly disabled only for the scripted advisor test. Corrected an initial harness tool_search query/queries argument mistake; it is not scored against entrants. S1-S20 refer, in order, to sources.json; pin URLs identify source evidence, while runtime outcomes and test counts below are direct local observations. Ranks favor broadly useful persistent installations when totals tie; no other seat's scores were consulted.",
  "plugins": [
    {
      "pr": 152596,
      "name": "coach",
      "validate": "Passed: catalog validate exit 0; enable and doctor exit 0, registering 2 tools and 7 hooks. Admission emitted a private-core-import warning for agent.usage_pricing.estimate_usage_cost in coach/advisor.py.",
      "inside": "Pinned https://github.com/Printaga/Hermes-Coach/tree/fb3e1d931dc87e441ad425d2830ea783e901bc85 (root; S3-S5). Loaded. Initial advertised tool list excludes its deferred tools, but corrected tool_search found coach_ask and coach_decide. Unconfigured ask returned a clear no-model error; empty question returned question_required; one-option decision returned options_count, without crashing. With the documented model configuration and override grants, the scripted advisor returned ok=true with advice and usage/cost accounting. /coach status and an unknown-setting error worked. Review hooks and Desktop code were read, but a real second-model quality improvement and Desktop interaction were not established.",
      "tests": "Final: 451 passed, 0 failed. Initial default run: 449 passed, 2 failed; rerunning with multiprocessing start method fork and GIT_CEILING_DIRECTORIES at the scratch temporary root resolved the sandbox AF_UNIX denial and enclosing-repository interference. No entrant source changes.",
      "useful": 4,
      "reliable": 4,
      "polish": 4,
      "creative": 3,
      "beyond": 5,
      "total": 25,
      "rank": 6,
      "best": "Its bounded completion-review gate and automatic feedback hooks make a second opinion part of Hermes execution, beyond an ordinary critic prompt (S3-S4).",
      "worst": "A second model requires explicit setup and extra inference, and the passing catalog check still warns about a private pricing import, leaving maintenance risk.",
      "blocked": null
    },
    {
      "pr": 153867,
      "name": "whos-listening",
      "validate": "Passed: catalog validate ok=true with all checks passed and no warnings; enable and doctor exit 0, 0 ordinary tools and 2 hooks (memory tools come from the selected provider).",
      "inside": "Pinned https://github.com/Web3B-eth/whos-listening/tree/a498dd769c4196d0e911b713d4ad7363c8154ba1 (root; S6-S8). Loaded; enable alone does not select its memory provider. After setting memory.provider=whos-listening and disabling both built-in memories as documented, Hermes offered remember, recall, forget and review. Direct private remember saved P1; recall returned that preference; invalid tier was rejected cleanly. A separate plan after tool_search held the private save for outside-content review and recall withheld the pending item. /memories settings warned when the provider was unselected; owner add and bad-command responses worked. Remote sync, Venice duplicate judging and a real provider-switch conversation were not run.",
      "tests": "Final: 524 passed, 0 failed. An earlier run had 523 passed and 1 snapshot-rewrite failure (test_the_copy_is_rewritten_if_it_changed_on_disk); an unchanged full rerun passed, so the earlier failure remains a flakiness caveat.",
      "useful": 5,
      "reliable": 4,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 27,
      "rank": 4,
      "best": "Route-aware memory storage and request scrubbing directly improve Hermes privacy across models and platforms (S7-S8).",
      "worst": "Its own limits disclose that compression and other auxiliary calls can bypass the guard and carry Private memories, so it is not a complete privacy boundary (S6).",
      "blocked": null
    },
    {
      "pr": 153844,
      "name": "fine-print",
      "validate": "Passed: catalog validate ok=true, all checks passed with no warnings; enable and doctor exit 0, registering 0 tools and 11 hooks.",
      "inside": "Pinned https://github.com/Joko-F/hermes-fine-print/tree/7b574f1b5c8dc8eafe3ea9f87ae6b0111853f0ba (root; S9-S11). Loaded as hooks plus /limits and a bounded system-prompt section. Three scripted calls completed: missing read_file returned its normal error; a search_files page of 50 results gained an actual _fine_print warning with offset=50 continuation advice; a short README read retained normal truncation metadata. /limits topics listed 81 limits; self-test reported 16 verified detectors, 19 fixture-only detectors and 46 reference-only limits, with no detector errors; nonexistent limit lookup returned a useful response. On this source build self-test correctly warned that its v0.23.0 formats were unverified against package version 0.0.0. User-facing Desktop card delivery was not exercised.",
      "tests": "487 passed, 8 skipped, 0 failed, using the repository tests/pytest.ini to avoid importing the plugin root as a standalone package. The first incorrect root-level invocation produced collection/setup errors; those were harness configuration errors, not the final result.",
      "useful": 5,
      "reliable": 5,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 29,
      "rank": 1,
      "best": "It made a real silent search-page cap visible inside the model result, with a concrete recovery step and no additional model or network dependency (S9-S10; local run).",
      "worst": "Coverage is deliberately partial and version-sensitive: the local self-test classified 46 limits as reference-only and 19 detectors as fixture-only (S11).",
      "blocked": null
    },
    {
      "pr": 153443,
      "name": "local-hands",
      "validate": "Passed: catalog validate exit 0; enable and doctor exit 0, registering 1 tool and 1 hook. Backend and CLI registration are present in the pinned code.",
      "inside": "Pinned https://github.com/noegng/local-hands/tree/62327c4174582e3e8ea2d0b1e8fc57d4f4a24842 (root; S12-S14). Loaded its hands terminal backend, deferred hands tool and /hands/CLI management surface. Corrected tool_search found hands. Both hands(status) and hands(use, not-paired) returned structured ok=false with Operation not permitted: this sandbox denies creation of the AF_UNIX control socket before machine selection. The conversation itself completed without crashing. No pairing, command approval or remote terminal/file execution succeeded here; TLS pinning, token hashing, approval timeout and no-TTY refusal were assessed from source rather than claimed as runtime successes (S12-S14).",
      "tests": "Final pytest against an unchanged copy with a Python-valid package directory name: 22 passed, 37 failed, 19 setup errors; all 56 unsuccessful cases raised PermissionError at AF_UNIX socket creation. The documented unittest runner on the original checkout ran 59 reported cases with 38 errors (21 passed); a class-level setup error prevents its 19 execution methods from running individually. Initial pytest on the hyphenated checkout also hit a root-package import issue. Main network/approval path remains unverified in this sandbox.",
      "useful": 5,
      "reliable": 3,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 27,
      "rank": 5,
      "best": "It connects an always-on Hermes brain to the user’s own computers through the existing terminal and file backend, with local approval rather than a new shell tool (S12-S14).",
      "worst": "The central remote-execution path could not be verified here because its Unix control socket is prohibited, and its tool exposed only a generic permission error (S13; local run).",
      "blocked": null
    },
    {
      "pr": 150800,
      "name": "hypnos",
      "validate": "Passed: catalog validate exit 0; enable and doctor exit 0, 0 tools and 9 hooks. Hermes chat warned that the provides_middleware manifest field was ignored, but the code-registered middleware loaded.",
      "inside": "Pinned https://github.com/0xNoramiya/hypnos/tree/0a809b9085fdeaed63fd909b927afcc41de512de (root; S15-S17). Loaded capture hooks, routing middleware, five chat commands and CLI, with no new tools. Both scripted reads ran, including a missing-file error; /hypnos status then reported 1 turn and 2 tool calls. Habits and brain export gave clean empty-state responses, and malformed /lethe gave usage. Its own CLI doctor reported the absent trainer interpreter and Docker with setup advice. A bounded one-cycle sleep completed cleanly with nothing_to_learn and no training. Routing, promotion, rollback, scrubber and worker code/tests were read; actual LoRA training, GPU performance and learned-model quality were not established.",
      "tests": "229 passed, 2 skipped, 0 failed after adding the test-only PyYAML dependency. Skips were the real-trainer tiny-night integration and Docker sandbox test. The passing night tests use a fake trainer, not demonstrated real fine-tuning (S17).",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "total": 28,
      "rank": 3,
      "best": "It implements a substantive learning loop—capture, replay, dream generation, LoRA training, held-out promotion and model routing—that changes Hermes itself (S15-S17 and pinned implementation).",
      "worst": "Its distinctive benefit depends on a separate training runtime and enough personal examples; real training was skipped here, and the router supports only Chat Completions and Anthropic Messages (S15-S17).",
      "blocked": null
    },
    {
      "pr": 150596,
      "name": "ariadne",
      "validate": "Passed: catalog validate exit 0; enable and doctor exit 0, 0 tools and 19 hook registrations. The provides_middleware manifest field warning did not prevent its three middleware registrations from loading.",
      "inside": "Pinned https://github.com/miracle352/hermes-ariadne/tree/128c95967a0761ad24d3d224441d1502c43bbb69 (root; S18-S20). Loaded middleware, /ariadne, CLI and Desktop code. Both scripted reads completed, including a missing-file error; enabling record through documented configuration produced a cassette with 1 turn, 3 model calls and 2 tools. CLI list worked; invalid cassette lookup failed clearly. Replaying that CLI-origin cassette reported surface/tool drift instead of claiming equivalence. The plugin’s separate real-Hermes TUI roundtrip then passed: 2 turns, 4 identical model calls, 3 identical tool calls, 1 auxiliary call served locally, 0 network leaks, 0 hook-contract violations, and no real tool executed during replay. Desktop controls and upgrade/bisect against a second Hermes version were not exercised.",
      "tests": "21 unit tests passed, 0 failed; real-Hermes tests/e2e/roundtrip.py also passed all assertions, including identical replay and no real tool execution. Broader scenario matrix, fork test and cross-version upgrade test were not run.",
      "useful": 4,
      "reliable": 5,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "total": 29,
      "rank": 2,
      "best": "Its real Hermes record/replay roundtrip reproduced model and tool requests offline while keeping tool execution frozen, giving users a practical regression check before upgrades (S19-S20; local run).",
      "worst": "CLI-origin recordings replay through a different surface and showed drift here, while streaming and dispatcher-internal hooks are explicitly outside faithful replay (S18).",
      "blocked": null
    }
  ]
}