{
  "batch": "deep-36",
  "setup": "Hermes cloned from NousResearch/hermes-agent main at 998498af138018125738330dd439e86b6b23b24a; installed once using uv sync --locked --python 3.14 (CPython 3.14.4); plugins tested at field.json SHA pins. RUN.md tryit.py used unmodified with Nous tests/fakes/fake_llm_provider.py: empty discovery plan, then 2–5 scripted calls per entry, isolated scratch Hermes homes, no real model credentials. Runtime bootstrapped managed Python 3.14.9. Supplementary local tests distinguish expected credential/input refusal from crashes. Test dependencies installed only in disposable scratch; deliverable is a judging report, not runnable plugin code. Scores are independent judgments; run observations are local evidence, source URLs identify the exact reviewed code. Optional tools may be deferred behind Hermes tool_search, so an empty added_by_plugin list alone is not a load failure.",
  "plugins": [
    {
      "pr": 150696,
      "name": "privacy-xray",
      "validate": "Passed: all 15 catalog checks, no warnings; enable and doctor also exit 0. Pinned aa2bf92140016d7cf2ce855d0fff98f46badbc94, plugins/privacy-xray (field.json).",
      "inside": "Loaded: privacy_xray tool, /xray command, pre_api_request/pre_auxiliary_call hooks, llm_request masking middleware, skill and dashboard backend. Five scripted calls completed: check returned 0 red/4 amber/24 green; map observed two real scripted-model requests to localhost; trace correctly reported no canary; find with short query refused because this harness did not resolve a local session surface; bogus action was rejected by Hermes schema validation without a crash. Empty-plan added tools list omitted deferred tools, but direct tool dispatch worked. No real vendor traffic or Desktop rendering tested. Supplementary /xray canary and trace worked; an invalid --hours value returned usage guidance.",
      "tests": "Passed: 79/79 using the supplied unittest script; initial pytest invocation hit package-name collection errors, resolved by using its documented unittest runner. Includes detectors, request masking, file modes, rotation, gateway restrictions, reports and audit fixes.",
      "useful": 5,
      "reliable": 5,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 29,
      "rank": 1,
      "best": "Makes Hermes's otherwise hidden main and auxiliary data flows visible with metadata-only auditing and a narrowly scoped request guard.",
      "worst": "Cannot block auxiliary requests or observe independent tool network traffic, and the finder was unavailable through this harness's unresolved session surface.",
      "blocked": null
    },
    {
      "pr": 150596,
      "name": "ariadne",
      "validate": "Passed catalog admission; enable and doctor passed. Pinned 128c95967a0761ad24d3d224441d1502c43bbb69, repository root.",
      "inside": "Loaded 16 declared hook types and llm_execution/tool_request/tool_execution middleware, /ariadne, CLI subcommands, Desktop and dashboard integrations; deliberately no model tool. Empty plan and two-call tryit plan succeeded (terminal printf and a clean read_file missing-path error). Repeated unmodified tryit with HERMES_ARIADNE_CONTROL set to recording: cassette contained one user turn, three model calls and two tool results. hermes ariadne play latest --json replayed Done with two tool_match events and no errors or live provider; reported drift on all three model requests. Diff identified CLI-to-TUI surface/prompt and deferred-tool catalog differences, so this was not an identical replay. A nonexistent cassette ID failed cleanly with guidance. Upgrade installation, bisect, live fork and rendered Desktop UI not exercised.",
      "tests": "21/21 unit tests passed, covering cassette codec/redaction, normalization, contract checks, safe replay configuration, exports and upgrade reporting. End-to-end recording/replay additionally completed with matched tools and reported environment drift rather than claiming identity.",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "total": 28,
      "rank": 2,
      "best": "Adds genuinely offline, tool-frozen session replay and request-level regression evidence through Hermes execution middleware, with a substantial debugging UI and CLI.",
      "worst": "The CLI recording replayed through TUI and a different tool catalog, producing three drift reports even on the same Hermes checkout; upgrade verdicts still need careful environment interpretation.",
      "blocked": null
    },
    {
      "pr": 150353,
      "name": "fluent",
      "validate": "Passed catalog admission with no failed checks; enable and doctor passed. Pinned 182c4301f2437e784e9022f1f38576e1d68c3f5f, repository root.",
      "inside": "Loaded three hooks, /fluent command, @cli context reference and dashboard, with no new agent tool. Original tryit plan ran git --version and an invalid unadopted Git flag normally. Supplementary real Hermes command dispatch adopted Git 2.53.0 (155 commands, 3174 options); terminal accepted git --version, blocked invented git diff --definitely-not-a-flag, but also falsely blocked valid git diff --staged. Invalid mode returned choices cleanly. Desktop/dashboard rendering and approval interaction not exercised.",
      "tests": "435 passed, 1 failed, 106 skipped in pytest. Failure: test_valid_commands_are_never_blocked[git diff --staged], independently reproduced through real Hermes dispatch. Skips include unavailable program fixtures; no claim of full coverage on this machine.",
      "useful": 5,
      "reliable": 3,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 27,
      "rank": 3,
      "best": "Learns the exact installed CLI and intercepts shell mistakes before execution through Hermes hooks rather than asking the model to remember a manual.",
      "worst": "The high-confidence guard rejects valid git diff --staged, so a common legitimate workflow is blocked even after successful adoption.",
      "blocked": null
    },
    {
      "pr": 151929,
      "name": "promise-keeper",
      "validate": "Passed catalog admission; enable and doctor passed. Pinned ab91fee21f607d5515a8e007cf615fc285718dd0, repository root. Hermes emits a nonfatal provides_middleware manifest warning, while runtime middleware registration succeeds. Catalog warning: private core import tools.terminal_tool.get_session_cwd in runtime code, plus private imports in its live_e2e test script (rule 9); no check failed.",
      "inside": "Loaded pre_llm_call/transform_llm_output/pre_tool_call hooks, tool_execution middleware, /promise, and Desktop/dashboard controls; no model-facing tool. Tryit allowed a scratch write, blocked hermes plugins disable promise-keeper with built-in R0, and returned a clean missing-file result. Supplementary real Hermes /promise add path activated a local rule; write_file into its protected folder was blocked with the user's quotation, an outside write succeeded, and an unknown confirmation ID returned a useful error. File existence confirmed both outcomes. UI and interactive ask approvals not exercised.",
      "tests": "660 passed, 15 failed, 8 skipped, 2 xfailed. Failures include Windows shell/backslash/caret/trailing-dot cases and git stash/checkout -f/reset --hard fallback expectations; these are the entrant's own red-team assertions, not proof every shell variant bypasses protection in every deployment.",
      "useful": 5,
      "reliable": 3,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 27,
      "rank": 4,
      "best": "Turns a user's persistent rules into deterministic checks outside the model, with quoted reasons, human-controlled changes and working self-protection.",
      "worst": "Fifteen red-team assertions fail, leaving material gaps in the broad shell-enforcement surface despite successful direct-file protection.",
      "blocked": null
    },
    {
      "pr": 153766,
      "name": "hermes-ussd",
      "validate": "Passed catalog admission, dependency installation, enable and doctor. Pinned 80ccfd0b1dfcf01d07aae72eb63a13242ca93af6, repository root.",
      "inside": "Loaded africastalking platform adapter, CLI commands, /ussd status and an SMS-reply skill; intentionally no agent-callable send tool. Three planned terminal calls inside Hermes worked as expected: hermes ussd status showed empty outbox; doctor exited 1 with explicit AT_USERNAME/setup guidance rather than hanging; purge --number invalid exited 2 with an invalid-number error. Real-model, carrier and paid SMS delivery were not exercised because no Africa's Talking account, phone or key was supplied. The supplied tests exercised local webhook, outbox and adapter flows with test doubles, not a real carrier.",
      "tests": "Passed all 300 tests (300 collected; full pytest run exit 0, no skips/failures shown). Installed declared aiohttp plus test-only hypothesis/pytest-asyncio after the initial collection attempt lacked them. Coverage includes encoding properties, menus, settings, allowlists, webhook authentication, retry/cap/outbox behavior and real Hermes class integration with a fake carrier.",
      "useful": 3,
      "reliable": 5,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 27,
      "rank": 5,
      "best": "Extends Hermes to ordinary feature phones with an asynchronous USSD-to-agent-to-SMS platform, store-first replies and unusually thorough cost and access controls.",
      "worst": "Its audience is constrained by a regional carrier integration and operator provisioning, and actual carrier delivery remains unverified without an account and reachable callback service.",
      "blocked": null
    },
    {
      "pr": 153225,
      "name": "agent-pool",
      "validate": "Passed all catalog checks, no catalog warnings; doctor registered 3 tools. Pinned 5b973a5c3dfd8633840bb438208211da1c78f3cf, repository root. Original tryit enable failed at uv lock: staged editable agent-pool package directory not found. Installing the same Git SHA into a separate fresh Hermes home with --yes-deps --enable succeeded. Installing after validation in the harness home reproduced the enable refusal, so that setup route remains unreliable; no plugin code was changed.",
      "inside": "Declared 3 tools, 10 gateway commands, pool registration and a Desktop console. Original empty-plan/three-call tryit sessions did not enable it: pool_status and pool_report_usage returned tool-not-found; the third call had no recorded result. Supplementary AIAgent with the same Nous fake model in the successfully installed fresh home completed four calls: pool_status returned configuration and a clear missing-Tor-sidecar warning; usage reporting with an invalid claim/negative token count failed closed on missing Tor; pool_pane_report wrote one local operation successfully; empty claim input also returned the Tor prerequisite error before input validation. No Tor daemon, sidecar, live Discord team or real bundle transfer was provisioned; none is claimed verified.",
      "tests": "412 passed, 48 failed, 11 skipped, 5 setup errors. Many failures/errors were environment-limited: bubblewrap NETLINK_ROUTE sockets and ClamAV socket operations denied, unavailable /tmp and Tor. Other failures included an installer-output assertion, TorNotReady exception incorrectly accessed as exc.value, and a test referring to an absent .hermes/reports document. These totals do not mean 53 independent production bugs. Declared crypto dependencies and test pyyaml were installed locally.",
      "useful": 3,
      "reliable": 2,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 23,
      "rank": 6,
      "best": "Builds a private agent-team collaboration platform with encrypted repository bundles, task claims, approval gates and a real Hermes Desktop console.",
      "worst": "The validation-then-enable route fails on this Hermes runtime, and the loaded status still labels ledger and identity ready while service construction can fail on missing Tor.",
      "blocked": null
    }
  ]
}