{
  "batch": "deep-x-06",
  "setup": "Hermes source https://github.com/NousResearch/hermes-agent at a84025ade22d2a4085f575d4353072e5b702635d; installed once under test/scratch/hermes with uv sync --locked --python 3.14 (CPython 3.14.4). plugins --help confirmed install, validate and doctor. Used RUN.md and tryit.py with Nous FakeLLMServer/AIAgent; empty plans first. Scratch copy of runner retains full validation output and scripted-provider request bodies; Permission Canvas trials additionally select its backend and explicit experimental native Linux setting. All homes and test data isolated in scratch. Local test observations below are this seat's measurements, not entrant claims; no vendor keys, voice device or graphical desktop used. Plugin revisions/folders are those in field.json. Dependencies for local tests were installed only in scratch; catalog validators used their own isolated dependency environments. Sources are indexed in sources.json; counts and actual-call outcomes are local observations. Ranking favors likely sustained use, balancing platform breadth and observed defects; scores are not the sole ordering key.",
  "plugins": [
    {
      "pr": 151341,
      "name": "permission-canvas",
      "validate": "Passed all 15 catalog checks, safe security scan, no warnings; enable and doctor passed. Pin 3a3086066051f5cc7a6ee52c20d62395b814da3f. Source: https://raw.githubusercontent.com/TANGBUDU/permission-canvas/3a3086066051f5cc7a6ee52c20d62395b814da3f/README.md",
      "inside": "Loaded; empty-plan AIAgent chat completed. Adds a computer-use provider and human /pc-canvas and /pc-approve commands, not a new model-facing tool. A second three-call AIAgent plan selected permission-canvas and explicit experimental native Linux placement: pc_canvas --check gave structured missing-policy, missing-driver and missing-Tkinter diagnostics; capture and invalid-action calls through tool_call both reported computer_use unavailable. No native desktop operation was verified here. Its real-Hermes loader tests did exercise provider selection and cross-process approval-command sharing with a test spy driver: yellow click withheld, human-command approval consumed once, repeat required approval again.",
      "tests": "314 passed, 9 skipped, 0 failed in 26.77 seconds, including both real-provider/command-loader parametrizations. Skipped coverage includes platform-dependent cases; no graphical Windows/cua-driver integration run.",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "total": 28,
      "rank": 2,
      "best": "Element-level screenshot and accessibility redaction plus exact one-use human approvals materially change Hermes computer use.",
      "worst": "Native desktop behavior is still a platform-sensitive prototype, and terminal/file/browser paths can bypass its explicitly limited backend boundary.",
      "blocked": null
    },
    {
      "pr": 152764,
      "name": "redactgate",
      "validate": "Passed all 15 catalog checks, safe security scan, no warnings; enable and doctor passed. Pin d4bb87d78381d0514cd8dd74525fea98a5b46d62. Manifest declares no new tools; code registers request/tool middleware, five hooks and /redact. Source: https://raw.githubusercontent.com/ctmakc/hermes-redactgate/d4bb87d78381d0514cd8dd74525fea98a5b46d62/README.md",
      "inside": "Loaded in real Hermes, empty-plan chat completed. Three scripted calls: write_file preserved synthetic Jane Doe/email locally; read_file returned PERSON/EMAIL placeholders to the provider; missing-path read returned a structured not_found result. All four recorded main-model requests contained neither raw synthetic name nor email. This trial used raw synthetic values in the scripted write arguments; restoration of model-issued placeholders is additionally covered by the passing real-Hermes integration suite. Adds no model-facing tools. Auxiliary calls are observed, not universally rewritten; /redact harden and configured block_types are separate opt-ins. Collision mechanism source: https://raw.githubusercontent.com/ctmakc/hermes-redactgate/d4bb87d78381d0514cd8dd74525fea98a5b46d62/vault.py (SessionVault.token keeps an existing token without checking its value).",
      "tests": "63 passed, 0 failed (4 warnings), including 4 real-Hermes integration tests; pytest completed in 4.00 seconds. Additional deterministic vault check failed: after 2,414 distinct synthetic emails, two shared the same six-hex EMAIL token and the later value restored to the earlier email.",
      "useful": 5,
      "reliable": 3,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "rank": 4,
      "best": "Offline, dependency-free middleware protected actual model requests and tool results while preserving useful local files.",
      "worst": "A reproduced 24-bit placeholder collision restored the wrong email, and auxiliary calls remain an explicitly disclosed privacy gap.",
      "blocked": null
    },
    {
      "pr": 151301,
      "name": "confidential-mode",
      "validate": "Passed all 15 catalog checks, enable and doctor. Safe security scan and no core override; warning names 31 private-core import sites centralized in hermes_compat.py, a maintenance risk disclosed by the entrant. Pin e407486a2227be8cafe7b4350586f3016ebff573. Source: https://raw.githubusercontent.com/ainsleys/hermes-confidential-mode/e407486a2227be8cafe7b4350586f3016ebff573/README.md",
      "inside": "Loaded and empty-plan chat completed; adds two attested provider profiles, eleven hooks, execution middleware, /confidential and CLI, not model-facing tools. Three scripted terminal calls returned observing status, an actionable on-host preflight refusal for auxiliary.vision.provider=auto, and a clean argparse error for invalid-tier. A supplementary run explicitly routed vision to the local scripted provider and locked on-host-only before AIAgent: local file read succeeded, web_search was blocked because an unattended run could not obtain human approval, and a missing file returned structured not_found. Receipt CLI without a session id returned usage rather than a receipt. Vendor attestation/encryption was inspected and exercised only through entrant fixtures/tests, not against a live keyed vendor.",
      "tests": "Final full serial run with declared vendor dependencies installed: 3084 passed, 4 failed, 8 skipped, 6 xfailed in 513.73 seconds. Failures: default ledger test demands WAL while Hermes selects DELETE for this linked SQLite; nested-PTY self-approval test hard-codes unavailable /tmp; allowlist_from_the_live_registry expects yb_search_sticker; builtin-tool data-flow inventory lists five yb_* tools absent from this Hermes checkout. Initial pre-dependency run had 2849 passed, 20 failed, 24 skipped, 6 xfailed; those vendor-import failures cleared after installing the declared dependencies. xdist was unusable because a parametrized id contains a process-specific object address, so the final counts are serial.",
      "useful": 4,
      "reliable": 4,
      "polish": 4,
      "creative": 5,
      "beyond": 5,
      "total": 27,
      "rank": 3,
      "best": "Real execution middleware successfully enforced an on-host lock and stopped an unattended off-host tool before dispatch.",
      "worst": "Its extensive security boundary still excludes shell/code traffic and cannot stop off-tier auxiliary calls, while 31 private-core dependencies complicate compatibility.",
      "blocked": null
    },
    {
      "pr": 153640,
      "name": "patient-research",
      "validate": "Not run: pinned repository could not be read.",
      "inside": "Not run: no pinned plugin code available.",
      "tests": "Not run: pinned repository inaccessible.",
      "useful": 0,
      "reliable": 0,
      "polish": 0,
      "creative": 0,
      "beyond": 0,
      "total": 0,
      "rank": 6,
      "best": "No code-based merits can be established at the assigned pin.",
      "worst": "The assigned repository and pinned raw manifest return HTTP 404, preventing inspection or execution.",
      "blocked": "Could not read https://github.com/solo386/patient-research at 8642b35bfc6a169ad20cc2f343f8c6dd34d82681, subfolder plugin: git fetch returned an authentication/access error, and both https://api.github.com/repos/solo386/patient-research and the pinned raw plugin/plugin.yaml returned HTTP 404."
    },
    {
      "pr": 150160,
      "name": "helm",
      "validate": "Passed catalog admission, enable and doctor; safe security scan. Desktop surface passed with review warnings about 27 app-markup CSS selectors, which are theme CSS rather than DOM access in the inspected source. Pin 0f68da9b485870059be2bef71f284a81ad771fdd. Source: https://raw.githubusercontent.com/ASV-Labs/hermes-helm/0f68da9b485870059be2bef71f284a81ad771fdd/README.md",
      "inside": "Loaded in real Hermes and completed empty-plan chat; intentionally registers no agent tools/hooks/commands. Two scripted terminal calls inside AIAgent exercised its actual sprite router functions: manifest listed 191 sprites; batch accepted base_blue.png and returned no data for nonexistent nope.png and traversal ../x.png. These are backend checks, not an end-to-end Hermes Desktop run. The JS SDK-mock suite registered 31 contributions and rendered 13 components; a real graphical Desktop was unavailable.",
      "tests": "npm ci and npm test passed: 31 contributions, 13 rendered components; direct FastAPI TestClient assertion script passed, 191 sprites available (not a counted pytest suite).",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 3,
      "beyond": 5,
      "total": 26,
      "rank": 5,
      "best": "A complete, well-documented Desktop control desk brings cross-bot approvals, questions and task plans into useful SDK contributions.",
      "worst": "Its benefit is limited to Desktop users, and real Desktop event routing and human approval clicks were not exercised here.",
      "blocked": null
    },
    {
      "pr": 152458,
      "name": "housemate",
      "validate": "Passed catalog admission with no warnings; enable and doctor passed, registering seven tools and one hook plus the deferred platform, commands and Desktop surface. Pin 09c1a1d0a88bfa8762cee6b3ccf94271b0f172c5. Source: https://raw.githubusercontent.com/RaYYeR220/housemate/09c1a1d0a88bfa8762cee6b3ccf94271b0f172c5/README.md",
      "inside": "Loaded and empty-plan chat completed. Deferred tools were discoverable through tool_search; initial rooms/set/list calls cleanly returned hub-not-running and a negative duration failed schema validation. Supplementary AIAgent run used the pinned real Hub, ControlServer, SQLite store, room registry and TimerService in a separate loopback process: five calls listed Kitchen/Office, created a 600-second tea timer, listed the same timer, rejected seconds=-1 before invocation, and cancelled it successfully. No real microphone, satellite, STT/TTS provider, gateway adapter lifecycle or spoken reply was exercised by this supplemental run; TLS was off only for its isolated loopback hub fixture.",
      "tests": "Initial run: 545 passed, 24 failed, 1 skipped; 23 failures were Hermes tools package shadowed by plugin tools.py. Re-run after preimporting real Hermes tools and running from tests/: 568 passed, 1 failed, 1 skipped in 21.73 seconds. Remaining failure: test_server_tls_context_ignores_an_injected_client_sslcontext expected minimum_version >= TLSv1_2 but observed MINIMUM_SUPPORTED on this interpreter. Plugin source was not changed.",
      "useful": 5,
      "reliable": 3,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 27,
      "rank": 1,
      "best": "Turns spare browser devices into a practical room-aware Hermes platform with deterministic timers and a substantial Desktop and approval interface.",
      "worst": "The sole remaining test failure undermines its TLS-default assumption, while the complete voice-and-device path remains unverified here.",
      "blocked": null
    }
  ]
}