{
  "batch": "deep-60",
  "setup": "Installed real NousResearch/hermes-agent source at 8ec227311bbbe7863d3982ce8f35012d6a59e7d7 with uv sync --locked --python 3.14 (Python 3.14.10); plugins CLI exposes install, validate and doctor. Used the supplied tryit.py and Nous FakeLLMServer/AIAgent, with disposable homes, no real model credentials, and localhost excluded from the proxy. Also used a scratch copy of tryit.py omitting redundant --install-deps after installing declared dependencies in the locked environment; its extra_config enables in_process isolation and selects Samskara memory for the planned run. Complete admission JSON was separately collected with the unchanged Hermes validate_plugin_dir API because tryit truncates command stdout. All six original empty-plan runs completed with validate/enable/doctor/chat exit 0. Prepared retries without dependency preparation hit the 120-second enable deadline; additional successful AIAgent probes set the normal plugins.enabled configuration explicitly. Disabled-tool errors from those timed-out retries are not treated as plugin failures. All tests and probes are local observations at the pinned commits; sources.json supplies the corresponding source material. Device/account-dependent paths are explicitly unverified below.",
  "plugins": [
    {
      "pr": 150251,
      "name": "gemini-live",
      "validate": "Passed all catalog checks at b6c2f24354aeed1a8b5651b62944d14ed96a472d; safe scan, no warnings, Desktop SDK surface check passed. Source: https://github.com/cineglobe/hermes-gemini-live/tree/b6c2f24354aeed1a8b5651b62944d14ed96a472d .",
      "inside": "Original supplied tryit.py: validation, enable, doctor and chat all exited 0. Adds CLI, Desktop voice engine/page and dashboard API, no ordinary agent tools. A configured real-Hermes conversation made 3 terminal calls to its CLI: doctor and models returned clear missing-Gemini-key errors, and invalid-command returned usage error without crashing. Separate backend TestClient probes: status 200/configured=false; models 200 with missing-key reason; session 412; unknown server tool returned structured error. Live duplex audio, ephemeral-token issuance, reconnection and Desktop rendering were not exercised without a Gemini account/microphone.",
      "tests": "Python: 117 passed, 10 failed. Failures concern read-only shell path rules: /root and hidden-ancestor scratch paths are deliberately rejected, several expected outside-path reasons differ, and /etc/os-release resolving outside /etc is refused. JavaScript engine: 39 passed. Generated-bundle freshness check passed. Reviewed README, manifest, server/secret handling, read-only command policy, browser engine and tests.",
      "useful": 5,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 2,
      "best": "A polished duplex voice layer can delegate to the user's real Hermes, monitor its threads and preserve its approval flow across Desktop and the dashboard.",
      "worst": "The actual voice session remains unverified without Gemini, and its optional read-only shell refuses legitimate paths in this environment.",
      "blocked": null
    },
    {
      "pr": 149823,
      "name": "samskara",
      "validate": "Passed all catalog checks at d807e875da3176301db9955e0352c9c8adb6cb01; no warnings. Source: https://github.com/SebaWag/hermes-samskara/tree/d807e875da3176301db9955e0352c9c8adb6cb01 .",
      "inside": "Original supplied tryit.py: validate, enable, doctor and chat all exited 0. Real Hermes doctor passed: memory provider registered. Empty/default run added no tools until memory.provider=samskara was selected. With the documented embedded://seed/demo.ttl endpoint, Hermes offered five memory tools. All five planned calls ran: PostgreSQL search returned 3 entities/15 facts with provenance; write queued one triple; DELETE was refused; status was healthy; conflicts returned none. No external graph server or extraction model was exercised. The configured provider conversation completed.",
      "tests": "150 passed after excluding localhost from the network proxy; initial 147 passed/3 failed were proxy-distorted HTTP failure-mode tests. Reviewed README, minimal manifest, memory-provider lifecycle, journal, embedded backend, tests and secret-scope access.",
      "useful": 4,
      "reliable": 4,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "rank": 5,
      "best": "Auditable relational recall and conflict tracking extend Hermes memory itself, with a usable offline demonstration and a fully passing local suite.",
      "worst": "Writes wait for manual reindexing, the embedded backend supports only narrow read-only query shapes, and session-end extraction is skipped when the host exposes no ctx.llm.",
      "blocked": null
    },
    {
      "pr": 151411,
      "name": "salts-secrets",
      "validate": "Passed at 09c1b426b08e41380a47233c4f3cead36a017ecf, folder salts_secrets. Warnings: role-dependent public_notes_lookup and four hooks not registered in the default owner-role probe; security caution ssh_dir_access in privacy_rules.yaml:71 (a screening rule). Source: https://github.com/dotv-7/salts-secrets-plugin/tree/09c1b426b08e41380a47233c4f3cead36a017ecf/salts_secrets .",
      "inside": "Original supplied tryit.py: validate, enable, doctor and chat all exited 0. Doctor passed, registering owner note tool and 2 hooks; declared guest hooks are conditional. With explicit plugins.enabled and in_process configuration, real Hermes loaded it: all 3 owner calls completed, a safe note became pending human /salt approve, empty text was refused, and password-bearing text was refused. A separate public-role AIAgent probe completed 3 calls: public lookup returned no documents; terminal and file:// image access were blocked by the guest gate. End-to-end Telegram/Discord routing, owner approval and publication/revocation were not exercised.",
      "tests": "374 passed, 3 failed, 1 skipped. All 3 failures were isolation-audit expectations: scratch guest working directories inherit the enclosing Git repository, which the plugin correctly flags for possible context injection. Reviewed README, manifest, owner/public registration, note approval, gateway routing, guest allowlist, privacy rules, tests and dashboard.",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "total": 28,
      "rank": 3,
      "best": "A real profile and tool boundary lets people share their own Hermes while owner-approved context stays distinct from private memory.",
      "worst": "Deployment requires an in-process multiplexed gateway and careful public-profile isolation, and live gateway routing plus revocation remain unverified here.",
      "blocked": null
    },
    {
      "pr": 151968,
      "name": "tickover",
      "validate": "Passed all catalog checks at 45ebd2bd447504fb6229088fda1b3cab052aa982; safe scan, no admission warnings. Source: https://github.com/rapha18th/tickover/tree/45ebd2bd447504fb6229088fda1b3cab052aa982 .",
      "inside": "Original supplied tryit.py: validate, enable, doctor and empty chat all exited 0; doctor registered 10 tools and 3 hooks. In a configured real-Hermes conversation all 4 planned calls ran: fleet gave useful empty-state guidance; hear(top=0) and listen(seconds=0) returned bounded input errors; selftest succeeded using actual local encoders/ffmpeg, with all 4 checks true (healthy, drift, start, no change) and a generated card. The synthetic faulty fan was correctly located in 3.9-8 kHz. No microphone was available; actual machinery, physical recording and outbound repairer alerts were not exercised. Runtime warns about the ignored provides_middleware field (empty here), without preventing loading.",
      "tests": "Full suite after runtime dependencies: 172 passed, 5 failed, 1 skipped. Installed the declared development dependencies and reran the 5 failures: 2 passed (PyYAML import checks), 3 remained. Remaining failures: crafted-playlist baseline fails under bundled ffmpeg 9.0.2; RTSP relay test times out; device-fault repairer notification test fails at watch setup. The successful real-Hermes selftest is additional evidence. Reviewed README/manifest, encoder pins and hash verification, tools, source allowlist/relay limits, cron/notification hooks, storage and tests.",
      "useful": 3,
      "reliable": 3,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "total": 26,
      "rank": 6,
      "best": "Local acoustic verification gives Hermes a new physical-world feedback channel, and its synthetic healthy/drift/start/no-change selftest worked inside the agent.",
      "worst": "It targets a narrower hardware-maintenance audience, and stream/security-fixture plus repairer-alert regressions remain unresolved despite the working local audio path.",
      "blocked": null
    },
    {
      "pr": 154010,
      "name": "own-compute",
      "validate": "Passed all catalog checks at 97522db41e7f6352089bd350df45bc64267a71e6; safe scan, no warnings, Desktop SDK surface check passed. Catalog name own-compute; manifest/runtime id cloud-compute. Source: https://github.com/Siioh/hermes-own-compute/tree/97522db41e7f6352089bd350df45bc64267a71e6 .",
      "inside": "Original supplied tryit.py: validate, enable, doctor and empty chat all exited 0. Loaded in real Hermes using plugins.enabled=[cloud-compute] with in_process isolation; empty conversation completed and the plugin injected compute context. Declares 15 tools plus hooks, a guide and Desktop/dashboard page; tools are deferred rather than added to the initial offered list. Four planned calls completed: compute_status returned unconfigured status/templates/setup guidance, compute_host types described SSH/Slurm/EC2/local hosts, list returned no hosts, and an invalid action returned structured guidance. No actual SSH host, Docker daemon, Slurm cluster or AWS account was supplied; provisioning, transfers, approvals and billing were not exercised live.",
      "tests": "Full suite: 3172 passed, 15 failed, 30 skipped. Repeated the 15 failures with localhost proxy bypass and Git ceiling: 2 passed, 13 still failed. Remaining failures include Git repo detection under an enclosing checkout, agent stop/collect reporting not_fetched for missing stub repo paths, timer rearming, and a saved-folder notification assertion. The two repaired failures were local HTTP readiness/display probes. Reviewed manifest, README, modular registration, host tools, secret-scope resolver, approval hooks, agent lifecycle, Desktop and tests.",
      "useful": 4,
      "reliable": 3,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "rank": 4,
      "best": "A substantial compute control plane brings a user's own SSH machines, containers, cluster jobs and remote agents into Hermes with contextual tools and a detailed UI.",
      "worst": "Agent stop/collection and other regression tests still fail here, while the actual remote lifecycle and cost controls remain unverified without configured infrastructure.",
      "blocked": null
    },
    {
      "pr": 152450,
      "name": "heats",
      "validate": "Passed all catalog checks at 2bcee10e9ca6489b7fd23bb0362a36eacdae269f; safe scan, no admission warnings, Desktop SDK check passed. Runtime additionally warns that provides_middleware is an unknown manifest field, but register_middleware still executes and the plugin loads. Source: https://github.com/EconLearn/hermes-heats/tree/2bcee10e9ca6489b7fd23bb0362a36eacdae269f .",
      "inside": "Original supplied tryit.py: validate, enable, doctor and empty chat all exited 0. Loaded in real Hermes with 3 deferred tools, tool_request middleware, pre/post-tool and pre_verify hooks, CLI/skill and Desktop/dashboard surfaces. Three planned calls completed cleanly: heats_run on an isolated Git fixture without tests requested an explicit check; heats_wait and heats_apply refused unknown run ids. Separate real AIAgent referee probes read then edited a tracked scratch calculator: the bad implementation produced 0 passed/1 failed and 3 measured correction nudges; the scripted model then hit its iteration budget. Restoring the implementation produced 1 passed/0 failed, no nudge, and a completed conversation. A live multi-subagent race and winner application were not exercised.",
      "tests": "211 passed after setting a Git ceiling so loose-file fixtures do not inherit the enclosing repository; the first run was 206 passed/5 failed for that reason. Two additional real-Hermes referee scenarios verified both red and green tests. Reviewed README, manifest, lifecycle adapter, worktree runner/isolation, test-protection/scoring/apply logic, approval gate, referee and tests.",
      "useful": 5,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 1,
      "best": "The measured pre_verify referee improves the user's existing coding agent directly, and the real-Hermes red/green probes demonstrated its central safeguard.",
      "worst": "Parallel attempts can spend substantial model time, and full race isolation, cancellation and winner application were covered by tests/code rather than a live subagent race here.",
      "blocked": null
    }
  ]
}