{
  "batch": "deep-48",
  "setup": "Installed unmodified NousResearch/hermes-agent source at f0066531cefb84421abe9f5e645cbcd7afa63848 with uv sync --locked --python 3.14; Python 3.14.10. Verified hermes plugins exposes install, validate and doctor. Fetched every entry at field.json's exact SHA. Ran the supplied tryit.py first with [] and then planned calls against Hermes' own FakeLLMServer, with fresh isolated homes. Added test-only dependencies in scratch. Supplemental conversations use the same upstream AIAgent driver and fake provider, retain the checked homes, set documented operator configuration and record full requests; no plugin source was repaired. All reported test outcomes are this seat's observations, not README claims. No real model, service credentials, GPU grant or desktop account was used. Tool discovery is deferred: an empty first-request tool delta alone does not mean a plugin failed to register. Sources identify the pinned code and its claims; runtime counts below are local observations.",
  "plugins": [
    {
      "pr": 150110,
      "name": "hermes-colab",
      "validate": "Passed all 15 catalog checks with --install-deps, including matching tools/hooks, security scan (safe), OAuth declaration and no core override; no warnings. Enable and doctor passed.",
      "inside": "Pinned PietrangeloManco/hermes-colab@f05cf934fff3b446f2e00cc32e19432e9e41e87b. Loaded 8 tools, pre_llm_call completion hook, and shipped Colab Desktop page. In real Hermes, colab_doctor returned ready=false with exact Google OAuth-client setup guidance; colab_status(all=true) returned an empty run list; a nonexistent script returned a structured error; an invalid accelerator was rejected before invocation. No call crashed or prompted for a sign-in. Read the supervisor, bundle/path policy, SQLite leases, OAuth transport, sync/resume implementation and tests. Actual Google assignment, GPU execution, resumed checkpoints and Desktop interaction were not verified without a Google grant.",
      "tests": "Initial full suite: 61 passed, 5 skipped because the local Jupyter server fixture did not start. After allowing that isolated server to start under this host, all 5 integration tests failed on kernel handshake/loss paths. A separate logged reproduction showed the kernel aborting with ZeroMQ Operation not permitted (src/ip_resolver.cpp:542), so these are host-restricted integration results, not proof that Colab execution works or is broken. A single kernel diagnostic also failed. Unit coverage for paths, secret exclusion, leases, sync, setup, notifications and tool errors passed.",
      "useful": 3,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "rank": 4,
      "best": "A readable, profile-aware GPU job supervisor with checkpoint sync, runtime recovery and chat/Desktop status extends Hermes into useful remote compute.",
      "worst": "It needs a user-created Google OAuth client, and its most valuable real-runtime and recovery paths could not be verified in this restricted host.",
      "blocked": null
    },
    {
      "pr": 153872,
      "name": "local-scout",
      "validate": "Catalog validation passed all 15 checks with --install-deps, security scan safe and no warnings; doctor passed with 9 tools/9 hooks. First empty-plan enable passed, but planned-run enable and retries, including a separate fresh home, failed: uv could not find an editable hermes-local-scout package under the generated workspace/plugin-sources path. Normal planned conversations therefore could not see Scout tools. Reliable is capped at 2 for this reproducible enable failure.",
      "inside": "Pinned Angerosteles/local-scout@c1e732ba4e9d431a711ea141c0095352a95bd7a2. Read its request/execution middleware, output transforms, detectors, consent/profile storage and optional learning paths. To separate code behavior from failed admission, a supplemental real-Hermes conversation explicitly enabled it in that isolated profile with already-installed dependencies, then ran its shipped operator CLI to grant profile masking/report consent; this is not a successful normal install. Five calls: valid savings calculation returned net=7 USD and break_even_n=125; read_file changed the synthetic contact to [EMAIL] and [CARD]; scout_report returned category/coverage counts without raw values; triage lacking dataset consent returned CONSENT_REQUIRED; negative n was rejected before invocation. All six captured provider requests lacked the original synthetic email and card. The user prompt was masked too. No supplementary call crashed. No broad detection accuracy, learned triage savings or live auxiliary masking guarantee is inferred from this small case.",
      "tests": "With an exact scratch copy named plugin (the tests import that package), 282 passed, 1 failed, 4 skipped, plus 160 passed subtests; HERMES_SOURCE pointed at the installed source. Earlier attempts had package-layout collection errors and are not counted as plugin failures. The single failing Hybrid.test_artifacts_reload_and_decision_measured_outcome was diagnosed by replaying each branch: logreg, knn and embed_search trained successfully; own_cnn cleanly returned PLAN_ONLY / unavailable: torch is missing (optional dependency). No network dataset accuracy claims were reproduced.",
      "useful": 5,
      "reliable": 2,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 25,
      "rank": 5,
      "best": "Its consent-controlled local masking demonstrably removed synthetic personal values before the provider saw them, addressing a recurring need for ordinary Hermes users.",
      "worst": "Normal enabling repeatedly failed in Hermes dependency-workspace generation, and the functional masking result required explicit isolated-profile configuration around that failure.",
      "blocked": null
    },
    {
      "pr": 150346,
      "name": "construct",
      "validate": "Passed all 16 catalog checks with --install-deps, including security scan (safe), declared capabilities, no core override and Desktop SDK surface; no warnings. Enable and doctor passed.",
      "inside": "Pinned rtmandreyev/construct@24a1bc838076c94728376850d461b383d003caa9, folder hermes/construct. Loaded 12 tools, pre_tool_call approval hook, /construct, a craft skill, Desktop viewers and dashboard routes; engine and viewer source are bundled. In real Hermes, construct_start created a hardcover book project. A corrected four-call follow-up used the returned project id: check returned zero errors/warnings; build(thumbnail=false) emitted an audited interactive page with 7 parts; export(format=glb) produced a 25,560-byte GLB and read it back as 7 named parts with an opening animation; an invalid format was rejected before invocation. The initial plan had omitted the project id and requested gltf rather than glb; its clean errors were test-plan mistakes, not plugin defects. FreeCAD/STEP, NetLogo and films were not verified.",
      "tests": "Ran the repository Hermes-plugin and Hermes-NetLogo suites: 39 passed, 3 failed, 6 skipped, plus 18 passed subtests. Failures: test_cd_end_to_end lacked a thumbnail; test_film_runs_as_a_job reported browser startup exit 1; test_jobs_stop_on_cancel_and_unload had already-failed film jobs rather than cancellable ones. Treat browser-dependent paths as unverified here; the local scene/build/GLB path succeeded independently. Optional FreeCAD/NetLogo scenarios were skipped. The rest of the standalone engine suite was read selectively, not run in full.",
      "useful": 3,
      "reliable": 4,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "total": 27,
      "rank": 3,
      "best": "One checked, provenance-aware scene really became an interactive Hermes artifact and an animated export, with guarded local execution and a substantial finished Desktop surface.",
      "worst": "The core creation path works, but the audience is specialized and browser/CAD/NetLogo dependencies leave much of the advertised output range unverified here.",
      "blocked": null
    },
    {
      "pr": 153221,
      "name": "capalis",
      "validate": "Passed: hermes plugins validate --install-deps --json, all 15 checks including capability declarations, security scan (safe) and no core override; no warnings. Enable and doctor passed.",
      "inside": "Pinned Femtech-web/Capalis@78843af2e15812fc391a74fe1cbcf2ac82a03de1, folder plugin. Loaded 7 tools, pre_gateway_dispatch hook and /capalis owner command. In real Hermes, status showed an intact empty store; setup named absent service tokens with fixes; capability comparison described issue.create and its setup requirement; compiling an empty draft returned E_INTENT/E_TRIGGER/E_STEPS; listing recipes returned []. All five calls returned without a crash. Live external writes/read-back and gateway delivery were not exercised without accounts.",
      "tests": "303 passed, 2 failed, 1 skipped across the repository suite. The failures were test_reporter_hears_back_when_the_issue_is_closed and test_slack_intake_replies_in_the_reports_thread: immediate post-close advance remained WAITING instead of VERIFIED. The opt-in real-provider sandbox was skipped. These failures concern completion polling, not evidence of an unauthorized write.",
      "useful": 4,
      "reliable": 3,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 25,
      "rank": 6,
      "best": "A version-bound approval, deterministic compilation, simulated writes, independent read-back and drift quarantine form a substantial Hermes automation engine.",
      "worst": "Two completion-notification tests failed, and capability comparison called a tokenless GitHub writer runnable while separately acknowledging that authentication was missing.",
      "blocked": null
    },
    {
      "pr": 153844,
      "name": "fine-print",
      "validate": "Passed all 15 catalog checks with --install-deps, including security scan (safe), declarations and no core override; no warnings. Enable and doctor passed.",
      "inside": "Pinned Joko-F/hermes-fine-print@7b574f1b5c8dc8eafe3ea9f87ae6b0111853f0ba. Loaded 11 hooks, /limits and a bounded system-prompt section; adds no model tools by design. Real Hermes search_files over 120 matching fixture files returned a 50-entry page with an added _fine_print continuation warning. Reading a 12,001-byte single-line fixture produced truncated_lines=true; a missing-file read returned a normal error without annotation or a crash. A second recorded conversation confirmed the warning reached the model. The real Hermes /limits handler also ran self-test, topics, a known-limit explanation, an unknown-id error and a self-contained 7 KB HTML history report. Self-test exposed 16 verified and 19 fixture-only detectors with no hook errors; the source install reports package version 0.0.0, so it correctly warned that formats were unverified for that version.",
      "tests": "487 passed, 8 skipped (495 collected) in the shipped pytest suite; counts taken from its double-quiet progress output. The fixture tests deliberately skip reference-only rules without a detector or committed shape. Inspected real-core fixtures, false-positive, recovery, ledger, concurrency, report and output-budget coverage.",
      "useful": 5,
      "reliable": 5,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "total": 30,
      "rank": 1,
      "best": "It demonstrably makes silent context loss visible to both Hermes and its user with bounded notes, actionable recovery guidance and no model or network cost.",
      "worst": "Detectors depend on specific Hermes result formats, and the documented fixture-only rules cannot yet justify the same confidence as observed live cases.",
      "blocked": null
    },
    {
      "pr": 151341,
      "name": "permission-canvas",
      "validate": "Passed all 15 catalog checks with --install-deps, including security scan (safe) and no core override; no warnings. Enable and doctor passed (backend kind; zero model tools/hooks is expected).",
      "inside": "Pinned TANGBUDU/permission-canvas@3a3086066051f5cc7a6ee52c20d62395b814da3f. Real Hermes discovered and selected its ComputerUseProvider and registered /pc-canvas and /pc-approve. Tried capture, list_windows and an invalid computer_use action through tryit; the core tool was unavailable, so the conversation stopped on invalid-tool calls rather than exercising a desktop. With computer_use.backend=permission-canvas and the required explicit non-Windows permission_canvas.desktop=native, provider availability was false without cua-driver. Human approval listing returned No pending approvals; a nonexistent id returned Unknown or expired. Starting the backend was refused by ControlBusy: Canvas did not finish its handoff; reopen it and resume. This is its first-run pause, not evidence of a completed desktop action. No real OS screenshot, human GUI interaction or input was verified here.",
      "tests": "314 passed, 9 skipped in the documented unit/protocol/real-loader suite, using HERMES_SRC and the Hermes interpreter. Includes fail-closed redaction, policy, exact one-time approval, focus, transport and state-file regression coverage with synthetic desktop adapters. Native Windows and live model scenarios were not run; this host has no configured cua-driver/desktop.",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 5,
      "beyond": 5,
      "total": 28,
      "rank": 2,
      "best": "Element-level read, act, approve and hide permissions enforced below the model, with a private human canvas, are a substantial improvement to Hermes computer use.",
      "worst": "Protection only covers this backend, requires a trusted policy directory and correctly grounded accessibility geometry, and the native desktop path remained unverified here.",
      "blocked": null
    }
  ]
}