{
  "batch": "trial-2",
  "setup": "Hermes source cloned once at 8ec227311bbbe7863d3982ce8f35012d6a59e7d7; uv sync --locked --python 3.14 succeeded with Python 3.14.4. CLI exposes install, validate and doctor. Supplied RUN.md and tryit.py used; empty-plan run and 2-5 scripted calls attempted per entry with Nous FakeLLMServer/AIAgent, isolated scratch profiles and no real credentials. pytest, pytest-asyncio, PyYAML and Hermes-pinned mcp 2.3.0 added only to the scratch environment for checks. Runtime observations below are this seat's executions, not claims made by entrants. Sources are the URLs in sources.json; field.json supplies each exact repo/SHA/subdir. No reference entries are in this six-entry batch. Desktop rendering, Apple Silicon inference, authenticated services and a real car are not established by the scripted CLI model. The runner counts any returned tool message as ran, including unknown-tool errors; these are distinguished below. Its initial concurrent bare-tools cache was invalid, so built-in tools reported as added were disregarded and registration/dispatch checked separately. Judgments apply the supplied 1-5 rubric, not a prediction of an official result.",
  "plugins": [
    {
      "pr": 151933,
      "name": "apphitlist",
      "validate": "Passed: hermes plugins validate --install-deps --json exit 0; portable manifest, fields, config-schema and security-scan checks pass, scanner safe; enable and Doctor exit 0.",
      "inside": "Pinned mmdincer/apphitlist-mcp@a7ef3444bd7b6821f34908e802a71a2fdde06e50 (root). Empty-plan conversation completed; Doctor registers 0 native tools/0 hooks. Portable MCP endpoint plus bundled apphitlist skill, not server implementation. Planned search_apps(query=fitness,limit=2) and bad get_app(app=\"\") both return tool-does-not-exist inside Hermes; neither reached AppHitlist. No API key/account was supplied or fabricated; authenticated nine-tool service and its reliability could not be tested. README explicitly says the remote server source is absent; mcp.json supplies its URL without authentication configuration. Sources: pinned README.md, mcp.json and field.json in sources.json.",
      "tests": "None: no test files or local server implementation at this pin; two Hermes call attempts produced unknown-tool responses, not successful research calls.",
      "useful": 3,
      "reliable": 2,
      "polish": 3,
      "creative": 2,
      "beyond": 3,
      "total": 16,
      "best": "Gives indie app developers a clearly documented remote route to app-market, advertising and complaint research that a prompt alone cannot retrieve.",
      "worst": "The pinned package is mostly setup documentation and a skill around a closed remote service, and no research tool was usable in the credential-free Hermes run; Reliable is capped at 2 for that observed runtime gap.",
      "blocked": null
    },
    {
      "pr": 149183,
      "name": "clack",
      "validate": "Passed: admission exit 0, declared nine tools match isolated registration, scanner safe and no core override; no warnings. Enable and Doctor exit 0. Manual rule-11 review nonetheless finds file-based credentials in client.py; passing scanner does not certify that policy.",
      "inside": "Pinned adroidian/clack@a965b33051d35bcef1dd3fa15255d34aacdba99a, plugins/hermes-clack. Loaded nine clack_* tools and bundled messaging skill; tools are deferred behind Hermes discovery rather than initially offered. Four scripted calls dispatched through real Hermes: whoami returned enrolled=false without network; reconcile(status) returned pending=false; poll(limit=0) was rejected by Hermes schema validation before handler; send(to=research-agent,text=release-note request) returned ok=false with generic safe-failure text before enrollment. No relay identity, trusted inviter or account was fabricated, so actual enrollment, encryption delivery and ack are supported by offline tests/code, not a live-service run. Sources: pinned README.md and client.py; registration and dispatch are local execution evidence.",
      "tests": "Passed: documented python -m unittest discover -s plugins/hermes-clack/tests -v, 38 tests in 14.570s, including adversarial relay, invitation, crypto, durable outbox and credential-permission cases. Initial pytest invocation had 38 setup errors importing the hyphenated package; using the documented runner resolved those harness errors.",
      "useful": 3,
      "reliable": 2,
      "polish": 4,
      "creative": 4,
      "beyond": 4,
      "total": 21,
      "best": "Adds encrypted agent coordination with durable outbox reconciliation, explicit acknowledgements and independent relay/inviter verification rather than merely messaging instructions.",
      "worst": "client.py directly reads identity.key and decrypts a service token from identity.json instead of Hermes secret storage, violating catalog rule 11 despite careful file permissions; unconfigured send also hides the actionable cause behind a generic failure.",
      "blocked": null
    },
    {
      "pr": 149187,
      "name": "token-dashboard",
      "validate": "Passed: admission exit 0; security scan safe and Desktop SDK-surface check passed. Capability probe explicitly skipped because this is manifest/JS-only. Enable exit 0; CLI Doctor exit 1 (No __init__.py), so CLI nonloading is not presented as proof that Desktop cannot load it.",
      "inside": "Pinned Whydiriki700/token-dashboard@39949565742a74d00ec03c9921c7ef3eee208d88 (root). Empty-plan Hermes conversation completed, but CLI loader logs No __init__.py and adds no Python tools. Code registers a right pane and statusBar.right chip through the Desktop SDK and reads focusedUsage plus session.list. CLI tool-discovery probes cannot exercise that UI; corrected search/invalid-description attempts are discovery checks, not plugin calls. A supplementary Node VM using SDK/React stubs confirmed both registrations and pricing/normalization helpers; real Desktop rendering and live gateway usage were not available. Sources: pinned README.md and desktop/plugin.js.",
      "tests": "None supplied. Supplementary checks passed: node --check desktop/plugin.js and 6 Node VM assertions (mini pricing, backend cost precedence, absent usage, alternate token fields, unknown-model fallback, two UI registrations). These do not replace a real Desktop integration test.",
      "useful": 4,
      "reliable": 2,
      "polish": 3,
      "creative": 2,
      "beyond": 4,
      "total": 19,
      "best": "Surfaces focused-session usage and cost in Hermes Desktop without sending data elsewhere, and honestly avoids inventing historical session token totals.",
      "worst": "No shipped tests or demo prove the Desktop path, CLI Doctor cannot load this JS-only plugin (conservative Reliable cap 2), and unknown models use a generic estimated rate while the visible status chip shows tokens rather than the README-promised always-visible cost.",
      "blocked": null
    },
    {
      "pr": 152550,
      "name": "brainvault",
      "validate": "Passed: admission exit 0, isolated memory-provider registration, security scan safe, no core override and Desktop-surface checks passed; no requires_hermes floor declared. Enable and Doctor exit 0. Manual review finds rule-11 violation in _core.py key storage even though scanner passes.",
      "inside": "Pinned xBuildyTeam/wave-os-hermes@76fc8fe297b6f75362e54e7ba5c45c6c39179c81, brainvault/. Loaded memory provider and /brainvault command. Original empty-plan/default-provider run exposes no BrainVault tools; following README by setting memory.provider=brainvault in a scratch-only variant of tryit.py exposes 11 tools. Four calls then all return: status reports unpaired/no wallet; remember saves a private release-note preference; recall retrieves the exact saved text labelled data, not instructions; invalid set_wallet(address=invalid-address) returns a clear format error without setting a wallet. No account/wallet, pairing, external minting or cross-machine restore was attempted. Code uses read-only chain requests and leaves signing/mint approval to external wallet UI. Sources: pinned README.md, __init__.py and _core.py.",
      "tests": "Passed: python -m pytest tests -q, 32 passed and 3 skipped in 5.68s (live Theta/Wave checks opt in with BV_LIVE=1). Includes PV1 vectors, tamper/wrong-key rejection, secret screening, capture/recall and memory-provider behavior; does not demonstrate a live wallet restore.",
      "useful": 4,
      "reliable": 2,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 24,
      "best": "Extends Hermes memory itself with working local capture/recall, provenance and secret screening, plus an implemented encrypted, fingerprint-checked path toward owner-held portable memory.",
      "worst": "_core.py directly reads decrypt-only secrets from agent.key or OS keyring instead of Hermes secret scope, violating catalog rule 11; its distinctive pairing/mint/restore flow still requires an external wallet and was not demonstrated here.",
      "blocked": null
    },
    {
      "pr": 151709,
      "name": "pit-crew",
      "validate": "Passed: admission --install-deps exit 0; six declared tools and three hooks match, scanner safe, no core override and Desktop surface passed. Warning: private core imports in safety/gate.py (tools.approval), settings.py (Hermes settings writer), watcher/delivery.py (send_message_tool). Enable and Doctor exit 0; these compatibility warnings preclude a perfect reliability score.",
      "inside": "Pinned trevornk/pit-crew@91dec4414b60f7a90b5bdd561d336604b1cc630e (root). Empty plan loads six car_* tools, three hooks, /car, bundled skill, dashboard/Desktop integrations and background-watcher code. Five scripted calls dispatched inside Hermes: car_status returns explicitly simulated battery/security/freshness; car_history(kind=trips,days=7) returns labelled simulated/recorded history; car_readiness(departure=tomorrow 07:30,trip_miles=30) returns ready with assumptions; car_command(lock) returns dry_run and says no car was contacted; car_command(fly) returns invalid with unknown-command text and an audit record. A separate three-call Hermes follow-up successfully disabled left_unlocked_away, listed it as disabled, and rejected an unknown rule; thus the standalone rule-readback test failures did not reproduce with the real PluginContext. No real car/account was used. Approval and stale/motion handling are implemented and covered by code/tests, but this five-call run did not demonstrate a human-approved live unlock or physical readback. Sources: pinned README.md, __init__.py and safety/gate.py.",
      "tests": "Failed in Hermes environment: after installing missing PyYAML, python -m pytest tests -q completed with 833 passed, 8 failed, 79 skipped in 142.48s. Failures: audit-page rule validation, impossible-trip alert, dashboard/rule edit readback, low-tire override, two fast-loss threshold cases and tool rule-disable. Several involve tests using standalone/fake settings while the real Hermes settings writer is importable; they are reported rather than silently dismissed. Earlier collection had two missing-yaml errors, resolved before this full run. Live car and some Hermes/browser integration cases skipped.",
      "useful": 4,
      "reliable": 3,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "best": "A complete proactive hardware companion combines persistent alerts, trip readiness, freshness-aware multi-source state, a simulator and approval-aware commands with Desktop and dashboard pages.",
      "worst": "The large test suite still has eight failures in this Hermes environment and admission warns about private core imports; simulator success cannot establish safe behavior against a real car or every supported integration.",
      "blocked": null
    },
    {
      "pr": 150815,
      "name": "omlx",
      "validate": "Passed: admission --install-deps exit 0, provider registration and dependency checks passed, scanner safe, no core override, Desktop surface and declarations passed; no warnings. Isolation audit explicitly requires in-process execution because create_client returns a live SDK client, consistent with README. Enable and Doctor exit 0, provider omlx registered.",
      "inside": "Pinned vitaliy-blprnt/hermes-plugin-omlx@eb748caf0a3080cd3b8d9426eb995d2722ec3406 (root). Empty-plan conversation completed; registers named model provider, authenticated dashboard API and Desktop settings UI, no agent tools. Corrected scripted CLI tool-search/invalid-description probes confirm no callable omlx tool; they do not test inference. Then exercised real Hermes web_server app with TestClient and its normal scratch-session authentication: GET /api/plugins/omlx/status returned 200, supported=false, no installed/running runtime and an Apple Silicon requirement; POST /start returned 400 with the same actionable requirement; malformed POST /download repo=../../invalid returned 422 schema validation. No fake Mac, downloaded native runtime, real generation, model activation or rendered Desktop UI was claimed. Sources: pinned README.md, provider.py and dashboard/plugin_api.py.",
      "tests": "Passed: python -m pytest tests -q, 30 passed in 2.74s, covering inert discovery, guarded provider client, profile/origin checks, ownership, model deletion, job scoping and cancellation/publication behavior. Three additional calls through actual Hermes web app returned expected 200/400/422. Hardware-native install/generation not run on this Linux host; author verification notes are not counted as this seat's execution.",
      "useful": 4,
      "reliable": 4,
      "polish": 5,
      "creative": 3,
      "beyond": 5,
      "total": 26,
      "best": "Makes local Apple Silicon inference a managed Hermes provider and Desktop workflow, with profile-scoped storage, credential/ownership guards, cancellation and validated model selection.",
      "worst": "Actual runtime installation, generation and rendered Desktop operation remain unverified here because macOS 15+ Apple Silicon is required, and the provider cannot run under strict host-process isolation.",
      "blocked": null
    }
  ]
}