{
  "batch": "deep-x-01",
  "setup": "Hermes source cloned from NousResearch/hermes-agent at a84025ade22d2a4085f575d4353072e5b702635d; uv sync --locked --python 3.14 installed Python 3.14.10 and the locked runtime. CLI exposed install, validate and doctor. Used the supplied tryit.py and Nous FakeLLMServer/AIAgent with isolated task-local Hermes homes, no real model or account. Initial parallel baselines shared a directory and hit loopback proxy errors; discarded those attempts. Final runs use separate homes and loopback proxy bypass. --install-deps entered slow PM provisioning; final admission ran the same Hermes catalog validator without that flag after installing declared dependencies (DuckDB 2.0.0, matplotlib 3.11.2), plus pytest and PyYAML for tests. Set HERMES_DISABLE_LAZY_INSTALLS=1 to suppress unrelated startup updates; enabling still prepared Hermes-managed dependencies. Empty-plan discovery preceded planned calls. For functional runs the runner preserved dependency runtimes, set config version 12 and disabled tool-search deferral so registered plugin tools reached the scripted model directly; plugin code was unchanged. UI-only plugins were exercised via their local routes/commands from Hermes tools, not misrepresented as chat tools. Python tests ran without a Hermes-source PYTHONPATH. Tests requiring host inspection were excluded; none of that information is evidence. All execution findings below are my local observations; sources.json pins the code and documentation behind the assessments. No Desktop browser, microphone, phone or live paid provider was exercised. Rankings compare only this six-plugin batch, not the whole hackathon.",
  "plugins": [
    {
      "pr": 152691,
      "name": "grimoire",
      "validate": "Passed: Hermes catalog validator exit 0; all 16 checks, no warnings, security safe; doctor exit 0 registered 8 tools and 6 hooks. Pinned bettersandeep/grimoire@4824773040b131f3179e5896d71d1435908758a7 (repository root).",
      "inside": "Loaded in real Hermes (enable, doctor and conversation exit 0). Empty plan loaded successfully; default tool search deferred plugin schemas, then the functional run with deferral off offered all 8 Grimoire tools. All 5 planned calls ran: attached a synthetic four-row sales CSV, profiled it (100/100 health), grouped revenue correctly as North=250 and South=200 with receipt q1, rejected DELETE with a read-only error, and returned pending for a merge until the user explicitly approves. Source CSV remained unchanged. Query guard locks DuckDB external access and allowed paths. Desktop charts/learning UI read in source, not visually exercised. A separate real-Hermes scripted reply probe sourced 250 from q2 but deliberately asserted 999; the output hook correctly appended No source query for: 999 and Verdict: needs revision.",
      "tests": "Passed: python -I selftest.py printed selftest passed (21 named self-test groups, including query safety, receipt checking, data quality, metrics, joins and restart cases). No pytest case count is claimed for this assertion-based runner.",
      "useful": 5,
      "reliable": 5,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 29,
      "rank": 1,
      "best": "A useful local analytics product that combines guarded SQL, persistent query receipts, answer-checking hooks and a substantial Desktop data workbench.",
      "worst": "The verifier checks digit claims rather than semantic truth, small values can match by coincidence, and the documented limits leave text and ranking claims unchecked.",
      "blocked": null
    },
    {
      "pr": 154055,
      "name": "augur",
      "validate": "Passed: all 16 catalog checks, no warnings, security safe; enable and doctor exit 0, registering 4 tools and 6 hooks. Pinned Bruno-coder21/augur@6e775b23e2a322c8eee32db4074a44cb85e86e89 (repository root).",
      "inside": "Loaded in real Hermes. Empty plan passed; with tool-search deferral off it offered augur_park, augur_queue, augur_done and augur_report. All 5 planned calls ran: parked a research task as item 1, listed it, moved it to needs_you, returned a clear error for id -999, and generated an actual weekly report PNG with one item needing the user. Read the classification, nudge, metering, Desktop chip and Friday guard implementations; did not claim a real model classified priorities or a scheduled Friday execution completed.",
      "tests": "Passed on full rerun: 101 pytest tests. Initial parallel run: 100 passed, 1 failed because test_late_label_appends_a_note_instead_of_waiting took 0.300043s against a <0.3s assertion; full rerun passed in 34.92s. Tests cover classification parsing/errors, nudge timing, queue/report behavior, registration and Friday-run path/tool restrictions. No live-provider semantic evaluation.",
      "useful": 4,
      "reliable": 5,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 4,
      "best": "A coherent focus assistant that turns passive priority tracking into a persistent queue, bounded nudges, weekly reports and guarded research-and-draft automation.",
      "worst": "Every substantial message can add classifier cost and latency, while the scripted runtime cannot establish real-model labeling accuracy or a completed scheduled Friday run.",
      "blocked": null
    },
    {
      "pr": 153900,
      "name": "annotary",
      "validate": "Passed: all 16 admission checks, security safe. One warning: annotary_hermes/webapp.py:24 imports private gateway.webapp_listener.webapp_settings (catalog rule 9 compatibility warning). Enable, doctor and conversation exit 0; doctor registered 10 tools. Pinned josh-andrews-dev/annotary-hermes@253abf811b4be90f57834f8233730fbc688e7ced (repository root).",
      "inside": "Loaded in real Hermes. Empty plan succeeded; functional run offered all 10 tools. All 5 planned calls ran against the real checksum-pinned Annotary v0.5.0 service in a throwaway task-local Hermes root: shared Release plan as version 1, read exactly the Markdown shared, anchored comment t1 to the requested passage, listed that unresolved thread, and rejected doc=-invalid with doc must be a Document id or Name. This establishes real document/version/thread operations; did not claim a full human Send back -> model edit -> phone approval loop or visually test the Desktop page. Read editor-only versioning, review guards, opt-in gateway wake consent and profile reconciliation code.",
      "tests": "Passed: 330 Python tests, 3 skipped, 1 deliberately deselected process-inspection test; tests exercised a real private pinned Annotary server, including review/version authorization, bridge routes, enablement and profiles. Desktop Node tests initially lacked the Hermes SDK stand-in; rerun using the repository's documented --import tests/desktop/register.mjs loader passed all 29 tests. No real-provider live review-loop test.",
      "useful": 5,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 2,
      "best": "Persistent line comments, versions, editor permissions and consented wake-ups make human review a first-class workflow across Hermes agents and clients.",
      "worst": "Per-profile installation and wake-up consent add setup work, and admission warns that a private gateway.webapp_listener import can break across Hermes releases.",
      "blocked": null
    },
    {
      "pr": 149244,
      "name": "lineage",
      "validate": "Passed: all 16 catalog checks, no warnings, security safe; enable, doctor and conversation exit 0. Doctor confirmed 10 observer hooks, no tools; adds slash/CLI commands and Desktop provenance surfaces. Pinned P0GL0L/hermes-lineage@61de70980f4322838a9b53f4802623146f1046c2 (repository root).",
      "inside": "Loaded in real Hermes; empty plan correctly added no chat tools. All 5 planned calls ran: read a four-row CSV, wrote a Markdown report naming it, hermes lineage show linked the artifact to the evidenced CSV and fake-model call, verify reported fresh, and a missing-file read returned a clean ordinary tool error. Further checks used this actual captured graph: removing the CSV produced source status missing but summary fresh; changing North 100 to 900 produced stale; a new real Hermes read of the changed CSV (agent exit 0, old report untouched) then made verify incorrectly report fresh/unchanged. The shared source node hash is overwritten by later reads and historical artifact edges do not retain their original source hash. Restored the synthetic CSV afterward. No remote refetch or visual Desktop test.",
      "tests": "Passed: 19 Python tests after installing missing test-only PyYAML (initially 18 passed, 1 import failure); 2 Desktop Node tests passed after installing the declared tests-js dependencies and using its SDK shim. Supplemental real-capture checks reproduced two freshness defects not covered by the supplied suite: missing source -> fresh, and changed source reread -> old artifact fresh.",
      "useful": 4,
      "reliable": 3,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "rank": 5,
      "best": "Passive observer hooks produce readable, hashed provenance receipts and distinguish observed, evidenced and inferred links without asking the agent to narrate its own work.",
      "worst": "A missing source still yields fresh, and rereading a changed source overwrites its shared hash so an unchanged old report incorrectly becomes fresh again.",
      "blocked": null
    },
    {
      "pr": 150251,
      "name": "gemini-live",
      "validate": "Passed: all 16 catalog checks, no warnings, security safe; enable, doctor and conversation exit 0. No agent chat tools/hooks is intentional: registers a CLI command plus Desktop/dashboard surfaces and backend routes. Pinned cineglobe/hermes-gemini-live@b6c2f24354aeed1a8b5651b62944d14ed96a472d (repository root).",
      "inside": "Loaded in real Hermes; empty plan correctly added no model tools. All 4 planned Hermes terminal calls exercised the installed plugin's actual FastAPI router using its local TestClient: status returned HTTP 200 with configured=false, session creation returned a clear HTTP 412 asking for a Gemini key, builtin memory recall returned HTTP 200 and No matching memories, and non-object tool args returned HTTP 400. No Google request, microphone, iPhone, voice call or live Hermes thread delegation was made; their implementations and offline tests were read instead. Code resolves keys through Hermes get_secret and constrains ephemeral tokens server-side; voice approvals are user responses rather than model-callable approvals.",
      "tests": "Core/API subset passed: 21 Python tests, 1 host-check test deselected. Broader Python run: 114 passed, 10 failed, 3 host-inspection tests deselected; nine failures involve strict path rejection under the hidden task-workspace ancestor, and one is an absolute-path allowlist assertion, not a failed Google call. Did not inspect host paths to explain that assertion. All 39 offline JS engine checks passed (scripted WebSocket/gateway), including interruptions, queueing, approval boundaries and quota handling; generated-bundle build.py --check passed. Real Google Live integration not run without a key.",
      "useful": 5,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 3,
      "best": "A full-duplex voice control plane combines live conversation, background Hermes work, thread steering and user approval handling instead of merely transcribing a prompt.",
      "worst": "Live audio and delegation remain unverified without a Google key and real clients, while its strict file guard rejects legitimate files beneath hidden workspace ancestors.",
      "blocked": null
    },
    {
      "pr": 153146,
      "name": "hermes-regress",
      "validate": "Passed: all 16 catalog checks, no warnings, security safe; enable, doctor and conversation exit 0. Registers 4 hooks and /regress, intentionally no model tools. Pinned harishkotra/hermes-regress@877c6dfdfd5e80bb1dca79a67b3081f9764545b5 (repository root).",
      "inside": "Loaded in real Hermes; empty plan correctly added no chat tools. All 5 planned calls ran: read a synthetic CSV, wrote a report, invoked rg_cli status, requested a nonexistent case (clean No case matching response), and tried a missing file (clean ordinary tool error). At turn end capture wrote one real case containing all 5 tool calls. Supplemental AIAgent privacy probe with an explicitly synthetic password found it stored unchanged in prompt, answer and write_file args. A second real-Hermes armed-fixture probe changed write_file content at the same path: the plugin blocked the actual write and preserved the sentinel, but supplied the original fixture anyway because args_key discards content. Both agents exited 0. This tested real capture and fixture interception, not a complete cross-model /regress run or its CLI preflight.",
      "tests": "Passed: all 18 supplied unittest tests. Additional code-level reproductions showed only tool results were redacted and different write_file content scored as exact. Additional actual-Hermes probes confirmed plaintext synthetic password retention in three case fields and wrong-content fixture matching while preventing the real side effect. These defects are absent from the supplied passing tests.",
      "useful": 4,
      "reliable": 1,
      "polish": 3,
      "creative": 4,
      "beyond": 5,
      "total": 22,
      "rank": 6,
      "best": "Capturing personal agent sessions and replaying their tool decisions behind fixture-blocking hooks directly addresses the risk of changing models or configurations.",
      "worst": "Automatic capture retains synthetic passwords in prompts, answers and arguments despite its redaction promise, and same-path writes with different content incorrectly receive an exact fixture match.",
      "blocked": null
    }
  ]
}