{
  "batch": "deep-34",
  "setup": "Hermes source fdde27e7dae174cb4ba842589e92508852b7dd9f installed once with uv sync --locked --python 3.14 (Python 3.14.10); plugins help confirmed install/validate/doctor. Each entry was fetched at field.json SHA. All installation, source inspection and execution used test/scratch. Used supplied tryit.py and Nous tests/fakes/fake_llm_provider.py with real AIAgent; initial HTTP 502s were fixed by NO_PROXY for localhost. The new PM dependency-admission worker stalled, so subsequent harness runs used dependencies preinstalled in that venv, omitted --install-deps, seeded plugins.enabled in disposable homes, and set HERMES_DISABLE_LAZY_INSTALLS=1 to prevent update bootstrap. Normal fresh-install admission is therefore not certified. Catalog validate, enable on the seeded selection, doctor and conversations ran without replacing plugin code or Hermes runtime APIs. Non-memory plugin tools were deferred by Hermes Tool Search, so exercised through its tool_call bridge; an empty added_by_plugin list is not a failed load. Memory-provider setup additionally selected whos-listening and disabled built-in memory per its README. Tests used isolated homes and temporary directories, no real accounts or hardware. Scores reflect pinned code plus local observations; sources.json supplies pinned upstream sources, while execution counts are locally observed, not claims of upstream certification. Rank is a separate retention judgement: Lineage’s passive, verified workflow leads; memory privacy follows, then checkpoint safety with the observed cwd bug. Roam and Wings rank above Lens despite its higher total because the glasses/iPhone/Gemini combination narrows the audience substantially.",
  "plugins": [
    {
      "pr": 153316,
      "name": "hermes-lens",
      "validate": "Passed: hermes plugins validate --json exit 0, security scan safe; warning for private tools.approval imports in adapter.py:494,601. Dependencies preinstalled; see setup.",
      "inside": "Pinned brukchip/hermes-lens@487fb7c911fa91e3f3768ce863b77f030ab23b56, repository root. Loaded: doctor exit 0; registers hermes_lens platform, one pre_tool_call approval hook and lens CLI, no model tools. Empty-plan conversation completed. Three terminal calls inside AIAgent: lens --help succeeded; lens try reported missing access token with setup remedy (exit 1); invalid subcommand returned a clear usage error (exit 2). No call crashed. Gemini grant minting against Google, iOS build, glasses and live voice were not exercised.",
      "tests": "Passed 67 Python tests, including HTTP adapter, grants, replay protection and approval handling; 2 warnings. Swift app-core tests not run: Swift compiler unavailable on this Linux runtime.",
      "useful": 3,
      "reliable": 4,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 26,
      "rank": 6,
      "best": "A polished glasses-and-iPhone channel connects the user’s own Hermes memory and tools to voice, with real per-call grants and tap-only approvals.",
      "worst": "It requires compatible glasses, an iPhone build, Gemini access and HTTPS setup, and the live hardware path remains unverified here.",
      "blocked": null
    },
    {
      "pr": 149244,
      "name": "lineage",
      "validate": "Passed: catalog validation exit 0, security scan safe, Desktop surface accepted and no warnings; doctor exit 0 reports 10 hooks and no tools.",
      "inside": "Pinned P0GL0L/hermes-lineage@61de70980f4322838a9b53f4802623146f1046c2, repository root. Loaded observer hooks, lineage CLI/slash command and Desktop graph. Inside real Hermes, read revenue.csv, wrote a note citing its path, and received a clean missing-file result on bad input. The real lineage CLI then showed the note’s write_file producer/model and one evidenced source, wrote a receipt, and verified artifact hash, receipt hash and local signature. Changing the source CSV made verify return stale with sources_changed=1 (exit 2); querying an unknown artifact returned nothing recorded (exit 1).",
      "tests": "Passed 19 Python tests and 2 Node Desktop contract/rendering tests. Node tests used the repository’s SDK shim and fixtures, not a running Desktop application.",
      "useful": 4,
      "reliable": 5,
      "polish": 5,
      "creative": 4,
      "beyond": 5,
      "total": 28,
      "rank": 1,
      "best": "Automatic, verifiable provenance receipts and a usable Desktop graph make agent-produced files auditable without requiring the model to narrate its own work.",
      "worst": "A source read earlier is only an explicitly labelled inference, and the local HMAC signature is not an independent third-party attestation.",
      "blocked": null
    },
    {
      "pr": 153867,
      "name": "whos-listening",
      "validate": "Passed: catalog validator exit 0, security scan safe, no warnings; doctor exit 0 with two hooks, llm_request middleware and memory-provider registration.",
      "inside": "Pinned Web3B-eth/whos-listening@a498dd769c4196d0e911b713d4ad7363c8154ba1, repository root. Loaded with memory.provider selected and built-in memory disabled. Empty-plan conversation completed. Four direct memory calls inside Hermes: saved private window-seat preference as P1, recalled P1 with private tags, listed it through review(all), and rejected forget(not-an-id) cleanly. Duplicate model judging disabled for deterministic testing; no real Venice account or GitHub sync service exercised. In a second real conversation, custom was explicitly marked Shared: recall and review withheld the existing Private P1, forget(P1) returned the same missing-memory error, and none of four recorded model requests contained its private text.",
      "tests": "Passed all 524 Python tests, covering provider/guard, consent, route classification, review, duplicate handling, persistence and sync; no failures.",
      "useful": 5,
      "reliable": 4,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 27,
      "rank": 2,
      "best": "Provider-, gateway- and platform-aware memory filtering changes what Hermes can safely remember across trusted and shared routes.",
      "worst": "Privacy depends on correct setup and tagged-memory boundaries; ordinary user text, model paraphrases and auxiliary calls are not a general data-loss-prevention boundary.",
      "blocked": null
    },
    {
      "pr": 150563,
      "name": "hermes-roam",
      "validate": "Passed: catalog validator exit 0 and security scan safe; private core import warning for settings.py:408 hermes_cli.version_info.get_version_info. Doctor exit 0 registers three tools, three hooks, CLI/slash command and setup skill.",
      "inside": "Pinned vcruz305/hermes-roam@8b9fba41fea77f2fe2d6d572cffd43039d2da11f, repository root. Loaded in real Hermes via deferred tools. Initial status/push named missing local_dir; invalid scope returned ValueError; apply=true was refused by default. With a scratch local backend, freshly generated encryption key and real hermes roam init, roam_push published encrypted generation 1 (3 files, 1508-byte bundle), roam_status reported clean and current, roam_pull previewed generation 1 without applying, and invalid scope was rejected. Real S3/R2 and remote Git accounts were not tested; no other user home was synced.",
      "tests": "Broad Python run: 608 passed, 2 failed, 1 skipped, 2 deselected. Failures: 20,000-path walk took 6.01s against a 2s bound; CLI status exceeded its 80-column assertion. Skip: optional realistic-home fixture absent. Two blackholed private-network timing probes excluded after sandbox network denial; mocked S3 and local Git backend tests were included. Targeted recheck with less concurrent load: performance test passed, CLI-width test still failed (1 passed, 1 failed).",
      "useful": 4,
      "reliable": 4,
      "polish": 4,
      "creative": 3,
      "beyond": 5,
      "total": 25,
      "rank": 4,
      "best": "Encrypted, host-aware home sync combines previewable changes, conflict handling and human-controlled apply across personal Hermes installations.",
      "worst": "CLI status still violates its 80-column test, cloud backends were not exercised against live services, and multi-device setup requires key and backend management.",
      "blocked": null
    },
    {
      "pr": 153781,
      "name": "wings",
      "validate": "Passed: catalog validator exit 0, security scan safe and Desktop surface accepted; warns about private hermes_state_sessions.INTERNAL_LISTING_SOURCES and hermes_cli.profiles.list_profiles imports. In-process only for dashboard auth providers.",
      "inside": "Pinned e-shizz/wings@0bed887bd06dd356995cabdfc4bdd5af5ac3dd98, folder plugin. Loaded: doctor exit 0, two tools plus CLI/dashboard/auth surfaces. Empty conversation completed. Via real Hermes tool_call: wings_status returned an uninitialized network without leaking keys; both nonexistent-device and ../invalid revocations returned {\"error\":\"'devices'\"}, a caught KeyError on missing setup, not an actionable error. No WireGuard interface, privileged helper, real peer or phone was provisioned. Reliability capped at 2 for this handler failure.",
      "tests": "Seven standalone Python test scripts passed (core, mesh, net trust, phone path, router, start job, trust). Root-helper shell checks: 85 passed, 1 failed; failure assumes the current test user differs from root, so the wrong-owner fixture is invalid under privileged execution. Actual multi-device networking not tested.",
      "useful": 4,
      "reliable": 2,
      "polish": 4,
      "creative": 4,
      "beyond": 5,
      "total": 24,
      "rank": 5,
      "best": "WireGuard pairing, device identity and Desktop/phone access make a fleet of personal Hermes installations usable without a coordination service.",
      "worst": "Revoking before setup hits a caught KeyError and returns only 'devices'; privileged installation and real network recovery remain unverified.",
      "blocked": null
    },
    {
      "pr": 149553,
      "name": "hermes-time-machine",
      "validate": "Passed: catalog validator exit 0, security scan safe, no warnings; doctor exit 0 registers five tools and five hooks.",
      "inside": "Pinned ddaikodaiko/hermes-time-machine@1ac9c78e539dd518d6c8d8ffd64395b72bab30d4, repository root. Loaded in real Hermes with deferred atm tools. Checkpoint/list/preview worked and ../invalid checkpoint was rejected cleanly. Found a real configuration mismatch: with protected workspace different from session cwd, writing inside the protected workspace was labelled outside-workspace and blocked for approval; guard.py passes session_dir to effects.preflight. With cwd aligned, a five-call read/checkpoint/write/rewind-preview/bad-ID flow succeeded: write carried an automatic checkpoint annotation, reused the unchanged file blob, and preview reported app.txt modified. An earlier unread overwrite was correctly blocked by Hermes core and is not counted as a plugin defect. No external publish or deploy was performed; effect memory and approval flow were exercised in its real-PluginManager integration tests.",
      "tests": "Passed 122 unittest tests, including real Hermes PluginManager integration, after isolating temporary paths from the enclosing Git repository. Initial runs had fixture-dependent failures: ancestor Git discovery in a no-Git fixture, then intentional display truncation with excessively long temporary paths; neither required plugin changes.",
      "useful": 5,
      "reliable": 3,
      "polish": 4,
      "creative": 5,
      "beyond": 5,
      "total": 27,
      "rank": 3,
      "best": "Agent-visible checkpoints plus approval gates and persistent external-effect memory address the dangerous gap between rewinding a conversation and undoing the world.",
      "worst": "Effect classification uses session cwd instead of the configured protected workspace, falsely treating a protected-file edit as irreversible when those paths differ.",
      "blocked": null
    }
  ]
}