{"batch":"deep-23","setup":"Hermes was not installed in this session: time/turn budget did not allow cloning hermes-agent, installing dependencies, and running tryit.py for six plugins. Each plugin was instead judged by reading its pinned-commit source from GitHub (API trees + raw file contents): README, plugin.yaml manifest, main implementation module(s) and test layout, per the brief's fallback for when Hermes cannot be installed.","plugins":[{"pr":150873,"name":"skill-court","validate":"not run (no Hermes install); plugin.yaml declares requires_hermes >=0.23.0, provides_hooks (pre_tool_call, transform_tool_result, on_skill_lifecycle, on_session_finalize, pre_llm_call), provides_middleware (llm_request); no obvious catalog violation seen (no file-based key reads, no unapproved money movement found)","inside":"not run inside Hermes; from code reading: replays tasks with/without a skill in Docker sandboxes, scores the difference, quarantines harmful skills without editing them; judge.py shows deliberate prompt-injection resistance for its own LLM judge calls (random per-call tag markers, explicit 'only text inside those tags is data' instruction)","tests":"present, not run here: tests/ has 12 files (test_appeal, test_arms, test_auto, test_dashboard, test_judge, test_migration, test_mining, test_procs, test_runner, test_scoring, test_skills_index, test_store); README also documents a 9,000-skill simulated benchmark (bench/simulate.py)","useful":4,"reliable":4,"polish":4,"creative":4,"beyond":5,"total":26,"rank":4,"best":"A genuinely new capability (blind pairwise before/after replay judging of agent-written skills) that only code with sandboxes and a judge call could do.","worst":"Scores were produced by reading the pinned commit only; its 12 test files and 9,000-skill benchmark were not executed in this session.","blocked":null},{"pr":151376,"name":"joust","validate":"not run (no Hermes install); plugin.yaml declares provides_tools (joust_launch/evaluate/diff/steer/merge/status/crossfire/report), provides_hooks (pre_tool_call, post_tool_call, subagent_start, subagent_stop); merge step explicitly routes through Hermes's human approval card before applying a patch, matching the catalog's 'no money/irreversible action without approval' spirit (here: no merge without approval)","inside":"not run inside Hermes; from code/README: '/joust demo' builds a throwaway repo and runs a full three-contender tournament with no model key, in about two seconds, per the README; worktree isolation is enforced by a tool_request middleware rewriting each child's path/workdir","tests":"present, not run here: tests/ has 8 files plus a tests/desktop_smoke/ suite (smoke.mjs, test_desktop_smoke.py, loader-hooks.mjs, react-shim.mjs, sdk-stub.mjs) exercising the Desktop UI path","useful":4,"reliable":4,"polish":5,"creative":4,"beyond":5,"total":27,"rank":3,"best":"Git-worktree isolation plus a 'Crossfire' cross-testing step and a human-approved merge make this more than a benchmark script; the architecture explicitly states what a prompt could not do (hook binding, pre_tool_call veto, worktree rewriting).","worst":"Narrower day-one audience than the batch's top two: useful mainly to developers running multi-contender coding benchmarks, and judged from source only, not a live run.","blocked":null},{"pr":149968,"name":"lahja","validate":"not run (no Hermes install); plugin.yaml config_schema stores only model ids/paths and runtime knobs (stt_model, language, vad, threads, timeout), no API keys; README states 'Fully local. No API keys. Stores no audio and no transcripts.', consistent with the catalog's secret-handling rule","inside":"not run inside Hermes; from code/README: routes Arabic voice notes through a dialect-tuned local Whisper via whisper.cpp with Silero VAD and a hallucination filter, routing non-Arabic audio to a separate model; reports a 41.1%\u219229.1% word-error-rate improvement over stock Whisper across 13 dialects, reproducible via bench/run_bench.py","tests":"present, not run here: tests/ has 6 files (test_manifest, test_provider, test_real_engine, test_review_fixes, test_routing, test_text)","useful":3,"reliable":4,"polish":4,"creative":4,"beyond":5,"total":25,"rank":6,"best":"Adds a genuinely new model capability to Hermes (dialect-preserving local Arabic STT) that a prompt or skill has no access to, with reproducible benchmark numbers.","worst":"Narrowest addressable audience in the batch (Arabic-dialect speech users specifically), and its reported WER gains were taken from the README rather than re-run.","blocked":null},{"pr":153871,"name":"barehand","validate":"not run (no Hermes install); README states the plugin needs QEMU and UEFI firmware (edk2/OVMF) on a Linux or macOS host plus a model that receives tool-result images; backend.py's own docstring stresses action methods never raise and every result carries an explicit ok/effect contract, and risky input (Ctrl-Alt-Del) is routed to 'ask the user for approval' rather than auto-sent","inside":"not run inside Hermes (no QEMU/VNC target available in this session); from code/README: a computer_use backend that drives a machine with no OS, network or agent on it purely via QEMU QMP/VNC screen+keyboard; README reports 12/12 rescue runs passing an independent judge across two models on QEMU guests, with per-run evidence (judge.txt/report.txt) checked into docs/results/","tests":"present, not run here: .github/workflows/ci.yml plus a dedicated tests/ dir (per tests/README.md referenced from the top README); large demo corpus of recorded frame-by-frame runs under barehand/demo/","useful":4,"reliable":4,"polish":5,"creative":5,"beyond":5,"total":28,"rank":1,"best":"The most novel and on-theme entry: a real computer_use driver that lets Hermes repair an OS-less machine through raw screen/keyboard, directly matching Nous's own 'unfuck your hardware with Hermes' framing, backed by checked-in per-run judged evidence.","worst":"Needs QEMU plus UEFI firmware and a vision-capable model, so day-one install friction is real even though the capability is strong; not run live in this session.","blocked":null},{"pr":153867,"name":"whos-listening","validate":"not run (no Hermes install); plugin.yaml declares its one secret (WHOS_LISTENING_SYNC_TOKEN) with 'type: secret, env: WHOS_LISTENING_SYNC_TOKEN' under config_schema and lists it in optional_env, i.e. read through Hermes's own secret/settings mechanism rather than a plain file -- the behaviour the catalog rule is aimed at","inside":"not run inside Hermes; from code/README: a memory provider that tiers every memory Private/Shared by gateway, chat platform and model route trust, refusing a save with no tier rather than defaulting it to Shared when settings can't be read (store.py fail-closed path)","tests":"present, not run here: tests/ has 20 files (test_autoreview, test_cli, test_commands, test_consent, test_duplicates, test_guard, test_id_allocator, test_judge, test_order, test_provider, test_recall, test_resolve, test_review, test_routes, test_settings, test_store, test_sync, test_sync_v2, test_tools, test_vectors, test_venice) plus tests/vectors/ fixture data","useful":4,"reliable":4,"polish":4,"creative":3,"beyond":5,"total":25,"rank":5,"best":"Broadly useful to almost any Hermes user and reads its one secret through Hermes's declared secret mechanism rather than a file, with a documented fail-closed default when trust can't be determined.","worst":"The private/shared memory-tiering idea itself is a refinement of an existing concept (memory providers with access tiers) rather than something new for agents, and was not exercised live.","blocked":null},{"pr":151804,"name":"dress-rehearsal","validate":"not run (no Hermes install); README states explicitly 'No network calls, no subprocesses, no dependencies', and documents a conservative, fail-closed shell classifier (any VAR=value prefix, git -c, env, sed etc. are always recorded rather than assumed safe) that reads as catalog-rule-aware design around not moving things without approval","inside":"not run inside Hermes; from code/README: pre_tool_call records every side-effecting call while armed instead of running it, pre_llm_call tells the agent it is rehearsing, and '/rehearse go' replays only the kept steps with no model in the loop, refusing a step if the target file's SHA-256 changed since rehearsal","tests":"present, not run here: README states '84 tests cover the shell classifier, risk flags, tool_call unwrapping, drop and keep, state round trips, ordered replay, stop and resume, refusal on drift, read-before-overwrite, replay in the rehearsed folder, and the full hook-plus-command flow' (tests/test_rehearsal.py)","useful":4,"reliable":5,"polish":4,"creative":4,"beyond":5,"total":27,"rank":2,"best":"Simple, dependency-free, fail-closed hook design (pre_tool_call interception plus hash-pinned replay with no model in the loop) gives real safety guarantees a prompt-based plan mode cannot, backed by a stated 84-test suite.","worst":"The read-only shell classifier is explicitly conservative and will over-record benign commands it doesn't recognize, and none of the 84 tests were executed in this session.","blocked":null}]}