Case study · 05
Hermes Agent
A personal AI assistant I text over iMessage, with a local memory layer and the ability to hand coding to Claude Code.
- Role
- Personal project
- Status
- Active build
- Read
- 3 min
- Stack
- Hermes AgentOpenRouterOllamaHindsightiMessageClaude CodeNode
3
models in the fallback chain
Problem
I wanted one assistant that lives where I already message and can act, without being locked to a single model or a single vendor.
Approach
- 01The model choice came from a test, not a preference. A local 20B model failed a real tool-calling task (9 API calls, 68 seconds, no completion, reasoning leaking into the reply). DeepSeek via OpenRouter passed the same task for a fraction of a cent, so the chain is DeepSeek, then Gemini, then a local Ollama model.
- 02Messaging moved from a self-bridged setup to a managed iMessage line, because the old gateway could see every conversation.
- 03When a job finishes, the delegate script texts the result straight back through Hermes's own send command, with no LLM in the loop, so a finished job can't be paraphrased. End to end, a simple handed-off job came back as a text in 11 seconds in testing.
- 04Delegates coding work to Claude Code through a dedicated tool, so handing off is enforced in code rather than left to the prompt. Jobs run one at a time with spend caps, retries and follow-ups that continue the same conversation.
- 05Memory is a local Hindsight server: it recalls before every turn and stores after it, across chats, so a new thread doesn't start from zero. It replaced a built-in store that had saved nothing. The first recall of each chat took 9 seconds, past an 8-second timeout, so memory was silently skipped; on an always-on server it takes 0.28 seconds.
- 06An honesty check on every reply flags claims of work that no tool actually did.
- 07I can watch and steer it from three phone apps I built: a live board of its Claude Code jobs, a homelab console, and its own dashboard behind a login gate.
- 08Grew out of an earlier, fully offline voice assistant on a Raspberry Pi.
The model choice came from a test, not a preference.
See it run
- me ›Fix the failing date test in the sync script and text me when it's green.
- hermes ›That's coding work, so I'm handing it to Claude Code.
- hermes ›Background job started: one at a time, spend cap on.
- claude code ›Reading the test, fixing the date parsing, running the suite. All tests pass.
- relay ›Job finished. Result sent as-is, with no model in the loop.
- relay ›Delivered to the same iMessage thread.
Architecture
What's inside
Core services04
- Managed iMessage line
- The messaging channel Hermes lives on.
- Model chain
- The models Hermes reasons with, in fallback order.
- Memory server
- Hermes's long-term memory, running locally.
- Claude Code
- The worker that does the actual coding.
Custom apps I built05
- Hermes
- An always-on assistant I text over iMessage, with memory, that hands real work to Claude Code.
- Claude delegate + relay
- The plumbing that turns a request into a coding job and texts the result back.
- Hermes console
- Hermes's own web dashboard on the phone, behind a login gate I wrote.
- Claude OS
- A live dashboard of every Claude Code session, with a command launcher, on my phone.
- Dead-man check
- The check that watches the watchers, from the laptop.
Phone apps
Three apps I built to run it from my phone
All three install to the home screen, live only on my private network, refuse any login but mine, and are checked every 5 minutes from the n8n Pi, which texts me through Hermes if one stops answering.
A live board of every Claude Code session on the laptop, including the ones Hermes starts.
- Sessions sorted into Working, Needs you, Idle and Ended, each with context used, cost and the last tool, updated live.
- Spend today, cost per hour, and the 5-hour and weekly limit rings, with a warning before I hit one.
- A searchable launcher for every command and skill that can run one in a chosen terminal session.
Safety: Run only types an allow-listed command into a session that already exists, after a confirm step. Nothing typed reaches a shell.
How it all connects
The homelab and Hermes are one system. Pick a part to see what it talks to, or trace a flow end to end.
- custom apps
- 8
- n8n workflows
- 4
- commands
- 10
- parts
- 41
- connections
- 72
01Devices6
Laptop
The docked laptop at home: main copy of the notes, and where Hermes and my apps run.
- Runs Hermes, Claude Code, the phone dashboards and the design processor as background services.
- Publishes its own metrics, so it is watched like every other node.
- macOS
- launchd
- Node
- tmux
- Ollama
Connects to
- Private mesh VPN · joins
- Hermes · runs
- Dead-man check · runs every 3 minutes
- Notes vault · keeps the main copy
- Design Inbox processor · runs every 10 minutes
- Claude OS · hosts
- Homelab OS · hosts
- Hermes console · hosts
- Watchdogs · alerts over the home network
NAS
Always-on storage on a mirrored pair of drives, and the Docker host for infrastructure.
- Hosts the notes-sync database and the monitoring stack, so they outlive any one laptop.
- Mirrored drives: one disk can fail without losing the notes.
- RAID 1
- Docker
Connects to
- Private mesh VPN · joins
- Encrypted notes sync · hosts
- Prometheus + Grafana · hosts
- Uptime Kuma · hosts
n8n Pi
A Raspberry Pi 5 that runs the automations and the watchdogs.
- Keeps its own copy of the notes with a headless Obsidian, so workflows can edit notes while the laptop sleeps.
- Runs the watchdog scripts, deliberately on a different machine from the one they alert through.
- Raspberry Pi 5
- Docker
- cron
Connects to
- Private mesh VPN · joins
- n8n · runs
- Headless Obsidian · runs
- Watchdogs · runs every 5 minutes
- Vault maintenance · runs link and health scripts
HA Pi
A Raspberry Pi 5 dedicated to the smart home.
- Isolates Home Assistant so automations and the house never share a failure.
- Matter devices on their own node.
- Raspberry Pi 5
- Home Assistant OS
Connects to
- Home Assistant · runs
Dashboard Pi
A Raspberry Pi 4 driving a rack touchscreen.
- A kiosk that shows fleet health at a glance, no laptop needed.
- Boots straight into the dashboards.
- Raspberry Pi 4
- Chromium kiosk
Connects to
- Prometheus + Grafana · shows the dashboards
Phone & iPad
Where I text Hermes, check the dashboards and capture ideas.
- Every app I built installs to the home screen as a web app.
- Notes stay in sync here too, end to end encrypted.
- iMessage
- Shortcuts
- Obsidian
Connects to
- Private mesh VPN · joins
- Managed iMessage line · texts Hermes; replies land in one thread
- Claude OS · home-screen app
- Homelab OS · home-screen app
- Hermes console · home-screen app
- Encrypted notes sync · LiveSync on every device
- Design inbox capture · share sheet
02Core services13
Private mesh VPN
One private network that every device joins.
- Nothing is exposed to the public internet; the apps are reachable only inside it.
- Gives the phone apps and the notes database trusted HTTPS.
- Tailscale
Connects to
- Laptop · joins
- NAS · joins
- n8n Pi · joins
- Phone & iPad · joins
Managed iMessage line
The messaging channel Hermes lives on.
- Replaced a self-bridged gateway that could see every conversation.
- Carries both chat and alerts, so everything arrives in one thread.
- iMessage
Connects to
- Phone & iPad · texts Hermes; replies land in one thread
- Hermes · messages in; replies and alerts out
- Claude delegate + relay · relays the result, no model
- Watchdogs · fallback: sends direct, tagged via Pi
Model chain
The models Hermes reasons with, in fallback order.
- DeepSeek first, then Gemini, then a local model as the offline last resort.
- Picked by a real tool-calling test, not by preference.
- OpenRouter
- DeepSeek
- Gemini
- Ollama
Connects to
- Hermes · reasons with, in fallback order
Memory server
Hermes's long-term memory, running locally.
- Recalls before each turn and stores after it.
- Stays on my hardware.
- Hindsight
Connects to
- Hermes · recalls and stores
Claude Code
The worker that does the actual coding.
- Runs headless for Hermes jobs, or in a terminal I can watch from the phone.
- Also does the read-only summarising for the design processor.
- Claude Code
- tmux
Connects to
- Claude delegate + relay · starts a background job
- Claude OS · live session board
- Design Inbox processor · summarises with read-only tools
Notes vault
My notes: the knowledge base the agents read and the workflows write.
- One vault shared by me, Hermes, n8n and the design processor.
- Each automated note has exactly one writer, so sync conflicts can't happen.
- Obsidian
- Markdown
Connects to
- Laptop · keeps the main copy
- Encrypted notes sync · LiveSync, end to end encrypted
- /done · writes the session log
- /process-inbox · files the inbox
- Design Inbox processor · writes the design hub
- /design-review · triages the design hub monthly
Encrypted notes sync
A self-hosted sync database that is the single source of truth for the vault.
- Devices sync to the database, never to each other.
- End to end encrypted; replaced a consumer cloud drive.
- CouchDB
- Self-hosted LiveSync
Connects to
- NAS · hosts
- Notes vault · LiveSync, end to end encrypted
- Headless Obsidian · Pi's copy syncs every 5 minutes
- Phone & iPad · LiveSync on every device
Headless Obsidian
Obsidian running without a screen on the n8n Pi, with a local API.
- Lets n8n edit notes the same way I would.
- A launcher restarts it within minutes if it stops.
- Obsidian
- Local REST API
Connects to
- n8n Pi · runs
- Encrypted notes sync · Pi's copy syncs every 5 minutes
- Todoist ↔ notes task sync · edits the task notes
- Design inbox capture · appends to raw captures
- Design Inbox processor · reads raw captures from the Pi
n8n
The workflow engine behind the automations.
- Runs in Docker on its own Pi.
- Every scheduled workflow is watched for going quiet.
- n8n
- Docker
Connects to
- n8n Pi · runs
- Watchdogs · checks health and quiet schedules
- Homelab OS · drills into runs
- Todoist ↔ notes task sync · runs every 6 hours
- Vault maintenance · runs daily
- Smart email sorter · runs hourly
- Design inbox capture · runs on each share
Prometheus + Grafana
Metrics from every machine every 15 seconds, and the dashboards on top.
- Custom metrics for notes-sync age and how long since each workflow last ran.
- Dashboards provisioned from files, so an update can't wipe them.
- Prometheus
- Grafana
- node_exporter
Connects to
- NAS · hosts
- Dashboard Pi · shows the dashboards
- Watchdogs · writes custom metrics
- Dead-man check · asks: is everything up?
- Homelab OS · reads host metrics
Uptime Kuma
An up/down board for every service.
- Deliberately sends no notifications: alerts have one path, through Hermes.
- The watchdogs push heartbeats to it.
- Uptime Kuma
Connects to
- NAS · hosts
- Watchdogs · pushes heartbeats
Home Assistant
The smart-home hub.
- Controllable from Homelab OS on the phone.
- Matter devices on a dedicated node.
- Home Assistant
- Matter
Connects to
- HA Pi · runs
- Homelab OS · controls
Todoist
Where I capture tasks on the go.
- Kept in step with the task lists in my notes by an n8n workflow.
- Tasks can start anywhere; the notes keep the context around them.
- Todoist API
Connects to
- Todoist ↔ notes task sync · tasks in and out
03Custom apps I built8
Hermes
An always-on assistant I text over iMessage, with memory, that hands real work to Claude Code.
- Delegation is enforced in code with a dedicated tool, not left to the prompt.
- An honesty check on every reply flags claims of work no tool actually did.
- Three-model fallback chain, plus a routing log kept as future training data.
- Hermes Agent
- launchd
- tmux
Connects to
- Laptop · runs
- Managed iMessage line · messages in; replies and alerts out
- Model chain · reasons with, in fallback order
- Memory server · recalls and stores
- Claude delegate + relay · calls its Claude Code tool
- Dead-man check · alerts through
- Hermes console · chat, sessions, models and schedules
- Design Inbox processor · texts me on failure
Claude delegate + relay
The plumbing that turns a request into a coding job and texts the result back.
- Background jobs one at a time, with spend caps, retries and follow-ups that continue the same conversation.
- The result is relayed as-is, skipping the model, so it can't be paraphrased.
- One locked-down entry point is the only way to send a message.
- Hermes plugin
- Claude Code
- bash
Connects to
- Hermes · calls its Claude Code tool
- Claude Code · starts a background job
- Managed iMessage line · relays the result, no model
- /handoff · can start a job with
- /context-budget · has run as a job
- /graphify-setup · has run headless
Watchdogs
Scripts on the n8n Pi that decide something is wrong and tell me.
- Check n8n, container restarts, the notes copy, the laptop and the phone apps every 5 minutes.
- Alert after 2 bad checks in a row and once on recovery, so one blip never pages me.
- Flag any scheduled workflow that has gone quiet, and write their own metrics.
- bash
- Python
- cron
- Prometheus
Connects to
- n8n Pi · runs every 5 minutes
- Laptop · alerts over the home network
- Managed iMessage line · fallback: sends direct, tagged via Pi
- Prometheus + Grafana · writes custom metrics
- Uptime Kuma · pushes heartbeats
- n8n · checks health and quiet schedules
Dead-man check
The check that watches the watchers, from the laptop.
- Every 3 minutes, asks Prometheus whether every Pi, the NAS and the watchdog are alive.
- If Prometheus can't be reached at all, that alone means the NAS is probably down.
- launchd
- Prometheus API
Connects to
- Laptop · runs every 3 minutes
- Prometheus + Grafana · asks: is everything up?
- Hermes · alerts through
Claude OS
A live dashboard of every Claude Code session, with a command launcher, on my phone.
- A board of sessions by state (working, needs you, idle), with context, cost and usage limits.
- Runs an allow-listed command in an existing session after a confirm step.
- Built as an observe-only Claude Code mod plus a zero-dependency web app.
- TypeScript
- Node
- Server-Sent Events
Connects to
- Laptop · hosts
- Phone & iPad · home-screen app
- Claude Code · live session board
- /done · runs from the phone
- /handoff · runs from the phone
- /checkpoint · runs from the phone
- /context-budget · runs from the phone
- /prune · runs from the phone
- /new-project · runs from the phone
- /process-inbox · runs from the phone
- /graphify-setup · runs from the phone
- /model-route · runs from the phone
Homelab OS
A phone and iPad console for the whole homelab.
- Hosts with charts and service checks every 30 seconds, with a week of history.
- n8n drill-down from workflow to run to each node's timeline.
- Home Assistant rooms, lights, scenes and automations, with container logs and restarts for the Pi.
- Guarded actions: allow-listed, confirmed, and nothing typed ever reaches a shell.
- Node
- vanilla JS
- Prometheus API
- n8n API
- Home Assistant API
Connects to
- Laptop · hosts
- Phone & iPad · home-screen app
- Prometheus + Grafana · reads host metrics
- n8n · drills into runs
- Home Assistant · controls
Hermes console
Hermes's own web dashboard on the phone, behind a login gate I wrote.
- The dashboard only answers the laptop itself, so the gate checks my login, rewrites the host and passes HTTP and WebSockets through.
- It holds API keys, so anything that isn't my login on the private network is turned away.
- Adds the home-screen icon and the icon editor shared with the other phone apps.
- Node
- WebSockets
- Reverse proxy
Connects to
- Laptop · hosts
- Phone & iPad · home-screen app
- Hermes · chat, sessions, models and schedules
Design Inbox processor
Turns links I share from the phone into summarised entries in my design hub.
- Videos are downloaded, framed and transcribed; Claude summarises with read-only tools and Python writes the note.
- Reads captures straight from the Pi, so it never waits on sync.
- After 3 failed tries it saves the bare link and texts me through Hermes.
- Python
- yt-dlp
- ffmpeg
- mlx-whisper
Connects to
- Laptop · runs every 10 minutes
- Headless Obsidian · reads raw captures from the Pi
- Claude Code · summarises with read-only tools
- Notes vault · writes the design hub
- Hermes · texts me on failure
04Commands10
/done
End-of-session wrap-up.
- Writes the session log to my notes so the next session starts with context.
- Pinned in Claude OS.
- Claude Code command
Connects to
- Claude OS · runs from the phone
- Notes vault · writes the session log
/handoff
Context-rot recovery: hands the work to a fresh session.
- Hermes jobs can start with it.
- Pinned in Claude OS.
- Claude Code command
Connects to
- Claude delegate + relay · can start a job with
- Claude OS · runs from the phone
/checkpoint
Saves the current session state to a file.
- Lets long work survive a restart.
- Pinned in Claude OS.
- Claude Code command
Connects to
- Claude OS · runs from the phone
/context-budget
Reports how the context window is being used.
- Has run as a real command inside a Hermes job.
- Pinned in Claude OS.
- Claude Code command
Connects to
- Claude delegate + relay · has run as a job
- Claude OS · runs from the phone
/prune
Shrinks context use without losing what matters.
- Keeps long sessions cheap.
- Pinned in Claude OS.
- Claude Code command
Connects to
- Claude OS · runs from the phone
/new-project
Kicks off a project end to end.
- Repo, project rules, knowledge graph, test and security baseline, and a notes hub in one go.
- Pinned in Claude OS.
- Claude Code command
Connects to
- Claude OS · runs from the phone
/process-inbox
Files the notes inbox into the right places.
- Keeps the vault tidy for the agents that read it.
- Pinned in Claude OS.
- Claude Code command
Connects to
- Claude OS · runs from the phone
- Notes vault · files the inbox
/graphify-setup
Builds a knowledge graph for a codebase.
- Has been run headless as a Hermes job.
- Pinned in Claude OS.
- Claude Code command
Connects to
- Claude delegate + relay · has run headless
- Claude OS · runs from the phone
/model-route
Picks the right model tier for the task at hand.
- Spends the expensive model only where it pays.
- Pinned in Claude OS.
- Claude Code command
Connects to
- Claude OS · runs from the phone
/design-review
The monthly triage of the design inbox.
- Closes the loop on everything the processor captured.
- Turns saved links into decisions.
- Claude Code command
Connects to
- Notes vault · triages the design hub monthly
05n8n workflows4
Todoist ↔ notes task sync
Keeps Todoist and the task lists in my notes in step, every 6 hours.
- New Todoist tasks land in the matching note; finished ones get ticked.
- Ticking a synced task in my notes closes it in Todoist.
- A reconcile pass follows date changes and deleted tasks.
- n8n
- Todoist API
- Obsidian API
Connects to
- n8n · runs every 6 hours
- Todoist · tasks in and out
- Headless Obsidian · edits the task notes
Vault maintenance
A daily pass that links related notes and runs a health check on the vault.
- Adds connections between notes so the agents can find them.
- Fixes banners and icons as it goes.
- n8n
- Python
Connects to
- n8n · runs daily
- n8n Pi · runs link and health scripts
Smart email sorter
An hourly workflow that sorts incoming email.
- Runs on a schedule, watched for going quiet like every other workflow.
- Every run is visible in Homelab OS's n8n drill-down.
- n8n
Connects to
- n8n · runs hourly
Design inbox capture
The webhook behind the phone's share-sheet shortcut.
- Checks a token, cleans the input and appends it to a raw-captures note.
- Works even when the laptop is off.
- n8n
- Webhook
- Obsidian API
Connects to
- n8n · runs on each share
- Phone & iPad · share sheet
- Headless Obsidian · appends to raw captures
Trace a flow
Text Hermes, code ships
- 01Phone & iPadI text Hermes from my phone, like any other chat.
- 02Managed iMessage lineThe managed iMessage line carries it to Hermes on the laptop.
- 03HermesHermes's model sees coding work and calls its Claude Code tool. I can also skip the model and type /claude.
- 04Claude delegate + relayThe delegate starts a background job, one at a time, with a spend cap.
- 05Claude CodeClaude Code does the work, headless.
- 06Claude delegate + relayWhen the job finishes, the relay picks up the result.
- 07Managed iMessage lineIt goes out as-is, skipping the model, so a finished job can't be paraphrased.
- 08Phone & iPadThe result lands in the same thread.
Something breaks at 3am
- 01Dead-man checkEvery 3 minutes, a dead-man check on the laptop asks Prometheus whether every Pi, the NAS and the watchdog are alive.
- 02Prometheus + GrafanaPrometheus on the NAS answers. If it can't be reached at all, that alone says the NAS is probably down.
- 03WatchdogsOn the n8n Pi, watchdogs check n8n, containers, the notes copy and the laptop every 5 minutes, and alert after 2 bad checks in a row.
- 04LaptopThe alert goes over the home network to the laptop. If the laptop or Hermes can't deliver, the Pi sends straight to the iMessage line, tagged via Pi.
- 05HermesHermes passes it on through its send path. No model touches it.
- 06Managed iMessage lineOut through the managed iMessage line.
- 07Phone & iPadOne message when something breaks, one when it recovers.
A task syncs everywhere
- 01TodoistI add or finish a task in Todoist.
- 02Todoist ↔ notes task syncEvery 6 hours, an n8n workflow picks up the change. Ticking a synced task in my notes closes it in Todoist too.
- 03Headless ObsidianIt edits the task note through headless Obsidian on the n8n Pi.
- 04Encrypted notes syncThe Pi's copy syncs to the encrypted database on the NAS every 5 minutes.
- 05Phone & iPadLaptop, phone and iPad pull it in through LiveSync.
Save a design idea
- 01Phone & iPadI share a link from the phone or iPad to the Design Inbox shortcut.
- 02Design inbox captureAn n8n webhook checks the token and cleans the input. This works even when the laptop is off.
- 03Headless ObsidianIt appends the link to a raw-captures note. n8n is that note's only writer.
- 04Design Inbox processorEvery 10 minutes, the laptop's processor reads the captures straight from the Pi.
- 05Claude CodeVideos are transcribed; Claude summarises with read-only tools.
- 06Design Inbox processorPython turns the summary into a structured entry.
- 07Notes vaultIt lands in the design hub's inbox, and syncs to every device.
Results
| Before | 9 s |
|---|---|
| After | 0.28 s |
- Running live, with memory that carries across chats.
- Finished coding jobs come back as a text, word for word; a simple one took 11 seconds end to end in testing.
- Memory recall per new chat: 9 seconds down to 0.28.
- Every incident gets a written root cause; three are below, each now with a guard that stops it happening again.
What broke, and the fix
Incident 01
The fallback that never fell back
FixProvider names are now validated against the real list.
What happenedHideImpactThe local fallback model never worked for the first weeks.
Blind spotThe provider name was invalid, and the config failed over silently.
FoundThe CLI's doctor command.
FixProvider names are now validated against the real list.
Incident 02
A message storm
FixA restart-loop breaker, and a rule never to restart mid-conversation.
What happenedHideTriggerDuplicate webhooks, self-chat, and redelivery after restarts.
ImpactThe same messages arrived again and again.
FixA restart-loop breaker, and a rule never to restart mid-conversation.
Incident 03
An update that broke tool calls
FixRolled back. Updates now happen at the start of a session and are re-verified.
What happenedHideTriggerA 739-commit upstream update.
ImpactTool-call output broke.
FixRolled back. Updates now happen at the start of a session and are re-verified.
What I'd do next
- Spawn other agents and talk to them directly, not only to Hermes.
- Connectors for Google Workspace, Todoist and Home Assistant, with approval required before anything is sent or changed.