Skip to content

Latest commit

 

History

History
432 lines (353 loc) · 16.8 KB

setup.md

File metadata and controls

432 lines (353 loc) · 16.8 KB

Setup

Prerequisites

  • Elixir 1.19+ / Erlang OTP 27+ — brew install elixir on macOS.
  • Postgres 14+ with the pgvector extension.
    brew install postgresql@18 pgvector
    brew services start postgresql@18
  • An OpenAI-compatible or Anthropic-compatible LLM endpoint. Three required env vars wire the chat client up — see "First boot" below for worked examples.
    • LLM_API_KEY — bearer token sent as Authorization: Bearer ….
    • LLM_API_BASE — base URL the provider request hangs off (e.g. https://api.openai.com/v1, https://api.kimi.com/coding/v1, https://api.deepseek.com/v1, https://api.minimax.io/anthropic, http://localhost:11434/v1 for Ollama, …).
    • LLM_MODEL — model id sent in the request body (e.g. gpt-4o-mini, kimi-for-coding, deepseek-chat, MiniMax-M3).
    • (optional) LLM_API_FORMAT — openai (default) calls /chat/completions; anthropic calls /v1/messages under LLM_API_BASE and uses Anthropic content/tool/thinking blocks.
    • (optional) LLM_USER_AGENT (default neoharness/0.1).
    • (optional) LLM_THINKING_STYLE — anthropic to emit {"thinking": {"type": "enabled" | "disabled"}} (Kimi, DeepSeek reasoning), openai to emit reasoning_effort: "medium" | "minimal" (gpt-5 / o-series), unset/none to drop the field. This only affects LLM_API_FORMAT=openai; the Anthropic adapter maps harness thinking to {"thinking": {"type": "adaptive" | "disabled"}}.
    • (optional) LLM_MAX_TOKENS — response-token cap for both API formats. Anthropic-compatible APIs require a value and default to 4096 when omitted; OpenAI-compatible APIs omit the field when unset and use the provider default. Set 8192 or 16384 when your model supports longer visible replies.
    • (optional) LLM_PROVIDER_LABEL — string surfaced as llm.provider in OpenTelemetry spans (default openai-compat).
  • (optional) OPENAI_API_KEY for (a) semantic memory search with text-embedding-3-small, (b) voice-note transcription via Whisper (whisper-1 by default; override with OPENAI_WHISPER_MODEL), and (c) OpenAI-backed image_generate / image_edit. Without the key, memory falls back to Postgres full-text search, inbound voice notes are surfaced to the model as [voice message — transcription unavailable] markers, and OpenAI image tools hide themselves.
  • (optional) MINIMAX_API_KEY for MiniMax media tools: image_generate provider=minimax, voice_generate, and music_generate. If unset and LLM_API_BASE already points at MiniMax, these tools reuse LLM_API_KEY.
  • (optional) MAPTILER_API_KEY for /atlas map tiles (client-side MapLibre rendering) and the place_lookup tool. Atlas extracts place names from persona memories and geocodes them with MapTiler when you click "Refresh from memories"; place_lookup geocodes a single query and renders a web chat place card. Without a key, /atlas falls back to the decorative SVG. Korea-first defaults are ATLAS_GEOCODING_COUNTRY=kr and ATLAS_GEOCODING_LANGUAGE=ko,en; set ATLAS_GEOCODING_PROXIMITY=<lon>,<lat> to bias ambiguous searches around a home/work area. Optional ATLAS_MAP_TILES_KEY (a separate, domain-restricted browser key for quota isolation — falls back to MAPTILER_API_KEY) and ATLAS_MAP_STYLE (default dataviz-dark) override the tiles key and map style respectively.
  • (optional) TELEGRAM_BOT_TOKEN, DISCORD_BOT_TOKEN for chat connectors. Unset → connectors silently skip boot.
  • (optional) SCHEDULE_TIMEZONE — IANA zone (e.g. America/Los_Angeles) used to interpret every cron expression in schedules.jsonc. Default Etc/UTC. Restart to apply.
  • The built-in Web connector (the Phoenix LiveView chat) is always on — no external service, no supervisor child. To disable it, config :neoharness, Neoharness.Connectors.Web, enabled: false in config/config.exs; ask_user will then skip the :web branch when deciding where to deliver questions.

First boot

cp .env.example .env
# edit .env — LLM_API_KEY, LLM_API_BASE, LLM_MODEL are required

mix deps.get
mix ecto.setup      # creates the db and runs every migration
iex -S mix

config/runtime.exs auto-loads .env from the project root at boot via Dotenvy — no source .env needed. Anything already exported in your shell wins, so prod / CI env vars override the file. Dotenvy handles the usual dialect (export prefix, quoted values, # comments, multiline values).

Worked env-var blocks:

# OpenAI
export LLM_API_KEY=sk-...
export LLM_API_BASE=https://api.openai.com/v1
export LLM_API_FORMAT=openai
export LLM_MODEL=gpt-4o-mini
export LLM_THINKING_STYLE=openai
export LLM_PROVIDER_LABEL=openai

# MiniMax M3 (Anthropic-compatible, recommended for interleaved thinking/tool use)
export LLM_API_KEY=...
export LLM_API_BASE=https://api.minimax.io/anthropic
export LLM_API_FORMAT=anthropic
export LLM_MODEL=MiniMax-M3
export LLM_MAX_TOKENS=8192
export LLM_CONTEXT_CUTOFF_TOKENS=800000
export LLM_PROVIDER_LABEL=minimax-m3
# Optional MiniMax media generation:
export IMAGE_GENERATION_PROVIDER=minimax
export MINIMAX_API_KEY=...          # can match LLM_API_KEY
export MINIMAX_VOICE_ID=...

# Optional Memory Atlas geocoding and map tiles:
export MAPTILER_API_KEY=...
export ATLAS_GEOCODING_COUNTRY=kr
export ATLAS_GEOCODING_LANGUAGE=ko,en
export ATLAS_GEOCODING_PROXIMITY=127.0276,37.4979
# Optional Atlas map (browser tiles); falls back to MAPTILER_API_KEY:
export ATLAS_MAP_TILES_KEY=your-domain-restricted-key
export ATLAS_MAP_STYLE=dataviz-dark

# Kimi for Coding
export LLM_API_KEY=...
export LLM_API_BASE=https://api.kimi.com/coding/v1
export LLM_API_FORMAT=openai
export LLM_MODEL=kimi-for-coding
export LLM_USER_AGENT=claude-code/2.1.118
export LLM_THINKING_STYLE=anthropic
export LLM_MAX_TOKENS=8192
export LLM_PROVIDER_LABEL=kimi-for-coding

On a healthy boot you'll see something like:

MCP: 0 servers, 0 tools
skills: loaded 0 from /Users/you/.neoharness/skills
Running NeoharnessWeb.Endpoint with cowboy 2.14.2 at 127.0.0.1:4000

From here:

  • Open http://localhost:4000 for the web UI.
  • From iex: Neoharness.say("hello") talks to the default main persona.
  • http://localhost:4000/docs — these guides rendered as a browsable site (the markdown files are bundled into the app at compile time, so the same URL works from a release).
  • http://localhost:4000/dashboard — Phoenix LiveDashboard. Tabs include Home, Metrics, Processes, Ports, OS Data (via :os_mon — CPU, memory, disk), Request Logger (enable a per-session log with a URL param), Ecto Stats (via ecto_psql_extras — slow queries, index usage, cache hit rate).
  • http://localhost:4000/oban — Oban Web for the job queue (heartbeats, schedules).
  • http://localhost:4000/errors — ErrorTracker. Every captured exception (web requests, Oban jobs, and agent/subagent task crashes) grouped with occurrence counts and stacktraces, stored in Postgres.

Multimodal attachments

Inbound images, video, voice notes, and files land under priv/uploads/<conv_id>/<sha256>.<ext> (gitignored) and are served back to the web UI via Plug.Static at /uploads/. Single-host storage — fine for local dev, swap for object storage if you go multi-host.

The web composer has a paperclip button (file picker) and a microphone button driven by MediaRecorder in the browser. Telegram and Discord ingest media inbound automatically (photo / voice / video / video_note / document / animation on Telegram; any attachment on Discord). Outbound: the agent uses send_image / send_file tools to attach media to its replies — see docs/tools.md.

Generated media follows the same path. image_generate can use either OpenAI or MiniMax and returns a send_image-ready path. MiniMax voice_generate and music_generate return audio paths that should be delivered with send_file.

Disabling vision or video

If your model does not support images or videos, set the matching env var to false in .env:

LLM_SUPPORTS_IMAGE=false    # rejects inbound images, strips them from history
LLM_SUPPORTS_VIDEO=false    # rejects inbound videos, strips them from history

Both default to true. When a flag is false:

  • Fresh uploads of that media type are rejected immediately with a user-visible message (flash on web, reply on Telegram/Discord).
  • Old messages in history are rebuilt as text markers ([image: name]) instead of multimodal blocks, so the LLM never sees content it can't handle.
  • If you leave a flag as true but the model still rejects the media, the turn surfaces a normal error — it's your responsibility to match the flags to the model's actual capabilities.

Audio and files are always allowed; they are downgraded to text markers automatically.

Provider Files API (videos on Kimi)

Most providers accept inline base64 for both images and videos in image_url / video_url content blocks. Moonshot/Kimi is the exception: it rejects inline base64 video with invalid part type: video_url and requires the asset to be uploaded via its Files API first, then referenced by ms://<file_id> in the chat message. Set:

LLM_FILES_API=moonshot

When this is set, every user turn that carries a video attachment is pre-uploaded to POST <LLM_API_BASE>/files (purpose=video), the returned file_id is cached in an ETS table by sha256 for the BEAM lifetime so re-sends of the same video skip the round-trip, and Message.to_api/1 emits the ms://<id> reference instead of base64. Upload failures fall back to inline base64 (which Kimi will then reject — but the failure is logged and surfaced).

Leave LLM_FILES_API unset for OpenAI / DeepSeek / MiniMax M3's Anthropic path / providers that handle inline base64 — videos go straight into the request body. MiniMax M3's Anthropic adapter emits image / video blocks with base64 source objects.

Config directories

Neoharness reads user-owned config from ~/.neoharness/:

~/.neoharness/
├── skills/                # skills shared by every persona (optional)
├── mcp.json               # MCP servers (optional)
└── agents/
    └── <persona>/         # one directory per agent — at least one must exist
        ├── SOUL.md        # personality
        ├── schedules.jsonc  # heartbeats / cron (optional)
        ├── connectors.jsonc # which telegram chats / discord channels this persona listens on
        └── skills/        # persona-scoped skills (optional)

Every agent is a persona at agents/<name>/ — there is no top-level SOUL.md/schedules.jsonc fallback. On first boot, neoharness auto-creates agents/main/ with placeholder SOUL.md, an empty connectors.jsonc, and a schedules.jsonc containing a 30-minute heartbeat (tick). Edit any of those files; the next turn (or next heartbeat tick) sees the changes.

Optional starter files for the shared root:

cp examples/mcp.json ~/.neoharness/

Verifying reliability

# Each persona's agent is supervised — kill it and the supervisor restarts it.
:sys.terminate(Process.whereis(Neoharness.Agent.Manager.via("main:main")), :kaboom)
Neoharness.say("still alive?")  # works

Deployment

Neoharness is single-tenant by design: one BEAM process, one Postgres, one user. Deployment means one VPS per user — friends who want to try it stand up their own.

The shipped path is mix release on a small Debian/Ubuntu VPS joined to a Tailscale tailnet. No public DNS, no public ports, no Let's Encrypt, no in-app auth. The tailnet is the perimeter: only people you've invited can reach the box at all, and traffic is WireGuard-encrypted end-to-end. For sharing with a friend, you invite them to the tailnet and hand them a http://<host>.<tailnet>.ts.net URL.

The release runs as your VPS user, so shell_exec and the rest of the host-touching tooling work as designed — the agent has a real Linux environment to operate in.

Sample artifact: rel/neoharness.service.example (systemd unit).

VPS quick start (Debian/Ubuntu, Tailscale)

A 1 vCPU / 2 GB box is plenty.

1. Provision the OS dependencies.

# Postgres 18 with pgvector. apt's `postgresql` typically lags; use
# the postgres.org apt repo if you want pg18 specifically.
sudo apt-get install -y postgresql postgresql-18-pgvector
sudo -u postgres createuser neoharness --pwprompt
sudo -u postgres createdb -O neoharness neoharness_prod

# Erlang/Elixir to build the release. Use asdf / mise / distro
# packages — anything that gives you Elixir 1.19+ / OTP 27+.
sudo apt-get install -y build-essential git curl libssl-dev libncurses5-dev

# Tailscale.
curl -fsSL https://tailscale.com/install.sh | sh

# Application user.
sudo useradd --create-home --shell /bin/bash neoharness

2. Join the tailnet.

sudo tailscale up --hostname=neoharness --ssh

Authorize the device in the Tailscale admin console, then note the MagicDNS name (neoharness.<your-tailnet>.ts.net). Confirm:

tailscale ip -4
tailscale status

3. Build the release on the VPS.

sudo -iu neoharness
git clone https://github.com/<you>/neoharness.git ~/build
cd ~/build
mix deps.get --only prod
MIX_ENV=prod mix release
mkdir -p /opt/neoharness/current
cp -a _build/prod/rel/neoharness/. /opt/neoharness/current/

Re-run those last four lines + systemctl restart neoharness to upgrade. There's no hot-code-load story; restarts are cheap.

4. Write /opt/neoharness/.env.

LLM_API_KEY=...
LLM_API_BASE=https://api.openai.com/v1
LLM_MODEL=gpt-4o-mini

DATABASE_URL=ecto://neoharness:<password>@localhost/neoharness_prod
SECRET_KEY_BASE=<mix phx.gen.secret 64>
SIGNING_SALT=<mix phx.gen.secret 32>

# MagicDNS name from `tailscale status`. Used for absolute URLs the
# LiveView generates — must match what friends type in.
PHX_HOST=neoharness.<your-tailnet>.ts.net
PORT=4000
sudo chown neoharness:neoharness /opt/neoharness/.env
sudo chmod 600 /opt/neoharness/.env

5. Lock the firewall down to the tailnet.

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow in on tailscale0
sudo ufw allow in on tailscale0 to any port 4000
sudo ufw allow 22/tcp     # drop this if you only SSH via `tailscale ssh`
sudo ufw enable

The endpoint binds 0.0.0.0:4000, but the only interface that can reach it is tailscale0. The public IP answers nothing on 4000.

6. Wire up systemd.

sudo cp /opt/neoharness/current/rel/neoharness.service.example \
        /etc/systemd/system/neoharness.service
sudo systemctl daemon-reload
sudo systemctl enable --now neoharness
journalctl -u neoharness -f   # expect "Running NeoharnessWeb.Endpoint"

Neoharness.Release.migrate/0 runs at the top of bin/start, so the init migration applies automatically on first boot (and any future ones on upgrade).

7. Visit it.

From any device on your tailnet: http://neoharness.<your-tailnet>.ts.net:4000. LiveDashboard at /dashboard and Oban Web at /oban are reachable on the same URL.

Release troubleshooting

On boot, neoharness checks that the release can load core Elixir protocol implementation modules such as Collectable.BitString and Inspect.PID. If startup fails with missing required Elixir protocol implementation modules, rebuild the release and recopy the complete _build/prod/rel/neoharness/ directory. A partial or mismatched release copy can otherwise surface later as noisy FORMATTER CRASH logs from System.cmd/3 or Logger instead of a clear startup failure.

Sharing with friends

In the Tailscale admin console, invite the friend's email to your tailnet (or share the node directly via "Share node"). They install Tailscale on their device, accept the invite, and the MagicDNS URL just resolves for them. Revoking access is the inverse: remove them from the tailnet, the URL stops resolving.

There's still no in-app authorization, so anyone with tailnet access to the box drives the same agent — they see your conversations, can trigger your tools, etc. If you want isolated state per friend, give each friend their own VPS; the codebase is single-tenant.

Backups

State that survives a rebuild lives in two places:

  • The Postgres database (messages, memories, conversations, schedules, Oban jobs).
  • /home/neoharness/.neoharness/ (per-persona SOUL.md, schedules.jsonc, per-persona skills, MCP config).

A nightly pg_dump of neoharness_prod plus a tar of ~/.neoharness to an off-box destination is the minimum survivor set. priv/uploads/ is recoverable from message history if you don't back it up; back it up if you care about exact bytes.