A.M.P. Troubleshooting Guide
When (not if) something goes wrong — Ala Malū Privacy Field Reference
Page 1 of 4 · Revised 2026-08-21
Quick Reference — Key Addresses & Log Files
Blue Triangle — BT (Cloud Hub · DigitalOcean NYC1)
# SSH into Blue Triangle ssh -i ~/.ssh/amp-deploy [email protected] # Check all running containers docker ps # Tail dashboard API logs docker logs -f amp-api # Tail Telegram bridge logs docker logs -f amp-bridge # Bridge health check curl http://100.113.114.32:5001 # Check relay failover state curl http://100.107.129.93:5001/relay-failover
Sea-Relay (Subscriber WireGuard · Vultr Seattle)
# SSH into Sea-Relay ssh -i ~/.ssh/amp-deploy [email protected] # Subscriber WireGuard (wg0 on :443) wg show wg0 latest-handshakes # Check puka back-tunnel handshakes for i in 20 21 22 23 24 25 26 27; do echo -n "wg$i: "; wg show wg$i latest-handshakes 2>/dev/null | awk '{age=systime()-$2; print age"s"}' || echo "not up" done # Check relay failover state cat /var/lib/amp/relay-failover-state.json | python3 -m json.tool # Check relay API curl http://localhost:5001/wg-handshakes curl http://localhost:5001/relay-failover # UFW status (must have 443/udp open) ufw status
Puka Nodes (Subscriber Exit Nodes)
# SSH via Tailscale (amp-deploy key) ssh -i ~/.ssh/amp-deploy pi@<tailscale-ip-of-puka> # Check WireGuard status (back-tunnel to sea-relay) sudo wg show wg0 # Check back-tunnel handshake age sudo wg show wg0 latest-handshakes # Restart WireGuard back-tunnel sudo systemctl restart wg-quick@wg0 # Check heartbeat log tail -20 /var/log/amp-heartbeat.log # Check serial and WireGuard conf cat /etc/amp-serial cat /etc/wireguard/wg0.conf # AllowedIPs must be 10.8.0.0/16 (not /24) grep AllowedIPs /etc/wireguard/wg0.conf
NodeLocation / OwnerTailscale IPStatusWG Back-Tunnel
BTBlue Triangle · DO NYC1100.113.114.32HubN/A
Sea-RelaySea-Relay · Vultr Seattle100.107.129.93Relaywg0 :443 subscriber, wg20–27 pukas
Node 20Leonard · Kent, WA (Alpha)100.110.69.51Activesea-relay wg20 :51840
Node 21Papa/Ronie · Waimea, HI100.116.250.85Activesea-relay wg21 :51841
Node 22Rylen · Nanakuli, HI100.86.244.86Not Deployedsea-relay wg22 :51842
Node 23Tammy · Waianae, HI100.73.221.51Not Deployedsea-relay wg23 :51843
Node 24Chester · Puyallup, WA100.110.102.89Activesea-relay wg24 :51844
Node 25Junas · Waipahu, HI100.68.72.59Activesea-relay wg25 :51845
Node 26Lenette · Beaverton, OR100.73.178.80On Holdsea-relay wg26 :51846
Node 27Jason · Kent, WA100.105.202.0Activesea-relay wg27 :51847

1 — Spoke Node Issues (Hawaii Pis)

🔴Telegram alert received: "Node Offline" — node hasn't checked in for 10+ minutes
  1. Check if it's a power issue — ask the family member if the Pi's LEDs are on. Green LED = running; no lights = unplugged or dead power supply.
  2. Check Tailscale from your laptop — if the node is still in the mesh, it has internet:
tailscale status # Look for the node's hostname (AMP...). If it shows "offline" → internet/Pi issue
  1. If Tailscale shows online but heartbeat is still missing — SSH in and check the cron:
ssh pi@<tailscale-ip> crontab -l # Confirm "*/5 * * * * /usr/local/bin/amp-heartbeat.sh" is present cat /var/log/amp-heartbeat.log | tail -20 # Look for errors sudo /usr/local/bin/amp-heartbeat.sh # Run manually to test
  1. If the node is truly unreachable — have the family member unplug it, wait 15 seconds, plug back in. The heartbeat cron will resume automatically. Alert will auto-clear when BT receives the next heartbeat.

Alert auto-clears when heartbeat resumes. No manual database cleanup needed — the alert.py script handles recovery detection automatically.

🔑Can't SSH into a spoke node
  1. Confirm you're on a Tailscale-connected network (your laptop must be on the mesh).
  2. Find the node's Tailscale IP: tailscale status — look for the AMP hostname.
  3. Try Tailscale SSH directly: tailscale ssh pi@AMPxxxxxxxx
  4. If that fails, check Tailscale service on the spoke: ask family to reboot the Pi, which restarts tailscaled automatically.
  5. If still unreachable — the Pi is likely powered off or lost internet. See "Node Offline" scenario above.

Password auth is disabled. Only SSH keys work (Leonard's key + BT deploy key). Never share SSH passwords — there are none.

🌐Tailscale is not connected on the spoke
  1. SSH in via local network if possible (connect to family's WiFi) or ask them to hold the Pi near the router.
  2. Check Tailscale status and logs:
sudo systemctl status tailscaled sudo tailscale status sudo journalctl -u tailscaled -n 50
  1. If "needs login" or "not authorized" — the auth key expired. Generate a new reusable auth key at tailscale.com/admin/settings/keys and re-authenticate:
sudo tailscale up \ --authkey=tskey-auth-xxxxx \ --advertise-exit-node \ --advertise-tags=tag:exit-node
  1. If tailscaled won't start: sudo systemctl restart tailscaled — then re-run step 3.
⏰Heartbeat cron is not sending (log is old or missing)
  1. Check if the cron entry exists: crontab -l (should show */5 * * * * /usr/local/bin/amp-heartbeat.sh)
  2. Check if cron service is running: sudo systemctl status cron
  3. Test the heartbeat script manually:
sudo /usr/local/bin/amp-heartbeat.sh # Check what it outputs — look for WAN IP or error messages
  1. If /etc/amp-serial is missing or empty, the heartbeat can't identify itself. Re-run first-boot if needed (see below).
  2. Reinstall cron entry if missing:
(crontab -l 2>/dev/null; echo "*/5 * * * * /usr/local/bin/amp-heartbeat.sh") | crontab -
A.M.P. Troubleshooting Guide
First-Boot Problems · Blue Triangle · Telegram Bridge · Registration Conflicts
Page 2 of 4 · Revised 2026-08-21

2 — First-Boot Problems

🥾First boot didn't complete — node has no serial, no Tailscale, no heartbeat
  1. SSH in locally (connect to family's WiFi, or use default pi/raspberry credentials if it's a fresh image). Check the log:
cat /var/log/amp-firstboot.log # Look for the last successful line and where it stopped
  1. Check if the guard file exists — if it does, the script thinks it already ran:
ls -la /etc/amp-firstboot-done # Exists = script won't re-run
  1. If the script stopped at Tailscale setup — the auth key file was missing or wrong:
ls /boot/amp-tailscale.key # Must exist with a valid key # If missing, create the key file: echo "tskey-auth-xxxxx" | sudo tee /boot/amp-tailscale.key
  1. To re-run first-boot completely (resets everything):
sudo rm /etc/amp-firstboot-done # Remove guard flag sudo bash /usr/local/bin/first-boot.sh # Re-run (or reboot)

Removing the guard file and rebooting is safe for a fresh Pi — it won't create a duplicate serial if the MAC isn't registered yet. If it IS already registered, the API returns 200 OK (idempotent).

💥Registration returned 409 CONFLICT — serial collision detected
  1. This means the MAC address was already registered under a different serial — usually because the SD card was re-flashed and the Pi booted fresh, generating a new salt.
  2. Check the audit_logs table on Blue Triangle:
ssh [email protected] docker exec -it amp-db psql -U amp -d ampdb -c \ "SELECT * FROM audit_logs WHERE hardware_mac = '<MAC>' ORDER BY timestamp DESC LIMIT 10;"
  1. Decide: do you want to keep the old serial or accept the new one?
    • Keep old serial: Delete the new record from gateways and reinstall the original serial file on the Pi: echo "AMPXXXXXXXX" | sudo tee /etc/amp-serial
    • Accept new serial: Update the gateways row with the new serial. Update /etc/amp-serial on the Pi to match.

3 — Blue Triangle (Cloud Hub) Issues

🔴Blue Triangle API unreachable — spokes can't register or send heartbeats
  1. SSH into BT (you must be on Tailscale): ssh [email protected]
  2. If SSH fails, BT may be down. Open DigitalOcean console in browser → Power → Power cycle the droplet.
  3. Once SSH works, check Docker containers:
docker ps -a # All containers should show "Up". If any show "Exited": docker start amp-dashboard docker start amp-db # Or restart all: cd /opt/amp && docker compose up -d
  1. Check if the API is responding locally on BT:
curl http://127.0.0.1:5000/api/health # Should return {"status":"ok"} — if not, check amp-dashboard logs docker logs --tail=50 amp-dashboard
  1. Check Tailscale on BT is still connected: tailscale status
🐳Docker container is down or crashing in a restart loop
docker ps -a # Check exit codes and status docker logs --tail=100 <container> # Read the error docker inspect <container> | grep -A5 '"State"' # Check restart count
cd /opt/amp && docker compose down && docker compose up -d # Full restart
🌊Telegram war room flooded with hundreds of tailscale_restart / reboot alerts after BT recovery

This happens when BT has been offline for hours — pukas queue events locally and flush everything at once when BT comes back, overwhelming the API's 2 Gunicorn workers.

  1. Snooze puka alerts immediately (works via aikai over LAN, no Tailscale needed):
ssh -i ~/.ssh/id_ed25519 [email protected] python3 -c "import json,time; open('/tmp/amp-snooze.json','w').write(json.dumps({'snoozed':True,'until_ts':int(time.time())+7200,'until':'2h from now'}))" # Or type /snooze 120 in the war room if bridge is responsive
  1. Block Telegram outbound from BT to stop the spam immediately:
ssh [email protected] iptables -I FORWARD -d 149.154.160.0/20 -j DROP iptables -I FORWARD -d 91.108.4.0/22 -j DROP
  1. Clear event queues on all pukas to stop new events being sent:
for IP in 100.110.69.51 100.116.250.85 100.86.244.86 100.73.221.51; do ssh -i ~/.ssh/amp-deploy pi@$IP "sudo rm -f /tmp/amp-events-queue" done
  1. Wait for API to clear (poll for HTTP 200), then unblock Telegram:
until curl -s --max-time 5 http://172.19.0.3:5000/api/gateways -o /dev/null -w '%{http_code}' | grep -q 200; do sleep 10; done iptables -D FORWARD -d 149.154.160.0/20 -j DROP iptables -D FORWARD -d 91.108.4.0/22 -j DROP

If puka processes are stuck mid-flush (SSH to pukas also timing out), wait it out — they will drain naturally in 15–30 min. The event queue is finite.

📊Hōʻike (hoike.alamalu.com:5000) returns connection timeout / HTTP 000
  1. Check if the API is responding at all:
ssh [email protected] curl -s --max-time 8 http://172.19.0.3:5000/api/gateways -o /dev/null -w '%{http_code}' # HTTP 200 = API fine, DNS/proxy issue. HTTP 000 = API overwhelmed or down.
  1. If API is overwhelmed (usually after event flood) — check logs and restart:
docker logs amp-api --tail=20 # Look for "WORKER TIMEOUT" or OOM errors cd /opt/amp/blue-triangle && docker compose restart api sleep 15 && curl -s --max-time 8 http://172.19.0.3:5000/api/gateways -o /dev/null -w '%{http_code}'
  1. If Docker containers can't reach internet after any iptables flush — rebuild Docker NAT rules:
systemctl restart docker # Rebuilds iptables NAT — containers can reach internet again # Verify: docker exec amp-api python3 -c "import urllib.request; print(urllib.request.urlopen('https://api.telegram.org',timeout=5).status)"
  1. If Hōʻike loads but shows stale data — check heartbeats are flowing:
docker exec amp-db psql -U amp -d amp -c "SELECT node_id, last_telemetry_seen FROM gateways ORDER BY node_id"
🔗wg_aikai failover tunnel won't establish handshake with aikai
  1. Check BT's wg_aikai config has Table = off (CRITICAL — without it, all BT outbound traffic routes through aikai, breaking internet for all Docker containers):
cat /etc/wireguard/wg_aikai.conf | grep -E 'Table|Address|ListenPort' # Must show: Table = off # If missing: wg-quick down wg_aikai, add "Table = off" under [Interface], wg-quick up wg_aikai
  1. Verify aikai's endpoint uses BT's PUBLIC IP (not Tailscale) — Tailscale ACLs block UDP 51845:
ssh -i ~/.ssh/id_ed25519 [email protected] sudo grep Endpoint /etc/wireguard/wg_failover.conf # Must show: Endpoint = 159.65.218.220:51845 (NOT 100.113.114.32) # Fix: sudo sed -i 's/100.113.114.32/159.65.218.220/' /etc/wireguard/wg_failover.conf sudo systemctl restart wg-quick@wg_failover
  1. Verify UDP 51845 is open in DO cloud firewall (Networking → Firewalls → amp-no-front-door).
  2. Check handshake on both sides:
wg show wg_aikai latest-handshakes # On BT — should be < 180 seconds ago ssh [email protected] "sudo wg show wg_failover latest-handshakes" # On aikai

4 — Telegram Bridge Not Responding (Ala-C / Ala-G silent)

✈Sent a Telegram message and got no response from Ala-C or Ala-G
  1. Quick health check — if this returns anything other than {"status":"up"}, the bridge is down:
curl http://100.113.114.32:5001 # Also check from BT itself to rule out network: ssh [email protected] "curl http://127.0.0.1:5001"
  1. Check bridge container status and restart it:
ssh [email protected] docker ps | grep bridge # Is it running? docker logs --tail=50 amp-bridge # Look for errors docker restart amp-bridge # Restart it
  1. If the bridge logs show "403 Forbidden" from Telegram — a bot token has expired or been revoked. Generate a new token in BotFather and update /opt/amp/.env, then restart.
  2. If the bridge logs show Anthropic API errors — check your API key balance/status at console.anthropic.com
  3. If the bridge is running but only ONE bot responds — one bot token is bad. Check which one is failing in the logs and fix that token.

/clear in the group chat resets the in-memory conversation history. Useful if the bots are giving confused or looping responses.

A.M.P. Troubleshooting Guide
Client Device Issues · Nessus · Subscriber Portal · Nuclear Recovery
Page 3 of 4 · Revised 2026-08-21

5 — Family Device / Subscriber Issues

📱Family member's device isn't routing through the exit node — no privacy protection
  1. On their phone/laptop — open the Tailscale app. Check that:
    • Tailscale is connected (toggle is ON)
    • "Use exit node" is selected, and their home Pi is chosen (e.g., AMP56F020949F)
  2. Confirm the Pi itself is advertising as an exit node:
ssh pi@<spoke-tailscale-ip> sudo tailscale status # Should show: offers exit node # If not advertising, re-run: sudo tailscale up --advertise-exit-node --advertise-tags=tag:exit-node
  1. On the Tailscale admin console (tailscale.com/admin) — confirm the exit node is "Approved". New exit node advertisements must be manually approved.
  2. Verify internet is routing through the Pi: have the family member visit api.ipify.org — their IP should match the Pi's WAN IP shown in the dashboard.
🐢Internet is slow when connected through the exit node
  1. Check the Pi's load and temperature (they're fanless — heat causes throttling):
ssh pi@<spoke-tailscale-ip> vcgencmd measure_temp # Pi temperature — should be under 65°C top -bn1 | head -5 # CPU load — should be under 60% cat /proc/net/dev # Network throughput counters
  1. If temperature is high (>75°C) — Pi is throttling. Check that the Argon NEO case heatsink is properly seated and the unit has airflow around it. Avoid placing in enclosed spaces.
  2. Test the Pi's own internet speed (eliminates the home ISP as the bottleneck):
curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python3 -
  1. If Pi speed is fine but VPN is slow — check Tailscale relay vs. direct connection:
sudo tailscale ping <client-tailscale-ip> # "via DERP" = relayed (slow). "direct" = direct UDP (fast) # Direct should form within 30 seconds after both sides are connected
🌐Can't reach puka.alamalu.com or setup.alamalu.com
  1. Check NPM (Nginx Proxy Manager) is running on BT: docker ps | grep proxy
  2. Check NPM proxy host configuration at http://100.113.114.32:81 (NPM admin UI)
  3. Check DNS: nslookup puka.alamalu.com — should point to Blue Triangle's public IP
  4. Check SSL cert status in NPM — expired certs will show certificate errors. Use NPM to renew Let's Encrypt certs.
ssh [email protected] docker logs --tail=30 amp-proxy # Look for proxy errors docker restart amp-proxy # Restart if necessary

6 — Nessus Scan Issues

🔬Nessus scan fails to reach the Pi or returns incomplete results
  1. Nessus runs on Blue Triangle at https://100.113.114.32:8834 — must be on Tailscale to access.
  2. The scan target is the Pi's local IP (on Kent Lab network during burn-in). Ensure Pi is on the same network as BT's scan range or use Tailscale IP as target.
  3. If Nessus says "Host unreachable" — check that the Pi is on (LEDs), has internet, and you can ping it from BT:
ssh [email protected] ping <pi-ip-or-tailscale-ip> -c 3
  1. If Nessus service itself is down on BT:
ssh [email protected] sudo systemctl status nessusd sudo systemctl start nessusd # Nessus runs as host service, NOT Docker

7 — Nuclear Recovery Procedures

☢Blue Triangle droplet is completely unreachable — cannot SSH, Tailscale shows offline
  1. Step 1 — netplan fix (try this first). Most likely cause after account suspension/reactivation: netplan vs. legacy ifupdown conflict. Use DO web console (Actions → Web Console) and run:
netplan --debug generate netplan --debug apply # Restores inbound connectivity immediately. No reboot needed.
  1. If web console shows "timed out waiting for handshake" — file a DO support ticket: "SSH listening on 0.0.0.0:22, no firewall, netplan apply run, connections still time out. Please reset backend networking."
  2. SSH key check — after restoring connectivity, verify ~/.ssh/authorized_keys contains the amp-deploy key. If missing, add it (or enable password auth temporarily):
cat ~/.ssh/authorized_keys # Should contain: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA92/nhiyYQqT16MEumPW+3mRMXtLZXCNqBOdxvYS8S6 amp-deploy # If missing, append it: echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA92/nhiyYQqT16MEumPW+3mRMXtLZXCNqBOdxvYS8S6 amp-deploy" >> ~/.ssh/authorized_keys # Temporary password auth (remove after use): echo 'PasswordAuthentication yes' > /etc/ssh/sshd_config.d/70-enable-password.conf && systemctl restart ssh
  1. After reboot, Tailscale should auto-reconnect. If auth expired:
tailscale up --authkey=tskey-auth-xxxxx --advertise-tags=tag:blue-triangle --ssh
  1. Once SSH works — restart Docker stack and verify:
cd /opt/amp/blue-triangle && docker compose up -d docker ps # All 5 containers should show "Up" # If containers can't reach internet after iptables changes: systemctl restart docker
  1. If the entire droplet is gone — rebuild from scratch using DR/README.md in the repo (full rebuild runbook including event flood warning).

Pukas stay operational even when BT is offline — they use direct Telegram alerts as fallback. Family internet access is unaffected. Use /snooze 120 in the war room to silence alert noise during recovery.

💾Need to rebuild a spoke node from scratch (SD card corrupted / Pi replaced)
  1. Flash a fresh Raspberry Pi OS Bullseye image to a new SD card.
  2. Add to /boot: empty ssh file + amp-tailscale.key with a valid reusable auth key.
  3. Ensure first-boot.sh is installed as systemd one-shot service in the image.
  4. Boot Pi — first-boot runs automatically. It will re-register with BT API using the new MAC.
    • If same MAC as before: gets same serial back (idempotent) ✓
    • If new hardware (different MAC): gets new serial, new node_id — update gateways table to reassign to correct subscriber
  5. Check the first-boot log to confirm success: tail -30 /var/log/amp-firstboot.log

8 — Symptom Quick-Lookup

What You SeeMost Likely CauseFirst Command to RunPage
🔴 Telegram "Node Offline" alert Pi unplugged or lost internet tailscale status on laptop P.1
Telegram bots not responding Bridge container down curl http://100.113.114.32:5001 P.2
SSH to spoke refused Tailscale disconnected on spoke tailscale status on laptop P.1
Heartbeat log is stale Cron not running or missing crontab -l on spoke P.1
First boot log stops mid-way Missing Tailscale key file ls /boot/amp-tailscale.key P.2
409 CONFLICT at registration SD re-flashed, MAC already registered Check audit_logs table in Postgres P.2
Docker containers showing Exited BT rebooted without auto-start set docker compose up -d on BT P.2
Internet slow through exit node Pi thermal throttle or DERP relay vcgencmd measure_temp on spoke P.3
Exit node not showing in Tailscale app Not approved in Tailscale admin Approve at tailscale.com/admin P.3
puka.alamalu.com unreachable NPM down or SSL cert expired docker ps | grep proxy on BT P.3
Blue Triangle SSH unreachable Tailscale down on BT or droplet frozen Use DigitalOcean web console P.3
Telegram flooded with 100s of alerts after BT recovery Pukas flushing 11-day backlog of queued events /snooze 120 in war room, then clear puka queues P.2
Hōʻike port 5000 times out / HTTP 000 Gunicorn workers overwhelmed by event flood docker compose restart api on BT P.2
All BT outbound broken after wg_aikai up wg_aikai missing Table = off — hijacks default route wg-quick down wg_aikai, add Table = off, bring back up P.2
wg_aikai handshake stuck at 0 (never connects) aikai endpoint uses Tailscale IP — blocked by ACLs Change endpoint to 159.65.218.220:51845 in wg_failover.conf P.2
Docker containers can't reach internet iptables/nftables flushed — Docker NAT rules wiped systemctl restart docker on BT P.2
Nessus won't start nessusd service not running sudo systemctl start nessusd P.3
Required Environment Variables — /opt/amp/.env on Blue Triangle
VariableUsed ByWhere to Get / Check
DATABASE_URLDashboard API, Alert scriptPostgres connection string on BT: postgresql://amp:<pass>@amp-db/ampdb
TELEGRAM_CLAUDE_TOKENAlert script, Telegram bridge (Ala-C bot)BotFather → Ala-C bot token
TELEGRAM_GEMINI_TOKENTelegram bridge (Ala-G bot)BotFather → Ala-G bot token
TELEGRAM_GROUP_CHAT_IDAlert script, Telegram bridgeWar Room group chat ID (negative number)
ANTHROPIC_API_KEYTelegram bridgeconsole.anthropic.com → API Keys

If any key is missing or wrong, the container will crash on startup. Check with: docker logs amp-bridge --tail=20 for KeyError or authentication error messages.

A.M.P. Troubleshooting Guide
Sea-Relay · Subscriber WireGuard · Relay Failover · Fleet Digest
Page 4 of 4 · Revised 2026-08-21

9 — Sea-Relay Issues (Vultr Seattle · 66.42.72.17)

Sea-relay is the subscriber-facing WireGuard relay. All mobile subscribers connect to relay.alamalu.com:443 (UDP). Each puka maintains a back-tunnel (wg20–wg27) to sea-relay. Subscribers never connect to BT directly.

🔴All subscribers disconnected — sea-relay unreachable or wg0 down
  1. SSH into sea-relay and check wg0 status:
ssh -i ~/.ssh/amp-deploy [email protected] ip link show wg0 # Should show UP wg show wg0 # Should show peers and recent handshakes ufw status # 443/udp must be ALLOW
  1. If wg0 is down, restart it:
systemctl restart wg-quick@wg0 wg show wg0 # Verify it came back up
  1. Check that amp-wg-sync.sh is running and has the correct ListenPort:
grep ListenPort /etc/wireguard/wg0.conf # Must show: ListenPort = 443 (NOT 51820 — that breaks all connections) # If wrong, edit wg0.conf, set ListenPort = 443, then: wg set wg0 listen-port 443 cat /etc/cron.d/amp-wg-sync # Sync cron must exist
  1. Verify UFW rules — all five must be present:
ufw status numbered # Required rules: # 22/tcp ALLOW (SSH) # 443/udp ALLOW (subscriber WireGuard) # 51820/udp ALLOW (legacy subscriber configs pre-port-443) # 51840:51847/udp ALLOW (puka back-tunnels) # 53 on wg0 ALLOW IN (AdGuard DNS for subscribers — CRITICAL) # If DNS rule missing: ufw allow in on wg0 to any port 53

Missing the DNS rule breaks all new domain lookups for subscribers (browsing, app updates, RCS). Cached IPs still work so Telegram reconnects appear fine, masking the issue.

📡Subscriber connected but no internet — browsing fails, DNS not resolving
  1. Check if AdGuard is running on sea-relay:
ssh -i ~/.ssh/amp-deploy [email protected] systemctl status AdGuardHome # Should be active (running). If not: systemctl start AdGuardHome
  1. Check AdGuard is bound to the right address:
ss -ulnp | grep 53 # Should show 10.8.0.1:53 bound (AdGuard DNS for wg0 subscribers)
  1. Check the back-tunnel to the subscriber's assigned puka is up (stale = no internet):
# Find which puka the subscriber's IP belongs to (e.g., 10.8.25.x = puka 25) wg show wg25 latest-handshakes # If handshake age > 120s → puka tunnel is stale → relay failover should kick in cat /var/lib/amp/relay-failover-state.json
  1. Verify kernel route for that puka's table exists:
# Example for puka 25: ip route show table 25 # Should show: default dev wg25 (or wg21 if failover is active) ip rule show | grep "25:" # Should show: from 10.8.25.0/24 lookup 25
🔀Relay failover triggered but subscribers not rerouting correctly
  1. Check the failover state file — it shows what the relay decided:
ssh -i ~/.ssh/amp-deploy [email protected] cat /var/lib/amp/relay-failover-state.json | python3 -m json.tool # Look for: "status": "stale" and "route": "wgN" where N ≠ tunnel number
  1. Check the relay failover timer is running:
systemctl status amp-relay-failover.timer systemctl status amp-relay-failover.service # Timer should be active; service should show "Last run" within 30s # If timer stopped: systemctl start amp-relay-failover.timer
  1. Manually trigger failover check:
bash /usr/local/bin/amp-relay-failover.sh cat /var/lib/amp/relay-failover-state.json | python3 -m json.tool
  1. Verify AllowedIPs on the target failover puka allows the stale puka's subnet:
# SSH to the fallback puka (e.g. puka 21 if puka 25 is stale) ssh -i ~/.ssh/amp-deploy [email protected] grep AllowedIPs /etc/wireguard/wg0.conf # MUST be: AllowedIPs = 10.8.0.0/16 (NOT /24) # If /24: sudo wg set wg0 peer <sea-relay-pubkey> allowed-ips 10.8.0.0/16 # Also check kernel route: ip route show | grep 10.8 # If 10.8.0.0/16 not present: sudo ip route add 10.8.0.0/16 dev wg0

AllowedIPs = 10.8.0.0/16 is required on every failover target. WireGuard silently drops packets from foreign subnets (10.8.25.x hitting puka 21's /24 allowlist) without it. Pukas 22 and 23 still need this fix when they come back online.

📊Fleet Health Telegram digest shows wrong data or all pukas red
  1. The fleet digest on BT reads from the sea-relay API — check sea-relay is reachable from BT:
ssh -i ~/.ssh/amp-deploy [email protected] curl -sf --max-time 5 http://100.107.129.93:5001/wg-handshakes | python3 -m json.tool # Should return nodes with handshake ages
  1. If the relay API (port 5001) is down on sea-relay:
ssh -i ~/.ssh/amp-deploy [email protected] ps aux | grep amp-relay-api # Restart if not running: pkill -f amp-relay-api.py python3 /usr/local/bin/amp-relay-api.py &
  1. Run the digest manually to test it:
ssh -i ~/.ssh/amp-deploy [email protected] bash /usr/local/bin/amp-health-digest.sh # Watch Telegram war room for the message
📱Subscriber gets "connection timed out" or can't connect at all (new QR scan)
  1. Verify the QR code was generated with the correct endpoint — must be relay.alamalu.com:443 (UDP), not BT's IP or old port 51820.
  2. Check the subscriber's WireGuard config on their device: Settings → WireGuard → peer endpoint. Should show relay.alamalu.com:443.
  3. If on a restrictive network (T-Mobile carrier), port 443 is required. Old configs on port 51820 fail. Have them rescan QR.
  4. Verify the subscriber's peer exists on sea-relay wg0:
ssh -i ~/.ssh/amp-deploy [email protected] wg show wg0 peers # Should list the subscriber's public key # If missing, amp-wg-sync.sh hasn't synced yet — wait up to 60s or run: bash /usr/local/bin/amp-wg-sync.sh
  1. If on hotel/airport WiFi that blocks UDP entirely — no WireGuard workaround without wstunnel (TCP wrapper). Short-term: tether to mobile data.

10 — Sea-Relay Symptom Quick-Lookup

What You SeeMost Likely CauseFirst Command to Run
All subscribers disconnected at once sea-relay wg0 down or UFW blocking wg show wg0 on sea-relay
Subscriber connected, no browsing (DNS fails) AdGuard down OR UFW missing DNS rule on wg0 systemctl status AdGuardHome on sea-relay
Subscriber exits wrong location (failover active) Back-tunnel to home puka is stale cat /var/lib/amp/relay-failover-state.json
Failover not rerouting traffic after puka goes down Failover target puka AllowedIPs still /24 (not /16) grep AllowedIPs /etc/wireguard/wg0.conf on fallback puka
Relay failover script not running Timer stopped or service failed systemctl status amp-relay-failover.timer
Fleet digest shows all pukas 🔴 sea-relay API port 5001 unreachable from BT curl http://100.107.129.93:5001/wg-handshakes from BT
Subscriber on T-Mobile can't connect Old config on port 51820 (T-Mobile blocks it) Have subscriber rescan QR — new configs use port 443
New subscriber not showing up on sea-relay amp-wg-sync.sh hasn't run yet (runs every 60s) bash /usr/local/bin/amp-wg-sync.sh on sea-relay
Puka back-tunnel stale — wg show shows old handshake Puka wg0 service needs restart or ISP flap sudo systemctl restart wg-quick@wg0 on puka
Hōʻike shows "Puka X · Location" as exit node (not home puka) Relay failover is active — home puka is offline Normal — failover working as designed. Check home puka status.
Sea-Relay Key Files & Services
File / ServicePurposeNotes
/etc/wireguard/wg0.confSubscriber-facing WireGuard (port 443)ListenPort must be 443. MTU 1280. DNS 10.8.0.1.
/usr/local/bin/amp-wg-sync.shSyncs subscriber peers from BT every 60sCron in /etc/cron.d/amp-wg-sync. Must have ListenPort=443 in generated conf.
/usr/local/bin/amp-relay-failover.shMesh-aware failover — runs every 30sWrites state to /var/lib/amp/relay-failover-state.json.
/usr/local/bin/amp-relay-api.pyHTTP API on port 5001Endpoints: /wg-handshakes, /wg-connections, /relay-failover
amp-relay-failover.timersystemd timer — fires every 30ssystemctl status amp-relay-failover.timer
AdGuardHomeAd-blocking DNS for subscribersBinds to 10.8.0.1:53 and 10.8.0.1:3000 (web UI)
/var/lib/amp/relay-failover-state.jsonCurrent failover state for all pukasRead by BT's list_gateways() to show exit puka in Hōʻike