Instances
BT — Blue Triangle (Newark)
| Web UI | http://100.113.114.32:3000 |
| DNS | 100.113.114.32:53 |
| Subnet filter | 10.8.{node_id}.x per puka |
| Role | Primary stats & digest engine |
Monthly subscriber digest pulls stats from this instance and apportions blocked-query counts by puka subnet. /opt/amp/monthly-digest.py
Sea-Relay (Seattle)
| Web UI | http://100.107.129.93:3000 |
| AdGuard DNS | 127.0.0.1:5300 (localhost only) |
| dnsdist proxy | 10.8.0.1:53 + 100.107.129.93:53 |
| AGH config | /opt/AdGuardHome/AdGuardHome.yaml |
| dnsdist config | /etc/dnsdist/dnsdist.conf |
dnsdist sits in front of AdGuard on port 53. AdGuard binds to localhost:5300 only. Subscriber WG configs set DNS = 10.8.0.1 — hits dnsdist → AdGuard.
Credentials
Both instances share the same amp user and password. Stored locally at:
~/Projects/Alamalu/adguard/.env
# AGH_URL = BT instance
# AGH_RELAY_URL = sea-relay instance
# AGH_USER / AGH_ADMIN_PW
Never store or transmit credentials in chat. Edit .env directly; sync between servers via SSH pipeline only.
DNS Flow — Who Queries What
Subscriber traffic (on VPN)
Subscriber device
→
wg0 tunnel
→
10.8.0.1:53
(sea-relay AdGuard)
→
DoH 1.1.1.1 / 8.8.8.8
Personal devices (Tailscale DNS override active)
Mac / phone
→
100.100.100.100
(Tailscale MagicDNS)
→
100.107.129.93:53
(sea-relay AdGuard)
→
DoH upstreams
Sea-relay preferred over BT while BT is NYC-based (~83ms vs ~19ms from Kent WA). Revisit after BT migrates west.
Pukas (THIS IS THE RISKY PATH)
puka (Raspberry Pi)
→
100.100.100.100
(Tailscale MagicDNS)
→
sea-relay AdGuard
(DNS override active)
→
DoH upstreams
Pukas run Tailscale. The global DNS override routes all puka DNS through sea-relay's AdGuard — including the heartbeat script's WAN IP lookups. If sea-relay's AdGuard goes down, pukas lose DNS and stop heartbeating.
Tailscale DNS Override — Caveats
dnsdist handles AdGuard restarts automatically. No need to toggle the Tailscale DNS override before restarting AdGuard. dnsdist detects the outage in ~6s and routes to 1.1.1.1 until AdGuard recovers.
Where the setting lives: Tailscale admin console → DNS → "Override DNS servers" → set to 100.107.129.93. This is set-and-forget — you only need to touch it if you're changing the nameserver itself.
Domain forwarding rules (critical — must stay in place)
Added to /opt/AdGuardHome/AdGuardHome.yaml on sea-relay:
upstream_dns:
- '[/blue-triangle/]100.100.100.100' # resolve Tailscale hostname
- '[/ts.net/]100.100.100.100' # resolve *.tailaae4a6.ts.net
- https://1.1.1.1/dns-query
- https://8.8.8.8/dns-query
Without these, blue-triangle and all .ts.net hostnames fail to resolve for pukas. This breaks Tailscale SSH ACLs and inter-node communication.
Heartbeat Script — DNS Dependency Map
Script: /usr/local/bin/amp-heartbeat.sh on each puka. Runs every 5 min via cron.
WAN IP resolution chain
# Chain stops at first success
WAN_IP=$(
curl http://checkip.amazonaws.com # DNS-dependent ← breaks with override
|| curl https://ifconfig.me # DNS-dependent ← breaks with override
|| curl https://1.1.1.1/cdn-cgi/trace | awk '/^ip=/{print $2}' # IP-only, no DNS
|| echo ""
)
If WAN_IP is empty after all three, the script exits 1 — no heartbeat sent. The https://1.1.1.1/cdn-cgi/trace fallback is the DNS-free safety net. Must use https:// — Cloudflare 301-redirects http://, which curl follows to HTML not trace data.
Fixed hardcoded values (DNS-independent as of 2026-08-22)
BT_API="http://100.113.114.32:5000" # was: http://blue-triangle:5000
If WAN_IP is empty → exit 1 → no heartbeat. This is the silent failure mode. The cron job completes without error, but BT receives no telemetry. Hōʻike shows the puka as OFFLINE after the stale threshold.
Diagnostic Pattern — Synchronized Fleet Outage
All pukas offline at the same Hōʻike timestamp = DNS failure
| Pattern |
Cause |
First check |
| All pukas last seen at same :XX:00 timestamp |
DNS went down at cron boundary — all pukas failed WAN_IP at same tick |
sea-relay AdGuard status |
| Pukas go offline staggered over several minutes |
ISP or per-puka issue, not global DNS |
Check individual puka logs |
| One or two pukas offline, others fine |
Local ISP/power at that site |
Ping puka Tailscale IP, check logs |
Why synchronized? Each puka's cron fires at the same wall-clock minute (e.g., :00, :05, :10…). If DNS breaks between two cron ticks, ALL pukas fail their NEXT cron tick simultaneously — producing an identical "last seen" timestamp across the entire fleet.
dnsdist Proxy — Automatic Fallback
Architecture (as of 2026-08-23)
Client DNS query
(subscriber or Tailscale)
→
dnsdist
10.8.0.1:53 / 100.107.129.93:53
→
AdGuard Home
127.0.0.1:5300 (primary)
→
DoH upstreams
dnsdist
→
1.1.1.1:53 (fallback — AdGuard down only)
dnsdist health-checks AdGuard every 2s. After 3 consecutive failures (~6s), traffic switches to 1.1.1.1 (ad blocking paused). When AdGuard recovers, dnsdist restores it as primary automatically.
/etc/dnsdist/dnsdist.conf
setLocal('10.8.0.1:53')
addLocal('100.107.129.93:53')
setACL({'0.0.0.0/0', '::/0'})
newServer({address='127.0.0.1:5300', order=1, checkName='dns.google',
checkInterval=2000, maxCheckFailures=3, name='adguard'})
newServer({address='1.1.1.1:53', order=2, checkName='dns.google',
checkInterval=0, name='cloudflare-fallback'})
Quick Runbook
Scenario: Sea-relay AdGuard needs a restart
1
Just restart AdGuard — no Tailscale toggle needed.
dnsdist detects the outage in ~6s and routes to 1.1.1.1 automatically. Fleet heartbeat is unaffected.
2
Restart AdGuard on sea-relay:
ssh -i ~/.ssh/amp-deploy [email protected] 'systemctl restart AdGuardHome'3
Confirm AdGuard is up and DNS resolves:
ssh -i ~/.ssh/amp-deploy [email protected] 'systemctl is-active AdGuardHome && dig +short @100.107.129.93 google.com'4
Verify ad blocking is restored (should return 0.0.0.0):
dig +short @10.8.0.1 doubleclick.net
Useful diagnostics
# Check heartbeat log on a puka
ssh -i ~/.ssh/amp-deploy pi@<tailscale_ip> 'tail -20 /var/log/amp-heartbeat.log'
# Test DNS-free WAN IP fallback from a puka
ssh -i ~/.ssh/amp-deploy pi@<tailscale_ip> 'curl -s https://1.1.1.1/cdn-cgi/trace | grep "^ip="'
# Test DNS resolution from a puka (should return an IP if AdGuard is working)
ssh -i ~/.ssh/amp-deploy pi@<tailscale_ip> 'curl -s --max-time 5 http://checkip.amazonaws.com'
# Sea-relay AdGuard service status
ssh -i ~/.ssh/amp-deploy [email protected] 'systemctl status AdGuardHome --no-pager'
UFW rules required on sea-relay for DNS to work
ufw allow in on tailscale0 to any port 53 # DNS for Tailscale devices (incl. pukas)
ufw allow in on tailscale0 to any port 3000 proto tcp # Web UI via Tailscale only
ufw allow in on wg0 to any port 53 # DNS for subscribers on VPN
Port 3000 is blocked on the public interface (enp1s0) — only accessible via Tailscale IP.