Layout, Validation, and Operations Runbook

/srv is the service runtime area

Keep each service in its own directory. For a new project, first inspect /srv and confirm that the intended directory does not already exist:

[REMOTE: VPS]

ls -ld /srv
sudo find /srv -mindepth 1 -maxdepth 2 -printf '%M %u:%g %p\n'
# Create only if the path is absent, not a symlink or mount.
sudo mkdir -- /srv/example-api &&
  sudo chown -- "$VPS_USER:$VPS_USER" /srv/example-api

Here mkdir deliberately omits -p: an existing path makes it fail, so && prevents the ownership change. Confirm that $VPS_USER is the intended deployment account and that its same-named group exists. On a shared host, also prevent concurrent changes between inspection and creation.

Never run chown -R over all of /srv. Recursive ownership changes reach service data as well as configuration; container volumes may require particular numeric UID/GID values, and backups may need root-only access. For an existing project, inspect its service-specific ownership requirements before changing any explicitly named path. An empty /srv/backups directory is not a backup system: backups still need a schedule, versions, off-site copies, and restore tests.

Three common systemd questions

systemctl is-active docker       # Current unit state
systemctl is-enabled docker      # Unit-file enablement
systemctl status docker --no-pager

systemctl is-active reports the current unit state; systemctl is-enabled reports unit-file enablement. Enablement alone does not prove that a service will start successfully at boot. Dependencies or socket activation can also start a service. When a service fails, use:

sudo journalctl -u docker --since today --no-pager
sudo journalctl -u tailscaled --since today --no-pager
sudo journalctl -u ssh --since today --no-pager
systemctl --failed --no-pager

Logs may contain IP addresses, usernames, paths, or application data. Redact sensitive information before sharing them.

Routine software updates

[REMOTE: VPS]

sudo apt update
apt list --upgradable
sudo apt upgrade

Read the upgrade summary first. If automation is needed, simulate the changes with apt-get -s before applying them, especially risky changes. Ubuntu phased updates may intentionally defer some ordinary updates: this is a staged rollout, not a dependency conflict.

Check whether a reboot is required:

if [ -f /var/run/reboot-required ]; then
  cat /var/run/reboot-required
else
  echo "No reboot required"
fi

Before a needed reboot, verify the current OpenSSH-over-Tailscale path, keep the session open, and run sudo reboot. Once the machine returns, reconnect and recheck the kernel and failed services.

Validation checklist

[REMOTE: VPS]

hostnamectl
timedatectl
nproc
free -h
df -h /

ip -brief addr
ip route

tailscale status
tailscale ip -4
systemctl is-active tailscaled

sudo ufw status verbose
sudo ufw status numbered

sudo sshd -t
sudo sshd -T | grep -E \
  '^(permitrootlogin|passwordauthentication|kbdinteractiveauthentication|pubkeyauthentication|authenticationmethods) '

docker --version
docker compose version
systemctl is-active docker
docker info --format \
  'ServerVersion={{.ServerVersion}} Driver={{.Driver}} CgroupDriver={{.CgroupDriver}}'
docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Ports}}\t{{.Status}}'
docker compose -f /srv/caddy/compose.yaml ps
docker compose -f /srv/uptime-kuma/compose.yaml ps
docker compose -f /srv/rsshub/compose.yaml ps
docker network inspect infra-edge

systemctl --failed --no-pager
find /srv -maxdepth 2 -printf '%M %u:%g %p\n' | sort

Compare the effective SSH values with the key-only policy, including the conditions of any applicable Match blocks. Then verify access from WSL2:

[LOCAL: WSL2]

# Must succeed
ssh -o ConnectTimeout=10 "$VPS_USER@$TAILSCALE_IPV4"

# Must fail
ssh -o ConnectionAttempts=1 -o ConnectTimeout=5 \
  "$VPS_USER@$VPS_PUBLIC_IPV4"

# Both public HTTPS health checks must succeed
curl --fail https://infra.example.com/healthz
curl --fail https://rss.example.com/healthz

# RSSHub port 1200 must not be directly accessible via any host address
curl --connect-timeout 5 --max-time 7 \
  "http://$VPS_PUBLIC_IPV4:1200/"
curl --connect-timeout 5 --max-time 7 \
  "http://$TAILSCALE_IPV4:1200/"

Both public HTTPS health checks must pass normal certificate validation. The Tailscale SSH connection must succeed, the public-IP SSH attempt must fail, and direct public and Tailscale requests to RSSHub port 1200 must fail.

A bare 1200/tcp in docker ps indicates an internal image port. A host publication looks like 0.0.0.0:1200->1200/tcp and would fail this design. The accepted state is Caddy publishing host ports 80/443, Kuma bound exactly to 127.0.0.1:3001, and no RSSHub host binding.

Suggested check cadence

Frequency	Checks
After every change	A new SSH session, the affected service’s state, UFW, and docker ps
Weekly	Upgradable packages, failed units, disk space, and container state
Monthly	Tailscale devices, SSH keys, exposed ports, and log growth
Every deployment	Compose diff, port bindings, volumes, health checks, and rollback plan; also the affected service’s state, UFW, and docker ps
Periodic drills	Restore from backups, rather than only checking that backup files exist

Incident priorities

1. Preserve access: do not close the last working session.
2. Stop expanding the scope: do not keep trying “fixes” that overwrite each other.
3. Identify the failing layer: check routing, ports, the firewall, the daemon, and authentication in turn.
4. Collect non-secret evidence: gather state, exit codes, timestamps, and recent logs.
5. Restore the most recent safe state: prefer restoring temporary access to resetting the system.
6. Verify recovery: use a fresh connection or a complete request.

Use Tmux for long operations so a local network interruption does not take away the terminal during an upgrade or investigation. Tmux keeps remote shell processes alive across SSH disconnects, but not host reboots. It does not replace a second SSH access path.

Boundaries for the current services

The three current Compose projects—Caddy, Uptime Kuma, and RSSHub—have independent lifecycles. Operate from the matching /srv/<service> directory, inspect docker compose ps and logs first, then decide whether a restart is needed. Do not restart all containers to troubleshoot one application or use a global Docker prune.

The RSSHub production shadow deployment covers image updates, ACCESS_KEY, real-route verification, canary checks, and rollback steps. The existing feed consumer, its persistent data, and its scheduler have not migrated. The consumer switchover and persistent-data backup and restore work have not happened yet. A healthy VPS RSSHub does not mean the downstream consumer has migrated.

My reading