Document secure engine operations

2026-07-23 16:02:37 -07:00
parent 899351552d
commit 68d76a59df
2 changed files with 155 additions and 1 deletions
@@ -34,7 +34,11 @@ npm run build
 npm start
 ```

-The control plane listens on port `3000`.
+The development server listens on port `3000` by default. In the current production deployment, the Node service listens privately on `127.0.0.1:3001`; Nginx exposes the control plane on port 80 and the active rendered release on port 3000.
+
+## Operations
+
+See [Website Engine Operations](docs/OPERATIONS.md) for the verified private-network topology, deployment procedure, service checks, key fingerprints, credential-handling rules, and recovery guidance. The operations document intentionally contains no passwords, tokens, or private-key material.

 ## Build artifacts

@@ -0,0 +1,150 @@
+# Website Engine Operations
+
+This document records the verified deployment and access design without containing passwords, tokens, private keys, or credential values.
+
+## Current topology
+
+The current deployment is **unstaged**. The active release is the live site.
+
+| Purpose | Address | Exposure |
+| --- | --- | --- |
+| Engine control plane | `http://10.138.2.46/` | Private network, Nginx port 80 |
+| Rendered live site | `http://10.138.2.46:3000/` | Private network, Nginx port 3000 |
+| Control-plane Node service | `http://127.0.0.1:3001/` | Engine container only |
+| Gitea web/API | `http://10.138.2.42:3000/` | Private network |
+| Gitea SSH | `[email protected]:22` | Private network |
+| Public Gitea | `https://git.labyricorn.com/` | Public entry point |
+
+Port 80 proxies to the private Node listener. Port 3000 serves files through the active-release symlink; it does not proxy to Node.
+
+In a future staged deployment, port 3000 may be assigned to the preview candidate instead. That behavior is not configured in the current unstaged deployment.
+
+## Server layout
+
+| Item | Location or name |
+| --- | --- |
+| Source checkout | `/opt/website-engine-control-plane` |
+| Persistent builds | `/var/lib/website-engine/builds` |
+| Active-release pointer | `/var/lib/website-engine/builds/current` |
+| Service account | `website-engine` |
+| systemd service | `labyricorn-control-plane.service` |
+| Nginx site | `/etc/nginx/sites-available/website-engine` |
+| Versioned service definition | `deploy/labyricorn-control-plane.service` |
+| Versioned Nginx definition | `deploy/nginx-website-engine.conf` |
+
+The systemd service is enabled across reboots. Nginx and the control-plane service must both be active for the control plane to work. Nginx alone serves the activated static release on port 3000.
+
+## Private Gitea access
+
+The engine checkout uses this read-only private SSH remote:
+
+```text
+[email protected]:Labyricorn/git-website-engine-control-plane.git
+```
+
+The Gitea SSH username is `gitea`, not the common default `git`.
+
+The deploy key is named `website-engine-10.138.2.46` in Gitea. Its public fingerprint is:
+
+```text
+SHA256:RtuazmPwcRLY6PbgtEaEELoQk8nN7OfsMyaB7Sa++J0
+```
+
+The pinned Gitea ED25519 host fingerprint is:
+
+```text
+SHA256:XM3wWREeQesVBhLltXXnGCmpCr6M1bWH+545f792tGc
+```
+
+Credential files on the engine server:
+
+| File | Owner/mode | Purpose |
+| --- | --- | --- |
+| `/var/lib/website-engine/.ssh/gitea_ed25519` | `website-engine`, `0600` | Private deploy key; never display or copy into Git |
+| `/var/lib/website-engine/.ssh/gitea_ed25519.pub` | `website-engine`, `0644` | Public deploy key |
+| `/var/lib/website-engine/.ssh/known_hosts` | `website-engine`, `0600` | Pinned Gitea host key |
+
+The repository-local `core.sshCommand` selects this identity, requires the pinned host key, disables interactive prompting, and refuses unrecognized host keys. The deploy key has read access only.
+
+Older protected HTTP credentials remain outside the repository under `/root/.config/gitea/`. They are not needed for deployment fetches now that private SSH works. Never print, commit, or copy their values into commands, logs, issues, or documentation.
+
+## Verified access matrix
+
+| Path | Result |
+| --- | --- |
+| Engine to `http://10.138.2.42:3000/` | HTTP 200 |
+| Engine to the private Gitea API | HTTP 200 |
+| Anonymous private HTTP Git read | HTTP 401, expected for the private repository |
+| SSH as `[email protected]` | Rejected; incorrect Gitea SSH user |
+| SSH as `[email protected]` | Authenticated with the deploy key |
+| Private SSH `git ls-remote` | Successful |
+| Engine checkout versus `origin/main` | Synchronized when last verified |
+| GPG key for Git authentication | Not applicable; GPG signs commits but does not authenticate Git transport |
+
+## Release and deployment flow
+
+1. Verify changes locally with `npm test`, `npm run lint`, and `npm run build`.
+2. Commit and push `main` to Gitea.
+3. On the engine server, update only with a fast-forward pull:
+
+   ```bash
+   git -C /opt/website-engine-control-plane pull --ff-only
+   ```
+
+4. Install locked dependencies and rebuild when application code or dependencies changed:
+
+   ```bash
+   cd /opt/website-engine-control-plane
+   npm ci --no-audit --no-fund
+   npm test
+   npm run lint
+   npm run build
+   ```
+
+5. Install changed versioned service or Nginx configuration, validate it, and restart only the affected service.
+6. Trigger and validate an engine build before activation.
+7. Activate the release. Activation updates the `current` symlink atomically, and Nginx begins serving it on port 3000 without copying files into the Nginx configuration directory.
+
+Do not deploy an unpushed working tree or bypass `--ff-only`. The server checkout should remain reproducible from Gitea.
+
+## Verification checks
+
+Run these checks on the engine server after deployment:
+
+```bash
+systemctl is-active labyricorn-control-plane.service nginx
+ss -lntp | grep -E '(:80|:3000|:3001)'
+curl -fsS http://127.0.0.1:3001/api/health
+curl -fsSI http://127.0.0.1:3000/
+readlink -f /var/lib/website-engine/builds/current
+git -C /opt/website-engine-control-plane rev-list --left-right --count HEAD...origin/main
+```
+
+Expected listeners:
+
+- Nginx on `0.0.0.0:80` and `0.0.0.0:3000`, with IPv6 equivalents;
+- Node on `127.0.0.1:3001` only.
+
+An unknown rendered-site route should return HTTP 404 and the generated `404.html` page.
+
+## Security rules
+
+- Add only public keys to Gitea. Never transfer or paste the private deploy key.
+- Keep the deploy key read-only unless a separately reviewed workflow genuinely requires pushes from the engine.
+- Prefer the private SSH remote for engine-to-Gitea Git traffic.
+- Treat plain private HTTP as unencrypted. Do not send tokens through it when SSH can perform the operation.
+- Do not disable strict host-key checking. If the Gitea host fingerprint changes, stop and verify the new fingerprint from the Gitea console before updating `known_hosts`.
+- Do not place secrets in Git remotes, environment files committed to Git, service definitions, command histories, or documentation.
+- Preserve the dedicated `website-engine` service account and its restrictive file permissions.
+
+## Key rotation
+
+1. Generate a new key under a new filename on the engine server.
+2. Add only the new public key to Gitea as a read-only deploy key.
+3. Verify SSH authentication and `git ls-remote` with the new key.
+4. Update the repository-local `core.sshCommand` to the new key.
+5. Verify a normal fetch through the private remote.
+6. Revoke the old deploy key in Gitea.
+7. Remove the old private key only after the new path has been verified end to end.
+
+Do not overwrite the active private key in place during rotation; retaining the old key until verification provides a safe rollback path.