git-website-engine-control-plane/docs/OPERATIONS.md

# Website Engine Operations

This document records the verified deployment and access design without containing passwords, tokens, private keys, or credential values.

## Current topology

The current deployment is **unstaged**. The active release is the live site.

| Purpose | Address | Exposure |
| --- | --- | --- |
| Engine control plane | `http://10.138.2.46/` | Private network, Nginx port 80 |
| Rendered live site | `http://10.138.2.46:3000/` | Private network, Nginx port 3000 |
| Control-plane Node service | `http://127.0.0.1:3001/` | Engine container only |
| Gitea web/API | `http://10.138.2.42:3000/` | Private network |
| Gitea SSH | `[email protected]:22` | Private network |
| Public Gitea | `https://git.labyricorn.com/` | Public entry point |

Port 80 proxies to the private Node listener. Port 3000 serves files through the active-release symlink; it does not proxy to Node.

In a future staged deployment, port 3000 may be assigned to the preview candidate instead. That behavior is not configured in the current unstaged deployment.

## Server layout

| Item | Location or name |
| --- | --- |
| Source checkout | `/opt/website-engine-control-plane` |
| Persistent builds | `/var/lib/website-engine/builds` |
| Active-release pointer | `/var/lib/website-engine/builds/current` |
| Service account | `website-engine` |
| systemd service | `labyricorn-control-plane.service` |
| Nginx site | `/etc/nginx/sites-available/website-engine` |
| Versioned service definition | `deploy/labyricorn-control-plane.service` |
| Versioned Nginx definition | `deploy/nginx-website-engine.conf` |

The systemd service is enabled across reboots. Nginx and the control-plane service must both be active for the control plane to work. Nginx alone serves the activated static release on port 3000.

## Private Gitea access

The engine checkout uses this read-only private SSH remote:

```text
[email protected]:Labyricorn/git-website-engine-control-plane.git
```

The Gitea SSH username is `gitea`, not the common default `git`.

The deploy key is named `website-engine-10.138.2.46` in Gitea. Its public fingerprint is:

```text
SHA256:RtuazmPwcRLY6PbgtEaEELoQk8nN7OfsMyaB7Sa++J0
```

The pinned Gitea ED25519 host fingerprint is:

```text
SHA256:XM3wWREeQesVBhLltXXnGCmpCr6M1bWH+545f792tGc
```

Credential files on the engine server:

| File | Owner/mode | Purpose |
| --- | --- | --- |
| `/var/lib/website-engine/.ssh/gitea_ed25519` | `website-engine`, `0600` | Private deploy key; never display or copy into Git |
| `/var/lib/website-engine/.ssh/gitea_ed25519.pub` | `website-engine`, `0644` | Public deploy key |
| `/var/lib/website-engine/.ssh/known_hosts` | `website-engine`, `0600` | Pinned Gitea host key |

The repository-local `core.sshCommand` selects this identity, requires the pinned host key, disables interactive prompting, and refuses unrecognized host keys. The deploy key has read access only.

Older protected HTTP credentials remain outside the repository under `/root/.config/gitea/`. They are not needed for deployment fetches now that private SSH works. Never print, commit, or copy their values into commands, logs, issues, or documentation.

## Verified access matrix

| Path | Result |
| --- | --- |
| Engine to `http://10.138.2.42:3000/` | HTTP 200 |
| Engine to the private Gitea API | HTTP 200 |
| Anonymous private HTTP Git read | HTTP 401, expected for the private repository |
| SSH as `[email protected]` | Rejected; incorrect Gitea SSH user |
| SSH as `[email protected]` | Authenticated with the deploy key |
| Private SSH `git ls-remote` | Successful |
| Engine checkout versus `origin/main` | Synchronized when last verified |
| GPG key for Git authentication | Not applicable; GPG signs commits but does not authenticate Git transport |

## Release and deployment flow

1. Verify changes locally with `npm test`, `npm run lint`, and `npm run build`.
2. Commit and push `main` to Gitea.
3. On the engine server, update only with a fast-forward pull:

   ```bash
   git -C /opt/website-engine-control-plane pull --ff-only
   ```

4. Install locked dependencies and rebuild when application code or dependencies changed:

   ```bash
   cd /opt/website-engine-control-plane
   npm ci --no-audit --no-fund
   npm test
   npm run lint
   npm run build
   ```

5. Install changed versioned service or Nginx configuration, validate it, and restart only the affected service.
6. Trigger and validate an engine build before activation.
7. Activate the release. Activation updates the `current` symlink atomically, and Nginx begins serving it on port 3000 without copying files into the Nginx configuration directory.

Do not deploy an unpushed working tree or bypass `--ff-only`. The server checkout should remain reproducible from Gitea.

## Verification checks

Run these checks on the engine server after deployment:

```bash
systemctl is-active labyricorn-control-plane.service nginx
ss -lntp | grep -E '(:80|:3000|:3001)'
curl -fsS http://127.0.0.1:3001/api/health
curl -fsSI http://127.0.0.1:3000/
readlink -f /var/lib/website-engine/builds/current
git -C /opt/website-engine-control-plane rev-list --left-right --count HEAD...origin/main
```

Expected listeners:

- Nginx on `0.0.0.0:80` and `0.0.0.0:3000`, with IPv6 equivalents;
- Node on `127.0.0.1:3001` only.

An unknown rendered-site route should return HTTP 404 and the generated `404.html` page.

## Security rules

- Add only public keys to Gitea. Never transfer or paste the private deploy key.
- Keep the deploy key read-only unless a separately reviewed workflow genuinely requires pushes from the engine.
- Prefer the private SSH remote for engine-to-Gitea Git traffic.
- Treat plain private HTTP as unencrypted. Do not send tokens through it when SSH can perform the operation.
- Do not disable strict host-key checking. If the Gitea host fingerprint changes, stop and verify the new fingerprint from the Gitea console before updating `known_hosts`.
- Do not place secrets in Git remotes, environment files committed to Git, service definitions, command histories, or documentation.
- Preserve the dedicated `website-engine` service account and its restrictive file permissions.

## Key rotation

1. Generate a new key under a new filename on the engine server.
2. Add only the new public key to Gitea as a read-only deploy key.
3. Verify SSH authentication and `git ls-remote` with the new key.
4. Update the repository-local `core.sshCommand` to the new key.
5. Verify a normal fetch through the private remote.
6. Revoke the old deploy key in Gitea.
7. Remove the old private key only after the new path has been verified end to end.

Do not overwrite the active private key in place during rotation; retaining the old key until verification provides a safe rollback path.

## Repository theme build contract

Production presentation is owned by `.theme/` in the repository selected by `packages/site-definition/site.yml`. There is no built-in production renderer or preview fallback. Configuration is Git-owned and read-only in control-plane v1; `PUT /api/site-config` returns `405 E_CONFIG_READ_ONLY`.

Each build resolves `LABYRICORN_SITE_DEFINITION_REF` (default `HEAD`) once to a full commit and materializes an archive of that commit. Set these variables when the site definition is not the application checkout:

```text
LABYRICORN_SITE_DEFINITION_REPOSITORY=/srv/site-definition
LABYRICORN_SITE_DEFINITION_REF=refs/heads/main
LABYRICORN_SITE_DEFINITION_PATH=packages/site-definition
LABYRICORN_BUILD_ROOT=/var/lib/website-engine/builds
```

The artifact ID is derived from the canonical input descriptor. Operational run IDs and execution time do not enter artifact bytes. The release layout is:

```text
<build-root>/
├── releases/sha256-<input-digest>/
├── staging -> releases/sha256-<input-digest>/
└── current -> releases/sha256-<input-digest>/
```

Staging and promotion verify every entry in `checksums.json`. A modified artifact is rejected. Promotion updates `current`; it does not render or copy a second release. On startup, valid `staging` and `current` pointers are rediscovered and verified.

### Theme failure behavior

- Missing or invalid `.theme/theme.yml`: the build fails before a release directory is created.
- Unknown manifest keys, unsafe paths, missing templates/assets, or strict-Liquid errors: the build fails closed.
- No selected staging/live release: `/api/live-site/html` returns `503 E_RELEASE_UNAVAILABLE`.
- Unknown route: the selected artifact's `404.html` is returned with HTTP 404.
- Rollback activates a previously verified immutable release; it never recreates or mutates release bytes.

Repository themes may contain Liquid templates and manifest-declared browser assets. They may not execute build programs, declare custom filters/tags, use dynamic includes, emit inline scripts/event handlers/styles, or reference undeclared local dependencies. HTTPS fonts require a pinned SHA-256 checksum and are cached only after verification.