From 2699afd5607bc08537e7fea7a02242bd3ea4058c Mon Sep 17 00:00:00 2001 From: Labyricorn Deployment Date: Tue, 11 Aug 2026 20:07:43 -0700 Subject: [PATCH] Expand deployment operations runbook --- README.md | 536 ++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 521 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index d4afbbd..27e4387 100644 --- a/README.md +++ b/README.md @@ -1,34 +1,540 @@ -# Labyricorn +# Labyricorn website and deployment runbook -Source for the Labyricorn static website, built with Lektor. +This repository is the canonical source for `www.labyricorn.com`. It contains +the editable Lektor project—not generated production files—and is also the +primary operations guide for building, deploying, verifying, rolling back, and +troubleshooting the site. + +## Quick reference + +| Item | Value | +| --- | --- | +| Production URL | `https://www.labyricorn.com` | +| Canonical Git repository | `https://git.labyricorn.com/Labyricorn/labyricorn-site.git` | +| Production branch | `main` | +| Production host | `ubuntu2-47` (`10.138.2.47`) | +| Repository checkout | `/srv/labyricorn/repo` | +| Active document root | `/srv/labyricorn/current` | +| Release storage | `/srv/labyricorn/releases` | +| Deployment command | `sudo deploy-labyricorn` | +| Rollback listing | `sudo rollback-labyricorn` | +| Rollback command | `sudo rollback-labyricorn RELEASE_NAME` | +| Deployment log | `/srv/labyricorn/logs/deploy.log` | +| nginx site config | `/etc/nginx/sites-available/labyricorn` | +| Lektor executable | `/srv/labyricorn/.local/bin/lektor` | +| Deployment account | `labyricorn-deploy` | + +## Architecture + +```text +Gitea: Labyricorn/labyricorn-site (main) + | + | authenticated Git fetch + v + /srv/labyricorn/repo + | + | Lektor static build + v + /srv/labyricorn/releases/- + | + | validated, atomic symlink switch + v + /srv/labyricorn/current + | + | read-only static files + v + nginx :80 + | + | Cloudflare proxy and public TLS + v + https://www.labyricorn.com +``` + +Gitea is the source of truth. The production checkout is disposable state and +must not contain unpublished edits during deployment. Lektor's development +server is not part of production serving; nginx serves generated files only. + +## Repository contents + +```text +Labyricorn.lektorproject Lektor project definition +content/ Editable site content (`contents.lr` files) +models/ Lektor content models +templates/ Jinja templates +assets/static/ CSS, favicon, and other static source assets +.gitignore Excludes generated and local files +README.md This runbook +``` + +Generated output such as `build/`, `dist/`, and `.lektor/` is intentionally +ignored and must not be committed. + +## Routine content workflow + +The preferred workflow is to edit in a normal Git checkout, review the changes, +push them to `main`, and then deploy on the production host. + +```bash +git switch main +git pull --ff-only origin main + +# Edit content, templates, models, or assets. + +lektor build --output-path build +git status +git add +git commit -m "Describe the site change" +git push origin main +``` + +Then deploy on `ubuntu2-47`: + +```bash +sudo deploy-labyricorn +``` + +Do not edit generated files under `/srv/labyricorn/current` or +`/srv/labyricorn/releases`. They will be replaced by deployment and are not +source-controlled. ## Local development -Install Lektor in an isolated environment, then run: +Install Lektor in an isolated Python environment. For example: + +```bash +python3 -m venv .venv +. .venv/bin/activate +python -m pip install --upgrade pip +python -m pip install lektor +``` + +Run the local editor and preview server: ```bash lektor server ``` -For a production-style build: +Run a production-style build: ```bash lektor build --output-path build +test -f build/index.html ``` -Generated output is intentionally not committed. Production deployment builds -an immutable release and atomically updates the nginx document-root symlink. +The project currently has no third-party Lektor packages or plugins. +## Production filesystem and permissions -## Production repository +```text +/srv/labyricorn/ +├── repo/ Git working copy (not web-accessible) +├── releases/ Successful immutable builds +├── current -> releases/ +├── logs/ +│ ├── deploy.log +│ └── deploy.lock +├── .local/bin/lektor pipx-installed Lektor +├── .netrc Git HTTPS credential (0600; secret) +├── .ssh/ Reserved deployment SSH material +└── README-DEPLOYMENT.md Supplemental host-local notes +``` -Git origin: https://git.labyricorn.com/Labyricorn/labyricorn-site.git -Production branch: main -Credential purpose: authenticated private repository access -Credential location: /srv/labyricorn/.netrc (mode 0600) +The `labyricorn-deploy` account owns the checkout, releases, logs, Lektor +environment, and Git credential. nginx's `www-data` account can traverse the +base path and read active generated files, but cannot read the source checkout +or credential. nginx cannot modify either source or releases. -## Future admin workflow +## Deployment behavior -The production deployment account has authenticated Git write access so a future -local-only administration workflow can commit and push content changes. Lektor -does not push automatically; that operation must remain explicit and controlled. +`/usr/local/bin/deploy-labyricorn` performs the following sequence: + +1. Changes from root to the unprivileged `labyricorn-deploy` account. +2. Acquires an exclusive `flock` lock to prevent concurrent deployments. +3. Refuses to proceed if the working tree contains any tracked or untracked + changes. +4. Fetches and prunes `origin`. +5. Verifies that `origin/main` exists. +6. Switches to local `main` and resets it exactly to `origin/main`. +7. Locates exactly one `.lektorproject` file instead of assuming its name. +8. Creates a unique release named with UTC time and the 12-character commit ID. +9. Runs Lektor with an explicit output directory. +10. Requires the build to succeed and contain `index.html`. +11. Writes the full commit ID to `.labyricorn-commit` in the release. +12. Atomically replaces `/srv/labyricorn/current` with a relative symlink to + the new release. +13. Keeps the active release plus recent prior releases, targeting five total. +14. Logs start, failure, success, release name, and commit. + +Run it with: + +```bash +sudo deploy-labyricorn +``` + +A failed build is removed before activation. The current symlink is not changed, +so the previously working site remains online. + +### Deployment preflight + +```bash +sudo -u labyricorn-deploy env HOME=/srv/labyricorn \ + git -C /srv/labyricorn/repo status --short --branch + +sudo -u labyricorn-deploy env HOME=/srv/labyricorn \ + git -C /srv/labyricorn/repo fetch origin +``` + +The status must be clean. Do not bypass a dirty-tree refusal. Determine whether +the files are legitimate unpublished admin edits, then commit and push them or +remove them deliberately before deployment. + +## Verify a deployment + +```bash +readlink -f /srv/labyricorn/current +cat /srv/labyricorn/current/.labyricorn-commit + +sudo -u labyricorn-deploy env HOME=/srv/labyricorn \ + git -C /srv/labyricorn/repo rev-parse HEAD + +sudo -u labyricorn-deploy env HOME=/srv/labyricorn \ + git -C /srv/labyricorn/repo ls-remote origin refs/heads/main + +nginx -t +systemctl is-active nginx + +curl -I -H 'Host: www.labyricorn.com' \ + -H 'X-Forwarded-Proto: https' http://127.0.0.1/ + +curl -I https://www.labyricorn.com/ +``` + +The commit in the active release, local `HEAD`, and remote `main` should match. +The local nginx request with `X-Forwarded-Proto: https` should return `200`. +Public HTTP should redirect to HTTPS, and public HTTPS should return `200`. + +## Rollback + +List releases and identify the current one: + +```bash +sudo rollback-labyricorn +``` + +Switch atomically to a previous successful release: + +```bash +sudo rollback-labyricorn 20260812T024742Z-92475d2ec759 +``` + +Use a release name printed by the listing command. Rollback does not build, +fetch, or alter Git. It only changes the `current` symlink and records the event +in the deployment log. + +Verify afterward: + +```bash +readlink -f /srv/labyricorn/current +cat /srv/labyricorn/current/.labyricorn-commit +curl -I -H 'Host: www.labyricorn.com' \ + -H 'X-Forwarded-Proto: https' http://127.0.0.1/ +``` + +A normal deployment after rollback rebuilds and reactivates the latest remote +`main` commit. + +## nginx and URL handling + +The site configuration is: + +```text +/etc/nginx/sites-available/labyricorn +/etc/nginx/sites-enabled/labyricorn -> ../sites-available/labyricorn +``` + +nginx serves `/srv/labyricorn/current` and supports: + +- `index.html` at directory-style URLs; +- direct static assets; +- optional `.html` fallback; +- generated `/404/index.html` with an HTTP `404` response; +- disabled directory browsing; +- rejection of hidden paths and common executable-script extensions; +- dedicated access and error logs. + +Always validate before reloading nginx: + +```bash +nginx -t +systemctl reload nginx +``` + +nginx logs: + +```text +/var/log/nginx/labyricorn-access.log +/var/log/nginx/labyricorn-error.log +``` + +## HTTPS, Cloudflare, and DNS + +Current production HTTPS terminates at Cloudflare. The origin nginx process +currently listens on port `80`, not `443`. For `www.labyricorn.com`, nginx uses +Cloudflare's `X-Forwarded-Proto` header to distinguish public HTTP from HTTPS: + +- public HTTP redirects to `https://www.labyricorn.com`; +- Cloudflare-originated HTTPS requests are served without a redirect loop; +- `labyricorn.com` is configured at nginx to redirect to the canonical HTTPS + `www` hostname. + +The apex `labyricorn.com` did not resolve during initial setup. Its redirect will +not work publicly until DNS/Cloudflare has an apex record routed to this origin. + +Certbot and the nginx integration are installed, and `certbot.timer` is enabled, +but no origin certificate was issued during setup. Do not run Certbot blindly +behind Cloudflare. First decide whether the desired origin mode is a Cloudflare +Origin CA certificate, Let's Encrypt with a compatible challenge, or direct DNS +without the proxy. After changing TLS, test both origin and public behavior for +redirect loops. + +Useful checks: + +```bash +getent ahosts www.labyricorn.com +getent ahosts labyricorn.com +curl -I http://www.labyricorn.com/ +curl -I https://www.labyricorn.com/ +systemctl status certbot.timer +certbot certificates +``` + +## Firewall and exposed services + +UFW is active with default incoming denial. Allowed services are: + +- OpenSSH: TCP 22 +- nginx HTTP/HTTPS profile: TCP 80 and 443 + +Inspect current state: + +```bash +ufw status verbose +ss -lntup +``` + +Lektor's development port must never be opened in UFW or bound to `0.0.0.0`. + +## Optional server-side Lektor admin + +An optional unit exists at: + +```text +/etc/systemd/system/lektor-admin.service +``` + +Current state: **disabled and stopped**. When started, it runs as +`labyricorn-deploy` and binds only to `127.0.0.1:5000`. It is not proxied by +nginx and production remains independent of it. + +Start it temporarily: + +```bash +systemctl start lektor-admin +systemctl status lektor-admin +ss -lntp | grep ':5000' +``` + +Access it securely through an SSH tunnel from an administrator workstation: + +```bash +ssh -L 5000:127.0.0.1:5000 root@10.138.2.47 +``` + +Then browse to `http://127.0.0.1:5000` on that workstation. + +Stop it when finished: + +```bash +systemctl stop lektor-admin +``` + +### Committing admin edits made on the server + +Lektor can edit files, but it does not automatically create Git commits or push +them. The deployment account intentionally has authenticated Git write access so +a controlled admin workflow can publish server-side edits. + +Review everything before committing: + +```bash +sudo -u labyricorn-deploy env HOME=/srv/labyricorn bash +cd /srv/labyricorn/repo +git status +git diff +git add +git commit -m "Describe the admin edit" +git push origin main +exit +``` + +Then run `sudo deploy-labyricorn`. The deployment script will refuse to run +while uncommitted admin edits remain. This is deliberate protection against +silently discarding edits when it resets to `origin/main`. + +Before enabling persistent or public admin access, add authentication and make a +separate security decision. Do not publish port 5000 directly and do not proxy +an unauthenticated editor through nginx. + +## Git authentication and secrets + +The private Gitea repository is accessed over HTTPS because the Gitea SSH port +was not reachable through `git.labyricorn.com` during setup. + +Credential purpose and location: + +| Purpose | Location | Permissions | +| --- | --- | --- | +| Gitea HTTPS fetch/push | `/srv/labyricorn/.netrc` | `0600`, owned by `labyricorn-deploy` | +| Reserved SSH identity | `/srv/labyricorn/.ssh/id_ed25519` | `0600`, owned by `labyricorn-deploy` | + +Never place token contents, passwords, or private keys in this README, Git URLs, +shell scripts, logs, commits, issue trackers, or command history. + +Check credential-file metadata without printing its contents: + +```bash +stat -c '%A %U:%G %n' /srv/labyricorn/.netrc +``` + +If the Gitea token is rotated, replace `.netrc` as `labyricorn-deploy`, preserve +mode `0600`, and verify both fetch and push access before revoking the old token. +Avoid printing either credential during rotation. + +## Logs and diagnostics + +```bash +# Deployment and rollback history +tail -n 100 /srv/labyricorn/logs/deploy.log + +# nginx service and request errors +journalctl -u nginx --since today +tail -n 100 /var/log/nginx/labyricorn-error.log + +# Optional admin service +journalctl -u lektor-admin --since today + +# Repository state +sudo -u labyricorn-deploy env HOME=/srv/labyricorn \ + git -C /srv/labyricorn/repo status + +# Active release +readlink -f /srv/labyricorn/current +cat /srv/labyricorn/current/.labyricorn-commit + +# Services and sockets +systemctl status nginx +systemctl status certbot.timer +ss -lntup +``` + +## Troubleshooting + +### Deployment says the repository is dirty + +Inspect as the deployment account: + +```bash +sudo -u labyricorn-deploy env HOME=/srv/labyricorn \ + git -C /srv/labyricorn/repo status --short +``` + +If these are intended admin edits, review, commit, and push them. Otherwise, +identify their origin before removing anything. Never use a destructive reset +until you are certain no work needs to be preserved. + +### Git fetch or push fails + +Check DNS, HTTPS, the remote URL, and credential permissions without displaying +the secret: + +```bash +getent ahosts git.labyricorn.com +curl -I https://git.labyricorn.com/ +sudo -u labyricorn-deploy env HOME=/srv/labyricorn \ + git -C /srv/labyricorn/repo remote -v +stat -c '%A %U:%G %n' /srv/labyricorn/.netrc +``` + +An authentication failure usually means the token was revoked, expired, or +lacks repository access. + +### Lektor build fails + +Run a disposable diagnostic build; do not build into `current`: + +```bash +test_dir=$(mktemp -d /srv/labyricorn/releases/.diagnostic.XXXXXX) +chown labyricorn-deploy:labyricorn-deploy "$test_dir" +sudo -u labyricorn-deploy env HOME=/srv/labyricorn \ + /srv/labyricorn/.local/bin/lektor \ + --project /srv/labyricorn/repo/Labyricorn.lektorproject \ + build --output-path "$test_dir" +``` + +Inspect the error and source. Remove the diagnostic directory only after +confirming its exact path and that it is not referenced by `current`. + +### nginx returns an error + +```bash +nginx -t +systemctl status nginx +readlink -f /srv/labyricorn/current +test -f /srv/labyricorn/current/index.html +tail -n 100 /var/log/nginx/labyricorn-error.log +``` + +Do not point nginx at the Git checkout or start Lektor as a production server. + +### Public HTTPS loops or behaves differently from local HTTP + +Check Cloudflare proxy mode and the forwarded protocol behavior: + +```bash +curl -I -H 'Host: www.labyricorn.com' http://127.0.0.1/ +curl -I -H 'Host: www.labyricorn.com' \ + -H 'X-Forwarded-Proto: https' http://127.0.0.1/ +curl -I http://www.labyricorn.com/ +curl -I https://www.labyricorn.com/ +``` + +The first origin request should redirect; the simulated Cloudflare HTTPS request +should return the site. A different result indicates a proxy/header or nginx +configuration problem. + +## Future automatic deployment + +Manual deployment is complete and remains the supported path. Automatic +deployment is not configured. + +The preferred future design is a Gitea Action triggered only by pushes to +`main`, invoking `deploy-labyricorn` through a dedicated, narrowly restricted +SSH identity. A webhook receiver is acceptable only if it authenticates a +secret, verifies the repository and branch, never executes payload text, and +runs with minimal privileges. Never expose a general shell endpoint or an +unauthenticated HTTP deployment URL. + +## Change-control checklist + +Before changing deployment, nginx, TLS, firewall, credentials, or admin access: + +1. Inspect the live state and preserve unrelated configuration. +2. Back up the specific file being changed. +3. Validate syntax before reload or restart. +4. Keep SSH reachable before firewall changes. +5. Verify local origin behavior and public Cloudflare behavior separately. +6. Record operational changes in this README. +7. Commit and push source documentation to `main`. + +This README describes the current verified installation on `ubuntu2-47`. Update +it whenever the operational architecture changes.