Labyricorn website and deployment runbook

This repository is the canonical source for www.labyricorn.com. It contains the editable Lektor project—not generated production files—and is also the primary operations guide for building, deploying, verifying, rolling back, and troubleshooting the site.

Quick reference

Item	Value
Production URL	https://www.labyricorn.com
Canonical Git repository	https://git.labyricorn.com/Labyricorn/labyricorn-site.git
Production branch	main
Production host	ubuntu2-47 (10.138.2.47)
Repository checkout	/srv/labyricorn/repo
Active document root	/srv/labyricorn/current
Release storage	/srv/labyricorn/releases
Deployment command	sudo deploy-labyricorn
Rollback listing	sudo rollback-labyricorn
Rollback command	sudo rollback-labyricorn RELEASE_NAME
Deployment log	/srv/labyricorn/logs/deploy.log
Remote-project cache	/srv/labyricorn/project-cache
nginx site config	/etc/nginx/sites-available/labyricorn
Lektor executable	/srv/labyricorn/.local/bin/lektor
Deployment account	labyricorn-deploy

Architecture

Gitea: Labyricorn/labyricorn-site (main)     public project repositories
              |                                        |
              | authenticated Git fetch                | read-only Git/API sync
              v                                        v
 /srv/labyricorn/repo                    /srv/labyricorn/project-cache
              |                                        |
              +---------- validated build input -------+
                               |
                               | isolated Lektor build
                               v
 /srv/labyricorn/releases/<UTC timestamp>-<commit>
              |
              | validated, atomic symlink switch
              v
 /srv/labyricorn/current
              |
              | read-only static files
              v
            nginx :80
              |
              | Cloudflare proxy and public TLS
              v
 https://www.labyricorn.com

Gitea is the source of truth. The production checkout is disposable state and must not contain unpublished edits during deployment. Lektor's development server is not part of production serving; nginx serves generated files only. Remote project repositories remain authoritative for their own exhibition and devlog records. They are never checked out into the site repository and cannot modify trusted site models, templates, scripts, or content.

Repository contents

Labyricorn.lektorproject  Lektor project definition
content/                  Editable site content (`contents.lr` files)
content/tags/             Controlled tag vocabulary and dedicated tag routes
models/                   Lektor content models
templates/                Jinja templates
assets/static/            CSS, filtering JavaScript, favicon, and static assets
configs/project-sources.ini  Approved public remote-project registry
scripts/project_sources.py  Remote-content validator and importer
scripts/build_with_projects.py  Isolated build/preview entry point
ops/deploy-labyricorn     Versioned copy of the production deployment command
tests/                    Importer validation tests
.gitea/workflows/         Push, schedule, and manual deployment automation
.gitignore                Excludes generated and local files
README.md                 This runbook

Generated output such as build/, dist/, and .lektor/ is intentionally ignored and must not be committed.

Routine content workflow

Edit in a normal Git checkout, review the changes, and push them to main. A successful push triggers the repository's Gitea Action, which builds and atomically activates that exact revision on the production host.

git switch main
git pull --ff-only origin main

# Edit content, templates, models, or assets.

python scripts/build_with_projects.py --output-path build
git status
git add <reviewed-files>
git commit -m "Describe the site change"
git push origin main

Watch the run under Actions in Gitea. Manual deployment remains available for recovery or controlled operations:

sudo deploy-labyricorn

Do not edit generated files under /srv/labyricorn/current or /srv/labyricorn/releases. They will be replaced by deployment and are not source-controlled.

Tags and filtering workflow

content/tags/ is the single source of truth for the controlled tag vocabulary. Each child directory is a normal Lektor record whose directory name is the canonical lowercase, hyphenated slug used in entry data and URLs:

content/tags/deployment/contents.lr  ->  /tags/deployment/

The tag record supplies the editor label and public description:

_model: tag
---
title: Deployment
---
summary: Build, release, rollout, verification, and rollback workflows.

The entry model reads its checkbox choices directly from these records. Blog entries and articles store the selected slugs as a comma-separated tags value:

tags: lektor, deployment, operations

To add a tag, create its content/tags/<slug>/contents.lr record, run a build, and then select it on the relevant entries. Do not add a slug directly to an entry before its tag record exists. To rename or retire a tag, update every referencing entry deliberately and verify that no old slug remains before renaming or removing its record.

/tags/ lists every approved tag and counts its usage across projects, blog entries, and articles. Dedicated tag URLs combine matching records from all three sections in one chronological discovery feed. Tag links are normal links and work without JavaScript. On the Projects, Blog, and Articles listing pages, assets/static/tags.js progressively adds instant filtering and keeps the selected filter in the tag query parameter.

Remote-project languages and declared technologies are normalized to slugs and matched only against this controlled vocabulary. Their approved matches become clickable tags and a deduplicated taxonomy_tags field used by filters, counts, and tag pages. A label that has no approved record remains visible and non-clickable in mustard, produces a build warning, and is excluded from tag discovery. Remote data never creates a tag record automatically.

After taxonomy or filtering changes:

1. Run python scripts/build_with_projects.py --output-path build.
2. Verify /tags/, at least one dedicated tag URL, and all three section listings.
3. Confirm tag counts cover projects, blog entries, and articles; every stored entry slug has a matching tag record; and unapproved project labels are not counted.
4. Test filtering with JavaScript enabled and confirm tag links remain usable without JavaScript.
5. Check keyboard operation plus desktop and narrow layouts.
6. If filtering JavaScript or tag styles changed, increment the corresponding cache-busting version in templates/base.html.

Repository-owned project pages and devlogs

The site can publish a project page whose source lives in the project's own public Git repository. Thinkloom is the first configured source:

https://git.labyricorn.com/Labyricorn/thinkloom-openai-hackathon
  .labyricorn/
  ├── AGENTS.md
  ├── project/
  │   ├── AGENTS.md
  │   ├── contents.lr
  │   └── icon.png
  └── devlog/
      ├── AGENTS.md
      ├── contents.lr
      └── <entry-slug>/contents.lr

The public routes are /projects/thinkloom/, /projects/thinkloom/devlog/, and /projects/thinkloom/devlog/<entry-slug>/. The project repository owns the native Lektor records and approved images. This site owns their models, templates, layout, repository-information card, and import policy.

configs/project-sources.ini is the allowlist. Each source declares a stable project ID, credential-free HTTPS Git URL, public web/API URL, and branch. Add or remove a source only through a reviewed site change. The importer also checks the provider API and refuses a repository reported as private. Remote Git commands run with an isolated empty home/config and interactive credential lookup disabled, so the production account's site-repository credential cannot silently turn a private project fetch into an authenticated one. An anonymous API response of 401, 403, or 404 is treated as a visibility failure rather than a metadata-cache condition.

The importer reads only these paths:

• .labyricorn/project/contents.lr
• .labyricorn/project/*.{png,jpg,jpeg,webp}
• .labyricorn/devlog/contents.lr
• .labyricorn/devlog/<entry-slug>/contents.lr
• approved image types within a devlog entry directory

Everything else is ignored, including remote README.md and AGENTS.md instructions. Symbolic links, submodules, executable files, raw HTML, unsupported models or schema versions, invalid slugs, unknown paths, out-of-tree references, and invalid image signatures are rejected. Limits are 100 imported files, 5 MiB per file, and 20 MiB total per project snapshot. Devlog source_commit values must exist in the repository and be ancestors of the imported branch revision.

Every build uses a temporary copy of trusted site source, materializes validated remote records into that copy, and invokes Lektor there. Imported files never enter the site working tree. Git mirrors, state, and the last validated snapshot live under the project cache. If a new snapshot or network fetch fails, the build uses the last-known-good validated revision. Initial publication fails when no valid snapshot exists. The generated .labyricorn-projects.json records the site revision, imported project revisions, metadata digests, synchronization status, and synchronization time.

The project card combines repository-owned content with public provider and Git metadata: source and README links, default branch, latest commit, license, languages, open issues, stars, forks, latest release, imported revision, and sync time. Approved language and technology labels link into the shared tag taxonomy, allowing a project to lead visitors to related writing. API metadata failure uses cached metadata and is identified in the sync status; it never causes an unvalidated new snapshot to replace a working one.

To preview or build with remote project content, use the wrapper instead of calling Lektor directly:

python scripts/build_with_projects.py --serve
python scripts/build_with_projects.py --output-path build
python -m unittest discover -s tests -v

The default development cache is .cache/project-sources, which is ignored by Git. Lektor must be installed because the importer deliberately uses Lektor's native record parser and the wrapper invokes the Lektor build/server. No custom Lektor plugin or third-party project-import package is used.

Local development

Install Lektor in an isolated Python environment. For example:

python3 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install lektor

Run the local editor and preview server with validated project content:

python scripts/build_with_projects.py --serve

Run a production-style build:

python scripts/build_with_projects.py --output-path build
test -f build/index.html

The project currently has no third-party Lektor packages or plugins.

Production filesystem and permissions

/srv/labyricorn/
├── repo/                 Git working copy (not web-accessible)
├── releases/             Successful immutable builds
├── current -> releases/<release>
├── logs/
│   ├── deploy.log
│   └── deploy.lock
├── project-cache/        Bare mirrors, metadata, and last-known-good state
├── .local/bin/lektor     pipx-installed Lektor
├── .netrc                Git HTTPS credential (0600; secret)
├── .ssh/                 Reserved deployment SSH material
└── README-DEPLOYMENT.md  Supplemental host-local notes

The labyricorn-deploy account owns the checkout, releases, logs, Lektor environment, and Git credential. nginx's www-data account can traverse the base path and read active generated files, but cannot read the source checkout or credential. nginx cannot modify either source or releases.

Deployment behavior

/usr/local/bin/deploy-labyricorn performs the following sequence:

1. Changes from root to the unprivileged labyricorn-deploy account.
2. Acquires an exclusive flock lock to prevent concurrent deployments.
3. Refuses to proceed if the working tree contains any tracked or untracked changes.
4. Fetches and prunes origin.
5. Verifies that origin/main exists.
6. Switches to local main and resets it exactly to origin/main.
7. Creates a unique release named with UTC time and the 12-character commit ID.
8. Runs scripts/build_with_projects.py with the persistent project cache and an explicit output directory.
9. Fetches, validates, and imports each approved public project source into an isolated temporary workspace before invoking Lektor.
10. Requires index.html and .labyricorn-projects.json.
11. Writes the full site commit ID to .labyricorn-commit.
12. Compares the site commit plus project commit/metadata digests with the active release and removes the candidate as an unchanged no-op when equal.
13. Atomically replaces /srv/labyricorn/current with a relative symlink to the new release.
14. Keeps the active release plus recent prior releases, targeting five total.
15. Logs start, failure, unchanged, success, release name, and commit.

Run it with:

sudo deploy-labyricorn

A failed build is removed before activation. The current symlink is not changed, so the previously working site remains online.

Deployment preflight

sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo status --short --branch

sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo fetch origin

The status must be clean. Do not bypass a dirty-tree refusal. Determine whether the files are legitimate unpublished admin edits, then commit and push them or remove them deliberately before deployment.

Verify a deployment

readlink -f /srv/labyricorn/current
cat /srv/labyricorn/current/.labyricorn-commit

sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo rev-parse HEAD

sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo ls-remote origin refs/heads/main

nginx -t
systemctl is-active nginx

curl -I -H 'Host: www.labyricorn.com' \
  -H 'X-Forwarded-Proto: https' http://127.0.0.1/

curl -I https://www.labyricorn.com/

The commit in the active release, local HEAD, and remote main should match. The local nginx request with X-Forwarded-Proto: https should return 200. Public HTTP should redirect to HTTPS, and public HTTPS should return 200.

Rollback

List releases and identify the current one:

sudo rollback-labyricorn

Switch atomically to a previous successful release:

sudo rollback-labyricorn 20260812T024742Z-92475d2ec759

Use a release name printed by the listing command. Rollback does not build, fetch, or alter Git. It only changes the current symlink and records the event in the deployment log.

Verify afterward:

readlink -f /srv/labyricorn/current
cat /srv/labyricorn/current/.labyricorn-commit
curl -I -H 'Host: www.labyricorn.com' \
  -H 'X-Forwarded-Proto: https' http://127.0.0.1/

A normal deployment after rollback rebuilds and reactivates the latest remote main commit.

nginx and URL handling

The site configuration is:

/etc/nginx/sites-available/labyricorn
/etc/nginx/sites-enabled/labyricorn -> ../sites-available/labyricorn

nginx serves /srv/labyricorn/current and supports:

• index.html at directory-style URLs;
• direct static assets;
• optional .html fallback;
• generated /404/index.html with an HTTP 404 response;
• disabled directory browsing;
• rejection of hidden paths and common executable-script extensions;
• dedicated access and error logs.

Always validate before reloading nginx:

nginx -t
systemctl reload nginx

nginx logs:

/var/log/nginx/labyricorn-access.log
/var/log/nginx/labyricorn-error.log

HTTPS, Cloudflare, and DNS

Current production HTTPS terminates at Cloudflare. The origin nginx process currently listens on port 80, not 443. For www.labyricorn.com, nginx uses Cloudflare's X-Forwarded-Proto header to distinguish public HTTP from HTTPS:

• public HTTP redirects to https://www.labyricorn.com;
• Cloudflare-originated HTTPS requests are served without a redirect loop;
• labyricorn.com is configured at nginx to redirect to the canonical HTTPS www hostname.

The apex labyricorn.com did not resolve during initial setup. Its redirect will not work publicly until DNS/Cloudflare has an apex record routed to this origin.

Certbot and the nginx integration are installed, and certbot.timer is enabled, but no origin certificate was issued during setup. Do not run Certbot blindly behind Cloudflare. First decide whether the desired origin mode is a Cloudflare Origin CA certificate, Let's Encrypt with a compatible challenge, or direct DNS without the proxy. After changing TLS, test both origin and public behavior for redirect loops.

Useful checks:

getent ahosts www.labyricorn.com
getent ahosts labyricorn.com
curl -I http://www.labyricorn.com/
curl -I https://www.labyricorn.com/
systemctl status certbot.timer
certbot certificates

Firewall and exposed services

UFW is active with default incoming denial. Allowed services are:

• OpenSSH: TCP 22
• nginx HTTP/HTTPS profile: TCP 80 and 443

Inspect current state:

ufw status verbose
ss -lntup

Lektor's development port must never be opened in UFW or bound to 0.0.0.0.

Optional server-side Lektor admin

An optional unit exists at:

/etc/systemd/system/lektor-admin.service

Current state: disabled and stopped. When started, it runs as labyricorn-deploy and binds only to 127.0.0.1:5000. It is not proxied by nginx and production remains independent of it.

Start it temporarily:

systemctl start lektor-admin
systemctl status lektor-admin
ss -lntp | grep ':5000'

Access it securely through an SSH tunnel from an administrator workstation:

ssh -L 5000:127.0.0.1:5000 [email protected]

Then browse to http://127.0.0.1:5000 on that workstation.

Stop it when finished:

systemctl stop lektor-admin

Committing admin edits made on the server

Lektor can edit files, but it does not automatically create Git commits or push them. The deployment account intentionally has authenticated Git write access so a controlled admin workflow can publish server-side edits.

Review everything before committing:

sudo -u labyricorn-deploy env HOME=/srv/labyricorn bash
cd /srv/labyricorn/repo
git status
git diff
git add <reviewed-files>
git commit -m "Describe the admin edit"
git push origin main
exit

Then run sudo deploy-labyricorn. The deployment script will refuse to run while uncommitted admin edits remain. This is deliberate protection against silently discarding edits when it resets to origin/main.

Before enabling persistent or public admin access, add authentication and make a separate security decision. Do not publish port 5000 directly and do not proxy an unauthenticated editor through nginx.

Git authentication and secrets

The site repository is accessed over authenticated HTTPS because the Gitea SSH port was not reachable through git.labyricorn.com during setup. Approved remote project repositories, including Thinkloom, are public and are fetched without embedding credentials in their configured URLs.

Credential purpose and location:

Purpose	Location	Permissions
Gitea HTTPS fetch/push	/srv/labyricorn/.netrc	0600, owned by labyricorn-deploy
Reserved SSH identity	/srv/labyricorn/.ssh/id_ed25519	0600, owned by labyricorn-deploy

Never place token contents, passwords, or private keys in this README, Git URLs, shell scripts, logs, commits, issue trackers, or command history.

Check credential-file metadata without printing its contents:

stat -c '%A %U:%G %n' /srv/labyricorn/.netrc

If the Gitea token is rotated, replace .netrc as labyricorn-deploy, preserve mode 0600, and verify both fetch and push access before revoking the old token. Avoid printing either credential during rotation.

Logs and diagnostics

# Deployment and rollback history
tail -n 100 /srv/labyricorn/logs/deploy.log

# nginx service and request errors
journalctl -u nginx --since today
tail -n 100 /var/log/nginx/labyricorn-error.log

# Optional admin service
journalctl -u lektor-admin --since today

# Repository state
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo status

# Active release
readlink -f /srv/labyricorn/current
cat /srv/labyricorn/current/.labyricorn-commit

# Services and sockets
systemctl status nginx
systemctl status certbot.timer
ss -lntup

Troubleshooting

Deployment says the repository is dirty

Inspect as the deployment account:

sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo status --short

If these are intended admin edits, review, commit, and push them. Otherwise, identify their origin before removing anything. Never use a destructive reset until you are certain no work needs to be preserved.

Git fetch or push fails

Check DNS, HTTPS, the remote URL, and credential permissions without displaying the secret:

getent ahosts git.labyricorn.com
curl -I https://git.labyricorn.com/
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo remote -v
stat -c '%A %U:%G %n' /srv/labyricorn/.netrc

An authentication failure usually means the token was revoked, expired, or lacks repository access.

Lektor build fails

Run a disposable diagnostic build; do not build into current:

test_dir=$(mktemp -d /srv/labyricorn/releases/.diagnostic.XXXXXX)
chown labyricorn-deploy:labyricorn-deploy "$test_dir"
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  /srv/labyricorn/.local/pipx/venvs/lektor/bin/python \
  /srv/labyricorn/repo/scripts/build_with_projects.py \
  --site-root /srv/labyricorn/repo \
  --cache-path /srv/labyricorn/project-cache \
  --output-path "$test_dir" \
  --lektor /srv/labyricorn/.local/bin/lektor

Inspect the error and source. Remove the diagnostic directory only after confirming its exact path and that it is not referenced by current.

nginx returns an error

nginx -t
systemctl status nginx
readlink -f /srv/labyricorn/current
test -f /srv/labyricorn/current/index.html
tail -n 100 /var/log/nginx/labyricorn-error.log

Do not point nginx at the Git checkout or start Lektor as a production server.

Public HTTPS loops or behaves differently from local HTTP

Check Cloudflare proxy mode and the forwarded protocol behavior:

curl -I -H 'Host: www.labyricorn.com' http://127.0.0.1/
curl -I -H 'Host: www.labyricorn.com' \
  -H 'X-Forwarded-Proto: https' http://127.0.0.1/
curl -I http://www.labyricorn.com/
curl -I https://www.labyricorn.com/

The first origin request should redirect; the simulated Cloudflare HTTPS request should return the site. A different result indicates a proxy/header or nginx configuration problem.

Remote project synchronization fails

Inspect the deployment log and active manifest without editing the cache:

tail -n 100 /srv/labyricorn/logs/deploy.log
cat /srv/labyricorn/current/.labyricorn-projects.json

current means the latest remote revision and provider metadata validated. metadata-cached means Git content is current but provider metadata fell back to its cache. last-known-good means the latest fetch or candidate snapshot could not safely replace the previously validated commit. A first import with no valid cached snapshot fails the deployment. Do not bypass validation by copying remote files into content/, the build workspace, or an active release.

Automatic deployment and project refresh

The workflow at .gitea/workflows/deploy.yml runs for three event types:

• every push to site main;
• every 30 minutes (*/30 * * * *, evaluated by Gitea in UTC);
• an authenticated manual workflow_dispatch.

All events run the same one-job deployment:

1. Run on the repository-scoped runner labelled labyricorn-deploy.
2. Execute the existing root-owned deploy-labyricorn command.
3. Confirm /srv/labyricorn/current/.labyricorn-commit equals the workflow's site commit.
4. Request the site through nginx on the local origin and fail if it is unhealthy.

The action intentionally does not check out the repository into its own workspace. The deployment script fetches origin/main using the dedicated labyricorn-deploy account, validates a clean checkout, builds a new release, and atomically changes the current symlink. Concurrent action runs share the labyricorn-production concurrency group, and the deployment script also uses a filesystem lock. A scheduled/manual run updates the site when an approved project commit or public metadata digest changes. When neither the site nor any project input changed, it validates and builds a candidate, removes it, logs an unchanged result, and leaves the active symlink untouched.

Trigger a refresh

From Gitea, open Labyricorn/labyricorn-site → Actions → Deploy production and choose Run workflow for main.

Automation and coding assistants may dispatch the same workflow through Gitea's authenticated API after pushing project publishing content:

export LABYRICORN_REFRESH_TOKEN='set this outside Git and shell history'
curl --fail --silent --show-error \
  -X POST \
  -H "Authorization: token $LABYRICORN_REFRESH_TOKEN" \
  -H 'Content-Type: application/json' \
  --data '{"ref":"main"}' \
  https://git.labyricorn.com/api/v1/repos/Labyricorn/labyricorn-site/actions/workflows/deploy.yml/dispatches
unset LABYRICORN_REFRESH_TOKEN

The token must be supplied through an approved credential store or ephemeral environment and must have permission to dispatch Actions for the site repository. Never put it in the URL, a project repository, AGENTS.md, logs, or committed scripts. Send at most one dispatch after the relevant project push, then verify the Action and public project revision. The scheduled run is the fallback when no refresh credential is available. This is deliberately not an unauthenticated public webhook or long-running URL listener, so it cannot be spammed by anonymous requests.

Runner installation

The production host runs official Gitea Runner 3.0.0 directly on Linux; Docker is not installed or required for this workflow.

Item	Value
systemd unit	gitea-runner.service
service account	gitea-runner
runner name	labyricorn-production
registration scope	Labyricorn/labyricorn-site repository
execution label	labyricorn-deploy:host
runner binary	/usr/local/bin/gitea-runner
runner configuration	/etc/gitea-runner/config.yaml
registration state	/var/lib/gitea-runner/.runner
job workspace	/var/lib/gitea-runner/work
sudo policy	/etc/sudoers.d/gitea-runner-labyricorn

The runner account has no general administrative access. Its sudo policy allows only this exact command without a password:

/usr/local/bin/deploy-labyricorn

Useful checks:

systemctl status gitea-runner
journalctl -u gitea-runner -n 100 --no-pager
sudo -l -U gitea-runner

If the runner must be replaced, delete or disable it in Repository Settings → Actions → Runners, generate a fresh repository registration token, and register the replacement. Registration tokens and .runner contents are credentials: never commit or print them. A push can still be deployed manually with sudo deploy-labyricorn while the runner is unavailable.

Change-control checklist

Before changing deployment, nginx, TLS, firewall, credentials, or admin access:

1. Inspect the live state and preserve unrelated configuration.
2. Back up the specific file being changed.
3. Validate syntax before reload or restart.
4. Keep SSH reachable before firewall changes.
5. Verify local origin behavior and public Cloudflare behavior separately.
6. Record operational changes in this README.
7. Commit and push source documentation to main.

This README describes the current verified installation on ubuntu2-47. Update it whenever the operational architecture changes.