labyricorn-site/README.md

# Labyricorn website and deployment runbook

This repository is the canonical source for `www.labyricorn.com`. It contains
the editable Lektor project—not generated production files—and is also the
primary operations guide for building, deploying, verifying, rolling back, and
troubleshooting the site.

## Quick reference

| Item | Value |
| --- | --- |
| Production URL | `https://www.labyricorn.com` |
| Canonical Git repository | `https://git.labyricorn.com/Labyricorn/labyricorn-site.git` |
| Production branch | `main` |
| Production host | `ubuntu2-47` (`10.138.2.47`) |
| Repository checkout | `/srv/labyricorn/repo` |
| Active document root | `/srv/labyricorn/current` |
| Release storage | `/srv/labyricorn/releases` |
| Deployment command | `sudo deploy-labyricorn` |
| Rollback listing | `sudo rollback-labyricorn` |
| Rollback command | `sudo rollback-labyricorn RELEASE_NAME` |
| Deployment log | `/srv/labyricorn/logs/deploy.log` |
| nginx site config | `/etc/nginx/sites-available/labyricorn` |
| Lektor executable | `/srv/labyricorn/.local/bin/lektor` |
| Deployment account | `labyricorn-deploy` |

## Architecture

```text
Gitea: Labyricorn/labyricorn-site (main)
              |
              | authenticated Git fetch
              v
 /srv/labyricorn/repo
              |
              | Lektor static build
              v
 /srv/labyricorn/releases/<UTC timestamp>-<commit>
              |
              | validated, atomic symlink switch
              v
 /srv/labyricorn/current
              |
              | read-only static files
              v
            nginx :80
              |
              | Cloudflare proxy and public TLS
              v
 https://www.labyricorn.com
```

Gitea is the source of truth. The production checkout is disposable state and
must not contain unpublished edits during deployment. Lektor's development
server is not part of production serving; nginx serves generated files only.

## Repository contents

```text
Labyricorn.lektorproject  Lektor project definition
content/                  Editable site content (`contents.lr` files)
content/tags/             Controlled tag vocabulary and dedicated tag routes
models/                   Lektor content models
templates/                Jinja templates
assets/static/            CSS, filtering JavaScript, favicon, and static assets
.gitignore                Excludes generated and local files
README.md                 This runbook
```

Generated output such as `build/`, `dist/`, and `.lektor/` is intentionally
ignored and must not be committed.

## Routine content workflow

Edit in a normal Git checkout, review the changes, and push them to `main`.
A successful push triggers the repository's Gitea Action, which builds and
atomically activates that exact revision on the production host.

```bash
git switch main
git pull --ff-only origin main

# Edit content, templates, models, or assets.

lektor build --output-path build
git status
git add <reviewed-files>
git commit -m "Describe the site change"
git push origin main
```

Watch the run under **Actions** in Gitea. Manual deployment remains available
for recovery or controlled operations:

```bash
sudo deploy-labyricorn
```

Do not edit generated files under `/srv/labyricorn/current` or
`/srv/labyricorn/releases`. They will be replaced by deployment and are not
source-controlled.

## Tags and filtering workflow

`content/tags/` is the single source of truth for the controlled tag
vocabulary. Each child directory is a normal Lektor record whose directory name
is the canonical lowercase, hyphenated slug used in entry data and URLs:

```text
content/tags/deployment/contents.lr  ->  /tags/deployment/
```

The tag record supplies the editor label and public description:

    _model: tag
    ---
    title: Deployment
    ---
    summary: Build, release, rollout, verification, and rollback workflows.

The entry model reads its checkbox choices directly from these records. Blog
entries and articles store the selected slugs as a comma-separated `tags`
value:

```text
tags: lektor, deployment, operations
```

To add a tag, create its `content/tags/<slug>/contents.lr` record, run a build,
and then select it on the relevant entries. Do not add a slug directly to an
entry before its tag record exists. To rename or retire a tag, update every
referencing entry deliberately and verify that no old slug remains before
renaming or removing its record.

`/tags/` lists every approved tag and counts its usage across blog entries and
articles. Dedicated tag URLs list matching entries from both sections. Tag
links are normal links and work without JavaScript. On the Blog and Articles
listing pages, `assets/static/tags.js` progressively adds instant filtering and
keeps the selected filter in the `tag` query parameter.

After taxonomy or filtering changes:

1. Run `lektor build --output-path build`.
2. Verify `/tags/`, at least one dedicated tag URL, and both section listings.
3. Confirm tag counts cover blog entries and articles and that every entry slug
   has a matching tag record.
4. Test filtering with JavaScript enabled and confirm tag links remain usable
   without JavaScript.
5. Check keyboard operation plus desktop and narrow layouts.
6. If filtering JavaScript or tag styles changed, increment the corresponding
   cache-busting version in `templates/base.html`.

## Local development

Install Lektor in an isolated Python environment. For example:

```bash
python3 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install lektor
```

Run the local editor and preview server:

```bash
lektor server
```

Run a production-style build:

```bash
lektor build --output-path build
test -f build/index.html
```

The project currently has no third-party Lektor packages or plugins.

## Production filesystem and permissions

```text
/srv/labyricorn/
├── repo/                 Git working copy (not web-accessible)
├── releases/             Successful immutable builds
├── current -> releases/<release>
├── logs/
│   ├── deploy.log
│   └── deploy.lock
├── .local/bin/lektor     pipx-installed Lektor
├── .netrc                Git HTTPS credential (0600; secret)
├── .ssh/                 Reserved deployment SSH material
└── README-DEPLOYMENT.md  Supplemental host-local notes
```

The `labyricorn-deploy` account owns the checkout, releases, logs, Lektor
environment, and Git credential. nginx's `www-data` account can traverse the
base path and read active generated files, but cannot read the source checkout
or credential. nginx cannot modify either source or releases.

## Deployment behavior

`/usr/local/bin/deploy-labyricorn` performs the following sequence:

1. Changes from root to the unprivileged `labyricorn-deploy` account.
2. Acquires an exclusive `flock` lock to prevent concurrent deployments.
3. Refuses to proceed if the working tree contains any tracked or untracked
   changes.
4. Fetches and prunes `origin`.
5. Verifies that `origin/main` exists.
6. Switches to local `main` and resets it exactly to `origin/main`.
7. Locates exactly one `.lektorproject` file instead of assuming its name.
8. Creates a unique release named with UTC time and the 12-character commit ID.
9. Runs Lektor with an explicit output directory.
10. Requires the build to succeed and contain `index.html`.
11. Writes the full commit ID to `.labyricorn-commit` in the release.
12. Atomically replaces `/srv/labyricorn/current` with a relative symlink to
    the new release.
13. Keeps the active release plus recent prior releases, targeting five total.
14. Logs start, failure, success, release name, and commit.

Run it with:

```bash
sudo deploy-labyricorn
```

A failed build is removed before activation. The current symlink is not changed,
so the previously working site remains online.

### Deployment preflight

```bash
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo status --short --branch

sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo fetch origin
```

The status must be clean. Do not bypass a dirty-tree refusal. Determine whether
the files are legitimate unpublished admin edits, then commit and push them or
remove them deliberately before deployment.

## Verify a deployment

```bash
readlink -f /srv/labyricorn/current
cat /srv/labyricorn/current/.labyricorn-commit

sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo rev-parse HEAD

sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo ls-remote origin refs/heads/main

nginx -t
systemctl is-active nginx

curl -I -H 'Host: www.labyricorn.com' \
  -H 'X-Forwarded-Proto: https' http://127.0.0.1/

curl -I https://www.labyricorn.com/
```

The commit in the active release, local `HEAD`, and remote `main` should match.
The local nginx request with `X-Forwarded-Proto: https` should return `200`.
Public HTTP should redirect to HTTPS, and public HTTPS should return `200`.

## Rollback

List releases and identify the current one:

```bash
sudo rollback-labyricorn
```

Switch atomically to a previous successful release:

```bash
sudo rollback-labyricorn 20260812T024742Z-92475d2ec759
```

Use a release name printed by the listing command. Rollback does not build,
fetch, or alter Git. It only changes the `current` symlink and records the event
in the deployment log.

Verify afterward:

```bash
readlink -f /srv/labyricorn/current
cat /srv/labyricorn/current/.labyricorn-commit
curl -I -H 'Host: www.labyricorn.com' \
  -H 'X-Forwarded-Proto: https' http://127.0.0.1/
```

A normal deployment after rollback rebuilds and reactivates the latest remote
`main` commit.

## nginx and URL handling

The site configuration is:

```text
/etc/nginx/sites-available/labyricorn
/etc/nginx/sites-enabled/labyricorn -> ../sites-available/labyricorn
```

nginx serves `/srv/labyricorn/current` and supports:

- `index.html` at directory-style URLs;
- direct static assets;
- optional `.html` fallback;
- generated `/404/index.html` with an HTTP `404` response;
- disabled directory browsing;
- rejection of hidden paths and common executable-script extensions;
- dedicated access and error logs.

Always validate before reloading nginx:

```bash
nginx -t
systemctl reload nginx
```

nginx logs:

```text
/var/log/nginx/labyricorn-access.log
/var/log/nginx/labyricorn-error.log
```

## HTTPS, Cloudflare, and DNS

Current production HTTPS terminates at Cloudflare. The origin nginx process
currently listens on port `80`, not `443`. For `www.labyricorn.com`, nginx uses
Cloudflare's `X-Forwarded-Proto` header to distinguish public HTTP from HTTPS:

- public HTTP redirects to `https://www.labyricorn.com`;
- Cloudflare-originated HTTPS requests are served without a redirect loop;
- `labyricorn.com` is configured at nginx to redirect to the canonical HTTPS
  `www` hostname.

The apex `labyricorn.com` did not resolve during initial setup. Its redirect will
not work publicly until DNS/Cloudflare has an apex record routed to this origin.

Certbot and the nginx integration are installed, and `certbot.timer` is enabled,
but no origin certificate was issued during setup. Do not run Certbot blindly
behind Cloudflare. First decide whether the desired origin mode is a Cloudflare
Origin CA certificate, Let's Encrypt with a compatible challenge, or direct DNS
without the proxy. After changing TLS, test both origin and public behavior for
redirect loops.

Useful checks:

```bash
getent ahosts www.labyricorn.com
getent ahosts labyricorn.com
curl -I http://www.labyricorn.com/
curl -I https://www.labyricorn.com/
systemctl status certbot.timer
certbot certificates
```

## Firewall and exposed services

UFW is active with default incoming denial. Allowed services are:

- OpenSSH: TCP 22
- nginx HTTP/HTTPS profile: TCP 80 and 443

Inspect current state:

```bash
ufw status verbose
ss -lntup
```

Lektor's development port must never be opened in UFW or bound to `0.0.0.0`.

## Optional server-side Lektor admin

An optional unit exists at:

```text
/etc/systemd/system/lektor-admin.service
```

Current state: **disabled and stopped**. When started, it runs as
`labyricorn-deploy` and binds only to `127.0.0.1:5000`. It is not proxied by
nginx and production remains independent of it.

Start it temporarily:

```bash
systemctl start lektor-admin
systemctl status lektor-admin
ss -lntp | grep ':5000'
```

Access it securely through an SSH tunnel from an administrator workstation:

```bash
ssh -L 5000:127.0.0.1:5000 [email protected]
```

Then browse to `http://127.0.0.1:5000` on that workstation.

Stop it when finished:

```bash
systemctl stop lektor-admin
```

### Committing admin edits made on the server

Lektor can edit files, but it does not automatically create Git commits or push
them. The deployment account intentionally has authenticated Git write access so
a controlled admin workflow can publish server-side edits.

Review everything before committing:

```bash
sudo -u labyricorn-deploy env HOME=/srv/labyricorn bash
cd /srv/labyricorn/repo
git status
git diff
git add <reviewed-files>
git commit -m "Describe the admin edit"
git push origin main
exit
```

Then run `sudo deploy-labyricorn`. The deployment script will refuse to run
while uncommitted admin edits remain. This is deliberate protection against
silently discarding edits when it resets to `origin/main`.

Before enabling persistent or public admin access, add authentication and make a
separate security decision. Do not publish port 5000 directly and do not proxy
an unauthenticated editor through nginx.

## Git authentication and secrets

The private Gitea repository is accessed over HTTPS because the Gitea SSH port
was not reachable through `git.labyricorn.com` during setup.

Credential purpose and location:

| Purpose | Location | Permissions |
| --- | --- | --- |
| Gitea HTTPS fetch/push | `/srv/labyricorn/.netrc` | `0600`, owned by `labyricorn-deploy` |
| Reserved SSH identity | `/srv/labyricorn/.ssh/id_ed25519` | `0600`, owned by `labyricorn-deploy` |

Never place token contents, passwords, or private keys in this README, Git URLs,
shell scripts, logs, commits, issue trackers, or command history.

Check credential-file metadata without printing its contents:

```bash
stat -c '%A %U:%G %n' /srv/labyricorn/.netrc
```

If the Gitea token is rotated, replace `.netrc` as `labyricorn-deploy`, preserve
mode `0600`, and verify both fetch and push access before revoking the old token.
Avoid printing either credential during rotation.

## Logs and diagnostics

```bash
# Deployment and rollback history
tail -n 100 /srv/labyricorn/logs/deploy.log

# nginx service and request errors
journalctl -u nginx --since today
tail -n 100 /var/log/nginx/labyricorn-error.log

# Optional admin service
journalctl -u lektor-admin --since today

# Repository state
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo status

# Active release
readlink -f /srv/labyricorn/current
cat /srv/labyricorn/current/.labyricorn-commit

# Services and sockets
systemctl status nginx
systemctl status certbot.timer
ss -lntup
```

## Troubleshooting

### Deployment says the repository is dirty

Inspect as the deployment account:

```bash
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo status --short
```

If these are intended admin edits, review, commit, and push them. Otherwise,
identify their origin before removing anything. Never use a destructive reset
until you are certain no work needs to be preserved.

### Git fetch or push fails

Check DNS, HTTPS, the remote URL, and credential permissions without displaying
the secret:

```bash
getent ahosts git.labyricorn.com
curl -I https://git.labyricorn.com/
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo remote -v
stat -c '%A %U:%G %n' /srv/labyricorn/.netrc
```

An authentication failure usually means the token was revoked, expired, or
lacks repository access.

### Lektor build fails

Run a disposable diagnostic build; do not build into `current`:

```bash
test_dir=$(mktemp -d /srv/labyricorn/releases/.diagnostic.XXXXXX)
chown labyricorn-deploy:labyricorn-deploy "$test_dir"
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  /srv/labyricorn/.local/bin/lektor \
  --project /srv/labyricorn/repo/Labyricorn.lektorproject \
  build --output-path "$test_dir"
```

Inspect the error and source. Remove the diagnostic directory only after
confirming its exact path and that it is not referenced by `current`.

### nginx returns an error

```bash
nginx -t
systemctl status nginx
readlink -f /srv/labyricorn/current
test -f /srv/labyricorn/current/index.html
tail -n 100 /var/log/nginx/labyricorn-error.log
```

Do not point nginx at the Git checkout or start Lektor as a production server.

### Public HTTPS loops or behaves differently from local HTTP

Check Cloudflare proxy mode and the forwarded protocol behavior:

```bash
curl -I -H 'Host: www.labyricorn.com' http://127.0.0.1/
curl -I -H 'Host: www.labyricorn.com' \
  -H 'X-Forwarded-Proto: https' http://127.0.0.1/
curl -I http://www.labyricorn.com/
curl -I https://www.labyricorn.com/
```

The first origin request should redirect; the simulated Cloudflare HTTPS request
should return the site. A different result indicates a proxy/header or nginx
configuration problem.

## Automatic deployment

Pushes to `main` automatically deploy through Gitea Actions. The workflow is
versioned with the site at `.gitea/workflows/deploy.yml` and has one job:

1. Run on the repository-scoped runner labelled `labyricorn-deploy`.
2. Execute the existing root-owned `deploy-labyricorn` command.
3. Confirm `/srv/labyricorn/current/.labyricorn-commit` equals the pushed commit.
4. Request the site through nginx on the local origin and fail if it is unhealthy.

The action intentionally does not check out the repository into its own
workspace. The deployment script fetches `origin/main` using the dedicated
`labyricorn-deploy` account, validates a clean checkout, builds a new release,
and atomically changes the `current` symlink. Concurrent action runs share the
`labyricorn-production` concurrency group, and the deployment script also uses
a filesystem lock.

### Runner installation

The production host runs official Gitea Runner 3.0.0 directly on Linux; Docker
is not installed or required for this workflow.

| Item | Value |
|---|---|
| systemd unit | `gitea-runner.service` |
| service account | `gitea-runner` |
| runner name | `labyricorn-production` |
| registration scope | `Labyricorn/labyricorn-site` repository |
| execution label | `labyricorn-deploy:host` |
| runner binary | `/usr/local/bin/gitea-runner` |
| runner configuration | `/etc/gitea-runner/config.yaml` |
| registration state | `/var/lib/gitea-runner/.runner` |
| job workspace | `/var/lib/gitea-runner/work` |
| sudo policy | `/etc/sudoers.d/gitea-runner-labyricorn` |

The runner account has no general administrative access. Its sudo policy allows
only this exact command without a password:

```text
/usr/local/bin/deploy-labyricorn
```

Useful checks:

```bash
systemctl status gitea-runner
journalctl -u gitea-runner -n 100 --no-pager
sudo -l -U gitea-runner
```

If the runner must be replaced, delete or disable it in **Repository Settings →
Actions → Runners**, generate a fresh repository registration token, and
register the replacement. Registration tokens and `.runner` contents are
credentials: never commit or print them. A push can still be deployed manually
with `sudo deploy-labyricorn` while the runner is unavailable.

## Change-control checklist

Before changing deployment, nginx, TLS, firewall, credentials, or admin access:

1. Inspect the live state and preserve unrelated configuration.
2. Back up the specific file being changed.
3. Validate syntax before reload or restart.
4. Keep SSH reachable before firewall changes.
5. Verify local origin behavior and public Cloudflare behavior separately.
6. Record operational changes in this README.
7. Commit and push source documentation to `main`.

This README describes the current verified installation on `ubuntu2-47`. Update
it whenever the operational architecture changes.