labyricorn-site/README.md

# Labyricorn website and deployment runbook

This repository is the canonical source for `www.labyricorn.com`. It contains
the editable Lektor project—not generated production files—and is also the
primary operations guide for building, deploying, verifying, rolling back, and
troubleshooting the site.

## Quick reference

| Item | Value |
| --- | --- |
| Production URL | `https://www.labyricorn.com` |
| Canonical Git repository | `https://git.labyricorn.com/Labyricorn/labyricorn-site.git` |
| Production branch | `main` |
| Production host | `ubuntu2-47` (`10.138.2.47`) |
| Repository checkout | `/srv/labyricorn/repo` |
| Active document root | `/srv/labyricorn/current` |
| Release storage | `/srv/labyricorn/releases` |
| Deployment command | `sudo deploy-labyricorn` |
| Rollback listing | `sudo rollback-labyricorn` |
| Rollback command | `sudo rollback-labyricorn RELEASE_NAME` |
| Deployment log | `/srv/labyricorn/logs/deploy.log` |
| nginx site config | `/etc/nginx/sites-available/labyricorn` |
| Lektor executable | `/srv/labyricorn/.local/bin/lektor` |
| Deployment account | `labyricorn-deploy` |

## Architecture

```text
Gitea: Labyricorn/labyricorn-site (main)
              |
              | authenticated Git fetch
              v
 /srv/labyricorn/repo
              |
              | Lektor static build
              v
 /srv/labyricorn/releases/<UTC timestamp>-<commit>
              |
              | validated, atomic symlink switch
              v
 /srv/labyricorn/current
              |
              | read-only static files
              v
            nginx :80
              |
              | Cloudflare proxy and public TLS
              v
 https://www.labyricorn.com
```

Gitea is the source of truth. The production checkout is disposable state and
must not contain unpublished edits during deployment. Lektor's development
server is not part of production serving; nginx serves generated files only.

## Repository contents

```text
Labyricorn.lektorproject  Lektor project definition
content/                  Editable site content (`contents.lr` files)
models/                   Lektor content models
templates/                Jinja templates
assets/static/            CSS, favicon, and other static source assets
.gitignore                Excludes generated and local files
README.md                 This runbook
```

Generated output such as `build/`, `dist/`, and `.lektor/` is intentionally
ignored and must not be committed.

## Routine content workflow

The preferred workflow is to edit in a normal Git checkout, review the changes,
push them to `main`, and then deploy on the production host.

```bash
git switch main
git pull --ff-only origin main

# Edit content, templates, models, or assets.

lektor build --output-path build
git status
git add <reviewed-files>
git commit -m "Describe the site change"
git push origin main
```

Then deploy on `ubuntu2-47`:

```bash
sudo deploy-labyricorn
```

Do not edit generated files under `/srv/labyricorn/current` or
`/srv/labyricorn/releases`. They will be replaced by deployment and are not
source-controlled.

## Local development

Install Lektor in an isolated Python environment. For example:

```bash
python3 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install lektor
```

Run the local editor and preview server:

```bash
lektor server
```

Run a production-style build:

```bash
lektor build --output-path build
test -f build/index.html
```

The project currently has no third-party Lektor packages or plugins.

## Production filesystem and permissions

```text
/srv/labyricorn/
├── repo/                 Git working copy (not web-accessible)
├── releases/             Successful immutable builds
├── current -> releases/<release>
├── logs/
│   ├── deploy.log
│   └── deploy.lock
├── .local/bin/lektor     pipx-installed Lektor
├── .netrc                Git HTTPS credential (0600; secret)
├── .ssh/                 Reserved deployment SSH material
└── README-DEPLOYMENT.md  Supplemental host-local notes
```

The `labyricorn-deploy` account owns the checkout, releases, logs, Lektor
environment, and Git credential. nginx's `www-data` account can traverse the
base path and read active generated files, but cannot read the source checkout
or credential. nginx cannot modify either source or releases.

## Deployment behavior

`/usr/local/bin/deploy-labyricorn` performs the following sequence:

1. Changes from root to the unprivileged `labyricorn-deploy` account.
2. Acquires an exclusive `flock` lock to prevent concurrent deployments.
3. Refuses to proceed if the working tree contains any tracked or untracked
   changes.
4. Fetches and prunes `origin`.
5. Verifies that `origin/main` exists.
6. Switches to local `main` and resets it exactly to `origin/main`.
7. Locates exactly one `.lektorproject` file instead of assuming its name.
8. Creates a unique release named with UTC time and the 12-character commit ID.
9. Runs Lektor with an explicit output directory.
10. Requires the build to succeed and contain `index.html`.
11. Writes the full commit ID to `.labyricorn-commit` in the release.
12. Atomically replaces `/srv/labyricorn/current` with a relative symlink to
    the new release.
13. Keeps the active release plus recent prior releases, targeting five total.
14. Logs start, failure, success, release name, and commit.

Run it with:

```bash
sudo deploy-labyricorn
```

A failed build is removed before activation. The current symlink is not changed,
so the previously working site remains online.

### Deployment preflight

```bash
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo status --short --branch

sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo fetch origin
```

The status must be clean. Do not bypass a dirty-tree refusal. Determine whether
the files are legitimate unpublished admin edits, then commit and push them or
remove them deliberately before deployment.

## Verify a deployment

```bash
readlink -f /srv/labyricorn/current
cat /srv/labyricorn/current/.labyricorn-commit

sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo rev-parse HEAD

sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo ls-remote origin refs/heads/main

nginx -t
systemctl is-active nginx

curl -I -H 'Host: www.labyricorn.com' \
  -H 'X-Forwarded-Proto: https' http://127.0.0.1/

curl -I https://www.labyricorn.com/
```

The commit in the active release, local `HEAD`, and remote `main` should match.
The local nginx request with `X-Forwarded-Proto: https` should return `200`.
Public HTTP should redirect to HTTPS, and public HTTPS should return `200`.

## Rollback

List releases and identify the current one:

```bash
sudo rollback-labyricorn
```

Switch atomically to a previous successful release:

```bash
sudo rollback-labyricorn 20260812T024742Z-92475d2ec759
```

Use a release name printed by the listing command. Rollback does not build,
fetch, or alter Git. It only changes the `current` symlink and records the event
in the deployment log.

Verify afterward:

```bash
readlink -f /srv/labyricorn/current
cat /srv/labyricorn/current/.labyricorn-commit
curl -I -H 'Host: www.labyricorn.com' \
  -H 'X-Forwarded-Proto: https' http://127.0.0.1/
```

A normal deployment after rollback rebuilds and reactivates the latest remote
`main` commit.

## nginx and URL handling

The site configuration is:

```text
/etc/nginx/sites-available/labyricorn
/etc/nginx/sites-enabled/labyricorn -> ../sites-available/labyricorn
```

nginx serves `/srv/labyricorn/current` and supports:

- `index.html` at directory-style URLs;
- direct static assets;
- optional `.html` fallback;
- generated `/404/index.html` with an HTTP `404` response;
- disabled directory browsing;
- rejection of hidden paths and common executable-script extensions;
- dedicated access and error logs.

Always validate before reloading nginx:

```bash
nginx -t
systemctl reload nginx
```

nginx logs:

```text
/var/log/nginx/labyricorn-access.log
/var/log/nginx/labyricorn-error.log
```

## HTTPS, Cloudflare, and DNS

Current production HTTPS terminates at Cloudflare. The origin nginx process
currently listens on port `80`, not `443`. For `www.labyricorn.com`, nginx uses
Cloudflare's `X-Forwarded-Proto` header to distinguish public HTTP from HTTPS:

- public HTTP redirects to `https://www.labyricorn.com`;
- Cloudflare-originated HTTPS requests are served without a redirect loop;
- `labyricorn.com` is configured at nginx to redirect to the canonical HTTPS
  `www` hostname.

The apex `labyricorn.com` did not resolve during initial setup. Its redirect will
not work publicly until DNS/Cloudflare has an apex record routed to this origin.

Certbot and the nginx integration are installed, and `certbot.timer` is enabled,
but no origin certificate was issued during setup. Do not run Certbot blindly
behind Cloudflare. First decide whether the desired origin mode is a Cloudflare
Origin CA certificate, Let's Encrypt with a compatible challenge, or direct DNS
without the proxy. After changing TLS, test both origin and public behavior for
redirect loops.

Useful checks:

```bash
getent ahosts www.labyricorn.com
getent ahosts labyricorn.com
curl -I http://www.labyricorn.com/
curl -I https://www.labyricorn.com/
systemctl status certbot.timer
certbot certificates
```

## Firewall and exposed services

UFW is active with default incoming denial. Allowed services are:

- OpenSSH: TCP 22
- nginx HTTP/HTTPS profile: TCP 80 and 443

Inspect current state:

```bash
ufw status verbose
ss -lntup
```

Lektor's development port must never be opened in UFW or bound to `0.0.0.0`.

## Optional server-side Lektor admin

An optional unit exists at:

```text
/etc/systemd/system/lektor-admin.service
```

Current state: **disabled and stopped**. When started, it runs as
`labyricorn-deploy` and binds only to `127.0.0.1:5000`. It is not proxied by
nginx and production remains independent of it.

Start it temporarily:

```bash
systemctl start lektor-admin
systemctl status lektor-admin
ss -lntp | grep ':5000'
```

Access it securely through an SSH tunnel from an administrator workstation:

```bash
ssh -L 5000:127.0.0.1:5000 [email protected]
```

Then browse to `http://127.0.0.1:5000` on that workstation.

Stop it when finished:

```bash
systemctl stop lektor-admin
```

### Committing admin edits made on the server

Lektor can edit files, but it does not automatically create Git commits or push
them. The deployment account intentionally has authenticated Git write access so
a controlled admin workflow can publish server-side edits.

Review everything before committing:

```bash
sudo -u labyricorn-deploy env HOME=/srv/labyricorn bash
cd /srv/labyricorn/repo
git status
git diff
git add <reviewed-files>
git commit -m "Describe the admin edit"
git push origin main
exit
```

Then run `sudo deploy-labyricorn`. The deployment script will refuse to run
while uncommitted admin edits remain. This is deliberate protection against
silently discarding edits when it resets to `origin/main`.

Before enabling persistent or public admin access, add authentication and make a
separate security decision. Do not publish port 5000 directly and do not proxy
an unauthenticated editor through nginx.

## Git authentication and secrets

The private Gitea repository is accessed over HTTPS because the Gitea SSH port
was not reachable through `git.labyricorn.com` during setup.

Credential purpose and location:

| Purpose | Location | Permissions |
| --- | --- | --- |
| Gitea HTTPS fetch/push | `/srv/labyricorn/.netrc` | `0600`, owned by `labyricorn-deploy` |
| Reserved SSH identity | `/srv/labyricorn/.ssh/id_ed25519` | `0600`, owned by `labyricorn-deploy` |

Never place token contents, passwords, or private keys in this README, Git URLs,
shell scripts, logs, commits, issue trackers, or command history.

Check credential-file metadata without printing its contents:

```bash
stat -c '%A %U:%G %n' /srv/labyricorn/.netrc
```

If the Gitea token is rotated, replace `.netrc` as `labyricorn-deploy`, preserve
mode `0600`, and verify both fetch and push access before revoking the old token.
Avoid printing either credential during rotation.

## Logs and diagnostics

```bash
# Deployment and rollback history
tail -n 100 /srv/labyricorn/logs/deploy.log

# nginx service and request errors
journalctl -u nginx --since today
tail -n 100 /var/log/nginx/labyricorn-error.log

# Optional admin service
journalctl -u lektor-admin --since today

# Repository state
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo status

# Active release
readlink -f /srv/labyricorn/current
cat /srv/labyricorn/current/.labyricorn-commit

# Services and sockets
systemctl status nginx
systemctl status certbot.timer
ss -lntup
```

## Troubleshooting

### Deployment says the repository is dirty

Inspect as the deployment account:

```bash
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo status --short
```

If these are intended admin edits, review, commit, and push them. Otherwise,
identify their origin before removing anything. Never use a destructive reset
until you are certain no work needs to be preserved.

### Git fetch or push fails

Check DNS, HTTPS, the remote URL, and credential permissions without displaying
the secret:

```bash
getent ahosts git.labyricorn.com
curl -I https://git.labyricorn.com/
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo remote -v
stat -c '%A %U:%G %n' /srv/labyricorn/.netrc
```

An authentication failure usually means the token was revoked, expired, or
lacks repository access.

### Lektor build fails

Run a disposable diagnostic build; do not build into `current`:

```bash
test_dir=$(mktemp -d /srv/labyricorn/releases/.diagnostic.XXXXXX)
chown labyricorn-deploy:labyricorn-deploy "$test_dir"
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  /srv/labyricorn/.local/bin/lektor \
  --project /srv/labyricorn/repo/Labyricorn.lektorproject \
  build --output-path "$test_dir"
```

Inspect the error and source. Remove the diagnostic directory only after
confirming its exact path and that it is not referenced by `current`.

### nginx returns an error

```bash
nginx -t
systemctl status nginx
readlink -f /srv/labyricorn/current
test -f /srv/labyricorn/current/index.html
tail -n 100 /var/log/nginx/labyricorn-error.log
```

Do not point nginx at the Git checkout or start Lektor as a production server.

### Public HTTPS loops or behaves differently from local HTTP

Check Cloudflare proxy mode and the forwarded protocol behavior:

```bash
curl -I -H 'Host: www.labyricorn.com' http://127.0.0.1/
curl -I -H 'Host: www.labyricorn.com' \
  -H 'X-Forwarded-Proto: https' http://127.0.0.1/
curl -I http://www.labyricorn.com/
curl -I https://www.labyricorn.com/
```

The first origin request should redirect; the simulated Cloudflare HTTPS request
should return the site. A different result indicates a proxy/header or nginx
configuration problem.

## Future automatic deployment

Manual deployment is complete and remains the supported path. Automatic
deployment is not configured.

The preferred future design is a Gitea Action triggered only by pushes to
`main`, invoking `deploy-labyricorn` through a dedicated, narrowly restricted
SSH identity. A webhook receiver is acceptable only if it authenticates a
secret, verifies the repository and branch, never executes payload text, and
runs with minimal privileges. Never expose a general shell endpoint or an
unauthenticated HTTP deployment URL.

## Change-control checklist

Before changing deployment, nginx, TLS, firewall, credentials, or admin access:

1. Inspect the live state and preserve unrelated configuration.
2. Back up the specific file being changed.
3. Validate syntax before reload or restart.
4. Keep SSH reachable before firewall changes.
5. Verify local origin behavior and public Cloudflare behavior separately.
6. Record operational changes in this README.
7. Commit and push source documentation to `main`.

This README describes the current verified installation on `ubuntu2-47`. Update
it whenever the operational architecture changes.