# Labyricorn website and deployment runbook

This repository is the canonical source for `www.labyricorn.com`. It contains
the editable Lektor project—not generated production files—and is also the
primary operations guide for building, deploying, verifying, rolling back, and
troubleshooting the site.

## Quick reference

| Item | Value |
| --- | --- |
| Production URL | `https://www.labyricorn.com` |
| Canonical Git repository | `https://git.labyricorn.com/Labyricorn/labyricorn-site.git` |
| Production branch | `main` |
| Production host | `ubuntu2-47` (`10.138.2.47`) |
| Repository checkout | `/srv/labyricorn/repo` |
| Active document root | `/srv/labyricorn/current` |
| Release storage | `/srv/labyricorn/releases` |
| Deployment command | `sudo deploy-labyricorn` |
| Rollback listing | `sudo rollback-labyricorn` |
| Rollback command | `sudo rollback-labyricorn RELEASE_NAME` |
| Deployment log | `/srv/labyricorn/logs/deploy.log` |
| Remote-project cache | `/srv/labyricorn/project-cache` |
| nginx site config | `/etc/nginx/sites-available/labyricorn` |
| Lektor executable | `/srv/labyricorn/.local/bin/lektor` |
| Deployment account | `labyricorn-deploy` |

## Architecture

```text
Gitea: Labyricorn/labyricorn-site (main)     public project repositories
              |                                        |
              | authenticated Git fetch                | read-only Git/API sync
              v                                        v
 /srv/labyricorn/repo                    /srv/labyricorn/project-cache
              |                                        |
              +---------- validated build input -------+
                               |
                               | isolated Lektor build
                               v
 /srv/labyricorn/releases/<UTC timestamp>-<commit>
              |
              | validated, atomic symlink switch
              v
 /srv/labyricorn/current
              |
              | read-only static files
              v
            nginx :80
              |
              | Cloudflare proxy and public TLS
              v
 https://www.labyricorn.com
```

Gitea is the source of truth. The production checkout is disposable state and
must not contain unpublished edits during deployment. Lektor's development
server is not part of production serving; nginx serves generated files only.
Remote project repositories remain authoritative for their own exhibition and
devlog records. They are never checked out into the site repository and cannot
modify trusted site models, templates, scripts, or content.

## Repository contents

```text
Labyricorn.lektorproject  Lektor project definition
content/                  Editable site content (`contents.lr` files)
content/tags/             Tracked tag registry and dedicated tag routes
models/                   Lektor content models
templates/                Jinja templates
assets/static/            CSS, filtering JavaScript, favicon, and static assets
configs/project-sources.ini  Approved public remote-project registry
scripts/project_sources.py  Remote-content validator and importer
scripts/build_with_projects.py  Isolated build/preview entry point
scripts/trigger_project_refresh.py  Authenticated remote-project refresh client
ops/deploy-labyricorn     Versioned copy of the production deployment command
tests/                    Importer validation tests
.gitea/workflows/         Push, schedule, and manual deployment automation
.gitignore                Excludes generated and local files
README.md                 This runbook
```

Generated output such as `build/`, `dist/`, and `.lektor/` is intentionally
ignored and must not be committed.

## Routine content workflow

Edit in a normal Git checkout, review the changes, and push them to `main`.
A successful push triggers the repository's Gitea Action, which builds and
atomically activates that exact revision on the production host.

```bash
git switch main
git pull --ff-only origin main

# Edit content, templates, models, or assets.

python scripts/build_with_projects.py --output-path build
git status
git add <reviewed-files>
git commit -m "Describe the site change"
git push origin main
```

Watch the run under **Actions** in Gitea. Manual deployment remains available
for recovery or controlled operations:

```bash
sudo deploy-labyricorn
```

Do not edit generated files under `/srv/labyricorn/current` or
`/srv/labyricorn/releases`. They will be replaced by deployment and are not
source-controlled.

## Tags and filtering workflow

`content/tags/` is the tracked tag registry. Each child directory is a normal
Lektor record whose directory name is the canonical lowercase, hyphenated slug
used in entry data and URLs:

```text
content/tags/deployment/contents.lr  ->  /tags/deployment/
```

The tag record supplies the editor label and public description:

    _model: tag
    ---
    title: Deployment
    ---
    summary: Build, release, rollout, verification, and rollback workflows.

Blog entries and articles store all tag slugs in one comma-separated `tags`
value. The entry model reads its tracked checkbox choices from the registry,
while the focused entry editors also accept arbitrary additional tags:

```text
tags: lektor, deployment, operations
```

A tag with a matching registry record is tracked: it links to a dedicated tag
page and participates in tag discovery. A tag without a matching record remains
visible using the existing nonlinked presentation but is excluded from tracked
discovery. Tracking an already-used freeform slug, or untracking an existing
slug, changes that behavior on the next build without rewriting content
entries. Slug renaming is different: it can require deliberately updating every
reference and is not provided by the tag editor.

`/tags/` lists every approved tag and counts its usage across projects, project
devlog entries, blog entries, and articles. Dedicated tag URLs combine matching
records from all four content types in one chronological discovery feed. Tag
links are normal links and work without JavaScript. On the Projects, Blog, and
Articles listing pages, `assets/static/tags.js` progressively adds instant
filtering and keeps the selected filter in the `tag` query parameter.

Remote-project languages and declared technologies are normalized to slugs and
matched only against this controlled vocabulary. Their approved matches become
clickable tags and a deduplicated `taxonomy_tags` field used by filters, counts,
and tag pages. Devlog topics follow the same rule and populate derived
`topic_tag_slugs` used by tag feeds and counts. A label that has no approved
record remains visible and non-clickable in mustard, produces a build warning,
and is excluded from tag discovery. Remote data never creates a tag record
automatically.

After taxonomy or filtering changes:

1. Run `python scripts/build_with_projects.py --output-path build`.
2. Verify `/tags/`, at least one dedicated tag URL, the three top-level section
   listings, and a project devlog index.
3. Confirm tracked-tag counts cover projects, devlog entries, blog entries, and
   articles; freeform entry tags remain visible but unlinked; and unapproved
   remote labels are not counted.
4. Test filtering with JavaScript enabled and confirm tag links remain usable
   without JavaScript.
5. Check keyboard operation plus desktop and narrow layouts.
6. If filtering JavaScript or tag styles changed, increment the corresponding
   cache-busting version in `templates/base.html`.

## Repository-owned project pages and devlogs

The site can publish a project page whose source lives in the project's own
public Git repository. Thinkloom is the first configured source:

```text
https://git.labyricorn.com/Labyricorn/thinkloom-openai-hackathon
  .labyricorn/
  ├── AGENTS.md
  ├── project/
  │   ├── AGENTS.md
  │   ├── contents.lr
  │   └── icon.png
  └── devlog/
      ├── AGENTS.md
      ├── contents.lr
      └── <entry-slug>/contents.lr
```

The public routes are `/projects/thinkloom/`,
`/projects/thinkloom/devlog/`, and
`/projects/thinkloom/devlog/<entry-slug>/`. The project repository owns the
native Lektor records and approved images. This site owns their models,
templates, layout, repository-information card, and import policy. The latest
imported devlog entry from each project also joins the homepage Recent Activity
feed, ordered with projects, articles, and blog entries by publication date.
Older devlog entries remain available on each project's complete devlog. The
feed initially shows six items and reveals up to six more per button activation.
This is a progressive enhancement implemented by `assets/static/activity.js`;
without JavaScript, the complete semantic feed remains visible.

`configs/project-sources.ini` is the allowlist. Each source declares a stable
project ID, credential-free HTTPS Git URL, public web/API URL, and branch. Add
or remove a source only through a reviewed site change. The importer also
checks the provider API and refuses a repository reported as private.
Remote Git commands run with an isolated empty home/config and interactive
credential lookup disabled, so the production account's site-repository
credential cannot silently turn a private project fetch into an authenticated
one. An anonymous API response of 401, 403, or 404 is treated as a visibility
failure rather than a metadata-cache condition.

The importer reads only these paths:

- `.labyricorn/project/contents.lr`
- `.labyricorn/project/*.{png,jpg,jpeg,webp}`
- `.labyricorn/devlog/contents.lr`
- `.labyricorn/devlog/<entry-slug>/contents.lr`
- approved image types within a devlog entry directory

Everything else is ignored, including remote `README.md` and `AGENTS.md`
instructions. Symbolic links, submodules, executable files, raw HTML,
unsupported models or schema versions, invalid slugs, unknown paths,
out-of-tree references, and invalid image signatures are rejected. Limits are
100 imported files, 5 MiB per file, and 20 MiB total per project snapshot.
Devlog `source_commit` values must exist in the repository and be ancestors of
the imported branch revision.

Every build uses a temporary copy of trusted site source, materializes validated
remote records into that copy, and invokes Lektor there. Imported files never
enter the site working tree. Git mirrors, state, and the last validated snapshot
live under the project cache. If a new snapshot or network fetch fails, the
build uses the last-known-good validated revision. Initial publication fails
when no valid snapshot exists. The generated
`.labyricorn-projects.json` records the site revision, imported project
revisions, metadata digests, synchronization status, and synchronization time.

The project card combines repository-owned content with public provider and Git
metadata: source and README links, default branch, latest commit, license,
languages, open issues, stars, forks, latest release, imported revision, and
sync time. Approved language and technology labels link into the shared tag
taxonomy, as do approved devlog topics, allowing either a project or a
development milestone to lead visitors to related writing. API metadata failure
uses cached metadata and is identified in the
sync status; it never causes an unvalidated new snapshot to replace a working
one.

To preview or build with remote project content, use the wrapper instead of
calling Lektor directly:

```bash
python scripts/build_with_projects.py --serve
python scripts/build_with_projects.py --output-path build
python -m unittest discover -s tests -v
```

The default development cache is `.cache/project-sources`, which is ignored by
Git. Lektor must be installed because the importer deliberately uses Lektor's
native record parser and the wrapper invokes the Lektor build/server. No custom
Lektor plugin or third-party project-import package is used.

## Local development

Install Lektor in an isolated Python environment. For example:

```bash
python3 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install lektor
```

Run the local editor and preview server with validated project content:

```bash
python scripts/build_with_projects.py --serve
```

Run a production-style build:

```bash
python scripts/build_with_projects.py --output-path build
test -f build/index.html
```

The project currently has no third-party Lektor packages or plugins.

### Local management launcher

To check the checkout's Git synchronization state and open the focused desktop
editors from one small window, run:

```bash
python scripts/management_launcher.py
```

The launcher locates the repository containing its script, fetches metadata
from the configured `origin`, and reports the current branch, upstream,
working-tree changes, and ahead/behind state in plain language. It offers only
fast-forward pulls for a clean, remote-ahead checkout and normal, non-force
pushes. **Commit & Push** captures every path to be staged, previews the first
40 with a count of any additional paths, requires a commit message for
working-tree changes, warns that pushing `main` deploys production, and
proceeds only after explicit confirmation. If the checkout already has
local commits, the same control can push them without creating another commit.
Diverged histories, remote-ahead publishing, missing tracking configuration,
and dirty pulls require manual review. Refresh rechecks the state without
restarting the window. Each editor is started as a separate Python process and
remains directly runnable with its documented command.

The management launcher and standalone devlog editor use Git from `PATH` when
it is available. On Windows, they fall back to
`support/PortableGit/cmd/git.exe`; the bundled runtime explicitly uses its
included Git Credential Manager rather than the first-use helper selector.
Authentication is established per workstation through the normal credential
manager flow. Never place a username or token in the repository remote URL.
If Gitea rejects a saved credential, the launcher offers to clear only the
credential for the configured HTTPS host and refresh again. Accepting that
prompt permits the normal Git Credential Manager sign-in flow during the
recovery refresh or the next explicit **Commit & Push** operation; it does not
alter working files or local commits. Routine background refreshes remain
non-interactive.

### Local blog editor

For a focused desktop form that creates and edits records under `content/blog/`,
run:

```bash
python scripts/blog_editor.py
```

The editor uses Tkinter from the Python standard library. It presents tracked
tag choices from `content/tags/`, accepts normalized additional/freeform tags,
writes normal `contents.lr` records to the working tree, and can delete a
loaded entry after explicit confirmation. Entry
deletion removes its complete record directory, including attachments, so the
confirmation identifies additional files before proceeding. The editor does
not commit, push, build, or deploy changes. Review saved or deleted records with
the routine content workflow above.

For a saved entry, **Choose Narration WAV…** converts a selected WAV using
`ffmpeg` from `PATH`, or the repository-provided `support/ffmpeg.exe` when it
is not installed system-wide, and writes the entry-local `narration.mp3`
attachment that the existing narration player recognizes. It uses a fixed
44.1 kHz mono, 96 kbps MP3 preset; replacing or removing narration requires
confirmation. If `ffmpeg` is unavailable or conversion fails, the existing
narration attachment is left unchanged.

### Local tracked tag editor

To create, edit, or untrack records in the tracked tag registry, run:

```bash
python scripts/tag_editor.py
```

The editor manages only `content/tags/<slug>/contents.lr`. Creating a tracked
record does not rewrite existing content that already uses the slug. Untracking
requires confirmation, removes only that tracked record, and leaves all tag
values in blog, article, project, and devlog content untouched. Display names
and summaries can be edited, but slugs cannot be renamed in this editor.

### Local article editor

For a focused desktop form that creates and edits records under
`content/articles/`, run:

```bash
python scripts/article_editor.py
```

The article editor follows the existing cover convention: a first Markdown
image referencing an article-local `cover-image.png` or `cover-image.jpg`.
Choose a PNG, JPG, or JPEG in the editor and it is copied beside `contents.lr`
with that canonical name. Replacing or removing a cover requires confirmation.
Tracked tags are selected from `content/tags/`; arbitrary additional tags may
be entered without creating tracked records.
The editor does not commit, push, build, or deploy changes; review all working
tree changes with the routine content workflow above.

The article editor also has the same saved-entry narration control: it converts
a selected WAV to the article-local `narration.mp3` attachment using `ffmpeg`
from `PATH` or `support/ffmpeg.exe`, with the fixed 44.1 kHz mono, 96 kbps MP3
preset. Replacement and removal require confirmation; failed conversion leaves
any existing narration unchanged.

### Local project editor

To add, edit, or remove a site-approved remote project source, run:

```bash
python scripts/project_editor.py
```

The project editor asks for one public GitHub or Gitea repository web URL and
derives the anonymous Git and provider API URLs stored in
`configs/project-sources.ini`. Project prose, technology labels, images, and
devlog records remain owned by the external project repository under
`.labyricorn/`. Its explicit **Check Devlog Status** action performs the
importer's public-provider and remote-snapshot validation against a disposable
read-only Git mirror, then removes that mirror. When setup is missing, the
editor directs the user to copy the standalone `scripts/devlog_editor.py` into
the project checkout, initialize and author the devlog, publish the changes, and
recheck. It never edits, commits, or pushes an external project repository.

Project Editor requires the Lektor Python package because it uses the site's
authoritative importer directly. If the launching Python cannot import Lektor,
the editor automatically checks `LABYRICORN_LEKTOR_PYTHON`, a project-local
`.venv`, common pipx Lektor environments, and the environment containing a
`lektor` command. It relaunches itself with the first runtime that can import
both Lektor and Tkinter. Set `LABYRICORN_LEKTOR_PYTHON` to an explicit Python
executable when Lektor is installed somewhere else.

## Production filesystem and permissions

```text
/srv/labyricorn/
├── repo/                 Git working copy (not web-accessible)
├── releases/             Successful immutable builds
├── current -> releases/<release>
├── logs/
│   ├── deploy.log
│   └── deploy.lock
├── project-cache/        Bare mirrors, metadata, and last-known-good state
├── .local/bin/lektor     pipx-installed Lektor
├── .netrc                Git HTTPS credential (0600; secret)
├── .ssh/                 Reserved deployment SSH material
└── README-DEPLOYMENT.md  Supplemental host-local notes
```

The `labyricorn-deploy` account owns the checkout, releases, logs, Lektor
environment, and Git credential. nginx's `www-data` account can traverse the
base path and read active generated files, but cannot read the source checkout
or credential. nginx cannot modify either source or releases.

## Deployment behavior

`/usr/local/bin/deploy-labyricorn` performs the following sequence:

1. Changes from root to the unprivileged `labyricorn-deploy` account.
2. Acquires an exclusive `flock` lock to prevent concurrent deployments.
3. Refuses to proceed if the working tree contains any tracked or untracked
   changes.
4. Fetches and prunes `origin`.
5. Verifies that `origin/main` exists.
6. Switches to local `main` and resets it exactly to `origin/main`.
7. Creates a unique release named with UTC time and the 12-character commit ID.
8. Runs `scripts/build_with_projects.py` with the persistent project cache and
   an explicit output directory.
9. Fetches, validates, and imports each approved public project source into an
   isolated temporary workspace before invoking Lektor.
10. Requires `index.html` and `.labyricorn-projects.json`.
11. Writes the full site commit ID to `.labyricorn-commit`.
12. Compares the site commit plus project commit/metadata digests with the
    active release and removes the candidate as an unchanged no-op when equal.
13. Atomically replaces `/srv/labyricorn/current` with a relative symlink to
    the new release.
14. Keeps the active release plus recent prior releases, targeting five total.
15. Logs start, failure, unchanged, success, release name, and commit.

Run it with:

```bash
sudo deploy-labyricorn
```

A failed build is removed before activation. The current symlink is not changed,
so the previously working site remains online.

### Deployment preflight

```bash
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo status --short --branch

sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo fetch origin
```

The status must be clean. Do not bypass a dirty-tree refusal. Determine whether
the files are legitimate unpublished admin edits, then commit and push them or
remove them deliberately before deployment.

## Verify a deployment

```bash
readlink -f /srv/labyricorn/current
cat /srv/labyricorn/current/.labyricorn-commit

sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo rev-parse HEAD

sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo ls-remote origin refs/heads/main

nginx -t
systemctl is-active nginx

curl -I -H 'Host: www.labyricorn.com' \
  -H 'X-Forwarded-Proto: https' http://127.0.0.1/

curl -I https://www.labyricorn.com/
```

The commit in the active release, local `HEAD`, and remote `main` should match.
The local nginx request with `X-Forwarded-Proto: https` should return `200`.
Public HTTP should redirect to HTTPS, and public HTTPS should return `200`.

## Rollback

List releases and identify the current one:

```bash
sudo rollback-labyricorn
```

Switch atomically to a previous successful release:

```bash
sudo rollback-labyricorn 20260812T024742Z-92475d2ec759
```

Use a release name printed by the listing command. Rollback does not build,
fetch, or alter Git. It only changes the `current` symlink and records the event
in the deployment log.

Verify afterward:

```bash
readlink -f /srv/labyricorn/current
cat /srv/labyricorn/current/.labyricorn-commit
curl -I -H 'Host: www.labyricorn.com' \
  -H 'X-Forwarded-Proto: https' http://127.0.0.1/
```

A normal deployment after rollback rebuilds and reactivates the latest remote
`main` commit.

## nginx and URL handling

The site configuration is:

```text
/etc/nginx/sites-available/labyricorn
/etc/nginx/sites-enabled/labyricorn -> ../sites-available/labyricorn
```

nginx serves `/srv/labyricorn/current` and supports:

- `index.html` at directory-style URLs;
- direct static assets;
- optional `.html` fallback;
- generated `/404/index.html` with an HTTP `404` response;
- disabled directory browsing;
- rejection of hidden paths and common executable-script extensions;
- dedicated access and error logs.

Always validate before reloading nginx:

```bash
nginx -t
systemctl reload nginx
```

nginx logs:

```text
/var/log/nginx/labyricorn-access.log
/var/log/nginx/labyricorn-error.log
```

## HTTPS, Cloudflare, and DNS

Current production HTTPS terminates at Cloudflare. The origin nginx process
currently listens on port `80`, not `443`. For `www.labyricorn.com`, nginx uses
Cloudflare's `X-Forwarded-Proto` header to distinguish public HTTP from HTTPS:

- public HTTP redirects to `https://www.labyricorn.com`;
- Cloudflare-originated HTTPS requests are served without a redirect loop;
- `labyricorn.com` is configured at nginx to redirect to the canonical HTTPS
  `www` hostname.

The apex `labyricorn.com` did not resolve during initial setup. Its redirect will
not work publicly until DNS/Cloudflare has an apex record routed to this origin.

Certbot and the nginx integration are installed, and `certbot.timer` is enabled,
but no origin certificate was issued during setup. Do not run Certbot blindly
behind Cloudflare. First decide whether the desired origin mode is a Cloudflare
Origin CA certificate, Let's Encrypt with a compatible challenge, or direct DNS
without the proxy. After changing TLS, test both origin and public behavior for
redirect loops.

Useful checks:

```bash
getent ahosts www.labyricorn.com
getent ahosts labyricorn.com
curl -I http://www.labyricorn.com/
curl -I https://www.labyricorn.com/
systemctl status certbot.timer
certbot certificates
```

## Firewall and exposed services

UFW is active with default incoming denial. Allowed services are:

- OpenSSH: TCP 22
- nginx HTTP/HTTPS profile: TCP 80 and 443

Inspect current state:

```bash
ufw status verbose
ss -lntup
```

Lektor's development port must never be opened in UFW or bound to `0.0.0.0`.

## Optional server-side Lektor admin

An optional unit exists at:

```text
/etc/systemd/system/lektor-admin.service
```

Current state: **disabled and stopped**. When started, it runs as
`labyricorn-deploy` and binds only to `127.0.0.1:5000`. It is not proxied by
nginx and production remains independent of it.

Start it temporarily:

```bash
systemctl start lektor-admin
systemctl status lektor-admin
ss -lntp | grep ':5000'
```

Access it securely through an SSH tunnel from an administrator workstation:

```bash
ssh -L 5000:127.0.0.1:5000 root@10.138.2.47
```

Then browse to `http://127.0.0.1:5000` on that workstation.

Stop it when finished:

```bash
systemctl stop lektor-admin
```

### Committing admin edits made on the server

Lektor can edit files, but it does not automatically create Git commits or push
them. The deployment account intentionally has authenticated Git write access so
a controlled admin workflow can publish server-side edits.

Review everything before committing:

```bash
sudo -u labyricorn-deploy env HOME=/srv/labyricorn bash
cd /srv/labyricorn/repo
git status
git diff
git add <reviewed-files>
git commit -m "Describe the admin edit"
git push origin main
exit
```

Then run `sudo deploy-labyricorn`. The deployment script will refuse to run
while uncommitted admin edits remain. This is deliberate protection against
silently discarding edits when it resets to `origin/main`.

Before enabling persistent or public admin access, add authentication and make a
separate security decision. Do not publish port 5000 directly and do not proxy
an unauthenticated editor through nginx.

## Git authentication and secrets

The site repository is accessed over authenticated HTTPS because the Gitea SSH
port was not reachable through `git.labyricorn.com` during setup. Approved
remote project repositories, including Thinkloom, are public and are fetched
without embedding credentials in their configured URLs.

Credential purpose and location:

| Purpose | Location | Permissions |
| --- | --- | --- |
| Gitea HTTPS fetch/push | `/srv/labyricorn/.netrc` | `0600`, owned by `labyricorn-deploy` |
| Reserved SSH identity | `/srv/labyricorn/.ssh/id_ed25519` | `0600`, owned by `labyricorn-deploy` |

Never place token contents, passwords, or private keys in this README, Git URLs,
shell scripts, logs, commits, issue trackers, or command history.

Check credential-file metadata without printing its contents:

```bash
stat -c '%A %U:%G %n' /srv/labyricorn/.netrc
```

If the Gitea token is rotated, replace `.netrc` as `labyricorn-deploy`, preserve
mode `0600`, and verify both fetch and push access before revoking the old token.
Avoid printing either credential during rotation.

## Logs and diagnostics

```bash
# Deployment and rollback history
tail -n 100 /srv/labyricorn/logs/deploy.log

# nginx service and request errors
journalctl -u nginx --since today
tail -n 100 /var/log/nginx/labyricorn-error.log

# Optional admin service
journalctl -u lektor-admin --since today

# Repository state
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo status

# Active release
readlink -f /srv/labyricorn/current
cat /srv/labyricorn/current/.labyricorn-commit

# Services and sockets
systemctl status nginx
systemctl status certbot.timer
ss -lntup
```

## Troubleshooting

### Deployment says the repository is dirty

Inspect as the deployment account:

```bash
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo status --short
```

If these are intended admin edits, review, commit, and push them. Otherwise,
identify their origin before removing anything. Never use a destructive reset
until you are certain no work needs to be preserved.

### Git fetch or push fails

Check DNS, HTTPS, the remote URL, and credential permissions without displaying
the secret:

```bash
getent ahosts git.labyricorn.com
curl -I https://git.labyricorn.com/
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo remote -v
stat -c '%A %U:%G %n' /srv/labyricorn/.netrc
```

An authentication failure usually means the token was revoked, expired, or
lacks repository access.

### Lektor build fails

Run a disposable diagnostic build; do not build into `current`:

```bash
test_dir=$(mktemp -d /srv/labyricorn/releases/.diagnostic.XXXXXX)
chown labyricorn-deploy:labyricorn-deploy "$test_dir"
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  /srv/labyricorn/.local/pipx/venvs/lektor/bin/python \
  /srv/labyricorn/repo/scripts/build_with_projects.py \
  --site-root /srv/labyricorn/repo \
  --cache-path /srv/labyricorn/project-cache \
  --output-path "$test_dir" \
  --lektor /srv/labyricorn/.local/bin/lektor
```

Inspect the error and source. Remove the diagnostic directory only after
confirming its exact path and that it is not referenced by `current`.

### nginx returns an error

```bash
nginx -t
systemctl status nginx
readlink -f /srv/labyricorn/current
test -f /srv/labyricorn/current/index.html
tail -n 100 /var/log/nginx/labyricorn-error.log
```

Do not point nginx at the Git checkout or start Lektor as a production server.

### Public HTTPS loops or behaves differently from local HTTP

Check Cloudflare proxy mode and the forwarded protocol behavior:

```bash
curl -I -H 'Host: www.labyricorn.com' http://127.0.0.1/
curl -I -H 'Host: www.labyricorn.com' \
  -H 'X-Forwarded-Proto: https' http://127.0.0.1/
curl -I http://www.labyricorn.com/
curl -I https://www.labyricorn.com/
```

The first origin request should redirect; the simulated Cloudflare HTTPS request
should return the site. A different result indicates a proxy/header or nginx
configuration problem.

### Remote project synchronization fails

Inspect the deployment log and active manifest without editing the cache:

```bash
tail -n 100 /srv/labyricorn/logs/deploy.log
cat /srv/labyricorn/current/.labyricorn-projects.json
```

`current` means the latest remote revision and provider metadata validated.
`metadata-cached` means Git content is current but provider metadata fell back
to its cache. `last-known-good` means the latest fetch or candidate snapshot
could not safely replace the previously validated commit. A first import with
no valid cached snapshot fails the deployment. Do not bypass validation by
copying remote files into `content/`, the build workspace, or an active release.

## Automatic deployment and project refresh

The workflow at `.gitea/workflows/deploy.yml` runs for three event types:

- every push to site `main`;
- every 30 minutes (`*/30 * * * *`, evaluated by Gitea in UTC);
- an authenticated manual `workflow_dispatch`.

All events run the same one-job deployment:

1. Run on the repository-scoped runner labelled `labyricorn-deploy`.
2. Execute the existing root-owned `deploy-labyricorn` command.
3. Confirm `/srv/labyricorn/current/.labyricorn-commit` equals the workflow's
   site commit.
4. Request the site through nginx on the local origin and fail if it is unhealthy.

The action intentionally does not check out the repository into its own
workspace. The deployment script fetches `origin/main` using the dedicated
`labyricorn-deploy` account, validates a clean checkout, builds a new release,
and atomically changes the `current` symlink. Concurrent action runs share the
`labyricorn-production` concurrency group, and the deployment script also uses
a filesystem lock. A scheduled/manual run updates the site when an approved
project commit or public metadata digest changes. When neither the site nor any
project input changed, it validates and builds a candidate, removes it, logs an
`unchanged` result, and leaves the active symlink untouched.

### Trigger a refresh

From Gitea, open **Labyricorn/labyricorn-site → Actions → Deploy production**
and choose **Run workflow** for `main`.

Automation and coding assistants may dispatch the same workflow through
Gitea's authenticated API after pushing project publishing content. With
`LABYRICORN_REFRESH_TOKEN` supplied by an approved credential store or
ephemeral environment, run:

```bash
python scripts/trigger_project_refresh.py
```

Use `python scripts/trigger_project_refresh.py --dry-run` to inspect the fixed
site workflow target without reading the token or making a network request.
The script sends only `{"ref":"main"}`, keeps the credential out of the URL
and request body, and reports whether Gitea accepted the dispatch. It does not
wait for or claim a successful deployment.

The token must be supplied through an approved credential store or ephemeral
environment and must have permission to dispatch Actions for the site
repository. Never put it in the URL, a project repository, `AGENTS.md`, logs,
or committed scripts. Send at most one dispatch after the relevant project
push, then verify the Action and public project revision. The scheduled run is
the fallback when no refresh credential is available. This is deliberately not
an unauthenticated public webhook or long-running URL listener, so it cannot be
spammed by anonymous requests.

### Runner installation

The production host runs official Gitea Runner 3.0.0 directly on Linux; Docker
is not installed or required for this workflow.

| Item | Value |
|---|---|
| systemd unit | `gitea-runner.service` |
| service account | `gitea-runner` |
| runner name | `labyricorn-production` |
| registration scope | `Labyricorn/labyricorn-site` repository |
| execution label | `labyricorn-deploy:host` |
| runner binary | `/usr/local/bin/gitea-runner` |
| runner configuration | `/etc/gitea-runner/config.yaml` |
| registration state | `/var/lib/gitea-runner/.runner` |
| job workspace | `/var/lib/gitea-runner/work` |
| sudo policy | `/etc/sudoers.d/gitea-runner-labyricorn` |

The runner account has no general administrative access. Its sudo policy allows
only this exact command without a password:

```text
/usr/local/bin/deploy-labyricorn
```

Useful checks:

```bash
systemctl status gitea-runner
journalctl -u gitea-runner -n 100 --no-pager
sudo -l -U gitea-runner
```

If the runner must be replaced, delete or disable it in **Repository Settings →
Actions → Runners**, generate a fresh repository registration token, and
register the replacement. Registration tokens and `.runner` contents are
credentials: never commit or print them. A push can still be deployed manually
with `sudo deploy-labyricorn` while the runner is unavailable.

## Change-control checklist

Before changing deployment, nginx, TLS, firewall, credentials, or admin access:

1. Inspect the live state and preserve unrelated configuration.
2. Back up the specific file being changed.
3. Validate syntax before reload or restart.
4. Keep SSH reachable before firewall changes.
5. Verify local origin behavior and public Cloudflare behavior separately.
6. Record operational changes in this README.
7. Commit and push source documentation to `main`.

This README describes the current verified installation on `ubuntu2-47`. Update
it whenever the operational architecture changes.

User Note: Check 'C:\Users\-username-\AppData\Local\pipx\pipx\venvs\lektor' for existing lektor install