Labyricorn website and deployment runbook

This repository is the canonical source for www.labyricorn.com. It contains the editable Lektor project—not generated production files—and is also the primary operations guide for building, deploying, verifying, rolling back, and troubleshooting the site.

Quick reference

Item	Value
Production URL	https://www.labyricorn.com
Canonical Git repository	https://git.labyricorn.com/Labyricorn/labyricorn-site.git
Production branch	main
Production host	ubuntu2-47 (10.138.2.47)
Repository checkout	/srv/labyricorn/repo
Active document root	/srv/labyricorn/current
Release storage	/srv/labyricorn/releases
Deployment command	sudo deploy-labyricorn
Rollback listing	sudo rollback-labyricorn
Rollback command	sudo rollback-labyricorn RELEASE_NAME
Deployment log	/srv/labyricorn/logs/deploy.log
Remote-project cache	/srv/labyricorn/project-cache
nginx site config	/etc/nginx/sites-available/labyricorn
Lektor executable	/srv/labyricorn/.local/bin/lektor
Deployment account	labyricorn-deploy

Architecture

Gitea: Labyricorn/labyricorn-site (main)     public project repositories
              |                                        |
              | authenticated Git fetch                | read-only Git/API sync
              v                                        v
 /srv/labyricorn/repo                    /srv/labyricorn/project-cache
              |                                        |
              +---------- validated build input -------+
                               |
                               | isolated Lektor build
                               v
 /srv/labyricorn/releases/<UTC timestamp>-<commit>
              |
              | validated, atomic symlink switch
              v
 /srv/labyricorn/current
              |
              | read-only static files
              v
            nginx :80
              |
              | Cloudflare proxy and public TLS
              v
 https://www.labyricorn.com

Gitea is the source of truth. The production checkout is disposable state and must not contain unpublished edits during deployment. Lektor's development server is not part of production serving; nginx serves generated files only. Remote project repositories remain authoritative for their own exhibition and devlog records. They are never checked out into the site repository and cannot modify trusted site models, templates, scripts, or content.

Repository contents

Labyricorn.lektorproject  Lektor project definition
content/                  Editable site content (`contents.lr` files)
content/tags/             Tracked tag registry and dedicated tag routes
models/                   Lektor content models
templates/                Jinja templates
assets/static/            CSS, filtering JavaScript, favicon, and static assets
configs/project-sources.ini  Approved public remote-project registry
scripts/project_sources.py  Remote-content validator and importer
scripts/build_with_projects.py  Isolated build/preview entry point
scripts/labyricorn_mcp_server.py  Local stdio MCP launcher
labyricorn_mcp/             Repository-scoped MCP protocol and domain adapters
scripts/trigger_project_refresh.py  Authenticated remote-project refresh client
ops/deploy-labyricorn     Versioned copy of the production deployment command
tests/                    Importer validation tests
.gitea/workflows/         Push, schedule, and manual deployment automation
.gitignore                Excludes generated and local files
README.md                 This runbook

Generated output such as build/, dist/, and .lektor/ is intentionally ignored and must not be committed.

Public analytics

Normal public pages load the deferred Umami Cloud tracker once from the shared templates/base.html layout:

<script defer src="https://cloud.umami.is/script.js" data-website-id="a89a54ce-3208-4655-80dc-5480f85d5dce"></script>

This is baseline page tracking only. The repository does not host an analytics backend, add custom analytics events, or include the tracker in local authoring and build interfaces.

The public privacy disclosure is the normal Lektor page record at content/privacy/contents.lr, published at /privacy/ and linked from the shared site footer. It describes the current cookie-free analytics implementation, aggregate usage data, external links, and the policy for any future anonymous feature measurements. The page's visible Last updated date comes from its updated content metadata; revise that value whenever the disclosure materially changes. The site does not display analytics cookie or consent UI for this implementation.

Atom and RSS feeds

The project-aware build generates Atom 1.0 and RSS 2.0 feeds from one shared feed catalog in scripts/feed_generator.py. Feed XML is materialized only in the isolated build workspace and is then handled as normal Lektor static output; generated feed files are never added to the trusted source tree or Git.

Scope	Atom	RSS
Recent Activity	/feed.xml	/rss.xml
Articles	/articles/feed.xml	/articles/rss.xml
Blog	/blog/feed.xml	/blog/rss.xml
Project history	/projects/<project-id>/feed.xml	/projects/<project-id>/rss.xml

Recent Activity paths are selected by the feed catalog and added to the isolated homepage record before Lektor renders it, so the homepage and both feed formats use the same public-content and latest-devlog-per-project rules. General feeds are limited to the newest 50 selected records. Each project feed is untruncated and intentionally renders the project overview first, followed by its public devlog entries from oldest to newest. Canonical record URLs are stable entry IDs/GUIDs, and rendered body links and media URLs are made absolute under https://www.labyricorn.com/.

Always use the existing wrapper for builds and previews so validated remote projects, homepage activity data, and feeds are prepared together:

python scripts/build_with_projects.py --output-path build
python scripts/build_with_projects.py --serve

After feed changes, parse every generated XML file, compare Atom entry IDs with RSS GUIDs for each scope, confirm hidden records are absent, and check at least two project feeds for overview-first chronological history. Also verify the homepage, Articles, Blog, and project pages expose only their contextual feed autodiscovery and visible links.

Top-level listing heroes

The three primary listing routes each render one design-specific circuit hero through templates/section.html:

Route	Hero treatment
/projects/	Green system-architecture and project-matrix display
/articles/	Cyan intelligence-briefing and knowledge-system display
/blog/	Purple transmission and live-signal display

These heroes apply only to the top-level Projects, Articles, and Blog listing pages. Individual project pages, project devlogs, articles, and blog posts keep their existing templates and headers. Each hero preserves its own SVG circuit paths, status module, copy, timing, and responsive behavior while sharing namespaced CSS in assets/static/style.css. The design declares a 175-pixel minimum height rather than a fixed height, so its complete content determines the final rendered height. At viewports up to 760 pixels wide, each hero's copy and decorative panel use the design's single-column layout.

assets/static/section-heroes.js supplies the circuit-packet and intermittent glitch animations. The complete static hero remains available when JavaScript or the Web Animations API is unavailable, and motion is disabled when the user prefers reduced motion. When hero styles or behavior change, increment the corresponding style.css or section-heroes.js cache-busting version in templates/base.html.

Homepage tag discovery

The homepage ends with an Explore the Signal topic matrix immediately after Recent Activity and Featured Projects and before the site footer. It is a server-rendered discovery route over the canonical child records under content/tags/; it does not maintain a separate topic list. Each topic is a normal link to its dedicated /tags/<slug>/ page and includes its current published usage count.

Counts use the same discovery scope as the tag directory and dedicated tag pages: projects, project devlog entries, blog entries, and articles. The matrix uses four restrained visual levels for counts of 0–1, 2–3, 4–7, and 8 or more. The displayed count remains exact while the highest visual level is capped, so a frequently used tag cannot dominate the layout. Tag records and content changes update the matrix automatically on the next normal project-aware build.

assets/static/tags.js progressively updates the Tag Summary readout when a visitor hovers or focuses a topic. The summary is optional enhancement only: every link remains visible, keyboard accessible, touch friendly, and usable with JavaScript disabled. Styling lives in assets/static/style.css, uses the existing Labyricorn gold treatment, wraps without horizontal scrolling, and removes the small focus/hover lift for reduced-motion users. When these assets change, increment their cache-busting versions in templates/base.html.

After changing the homepage matrix:

1. Run python scripts/build_with_projects.py --output-path build with the Lektor runtime documented below.
2. Confirm the generated homepage topic count matches the tracked registry and every topic links to its existing tag page with the expected usage count.
3. Check desktop and narrow layouts, keyboard focus, the summary enhancement, normal link navigation, and the transition into the footer.
4. Run python -m unittest discover -s tests -v and git diff --check.

Routine content workflow

Edit in a normal Git checkout, review the changes, and push them to main. A successful push triggers the repository's Gitea Action, which builds and atomically activates that exact revision on the production host.

git switch main
git pull --ff-only origin main

# Edit content, templates, models, or assets.

python scripts/build_with_projects.py --output-path build
git status
git add <reviewed-files>
git commit -m "Describe the site change"
git push origin main

Watch the run under Actions in Gitea. Manual deployment remains available for recovery or controlled operations:

sudo deploy-labyricorn

Do not edit generated files under /srv/labyricorn/current or /srv/labyricorn/releases. They will be replaced by deployment and are not source-controlled.

Tags and filtering workflow

content/tags/ is the tracked tag registry. Each child directory is a normal Lektor record whose directory name is the canonical lowercase, hyphenated slug used in entry data and URLs:

content/tags/deployment/contents.lr  ->  /tags/deployment/

The tag record supplies the editor label and public description:

_model: tag
---
title: Deployment
---
summary: Build, release, rollout, verification, and rollback workflows.

Blog entries and articles store all tag slugs in one comma-separated tags value. The entry model reads its tracked checkbox choices from the registry, while the focused entry editors also accept arbitrary additional tags:

tags: lektor, deployment, operations

A tag with a matching registry record is tracked: it links to a dedicated tag page and participates in tag discovery. A tag without a matching record remains visible using the existing nonlinked presentation but is excluded from tracked discovery. Tracking an already-used freeform slug, or untracking an existing slug, changes that behavior on the next build without rewriting content entries. Slug renaming is different: it can require deliberately updating every reference and is not provided by the tag editor.

/tags/ lists every approved tag and counts its usage across projects, project devlog entries, blog entries, and articles. Dedicated tag URLs combine matching records from all four content types in one chronological discovery feed. Tag links are normal links and work without JavaScript. On the Projects, Blog, and Articles listing pages, assets/static/tags.js progressively adds instant filtering and keeps the selected filter in the tag query parameter.

Remote-project languages and declared technologies are normalized to slugs and matched only against this controlled vocabulary. Their approved matches become clickable tags and a deduplicated taxonomy_tags field used by filters, counts, and tag pages. Devlog topics follow the same rule and populate derived topic_tag_slugs used by tag feeds and counts. A label that has no approved record remains visible and non-clickable in mustard, produces a build warning, and is excluded from tag discovery. Remote data never creates a tag record automatically.

After taxonomy or filtering changes:

1. Run python scripts/build_with_projects.py --output-path build.
2. Verify /tags/, at least one dedicated tag URL, the three top-level section listings, and a project devlog index.
3. Confirm tracked-tag counts cover projects, devlog entries, blog entries, and articles; freeform entry tags remain visible but unlinked; and unapproved remote labels are not counted.
4. Test filtering with JavaScript enabled and confirm tag links remain usable without JavaScript.
5. Check keyboard operation plus desktop and narrow layouts.
6. If filtering JavaScript or tag styles changed, increment the corresponding cache-busting version in templates/base.html.

Repository-owned project pages and devlogs

The site can publish a project page whose source lives in the project's own public Git repository. Thinkloom is the first configured source:

https://git.labyricorn.com/Labyricorn/thinkloom-openai-hackathon
  .labyricorn/
  ├── AGENTS.md
  ├── project/
  │   ├── AGENTS.md
  │   ├── contents.lr
  │   └── icon.png
  └── devlog/
      ├── AGENTS.md
      ├── contents.lr
      └── <entry-slug>/contents.lr

The public routes are /projects/thinkloom/, /projects/thinkloom/devlog/, and /projects/thinkloom/devlog/<entry-slug>/. The project repository owns the native Lektor records and approved images. This site owns their models, templates, layout, repository-information card, and import policy. The latest imported devlog entry from each project also joins the homepage Recent Activity feed, ordered with projects, articles, and blog entries by publication date. Older devlog entries remain available on each project's complete devlog. The feed initially shows six items and reveals up to six more per button activation. This is a progressive enhancement implemented by assets/static/activity.js; without JavaScript, the complete semantic feed remains visible.

configs/project-sources.ini is the allowlist. Each source declares a stable project ID, credential-free HTTPS Git URL, public web/API URL, and branch. Add or remove a source only through a reviewed site change. The importer also checks the provider API and refuses a repository reported as private. Remote Git commands run with an isolated empty home/config and interactive credential lookup disabled, so the production account's site-repository credential cannot silently turn a private project fetch into an authenticated one. An anonymous API response of 401, 403, or 404 is treated as a visibility failure rather than a metadata-cache condition.

The importer reads only these paths:

• .labyricorn/project/contents.lr
• .labyricorn/project/*.{png,jpg,jpeg,webp}
• .labyricorn/devlog/contents.lr
• .labyricorn/devlog/<entry-slug>/contents.lr
• approved image types within a devlog entry directory

Everything else is ignored, including remote README.md and AGENTS.md instructions. Symbolic links, submodules, executable files, raw HTML, unsupported models or schema versions, invalid slugs, unknown paths, out-of-tree references, and invalid image signatures are rejected. Limits are 100 imported files, 5 MiB per file, and 20 MiB total per project snapshot. Devlog source_commit values must exist in the repository and be ancestors of the imported branch revision.

Every build uses a temporary copy of trusted site source, materializes validated remote records into that copy, and invokes Lektor there. Imported files never enter the site working tree. Git mirrors, state, and the last validated snapshot live under the project cache. If a new snapshot or network fetch fails, the build uses the last-known-good validated revision. Initial publication fails when no valid snapshot exists. The generated .labyricorn-projects.json records the site revision, imported project revisions, metadata digests, synchronization status, and synchronization time.

The project card combines repository-owned content with public provider and Git metadata: source and README links, default branch, latest commit, license, languages, open issues, stars, forks, latest release, imported revision, and sync time. Approved language and technology labels link into the shared tag taxonomy, as do approved devlog topics, allowing either a project or a development milestone to lead visitors to related writing. API metadata failure uses cached metadata and is identified in the sync status; it never causes an unvalidated new snapshot to replace a working one.

To preview or build with remote project content, use the wrapper instead of calling Lektor directly:

python scripts/build_with_projects.py --serve
python scripts/build_with_projects.py --output-path build
python -m unittest discover -s tests -v

The default development cache is .cache/project-sources, which is ignored by Git. Lektor must be installed because the importer deliberately uses Lektor's native record parser and the wrapper invokes the Lektor build/server. No custom Lektor plugin or third-party project-import package is used.

Local development

Install Lektor in an isolated Python environment. For example:

python3 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install lektor

Run the local editor and preview server with validated project content:

python scripts/build_with_projects.py --serve

Run a production-style build:

python scripts/build_with_projects.py --output-path build
test -f build/index.html

The project currently has no third-party Lektor packages or plugins.

Local Labyricorn MCP server

Labyricorn MCP is a local, stdio-based, repository-scoped management interface for this checkout. An MCP-capable AI client launches it as a child process and receives structured tools for repository information, models, content, tracked tags, validated project snapshots, allowlisted design files, local validation, and constrained Git operations.

It is not the Labyricorn website server, a deployment or remote-administration service, a general filesystem MCP, a shell MCP, or a production infrastructure controller. It creates no listener, HTTP endpoint, OAuth flow, persistent service, scheduled job, or additional deployment mechanism. The boundary stays:

MCP-capable AI client
        |
      stdio
        |
Labyricorn MCP process
        |
local Labyricorn Lektor repository
        |
 explicit ordinary Git push
        |
      Gitea
        |
existing external update/deployment system
        |
live site

The implementation uses Python's standard library and the repository's existing article, blog, tracked-tag, project-import, build, and Git helpers. It adds no Python package dependency. Launch it from a terminal with:

python scripts/labyricorn_mcp_server.py --repository /absolute/path/to/labyricorn-site

On Windows, use the existing Lektor pipx Python when available. Project-cache inspection and local builds need Lektor in the server runtime; the usual pipx location is:

%LOCALAPPDATA%\pipx\pipx\venvs\lektor\Scripts\python.exe

For an MCP client that accepts the common mcpServers configuration shape, use absolute paths. A Windows example is:

{
  "mcpServers": {
    "labyricorn": {
      "command": "C:\\Users\\YOUR-NAME\\AppData\\Local\\pipx\\pipx\\venvs\\lektor\\Scripts\\python.exe",
      "args": [
        "C:\\absolute\\path\\to\\labyricorn-site\\scripts\\labyricorn_mcp_server.py",
        "--repository",
        "C:\\absolute\\path\\to\\labyricorn-site"
      ]
    }
  }
}

The client must launch the process directly; stdout is reserved exclusively for newline-delimited MCP JSON-RPC messages and diagnostics go to stderr. The server supports MCP protocol revisions 2025-11-25, 2025-06-18, 2025-03-26, and 2024-11-05.

The focused v1 tool surface includes:

• repository/model/content reads plus safe article and blog create/update;
• tracked-tag list, usage, create, update, and explicit untrack;
• allowlisted project sources and locally cached snapshots validated through the existing importer, plus an explicit networked source-validation check;
• inspection of templates and static text assets, with hash-guarded updates only under templates/ and assets/static/;
• the existing isolated project-aware build, unit tests, diff check, and a consolidated validation result;
• Git status, diff, and log; clean fast-forward pull; reviewed-path-only commit; and normal upstream push.

Mutations use repository-relative path containment and reject traversal, out-of-tree resolution, and linked-path writes. Content and design updates require the SHA-256 returned by the preceding read, so a concurrent human edit causes FILE_CHANGED instead of being overwritten. Commits and pushes are separate explicit actions. Commit accepts only reviewed changed paths, refuses credential-like files and unrelated staged changes, and never pushes. Push cannot force or rewrite history. A push from main requires an explicit acknowledgement that Gitea's existing external site-update workflow may run. The MCP itself never calls, inspects, restarts, or reconfigures production.

Content deletion and tracked-tag slug renaming are intentionally absent from v1. Untracking a tag removes only its registry record and preserves every content reference, matching the existing tracked-tag editor.

Local management launcher

To check the checkout's Git synchronization state and open the focused desktop editors from one small window, run:

python scripts/management_launcher.py

The launcher locates the repository containing its script, fetches metadata from the configured origin, and reports the current branch, upstream, working-tree changes, and ahead/behind state in plain language. It offers only fast-forward pulls for a clean, remote-ahead checkout and normal, non-force pushes. Commit & Push captures every path to be staged, previews the first 40 with a count of any additional paths, requires a commit message for working-tree changes, warns that pushing main deploys production, and proceeds only after explicit confirmation. If the checkout already has local commits, the same control can push them without creating another commit. Diverged histories, remote-ahead publishing, missing tracking configuration, and dirty pulls require manual review. Refresh rechecks the state without restarting the window. Each editor is started as a separate Python process and remains directly runnable with its documented command.

The management launcher and standalone devlog editor use Git from PATH when it is available. On Windows, they fall back to support/PortableGit/cmd/git.exe; the bundled runtime explicitly uses its included Git Credential Manager rather than the first-use helper selector. Authentication is established per workstation through the normal credential manager flow. Never place a username or token in the repository remote URL. If Gitea rejects a saved credential, the launcher offers to clear only the credential for the configured HTTPS host and refresh again. Accepting that prompt permits the normal Git Credential Manager sign-in flow during the recovery refresh or the next explicit Commit & Push operation; it does not alter working files or local commits. Routine background refreshes remain non-interactive.

Local blog editor

For a focused desktop form that creates and edits records under content/blog/, run:

python scripts/blog_editor.py

The editor uses Tkinter from the Python standard library. It presents tracked tag choices from content/tags/, accepts normalized additional/freeform tags, writes normal contents.lr records to the working tree, and can delete a loaded entry after explicit confirmation. Entry deletion removes its complete record directory, including attachments, so the confirmation identifies additional files before proceeding. The editor does not commit, push, build, or deploy changes. Review saved or deleted records with the routine content workflow above.

For a saved entry, Choose Narration WAV… converts a selected WAV using ffmpeg from PATH, or the repository-provided support/ffmpeg.exe when it is not installed system-wide, and writes the entry-local narration.mp3 attachment that the existing narration player recognizes. It uses a fixed 44.1 kHz mono, 96 kbps MP3 preset; replacing or removing narration requires confirmation. If ffmpeg is unavailable or conversion fails, the existing narration attachment is left unchanged.

Local tracked tag editor

To create, edit, or untrack records in the tracked tag registry, run:

python scripts/tag_editor.py

The editor manages only content/tags/<slug>/contents.lr. Creating a tracked record does not rewrite existing content that already uses the slug. Untracking requires confirmation, removes only that tracked record, and leaves all tag values in blog, article, project, and devlog content untouched. Display names and summaries can be edited, but slugs cannot be renamed in this editor.

Local article editor

For a focused desktop form that creates and edits records under content/articles/, run:

python scripts/article_editor.py

The article editor follows the existing cover convention: a first Markdown image referencing an article-local cover-image.png or cover-image.jpg. Choose a PNG, JPG, or JPEG in the editor and it is copied beside contents.lr with that canonical name. Replacing or removing a cover requires confirmation. Tracked tags are selected from content/tags/; arbitrary additional tags may be entered without creating tracked records. The editor does not commit, push, build, or deploy changes; review all working tree changes with the routine content workflow above.

The article editor also has the same saved-entry narration control: it converts a selected WAV to the article-local narration.mp3 attachment using ffmpeg from PATH or support/ffmpeg.exe, with the fixed 44.1 kHz mono, 96 kbps MP3 preset. Replacement and removal require confirmation; failed conversion leaves any existing narration unchanged.

Local project editor

To add, edit, or remove a site-approved remote project source, run:

python scripts/project_editor.py

The project editor asks for one public GitHub or Gitea repository web URL and derives the anonymous Git and provider API URLs stored in configs/project-sources.ini. Project prose, technology labels, images, and devlog records remain owned by the external project repository under .labyricorn/. Its explicit Check Devlog Status action performs the importer's public-provider and remote-snapshot validation against a disposable read-only Git mirror, then removes that mirror. When setup is missing, the editor directs the user to copy the standalone scripts/devlog_editor.py into the project checkout, initialize and author the devlog, publish the changes, and recheck. It never edits, commits, or pushes an external project repository.

Project Editor requires the Lektor Python package because it uses the site's authoritative importer directly. If the launching Python cannot import Lektor, the editor automatically checks LABYRICORN_LEKTOR_PYTHON, a project-local .venv, common pipx Lektor environments, and the environment containing a lektor command. It relaunches itself with the first runtime that can import both Lektor and Tkinter. Set LABYRICORN_LEKTOR_PYTHON to an explicit Python executable when Lektor is installed somewhere else.

Production filesystem and permissions

/srv/labyricorn/
├── repo/                 Git working copy (not web-accessible)
├── releases/             Successful immutable builds
├── current -> releases/<release>
├── logs/
│   ├── deploy.log
│   └── deploy.lock
├── project-cache/        Bare mirrors, metadata, and last-known-good state
├── .local/bin/lektor     pipx-installed Lektor
├── .netrc                Git HTTPS credential (0600; secret)
├── .ssh/                 Reserved deployment SSH material
└── README-DEPLOYMENT.md  Supplemental host-local notes

The labyricorn-deploy account owns the checkout, releases, logs, Lektor environment, and Git credential. nginx's www-data account can traverse the base path and read active generated files, but cannot read the source checkout or credential. nginx cannot modify either source or releases.

Deployment behavior

/usr/local/bin/deploy-labyricorn performs the following sequence:

1. Changes from root to the unprivileged labyricorn-deploy account.
2. Acquires an exclusive flock lock to prevent concurrent deployments.
3. Refuses to proceed if the working tree contains any tracked or untracked changes.
4. Fetches and prunes origin.
5. Verifies that origin/main exists.
6. Switches to local main and resets it exactly to origin/main.
7. Creates a unique release named with UTC time and the 12-character commit ID.
8. Runs scripts/build_with_projects.py with the persistent project cache and an explicit output directory.
9. Fetches, validates, and imports each approved public project source into an isolated temporary workspace before invoking Lektor.
10. Requires index.html and .labyricorn-projects.json.
11. Writes the full site commit ID to .labyricorn-commit.
12. Compares the site commit plus project commit/metadata digests with the active release and removes the candidate as an unchanged no-op when equal.
13. Atomically replaces /srv/labyricorn/current with a relative symlink to the new release.
14. Keeps the active release plus recent prior releases, targeting five total.
15. Logs start, failure, unchanged, success, release name, and commit.

Run it with:

sudo deploy-labyricorn

A failed build is removed before activation. The current symlink is not changed, so the previously working site remains online.

Deployment preflight

sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo status --short --branch

sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo fetch origin

The status must be clean. Do not bypass a dirty-tree refusal. Determine whether the files are legitimate unpublished admin edits, then commit and push them or remove them deliberately before deployment.

Verify a deployment

readlink -f /srv/labyricorn/current
cat /srv/labyricorn/current/.labyricorn-commit

sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo rev-parse HEAD

sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo ls-remote origin refs/heads/main

nginx -t
systemctl is-active nginx

curl -I -H 'Host: www.labyricorn.com' \
  -H 'X-Forwarded-Proto: https' http://127.0.0.1/

curl -I https://www.labyricorn.com/

The commit in the active release, local HEAD, and remote main should match. The local nginx request with X-Forwarded-Proto: https should return 200. Public HTTP should redirect to HTTPS, and public HTTPS should return 200.

Rollback

List releases and identify the current one:

sudo rollback-labyricorn

Switch atomically to a previous successful release:

sudo rollback-labyricorn 20260812T024742Z-92475d2ec759

Use a release name printed by the listing command. Rollback does not build, fetch, or alter Git. It only changes the current symlink and records the event in the deployment log.

Verify afterward:

readlink -f /srv/labyricorn/current
cat /srv/labyricorn/current/.labyricorn-commit
curl -I -H 'Host: www.labyricorn.com' \
  -H 'X-Forwarded-Proto: https' http://127.0.0.1/

A normal deployment after rollback rebuilds and reactivates the latest remote main commit.

nginx and URL handling

The site configuration is:

/etc/nginx/sites-available/labyricorn
/etc/nginx/sites-enabled/labyricorn -> ../sites-available/labyricorn

nginx serves /srv/labyricorn/current and supports:

• index.html at directory-style URLs;
• direct static assets;
• optional .html fallback;
• generated /404/index.html with an HTTP 404 response;
• disabled directory browsing;
• rejection of hidden paths and common executable-script extensions;
• dedicated access and error logs.

Always validate before reloading nginx:

nginx -t
systemctl reload nginx

nginx logs:

/var/log/nginx/labyricorn-access.log
/var/log/nginx/labyricorn-error.log

HTTPS, Cloudflare, and DNS

Current production HTTPS terminates at Cloudflare. The origin nginx process currently listens on port 80, not 443. For www.labyricorn.com, nginx uses Cloudflare's X-Forwarded-Proto header to distinguish public HTTP from HTTPS:

• public HTTP redirects to https://www.labyricorn.com;
• Cloudflare-originated HTTPS requests are served without a redirect loop;
• labyricorn.com is configured at nginx to redirect to the canonical HTTPS www hostname.

The apex labyricorn.com did not resolve during initial setup. Its redirect will not work publicly until DNS/Cloudflare has an apex record routed to this origin.

Certbot and the nginx integration are installed, and certbot.timer is enabled, but no origin certificate was issued during setup. Do not run Certbot blindly behind Cloudflare. First decide whether the desired origin mode is a Cloudflare Origin CA certificate, Let's Encrypt with a compatible challenge, or direct DNS without the proxy. After changing TLS, test both origin and public behavior for redirect loops.

Useful checks:

getent ahosts www.labyricorn.com
getent ahosts labyricorn.com
curl -I http://www.labyricorn.com/
curl -I https://www.labyricorn.com/
systemctl status certbot.timer
certbot certificates

Firewall and exposed services

UFW is active with default incoming denial. Allowed services are:

• OpenSSH: TCP 22
• nginx HTTP/HTTPS profile: TCP 80 and 443

Inspect current state:

ufw status verbose
ss -lntup

Lektor's development port must never be opened in UFW or bound to 0.0.0.0.

Optional server-side Lektor admin

An optional unit exists at:

/etc/systemd/system/lektor-admin.service

Current state: disabled and stopped. When started, it runs as labyricorn-deploy and binds only to 127.0.0.1:5000. It is not proxied by nginx and production remains independent of it.

Start it temporarily:

systemctl start lektor-admin
systemctl status lektor-admin
ss -lntp | grep ':5000'

Access it securely through an SSH tunnel from an administrator workstation:

ssh -L 5000:127.0.0.1:5000 [email protected]

Then browse to http://127.0.0.1:5000 on that workstation.

Stop it when finished:

systemctl stop lektor-admin

Committing admin edits made on the server

Lektor can edit files, but it does not automatically create Git commits or push them. The deployment account intentionally has authenticated Git write access so a controlled admin workflow can publish server-side edits.

Review everything before committing:

sudo -u labyricorn-deploy env HOME=/srv/labyricorn bash
cd /srv/labyricorn/repo
git status
git diff
git add <reviewed-files>
git commit -m "Describe the admin edit"
git push origin main
exit

Then run sudo deploy-labyricorn. The deployment script will refuse to run while uncommitted admin edits remain. This is deliberate protection against silently discarding edits when it resets to origin/main.

Before enabling persistent or public admin access, add authentication and make a separate security decision. Do not publish port 5000 directly and do not proxy an unauthenticated editor through nginx.

Git authentication and secrets

The site repository is accessed over authenticated HTTPS because the Gitea SSH port was not reachable through git.labyricorn.com during setup. Approved remote project repositories, including Thinkloom, are public and are fetched without embedding credentials in their configured URLs.

Credential purpose and location:

Purpose	Location	Permissions
Gitea HTTPS fetch/push	/srv/labyricorn/.netrc	0600, owned by labyricorn-deploy
Reserved SSH identity	/srv/labyricorn/.ssh/id_ed25519	0600, owned by labyricorn-deploy

Never place token contents, passwords, or private keys in this README, Git URLs, shell scripts, logs, commits, issue trackers, or command history.

Check credential-file metadata without printing its contents:

stat -c '%A %U:%G %n' /srv/labyricorn/.netrc

If the Gitea token is rotated, replace .netrc as labyricorn-deploy, preserve mode 0600, and verify both fetch and push access before revoking the old token. Avoid printing either credential during rotation.

Logs and diagnostics

# Deployment and rollback history
tail -n 100 /srv/labyricorn/logs/deploy.log

# nginx service and request errors
journalctl -u nginx --since today
tail -n 100 /var/log/nginx/labyricorn-error.log

# Optional admin service
journalctl -u lektor-admin --since today

# Repository state
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo status

# Active release
readlink -f /srv/labyricorn/current
cat /srv/labyricorn/current/.labyricorn-commit

# Services and sockets
systemctl status nginx
systemctl status certbot.timer
ss -lntup

Troubleshooting

Deployment says the repository is dirty

Inspect as the deployment account:

sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo status --short

If these are intended admin edits, review, commit, and push them. Otherwise, identify their origin before removing anything. Never use a destructive reset until you are certain no work needs to be preserved.

Git fetch or push fails

Check DNS, HTTPS, the remote URL, and credential permissions without displaying the secret:

getent ahosts git.labyricorn.com
curl -I https://git.labyricorn.com/
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  git -C /srv/labyricorn/repo remote -v
stat -c '%A %U:%G %n' /srv/labyricorn/.netrc

An authentication failure usually means the token was revoked, expired, or lacks repository access.

Lektor build fails

Run a disposable diagnostic build; do not build into current:

test_dir=$(mktemp -d /srv/labyricorn/releases/.diagnostic.XXXXXX)
chown labyricorn-deploy:labyricorn-deploy "$test_dir"
sudo -u labyricorn-deploy env HOME=/srv/labyricorn \
  /srv/labyricorn/.local/pipx/venvs/lektor/bin/python \
  /srv/labyricorn/repo/scripts/build_with_projects.py \
  --site-root /srv/labyricorn/repo \
  --cache-path /srv/labyricorn/project-cache \
  --output-path "$test_dir" \
  --lektor /srv/labyricorn/.local/bin/lektor

Inspect the error and source. Remove the diagnostic directory only after confirming its exact path and that it is not referenced by current.

nginx returns an error

nginx -t
systemctl status nginx
readlink -f /srv/labyricorn/current
test -f /srv/labyricorn/current/index.html
tail -n 100 /var/log/nginx/labyricorn-error.log

Do not point nginx at the Git checkout or start Lektor as a production server.

Public HTTPS loops or behaves differently from local HTTP

Check Cloudflare proxy mode and the forwarded protocol behavior:

curl -I -H 'Host: www.labyricorn.com' http://127.0.0.1/
curl -I -H 'Host: www.labyricorn.com' \
  -H 'X-Forwarded-Proto: https' http://127.0.0.1/
curl -I http://www.labyricorn.com/
curl -I https://www.labyricorn.com/

The first origin request should redirect; the simulated Cloudflare HTTPS request should return the site. A different result indicates a proxy/header or nginx configuration problem.

Remote project synchronization fails

Inspect the deployment log and active manifest without editing the cache:

tail -n 100 /srv/labyricorn/logs/deploy.log
cat /srv/labyricorn/current/.labyricorn-projects.json

current means the latest remote revision and provider metadata validated. metadata-cached means Git content is current but provider metadata fell back to its cache. last-known-good means the latest fetch or candidate snapshot could not safely replace the previously validated commit. A first import with no valid cached snapshot fails the deployment. Do not bypass validation by copying remote files into content/, the build workspace, or an active release.

Automatic deployment and project refresh

The workflow at .gitea/workflows/deploy.yml runs for three event types:

• every push to site main;
• every 30 minutes (*/30 * * * *, evaluated by Gitea in UTC);
• an authenticated manual workflow_dispatch.

All events run the same one-job deployment:

1. Run on the repository-scoped runner labelled labyricorn-deploy.
2. Execute the existing root-owned deploy-labyricorn command.
3. Confirm /srv/labyricorn/current/.labyricorn-commit equals the workflow's site commit.
4. Request the site through nginx on the local origin and fail if it is unhealthy.

The action intentionally does not check out the repository into its own workspace. The deployment script fetches origin/main using the dedicated labyricorn-deploy account, validates a clean checkout, builds a new release, and atomically changes the current symlink. Concurrent action runs share the labyricorn-production concurrency group, and the deployment script also uses a filesystem lock. A scheduled/manual run updates the site when an approved project commit or public metadata digest changes. When neither the site nor any project input changed, it validates and builds a candidate, removes it, logs an unchanged result, and leaves the active symlink untouched.

Trigger a refresh

From Gitea, open Labyricorn/labyricorn-site → Actions → Deploy production and choose Run workflow for main.

Automation and coding assistants may dispatch the same workflow through Gitea's authenticated API after pushing project publishing content. With LABYRICORN_REFRESH_TOKEN supplied by an approved credential store or ephemeral environment, run:

python scripts/trigger_project_refresh.py

Use python scripts/trigger_project_refresh.py --dry-run to inspect the fixed site workflow target without reading the token or making a network request. The script sends only {"ref":"main"}, keeps the credential out of the URL and request body, and reports whether Gitea accepted the dispatch. It does not wait for or claim a successful deployment.

The token must be supplied through an approved credential store or ephemeral environment and must have permission to dispatch Actions for the site repository. Never put it in the URL, a project repository, AGENTS.md, logs, or committed scripts. Send at most one dispatch after the relevant project push, then verify the Action and public project revision. The scheduled run is the fallback when no refresh credential is available. This is deliberately not an unauthenticated public webhook or long-running URL listener, so it cannot be spammed by anonymous requests.

Runner installation

The production host runs official Gitea Runner 3.0.0 directly on Linux; Docker is not installed or required for this workflow.

Item	Value
systemd unit	gitea-runner.service
service account	gitea-runner
runner name	labyricorn-production
registration scope	Labyricorn/labyricorn-site repository
execution label	labyricorn-deploy:host
runner binary	/usr/local/bin/gitea-runner
runner configuration	/etc/gitea-runner/config.yaml
registration state	/var/lib/gitea-runner/.runner
job workspace	/var/lib/gitea-runner/work
sudo policy	/etc/sudoers.d/gitea-runner-labyricorn

The runner account has no general administrative access. Its sudo policy allows only this exact command without a password:

/usr/local/bin/deploy-labyricorn

Useful checks:

systemctl status gitea-runner
journalctl -u gitea-runner -n 100 --no-pager
sudo -l -U gitea-runner

If the runner must be replaced, delete or disable it in Repository Settings → Actions → Runners, generate a fresh repository registration token, and register the replacement. Registration tokens and .runner contents are credentials: never commit or print them. A push can still be deployed manually with sudo deploy-labyricorn while the runner is unavailable.

Change-control checklist

Before changing deployment, nginx, TLS, firewall, credentials, or admin access:

1. Inspect the live state and preserve unrelated configuration.
2. Back up the specific file being changed.
3. Validate syntax before reload or restart.
4. Keep SSH reachable before firewall changes.
5. Verify local origin behavior and public Cloudflare behavior separately.
6. Record operational changes in this README.
7. Commit and push source documentation to main.

This README describes the current verified installation on ubuntu2-47. Update it whenever the operational architecture changes.

User Note: Check 'C:\Users-username-\AppData\Local\pipx\pipx\venvs\lektor' for existing lektor install