Labyricorn/ThinkStorm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
94ff1e4408bf938c06f5a3e7db926878c34b0983
ThinkStorm/README.md
T

11 KiB
Raw Blame History

⚡ ThinkStorm

Frictionless Anonymous Idea Intake, Autonomous AI Research, and Project Incubation Engine.

ThinkStorm captures unformed ideas with zero friction, preserves original submissions immutably, and orchestrates self-hosted AI models to research the landscape, synthesize feasibility, and incubate full Gitea projects.

🌟 Key Features

1. Frictionless Anonymous Intake

  • Zero Required Metadata: Submit raw, free-form thoughts without mandatory titles, categories, or tags.
  • Embedded URL Isolation & Safety Scanner: URLs inside submissions are extracted and evaluated via VirusTotal. Unsafe or suspicious links are automatically quarantined for administrator review before any model processing runs.
  • Immutable Prompt Preservation: The original submission prompt is permanently locked in the database and displayed verbatim across all views.

2. Multi-Stage Research & Evaluation Pipeline

  • Title & Summary Extraction: AI extracts semantic titles, executive summaries, and initial taxonomies.
  • Duplicate & Relationship Mapping: Automatically identifies conceptual overlaps and clusters related ideas.
  • Prior Art & Competitor Search: Queries live web indices via SearXNG and Perplexica to discover similar projects, existing tools, and architectural differentiators.
  • Deep Technical Feasibility & Risk Critique: Generates feasibility scores (0–10), technical constraints, implementation roadmaps, and risk matrices.

3. Canonical Gitea Project Repositories

  • Every incubated idea is automatically provisioned as a full Git repository under the thinkstorm organization on Gitea (e.g. https://git.labyricorn.com/thinkstorm/ts-0032).
  • Repositories include:
    • README.md: Executive summary, prompt quotation, lifecycle badge, and project structure guide.
    • metadata.json: Machine-readable metadata schema.
    • research/: prior-art.md, analysis.md, feasibility.md.
    • outputs/: Multi-modal work track deliverables organized by track.
    • provenance/: Execution run logs and token attribution telemetry.
  • Full Git history, branch management, issue tracking, and SSH/HTTPS cloning (git clone ...) are natively supported.

4. Multi-Modal Work Tracks & Versioning

  • Ideas can be claimed by authenticated users to unlock specialized Work Tracks:
    • 📄 Article & Whitepaper: Structured long-form analysis and publication drafts.
    • 💻 Coding Project: Software architectures, API specs, and codebase scaffolds.
    • ✍️ Blog Entry: Conversational announcement posts and technical retrospectives.
  • Iterative Deliverable Versioning: Subsequent workflow runs archive previous outputs as versions (v1, v2, v3) with model attribution, allowing fine-grained deletion and draft comparison.

5. Authentication & Access Control

  • Gitea OAuth2 Single Sign-On: Log in with your Gitea account with automatic profile provisioning.
  • Local Fallback Authentication: Secure bcrypt password authentication for administrative and local accounts.
  • Role-Based Access Control (RBAC): Supports ANONYMOUS, USER, and ADMIN permission tiers.

6. Administration, Queue & Provenance

  • Background Job Queue: Asynchronous, non-blocking pipeline execution with foreground prioritization and retry controls.
  • Prompt Catalog & Semantic Versioning: Edit prompt templates dynamically with version tracking and profile activation.
  • Token Accounting & Model Telemetry: Comprehensive auditing of input/output token usage, provider durations, and execution status.

🏗️ Architecture & Technology Stack

graph TD
    User([User / Browser]) -->|Submit / Browse| Web[FastAPI Web Server]
    Web -->|Store / Query| DB[(SQLite WAL Database)]
    Web -->|Enqueue Jobs| Queue[Background Queue Worker]

    Queue --> P1[1. Safety & URL Evaluator]
    Queue --> P2[2. Semantic Title & Tag Extractor]
    Queue --> P3[3. Duplicate & Cluster Detector]
    Queue --> P4[4. Prior Art Search]
    Queue --> P5[5. Feasibility & Risk Synthesis]

    P1 -->|Scan URLs| VT[VirusTotal API]
    P4 -->|Web Search| SearXNG[SearXNG & Perplexica]
    P2 & P3 & P4 & P5 -->|LLM Synthesis| Omni[OmniRoute LLM Router]

    Queue -->|Push Dossier Repo| Gitea[Gitea Project Host]
    Queue -->|Push Gist Snippets| OpenGist[OpenGist Service]
  • Backend: Python 3.13, FastAPI, Uvicorn, AsyncIO, Pydantic
  • Database: SQLite 3 with Write-Ahead Logging (WAL), Foreign Keys, and automated migrations
  • Frontend: Jinja2 Templates, Vanilla Modern CSS (Dark Mode, Glassmorphism), Modular JavaScript
  • AI Orchestration: OmniRoute (Ollama, OpenAI, Anthropic, vLLM endpoints)
  • Web Search: SearXNG (Meta-search API) & Perplexica (Search Backend)
  • Code & Version Control: Gitea (OAuth2 SSO & Git Repositories), OpenGist (Gist Dossiers)
  • Security & Safety: VirusTotal API (URL domain risk analysis)

📁 Repository Structure

.
├── thinkstorm/                  # Core application package
│   ├── main.py                  # FastAPI application entrypoint & middleware
│   ├── config.py                # Service configuration & dynamic .env loader
│   ├── database.py              # SQLite database schema, migrations & seed defaults
│   ├── models.py                # Dataclasses, Enums, and Pydantic models
│   ├── auth.py                  # Password hashing, JWT sessions & RBAC
│   ├── api/                     # REST API routers
│   │   ├── ideas.py             # Idea submission, claiming, work tracks & sync
│   │   ├── auth_routes.py       # Local login & Gitea OAuth2 SSO callback
│   │   └── admin.py             # Service testing, queue retry, quarantine moderation
│   ├── processors/              # Pipeline execution engine
│   │   └── pipeline.py          # Bounded processors 1-5 & work track workflows
│   ├── prompts/                 # Prompt definitions & profile catalog
│   │   └── catalog.py           # System prompts, profile alignment & versioning
│   ├── queue/                   # Background job queue
│   │   └── worker.py            # Async worker queue with foreground priority
│   ├── services/                # External service adapters
│   │   ├── gitea.py             # Gitea repository creation & file sync
│   │   ├── opengist.py          # OpenGist REST sync adapter
│   │   ├── omniroute.py         # LLM completion & streaming adapter
│   │   ├── searxng.py           # Live meta-search adapter
│   │   ├── perplexica.py        # Perplexica search backend adapter
│   │   └── virustotal.py        # URL reputation scanner adapter
│   ├── templates/               # Jinja2 HTML templates
│   │   ├── base.html            # Core layout, navigation & status bar
│   │   ├── index.html           # Frictionless hero intake page
│   │   ├── ideas.html           # Filterable ideas list with lifecycle states
│   │   ├── idea_detail.html     # Comprehensive idea dossier & work tracks
│   │   ├── login.html           # Gitea SSO & local login portal
│   │   ├── prompt_catalog.html  # Prompt template editor & version manager
│   │   └── admin.html           # Service health & queue management
│   └── static/                  # Static assets
│       ├── css/style.css        # Glassmorphic dark design system
│       └── js/app.js            # Interactive tab navigation, toasts & sync
├── tests/                       # Test suite
│   ├── test_api.py              # API endpoint & permission tests
│   └── test_artifact_versioning.py # Versioning & lifecycle tests
├── .labyricorn/                 # Labyricorn exhibition & devlog space
│   ├── project/contents.lr      # Project exhibition record
│   └── devlog/                  # Public development logs
├── data/                        # Persistent local data & SQLite store
└── README.md                    # This documentation file

⚙️ Configuration (.env)

ThinkStorm loads environment settings from /root/.env, .env, and thinkstorm/.env:

Variable Description Default / Example
OMNIROUTE_URL OmniRoute LLM API base endpoint https://llm.example.com
OMNIROUTE_API_KEY Bearer token for LLM routing <your-omniroute-api-key>
SEARXNG_URL SearXNG meta-search instance https://search.example.com
PERPLEXICA_URL Perplexica backend instance https://perplexica.example.com
GITEA_URL Gitea instance public URL https://git.example.com
GITEA_API_TOKEN Gitea admin/user access token <your-gitea-api-token>
GITEA_CLIENT_ID Gitea OAuth2 Application Client ID <your-gitea-client-id>
GITEA_CLIENT_SECRET Gitea OAuth2 Application Secret <your-gitea-client-secret>
OPENGIST_URL OpenGist public URL https://gist.example.com
OPENGIST_API_TOKEN OpenGist Personal Access Token <your-opengist-api-token>
VIRUSTOTAL_API_KEY VirusTotal API Key for URL safety <your-virustotal-api-key>
ADMIN_BOOTSTRAP_KEY Default admin account password <your-admin-bootstrap-password>

🚀 Getting Started

1. Installation

# Clone the repository
git clone https://git.labyricorn.com/thinkstorm/thinkstorm.git
cd thinkstorm

# Install Python dependencies
pip install fastapi uvicorn pydantic jinja2 requests pytest pytest-asyncio httpx

2. Running the Server

# Start ThinkStorm on port 8000
python3 -m uvicorn thinkstorm.main:app --host 0.0.0.0 --port 8000 --proxy-headers --forwarded-allow-ips='*'

Access the application in your browser at http://localhost:8000 (or your domain e.g. https://ts.labyricorn.com).

3. Running Automated Tests

PYTHONPATH=. pytest tests/ -v

📖 API Endpoints Summary

  • POST /api/ideas: Anonymous free-form idea submission.
  • GET /api/ideas: Filterable list of ideas by lifecycle state, category, or tag.
  • GET /api/ideas/{idea_id}: Full idea details, provenance, research, and work tracks.
  • POST /api/ideas/{idea_id}/claim: Claim an available idea for incubation.
  • POST /api/ideas/{idea_id}/activate: Activate an idea into active development.
  • POST /api/ideas/{idea_id}/sync-gitea: Push or re-sync dossier files to Gitea repository.
  • POST /api/ideas/work-tracks/{track_id}/activate: Run or re-run a work track workflow.
  • DELETE /api/ideas/work-tracks/outputs/{output_id}: Delete specific deliverable version.
  • POST /api/auth/login: Local username/password authentication.
  • GET /api/auth/gitea/url: Get Gitea OAuth2 authorization URL.
  • GET /auth/gitea/callback: OAuth2 redirect callback & session issuer.
  • POST /api/admin/services/{service_id}/test: Real-time health check for external services.

📄 License & Attribution

Developed by Labyricorn. Licensed under the MIT License.

Reference in New Issue View Git Blame Copy Permalink