Add project exhibition, development logs, and complete master README
This commit is contained in:
@@ -0,0 +1,200 @@
|
||||
# ⚡ ThinkStorm
|
||||
|
||||
> **Frictionless Anonymous Idea Intake, Autonomous AI Research, and Project Incubation Engine.**
|
||||
|
||||
ThinkStorm captures unformed ideas with zero friction, preserves original submissions immutably, and orchestrates self-hosted AI models to research the landscape, synthesize feasibility, and incubate full Gitea projects.
|
||||
|
||||
---
|
||||
|
||||
## 🌟 Key Features
|
||||
|
||||
### 1. Frictionless Anonymous Intake
|
||||
- **Zero Required Metadata**: Submit raw, free-form thoughts without mandatory titles, categories, or tags.
|
||||
- **Embedded URL Isolation & Safety Scanner**: URLs inside submissions are extracted and evaluated via VirusTotal. Unsafe or suspicious links are automatically quarantined for administrator review before any model processing runs.
|
||||
- **Immutable Prompt Preservation**: The original submission prompt is permanently locked in the database and displayed verbatim across all views.
|
||||
|
||||
### 2. Multi-Stage Research & Evaluation Pipeline
|
||||
- **Title & Summary Extraction**: AI extracts semantic titles, executive summaries, and initial taxonomies.
|
||||
- **Duplicate & Relationship Mapping**: Automatically identifies conceptual overlaps and clusters related ideas.
|
||||
- **Prior Art & Competitor Search**: Queries live web indices via SearXNG and Perplexica to discover similar projects, existing tools, and architectural differentiators.
|
||||
- **Deep Technical Feasibility & Risk Critique**: Generates feasibility scores (0–10), technical constraints, implementation roadmaps, and risk matrices.
|
||||
|
||||
### 3. Canonical Gitea Project Repositories
|
||||
- Every incubated idea is automatically provisioned as a full Git repository under the **`thinkstorm`** organization on Gitea (e.g. `https://git.labyricorn.com/thinkstorm/ts-0032`).
|
||||
- Repositories include:
|
||||
- `README.md`: Executive summary, prompt quotation, lifecycle badge, and project structure guide.
|
||||
- `metadata.json`: Machine-readable metadata schema.
|
||||
- `research/`: `prior-art.md`, `analysis.md`, `feasibility.md`.
|
||||
- `outputs/`: Multi-modal work track deliverables organized by track.
|
||||
- `provenance/`: Execution run logs and token attribution telemetry.
|
||||
- Full Git history, branch management, issue tracking, and SSH/HTTPS cloning (`git clone ...`) are natively supported.
|
||||
|
||||
### 4. Multi-Modal Work Tracks & Versioning
|
||||
- Ideas can be claimed by authenticated users to unlock specialized Work Tracks:
|
||||
- 📄 **Article & Whitepaper**: Structured long-form analysis and publication drafts.
|
||||
- 💻 **Coding Project**: Software architectures, API specs, and codebase scaffolds.
|
||||
- ✍️ **Blog Entry**: Conversational announcement posts and technical retrospectives.
|
||||
- **Iterative Deliverable Versioning**: Subsequent workflow runs archive previous outputs as versions (`v1`, `v2`, `v3`) with model attribution, allowing fine-grained deletion and draft comparison.
|
||||
|
||||
### 5. Authentication & Access Control
|
||||
- **Gitea OAuth2 Single Sign-On**: Log in with your Gitea account with automatic profile provisioning.
|
||||
- **Local Fallback Authentication**: Secure bcrypt password authentication for administrative and local accounts.
|
||||
- **Role-Based Access Control (RBAC)**: Supports `ANONYMOUS`, `USER`, and `ADMIN` permission tiers.
|
||||
|
||||
### 6. Administration, Queue & Provenance
|
||||
- **Background Job Queue**: Asynchronous, non-blocking pipeline execution with foreground prioritization and retry controls.
|
||||
- **Prompt Catalog & Semantic Versioning**: Edit prompt templates dynamically with version tracking and profile activation.
|
||||
- **Token Accounting & Model Telemetry**: Comprehensive auditing of input/output token usage, provider durations, and execution status.
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Architecture & Technology Stack
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
User([User / Browser]) -->|Submit / Browse| Web[FastAPI Web Server]
|
||||
Web -->|Store / Query| DB[(SQLite WAL Database)]
|
||||
Web -->|Enqueue Jobs| Queue[Background Queue Worker]
|
||||
|
||||
Queue --> P1[1. Safety & URL Evaluator]
|
||||
Queue --> P2[2. Semantic Title & Tag Extractor]
|
||||
Queue --> P3[3. Duplicate & Cluster Detector]
|
||||
Queue --> P4[4. Prior Art Search]
|
||||
Queue --> P5[5. Feasibility & Risk Synthesis]
|
||||
|
||||
P1 -->|Scan URLs| VT[VirusTotal API]
|
||||
P4 -->|Web Search| SearXNG[SearXNG & Perplexica]
|
||||
P2 & P3 & P4 & P5 -->|LLM Synthesis| Omni[OmniRoute LLM Router]
|
||||
|
||||
Queue -->|Push Dossier Repo| Gitea[Gitea Project Host]
|
||||
Queue -->|Push Gist Snippets| OpenGist[OpenGist Service]
|
||||
```
|
||||
|
||||
- **Backend**: Python 3.13, FastAPI, Uvicorn, AsyncIO, Pydantic
|
||||
- **Database**: SQLite 3 with Write-Ahead Logging (`WAL`), Foreign Keys, and automated migrations
|
||||
- **Frontend**: Jinja2 Templates, Vanilla Modern CSS (Dark Mode, Glassmorphism), Modular JavaScript
|
||||
- **AI Orchestration**: OmniRoute (Ollama, OpenAI, Anthropic, vLLM endpoints)
|
||||
- **Web Search**: SearXNG (Meta-search API) & Perplexica (Search Backend)
|
||||
- **Code & Version Control**: Gitea (OAuth2 SSO & Git Repositories), OpenGist (Gist Dossiers)
|
||||
- **Security & Safety**: VirusTotal API (URL domain risk analysis)
|
||||
|
||||
---
|
||||
|
||||
## 📁 Repository Structure
|
||||
|
||||
```
|
||||
.
|
||||
├── thinkstorm/ # Core application package
|
||||
│ ├── main.py # FastAPI application entrypoint & middleware
|
||||
│ ├── config.py # Service configuration & dynamic .env loader
|
||||
│ ├── database.py # SQLite database schema, migrations & seed defaults
|
||||
│ ├── models.py # Dataclasses, Enums, and Pydantic models
|
||||
│ ├── auth.py # Password hashing, JWT sessions & RBAC
|
||||
│ ├── api/ # REST API routers
|
||||
│ │ ├── ideas.py # Idea submission, claiming, work tracks & sync
|
||||
│ │ ├── auth_routes.py # Local login & Gitea OAuth2 SSO callback
|
||||
│ │ └── admin.py # Service testing, queue retry, quarantine moderation
|
||||
│ ├── processors/ # Pipeline execution engine
|
||||
│ │ └── pipeline.py # Bounded processors 1-5 & work track workflows
|
||||
│ ├── prompts/ # Prompt definitions & profile catalog
|
||||
│ │ └── catalog.py # System prompts, profile alignment & versioning
|
||||
│ ├── queue/ # Background job queue
|
||||
│ │ └── worker.py # Async worker queue with foreground priority
|
||||
│ ├── services/ # External service adapters
|
||||
│ │ ├── gitea.py # Gitea repository creation & file sync
|
||||
│ │ ├── opengist.py # OpenGist REST sync adapter
|
||||
│ │ ├── omniroute.py # LLM completion & streaming adapter
|
||||
│ │ ├── searxng.py # Live meta-search adapter
|
||||
│ │ ├── perplexica.py # Perplexica search backend adapter
|
||||
│ │ └── virustotal.py # URL reputation scanner adapter
|
||||
│ ├── templates/ # Jinja2 HTML templates
|
||||
│ │ ├── base.html # Core layout, navigation & status bar
|
||||
│ │ ├── index.html # Frictionless hero intake page
|
||||
│ │ ├── ideas.html # Filterable ideas list with lifecycle states
|
||||
│ │ ├── idea_detail.html # Comprehensive idea dossier & work tracks
|
||||
│ │ ├── login.html # Gitea SSO & local login portal
|
||||
│ │ ├── prompt_catalog.html # Prompt template editor & version manager
|
||||
│ │ └── admin.html # Service health & queue management
|
||||
│ └── static/ # Static assets
|
||||
│ ├── css/style.css # Glassmorphic dark design system
|
||||
│ └── js/app.js # Interactive tab navigation, toasts & sync
|
||||
├── tests/ # Test suite
|
||||
│ ├── test_api.py # API endpoint & permission tests
|
||||
│ └── test_artifact_versioning.py # Versioning & lifecycle tests
|
||||
├── .labyricorn/ # Labyricorn exhibition & devlog space
|
||||
│ ├── project/contents.lr # Project exhibition record
|
||||
│ └── devlog/ # Public development logs
|
||||
├── data/ # Persistent local data & SQLite store
|
||||
└── README.md # This documentation file
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ Configuration (`.env`)
|
||||
|
||||
ThinkStorm loads environment settings from `/root/.env`, `.env`, and `thinkstorm/.env`:
|
||||
|
||||
| Variable | Description | Default / Example |
|
||||
| :--- | :--- | :--- |
|
||||
| `OMNIROUTE_URL` | OmniRoute LLM API base endpoint | `https://llm.example.com` |
|
||||
| `OMNIROUTE_API_KEY` | Bearer token for LLM routing | `<your-omniroute-api-key>` |
|
||||
| `SEARXNG_URL` | SearXNG meta-search instance | `https://search.example.com` |
|
||||
| `PERPLEXICA_URL` | Perplexica backend instance | `https://perplexica.example.com` |
|
||||
| `GITEA_URL` | Gitea instance public URL | `https://git.example.com` |
|
||||
| `GITEA_API_TOKEN` | Gitea admin/user access token | `<your-gitea-api-token>` |
|
||||
| `GITEA_CLIENT_ID` | Gitea OAuth2 Application Client ID | `<your-gitea-client-id>` |
|
||||
| `GITEA_CLIENT_SECRET` | Gitea OAuth2 Application Secret | `<your-gitea-client-secret>` |
|
||||
| `OPENGIST_URL` | OpenGist public URL | `https://gist.example.com` |
|
||||
| `OPENGIST_API_TOKEN` | OpenGist Personal Access Token | `<your-opengist-api-token>` |
|
||||
| `VIRUSTOTAL_API_KEY` | VirusTotal API Key for URL safety | `<your-virustotal-api-key>` |
|
||||
| `ADMIN_BOOTSTRAP_KEY` | Default admin account password | `<your-admin-bootstrap-password>` |
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Getting Started
|
||||
|
||||
### 1. Installation
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://git.labyricorn.com/thinkstorm/thinkstorm.git
|
||||
cd thinkstorm
|
||||
|
||||
# Install Python dependencies
|
||||
pip install fastapi uvicorn pydantic jinja2 requests pytest pytest-asyncio httpx
|
||||
```
|
||||
|
||||
### 2. Running the Server
|
||||
```bash
|
||||
# Start ThinkStorm on port 8000
|
||||
python3 -m uvicorn thinkstorm.main:app --host 0.0.0.0 --port 8000 --proxy-headers --forwarded-allow-ips='*'
|
||||
```
|
||||
|
||||
Access the application in your browser at **`http://localhost:8000`** (or your domain e.g. **`https://ts.labyricorn.com`**).
|
||||
|
||||
### 3. Running Automated Tests
|
||||
```bash
|
||||
PYTHONPATH=. pytest tests/ -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📖 API Endpoints Summary
|
||||
|
||||
- `POST /api/ideas`: Anonymous free-form idea submission.
|
||||
- `GET /api/ideas`: Filterable list of ideas by lifecycle state, category, or tag.
|
||||
- `GET /api/ideas/{idea_id}`: Full idea details, provenance, research, and work tracks.
|
||||
- `POST /api/ideas/{idea_id}/claim`: Claim an available idea for incubation.
|
||||
- `POST /api/ideas/{idea_id}/activate`: Activate an idea into active development.
|
||||
- `POST /api/ideas/{idea_id}/sync-gitea`: Push or re-sync dossier files to Gitea repository.
|
||||
- `POST /api/ideas/work-tracks/{track_id}/activate`: Run or re-run a work track workflow.
|
||||
- `DELETE /api/ideas/work-tracks/outputs/{output_id}`: Delete specific deliverable version.
|
||||
- `POST /api/auth/login`: Local username/password authentication.
|
||||
- `GET /api/auth/gitea/url`: Get Gitea OAuth2 authorization URL.
|
||||
- `GET /auth/gitea/callback`: OAuth2 redirect callback & session issuer.
|
||||
- `POST /api/admin/services/{service_id}/test`: Real-time health check for external services.
|
||||
|
||||
---
|
||||
|
||||
## 📄 License & Attribution
|
||||
|
||||
Developed by **Labyricorn**. Licensed under the [MIT License](LICENSE).
|
||||
Reference in New Issue
Block a user