From 3a68b0fa0705b5cdbae89d4b47cf4ba6342fd998 Mon Sep 17 00:00:00 2001 From: mjalmoudy94 Date: Tue, 13 Oct 2026 22:57:06 +0330 Subject: [PATCH] docs: v2 plan (shared plan + foundation + 3 agent TODO files) Co-Authored-By: Claude Sonnet 5.5 --- docs/plan/PLAN-v2.md | 266 +++++++++++++++++++++++ docs/plan/TODO-0-foundation.md | 60 +++++ docs/plan/TODO-agent-1-org-platform.md | 129 +++++++++++ docs/plan/TODO-agent-2-ai-workforce.md | 155 +++++++++++++ docs/plan/TODO-agent-3-delivery-infra.md | 177 +++++++++++++++ 5 files changed, 787 insertions(+) create mode 100644 docs/plan/PLAN-v2.md create mode 100644 docs/plan/TODO-0-foundation.md create mode 100644 docs/plan/TODO-agent-1-org-platform.md create mode 100644 docs/plan/TODO-agent-2-ai-workforce.md create mode 100644 docs/plan/TODO-agent-3-delivery-infra.md diff --git a/docs/plan/PLAN-v2.md b/docs/plan/PLAN-v2.md new file mode 100644 index 0000000..ac94849 --- /dev/null +++ b/docs/plan/PLAN-v2.md @@ -0,0 +1,266 @@ +# Smart Org v2 — Plan (shared by all agents) + +Date: 2026-10-13 · Base branch: `dev` · Read this file fully before your own TODO file. + +This file is the **single source of truth** for the v2 work. Three agents work in parallel, each from +its own TODO file: + +| File | Agent | Area | +|---|---|---| +| `docs/plan/TODO-0-foundation.md` | Claude (runs FIRST, before the agents) | shared contracts, DB foundation, wiring | +| `docs/plan/TODO-agent-1-org-platform.md` | Agent 1 | invites, org wizard, app profile, sharing, owner model grants, memory, MCP | +| `docs/plan/TODO-agent-2-ai-workforce.md` | Agent 2 | admin (manager) AI, employees, dynamic pipelines, agent loop, test/commit/push | +| `docs/plan/TODO-agent-3-delivery-infra.md` | Agent 3 | docker sandbox, git workspaces, live CI/CD, demo server, Caddy domains, SSH/FTP deploy | + +--- + +## 1. Product vision (what the owner asked for) + +1. **One platform owner** controls the whole application and its infrastructure: AI accounts, git, + application rules, adding users, domains, sandboxes, models. Nobody else touches infra. +2. An **invited user** lands on an **organization creation wizard**. The wizard connects the org to **our + git (Gitea)** or **the user's own git** and asks for the application type, language, framework, and so on. + **One organization = exactly one application.** A user can create several orgs and can **share** an + org with other people so they help manage it. +3. The wizard also asks (optionally) for **SSH or FTP** details to deploy the app to **production**. + The platform can also run the app inside a **sandbox as a live demo**. +4. The user gives orders to the **admin AI (manager)**. The manager brings in **AI employees**, built + only from the **models the owner grants to that org**. They do the task, **test it and push it**. +5. A **live CI/CD** mechanism shows the deployment on our local **demo sandbox server**, live. +6. **Caddy** routes different domains to the applications. +7. This is the **web version of `/projects/agybot`** (Telegram bot). Reuse agybot's proven ideas, not its code. +8. Every org **improves itself** through **memories**: how its users like to manage the org, and how the + app's logic and code work. It also exposes **MCP** so AI agents can control our app and read or push data. +9. **Main difference from agybot:** users never touch infrastructure or AI accounts. They only give + orders and tasks. Tasks run through **dynamic pipelines**. + +## 2. What already exists (audited 2026-10-13, the code is the truth, not old TODO.md ticks) + +- **Transport:** the API is WebSocket-RPC only. `app/main.py` exposes only `/health*` and `/api/ws`. Every + handler lives in an internal FastAPI app (`app/interfaces/realtime/rpc_app.py`) that is reached + in-process. Realtime push uses channels `org:`, `task:`, `run:`, `thread:`, + `session:`, `user:`, `admin` (`app/application/ports/realtime.py`, authz in + `app/interfaces/realtime/authz.py`). +- **Roles:** platform roles `owner` (exactly one, enforced by a DB partial unique index), `manager` and + `user`. Org roles are `owner > admin > manager > member > viewer`. There is no public signup. Accounts + come from the owner (`/admin/users`) or from an org invite link. +- **Admin console:** agy (Google OAuth) accounts with pool rotation and 429 cooldown, plus model + catalog, routing policies, rate cards, usage and error console. +- **Projects:** `projects` holds only name/slug/description/default_branch. There is no app profile and + no "1 org = 1 app" rule. Gitea adapter (real) and mock. Preview domains through the Caddy adapter + (real) and mock. +- **Sandbox:** full CRUD API, but only Daytona/E2B adapters (untested live) and a mock that runs nothing. +- **Workforce:** templates, employees, threads, tasks, runs. **The run engine makes ONE LLM call per + task and stores the text.** Tool calls are recorded, never executed. No employee edits code, runs + tests, commits or pushes. +- **Missing entirely:** org wizard, app profile, external git, SSH/FTP deploy, CI/CD, demo server, docker + sandbox, memory system, MCP, per-org model grants. +- **Tests:** about 233 API tests (pytest, DB via `Base.metadata.create_all` in `tests/conftest.py`) and 64 + web tests (vitest). Five-step alembic chain ending at `d4e5f6a7b8c9`. + +## 3. What we take from agybot (concepts only, never import it) + +| agybot | v2 equivalent | Owner | +|---|---|---| +| Roles `manager, planner, programmer, reviewer, tester` (+ optional critic) | same role keys | Agent 2 | +| `[[TAG attr="..."]]body[[/TAG]]` protocol, per-role allowed tags | same, in `app/application/engine/protocol.py` | Agent 2 | +| Pipelines `plan_code_test` (default), `plan_code_review_test`, `code_test`, `analysis_only` | same names | Agent 2 | +| Pipeline choice: manager's `pipeline=` > task-kind default > app default; review pipeline needs an active reviewer | same | Agent 2 | +| Task statuses `pending_manager_review → pending_analysis → pending_implementation → pending_review → pending_test → pending_final_review → merge_request → completed` (+ `pending_user_decision`, `error_review`, `paused`, `cancelled`, `failed`) | same, with optimistic `expected_from` | Agent 2 | +| Gates: real test run after WORK_REPORT, green gate (waiving needs a written reason), diff guard, no-progress stop (2 identical diffs), max cycles, transient-error retries, escalate model on red | same | Agent 2 | +| Worktree per task `task/`, merge into `dev`, push | same, through `RepoWorkspacePort` | Agent 3 (port), Agent 2 (use) | +| Manager `HIRE/UPDATE_EMPLOYEE/FIRE` (max 3 changes per turn, max 20 employees), only on org-shared models | same, only on **owner-granted** models | Agent 2 | +| Org knowledge `[[KNOWLEDGE]]` (max 3 per turn, caps 1,500 chars/key and 6,000/org), lessons with 30-day TTL, project summaries per task | memory system | Agent 1 | +| `app_control_mcp` / `remote_dev_mcp` | MCP gateway with scoped tokens | Agent 1 | +| `caddy_manager` (single-label hosts under root, localhost targets, validate→revert→reload) | existing `CaddyDomainRouterAdapter` | Agent 3 | +| `ssh_manager.push_project`, `ftp_manager.deploy_project` | deploy adapters | Agent 3 | +| `org_live_stream` (ring buffer, polled) | realtime push events (no polling) | Agents 2 & 3 | + +**agybot bugs we must NOT repeat:** deploy publishing the wrong code silently; `git_publish` stuck +mid-merge; observe mode applying real state changes; a failed/cancelled dependency deadlocking its +dependents; manager chat cost not counted in the budget; git operations without the repo lock; about +870 silent `except Exception` blocks; plaintext credentials; the AI running directly on the host; +`git status --porcelain` hiding new dirs (always use `-uall`). + +## 4. Architecture decisions (fixed, do not re-decide) + +- **D1. 1 org = 1 app.** The app is the existing `projects` row (the table name stays). `projects` gets + `UNIQUE(organization_id)`. The UI calls it "Application" (اپلیکیشن). Route `/projects` becomes `/app`. +- **D2. App profile.** These are columns on `projects`: app_type, language, framework, runtime_version, + package_manager, setup/build/test/start commands, app_port, healthcheck_path, git_source + (`platform|external`), external_repo_url, wizard_completed_at. External git credentials and deploy + targets live in their own tables, **encrypted at rest** with the existing crypto helper. They are never + returned raw to the frontend (only `has_secret: true` and masked values). +- **D3. Owner model grants.** `org_model_grants(org_id, ai_model_id, daily_credit_cap, is_active)`. The + engine may only hire or run on granted models. Users never see model names; the UI shows the + employee's role and a neutral "tier" label. +- **D4. Execution engine = in-process agent loop.** It uses the existing `ModelProviderPort` (owner's agy + accounts, pool rotation) with the agybot tag protocol for actions **and** tools. Tools run **inside the + org's sandbox container** through `SandboxProviderPort`. There are no agent CLIs in v1, so AI + credentials never leave the server and users never see them. +- **D5. Sandbox = local Docker** (`SANDBOX_PROVIDER=docker`, the binary is configurable so podman works + too). Per app there is one **workspace container** (employees work in it) and one **demo container** + (runs the latest green build). Hardened: non-root, `--cap-drop ALL`, `no-new-privileges`, cpu/mem/pids + limits, ports published on `127.0.0.1` only. +- **D6. Git is done by the platform, never by the AI.** `RepoWorkspacePort` runs git **on the host** with + a gitdir that is **never mounted** into the container. Git is always called with explicit + `--git-dir/--work-tree`, hooks disabled (`core.hooksPath=/dev/null`), `-uall`, `core.quotePath=false`, + and a per-repo lock. Remote credentials go through `GIT_ASKPASS` or a temporary key file and **never + appear in a URL, log or exception**. +- **D7. CI/CD is platform-native.** A push to `dev` (by the engine or seen by polling the remote) creates + a `pipeline_run`. Steps: checkout → setup → build → test → package → deploy demo → healthcheck → route. + Logs stream live as realtime events on `org:` and a bounded tail is stored. A failed run keeps the + previous demo running. +- **D8. Production deploy is always human-triggered** (org owner/admin, with confirmation and audit). + SSH: release dir + `current` symlink + post-deploy command + rollback. FTP/FTPS: upload a release, keep + the previous one for rollback. User-supplied hosts are SSRF-checked (no private, loopback or link-local + IPs unless the owner allowlists them). SSH host key is pinned on first successful test (TOFU). +- **D9. Domains.** Each app gets an automatic demo host `.`. Only the + **owner** can attach extra hosts. Targets are always `127.0.0.1:`. The existing rules in + `app/domain/platform/rules.py` stay authoritative. +- **D10. Memory** (`org_memories`) is the org's long-term self-knowledge: rules, preferences, lessons, + facts and the code map, with caps, dedupe, TTL for lessons and pinning. Users can view, edit, pin and + delete it. The engine reads it through `MemoryPort.build_context` and feeds it through + `MemoryPort.on_task_finished`. +- **D11. MCP gateway** is a separate HTTP entry point (`services/mcp-gateway`) using the official `mcp` + Python SDK (streamable HTTP). Bearer tokens are hashed, scoped, org-bound and expire. Every call is + audited and org-enforced. +- **D12. Platform invites.** The owner can create an invite link for a **new platform account**, which + becomes role `manager` and may create orgs. A logged-in manager with no org is sent to the wizard. Org + invites (join an existing org) stay as they are. +- **D13. Automation safety.** Tasks run automatically up to merge into `dev`, followed by a demo deploy. + Nothing goes to production without a human click. No code path may flip that default. + +## 5. Contracts between agents (created by Claude in TODO-0, then FROZEN) + +All contracts live in `apps/api/app/application/ports/` and are `typing.Protocol`s with a +working default implementation, so each agent can test alone. **Agents must not change a contract +file.** If you need a change, write a `CHANGE REQUEST` in your phase report, keep coding against the +current signature, and add a local adapter or shim in your own files. + +```python +# ports/repo_workspace.py — implemented by Agent 3, used by Agent 2 (and Agent 1 for test_remote) +class RepoWorkspacePort(Protocol): + async def ensure_workspace(self, *, project_id: str) -> WorkspaceRef: ... # clone/fetch app repo on host + async def create_task_worktree(self, *, project_id: str, task_id: str, base: str = "dev") -> WorktreeRef: ... + async def diff(self, *, worktree: WorktreeRef) -> DiffResult: ... # text + changed files (-uall) + async def commit(self, *, worktree: WorktreeRef, message: str, author: str) -> str | None: ... # sha or None if clean + async def sync_with_base(self, *, worktree: WorktreeRef) -> MergeOutcome: ... # merge base into task branch + async def merge_into(self, *, project_id: str, branch: str, target: str = "dev") -> MergeOutcome: ... + async def push(self, *, project_id: str, branch: str) -> PushOutcome: ... + async def remove_worktree(self, *, worktree: WorktreeRef) -> None: ... + async def head(self, *, project_id: str, branch: str) -> str | None: ... + async def test_remote(self, *, url: str, credentials: RemoteCredentials) -> RemoteCheck: ... +# WorktreeRef has: project_id, task_id, branch, host_path, sandbox_path (path inside the workspace container) + +# ports/delivery_hooks.py — implemented by Agent 3, called by Agent 1 (app created) and Agent 2 (pushed) +class DeliveryHooks(Protocol): + async def on_app_created(self, *, organization_id: str, project_id: str) -> None: ... + async def on_branch_pushed(self, *, organization_id: str, project_id: str, branch: str, sha: str, source: str) -> None: ... + +# ports/deploy_target.py — implemented by Agent 3, called by Agent 1's wizard +class DeployTargetTester(Protocol): + async def test_connection(self, *, target: DeployTargetSpec) -> DeployCheck: ... # ok, message, fingerprint + +# ports/memory.py — implemented by Agent 1, used by Agent 2 +class MemoryPort(Protocol): + async def build_context(self, *, organization_id: str, role: str, query: str, budget_chars: int = 2500) -> str: ... + async def record(self, *, organization_id: str, kind: str, content: str, source: str, + evidence_task_id: str | None = None, key: str | None = None) -> str | None: ... + async def on_task_finished(self, *, organization_id: str, task: TaskOutcome) -> None: ... + +# ports/model_grants.py — implemented by Agent 1, used by Agent 2 +class ModelGrantPort(Protocol): + async def granted_models(self, *, organization_id: str) -> list[GrantedModel]: ... # id, capabilities, tier, caps + async def is_granted(self, *, organization_id: str, ai_model_id: str) -> bool: ... + +# ports/workforce_bootstrap.py — implemented by Agent 2, called by Agent 1's wizard +class WorkforceBootstrapPort(Protocol): + async def bootstrap_team(self, *, organization_id: str, project_id: str) -> None: ... # creates the manager + default team +``` + +Existing contracts that stay as they are: `SandboxProviderPort` (Agent 3 adds the docker adapter), +`GitProviderPort`, `DomainRouterPort`, `ModelProviderPort`, `EventPublisher`/`RealtimeEvent`. + +**Realtime event names** (use the existing `RealtimeEvent` + `get_publisher()`): + +| Channel | Event types | Producer | +|---|---|---| +| `task:` | `task.status`, `task.step`, `task.gate` | Agent 2 | +| `run:` | `run.log`, `run.tool`, `run.status` | Agent 2 | +| `org:` | `task.created`, `team.changed` | Agent 2 | +| `org:` | `pipeline.queued`, `pipeline.step`, `pipeline.log`, `pipeline.status`, `release.live`, `deploy.status` | Agent 3 | +| `org:` | `memory.changed`, `org.updated` | Agent 1 | + +Log events are **batched** (at most one event per 500 ms per stream, max 200 lines per event). + +## 6. File ownership (no two agents edit the same file) + +| Area | Agent 1 | Agent 2 | Agent 3 | +|---|---|---|---| +| API routers | `app_setup.py`*, `memory.py`*, `mcp_tokens.py`*, `admin_grants.py`*, `organizations.py`, `invites.py`, `admin_users.py`, `auth.py`, `knowledge.py` | `workforce.py` | `projects.py`, `sandbox.py`, `delivery.py`*, `admin_domains.py`* | +| Services | `org_setup_service.py`*, `memory_service.py`*, `model_grant_service.py`*, `mcp_service.py`*, `platform_users.py`, `knowledge_service.py` | `workforce_service.py`, `app/application/engine/**`* | `project_service.py`, `sandbox_service.py`, `delivery_service.py`*, `deploy_service.py`* | +| Adapters | — | — | everything in `app/infrastructure/adapters/` (incl. `registry.py`, `mock.py`) | +| Repositories | `identity.py`, `knowledge.py`, `memory.py`* | `workforce.py` | `projects.py`, `sandbox.py`, `delivery.py`* | +| Domain | `identity/`, `knowledge/`, `memory/`*, `projects/catalog.py`* | `workforce/` | `platform/`, `sandbox/`, `projects/models.py` | +| DB tables | `db/models_org.py`* | `db/models_engine.py`* | `db/models_delivery.py`* | +| Wiring | `container_org.py`* | `container_engine.py`* | `container_delivery.py`* | +| Workers | — | `workers/ai_runtime.py` | `workers/sandbox_gateway.py`, `workers/delivery.py`* | +| Other API | `app/interfaces/mcp/**`*, `services/mcp-gateway/**` | — | — | +| Web pages `apps/web/src/app/` | `onboarding/`, `accept-invite/`, `(app)/settings/`, `(app)/memory/`*, `(app)/knowledge/`, `(app)/admin/users/`, `(app)/admin/model-grants/`*, `(app)/admin/mcp/`*, `(app)/admin/organizations/` | `(app)/manager/`, `(app)/tasks/`, `(app)/workforce/`, `(app)/traces/`, `(app)/dashboard/` | `(app)/app/`*, `(app)/projects/` (delete after moving), `(app)/admin/domains/`*, `(app)/admin/sandbox/`*, `components/sandbox-panel.tsx` | +| Web API client | `lib/api-org.ts`* | `lib/api-engine.ts`* | `lib/api-delivery.ts`* | +| Tests | `tests/test_org_*.py`, `test_memory*.py`, `test_mcp*.py`, `test_model_grants*.py`, `test_invites.py`, `test_platform_hierarchy.py`, web `src/test/org-*.test.tsx` | `tests/test_engine_*.py`, `test_workforce*.py`, `test_workers.py`, web `src/test/engine-*.test.tsx` | `tests/test_docker_*.py`, `test_git_workspace*.py`, `test_delivery*.py`, `test_deploy*.py`, `test_project_repo.py`, `test_domain_isolation.py`, `test_platform_rules.py`, web `src/test/delivery-*.test.tsx` | +| Alembic (one file per agent, more only if chained off your own) | `a1b2c3d4e5f1_org_platform.py` | `a2b3c4d5e6f2_ai_workforce.py` | `a3b4c5d6e7f3_delivery_infra.py` | + +`*` = new file (Claude creates an empty stub in TODO-0, the agent fills it). + +**Locked files (Claude only):** `db/models.py`, `container.py`, `rpc_app.py`, `workers/__main__.py`, +`ports/*.py` contracts, `tests/conftest.py`, `pyproject.toml`, `package.json`, `pnpm-lock.yaml`, +`apps/web/src/lib/api.ts`, `apps/web/src/lib/nav.ts`, `components/AppShell.tsx`, `alembic/env.py`, root +`README.md`/`PLAN.md`/`TODO.md`, `docs/plan/PLAN-v2.md`, every other agent's TODO file. + +**Append-only blocks.** `app/core/config.py` and `.env.example` each have three marked blocks +(`# >>> agent-1` … `# <<< agent-1`, and so on). You may add lines **only inside your own block**. + +Each new migration uses `down_revision = "a0f0e1d2c3b4"` (the foundation). Claude adds the merge +revision when the branches are merged. + +## 7. Working rules for every agent + +1. Work **only** in your own git worktree and branch (given in your TODO file). Never `cd` outside it and + never write to `/tmp`. Put scratch scripts in `./_scratch/` inside your worktree and delete them at the end. +2. **Do NOT commit, push, merge or rebase.** Claude reviews and commits each phase. +3. At the **end of each phase**, stop and write `docs/plan/reports/agent--phase-.md` containing: + what you did, files changed, test output summary, deviations, open questions, and `CHANGE REQUEST`s. + Then wait. +4. Before declaring a phase done, run **all** checks (not only your tests) and paste the summary lines: + ```bash + cd apps/api && .venv/bin/python -m pytest -q && .venv/bin/ruff check . && .venv/bin/mypy app + cd apps/web && pnpm test && pnpm exec tsc --noEmit && pnpm lint + ``` + Every existing test must stay green. Never delete or weaken an existing test to make it pass. If an + existing test encodes behaviour that this plan changes on purpose, update it and explain why in the report. +5. Every multi-tenant query is filtered by `organization_id`. Every destructive or production action has + a confirmation and an audit event. No secret ever goes to the frontend, a log, an exception message, a + realtime event or an LLM prompt. +6. No silent `except Exception: pass`. Catch narrow errors, or record them through + `app.infrastructure.error_log`. +7. Users (non-owner) never see model names, providers, accounts, containers, ports or raw infra + errors. Translate them to plain Persian messages. +8. UI language is Persian and RTL. Reuse `packages/ui` components and the existing motion classes. +9. Match the existing code style: hexagonal layers (domain has no FastAPI/SQLAlchemy imports), type + hints everywhere, docstrings in the same tone as nearby code. +10. Do not run `agy`, do not call real external services in tests (use fakes). Integration tests that need + docker, network or real git remotes must be skipped unless an env flag is set. + +## 8. Integration (Claude, after the agents) + +1. Review each phase report and diff, run the full checks, and commit on the agent's branch. +2. Merge the branches into `dev` one by one: Agent 3 → Agent 1 → Agent 2. Run the full tests after **each** + merge. +3. Add the alembic merge revision for the three heads and run `alembic upgrade head` + `alembic check`. +4. Replace the Null/default contract implementations in `container.py` with the real ones and run the + end-to-end smoke test: + wizard → app created → demo domain → manager order → task runs → tests → push → pipeline → demo live + → production deploy (to a local SSH/FTP test server). +5. Update `README.md`, `TODO.md` and the architecture doc, then commit. diff --git a/docs/plan/TODO-0-foundation.md b/docs/plan/TODO-0-foundation.md new file mode 100644 index 0000000..4014cf0 --- /dev/null +++ b/docs/plan/TODO-0-foundation.md @@ -0,0 +1,60 @@ +# TODO-0 — Foundation (Claude does this BEFORE the three agents start) + +Goal: put every shared or locked file in its final shape, so the three agents never edit the same +file. When this is done, `dev` has a foundation commit and three worktrees exist. + +Read `docs/plan/PLAN-v2.md` §4–§6 first. Everything here is mechanical; no product logic. + +## F1. Environment +- [ ] `pnpm install` at root; `pnpm api:venv`; confirm the baseline: `pytest -q`, `ruff`, `mypy`, `pnpm test`, `tsc`. Record the counts. +- [ ] Add API deps to `apps/api/pyproject.toml`: `asyncssh`, `aioftp`, `mcp` (official SDK). Install and re-run the baseline. + +## F2. DB foundation (`db/models.py` + migration `a0f0e1d2c3b4_v2_foundation.py`, down_revision `d4e5f6a7b8c9`) +- [ ] `projects`: add app-profile columns (all nullable or defaulted): `app_type`, `language`, `framework`, + `runtime_version`, `package_manager`, `setup_command`, `build_command`, `test_command`, `start_command`, + `app_port` (int), `healthcheck_path` (default `/`), `git_source` (`platform|external`, default `platform`), + `external_repo_url`, `wizard_completed_at`. +- [ ] `projects`: `UniqueConstraint("organization_id", name="uq_project_one_per_org")`. Fix the seed and any test that creates 2 projects in one org. +- [ ] New table `project_external_git` (project_id PK/FK, auth_type `token|ssh_key|none`, username, secret_encrypted, known_hosts, updated_at). +- [ ] New table `deploy_targets` (id, organization_id, project_id, kind `ssh|sftp|ftp|ftps`, host, port, username, + auth_type `password|key`, secret_encrypted, remote_path, post_deploy_command, host_fingerprint, is_active, + last_tested_at, last_test_ok, last_test_message, created_at). At most one active target per project. +- [ ] New table `org_model_grants` (id, organization_id, ai_model_id, daily_credit_cap nullable, is_active, created_by, created_at; unique org+model). +- [ ] At the end of `models.py`, import `models_org`, `models_engine`, `models_delivery` (three empty modules with a docstring naming the owner agent). +- [ ] `alembic upgrade head`, `alembic downgrade -1`, `alembic upgrade head`, `alembic check`, all clean. + +## F3. Contracts (`app/application/ports/`), exactly as in PLAN-v2 §5 +- [ ] `repo_workspace.py` (+ dataclasses `WorkspaceRef`, `WorktreeRef`, `DiffResult`, `MergeOutcome`, `PushOutcome`, `RemoteCredentials`, `RemoteCheck`) and an in-memory `FakeRepoWorkspace` for tests. +- [ ] `delivery_hooks.py` + `NullDeliveryHooks` (records calls in a list for tests). +- [ ] `deploy_target.py` (`DeployTargetSpec`, `DeployCheck`) + `FakeDeployTargetTester` (ok unless host contains "fail"). +- [ ] `memory.py` (`TaskOutcome`) + `NullMemory` (empty context, record returns None). +- [ ] `model_grants.py` (`GrantedModel`) + `AllActiveModelsGrants` (today's behaviour: every active model). +- [ ] `workforce_bootstrap.py` + `NullWorkforceBootstrap`. +- [ ] `app/domain/projects/profile.py`: frozen dataclass `AppProfile` + `AppProfile.from_row(...)`. + +## F4. Wiring stubs +- [ ] `app/application/container_org.py`, `container_engine.py`, `container_delivery.py`: each holds the builder functions for its port(s), returning the Null/Fake implementations for now. `container.py` re-exports them. +- [ ] Empty routers (`router = APIRouter()`) registered in `rpc_app.py`: `app_setup`, `memory`, `mcp_tokens`, `admin_grants`, `delivery`, `admin_domains`. +- [ ] `app/workers/delivery.py` stub (one no-op pass) registered in `workers/__main__.py` as `delivery`; `services/delivery-worker/README.md` + `pnpm worker:delivery` script. +- [ ] `services/mcp-gateway/README.md` placeholder + `pnpm mcp:dev` script (points at `app.interfaces.mcp.server:main`, stub module that raises a clear "not implemented yet"). +- [ ] `config.py` and `.env.example`: three marked blocks `# >>> agent-1 … # <<< agent-1` (likewise 2 and 3). Pre-fill the obvious settings: + - agent-3: `SANDBOX_DOCKER_BIN=docker`, `SANDBOX_WORKSPACES_ROOT`, `SANDBOX_DEFAULT_CPUS=1`, `SANDBOX_DEFAULT_MEMORY_MB=1024`, `DEMO_PORT_START=8100`, `DEMO_PORT_END=8999`, `DELIVERY_POLL_SECONDS=60`, `DEPLOY_ALLOW_PRIVATE_HOSTS=false`, `ARTIFACTS_ROOT`. + - agent-2: `ENGINE_MAX_TOOL_CALLS_PER_TURN=40`, `ENGINE_TURN_TIMEOUT_SECONDS=900`, `ENGINE_MAX_CYCLES=6`, `ENGINE_MAX_EMPLOYEES=20`, `ENGINE_TICK_SECONDS=10`. + - agent-1: `MCP_GATEWAY_HOST=127.0.0.1`, `MCP_GATEWAY_PORT=8765`, `MCP_TOKEN_TTL_DAYS=90`, `MEMORY_ORG_CHAR_CAP=60000`, `PLATFORM_INVITE_TTL_HOURS=72`. + +## F5. Web stubs +- [ ] `lib/api-org.ts`, `lib/api-engine.ts`, `lib/api-delivery.ts`: each imports `apiRequest` from `./api` and exports an empty object `orgApi` / `engineApi` / `deliveryApi`. +- [ ] `nav.ts`: rename the projects entry to `{ href: "/app", label: "اپلیکیشن", hint: "دمو، استقرار، Git" }`, rename knowledge to `{ href: "/memory", label: "حافظهٔ سازمان", ... }`; admin entries `/admin/model-grants` (مدل‌های سازمان‌ها), `/admin/domains` (دامنه‌ها), `/admin/sandbox` (سندباکس‌ها), `/admin/mcp` (MCP). +- [ ] Placeholder pages (using `PlaceholderPage`) for `(app)/app`, `(app)/memory`, `(app)/admin/model-grants`, `(app)/admin/domains`, `(app)/admin/sandbox`, `(app)/admin/mcp`. `(app)/projects` keeps working until Agent 3 removes it. + +## F6. Finish +- [ ] Baseline checks are all green with the same counts as F1 (plus any new tests). +- [ ] Commit on `dev`: `v2 foundation: shared contracts, app profile schema, wiring stubs`. +- [ ] Create the worktrees from that commit: + ```bash + git worktree add ../smart-org-wt/agent1 -b agent1/org-platform + git worktree add ../smart-org-wt/agent2 -b agent2/ai-workforce + git worktree add ../smart-org-wt/agent3 -b agent3/delivery-infra + ``` + In each one: `pnpm install` and `pnpm api:venv` so the agent can run the checks immediately. +- [ ] `mkdir docs/plan/reports` (tracked with a `.gitkeep`). diff --git a/docs/plan/TODO-agent-1-org-platform.md b/docs/plan/TODO-agent-1-org-platform.md new file mode 100644 index 0000000..2ed3967 --- /dev/null +++ b/docs/plan/TODO-agent-1-org-platform.md @@ -0,0 +1,129 @@ +# TODO — Agent 1: Org Platform (invites, wizard, app profile, sharing, model grants, memory, MCP) + +- **Worktree:** `/projects/smart-org-wt/agent1` · **Branch:** `agent1/org-platform` (already created, already installed) +- **Read first:** `docs/plan/PLAN-v2.md` (all of it). Then skim: `apps/api/app/interfaces/http/routers/organizations.py`, + `invites.py`, `admin_users.py`, `app/application/services/platform_users.py`, `app/domain/identity/models.py`, + `apps/web/src/app/onboarding/page.tsx`, `apps/web/src/app/(app)/settings/`. +- **Your files:** only the "Agent 1" column of PLAN-v2 §6, your block in `config.py` / `.env.example`, and the + migration `alembic/versions/a1b2c3d4e5f1_org_platform.py` (`down_revision = "a0f0e1d2c3b4"`). +- **Never touch:** the locked files and the other agents' files (PLAN-v2 §6). Need something there? → `CHANGE REQUEST` in your report. +- **Do NOT commit, push, merge or rebase.** Never `cd` outside the worktree, never write to `/tmp`. +- **After each phase:** run all checks (PLAN-v2 §7.4), write `docs/plan/reports/agent-1-phase-.md`, then STOP and wait. + +You consume these contracts (do not change them): `DeployTargetTester`, `RepoWorkspacePort.test_remote`, +`DeliveryHooks.on_app_created`, `WorkforceBootstrapPort.bootstrap_team`. Get them only from the builders in +`app/application/container.py`. In tests use the Fake/Null versions from the ports package. + +You implement these contracts: `MemoryPort`, `ModelGrantPort`. Your real classes are wired in +`container_org.py`. + +--- + +## Phase 1 — Platform invites, roles and org sharing + +Backend +- [ ] 1.1 New table `platform_invites` (in `db/models_org.py`): id, token_hash, email (optional), created_by (owner), role (`manager` only for now), expires_at (`PLATFORM_INVITE_TTL_HOURS`), used_at, used_by_user_id, revoked_at. Migration in your file. +- [ ] 1.2 Owner-only endpoints in `admin_users.py`: `POST /admin/platform-invites` (returns `invite_url` once, token shown once), `GET /admin/platform-invites`, `POST /admin/platform-invites/{id}/revoke`. Audit every action. +- [ ] 1.3 In `invites.py`, extend `GET /invites/{token}` and `POST /invites/accept` to recognise a platform invite: if no account exists, create a `manager` account with the chosen password; if the email exists, require login (same rule as org invites: an invite never resets a password). Single use, constant-time compare, expired or revoked → the same generic 404. +- [ ] 1.4 `GET /auth/me` (you own `auth.py`) gains `needs_onboarding: bool` = platform role is `manager`, the user has zero org memberships, and they are not the owner. +- [ ] 1.5 Org sharing in `organizations.py`: `POST /organizations/{id}/transfer-ownership` (owner → an existing admin member, in one transaction, audited), `POST /organizations/{id}/leave` (the last owner cannot leave), and `GET /organizations/{id}/invites` + `DELETE .../invites/{invite_id}` (list and revoke pending org invites). Keep the role ladder `owner > admin > manager > member > viewer`. +- [ ] 1.6 Tests: `tests/test_org_platform_invites.py`, `tests/test_org_sharing.py`. Cover: non-owner cannot create platform invites; token single use; expired/revoked; existing-email path; transfer + leave edge cases; tenant isolation (a member of org A cannot list org B invites). + +Frontend +- [ ] 1.7 `/admin/users`: a "دعوت مدیر جدید" button → modal → shows the link once with a copy button; a list of pending invites with revoke. +- [ ] 1.8 `/accept-invite`: handle the platform-invite variant (title "ساخت حساب مدیر"). +- [ ] 1.9 After login: if `needs_onboarding`, redirect to `/onboarding`. Do this in the `onboarding` page and in a small hook in `lib/api-org.ts`. `AppShell` is locked, so if a redirect there is unavoidable, write a CHANGE REQUEST. +- [ ] 1.10 `(app)/settings`: a "اشتراک‌گذاری سازمان" section: members with role selector, invite form (email + role), pending invites with revoke, transfer ownership (confirmation modal that requires typing the org slug), leave org. +- [ ] 1.11 Web tests `src/test/org-sharing.test.tsx`. + +**Phase 1 done when:** the owner creates a link → a new person opens it, sets a password → logs in → lands on `/onboarding`. All checks are green. + +--- + +## Phase 2 — Org creation wizard + app profile (1 org = 1 app) + +Backend +- [ ] 2.1 `app/domain/projects/catalog.py`: pure-Python catalog. `APP_TYPES` (`web_app`, `api`, `static_site`, `bot`, `worker`, `other`) and `LANGUAGES` → frameworks → defaults (runtime_version, package_manager, setup/build/test/start commands, app_port, healthcheck_path, docker base image name). Cover at least: + - node: next, express, nestjs, vite-react + - python: fastapi, django, flask + - php: laravel, plain + - go: plain + - static: html + Add a function `defaults_for(app_type, language, framework) -> ProfileDefaults`. Unit tests for every entry. +- [ ] 2.2 New table `org_setup_drafts` (user_id, step, data_json **without secrets**, updated_at). Secrets typed during the wizard are held only in the final submit request, never in the draft. +- [ ] 2.3 `org_setup_service.py` + router `app_setup.py`: + - `GET /setup/catalog`. + - `GET/PUT /setup/draft`. + - `POST /setup/check-git`: external URL + credentials → `RepoWorkspacePort.test_remote`. Validate the URL **format** first in your own service: `https://` or `ssh://`/`git@` only, no credentials embedded in the URL, no IP-literal localhost/private hosts. The DNS-resolved SSRF check happens inside `test_remote` (Agent 3); do not import Agent 3's code. + - `POST /setup/check-deploy`: → `DeployTargetTester.test_connection`. + - `POST /setup/complete`: in ONE transaction, create the org (caller = org owner), the project with its app profile (`wizard_completed_at=now`), the encrypted external git credentials or the platform Gitea repo through the existing `GitProviderPort.ensure_project_repo`, the optional deploy target (encrypted), and the default model grants (Phase 3; until then, none). After commit, call `WorkforceBootstrapPort.bootstrap_team` and `DeliveryHooks.on_app_created`. Each hook is called once; a hook failure is recorded with `error_log` and does not undo the org. + - Slug is unique (reuse the existing slug rules). Audit `org.created_via_wizard`. +- [ ] 2.4 App profile editing for org owner/admin: `GET/PATCH /organizations/{org}/app/profile` (commands, port, healthcheck, framework), `PUT /organizations/{org}/app/deploy-target` (+ test), `PUT /organizations/{org}/app/git` (switch platform ↔ external is NOT allowed after creation; only credential rotation). Secrets are write-only: responses return `has_secret` and masked usernames only. +- [ ] 2.5 Enforce 1 org = 1 app: any other path that creates a second project returns a clear 409 (the DB constraint exists already; make the service error friendly). Note: `projects.py` is Agent 3's file. Do NOT edit it; only your service/router may create the app. +- [ ] 2.6 Tests `tests/test_org_wizard.py`: happy path platform git; happy path external git with a fake `test_remote`; a deploy target with a failing test does not block completion (it is saved as `last_test_ok=false`) but the user sees the warning; secrets never in any response, draft, audit metadata or log (assert by searching the captured output and DB rows for the plaintext secret); a second app → 409; a user with role `user` cannot complete the wizard; SSRF cases for the git URL. + +Frontend +- [ ] 2.7 Rebuild `/onboarding` as a 5-step wizard (stepper, back/next, draft auto-save, refresh resumes): + 1. Organization: name, slug (auto-suggest). + 2. Application: name, type, language, framework (cards from the catalog), "advanced" accordion with prefilled commands, port and healthcheck. + 3. Git: "Git پلتفرم (پیشنهادی)" or "Git خودم" (URL + token or SSH key + "بررسی اتصال" button). + 4. Production (optional, skippable): SSH / SFTP / FTP / FTPS fields + "تست اتصال". Show a clear note that production deploys only happen when they press the button. + 5. Review → create → success screen with the demo URL placeholder and a "برو به مدیر AI" button. +- [ ] 2.8 `(app)/settings` → "اپلیکیشن" tab: edit the profile, rotate git credentials, edit or test the deploy target. +- [ ] 2.9 Web tests `src/test/org-wizard.test.tsx` (step navigation, validation, secrets never placed in the draft payload). + +**Phase 2 done when:** a fresh manager completes the wizard in the browser and lands on the dashboard of the new org. The DB has exactly one project with a full profile. The Null hooks recorded one `bootstrap_team` and one `on_app_created` call. + +--- + +## Phase 3 — Owner model grants (employees only use what the owner provides) + +- [ ] 3.1 `model_grant_service.py` implementing `ModelGrantPort` over `org_model_grants` + `ai_models` (only active models, only active grants). `GrantedModel.tier` is a neutral label derived from capabilities (`سریع`, `متعادل`, `قوی`). Never the model name. +- [ ] 3.2 Owner endpoints `admin_grants.py`: list grants per org, grant/revoke model, set daily credit cap, and "default grant set for new orgs" (new table `default_model_grants`, or a column in your own table; do NOT edit `ai_models`). Audit all. +- [ ] 3.3 Wizard `complete` copies the default grant set into the new org. +- [ ] 3.4 Owner org policy (`org_policies` table, yours): max_employees (≤ `ENGINE_MAX_EMPLOYEES`), monthly credit budget, automation_enabled (bool). Endpoints under `admin_grants.py`. Expose `get_org_policy(org_id)` in `model_grant_service.py` for Agent 2 to read later. Write a CHANGE REQUEST so Claude adds it to the `ModelGrantPort` contract at integration. +- [ ] 3.5 Wire your real `ModelGrantPort` in `container_org.py`. +- [ ] 3.6 UI `/admin/model-grants`: org list → grant matrix (models × checkbox + cap), default set editor, policy editor. Org users see nothing new. +- [ ] 3.7 Tests `tests/test_model_grants.py`: non-owner forbidden everywhere; revoked or inactive model not returned; a grant on an org you query for another org is never returned (isolation); defaults copied on wizard completion. + +--- + +## Phase 4 — Org memory (self-improving org) + +- [ ] 4.1 Domain `app/domain/memory/`: `MemoryKind` (`rule`, `preference`, `lesson`, `fact`, `code_map`), `MemorySource` (`user`, `manager`, `system`, `reflection`), validation (key ≤ 80 chars, content ≤ 1,500 chars), normalisation + content hash for dedupe. +- [ ] 4.2 Table `org_memories` (in `models_org.py`): id, organization_id, kind, key (nullable), content, source, evidence_task_id, pinned, confidence (0..1), expires_at (lessons: +30 days unless pinned or confirmed), hits, last_used_at, created_by_user_id, created_at, updated_at, deleted_at (soft). Index `(organization_id, kind)`. +- [ ] 4.3 `memory_service.py` implementing `MemoryPort`: + - `record`: dedupe by hash or same key (update instead of insert), org cap `MEMORY_ORG_CHAR_CAP` (evict oldest unpinned lessons first), max 3 writes per task. + - `build_context(role, query, budget)`: pinned rules and preferences first, then the top memories by keyword overlap with `query` and recency. Role filter: `code_map` and `fact` only for planner/programmer/reviewer/tester; preferences for the manager. Returns a compact Persian/English block within `budget_chars`; updates `hits/last_used_at`. + - `on_task_finished(TaskOutcome)`: one cheap model call (through the existing ai_control routing in "cheap" mode and `ModelProviderPort`, no new adapter) asking for ≤ 2 lessons + a 1-paragraph code-map delta. It stores them with `source=reflection`, `confidence=0.5`. If the call fails, it records the error and returns (it never raises into the engine). Record usage for that call in the usage ledger like other model calls. + - Expire job: function `expire_memories()` (called from the existing billing worker? NO, that is not your file → expose the function and write a CHANGE REQUEST for a worker hook). +- [ ] 4.4 Router `memory.py`: list (filter by kind, search), create (users can "teach" a rule or preference), edit, pin/unpin, delete (soft, audited), and a `confirm` action (lesson → confidence 1, no expiry). Roles: viewer can read; member+ can add preferences; admin+ can edit or delete anything. +- [ ] 4.5 Fold the existing knowledge sources into the same page: keep `knowledge.py` endpoints working and show sources as a second tab. +- [ ] 4.6 UI `/memory` (replace the placeholder; also redirect `/knowledge` → `/memory`): tabs "قوانین و ترجیحات", "درس‌های آموخته", "نقشهٔ کد", "منابع دانش"; inline edit; pin; confirm a lesson; "به سازمان یاد بده" input. Publish `memory.changed` on `org:` and refresh live. +- [ ] 4.7 Wire the real `MemoryPort` in `container_org.py`. +- [ ] 4.8 Tests `tests/test_memory.py`: dedupe, caps and eviction, TTL, role filtering, budget respected, isolation, reflection failure does not raise, no secrets: run content through `app/core/redact.py` before storing and test it. + +--- + +## Phase 5 — MCP gateway (AI agents control our app) + +- [ ] 5.1 Table `mcp_tokens` (id, organization_id, name, token_hash, prefix (first 6 chars for display), scopes CSV, created_by, expires_at, last_used_at, revoked_at). Tokens look like `smo_` and are shown once. +- [ ] 5.2 `mcp_service.py`: issue, list, revoke, verify (constant-time; expired or revoked → reject), scope check. +- [ ] 5.3 `app/interfaces/mcp/server.py` with the official `mcp` SDK (streamable HTTP, bound to `MCP_GATEWAY_HOST:MCP_GATEWAY_PORT`) and `main()` for `pnpm mcp:dev`. Tools (each resolves the org from the token, never from arguments): + - `app_get_profile` (`app:read`) + - `tasks_list` / `tasks_get` (`tasks:read`) + - `tasks_create` (`tasks:write`): goes to the manager as a user order through the **current** public `WorkforceService.create_task` signature; read it, do not edit it (Agent 2 owns it; Claude re-wires at integration if it changes) + - `memory_search` (`memory:read`) / `memory_write` (`memory:write`, kinds `fact|preference` only, `source=user`) + - `knowledge_search` (`memory:read`) + - `pipeline_status` (`delivery:read`): register the tool but return `{"available": false}`. Agent 3's tables do not exist in your branch; Claude wires it at integration + Every call is audited (`mcp.tool_call`, token prefix, tool, ok/error), rate limited (60/min per token, in-memory), and its output passes through `redact.py`. +- [ ] 5.4 `services/mcp-gateway/README.md`: how to run it, the Caddy host suggestion `mcp.` (Agent 3/owner configures it), and a client config snippet. +- [ ] 5.5 Router `mcp_tokens.py`: org admin+ can create, list and revoke org tokens. Owner endpoint to disable MCP globally or per org (setting stored in your `org_policies`). +- [ ] 5.6 UI: `(app)/settings` → "یکپارچه‌سازی (MCP)" tab (create with scope checkboxes, show-once modal with a copy button, list, revoke); `/admin/mcp` (global switch, per-org switch, token counts). +- [ ] 5.7 Tests `tests/test_mcp.py`: use the SDK's in-process client or call the tool functions directly. Cover: a token from org A can never read org B; scope enforcement; revoked/expired; rate limit; audit row written; no secret in outputs. + +--- + +## Final report (after Phase 5) +`docs/plan/reports/agent-1-final.md`: summary, all endpoints added (method + path + role), tables added, the +CHANGE REQUESTs still open, manual QA steps for the wizard and MCP, known limitations. diff --git a/docs/plan/TODO-agent-2-ai-workforce.md b/docs/plan/TODO-agent-2-ai-workforce.md new file mode 100644 index 0000000..f046643 --- /dev/null +++ b/docs/plan/TODO-agent-2-ai-workforce.md @@ -0,0 +1,155 @@ +# TODO — Agent 2: AI Workforce Engine (manager AI, employees, dynamic pipelines, test/commit/push) + +- **Worktree:** `/projects/smart-org-wt/agent2` · **Branch:** `agent2/ai-workforce` (already created, already installed) +- **Read first:** `docs/plan/PLAN-v2.md` (all of it, especially §3 and §5). Then read fully: + `apps/api/app/application/services/workforce_service.py`, `app/domain/workforce/models.py` + `tools.py`, + `app/interfaces/http/routers/workforce.py`, `app/infrastructure/repositories/workforce.py`, + `app/application/ports/model_provider.py`, `app/application/ports/realtime.py`, `app/workers/ai_runtime.py`, + `tests/test_workforce.py`. +- **Reference (concepts only, NEVER import, never edit):** `/projects/agybot/org_manager.py` (lines ~23-101 + roles/pipelines/statuses, `_WORK_TRANSITIONS` ~413) and `/projects/agybot/org_orchestrator.py` + (`ROLE_TAGS`/`MODE_TAGS` ~206-223, `STATUS_ROLE` ~178, `DEFAULT_EFFORT_BY_ROLE` ~292, + `DIRECT_MANAGER_SYSTEM_PROMPT` ~662, `_manager_pipeline` ~7671). Read with grep or by line range; those files are huge. +- **Your files:** only the "Agent 2" column of PLAN-v2 §6, your block in `config.py` / `.env.example`, and the + migration `alembic/versions/a2b3c4d5e6f2_ai_workforce.py` (`down_revision = "a0f0e1d2c3b4"`). + New tables go in `app/infrastructure/db/models_engine.py`. **`db/models.py` is locked**, so new task and + employee fields go into new 1:1 side tables (see 1.3), never into `tasks`/`employees` columns. +- **Never touch:** the locked files and the other agents' files. Need something there? → `CHANGE REQUEST` in your report. +- **Do NOT commit, push, merge or rebase.** Never `cd` outside the worktree, never write to `/tmp`. +- **After each phase:** run all checks (PLAN-v2 §7.4), write `docs/plan/reports/agent-2-phase-.md`, then STOP and wait. + +You consume: `ModelProviderPort` (existing), `ModelGrantPort`, `MemoryPort`, `RepoWorkspacePort`, +`SandboxProviderPort` (existing), `DeliveryHooks.on_branch_pushed`, `EventPublisher`. Get them only from +the builders in `app/application/container.py`. Tests use the Fake/Null versions plus a **scripted fake model** +(you write it in `tests/engine_fakes.py`: returns queued responses in order and records the prompts it got). + +You implement: `WorkforceBootstrapPort` (wired in `container_engine.py`). + +**Important behaviour rules** +- The AI never runs git. It edits files and runs commands inside the sandbox; the platform does git through `RepoWorkspacePort`. +- Every model call writes a usage event exactly as the current `start_run` does (keep that code path). + Manager chat calls count too (agybot forgot this). +- Users never see model names. Store `ai_model_id` internally; APIs for non-owners return `tier` only. +- Nothing in this engine deploys to production. The furthest it goes is "merged and pushed to `dev`". + +--- + +## Phase 1 — Engine core: protocol, state machine, pipelines (pure logic, no I/O) + +Create package `app/application/engine/` (and `app/domain/workforce/` additions where the logic is pure): + +- [ ] 1.1 `domain/workforce/pipeline.py`: + - `Stage` enum: `pending_manager_review`, `pending_analysis`, `pending_implementation`, `pending_review`, `pending_test`, `pending_final_review`, `merge_request`, `completed`, plus side states `pending_user_decision`, `error_review`, `paused`, `cancelled`, `failed`. + - `PIPELINES`: `plan_code_test` (default), `plan_code_review_test`, `code_test`, `analysis_only` (no worktree, no merge). Each is an ordered list of stages. + - `TRANSITIONS: dict[Stage, frozenset[Stage]]` and `next_stage(pipeline, current, outcome)`. + - `STAGE_ROLE`: stage → (role, mode). Manager modes are `triage`, `final`, `error`. + - `resolve_pipeline(manager_choice, task_kind, app_default, has_active_reviewer)`: priority manager > kind > app default > `plan_code_test`. A review pipeline without an active reviewer falls back to `plan_code_test`. + - `TASK_KINDS`: `feature`, `bugfix`, `refactor`, `docs`, `analysis` → default pipeline + risk. + - `RoleKey.DEVELOPER` is our "programmer". Keep the existing enum values; do not rename them. +- [ ] 1.2 `engine/protocol.py`: parser for `[[TAG attr="v" ...]]body[[/TAG]]` and self-closing `[[TAG attr="v"/]]`. Tolerant of whitespace; rejects unknown tags; returns typed dataclasses. Allowed tags per role/mode: + - manager/triage: `ASSIGN`, `REJECT`, `ASK_USER`, `KNOWLEDGE` + - manager/final: `COMPLETE`, `REVISE`, `KNOWLEDGE` + - manager/error: `RESOLVE_ERROR`, `CANCEL`, `KNOWLEDGE` + - manager/chat: `CREATE_TASK`, `PAUSE_TASK`, `RESUME_TASK`, `CANCEL_TASK`, `HIRE`, `UPDATE_EMPLOYEE`, `FIRE`, `KNOWLEDGE` + - planner: `PLAN_SPEC`, `SPLIT`, `ASK_USER` + - developer: `WORK_REPORT` + - reviewer: `REVIEW_PASS`, `REVIEW_CHANGES` + - tester: `TEST_PASS`, `TEST_FAIL` + - tool tags (all working roles): `READ_FILE path`, `WRITE_FILE path` (body = content), `EDIT_FILE path` (body = `<<>>` search/replace blocks), `LIST_DIR path`, `SEARCH pattern path`, `RUN cmd timeout` + Disallowed tags are ignored and reported back to the model as an error line. Limits: max 3 `CREATE_TASK` per reply, max 3 team changes per turn, max 3 `KNOWLEDGE` per turn. +- [ ] 1.3 Tables in `models_engine.py`: + - `task_pipeline_state` (task_id PK/FK, pipeline, kind, risk, stage, cycle, max_cycles, branch, last_diff_hash, same_diff_count, retry_count, waiver_reason, parent_task_id, depends_on JSON, version int for optimistic updates, updated_at) + - `employee_profiles` (employee_id PK/FK, ai_model_id, effort, roles CSV, hired_by `user|manager`, busy_task_id) + - `engine_turns` (id, task_id, run_id, employee_id, role, mode, stage, prompt_chars, reply_excerpt ≤ 4000, tags JSON, tool_calls int, ok, error_kind, started_at, finished_at) + - `gate_results` (id, task_id, gate `tests|diff_guard|no_progress|green|lint`, ok, detail ≤ 8000, created_at) + Map the coarse `tasks.status` from the stage: new / running / waiting_approval / done / failed / cancelled. Existing APIs keep working. +- [ ] 1.4 `engine/state.py`: `advance(task_id, expected_from, to)` with an optimistic check (`version`). It fails cleanly when another worker moved the task first. A failed or cancelled **dependency** moves the dependents to `pending_user_decision`, never leaving them stuck (agybot bug M1). +- [ ] 1.5 Tests `tests/test_engine_protocol.py`, `tests/test_engine_pipeline.py`: every pipeline walks start→completed on the happy path; every illegal transition is refused; concurrent advance (two sessions) → exactly one wins; the dependency failure rule; tag parsing edge cases (nested brackets in a body, unknown tags, the per-role matrix, limits). + +--- + +## Phase 2 — Agent turn runner (one employee, one turn, real tools in the sandbox) + +- [ ] 2.1 `engine/prompt.py`, a prompt builder with a **stable prefix** (cache-friendly), in this order: + 1. platform rules (short, yours) + 2. role protocol (allowed tags + format examples) + 3. toolkit description + 4. app profile (from `AppProfile`: language, framework, commands; never secrets) + 5. `MemoryPort.build_context(org, role, query=task text, budget_chars=2500)` + 6. task brief (≤ 1,200 chars) + stage-specific context (the planner's spec for the developer; the **diff only** for the reviewer; test output for the tester) + 7. the conversation so far in this turn (tool results) + Hard cap on total characters; truncate oldest tool results first. +- [ ] 2.2 `engine/tools.py`: executes tool tags **inside the sandbox** through `SandboxProviderPort`, rooted at `WorktreeRef.sandbox_path`. + - Path safety: normalise; reject absolute paths outside the root, `..` escapes and anything under `.git`. + - `RUN` has a timeout (default 120s, max 600s) and output capped (last 8,000 chars). The command is passed to the sandbox as-is (the sandbox is the security boundary), but `git push`, `git remote` and `git config` are refused with a message ("the platform handles git"). + - `EDIT_FILE` applies search/replace blocks and fails with a clear message when OLD is not found exactly once. + - Each tool call writes a `tool_calls` row (existing table) with status `succeeded`/`failed`, so the existing approvals UI finally sees real executions. +- [ ] 2.3 `engine/runner.py`, `run_turn(task, employee, role, mode) -> TurnResult`. Loop: + - model call → parse tags → execute tools → feed results back → until a terminal protocol tag appears, or `ENGINE_MAX_TOOL_CALLS_PER_TURN`, or `ENGINE_TURN_TIMEOUT_SECONDS`. + - Model = the employee's `ai_model_id`, which must be granted (`ModelGrantPort.is_granted`; if not, fail with `error_kind=auth` and tell the manager). + - Error kinds `rate_limit`, `unavailable`, `network`, `timeout` are **transient** (retry later, up to 2 retries); `auth`, `model`, `protocol` are not. + - Writes `engine_turns`, `run_steps` and usage events, and publishes `run.log` / `run.tool` / `run.status` (batched, PLAN-v2 §5). + - Cancellation: `cancel_run` must stop the loop between steps. +- [ ] 2.4 Tests `tests/test_engine_runner.py` with the scripted fake model + `FakeRepoWorkspace` + a fake sandbox (dict filesystem + scripted command results; put it in `tests/engine_fakes.py`). Cover: the write → run → report loop; path escape refused; `git push` refused; the tool-call cap; timeout; transient vs fatal errors; usage events counted per model call; an ungranted model refused; no secret from settings ever appears in a prompt (assert on the fake model's recorded prompts). + +--- + +## Phase 3 — Manager AI (orders → tasks → team) + +- [ ] 3.1 Manager chat: rewrite `post_user_message` so the manager replies **and** may emit `CREATE_TASK` (title, kind, priority, depends_on, optional pipeline), `PAUSE_TASK` / `RESUME_TASK` / `CANCEL_TASK`, and `HIRE` / `UPDATE_EMPLOYEE` / `FIRE`. Tasks created this way get a `task_pipeline_state`, stage `pending_manager_review`, and are linked to the thread. The manager's prompt must ask for **atomic tasks that do not touch the same files**. Count chat usage. +- [ ] 3.2 Team management: `HIRE role= tier= name=` picks a granted model matching the tier/capabilities (`ModelGrantPort.granted_models`). Respect max employees (setting + later org policy), max 3 changes per turn, and "the last manager cannot be fired". Publish `team.changed`. Users can still hire and fire from the UI (existing endpoints) but only choose **role + tier**, never a model. +- [ ] 3.3 Triage turn (manager, mode triage): decide pipeline + assignee (`ASSIGN employee= pipeline=`), or `REJECT`, or `ASK_USER` (→ `pending_user_decision`, message posted in the thread). Default employee selection: least busy active employee with the needed role; `ASSIGN employee="name"` uses fuzzy name matching inside the org only. +- [ ] 3.4 Final turn (manager, mode final): sees the plan, a diff summary, gate results and the test output → `COMPLETE` (→ `merge_request`) or `REVISE` (→ back to implementation, cycle+1). The error turn (mode error) handles failed gates and errors → `RESOLVE_ERROR` (retry with a hint) or `CANCEL`. +- [ ] 3.5 `KNOWLEDGE` tags → `MemoryPort.record(kind=fact|rule, source=manager)`. +- [ ] 3.6 `WorkforceBootstrapPort.bootstrap_team(org, project)`: creates one manager + one planner + one developer + one tester on the strongest/balanced/fast granted tiers (whatever is granted; if nothing is granted, create the employees "inactive" and post a system message "منتظر تخصیص مدل توسط مالک سامانه"). Idempotent. Wire it in `container_engine.py`. +- [ ] 3.7 Tests `tests/test_engine_manager.py`: an order → ≤ 3 tasks with deps; HIRE limits; FIRE the last manager refused; ASK_USER path; idempotent bootstrap; cross-org isolation (the manager of org A can never assign an employee of org B, even by exact name). + +--- + +## Phase 4 — Pipeline execution, gates, commit, merge, push + +- [ ] 4.1 `engine/scheduler.py`, `tick()`. Each tick: + - finds tasks whose stage needs a turn and whose deps are completed, and a free employee with the role (one task per employee at a time) + - honours org automation (if Agent 1's policy says `automation_enabled=false`, tasks wait in `pending_manager_review`; read it through `ModelGrantPort` when the CHANGE REQUEST lands, until then assume enabled) + - runs turns concurrently with a per-org cap (3) and a global cap (setting) + `workers/ai_runtime.py` calls `tick()` in its loop every `ENGINE_TICK_SECONDS` (keep the one-pass mode for cron). Safe with two worker processes thanks to the optimistic `advance`. +- [ ] 4.2 Worktree lifecycle: on the first developer/planner turn, `RepoWorkspacePort.create_task_worktree(project, task, base="dev")`; the branch is `task/`. `analysis_only` gets none. The worktree is removed after completion, cancellation or failure. +- [ ] 4.3 Gates after `WORK_REPORT` (each writes a `gate_results` row and publishes `task.gate`): + - **diff guard**: `RepoWorkspacePort.diff`. Fail on changes under `.git`, CI files (`.github/`, `.gitlab-ci.yml`, `.gitea/`), symlinks pointing outside the repo, or files > 1 MB. An empty diff for a code task also fails. + - **no-progress**: hash the diff; 2 identical hashes in a row → `error_review`. + - **tests**: run the app profile's `test_command` in the sandbox (timeout 600s), store the tail of the output. Red → back to the developer with the failure tail (one escalation to a stronger granted tier on the 2nd red), counting cycles. Green → tester stage. + - **green gate**: the tester's `TEST_PASS` is accepted only if the tests gate is green, OR `TEST_PASS waive="reason"` with a non-empty reason (stored in `waiver_reason`, shown in UI). + - `max_cycles` reached → `error_review`. +- [ ] 4.4 Commit + merge + push on `merge_request`: + - `commit` (message: `task/: `, author = employee display name + platform email), then `sync_with_base`, then re-run the tests if base changed (a semantic conflict is only caught by retests). + - Then `merge_into(target="dev")` and `push(branch="dev")`. + - A conflict → back to the developer with the conflicting files (max 3 times, then `error_review`). + - On success: stage `completed`, `DeliveryHooks.on_branch_pushed(org, project, "dev", sha, source="engine")`, `MemoryPort.on_task_finished(TaskOutcome)` (never let its failure fail the task). +- [ ] 4.5 Budget: before each turn, check the org credit wallet (existing billing service public functions; read, do not edit) and the per-task cap. Out of budget → `paused` with a Persian reason. +- [ ] 4.6 Tests `tests/test_engine_flow.py`, the end-to-end engine with all fakes. The scripted model plays manager → planner → developer → tester → manager; the fake sandbox makes tests red once and then green. Assert the stage sequence, gate rows, cycles, the commit/merge/push calls on `FakeRepoWorkspace`, `on_branch_pushed` called once with the right sha, `on_task_finished` called once, and usage events. Plus the failure paths: no-progress, max cycles, merge conflict ×3, cancel mid-turn, a dependency failure. + +--- + +## Phase 5 — UI (orders, live task view, team) + +- [ ] 5.1 `lib/api-engine.ts`: typed client functions for everything new (task detail with pipeline state, turns, gates, team with tier, hire/fire with role+tier). +- [ ] 5.2 `/manager`: the chat shows task cards created by the manager inline (title, stage badge, link), live via `org:` + `thread:` channels; an "در انتظار تصمیم شما" banner for `pending_user_decision` with a reply box. +- [ ] 5.3 `/tasks`: a board grouped by stage (columns: بررسی مدیر، تحلیل، پیاده‌سازی، بازبینی، تست، ادغام، انجام‌شده، نیازمند تصمیم) with a live update; filters; pause/resume/cancel (with confirm). +- [ ] 5.4 `/tasks/[id]`: + - a pipeline stepper (stages of its pipeline, current one highlighted) + - the turn timeline (who, role, when, outcome) + - a **live log** panel (subscribe to `run:<id>`, auto-scroll, pause-scroll) + - gate results (green/red with expandable output) + - a diff viewer (read-only, from the latest diff) + - the waiver reason when present + - a link to the pipeline run in `/app` once pushed (just a link with the sha; that page is Agent 3's) +- [ ] 5.5 `/workforce`: employees with role, tier, status, current task; hire (role + tier) and fire with confirm. **No model names anywhere** for non-owners (test it). +- [ ] 5.6 `/dashboard` + `/traces`: show active tasks, a stage distribution and recent completions; keep the existing widgets working. +- [ ] 5.7 Web tests `src/test/engine-*.test.tsx`: board grouping, live log append and batching, no model name rendered (render with a fixture that contains a model name and assert it is absent). + +--- + +## Final report (after Phase 5) +`docs/plan/reports/agent-2-final.md`: summary, the engine architecture (one diagram in text), the tag protocol +reference table, every endpoint added or changed, tables, settings used, CHANGE REQUESTs still open (expected: +org policy in `ModelGrantPort`, the worker hook for memory expiry), manual QA steps, known limitations. diff --git a/docs/plan/TODO-agent-3-delivery-infra.md b/docs/plan/TODO-agent-3-delivery-infra.md new file mode 100644 index 0000000..adfc4f2 --- /dev/null +++ b/docs/plan/TODO-agent-3-delivery-infra.md @@ -0,0 +1,177 @@ +# TODO — Agent 3: Delivery & Infra (docker sandbox, git workspaces, live CI/CD, demo server, Caddy, SSH/FTP) + +- **Worktree:** `/projects/smart-org-wt/agent3` · **Branch:** `agent3/delivery-infra` (already created, already installed) +- **Read first:** `docs/plan/PLAN-v2.md` (all of it, especially D5–D9). Then read fully: + `apps/api/app/application/ports/sandbox_provider.py`, `git_provider.py`, `domain_router.py`, `realtime.py`, + `app/infrastructure/adapters/registry.py`, `mock.py`, `caddy.py`, `gitea.py`, `sandbox_remote.py`, + `app/application/services/sandbox_service.py`, `project_service.py`, `app/domain/platform/rules.py`, + `docs/current-infra-notes.md`, `tests/test_project_repo.py`, `tests/test_domain_isolation.py`. +- **Reference (concepts only, NEVER import, never edit):** `/projects/agybot/caddy_manager.py`, `port_registry.py`, + `ssh_manager.py`, `ftp_manager.py`, `git_manager.py`. +- **Your files:** only the "Agent 3" column of PLAN-v2 §6 (including ALL of `app/infrastructure/adapters/`), your + block in `config.py` / `.env.example`, and the migration `alembic/versions/a3b4c5d6e7f3_delivery_infra.py` + (`down_revision = "a0f0e1d2c3b4"`). New tables go in `app/infrastructure/db/models_delivery.py`. **`db/models.py` + is locked**, so extra data about existing sandbox tables goes into new side tables. +- **Never touch:** the locked files and the other agents' files. Need something there? → `CHANGE REQUEST` in your report. +- **Do NOT commit, push, merge or rebase.** Never `cd` outside the worktree, never write to `/tmp`: tests use pytest's + `tmp_path` (allowed) and the worktree's `./_scratch/`. +- **After each phase:** run all checks (PLAN-v2 §7.4), write `docs/plan/reports/agent-3-phase-<K>.md`, then STOP and wait. + +You implement: `RepoWorkspacePort`, `DeliveryHooks`, `DeployTargetTester`, plus a docker `SandboxProviderPort` +adapter. Wire them in `container_delivery.py` and `registry.py` behind settings (mock remains the default +for tests). + +**Security rules specific to you** +- Nothing runs on the host except: `git` (via the workspace adapter, hardened per D6), the docker/podman CLI, and Caddy reload (existing adapter). +- Every subprocess: `asyncio.create_subprocess_exec` (never `shell=True`), `start_new_session=True`, kill the process group on timeout, drain stdout and stderr concurrently (avoid the 64 KB pipe deadlock), and pass explicit `env` (never inherit secrets). +- Credentials never appear in argv, URLs, logs, exceptions, realtime events or DB columns other than `*_encrypted`. Use `GIT_ASKPASS` with a 0700 helper script that reads a 0600 temp file in the worktree's `./_scratch`-like private dir under `SANDBOX_WORKSPACES_ROOT`, deleted in `finally`. +- Ports and hosts must pass `app/domain/platform/rules.py` (protected ports, single-label hosts, localhost targets). + +--- + +## Phase 1 — Docker sandbox adapter + +- [ ] 1.1 `adapters/docker_sandbox.py`, `DockerSandboxAdapter` implementing `SandboxProviderPort` with the CLI in `SANDBOX_DOCKER_BIN` (docker or podman, the same flags): + - `create_workspace`: creates a host dir `SANDBOX_WORKSPACES_ROOT/<org_id>/<project_id>/work` (0700, owned by the platform user). + - `start_session`: `docker run -d` the **workspace container**: + - image from the app profile language: `node:20-bookworm`, `python:3.13-bookworm`, `php:8.3-cli`, `golang:1.23`, fallback `debian:bookworm`; make the mapping a dict in your config block / domain + - `--user 1000:1000 --cap-drop ALL --security-opt no-new-privileges --pids-limit 512 --cpus $SANDBOX_DEFAULT_CPUS --memory ${MB}m` + - `--read-only` with `--tmpfs /tmp` and `--tmpfs /home/app` + - bind-mount **only the work dir** (never the git dir) to `/workspace` + - label `smart-org.org=<id>`, `smart-org.project=<id>`, `smart-org.role=workspace` + - network: the default bridge (package installs need egress), with no published ports + - `run_command`: `docker exec -w <cwd> <ctr> sh -lc <cmd>`, with timeout, output cap, exit code, and an optional async `on_output(chunk)` callback for streaming (add it as an adapter-level extra method; the port stays unchanged). + - `read_file` / `write_file`: through the host bind mount with path safety (`resolve()` stays inside the work dir; reject symlinks that escape). + - `pause` / `resume` / `destroy`: stop/start/rm. `snapshot`: `docker commit` (tagged per org) or raise a clear "not supported" error, your choice, documented. + - `get_preview_url`: delegated to the domain layer (Phase 4). +- [ ] 1.2 `registry.py`: `SANDBOX_PROVIDER=docker` → this adapter. The mock stays the default. +- [ ] 1.3 The `sandbox_gateway` worker: also reap containers with the `smart-org.*` label that have no live session row (orphans), audited. +- [ ] 1.4 Hide infra from non-owners: + - `sandbox.py` command/file endpoints become **owner-only** (or admin+ of the org with a setting off by default; prefer owner-only). + - Keep the read-only status endpoints for org members. + - Update `components/sandbox-panel.tsx` to show only status to non-owners. + - Update existing tests that assumed member access, and explain each change in your report. +- [ ] 1.5 Tests `tests/test_docker_sandbox.py`: unit tests with a **fake process runner** (inject the subprocess function) that assert the exact argv (hardening flags present, no secrets, mount of the work dir only), timeout kill, output cap, path escape refused. Integration tests marked `@pytest.mark.docker`, skipped unless `SANDBOX_DOCKER_IT=1`: real start → exec `echo` → write/read file → destroy. + +--- + +## Phase 2 — Git workspaces (`RepoWorkspacePort`, the platform does git, D6) + +- [ ] 2.1 `adapters/git_workspace.py`, `LocalGitWorkspaceAdapter`. Layout per project under `SANDBOX_WORKSPACES_ROOT/<org>/<project>/`: + - `git/`: a bare-ish gitdir, **never mounted** + - `work/`: the main working tree for the dev branch, mounted at `/workspace` + - `work/.worktrees/<task_id>/`: task worktrees, visible in the container at `/workspace/.worktrees/<task_id>` + - `WorktreeRef.sandbox_path` = that in-container path + Note: the `.worktrees` dir must be excluded via `info/exclude` in the gitdir, not the user's `.gitignore`. +- [ ] 2.2 Every git call is `git --git-dir=<gitdir> --work-tree=<path> -c core.hooksPath=/dev/null -c core.fsmonitor=false -c core.quotePath=false -c protocol.file.allow=never -c safe.directory=*`, with env `GIT_CONFIG_NOSYSTEM=1`, `HOME=<private dir>`, `GIT_TERMINAL_PROMPT=0`. Never trust the `.git` *file* inside a worktree; always pass explicit dirs. + - Status: `status --porcelain=v1 -uall -z`. + - One `asyncio.Lock` per project id held for every mutating operation (agybot M14). Remove a stale `index.lock` only if older than 30s. +- [ ] 2.3 Remotes: + - **platform** git: the URL and credentials come from the existing Gitea adapter / `project_git_credentials` (decrypt via the existing crypto helper). + - **external** git: `project_external_git` (token → `GIT_ASKPASS`; ssh_key → temp key file 0600 + `GIT_SSH_COMMAND="ssh -i <key> -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=<per-project known_hosts>"`; store `known_hosts` on the first `test_remote` = TOFU, in the existing column). + - `test_remote`: `git ls-remote --heads` with a 20s timeout; returns ok, default branch, branch list, and a friendly error (auth failed / not found / unreachable) **without echoing the URL credentials or stderr secrets**. + - SSRF check on external hosts (resolve DNS, reject private, loopback and link-local unless `DEPLOY_ALLOW_PRIVATE_HOSTS`); put the check in `domain/platform/rules.py` (yours). It runs inside `test_remote` and before every deploy connection; Agent 1 only does format validation and relies on your check. +- [ ] 2.4 Implement every `RepoWorkspacePort` method per PLAN-v2 §5. + - `ensure_workspace` clones on first use, fetches afterwards. It creates `dev` from the default branch if missing; for platform repos also push `dev`. + - `merge_into` does a `--no-ff` merge in the main work tree under the lock and returns the conflict file list on failure, after `merge --abort`. Never leave a half-merge (agybot H3). + - `push` returns ok/rejected (non-fast-forward → fetch + report, never force). +- [ ] 2.5 Wire it in `container_delivery.py` (setting `GIT_WORKSPACE=local|fake`, fake default in tests). +- [ ] 2.6 Tests `tests/test_git_workspace.py` with **real git** and a local bare repo in `tmp_path` as "remote" (no network): + - the full flow ensure → worktree → write → diff (new dir appears thanks to `-uall`) → commit → sync → merge → push → head + - a conflict path leaves a clean tree + - hooks are NOT executed (plant a `post-commit` hook that writes a file and assert it is absent) + - a `.git` file tampered to point elsewhere is ignored + - credentials are never in argv (capture the runner calls) + - concurrent merges are serialised + +--- + +## Phase 3 — Live CI/CD + demo server + +- [ ] 3.1 Tables (`models_delivery.py`): + - `pipeline_runs` (id, org, project, trigger `engine|poll|manual`, branch, sha, status `queued|running|succeeded|failed|cancelled|superseded`, started/finished, error_summary) + - `pipeline_steps` (id, run_id, index, name, status, started/finished, exit_code, log_tail ≤ 64 KB) + - `releases` (id, org, project, run_id, sha, artifact_path, created_at, is_demo_live) + - `app_runtimes` (project_id PK, demo_container_id, host_port, demo_host, status, current_release_id, updated_at) +- [ ] 3.2 `delivery_service.py` implementing `DeliveryHooks`: + - `on_app_created`: allocate a demo port in `DEMO_PORT_START..END` (reuse the existing preview port rules), claim the demo host `<org-slug>.<PREVIEW_ROOT_DOMAIN>` through the existing `DomainRouterPort`, create `app_runtimes`, `ensure_workspace`, and queue a first pipeline run if the repo has a `dev` head. + - `on_branch_pushed`: queue a run (one active run per project; a newer queued run supersedes an older queued one). +- [ ] 3.3 `workers/delivery.py`: loop (`--loop`) that executes queued runs and polls remotes every `DELIVERY_POLL_SECONDS` (`head` vs the last run sha) for pushes made outside the engine (users pushing to their own git). + - Steps, each in a **fresh build container** (same image mapping and hardening, work dir = a clean checkout of the sha in a per-run dir): + 1. `checkout` + 2. `setup` (setup_command) + 3. `build` (build_command, skipped if empty) + 4. `test` (test_command, skipped if empty) + 5. `package` (tar.gz of the run dir excluding `.git` and `node_modules` → `ARTIFACTS_ROOT/<org>/<project>/<sha>.tar.gz`) + 6. `deploy_demo`: start the new demo container with `start_command`, port mapping `127.0.0.1:<host_port>:<app_port>`, the same hardening, label `role=demo` + 7. `healthcheck`: HTTP GET `http://127.0.0.1:<host_port><healthcheck_path>` with 20 retries × 3s; any 2xx/3xx = ok + 8. `route`: `configure_reverse_proxy` for the demo host → `127.0.0.1:<host_port>` + - **Blue/green:** start the new container on a temporary second port, healthcheck it, then switch the Caddy route, then stop the old one. If any step fails, the old demo keeps running untouched and the run is `failed` with a Persian `error_summary`. +- [ ] 3.4 Live logs: every step streams output via the `on_output` callback → batched `pipeline.log` events on `org:<id>` (≤ 1 event / 500 ms, ≤ 200 lines), plus `pipeline.step` and `pipeline.status`; `release.live` when switched. Store the step `log_tail` (last 64 KB). Pass all output through `app/core/redact.py` before publishing or storing. +- [ ] 3.5 Router `delivery.py` (org member read, admin+ actions): list runs, run detail with steps and tails, `POST /delivery/runs` (manual re-run of the dev head, admin+), cancel the queued/running run, list releases, rollback the demo to a previous release (admin+, confirm, audit), and the runtime status (demo URL, live release, health). +- [ ] 3.6 Tests `tests/test_delivery.py` with the fake runner + `FakeRepoWorkspace` + mock domain router: + - the happy path step sequence and events + - a failed build keeps the old demo + - healthcheck failure → no route switch + - supersede logic + - poll detects an external push + - logs redacted (plant a secret in the build output) + - tenant isolation on every endpoint + - an optional `@pytest.mark.docker` end-to-end with a tiny static app (`python -m http.server`) + +--- + +## Phase 4 — Domains (Caddy) owned by the platform owner + +- [ ] 4.1 `admin_domains.py` (owner only): + - list all routes (host → org/app → target port → status) + - attach an extra host to an org's app: a single label under the root via the existing rules, OR a full custom FQDN when `ALLOW_CUSTOM_DOMAINS=true` (strict FQDN regex, no wildcard, no protected hosts such as `git`, `mcp`, `panel`, `api`, root) + - detach (confirm + audit) + - re-sync all routes from the DB to Caddy (idempotent) + Users cannot add domains (the existing member-facing preview-domain endpoints in `projects.py` become owner-only, and the tests are updated with reasons). +- [ ] 4.2 The Caddy adapter keeps validate → revert → reload. Add a `routes_fingerprint` check so re-sync does nothing when nothing changed. +- [ ] 4.3 The demo host is created automatically in `on_app_created` (Phase 3). Record the MCP gateway host suggestion (`mcp.<root>` → `MCP_GATEWAY_PORT`) as a protected system route that the owner can enable from this page. +- [ ] 4.4 Tests `tests/test_admin_domains.py`: non-owner forbidden; protected hosts refused; invalid FQDNs refused; detach audited; re-sync idempotent (mock router call count). + +--- + +## Phase 5 — Production deploy over SSH / SFTP / FTP / FTPS (always human-triggered) + +- [ ] 5.1 `adapters/deploy_ssh.py` (asyncssh), `DeployTargetTester` + `SshDeployer`: + - Test: connect (15s timeout) → host-key TOFU (store the fingerprint on the first success; afterwards a mismatch = hard fail "کلید میزبان تغییر کرده"), check `remote_path` exists and is writable. + - Deploy: upload the release artifact → `remote_path/releases/<ts>-<sha7>/` → extract → run `post_deploy_command` with `cwd` there (timeout 600s, streamed output) → atomically switch the `remote_path/current` symlink → keep the last 5 releases. + - Rollback = re-point `current` to the previous release + rerun post_deploy. +- [ ] 5.2 `adapters/deploy_ftp.py` (aioftp; FTPS via explicit TLS), `DeployTargetTester` + `FtpDeployer`: + - Test: login, cwd `remote_path`, write and delete a probe file. + - Deploy: upload the build output (the build dir is set by the framework catalog default or `dist/`, `build/`, `public/`, else the whole artifact) into `remote_path` after downloading a **backup listing** of the files it overwrites into `ARTIFACTS_ROOT/.../ftp-backup-<ts>.tar.gz`. + - Rollback = re-upload that backup. Document the limits of FTP (no atomic switch). +- [ ] 5.3 SSRF check for every target host (the Phase 2.3 helper). A combined `DeployTargetTester` dispatching on `kind`, wired in `container_delivery.py`. +- [ ] 5.4 `deploy_service.py` + endpoints in `delivery.py`: + - `POST /delivery/production/deploy` with `release_id` (default: the live demo release). Org **owner/admin only**; requires a `confirm: "<org-slug>"` field; one production deploy per project at a time. + - Status: `deploy.status` events and a `production_deploys` table (id, release, target, status, started/finished, log_tail, rolled_back_from). + - `POST /delivery/production/rollback`. + - Every action audited. Credentials decrypted only inside the adapter call. +- [ ] 5.5 Tests `tests/test_deploy.py`: fake SSH/FTP clients injected. Cover: the release/symlink sequence, keep-5 pruning, rollback, host-key mismatch refused, SSRF refused, a member (non-admin) refused, the confirm slug mismatch refused, no password in logs/events/DB columns other than encrypted ones. Optional integration (`DEPLOY_IT=1`) against a local `sshd`/`pure-ftpd` container. + +--- + +## Phase 6 — UI: the "Application" page (replaces `/projects`) + +- [ ] 6.1 `(app)/app/page.tsx` (+ components under `(app)/app/_components/`): + - **Header:** app name, type/language/framework badges, demo URL (open button), live release sha + time, health dot. + - **Pipeline runs:** list with status; clicking one opens the run view: a step list with status and duration, and a **live log console** (subscribe `org:<id>`, filter by run id, ANSI colours stripped, auto-scroll with pause, and a "copy" button). + - **Releases:** history; "rollback demo" (admin+, confirm). + - **Production:** target summary (kind, host, path, last test result; no secrets), the "استقرار در production" button (admin+, modal requiring the org slug), a deploy live log, production history + rollback. + - **Git:** repo link (platform: Gitea URL; external: the URL without credentials), `dev` head sha, last poll time. No branch/PR/port controls for users. +- [ ] 6.2 Remove `(app)/projects` (redirect `/projects*` → `/app`) and drop the member-facing port/branch/PR/command controls. Owner-only debug tools move to `/admin/sandbox`. +- [ ] 6.3 `/admin/sandbox` (owner): running containers per org (workspace/demo/build), CPU/memory limits per org (existing `sandbox_resource_limits`), stop/restart a container (confirm + audit), and an image mapping view. `/admin/domains` (owner): the Phase 4 UI. +- [ ] 6.4 `lib/api-delivery.ts`: typed client for all of the above. +- [ ] 6.5 Web tests `src/test/delivery-*.test.tsx`: the log console batching and auto-scroll, the confirm-slug modal, no secret fields rendered, non-admin sees no deploy button. + +--- + +## Final report (after Phase 6) +`docs/plan/reports/agent-3-final.md`: summary, host requirements (docker/podman, Caddy, dirs and permissions, +the sudo/systemd needs of the Caddy reload), every setting, every endpoint added or changed (method + path + role), +tables, CHANGE REQUESTs still open, manual QA script (wizard app → push → watch live pipeline → open demo URL → +deploy to a local SSH test server → rollback), known limitations.