docs: v2 plan (shared plan + foundation + 3 agent TODO files)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
mjalmoudy94
co-authored by
Claude Sonnet 5.5
Claude Sonnet 5.5
parent
8eb631aedf
commit
3a68b0fa07
@@ -0,0 +1,266 @@
|
||||
# Smart Org v2 — Plan (shared by all agents)
|
||||
|
||||
Date: 2026-10-13 · Base branch: `dev` · Read this file fully before your own TODO file.
|
||||
|
||||
This file is the **single source of truth** for the v2 work. Three agents work in parallel, each from
|
||||
its own TODO file:
|
||||
|
||||
| File | Agent | Area |
|
||||
|---|---|---|
|
||||
| `docs/plan/TODO-0-foundation.md` | Claude (runs FIRST, before the agents) | shared contracts, DB foundation, wiring |
|
||||
| `docs/plan/TODO-agent-1-org-platform.md` | Agent 1 | invites, org wizard, app profile, sharing, owner model grants, memory, MCP |
|
||||
| `docs/plan/TODO-agent-2-ai-workforce.md` | Agent 2 | admin (manager) AI, employees, dynamic pipelines, agent loop, test/commit/push |
|
||||
| `docs/plan/TODO-agent-3-delivery-infra.md` | Agent 3 | docker sandbox, git workspaces, live CI/CD, demo server, Caddy domains, SSH/FTP deploy |
|
||||
|
||||
---
|
||||
|
||||
## 1. Product vision (what the owner asked for)
|
||||
|
||||
1. **One platform owner** controls the whole application and its infrastructure: AI accounts, git,
|
||||
application rules, adding users, domains, sandboxes, models. Nobody else touches infra.
|
||||
2. An **invited user** lands on an **organization creation wizard**. The wizard connects the org to **our
|
||||
git (Gitea)** or **the user's own git** and asks for the application type, language, framework, and so on.
|
||||
**One organization = exactly one application.** A user can create several orgs and can **share** an
|
||||
org with other people so they help manage it.
|
||||
3. The wizard also asks (optionally) for **SSH or FTP** details to deploy the app to **production**.
|
||||
The platform can also run the app inside a **sandbox as a live demo**.
|
||||
4. The user gives orders to the **admin AI (manager)**. The manager brings in **AI employees**, built
|
||||
only from the **models the owner grants to that org**. They do the task, **test it and push it**.
|
||||
5. A **live CI/CD** mechanism shows the deployment on our local **demo sandbox server**, live.
|
||||
6. **Caddy** routes different domains to the applications.
|
||||
7. This is the **web version of `/projects/agybot`** (Telegram bot). Reuse agybot's proven ideas, not its code.
|
||||
8. Every org **improves itself** through **memories**: how its users like to manage the org, and how the
|
||||
app's logic and code work. It also exposes **MCP** so AI agents can control our app and read or push data.
|
||||
9. **Main difference from agybot:** users never touch infrastructure or AI accounts. They only give
|
||||
orders and tasks. Tasks run through **dynamic pipelines**.
|
||||
|
||||
## 2. What already exists (audited 2026-10-13, the code is the truth, not old TODO.md ticks)
|
||||
|
||||
- **Transport:** the API is WebSocket-RPC only. `app/main.py` exposes only `/health*` and `/api/ws`. Every
|
||||
handler lives in an internal FastAPI app (`app/interfaces/realtime/rpc_app.py`) that is reached
|
||||
in-process. Realtime push uses channels `org:<id>`, `task:<id>`, `run:<id>`, `thread:<id>`,
|
||||
`session:<id>`, `user:<id>`, `admin` (`app/application/ports/realtime.py`, authz in
|
||||
`app/interfaces/realtime/authz.py`).
|
||||
- **Roles:** platform roles `owner` (exactly one, enforced by a DB partial unique index), `manager` and
|
||||
`user`. Org roles are `owner > admin > manager > member > viewer`. There is no public signup. Accounts
|
||||
come from the owner (`/admin/users`) or from an org invite link.
|
||||
- **Admin console:** agy (Google OAuth) accounts with pool rotation and 429 cooldown, plus model
|
||||
catalog, routing policies, rate cards, usage and error console.
|
||||
- **Projects:** `projects` holds only name/slug/description/default_branch. There is no app profile and
|
||||
no "1 org = 1 app" rule. Gitea adapter (real) and mock. Preview domains through the Caddy adapter
|
||||
(real) and mock.
|
||||
- **Sandbox:** full CRUD API, but only Daytona/E2B adapters (untested live) and a mock that runs nothing.
|
||||
- **Workforce:** templates, employees, threads, tasks, runs. **The run engine makes ONE LLM call per
|
||||
task and stores the text.** Tool calls are recorded, never executed. No employee edits code, runs
|
||||
tests, commits or pushes.
|
||||
- **Missing entirely:** org wizard, app profile, external git, SSH/FTP deploy, CI/CD, demo server, docker
|
||||
sandbox, memory system, MCP, per-org model grants.
|
||||
- **Tests:** about 233 API tests (pytest, DB via `Base.metadata.create_all` in `tests/conftest.py`) and 64
|
||||
web tests (vitest). Five-step alembic chain ending at `d4e5f6a7b8c9`.
|
||||
|
||||
## 3. What we take from agybot (concepts only, never import it)
|
||||
|
||||
| agybot | v2 equivalent | Owner |
|
||||
|---|---|---|
|
||||
| Roles `manager, planner, programmer, reviewer, tester` (+ optional critic) | same role keys | Agent 2 |
|
||||
| `[[TAG attr="..."]]body[[/TAG]]` protocol, per-role allowed tags | same, in `app/application/engine/protocol.py` | Agent 2 |
|
||||
| Pipelines `plan_code_test` (default), `plan_code_review_test`, `code_test`, `analysis_only` | same names | Agent 2 |
|
||||
| Pipeline choice: manager's `pipeline=` > task-kind default > app default; review pipeline needs an active reviewer | same | Agent 2 |
|
||||
| Task statuses `pending_manager_review → pending_analysis → pending_implementation → pending_review → pending_test → pending_final_review → merge_request → completed` (+ `pending_user_decision`, `error_review`, `paused`, `cancelled`, `failed`) | same, with optimistic `expected_from` | Agent 2 |
|
||||
| Gates: real test run after WORK_REPORT, green gate (waiving needs a written reason), diff guard, no-progress stop (2 identical diffs), max cycles, transient-error retries, escalate model on red | same | Agent 2 |
|
||||
| Worktree per task `task/<id>`, merge into `dev`, push | same, through `RepoWorkspacePort` | Agent 3 (port), Agent 2 (use) |
|
||||
| Manager `HIRE/UPDATE_EMPLOYEE/FIRE` (max 3 changes per turn, max 20 employees), only on org-shared models | same, only on **owner-granted** models | Agent 2 |
|
||||
| Org knowledge `[[KNOWLEDGE]]` (max 3 per turn, caps 1,500 chars/key and 6,000/org), lessons with 30-day TTL, project summaries per task | memory system | Agent 1 |
|
||||
| `app_control_mcp` / `remote_dev_mcp` | MCP gateway with scoped tokens | Agent 1 |
|
||||
| `caddy_manager` (single-label hosts under root, localhost targets, validate→revert→reload) | existing `CaddyDomainRouterAdapter` | Agent 3 |
|
||||
| `ssh_manager.push_project`, `ftp_manager.deploy_project` | deploy adapters | Agent 3 |
|
||||
| `org_live_stream` (ring buffer, polled) | realtime push events (no polling) | Agents 2 & 3 |
|
||||
|
||||
**agybot bugs we must NOT repeat:** deploy publishing the wrong code silently; `git_publish` stuck
|
||||
mid-merge; observe mode applying real state changes; a failed/cancelled dependency deadlocking its
|
||||
dependents; manager chat cost not counted in the budget; git operations without the repo lock; about
|
||||
870 silent `except Exception` blocks; plaintext credentials; the AI running directly on the host;
|
||||
`git status --porcelain` hiding new dirs (always use `-uall`).
|
||||
|
||||
## 4. Architecture decisions (fixed, do not re-decide)
|
||||
|
||||
- **D1. 1 org = 1 app.** The app is the existing `projects` row (the table name stays). `projects` gets
|
||||
`UNIQUE(organization_id)`. The UI calls it "Application" (اپلیکیشن). Route `/projects` becomes `/app`.
|
||||
- **D2. App profile.** These are columns on `projects`: app_type, language, framework, runtime_version,
|
||||
package_manager, setup/build/test/start commands, app_port, healthcheck_path, git_source
|
||||
(`platform|external`), external_repo_url, wizard_completed_at. External git credentials and deploy
|
||||
targets live in their own tables, **encrypted at rest** with the existing crypto helper. They are never
|
||||
returned raw to the frontend (only `has_secret: true` and masked values).
|
||||
- **D3. Owner model grants.** `org_model_grants(org_id, ai_model_id, daily_credit_cap, is_active)`. The
|
||||
engine may only hire or run on granted models. Users never see model names; the UI shows the
|
||||
employee's role and a neutral "tier" label.
|
||||
- **D4. Execution engine = in-process agent loop.** It uses the existing `ModelProviderPort` (owner's agy
|
||||
accounts, pool rotation) with the agybot tag protocol for actions **and** tools. Tools run **inside the
|
||||
org's sandbox container** through `SandboxProviderPort`. There are no agent CLIs in v1, so AI
|
||||
credentials never leave the server and users never see them.
|
||||
- **D5. Sandbox = local Docker** (`SANDBOX_PROVIDER=docker`, the binary is configurable so podman works
|
||||
too). Per app there is one **workspace container** (employees work in it) and one **demo container**
|
||||
(runs the latest green build). Hardened: non-root, `--cap-drop ALL`, `no-new-privileges`, cpu/mem/pids
|
||||
limits, ports published on `127.0.0.1` only.
|
||||
- **D6. Git is done by the platform, never by the AI.** `RepoWorkspacePort` runs git **on the host** with
|
||||
a gitdir that is **never mounted** into the container. Git is always called with explicit
|
||||
`--git-dir/--work-tree`, hooks disabled (`core.hooksPath=/dev/null`), `-uall`, `core.quotePath=false`,
|
||||
and a per-repo lock. Remote credentials go through `GIT_ASKPASS` or a temporary key file and **never
|
||||
appear in a URL, log or exception**.
|
||||
- **D7. CI/CD is platform-native.** A push to `dev` (by the engine or seen by polling the remote) creates
|
||||
a `pipeline_run`. Steps: checkout → setup → build → test → package → deploy demo → healthcheck → route.
|
||||
Logs stream live as realtime events on `org:<id>` and a bounded tail is stored. A failed run keeps the
|
||||
previous demo running.
|
||||
- **D8. Production deploy is always human-triggered** (org owner/admin, with confirmation and audit).
|
||||
SSH: release dir + `current` symlink + post-deploy command + rollback. FTP/FTPS: upload a release, keep
|
||||
the previous one for rollback. User-supplied hosts are SSRF-checked (no private, loopback or link-local
|
||||
IPs unless the owner allowlists them). SSH host key is pinned on first successful test (TOFU).
|
||||
- **D9. Domains.** Each app gets an automatic demo host `<org-slug>.<PREVIEW_ROOT_DOMAIN>`. Only the
|
||||
**owner** can attach extra hosts. Targets are always `127.0.0.1:<allocated port>`. The existing rules in
|
||||
`app/domain/platform/rules.py` stay authoritative.
|
||||
- **D10. Memory** (`org_memories`) is the org's long-term self-knowledge: rules, preferences, lessons,
|
||||
facts and the code map, with caps, dedupe, TTL for lessons and pinning. Users can view, edit, pin and
|
||||
delete it. The engine reads it through `MemoryPort.build_context` and feeds it through
|
||||
`MemoryPort.on_task_finished`.
|
||||
- **D11. MCP gateway** is a separate HTTP entry point (`services/mcp-gateway`) using the official `mcp`
|
||||
Python SDK (streamable HTTP). Bearer tokens are hashed, scoped, org-bound and expire. Every call is
|
||||
audited and org-enforced.
|
||||
- **D12. Platform invites.** The owner can create an invite link for a **new platform account**, which
|
||||
becomes role `manager` and may create orgs. A logged-in manager with no org is sent to the wizard. Org
|
||||
invites (join an existing org) stay as they are.
|
||||
- **D13. Automation safety.** Tasks run automatically up to merge into `dev`, followed by a demo deploy.
|
||||
Nothing goes to production without a human click. No code path may flip that default.
|
||||
|
||||
## 5. Contracts between agents (created by Claude in TODO-0, then FROZEN)
|
||||
|
||||
All contracts live in `apps/api/app/application/ports/` and are `typing.Protocol`s with a
|
||||
working default implementation, so each agent can test alone. **Agents must not change a contract
|
||||
file.** If you need a change, write a `CHANGE REQUEST` in your phase report, keep coding against the
|
||||
current signature, and add a local adapter or shim in your own files.
|
||||
|
||||
```python
|
||||
# ports/repo_workspace.py — implemented by Agent 3, used by Agent 2 (and Agent 1 for test_remote)
|
||||
class RepoWorkspacePort(Protocol):
|
||||
async def ensure_workspace(self, *, project_id: str) -> WorkspaceRef: ... # clone/fetch app repo on host
|
||||
async def create_task_worktree(self, *, project_id: str, task_id: str, base: str = "dev") -> WorktreeRef: ...
|
||||
async def diff(self, *, worktree: WorktreeRef) -> DiffResult: ... # text + changed files (-uall)
|
||||
async def commit(self, *, worktree: WorktreeRef, message: str, author: str) -> str | None: ... # sha or None if clean
|
||||
async def sync_with_base(self, *, worktree: WorktreeRef) -> MergeOutcome: ... # merge base into task branch
|
||||
async def merge_into(self, *, project_id: str, branch: str, target: str = "dev") -> MergeOutcome: ...
|
||||
async def push(self, *, project_id: str, branch: str) -> PushOutcome: ...
|
||||
async def remove_worktree(self, *, worktree: WorktreeRef) -> None: ...
|
||||
async def head(self, *, project_id: str, branch: str) -> str | None: ...
|
||||
async def test_remote(self, *, url: str, credentials: RemoteCredentials) -> RemoteCheck: ...
|
||||
# WorktreeRef has: project_id, task_id, branch, host_path, sandbox_path (path inside the workspace container)
|
||||
|
||||
# ports/delivery_hooks.py — implemented by Agent 3, called by Agent 1 (app created) and Agent 2 (pushed)
|
||||
class DeliveryHooks(Protocol):
|
||||
async def on_app_created(self, *, organization_id: str, project_id: str) -> None: ...
|
||||
async def on_branch_pushed(self, *, organization_id: str, project_id: str, branch: str, sha: str, source: str) -> None: ...
|
||||
|
||||
# ports/deploy_target.py — implemented by Agent 3, called by Agent 1's wizard
|
||||
class DeployTargetTester(Protocol):
|
||||
async def test_connection(self, *, target: DeployTargetSpec) -> DeployCheck: ... # ok, message, fingerprint
|
||||
|
||||
# ports/memory.py — implemented by Agent 1, used by Agent 2
|
||||
class MemoryPort(Protocol):
|
||||
async def build_context(self, *, organization_id: str, role: str, query: str, budget_chars: int = 2500) -> str: ...
|
||||
async def record(self, *, organization_id: str, kind: str, content: str, source: str,
|
||||
evidence_task_id: str | None = None, key: str | None = None) -> str | None: ...
|
||||
async def on_task_finished(self, *, organization_id: str, task: TaskOutcome) -> None: ...
|
||||
|
||||
# ports/model_grants.py — implemented by Agent 1, used by Agent 2
|
||||
class ModelGrantPort(Protocol):
|
||||
async def granted_models(self, *, organization_id: str) -> list[GrantedModel]: ... # id, capabilities, tier, caps
|
||||
async def is_granted(self, *, organization_id: str, ai_model_id: str) -> bool: ...
|
||||
|
||||
# ports/workforce_bootstrap.py — implemented by Agent 2, called by Agent 1's wizard
|
||||
class WorkforceBootstrapPort(Protocol):
|
||||
async def bootstrap_team(self, *, organization_id: str, project_id: str) -> None: ... # creates the manager + default team
|
||||
```
|
||||
|
||||
Existing contracts that stay as they are: `SandboxProviderPort` (Agent 3 adds the docker adapter),
|
||||
`GitProviderPort`, `DomainRouterPort`, `ModelProviderPort`, `EventPublisher`/`RealtimeEvent`.
|
||||
|
||||
**Realtime event names** (use the existing `RealtimeEvent` + `get_publisher()`):
|
||||
|
||||
| Channel | Event types | Producer |
|
||||
|---|---|---|
|
||||
| `task:<id>` | `task.status`, `task.step`, `task.gate` | Agent 2 |
|
||||
| `run:<id>` | `run.log`, `run.tool`, `run.status` | Agent 2 |
|
||||
| `org:<id>` | `task.created`, `team.changed` | Agent 2 |
|
||||
| `org:<id>` | `pipeline.queued`, `pipeline.step`, `pipeline.log`, `pipeline.status`, `release.live`, `deploy.status` | Agent 3 |
|
||||
| `org:<id>` | `memory.changed`, `org.updated` | Agent 1 |
|
||||
|
||||
Log events are **batched** (at most one event per 500 ms per stream, max 200 lines per event).
|
||||
|
||||
## 6. File ownership (no two agents edit the same file)
|
||||
|
||||
| Area | Agent 1 | Agent 2 | Agent 3 |
|
||||
|---|---|---|---|
|
||||
| API routers | `app_setup.py`*, `memory.py`*, `mcp_tokens.py`*, `admin_grants.py`*, `organizations.py`, `invites.py`, `admin_users.py`, `auth.py`, `knowledge.py` | `workforce.py` | `projects.py`, `sandbox.py`, `delivery.py`*, `admin_domains.py`* |
|
||||
| Services | `org_setup_service.py`*, `memory_service.py`*, `model_grant_service.py`*, `mcp_service.py`*, `platform_users.py`, `knowledge_service.py` | `workforce_service.py`, `app/application/engine/**`* | `project_service.py`, `sandbox_service.py`, `delivery_service.py`*, `deploy_service.py`* |
|
||||
| Adapters | — | — | everything in `app/infrastructure/adapters/` (incl. `registry.py`, `mock.py`) |
|
||||
| Repositories | `identity.py`, `knowledge.py`, `memory.py`* | `workforce.py` | `projects.py`, `sandbox.py`, `delivery.py`* |
|
||||
| Domain | `identity/`, `knowledge/`, `memory/`*, `projects/catalog.py`* | `workforce/` | `platform/`, `sandbox/`, `projects/models.py` |
|
||||
| DB tables | `db/models_org.py`* | `db/models_engine.py`* | `db/models_delivery.py`* |
|
||||
| Wiring | `container_org.py`* | `container_engine.py`* | `container_delivery.py`* |
|
||||
| Workers | — | `workers/ai_runtime.py` | `workers/sandbox_gateway.py`, `workers/delivery.py`* |
|
||||
| Other API | `app/interfaces/mcp/**`*, `services/mcp-gateway/**` | — | — |
|
||||
| Web pages `apps/web/src/app/` | `onboarding/`, `accept-invite/`, `(app)/settings/`, `(app)/memory/`*, `(app)/knowledge/`, `(app)/admin/users/`, `(app)/admin/model-grants/`*, `(app)/admin/mcp/`*, `(app)/admin/organizations/` | `(app)/manager/`, `(app)/tasks/`, `(app)/workforce/`, `(app)/traces/`, `(app)/dashboard/` | `(app)/app/`*, `(app)/projects/` (delete after moving), `(app)/admin/domains/`*, `(app)/admin/sandbox/`*, `components/sandbox-panel.tsx` |
|
||||
| Web API client | `lib/api-org.ts`* | `lib/api-engine.ts`* | `lib/api-delivery.ts`* |
|
||||
| Tests | `tests/test_org_*.py`, `test_memory*.py`, `test_mcp*.py`, `test_model_grants*.py`, `test_invites.py`, `test_platform_hierarchy.py`, web `src/test/org-*.test.tsx` | `tests/test_engine_*.py`, `test_workforce*.py`, `test_workers.py`, web `src/test/engine-*.test.tsx` | `tests/test_docker_*.py`, `test_git_workspace*.py`, `test_delivery*.py`, `test_deploy*.py`, `test_project_repo.py`, `test_domain_isolation.py`, `test_platform_rules.py`, web `src/test/delivery-*.test.tsx` |
|
||||
| Alembic (one file per agent, more only if chained off your own) | `a1b2c3d4e5f1_org_platform.py` | `a2b3c4d5e6f2_ai_workforce.py` | `a3b4c5d6e7f3_delivery_infra.py` |
|
||||
|
||||
`*` = new file (Claude creates an empty stub in TODO-0, the agent fills it).
|
||||
|
||||
**Locked files (Claude only):** `db/models.py`, `container.py`, `rpc_app.py`, `workers/__main__.py`,
|
||||
`ports/*.py` contracts, `tests/conftest.py`, `pyproject.toml`, `package.json`, `pnpm-lock.yaml`,
|
||||
`apps/web/src/lib/api.ts`, `apps/web/src/lib/nav.ts`, `components/AppShell.tsx`, `alembic/env.py`, root
|
||||
`README.md`/`PLAN.md`/`TODO.md`, `docs/plan/PLAN-v2.md`, every other agent's TODO file.
|
||||
|
||||
**Append-only blocks.** `app/core/config.py` and `.env.example` each have three marked blocks
|
||||
(`# >>> agent-1` … `# <<< agent-1`, and so on). You may add lines **only inside your own block**.
|
||||
|
||||
Each new migration uses `down_revision = "a0f0e1d2c3b4"` (the foundation). Claude adds the merge
|
||||
revision when the branches are merged.
|
||||
|
||||
## 7. Working rules for every agent
|
||||
|
||||
1. Work **only** in your own git worktree and branch (given in your TODO file). Never `cd` outside it and
|
||||
never write to `/tmp`. Put scratch scripts in `./_scratch/` inside your worktree and delete them at the end.
|
||||
2. **Do NOT commit, push, merge or rebase.** Claude reviews and commits each phase.
|
||||
3. At the **end of each phase**, stop and write `docs/plan/reports/agent-<N>-phase-<K>.md` containing:
|
||||
what you did, files changed, test output summary, deviations, open questions, and `CHANGE REQUEST`s.
|
||||
Then wait.
|
||||
4. Before declaring a phase done, run **all** checks (not only your tests) and paste the summary lines:
|
||||
```bash
|
||||
cd apps/api && .venv/bin/python -m pytest -q && .venv/bin/ruff check . && .venv/bin/mypy app
|
||||
cd apps/web && pnpm test && pnpm exec tsc --noEmit && pnpm lint
|
||||
```
|
||||
Every existing test must stay green. Never delete or weaken an existing test to make it pass. If an
|
||||
existing test encodes behaviour that this plan changes on purpose, update it and explain why in the report.
|
||||
5. Every multi-tenant query is filtered by `organization_id`. Every destructive or production action has
|
||||
a confirmation and an audit event. No secret ever goes to the frontend, a log, an exception message, a
|
||||
realtime event or an LLM prompt.
|
||||
6. No silent `except Exception: pass`. Catch narrow errors, or record them through
|
||||
`app.infrastructure.error_log`.
|
||||
7. Users (non-owner) never see model names, providers, accounts, containers, ports or raw infra
|
||||
errors. Translate them to plain Persian messages.
|
||||
8. UI language is Persian and RTL. Reuse `packages/ui` components and the existing motion classes.
|
||||
9. Match the existing code style: hexagonal layers (domain has no FastAPI/SQLAlchemy imports), type
|
||||
hints everywhere, docstrings in the same tone as nearby code.
|
||||
10. Do not run `agy`, do not call real external services in tests (use fakes). Integration tests that need
|
||||
docker, network or real git remotes must be skipped unless an env flag is set.
|
||||
|
||||
## 8. Integration (Claude, after the agents)
|
||||
|
||||
1. Review each phase report and diff, run the full checks, and commit on the agent's branch.
|
||||
2. Merge the branches into `dev` one by one: Agent 3 → Agent 1 → Agent 2. Run the full tests after **each**
|
||||
merge.
|
||||
3. Add the alembic merge revision for the three heads and run `alembic upgrade head` + `alembic check`.
|
||||
4. Replace the Null/default contract implementations in `container.py` with the real ones and run the
|
||||
end-to-end smoke test:
|
||||
wizard → app created → demo domain → manager order → task runs → tests → push → pipeline → demo live
|
||||
→ production deploy (to a local SSH/FTP test server).
|
||||
5. Update `README.md`, `TODO.md` and the architecture doc, then commit.
|
||||
@@ -0,0 +1,60 @@
|
||||
# TODO-0 — Foundation (Claude does this BEFORE the three agents start)
|
||||
|
||||
Goal: put every shared or locked file in its final shape, so the three agents never edit the same
|
||||
file. When this is done, `dev` has a foundation commit and three worktrees exist.
|
||||
|
||||
Read `docs/plan/PLAN-v2.md` §4–§6 first. Everything here is mechanical; no product logic.
|
||||
|
||||
## F1. Environment
|
||||
- [ ] `pnpm install` at root; `pnpm api:venv`; confirm the baseline: `pytest -q`, `ruff`, `mypy`, `pnpm test`, `tsc`. Record the counts.
|
||||
- [ ] Add API deps to `apps/api/pyproject.toml`: `asyncssh`, `aioftp`, `mcp` (official SDK). Install and re-run the baseline.
|
||||
|
||||
## F2. DB foundation (`db/models.py` + migration `a0f0e1d2c3b4_v2_foundation.py`, down_revision `d4e5f6a7b8c9`)
|
||||
- [ ] `projects`: add app-profile columns (all nullable or defaulted): `app_type`, `language`, `framework`,
|
||||
`runtime_version`, `package_manager`, `setup_command`, `build_command`, `test_command`, `start_command`,
|
||||
`app_port` (int), `healthcheck_path` (default `/`), `git_source` (`platform|external`, default `platform`),
|
||||
`external_repo_url`, `wizard_completed_at`.
|
||||
- [ ] `projects`: `UniqueConstraint("organization_id", name="uq_project_one_per_org")`. Fix the seed and any test that creates 2 projects in one org.
|
||||
- [ ] New table `project_external_git` (project_id PK/FK, auth_type `token|ssh_key|none`, username, secret_encrypted, known_hosts, updated_at).
|
||||
- [ ] New table `deploy_targets` (id, organization_id, project_id, kind `ssh|sftp|ftp|ftps`, host, port, username,
|
||||
auth_type `password|key`, secret_encrypted, remote_path, post_deploy_command, host_fingerprint, is_active,
|
||||
last_tested_at, last_test_ok, last_test_message, created_at). At most one active target per project.
|
||||
- [ ] New table `org_model_grants` (id, organization_id, ai_model_id, daily_credit_cap nullable, is_active, created_by, created_at; unique org+model).
|
||||
- [ ] At the end of `models.py`, import `models_org`, `models_engine`, `models_delivery` (three empty modules with a docstring naming the owner agent).
|
||||
- [ ] `alembic upgrade head`, `alembic downgrade -1`, `alembic upgrade head`, `alembic check`, all clean.
|
||||
|
||||
## F3. Contracts (`app/application/ports/`), exactly as in PLAN-v2 §5
|
||||
- [ ] `repo_workspace.py` (+ dataclasses `WorkspaceRef`, `WorktreeRef`, `DiffResult`, `MergeOutcome`, `PushOutcome`, `RemoteCredentials`, `RemoteCheck`) and an in-memory `FakeRepoWorkspace` for tests.
|
||||
- [ ] `delivery_hooks.py` + `NullDeliveryHooks` (records calls in a list for tests).
|
||||
- [ ] `deploy_target.py` (`DeployTargetSpec`, `DeployCheck`) + `FakeDeployTargetTester` (ok unless host contains "fail").
|
||||
- [ ] `memory.py` (`TaskOutcome`) + `NullMemory` (empty context, record returns None).
|
||||
- [ ] `model_grants.py` (`GrantedModel`) + `AllActiveModelsGrants` (today's behaviour: every active model).
|
||||
- [ ] `workforce_bootstrap.py` + `NullWorkforceBootstrap`.
|
||||
- [ ] `app/domain/projects/profile.py`: frozen dataclass `AppProfile` + `AppProfile.from_row(...)`.
|
||||
|
||||
## F4. Wiring stubs
|
||||
- [ ] `app/application/container_org.py`, `container_engine.py`, `container_delivery.py`: each holds the builder functions for its port(s), returning the Null/Fake implementations for now. `container.py` re-exports them.
|
||||
- [ ] Empty routers (`router = APIRouter()`) registered in `rpc_app.py`: `app_setup`, `memory`, `mcp_tokens`, `admin_grants`, `delivery`, `admin_domains`.
|
||||
- [ ] `app/workers/delivery.py` stub (one no-op pass) registered in `workers/__main__.py` as `delivery`; `services/delivery-worker/README.md` + `pnpm worker:delivery` script.
|
||||
- [ ] `services/mcp-gateway/README.md` placeholder + `pnpm mcp:dev` script (points at `app.interfaces.mcp.server:main`, stub module that raises a clear "not implemented yet").
|
||||
- [ ] `config.py` and `.env.example`: three marked blocks `# >>> agent-1 … # <<< agent-1` (likewise 2 and 3). Pre-fill the obvious settings:
|
||||
- agent-3: `SANDBOX_DOCKER_BIN=docker`, `SANDBOX_WORKSPACES_ROOT`, `SANDBOX_DEFAULT_CPUS=1`, `SANDBOX_DEFAULT_MEMORY_MB=1024`, `DEMO_PORT_START=8100`, `DEMO_PORT_END=8999`, `DELIVERY_POLL_SECONDS=60`, `DEPLOY_ALLOW_PRIVATE_HOSTS=false`, `ARTIFACTS_ROOT`.
|
||||
- agent-2: `ENGINE_MAX_TOOL_CALLS_PER_TURN=40`, `ENGINE_TURN_TIMEOUT_SECONDS=900`, `ENGINE_MAX_CYCLES=6`, `ENGINE_MAX_EMPLOYEES=20`, `ENGINE_TICK_SECONDS=10`.
|
||||
- agent-1: `MCP_GATEWAY_HOST=127.0.0.1`, `MCP_GATEWAY_PORT=8765`, `MCP_TOKEN_TTL_DAYS=90`, `MEMORY_ORG_CHAR_CAP=60000`, `PLATFORM_INVITE_TTL_HOURS=72`.
|
||||
|
||||
## F5. Web stubs
|
||||
- [ ] `lib/api-org.ts`, `lib/api-engine.ts`, `lib/api-delivery.ts`: each imports `apiRequest` from `./api` and exports an empty object `orgApi` / `engineApi` / `deliveryApi`.
|
||||
- [ ] `nav.ts`: rename the projects entry to `{ href: "/app", label: "اپلیکیشن", hint: "دمو، استقرار، Git" }`, rename knowledge to `{ href: "/memory", label: "حافظهٔ سازمان", ... }`; admin entries `/admin/model-grants` (مدلهای سازمانها), `/admin/domains` (دامنهها), `/admin/sandbox` (سندباکسها), `/admin/mcp` (MCP).
|
||||
- [ ] Placeholder pages (using `PlaceholderPage`) for `(app)/app`, `(app)/memory`, `(app)/admin/model-grants`, `(app)/admin/domains`, `(app)/admin/sandbox`, `(app)/admin/mcp`. `(app)/projects` keeps working until Agent 3 removes it.
|
||||
|
||||
## F6. Finish
|
||||
- [ ] Baseline checks are all green with the same counts as F1 (plus any new tests).
|
||||
- [ ] Commit on `dev`: `v2 foundation: shared contracts, app profile schema, wiring stubs`.
|
||||
- [ ] Create the worktrees from that commit:
|
||||
```bash
|
||||
git worktree add ../smart-org-wt/agent1 -b agent1/org-platform
|
||||
git worktree add ../smart-org-wt/agent2 -b agent2/ai-workforce
|
||||
git worktree add ../smart-org-wt/agent3 -b agent3/delivery-infra
|
||||
```
|
||||
In each one: `pnpm install` and `pnpm api:venv` so the agent can run the checks immediately.
|
||||
- [ ] `mkdir docs/plan/reports` (tracked with a `.gitkeep`).
|
||||
@@ -0,0 +1,129 @@
|
||||
# TODO — Agent 1: Org Platform (invites, wizard, app profile, sharing, model grants, memory, MCP)
|
||||
|
||||
- **Worktree:** `/projects/smart-org-wt/agent1` · **Branch:** `agent1/org-platform` (already created, already installed)
|
||||
- **Read first:** `docs/plan/PLAN-v2.md` (all of it). Then skim: `apps/api/app/interfaces/http/routers/organizations.py`,
|
||||
`invites.py`, `admin_users.py`, `app/application/services/platform_users.py`, `app/domain/identity/models.py`,
|
||||
`apps/web/src/app/onboarding/page.tsx`, `apps/web/src/app/(app)/settings/`.
|
||||
- **Your files:** only the "Agent 1" column of PLAN-v2 §6, your block in `config.py` / `.env.example`, and the
|
||||
migration `alembic/versions/a1b2c3d4e5f1_org_platform.py` (`down_revision = "a0f0e1d2c3b4"`).
|
||||
- **Never touch:** the locked files and the other agents' files (PLAN-v2 §6). Need something there? → `CHANGE REQUEST` in your report.
|
||||
- **Do NOT commit, push, merge or rebase.** Never `cd` outside the worktree, never write to `/tmp`.
|
||||
- **After each phase:** run all checks (PLAN-v2 §7.4), write `docs/plan/reports/agent-1-phase-<K>.md`, then STOP and wait.
|
||||
|
||||
You consume these contracts (do not change them): `DeployTargetTester`, `RepoWorkspacePort.test_remote`,
|
||||
`DeliveryHooks.on_app_created`, `WorkforceBootstrapPort.bootstrap_team`. Get them only from the builders in
|
||||
`app/application/container.py`. In tests use the Fake/Null versions from the ports package.
|
||||
|
||||
You implement these contracts: `MemoryPort`, `ModelGrantPort`. Your real classes are wired in
|
||||
`container_org.py`.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Platform invites, roles and org sharing
|
||||
|
||||
Backend
|
||||
- [ ] 1.1 New table `platform_invites` (in `db/models_org.py`): id, token_hash, email (optional), created_by (owner), role (`manager` only for now), expires_at (`PLATFORM_INVITE_TTL_HOURS`), used_at, used_by_user_id, revoked_at. Migration in your file.
|
||||
- [ ] 1.2 Owner-only endpoints in `admin_users.py`: `POST /admin/platform-invites` (returns `invite_url` once, token shown once), `GET /admin/platform-invites`, `POST /admin/platform-invites/{id}/revoke`. Audit every action.
|
||||
- [ ] 1.3 In `invites.py`, extend `GET /invites/{token}` and `POST /invites/accept` to recognise a platform invite: if no account exists, create a `manager` account with the chosen password; if the email exists, require login (same rule as org invites: an invite never resets a password). Single use, constant-time compare, expired or revoked → the same generic 404.
|
||||
- [ ] 1.4 `GET /auth/me` (you own `auth.py`) gains `needs_onboarding: bool` = platform role is `manager`, the user has zero org memberships, and they are not the owner.
|
||||
- [ ] 1.5 Org sharing in `organizations.py`: `POST /organizations/{id}/transfer-ownership` (owner → an existing admin member, in one transaction, audited), `POST /organizations/{id}/leave` (the last owner cannot leave), and `GET /organizations/{id}/invites` + `DELETE .../invites/{invite_id}` (list and revoke pending org invites). Keep the role ladder `owner > admin > manager > member > viewer`.
|
||||
- [ ] 1.6 Tests: `tests/test_org_platform_invites.py`, `tests/test_org_sharing.py`. Cover: non-owner cannot create platform invites; token single use; expired/revoked; existing-email path; transfer + leave edge cases; tenant isolation (a member of org A cannot list org B invites).
|
||||
|
||||
Frontend
|
||||
- [ ] 1.7 `/admin/users`: a "دعوت مدیر جدید" button → modal → shows the link once with a copy button; a list of pending invites with revoke.
|
||||
- [ ] 1.8 `/accept-invite`: handle the platform-invite variant (title "ساخت حساب مدیر").
|
||||
- [ ] 1.9 After login: if `needs_onboarding`, redirect to `/onboarding`. Do this in the `onboarding` page and in a small hook in `lib/api-org.ts`. `AppShell` is locked, so if a redirect there is unavoidable, write a CHANGE REQUEST.
|
||||
- [ ] 1.10 `(app)/settings`: a "اشتراکگذاری سازمان" section: members with role selector, invite form (email + role), pending invites with revoke, transfer ownership (confirmation modal that requires typing the org slug), leave org.
|
||||
- [ ] 1.11 Web tests `src/test/org-sharing.test.tsx`.
|
||||
|
||||
**Phase 1 done when:** the owner creates a link → a new person opens it, sets a password → logs in → lands on `/onboarding`. All checks are green.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Org creation wizard + app profile (1 org = 1 app)
|
||||
|
||||
Backend
|
||||
- [ ] 2.1 `app/domain/projects/catalog.py`: pure-Python catalog. `APP_TYPES` (`web_app`, `api`, `static_site`, `bot`, `worker`, `other`) and `LANGUAGES` → frameworks → defaults (runtime_version, package_manager, setup/build/test/start commands, app_port, healthcheck_path, docker base image name). Cover at least:
|
||||
- node: next, express, nestjs, vite-react
|
||||
- python: fastapi, django, flask
|
||||
- php: laravel, plain
|
||||
- go: plain
|
||||
- static: html
|
||||
Add a function `defaults_for(app_type, language, framework) -> ProfileDefaults`. Unit tests for every entry.
|
||||
- [ ] 2.2 New table `org_setup_drafts` (user_id, step, data_json **without secrets**, updated_at). Secrets typed during the wizard are held only in the final submit request, never in the draft.
|
||||
- [ ] 2.3 `org_setup_service.py` + router `app_setup.py`:
|
||||
- `GET /setup/catalog`.
|
||||
- `GET/PUT /setup/draft`.
|
||||
- `POST /setup/check-git`: external URL + credentials → `RepoWorkspacePort.test_remote`. Validate the URL **format** first in your own service: `https://` or `ssh://`/`git@` only, no credentials embedded in the URL, no IP-literal localhost/private hosts. The DNS-resolved SSRF check happens inside `test_remote` (Agent 3); do not import Agent 3's code.
|
||||
- `POST /setup/check-deploy`: → `DeployTargetTester.test_connection`.
|
||||
- `POST /setup/complete`: in ONE transaction, create the org (caller = org owner), the project with its app profile (`wizard_completed_at=now`), the encrypted external git credentials or the platform Gitea repo through the existing `GitProviderPort.ensure_project_repo`, the optional deploy target (encrypted), and the default model grants (Phase 3; until then, none). After commit, call `WorkforceBootstrapPort.bootstrap_team` and `DeliveryHooks.on_app_created`. Each hook is called once; a hook failure is recorded with `error_log` and does not undo the org.
|
||||
- Slug is unique (reuse the existing slug rules). Audit `org.created_via_wizard`.
|
||||
- [ ] 2.4 App profile editing for org owner/admin: `GET/PATCH /organizations/{org}/app/profile` (commands, port, healthcheck, framework), `PUT /organizations/{org}/app/deploy-target` (+ test), `PUT /organizations/{org}/app/git` (switch platform ↔ external is NOT allowed after creation; only credential rotation). Secrets are write-only: responses return `has_secret` and masked usernames only.
|
||||
- [ ] 2.5 Enforce 1 org = 1 app: any other path that creates a second project returns a clear 409 (the DB constraint exists already; make the service error friendly). Note: `projects.py` is Agent 3's file. Do NOT edit it; only your service/router may create the app.
|
||||
- [ ] 2.6 Tests `tests/test_org_wizard.py`: happy path platform git; happy path external git with a fake `test_remote`; a deploy target with a failing test does not block completion (it is saved as `last_test_ok=false`) but the user sees the warning; secrets never in any response, draft, audit metadata or log (assert by searching the captured output and DB rows for the plaintext secret); a second app → 409; a user with role `user` cannot complete the wizard; SSRF cases for the git URL.
|
||||
|
||||
Frontend
|
||||
- [ ] 2.7 Rebuild `/onboarding` as a 5-step wizard (stepper, back/next, draft auto-save, refresh resumes):
|
||||
1. Organization: name, slug (auto-suggest).
|
||||
2. Application: name, type, language, framework (cards from the catalog), "advanced" accordion with prefilled commands, port and healthcheck.
|
||||
3. Git: "Git پلتفرم (پیشنهادی)" or "Git خودم" (URL + token or SSH key + "بررسی اتصال" button).
|
||||
4. Production (optional, skippable): SSH / SFTP / FTP / FTPS fields + "تست اتصال". Show a clear note that production deploys only happen when they press the button.
|
||||
5. Review → create → success screen with the demo URL placeholder and a "برو به مدیر AI" button.
|
||||
- [ ] 2.8 `(app)/settings` → "اپلیکیشن" tab: edit the profile, rotate git credentials, edit or test the deploy target.
|
||||
- [ ] 2.9 Web tests `src/test/org-wizard.test.tsx` (step navigation, validation, secrets never placed in the draft payload).
|
||||
|
||||
**Phase 2 done when:** a fresh manager completes the wizard in the browser and lands on the dashboard of the new org. The DB has exactly one project with a full profile. The Null hooks recorded one `bootstrap_team` and one `on_app_created` call.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Owner model grants (employees only use what the owner provides)
|
||||
|
||||
- [ ] 3.1 `model_grant_service.py` implementing `ModelGrantPort` over `org_model_grants` + `ai_models` (only active models, only active grants). `GrantedModel.tier` is a neutral label derived from capabilities (`سریع`, `متعادل`, `قوی`). Never the model name.
|
||||
- [ ] 3.2 Owner endpoints `admin_grants.py`: list grants per org, grant/revoke model, set daily credit cap, and "default grant set for new orgs" (new table `default_model_grants`, or a column in your own table; do NOT edit `ai_models`). Audit all.
|
||||
- [ ] 3.3 Wizard `complete` copies the default grant set into the new org.
|
||||
- [ ] 3.4 Owner org policy (`org_policies` table, yours): max_employees (≤ `ENGINE_MAX_EMPLOYEES`), monthly credit budget, automation_enabled (bool). Endpoints under `admin_grants.py`. Expose `get_org_policy(org_id)` in `model_grant_service.py` for Agent 2 to read later. Write a CHANGE REQUEST so Claude adds it to the `ModelGrantPort` contract at integration.
|
||||
- [ ] 3.5 Wire your real `ModelGrantPort` in `container_org.py`.
|
||||
- [ ] 3.6 UI `/admin/model-grants`: org list → grant matrix (models × checkbox + cap), default set editor, policy editor. Org users see nothing new.
|
||||
- [ ] 3.7 Tests `tests/test_model_grants.py`: non-owner forbidden everywhere; revoked or inactive model not returned; a grant on an org you query for another org is never returned (isolation); defaults copied on wizard completion.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Org memory (self-improving org)
|
||||
|
||||
- [ ] 4.1 Domain `app/domain/memory/`: `MemoryKind` (`rule`, `preference`, `lesson`, `fact`, `code_map`), `MemorySource` (`user`, `manager`, `system`, `reflection`), validation (key ≤ 80 chars, content ≤ 1,500 chars), normalisation + content hash for dedupe.
|
||||
- [ ] 4.2 Table `org_memories` (in `models_org.py`): id, organization_id, kind, key (nullable), content, source, evidence_task_id, pinned, confidence (0..1), expires_at (lessons: +30 days unless pinned or confirmed), hits, last_used_at, created_by_user_id, created_at, updated_at, deleted_at (soft). Index `(organization_id, kind)`.
|
||||
- [ ] 4.3 `memory_service.py` implementing `MemoryPort`:
|
||||
- `record`: dedupe by hash or same key (update instead of insert), org cap `MEMORY_ORG_CHAR_CAP` (evict oldest unpinned lessons first), max 3 writes per task.
|
||||
- `build_context(role, query, budget)`: pinned rules and preferences first, then the top memories by keyword overlap with `query` and recency. Role filter: `code_map` and `fact` only for planner/programmer/reviewer/tester; preferences for the manager. Returns a compact Persian/English block within `budget_chars`; updates `hits/last_used_at`.
|
||||
- `on_task_finished(TaskOutcome)`: one cheap model call (through the existing ai_control routing in "cheap" mode and `ModelProviderPort`, no new adapter) asking for ≤ 2 lessons + a 1-paragraph code-map delta. It stores them with `source=reflection`, `confidence=0.5`. If the call fails, it records the error and returns (it never raises into the engine). Record usage for that call in the usage ledger like other model calls.
|
||||
- Expire job: function `expire_memories()` (called from the existing billing worker? NO, that is not your file → expose the function and write a CHANGE REQUEST for a worker hook).
|
||||
- [ ] 4.4 Router `memory.py`: list (filter by kind, search), create (users can "teach" a rule or preference), edit, pin/unpin, delete (soft, audited), and a `confirm` action (lesson → confidence 1, no expiry). Roles: viewer can read; member+ can add preferences; admin+ can edit or delete anything.
|
||||
- [ ] 4.5 Fold the existing knowledge sources into the same page: keep `knowledge.py` endpoints working and show sources as a second tab.
|
||||
- [ ] 4.6 UI `/memory` (replace the placeholder; also redirect `/knowledge` → `/memory`): tabs "قوانین و ترجیحات", "درسهای آموخته", "نقشهٔ کد", "منابع دانش"; inline edit; pin; confirm a lesson; "به سازمان یاد بده" input. Publish `memory.changed` on `org:<id>` and refresh live.
|
||||
- [ ] 4.7 Wire the real `MemoryPort` in `container_org.py`.
|
||||
- [ ] 4.8 Tests `tests/test_memory.py`: dedupe, caps and eviction, TTL, role filtering, budget respected, isolation, reflection failure does not raise, no secrets: run content through `app/core/redact.py` before storing and test it.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — MCP gateway (AI agents control our app)
|
||||
|
||||
- [ ] 5.1 Table `mcp_tokens` (id, organization_id, name, token_hash, prefix (first 6 chars for display), scopes CSV, created_by, expires_at, last_used_at, revoked_at). Tokens look like `smo_<random>` and are shown once.
|
||||
- [ ] 5.2 `mcp_service.py`: issue, list, revoke, verify (constant-time; expired or revoked → reject), scope check.
|
||||
- [ ] 5.3 `app/interfaces/mcp/server.py` with the official `mcp` SDK (streamable HTTP, bound to `MCP_GATEWAY_HOST:MCP_GATEWAY_PORT`) and `main()` for `pnpm mcp:dev`. Tools (each resolves the org from the token, never from arguments):
|
||||
- `app_get_profile` (`app:read`)
|
||||
- `tasks_list` / `tasks_get` (`tasks:read`)
|
||||
- `tasks_create` (`tasks:write`): goes to the manager as a user order through the **current** public `WorkforceService.create_task` signature; read it, do not edit it (Agent 2 owns it; Claude re-wires at integration if it changes)
|
||||
- `memory_search` (`memory:read`) / `memory_write` (`memory:write`, kinds `fact|preference` only, `source=user`)
|
||||
- `knowledge_search` (`memory:read`)
|
||||
- `pipeline_status` (`delivery:read`): register the tool but return `{"available": false}`. Agent 3's tables do not exist in your branch; Claude wires it at integration
|
||||
Every call is audited (`mcp.tool_call`, token prefix, tool, ok/error), rate limited (60/min per token, in-memory), and its output passes through `redact.py`.
|
||||
- [ ] 5.4 `services/mcp-gateway/README.md`: how to run it, the Caddy host suggestion `mcp.<root>` (Agent 3/owner configures it), and a client config snippet.
|
||||
- [ ] 5.5 Router `mcp_tokens.py`: org admin+ can create, list and revoke org tokens. Owner endpoint to disable MCP globally or per org (setting stored in your `org_policies`).
|
||||
- [ ] 5.6 UI: `(app)/settings` → "یکپارچهسازی (MCP)" tab (create with scope checkboxes, show-once modal with a copy button, list, revoke); `/admin/mcp` (global switch, per-org switch, token counts).
|
||||
- [ ] 5.7 Tests `tests/test_mcp.py`: use the SDK's in-process client or call the tool functions directly. Cover: a token from org A can never read org B; scope enforcement; revoked/expired; rate limit; audit row written; no secret in outputs.
|
||||
|
||||
---
|
||||
|
||||
## Final report (after Phase 5)
|
||||
`docs/plan/reports/agent-1-final.md`: summary, all endpoints added (method + path + role), tables added, the
|
||||
CHANGE REQUESTs still open, manual QA steps for the wizard and MCP, known limitations.
|
||||
@@ -0,0 +1,155 @@
|
||||
# TODO — Agent 2: AI Workforce Engine (manager AI, employees, dynamic pipelines, test/commit/push)
|
||||
|
||||
- **Worktree:** `/projects/smart-org-wt/agent2` · **Branch:** `agent2/ai-workforce` (already created, already installed)
|
||||
- **Read first:** `docs/plan/PLAN-v2.md` (all of it, especially §3 and §5). Then read fully:
|
||||
`apps/api/app/application/services/workforce_service.py`, `app/domain/workforce/models.py` + `tools.py`,
|
||||
`app/interfaces/http/routers/workforce.py`, `app/infrastructure/repositories/workforce.py`,
|
||||
`app/application/ports/model_provider.py`, `app/application/ports/realtime.py`, `app/workers/ai_runtime.py`,
|
||||
`tests/test_workforce.py`.
|
||||
- **Reference (concepts only, NEVER import, never edit):** `/projects/agybot/org_manager.py` (lines ~23-101
|
||||
roles/pipelines/statuses, `_WORK_TRANSITIONS` ~413) and `/projects/agybot/org_orchestrator.py`
|
||||
(`ROLE_TAGS`/`MODE_TAGS` ~206-223, `STATUS_ROLE` ~178, `DEFAULT_EFFORT_BY_ROLE` ~292,
|
||||
`DIRECT_MANAGER_SYSTEM_PROMPT` ~662, `_manager_pipeline` ~7671). Read with grep or by line range; those files are huge.
|
||||
- **Your files:** only the "Agent 2" column of PLAN-v2 §6, your block in `config.py` / `.env.example`, and the
|
||||
migration `alembic/versions/a2b3c4d5e6f2_ai_workforce.py` (`down_revision = "a0f0e1d2c3b4"`).
|
||||
New tables go in `app/infrastructure/db/models_engine.py`. **`db/models.py` is locked**, so new task and
|
||||
employee fields go into new 1:1 side tables (see 1.3), never into `tasks`/`employees` columns.
|
||||
- **Never touch:** the locked files and the other agents' files. Need something there? → `CHANGE REQUEST` in your report.
|
||||
- **Do NOT commit, push, merge or rebase.** Never `cd` outside the worktree, never write to `/tmp`.
|
||||
- **After each phase:** run all checks (PLAN-v2 §7.4), write `docs/plan/reports/agent-2-phase-<K>.md`, then STOP and wait.
|
||||
|
||||
You consume: `ModelProviderPort` (existing), `ModelGrantPort`, `MemoryPort`, `RepoWorkspacePort`,
|
||||
`SandboxProviderPort` (existing), `DeliveryHooks.on_branch_pushed`, `EventPublisher`. Get them only from
|
||||
the builders in `app/application/container.py`. Tests use the Fake/Null versions plus a **scripted fake model**
|
||||
(you write it in `tests/engine_fakes.py`: returns queued responses in order and records the prompts it got).
|
||||
|
||||
You implement: `WorkforceBootstrapPort` (wired in `container_engine.py`).
|
||||
|
||||
**Important behaviour rules**
|
||||
- The AI never runs git. It edits files and runs commands inside the sandbox; the platform does git through `RepoWorkspacePort`.
|
||||
- Every model call writes a usage event exactly as the current `start_run` does (keep that code path).
|
||||
Manager chat calls count too (agybot forgot this).
|
||||
- Users never see model names. Store `ai_model_id` internally; APIs for non-owners return `tier` only.
|
||||
- Nothing in this engine deploys to production. The furthest it goes is "merged and pushed to `dev`".
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Engine core: protocol, state machine, pipelines (pure logic, no I/O)
|
||||
|
||||
Create package `app/application/engine/` (and `app/domain/workforce/` additions where the logic is pure):
|
||||
|
||||
- [ ] 1.1 `domain/workforce/pipeline.py`:
|
||||
- `Stage` enum: `pending_manager_review`, `pending_analysis`, `pending_implementation`, `pending_review`, `pending_test`, `pending_final_review`, `merge_request`, `completed`, plus side states `pending_user_decision`, `error_review`, `paused`, `cancelled`, `failed`.
|
||||
- `PIPELINES`: `plan_code_test` (default), `plan_code_review_test`, `code_test`, `analysis_only` (no worktree, no merge). Each is an ordered list of stages.
|
||||
- `TRANSITIONS: dict[Stage, frozenset[Stage]]` and `next_stage(pipeline, current, outcome)`.
|
||||
- `STAGE_ROLE`: stage → (role, mode). Manager modes are `triage`, `final`, `error`.
|
||||
- `resolve_pipeline(manager_choice, task_kind, app_default, has_active_reviewer)`: priority manager > kind > app default > `plan_code_test`. A review pipeline without an active reviewer falls back to `plan_code_test`.
|
||||
- `TASK_KINDS`: `feature`, `bugfix`, `refactor`, `docs`, `analysis` → default pipeline + risk.
|
||||
- `RoleKey.DEVELOPER` is our "programmer". Keep the existing enum values; do not rename them.
|
||||
- [ ] 1.2 `engine/protocol.py`: parser for `[[TAG attr="v" ...]]body[[/TAG]]` and self-closing `[[TAG attr="v"/]]`. Tolerant of whitespace; rejects unknown tags; returns typed dataclasses. Allowed tags per role/mode:
|
||||
- manager/triage: `ASSIGN`, `REJECT`, `ASK_USER`, `KNOWLEDGE`
|
||||
- manager/final: `COMPLETE`, `REVISE`, `KNOWLEDGE`
|
||||
- manager/error: `RESOLVE_ERROR`, `CANCEL`, `KNOWLEDGE`
|
||||
- manager/chat: `CREATE_TASK`, `PAUSE_TASK`, `RESUME_TASK`, `CANCEL_TASK`, `HIRE`, `UPDATE_EMPLOYEE`, `FIRE`, `KNOWLEDGE`
|
||||
- planner: `PLAN_SPEC`, `SPLIT`, `ASK_USER`
|
||||
- developer: `WORK_REPORT`
|
||||
- reviewer: `REVIEW_PASS`, `REVIEW_CHANGES`
|
||||
- tester: `TEST_PASS`, `TEST_FAIL`
|
||||
- tool tags (all working roles): `READ_FILE path`, `WRITE_FILE path` (body = content), `EDIT_FILE path` (body = `<<<OLD … === … NEW>>>` search/replace blocks), `LIST_DIR path`, `SEARCH pattern path`, `RUN cmd timeout`
|
||||
Disallowed tags are ignored and reported back to the model as an error line. Limits: max 3 `CREATE_TASK` per reply, max 3 team changes per turn, max 3 `KNOWLEDGE` per turn.
|
||||
- [ ] 1.3 Tables in `models_engine.py`:
|
||||
- `task_pipeline_state` (task_id PK/FK, pipeline, kind, risk, stage, cycle, max_cycles, branch, last_diff_hash, same_diff_count, retry_count, waiver_reason, parent_task_id, depends_on JSON, version int for optimistic updates, updated_at)
|
||||
- `employee_profiles` (employee_id PK/FK, ai_model_id, effort, roles CSV, hired_by `user|manager`, busy_task_id)
|
||||
- `engine_turns` (id, task_id, run_id, employee_id, role, mode, stage, prompt_chars, reply_excerpt ≤ 4000, tags JSON, tool_calls int, ok, error_kind, started_at, finished_at)
|
||||
- `gate_results` (id, task_id, gate `tests|diff_guard|no_progress|green|lint`, ok, detail ≤ 8000, created_at)
|
||||
Map the coarse `tasks.status` from the stage: new / running / waiting_approval / done / failed / cancelled. Existing APIs keep working.
|
||||
- [ ] 1.4 `engine/state.py`: `advance(task_id, expected_from, to)` with an optimistic check (`version`). It fails cleanly when another worker moved the task first. A failed or cancelled **dependency** moves the dependents to `pending_user_decision`, never leaving them stuck (agybot bug M1).
|
||||
- [ ] 1.5 Tests `tests/test_engine_protocol.py`, `tests/test_engine_pipeline.py`: every pipeline walks start→completed on the happy path; every illegal transition is refused; concurrent advance (two sessions) → exactly one wins; the dependency failure rule; tag parsing edge cases (nested brackets in a body, unknown tags, the per-role matrix, limits).
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Agent turn runner (one employee, one turn, real tools in the sandbox)
|
||||
|
||||
- [ ] 2.1 `engine/prompt.py`, a prompt builder with a **stable prefix** (cache-friendly), in this order:
|
||||
1. platform rules (short, yours)
|
||||
2. role protocol (allowed tags + format examples)
|
||||
3. toolkit description
|
||||
4. app profile (from `AppProfile`: language, framework, commands; never secrets)
|
||||
5. `MemoryPort.build_context(org, role, query=task text, budget_chars=2500)`
|
||||
6. task brief (≤ 1,200 chars) + stage-specific context (the planner's spec for the developer; the **diff only** for the reviewer; test output for the tester)
|
||||
7. the conversation so far in this turn (tool results)
|
||||
Hard cap on total characters; truncate oldest tool results first.
|
||||
- [ ] 2.2 `engine/tools.py`: executes tool tags **inside the sandbox** through `SandboxProviderPort`, rooted at `WorktreeRef.sandbox_path`.
|
||||
- Path safety: normalise; reject absolute paths outside the root, `..` escapes and anything under `.git`.
|
||||
- `RUN` has a timeout (default 120s, max 600s) and output capped (last 8,000 chars). The command is passed to the sandbox as-is (the sandbox is the security boundary), but `git push`, `git remote` and `git config` are refused with a message ("the platform handles git").
|
||||
- `EDIT_FILE` applies search/replace blocks and fails with a clear message when OLD is not found exactly once.
|
||||
- Each tool call writes a `tool_calls` row (existing table) with status `succeeded`/`failed`, so the existing approvals UI finally sees real executions.
|
||||
- [ ] 2.3 `engine/runner.py`, `run_turn(task, employee, role, mode) -> TurnResult`. Loop:
|
||||
- model call → parse tags → execute tools → feed results back → until a terminal protocol tag appears, or `ENGINE_MAX_TOOL_CALLS_PER_TURN`, or `ENGINE_TURN_TIMEOUT_SECONDS`.
|
||||
- Model = the employee's `ai_model_id`, which must be granted (`ModelGrantPort.is_granted`; if not, fail with `error_kind=auth` and tell the manager).
|
||||
- Error kinds `rate_limit`, `unavailable`, `network`, `timeout` are **transient** (retry later, up to 2 retries); `auth`, `model`, `protocol` are not.
|
||||
- Writes `engine_turns`, `run_steps` and usage events, and publishes `run.log` / `run.tool` / `run.status` (batched, PLAN-v2 §5).
|
||||
- Cancellation: `cancel_run` must stop the loop between steps.
|
||||
- [ ] 2.4 Tests `tests/test_engine_runner.py` with the scripted fake model + `FakeRepoWorkspace` + a fake sandbox (dict filesystem + scripted command results; put it in `tests/engine_fakes.py`). Cover: the write → run → report loop; path escape refused; `git push` refused; the tool-call cap; timeout; transient vs fatal errors; usage events counted per model call; an ungranted model refused; no secret from settings ever appears in a prompt (assert on the fake model's recorded prompts).
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Manager AI (orders → tasks → team)
|
||||
|
||||
- [ ] 3.1 Manager chat: rewrite `post_user_message` so the manager replies **and** may emit `CREATE_TASK` (title, kind, priority, depends_on, optional pipeline), `PAUSE_TASK` / `RESUME_TASK` / `CANCEL_TASK`, and `HIRE` / `UPDATE_EMPLOYEE` / `FIRE`. Tasks created this way get a `task_pipeline_state`, stage `pending_manager_review`, and are linked to the thread. The manager's prompt must ask for **atomic tasks that do not touch the same files**. Count chat usage.
|
||||
- [ ] 3.2 Team management: `HIRE role= tier= name=` picks a granted model matching the tier/capabilities (`ModelGrantPort.granted_models`). Respect max employees (setting + later org policy), max 3 changes per turn, and "the last manager cannot be fired". Publish `team.changed`. Users can still hire and fire from the UI (existing endpoints) but only choose **role + tier**, never a model.
|
||||
- [ ] 3.3 Triage turn (manager, mode triage): decide pipeline + assignee (`ASSIGN employee= pipeline=`), or `REJECT`, or `ASK_USER` (→ `pending_user_decision`, message posted in the thread). Default employee selection: least busy active employee with the needed role; `ASSIGN employee="name"` uses fuzzy name matching inside the org only.
|
||||
- [ ] 3.4 Final turn (manager, mode final): sees the plan, a diff summary, gate results and the test output → `COMPLETE` (→ `merge_request`) or `REVISE` (→ back to implementation, cycle+1). The error turn (mode error) handles failed gates and errors → `RESOLVE_ERROR` (retry with a hint) or `CANCEL`.
|
||||
- [ ] 3.5 `KNOWLEDGE` tags → `MemoryPort.record(kind=fact|rule, source=manager)`.
|
||||
- [ ] 3.6 `WorkforceBootstrapPort.bootstrap_team(org, project)`: creates one manager + one planner + one developer + one tester on the strongest/balanced/fast granted tiers (whatever is granted; if nothing is granted, create the employees "inactive" and post a system message "منتظر تخصیص مدل توسط مالک سامانه"). Idempotent. Wire it in `container_engine.py`.
|
||||
- [ ] 3.7 Tests `tests/test_engine_manager.py`: an order → ≤ 3 tasks with deps; HIRE limits; FIRE the last manager refused; ASK_USER path; idempotent bootstrap; cross-org isolation (the manager of org A can never assign an employee of org B, even by exact name).
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Pipeline execution, gates, commit, merge, push
|
||||
|
||||
- [ ] 4.1 `engine/scheduler.py`, `tick()`. Each tick:
|
||||
- finds tasks whose stage needs a turn and whose deps are completed, and a free employee with the role (one task per employee at a time)
|
||||
- honours org automation (if Agent 1's policy says `automation_enabled=false`, tasks wait in `pending_manager_review`; read it through `ModelGrantPort` when the CHANGE REQUEST lands, until then assume enabled)
|
||||
- runs turns concurrently with a per-org cap (3) and a global cap (setting)
|
||||
`workers/ai_runtime.py` calls `tick()` in its loop every `ENGINE_TICK_SECONDS` (keep the one-pass mode for cron). Safe with two worker processes thanks to the optimistic `advance`.
|
||||
- [ ] 4.2 Worktree lifecycle: on the first developer/planner turn, `RepoWorkspacePort.create_task_worktree(project, task, base="dev")`; the branch is `task/<task_id>`. `analysis_only` gets none. The worktree is removed after completion, cancellation or failure.
|
||||
- [ ] 4.3 Gates after `WORK_REPORT` (each writes a `gate_results` row and publishes `task.gate`):
|
||||
- **diff guard**: `RepoWorkspacePort.diff`. Fail on changes under `.git`, CI files (`.github/`, `.gitlab-ci.yml`, `.gitea/`), symlinks pointing outside the repo, or files > 1 MB. An empty diff for a code task also fails.
|
||||
- **no-progress**: hash the diff; 2 identical hashes in a row → `error_review`.
|
||||
- **tests**: run the app profile's `test_command` in the sandbox (timeout 600s), store the tail of the output. Red → back to the developer with the failure tail (one escalation to a stronger granted tier on the 2nd red), counting cycles. Green → tester stage.
|
||||
- **green gate**: the tester's `TEST_PASS` is accepted only if the tests gate is green, OR `TEST_PASS waive="reason"` with a non-empty reason (stored in `waiver_reason`, shown in UI).
|
||||
- `max_cycles` reached → `error_review`.
|
||||
- [ ] 4.4 Commit + merge + push on `merge_request`:
|
||||
- `commit` (message: `task/<id>: <title>`, author = employee display name + platform email), then `sync_with_base`, then re-run the tests if base changed (a semantic conflict is only caught by retests).
|
||||
- Then `merge_into(target="dev")` and `push(branch="dev")`.
|
||||
- A conflict → back to the developer with the conflicting files (max 3 times, then `error_review`).
|
||||
- On success: stage `completed`, `DeliveryHooks.on_branch_pushed(org, project, "dev", sha, source="engine")`, `MemoryPort.on_task_finished(TaskOutcome)` (never let its failure fail the task).
|
||||
- [ ] 4.5 Budget: before each turn, check the org credit wallet (existing billing service public functions; read, do not edit) and the per-task cap. Out of budget → `paused` with a Persian reason.
|
||||
- [ ] 4.6 Tests `tests/test_engine_flow.py`, the end-to-end engine with all fakes. The scripted model plays manager → planner → developer → tester → manager; the fake sandbox makes tests red once and then green. Assert the stage sequence, gate rows, cycles, the commit/merge/push calls on `FakeRepoWorkspace`, `on_branch_pushed` called once with the right sha, `on_task_finished` called once, and usage events. Plus the failure paths: no-progress, max cycles, merge conflict ×3, cancel mid-turn, a dependency failure.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — UI (orders, live task view, team)
|
||||
|
||||
- [ ] 5.1 `lib/api-engine.ts`: typed client functions for everything new (task detail with pipeline state, turns, gates, team with tier, hire/fire with role+tier).
|
||||
- [ ] 5.2 `/manager`: the chat shows task cards created by the manager inline (title, stage badge, link), live via `org:` + `thread:` channels; an "در انتظار تصمیم شما" banner for `pending_user_decision` with a reply box.
|
||||
- [ ] 5.3 `/tasks`: a board grouped by stage (columns: بررسی مدیر، تحلیل، پیادهسازی، بازبینی، تست، ادغام، انجامشده، نیازمند تصمیم) with a live update; filters; pause/resume/cancel (with confirm).
|
||||
- [ ] 5.4 `/tasks/[id]`:
|
||||
- a pipeline stepper (stages of its pipeline, current one highlighted)
|
||||
- the turn timeline (who, role, when, outcome)
|
||||
- a **live log** panel (subscribe to `run:<id>`, auto-scroll, pause-scroll)
|
||||
- gate results (green/red with expandable output)
|
||||
- a diff viewer (read-only, from the latest diff)
|
||||
- the waiver reason when present
|
||||
- a link to the pipeline run in `/app` once pushed (just a link with the sha; that page is Agent 3's)
|
||||
- [ ] 5.5 `/workforce`: employees with role, tier, status, current task; hire (role + tier) and fire with confirm. **No model names anywhere** for non-owners (test it).
|
||||
- [ ] 5.6 `/dashboard` + `/traces`: show active tasks, a stage distribution and recent completions; keep the existing widgets working.
|
||||
- [ ] 5.7 Web tests `src/test/engine-*.test.tsx`: board grouping, live log append and batching, no model name rendered (render with a fixture that contains a model name and assert it is absent).
|
||||
|
||||
---
|
||||
|
||||
## Final report (after Phase 5)
|
||||
`docs/plan/reports/agent-2-final.md`: summary, the engine architecture (one diagram in text), the tag protocol
|
||||
reference table, every endpoint added or changed, tables, settings used, CHANGE REQUESTs still open (expected:
|
||||
org policy in `ModelGrantPort`, the worker hook for memory expiry), manual QA steps, known limitations.
|
||||
@@ -0,0 +1,177 @@
|
||||
# TODO — Agent 3: Delivery & Infra (docker sandbox, git workspaces, live CI/CD, demo server, Caddy, SSH/FTP)
|
||||
|
||||
- **Worktree:** `/projects/smart-org-wt/agent3` · **Branch:** `agent3/delivery-infra` (already created, already installed)
|
||||
- **Read first:** `docs/plan/PLAN-v2.md` (all of it, especially D5–D9). Then read fully:
|
||||
`apps/api/app/application/ports/sandbox_provider.py`, `git_provider.py`, `domain_router.py`, `realtime.py`,
|
||||
`app/infrastructure/adapters/registry.py`, `mock.py`, `caddy.py`, `gitea.py`, `sandbox_remote.py`,
|
||||
`app/application/services/sandbox_service.py`, `project_service.py`, `app/domain/platform/rules.py`,
|
||||
`docs/current-infra-notes.md`, `tests/test_project_repo.py`, `tests/test_domain_isolation.py`.
|
||||
- **Reference (concepts only, NEVER import, never edit):** `/projects/agybot/caddy_manager.py`, `port_registry.py`,
|
||||
`ssh_manager.py`, `ftp_manager.py`, `git_manager.py`.
|
||||
- **Your files:** only the "Agent 3" column of PLAN-v2 §6 (including ALL of `app/infrastructure/adapters/`), your
|
||||
block in `config.py` / `.env.example`, and the migration `alembic/versions/a3b4c5d6e7f3_delivery_infra.py`
|
||||
(`down_revision = "a0f0e1d2c3b4"`). New tables go in `app/infrastructure/db/models_delivery.py`. **`db/models.py`
|
||||
is locked**, so extra data about existing sandbox tables goes into new side tables.
|
||||
- **Never touch:** the locked files and the other agents' files. Need something there? → `CHANGE REQUEST` in your report.
|
||||
- **Do NOT commit, push, merge or rebase.** Never `cd` outside the worktree, never write to `/tmp`: tests use pytest's
|
||||
`tmp_path` (allowed) and the worktree's `./_scratch/`.
|
||||
- **After each phase:** run all checks (PLAN-v2 §7.4), write `docs/plan/reports/agent-3-phase-<K>.md`, then STOP and wait.
|
||||
|
||||
You implement: `RepoWorkspacePort`, `DeliveryHooks`, `DeployTargetTester`, plus a docker `SandboxProviderPort`
|
||||
adapter. Wire them in `container_delivery.py` and `registry.py` behind settings (mock remains the default
|
||||
for tests).
|
||||
|
||||
**Security rules specific to you**
|
||||
- Nothing runs on the host except: `git` (via the workspace adapter, hardened per D6), the docker/podman CLI, and Caddy reload (existing adapter).
|
||||
- Every subprocess: `asyncio.create_subprocess_exec` (never `shell=True`), `start_new_session=True`, kill the process group on timeout, drain stdout and stderr concurrently (avoid the 64 KB pipe deadlock), and pass explicit `env` (never inherit secrets).
|
||||
- Credentials never appear in argv, URLs, logs, exceptions, realtime events or DB columns other than `*_encrypted`. Use `GIT_ASKPASS` with a 0700 helper script that reads a 0600 temp file in the worktree's `./_scratch`-like private dir under `SANDBOX_WORKSPACES_ROOT`, deleted in `finally`.
|
||||
- Ports and hosts must pass `app/domain/platform/rules.py` (protected ports, single-label hosts, localhost targets).
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Docker sandbox adapter
|
||||
|
||||
- [ ] 1.1 `adapters/docker_sandbox.py`, `DockerSandboxAdapter` implementing `SandboxProviderPort` with the CLI in `SANDBOX_DOCKER_BIN` (docker or podman, the same flags):
|
||||
- `create_workspace`: creates a host dir `SANDBOX_WORKSPACES_ROOT/<org_id>/<project_id>/work` (0700, owned by the platform user).
|
||||
- `start_session`: `docker run -d` the **workspace container**:
|
||||
- image from the app profile language: `node:20-bookworm`, `python:3.13-bookworm`, `php:8.3-cli`, `golang:1.23`, fallback `debian:bookworm`; make the mapping a dict in your config block / domain
|
||||
- `--user 1000:1000 --cap-drop ALL --security-opt no-new-privileges --pids-limit 512 --cpus $SANDBOX_DEFAULT_CPUS --memory ${MB}m`
|
||||
- `--read-only` with `--tmpfs /tmp` and `--tmpfs /home/app`
|
||||
- bind-mount **only the work dir** (never the git dir) to `/workspace`
|
||||
- label `smart-org.org=<id>`, `smart-org.project=<id>`, `smart-org.role=workspace`
|
||||
- network: the default bridge (package installs need egress), with no published ports
|
||||
- `run_command`: `docker exec -w <cwd> <ctr> sh -lc <cmd>`, with timeout, output cap, exit code, and an optional async `on_output(chunk)` callback for streaming (add it as an adapter-level extra method; the port stays unchanged).
|
||||
- `read_file` / `write_file`: through the host bind mount with path safety (`resolve()` stays inside the work dir; reject symlinks that escape).
|
||||
- `pause` / `resume` / `destroy`: stop/start/rm. `snapshot`: `docker commit` (tagged per org) or raise a clear "not supported" error, your choice, documented.
|
||||
- `get_preview_url`: delegated to the domain layer (Phase 4).
|
||||
- [ ] 1.2 `registry.py`: `SANDBOX_PROVIDER=docker` → this adapter. The mock stays the default.
|
||||
- [ ] 1.3 The `sandbox_gateway` worker: also reap containers with the `smart-org.*` label that have no live session row (orphans), audited.
|
||||
- [ ] 1.4 Hide infra from non-owners:
|
||||
- `sandbox.py` command/file endpoints become **owner-only** (or admin+ of the org with a setting off by default; prefer owner-only).
|
||||
- Keep the read-only status endpoints for org members.
|
||||
- Update `components/sandbox-panel.tsx` to show only status to non-owners.
|
||||
- Update existing tests that assumed member access, and explain each change in your report.
|
||||
- [ ] 1.5 Tests `tests/test_docker_sandbox.py`: unit tests with a **fake process runner** (inject the subprocess function) that assert the exact argv (hardening flags present, no secrets, mount of the work dir only), timeout kill, output cap, path escape refused. Integration tests marked `@pytest.mark.docker`, skipped unless `SANDBOX_DOCKER_IT=1`: real start → exec `echo` → write/read file → destroy.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Git workspaces (`RepoWorkspacePort`, the platform does git, D6)
|
||||
|
||||
- [ ] 2.1 `adapters/git_workspace.py`, `LocalGitWorkspaceAdapter`. Layout per project under `SANDBOX_WORKSPACES_ROOT/<org>/<project>/`:
|
||||
- `git/`: a bare-ish gitdir, **never mounted**
|
||||
- `work/`: the main working tree for the dev branch, mounted at `/workspace`
|
||||
- `work/.worktrees/<task_id>/`: task worktrees, visible in the container at `/workspace/.worktrees/<task_id>`
|
||||
- `WorktreeRef.sandbox_path` = that in-container path
|
||||
Note: the `.worktrees` dir must be excluded via `info/exclude` in the gitdir, not the user's `.gitignore`.
|
||||
- [ ] 2.2 Every git call is `git --git-dir=<gitdir> --work-tree=<path> -c core.hooksPath=/dev/null -c core.fsmonitor=false -c core.quotePath=false -c protocol.file.allow=never -c safe.directory=*`, with env `GIT_CONFIG_NOSYSTEM=1`, `HOME=<private dir>`, `GIT_TERMINAL_PROMPT=0`. Never trust the `.git` *file* inside a worktree; always pass explicit dirs.
|
||||
- Status: `status --porcelain=v1 -uall -z`.
|
||||
- One `asyncio.Lock` per project id held for every mutating operation (agybot M14). Remove a stale `index.lock` only if older than 30s.
|
||||
- [ ] 2.3 Remotes:
|
||||
- **platform** git: the URL and credentials come from the existing Gitea adapter / `project_git_credentials` (decrypt via the existing crypto helper).
|
||||
- **external** git: `project_external_git` (token → `GIT_ASKPASS`; ssh_key → temp key file 0600 + `GIT_SSH_COMMAND="ssh -i <key> -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=<per-project known_hosts>"`; store `known_hosts` on the first `test_remote` = TOFU, in the existing column).
|
||||
- `test_remote`: `git ls-remote --heads` with a 20s timeout; returns ok, default branch, branch list, and a friendly error (auth failed / not found / unreachable) **without echoing the URL credentials or stderr secrets**.
|
||||
- SSRF check on external hosts (resolve DNS, reject private, loopback and link-local unless `DEPLOY_ALLOW_PRIVATE_HOSTS`); put the check in `domain/platform/rules.py` (yours). It runs inside `test_remote` and before every deploy connection; Agent 1 only does format validation and relies on your check.
|
||||
- [ ] 2.4 Implement every `RepoWorkspacePort` method per PLAN-v2 §5.
|
||||
- `ensure_workspace` clones on first use, fetches afterwards. It creates `dev` from the default branch if missing; for platform repos also push `dev`.
|
||||
- `merge_into` does a `--no-ff` merge in the main work tree under the lock and returns the conflict file list on failure, after `merge --abort`. Never leave a half-merge (agybot H3).
|
||||
- `push` returns ok/rejected (non-fast-forward → fetch + report, never force).
|
||||
- [ ] 2.5 Wire it in `container_delivery.py` (setting `GIT_WORKSPACE=local|fake`, fake default in tests).
|
||||
- [ ] 2.6 Tests `tests/test_git_workspace.py` with **real git** and a local bare repo in `tmp_path` as "remote" (no network):
|
||||
- the full flow ensure → worktree → write → diff (new dir appears thanks to `-uall`) → commit → sync → merge → push → head
|
||||
- a conflict path leaves a clean tree
|
||||
- hooks are NOT executed (plant a `post-commit` hook that writes a file and assert it is absent)
|
||||
- a `.git` file tampered to point elsewhere is ignored
|
||||
- credentials are never in argv (capture the runner calls)
|
||||
- concurrent merges are serialised
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Live CI/CD + demo server
|
||||
|
||||
- [ ] 3.1 Tables (`models_delivery.py`):
|
||||
- `pipeline_runs` (id, org, project, trigger `engine|poll|manual`, branch, sha, status `queued|running|succeeded|failed|cancelled|superseded`, started/finished, error_summary)
|
||||
- `pipeline_steps` (id, run_id, index, name, status, started/finished, exit_code, log_tail ≤ 64 KB)
|
||||
- `releases` (id, org, project, run_id, sha, artifact_path, created_at, is_demo_live)
|
||||
- `app_runtimes` (project_id PK, demo_container_id, host_port, demo_host, status, current_release_id, updated_at)
|
||||
- [ ] 3.2 `delivery_service.py` implementing `DeliveryHooks`:
|
||||
- `on_app_created`: allocate a demo port in `DEMO_PORT_START..END` (reuse the existing preview port rules), claim the demo host `<org-slug>.<PREVIEW_ROOT_DOMAIN>` through the existing `DomainRouterPort`, create `app_runtimes`, `ensure_workspace`, and queue a first pipeline run if the repo has a `dev` head.
|
||||
- `on_branch_pushed`: queue a run (one active run per project; a newer queued run supersedes an older queued one).
|
||||
- [ ] 3.3 `workers/delivery.py`: loop (`--loop`) that executes queued runs and polls remotes every `DELIVERY_POLL_SECONDS` (`head` vs the last run sha) for pushes made outside the engine (users pushing to their own git).
|
||||
- Steps, each in a **fresh build container** (same image mapping and hardening, work dir = a clean checkout of the sha in a per-run dir):
|
||||
1. `checkout`
|
||||
2. `setup` (setup_command)
|
||||
3. `build` (build_command, skipped if empty)
|
||||
4. `test` (test_command, skipped if empty)
|
||||
5. `package` (tar.gz of the run dir excluding `.git` and `node_modules` → `ARTIFACTS_ROOT/<org>/<project>/<sha>.tar.gz`)
|
||||
6. `deploy_demo`: start the new demo container with `start_command`, port mapping `127.0.0.1:<host_port>:<app_port>`, the same hardening, label `role=demo`
|
||||
7. `healthcheck`: HTTP GET `http://127.0.0.1:<host_port><healthcheck_path>` with 20 retries × 3s; any 2xx/3xx = ok
|
||||
8. `route`: `configure_reverse_proxy` for the demo host → `127.0.0.1:<host_port>`
|
||||
- **Blue/green:** start the new container on a temporary second port, healthcheck it, then switch the Caddy route, then stop the old one. If any step fails, the old demo keeps running untouched and the run is `failed` with a Persian `error_summary`.
|
||||
- [ ] 3.4 Live logs: every step streams output via the `on_output` callback → batched `pipeline.log` events on `org:<id>` (≤ 1 event / 500 ms, ≤ 200 lines), plus `pipeline.step` and `pipeline.status`; `release.live` when switched. Store the step `log_tail` (last 64 KB). Pass all output through `app/core/redact.py` before publishing or storing.
|
||||
- [ ] 3.5 Router `delivery.py` (org member read, admin+ actions): list runs, run detail with steps and tails, `POST /delivery/runs` (manual re-run of the dev head, admin+), cancel the queued/running run, list releases, rollback the demo to a previous release (admin+, confirm, audit), and the runtime status (demo URL, live release, health).
|
||||
- [ ] 3.6 Tests `tests/test_delivery.py` with the fake runner + `FakeRepoWorkspace` + mock domain router:
|
||||
- the happy path step sequence and events
|
||||
- a failed build keeps the old demo
|
||||
- healthcheck failure → no route switch
|
||||
- supersede logic
|
||||
- poll detects an external push
|
||||
- logs redacted (plant a secret in the build output)
|
||||
- tenant isolation on every endpoint
|
||||
- an optional `@pytest.mark.docker` end-to-end with a tiny static app (`python -m http.server`)
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Domains (Caddy) owned by the platform owner
|
||||
|
||||
- [ ] 4.1 `admin_domains.py` (owner only):
|
||||
- list all routes (host → org/app → target port → status)
|
||||
- attach an extra host to an org's app: a single label under the root via the existing rules, OR a full custom FQDN when `ALLOW_CUSTOM_DOMAINS=true` (strict FQDN regex, no wildcard, no protected hosts such as `git`, `mcp`, `panel`, `api`, root)
|
||||
- detach (confirm + audit)
|
||||
- re-sync all routes from the DB to Caddy (idempotent)
|
||||
Users cannot add domains (the existing member-facing preview-domain endpoints in `projects.py` become owner-only, and the tests are updated with reasons).
|
||||
- [ ] 4.2 The Caddy adapter keeps validate → revert → reload. Add a `routes_fingerprint` check so re-sync does nothing when nothing changed.
|
||||
- [ ] 4.3 The demo host is created automatically in `on_app_created` (Phase 3). Record the MCP gateway host suggestion (`mcp.<root>` → `MCP_GATEWAY_PORT`) as a protected system route that the owner can enable from this page.
|
||||
- [ ] 4.4 Tests `tests/test_admin_domains.py`: non-owner forbidden; protected hosts refused; invalid FQDNs refused; detach audited; re-sync idempotent (mock router call count).
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Production deploy over SSH / SFTP / FTP / FTPS (always human-triggered)
|
||||
|
||||
- [ ] 5.1 `adapters/deploy_ssh.py` (asyncssh), `DeployTargetTester` + `SshDeployer`:
|
||||
- Test: connect (15s timeout) → host-key TOFU (store the fingerprint on the first success; afterwards a mismatch = hard fail "کلید میزبان تغییر کرده"), check `remote_path` exists and is writable.
|
||||
- Deploy: upload the release artifact → `remote_path/releases/<ts>-<sha7>/` → extract → run `post_deploy_command` with `cwd` there (timeout 600s, streamed output) → atomically switch the `remote_path/current` symlink → keep the last 5 releases.
|
||||
- Rollback = re-point `current` to the previous release + rerun post_deploy.
|
||||
- [ ] 5.2 `adapters/deploy_ftp.py` (aioftp; FTPS via explicit TLS), `DeployTargetTester` + `FtpDeployer`:
|
||||
- Test: login, cwd `remote_path`, write and delete a probe file.
|
||||
- Deploy: upload the build output (the build dir is set by the framework catalog default or `dist/`, `build/`, `public/`, else the whole artifact) into `remote_path` after downloading a **backup listing** of the files it overwrites into `ARTIFACTS_ROOT/.../ftp-backup-<ts>.tar.gz`.
|
||||
- Rollback = re-upload that backup. Document the limits of FTP (no atomic switch).
|
||||
- [ ] 5.3 SSRF check for every target host (the Phase 2.3 helper). A combined `DeployTargetTester` dispatching on `kind`, wired in `container_delivery.py`.
|
||||
- [ ] 5.4 `deploy_service.py` + endpoints in `delivery.py`:
|
||||
- `POST /delivery/production/deploy` with `release_id` (default: the live demo release). Org **owner/admin only**; requires a `confirm: "<org-slug>"` field; one production deploy per project at a time.
|
||||
- Status: `deploy.status` events and a `production_deploys` table (id, release, target, status, started/finished, log_tail, rolled_back_from).
|
||||
- `POST /delivery/production/rollback`.
|
||||
- Every action audited. Credentials decrypted only inside the adapter call.
|
||||
- [ ] 5.5 Tests `tests/test_deploy.py`: fake SSH/FTP clients injected. Cover: the release/symlink sequence, keep-5 pruning, rollback, host-key mismatch refused, SSRF refused, a member (non-admin) refused, the confirm slug mismatch refused, no password in logs/events/DB columns other than encrypted ones. Optional integration (`DEPLOY_IT=1`) against a local `sshd`/`pure-ftpd` container.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — UI: the "Application" page (replaces `/projects`)
|
||||
|
||||
- [ ] 6.1 `(app)/app/page.tsx` (+ components under `(app)/app/_components/`):
|
||||
- **Header:** app name, type/language/framework badges, demo URL (open button), live release sha + time, health dot.
|
||||
- **Pipeline runs:** list with status; clicking one opens the run view: a step list with status and duration, and a **live log console** (subscribe `org:<id>`, filter by run id, ANSI colours stripped, auto-scroll with pause, and a "copy" button).
|
||||
- **Releases:** history; "rollback demo" (admin+, confirm).
|
||||
- **Production:** target summary (kind, host, path, last test result; no secrets), the "استقرار در production" button (admin+, modal requiring the org slug), a deploy live log, production history + rollback.
|
||||
- **Git:** repo link (platform: Gitea URL; external: the URL without credentials), `dev` head sha, last poll time. No branch/PR/port controls for users.
|
||||
- [ ] 6.2 Remove `(app)/projects` (redirect `/projects*` → `/app`) and drop the member-facing port/branch/PR/command controls. Owner-only debug tools move to `/admin/sandbox`.
|
||||
- [ ] 6.3 `/admin/sandbox` (owner): running containers per org (workspace/demo/build), CPU/memory limits per org (existing `sandbox_resource_limits`), stop/restart a container (confirm + audit), and an image mapping view. `/admin/domains` (owner): the Phase 4 UI.
|
||||
- [ ] 6.4 `lib/api-delivery.ts`: typed client for all of the above.
|
||||
- [ ] 6.5 Web tests `src/test/delivery-*.test.tsx`: the log console batching and auto-scroll, the confirm-slug modal, no secret fields rendered, non-admin sees no deploy button.
|
||||
|
||||
---
|
||||
|
||||
## Final report (after Phase 6)
|
||||
`docs/plan/reports/agent-3-final.md`: summary, host requirements (docker/podman, Caddy, dirs and permissions,
|
||||
the sudo/systemd needs of the Caddy reload), every setting, every endpoint added or changed (method + path + role),
|
||||
tables, CHANGE REQUESTs still open, manual QA script (wizard app → push → watch live pipeline → open demo URL →
|
||||
deploy to a local SSH test server → rollback), known limitations.
|
||||
Reference in New Issue
Block a user