feat(integration): wire v2 workstreams together (engine policy/budget, sandbox, tenant checks, order route, docs)
Engine honours org policy and monthly budget, legacy runner skips engine tasks, sandbox session provisioned on demand, project fallback for threads, reviewer fallback, tenant checks on project ids, memory reflection respects grants, expiry worker, MCP pipeline_status, login redirect, sharing settings, e2e smoke. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
mjalmoudy94
co-authored by
Claude Sonnet 5.5
Claude Sonnet 5.5
parent
36564e3f2b
commit
fef5a9deab
@@ -278,3 +278,28 @@ revision when the branches are merged.
|
||||
wizard → app created → demo domain → manager order → task runs → tests → push → pipeline → demo live
|
||||
→ production deploy (to a local SSH/FTP test server).
|
||||
5. Update `README.md`, `TODO.md` and the architecture doc, then commit.
|
||||
|
||||
## 9. Status (2026-10-14)
|
||||
|
||||
Written on 2026-10-14 from the code, an alembic run on a scratch SQLite database, and the statuses in
|
||||
`todo-finish-plan.md`. A box is ticked only when its task is `status: done` or its check passed.
|
||||
|
||||
- [ ] §8.1–8.3 (partly done)
|
||||
- [x] Alembic: single head `b9c0d1e2f3a4`. `upgrade head`, a downgrade to `d4e5f6a7b8c9` and `upgrade head` run cleanly, and `alembic check` reports no new operations.
|
||||
- [x] The `agent2/ai-workforce` and `agent3/delivery-infra` branches are contained in `dev`.
|
||||
- [ ] `agent1/org-platform` (6 commits, `0016bb4`..`051bf1b`) is not an ancestor of `dev`, and its committed tree differs from `dev` in 138 files. Its phases reached `dev` as separate commits (`5095a29`, `5fb85d6`, `dee2f72`). Open: confirm that `dev` has everything from Agent 1, then merge or retire the branch.
|
||||
- [x] §8.4 done through T1–T26; T27 is this docs pass.
|
||||
- [x] Double execution fixed (T2): the legacy run drain skips tasks that own a pipeline state.
|
||||
- [x] Zero-wallet pause fixed (T3): only an overdrawn wallet pauses a task.
|
||||
- [x] The engine honours the owner's org policy: automation switch, monthly credit budget and `max_employees` (T1, T4, T5).
|
||||
- [x] A new organization gets a sandbox session that the engine starts on demand (T6).
|
||||
- [x] End-to-end smoke test `tests/test_v2_e2e_smoke.py` (T7; closer to reality in T26): a thread made without a project, the real `AiRuntimeWorker.drain()`, and a sandbox session that the engine starts. The production-deploy leg is not in it.
|
||||
- [x] Review fixes from the integration pass: the manager gives a task the org's project when the thread has none (T20); a review pipeline is chosen only with an active reviewer (T21); the scheduler tick reads a wide window, so skipped tasks cannot starve other organizations (T22); archived (fired) employees no longer count toward the hire cap (T23); a project id from another organization gets a 404 on threads and tasks, and the engine never uses another tenant's sandbox (T24); an explicit `next` (a pending invite) beats the onboarding redirect at login (T25).
|
||||
- [x] Fixed: the HTTP message route `POST /api/organizations/{org}/threads/{thread}/messages` built its manager without pipeline states and employee profiles (`workforce_service._manager_turns`), so a manager order sent through it made tasks the engine never picked up. The manager now gets both, as `container_engine._build_manager` gives them. `tests/test_integration_order_route.py` posts the order over HTTP and checks its pipeline state.
|
||||
- [ ] The production-deploy leg of the smoke test stays manual and has not been run. See `docs/plan/reports/agent-3-final.md` §6, steps 7–8, which need `DEPLOY_IT=1`.
|
||||
- [x] Contract extension `ModelGrantPort.get_org_policy` (T1), with `DefaultOrgPolicy` as the default.
|
||||
- [ ] Left to Agent 2 phase 5 (owned by the live phase-5 session; source: "Left to Agent 2" in `todo-finish-plan.md`):
|
||||
- Web: `apps/web/src/lib/api-engine.ts`, `(app)/manager/**`, `(app)/tasks/**`, `(app)/workforce/**`, `(app)/dashboard/**`, `(app)/traces/**`, `src/test/engine-*.test.tsx`.
|
||||
- Contracts: the engine event types in `@agy/contracts` (`task.step`, `task.gate`, `run.log`, `run.tool`, `team.changed`).
|
||||
- Docs: tick `docs/plan/TODO-agent-2-ai-workforce.md` and write `docs/plan/reports/agent-2-final.md`.
|
||||
- Open decisions: direct tasks (HTTP `POST /tasks`, MCP `tasks_create`) get no pipeline state and still run through the legacy `start_run`. A `paused` task has no resume path in the API yet.
|
||||
|
||||
@@ -6,55 +6,55 @@ file. When this is done, `dev` has a foundation commit and three worktrees exist
|
||||
Read `docs/plan/PLAN-v2.md` §4–§6 first. Everything here is mechanical; no product logic.
|
||||
|
||||
## F1. Environment
|
||||
- [ ] `pnpm install` at root; `pnpm api:venv`; confirm the baseline: `pytest -q`, `ruff`, `mypy`, `pnpm test`, `tsc`. Record the counts.
|
||||
- [ ] Add API deps to `apps/api/pyproject.toml`: `asyncssh`, `aioftp`, `mcp` (official SDK). Install and re-run the baseline.
|
||||
- [ ] `pnpm install` at root; `pnpm api:venv`; confirm the baseline: `pytest -q`, `ruff`, `mypy`, `pnpm test`, `tsc`. Record the counts. — open: apps/api/.venv in the main checkout is incomplete (bin/ is empty, no pyvenv.cfg), so `pnpm api:venv` cannot be confirmed there; the foundation-time counts are not written down anywhere in the repo.
|
||||
- [x] Add API deps to `apps/api/pyproject.toml`: `asyncssh`, `aioftp`, `mcp` (official SDK). Install and re-run the baseline. (baseline recorded at foundation time)
|
||||
|
||||
## F2. DB foundation (`db/models.py` + migration `a0f0e1d2c3b4_v2_foundation.py`, down_revision `d4e5f6a7b8c9`)
|
||||
- [ ] `projects`: add app-profile columns (all nullable or defaulted): `app_type`, `language`, `framework`,
|
||||
- [x] `projects`: add app-profile columns (all nullable or defaulted): `app_type`, `language`, `framework`,
|
||||
`runtime_version`, `package_manager`, `setup_command`, `build_command`, `test_command`, `start_command`,
|
||||
`app_port` (int), `healthcheck_path` (default `/`), `git_source` (`platform|external`, default `platform`),
|
||||
`external_repo_url`, `wizard_completed_at`.
|
||||
- [ ] `projects`: `UniqueConstraint("organization_id", name="uq_project_one_per_org")`. Fix the seed and any test that creates 2 projects in one org.
|
||||
- [ ] New table `project_external_git` (project_id PK/FK, auth_type `token|ssh_key|none`, username, secret_encrypted, known_hosts, updated_at).
|
||||
- [ ] New table `deploy_targets` (id, organization_id, project_id, kind `ssh|sftp|ftp|ftps`, host, port, username,
|
||||
- [x] `projects`: `UniqueConstraint("organization_id", name="uq_project_one_per_org")`. Fix the seed and any test that creates 2 projects in one org.
|
||||
- [x] New table `project_external_git` (project_id PK/FK, auth_type `token|ssh_key|none`, username, secret_encrypted, known_hosts, updated_at).
|
||||
- [x] New table `deploy_targets` (id, organization_id, project_id, kind `ssh|sftp|ftp|ftps`, host, port, username,
|
||||
auth_type `password|key`, secret_encrypted, remote_path, post_deploy_command, host_fingerprint, is_active,
|
||||
last_tested_at, last_test_ok, last_test_message, created_at). At most one active target per project.
|
||||
- [ ] New table `org_model_grants` (id, organization_id, ai_model_id, daily_credit_cap nullable, is_active, created_by, created_at; unique org+model).
|
||||
- [ ] At the end of `models.py`, import `models_org`, `models_engine`, `models_delivery` (three empty modules with a docstring naming the owner agent).
|
||||
- [ ] `alembic upgrade head`, `alembic downgrade -1`, `alembic upgrade head`, `alembic check`, all clean.
|
||||
- [x] New table `org_model_grants` (id, organization_id, ai_model_id, daily_credit_cap nullable, is_active, created_by, created_at; unique org+model).
|
||||
- [x] At the end of `models.py`, import `models_org`, `models_engine`, `models_delivery` (three empty modules with a docstring naming the owner agent). (empty at foundation time)
|
||||
- [x] `alembic upgrade head`, `alembic downgrade -1`, `alembic upgrade head`, `alembic check`, all clean. (re-run 2026-10-14 at head: upgrade head, downgrade to d4e5f6a7b8c9, upgrade head, check, all clean)
|
||||
|
||||
## F3. Contracts (`app/application/ports/`), exactly as in PLAN-v2 §5
|
||||
- [ ] `repo_workspace.py` (+ dataclasses `WorkspaceRef`, `WorktreeRef`, `DiffResult`, `MergeOutcome`, `PushOutcome`, `RemoteCredentials`, `RemoteCheck`) and an in-memory `FakeRepoWorkspace` for tests.
|
||||
- [ ] `delivery_hooks.py` + `NullDeliveryHooks` (records calls in a list for tests).
|
||||
- [ ] `deploy_target.py` (`DeployTargetSpec`, `DeployCheck`) + `FakeDeployTargetTester` (ok unless host contains "fail").
|
||||
- [ ] `memory.py` (`TaskOutcome`) + `NullMemory` (empty context, record returns None).
|
||||
- [ ] `model_grants.py` (`GrantedModel`) + `AllActiveModelsGrants` (today's behaviour: every active model).
|
||||
- [ ] `workforce_bootstrap.py` + `NullWorkforceBootstrap`.
|
||||
- [ ] `app/domain/projects/profile.py`: frozen dataclass `AppProfile` + `AppProfile.from_row(...)`.
|
||||
- [x] `repo_workspace.py` (+ dataclasses `WorkspaceRef`, `WorktreeRef`, `DiffResult`, `MergeOutcome`, `PushOutcome`, `RemoteCredentials`, `RemoteCheck`) and an in-memory `FakeRepoWorkspace` for tests.
|
||||
- [x] `delivery_hooks.py` + `NullDeliveryHooks` (records calls in a list for tests).
|
||||
- [x] `deploy_target.py` (`DeployTargetSpec`, `DeployCheck`) + `FakeDeployTargetTester` (ok unless host contains "fail").
|
||||
- [x] `memory.py` (`TaskOutcome`) + `NullMemory` (empty context, record returns None).
|
||||
- [x] `model_grants.py` (`GrantedModel`) + `AllActiveModelsGrants` (today's behaviour: every active model).
|
||||
- [x] `workforce_bootstrap.py` + `NullWorkforceBootstrap`.
|
||||
- [x] `app/domain/projects/profile.py`: frozen dataclass `AppProfile` + `AppProfile.from_row(...)`.
|
||||
|
||||
## F4. Wiring stubs
|
||||
- [ ] `app/application/container_org.py`, `container_engine.py`, `container_delivery.py`: each holds the builder functions for its port(s), returning the Null/Fake implementations for now. `container.py` re-exports them.
|
||||
- [ ] Empty routers (`router = APIRouter()`) registered in `rpc_app.py`: `app_setup`, `memory`, `mcp_tokens`, `admin_grants`, `delivery`, `admin_domains`.
|
||||
- [ ] `app/workers/delivery.py` stub (one no-op pass) registered in `workers/__main__.py` as `delivery`; `services/delivery-worker/README.md` + `pnpm worker:delivery` script.
|
||||
- [ ] `services/mcp-gateway/README.md` placeholder + `pnpm mcp:dev` script (points at `app.interfaces.mcp.server:main`, stub module that raises a clear "not implemented yet").
|
||||
- [ ] `config.py` and `.env.example`: three marked blocks `# >>> agent-1 … # <<< agent-1` (likewise 2 and 3). Pre-fill the obvious settings:
|
||||
- [x] `app/application/container_org.py`, `container_engine.py`, `container_delivery.py`: each holds the builder functions for its port(s), returning the Null/Fake implementations for now. `container.py` re-exports them. (foundation stubs; the builders now return the real services)
|
||||
- [x] Empty routers (`router = APIRouter()`) registered in `rpc_app.py`: `app_setup`, `memory`, `mcp_tokens`, `admin_grants`, `delivery`, `admin_domains`. (routers now hold their endpoints)
|
||||
- [x] `app/workers/delivery.py` stub (one no-op pass) registered in `workers/__main__.py` as `delivery`; `services/delivery-worker/README.md` + `pnpm worker:delivery` script. (the stub is now the real worker pass)
|
||||
- [x] `services/mcp-gateway/README.md` placeholder + `pnpm mcp:dev` script (points at `app.interfaces.mcp.server:main`, stub module that raises a clear "not implemented yet"). (the module now runs the real gateway)
|
||||
- [x] `config.py` and `.env.example`: three marked blocks `# >>> agent-1 … # <<< agent-1` (likewise 2 and 3). Pre-fill the obvious settings:
|
||||
- agent-3: `SANDBOX_DOCKER_BIN=docker`, `SANDBOX_WORKSPACES_ROOT`, `SANDBOX_DEFAULT_CPUS=1`, `SANDBOX_DEFAULT_MEMORY_MB=1024`, `DEMO_PORT_START=8100`, `DEMO_PORT_END=8999`, `DELIVERY_POLL_SECONDS=60`, `DEPLOY_ALLOW_PRIVATE_HOSTS=false`, `ARTIFACTS_ROOT`.
|
||||
- agent-2: `ENGINE_MAX_TOOL_CALLS_PER_TURN=40`, `ENGINE_TURN_TIMEOUT_SECONDS=900`, `ENGINE_MAX_CYCLES=6`, `ENGINE_MAX_EMPLOYEES=20`, `ENGINE_TICK_SECONDS=10`.
|
||||
- agent-1: `MCP_GATEWAY_HOST=127.0.0.1`, `MCP_GATEWAY_PORT=8765`, `MCP_TOKEN_TTL_DAYS=90`, `MEMORY_ORG_CHAR_CAP=60000`, `PLATFORM_INVITE_TTL_HOURS=72`.
|
||||
|
||||
## F5. Web stubs
|
||||
- [ ] `lib/api-org.ts`, `lib/api-engine.ts`, `lib/api-delivery.ts`: each imports `apiRequest` from `./api` and exports an empty object `orgApi` / `engineApi` / `deliveryApi`.
|
||||
- [ ] `nav.ts`: rename the projects entry to `{ href: "/app", label: "اپلیکیشن", hint: "دمو، استقرار، Git" }`, rename knowledge to `{ href: "/memory", label: "حافظهٔ سازمان", ... }`; admin entries `/admin/model-grants` (مدلهای سازمانها), `/admin/domains` (دامنهها), `/admin/sandbox` (سندباکسها), `/admin/mcp` (MCP).
|
||||
- [ ] Placeholder pages (using `PlaceholderPage`) for `(app)/app`, `(app)/memory`, `(app)/admin/model-grants`, `(app)/admin/domains`, `(app)/admin/sandbox`, `(app)/admin/mcp`. `(app)/projects` keeps working until Agent 3 removes it.
|
||||
- [x] `lib/api-org.ts`, `lib/api-engine.ts`, `lib/api-delivery.ts`: each imports `apiRequest` from `./api` and exports an empty object `orgApi` / `engineApi` / `deliveryApi`. (empty objects at foundation time)
|
||||
- [x] `nav.ts`: rename the projects entry to `{ href: "/app", label: "اپلیکیشن", hint: "دمو، استقرار، Git" }`, rename knowledge to `{ href: "/memory", label: "حافظهٔ سازمان", ... }`; admin entries `/admin/model-grants` (مدلهای سازمانها), `/admin/domains` (دامنهها), `/admin/sandbox` (سندباکسها), `/admin/mcp` (MCP).
|
||||
- [x] Placeholder pages (using `PlaceholderPage`) for `(app)/app`, `(app)/memory`, `(app)/admin/model-grants`, `(app)/admin/domains`, `(app)/admin/sandbox`, `(app)/admin/mcp`. `(app)/projects` keeps working until Agent 3 removes it. (placeholders at foundation time)
|
||||
|
||||
## F6. Finish
|
||||
- [ ] Baseline checks are all green with the same counts as F1 (plus any new tests).
|
||||
- [ ] Commit on `dev`: `v2 foundation: shared contracts, app profile schema, wiring stubs`.
|
||||
- [ ] Create the worktrees from that commit:
|
||||
- [x] Baseline checks are all green with the same counts as F1 (plus any new tests). (baseline recorded at foundation time)
|
||||
- [x] Commit on `dev`: `v2 foundation: shared contracts, app profile schema, wiring stubs`. (commit 0cd9194)
|
||||
- [x] Create the worktrees from that commit: (worktrees listed by git worktree list)
|
||||
```bash
|
||||
git worktree add /home/mamad/work/smart-org-wt/agent1 -b agent1/org-platform
|
||||
git worktree add /home/mamad/work/smart-org-wt/agent2 -b agent2/ai-workforce
|
||||
git worktree add /home/mamad/work/smart-org-wt/agent3 -b agent3/delivery-infra
|
||||
```
|
||||
In each one: `pnpm install` and `pnpm api:venv` so the agent can run the checks immediately.
|
||||
- [ ] `mkdir docs/plan/reports` (tracked with a `.gitkeep`).
|
||||
- [x] `mkdir docs/plan/reports` (tracked with a `.gitkeep`).
|
||||
|
||||
@@ -22,19 +22,19 @@ You implement these contracts: `MemoryPort`, `ModelGrantPort`. Your real classes
|
||||
## Phase 1 — Platform invites, roles and org sharing
|
||||
|
||||
Backend
|
||||
- [ ] 1.1 New table `platform_invites` (in `db/models_org.py`): id, token_hash, email (optional), created_by (owner), role (`manager` only for now), expires_at (`PLATFORM_INVITE_TTL_HOURS`), used_at, used_by_user_id, revoked_at. Migration in your file.
|
||||
- [ ] 1.2 Owner-only endpoints in `admin_users.py`: `POST /admin/platform-invites` (returns `invite_url` once, token shown once), `GET /admin/platform-invites`, `POST /admin/platform-invites/{id}/revoke`. Audit every action.
|
||||
- [ ] 1.3 In `invites.py`, extend `GET /invites/{token}` and `POST /invites/accept` to recognise a platform invite: if no account exists, create a `manager` account with the chosen password; if the email exists, require login (same rule as org invites: an invite never resets a password). Single use, constant-time compare, expired or revoked → the same generic 404.
|
||||
- [ ] 1.4 `GET /auth/me` (you own `auth.py`) gains `needs_onboarding: bool` = platform role is `manager`, the user has zero org memberships, and they are not the owner.
|
||||
- [ ] 1.5 Org sharing in `organizations.py`: `POST /organizations/{id}/transfer-ownership` (owner → an existing admin member, in one transaction, audited), `POST /organizations/{id}/leave` (the last owner cannot leave), and `GET /organizations/{id}/invites` + `DELETE .../invites/{invite_id}` (list and revoke pending org invites). Keep the role ladder `owner > admin > manager > member > viewer`.
|
||||
- [ ] 1.6 Tests: `tests/test_org_platform_invites.py`, `tests/test_org_sharing.py`. Cover: non-owner cannot create platform invites; token single use; expired/revoked; existing-email path; transfer + leave edge cases; tenant isolation (a member of org A cannot list org B invites).
|
||||
- [x] 1.1 New table `platform_invites` (in `db/models_org.py`): id, token_hash, email (optional), created_by (owner), role (`manager` only for now), expires_at (`PLATFORM_INVITE_TTL_HOURS`), used_at, used_by_user_id, revoked_at. Migration in your file.
|
||||
- [x] 1.2 Owner-only endpoints in `admin_users.py`: `POST /admin/platform-invites` (returns `invite_url` once, token shown once), `GET /admin/platform-invites`, `POST /admin/platform-invites/{id}/revoke`. Audit every action.
|
||||
- [x] 1.3 In `invites.py`, extend `GET /invites/{token}` and `POST /invites/accept` to recognise a platform invite: if no account exists, create a `manager` account with the chosen password; if the email exists, require login (same rule as org invites: an invite never resets a password). Single use, constant-time compare, expired or revoked → the same generic 404. (token matched by its hash, as org invites are)
|
||||
- [x] 1.4 `GET /auth/me` (you own `auth.py`) gains `needs_onboarding: bool` = platform role is `manager`, the user has zero org memberships, and they are not the owner.
|
||||
- [x] 1.5 Org sharing in `organizations.py`: `POST /organizations/{id}/transfer-ownership` (owner → an existing admin member, in one transaction, audited), `POST /organizations/{id}/leave` (the last owner cannot leave), and `GET /organizations/{id}/invites` + `DELETE .../invites/{invite_id}` (list and revoke pending org invites). Keep the role ladder `owner > admin > manager > member > viewer`.
|
||||
- [x] 1.6 Tests: `tests/test_org_platform_invites.py`, `tests/test_org_sharing.py`. Cover: non-owner cannot create platform invites; token single use; expired/revoked; existing-email path; transfer + leave edge cases; tenant isolation (a member of org A cannot list org B invites). (sharing cases are in tests/test_org_platform_invites.py; tests/test_org_sharing.py was not created)
|
||||
|
||||
Frontend
|
||||
- [ ] 1.7 `/admin/users`: a "دعوت مدیر جدید" button → modal → shows the link once with a copy button; a list of pending invites with revoke.
|
||||
- [ ] 1.8 `/accept-invite`: handle the platform-invite variant (title "ساخت حساب مدیر").
|
||||
- [ ] 1.9 After login: if `needs_onboarding`, redirect to `/onboarding`. Do this in the `onboarding` page and in a small hook in `lib/api-org.ts`. `AppShell` is locked, so if a redirect there is unavoidable, write a CHANGE REQUEST.
|
||||
- [ ] 1.10 `(app)/settings`: a "اشتراکگذاری سازمان" section: members with role selector, invite form (email + role), pending invites with revoke, transfer ownership (confirmation modal that requires typing the org slug), leave org.
|
||||
- [ ] 1.11 Web tests `src/test/org-sharing.test.tsx`.
|
||||
- [x] 1.7 `/admin/users`: a "دعوت مدیر جدید" button → modal → shows the link once with a copy button; a list of pending invites with revoke.
|
||||
- [x] 1.8 `/accept-invite`: handle the platform-invite variant (title "ساخت حساب مدیر").
|
||||
- [x] 1.9 After login: if `needs_onboarding`, redirect to `/onboarding`. Do this in the `onboarding` page and in a small hook in `lib/api-org.ts`. `AppShell` is locked, so if a redirect there is unavoidable, write a CHANGE REQUEST. (login redirect in login/page.tsx via fetchNeedsOnboarding in lib/api-org.ts, T12)
|
||||
- [x] 1.10 `(app)/settings`: a "اشتراکگذاری سازمان" section: members with role selector, invite form (email + role), pending invites with revoke, transfer ownership (confirmation modal that requires typing the org slug), leave org. (T13: settings/_components/sharing-section.tsx)
|
||||
- [x] 1.11 Web tests `src/test/org-sharing.test.tsx`. (accept-invite tests here; settings sharing tests in org-settings-sharing.test.tsx, T13)
|
||||
|
||||
**Phase 1 done when:** the owner creates a link → a new person opens it, sets a password → logs in → lands on `/onboarding`. All checks are green.
|
||||
|
||||
@@ -49,28 +49,28 @@ Backend
|
||||
- php: laravel, plain
|
||||
- go: plain
|
||||
- static: html
|
||||
Add a function `defaults_for(app_type, language, framework) -> ProfileDefaults`. Unit tests for every entry.
|
||||
- [ ] 2.2 New table `org_setup_drafts` (user_id, step, data_json **without secrets**, updated_at). Secrets typed during the wizard are held only in the final submit request, never in the draft.
|
||||
- [ ] 2.3 `org_setup_service.py` + router `app_setup.py`:
|
||||
Add a function `defaults_for(app_type, language, framework) -> ProfileDefaults`. Unit tests for every entry. — open: defaults_for exists in catalog.py, but no unit test covers the catalog entries; test_org_wizard.py only checks the endpoint and the next.js defaults.
|
||||
- [x] 2.2 New table `org_setup_drafts` (user_id, step, data_json **without secrets**, updated_at). Secrets typed during the wizard are held only in the final submit request, never in the draft.
|
||||
- [x] 2.3 `org_setup_service.py` + router `app_setup.py`:
|
||||
- `GET /setup/catalog`.
|
||||
- `GET/PUT /setup/draft`.
|
||||
- `POST /setup/check-git`: external URL + credentials → `RepoWorkspacePort.test_remote`. Validate the URL **format** first in your own service: `https://` or `ssh://`/`git@` only, no credentials embedded in the URL, no IP-literal localhost/private hosts. The DNS-resolved SSRF check happens inside `test_remote` (Agent 3); do not import Agent 3's code.
|
||||
- `POST /setup/check-deploy`: → `DeployTargetTester.test_connection`.
|
||||
- `POST /setup/complete`: in ONE transaction, create the org (caller = org owner), the project with its app profile (`wizard_completed_at=now`), the encrypted external git credentials or the platform Gitea repo through the existing `GitProviderPort.ensure_project_repo`, the optional deploy target (encrypted), and the default model grants (Phase 3; until then, none). After commit, call `WorkforceBootstrapPort.bootstrap_team` and `DeliveryHooks.on_app_created`. Each hook is called once; a hook failure is recorded with `error_log` and does not undo the org.
|
||||
- Slug is unique (reuse the existing slug rules). Audit `org.created_via_wizard`.
|
||||
- [ ] 2.4 App profile editing for org owner/admin: `GET/PATCH /organizations/{org}/app/profile` (commands, port, healthcheck, framework), `PUT /organizations/{org}/app/deploy-target` (+ test), `PUT /organizations/{org}/app/git` (switch platform ↔ external is NOT allowed after creation; only credential rotation). Secrets are write-only: responses return `has_secret` and masked usernames only.
|
||||
- [ ] 2.5 Enforce 1 org = 1 app: any other path that creates a second project returns a clear 409 (the DB constraint exists already; make the service error friendly). Note: `projects.py` is Agent 3's file. Do NOT edit it; only your service/router may create the app.
|
||||
- [ ] 2.6 Tests `tests/test_org_wizard.py`: happy path platform git; happy path external git with a fake `test_remote`; a deploy target with a failing test does not block completion (it is saved as `last_test_ok=false`) but the user sees the warning; secrets never in any response, draft, audit metadata or log (assert by searching the captured output and DB rows for the plaintext secret); a second app → 409; a user with role `user` cannot complete the wizard; SSRF cases for the git URL.
|
||||
- Slug is unique (reuse the existing slug rules). Audit `org.created_via_wizard`. (hook failures use record_error, the T11 change; the wizard audit row is organization.wizard_complete)
|
||||
- [x] 2.4 App profile editing for org owner/admin: `GET/PATCH /organizations/{org}/app/profile` (commands, port, healthcheck, framework), `PUT /organizations/{org}/app/deploy-target` (+ test), `PUT /organizations/{org}/app/git` (switch platform ↔ external is NOT allowed after creation; only credential rotation). Secrets are write-only: responses return `has_secret` and masked usernames only.
|
||||
- [x] 2.5 Enforce 1 org = 1 app: any other path that creates a second project returns a clear 409 (the DB constraint exists already; make the service error friendly). Note: `projects.py` is Agent 3's file. Do NOT edit it; only your service/router may create the app. (409 test: test_second_app_for_same_org_returns_409)
|
||||
- [ ] 2.6 Tests `tests/test_org_wizard.py`: happy path platform git; happy path external git with a fake `test_remote`; a deploy target with a failing test does not block completion (it is saved as `last_test_ok=false`) but the user sees the warning; secrets never in any response, draft, audit metadata or log (assert by searching the captured output and DB rows for the plaintext secret); a second app → 409; a user with role `user` cannot complete the wizard; SSRF cases for the git URL. — open: the secrets test checks responses, audit and draft; it does not capture log output or scan DB rows for the plaintext secret.
|
||||
|
||||
Frontend
|
||||
- [ ] 2.7 Rebuild `/onboarding` as a 5-step wizard (stepper, back/next, draft auto-save, refresh resumes):
|
||||
- [ ] 2.7 Rebuild `/onboarding` as a 5-step wizard (stepper, back/next, draft auto-save, refresh resumes): — open: step 5's success screen (onboarding/page.tsx) has no demo URL placeholder; the other steps are in place.
|
||||
1. Organization: name, slug (auto-suggest).
|
||||
2. Application: name, type, language, framework (cards from the catalog), "advanced" accordion with prefilled commands, port and healthcheck.
|
||||
3. Git: "Git پلتفرم (پیشنهادی)" or "Git خودم" (URL + token or SSH key + "بررسی اتصال" button).
|
||||
4. Production (optional, skippable): SSH / SFTP / FTP / FTPS fields + "تست اتصال". Show a clear note that production deploys only happen when they press the button.
|
||||
5. Review → create → success screen with the demo URL placeholder and a "برو به مدیر AI" button.
|
||||
- [ ] 2.8 `(app)/settings` → "اپلیکیشن" tab: edit the profile, rotate git credentials, edit or test the deploy target.
|
||||
- [ ] 2.9 Web tests `src/test/org-wizard.test.tsx` (step navigation, validation, secrets never placed in the draft payload).
|
||||
- [x] 2.8 `(app)/settings` → "اپلیکیشن" tab: edit the profile, rotate git credentials, edit or test the deploy target.
|
||||
- [ ] 2.9 Web tests `src/test/org-wizard.test.tsx` (step navigation, validation, secrets never placed in the draft payload). — open: there is no validation case; step navigation, slug suggestion and the secret-free draft are covered.
|
||||
|
||||
**Phase 2 done when:** a fresh manager completes the wizard in the browser and lands on the dashboard of the new org. The DB has exactly one project with a full profile. The Null hooks recorded one `bootstrap_team` and one `on_app_created` call.
|
||||
|
||||
@@ -78,49 +78,49 @@ Frontend
|
||||
|
||||
## Phase 3 — Owner model grants (employees only use what the owner provides)
|
||||
|
||||
- [ ] 3.1 `model_grant_service.py` implementing `ModelGrantPort` over `org_model_grants` + `ai_models` (only active models, only active grants). `GrantedModel.tier` is a neutral label derived from capabilities (`سریع`, `متعادل`, `قوی`). Never the model name.
|
||||
- [ ] 3.2 Owner endpoints `admin_grants.py`: list grants per org, grant/revoke model, set daily credit cap, and "default grant set for new orgs" (new table `default_model_grants`, or a column in your own table; do NOT edit `ai_models`). Audit all.
|
||||
- [ ] 3.3 Wizard `complete` copies the default grant set into the new org.
|
||||
- [ ] 3.4 Owner org policy (`org_policies` table, yours): max_employees (≤ `ENGINE_MAX_EMPLOYEES`), monthly credit budget, automation_enabled (bool). Endpoints under `admin_grants.py`. Expose `get_org_policy(org_id)` in `model_grant_service.py` for Agent 2 to read later. Write a CHANGE REQUEST so Claude adds it to the `ModelGrantPort` contract at integration.
|
||||
- [ ] 3.5 Wire your real `ModelGrantPort` in `container_org.py`.
|
||||
- [ ] 3.6 UI `/admin/model-grants`: org list → grant matrix (models × checkbox + cap), default set editor, policy editor. Org users see nothing new.
|
||||
- [ ] 3.7 Tests `tests/test_model_grants.py`: non-owner forbidden everywhere; revoked or inactive model not returned; a grant on an org you query for another org is never returned (isolation); defaults copied on wizard completion.
|
||||
- [x] 3.1 `model_grant_service.py` implementing `ModelGrantPort` over `org_model_grants` + `ai_models` (only active models, only active grants). `GrantedModel.tier` is a neutral label derived from capabilities (`سریع`, `متعادل`, `قوی`). Never the model name.
|
||||
- [x] 3.2 Owner endpoints `admin_grants.py`: list grants per org, grant/revoke model, set daily credit cap, and "default grant set for new orgs" (new table `default_model_grants`, or a column in your own table; do NOT edit `ai_models`). Audit all.
|
||||
- [x] 3.3 Wizard `complete` copies the default grant set into the new org.
|
||||
- [x] 3.4 Owner org policy (`org_policies` table, yours): max_employees (≤ `ENGINE_MAX_EMPLOYEES`), monthly credit budget, automation_enabled (bool). Endpoints under `admin_grants.py`. Expose `get_org_policy(org_id)` in `model_grant_service.py` for Agent 2 to read later. Write a CHANGE REQUEST so Claude adds it to the `ModelGrantPort` contract at integration. (the ModelGrantPort contract change is T1: get_org_policy)
|
||||
- [x] 3.5 Wire your real `ModelGrantPort` in `container_org.py`.
|
||||
- [x] 3.6 UI `/admin/model-grants`: org list → grant matrix (models × checkbox + cap), default set editor, policy editor. Org users see nothing new.
|
||||
- [x] 3.7 Tests `tests/test_model_grants.py`: non-owner forbidden everywhere; revoked or inactive model not returned; a grant on an org you query for another org is never returned (isolation); defaults copied on wizard completion.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Org memory (self-improving org)
|
||||
|
||||
- [ ] 4.1 Domain `app/domain/memory/`: `MemoryKind` (`rule`, `preference`, `lesson`, `fact`, `code_map`), `MemorySource` (`user`, `manager`, `system`, `reflection`), validation (key ≤ 80 chars, content ≤ 1,500 chars), normalisation + content hash for dedupe.
|
||||
- [ ] 4.2 Table `org_memories` (in `models_org.py`): id, organization_id, kind, key (nullable), content, source, evidence_task_id, pinned, confidence (0..1), expires_at (lessons: +30 days unless pinned or confirmed), hits, last_used_at, created_by_user_id, created_at, updated_at, deleted_at (soft). Index `(organization_id, kind)`.
|
||||
- [ ] 4.3 `memory_service.py` implementing `MemoryPort`:
|
||||
- [x] 4.1 Domain `app/domain/memory/`: `MemoryKind` (`rule`, `preference`, `lesson`, `fact`, `code_map`), `MemorySource` (`user`, `manager`, `system`, `reflection`), validation (key ≤ 80 chars, content ≤ 1,500 chars), normalisation + content hash for dedupe.
|
||||
- [x] 4.2 Table `org_memories` (in `models_org.py`): id, organization_id, kind, key (nullable), content, source, evidence_task_id, pinned, confidence (0..1), expires_at (lessons: +30 days unless pinned or confirmed), hits, last_used_at, created_by_user_id, created_at, updated_at, deleted_at (soft). Index `(organization_id, kind)`.
|
||||
- [x] 4.3 `memory_service.py` implementing `MemoryPort`:
|
||||
- `record`: dedupe by hash or same key (update instead of insert), org cap `MEMORY_ORG_CHAR_CAP` (evict oldest unpinned lessons first), max 3 writes per task.
|
||||
- `build_context(role, query, budget)`: pinned rules and preferences first, then the top memories by keyword overlap with `query` and recency. Role filter: `code_map` and `fact` only for planner/programmer/reviewer/tester; preferences for the manager. Returns a compact Persian/English block within `budget_chars`; updates `hits/last_used_at`.
|
||||
- `on_task_finished(TaskOutcome)`: one cheap model call (through the existing ai_control routing in "cheap" mode and `ModelProviderPort`, no new adapter) asking for ≤ 2 lessons + a 1-paragraph code-map delta. It stores them with `source=reflection`, `confidence=0.5`. If the call fails, it records the error and returns (it never raises into the engine). Record usage for that call in the usage ledger like other model calls.
|
||||
- Expire job: function `expire_memories()` (called from the existing billing worker? NO, that is not your file → expose the function and write a CHANGE REQUEST for a worker hook).
|
||||
- [ ] 4.4 Router `memory.py`: list (filter by kind, search), create (users can "teach" a rule or preference), edit, pin/unpin, delete (soft, audited), and a `confirm` action (lesson → confidence 1, no expiry). Roles: viewer can read; member+ can add preferences; admin+ can edit or delete anything.
|
||||
- [ ] 4.5 Fold the existing knowledge sources into the same page: keep `knowledge.py` endpoints working and show sources as a second tab.
|
||||
- [ ] 4.6 UI `/memory` (replace the placeholder; also redirect `/knowledge` → `/memory`): tabs "قوانین و ترجیحات", "درسهای آموخته", "نقشهٔ کد", "منابع دانش"; inline edit; pin; confirm a lesson; "به سازمان یاد بده" input. Publish `memory.changed` on `org:<id>` and refresh live.
|
||||
- [ ] 4.7 Wire the real `MemoryPort` in `container_org.py`.
|
||||
- [ ] 4.8 Tests `tests/test_memory.py`: dedupe, caps and eviction, TTL, role filtering, budget respected, isolation, reflection failure does not raise, no secrets: run content through `app/core/redact.py` before storing and test it.
|
||||
- Expire job: function `expire_memories()` (called from the existing billing worker? NO, that is not your file → expose the function and write a CHANGE REQUEST for a worker hook). (expire hook: T9, called from the billing pass)
|
||||
- [x] 4.4 Router `memory.py`: list (filter by kind, search), create (users can "teach" a rule or preference), edit, pin/unpin, delete (soft, audited), and a `confirm` action (lesson → confidence 1, no expiry). Roles: viewer can read; member+ can add preferences; admin+ can edit or delete anything.
|
||||
- [x] 4.5 Fold the existing knowledge sources into the same page: keep `knowledge.py` endpoints working and show sources as a second tab.
|
||||
- [x] 4.6 UI `/memory` (replace the placeholder; also redirect `/knowledge` → `/memory`): tabs "قوانین و ترجیحات", "درسهای آموخته", "نقشهٔ کد", "منابع دانش"; inline edit; pin; confirm a lesson; "به سازمان یاد بده" input. Publish `memory.changed` on `org:<id>` and refresh live.
|
||||
- [x] 4.7 Wire the real `MemoryPort` in `container_org.py`.
|
||||
- [x] 4.8 Tests `tests/test_memory.py`: dedupe, caps and eviction, TTL, role filtering, budget respected, isolation, reflection failure does not raise, no secrets: run content through `app/core/redact.py` before storing and test it.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — MCP gateway (AI agents control our app)
|
||||
|
||||
- [ ] 5.1 Table `mcp_tokens` (id, organization_id, name, token_hash, prefix (first 6 chars for display), scopes CSV, created_by, expires_at, last_used_at, revoked_at). Tokens look like `smo_<random>` and are shown once.
|
||||
- [ ] 5.2 `mcp_service.py`: issue, list, revoke, verify (constant-time; expired or revoked → reject), scope check.
|
||||
- [ ] 5.3 `app/interfaces/mcp/server.py` with the official `mcp` SDK (streamable HTTP, bound to `MCP_GATEWAY_HOST:MCP_GATEWAY_PORT`) and `main()` for `pnpm mcp:dev`. Tools (each resolves the org from the token, never from arguments):
|
||||
- [x] 5.1 Table `mcp_tokens` (id, organization_id, name, token_hash, prefix (first 6 chars for display), scopes CSV, created_by, expires_at, last_used_at, revoked_at). Tokens look like `smo_<random>` and are shown once.
|
||||
- [x] 5.2 `mcp_service.py`: issue, list, revoke, verify (constant-time; expired or revoked → reject), scope check.
|
||||
- [x] 5.3 `app/interfaces/mcp/server.py` with the official `mcp` SDK (streamable HTTP, bound to `MCP_GATEWAY_HOST:MCP_GATEWAY_PORT`) and `main()` for `pnpm mcp:dev`. Tools (each resolves the org from the token, never from arguments):
|
||||
- `app_get_profile` (`app:read`)
|
||||
- `tasks_list` / `tasks_get` (`tasks:read`)
|
||||
- `tasks_create` (`tasks:write`): goes to the manager as a user order through the **current** public `WorkforceService.create_task` signature; read it, do not edit it (Agent 2 owns it; Claude re-wires at integration if it changes)
|
||||
- `memory_search` (`memory:read`) / `memory_write` (`memory:write`, kinds `fact|preference` only, `source=user`)
|
||||
- `knowledge_search` (`memory:read`)
|
||||
- `pipeline_status` (`delivery:read`): register the tool but return `{"available": false}`. Agent 3's tables do not exist in your branch; Claude wires it at integration
|
||||
- `pipeline_status` (`delivery:read`): register the tool but return `{"available": false}`. Agent 3's tables do not exist in your branch; Claude wires it at integration (pipeline_status: T10)
|
||||
Every call is audited (`mcp.tool_call`, token prefix, tool, ok/error), rate limited (60/min per token, in-memory), and its output passes through `redact.py`.
|
||||
- [ ] 5.4 `services/mcp-gateway/README.md`: how to run it, the Caddy host suggestion `mcp.<root>` (Agent 3/owner configures it), and a client config snippet.
|
||||
- [ ] 5.5 Router `mcp_tokens.py`: org admin+ can create, list and revoke org tokens. Owner endpoint to disable MCP globally or per org (setting stored in your `org_policies`).
|
||||
- [ ] 5.6 UI: `(app)/settings` → "یکپارچهسازی (MCP)" tab (create with scope checkboxes, show-once modal with a copy button, list, revoke); `/admin/mcp` (global switch, per-org switch, token counts).
|
||||
- [ ] 5.7 Tests `tests/test_mcp.py`: use the SDK's in-process client or call the tool functions directly. Cover: a token from org A can never read org B; scope enforcement; revoked/expired; rate limit; audit row written; no secret in outputs.
|
||||
- [x] 5.4 `services/mcp-gateway/README.md`: how to run it, the Caddy host suggestion `mcp.<root>` (Agent 3/owner configures it), and a client config snippet.
|
||||
- [x] 5.5 Router `mcp_tokens.py`: org admin+ can create, list and revoke org tokens. Owner endpoint to disable MCP globally or per org (setting stored in your `org_policies`).
|
||||
- [x] 5.6 UI: `(app)/settings` → "یکپارچهسازی (MCP)" tab (create with scope checkboxes, show-once modal with a copy button, list, revoke); `/admin/mcp` (global switch, per-org switch, token counts).
|
||||
- [x] 5.7 Tests `tests/test_mcp.py`: use the SDK's in-process client or call the tool functions directly. Cover: a token from org A can never read org B; scope enforcement; revoked/expired; rate limit; audit row written; no secret in outputs.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -31,7 +31,7 @@ for tests).
|
||||
|
||||
## Phase 1 — Docker sandbox adapter
|
||||
|
||||
- [ ] 1.1 `adapters/docker_sandbox.py`, `DockerSandboxAdapter` implementing `SandboxProviderPort` with the CLI in `SANDBOX_DOCKER_BIN` (docker or podman, the same flags):
|
||||
- [x] 1.1 `adapters/docker_sandbox.py`, `DockerSandboxAdapter` implementing `SandboxProviderPort` with the CLI in `SANDBOX_DOCKER_BIN` (docker or podman, the same flags):
|
||||
- `create_workspace`: creates a host dir `SANDBOX_WORKSPACES_ROOT/<org_id>/<project_id>/work` (0700, owned by the platform user).
|
||||
- `start_session`: `docker run -d` the **workspace container**:
|
||||
- image from the app profile language: `node:20-bookworm`, `python:3.13-bookworm`, `php:8.3-cli`, `golang:1.23`, fallback `debian:bookworm`; make the mapping a dict in your config block / domain
|
||||
@@ -44,39 +44,39 @@ for tests).
|
||||
- `read_file` / `write_file`: through the host bind mount with path safety (`resolve()` stays inside the work dir; reject symlinks that escape).
|
||||
- `pause` / `resume` / `destroy`: stop/start/rm. `snapshot`: `docker commit` (tagged per org) or raise a clear "not supported" error, your choice, documented.
|
||||
- `get_preview_url`: delegated to the domain layer (Phase 4).
|
||||
- [ ] 1.2 `registry.py`: `SANDBOX_PROVIDER=docker` → this adapter. The mock stays the default.
|
||||
- [ ] 1.3 The `sandbox_gateway` worker: also reap containers with the `smart-org.*` label that have no live session row (orphans), audited.
|
||||
- [ ] 1.4 Hide infra from non-owners:
|
||||
- [x] 1.2 `registry.py`: `SANDBOX_PROVIDER=docker` → this adapter. The mock stays the default.
|
||||
- [x] 1.3 The `sandbox_gateway` worker: also reap containers with the `smart-org.*` label that have no live session row (orphans), audited.
|
||||
- [x] 1.4 Hide infra from non-owners:
|
||||
- `sandbox.py` command/file endpoints become **owner-only** (or admin+ of the org with a setting off by default; prefer owner-only).
|
||||
- Keep the read-only status endpoints for org members.
|
||||
- Update `components/sandbox-panel.tsx` to show only status to non-owners.
|
||||
- Update existing tests that assumed member access, and explain each change in your report.
|
||||
- [ ] 1.5 Tests `tests/test_docker_sandbox.py`: unit tests with a **fake process runner** (inject the subprocess function) that assert the exact argv (hardening flags present, no secrets, mount of the work dir only), timeout kill, output cap, path escape refused. Integration tests marked `@pytest.mark.docker`, skipped unless `SANDBOX_DOCKER_IT=1`: real start → exec `echo` → write/read file → destroy.
|
||||
- [ ] 1.5 Tests `tests/test_docker_sandbox.py`: unit tests with a **fake process runner** (inject the subprocess function) that assert the exact argv (hardening flags present, no secrets, mount of the work dir only), timeout kill, output cap, path escape refused. Integration tests marked `@pytest.mark.docker`, skipped unless `SANDBOX_DOCKER_IT=1`: real start → exec `echo` → write/read file → destroy. — open: no `@pytest.mark.docker` sandbox integration test (real start, exec, write/read, destroy) exists in tests/test_docker_sandbox.py; the only docker-marked test is test_delivery.py::test_docker_end_to_end_static_app.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Git workspaces (`RepoWorkspacePort`, the platform does git, D6)
|
||||
|
||||
- [ ] 2.1 `adapters/git_workspace.py`, `LocalGitWorkspaceAdapter`. Layout per project under `SANDBOX_WORKSPACES_ROOT/<org>/<project>/`:
|
||||
- [x] 2.1 `adapters/git_workspace.py`, `LocalGitWorkspaceAdapter`. Layout per project under `SANDBOX_WORKSPACES_ROOT/<org>/<project>/`:
|
||||
- `git/`: a bare-ish gitdir, **never mounted**
|
||||
- `work/`: the main working tree for the dev branch, mounted at `/workspace`
|
||||
- `work/.worktrees/<task_id>/`: task worktrees, visible in the container at `/workspace/.worktrees/<task_id>`
|
||||
- `WorktreeRef.sandbox_path` = that in-container path
|
||||
Note: the `.worktrees` dir must be excluded via `info/exclude` in the gitdir, not the user's `.gitignore`.
|
||||
- [ ] 2.2 Every git call is `git --git-dir=<gitdir> --work-tree=<path> -c core.hooksPath=/dev/null -c core.fsmonitor=false -c core.quotePath=false -c protocol.file.allow=never -c safe.directory=*`, with env `GIT_CONFIG_NOSYSTEM=1`, `HOME=<private dir>`, `GIT_TERMINAL_PROMPT=0`. Never trust the `.git` *file* inside a worktree; always pass explicit dirs.
|
||||
- [x] 2.2 Every git call is `git --git-dir=<gitdir> --work-tree=<path> -c core.hooksPath=/dev/null -c core.fsmonitor=false -c core.quotePath=false -c protocol.file.allow=never -c safe.directory=*`, with env `GIT_CONFIG_NOSYSTEM=1`, `HOME=<private dir>`, `GIT_TERMINAL_PROMPT=0`. Never trust the `.git` *file* inside a worktree; always pass explicit dirs.
|
||||
- Status: `status --porcelain=v1 -uall -z`.
|
||||
- One `asyncio.Lock` per project id held for every mutating operation (agybot M14). Remove a stale `index.lock` only if older than 30s.
|
||||
- [ ] 2.3 Remotes:
|
||||
- [ ] 2.3 Remotes: — open: the ssh_key first contact uses StrictHostKeyChecking=accept-new and the key is never stored. test_remote returns no host key (RemoteCheck has no field for it) and nothing writes project_external_git.known_hosts.
|
||||
- **platform** git: the URL and credentials come from the existing Gitea adapter / `project_git_credentials` (decrypt via the existing crypto helper).
|
||||
- **external** git: `project_external_git` (token → `GIT_ASKPASS`; ssh_key → temp key file 0600 + `GIT_SSH_COMMAND="ssh -i <key> -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=<per-project known_hosts>"`; store `known_hosts` on the first `test_remote` = TOFU, in the existing column).
|
||||
- `test_remote`: `git ls-remote --heads` with a 20s timeout; returns ok, default branch, branch list, and a friendly error (auth failed / not found / unreachable) **without echoing the URL credentials or stderr secrets**.
|
||||
- SSRF check on external hosts (resolve DNS, reject private, loopback and link-local unless `DEPLOY_ALLOW_PRIVATE_HOSTS`); put the check in `domain/platform/rules.py` (yours). It runs inside `test_remote` and before every deploy connection; Agent 1 only does format validation and relies on your check.
|
||||
- [ ] 2.4 Implement every `RepoWorkspacePort` method per PLAN-v2 §5.
|
||||
- [x] 2.4 Implement every `RepoWorkspacePort` method per PLAN-v2 §5.
|
||||
- `ensure_workspace` clones on first use, fetches afterwards. It creates `dev` from the default branch if missing; for platform repos also push `dev`.
|
||||
- `merge_into` does a `--no-ff` merge in the main work tree under the lock and returns the conflict file list on failure, after `merge --abort`. Never leave a half-merge (agybot H3).
|
||||
- `push` returns ok/rejected (non-fast-forward → fetch + report, never force).
|
||||
- [ ] 2.5 Wire it in `container_delivery.py` (setting `GIT_WORKSPACE=local|fake`, fake default in tests).
|
||||
- [ ] 2.6 Tests `tests/test_git_workspace.py` with **real git** and a local bare repo in `tmp_path` as "remote" (no network):
|
||||
- [x] 2.5 Wire it in `container_delivery.py` (setting `GIT_WORKSPACE=local|fake`, fake default in tests).
|
||||
- [x] 2.6 Tests `tests/test_git_workspace.py` with **real git** and a local bare repo in `tmp_path` as "remote" (no network):
|
||||
- the full flow ensure → worktree → write → diff (new dir appears thanks to `-uall`) → commit → sync → merge → push → head
|
||||
- a conflict path leaves a clean tree
|
||||
- hooks are NOT executed (plant a `post-commit` hook that writes a file and assert it is absent)
|
||||
@@ -88,15 +88,15 @@ for tests).
|
||||
|
||||
## Phase 3 — Live CI/CD + demo server
|
||||
|
||||
- [ ] 3.1 Tables (`models_delivery.py`):
|
||||
- [x] 3.1 Tables (`models_delivery.py`):
|
||||
- `pipeline_runs` (id, org, project, trigger `engine|poll|manual`, branch, sha, status `queued|running|succeeded|failed|cancelled|superseded`, started/finished, error_summary)
|
||||
- `pipeline_steps` (id, run_id, index, name, status, started/finished, exit_code, log_tail ≤ 64 KB)
|
||||
- `releases` (id, org, project, run_id, sha, artifact_path, created_at, is_demo_live)
|
||||
- `app_runtimes` (project_id PK, demo_container_id, host_port, demo_host, status, current_release_id, updated_at)
|
||||
- [ ] 3.2 `delivery_service.py` implementing `DeliveryHooks`:
|
||||
- [x] 3.2 `delivery_service.py` implementing `DeliveryHooks`:
|
||||
- `on_app_created`: allocate a demo port in `DEMO_PORT_START..END` (reuse the existing preview port rules), claim the demo host `<org-slug>.<PREVIEW_ROOT_DOMAIN>` through the existing `DomainRouterPort`, create `app_runtimes`, `ensure_workspace`, and queue a first pipeline run if the repo has a `dev` head.
|
||||
- `on_branch_pushed`: queue a run (one active run per project; a newer queued run supersedes an older queued one).
|
||||
- [ ] 3.3 `workers/delivery.py`: loop (`--loop`) that executes queued runs and polls remotes every `DELIVERY_POLL_SECONDS` (`head` vs the last run sha) for pushes made outside the engine (users pushing to their own git).
|
||||
- [x] 3.3 `workers/delivery.py`: loop (`--loop`) that executes queued runs and polls remotes every `DELIVERY_POLL_SECONDS` (`head` vs the last run sha) for pushes made outside the engine (users pushing to their own git).
|
||||
- Steps, each in a **fresh build container** (same image mapping and hardening, work dir = a clean checkout of the sha in a per-run dir):
|
||||
1. `checkout`
|
||||
2. `setup` (setup_command)
|
||||
@@ -107,9 +107,9 @@ for tests).
|
||||
7. `healthcheck`: HTTP GET `http://127.0.0.1:<host_port><healthcheck_path>` with 20 retries × 3s; any 2xx/3xx = ok
|
||||
8. `route`: `configure_reverse_proxy` for the demo host → `127.0.0.1:<host_port>`
|
||||
- **Blue/green:** start the new container on a temporary second port, healthcheck it, then switch the Caddy route, then stop the old one. If any step fails, the old demo keeps running untouched and the run is `failed` with a Persian `error_summary`.
|
||||
- [ ] 3.4 Live logs: every step streams output via the `on_output` callback → batched `pipeline.log` events on `org:<id>` (≤ 1 event / 500 ms, ≤ 200 lines), plus `pipeline.step` and `pipeline.status`; `release.live` when switched. Store the step `log_tail` (last 64 KB). Pass all output through `app/core/redact.py` before publishing or storing.
|
||||
- [ ] 3.5 Router `delivery.py` (org member read, admin+ actions): list runs, run detail with steps and tails, `POST /delivery/runs` (manual re-run of the dev head, admin+), cancel the queued/running run, list releases, rollback the demo to a previous release (admin+, confirm, audit), and the runtime status (demo URL, live release, health).
|
||||
- [ ] 3.6 Tests `tests/test_delivery.py` with the fake runner + `FakeRepoWorkspace` + mock domain router:
|
||||
- [x] 3.4 Live logs: every step streams output via the `on_output` callback → batched `pipeline.log` events on `org:<id>` (≤ 1 event / 500 ms, ≤ 200 lines), plus `pipeline.step` and `pipeline.status`; `release.live` when switched. Store the step `log_tail` (last 64 KB). Pass all output through `app/core/redact.py` before publishing or storing.
|
||||
- [x] 3.5 Router `delivery.py` (org member read, admin+ actions): list runs, run detail with steps and tails, `POST /delivery/runs` (manual re-run of the dev head, admin+), cancel the queued/running run, list releases, rollback the demo to a previous release (admin+, confirm, audit), and the runtime status (demo URL, live release, health).
|
||||
- [x] 3.6 Tests `tests/test_delivery.py` with the fake runner + `FakeRepoWorkspace` + mock domain router:
|
||||
- the happy path step sequence and events
|
||||
- a failed build keeps the old demo
|
||||
- healthcheck failure → no route switch
|
||||
@@ -123,50 +123,50 @@ for tests).
|
||||
|
||||
## Phase 4 — Domains (Caddy) owned by the platform owner
|
||||
|
||||
- [ ] 4.1 `admin_domains.py` (owner only):
|
||||
- [x] 4.1 `admin_domains.py` (owner only):
|
||||
- list all routes (host → org/app → target port → status)
|
||||
- attach an extra host to an org's app: a single label under the root via the existing rules, OR a full custom FQDN when `ALLOW_CUSTOM_DOMAINS=true` (strict FQDN regex, no wildcard, no protected hosts such as `git`, `mcp`, `panel`, `api`, root)
|
||||
- detach (confirm + audit)
|
||||
- re-sync all routes from the DB to Caddy (idempotent)
|
||||
Users cannot add domains (the existing member-facing preview-domain endpoints in `projects.py` become owner-only, and the tests are updated with reasons).
|
||||
- [ ] 4.2 The Caddy adapter keeps validate → revert → reload. Add a `routes_fingerprint` check so re-sync does nothing when nothing changed.
|
||||
- [ ] 4.3 The demo host is created automatically in `on_app_created` (Phase 3). Record the MCP gateway host suggestion (`mcp.<root>` → `MCP_GATEWAY_PORT`) as a protected system route that the owner can enable from this page.
|
||||
- [ ] 4.4 Tests `tests/test_admin_domains.py`: non-owner forbidden; protected hosts refused; invalid FQDNs refused; detach audited; re-sync idempotent (mock router call count).
|
||||
- [x] 4.2 The Caddy adapter keeps validate → revert → reload. Add a `routes_fingerprint` check so re-sync does nothing when nothing changed.
|
||||
- [ ] 4.3 The demo host is created automatically in `on_app_created` (Phase 3). Record the MCP gateway host suggestion (`mcp.<root>` → `MCP_GATEWAY_PORT`) as a protected system route that the owner can enable from this page. — open: the MCP route is only a read-only suggestion (GET /api/admin/domains/suggested, shown as enabled or disabled on the page). There is no enable action, and validate_extra_host refuses the protected label mcp, so the owner cannot enable it from this page.
|
||||
- [x] 4.4 Tests `tests/test_admin_domains.py`: non-owner forbidden; protected hosts refused; invalid FQDNs refused; detach audited; re-sync idempotent (mock router call count).
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Production deploy over SSH / SFTP / FTP / FTPS (always human-triggered)
|
||||
|
||||
- [ ] 5.1 `adapters/deploy_ssh.py` (asyncssh), `DeployTargetTester` + `SshDeployer`:
|
||||
- [x] 5.1 `adapters/deploy_ssh.py` (asyncssh), `DeployTargetTester` + `SshDeployer`:
|
||||
- Test: connect (15s timeout) → host-key TOFU (store the fingerprint on the first success; afterwards a mismatch = hard fail "کلید میزبان تغییر کرده"), check `remote_path` exists and is writable.
|
||||
- Deploy: upload the release artifact → `remote_path/releases/<ts>-<sha7>/` → extract → run `post_deploy_command` with `cwd` there (timeout 600s, streamed output) → atomically switch the `remote_path/current` symlink → keep the last 5 releases.
|
||||
- Rollback = re-point `current` to the previous release + rerun post_deploy.
|
||||
- [ ] 5.2 `adapters/deploy_ftp.py` (aioftp; FTPS via explicit TLS), `DeployTargetTester` + `FtpDeployer`:
|
||||
- [x] 5.2 `adapters/deploy_ftp.py` (aioftp; FTPS via explicit TLS), `DeployTargetTester` + `FtpDeployer`:
|
||||
- Test: login, cwd `remote_path`, write and delete a probe file.
|
||||
- Deploy: upload the build output (the build dir is set by the framework catalog default or `dist/`, `build/`, `public/`, else the whole artifact) into `remote_path` after downloading a **backup listing** of the files it overwrites into `ARTIFACTS_ROOT/.../ftp-backup-<ts>.tar.gz`.
|
||||
- Rollback = re-upload that backup. Document the limits of FTP (no atomic switch).
|
||||
- [ ] 5.3 SSRF check for every target host (the Phase 2.3 helper). A combined `DeployTargetTester` dispatching on `kind`, wired in `container_delivery.py`.
|
||||
- [ ] 5.4 `deploy_service.py` + endpoints in `delivery.py`:
|
||||
- [x] 5.3 SSRF check for every target host (the Phase 2.3 helper). A combined `DeployTargetTester` dispatching on `kind`, wired in `container_delivery.py`.
|
||||
- [x] 5.4 `deploy_service.py` + endpoints in `delivery.py`:
|
||||
- `POST /delivery/production/deploy` with `release_id` (default: the live demo release). Org **owner/admin only**; requires a `confirm: "<org-slug>"` field; one production deploy per project at a time.
|
||||
- Status: `deploy.status` events and a `production_deploys` table (id, release, target, status, started/finished, log_tail, rolled_back_from).
|
||||
- `POST /delivery/production/rollback`.
|
||||
- Every action audited. Credentials decrypted only inside the adapter call.
|
||||
- [ ] 5.5 Tests `tests/test_deploy.py`: fake SSH/FTP clients injected. Cover: the release/symlink sequence, keep-5 pruning, rollback, host-key mismatch refused, SSRF refused, a member (non-admin) refused, the confirm slug mismatch refused, no password in logs/events/DB columns other than encrypted ones. Optional integration (`DEPLOY_IT=1`) against a local `sshd`/`pure-ftpd` container.
|
||||
- [x] 5.5 Tests `tests/test_deploy.py`: fake SSH/FTP clients injected. Cover: the release/symlink sequence, keep-5 pruning, rollback, host-key mismatch refused, SSRF refused, a member (non-admin) refused, the confirm slug mismatch refused, no password in logs/events/DB columns other than encrypted ones. Optional integration (`DEPLOY_IT=1`) against a local `sshd`/`pure-ftpd` container.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — UI: the "Application" page (replaces `/projects`)
|
||||
|
||||
- [ ] 6.1 `(app)/app/page.tsx` (+ components under `(app)/app/_components/`):
|
||||
- [x] 6.1 `(app)/app/page.tsx` (+ components under `(app)/app/_components/`):
|
||||
- **Header:** app name, type/language/framework badges, demo URL (open button), live release sha + time, health dot.
|
||||
- **Pipeline runs:** list with status; clicking one opens the run view: a step list with status and duration, and a **live log console** (subscribe `org:<id>`, filter by run id, ANSI colours stripped, auto-scroll with pause, and a "copy" button).
|
||||
- **Releases:** history; "rollback demo" (admin+, confirm).
|
||||
- **Production:** target summary (kind, host, path, last test result; no secrets), the "استقرار در production" button (admin+, modal requiring the org slug), a deploy live log, production history + rollback.
|
||||
- **Git:** repo link (platform: Gitea URL; external: the URL without credentials), `dev` head sha, last poll time. No branch/PR/port controls for users.
|
||||
- [ ] 6.2 Remove `(app)/projects` (redirect `/projects*` → `/app`) and drop the member-facing port/branch/PR/command controls. Owner-only debug tools move to `/admin/sandbox`.
|
||||
- [ ] 6.3 `/admin/sandbox` (owner): running containers per org (workspace/demo/build), CPU/memory limits per org (existing `sandbox_resource_limits`), stop/restart a container (confirm + audit), and an image mapping view. `/admin/domains` (owner): the Phase 4 UI.
|
||||
- [ ] 6.4 `lib/api-delivery.ts`: typed client for all of the above.
|
||||
- [ ] 6.5 Web tests `src/test/delivery-*.test.tsx`: the log console batching and auto-scroll, the confirm-slug modal, no secret fields rendered, non-admin sees no deploy button.
|
||||
- [x] 6.2 Remove `(app)/projects` (redirect `/projects*` → `/app`) and drop the member-facing port/branch/PR/command controls. Owner-only debug tools move to `/admin/sandbox`.
|
||||
- [ ] 6.3 `/admin/sandbox` (owner): running containers per org (workspace/demo/build), CPU/memory limits per org (existing `sandbox_resource_limits`), stop/restart a container (confirm + audit), and an image mapping view. `/admin/domains` (owner): the Phase 4 UI. — open: no restart action. The backend and the page offer list and stop only (POST /api/admin/sandbox/containers/{id}/stop).
|
||||
- [ ] 6.4 `lib/api-delivery.ts`: typed client for all of the above. — open: /admin/sandbox calls apiRequest directly, and lib/api-delivery.ts has deliveryApi and domainsApi only, with no sandbox client.
|
||||
- [ ] 6.5 Web tests `src/test/delivery-*.test.tsx`: the log console batching and auto-scroll, the confirm-slug modal, no secret fields rendered, non-admin sees no deploy button. — open: no test for log batching or auto-scroll. The non-admin deploy-button case is covered by `hides the deploy actions from an org member who is not an admin` (`apps/web/src/test/delivery-app.test.tsx`, added in T14).
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user