feat(integration): prepaid billing, engine UI, credits console, direct tasks, v2 final

- Prepaid credit system: org wallet, debit on usage, chat gate, admin credit console
- Engine UI: /manager, /tasks, /tasks/[id], /workforce, /dashboard, /traces pages
- Direct task creation (HTTP POST /tasks, MCP tasks_create) at pending_manager_review
- Pause/resume/cancel/answer actions in engine API and UI
- Employee cap (7 per org default), review pipeline guard (active reviewer required)
- Scheduler: wide window prevents org starvation; archived employees excluded from cap
- Tenant isolation: project-id cross-org 404, engine never uses another tenant's sandbox
- Order route fix: manager built with pipeline states + employee profiles
- Contract extension: ModelGrantPort.get_org_policy with DefaultOrgPolicy
- Engine event types in @agy/contracts (task.step, task.gate, run.log, run.tool, team.changed)
- api-engine.ts typed client; api-credit.ts client
- Alembic migration c4d5e6f7a8b9: prepaid credit + engine UI tables
- 859 tests passed, 1 skipped; ruff/mypy/tsc/eslint/vitest all green

Refs: PLAN-v2.md §9 (T1–T43), todo-v2-final.md
This commit is contained in:
mjalmoudy94
2026-10-14 23:19:05 +03:30
parent fef5a9deab
commit d532929189
117 changed files with 13119 additions and 640 deletions
+9 -6
View File
@@ -290,7 +290,7 @@ Written on 2026-10-14 from the code, an alembic run on a scratch SQLite database
- [ ] `agent1/org-platform` (6 commits, `0016bb4`..`051bf1b`) is not an ancestor of `dev`, and its committed tree differs from `dev` in 138 files. Its phases reached `dev` as separate commits (`5095a29`, `5fb85d6`, `dee2f72`). Open: confirm that `dev` has everything from Agent 1, then merge or retire the branch.
- [x] §8.4 done through T1–T26; T27 is this docs pass.
- [x] Double execution fixed (T2): the legacy run drain skips tasks that own a pipeline state.
- [x] Zero-wallet pause fixed (T3): only an overdrawn wallet pauses a task.
- [x] Billing is prepaid (D-B): the owner credits each org, every usage event debits the wallet once, and a zero, negative or missing wallet pauses engine turns and the manager chat (T5–T13).
- [x] The engine honours the owner's org policy: automation switch, monthly credit budget and `max_employees` (T1, T4, T5).
- [x] A new organization gets a sandbox session that the engine starts on demand (T6).
- [x] End-to-end smoke test `tests/test_v2_e2e_smoke.py` (T7; closer to reality in T26): a thread made without a project, the real `AiRuntimeWorker.drain()`, and a sandbox session that the engine starts. The production-deploy leg is not in it.
@@ -298,8 +298,11 @@ Written on 2026-10-14 from the code, an alembic run on a scratch SQLite database
- [x] Fixed: the HTTP message route `POST /api/organizations/{org}/threads/{thread}/messages` built its manager without pipeline states and employee profiles (`workforce_service._manager_turns`), so a manager order sent through it made tasks the engine never picked up. The manager now gets both, as `container_engine._build_manager` gives them. `tests/test_integration_order_route.py` posts the order over HTTP and checks its pipeline state.
- [ ] The production-deploy leg of the smoke test stays manual and has not been run. See `docs/plan/reports/agent-3-final.md` §6, steps 7–8, which need `DEPLOY_IT=1`.
- [x] Contract extension `ModelGrantPort.get_org_policy` (T1), with `DefaultOrgPolicy` as the default.
- [ ] Left to Agent 2 phase 5 (owned by the live phase-5 session; source: "Left to Agent 2" in `todo-finish-plan.md`):
- Web: `apps/web/src/lib/api-engine.ts`, `(app)/manager/**`, `(app)/tasks/**`, `(app)/workforce/**`, `(app)/dashboard/**`, `(app)/traces/**`, `src/test/engine-*.test.tsx`.
- Contracts: the engine event types in `@agy/contracts` (`task.step`, `task.gate`, `run.log`, `run.tool`, `team.changed`).
- Docs: tick `docs/plan/TODO-agent-2-ai-workforce.md` and write `docs/plan/reports/agent-2-final.md`.
- Open decisions: direct tasks (HTTP `POST /tasks`, MCP `tasks_create`) get no pipeline state and still run through the legacy `start_run`. A `paused` task has no resume path in the API yet.
- [x] Engine UI, contracts and owner console (phase 5 and the follow-ups), tracked in `todo-v2-final.md`; its `status:` lines are the source of truth:
- [x] Contracts: the engine event types in `@agy/contracts` (`task.step`, `task.gate`, `run.log`, `run.tool`, `team.changed`) (T20).
- [x] `apps/web/src/lib/api-engine.ts` typed client (T21) and the phase-5 pages: `/manager` (T29–T30), `/tasks` (T31–T32), `/tasks/[id]` (T33–T36), `/workforce` (T37–T39), `/dashboard` and `/traces` (T40–T41).
- [x] Direct tasks (HTTP `POST /tasks`, MCP `tasks_create`) enter the engine at `pending_manager_review` (T14); the legacy `start_run` refuses them with 409.
- [x] Pause, resume, cancel and answer, in the engine API and in the UI, with confirm (T16, T18, T23).
- [x] Employee cap 7 per organization by default (T25).
- [x] Owner credit console `/admin/credits` and its client (T10–T11, T26–T27).
- [x] Docs: `docs/plan/TODO-agent-2-ai-workforce.md` ticked and `docs/plan/reports/agent-2-final.md` written (T43). Three boxes stay open there, each with its reason.
+29 -29
View File
@@ -38,7 +38,7 @@ You implement: `WorkforceBootstrapPort` (wired in `container_engine.py`).
Create package `app/application/engine/` (and `app/domain/workforce/` additions where the logic is pure):
- [ ] 1.1 `domain/workforce/pipeline.py`:
- [x] 1.1 `domain/workforce/pipeline.py`:
- `Stage` enum: `pending_manager_review`, `pending_analysis`, `pending_implementation`, `pending_review`, `pending_test`, `pending_final_review`, `merge_request`, `completed`, plus side states `pending_user_decision`, `error_review`, `paused`, `cancelled`, `failed`.
- `PIPELINES`: `plan_code_test` (default), `plan_code_review_test`, `code_test`, `analysis_only` (no worktree, no merge). Each is an ordered list of stages.
- `TRANSITIONS: dict[Stage, frozenset[Stage]]` and `next_stage(pipeline, current, outcome)`.
@@ -46,7 +46,7 @@ Create package `app/application/engine/` (and `app/domain/workforce/` additions
- `resolve_pipeline(manager_choice, task_kind, app_default, has_active_reviewer)`: priority manager > kind > app default > `plan_code_test`. A review pipeline without an active reviewer falls back to `plan_code_test`.
- `TASK_KINDS`: `feature`, `bugfix`, `refactor`, `docs`, `analysis` → default pipeline + risk.
- `RoleKey.DEVELOPER` is our "programmer". Keep the existing enum values; do not rename them.
- [ ] 1.2 `engine/protocol.py`: parser for `[[TAG attr="v" ...]]body[[/TAG]]` and self-closing `[[TAG attr="v"/]]`. Tolerant of whitespace; rejects unknown tags; returns typed dataclasses. Allowed tags per role/mode:
- [x] 1.2 `engine/protocol.py`: parser for `[[TAG attr="v" ...]]body[[/TAG]]` and self-closing `[[TAG attr="v"/]]`. Tolerant of whitespace; rejects unknown tags; returns typed dataclasses. Allowed tags per role/mode:
- manager/triage: `ASSIGN`, `REJECT`, `ASK_USER`, `KNOWLEDGE`
- manager/final: `COMPLETE`, `REVISE`, `KNOWLEDGE`
- manager/error: `RESOLVE_ERROR`, `CANCEL`, `KNOWLEDGE`
@@ -57,20 +57,20 @@ Create package `app/application/engine/` (and `app/domain/workforce/` additions
- tester: `TEST_PASS`, `TEST_FAIL`
- tool tags (all working roles): `READ_FILE path`, `WRITE_FILE path` (body = content), `EDIT_FILE path` (body = `<<<OLD … === … NEW>>>` search/replace blocks), `LIST_DIR path`, `SEARCH pattern path`, `RUN cmd timeout`
Disallowed tags are ignored and reported back to the model as an error line. Limits: max 3 `CREATE_TASK` per reply, max 3 team changes per turn, max 3 `KNOWLEDGE` per turn.
- [ ] 1.3 Tables in `models_engine.py`:
- [x] 1.3 Tables in `models_engine.py`:
- `task_pipeline_state` (task_id PK/FK, pipeline, kind, risk, stage, cycle, max_cycles, branch, last_diff_hash, same_diff_count, retry_count, waiver_reason, parent_task_id, depends_on JSON, version int for optimistic updates, updated_at)
- `employee_profiles` (employee_id PK/FK, ai_model_id, effort, roles CSV, hired_by `user|manager`, busy_task_id)
- `engine_turns` (id, task_id, run_id, employee_id, role, mode, stage, prompt_chars, reply_excerpt ≤ 4000, tags JSON, tool_calls int, ok, error_kind, started_at, finished_at)
- `gate_results` (id, task_id, gate `tests|diff_guard|no_progress|green|lint`, ok, detail ≤ 8000, created_at)
Map the coarse `tasks.status` from the stage: new / running / waiting_approval / done / failed / cancelled. Existing APIs keep working.
- [ ] 1.4 `engine/state.py`: `advance(task_id, expected_from, to)` with an optimistic check (`version`). It fails cleanly when another worker moved the task first. A failed or cancelled **dependency** moves the dependents to `pending_user_decision`, never leaving them stuck (agybot bug M1).
- [ ] 1.5 Tests `tests/test_engine_protocol.py`, `tests/test_engine_pipeline.py`: every pipeline walks start→completed on the happy path; every illegal transition is refused; concurrent advance (two sessions) → exactly one wins; the dependency failure rule; tag parsing edge cases (nested brackets in a body, unknown tags, the per-role matrix, limits).
- [x] 1.4 `engine/state.py`: `advance(task_id, expected_from, to)` with an optimistic check (`version`). It fails cleanly when another worker moved the task first. A failed or cancelled **dependency** moves the dependents to `pending_user_decision`, never leaving them stuck (agybot bug M1).
- [x] 1.5 Tests `tests/test_engine_protocol.py`, `tests/test_engine_pipeline.py`: every pipeline walks start→completed on the happy path; every illegal transition is refused; concurrent advance (two sessions) → exactly one wins; the dependency failure rule; tag parsing edge cases (nested brackets in a body, unknown tags, the per-role matrix, limits).
---
## Phase 2 — Agent turn runner (one employee, one turn, real tools in the sandbox)
- [ ] 2.1 `engine/prompt.py`, a prompt builder with a **stable prefix** (cache-friendly), in this order:
- [x] 2.1 `engine/prompt.py`, a prompt builder with a **stable prefix** (cache-friendly), in this order:
1. platform rules (short, yours)
2. role protocol (allowed tags + format examples)
3. toolkit description
@@ -79,7 +79,7 @@ Create package `app/application/engine/` (and `app/domain/workforce/` additions
6. task brief (≤ 1,200 chars) + stage-specific context (the planner's spec for the developer; the **diff only** for the reviewer; test output for the tester)
7. the conversation so far in this turn (tool results)
Hard cap on total characters; truncate oldest tool results first.
- [ ] 2.2 `engine/tools.py`: executes tool tags **inside the sandbox** through `SandboxProviderPort`, rooted at `WorktreeRef.sandbox_path`.
- [x] 2.2 `engine/tools.py`: executes tool tags **inside the sandbox** through `SandboxProviderPort`, rooted at `WorktreeRef.sandbox_path`.
- Path safety: normalise; reject absolute paths outside the root, `..` escapes and anything under `.git`.
- `RUN` has a timeout (default 120s, max 600s) and output capped (last 8,000 chars). The command is passed to the sandbox as-is (the sandbox is the security boundary), but `git push`, `git remote` and `git config` are refused with a message ("the platform handles git").
- `EDIT_FILE` applies search/replace blocks and fails with a clear message when OLD is not found exactly once.
@@ -89,20 +89,20 @@ Create package `app/application/engine/` (and `app/domain/workforce/` additions
- Model = the employee's `ai_model_id`, which must be granted (`ModelGrantPort.is_granted`; if not, fail with `error_kind=auth` and tell the manager).
- Error kinds `rate_limit`, `unavailable`, `network`, `timeout` are **transient** (retry later, up to 2 retries); `auth`, `model`, `protocol` are not.
- Writes `engine_turns`, `run_steps` and usage events, and publishes `run.log` / `run.tool` / `run.status` (batched, PLAN-v2 §5).
- Cancellation: `cancel_run` must stop the loop between steps.
- [ ] 2.4 Tests `tests/test_engine_runner.py` with the scripted fake model + `FakeRepoWorkspace` + a fake sandbox (dict filesystem + scripted command results; put it in `tests/engine_fakes.py`). Cover: the write → run → report loop; path escape refused; `git push` refused; the tool-call cap; timeout; transient vs fatal errors; usage events counted per model call; an ungranted model refused; no secret from settings ever appears in a prompt (assert on the fake model's recorded prompts).
- Cancellation: `cancel_run` must stop the loop between steps. — open: run.status is not published for engine runs (engine/scheduler.py sets the Run row's status without an event); run.log and run.tool are published, and stage moves publish task.step instead.
- [x] 2.4 Tests `tests/test_engine_runner.py` with the scripted fake model + `FakeRepoWorkspace` + a fake sandbox (dict filesystem + scripted command results; put it in `tests/engine_fakes.py`). Cover: the write → run → report loop; path escape refused; `git push` refused; the tool-call cap; timeout; transient vs fatal errors; usage events counted per model call; an ungranted model refused; no secret from settings ever appears in a prompt (assert on the fake model's recorded prompts).
---
## Phase 3 — Manager AI (orders → tasks → team)
- [ ] 3.1 Manager chat: rewrite `post_user_message` so the manager replies **and** may emit `CREATE_TASK` (title, kind, priority, depends_on, optional pipeline), `PAUSE_TASK` / `RESUME_TASK` / `CANCEL_TASK`, and `HIRE` / `UPDATE_EMPLOYEE` / `FIRE`. Tasks created this way get a `task_pipeline_state`, stage `pending_manager_review`, and are linked to the thread. The manager's prompt must ask for **atomic tasks that do not touch the same files**. Count chat usage.
- [ ] 3.2 Team management: `HIRE role= tier= name=` picks a granted model matching the tier/capabilities (`ModelGrantPort.granted_models`). Respect max employees (setting + later org policy), max 3 changes per turn, and "the last manager cannot be fired". Publish `team.changed`. Users can still hire and fire from the UI (existing endpoints) but only choose **role + tier**, never a model.
- [ ] 3.3 Triage turn (manager, mode triage): decide pipeline + assignee (`ASSIGN employee= pipeline=`), or `REJECT`, or `ASK_USER` (→ `pending_user_decision`, message posted in the thread). Default employee selection: least busy active employee with the needed role; `ASSIGN employee="name"` uses fuzzy name matching inside the org only.
- [ ] 3.4 Final turn (manager, mode final): sees the plan, a diff summary, gate results and the test output → `COMPLETE` (→ `merge_request`) or `REVISE` (→ back to implementation, cycle+1). The error turn (mode error) handles failed gates and errors → `RESOLVE_ERROR` (retry with a hint) or `CANCEL`.
- [ ] 3.5 `KNOWLEDGE` tags → `MemoryPort.record(kind=fact|rule, source=manager)`.
- [ ] 3.6 `WorkforceBootstrapPort.bootstrap_team(org, project)`: creates one manager + one planner + one developer + one tester on the strongest/balanced/fast granted tiers (whatever is granted; if nothing is granted, create the employees "inactive" and post a system message "منتظر تخصیص مدل توسط مالک سامانه"). Idempotent. Wire it in `container_engine.py`.
- [ ] 3.7 Tests `tests/test_engine_manager.py`: an order → ≤ 3 tasks with deps; HIRE limits; FIRE the last manager refused; ASK_USER path; idempotent bootstrap; cross-org isolation (the manager of org A can never assign an employee of org B, even by exact name).
- [x] 3.1 Manager chat: rewrite `post_user_message` so the manager replies **and** may emit `CREATE_TASK` (title, kind, priority, depends_on, optional pipeline), `PAUSE_TASK` / `RESUME_TASK` / `CANCEL_TASK`, and `HIRE` / `UPDATE_EMPLOYEE` / `FIRE`. Tasks created this way get a `task_pipeline_state`, stage `pending_manager_review`, and are linked to the thread. The manager's prompt must ask for **atomic tasks that do not touch the same files**. Count chat usage.
- [x] 3.2 Team management: `HIRE role= tier= name=` picks a granted model matching the tier/capabilities (`ModelGrantPort.granted_models`). Respect max employees (setting + later org policy), max 3 changes per turn, and "the last manager cannot be fired". Publish `team.changed`. Users can still hire and fire from the UI (existing endpoints) but only choose **role + tier**, never a model.
- [x] 3.3 Triage turn (manager, mode triage): decide pipeline + assignee (`ASSIGN employee= pipeline=`), or `REJECT`, or `ASK_USER` (→ `pending_user_decision`, message posted in the thread). Default employee selection: least busy active employee with the needed role; `ASSIGN employee="name"` uses fuzzy name matching inside the org only.
- [x] 3.4 Final turn (manager, mode final): sees the plan, a diff summary, gate results and the test output → `COMPLETE` (→ `merge_request`) or `REVISE` (→ back to implementation, cycle+1). The error turn (mode error) handles failed gates and errors → `RESOLVE_ERROR` (retry with a hint) or `CANCEL`.
- [x] 3.5 `KNOWLEDGE` tags → `MemoryPort.record(kind=fact|rule, source=manager)`.
- [x] 3.6 `WorkforceBootstrapPort.bootstrap_team(org, project)`: creates one manager + one planner + one developer + one tester on the strongest/balanced/fast granted tiers (whatever is granted; if nothing is granted, create the employees "inactive" and post a system message "منتظر تخصیص مدل توسط مالک سامانه"). Idempotent. Wire it in `container_engine.py`.
- [x] 3.7 Tests `tests/test_engine_manager.py`: an order → ≤ 3 tasks with deps; HIRE limits; FIRE the last manager refused; ASK_USER path; idempotent bootstrap; cross-org isolation (the manager of org A can never assign an employee of org B, even by exact name).
---
@@ -112,30 +112,30 @@ Create package `app/application/engine/` (and `app/domain/workforce/` additions
- finds tasks whose stage needs a turn and whose deps are completed, and a free employee with the role (one task per employee at a time)
- honours org automation (if Agent 1's policy says `automation_enabled=false`, tasks wait in `pending_manager_review`; read it through `ModelGrantPort` when the CHANGE REQUEST lands, until then assume enabled)
- runs turns concurrently with a per-org cap (3) and a global cap (setting)
`workers/ai_runtime.py` calls `tick()` in its loop every `ENGINE_TICK_SECONDS` (keep the one-pass mode for cron). Safe with two worker processes thanks to the optimistic `advance`.
- [ ] 4.2 Worktree lifecycle: on the first developer/planner turn, `RepoWorkspacePort.create_task_worktree(project, task, base="dev")`; the branch is `task/<task_id>`. `analysis_only` gets none. The worktree is removed after completion, cancellation or failure.
- [ ] 4.3 Gates after `WORK_REPORT` (each writes a `gate_results` row and publishes `task.gate`):
`workers/ai_runtime.py` calls `tick()` in its loop every `ENGINE_TICK_SECONDS` (keep the one-pass mode for cron). Safe with two worker processes thanks to the optimistic `advance`. — open: turns run one after another in tick() (a plain loop over the selected states); the per-org and global caps only bound the selection, there is no concurrent execution.
- [x] 4.2 Worktree lifecycle: on the first developer/planner turn, `RepoWorkspacePort.create_task_worktree(project, task, base="dev")`; the branch is `task/<task_id>`. `analysis_only` gets none. The worktree is removed after completion, cancellation or failure.
- [x] 4.3 Gates after `WORK_REPORT` (each writes a `gate_results` row and publishes `task.gate`):
- **diff guard**: `RepoWorkspacePort.diff`. Fail on changes under `.git`, CI files (`.github/`, `.gitlab-ci.yml`, `.gitea/`), symlinks pointing outside the repo, or files > 1 MB. An empty diff for a code task also fails.
- **no-progress**: hash the diff; 2 identical hashes in a row → `error_review`.
- **tests**: run the app profile's `test_command` in the sandbox (timeout 600s), store the tail of the output. Red → back to the developer with the failure tail (one escalation to a stronger granted tier on the 2nd red), counting cycles. Green → tester stage.
- **green gate**: the tester's `TEST_PASS` is accepted only if the tests gate is green, OR `TEST_PASS waive="reason"` with a non-empty reason (stored in `waiver_reason`, shown in UI).
- `max_cycles` reached → `error_review`.
- [ ] 4.4 Commit + merge + push on `merge_request`:
- [x] 4.4 Commit + merge + push on `merge_request`:
- `commit` (message: `task/<id>: <title>`, author = employee display name + platform email), then `sync_with_base`, then re-run the tests if base changed (a semantic conflict is only caught by retests).
- Then `merge_into(target="dev")` and `push(branch="dev")`.
- A conflict → back to the developer with the conflicting files (max 3 times, then `error_review`).
- On success: stage `completed`, `DeliveryHooks.on_branch_pushed(org, project, "dev", sha, source="engine")`, `MemoryPort.on_task_finished(TaskOutcome)` (never let its failure fail the task).
- [ ] 4.5 Budget: before each turn, check the org credit wallet (existing billing service public functions; read, do not edit) and the per-task cap. Out of budget → `paused` with a Persian reason.
- [ ] 4.6 Tests `tests/test_engine_flow.py`, the end-to-end engine with all fakes. The scripted model plays manager → planner → developer → tester → manager; the fake sandbox makes tests red once and then green. Assert the stage sequence, gate rows, cycles, the commit/merge/push calls on `FakeRepoWorkspace`, `on_branch_pushed` called once with the right sha, `on_task_finished` called once, and usage events. Plus the failure paths: no-progress, max cycles, merge conflict ×3, cancel mid-turn, a dependency failure.
- [x] 4.5 Budget: before each turn, check the org credit wallet (existing billing service public functions; read, do not edit) and the per-task cap. Out of budget → `paused` with a Persian reason.
- [ ] 4.6 Tests `tests/test_engine_flow.py`, the end-to-end engine with all fakes. The scripted model plays manager → planner → developer → tester → manager; the fake sandbox makes tests red once and then green. Assert the stage sequence, gate rows, cycles, the commit/merge/push calls on `FakeRepoWorkspace`, `on_branch_pushed` called once with the right sha, `on_task_finished` called once, and usage events. Plus the failure paths: no-progress, max cycles, merge conflict ×3, cancel mid-turn, a dependency failure. — open: tests/test_engine_flow.py has no max-cycles → error_review test and only one merge conflict (retry), not three; the blocking no-progress case, cancel mid-turn and the dependency-failure rule are asserted in test_engine_gates.py, test_engine_runner.py and test_engine_pipeline.py.
---
## Phase 5 — UI (orders, live task view, team)
- [ ] 5.1 `lib/api-engine.ts`: typed client functions for everything new (task detail with pipeline state, turns, gates, team with tier, hire/fire with role+tier).
- [ ] 5.2 `/manager`: the chat shows task cards created by the manager inline (title, stage badge, link), live via `org:` + `thread:` channels; an "در انتظار تصمیم شما" banner for `pending_user_decision` with a reply box.
- [ ] 5.3 `/tasks`: a board grouped by stage (columns: بررسی مدیر، تحلیل، پیاده‌سازی، بازبینی، تست، ادغام، انجام‌شده، نیازمند تصمیم) with a live update; filters; pause/resume/cancel (with confirm).
- [ ] 5.4 `/tasks/[id]`:
- [x] 5.1 `lib/api-engine.ts`: typed client functions for everything new (task detail with pipeline state, turns, gates, team with tier, hire/fire with role+tier).
- [x] 5.2 `/manager`: the chat shows task cards created by the manager inline (title, stage badge, link), live via `org:` + `thread:` channels; an "در انتظار تصمیم شما" banner for `pending_user_decision` with a reply box.
- [x] 5.3 `/tasks`: a board grouped by stage (columns: بررسی مدیر، تحلیل، پیاده‌سازی، بازبینی، تست، ادغام، انجام‌شده، نیازمند تصمیم) with a live update; filters; pause/resume/cancel (with confirm).
- [x] 5.4 `/tasks/[id]`:
- a pipeline stepper (stages of its pipeline, current one highlighted)
- the turn timeline (who, role, when, outcome)
- a **live log** panel (subscribe to `run:<id>`, auto-scroll, pause-scroll)
@@ -143,9 +143,9 @@ Create package `app/application/engine/` (and `app/domain/workforce/` additions
- a diff viewer (read-only, from the latest diff)
- the waiver reason when present
- a link to the pipeline run in `/app` once pushed (just a link with the sha; that page is Agent 3's)
- [ ] 5.5 `/workforce`: employees with role, tier, status, current task; hire (role + tier) and fire with confirm. **No model names anywhere** for non-owners (test it).
- [ ] 5.6 `/dashboard` + `/traces`: show active tasks, a stage distribution and recent completions; keep the existing widgets working.
- [ ] 5.7 Web tests `src/test/engine-*.test.tsx`: board grouping, live log append and batching, no model name rendered (render with a fixture that contains a model name and assert it is absent).
- [x] 5.5 `/workforce`: employees with role, tier, status, current task; hire (role + tier) and fire with confirm. **No model names anywhere** for non-owners (test it).
- [x] 5.6 `/dashboard` + `/traces`: show active tasks, a stage distribution and recent completions; keep the existing widgets working.
- [x] 5.7 Web tests `src/test/engine-*.test.tsx`: board grouping, live log append and batching, no model name rendered (render with a fixture that contains a model name and assert it is absent).
---
+216
View File
@@ -0,0 +1,216 @@
# Agent 2 — Final report: AI workforce engine
Scope: phases 1–5 of `docs/plan/TODO-agent-2-ai-workforce.md`, plus the prepaid billing and direct-task
changes made after them (section 10). Every fact here was checked against the code with grep or `ls`.
## 1. Status
- TODO boxes: **26 of 29 ticked**. Three stay open, each with its reason in the TODO:
- 2.3: `run.status` is not published for engine runs (run.log and run.tool are).
- 4.1: turns run one after another inside a tick; the per-org and global caps only bound what a tick selects.
- 4.6: `tests/test_engine_flow.py` lacks the max-cycles and three-conflict failure tests (section 11).
- Every other TODO item has its code, its tests and (for phase 5) its UI in the tree.
- CHANGE REQUESTs still open: none (section 8).
## 2. Architecture
```
order (manager chat, or POST /tasks)
-> manager chat (ManagerTurns.chat; may emit CREATE_TASK)
-> task + task_pipeline_state at pending_manager_review
-> scheduler tick; each stage move is one turn or one gate:
pending_manager_review manager triage: ASSIGN | REJECT | ASK_USER
pending_analysis planner: PLAN_SPEC | SPLIT (tools)
pending_implementation developer: WORK_REPORT (tools)
gates after it: diff guard, no-progress, tests
pending_review reviewer: REVIEW_PASS | REVIEW_CHANGES (plan_code_review_test only)
pending_test tester: TEST_PASS | TEST_FAIL; green gate or waiver
pending_final_review manager final: COMPLETE | REVISE (REVISE goes back, cycle + 1)
merge_request commit; sync with base (retest if base moved); merge into dev; push dev
-> DeliveryHooks.on_branch_pushed(branch="dev", source="engine")
completed MemoryPort.on_task_finished (its failure never fails the task)
side states: pending_user_decision, error_review (manager error mode: RESOLVE_ERROR | CANCEL),
paused (no credit or a budget cap), cancelled, failed
```
Pipelines (`app/domain/workforce/pipeline.py`, `PIPELINES`):
- `plan_code_test` (default): manager review, analysis, implementation, test, final review, merge, completed.
- `plan_code_review_test`: as above with a review stage after implementation; falls back to `plan_code_test`
when no reviewer is active.
- `code_test`: no planner stage.
- `analysis_only`: manager review, analysis, completed. No worktree and no merge.
The AI never runs git. `RepoWorkspacePort` does commit, sync, merge and push. The `RUN` tool refuses
`git push`, `git remote` and `git config`.
## 3. Tag protocol (`app/application/engine/protocol.py`, `ALLOWED_TAGS`)
| role / mode | allowed tags |
|---|---|
| manager / triage | ASSIGN, REJECT, ASK_USER, KNOWLEDGE |
| manager / final | COMPLETE, REVISE, KNOWLEDGE |
| manager / error | RESOLVE_ERROR, CANCEL, KNOWLEDGE |
| manager / chat | CREATE_TASK, PAUSE_TASK, RESUME_TASK, CANCEL_TASK, HIRE, UPDATE_EMPLOYEE, FIRE, KNOWLEDGE |
| planner / work | PLAN_SPEC, SPLIT, ASK_USER + toolkit |
| developer / work | WORK_REPORT, ASK_USER + toolkit |
| reviewer / work | REVIEW_PASS, REVIEW_CHANGES + toolkit |
| tester / work | TEST_PASS, TEST_FAIL + toolkit |
Toolkit (working roles only): `READ_FILE`, `WRITE_FILE`, `EDIT_FILE`, `LIST_DIR`, `SEARCH`, `RUN`.
Limits: at most 3 `CREATE_TASK` per reply, 3 team changes per turn (HIRE, UPDATE_EMPLOYEE, FIRE together),
3 `KNOWLEDGE` per turn. A disallowed tag is dropped and reported back to the model as an error line.
## 4. Endpoints added or changed
Engine (`app/interfaces/http/routers/engine.py`, prefix `/api/organizations/{org_id}/engine`):
- `GET /status`: the engine status, including the wallet balance. Every member may read it, so they can see
why work stopped.
- `GET /tasks`: the task board (engine tasks with stage, pipeline, kind, cycle and assignee).
- `GET /tasks/{task_id}`: task detail with pipeline state, turns, gates, last diff, waiver reason and merged sha.
- `GET /turns`: the organization's newest engine turns.
- `POST /tasks/{task_id}/pause`, `/resume`, `/cancel`, `/answer`: resume is refused while the wallet is at zero.
A task of another organization is a 404.
- `GET /team`: employees with role, tier, status and busy task. Non-owners see tier only, never a model.
- `POST /team/hire` (role + tier) and `POST /team/{employee_id}/fire`.
Owner credit console (`app/interfaces/http/routers/admin_credits.py`, platform owner only):
- `GET /api/admin/credits`: the balance of every organization.
- `POST /api/admin/orgs/{org_id}/credit`: a positive credit with a required reason, capped per request; writes one
history row of kind `owner_credit`.
- `GET /api/admin/orgs/{org_id}/credit-history`.
Billing lockdown (`app/interfaces/http/routers/billing.py`):
- Changed: `POST /api/organizations/{org_id}/billing/add-credit` is platform-owner only. Org admins are refused.
A negative amount claws back, but never leaves the wallet below zero. It writes one `adjustment` row.
- Changed: `POST /api/organizations/{org_id}/billing/payments/{payment_id}/settle` is platform-owner only. A
succeeded settle credits the wallet through the credit ledger as one `adjustment` row. An already-final
payment is returned untouched.
- Unchanged: `checkout` only creates a pending payment.
Workforce (`app/interfaces/http/routers/workforce.py`, `app/application/services/workforce_service.py`):
- Changed: `POST /api/organizations/{org_id}/tasks` creates a pipeline state at `pending_manager_review` with
kind `feature`, so the engine picks the task up. The MCP `tasks_create` tool calls the same `create_task`.
- Changed: the legacy run start (`WorkforceService.start_run`, the «اجرا» button) answers 409 for a task the
engine owns.
## 5. Tables
- `task_pipeline_state` (one row per engine task): pipeline, kind, risk, stage, cycle, max_cycles, branch,
last_diff_hash, same_diff_count, retry_count, waiver_reason, parent_task_id, depends_on (JSON), version
(optimistic lock), `last_diff` (capped at 20,000 characters), `merged_sha`.
- `employee_profiles` (one row per employee): `ai_model_id` (internal; never returned to non-owners), effort,
roles, `hired_by` (`user` or `manager`), `busy_task_id`.
- `engine_turns` (one row per employee turn): task, run, employee, role, mode, stage, prompt_chars,
`reply_excerpt` (at most 4,000 characters), tags (JSON), tool_calls, ok, error_kind, started_at, finished_at.
- `gate_results` (one row per gate run): gate (`tests`, `diff_guard`, `no_progress`, `green`, `lint`), ok,
detail (at most 8,000 characters).
- `credit_transactions` (`app/infrastructure/db/models_billing.py`): the history of each organization's prepaid
wallet, with owner adjustments and usage debits. Unique on `usage_event_id`.
- Migration of the engine tables: `alembic/versions/a2b3c4d5e6f2_ai_workforce.py`.
## 6. Settings (`app/core/config.py`)
Engine (`ENGINE_*`):
| setting | default | meaning |
|---|---|---|
| `ENGINE_MAX_TOOL_CALLS_PER_TURN` | 40 | tool calls allowed in one turn |
| `ENGINE_MAX_PROTOCOL_RETRIES` | 3 | retries after a protocol error in one turn |
| `ENGINE_TURN_TIMEOUT_SECONDS` | 900 | wall-clock limit for one turn |
| `ENGINE_MAX_CONCURRENT_TURNS` | 8 | global cap on selected turns per tick |
| `ENGINE_MAX_TURNS_PER_ORG` | 3 | per-organization cap on selected turns per tick |
| `ENGINE_TESTS_TIMEOUT_SECONDS` | 600 | timeout for the project's test command |
| `ENGINE_MAX_MERGE_CONFLICTS` | 3 | conflicts before `error_review` |
| `ENGINE_MAX_TASK_CREDITS` | 0.0 | per-task spend cap; 0 disables it |
| `ENGINE_MAX_CYCLES` | 6 | implementation cycles before `error_review` |
| `ENGINE_MAX_EMPLOYEES` | 20 | platform ceiling on employees per organization |
| `ENGINE_TICK_SECONDS` | 10.0 | pause between worker ticks in loop mode |
| `ENGINE_PROMPT_CHAR_CAP` | 24000 | hard cap on one turn's prompt; the oldest tool results are cut first |
| `ENGINE_TOOL_OUTPUT_CHARS` | 8000 | tail kept from one tool output |
| `ENGINE_REPLY_EXCERPT_CHARS` | 4000 | reply text stored in `engine_turns` |
| `ENGINE_LOG_FLUSH_SECONDS` | 0.5 | batching window for live log events |
| `ENGINE_LOG_MAX_LINES_PER_EVENT` | 200 | lines per `run.log` event |
Billing: `BILLING_STARTING_CREDIT` (default 0.0) is the credit a new organization starts with.
The per-organization employee cap is 7 by default (the org policy, section 10). `ENGINE_MAX_EMPLOYEES` is only
the ceiling.
## 7. Realtime events
- `task.step` (`engine/events.py`): on every stage move. Channels: `task:` and `org:`.
- `task.gate`: after each gate result. Channels: `task:` and `org:`.
- `run.log` and `run.tool`: batched run output and tool calls. Channels: `run:`, `task:` and `org:`.
- `team.changed`: on hire, fire and update. Channel: `org:`.
- `task.created`: when the manager creates a task (CREATE_TASK) and when `create_task` runs. Channel: `task:`.
- `run.status` is not published for engine runs (open 2.3).
## 8. CHANGE REQUESTs
None open. The two expected CRs have landed:
- The org policy is on `ModelGrantPort` (`app/application/ports/model_grants.py`, `get_org_policy`).
- The memory expiry hook runs in the billing worker pass (`app/workers/billing.py`, `MemoryRepository.expire`).
## 9. Manual QA (owner, on a real host)
These steps are not run by the agent. The owner runs them:
1. Owner credits an organization at `/admin/credits`.
2. Order work in `/manager`: the task cards appear inline, and the `/tasks` board moves live.
3. Open a task: stepper, turn timeline, live log, gates and diff. Pause, resume and cancel (each asks to confirm).
4. At zero credit, work pauses and resume is refused until credit is added.
5. On `/workforce`, hire and fire by role and tier. No model name appears for a member.
6. A pushed task shows its short sha, with a link to `/app`.
## 10. Changes outside the original TODO (owner-approved)
- Settle credits through the ledger: `BillingService.settle_payment` calls `credit_ledger.apply`.
- Wallet debit on usage: every usage row debits its organization's wallet once (`UsageLedgerRepository.record`,
keyed on the usage row id).
- Prepaid pause: a zero, negative or missing wallet pauses engine turns and the manager chat
(`Scheduler._budget_reason`, `balance <= 0`).
- Employee cap 7 per organization by default (`DEFAULT_MAX_EMPLOYEES` in `model_grant_service.py`); the admin
page default is 7 too.
- Owner credit page `/admin/credits`, with credit history.
- Direct tasks enter the engine: kind `feature` at `pending_manager_review`. The legacy run refuses them.
- Pause, resume, cancel and answer, in the API and in the UI (with confirm).
- Reference count in the web realtime client (`apps/web/src/lib/ws.ts`): a channel stays subscribed while any
live hook still holds it. The server hears about it only when the last holder leaves, and a reconnect replays
only the channels still held.
## 11. Known limitations and leftovers
Deliberate limitations:
- Turns run one after another inside a tick (open 4.1). The caps bound what a tick selects, not how many run at once.
- The stored diff is capped at 20,000 characters.
- A budget-paused task is not resumed automatically when credit is added. Resume it by hand; it is refused while
the credit is still zero.
- Invoices from the billing worker are informational under prepaid. They debit nothing; the usage ledger debits
the wallet.
- The manager chat is also refused at zero credit, because it costs model calls.
Open TODO items:
- 2.3: `run.status` is not published for engine runs. `engine/scheduler.py` sets the run row's status without an
event, so stage moves (`task.step`) are the only live signal.
- 4.1: no concurrent turns inside a tick (see above).
- 4.6: `tests/test_engine_flow.py` has no max-cycles test (`max_cycles` reaching `error_review`) and only one
merge conflict (retry), not three. Its no-progress case is only a passing gate row. The blocking no-progress
case is in `tests/test_engine_gates.py`. Cancel mid-turn is in `tests/test_engine_runner.py`. The dependency
failure rule is in `tests/test_engine_pipeline.py`.
Minor leftovers, known and not fixed:
- The billing page's payments card shows the payment adapter name (`mock`) to members.
- Paused tasks do not highlight the stage they paused at.
- Engine task pages do not show legacy approval cards.
- Two concurrent settles of one payment could credit twice: `PaymentRepository.set_status` is unconditional.
- The generic Persian word «مدل» appears in two non-owner hints.
- Manual real-environment QA (docker, caddy, ssh/ftp) is for the owner on a real host.
## 12. Deviations from the TODO wording
The behaviour is the same; the names or channels differ:
- 1.1: `TASK_KINDS` is the `TaskKind` enum plus `TASK_KIND_DEFAULTS` (`app/domain/workforce/pipeline.py`).
- 2.1: the memory context is built in `runner.py` (`_build_prompt`, budget 2,500) and passed to `prompt.py`.
- 3.1: `post_user_message` is `ManagerTurns.chat` (`app/application/engine/manager.py`).
- 5.4: the live log subscribes to `task:<id>` (`apps/web/src/app/(app)/tasks/[id]/_components/live-log.tsx`),
not `run:<id>`. The `task:` channel also carries `run.log`.
+10 -5
View File
@@ -22,7 +22,7 @@ and nothing reaches production without a human click.
| 1 | `DockerSandboxAdapter` (hardened workspace + demo containers), `process.py` (the single hardened subprocess runner), orphan reaping, infra hidden from non-owners |
| 2 | `LocalGitWorkspaceAdapter` — the platform's own git: per-project gitdir, worktrees per task, per-repo lock, hooks disabled, credentials only ever as a *path* |
| 3 | Live CI/CD: pipeline runs/steps/releases/runtimes, build containers, blue/green demo deploy, live batched logs, remote polling |
| 4 | Owner domain console: list/attach/detach/resync routes, custom FQDNs behind a setting, MCP gateway system route |
| 4 | Owner domain console: list/attach/detach/resync routes, custom FQDNs behind a setting, MCP gateway as a read-only suggestion (`GET /suggested`; the `mcp` label is protected, so it cannot be attached) |
| 5 | Production deploy: `asyncssh` + `aioftp` adapters with host-key TOFU, atomic symlink switch, FTP backup-first, `production_deploys` ledger |
| 6 | The Application page (four tabs + live build console + production panel), `/admin/domains`, `/admin/sandbox`, `/projects* → /app` redirect, typed client + contracts |
@@ -98,7 +98,7 @@ All paths are under `/api`.
| Method | Path | Notes |
|---|---|---|
| GET | `` | every route, app routes and system routes |
| GET | `` | every attached app route. The MCP gateway host is a read-only suggestion (`/suggested`), not a route the owner can enable: the label `mcp` is protected |
| POST | `` | attach a host (label, or a custom FQDN when the setting is on) |
| DELETE | `/{domain_id}` | detach, host typed back as confirmation |
| POST | `/sync` | idempotent re-sync from the database |
@@ -125,9 +125,12 @@ All four endpoints (ports, domains create/list/delete) became platform-owner onl
## 5. Tables (all in `models_delivery.py`)
`pipeline_runs`, `pipeline_steps`, `releases`, `app_runtimes`,
`production_deploys`, `system_routes`. Migration
`production_deploys`. Migration
`a3b4c5d6e7f3_delivery_infra` (`down_revision = "a0f0e1d2c3b4"`).
The MCP gateway has no table: `GET /api/admin/domains/suggested` computes its
suggestion from settings.
---
## 6. Manual QA script
@@ -157,8 +160,8 @@ Run against a real docker + caddy host with
8. **Production, FTP**: the same against a `pure-ftpd` container; expect a
`ftp-backup-<ts>.tar.gz` and, on rollback, that archive re-uploaded.
9. **Domains**: as the owner attach a second label; re-sync twice and check the
second call reports `up_to_date: true` with no Caddy reload. Enable the MCP
system route; `mcp.<root>` must answer through the gateway port.
second call reports `up_to_date: true` with no Caddy reload. Check that
`GET /api/admin/domains/suggested` shows `mcp.<root>` as a suggestion (`enabled=false`).
10. **SSRF**: attempt a production deploy to `127.0.0.1`, `10.0.0.1` and
`169.254.169.254`. All three must be refused before any connection.
11. **Tenant isolation**: a member of org B gets 404 on every org-A delivery and
@@ -213,3 +216,5 @@ Run against a real docker + caddy host with
(D6), but it also means Agent 2's engine must go through `RepoWorkspacePort`
for all of it, and a git failure surfaces to the customer as a Persian message
rather than a git exit code.
- **No sandbox restart.** `/admin/sandbox` lists and stops containers; there is
no restart action (TODO 6.3 open).