smart-org/docs/plan/reports/agent-2-final.md

Agent 2 — Final report: AI workforce engine

Scope: phases 1–5 of docs/plan/TODO-agent-2-ai-workforce.md, plus the prepaid billing and direct-task changes made after them (section 10). Every fact here was checked against the code with grep or ls.

1. Status

• TODO boxes: 26 of 29 ticked. Three stay open, each with its reason in the TODO:
  • 2.3: run.status is not published for engine runs (run.log and run.tool are).
  • 4.1: turns run one after another inside a tick; the per-org and global caps only bound what a tick selects.
  • 4.6: tests/test_engine_flow.py lacks the max-cycles and three-conflict failure tests (section 11).
• Every other TODO item has its code, its tests and (for phase 5) its UI in the tree.
• CHANGE REQUESTs still open: none (section 8).

2. Architecture

order (manager chat, or POST /tasks)
  -> manager chat (ManagerTurns.chat; may emit CREATE_TASK)
  -> task + task_pipeline_state at pending_manager_review
  -> scheduler tick; each stage move is one turn or one gate:
     pending_manager_review   manager triage: ASSIGN | REJECT | ASK_USER
     pending_analysis         planner: PLAN_SPEC | SPLIT (tools)
     pending_implementation   developer: WORK_REPORT (tools)
                              gates after it: diff guard, no-progress, tests
     pending_review           reviewer: REVIEW_PASS | REVIEW_CHANGES (plan_code_review_test only)
     pending_test             tester: TEST_PASS | TEST_FAIL; green gate or waiver
     pending_final_review     manager final: COMPLETE | REVISE (REVISE goes back, cycle + 1)
     merge_request            commit; sync with base (retest if base moved); merge into dev; push dev
                              -> DeliveryHooks.on_branch_pushed(branch="dev", source="engine")
     completed                MemoryPort.on_task_finished (its failure never fails the task)
  side states: pending_user_decision, error_review (manager error mode: RESOLVE_ERROR | CANCEL),
               paused (no credit or a budget cap), cancelled, failed

Pipelines (app/domain/workforce/pipeline.py, PIPELINES):

• plan_code_test (default): manager review, analysis, implementation, test, final review, merge, completed.
• plan_code_review_test: as above with a review stage after implementation; falls back to plan_code_test when no reviewer is active.
• code_test: no planner stage.
• analysis_only: manager review, analysis, completed. No worktree and no merge.

The AI never runs git. RepoWorkspacePort does commit, sync, merge and push. The RUN tool refuses git push, git remote and git config.

3. Tag protocol (app/application/engine/protocol.py, ALLOWED_TAGS)

role / mode	allowed tags
manager / triage	ASSIGN, REJECT, ASK_USER, KNOWLEDGE
manager / final	COMPLETE, REVISE, KNOWLEDGE
manager / error	RESOLVE_ERROR, CANCEL, KNOWLEDGE
manager / chat	CREATE_TASK, PAUSE_TASK, RESUME_TASK, CANCEL_TASK, HIRE, UPDATE_EMPLOYEE, FIRE, KNOWLEDGE
planner / work	PLAN_SPEC, SPLIT, ASK_USER + toolkit
developer / work	WORK_REPORT, ASK_USER + toolkit
reviewer / work	REVIEW_PASS, REVIEW_CHANGES + toolkit
tester / work	TEST_PASS, TEST_FAIL + toolkit

Toolkit (working roles only): READ_FILE, WRITE_FILE, EDIT_FILE, LIST_DIR, SEARCH, RUN.

Limits: at most 3 CREATE_TASK per reply, 3 team changes per turn (HIRE, UPDATE_EMPLOYEE, FIRE together), 3 KNOWLEDGE per turn. A disallowed tag is dropped and reported back to the model as an error line.

4. Endpoints added or changed

Engine (app/interfaces/http/routers/engine.py, prefix /api/organizations/{org_id}/engine):

• GET /status: the engine status, including the wallet balance. Every member may read it, so they can see why work stopped.
• GET /tasks: the task board (engine tasks with stage, pipeline, kind, cycle and assignee).
• GET /tasks/{task_id}: task detail with pipeline state, turns, gates, last diff, waiver reason and merged sha.
• GET /turns: the organization's newest engine turns.
• POST /tasks/{task_id}/pause, /resume, /cancel, /answer: resume is refused while the wallet is at zero. A task of another organization is a 404.
• GET /team: employees with role, tier, status and busy task. Non-owners see tier only, never a model.
• POST /team/hire (role + tier) and POST /team/{employee_id}/fire.

Owner credit console (app/interfaces/http/routers/admin_credits.py, platform owner only):

• GET /api/admin/credits: the balance of every organization.
• POST /api/admin/orgs/{org_id}/credit: a positive credit with a required reason, capped per request; writes one history row of kind owner_credit.
• GET /api/admin/orgs/{org_id}/credit-history.

Billing lockdown (app/interfaces/http/routers/billing.py):

• Changed: POST /api/organizations/{org_id}/billing/add-credit is platform-owner only. Org admins are refused. A negative amount claws back, but never leaves the wallet below zero. It writes one adjustment row.
• Changed: POST /api/organizations/{org_id}/billing/payments/{payment_id}/settle is platform-owner only. A succeeded settle credits the wallet through the credit ledger as one adjustment row. An already-final payment is returned untouched.
• Unchanged: checkout only creates a pending payment.

Workforce (app/interfaces/http/routers/workforce.py, app/application/services/workforce_service.py):

• Changed: POST /api/organizations/{org_id}/tasks creates a pipeline state at pending_manager_review with kind feature, so the engine picks the task up. The MCP tasks_create tool calls the same create_task.
• Changed: the legacy run start (WorkforceService.start_run, the «اجرا» button) answers 409 for a task the engine owns.

5. Tables

• task_pipeline_state (one row per engine task): pipeline, kind, risk, stage, cycle, max_cycles, branch, last_diff_hash, same_diff_count, retry_count, waiver_reason, parent_task_id, depends_on (JSON), version (optimistic lock), last_diff (capped at 20,000 characters), merged_sha.
• employee_profiles (one row per employee): ai_model_id (internal; never returned to non-owners), effort, roles, hired_by (user or manager), busy_task_id.
• engine_turns (one row per employee turn): task, run, employee, role, mode, stage, prompt_chars, reply_excerpt (at most 4,000 characters), tags (JSON), tool_calls, ok, error_kind, started_at, finished_at.
• gate_results (one row per gate run): gate (tests, diff_guard, no_progress, green, lint), ok, detail (at most 8,000 characters).
• credit_transactions (app/infrastructure/db/models_billing.py): the history of each organization's prepaid wallet, with owner adjustments and usage debits. Unique on usage_event_id.
• Migration of the engine tables: alembic/versions/a2b3c4d5e6f2_ai_workforce.py.

6. Settings (app/core/config.py)

Engine (ENGINE_*):

setting	default	meaning
ENGINE_MAX_TOOL_CALLS_PER_TURN	40	tool calls allowed in one turn
ENGINE_MAX_PROTOCOL_RETRIES	3	retries after a protocol error in one turn
ENGINE_TURN_TIMEOUT_SECONDS	900	wall-clock limit for one turn
ENGINE_MAX_CONCURRENT_TURNS	8	global cap on selected turns per tick
ENGINE_MAX_TURNS_PER_ORG	3	per-organization cap on selected turns per tick
ENGINE_TESTS_TIMEOUT_SECONDS	600	timeout for the project's test command
ENGINE_MAX_MERGE_CONFLICTS	3	conflicts before error_review
ENGINE_MAX_TASK_CREDITS	0.0	per-task spend cap; 0 disables it
ENGINE_MAX_CYCLES	6	implementation cycles before error_review
ENGINE_MAX_EMPLOYEES	20	platform ceiling on employees per organization
ENGINE_TICK_SECONDS	10.0	pause between worker ticks in loop mode
ENGINE_PROMPT_CHAR_CAP	24000	hard cap on one turn's prompt; the oldest tool results are cut first
ENGINE_TOOL_OUTPUT_CHARS	8000	tail kept from one tool output
ENGINE_REPLY_EXCERPT_CHARS	4000	reply text stored in engine_turns
ENGINE_LOG_FLUSH_SECONDS	0.5	batching window for live log events
ENGINE_LOG_MAX_LINES_PER_EVENT	200	lines per run.log event

Billing: BILLING_STARTING_CREDIT (default 0.0) is the credit a new organization starts with.

The per-organization employee cap is 7 by default (the org policy, section 10). ENGINE_MAX_EMPLOYEES is only the ceiling.

7. Realtime events

• task.step (engine/events.py): on every stage move. Channels: task: and org:.
• task.gate: after each gate result. Channels: task: and org:.
• run.log and run.tool: batched run output and tool calls. Channels: run:, task: and org:.
• team.changed: on hire, fire and update. Channel: org:.
• task.created: when the manager creates a task (CREATE_TASK) and when create_task runs. Channel: task:.
• run.status is not published for engine runs (open 2.3).

8. CHANGE REQUESTs

None open. The two expected CRs have landed:

• The org policy is on ModelGrantPort (app/application/ports/model_grants.py, get_org_policy).
• The memory expiry hook runs in the billing worker pass (app/workers/billing.py, MemoryRepository.expire).

9. Manual QA (owner, on a real host)

These steps are not run by the agent. The owner runs them:

1. Owner credits an organization at /admin/credits.
2. Order work in /manager: the task cards appear inline, and the /tasks board moves live.
3. Open a task: stepper, turn timeline, live log, gates and diff. Pause, resume and cancel (each asks to confirm).
4. At zero credit, work pauses and resume is refused until credit is added.
5. On /workforce, hire and fire by role and tier. No model name appears for a member.
6. A pushed task shows its short sha, with a link to /app.

10. Changes outside the original TODO (owner-approved)

• Settle credits through the ledger: BillingService.settle_payment calls credit_ledger.apply.
• Wallet debit on usage: every usage row debits its organization's wallet once (UsageLedgerRepository.record, keyed on the usage row id).
• Prepaid pause: a zero, negative or missing wallet pauses engine turns and the manager chat (Scheduler._budget_reason, balance <= 0).
• Employee cap 7 per organization by default (DEFAULT_MAX_EMPLOYEES in model_grant_service.py); the admin page default is 7 too.
• Owner credit page /admin/credits, with credit history.
• Direct tasks enter the engine: kind feature at pending_manager_review. The legacy run refuses them.
• Pause, resume, cancel and answer, in the API and in the UI (with confirm).
• Reference count in the web realtime client (apps/web/src/lib/ws.ts): a channel stays subscribed while any live hook still holds it. The server hears about it only when the last holder leaves, and a reconnect replays only the channels still held.

11. Known limitations and leftovers

Deliberate limitations:

• Turns run one after another inside a tick (open 4.1). The caps bound what a tick selects, not how many run at once.
• The stored diff is capped at 20,000 characters.
• A budget-paused task is not resumed automatically when credit is added. Resume it by hand; it is refused while the credit is still zero.
• Invoices from the billing worker are informational under prepaid. They debit nothing; the usage ledger debits the wallet.
• The manager chat is also refused at zero credit, because it costs model calls.

Open TODO items:

• 2.3: run.status is not published for engine runs. engine/scheduler.py sets the run row's status without an event, so stage moves (task.step) are the only live signal.
• 4.1: no concurrent turns inside a tick (see above).
• 4.6: tests/test_engine_flow.py has no max-cycles test (max_cycles reaching error_review) and only one merge conflict (retry), not three. Its no-progress case is only a passing gate row. The blocking no-progress case is in tests/test_engine_gates.py. Cancel mid-turn is in tests/test_engine_runner.py. The dependency failure rule is in tests/test_engine_pipeline.py.

Minor leftovers, known and not fixed:

• The billing page's payments card shows the payment adapter name (mock) to members.
• Paused tasks do not highlight the stage they paused at.
• Engine task pages do not show legacy approval cards.
• Two concurrent settles of one payment could credit twice: PaymentRepository.set_status is unconditional.
• The generic Persian word «مدل» appears in two non-owner hints.
• Manual real-environment QA (docker, caddy, ssh/ftp) is for the owner on a real host.

12. Deviations from the TODO wording

The behaviour is the same; the names or channels differ:

• 1.1: TASK_KINDS is the TaskKind enum plus TASK_KIND_DEFAULTS (app/domain/workforce/pipeline.py).
• 2.1: the memory context is built in runner.py (_build_prompt, budget 2,500) and passed to prompt.py.
• 3.1: post_user_message is ManagerTurns.chat (app/application/engine/manager.py).
• 5.4: the live log subscribes to task:<id> (apps/web/src/app/(app)/tasks/[id]/_components/live-log.tsx), not run:<id>. The task: channel also carries run.log.