smart-org/docs/plan/TODO-agent-3-delivery-infra.md

# TODO — Agent 3: Delivery & Infra (docker sandbox, git workspaces, live CI/CD, demo server, Caddy, SSH/FTP)

- **Worktree:** `/home/mamad/work/smart-org-wt/agent3` · **Branch:** `agent3/delivery-infra` (already created, already installed)
- **Read first:** `docs/plan/PLAN-v2.md` (all of it, especially D5–D9). Then read fully:
  `apps/api/app/application/ports/sandbox_provider.py`, `git_provider.py`, `domain_router.py`, `realtime.py`,
  `app/infrastructure/adapters/registry.py`, `mock.py`, `caddy.py`, `gitea.py`, `sandbox_remote.py`,
  `app/application/services/sandbox_service.py`, `project_service.py`, `app/domain/platform/rules.py`,
  `docs/current-infra-notes.md`, `tests/test_project_repo.py`, `tests/test_domain_isolation.py`.
- **Reference (concepts only, NEVER import, never edit):** `/projects/agybot/caddy_manager.py`, `port_registry.py`,
  `ssh_manager.py`, `ftp_manager.py`, `git_manager.py`.
- **Your files:** only the "Agent 3" column of PLAN-v2 §6 (including ALL of `app/infrastructure/adapters/`), your
  block in `config.py` / `.env.example`, and the migration `alembic/versions/a3b4c5d6e7f3_delivery_infra.py`
  (`down_revision = "a0f0e1d2c3b4"`). New tables go in `app/infrastructure/db/models_delivery.py`. **`db/models.py`
  is locked**, so extra data about existing sandbox tables goes into new side tables.
- **Never touch:** the locked files and the other agents' files. Need something there? → `CHANGE REQUEST` in your report.
- **Do NOT commit, push, merge or rebase.** Never `cd` outside the worktree, never write to `/tmp`: tests use pytest's
  `tmp_path` (allowed) and the worktree's `./_scratch/`.
- **After each phase:** run all checks (PLAN-v2 §7.4), write `docs/plan/reports/agent-3-phase-<K>.md`, then STOP and wait.

You implement: `RepoWorkspacePort`, `DeliveryHooks`, `DeployTargetTester`, plus a docker `SandboxProviderPort`
adapter. Wire them in `container_delivery.py` and `registry.py` behind settings (mock remains the default
for tests).

**Security rules specific to you**
- Nothing runs on the host except: `git` (via the workspace adapter, hardened per D6), the docker/podman CLI, and Caddy reload (existing adapter).
- Every subprocess: `asyncio.create_subprocess_exec` (never `shell=True`), `start_new_session=True`, kill the process group on timeout, drain stdout and stderr concurrently (avoid the 64 KB pipe deadlock), and pass explicit `env` (never inherit secrets).
- Credentials never appear in argv, URLs, logs, exceptions, realtime events or DB columns other than `*_encrypted`. Use `GIT_ASKPASS` with a 0700 helper script that reads a 0600 temp file in the worktree's `./_scratch`-like private dir under `SANDBOX_WORKSPACES_ROOT`, deleted in `finally`.
- Ports and hosts must pass `app/domain/platform/rules.py` (protected ports, single-label hosts, localhost targets).

---

## Phase 1 — Docker sandbox adapter

- [x] 1.1 `adapters/docker_sandbox.py`, `DockerSandboxAdapter` implementing `SandboxProviderPort` with the CLI in `SANDBOX_DOCKER_BIN` (docker or podman, the same flags):
  - `create_workspace`: creates a host dir `SANDBOX_WORKSPACES_ROOT/<org_id>/<project_id>/work` (0700, owned by the platform user).
  - `start_session`: `docker run -d` the **workspace container**:
    - image from the app profile language: `node:20-bookworm`, `python:3.13-bookworm`, `php:8.3-cli`, `golang:1.23`, fallback `debian:bookworm`; make the mapping a dict in your config block / domain
    - `--user 1000:1000 --cap-drop ALL --security-opt no-new-privileges --pids-limit 512 --cpus $SANDBOX_DEFAULT_CPUS --memory ${MB}m`
    - `--read-only` with `--tmpfs /tmp` and `--tmpfs /home/app`
    - bind-mount **only the work dir** (never the git dir) to `/workspace`
    - label `smart-org.org=<id>`, `smart-org.project=<id>`, `smart-org.role=workspace`
    - network: the default bridge (package installs need egress), with no published ports
  - `run_command`: `docker exec -w <cwd> <ctr> sh -lc <cmd>`, with timeout, output cap, exit code, and an optional async `on_output(chunk)` callback for streaming (add it as an adapter-level extra method; the port stays unchanged).
  - `read_file` / `write_file`: through the host bind mount with path safety (`resolve()` stays inside the work dir; reject symlinks that escape).
  - `pause` / `resume` / `destroy`: stop/start/rm. `snapshot`: `docker commit` (tagged per org) or raise a clear "not supported" error, your choice, documented.
  - `get_preview_url`: delegated to the domain layer (Phase 4).
- [x] 1.2 `registry.py`: `SANDBOX_PROVIDER=docker` → this adapter. The mock stays the default.
- [x] 1.3 The `sandbox_gateway` worker: also reap containers with the `smart-org.*` label that have no live session row (orphans), audited.
- [x] 1.4 Hide infra from non-owners:
  - `sandbox.py` command/file endpoints become **owner-only** (or admin+ of the org with a setting off by default; prefer owner-only).
  - Keep the read-only status endpoints for org members.
  - Update `components/sandbox-panel.tsx` to show only status to non-owners.
  - Update existing tests that assumed member access, and explain each change in your report.
- [ ] 1.5 Tests `tests/test_docker_sandbox.py`: unit tests with a **fake process runner** (inject the subprocess function) that assert the exact argv (hardening flags present, no secrets, mount of the work dir only), timeout kill, output cap, path escape refused. Integration tests marked `@pytest.mark.docker`, skipped unless `SANDBOX_DOCKER_IT=1`: real start → exec `echo` → write/read file → destroy. — open: no `@pytest.mark.docker` sandbox integration test (real start, exec, write/read, destroy) exists in tests/test_docker_sandbox.py; the only docker-marked test is test_delivery.py::test_docker_end_to_end_static_app.

---

## Phase 2 — Git workspaces (`RepoWorkspacePort`, the platform does git, D6)

- [x] 2.1 `adapters/git_workspace.py`, `LocalGitWorkspaceAdapter`. Layout per project under `SANDBOX_WORKSPACES_ROOT/<org>/<project>/`:
  - `git/`: a bare-ish gitdir, **never mounted**
  - `work/`: the main working tree for the dev branch, mounted at `/workspace`
  - `work/.worktrees/<task_id>/`: task worktrees, visible in the container at `/workspace/.worktrees/<task_id>`
  - `WorktreeRef.sandbox_path` = that in-container path
  Note: the `.worktrees` dir must be excluded via `info/exclude` in the gitdir, not the user's `.gitignore`.
- [x] 2.2 Every git call is `git --git-dir=<gitdir> --work-tree=<path> -c core.hooksPath=/dev/null -c core.fsmonitor=false -c core.quotePath=false -c protocol.file.allow=never -c safe.directory=*`, with env `GIT_CONFIG_NOSYSTEM=1`, `HOME=<private dir>`, `GIT_TERMINAL_PROMPT=0`. Never trust the `.git` *file* inside a worktree; always pass explicit dirs.
  - Status: `status --porcelain=v1 -uall -z`.
  - One `asyncio.Lock` per project id held for every mutating operation (agybot M14). Remove a stale `index.lock` only if older than 30s.
- [ ] 2.3 Remotes: — open: the ssh_key first contact uses StrictHostKeyChecking=accept-new and the key is never stored. test_remote returns no host key (RemoteCheck has no field for it) and nothing writes project_external_git.known_hosts.
  - **platform** git: the URL and credentials come from the existing Gitea adapter / `project_git_credentials` (decrypt via the existing crypto helper).
  - **external** git: `project_external_git` (token → `GIT_ASKPASS`; ssh_key → temp key file 0600 + `GIT_SSH_COMMAND="ssh -i <key> -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=<per-project known_hosts>"`; store `known_hosts` on the first `test_remote` = TOFU, in the existing column).
  - `test_remote`: `git ls-remote --heads` with a 20s timeout; returns ok, default branch, branch list, and a friendly error (auth failed / not found / unreachable) **without echoing the URL credentials or stderr secrets**.
  - SSRF check on external hosts (resolve DNS, reject private, loopback and link-local unless `DEPLOY_ALLOW_PRIVATE_HOSTS`); put the check in `domain/platform/rules.py` (yours). It runs inside `test_remote` and before every deploy connection; Agent 1 only does format validation and relies on your check.
- [x] 2.4 Implement every `RepoWorkspacePort` method per PLAN-v2 §5.
  - `ensure_workspace` clones on first use, fetches afterwards. It creates `dev` from the default branch if missing; for platform repos also push `dev`.
  - `merge_into` does a `--no-ff` merge in the main work tree under the lock and returns the conflict file list on failure, after `merge --abort`. Never leave a half-merge (agybot H3).
  - `push` returns ok/rejected (non-fast-forward → fetch + report, never force).
- [x] 2.5 Wire it in `container_delivery.py` (setting `GIT_WORKSPACE=local|fake`, fake default in tests).
- [x] 2.6 Tests `tests/test_git_workspace.py` with **real git** and a local bare repo in `tmp_path` as "remote" (no network):
  - the full flow ensure → worktree → write → diff (new dir appears thanks to `-uall`) → commit → sync → merge → push → head
  - a conflict path leaves a clean tree
  - hooks are NOT executed (plant a `post-commit` hook that writes a file and assert it is absent)
  - a `.git` file tampered to point elsewhere is ignored
  - credentials are never in argv (capture the runner calls)
  - concurrent merges are serialised

---

## Phase 3 — Live CI/CD + demo server

- [x] 3.1 Tables (`models_delivery.py`):
  - `pipeline_runs` (id, org, project, trigger `engine|poll|manual`, branch, sha, status `queued|running|succeeded|failed|cancelled|superseded`, started/finished, error_summary)
  - `pipeline_steps` (id, run_id, index, name, status, started/finished, exit_code, log_tail ≤ 64 KB)
  - `releases` (id, org, project, run_id, sha, artifact_path, created_at, is_demo_live)
  - `app_runtimes` (project_id PK, demo_container_id, host_port, demo_host, status, current_release_id, updated_at)
- [x] 3.2 `delivery_service.py` implementing `DeliveryHooks`:
  - `on_app_created`: allocate a demo port in `DEMO_PORT_START..END` (reuse the existing preview port rules), claim the demo host `<org-slug>.<PREVIEW_ROOT_DOMAIN>` through the existing `DomainRouterPort`, create `app_runtimes`, `ensure_workspace`, and queue a first pipeline run if the repo has a `dev` head.
  - `on_branch_pushed`: queue a run (one active run per project; a newer queued run supersedes an older queued one).
- [x] 3.3 `workers/delivery.py`: loop (`--loop`) that executes queued runs and polls remotes every `DELIVERY_POLL_SECONDS` (`head` vs the last run sha) for pushes made outside the engine (users pushing to their own git).
  - Steps, each in a **fresh build container** (same image mapping and hardening, work dir = a clean checkout of the sha in a per-run dir):
    1. `checkout`
    2. `setup` (setup_command)
    3. `build` (build_command, skipped if empty)
    4. `test` (test_command, skipped if empty)
    5. `package` (tar.gz of the run dir excluding `.git` and `node_modules` → `ARTIFACTS_ROOT/<org>/<project>/<sha>.tar.gz`)
    6. `deploy_demo`: start the new demo container with `start_command`, port mapping `127.0.0.1:<host_port>:<app_port>`, the same hardening, label `role=demo`
    7. `healthcheck`: HTTP GET `http://127.0.0.1:<host_port><healthcheck_path>` with 20 retries × 3s; any 2xx/3xx = ok
    8. `route`: `configure_reverse_proxy` for the demo host → `127.0.0.1:<host_port>`
  - **Blue/green:** start the new container on a temporary second port, healthcheck it, then switch the Caddy route, then stop the old one. If any step fails, the old demo keeps running untouched and the run is `failed` with a Persian `error_summary`.
- [x] 3.4 Live logs: every step streams output via the `on_output` callback → batched `pipeline.log` events on `org:<id>` (≤ 1 event / 500 ms, ≤ 200 lines), plus `pipeline.step` and `pipeline.status`; `release.live` when switched. Store the step `log_tail` (last 64 KB). Pass all output through `app/core/redact.py` before publishing or storing.
- [x] 3.5 Router `delivery.py` (org member read, admin+ actions): list runs, run detail with steps and tails, `POST /delivery/runs` (manual re-run of the dev head, admin+), cancel the queued/running run, list releases, rollback the demo to a previous release (admin+, confirm, audit), and the runtime status (demo URL, live release, health).
- [x] 3.6 Tests `tests/test_delivery.py` with the fake runner + `FakeRepoWorkspace` + mock domain router:
  - the happy path step sequence and events
  - a failed build keeps the old demo
  - healthcheck failure → no route switch
  - supersede logic
  - poll detects an external push
  - logs redacted (plant a secret in the build output)
  - tenant isolation on every endpoint
  - an optional `@pytest.mark.docker` end-to-end with a tiny static app (`python -m http.server`)

---

## Phase 4 — Domains (Caddy) owned by the platform owner

- [x] 4.1 `admin_domains.py` (owner only):
  - list all routes (host → org/app → target port → status)
  - attach an extra host to an org's app: a single label under the root via the existing rules, OR a full custom FQDN when `ALLOW_CUSTOM_DOMAINS=true` (strict FQDN regex, no wildcard, no protected hosts such as `git`, `mcp`, `panel`, `api`, root)
  - detach (confirm + audit)
  - re-sync all routes from the DB to Caddy (idempotent)
  Users cannot add domains (the existing member-facing preview-domain endpoints in `projects.py` become owner-only, and the tests are updated with reasons).
- [x] 4.2 The Caddy adapter keeps validate → revert → reload. Add a `routes_fingerprint` check so re-sync does nothing when nothing changed.
- [ ] 4.3 The demo host is created automatically in `on_app_created` (Phase 3). Record the MCP gateway host suggestion (`mcp.<root>` → `MCP_GATEWAY_PORT`) as a protected system route that the owner can enable from this page. — open: the MCP route is only a read-only suggestion (GET /api/admin/domains/suggested, shown as enabled or disabled on the page). There is no enable action, and validate_extra_host refuses the protected label mcp, so the owner cannot enable it from this page.
- [x] 4.4 Tests `tests/test_admin_domains.py`: non-owner forbidden; protected hosts refused; invalid FQDNs refused; detach audited; re-sync idempotent (mock router call count).

---

## Phase 5 — Production deploy over SSH / SFTP / FTP / FTPS (always human-triggered)

- [x] 5.1 `adapters/deploy_ssh.py` (asyncssh), `DeployTargetTester` + `SshDeployer`:
  - Test: connect (15s timeout) → host-key TOFU (store the fingerprint on the first success; afterwards a mismatch = hard fail "کلید میزبان تغییر کرده"), check `remote_path` exists and is writable.
  - Deploy: upload the release artifact → `remote_path/releases/<ts>-<sha7>/` → extract → run `post_deploy_command` with `cwd` there (timeout 600s, streamed output) → atomically switch the `remote_path/current` symlink → keep the last 5 releases.
  - Rollback = re-point `current` to the previous release + rerun post_deploy.
- [x] 5.2 `adapters/deploy_ftp.py` (aioftp; FTPS via explicit TLS), `DeployTargetTester` + `FtpDeployer`:
  - Test: login, cwd `remote_path`, write and delete a probe file.
  - Deploy: upload the build output (the build dir is set by the framework catalog default or `dist/`, `build/`, `public/`, else the whole artifact) into `remote_path` after downloading a **backup listing** of the files it overwrites into `ARTIFACTS_ROOT/.../ftp-backup-<ts>.tar.gz`.
  - Rollback = re-upload that backup. Document the limits of FTP (no atomic switch).
- [x] 5.3 SSRF check for every target host (the Phase 2.3 helper). A combined `DeployTargetTester` dispatching on `kind`, wired in `container_delivery.py`.
- [x] 5.4 `deploy_service.py` + endpoints in `delivery.py`:
  - `POST /delivery/production/deploy` with `release_id` (default: the live demo release). Org **owner/admin only**; requires a `confirm: "<org-slug>"` field; one production deploy per project at a time.
  - Status: `deploy.status` events and a `production_deploys` table (id, release, target, status, started/finished, log_tail, rolled_back_from).
  - `POST /delivery/production/rollback`.
  - Every action audited. Credentials decrypted only inside the adapter call.
- [x] 5.5 Tests `tests/test_deploy.py`: fake SSH/FTP clients injected. Cover: the release/symlink sequence, keep-5 pruning, rollback, host-key mismatch refused, SSRF refused, a member (non-admin) refused, the confirm slug mismatch refused, no password in logs/events/DB columns other than encrypted ones. Optional integration (`DEPLOY_IT=1`) against a local `sshd`/`pure-ftpd` container.

---

## Phase 6 — UI: the "Application" page (replaces `/projects`)

- [x] 6.1 `(app)/app/page.tsx` (+ components under `(app)/app/_components/`):
  - **Header:** app name, type/language/framework badges, demo URL (open button), live release sha + time, health dot.
  - **Pipeline runs:** list with status; clicking one opens the run view: a step list with status and duration, and a **live log console** (subscribe `org:<id>`, filter by run id, ANSI colours stripped, auto-scroll with pause, and a "copy" button).
  - **Releases:** history; "rollback demo" (admin+, confirm).
  - **Production:** target summary (kind, host, path, last test result; no secrets), the "استقرار در production" button (admin+, modal requiring the org slug), a deploy live log, production history + rollback.
  - **Git:** repo link (platform: Gitea URL; external: the URL without credentials), `dev` head sha, last poll time. No branch/PR/port controls for users.
- [x] 6.2 Remove `(app)/projects` (redirect `/projects*` → `/app`) and drop the member-facing port/branch/PR/command controls. Owner-only debug tools move to `/admin/sandbox`.
- [ ] 6.3 `/admin/sandbox` (owner): running containers per org (workspace/demo/build), CPU/memory limits per org (existing `sandbox_resource_limits`), stop/restart a container (confirm + audit), and an image mapping view. `/admin/domains` (owner): the Phase 4 UI. — open: no restart action. The backend and the page offer list and stop only (POST /api/admin/sandbox/containers/{id}/stop).
- [ ] 6.4 `lib/api-delivery.ts`: typed client for all of the above. — open: /admin/sandbox calls apiRequest directly, and lib/api-delivery.ts has deliveryApi and domainsApi only, with no sandbox client.
- [ ] 6.5 Web tests `src/test/delivery-*.test.tsx`: the log console batching and auto-scroll, the confirm-slug modal, no secret fields rendered, non-admin sees no deploy button. — open: no test for log batching or auto-scroll. The non-admin deploy-button case is covered by `hides the deploy actions from an org member who is not an admin` (`apps/web/src/test/delivery-app.test.tsx`, added in T14).

---

## Final report (after Phase 6)
`docs/plan/reports/agent-3-final.md`: summary, host requirements (docker/podman, Caddy, dirs and permissions,
the sudo/systemd needs of the Caddy reload), every setting, every endpoint added or changed (method + path + role),
tables, CHANGE REQUESTs still open, manual QA script (wizard app → push → watch live pipeline → open demo URL →
deploy to a local SSH test server → rollback), known limitations.