smart-org/docs/plan/TODO-agent-3-delivery-infra.md

TODO — Agent 3: Delivery & Infra (docker sandbox, git workspaces, live CI/CD, demo server, Caddy, SSH/FTP)

• Worktree: /home/mamad/work/smart-org-wt/agent3 · Branch: agent3/delivery-infra (already created, already installed)
• Read first: docs/plan/PLAN-v2.md (all of it, especially D5–D9). Then read fully: apps/api/app/application/ports/sandbox_provider.py, git_provider.py, domain_router.py, realtime.py, app/infrastructure/adapters/registry.py, mock.py, caddy.py, gitea.py, sandbox_remote.py, app/application/services/sandbox_service.py, project_service.py, app/domain/platform/rules.py, docs/current-infra-notes.md, tests/test_project_repo.py, tests/test_domain_isolation.py.
• Reference (concepts only, NEVER import, never edit): /projects/agybot/caddy_manager.py, port_registry.py, ssh_manager.py, ftp_manager.py, git_manager.py.
• Your files: only the "Agent 3" column of PLAN-v2 §6 (including ALL of app/infrastructure/adapters/), your block in config.py / .env.example, and the migration alembic/versions/a3b4c5d6e7f3_delivery_infra.py (down_revision = "a0f0e1d2c3b4"). New tables go in app/infrastructure/db/models_delivery.py. db/models.py is locked, so extra data about existing sandbox tables goes into new side tables.
• Never touch: the locked files and the other agents' files. Need something there? → CHANGE REQUEST in your report.
• Do NOT commit, push, merge or rebase. Never cd outside the worktree, never write to /tmp: tests use pytest's tmp_path (allowed) and the worktree's ./_scratch/.
• After each phase: run all checks (PLAN-v2 §7.4), write docs/plan/reports/agent-3-phase-<K>.md, then STOP and wait.

You implement: RepoWorkspacePort, DeliveryHooks, DeployTargetTester, plus a docker SandboxProviderPort adapter. Wire them in container_delivery.py and registry.py behind settings (mock remains the default for tests).

Security rules specific to you

• Nothing runs on the host except: git (via the workspace adapter, hardened per D6), the docker/podman CLI, and Caddy reload (existing adapter).
• Every subprocess: asyncio.create_subprocess_exec (never shell=True), start_new_session=True, kill the process group on timeout, drain stdout and stderr concurrently (avoid the 64 KB pipe deadlock), and pass explicit env (never inherit secrets).
• Credentials never appear in argv, URLs, logs, exceptions, realtime events or DB columns other than *_encrypted. Use GIT_ASKPASS with a 0700 helper script that reads a 0600 temp file in the worktree's ./_scratch-like private dir under SANDBOX_WORKSPACES_ROOT, deleted in finally.
• Ports and hosts must pass app/domain/platform/rules.py (protected ports, single-label hosts, localhost targets).

---

Phase 1 — Docker sandbox adapter

• 1.1 adapters/docker_sandbox.py, DockerSandboxAdapter implementing SandboxProviderPort with the CLI in SANDBOX_DOCKER_BIN (docker or podman, the same flags):
  • create_workspace: creates a host dir SANDBOX_WORKSPACES_ROOT/<org_id>/<project_id>/work (0700, owned by the platform user).
  • start_session: docker run -d the workspace container:
    • image from the app profile language: node:20-bookworm, python:3.13-bookworm, php:8.3-cli, golang:1.23, fallback debian:bookworm; make the mapping a dict in your config block / domain
    • --user 1000:1000 --cap-drop ALL --security-opt no-new-privileges --pids-limit 512 --cpus $SANDBOX_DEFAULT_CPUS --memory ${MB}m
    • --read-only with --tmpfs /tmp and --tmpfs /home/app
    • bind-mount only the work dir (never the git dir) to /workspace
    • label smart-org.org=<id>, smart-org.project=<id>, smart-org.role=workspace
    • network: the default bridge (package installs need egress), with no published ports
  • run_command: docker exec -w <cwd> <ctr> sh -lc <cmd>, with timeout, output cap, exit code, and an optional async on_output(chunk) callback for streaming (add it as an adapter-level extra method; the port stays unchanged).
  • read_file / write_file: through the host bind mount with path safety (resolve() stays inside the work dir; reject symlinks that escape).
  • pause / resume / destroy: stop/start/rm. snapshot: docker commit (tagged per org) or raise a clear "not supported" error, your choice, documented.
  • get_preview_url: delegated to the domain layer (Phase 4).
• 1.2 registry.py: SANDBOX_PROVIDER=docker → this adapter. The mock stays the default.
• 1.3 The sandbox_gateway worker: also reap containers with the smart-org.* label that have no live session row (orphans), audited.
• 1.4 Hide infra from non-owners:
  • sandbox.py command/file endpoints become owner-only (or admin+ of the org with a setting off by default; prefer owner-only).
  • Keep the read-only status endpoints for org members.
  • Update components/sandbox-panel.tsx to show only status to non-owners.
  • Update existing tests that assumed member access, and explain each change in your report.
• 1.5 Tests tests/test_docker_sandbox.py: unit tests with a fake process runner (inject the subprocess function) that assert the exact argv (hardening flags present, no secrets, mount of the work dir only), timeout kill, output cap, path escape refused. Integration tests marked @pytest.mark.docker, skipped unless SANDBOX_DOCKER_IT=1: real start → exec echo → write/read file → destroy. — open: no @pytest.mark.docker sandbox integration test (real start, exec, write/read, destroy) exists in tests/test_docker_sandbox.py; the only docker-marked test is test_delivery.py::test_docker_end_to_end_static_app.

---

Phase 2 — Git workspaces (RepoWorkspacePort, the platform does git, D6)

• 2.1 adapters/git_workspace.py, LocalGitWorkspaceAdapter. Layout per project under SANDBOX_WORKSPACES_ROOT/<org>/<project>/:
  • git/: a bare-ish gitdir, never mounted
  • work/: the main working tree for the dev branch, mounted at /workspace
  • work/.worktrees/<task_id>/: task worktrees, visible in the container at /workspace/.worktrees/<task_id>
  • WorktreeRef.sandbox_path = that in-container path Note: the .worktrees dir must be excluded via info/exclude in the gitdir, not the user's .gitignore.
• 2.2 Every git call is git --git-dir=<gitdir> --work-tree=<path> -c core.hooksPath=/dev/null -c core.fsmonitor=false -c core.quotePath=false -c protocol.file.allow=never -c safe.directory=*, with env GIT_CONFIG_NOSYSTEM=1, HOME=<private dir>, GIT_TERMINAL_PROMPT=0. Never trust the .git file inside a worktree; always pass explicit dirs.
  • Status: status --porcelain=v1 -uall -z.
  • One asyncio.Lock per project id held for every mutating operation (agybot M14). Remove a stale index.lock only if older than 30s.
• 2.3 Remotes: — open: the ssh_key first contact uses StrictHostKeyChecking=accept-new and the key is never stored. test_remote returns no host key (RemoteCheck has no field for it) and nothing writes project_external_git.known_hosts.
  • platform git: the URL and credentials come from the existing Gitea adapter / project_git_credentials (decrypt via the existing crypto helper).
  • external git: project_external_git (token → GIT_ASKPASS; ssh_key → temp key file 0600 + GIT_SSH_COMMAND="ssh -i <key> -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=<per-project known_hosts>"; store known_hosts on the first test_remote = TOFU, in the existing column).
  • test_remote: git ls-remote --heads with a 20s timeout; returns ok, default branch, branch list, and a friendly error (auth failed / not found / unreachable) without echoing the URL credentials or stderr secrets.
  • SSRF check on external hosts (resolve DNS, reject private, loopback and link-local unless DEPLOY_ALLOW_PRIVATE_HOSTS); put the check in domain/platform/rules.py (yours). It runs inside test_remote and before every deploy connection; Agent 1 only does format validation and relies on your check.
• 2.4 Implement every RepoWorkspacePort method per PLAN-v2 §5.
  • ensure_workspace clones on first use, fetches afterwards. It creates dev from the default branch if missing; for platform repos also push dev.
  • merge_into does a --no-ff merge in the main work tree under the lock and returns the conflict file list on failure, after merge --abort. Never leave a half-merge (agybot H3).
  • push returns ok/rejected (non-fast-forward → fetch + report, never force).
• 2.5 Wire it in container_delivery.py (setting GIT_WORKSPACE=local|fake, fake default in tests).
• 2.6 Tests tests/test_git_workspace.py with real git and a local bare repo in tmp_path as "remote" (no network):
  • the full flow ensure → worktree → write → diff (new dir appears thanks to -uall) → commit → sync → merge → push → head
  • a conflict path leaves a clean tree
  • hooks are NOT executed (plant a post-commit hook that writes a file and assert it is absent)
  • a .git file tampered to point elsewhere is ignored
  • credentials are never in argv (capture the runner calls)
  • concurrent merges are serialised

---

Phase 3 — Live CI/CD + demo server

• 3.1 Tables (models_delivery.py):
  • pipeline_runs (id, org, project, trigger engine|poll|manual, branch, sha, status queued|running|succeeded|failed|cancelled|superseded, started/finished, error_summary)
  • pipeline_steps (id, run_id, index, name, status, started/finished, exit_code, log_tail ≤ 64 KB)
  • releases (id, org, project, run_id, sha, artifact_path, created_at, is_demo_live)
  • app_runtimes (project_id PK, demo_container_id, host_port, demo_host, status, current_release_id, updated_at)
• 3.2 delivery_service.py implementing DeliveryHooks:
  • on_app_created: allocate a demo port in DEMO_PORT_START..END (reuse the existing preview port rules), claim the demo host <org-slug>.<PREVIEW_ROOT_DOMAIN> through the existing DomainRouterPort, create app_runtimes, ensure_workspace, and queue a first pipeline run if the repo has a dev head.
  • on_branch_pushed: queue a run (one active run per project; a newer queued run supersedes an older queued one).
• 3.3 workers/delivery.py: loop (--loop) that executes queued runs and polls remotes every DELIVERY_POLL_SECONDS (head vs the last run sha) for pushes made outside the engine (users pushing to their own git).
  • Steps, each in a fresh build container (same image mapping and hardening, work dir = a clean checkout of the sha in a per-run dir):
    1. checkout
    2. setup (setup_command)
    3. build (build_command, skipped if empty)
    4. test (test_command, skipped if empty)
    5. package (tar.gz of the run dir excluding .git and node_modules → ARTIFACTS_ROOT/<org>/<project>/<sha>.tar.gz)
    6. deploy_demo: start the new demo container with start_command, port mapping 127.0.0.1:<host_port>:<app_port>, the same hardening, label role=demo
    7. healthcheck: HTTP GET http://127.0.0.1:<host_port><healthcheck_path> with 20 retries × 3s; any 2xx/3xx = ok
    8. route: configure_reverse_proxy for the demo host → 127.0.0.1:<host_port>
  • Blue/green: start the new container on a temporary second port, healthcheck it, then switch the Caddy route, then stop the old one. If any step fails, the old demo keeps running untouched and the run is failed with a Persian error_summary.
• 3.4 Live logs: every step streams output via the on_output callback → batched pipeline.log events on org:<id> (≤ 1 event / 500 ms, ≤ 200 lines), plus pipeline.step and pipeline.status; release.live when switched. Store the step log_tail (last 64 KB). Pass all output through app/core/redact.py before publishing or storing.
• 3.5 Router delivery.py (org member read, admin+ actions): list runs, run detail with steps and tails, POST /delivery/runs (manual re-run of the dev head, admin+), cancel the queued/running run, list releases, rollback the demo to a previous release (admin+, confirm, audit), and the runtime status (demo URL, live release, health).
• 3.6 Tests tests/test_delivery.py with the fake runner + FakeRepoWorkspace + mock domain router:
  • the happy path step sequence and events
  • a failed build keeps the old demo
  • healthcheck failure → no route switch
  • supersede logic
  • poll detects an external push
  • logs redacted (plant a secret in the build output)
  • tenant isolation on every endpoint
  • an optional @pytest.mark.docker end-to-end with a tiny static app (python -m http.server)

---

Phase 4 — Domains (Caddy) owned by the platform owner

• 4.1 admin_domains.py (owner only):
  • list all routes (host → org/app → target port → status)
  • attach an extra host to an org's app: a single label under the root via the existing rules, OR a full custom FQDN when ALLOW_CUSTOM_DOMAINS=true (strict FQDN regex, no wildcard, no protected hosts such as git, mcp, panel, api, root)
  • detach (confirm + audit)
  • re-sync all routes from the DB to Caddy (idempotent) Users cannot add domains (the existing member-facing preview-domain endpoints in projects.py become owner-only, and the tests are updated with reasons).
• 4.2 The Caddy adapter keeps validate → revert → reload. Add a routes_fingerprint check so re-sync does nothing when nothing changed.
• 4.3 The demo host is created automatically in on_app_created (Phase 3). Record the MCP gateway host suggestion (mcp.<root> → MCP_GATEWAY_PORT) as a protected system route that the owner can enable from this page. — open: the MCP route is only a read-only suggestion (GET /api/admin/domains/suggested, shown as enabled or disabled on the page). There is no enable action, and validate_extra_host refuses the protected label mcp, so the owner cannot enable it from this page.
• 4.4 Tests tests/test_admin_domains.py: non-owner forbidden; protected hosts refused; invalid FQDNs refused; detach audited; re-sync idempotent (mock router call count).

---

Phase 5 — Production deploy over SSH / SFTP / FTP / FTPS (always human-triggered)

• 5.1 adapters/deploy_ssh.py (asyncssh), DeployTargetTester + SshDeployer:
  • Test: connect (15s timeout) → host-key TOFU (store the fingerprint on the first success; afterwards a mismatch = hard fail "کلید میزبان تغییر کرده"), check remote_path exists and is writable.
  • Deploy: upload the release artifact → remote_path/releases/<ts>-<sha7>/ → extract → run post_deploy_command with cwd there (timeout 600s, streamed output) → atomically switch the remote_path/current symlink → keep the last 5 releases.
  • Rollback = re-point current to the previous release + rerun post_deploy.
• 5.2 adapters/deploy_ftp.py (aioftp; FTPS via explicit TLS), DeployTargetTester + FtpDeployer:
  • Test: login, cwd remote_path, write and delete a probe file.
  • Deploy: upload the build output (the build dir is set by the framework catalog default or dist/, build/, public/, else the whole artifact) into remote_path after downloading a backup listing of the files it overwrites into ARTIFACTS_ROOT/.../ftp-backup-<ts>.tar.gz.
  • Rollback = re-upload that backup. Document the limits of FTP (no atomic switch).
• 5.3 SSRF check for every target host (the Phase 2.3 helper). A combined DeployTargetTester dispatching on kind, wired in container_delivery.py.
• 5.4 deploy_service.py + endpoints in delivery.py:
  • POST /delivery/production/deploy with release_id (default: the live demo release). Org owner/admin only; requires a confirm: "<org-slug>" field; one production deploy per project at a time.
  • Status: deploy.status events and a production_deploys table (id, release, target, status, started/finished, log_tail, rolled_back_from).
  • POST /delivery/production/rollback.
  • Every action audited. Credentials decrypted only inside the adapter call.
• 5.5 Tests tests/test_deploy.py: fake SSH/FTP clients injected. Cover: the release/symlink sequence, keep-5 pruning, rollback, host-key mismatch refused, SSRF refused, a member (non-admin) refused, the confirm slug mismatch refused, no password in logs/events/DB columns other than encrypted ones. Optional integration (DEPLOY_IT=1) against a local sshd/pure-ftpd container.

---

Phase 6 — UI: the "Application" page (replaces /projects)

• 6.1 (app)/app/page.tsx (+ components under (app)/app/_components/):
  • Header: app name, type/language/framework badges, demo URL (open button), live release sha + time, health dot.
  • Pipeline runs: list with status; clicking one opens the run view: a step list with status and duration, and a live log console (subscribe org:<id>, filter by run id, ANSI colours stripped, auto-scroll with pause, and a "copy" button).
  • Releases: history; "rollback demo" (admin+, confirm).
  • Production: target summary (kind, host, path, last test result; no secrets), the "استقرار در production" button (admin+, modal requiring the org slug), a deploy live log, production history + rollback.
  • Git: repo link (platform: Gitea URL; external: the URL without credentials), dev head sha, last poll time. No branch/PR/port controls for users.
• 6.2 Remove (app)/projects (redirect /projects* → /app) and drop the member-facing port/branch/PR/command controls. Owner-only debug tools move to /admin/sandbox.
• 6.3 /admin/sandbox (owner): running containers per org (workspace/demo/build), CPU/memory limits per org (existing sandbox_resource_limits), stop/restart a container (confirm + audit), and an image mapping view. /admin/domains (owner): the Phase 4 UI. — open: no restart action. The backend and the page offer list and stop only (POST /api/admin/sandbox/containers/{id}/stop).
• 6.4 lib/api-delivery.ts: typed client for all of the above. — open: /admin/sandbox calls apiRequest directly, and lib/api-delivery.ts has deliveryApi and domainsApi only, with no sandbox client.
• 6.5 Web tests src/test/delivery-*.test.tsx: the log console batching and auto-scroll, the confirm-slug modal, no secret fields rendered, non-admin sees no deploy button. — open: no test for log batching or auto-scroll. The non-admin deploy-button case is covered by hides the deploy actions from an org member who is not an admin (apps/web/src/test/delivery-app.test.tsx, added in T14).

---

Final report (after Phase 6)

docs/plan/reports/agent-3-final.md: summary, host requirements (docker/podman, Caddy, dirs and permissions, the sudo/systemd needs of the Caddy reload), every setting, every endpoint added or changed (method + path + role), tables, CHANGE REQUESTs still open, manual QA script (wizard app → push → watch live pipeline → open demo URL → deploy to a local SSH test server → rollback), known limitations.