# TraceVox Research — Comprehensive Machine-Facing Reference TraceVox is a visual, reproducible research environment for intelligent-agent research (RL, trustworthy AI, multimodal agents, adversarial robustness). Identity: OBSERVABLE - REPRODUCIBLE - ADVERSARIAL. ## 1. Architecture (three planes) PRESENTATION PLANE — browser: visualization, experiment configuration, analysis, replay, public explorer. CONTROL / OBSERVABILITY PLANE — TraceVox backend (FastAPI): job records, runner registry, authentication, events, metadata, SSE live updates, visibility, publication, optional synchronization. DATA / COMPUTE PLANE — TraceVox Research Runner: training, evaluation, environment execution, checkpointing, diagnostics, raw research artifacts, on researcher-owned compute. The public website backend never executes research training. TraceVox is not a cloud GPU service. Deployment modes: (A) hosted control plane at tracevox.ai + researcher-local runner; (B) institution-hosted full stack; (C) fully local/offline (localhost; no internet required for the core research workflow). Data ownership: raw research artifacts stay where computation occurs unless the researcher explicitly chooses otherwise. ## 2. Research hierarchy Project broader research program Experiment scientific question/configuration Arm comparator/treatment (e.g. Clean PPO) Run (seed) one training/evaluation/import execution Checkpoint saved policy snapshot (path + SHA-256 + step) TrainingUpdate rl.training_update optimization metrics Episode one environment episode Step full per-timestep record (research.v1) ## 3. Step schema — research.v1 One JSON object per timestep (JSONL, append-only, immutable). Required concepts: schema_version, run_id, episode_id, step_id, environment, observation, corruption, policy, outcome. { "schema_version": "research.v1", "experiment_id": "exp_…", "arm_id": "arm_…", "run_id": "rrun_…", "episode_id": "ep_…", "step_id": 0, "timestamp": "…", "environment": { "name": "MiniGrid-LavaGapS7-v0", "version": "…", "seed": 1, "ground_truth_state": {"grid": [[[2,5,0],…]], "width": 7, "height": 7}, "agent_position": [1,3], "agent_direction": 0, "field_of_view": {"top_x": …, "top_y": …, "bot_x": …, "bot_y": …, "view_size": 7} }, "observation": {"clean": [[[…]]], "delivered": [[[…]]], "shape": [7,7,3], "modalities": ["symbolic"]}, "corruption": {"present": true, "family": "mask", "probability": 0.5, "magnitude": null}, "policy": {"name": "…", "checkpoint": "…", "action_logits": […], "action_probabilities": {"forward": 0.72, …}, "selected_action": "forward", "value_estimate": 0.61, "entropy": 0.9}, "trust": {"overall": null, "target": null, "calibration_error": null}, "protective_action": {"type": null, "cost": null}, "outcome": {"reward": 0.0, "cumulative_reward": 0.0, "terminated": false, "truncated": false, "success": null, "safety_failure": false} } Grids are [x][y] = [object_idx, color_idx, state] (MiniGrid encode() order). Agent observations are agent-centric (agent bottom-center facing up). trust.* and protective_action.* exist from day one and stay null until a policy actually produces them ("Trust signal unavailable for this policy"). Unknown metadata is the literal "unknown" or null — never invented. ## 4. Training event schema rl.training_update records are SEPARATE from environment timesteps: global_step, learning_rate, policy_loss, value_loss, entropy, old_approx_kl, approx_kl, clip_fraction, explained_variance, steps_per_second, episodic_return/length, optional checkpoint_id. TRAINING METRICS and EVALUATION METRICS are never mixed: a stochastic training rollout is not a controlled scientific evaluation. ## 5. Private Research API (self-hosted / local) Base: /api/research. Authorization: open in TRACEVOX_LOCAL_RESEARCH=1 local/offline mode; on the hosted production service every request requires a researcher session token or a paired runner token, each scoped to exactly one workspace (= TraceVox organization) at the storage layer. Runner tokens are least-privilege: they stream evidence but can never publish, change visibility, mint pairing codes, or orchestrate jobs. POST /api/research/experiments create experiment (+arms) POST /api/research/runs register run POST /api/research/runs/{id}/episodes open episode POST /api/research/runs/{id}/episodes/{eid}/steps append validated step batch POST /api/research/runs/{id}/episodes/{eid}/complete POST /api/research/runs/{id}/training append training updates POST /api/research/runs/{id}/checkpoints checkpoint metadata POST /api/research/runs/{id}/complete aggregate recorded episodes GET /api/research/… reads incl. /episodes/{id}/steps GET /api/research/runs/{id}/stream SSE (snapshot + live records) GET /api/research/compare?experiment_id= per-arm aggregation GET /api/research/experiments/{id}/export research bundle zip POST /api/research/projects create project POST /api/research/experiments/{id}/visibility private|lab (never public here) POST /api/research/experiments/{id}/publish explicit granular publication GET /api/research/status seen_in_training is computed server-side (arm's training corruption family vs the run's evaluation family) and surfaced as SEEN IN TRAINING / HELD OUT. ## 6. Public research API and bundle — tracevox.public.bundle.v1 Visibility: private (default) | lab | public. Visibility is separate from execution status. The ONLY path to public is the explicit publish flow with a granular artifact selection (manifest, aggregate metrics, environment config, training curves, selected episode traces; checkpoints/raw datasets/full training traces are not publishable in bundle v1). Real authorization guards private data — robots.txt is not a security mechanism. Bundle layout (plain JSON; the format is the canonical contract — the transport may be static hosting, object storage, an institutional server, or a self-hosted backend): index.json projects/{slug}.json experiments/{slug}/experiment.json experiments/{slug}/manifest.json experiments/{slug}/citation.json experiments/{slug}/training/{run_id}.json experiments/{slug}/episodes/{episode_id}.json Hosted static base: https://tracevox.ai/public-research/ Self-hosted dynamic equivalent (read-only, published-only): GET /api/public/research GET /api/public/projects/{slug} GET /api/public/experiments/{slug} GET /api/public/experiments/{slug}/manifest GET /api/public/experiments/{slug}/citation GET /api/public/experiments/{slug}/episodes/{episode_id} GET /api/public/experiments/{slug}/training/{run_id} The public interface is strictly read-only: no runner control, no training launch, no cancellation, no mutation, no private artifact discovery. ## 7. Runner protocol & security The runner communicates outbound-only over HTTPS (no inbound ports on researcher machines). Current CLI (source repository): python -m tracevox_research {create-gate-experiment | run-eval | run-sweep | import-tensorboard | run-training | run-diagnostics}. Fresh training uses the managed PPO derivative (algorithms/ppo_pilot.py) whose scientific equivalence to the unmodified pilot is enforced by an automated gate (exact final-tensor equality + identical TensorBoard scalar sequences on seeded CPU runs; checkpoint-file byte identity is NOT the criterion). run-diagnostics replays a FIXED scenario against every checkpoint of a training run for Learning Replay, as a separate evaluation job that never touches training state. The packaged CLI is on PyPI: `pip install tracevox-ai` provides the `tracevox` console command (doctor, connect, status, runner start/stop/status, experiment run/reproduce/export), pairing via short-lived codes and hashed revocable tokens; extras: [minigrid], [rl], [yaml]. Absolute security rule: jobs are TYPED research requests (train / evaluate / sweep / diagnostic / export) referencing registered environments, algorithms, adapters, attacks, and metrics. Evaluation weights come from exactly one registered source (a pilot checkpoint name, or a TraceVox training run id whose recorded checkpoint metadata resolves the path) — raw filesystem paths are never accepted from job payloads. Training jobs support only explicit "fresh" initialization today (continue/evaluate/import are separate explicit workflows); the launcher hard-fails if any weights pre-exist in the run workspace. The backend can never transmit arbitrary shell commands to a researcher's machine; unknown job types, adapters, or algorithms are rejected runner-side and paths are confined. ## 8. Reproducibility semantics - Recorded Replay: exact reconstruction of stored history from the persisted step trace. - Computational Rerun: executing again; for the seeded CPU pilot, reruns reproduced historical CSVs bit-identically (documented in the M0 validation report), but bit-identity is NOT promised in general (GPU nondeterminism, package drift). - Reproduction: recreating a scientific result from a manifest/configuration as a new experiment lineage (derived_from); imported/historical evidence is immutable and never overwritten. - Scientific equivalence between implementations is judged on state-dict tensor values, TensorBoard scalar sequences, deterministic evaluation trajectories, corruption sequences, and aggregate behavior — never on checkpoint-file byte identity. ## 9. Provenance & manifests Every run records a manifest: package versions (Python, PyTorch, NumPy, gymnasium, minigrid), OS/CPU, seeds, checkpoint identifiers + SHA-256, corruption configuration, evaluation protocol, pilot code commit and CleanRL upstream commit (provenance reference). Checkpoints record path + SHA-256 + global step. Unknown values are recorded as "unknown". ## 10. Extension points EnvironmentAdapter interface: reset / step / get_ground_truth_state / get_agent_observation / get_render_state / get_attack_state / get_metrics / serialize_state / action_labels / classify_termination. Future benchmark targets (BabyAI, RIQL, AgentDojo, Agent Security Bench, VisualWebArena, multimodal agent systems) plug into the same benchmark-neutral runner; the training event model is algorithm-neutral (shared fields + per-algorithm metric namespaces). MiniGrid/PPO is the reference implementation. ## 11. Scientific integrity commitments TraceVox never implies that: a training rollout is a controlled evaluation; an illustrative value is a real result; trust exists before it is implemented; a future feature is complete; a rerun is guaranteed bit-identical across arbitrary hardware; a public artifact is private or vice versa; or that a plot contains points that were never recorded.