The RL Data Atlas lists every RL task source that Marin — the open training project whose job scheduler, dataset processing and app host this Atlas runs on — has declared, with its review state and evidence. A source is one pinned population of tasks, for example one Hugging Face dataset revision. A task is one RL problem stored as a TaskSpec: the prompt, the machine environment the agent works in, the answer contract and the grader (also called the verifier).
This page shows each path a dataset can take into the Atlas and what each outcome lets you rely on. Being listed is not the same as being approved for training: see the eligibility matrix, and the reader workflow for using an existing source. The counts come from the catalog snapshot that the Atlas currently serves, so they change when an operator loads a new snapshot. The normative rules live in the task curation reference; each step below links to its section and to the code that enforces it.
The paths
Solid edges are the path a candidate takes; dashed edges leave it for an outcome or an alternate route. Agentic tasks need a machine (a shell, files or tools); reply-only and conversation tasks skip steps 2 and 3. Select an outcome to list its sources.
the candidate's required pathexit into an outcome, or an alternate route such as the non-agentic bypass
What each outcome guarantees
The By source view labels every source with a Quality level — QUICK, SAMPLE or FULL — saying how much executed QA examined it, where independent QC by an upstream release also counts as QUICK because this repository did not execute it. Each source also carries a Result — GOOD, SOME_ISSUES or BAD — saying what that QA found. The rules live in Quality and result. Each row links to an evidence page showing the recorded pointers behind those labels; evidence that was never recorded, or that lives in a private release you cannot open, shows as missing or restricted.
Being admitted is evidence, not eligibility. A training run can select only sources that are runnable, not deprecated or excluded, whose manifests record a mode: full artifact, and that a selection recipe names. Oversight describes the execution evidence behind admission; SAMPLE oversight and a FULL-mode artifact are different axes, and selection requires the artifact. The Daytona label applies to admitted and runnable sources; it describes where a source can run, not whether it is admitted.
Eligibility at a glance: what lists a source in the Atlas, and what additionally selects it for training.
Source
Listed in the Atlas
Selectable for training additionally requires
Ordinary source
A declaration or a published snapshot that contains it.
Runnable and not deprecated or excluded; a mode: full artifact; a selection recipe naming it.
Grandfathered blend
Membership in the frozen GRANDFATHERED_MARINSKYRL_SOURCES list.
The same FULL-artifact and recipe rules. Only the format gate is waived; measured environments and Daytona counts are still not established.
Oversight not recorded (legacy)
A snapshot loaded before the oversight field existed, or a source without declared oversight.
The same FULL-artifact and recipe rules. Without oversight evidence this page cannot establish what covered the retained tasks; inspect the linked campaign record.
Outcome
You can rely on
You cannot rely on
Sources
Rejected
Nothing from it reaches training. A rejected agentic source fails runnable_sources(). A rejected task is a typed rejection whose reason the source manifest.json counts, and the surviving tasks continue.
Seeing it as a row here. Task-level rejections appear only in the source manifest and its Atlas source details.
—
Inventory only
A pinned reference, a declared row count where known and any independent QC that the upstream release recorded.
Any execution evidence. Quality shows QUICK (or its independent QC verdict as the Result). The source cannot be selected for training; it can still qualify through a later SAMPLE or FULL pass.
Grandfathered blend
Membership is exactly the frozen GRANDFATHERED_MARINSKYRL_SOURCES list: the Nemotron Ultra mixture components that predate the task-directory requirement. No new source can join the list.
A measured environment, a snapshot count or the environment-fidelity rules; these blends predate them and sit outside the current audit. Their training eligibility follows the same FULL-artifact rule as any other source.
Oversight not recorded
The source passed the format gate and is runnable. A non-agentic row may rest on a campaign record you can open in its source details.
A declared oversight level or its evidence. This snapshot does not establish execution coverage: absent metadata does not prove either that QA passed or that it never ran. An agentic source here still owes its SAMPLE or FULL QA (class (b)) before it stays runnable.
Excluded
The source notes say which defect or duplicate the review found. Its history stays in the Atlas.
Any of its tasks in a run. Excluded sources are not runnable.
Deprecated
A named replacement or withdrawal reason in the notes, and the retained history.
Any of its tasks in a run. runnable_sources() drops deprecated sources.
Admitted, SAMPLE
Every retained task passed the mechanical admission checks, including environment fidelity. Controls, verifier execution and a blinded review cover a documented sample of that retained cohort; the oversight evidence names the campaign, the recipe revision and the reviewed and admitted counts. Unsampled execution stays unverified.
That every task's verifier works. Without a positive control, neither label certifies solvability. Evaluation overlap may be recorded UNCHECKED, and admission can coexist with it.
Admitted, FULL
Every retained task ran end to end, including its verifier — declared oversight full, or executed controls that validated every eligible task.
Grader correctness beyond the controls and review: a live verifier can still mis-grade, and semantic review remains sampled. Selection removes exact heldout matches; it does not do semantic decontamination.
Daytona check: fits budget
The source's declared environments map to at most the snapshot limit of unique identities and pass the recorded capability rules. Static measurement only: no provisioning, quota headroom or rollout was tested.
Exceeds budget
A run would need more snapshots than the limit, or asks for something the capability rules refuse. The source stays ingested; gVisor or a rebuilt, digest-pinned image are options to validate, not guaranteed remedies.
Not measured
An agentic source without a recorded snapshot count: its Daytona behavior is unknown. Non-agentic sources provision no sandbox, so the label is N/A for them.
Source
Origin
Oversight
Daytona check
Oracle validation
The steps in detail
1 Declare the source
Each dataset is one RlDataSource in a module under datasets/, collected by all_sources() in sources.py. Its SourceInfo gives a stable ID, a title, an origin (where the rows came from, such as the Hugging Face organization), a family (the task family that selection quotas draw from), search tags and notes; the measured curriculum coverage is a separate record attached at load time, not another name for the family. Its input is pinned to an immutable revision.
A pipeline is the executable conversion procedure: one graph-building callable (usually process_rows) that reads the pinned rows and produces TaskSpecs and typed rejections. A source without a pipeline is Inventory: listed with its evidence, never admitted. The tag deprecated marks it Deprecated and excluded marks it Excluded; the notes carry the reason. Declaration, qualification and loading are independent: registering a source makes it visible; only a qualified release and an operator's load put its rows in the served snapshot.
An agentic source declares SourceInfo.task_format. A new agentic source must ingest TaskCompendium or Harbor task directories — in either format a task is one directory holding task.toml, its environment and its verifier. MARINSKYRL is Marin's internal blend format: JSONL rows produced by Marin's own RL pipeline before task directories existed, so a blend has no per-task directory to check. It is accepted only for the sources in GRANDFATHERED_MARINSKYRL_SOURCES, a frozen list of Nemotron Ultra components admitted before the requirement. The check raises for any other agentic source when the registry is built, so it fails before a run starts.
An agentic task's environment specification is part of the data, and the check runs per task: a task that fails is dropped and counted while the surviving tasks continue. Lowering translates a supported environment recipe into setup commands on a digest-pinned base; operations the lowering cannot express (COPY, ADD, USER, VOLUME, ENTRYPOINT and others) are refused. Fidelity preserves the declared recipe; it does not pin whatever unpinned external dependencies the recipe fetches from the network at build time. A task qualifies in one of two forms: built from the archive's own Dockerfile (docker_build), or lowered onto a pinned base with each RUN as a setup command and the recipe's working directory and environment variables. A shared image such as iris-task — the worker's own base image — never stands in for a task's environment. The one exception: when provably_no_op_against proves the Dockerfile changes nothing relative to that image (one FROM of its digest, no steps beyond WORKDIR, no copied fixtures), the task may declare the image, and the change ships as a new dataset version.
An agentic task that declares no environment is rejected; it would otherwise run in the host image. A source declaration cannot supply the missing environment.
A source is repaired when its published spec completely describes how its tasks run. A conversion may change delivery at ingestion, for example a reply-only rewrite that grades the assistant response in place of /app/answer.txt, but only with SAMPLE or FULL QA behind it, recorded as its own qualified transformation rather than an unmarked continuation. Audits sort sources into three classes:
(a) OK The spec describes runtime behavior and SAMPLE or FULL QA evidence exists.
(b) Needs QA The spec describes runtime behavior, but QA evidence is missing.
(c) Needs repair Runtime behavior differs from the spec: a substituted image, a missing environment resolved to the host, or a declared environment that the conversion ignores.
The curation driver requires --mode quick|sample|full. QUICK converts rows to TaskSpecs and records typed rejections; it reviews nothing and admits nothing, so a QUICK-only source has oversight none and stays inventory. SAMPLE reviews a bounded panel of at most 100 rows. FULL expands an accepted source to every row and verifies the kept tasks.
Mode and oversight are different axes. The mode is processing scope: how much of the source this pass examined. The oversight level, none, sample or full, is the weakest review the admission rests on, with evidence that points at the campaign, recipe revision and verification summary. A cohort whose setup ran on every task but whose verifier liveness ran on a sample is sample; a FULL-mode campaign can carry SAMPLE oversight. Independent QC by the upstream release does not raise the level.
Planned: oracle filtering. The policy this page is moving to runs an oracle agent over every task that ships an oracle, in both SAMPLE and FULL, and the admitted cohort becomes exactly the passing subset; execution routes through Harbor on DAYTONA when the task's snapshots fit the cap, or Harbor LOCAL otherwise. It is not implemented on this branch: today, executed controls sample at most 20 kept rows, and at a 95% control pass fraction the unsampled tasks skip validation.
SAMPLE or FULL QA must establish each of these, and the oversight evidence points at the result. One negative-control rule covers every grader: on an untouched workspace the verifier must return a well-formed negative verdict — exit code 0 and a trailing numeric reward below 1. A reward at or above 1 means the verifier rewards an unsolved workspace: the task is defective, not live. A crash, timeout or unparsable report is an infrastructure error, never a zero.
Conversion fidelity. Converted TaskSpecs match the archive's instruction, answer contract, environment and grader.
Controls. A golden or reference answer scores 1 wherever the source has one. An empty or wrong answer scores 0, unless the declaration records an expected partial reward for it. An empty answer that scores above 0 marks the task defective (check:empty).
Verifier execution. A reward of 0 alone does not prove the verifier ran; the negative-verdict rule above does. Executed controls sample at most 20 kept rows, and at a 95% control pass fraction the unsampled tasks skip validation.
Blinded review of a frozen sample, at least 20 tasks for SAMPLE, estimates the residual mis-grading rate: the audit-grader method seeds and hash-freezes the cases, conceals model, verdict and reviewer labels, and runs two independent reviews. It bounds the error rate on that sample; it does not certify every grader decision.
Drop accounting.input_rows equals converted rows plus rejections, and every rejection has a reason.
Evaluation overlap. Overlap with the tracked evaluation sets is checked with exact matching where a check exists, and recorded UNCHECKED otherwise. Admission may coexist with unchecked overlap; selection removes exact matches to the recipe's heldout set and does not establish semantic decontamination.
Sources without oracle solutions
When no row ships a solution, three checks replace the golden control. Payload preservation (offline): every file the native recipe copies appears in the rebuilt setup with identical bytes and mode. Setup execution (local Docker): the lowered setup commands, run in the declared base image, write those same files. Verifier liveness (sandbox): the task's own verifier runs in its graded environment on the untouched workspace and returns the well-formed negative verdict defined above — exit code 0, trailing reward below 1. A crash, timeout, unparsable report or a reward of 1 or more is a failure. Every sampled task must pass, 20 per base image by default. Each sampled task's recorded exit code and trailing reward form its liveness receipt. The result sets oracle_validation:
executed: shipped oracle solutions ran and scored their expected reward.
verifier_executed: no oracle exists, and the checks above passed. Successful solutions are not certified.
skipped: neither ran, so execution is not certified. This is the default for a source without oracles.
An LLM judge can be part of a task's grading semantics — distinct from the GLM reviewer that grades the curation panel, which is QA evidence, not the task's reward. A judged task whose judge model is empty, or whose rubric and settings are unstated, does not describe what grades it; the policy under development holds such a source at oversight none until it is repaired. To repair one, the repairer judges ten tasks from the frozen sample by hand, then pins the source to a judge tier. Planned: the tier table. Tiers are not yet implemented on this branch; the intended table (JUDGE_TIERS) is LIGHT for surface checks with a clear contract, MEDIUM for comparison with a reference or checklist, and HEAVY for multi-step verification that needs deep domain reasoning — every tier at temperature 0.7, or recording API default where the API cannot take a temperature.
What exists today is the judge contract the tiers will pin: the judge specification states the rubric, what the judge sees (question, answer, references, context and whether reasoning is removed), the context length, completion tokens and reasoning effort, the harness (API, prompts, label parsing), the turns, the request timeout and the retry budget. It says nothing about the endpoint: JudgeConnection supplies the URL and key at run time, so the connection can change independently of the judge specification. Revalidate grading behavior whenever the serving implementation or the effective model revision changes; matching a model name alone does not establish equal weights, revision or verdicts. End-to-end validation through the production grader must show a reference scores 1, a wrong or empty answer scores 0, agreement with the hand verdicts and a blinded review, and a parsed verdict for every request.
Marin's RL rollouts can run on Daytona, a hosted sandbox service whose organization caps how many snapshots (prebuilt environments) exist at once. This step is a static capacity and compatibility check, not a provisioning test: it materializes each task's declared agent and grader environments, derives their snapshot identities through the same snapshot_identity function that the Daytona factory uses to name snapshots, and applies the recorded capability rules. It makes no Daytona API calls and is never declared by hand, so it cannot observe current quota headroom, image availability, startup success or a completed rollout. A source exceeds the budget when its tasks need more unique snapshots than the limit, or when the capability rules refuse a task's declared requirements. A grader placed by a package lock runs in the local bubblewrap runtime on the worker, which Daytona cannot provision, so its snapshot count is irrelevant. An over-budget source stays ingested with its label.
Qualified, published and loaded are three separate operations, and only the last one changes the Atlas. A conversion first publishes its cohort to the open-athena Hugging Face organization as a private repository named open-athena/<family>-<format>-<YYYYMMDD>, in TaskCompendium format (native Harbor packages can also be imported directly). The card records provenance, excluded counts with reasons, the license and a research-artifact warning. Verification reads the upload back through the same login and compares file hashes. An importer that converts nothing brings the cohort into the catalog: import_prebuilt validates every TaskCompendium row under the admission predicates, and import_harbor_prebuilt requires a scored control receipt in controls.jsonl for every native Harbor package.
A FULL campaign with --publish-catalog writes the shared catalog snapshot. When a qualified release replaces an older source, the older source gets the deprecated tag and a note that names its replacement; a source can also be withdrawn without a replacement, with the reason in its notes. The Atlas changes only when an operator runs its loader on that snapshot, and the UI can legitimately disagree with the catalog until that load happens.
Admission does not put a source in a training run. A selection recipe (--selection RECIPE.json) names the sources and family quotas and pins a heldout set. Selection reads only final/*.parquet from sources whose manifests record mode: full; SAMPLE panels and QUICK conversions cannot supply the training population. Before sampling it removes tasks that match the complete heldout set by task ID, public text or exact conversation, and duplicates across sources. These are exact overlap checks.
Before a training run
Confirm the source is runnable, not deprecated or excluded, in the By source view.
Read its Quality and Result labels, then open the evidence page for the oversight pointers, control outcomes and reviewed/admitted counts behind them.
Check coverage: intended use, converted example count and leading curriculum area.
Check the backend: the Daytona check, plus the row's backend_capabilities map when it carries one. Package-lock graders run locally, never on Daytona.
Check the recorded evaluation-overlap status and the license; obtain access to the private release (Marin team members use their Hugging Face login; others ask the maintainers named in the source notes).
Write the selection recipe and require the FULL-mode artifact; the quick RL loop shows the handoff from the selection artifact to a training run.
To use data that is already in the Atlas you do not need to declare anything:
Find it. Search the By source view; filters hide QUICK-quality and BAD-result sources by default, and you can turn that off.
Identify the retained release. Open the source: the entry pins the dataset, its revision and the private converted release it came from.
Inspect the evidence. Open its evidence page for control, verifier and review coverage, and the campaign and recipe revision behind the oversight label.
Check the constraints. License, recorded evaluation-overlap status, and backend requirements (Daytona check, backend_capabilities when present).
Get access. Private open-athena releases need a Hugging Face login with membership; ask the maintainers named in the source notes if you are outside the team.
All stages: Python 3.12 or later, uv and Git. Clone the repository and run uv sync --all-packages (installation guide).
QUICK: a local computer. Public sources need no Hugging Face token.
SAMPLE with review: an OpenAI-compatible endpoint for a GLM-compatible review model — any endpoint serving the named model revision with an OpenAI chat API; the review model is QA tooling, distinct from a task's own judge. Pass --base-url, --model (default glm-5.3) and --model-revision, and put the key in GLM_BULK_TOKEN. Without --base-url the driver looks up Marin's internal relay job, which works only inside Marin's Iris cluster.
FULL and production: Iris access (authentication) and CoreWeave object-storage credentials in CW_KEY_ID and CW_KEY_SECRET.
Executable-environment cohorts: a Docker daemon, or a Linux dev box (below).
Marin team members can check everything at once with the onboard-marin skill. Never print a token or key in a log, report or commit.
Your first source
Declare it. Follow adding a dataset: add one RlDataSource to the nearest family module, pin its input, set config=CurationRecipe(...) and pipeline=process_rows, record the license in the notes, and register a new module in all_sources(). Agentic sources also declare task_format.
Test the conversion with a behavioral test in the family's test module:
uv run --with-editable './lib/taskcompendium[pipeline]' pytest \
experiments/post_training/task_curation/tests/test_<family>.py \
experiments/post_training/task_curation/tests/test_catalog.py
Find the invocation name.--source takes config.name, not the stable ID:
uv run python -c 'from experiments.post_training.task_curation.sources import runnable_sources; print("\n".join(runnable_sources()))'
Run QUICK locally and check that input_rows in each manifest.json equals converted_rows plus the sum of rejections:
Run SAMPLE with review. A 5-row panel is a conversion and review smoke check only; it does not meet the admission-review minimum of 20 tasks. Record the panel size and seed and carry both into the FULL campaign. QUICK outputs live under --output-root; SAMPLE and FULL retain artifacts under --cache-dir (locally) or --artifact-root, and a later pass reuses completed conversions from there.
Bubblewrap graders, Docker inside a pod and verifier liveness need Linux. scripts/iris/dev_cpu.py holds a privileged, non-preemptible CPU pod on a CoreWeave cluster, with the same container profile as curation workers. Name, size, priority and lifetime have no defaults. allocate returns once the pod runs; Iris ends the holder at --max-lifetime.
uv run python -m scripts.iris.dev_cpu --config lib/iris/config/cw-rno2a.yaml --name NAME \
allocate --cpu 16 --memory 64GB --disk 200GB --priority interactive --max-lifetime 259200
uv run python -m scripts.iris.dev_cpu --name NAME exec -- bash -lc 'bwrap --version; docker info'
uv run python -m scripts.iris.dev_cpu --name NAME cp-to ./grader.py /tmp/grader.py
uv run python -m scripts.iris.dev_cpu --name NAME release
Run FULL in an Iris job. Keep the SAMPLE run's --quality-sample-size and --seed so FULL reviews the same panel. Omit --run to print the plan with every resolved default. --publish-catalog requires FULL with assessment and no --cache-dir.
Iris runs default to 64 workers, a 16g coordinator, 32 normalized shards and the review cache <artifact-root>/rl-data/review-cache, whose keys hash the model revision and the complete request. Without --worker-image, workers use the cluster's default_task_image; pass a digest to record an exact image in the artifact identity.
Read the campaign report before loading: a failed FULL campaign also publishes a snapshot. Then load the Atlas:
uv run python infra/marina/apps/rl_data_catalog/load.py \
--app-id fb11c931-5861-4878-8bb5-a964d652b45f
Advanced authoring only: these tools build new task cohorts and register nothing as training-approved. Some cohorts need live containers and hidden-test separation, with positive and negative controls before any task is trusted. Each writes everything under the --output you choose:
Staged SWE-Smith:swesmith_staged pilots one parent (--pilot TASK:IMPORT:DEPENDENCY_COMMAND), swesmith_staged_cohort extends validated repositories, swesmith_staged_v2 builds shared-image repair handoffs, and swesmith_handoff validates a submitted workspace against the private contract. Gold patches and contracts stay under private/.
CalibForge:calibforge_environment, calibforge_fixtures, calibforge_native_references, calibforge_captured_fixtures, calibforge_equivalence and calibforge_release inventory, freeze, compare and package fixtures; calibforge_rebuild lowers recipes onto pinned bases.
Docker images these tools commit are host-local until an operator publishes them; exports record image IDs in local-images.json.
Where the evidence lives
What
Location
Pinned source bytes
The declared revision, cached under ~/.cache/marin or --cache-dir. On CoreWeave, set HF_HUB_DISABLE_XET=1.
Marina app fb11c931-5861-4878-8bb5-a964d652b45f, a Postgres copy of one catalog snapshot plus the curriculum release. Query it with uv run marina apps sql fb11c931-5861-4878-8bb5-a964d652b45f "SQL".
Lessons from earlier campaigns
Expect most first attempts to fail on job submission, credentials or grading before they touch the data. Use a new report directory for each attempt.
An inventory-only source reaches production only through a FULL campaign that publishes a snapshot containing it. Run it on a small runnable source if nothing else needs curating.
A failed campaign can move the catalog's CURRENT pointer. Read the report before loading.
Credentials given to the driver pod do not always reach the worker pods its coordinator starts. Check that the workers receive them.
Use an iris client that matches the deployed controller.
Retrying a failed review step returns the same cached judgments unless the grader or cache key changes.
Merging Atlas code changes nothing in production until someone runs marina release.
A task whose agent works in a machine: a shell, files or tools. Reply-only and conversation tasks need no agent machine.
Harbor, Harbor Hub
A task-directory format and runner: one directory per task with task.toml, an environment and a verifier. The Hub (hub.harborframework.com) hosts task releases; a digest pins one release's bytes.
TaskCompendium
Marin's task library and format; its TaskSpec is the canonical record of one task.
MarinSkyRL blend
Marin's internal RL format: JSONL rows from Marin's own RL pipeline, with no per-task directory. Accepted only for the frozen grandfather list.
Nemotron Ultra mix
The historical Marin RL mixture whose components make up the grandfather list; the listed source IDs are its admitted components.
Oracle, golden
A reference solution. A golden control grades it and expects reward 1.
Controls
Grader checks before admission: golden scores 1, empty or wrong scores 0.
Oversight level
The weakest review an admission rests on: none, sample or full. Distinct from the curation mode, which is processing scope.
Quality, Result
Atlas labels. Quality (QUICK, SAMPLE, FULL) is how much executed QA examined the source; independent upstream QC counts as QUICK. Result (GOOD, SOME_ISSUES, BAD) is what that QA found.
Lowering
Translating a supported environment recipe into setup commands on a digest-pinned base. Unsupported operations are refused; the declared recipe is preserved, not unpinned external dependency contents.
Shared image
A base image reused by many tasks (for example the worker's iris-task). A task may declare it only with a mechanical no-op proof.
Snapshot (catalog)
One published state of the curation catalog. The Atlas serves one at a time.
Snapshot (Daytona)
A prebuilt sandbox environment. Each unique agent or grader environment is one; snapshot_identity is the production naming rule.
Runnable
Has a pipeline and an Available status, and passes the format gate. Runnable for curation is not the same as runnable on a backend, and neither implies training-eligible.
Campaign
One driver invocation of the source procedure over selected sources; a coordinator schedules workers that process normalized shards.
Cohort
A named set of tasks published together as a versioned release. Authoring tools write candidate cohorts that register nothing; the catalog's admitted and excluded cohorts are curation populations.
Selection recipe
A JSON file that names sources and quotas for training and pins the heldout set.
CurationCatalog, CURRENT
The typed reader over one FineStore catalog view; CURRENT is its moving pointer to the latest published snapshot, which a load selects and copies.
Artifact identity
The resolved input, converter, grader and execution identities recorded in identity.json; artifacts are named <name>-<hash> under the artifact root.
Curriculum release
A published embedding-and-matching pass that measures each source's subject areas; it informs coverage display, not admission.
marina release vs loader
marina release deploys the Atlas app's code; the loader imports catalog data into its Postgres. Neither implies the other.
Task Trove, Snowball Smartie
Open-athena cohorts. Task Trove is an active source family in the catalog; Snowball Smartie is an earlier campaign whose artifacts and REPRODUCING.md remain published.
Iris, Zephyr, Marina
Marin's job scheduler, dataset-processing framework and internal app host.