Implement "learning to discover at test time" Erdos env + training code#1

Open

mormio wants to merge 48 commits intomainfrom

Commits on Mar 31, 2026

feat: add Erdős Discovery environment + entropic adaptive-β advantage estimator
Morgane Moss
committed
docs: add PUCT buffer utility, Erdős discovery README, update main README
Morgane Moss
committed
feat: add TTT-Discover run script and config for Erdős GRPO training
Morgane Moss
committed
feat: add debug config and SLURM launcher for Erdős TTT-Discover
Morgane Moss
committed
feat: add inline mode for ErdosDiscoveryEnvironment, simplify debug launch
Morgane Moss
committed
fix: run_discover.py config loading, setup/grpo_train signatures, debug iterations
Morgane Moss
committed

Commits on Apr 1, 2026

fix: v0.5.0 container compatibility - handle missing register_omegaconf_resolvers, add launch scripts and shim for container deployment
Morgane Moss
committed
fix: register mul/div OmegaConf resolvers for v0.5.0 compat
Morgane Moss
committed
fix: sync step method for Ray actor event loop compatibility
Morgane Moss
committed
fix: observations must include content key for rollout engine
Morgane Moss
committed
fix: disable CPU offload for debug (1.5B fits on GPU)
Morgane Moss
committed
docs: add LESSONS_LEARNED.md for Erdős TTT-Discover on NeMo RL
Morgane Moss
committed
feat: add 120B Nemotron Super launch config for Erdős TTT-Discover
Morgane Moss
committed
fix: inherit from grpo_superv3.yaml for correct Megatron+NemotronH config
Morgane Moss
committed
fix: set async_engine false for 120B to avoid engine core crash
Morgane Moss
committed
fix: batch size 504 (8×63) divisible by DP=12
Morgane Moss
committed
fix: version-agnostic setup() unpacking for super-v3 container compat
Morgane Moss
committed
fix: explicit setup→grpo_train wiring for super-v3 container (11 return values)
Morgane Moss
committed
fix: reduce seq length to 4096 to avoid OOM during training
Morgane Moss
committed
feat: port exact reference TTT-Discover env + prompts
Morgane Moss
committed
feat: 8k seq len, wandb logging, erdos/ metrics with max reward + valid rate
Morgane Moss
committed
fix: use ThreadPoolExecutor timeout instead of signal.alarm (broken in Ray actors)
Morgane Moss
committed
feat: add timestamped progress logging to reward computation
Morgane Moss
committed
fix: remove stale _Timeout reference that crashed env actor on bad model output
Morgane Moss
committed
fix: use print() instead of logger.info() for Ray actor visibility
Morgane Moss
committed
feat: prominent step-level logging with max_reward, best_C5, global_best_C5
Morgane Moss
committed
fix: disable validation to prevent max_val_samples None crash at step 5
Morgane Moss
committed

Commits on Apr 2, 2026

fix: disable checkpointing (async writer crashes at step 10)
Morgane Moss
committed
feat: scale to 10 nodes + 16k seq len (CP=2), save outputs to JSONL per step
Morgane Moss
committed
fix: fully disable checkpointing with null checkpoint_must_save_by
Morgane Moss
committed
feat: debug config for step 10 hang repro (Qwen 1.5B, 1 node, 16k, 15 steps)
Morgane Moss
committed
fix: ensure checkpointing config exists for both v0.5.0 and super-v3 containers
Morgane Moss
committed
fix: inject checkpointing into master_config returned by setup() (not just input config)
Morgane Moss
committed
debug: print setup() return types to fix unpacking order
Morgane Moss
committed
fix: correct v0.5.0 setup() unpacking order (clusters at [2], not dataloader)
Morgane Moss
committed
fix: debug config back to 4k (16k OOMs on 1 node with 1.5B + LoRA)
Morgane Moss
committed
120B at 4k context, 8 nodes, 50 steps, checkpointing fully disabled
Morgane Moss
committed
fix: use multiprocessing.Process + kill() for hard sandbox timeout (threads cant be killed)
Morgane Moss
committed
fix: sandbox timeout 1000s matching paper, not 120s
Morgane Moss
committed
fix: clean subprocess sandbox - BaseException for alarm, SIGTERM before SIGKILL, 1000s timeout
Morgane Moss
committed

Commits on Apr 3, 2026

feat: stateful PUCT sampler integrated into RL env + dataset
Morgane Moss
committed
config: 8 nodes, 16k seq, CP=2, copy erdos_ref_puct_sampler to container
Morgane Moss
committed
cleanup: config naming, PUCT log dir, remove stale puct_buffer copy
Morgane Moss
committed
fix: point config to instruct model (no Base)
Morgane Moss
committed
script to convert ds to nemo rl/sft format
Morgane Moss
committed

Commits on Apr 7, 2026

cleanup: remove debug configs, shim copies, scratch files, and Gym submodule
Morgane Moss
committed
cleanup: remove unused puct_buffer.py and ray.sub.bak
Morgane Moss
committed
restore Gym submodule to match main (avoid merge conflict)
Morgane Moss
committed