
Pre-Training AuroraGPT
on Intel XPUs
Training, validation, and recovery at scale
Aurora, at production scale
- Racks
- 166
- Nodes
- 10,624
- GPUs
- 63,744Max 1550 · 2 XPUs
- CPUs
- 21,248Xeon Max
- Network
- Slingshot11 · 8 endpoints
- Memory
- 20.4 PB
- Storage
- 230 PBDAOS · 31 TB/s
~10,000 sq ft · 600 tons · built by Intel + HPE · one of the first exascale systems
Aurora cabinets, Argonne Leadership Computing Facility
One training program across vendors
🍋 ezpz: Write once, run anywhere.
[ github] [ezpz.cool]
# demo.py
import torch
import ezpz
rank = ezpz.setup_torch()
size = ezpz.get_world_size()
dev = ezpz.get_torch_device()
layer = torch.nn.Linear(128, 10)
model = ezpz.wrap_model(layer.to(dev))
print(f"Rank {rank}/{size}")$ ezpz launch python3 demo.py # 2N Aurora
Rank 0/24
Rank 1/24ezpz automatically picks the backend and handles the distributed launch + setup. See 🚀 ezpz launch.AuroraGPT production loss
2B stage 2 continues past 4.67T on the dolmino mix: 6.08T (256N) and 6.93T (512N). 2B reference: 7.77T tokens. Optimizer: SophiaG at 50M tokens per batch.
Living on the bleeding edge
how to keep your sanity when the software keeps moving
The run looked healthy. The step outputs disagreed — so every resync from upstream:main into ezpz passes a bit-exact gate before it ships, not a post-hoc plot comparison.
One stalled host holds up the step
ezpz launch --auto-retry
Thank you
AuroraGPT trains open foundation models for science.
ezpz + TorchTitan carry the run across devices, systems, and failures.
Acknowledgements
This research used resources of the Argonne Leadership Computing Facility at Argonne National Laboratory, a U.S. Department of Energy Office of Science User Facility, under Contract DE-AC02-06CH11357.
Aurora was built by HPE and Intel. Thanks to the ALCF operations and user-support teams; the Intel torch.xpu, oneCCL, and oneAPI teams; and the AuroraGPT collaborators.
Backup: held-out evaluation results
Accuracy vs. trillion tokens. Dashed grey = 2B MDS reference · pink = 2B 256N · blue dashed = 2B 512N · green = 20B 512N.
Backup: SophiaG at 50M tokens per batch
W&B aurora_gpt/AuroraGPT, optimizer-experiments/Megatron-DeepSpeed outdir, 12-layer gemma-7B tokenizer, rope θ=50k, zero-stage 0, no activation checkpointing, mbs=1, grad-accum ≥ 2, 50M-token batches, world size 3072, torch ≥ 2.8, olmo-mix-1124. Resume chains from the campaign’s auto-retry are merged longest-run-first. The SophiaG grad-norm trace shows brief 1–3-step spikes to 10–114 (max 113.7; most are a single logged step) scattered through healthy runs, not tied to a single resume boundary; the smoothed line uses a median pre-filter and the spike heights should not be read as real gradient magnitudes. Log scale; smoothed with the raw trace faint behind. Source: scripts/fetch-sc26-optimizer.py.
Backup: post-training the 2B — SFT
SFT (tulu_math_uc_mix): loss / grad-norm / LR / token accuracy / entropy / tokens seen vs. global step. Final loss 0.77 over ~4.5B tokens; loss curve from the failover-chain logs (72 logged steps).
SFT gives the capability; the base model is the ceiling.
Backup: post-training the 2B — GRPO
GRPO on the SFT’d 2B (sum_digits, 1000 steps, 8 nodes): accuracy reward ~0.45 → ~0.77 (raw trace max ~0.95); format reward saturated by ~step 60. Raw trace lightly decimated; trainer state lives on Sunspot.
GRPO polishes an already-solved task. Full write-up: POST-TRAINING-2B.md
Backup: Recovery at job, node, and process scope