QR code linking to samf.sh/talks/2026/11/20samf.sh/talks/2026/11/20Aurora at the Argonne Leadership Computing Facility
AuroraGPT · SC26

Pre-Training AuroraGPT
on Intel XPUs

Training, validation, and recovery at scale

Sam ForemanNathan Nichols · Varuni Sastry · Sam Wheeler · Khalid HossainHuihuo Zheng · Murali Emani · Filippo Simini · Marieme Ngom · Ethan Wong · Venkatram Vishwanath
4th Annual Trillion Parameter Workshop · November 20, 2026

Aurora, at production scale

Racks
166
Nodes
10,624
GPUs
63,744Max 1550 · 2 XPUs
CPUs
21,248Xeon Max
Network
Slingshot11 · 8 endpoints
Memory
20.4 PB
Storage
230 PBDAOS · 31 TB/s

~10,000 sq ft · 600 tons · built by Intel + HPE · one of the first exascale systems

Aurora cabinets, Argonne Leadership Computing Facility

One training program across vendors

🍋 ezpz: Write once, run anywhere.
[ github] [ezpz.cool]

# demo.py
import torch
import ezpz
rank = ezpz.setup_torch()
size = ezpz.get_world_size()
dev = ezpz.get_torch_device()
layer = torch.nn.Linear(128, 10)
model = ezpz.wrap_model(layer.to(dev))
print(f"Rank {rank}/{size}")
$ ezpz launch python3 demo.py  # 2N Aurora
Rank 0/24
Rank 1/24
%%{init: {'themeVariables': {'fontSize': '22px'}, 'flowchart': {'nodeSpacing': 40, 'rankSpacing': 36, 'padding': 10}}}%% flowchart LR EZ{"<span style='font-weight:700'>ezpz launch</span>"} EZ --> AURORA["<span style='font-weight:700'>Aurora</span><br/><i>PBS · Intel</i>"] EZ --> POLARIS["<span style='font-weight:700'>Polaris</span><br/><i>PBS · NVIDIA</i>"] EZ --> PM["<span style='font-weight:700'>Perlmutter</span><br/><i>SLURM · NVIDIA</i>"] EZ --> FRONTIER["<span style='font-weight:700'>Frontier</span><br/><i>SLURM · AMD</i>"] EZ --> MAC["<span style='font-weight:700'>Multi-CPU</span>"] EZ --> MPS["<span style='font-weight:700'>Multi-MPS*</span><br/><i>MPS · Apple</i>"] classDef hub fill:none,stroke:currentColor,color:currentColor,stroke-width:1.8px classDef aurora stroke-width:1.5px classDef polaris stroke-width:1.5px classDef perlmutter stroke-width:1.5px classDef frontier stroke-width:1.5px classDef mac stroke-width:1.5px classDef mps stroke-width:1.5px class EZ hub class AURORA aurora class POLARIS polaris class PM perlmutter class FRONTIER frontier class MAC mac class MPS mps
ezpz automatically picks the backend and handles the distributed launch + setup. See 🚀 ezpz launch.

AuroraGPT production loss

AuroraGPT training-loss trajectories versus tokens consumed, including 2B stage-2 dolmino continuations

2B stage 2 continues past 4.67T on the dolmino mix: 6.08T (256N) and 6.93T (512N). 2B reference: 7.77T tokens. Optimizer: SophiaG at 50M tokens per batch.

Living on the bleeding edge

how to keep your sanity when the software keeps moving

%%{init: {'gitGraph': {'mainBranchName': 'upstream:main', 'showCommitLabel': false}}}%% gitGraph LR: commit id: "base" msg: "upstream HEAD — ezpz forks here" branch ezpz order: 1 commit id: "e1" msg: "ezpz: XPU + oneCCL" commit id: "e2" msg: "experiments/ezpz fork" checkout "upstream:main" commit id: "u2" msg: "upstream: a refactor lands" checkout ezpz merge "upstream:main" tag: "resync" commit id: "gate1" type: REVERSE tag: "diff ≠ 0" msg: "50-step smoke: loss disagrees" commit id: "fix" msg: "bisect → fix" commit id: "gate2" type: HIGHLIGHT tag: "bit-exact" msg: "re-run: bits match"

The run looked healthy. The step outputs disagreed — so every resync from upstream:main into ezpz passes a bit-exact gate before it ships, not a post-hoc plot comparison.

One stalled host holds up the step

ezpz launch --auto-retry
Eight training hosts exchange gradients. After a stall, a spare replaces one host and the trainer reloads a checkpoint.
Batch shards 0–7 Checkpoint / spare
Example attempt step 3003 · distributing batch shards

Thank you

AuroraGPT trains open foundation models for science.

ezpz + TorchTitan carry the run across devices, systems, and failures.

Acknowledgements

This research used resources of the Argonne Leadership Computing Facility at Argonne National Laboratory, a U.S. Department of Energy Office of Science User Facility, under Contract DE-AC02-06CH11357.

Aurora was built by HPE and Intel. Thanks to the ALCF operations and user-support teams; the Intel torch.xpu, oneCCL, and oneAPI teams; and the AuroraGPT collaborators.

Backup: held-out evaluation results

HellaSwag accuracy versus trillion tokens
ARC-Easy accuracy versus trillion tokens
ARC-Challenge accuracy versus trillion tokens
Winogrande accuracy versus trillion tokens

Accuracy vs. trillion tokens. Dashed grey = 2B MDS reference · pink = 2B 256N · blue dashed = 2B 512N · green = 20B 512N.

Backup: SophiaG at 50M tokens per batch

W&B aurora_gpt/AuroraGPT, optimizer-experiments/Megatron-DeepSpeed outdir, 12-layer gemma-7B tokenizer, rope θ=50k, zero-stage 0, no activation checkpointing, mbs=1, grad-accum ≥ 2, 50M-token batches, world size 3072, torch ≥ 2.8, olmo-mix-1124. Resume chains from the campaign’s auto-retry are merged longest-run-first. The SophiaG grad-norm trace shows brief 1–3-step spikes to 10–114 (max 113.7; most are a single logged step) scattered through healthy runs, not tied to a single resume boundary; the smoothed line uses a median pre-filter and the spike heights should not be read as real gradient magnitudes. Log scale; smoothed with the raw trace faint behind. Source: scripts/fetch-sc26-optimizer.py.

Backup: post-training the 2B — SFT

SFT (tulu_math_uc_mix): loss / grad-norm / LR / token accuracy / entropy / tokens seen vs. global step. Final loss 0.77 over ~4.5B tokens; loss curve from the failover-chain logs (72 logged steps).

SFT gives the capability; the base model is the ceiling.

Backup: post-training the 2B — GRPO

GRPO on the SFT’d 2B (sum_digits, 1000 steps, 8 nodes): accuracy reward ~0.45 → ~0.77 (raw trace max ~0.95); format reward saturated by ~step 60. Raw trace lightly decimated; trainer state lives on Sunspot.

GRPO polishes an already-solved task. Full write-up: POST-TRAINING-2B.md

Backup: Recovery at job, node, and process scope

flowchart TB subgraph job["📋 JOB scope — PBS · hours"] JOB_TXT["<i>crash / walltime</i> →<br/><span style='font-weight:700'>chained resubmit</span>"] subgraph node["🖥 NODE scope — failover wrapper · minutes"] NODE_TXT["<i>bad host detected</i> →<br/><span style='font-weight:700'>swap from spare pool</span>"] subgraph proc["⏱ PROCESS scope — ezpz launch · seconds"] PROC_TXT["<i>stdout idle ≥ N s</i> →<br/><span style='font-weight:700'>kill + backoff</span>"] end end end classDef jobC stroke-width:1.5px classDef nodeC stroke-width:1.5px classDef procC stroke-width:1.5px classDef jobTxt fill:transparent,stroke:none classDef nodeTxt fill:transparent,stroke:none classDef procTxt fill:transparent,stroke:none class JOB_TXT jobTxt class NODE_TXT nodeTxt class PROC_TXT procTxt class job jobC class node nodeC class proc procC
▦ JOB scope PBS · hours
crash / walltimechained resubmit
⬡ NODE scope failover wrapper · minutes
bad host detectedswap from spare pool
◉ PROCESS scope ezpz launch · seconds
stdout idle ≥ N skill + backoff
1
1
 Slides
Theme
▣ Deck
 Font
 Code Font
 Table of Contents
    Keybinds
    j / →Next slide / fragment k / ←Previous slide / fragment SpaceAdvance (fragments first) g / GFirst / last slide Ctrl+.Open speaker notes window EscToggle overview / close dialog fFullscreen pPrint / export to PDF ? / : / mThis dialog