 Command

Sam Foreman's personal site. Vim-style keybinds for navigation; theme + font pickers below.

Theme
 Font Body Code
Reader
Keybinds
Navigation
j / ↓ Next item k / ↑ Previous item g First item in region G Last item in region zz Center focused item h / l Sidebar / main content ] / [ Next/previous heading } / { Next/previous block d / u Half-page down/up
Layout
<zh> Toggle sidebar <zr> Toggle reader view <zj> / <zk> Focus main / actions ⇧C / ⇧E  ·  <zM> / <zR> Collapse / expand all sections
Dialogs
⌃P / : Command palette ⌃X Theme picker / Search ? Show keybinds ⌃N / ⌃P Next/prev search result Esc Close dialog / exit reader
History
n Next document b Previous document ⌃O History back ⌃I History forward
Sections
a about p posts t talks m more s style
 Search
about: Sam Foreman ideas: 💡 Ideas more: ➕ More now: Now posts: 📬 Posts posts/2023/12/05: 🔳 l2hmc-qcd Example: 4D SU(3) posts/2025: 📆 2025 posts/2025/04/28: 🔥 Building PyTorch 2.6 from Source on Aurora posts/2025/05/03: 🚧 Frameworks Issue with numpy \› 2 posts/2025/06: 06 posts/2025/06/01: 📰 Nice Headings posts/2025/06/02: 🧜‍♀️ Mermaid posts/2025/06/14: 🏗️ Building PyTorch 2.8 from Source on Aurora posts/2025/09/12: 🍹 BlendCorpus + TorchTitan @ ALCF posts/2025/09/17: 📊 pbs-tui: TUI for PBS Job Scheduler Monitoring posts/2025/10/06: 🎨 Mixing Between Distributions While Training posts/2025/11/12: 🧊 Cooling Down Checkpoints: Best Practices for Model Evaluation posts/2026/01/07: 🎉 Happy New Year! posts/2026/01/10: 🍋 ezpz: distributed PyTorch across any hardware posts/2026/02/28: ⏱️ Comparing Launchers on Aurora posts/2026/02/28: ## torchrun posts/2026/02/28: ## ezpz posts/2026/04/27: Pre-Training AuroraGPT with TorchTitan posts/2026/04/27: ## Two-Week Summary (Apr 12–27, 2026) posts/2026/04/27: ## Detailed Breakdown posts/2026/04/27: ### Week 1: Apr 12–18 — Benchmarking, LR Finder, XPU Fixes posts/2026/04/27: #### Benchmarking (Apr 12–15) posts/2026/04/27: #### LR Finder (Apr 12–14) posts/2026/04/27: #### Scaling Study (Apr 12) posts/2026/04/27: #### Upstream Syncs (Apr 12–18, syncs 6–14) posts/2026/04/27: #### XPU Bug Fixes (Apr 18) posts/2026/04/27: #### RL Experiment (Apr 18) posts/2026/04/27: ### Week 1.5: Apr 18–25 — Production Readiness posts/2026/04/27: #### Torch 2.12 Benchmarks (Apr 18) posts/2026/04/27: #### LR Finder Extensions (Apr 20–21) posts/2026/04/27: #### XPU Fixes (Apr 23) posts/2026/04/27: #### Torch 2.13 Environment (Apr 25) posts/2026/04/27: #### 2B Scaling Study on Torch 2.13 (Apr 25) posts/2026/04/27: #### Production Training (Apr 25) posts/2026/04/27: ### Week 2: Apr 26–27 — Optimizer Competition posts/2026/04/27: #### RL Multi-Task Refactor (Apr 26) posts/2026/04/27: #### Docs Reorganization (Apr 26) posts/2026/04/27: #### Generic HF Dataset Streaming (Apr 26) posts/2026/04/27: #### New Optimizers (Apr 26) posts/2026/04/27: #### Architecture Tweaks (Apr 26–27) posts/2026/04/27: ## Competition Results posts/2026/04/27: ### Round 1–3: Speedrun — 2N, GBS=48, 1000 steps posts/2026/04/27: ### 10B Full Training — 8N, GBS=384, ~3,178 steps posts/2026/04/27: ### Round 4: Reproducible Speedrun — 2N, GAS=8, GBS=384, 1000 steps posts/2026/04/27: ## Key Discoveries posts/2026/04/27: ## Infrastructure Built posts/2026/04/27: ## High-Level posts/2026/04/27: ## Detailed Breakdown posts/2026/04/27: ### Week 1: Apr 12–18 — Benchmarking, LR Finder, XPU Fixes posts/2026/04/27: #### Benchmarking (Apr 12–15) posts/2026/04/27: #### LR Finder (Apr 12–14) posts/2026/04/27: #### Scaling Study (Apr 12) posts/2026/04/27: #### Upstream Syncs (Apr 12–18, syncs 6–14) posts/2026/04/27: #### XPU Bug Fixes (Apr 18) posts/2026/04/27: #### RL Experiment (Apr 18) posts/2026/04/27: ### Week 1.5: Apr 18–25 — Production Readiness posts/2026/04/27: #### Torch 2.12 Benchmarks (Apr 18) posts/2026/04/27: #### LR Finder Extensions (Apr 20–21) posts/2026/04/27: #### XPU Fixes (Apr 23) posts/2026/04/27: #### Torch 2.13 Environment (Apr 25) posts/2026/04/27: #### 2B Scaling Study on Torch 2.13 (Apr 25) posts/2026/04/27: #### Production Training (Apr 25) posts/2026/04/27: ### Week 2: Apr 26–27 — Optimizer Competition posts/2026/04/27: #### RL Multi-Task Refactor (Apr 26) posts/2026/04/27: #### Docs Reorganization (Apr 26) posts/2026/04/27: #### Generic HF Dataset Streaming (Apr 26) posts/2026/04/27: #### New Optimizers (Apr 26) posts/2026/04/27: #### Architecture Tweaks (Apr 26–27) posts/2026/04/27: ## Competition Results posts/2026/04/27: ### Round 1–3: 1000-step speedruns, 2 nodes, GBS=48 (17 configs) posts/2026/04/27: ### Round 4 (10B full training, 8 nodes, GBS=384, 5 configs) posts/2026/04/27: ### Round 5 (2 nodes, GAS=8, GBS=384, local dataset, 8 configs — in progress) posts/2026/04/27: ## Key Discoveries posts/2026/04/27: ## Infrastructure Built posts/2026/05/01: Running 50k Python Processes on Aurora with ezpz yeet posts/2026/06/27: Local AI Apps on ALCF: Argo, Inference Endpoints, and One Gateway posts/2026/06/28: Migrating from Quarto to Astro: samforeman.me → samf.sh posts/2026/08/08: Pre-Training LLMs on a Supercomputer posts/2026/09/22: Working From Anywhere: Persistent Access to Compute and Context posts/2026/09/25: A Small Service Mesh for My Macs and Supercomputers posts/2026/10/16: ezpz from anywhere: remote submission via the ALCF IRI API posts/2026/11/11: Benchmarking ALCF systems with torch-hammer posts/2026/11/16: ⚖️ inkling-2 vs muse-spark-bf16 — GSM8K + HumanEval on ALCF posts/ai-for-physics: ⚛️ AI for Physics posts/ai-for-physics/diffusion: 🎲 MCMC + Diffusion Sampling posts/ai-for-physics/l2hmc-qcd: 🎢 L2HMC for LQCD posts/ai-for-physics/l2hmc-qcd/2du1: 🎢 l2hmc-qcd Example: 2D U(1) posts/auroragpt: 🤖 AuroraGPT posts/auroragpt/aurora-gpt: 🏎️ Megatron-DeepSpeed on Intel XPU posts/auroragpt/checkpoints: 💾 Converting Checkpoints posts/auroragpt/determinstic-flash-attn/deterministic-flash-attn: 🎰 Deterministic flash-attn posts/auroragpt/flash-attn-sunspot: 📸 flash-attn on Sunspot posts/auroragpt/long-sequences: 🚂 Loooooooong Sequence Lengths posts/auroragpt/mpi4py-reproducer: 🐛 mpi4py bug on Sunspot posts/auroragpt/spike-skipper: 🏔️ Spike Skipper posts/auroragpt/startup-times: 🐢 Starting Up Distributed Training on Aurora posts/auroragpt/startup-times: ## Response posts/auroragpt/startup-times: ### Measuring / Calculating Startup Time posts/auroragpt/startup-times: ## Minimal Working Example posts/croon: croon: Synced Lyrics in the Terminal, for Whatever Is Playing posts/dope-slides: 💅 How to Make Dope Slides posts/drafts/2025/09/22: 📝 2025 Annual Report posts/ezpz-at-alcf: 🍋 ezpz @ ALCF posts/ezpz-v1: 📝 ezpz-v1 posts/globusfs: globusfs: an fsspec Filesystem for Globus Collections posts/jupyter: 📗 Jupyter posts/jupyter/test: 🏁 l2hmc Example: 2D $U(1)$ posts/resume: 🧑🏻‍💻 Sam Foreman’s Résumé posts/svgbob: 🫥 svgbob posts/torchtune-aurora: 🪛 Torchtune on Aurora posts/torchtune-patch-aurora: 🚑 Torchtune Patch on Aurora posts/wandb-tui: wandb-tui: Comparing W&B Runs Without Leaving the Terminal projects: 📚 Projects talks: 🎙️ Talks talks/2025/09/24: Training Foundation Models on Supercomputers talks/2025/10/08: AERIS: Argonne's Earth Systems Model talks/2025/10/15: Training Foundation Models on Supercomputers talks/2025/10/24: Training Foundation Models on Supercomputers talks/2025/12/16: AuroraGPT: Training Foundation Models on Supercomputers talks/2026/06/03: Production Pre-Training at Scale: The Good, the Bad, and the Restarts talks/2026/07/14: Pre-Training AuroraGPT at Scale on Aurora talks/2026/08/03: Pre-Training LLMs on a Supercomputer talks/2026/10/15: Profiling and Optimization of AI Training Workloads talks/2026/10/21: Scaling PyTorch on Intel XPU: Field Notes from AI for Science on Aurora talks/2026/10/30: Towards Resilient Training and Fault-Tolerant Software at ALCF talks/2026/11/20: Pre-Training AuroraGPT on Intel XPUs talks/ai-for-science-2024: Parallel Training Methods talks/alcf-hpc-workshop-2024/alcf-hpc-workshop-2024: Deep Learning and Foundation Models at Scale talks/aurora-gpt-fm-for-electric-grid/auroragpt-fm-for-electric-grid: AuroraGPT: Foundation Models for Science talks/auroragpt-siam25: AuroraGPT talks/auroragpt/alcf-hpc-workshop-2024/auroragpt-alcf-hands-on-hpc-workshop-2024: AuroraGPT: ANL's General Purpose Scientific LLM talks/demo-slides: AuroraGPT: Training Foundation Models on Supercomputers talks/hpc-user-forum/auroragpt: AuroraGPT talks/incite-hackathon-2025: ALCF Incite Hackathon 2025 talks/incite-hackathon-2025/auroragpt: LLMs on Aurora: Overview talks/incite-hackathon-2025/ezpz: LLMs on Aurora: Hands-On talks/llms-at-scale: Training LLMs at Scale talks/llms-on-polaris: Training LLMs on Polaris talks/openskai25: Open SkAI2025 talks/openskai25/ai4science: Scientific AI at Scale: AuroraGPT talks/openskai25/training: Scientific AI at Scale: Distributed Training webtui: Style webtui/components/accordion: Accordion webtui/components/badge: Badge webtui/components/button: Button webtui/components/checkbox: Checkbox webtui/components/dialog: Dialog webtui/components/input: Input webtui/components/popover: Popover webtui/components/pre: Pre webtui/components/progress: Progress webtui/components/radio: Radio webtui/components/range: Range webtui/components/separator: Separator webtui/components/spinner: Spinner webtui/components/switch: Switch webtui/components/table: Table webtui/components/textarea: Textarea webtui/components/tooltip: Popover webtui/components/typography: Typography webtui/components/view: View webtui/contributing/contributing: Contributing webtui/contributing/contributing: ## Local Development webtui/contributing/contributing: ## Issues webtui/contributing/contributing: ## Pull Requests webtui/contributing/style-guide: Style Guide webtui/contributing/style-guide: ## CSS Units webtui/contributing/style-guide: ## Selectors webtui/contributing/style-guide: ## Documentation webtui/installation/astro: Astro webtui/installation/astro: ## Scoping webtui/installation/astro: ### Frontmatter Imports webtui/installation/astro: ### ‹style› tag webtui/installation/astro: ### Full Library Import webtui/installation/nextjs: Next.js webtui/installation/vite: Vite webtui/plugins/plugin-dev: Developing Plugins webtui/plugins/plugin-dev: ### Style Layers webtui/plugins/plugin-nf: Nerd Font Plugin webtui/plugins/theme-catppuccin: Catppuccin Theme webtui/plugins/theme-custom: Custom Theme webtui/plugins/theme-everforest: Everforest Theme webtui/plugins/theme-gruvbox: Gruvbox Theme webtui/plugins/theme-nord: Nord Theme webtui/plugins/theme-vitesse: Vitesse Theme webtui/start/ascii-boxes: ASCII Boxes webtui/start/changelog: Changelog webtui/start/installation: Installation webtui/start/installation: ## Installation webtui/start/installation: ## Using CSS webtui/start/installation: ## Using ESM webtui/start/installation: ## Using a CDN webtui/start/installation: ## Full Library Import webtui/start/installation: ### CSS webtui/start/installation: ### ESM webtui/start/installation: ### CDN webtui/start/intro: Introduction webtui/start/intro: ## Features webtui/start/plugins: Plugins webtui/start/plugins: ## Official Plugins webtui/start/plugins: ### Themes webtui/start/plugins: ## Community Plugins webtui/start/theming: Theming webtui/start/theming: ## CSS Variables webtui/start/theming: ### Font Styles webtui/start/theming: ### Colors webtui/start/theming: ### Light & Dark webtui/start/theming: ## Theme Plugins webtui/start/theming: ### Using Multiple Theme Accents webtui/start/tuis-vs-guis: TUIs vs GUIs webtui/start/tuis-vs-guis: ## Monospace Fonts webtui/start/tuis-vs-guis: ## Character Cells
 Theme Current: Light j/k or ↑/↓ + Enter

⚖️ inkling-2 vs muse-spark-bf16 — GSM8K + HumanEval on ALCF

Head-to-head GSM8K and HumanEval on ALCF minerva: inkling-2 (inkling-2-small, 1M ctx) vs muse-spark-bf16, via lm-eval through the llm-rosetta gateway. Numbers, rollouts, and the harness fix that turned 0.0 into 0.85.

Two models on the same box, same harness, same prompts. One wins math by 21 points. Code is closer.

Numbers

Benchmarkinkling-2 (inkling-2-small, 1M ctx)muse-spark-bf16[1m]ΔWinner
GSM8K flex (n=200)0.945 ± 0.0160.730 ± 0.032+0.215inkling-2
GSM8K strict (n=200)0.935 ± 0.0180.725 ± 0.032+0.210inkling-2
HumanEval pass@1 (n=164)0.848 ± 0.0280.787 ± 0.032+0.061inkling-2

The GSM8K gap is 6σ. The HumanEval gap is 1.4σ pooled. The pattern matters more than either interval alone: large reasoning lead, small code lead.

How it was run

flowchart LR A@{ shape: stadium, label: "ALCF minerva<br/>inkling-2-small<br/>muse-spark-bf16" } --> B@{ shape: rect, label: "llm-rosetta gateway<br/>:8765 /v1/chat/completions" } B --> C@{ shape: rect, label: "lm-eval local-chat-completions<br/>apply_chat_template=true<br/>num_concurrent=4" } C --> D1@{ shape: rect, label: "GSM8K<br/>200 sampled<br/>flex + strict" } C --> D2@{ shape: rect, label: "HumanEval<br/>164 full set<br/>fenced-block extract" } D1 --> E@{ shape: stadium, label: "pass@1 + stderr" } D2 --> E
SettingValue
Runnerllm-rosetta gateway on minerva (http://127.0.0.1:8765/v1/chat/completions)
Harnesslm-evaluation-harness via local-chat-completions, temperature=0, max_tokens=4096, num_concurrent=4
GSM8Klm-eval gsm8k flex + strict exact_match, apply_chat_template
HumanEvalcustom humaneval_chat.yaml — single ```python fenced block, build_predictions_code, until: ["\n\n\n"]

Token usage from the rollout captures:

Promptinkling-2 reasoningmuse-spark-bf16 reasoning
GSM8K easy (48 clips)75447
GSM8K hard (6 cats)110496
HumanEval has_close_elements8461314
HumanEval separate_paren_groups3997 at 4096, 8190 at 8192 (length)4093 at 4096, 8189 at 8192 (length)

inkling-2 is terse on math. muse-spark-bf16 reasons longer even when the answer is one division. Both blow the budget on separate_paren_groups at both 4096 and 8192. That last row is a harness limit, not a capability signal — the 164-problem aggregate still clears because most docs are short.

flowchart TD Q@{ shape: stadium, label: "Question" } Q --> R1@{ shape: rect, label: "inkling-2<br/>short CoT<br/>75-110 tok" } Q --> R2@{ shape: rect, label: "muse-spark<br/>long CoT<br/>447-496 tok" } R1 --> A@{ shape: stadium, label: "Same answer<br/>72 / $150" } R2 --> A
xychart-beta title "GSM8K and HumanEval — pass rate" x-axis ["GSM8K flex", "GSM8K strict", "HumanEval"] y-axis "pass rate" 0 --> 1 bar "inkling-2" [0.945, 0.935, 0.848] bar "muse-spark-bf16" [0.73, 0.725, 0.787]
xychart-beta title "inkling-2 lead in points" x-axis ["GSM8K flex", "HumanEval"] y-axis "delta" 0 --> 0.25 bar "lead (points)" [0.215, 0.061]

Rollouts

All rollouts are live POST /v1/chat/completions captures at temperature=0, max_tokens=4096 unless noted, written to /tmp/rollout-*.txt on 2026-11-16. No edits.

GSM8K — easy

Q: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? A: Let’s think step by step.

inkling-2 — 139 chars, 75 reasoning tok, stop:

Natalia sold 48 clips in April.
In May, she sold half as many: 48 / 2 = 24 clips.
Altogether: 48 + 24 = 72 clips.

**Answer: 72 clips**

muse-spark-bf16 — 143 chars, 447 reasoning tok, stop:

In April Natalia sold 48 clips.

In May she sold half as many as in April:
48 / 2 = 24 clips.

Altogether in April and May:
48 + 24 = 72 clips.

Both correct. inkling-2 used 6x fewer reasoning tokens for the same result and adds **Answer:** markup. muse-spark-bf16 is plain prose.

GSM8K — hard

Q: Janessa has 3 cats. She buys 10 new toys for each cat and each toy costs 4.Afterbuyingallthetoys,shewasgiven3morecatsfromaneighborwhowasmovingaway.Nowshedecidestobuyanothertoyforeachcatshenowowns,witheachnewtoycosting4. After buying all the toys, she was given 3 more cats from a neighbor who was moving away. Now she decides to buy another toy for each cat she now owns, with each new toy costing 5. How much did Janessa spend in total on toys? A: Let’s think step by step.

inkling-2 — 241 chars, 110 reasoning tok, stop:

Janessa starts with 3 cats.

1. Toys for original cats: 3 cats × 10 toys = 30 toys
2. Cost: 30 × $4 = $120
3. She gets 3 more cats: 3 + 3 = 6 cats
4. One new toy per cat: 6 × $5 = $30
5. Total spent: $120 + $30 = $150

Answer: $150

muse-spark-bf16 — 281 chars, 496 reasoning tok, stop:

Janessa spent **$150** in total.

Step by step:
1. Initial toys: 3 cats * 10 toys = 30 toys
2. Cost of initial toys: 30 * $4 = $120
3. Total cats after gift: 3 + 3 = 6 cats
4. New toys: 6 cats * 1 toy = 6 toys
5. Cost of new toys: 6 * $5 = $30
6. Total spent: $120 + $30 = **$150**

Both correct. Same 5-step arithmetic. muse-spark-bf16 spells out 6 cats * 1 toy = 6 toys as its own step. inkling-2 folds it into the cost line.

HumanEval — has_close_elements

Same fenced-block prompt for both: Complete the function. Reply with the full implementation inside a single ```python block.

inkling-2 — 550 chars, 846 reasoning tok, stop:

from typing import List


def has_close_elements(numbers: List[float], threshold: float) -> bool:
    """ Check if in given list of numbers, are any two numbers closer to each other than
    given threshold.
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
    True
    """
    sorted_numbers = sorted(numbers)
    for i in range(len(sorted_numbers) - 1):
        if sorted_numbers[i + 1] - sorted_numbers[i] < threshold:
            return True
    return False

muse-spark-bf16 — 601 chars, 1314 reasoning tok, stop:

from typing import List


def has_close_elements(numbers: List[float], threshold: float) -> bool:
    """ Check if in given list of numbers, are any two numbers closer to each other than
    given threshold.
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
    True
    """
    if len(numbers) < 2:
        return False
    numbers_sorted = sorted(numbers)
    for i in range(len(numbers_sorted) - 1):
        if abs(numbers_sorted[i + 1] - numbers_sorted[i]) < threshold:
            return True
    return False

Both correct. Same O(n log n) sort-then-scan. muse-spark-bf16 adds a len < 2 guard and abs() over the sorted delta. Harmless. inkling-2 is tighter.

HumanEval — separate_paren_groups

separate_paren_groups('( ) (( )) (( )( ))') -> ['( )', '(( ))', '(( )( ))']

inkling-2 — 418 visible chars at 4096 (3997 reasoning tok, length), 0 visible at 8192 (8190 reasoning tok, length):

from typing import List

def separate_paren_groups(paren_string: str) -> List[str]:
    """ Input to this function is a string containing multiple groups of nested parentheses. Your goal is to
    separate those group into separate strings and return the list of those.
    Separated groups are balanced (each open brace is properly closed) and not nested within each other
    Ignore any spaces in the input

Truncated mid-docstring. Reasoning burned the window at both budgets.

muse-spark-bf16 — 0 visible at 4096 (4093 reasoning tok, length), 0 visible at 8192 (8189 reasoning tok, length):

(empty — reasoning filled the window at both 4096 and 8192)

Same pathology. This doc overtriggers chain-of-thought on both models. It is an outlier: the 164-problem aggregate still scored 0.85 / 0.79 because most docs are short. Raising max_tokens beyond 8192 or adding a concise system prompt recovers the visible code. max_gen_toks=4096 in the harness is the bottleneck here.

sequenceDiagram participant U as User Prompt participant R as Reasoning (think) participant V as Visible (fenced block) U->>R: has_close_elements (846/1314 tok) — room left R->>V: Full function, stop U->>R: separate_paren_groups (8190/8189 tok) — no room at 8192 R->>V: Truncated / empty, length Note over V: Aggregate still 0.85 / 0.79 — most docs are short CoT

What to make of it

If you need math reasoning on ALCF, inkling-2 is the pick. 21 points on GSM8K, 6x fewer reasoning tokens on easy problems, and flex vs strict within a point — extraction is clean.

If you need code, both clear 0.78 pass@1. inkling-2 leads by 6 points but that is only 1.4σ pooled. Call it leaning inkling-2, not dominance. Style split is conciseness vs defensive guards.

Harness notes

  • Never leave generation_kwargs.until unset on generate_until chat tasks. lm-eval injects fewshot_delimiter = "\n\n" and ```python blocks die at the first blank line. Pin until: ["\n\n\n"] explicitly. The TaskConfig.__post_init__ path that does this is at lm_eval/config/task.py:142.
  • max_tokens=4096 is tight for reasoning models. Some HumanEval docs burn 4000+ tok in <think>. Even 8192 was not enough for separate_paren_groups. Cap reasoning or raise max_gen_toks if you need per-sample rollouts.
  • Flex vs strict on GSM8K barely diverged. That is a good sign for extraction.

Repro

# GSM8K (n=200)
lm-eval run --model local-chat-completions \
  --model_args "model=alcf-minerva/inkling-2,base_url=http://127.0.0.1:8765/v1/chat/completions,num_concurrent=4,timeout=600" \
  --tasks gsm8k --limit 200 --apply_chat_template \
  --output_path /tmp/lm-eval-results/inkling2-gsm8k --log_samples

# HumanEval (n=164) — humaneval_chat.yaml must have generation_kwargs.until = ["\n\n\n"]
HF_ALLOW_CODE_EVAL=1 lm-eval run --model local-chat-completions \
  --model_args "model=alcf-minerva/inkling-2,base_url=http://127.0.0.1:8765/v1/chat/completions,num_concurrent=4,timeout=600" \
  --tasks humaneval_chat --include_path /tmp/lm-eval-tasks \
  --apply_chat_template --confirm_run_unsafe_code \
  --output_path /tmp/lm-eval-results/inkling2-humaneval --log_samples

# repeat with model=alcf-minerva/muse-spark-bf16 for the other row

Raw results_*.json + samples_*.jsonl land in /tmp/lm-eval-results/ — /tmp clears on reboot, so move to outputs/ if you want them durable. Verbatim rollouts for this post at /tmp/rollout-{inkling2,muse}-{gsm8k-easy,gsm8k-hard,humaneval-0,humaneval-1}.txt.

 samf.sh / posts / 2026 / 11 / 16 · Top 1:1