145 lines
10 KiB
Markdown
145 lines
10 KiB
Markdown
# "Mountain 3 crash" (speedrun community) – investigation 2026-11-08
|
||
|
||
Short version: **Mountain 3 = `t1l5m003`** (world 5, chapter 3). Its boulder spawner
|
||
(`RollingBoulderMountain`) can end while a boulder it spawned is still alive. When that boulder
|
||
finishes its path, it decrements `parent.v13` of the dead spawner. That write
|
||
(`-= 1.0`, i.e. `-0x1000`) goes into freed or reused memory (use after free). If it hits a header of
|
||
the game's local heap, a later `FreeLocalMem` crashes. That matches the community crash dump.
|
||
The stale write is **verified in game**. A full crash was **not** reproduced. A one-word level-data
|
||
patch removes the stale write (verified in game).
|
||
|
||
## Evidence from the community
|
||
|
||
* DebugDiag dump (English exe, 2000-10-17): access violation **writing** `0x0000340d` at
|
||
`groove+0x1ad52` = VA `0x41AD52`, thread 0, Argonaut name `FreeLocalMem`.
|
||
* English `0x41AD10..0x41AD89` is byte-identical to German **`FUN_0041ada0`** (only the rel32 of
|
||
the "ITS NOT USED! %08x" debug printf differs). Crash instruction = German `0x41ADE2`
|
||
`mov [eax+4], esi`.
|
||
* The local heap (`FUN_0041af00` init, `FUN_0041ae20` alloc, `FUN_0041ada0` free) holds blocks with a
|
||
12-byte header `{next, prev, u16 size (dwords), u16 used}`. The blocks are contiguous
|
||
(`next == hdr + (size+3)*4`). On free, the routine merges with the **next** block if
|
||
`next.used == 0`, then does `next->next->prev = hdr`. The crash means the header after the
|
||
freed block had `used == 0` and `next == 0x3409`. 0x3409 is not 4-aligned, so it can't be a real
|
||
block pointer. **The heap metadata was corrupted earlier, and the free only shows it.**
|
||
* Clips: index.mp4 is in a snowy canyon with a pushable boulder (red X), wall shards
|
||
(`SmashWallShrapnel`), a ruby-idol pickup (HUD), and a crash at the cave about 6 s later.
|
||
index2.mp4 is the Kronk ice race (`KronkOnIceDoor`/`KronkOnIce`, same WAD): black screen about
|
||
2 s after the race starts. Both are in `t1l5m003`. Our boot to (74.6, 14.8, 6.3) shows the same canyon
|
||
and boulder as clip 1, so clip 1 is most likely the `#47` push-boulder section (hypothesis,
|
||
based on what the scenes look like).
|
||
* Side note: the heap is `operator_new(0x200000)` but is initialised with size `0x800000` into a u16
|
||
field, so it is really 0xFFFD dwords (≈256 KB). Mountain 3 uses about 88 KB of it, so the heap
|
||
is not exhausted. No leak was seen: block count stayed stable over minutes.
|
||
|
||
## The bug (strat code of unit 1 of t1l5m003, verified by disassembly)
|
||
|
||
`RollingBoulderMountain` (6 MAP instances, params v0..v9). `v13` is set to 0 and never
|
||
incremented anywhere. The only other access is the boulder's `parent.v13 -= 1.0`
|
||
(`pushParentVarAddr v13; dec`) in `L_0574`, which runs when the boulder ends its path or falls:
|
||
|
||
```
|
||
spawner: ... spawn loop (L_0c24: spawn<7,31,4,0x14c000,0>(…RollingBoulderForMountain)) ...
|
||
freezeOffCamera 1
|
||
L_0d90: pushVar v13 / pushShort 0 / eq / jzYield L_0d90 / end ; ends at once when v13 == 0
|
||
boulder: L_0574: parent.v13 -= 1.0 ; ... ; remove
|
||
```
|
||
|
||
* With `v1 == 0` (instances #17 and #35) or `v4 != 0` (#22 and #47, the pushable "red X" boulders,
|
||
which wait to be charged), the spawner spawns **one** boulder and **ends in the same frame**
|
||
(v13 is still 0).
|
||
* Instances #1 and #27 (`v1` = 8.3 / 8.75) loop: a boulder about every 8–10 s, until the player
|
||
is near the path start. They end the same way if no boulder has died yet when the loop exits.
|
||
* Strat removal (strat update loop `FUN_0054c4a0`) frees the var block
|
||
(`+0xF4`) and the collision block (`+0xC0`) and puts the struct on the free list (LIFO). It does
|
||
**not** clear the children's `+0xC` parent pointers. The parent opcodes (`0x553750`/`0x553770`/
|
||
`0x553a40`) have no check.
|
||
* So the boulder's write goes to `deadSpawner.vars + 0x34`: either memory of the freed var block
|
||
(now part of a free block or of another allocation, possibly a **block header**), or, if the
|
||
struct was already reused, `v13` of another strat. For a boulder, `v13` is its sound address
|
||
(`v13 = sound:4`, used by `playSoundChannel`).
|
||
* "Predictable intervals" (runner's guess) fits: the stale writes happen a fixed time after you
|
||
pass a spawner, or after you push a boulder, when that boulder reaches the end of its path.
|
||
Whether a write hurts depends on the heap layout at that moment, so it depends on timing.
|
||
It is not deterministic across runs.
|
||
|
||
## In-game verification (apptest, Xvfb :26, sessions `mtn3-*`)
|
||
|
||
Tooling (throwaway, see `notes/dev/mountain3-diag/`): a test-only mod `Mtn3DiagMod` (loaded through
|
||
`APPTEST_CP` + `-Dllamamod.extraMods`). It installs a code cave (`cave.asm`, nasm) that replaces the
|
||
`pushParentVarAddr` and `popParentVar` handlers. They log every access, flag **stale** ones
|
||
(parent `+0xEC == 0`, i.e. removed) together with the block that the target address falls in, and
|
||
walk the whole heap on every alloc and free (`next/prev/used` consistency).
|
||
`rd.py` reads the log by attaching `winedbg` for a moment (stdin script `attach`/`x`/`detach`;
|
||
works under ptrace_scope 1). `apptest.py` in the agent copy got an env switch
|
||
`APPTEST_IGNORE_WINEDBG` so the short attach is not taken for a crash (not copied back).
|
||
A gdb-over-winedbg monitor with breakpoints was too slow: the game froze.
|
||
|
||
Boot-to-level `start=29.5, 14, 23.2, 180 @ 5/3` (next to spawners #1 and #17), no input, 70–85 s:
|
||
|
||
| | runs | runs with a stale write | heap anomalies | crash |
|
||
|---|---|---|---|---|
|
||
| original WAD | 6 (1 manual + 5) | 5 | 0 | 0 |
|
||
| level fix (below) | 3 | 0 | 0 | 0 |
|
||
|
||
Every stale record was the same and deterministic for this start (logged by the cave):
|
||
`RollingBoulderForMountain` → parent `RollingBoulderMountain` (struct `+0xEC == 0`, removed), `v13`,
|
||
target = `freed var block + 0x34`. At the time of the write this address was inside a **used
|
||
16-dword block** (a `0x40000` collision block, most likely the next boulder's), at offset +0x40 from its
|
||
header, which is 12 bytes before the next block header. In this layout it corrupts collision data
|
||
(u16 fields `+0x34/+0x36` become 0xF000/0xFFFF), not the heap. Any other layout (other timing,
|
||
other route, other spawns in between, e.g. an 11-dword allocation such as a `0x28000` pickup's
|
||
collision block puts a remainder header's size/used word exactly there) turns the same write into
|
||
header corruption. **Not verified:** a crash caused by this write. No natural run crashed, and the
|
||
real runner route (from the chapter start at full speed, pushing #22/#47, then the race) could not
|
||
be driven reliably by key scripts (the push needs speed and alignment; several attempts missed).
|
||
Also seen in game: strat structs are recycled LIFO (the dead spawner's struct was later reused by a
|
||
boulder).
|
||
|
||
Other checks in this WAD: every `collisionOn` user is spawned or placed with collision type flags
|
||
(no `typeFlags 0 + collisionOn` case like in the arena), and all `extra` operands are 0. Other
|
||
parent-variable writers (`KronkOnIcePickup` → `parent.v7`, `CheckPointHead`, `RubyIdolStatue`,
|
||
`AxeGuardAxe`) have parents that do not end while they live, as far as the code shows. That was
|
||
checked by reading the code only; the cave saw no stale access from them.
|
||
|
||
Side finding (not part of the crash): boot-to-level `start=28.5, 10.5, 22.5, 0 @ 5/3` passes the
|
||
floor check, but the game crashes while loading (`page fault read 0x00000002 at 0x405527`, floor/
|
||
tile lookup in `FUN_004054c0`). Out of scope here.
|
||
|
||
## Fix / avoidance
|
||
|
||
* **Level data (tested):** in `RollingBoulderMountain`, change `pushShort 0` before `eq` to
|
||
`pushShort 0.5` (word `0x00000044` → `0x08000044`, the only match of the 6-word pattern
|
||
`0x1002f 0xd0004 0x44 0x4e 0xfffc0012 0xb4`, file offset 0x4a0a27 in the German
|
||
`t1l5m003.wad`). `v13` only takes whole numbers (0, -1, -2 …), so the spawner never ends. It
|
||
stays frozen off camera and costs nothing, and every boulder writes into a live spawner. The
|
||
boulders still roll (in-game: 6–7 normal parent writes per run, 0 stale). `notes/dev/mountain3-diag/wadfix.py IN OUT`.
|
||
The English CD's WAD is not checked; the patch looks for the pattern and refuses if it isn't unique.
|
||
* **Exe (general, untested):** let removal (`+0xE8 & 4` path) clear `+0xC` of all strats whose
|
||
parent is the removed one (pool `DAT_006d9dc0`, stride 0x154), and let the parent opcodes
|
||
redirect a NULL or removed parent to a dummy dword. This also covers similar bugs in other levels.
|
||
* **Runners (hypothesis):** there's no reliable avoidance. The write happens when the boulders of
|
||
#17 and #35 (spawned when you pass their trigger spots) and the pushed boulders #22 and #47 reach
|
||
the end of their path. Not pushing #22/#47 removes two of the four guaranteed stale writes, if
|
||
the route allows that.
|
||
|
||
Diag tooling notes: the files in `notes/dev/mountain3-diag/` are as used, with absolute paths
|
||
of the agent job dir. The mod writes `addresses.txt` and expects `cave_<fix>_<CAVE>_<DATA>.bin`
|
||
(`nasm -f bin -DCAVE=0x6ef0e0 -DDATA=0x70f010 -DFIX=0 cave.asm`; these addresses are valid with
|
||
boot-to-level + mtn3-diag on the German exe). This is only for the German exe; nothing is registered in the app.
|
||
|
||
## Discord answer (draft)
|
||
|
||
> We looked into the Mountain 3 crash. Your dump crashes in the game's own memory manager
|
||
> (`FreeLocalMem`, writing to 0x340d). That is the symptom: its bookkeeping was overwritten
|
||
> earlier. **Verified:** in Mountain 3 the rolling-boulder spawner can delete itself while
|
||
> its boulder is still rolling. When that boulder reaches the end of its path (or falls off), it
|
||
> writes into the spawner's memory, which has already been freed and reused. We saw this
|
||
> write in the game with a debug build, and a one-value level patch makes it go away.
|
||
> **Not verified yet:** that this exact write causes your crash. We never got the game to
|
||
> crash. Where the write lands depends on what else is in memory at that moment, so it
|
||
> breaks things only sometimes, and the crash can come seconds later somewhere else
|
||
> (e.g. in the Kronk race). That would explain why it's Mountain 3 only, at different spots,
|
||
> without an error message, and "every so often" around boulders. The likely triggers are the
|
||
> one-shot boulders near the start/top of the slope and the two red-X push boulders. We'll
|
||
> probably ship the fix in LlamaMod.
|