XTD/XTT Parsing: Incorrect bit layouts for normals and splat alpha channels

[coconutbird]

Summary

After comparing Foundry's XTD/XTT parsing logic against IDA disassembly of the Halo Wars DE PC binary, I found several discrepancies in bit extraction that cause incorrect data decoding.

1. Splat Alpha Channel Extraction Order (XTT)

Game's Implementation (untile_xbox360_alpha_texture @ 0x1407e34e0):

// From IDA decompilation:
*a1 = byte_1411E4B80[*(_WORD *)(a2 + 2 * v23) & 0xF];         // Channel 0: bits 0-3
a1[1] = byte_1411E4B80[(unsigned __int64)v24 >> 12];          // Channel 1: bits 12-15
a1[2] = byte_1411E4B80[(v24 >> 8) & 0xF];                     // Channel 2: bits 8-11
a1[3] = byte_1411E4B80[(unsigned __int8)v24 >> 4];            // Channel 3: bits 4-7

Foundry's Implementation (TerrainIO.cs):

int pa = (pixel >> 12) & 0xF;  // Alpha → layer 3
int pr = (pixel >> 8) & 0xF;   // Red → layer 0
int pg = (pixel >> 4) & 0xF;   // Green → layer 1
int pb = (pixel >> 0) & 0xF;   // Blue → layer 2

Comparison:

Channel Index	Game (IDA)	Foundry	Status
0	bits 0-3	bits 8-11 (Red)	❌ Wrong
1	bits 12-15	bits 4-7 (Green)	❌ Wrong
2	bits 8-11	bits 0-3 (Blue)	❌ Wrong
3	bits 4-7	bits 12-15 (Alpha)	❌ Wrong

Impact: Splat texture layers are mapped to wrong terrain areas, causing incorrect texture blending.

---

2. Normal Bit Layout (XTD)

Game's Implementation (from IDA shader constant comments):

// Bit Layout per 32-bit value:
//   X: bits 22-31 (10 bits)
//   Y: bits 11-21 (10 bits) 
//   Z: bits 0-10  (10 bits)

Foundry's Implementation (Xtd.cs):

uint x = v >> 20 & kBitMask10;  // bits 20-29
uint y = v >> 10 & kBitMask10;  // bits 10-19
uint z = v >> 0  & kBitMask10;  // bits 0-9

Comparison:

Component	Game (IDA)	Foundry	Status
X	bits 22-31	bits 20-29	❌ Off by 2
Y	bits 11-21	bits 10-19	❌ Off by 1
Z	bits 0-10	bits 0-9	✅ Close

Impact: Normals are decoded incorrectly, causing incorrect lighting on terrain.

---

Suggested Fixes

Splat Alpha (TerrainIO.cs):

// Correct channel order from game:
int ch0 = (pixel >> 0) & 0xF;   // bits 0-3
int ch1 = (pixel >> 12) & 0xF;  // bits 12-15
int ch2 = (pixel >> 8) & 0xF;   // bits 8-11
int ch3 = (pixel >> 4) & 0xF;   // bits 4-7

Normals (Xtd.cs):

uint x = (v >> 22) & kBitMask10;  // bits 22-31
uint y = (v >> 11) & kBitMask10;  // bits 11-20
uint z = v & kBitMask10;          // bits 0-9

---

Verification Method

These findings were verified by:

1. Disassembling the Halo Wars DE PC binary using IDA Pro
2. Decompiling untile_xbox360_alpha_texture (0x1407e34e0) for splat alpha
3. Analyzing BTerrainIOLoader__loadXTDInternal (0x140668db0) for position/normal layouts
4. Checking the 4-bit to 8-bit expansion LUT at 0x1411E4B80: [0x00, 0x11, 0x22, ..., 0xFF]

Happy to provide more details or IDA screenshots if helpful.