ArrowBuffer allocates a multiple of 64 plus 8 bytes.

[willtemperley]

Describe the enhancement requested

In ArrowBuffer, alignTo64 always aligns to a multiple of 64 plus 8. I just can't think of a reason for this. The naming alignTo64 seems to point ot this being an oversight.

private static func alignTo64(_ length: UInt) -> UInt {
    let bufAlignment = length % 64
    if bufAlignment != 0 {
        return length + (64 - bufAlignment) + 8
    }

    return length + 8
}

Oddly this behaviour is tested for which indicates it might have been intentional but I really can't see a reason.

    func testPrimitiveArray() throws {
.
        let arrayBuilder: NumberArrayBuilder<UInt8> = try ArrowArrayBuilders.loadNumberArrayBuilder()
        for index in 0..<100 {
            arrayBuilder.append(UInt8(index))
        }

        XCTAssertEqual(arrayBuilder.nullCount, 0)
        arrayBuilder.append(nil)
        XCTAssertEqual(arrayBuilder.length, 101)
        XCTAssertEqual(arrayBuilder.capacity, 136) // Why not 128??
   }

[maltzsama]

Hey @willtemperley, great catch!

You are completely right. There is no basis in the Apache Arrow memory specification for this arbitrary +8 byte overhead. It seems like a legacy oversight that slipped through and was mistakenly codified into the unit tests. In datasets with a large number of columns or frequent small buffer allocations, this undocumented padding just leads to unnecessary memory bloat.

I have just opened a PR to remove this padding and strictly enforce the 64-byte alignment. I also updated the testPrimitiveArray unit test you pointed out so it now correctly asserts a capacity of 128 instead of 136.
PR: #144
Thanks for documenting this!