Add DNG, CR2, NEF, and ARW RAW format support - #61
Conversation
|
To clarify: I don't claim to understand all the magic behind exiftool's parsing behavior but to your point, there must be some hard-earned wisdom there so I saw it best to replicate the logic. |
RAW files are structurally TIFF — the new imageDecoderRAW reuses the existing EXIF parser with SubIFD following, so all four formats get EXIF/IPTC/XMP/CONFIG support with minimal new code. CONFIG dimensions are resolved by scanning IFD0, ExifIFD, and SubIFDs for the largest image, with DNG DefaultCropSize taking priority. Implements exiftool-compatible context-dependent tag naming for strip/thumbnail offset tags (0x0111, 0x0117, 0x0201, 0x0202), which get different names depending on IFD, format, Compression value, and SubfileType. This is necessary for all four RAW formats to pass golden comparison against exiftool output. Closes #59 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #61 +/- ##
==========================================
- Coverage 78.25% 78.16% -0.10%
==========================================
Files 15 16 +1
Lines 1890 2079 +189
==========================================
+ Hits 1479 1625 +146
- Misses 279 317 +38
- Partials 132 137 +5 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
I don't understand all of it either, and I certainly don't agree with all of his choices. But there's som much odd image metadata out in the wild, so without any reference output, I'd be lost. 30 MB test data is too much. Especially considering the relatively poor test coverage in this PR (for the exif changes, that is, see above), which means that we either have 1. Too few test files or 2. Too much code. I have not thought about this too hard, but my initial thinking is that I'm not convinced about how useful metadata information about preview images is for us. |
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Pushed a follow-up commit: Removed context-dependent tag naming (~100 lines deleted from Truncated test images to metadata-only (32 MB → 275 KB). The strip/tile image data is never read by the metadata decoder, so truncating past the IFD entries is safe. Golden files are unchanged. SubIFD following (0x014a) is kept — without it, DNG/NEF/ARW report IFD0 thumbnail metadata (e.g. ImageWidth=128) instead of the main raw image. Golden test results after this commit:
Both are still exercised by |
|
Thanks, this looks great. I will wrap this up in #62. As to test data, I have thought about pulling in some extended data sets for CI builds (and development), which I guess would also make some of the licensing questions easier to reason about, but that needs to wait for another day. |
Closes #59.
Summary
Adds DNG, CR2, NEF, and ARW RAW format support. All four are TIFF containers —
imageDecoderRAWvalidates the TIFF header then hands off to the existing EXIF/IPTC/XMP decoders. All four pass golden comparison against exiftool.Context-dependent tag naming
The RAW decoder itself is straightforward. The hard part was that exiftool gives the same tag ID different names depending on IFD, Compression, and SubfileType. Without replicating this, CR2 and ARW golden tests fail.
The switch statements in
metadecoder_exif.go(lines 354–425) are derived from exiftool'slib/Image/ExifTool/Exif.pm. We implement the subset relevant to our formats:Tags 0x0201/0x0202 (exiftool has 10 condition entries, we implement 4):
dirName == "IFD1"dirName == "SubIFD"HasPrefix(dirName, "SubIFD")Tags 0x0111/0x0117 (exiftool has 7 condition entries, we implement 5):
comp == 34892comp == 52546format == CR2 && dirName == "IFD0"(DNG|TIFF) && comp == 7 && sft != 0Skipped entries are for formats we don't support (MRW, NRW, SRW, RIFF/MOV).
currentCompressionandcurrentSubfileTypeare tracked per-IFD and reset at eachdecodeTags()call — safe because TIFF requires ascending tag order, and both precede the offset tags.4 of 9 branches are exercised by golden tests (CR2, ARW, DNG, NEF). The other 5 lack test files — they require lossy DNG, JPEG XL DNG, or newer Nikon NEFs where the smallest CC0 samples are 22–94 MB. These branches are straightforward value comparisons directly translated from exiftool.
Other changes to shared code
helpers.go—convertNumbersToSpaceLimitedreturns single values as-is instead of "". Fixes single-value LONG tags (e.g. BlackLevel).gen/main.go—identifyfailures now non-fatal (ImageMagick can't read RAW without libraw delegates). Golden infra already handles missing.config.json.Test images
CC0 samples from raw.pixls.us, smallest available per manufacturer. Total ~32 MB:
Is ~32 MB of test data acceptable, or would you prefer I find smaller samples? Cropping/downsampling risks altering IFD structure.
Tests
TestDecodeRAW/TestDecodeRAWConfig— EXIF and CONFIG assertions for all four formatsgoldenSkip). "Binary data" tags skipped.