Skip to content

Add DNG, CR2, NEF, and ARW RAW format support - #61

Closed
tonimelisma wants to merge 2 commits into
bep:mainfrom
tonimelisma:raw-format-support
Closed

Add DNG, CR2, NEF, and ARW RAW format support#61
tonimelisma wants to merge 2 commits into
bep:mainfrom
tonimelisma:raw-format-support

Conversation

@tonimelisma

Copy link
Copy Markdown
Contributor

Closes #59.

Summary

Adds DNG, CR2, NEF, and ARW RAW format support. All four are TIFF containers — imageDecoderRAW validates the TIFF header then hands off to the existing EXIF/IPTC/XMP decoders. All four pass golden comparison against exiftool.

Context-dependent tag naming

The RAW decoder itself is straightforward. The hard part was that exiftool gives the same tag ID different names depending on IFD, Compression, and SubfileType. Without replicating this, CR2 and ARW golden tests fail.

The switch statements in metadecoder_exif.go (lines 354–425) are derived from exiftool's lib/Image/ExifTool/Exif.pm. We implement the subset relevant to our formats:

Tags 0x0201/0x0202 (exiftool has 10 condition entries, we implement 4):

Our condition Exiftool equivalent Result
dirName == "IFD1" Entry 1 ThumbnailOffset/Length
dirName == "SubIFD" Entry 6 JpgFromRawStart/Length
HasPrefix(dirName, "SubIFD") Entries 8–9 OtherImageStart/Length
default (IFD0, etc.) Entry 10 (fallback) PreviewImageStart/Length

Tags 0x0111/0x0117 (exiftool has 7 condition entries, we implement 5):

Our condition Exiftool equivalent Result
comp == 34892 Entry 2 OtherImageStart/Length
comp == 52546 Entry 3 PreviewJXLStart/Length
format == CR2 && dirName == "IFD0" Entry 5 PreviewImageStart/Length
(DNG|TIFF) && comp == 7 && sft != 0 Entry 6 PreviewImageStart/Length
default Entry 4 StripOffsets/ByteCounts

Skipped entries are for formats we don't support (MRW, NRW, SRW, RIFF/MOV). currentCompression and currentSubfileType are tracked per-IFD and reset at each decodeTags() call — safe because TIFF requires ascending tag order, and both precede the offset tags.

4 of 9 branches are exercised by golden tests (CR2, ARW, DNG, NEF). The other 5 lack test files — they require lossy DNG, JPEG XL DNG, or newer Nikon NEFs where the smallest CC0 samples are 22–94 MB. These branches are straightforward value comparisons directly translated from exiftool.

Other changes to shared code

  • helpers.goconvertNumbersToSpaceLimited returns single values as-is instead of "". Fixes single-value LONG tags (e.g. BlackLevel).
  • gen/main.goidentify failures now non-fatal (ImageMagick can't read RAW without libraw delegates). Golden infra already handles missing .config.json.

Test images

CC0 samples from raw.pixls.us, smallest available per manufacturer. Total ~32 MB:

File Camera Size
sample.dng Canon PowerShot SD450 6.3 MB
sample.cr2 Canon PowerShot S90 13 MB
sample.nef Nikon D1 3.9 MB
sample.arw Sony DSLR-A330 9.2 MB

Is ~32 MB of test data acceptable, or would you prefer I find smaller samples? Cropping/downsampling risks altering IFD structure.

Tests

  • TestDecodeRAW / TestDecodeRAWConfig — EXIF and CONFIG assertions for all four formats
  • Golden comparison passes for all four (none in goldenSkip). "Binary data" tags skipped.
  • Fuzz tests and benchmarks for all four formats

@tonimelisma

Copy link
Copy Markdown
Contributor Author

To clarify: I don't claim to understand all the magic behind exiftool's parsing behavior but to your point, there must be some hard-earned wisdom there so I saw it best to replicate the logic.

RAW files are structurally TIFF — the new imageDecoderRAW reuses the
existing EXIF parser with SubIFD following, so all four formats get
EXIF/IPTC/XMP/CONFIG support with minimal new code.

CONFIG dimensions are resolved by scanning IFD0, ExifIFD, and SubIFDs
for the largest image, with DNG DefaultCropSize taking priority.

Implements exiftool-compatible context-dependent tag naming for
strip/thumbnail offset tags (0x0111, 0x0117, 0x0201, 0x0202), which
get different names depending on IFD, format, Compression value, and
SubfileType. This is necessary for all four RAW formats to pass golden
comparison against exiftool output.

Closes #59

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@codecov-commenter

codecov-commenter commented Feb 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 76.26263% with 47 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.16%. Comparing base (69db29f) to head (ea79203).

Files with missing lines Patch % Lines
imagedecoder_raw.go 77.96% 35 Missing and 4 partials ⚠️
metadecoder_exif.go 53.84% 4 Missing and 2 partials ⚠️
gen/main.go 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #61      +/-   ##
==========================================
- Coverage   78.25%   78.16%   -0.10%     
==========================================
  Files          15       16       +1     
  Lines        1890     2079     +189     
==========================================
+ Hits         1479     1625     +146     
- Misses        279      317      +38     
- Partials      132      137       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bep

bep commented Feb 23, 2026

Copy link
Copy Markdown
Owner

To clarify: I don't claim to understand all the magic behind exiftool's parsing behavior but to your point, there must be some hard-earned wisdom there so I saw it best to replicate the logic.

I don't understand all of it either, and I certainly don't agree with all of his choices. But there's som much odd image metadata out in the wild, so without any reference output, I'd be lost.

30 MB test data is too much. Especially considering the relatively poor test coverage in this PR (for the exif changes, that is, see above), which means that we either have 1. Too few test files or 2. Too much code. I have not thought about this too hard, but my initial thinking is that I'm not convinced about how useful metadata information about preview images is for us.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@tonimelisma

Copy link
Copy Markdown
Contributor Author

Pushed a follow-up commit:

Removed context-dependent tag naming (~100 lines deleted from metadecoder_exif.go). The switch blocks that renamed tags based on IFD context / compression / format are gone, along with the currentCompression/currentSubfileType state tracking. Tags 0x0201/0x0202 now always use their default TIFF names (ThumbnailOffset/ThumbnailLength) and 0x0111/0x0117 always use StripOffsets/StripByteCounts. Patch coverage on decodeTag goes from 55% to 91%.

Truncated test images to metadata-only (32 MB → 275 KB). The strip/tile image data is never read by the metadata decoder, so truncating past the IFD entries is safe. Golden files are unchanged.

SubIFD following (0x014a) is kept — without it, DNG/NEF/ARW report IFD0 thumbnail metadata (e.g. ImageWidth=128) instead of the main raw image.

Golden test results after this commit:

  • DNG and NEF pass golden comparison (not in goldenSkip).
  • CR2 is in goldenSkip — it chains 4 IFDs via next-IFD pointers (not SubIFDs), so we only reach IFD0/IFD1.
  • ARW is in goldenSkip — exiftool renames IFD0's 0x0201/0x0202 to PreviewImageStart/Length; our default names collide with IFD1's ThumbnailLength.

Both are still exercised by TestDecodeRAW and TestDecodeRAWConfig.

@bep

bep commented Feb 26, 2026

Copy link
Copy Markdown
Owner

Thanks, this looks great. I will wrap this up in #62.

As to test data, I have thought about pulling in some extended data sets for CI builds (and development), which I guess would also make some of the licensing questions easier to reason about, but that needs to wait for another day.

@bep bep closed this Feb 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add TIFF-based camera RAW metadata support (DNG, CR2, NEF, ARW)

3 participants