NaNs are in the background mipmaps.

[bhouston]

    // Sample framebuffer to get pixel the refracted ray hits.
    vec4 transmittedLight = getTransmissionSample(
      backgroundTexture,
      refractionCoords,
      material.specularRoughness,
      ior
    );
    DEBUG_OUTPUT(39, transmittedLight.rgb);

The NaNs must be coming from the mipmap.