低级视觉任务与高级视觉任务结合

[meton-robean]

总概

1. high low level task 结合可以分为两种：1)利用高级任务去辅助低级任务 2）研究低级任务输出如何更好服务后续高级任务

---

Segmentation-Aware Image Denoising without Knowing True Segmentation
论文阅读记录 2019 prprint

这篇论文利用语义网络信息来辅助去噪，但是语意网路那边不借助标签监督信息，这区别与传统的联合训练方法)

On the other hand, in the metric of PSNR,
a model trained by minimizing MSE on the image domain
should always outperform a model trained by minimizing a
hybrid weighted loss. Therefore, we emphasize that the goal of
our following experiments is not to pursue the highest PSNR,
but to quantitatively demonstrate the different behaviors between
models with and without segmentation awareness.

论文中这段话说明了 采用了joint training 后 psnr大部分时候比不过直接使用MSE训练的去噪模型
从实验表格看确实如此，说明了high-level和low-level task之间的gap. 但是差距似乎不大在论文所提供的实验中. (CDN-CNN-B是直接训练的去噪模型， 其余两个是采用了两种不同方式的联合训练的模型：

[meton-robean]

Zhiwen Fan, Liyan Sun, Xinghao Ding, Yue Huang, Congbo Cai, and
John Paisley. A segmentation-aware deep fusion network for compressed
sensing mri. arXiv preprint arXiv:1804.01210, 2018.

这篇是预训练的语义分割网络帮 医学图像处理任务
做法是采用了fixed 的pretrianed脑图语义分割模型，提供一些中间feature信息concat到要解决的医学影片重构任务的网络中， 去帮组它获得更好的重构性能。 本质上 应该属于 high-level辅助 low-level task.

[meton-robean]

[20]Ding Liu, Bihan Wen, Xianming Liu, Zhangyang Wang, and Thomas S
Huang. When image denoising meets high-level vision tasks: A deep
learning approach. arXiv preprint arXiv:1706.04284, 2017.
后来中了IJCAI2018

这篇是去噪和高级视觉任务网络联合训练

First we initialize the network for high�level vision task from a network that is well-trained in the
noiseless setting. We train the cascade of two networks in an
end-to-end manner while fixing the weights in the network
for high-level vision task.

such a training strategy is to make
the trained denoising network robust enough without losing
the generality for various high-level vision tasks.

其中高级网络的参数是固定的，但是利用了高级网络的featrue maps作感知损失和高级网络原本的监督信息 交叉熵loss 以及去噪的loss， 最后参数更新只更新去噪网络的。 这样做的原因是增加泛化

他们用了高级网络之后，发现psnr那些也降了，然后他们的理由是：


他们就说不是最求高的psnr而是探究两者的影响，然后后面他们贴了图说高级网络对去噪有帮组，是从视觉效果来说的， 他们贴的图都是在说视觉效果人眼看起来更好，然后他们主观打分，说联合训练后数据集中的去噪视觉效果更好了。 但是实际上他们的psnr打不过

[meton-robean]

AOD-Net: All-in-One Dehazing Network
In Proceedings of the IEEE
International Conference on Computer Vision, volume 1, page 7, 2017

一个CNN dehaze 网络，顺便探究了dehaze对图像检测的影响。 b是baseline + object dect without joint training; c 是 作者网络 + object det wthout JT; d 是作者网络+目标检测 联合训练。 从而说明联合训练可以提高目标检测，但是d可以看出去雾效果没有那么好，但是目标检测效果却最好，说明联合训练一定程度影响low-level task性能

[meton-robean]

Segmentation-Aware Convolutional Networks Using Local Attention Masks ICCV2017

This work introduces Segmentation-Aware Convolutional
Networks, a direct generalization of standard CNNs
that allows us to seamlessly accommodate segmentation information
throughout a deep architecture. Our approach
avoids feature blurring before it happens, rather than fixing
it post-hoc. The full architecture can be trained end-to end.

作者提出将分割网络的信息嵌入到其他任务的网络中的方法，从而可以减少输出结果的平滑性。 本质上是一种end2end的联合训练。

[meton-robean]

On the Relation between Color Image Denoising and Classification
2016
这篇文章在denoise网络结构后面接上目标检测的网络结构做end2end训练. （文中说是two-stage,其实是end2end, 通过这样最终获得了更好的去噪效果，本质上是利用high-level辅助low-level.
最后拓展实验发现某些经过去噪的图片再做目标检测，反而没有直接带噪识别的效果好，作者没有给出原因，只是发现这个现象。 但是我们可以推测，去噪一定程度上可能损失高级语义信息，导致这种现象？

First we convolve the color image
with high pass filters to capture the high frequencies and apply
our residual CNN model to obtain the intermediate result.
In the second stage, we adapt either AlexNet [14] or VGG16
[23] deep architecture from image classification and stack it
on top of our first stage model, along with introducing a novel
cost function inspired by Sobolev space equipped norm [1].
As the experiments will show, our proposed method overcomes
the regress-to-mean problem and enables the recovery of high
frequency details better than other denoising works.

Inspired by [10], we fully take advantage of sophisticated CNN architectures
proposed for image classification and adapt them
on top of our 3DR model as a second stage. We adapt
AlexNet [14] and VGG-16 [23] which are two widely used
CNN architectures. We intend to have an end-to-end model
and an output denoised image with the same size as the input
image.

[meton-robean]

Single Image Deraining: A Comprehensive Benchmark Analysis CVPR2019

这篇关注的是现有的去雨 state-of-the-arts 模型能不能真的帮到高级任务，文中用的是目标检测任务。结果发现： 去雨模型作为直接独立作为目标检测的前序，反而可能会恶化目标检测结果。 他们的结果说明了low, high task的gap.

Perhaps surprisingly at the first glance, we find that almost
all existing deraining algorithms will deteriorate
the detection performance compared to directly using
the rainy images for YOLO-V3, SSD-512, and RetinaNet.
Our observation concurs the conclusion of another
recent study (on dehazing) [51]: since those deraining algorithms
were not trained/optimized towards the end goal of
object detection, they are unnecessary to help this goal, and
the deraining process itself might have lost discriminative,
semantically meaningful true information

[meton-robean]

I Can See Clearly Now : Image Restoration via De-Raining

这篇也是直接将去雨模型作为独立的前序来去雨，然后去雨结果用于后续分割任务; 对比的对象是有雨无雨增广的处理。 这里显示前者可以改善情况，和前一篇Single Image Deraining: A Comprehensive Benchmark Analysis CVPR2019 结论似乎不太一致。

[meton-robean]

A Convolutional Network for Joint Deraining and Dehazing from A
Single Image for Autonomous Driving in Rain
IROS 2019

P了cityscape的数据集（rain streak + frog）
设计了可以去雨和去雾的网络，专门针对自动驾驶，道路场景的去雨
将去雨结果在目标检测和图像分割上看结果了，有一定效果。 没有使用 联合训练

[meton-robean]

Task-driven super resolution: Object detection in low-resolution images

也是将超分和目标检测的LOss 联合训练，来使得超分的结果真的有易于后续任务。

这篇有明确数据和文字表明提高高级任务LOSS在联合loss的比重之后，目标检测mAP提高了，但是PSNR显著下降。

As the relative value of β becomes larger, we get additional improvements, but at the cost of a
significant decline in PSNR (and as we see in Fig. 2 and in Section 4.4, in visual quality).

作者调节low-level loss和high-level loss比重，发现在一个比例下，这种联合训练的可以比纯high-level训练更加mAP高，但是PSNR还是会有些降低的，这是确定事实。作者最后得出结论：

Finally, the numbers in the table further illustrate that higher PSNR must not correspond to better
detection results

---

作者还探究了直接使用传统的超分model的输出用来做目标检测，对比使用联合训练的：

We see that reduction in resolution has a drastic effect on the mAP of the
detector, dropping it from 75.8 to 41.7 for 4× and 16.6 for 8×. This is presumably
due to both the actual loss of information, and the limitations of the
detector architecture which may miss small bounding boxes. The performance is
not significantly improved by non-task-driven SR methods, which in some cases
actually harm it further! However, our proposed TDSR approach obtains significantly
better results for both scaling factors, and recovers a significant fraction
of the detection accuracy lost in LR