时空域上下文学习的视频多帧质量增强方法

卷积神经网络（CNN）在视频增强方向取得了巨大的成功。现有的视频增强方法主要在空域探索图像内像素的相关性，忽略了连续帧之间的时域相似性。针对上述问题，提出一种基于时空域上下文学习的多帧质量增强方法（STMVE），即利用当前帧以及相邻多帧图像共同增强当前帧的质量。首先根据时域多帧图像直接预测得到当前帧的预测帧，然后利用预测帧对当前帧进行增强。其中，预测帧通过自适应可分离的卷积神经网络（ASCNN）得到；在后续增强中，设计了一种多帧卷积神经网络（MFCNN），利用早期融合架构来挖掘当前帧及其预测帧的时空域相关性，最终得到增强的当前帧。实验结果表明，所提出的STMVE方法在量化参数值37、32、27、22上，相对于H.265/HEVC，分别获得0.47、0.43、0.38、0.28 dB的性能增益；与多帧质量增强（MFQE）方法相比，平均获得0.17 dB的增益。 

Video multi-frame quality enhancement method via spatial-temporal context learning

Convolutional neural network (CNN) has achieved great success in the field of video enhancement. The existing video enhancement methods mainly explore the pixel correlations in spatial domain of an image, which ignores the temporal similarity between consecutive frames. To address the above issue, this paper proposes a multi-frame quality enhancement method, namely spatial-temporal multi-frame video enhancement (STMVE), through learning the spatial-temporal context of current frame. The basic idea of STMVE is utilizing the adjacent frames of current frame to help enhance the quality of current frame. To this end, the virtual frames of current frame are first predicted from its neighbouring frames and then current frame is enhanced by its virtual frames. And the adaptive separable convolutional neural network (ASCNN) is employed to generate the virtual frame. In the subsequent enhancement stage, a multi-frame CNN (MFCNN) is designed. An early-fusion CNN structure is developed to extract both temporal and spatial correlation between the current and virtual frames and output the enhanced current frame. The experimental results show that the proposed STMVE method obtains 0.47 dB, 0.43 dB, 0.38 dB and 0.28 dB PSNR gains compared with H.265/HEVC at quantized parameter values 37, 32, 27 and 22 respectively. Compared to the multi-frame quality enhancement (MFQE) method, an average 0.17 dB PSNR gain is obtained.