7.8 遥感图像场景分类可视化解释

7.8.1 场景分类训练过程

本节为遥感图像场景分类可视化。

示例任务为airplane、storagetanks及tenniscourt三个场景的分类任务。

场景分类训练过程示例代码点此下载。

7.8.2 分类过程可视化

加载可视化所需模块：

# Boilerplate imports.
import tensorflow.compat.v1 as tf
import numpy as np
import PIL.Image
from matplotlib import pylab as P
import pickle
import os
import tf_slim as slim
import inception_v3
import saliency
from mpl_toolkits.axes_grid1 import make_axes_locatable
import matplotlib.pyplot as plt
import config as cfg
import resnet
import cv2 as cv

WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

定义可视化所需函数：

# Boilerplate methods.
def ShowImage(im, title='', ax=None):
    if ax is None:
        P.figure()
    P.axis('off')
    # im = ((im + 1) * 127.5).astype(np.uint8)
    P.imshow(im)
    P.title(title)

def ShowGrayscaleImage(im, title='', ax=None):
    if ax is None:
        P.figure()
    P.axis('off')

    P.imshow(im, cmap=P.cm.gray, vmin=0, vmax=1)
    P.title(title)

def ShowHeatMap(im, title, ax=None):
    if ax is None:
        P.figure()
    P.axis('off')
    P.imshow(im, cmap='inferno')
    P.title(title)

def ShowDivergingImage(grad, title='', percentile=99, ax=None):
    if ax is None:
        fig, ax = P.subplots()
    else:
        fig = ax.figure

    P.axis('off')
    divider = make_axes_locatable(ax)
    cax = divider.append_axes('right', size='5%', pad=0.05)
    im = ax.imshow(grad, cmap=P.cm.coolwarm, vmin=-1, vmax=1)
    fig.colorbar(im, cax=cax, orientation='vertical')
    P.title(title)

def LoadImage(file_path):
    im = PIL.Image.open(file_path)
    im = np.asarray(im)
    return im / 127.5 - 1.0

加载训练好的场景分类模型：

graph = tf.Graph()

with graph.as_default():


    input_img = tf.placeholder(tf.float32,[None, cfg.image_height, cfg.image_width, 3])
    cls_score = resnet.resnet_base(input_img, scope_name=cfg.NET_NAME, is_training=False)
    init = tf.global_variables_initializer()
    output_dir = r'F:\learning\interpretation\Tensorflow-Resnet-Image-Classification-master\output_2'
    ckpt_file = tf.train.latest_checkpoint(output_dir)
    neuron_selector = tf.placeholder(tf.int32)
    y = cls_score[0][neuron_selector]
    prediction = tf.argmax(cls_score, 1)

    # with tf.Session(graph=graph) as sess:

    sess = tf.Session(graph=graph)
    saver = tf.train.Saver()
    saver.restore(sess, ckpt_file)

    variables = tf.global_variables()
    saver = tf.train.Saver(variables)
    init = tf.global_variables_initializer()

    sess.run(init)
    saver.restore(sess, ckpt_file)

WARNING:tensorflow:From F:\learning\interpretation\history-of-interpretation-master\resnet.py:25: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

WARNING:tensorflow:From F:\learning\interpretation\history-of-interpretation-master\resnet.py:61: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:From D:\anaconda\lib\site-packages\tensorflow_core\contrib\layers\python\layers\layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
INFO:tensorflow:Restoring parameters from F:\learning\interpretation\Tensorflow-Resnet-Image-Classification-master\output_2\output.model-2500
INFO:tensorflow:Restoring parameters from F:\learning\interpretation\Tensorflow-Resnet-Image-Classification-master\output_2\output.model-2500

使用加载的模型进行预测：

imagepath = r'F:\learning\interpretation\Tensorflow-Resnet-Image-Classification-master\UCMerced_LandUse\1\train\airplane\airplane35.tif'
test_image_single_1 = cv.imdecode(np.fromfile(imagepath, dtype=np.uint8), 1)
test_image_single = cv.resize(test_image_single_1, (cfg.image_height, cfg.image_width))
test_image_single = np.reshape(test_image_single, [1, cfg.image_height, cfg.image_width, 3])
im = np.reshape(test_image_single_1, [cfg.image_height, cfg.image_width, 3])
test_image_single.astype(np.float32)
cls_score_1 = sess.run(cls_score, feed_dict={input_img: test_image_single})

分类过程可视化方法:

示例一：

Vanilla Gradients(参考文献)：

论文介绍了CNN各层到底学到了原始图像的什么特征以及这些特征对最终预测结果的影响力度。该方法通过使用梯度上升方法来探索深度卷积神经网络内部的分类过程，通过计算每个类的预测概率相对于输入图像的梯度来对图像分类模型进行可视化。

在可视化结果中，每个像素为分类概率最大类别的预测概率相对于输入图像的梯度。

SmoothGrad(参考文献)：

与上述方法一样，此方法仍旧通过计算类别预测概率相对于输入图像的梯度进行图像分类过程的可视化。

但SmoothGrad方法通过在输入图像中添加噪声，然后对各个带有噪声的输入图像分别计算梯度，从而在视觉上锐化这些基于梯度的可视化结果。最后将所有可视化结果进行平均即可得到更清晰的结果。

# Construct the saliency object. This doesn't yet compute the saliency mask, it just sets up the necessary ops.
gradient_saliency = saliency.GradientSaliency(graph, sess, y, input_img)

# Compute the vanilla mask and the smoothed mask.
vanilla_mask_3d = gradient_saliency.GetMask(test_image_single[0], feed_dict = {neuron_selector: 1})
smoothgrad_mask_3d = gradient_saliency.GetSmoothedMask(test_image_single[0], feed_dict = {neuron_selector: 1})

# Call the visualization methods to convert the 3D tensors to 2D grayscale.
vanilla_mask_grayscale = saliency.VisualizeImageGrayscale(vanilla_mask_3d)
smoothgrad_mask_grayscale = saliency.VisualizeImageGrayscale(smoothgrad_mask_3d)

# Set up matplot lib figures.
ROWS = 1
COLS = 2
UPSCALE_FACTOR = 10
P.figure(figsize=(ROWS * UPSCALE_FACTOR, COLS * UPSCALE_FACTOR))

# Render the saliency masks.
ShowGrayscaleImage(vanilla_mask_grayscale, title='Vanilla Gradient', ax=P.subplot(ROWS, COLS, 1))
ShowGrayscaleImage(smoothgrad_mask_grayscale, title='SmoothGrad', ax=P.subplot(ROWS, COLS, 2))

示例二：

Guided Backpropogation（参考文献）：

在上述卷积神经网络可视化方法中，使用的网络结构以conv+ReLU+Maxpooling为主要实现方式，该论文中提出了一种新的完全由卷积层构成的神经网络，因此该论文引入了引导式反向传播(Guided-Backpropagation)。

导向反向传播相当于对普通的反向传播加了指导，限制了小于0的梯度的回传该反向传播，可在进行标准梯度上升时过滤掉传播时产生的负激活。

# Construct the saliency object. This doesn't yet compute the saliency mask, it just sets up the necessary ops.
# NOTE: GuidedBackprop creates a copy of the given graph to override the gradient.
# Don't construct too many of these!
guided_backprop = saliency.GuidedBackprop(graph, sess, y, input_img)

# Compute the vanilla mask and the smoothed mask.
vanilla_guided_backprop_mask_3d = guided_backprop.GetMask(test_image_single[0], feed_dict = {neuron_selector: 1})
smoothgrad_guided_backprop_mask_3d = guided_backprop.GetSmoothedMask(test_image_single[0], feed_dict = {neuron_selector: 1})

# Call the visualization methods to convert the 3D tensors to 2D grayscale.
vanilla_mask_grayscale = saliency.VisualizeImageGrayscale(vanilla_guided_backprop_mask_3d)
smoothgrad_mask_grayscale = saliency.VisualizeImageGrayscale(smoothgrad_guided_backprop_mask_3d)

# Set up matplot lib figures.
ROWS = 1
COLS = 2
UPSCALE_FACTOR = 10
P.figure(figsize=(ROWS * UPSCALE_FACTOR, COLS * UPSCALE_FACTOR))

# Render the saliency masks.
ShowGrayscaleImage(vanilla_mask_grayscale, title='Vanilla Guided Backprop', ax=P.subplot(ROWS, COLS, 1))
ShowGrayscaleImage(smoothgrad_mask_grayscale, title='SmoothGrad Guided Backprop', ax=P.subplot(ROWS, COLS, 2))

INFO:tensorflow:Restoring parameters from /tmp/guided_backprop_ckpt

示例三：

Integrated Gradients（参考文献）

积分梯度（Integrated Gradients）这种方法的提出是为了解决传统基于梯度的可解释性方法的一个缺陷——梯度饱和。在上述最原始的解释性方法中，假设神经网络的分类结果线性依赖于输入图片中的每个像素或特征。

然而，真正的神经网络高度非线性，某个像素或特征增强到一定程度后可能对网络决策的贡献达到饱和，继续增加不会带来决策分数的增加，导致输出对输入特征的梯度为0。

积分梯度法从通过对梯度沿不同路径积分，期望得到非饱和区非零梯度对决策重要性的贡献。

# Construct the saliency object. This doesn't yet compute the saliency mask, it just sets up the necessary ops.
integrated_gradients = saliency.IntegratedGradients(graph, sess, y, input_img)
blur_ig = saliency.BlurIG(graph, sess, y, input_img)

# Baseline is a black image for vanilla integrated gradients.
baseline = np.zeros(test_image_single[0].shape)
baseline.fill(-1)

# Compute the vanilla mask and the Blur IG mask.
vanilla_integrated_gradients_mask_3d = integrated_gradients.GetMask(
  test_image_single[0], feed_dict = {neuron_selector: 1}, x_steps=25, x_baseline=baseline)
blur_ig_mask_3d = blur_ig.GetMask(
  test_image_single[0], feed_dict = {neuron_selector: 1})

# Call the visualization methods to convert the 3D tensors to 2D grayscale.
vanilla_mask_grayscale = saliency.VisualizeImageGrayscale(vanilla_integrated_gradients_mask_3d)
blur_ig_mask_grayscale = saliency.VisualizeImageGrayscale(blur_ig_mask_3d)

# Set up matplot lib figures.
ROWS = 1
COLS = 2
UPSCALE_FACTOR = 10
P.figure(figsize=(ROWS * UPSCALE_FACTOR, COLS * UPSCALE_FACTOR))

# Render the saliency masks.
ShowGrayscaleImage(vanilla_mask_grayscale, title='Vanilla Integrated Gradients', ax=P.subplot(ROWS, COLS, 1))
ShowGrayscaleImage(blur_ig_mask_grayscale, title='Blur Integrated Gradients', ax=P.subplot(ROWS, COLS, 2))

示例四：

XRAI（参考文献）

在积分梯度的基础上，该文章提出了一种基于区域的解释性方法，用于输出图像分类的显著性结果。

该方法首先对输入图像进行非监督的图像过分割，然后在积分梯度的基础上计算各分割区域内部像素的显著性值。

最后结合分割区域的总显著性值，得到各区域的显著性排序。

# Construct the saliency object. This doesn't yet compute the saliency mask, it just sets up the necessary ops.
xrai_object = saliency.XRAI(graph, sess, y, input_img)

# Compute XRAI attributions with default parameters
xrai_attributions = xrai_object.GetMask(test_image_single[0], feed_dict={neuron_selector: 1})

# Set up matplot lib figures.
ROWS = 1
COLS = 3
UPSCALE_FACTOR = 20
P.figure(figsize=(ROWS * UPSCALE_FACTOR, COLS * UPSCALE_FACTOR))

# Show original image
ShowImage(im, title='Original Image', ax=P.subplot(ROWS, COLS, 1))

# Show XRAI heatmap attributions
ShowHeatMap(xrai_attributions, title='XRAI Heatmap', ax=P.subplot(ROWS, COLS, 2))

# Show most salient 30% of the image
mask = xrai_attributions > np.percentile(xrai_attributions, 70)
im_mask = np.array(im)
im_mask[~mask] = 0
ShowImage(im_mask, title='Top 30%', ax=P.subplot(ROWS, COLS, 3))

更多详细代码请见链接