<jupyter_start><jupyter_text>This is an exmaple of how DenseNet is build by Tensorflow 2<jupyter_code>DenseNet面试题:
1. 描述下什么是Densenet
    CVPR2017收录论文, DenseNet脱离了加深网络层数(ResNet)和加宽网络结构(Inception)来提升网络性能的定式思维,从特征的角度考虑,通过特征重用和旁路(Bypass)设置,既大幅度减少了网络的参数量,又在一定程度上缓解了gradient vanishing问题的产生. 它主要由DenseBlock+TransitionBlock构成, DenseBlock为密集连接层, 而TransitionBlock为连接两两DenseBlock的层.
    
2. 为什么DenseNet优秀
    (1).相比ResNet拥有更少的参数数量.
    (2).旁路加强了特征的重用.
    (3).网络更易于训练(提高了整个网络的信息和梯度流动),并具有一定的正则效果.
    (4).缓解了gradient vanishing和model degradation的问题.

3. 具体说明下特征复用是什么, 并如何缓解了gradient vanishing和model degradation的问题
      何恺明先生在提出ResNet时做出了这样的假设:若某一较深的网络多出另一较浅网络的若干层有能力学习到恒等映射,那么这一较深网络训练得到的模型性能一定不会弱于该浅层网络.通俗的说就是如果对某一网络中增添一些可以学到恒等映射的层组成新的网路,那么最差的结果也是新网络中的这些层在训练后成为恒等映射而不会影响原网络的性能.同样DenseNet在提出时也做过假设:与其多次学习冗余的特征,特征复用是一种更好的特征提取方式.

4. DenseNet中的DenseBlock(1)参数量如何计算
    DenseNet121中的Dense Block(1)有6层, 它们的参数量计算如下 (growth_rate=32)
        输入 (output shape=None, 56, 56, 64)
        层1 (output shape=None, 56, 56, 64+32=96): 
            BN (64*4=256) + Relu (0) + Conv (128*1*1*64=8192) + BN (128*4=512) + Relu (0) + Conv (32*3*3*128=36864)
        层2 (output shape=None, 56, 56, 64+32*2=128)：
            BN (96*4=384) + Relu (0) + Conv (128*1*1*96=12288) + BN (128*4=512) + Relu (0) + Conv (32*3*3*128=36864)
        层3 (output shape=None, 56, 56, 64+32*3=160)：
            BN (128*4=512) + Relu (0) + Conv (128*1*1*128=16384‬) + BN (128*4=512) + Relu (0) + Conv (32*3*3*128=36864)
        层4 (output shape=None, 56, 56, 64+32*4=192)：
            BN (160*4=640) + Relu (0) + Conv (128*1*1*160=20480) + BN (128*4=512) + Relu (0) + Conv (32*3*3*128=36864)
        层5 (output shape=None, 56, 56, 64+32*5=224)：
            BN (192*4=768) + Relu (0) + Conv (128*1*1*192=24576‬) + BN (128*4=512) + Relu (0) + Conv (32*3*3*128=36864)
        层6 (output shape=None, 56, 56, 64+32*6=256): 
            BN (224*4=896) + Relu (0) + Conv (128*1*1*224=28672) + BN (128*4=512) + Relu (0) + Conv (32*3*3*128=36864)
        输出 (output shape=None, 56, 56, 256)
        总和 = 8192+12288+16384+20480+24576+28672+36864*6+256+384+512+640+768+896+512*6 = 338304
    
5. DenseNet中的DenseBlock如何进行反向传播<jupyter_output><empty_output><jupyter_text>DenseNet的网络结构如下所示:![image.png](attachment:image.png) DenseNet的核心模块DenseBlock的结构如下所示:![image.png](attachment:image.png)<jupyter_code>import tensorflow as tf
import numpy as np
from tensorflow.keras import layers
from tensorflow.keras.models import Model

print(tf.__version__)
print(tf.keras.__version__)
class DenseBlock(layers.Layer):
    def __init__(self, blocks, growth_rate = 32, **kwargs):
        super().__init__(**kwargs)
        self.blocks = blocks
        self.growth_rate = growth_rate

    def build(self, input_shape):
        self.layer_bn_a, self.layer_bn_b = [], []
        self.layer_conv2D_a, self.layer_conv2D_b = [], []
        self.layer_relu = layers.Activation('relu')
        self.layer_concat = layers.Concatenate(axis = -1)
        for i in range(self.blocks):
            self.layer_bn_a.append(layers.BatchNormalization(epsilon=1.001e-5))
            self.layer_bn_b.append(layers.BatchNormalization(epsilon=1.001e-5))
            self.layer_conv2D_a.append(layers.Conv2D(4 * self.growth_rate, 1, padding='same', use_bias=False))
            self.layer_conv2D_b.append(layers.Conv2D(self.growth_rate, 3, padding='same', use_bias=False))
        super().build(input_shape)
        
    def call(self, inputs, training=True, **kwargs):
        x = inputs
        for i in range(self.blocks):
            x1 = self.layer_bn_a[i](x, training=training)
            x1 = self.layer_relu(x1)
            x1 = self.layer_conv2D_a[i](x1, training=training)
            x1 = self.layer_bn_b[i](x1, training=training)
            x1 = self.layer_relu(x1)
            x1 = self.layer_conv2D_b[i](x1, training=training)
            x = self.layer_concat([x, x1])
        return x
    

class TransitionBlock(layers.Layer):
    def __init__(self, reduction = 0.5, **kwargs):
        super().__init__(**kwargs)
        self.reduction = reduction
        
    def build(self, input_shape):
        self.layer_bn = layers.BatchNormalization(epsilon=1.001e-5)
        self.layer_relu = layers.Activation('relu')
        self.layer_conv2D = layers.Conv2D(int(input_shape[-1] * self.reduction), 1, padding='same', use_bias=False)
        self.layer_ap = layers.AveragePooling2D(2, strides=2)
        super().build(input_shape)  
    
    def call(self, inputs, training=None, **kwargs):
        x = self.layer_bn(inputs, training=training)
        x = self.layer_relu(x)
        x = self.layer_conv2D(x, training=training)
        x = self.layer_ap(x)
        return x
def DenseNet(blocks, input_shape=None, classes=1000, name = '', **kwargs):
    inputs = layers.Input(shape=input_shape)
    bn_axis = 3

    # 224,224,3 -> 112,112,64
    x = layers.ZeroPadding2D(padding=((3, 3), (3, 3)))(inputs)
    x = layers.Conv2D(64, 7, strides=2, use_bias=False, name='Conv1/Conv')(x)
    x = layers.BatchNormalization(epsilon=1.001e-5, name='Conv1/Bn')(x)
    x = layers.Activation('relu', name='Conv1/Relu')(x)
    
    # 112,112,64 -> 56,56,64
    x = layers.ZeroPadding2D(padding=((1, 1), (1, 1)))(x)
    x = layers.MaxPooling2D(3, strides=2, name='Pool1')(x)


    for i in range(len(blocks)):
        x = DenseBlock(blocks[i], name='DenseBlock' + str(i))(x)
        if i != len(blocks) - 1: # Skip Last TransitionBlock
            x = TransitionBlock(0.5, name='TransitionBlock' + str(i))(x)
            
    x = layers.BatchNormalization(epsilon=1.001e-5, name='Bn')(x)
    x = layers.Activation('relu', name='Relu')(x)

    x = layers.GlobalAveragePooling2D(name='Ap')(x)
    x = layers.Dense(classes, activation='softmax', name='Fc1000')(x)

    return Model(inputs, x, name=name)
def DenseNet121(input_shape=[224,224,3], classes=1000, name = 'densenet121', **kwargs):
    return DenseNet([6, 12, 24, 16], input_shape, classes, name, **kwargs)

model = DenseNet121()
model.summary()<jupyter_output>Model: "densenet121"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_7 (InputLayer)         [(None, 224, 224, 3)]     0         
_________________________________________________________________
zero_padding2d_12 (ZeroPaddi (None, 230, 230, 3)       0         
_________________________________________________________________
Conv1/Conv (Conv2D)          (None, 112, 112, 64)      9408      
_________________________________________________________________
Conv1/Bn (BatchNormalization (None, 112, 112, 64)      256       
_________________________________________________________________
Conv1/Relu (Activation)      (None, 112, 112, 64)      0         
_________________________________________________________________
zero_padding2d_13 (ZeroPaddi (None, 114, 114, 64)      0         
_______________________________________________________[...]
{'fraction_non_alphanumeric': 0.1448924348197749, 'fraction_numerical': 0.0908961390511469, 'mean_word_length': 3.736842105263158, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '564027', 'license': 'MIT', 'n_tokens_mistral': 3183, 'n_tokens_neox': 2893, 'n_words': 441, 'path': 'Classification/DenseNet.ipynb', 'repo_name': 'chendetong1993/DeepLearning', 'size': '7019'}