<jupyter_start><jupyter_text>A demonstration of backdrop masking<jupyter_code>from backdrop import *
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Simple masking example. Define x with 3 dimensions and let's say we want to mask on the 2nd dimension.<jupyter_code># Use requires_grad = True for demonstration so we can see what parts of the data are affected later.
x = Variable(torch.ones([1,4,4]),requires_grad = True)<jupyter_output><empty_output><jupyter_text>Define the masking layer with 50% masking probability.  Note mask_dims is the list of the dimensions for which we would want to implement masking.<jupyter_code>#In this case we want to mask along the second dimension of the input.
my_mask = mask_mod(mask_prob=0.5, mask_dims = [1])
my_mask.verbose = True #Prints out the generated mask every time.<jupyter_output><empty_output><jupyter_text>Pass the variable x through the mask. The mask is normalized by the number of its nonzero elements.<jupyter_code>y = my_mask(x)<jupyter_output>Generated mask with shape:  torch.Size([1, 4])
The generate mask is:  [2. 2. 0. 0.]
--------------------------------------------------<jupyter_text>y is identical to x. The mask is transparent during the forward pass.<jupyter_code>print(y)<jupyter_output>Variable containing:
(0 ,.,.) = 
  1  1  1  1
  1  1  1  1
  1  1  1  1
  1  1  1  1
[torch.FloatTensor of size 1x4x4]<jupyter_text>But it blocks the gradient along the backward pass. Note the normalization. The nonzero elements are the same as the mask's.<jupyter_code>y.sum().backward()
print(x.grad)
x.grad.data.zero_();<jupyter_output>Variable containing:
(0 ,.,.) = 
  2  2  0  0
  2  2  0  0
  2  2  0  0
  2  2  0  0
[torch.FloatTensor of size 1x4x4]<jupyter_text>More complicated masking example. Define x with 4 dimensions and mask along the last two.<jupyter_code># Use requires_grad = True for demonstration so we can see what parts of the data are affected later.
x = Variable(torch.randn([1,1,1024,1024]),requires_grad = True)<jupyter_output><empty_output><jupyter_text>Define two masking layer with 30% and 95% masking probability.<jupyter_code>mask_fine = mask_mod(mask_prob=0.3, mask_dims = [2,3])
mask_coarse = mask_mod(mask_prob=0.95, mask_dims = [2,3])<jupyter_output><empty_output><jupyter_text>Define a sequential network with the two masks in there.<jupyter_code># The strides and kernel size are chosen here to give a crisp mask definition. If kernel_size > stride, the masks will bleed into larger regions.

net = nn.Sequential(nn.Conv2d(in_channels=1, out_channels=64,kernel_size=5, stride=2,padding=2),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=2, stride=2),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=2, stride=2),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=2, stride=2),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=2, stride=2),nn.ReLU(),
                    mask_fine,
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=2, stride=2),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=2, stride=2),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=2, stride=2),nn.ReLU(),
                    mask_coarse)<jupyter_output><empty_output><jupyter_text>Pass the variable x through the network.<jupyter_code>y = net(x)
# dummy loss function
loss = y.sum()
loss.backward()<jupyter_output><empty_output><jupyter_text>Let's look at the gradients that made it all the way back to x.<jupyter_code>plt.figure(figsize=(10,10))
plt.imshow(x.grad.data.squeeze().cpu().abs().numpy(),vmax=0.0003)
plt.show()
x.grad.data.zero_();<jupyter_output><empty_output><jupyter_text>We see the product of the two masks at the different scales.  We can also specify where the mask is. (for diagnosis or variance checking)<jupyter_code>mask_coarse.mask_el=[0,0,2,2]

y = net(x)
# dummy loss function
loss = y.sum()
loss.backward()

plt.figure(figsize=(10,10))
plt.imshow(x.grad.data.squeeze().cpu().abs().numpy(),vmax=0.0003)
plt.show()
x.grad.data.zero_();

mask_coarse.mask_el=None # Revert back to random
# If the strides and kernel sizes are not chosen as before. 
# Same network just larger kernels for the later convs with padding to give the same size

net = nn.Sequential(nn.Conv2d(in_channels=1, out_channels=64,kernel_size=5, stride=2,padding=2),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=2, stride=2),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=2, stride=2),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=2, stride=2),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=2, stride=2),nn.ReLU(),
                    mask_fine,
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=3, stride=2,padding=1),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=3, stride=2,padding=1),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=3, stride=2,padding=1),nn.ReLU(),
                    mask_coarse)

mask_coarse.mask_el=[0,0,2,2]

y = net(x)
# dummy loss function
loss = y.sum()
loss.backward()

plt.figure(figsize=(10,10))
plt.imshow(x.grad.data.squeeze().cpu().abs().numpy(),vmax=0.0003)
plt.show()
x.grad.data.zero_();

mask_coarse.mask_el=None # Revert back to random<jupyter_output><empty_output><jupyter_text>As expected the receptor field is larger but the gradients drop off as we approach the boundary.<jupyter_code># If we also increase the kernel sizes for the smaller patches:

net = nn.Sequential(nn.Conv2d(in_channels=1, out_channels=64,kernel_size=5, stride=2,padding=2),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=3, stride=2,padding=1),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=3, stride=2,padding=1),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=3, stride=2,padding=1),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=3, stride=2,padding=1),nn.ReLU(),
                    mask_fine,
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=3, stride=2,padding=1),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=3, stride=2,padding=1),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=3, stride=2,padding=1),nn.ReLU(),
                    mask_coarse)

mask_coarse.mask_el=[0,0,2,2]

y = net(x)
# dummy loss function
loss = y.sum()
loss.backward()

plt.figure(figsize=(10,10))
plt.imshow(x.grad.data.squeeze().cpu().abs().numpy(),vmax=0.0003)
plt.show()
x.grad.data.zero_();

mask_coarse.mask_el=None # Revert back to random<jupyter_output><empty_output><jupyter_text>The receptor fields get completely washed out unless the mask probability is very high.<jupyter_code># Same as above but with higher fine mask probability.

mask_fine.mask_prob=0.85

net = nn.Sequential(nn.Conv2d(in_channels=1, out_channels=64,kernel_size=5, stride=2,padding=2),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=3, stride=2,padding=1),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=3, stride=2,padding=1),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=3, stride=2,padding=1),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=3, stride=2,padding=1),nn.ReLU(),
                    mask_fine,
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=3, stride=2,padding=1),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=3, stride=2,padding=1),nn.ReLU(),
                    nn.Conv2d(in_channels=64, out_channels=64,kernel_size=3, stride=2,padding=1),nn.ReLU(),
                    mask_coarse)

mask_coarse.mask_el=[0,0,2,2]

y = net(x)
# dummy loss function
loss = y.sum()
loss.backward()

plt.figure(figsize=(10,10))
plt.imshow(x.grad.data.squeeze().cpu().abs().numpy(),vmax=0.0003)
plt.show()
x.grad.data.zero_();

mask_coarse.mask_el=None # Revert back to random<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13561936402467964, 'fraction_numerical': 0.04568106312292359, 'mean_word_length': 3.993483412322275, 'pattern_counts': {'":': 0, '<': 54, '<?xml version=': 0, '>': 55, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '405715', 'license': 'MIT', 'n_tokens_mistral': 3194, 'n_tokens_neox': 2965, 'n_words': 600, 'path': 'backdrop-demo.ipynb', 'repo_name': 'dexgen/backdrop', 'size': '8428'}