<jupyter_start><jupyter_text>直接构造源句子、目标句子Embedding S: is the source sequence length, T: is the target sequence length,  N: is the batch size, E is the feature number<jupyter_code>src = torch.rand((10, 32, embed_dim)) # (S, N, E)
tgt = torch.rand((20, 32, embed_dim)) # (T, N, E)
out = transformer_model(src, tgt)
out.shape # (T, N, E)<jupyter_output><empty_output><jupyter_text>源句子、目标句子的构造 src_idx: (N,S) tgt_idx: (N,T)<jupyter_code>N=2
S=5
T=10
src_idx=torch.randint(0, 11, (N,S))
tgt_idx=torch.randint(0, 11, (N,T))
src_idx
tgt_idx
src_idx_reshaped=src_idx.permute(1,0)
tgt_idx_reshaped=tgt_idx.permute(1,0)
src_idx_reshaped
tgt_idx_reshaped<jupyter_output><empty_output><jupyter_text>Attention相关的输入参数构造 src_mask: (S,S), tgt_mask: (T, T)  memory_mask: (T, S)<jupyter_code>src_mask=memory_mask=None #encoder的暂时假设为None
tgt_mask=transformer_model.generate_square_subsequent_mask(sz=T)
tgt_mask
T,tgt_mask.shape
src = torch.rand((S, B, embed_dim)) # (S, N, E)
tgt = torch.rand((T, B, embed_dim)) # (T, N, E)
out = transformer_model(src, tgt,src_mask=src_mask,tgt_mask=tgt_mask,memory_mask=memory_mask)
out.shape # (T, N, E)<jupyter_output><empty_output><jupyter_text>Padding_mask相关的输入参数构造 src_key_padding_mask: (N,S), tgt_key_padding_mask: (N, T)  memory_key_padding_mask: (N, S)<jupyter_code>pad_idx=1
src_idx=torch.randint(0, 11, (N,S))
src_idx
src_key_padding_mask=(src_idx==pad_idx)
src_key_padding_mask # If a BoolTensor is provided, the positions with the value of True will be ignored while the position with the value of False will be unchanged.
tgt_idx=torch.randint(0, 11, (N,T))
tgt_idx
tgt_key_padding_mask=(tgt_idx==pad_idx)
tgt_key_padding_mask # If a BoolTensor is provided, the positions with the value of True will be ignored while the position with the value of False will be unchanged.
src = torch.rand((S, B, embed_dim)) # (S, N, E)
tgt = torch.rand((T, B, embed_dim)) # (T, N, E)
out = transformer_model(src, tgt,
                        src_mask=src_mask,tgt_mask=tgt_mask,memory_mask=memory_mask,
                        src_key_padding_mask=src_key_padding_mask,tgt_key_padding_mask=tgt_key_padding_mask,
                        memory_key_padding_mask=None
                       )
out.shape # (T, N, E)<jupyter_output><empty_output><jupyter_text>构造最后的Liner层<jupyter_code>vocab_size=11 # [0...10]
lin=nn.Linear(embed_dim,vocab_size)
ret=lin(out)
ret.shape
ret_reshaped=ret.permute(1,0,2)
ret_reshaped.shape
predict=ret_reshaped.argmax(dim=-1)
predict.shape
predict<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12554628526023043, 'fraction_numerical': 0.015097338100913786, 'mean_word_length': 6.235632183908046, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '311950', 'license': 'Apache-2.0', 'n_tokens_mistral': 1086, 'n_tokens_neox': 1075, 'n_words': 199, 'path': 'joeynmt/transformer_layers_todo.ipynb', 'repo_name': 'zhangjiekui/joeynmt', 'size': '2517'}