<jupyter_start><jupyter_text>week04: understanding the positional encoding_This notebook is brought to you by [<NAME>](https://www.linkedin.com/in/vladislav-goncharenko/)_<jupyter_code># If using Colab, uncomment this cell
! pip install plotly --upgrade
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

import plotly.express as px
plt.rcParams.update({'font.size': 14})<jupyter_output><empty_output><jupyter_text>Positional Encoding matrix components proposed in the article$$PE_{(pos,2i)} = \sin \left( \frac{pos}{10000^{2i/d_{\text{model}}}} \right) \sim$$$$\sim \sin \left( \exp \left( -\frac{2i}{d_{\text{model}}} \right) \cdot \text{pos} \right) =\sin(\omega \cdot t)$$$$ \\ $$$$PE_{(pos,2i+1)} =\cos(\dots) \sim \cos (\omega \cdot t)$$Let's treat $\text{pos}$ as time and number of embedding component as carrier frequency of our signal.Note that carrier frequencies decrease exponentionally.<jupyter_code>def make_carriers(d_mod, denom):
    return 1 / np.power(denom, np.arange(d_mod) / d_mod)
carriers = {i: make_carriers(10, 10**i) for i in (8, 4, 2)}

plt.figure(figsize=(15, 7))
for i, carrier in carriers.items():
    plt.plot(carrier, label=f'denominator: 10^{i}')
plt.legend()
plt.xlabel('Number of embedding component')
plt.ylabel('Carrier frequency')
plt.show()<jupyter_output><empty_output><jupyter_text>In fact different frequencies correspond to different "clocks" that we have to measure $\text{pos}$ as time.Model doesn't know $\text{pos}$ value directly but it sees all the "times" (in fact phases) of differend "clocks" ($\sin$s and $\cos$s of different frequencies)Having representative sutie of "clocks" we can definetly say _what time is it now ($\text{pos}$ value)_ for every given "moment"<jupyter_code>def make_pa_matrix(n_pos, d_mod, denom):
    res= np.empty((n_pos, d_mod))
    carriers = make_carriers(d_mod, denom)

    for pos in range(n_pos):
        if pos % 2:
            funct = np.sin
        else:
            funct = np.cos

        for i in range(d_mod):
            res[pos, i] = funct((pos // 2) * carriers[i])
    return res<jupyter_output><empty_output><jupyter_text>2d case<jupyter_code>pa2 = make_pa_matrix(1000, 2, 10**4)

plt.figure(figsize=(15, 12))
plt.scatter(pa2[:, 0], pa2[:, 1], c=np.arange(len(pa2)))
plt.colorbar()
plt.show()<jupyter_output><empty_output><jupyter_text>**Nice and harmonic picture, isn't it?**That's because the curve plotted is [Lissajou's curve](https://en.wikipedia.org/wiki/Lissajous_curve):![](https://upload.wikimedia.org/wikipedia/commons/5/5d/Lissajous_animation.gif)Curve implicitly specified by harmonic coordinates$$\left\{ \begin{align}  & x(t)=A\sin (at+\delta ) \\  & y(t)=B\sin (bt) \\ \end{align} \right.$$In our case $\delta = \pi / 2$  3d case<jupyter_code>pa3 = make_pa_matrix(250, 3, 2**3)
pa3_df = pd.DataFrame(
    np.concatenate((pa3, np.arange(len(pa3))[:, None]), axis=1),
    columns=['x', 'y', 'z', 'c'],
)
pa3_df.head()
fig = px.scatter_3d(pa3_df, x='x', y='y', z='z', color='c')
fig.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16699475065616798, 'fraction_numerical': 0.021653543307086614, 'mean_word_length': 4.886100386100386, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '229857', 'license': 'MIT', 'n_tokens_mistral': 1166, 'n_tokens_neox': 1099, 'n_words': 280, 'path': 'ML-MIPT-advanced/week1_04_Transformer_and_PoS_Tagging/week1_04_positional_encoding_carriers.ipynb', 'repo_name': 'kryvokhyzha/examples-and-courses', 'size': '3063'}