<jupyter_start><jupyter_text>Benchmark operation on a small rank three tensor.<jupyter_code>import sys
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
def r3py(a):
    """ Take a rank 3 python tensor and set all values to one. """
    I, J, T = len(a[0][0]), len(a[0]), len(a)
    for i in range(I):
        for j in range(J):
            for t in range(T):
                a[t][j][i] = 1
a = [[[0 for _ in range(1024)] for _ in range(1024)] for _ in range(10)]
sys.getsizeof(a[0][0]) * len(a[0]) * len(a)<jupyter_output><empty_output><jupyter_text>Around 8 bytes per entry for a pointer. An integer takes up 28 bytes.<jupyter_code>sys.getsizeof(0), sys.getsizeof(1), sys.getsizeof(2**3), sys.getsizeof(2**8), sys.getsizeof(sys.maxsize)
s = pd.Series([sys.getsizeof(2**i) for i in range(0, 128, 1)])
ax = s.plot(kind='line', style='b-', grid=True, title='integers in memory')
ax.set_xlabel('2^n')
ax.set_ylabel('sizeof')
plt.plot()<jupyter_output><empty_output><jupyter_text>The 1024x1024x10 pure python array takes up at least 92405760 * 28 bytes (around 2G) in memory.Let's set the value of each element to some value.<jupyter_code>%timeit -r4 r3py(a) # 2.22 s ± 45.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)<jupyter_output>2.28 s ± 76.5 ms per loop (mean ± std. dev. of 4 runs, 1 loop each)<jupyter_text>How about floats?> One of the major challenges in writing (somewhat) large-scale Python programs is to keep memory usage at a minimum.<jupyter_code>sys.getsizeof(2.0**1023)
sys.float_info<jupyter_output><empty_output><jupyter_text>Numpy version<jupyter_code>import numpy as np
ar = np.zeros((1024, 1024, 10))
ar.nbytes
ar.dtype<jupyter_output><empty_output><jupyter_text>The numpy version is much more compact: 83886080 bytes (around 80M). We can set an explicit type.<jupyter_code>ar = np.zeros((1024, 1024, 10), dtype='uint8')
ar.nbytes<jupyter_output><empty_output><jupyter_text>Numpy float information.<jupyter_code>pd.DataFrame([(np.finfo(f).dtype, np.finfo(f).nexp, np.finfo(f).nmant, np.finfo(f).min, np.finfo(f).max)
              for f in (np.float32, np.float64, np.float128, float)], columns=['name', 'nexp', 'nmant', 'min', 'max'])
# Let work with ints.
ar = np.zeros((1024, 1024, 10), dtype='uint8')
%timeit ar + 1 # 4.83 ms ± 274 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)<jupyter_output>4.66 ms ± 541 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)<jupyter_text>PyTorch and GPU<jupyter_code>import torch
torch.IntTensor(5, 3).zero_()
torch.IntTensor(5, 3).zero_() + 1
t = torch.IntTensor(1024, 1024, 10).zero_()
%timeit t + 1 # 19.4 ms ± 1.3 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)
# t.cuda?<jupyter_output><empty_output><jupyter_text>> Returns a copy of this object in CUDA memory.> If this object is already in CUDA memory and on the correct device, then no copy is performed and the original object is returned.* OpenCL?<jupyter_code>if torch.cuda.device_count():
    gt = t.cuda()
else:
    print("GPU? 乁( ◔ ౪◔)「      ┑(￣Д ￣)┍ ")
    pass # AssertionError: Torch not compiled with CUDA enabled<jupyter_output>GPU? 乁( ◔ ౪◔)「      ┑(￣Д ￣)┍<jupyter_text>Note: Next cells will **only** work on **GPU**.<jupyter_code>gt = t.cuda()
type(gt)
gt.size()
gt.get_device()
gt.element_size()
gt.is_cuda
%timeit gt + 1 # 2.42 ms ± 8.06 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
one = torch.IntTensor([1])
one
one.is_cuda
oneg = one.cuda()
oneg.is_cuda
%timeit gt + oneg # 2.5 ms ± 7.31 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1487279843444227, 'fraction_numerical': 0.0561923399496785, 'mean_word_length': 4.547286821705426, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '557925', 'license': 'MIT', 'n_tokens_mistral': 1559, 'n_tokens_neox': 1389, 'n_words': 429, 'path': '1 Benchmarks (python vs numpy).ipynb', 'repo_name': 'miku/pytorch-tour', 'size': '3577'}