<jupyter_start><jupyter_text>Estimate Time--- Learning ObjectivesThe goal of this lab is to estimate compute time needed for an end to end training run.**Motivation**: In order to request computing resources for a training job to run on a super computing cluster, one must provide information such as, the number of nodes/gpus requested as well as the estimated time for an end-to-end training job.Note: now that we obtained the toy text data via webscraping from the previous lab, the next step is to request compute resources. In order to train a very large language model, large compute resources must be requested and approved in advance. For the cluster admin to allocate resources to the training job, the applicant must provide minimal information such as the number of GPUs necessary for the training job to run as well as estimate how long, i.e the time it takes, to compute one end to end training run.This is what we are trying to achieve in this notebook, estimate training time per given parameters below.Training time (in seconds) is approximated with this equation : 8*T*P/n*X- T = dataset size measured in numbers of tokens in the dataset- P = model parameters for GPT3 variants- n = number of GPUs in the compute cluster- x = achieved teraflops per GPU The above equation was extracted from this paper : [Efficient Large-Scale Language Model Training on GPU Clusters](https://arxiv.org/pdf/2104.04473.pdf)---------------------------------------------------------------------------------------------------Assets provided below for you convenience :  --- Sanity check - Two scenarios were extracted from the screenshot of the paper : [Efficient Large-Scale Language Model Training on GPU Clusters](https://arxiv.org/pdf/2104.04473.pdf) **Scenario 1** -T = 300 billion tokens  assumed data size measured in tokensP = 175 billion GPT3 modeln = 1024 GPUsx = 140 teraFLOP/s per GPUQuestion : How many hours/ days will you need given the scenaio above for you to compute an end to end training job ?Answer : We should observe around **34 days** for an end to end training run**Scenario 2** - T =  450 billion tokens  P = 1 trillion parameters GPT 3 modeln = 3072 GPUsx = 163 teraFLOP/s per GPU Question: How many hours/ days will you need given this scenario above for you to compute an end to end training job ?Answer: We should observe around **84 days** for an end to end training run The below code block wraps the two scenarios within a function for automation.<jupyter_code># The following code block contain automatic function which assists the calculation of time-to-compute for an end to end training run.
import numpy as np
# T = dataset size measured in numbers of tokens in the dataset
# P = model parameters for GPT3 varients
# n = number of GPUs in the compute cluster
# x = achieved teraflops per GPU 

def calculate_days_needed(T , P , n ,x):
    if x is None:
        return 'not a good SuperPOD use case, let us try a bigger model :)'
    else:        
        tot=8*T*P
        div=n*x
        compute_sec=tot/div
        #convert compute seconds to days
        to_days=round(compute_sec/(3600*24),1)
        return to_days
## sanity check against the two scenarios above 
T=[300*1e+9, 450*1e+9]
n=[1024,3072]
GPT3_models_labels=[  'gpt3_175B','gpt3_1Trillion']
GPT3_model_params=[ 175*1e+9,1*1e+12 ]
GPT3_model_params_str=['175 Billion','1Trillion']
#according to the table above
GPT3_X=[140*1e+12,163*1e+12]
print("all below are measured with dataset size **300 billion** measured in tokens \n")
scene=1
for gpt3_name, gpt3_params, gpt3_param_str, x, n_,t in zip(GPT3_models_labels,GPT3_model_params,GPT3_model_params_str, GPT3_X ,n,T):
    days_needed=calculate_days_needed(t,gpt3_params,n_,x)
    print(" ----------------------------scenario {}-----------------------------------".format(scene))
    print(" language model :{} with {} number of parameters , it will need {} days to compute \n".format(gpt3_name, gpt3_param_str, str(days_needed)))
    scene+=1<jupyter_output><empty_output><jupyter_text>Below is an example of expected outputs :     ----------------------------scenario 1-----------------------------------     language model :gpt3_175B with 175 Billion number of parameters , it will need 33.9 days to compute      ----------------------------scenario 2-----------------------------------     language model :gpt3_1Trillion with 1Trillion number of parameters , it will need 83.2 days to compute ---**Exercise** -For a GPT3 model size of 70B parameters with approximately 300 Billion tokens in an existing dataset, you have requested 1/4 of the total number of GPUs available in [BerzeLiUs](https://www.nsc.liu.se/support/systems/berzelius-getting-started/).Question -How many hours/days would you need to do an end to end training run ?<jupyter_code>T=<FILL_IN> 
p=<FILL_IN> 
n=<FILL_IN> 
x=<FILL_IN> 
gpt3_params=<FILL_IN> 
calculate_days_needed(T,gpt3_params,n,x)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12593491004649282, 'fraction_numerical': 0.03173640590256721, 'mean_word_length': 3.3403508771929826, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '477483', 'license': 'Apache-2.0', 'n_tokens_mistral': 1463, 'n_tokens_neox': 1344, 'n_words': 618, 'path': 'ai/Megatron/English/Python/jupyter_notebook/Lab1-2_EstimateComputeDaysNeeded.ipynb', 'repo_name': 'programmah/gpubootcamp', 'size': '4947'}