<jupyter_start><jupyter_text>Create dataset for DDM DDM is a temporal model, meaning it should be trained to a specific period of time. Because our imagery cadence is inconsistent (e.g., we don't reliably haev 1 image per day or per week for each location), we modify the dataset to be organized by week. Each location should have one image per week, 52 images in total. If a location has more than one image on a given week, we randomly select one to use. If it has no images on a given week, we use the image from the previous week. The resulting dataset allows to us to train a DDM model for each week of the year.<jupyter_code>truth_df = pd.read_csv('../local/truth.csv')
truth_df.head()
def extract_week(img):
    """Extract the week from the image name"""
    
    dat = img.split(os.sep)[-1].split('_')[0]
    dat = datetime.strptime(dat, '%Y%m%d')
    week = dat.isocalendar()[1]
    month = dat.month
    return week, month

time_periods = {}

for i, row in tqdm(truth_df.iterrows()): 
    
    if row.label == 2:
        continue 
        
    time_periods[row.location_name] = {f"week_{i}" : {} for i in range(1,53)}
    
    ims = sorted(glob.glob(os.path.join(row.dir, '*.tif')))
    im0 = ims[0]
    
    times = [extract_week(im) for im in ims]
    weeks = [w for w, _ in times]
    
    weeks, inds = np.unique(weeks, return_index=True)
    ims = np.array(ims)[inds]
        
    if weeks[0] != 1: 
        weeks = np.insert(weeks, 0, 1)
        ims = np.insert(ims, 0, im0)
            
    for i in range(len(ims)):
        
        with rasterio.open(ims[i]) as src:
            b, g, r, _ = src.read()

        end = 53 if i == len(ims)-1 else weeks[i+1]
        for w in range(weeks[i], end):
            
            time_periods[row.location_name][f"week_{w}"] = {
                "r": r,
                "g": g,
                "b": b
            }
            
    time_periods[row.location_name]['images'] = ims

    

with open('time_periods.p', 'wb') as f: 
    pickle.dump(time_periods, f)
np.unique(weeks)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12203389830508475, 'fraction_numerical': 0.010653753026634382, 'mean_word_length': 2.398026315789474, 'pattern_counts': {'":': 3, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '50514', 'license': 'MIT', 'n_tokens_mistral': 722, 'n_tokens_neox': 685, 'n_words': 228, 'path': 'notebooks/ddm_setup.ipynb', 'repo_name': 'reglab/building-expansion', 'size': '2065'}