<jupyter_start><jupyter_text>other things, questions that are worth asking to get a high-level view:- what types of crimes (IUCR) have had the largest (relative, or percentage) drop/rise year on year?- what locations (Community.Area) have had the largest (relative, or percentage) drop/rise year on year?  question 2<jupyter_code># to cluster districts by crimes, I will represent each district as a vector with the counts for each crime 
# that happened in there

# but District is a floating point number
# how many different Districts are there?
districts = df.District.unique()

# ok so even though it's a float it's probably a categorical column

# remove NaNs because they may be noise
not_nans = np.invert(np.isnan(districts))

districts = districts[not_nans]
districts

# we need this because some districts may not have had all types of crime
# in which case the vector would not match the others
# (they need to be in the same order too)
all_iucrs = df["IUCR"].unique()

all_iucrs.sort()

all_iucrs
district_vectors = []

for district_index in districts:
    
    iucrs = df[df["District"] == district_index].groupby("IUCR")["IUCR"]
    
    vector = []
    
    for iucr_name in all_iucrs:
        
        # if there's a key error, it means this district has had no crimes of this IUCR code
        try:
            count = len(iucrs.get_group(iucr_name))
        except KeyError:
            count = 0
            
        vector.append(count)    
     
    district_vectors.append(vector)
        
X = np.vstack(district_vectors)
X.shape
# we normalize it to prevent large absolute values from affecting more
sc = preprocessing.MinMaxScaler()
X_scaled = sc.fit_transform(X)
kmeans = KMeans(n_clusters=2, random_state=0).fit(X_scaled)
kmeans.cluster_centers_<jupyter_output><empty_output><jupyter_text>what I would do if I had more time- try to project the points into a lower dimensionality to see whether they make sense and whether we can get any insight from looking at it- use other clustering methods- use cluster metrics (such as entropy) to see whether the clusters are good at splitting the data.  question 3 general strategy for a quick model: 1) extract features, train  - every date becomes: number of days from today   - this is because the "test set" is the next 6 months (from the end of the dataset), so we don't have data in that time frame, but if we consider the number of days from today, we still get some generalization and will be able to spot trends    - districts are one hot encoded  - murders are encoded with a 1 if IUCR=='HOMICIDE', 0 otherwise  2) at inference time, generate rows with all districts and murder == 1, for the 6 months. (6 x 50 == 300 so it's not much data) then we run a simple algorithm like Logistic Regression and get the probability, for each (district, month) that murder == 1. Then we select the district with the highest probability.<jupyter_code># extract features and try out a simple model just to get some results quickly

# one-hot-encode categorical features
districts = pd.get_dummies(df["District"])

## DID NOT FINISH<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06630365150544523, 'fraction_numerical': 0.006085842408712364, 'mean_word_length': 3.887323943661972, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '559359', 'license': 'MIT', 'n_tokens_mistral': 916, 'n_tokens_neox': 861, 'n_words': 418, 'path': 'python3/notebooks/crimes/task.ipynb', 'repo_name': 'tobby2002/python-sandbox', 'size': '3122'}