<jupyter_start><jupyter_text>This notebook was inspired by @kilian's excellent notebook: https://www.kaggle.com/batzner/gini-coefficient-an-intuitive-explanation, especially the alternative way of computing gini at the end.  I simply implemented the computation directly.  The code uses Numba to make it run fast.<jupyter_code>from numba import jit

@jit
def eval_gini(y_true, y_prob):
    y_true = np.asarray(y_true)
    y_true = y_true[np.argsort(y_prob)]
    ntrue = 0
    gini = 0
    delta = 0
    n = len(y_true)
    for i in range(n-1, -1, -1):
        y_i = y_true[i]
        ntrue += y_i
        gini += y_i * delta
        delta += 1 - y_i
    gini = 1 - 2 * gini / (ntrue * (n - ntrue))
    return gini<jupyter_output><empty_output><jupyter_text>How efficient is the above code?  Let's compare it with code taken from another excellent notebook by <NAME>: https://www.kaggle.com/tezdhar/faster-gini-calculation . His code is reproduced below.<jupyter_code>#Remove redundant calls
def ginic(actual, pred):
    actual = np.asarray(actual) #In case, someone passes Series or list
    n = len(actual)
    a_s = actual[np.argsort(pred)]
    a_c = a_s.cumsum()
    giniSum = a_c.sum() / a_s.sum() - (n + 1) / 2.0
    return giniSum / n
 
def gini_normalizedc(a, p):
    if p.ndim == 2:#Required for sklearn wrapper
        p = p[:,1] #If proba array contains proba for both 0 and 1 classes, just pick class 1
    return ginic(a, p) / ginic(a, a)<jupyter_output><empty_output><jupyter_text>Let's create a random test set roughly as large as the train data set.<jupyter_code>a = np.random.randint(0,2,600000)
p = np.random.rand(600000)
print(a[10:15], p[10:15])<jupyter_output>[1 0 1 0 1] [ 0.98810261  0.10452699  0.2870508   0.79100942  0.54620356]<jupyter_text>As sanity check, let's compare the output of the tow methods.<jupyter_code>gini_normalizedc(a, p)
eval_gini(a, p)
gini_normalizedc(a, p) - eval_gini(a, p)<jupyter_output><empty_output><jupyter_text>Looks fine, difference is negligible.  Let's time them now.<jupyter_code>%%timeit 
gini_normalizedc(a,p)
%%timeit 
eval_gini(a,p)<jupyter_output>10 loops, best of 3: 73.4 ms per loop<jupyter_text>OK, the speedup is not that large, but there is a speedup still.  Note that my code only handles binary values for y_true while Mohsin's code is more general. The speedup looks better if we factor the sorting time out.  Let's measure it.<jupyter_code>%timeit np.argsort(p)<jupyter_output>10 loops, best of 3: 66 ms per loop
{'fraction_non_alphanumeric': 0.10829307568438003, 'fraction_numerical': 0.03985507246376811, 'mean_word_length': 3.9799599198396796, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '92146', 'license': 'MIT', 'n_tokens_mistral': 1004, 'n_tokens_neox': 917, 'n_words': 275, 'path': '10 poer sugero safe driver prediction/extremely-fast-gini-computation.ipynb', 'repo_name': 'MLVPRASAD/KaggleProjects', 'size': '2490'}