<jupyter_start><jupyter_text>Someone asked how to generate outputs to use with  [LibFFM](https://github.com/guestwalk/libffm)So all I do is to use pandas cuts for the numerics to turn them into categories.  Feel free to try using them as straight numerics if you wish.  I have tried to make it as generic as possible so you can use it on other competitons going forward!<jupyter_code>import math
import numpy as np
import pandas as pd
train = pd.read_csv('../input/train.csv')
test = pd.read_csv('../input/test.csv')
test.insert(1,'target',0)
print(train.shape)
print(test.shape)
x = pd.concat([train,test])
x = x.reset_index(drop=True)
unwanted = x.columns[x.columns.str.startswith('ps_calc_')]
x.drop(unwanted,inplace=True,axis=1)
features = x.columns[2:]
categories = []
for c in features:
    trainno = len(x.loc[:train.shape[0],c].unique())
    testno = len(x.loc[train.shape[0]:,c].unique())
    print(c,trainno,testno)<jupyter_output>ps_ind_01 8 8
ps_ind_02_cat 5 5
ps_ind_03 12 12
ps_ind_04_cat 3 3
ps_ind_05_cat 8 8
ps_ind_06_bin 2 2
ps_ind_07_bin 2 2
ps_ind_08_bin 2 2
ps_ind_09_bin 2 2
ps_ind_10_bin 2 2
ps_ind_11_bin 2 2
ps_ind_12_bin 2 2
ps_ind_13_bin 2 2
ps_ind_14 5 5
ps_ind_15 14 14
ps_ind_16_bin 2 2
ps_ind_17_bin 2 2
ps_ind_18_bin 2 2
ps_reg_01 10 10
ps_reg_02 19 19
ps_reg_03 5013 5046
ps_car_01_cat 13 13
ps_car_02_cat 3 3
ps_car_03_cat 3 3
ps_car_04_cat 10 10
ps_car_05_cat 3 3
ps_car_06_cat 18 18
ps_car_07_cat 3 3
ps_car_08_cat 2 2
ps_car_09_cat 6 6
ps_car_10_cat 3 3
ps_car_11_cat 104 104
ps_car_11 5 5
ps_car_12 184 201
ps_car_13 70482 83769
ps_car_14 850 885
ps_car_15 15 15<jupyter_text>OK so let us turn the big boys into categories<jupyter_code>x.loc[:,'ps_reg_03'] = pd.cut(x['ps_reg_03'], 50,labels=False)
x.loc[:,'ps_car_12'] = pd.cut(x['ps_car_12'], 50,labels=False)
x.loc[:,'ps_car_13'] = pd.cut(x['ps_car_13'], 50,labels=False)
x.loc[:,'ps_car_14'] =  pd.cut(x['ps_car_14'], 50,labels=False)
x.loc[:,'ps_car_15'] =  pd.cut(x['ps_car_15'], 50,labels=False)
test = x.loc[train.shape[0]:].copy()
train = x.loc[:train.shape[0]].copy()
#Always good to shuffle for SGD type optimizers
train = train.sample(frac=1).reset_index(drop=True)
train.drop('id',inplace=True,axis=1)
test.drop('id',inplace=True,axis=1)
train.head()
test.head()<jupyter_output><empty_output><jupyter_text>All out parameters are categories - if you want to try numerics please scale them first!<jupyter_code>categories = train.columns[1:]
numerics = []
currentcode = len(numerics)
catdict = {}
catcodes = {}
for x in numerics:
    catdict[x] = 0
for x in categories:
    catdict[x] = 1

noofrows = train.shape[0]
noofcolumns = len(features)
with open("alltrainffm.txt", "w") as text_file:
    for n, r in enumerate(range(noofrows)):
        if((n%100000)==0):
            print('Row',n)
        datastring = ""
        datarow = train.iloc[r].to_dict()
        datastring += str(int(datarow['target']))


        for i, x in enumerate(catdict.keys()):
            if(catdict[x]==0):
                datastring = datastring + " "+str(i)+":"+ str(i)+":"+ str(datarow[x])
            else:
                if(x not in catcodes):
                    catcodes[x] = {}
                    currentcode +=1
                    catcodes[x][datarow[x]] = currentcode
                elif(datarow[x] not in catcodes[x]):
                    currentcode +=1
                    catcodes[x][datarow[x]] = currentcode

                code = catcodes[x][datarow[x]]
                datastring = datastring + " "+str(i)+":"+ str(int(code))+":1"
        datastring += '\n'
        text_file.write(datastring)
        
noofrows = test.shape[0]
noofcolumns = len(features)
with open("alltestffm.txt", "w") as text_file:
    for n, r in enumerate(range(noofrows)):
        if((n%100000)==0):
            print('Row',n)
        datastring = ""
        datarow = test.iloc[r].to_dict()
        datastring += str(int(datarow['target']))


        for i, x in enumerate(catdict.keys()):
            if(catdict[x]==0):
                datastring = datastring + " "+str(i)+":"+ str(i)+":"+ str(datarow[x])
            else:
                if(x not in catcodes):
                    catcodes[x] = {}
                    currentcode +=1
                    catcodes[x][datarow[x]] = currentcode
                elif(datarow[x] not in catcodes[x]):
                    currentcode +=1
                    catcodes[x][datarow[x]] = currentcode

                code = catcodes[x][datarow[x]]
                datastring = datastring + " "+str(i)+":"+ str(int(code))+":1"
        datastring += '\n'
        text_file.write(datastring)<jupyter_output>Row 0
Row 100000
Row 200000
Row 300000
Row 400000
Row 500000
Row 0
Row 100000
Row 200000
Row 300000
Row 400000
Row 500000
Row 600000
Row 700000
Row 800000<jupyter_text>Once you have built the libffm just use* ./ffm-train alltrainffm.txt* ./ffm-predict alltestffm.txt alltrainffm.txt.model output.txt We haven't shuffled test to we can just create the submission as follows<jupyter_code># sub = pd.read_csv('../input/sample_submission.csv')
# outputs = pd.read_csv('output.txt',header=None)
# outputs.columns = ['target']
# sub.target = outputs.target.ravel()
# sub.to_csv('libffmsubmission.csv',index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13370102700646633, 'fraction_numerical': 0.06428299733739064, 'mean_word_length': 3.300081766148814, 'pattern_counts': {'":': 8, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '19369', 'license': 'MIT', 'n_tokens_mistral': 2236, 'n_tokens_neox': 1941, 'n_words': 458, 'path': '10 poer sugero safe driver prediction/libffm-generator-lb-280.ipynb', 'repo_name': 'MLVPRASAD/KaggleProjects', 'size': '5258'}