<jupyter_start><jupyter_text>Kolumny do usunięcia* STDs:cervical condylomatosis* STDs:vaginal condylomatosis* STDs:pelvic inflammatory disease* STDs:genital herpes* STDs:molluscum contagiosum* STDs:AIDS* STDs:Hepatitis B* STDs:HPV* Dx:CINPróg jest następujący - poniżej 15 odpowiedzi innych od dominującej i NA.<jupyter_code>pd.options.display.max_columns = None
dane.head()
dane = dane.drop(['STDs:cervical condylomatosis', 
         'STDs:vaginal condylomatosis', 
         'STDs:pelvic inflammatory disease', 
         'STDs:genital herpes',
         'STDs:molluscum contagiosum',
         'STDs:AIDS',
         'STDs:Hepatitis B',
         'STDs:HPV', 'Dx:CIN'], axis=1)
dane.head()<jupyter_output><empty_output><jupyter_text>Uzupełnianie braków w kolumnach kategorycznych Sposób obróbki kolumn: Dodatkowa kategoria na brak danych i wypełnienie braków 0- Smokes- Hormonal Contraceptives- IUD- STDs- STDs (number)- STDs:condylomatosis- STDs:vulvo-perineal condylomatosis- STDs:syphilis- STDs:HIV Zmapowanie wartości >1 - STDs (number) >2 -> 2- STDs: Number of diagnosis >1 -> 1<jupyter_code>def column_nodata(df, column_name):
    df[column_name + "_null"] = df[column_name].apply(lambda x: 1 if pd.isnull(x) else 0)
    df[column_name] = df[column_name].fillna(0)

def replace_in_column(df, column_name, src, dst):
    df[column_name] = df[column_name].replace(to_replace=src, value=dst)

replace_in_column(dane, 'STDs (number)', [3, 4], 2)
replace_in_column(dane, 'STDs: Number of diagnosis', [2,3], 1)

nodata_categories = [
    'Smokes',
    'Hormonal Contraceptives',
    'IUD',
    'STDs',
    'STDs (number)',
    'STDs:condylomatosis',
    'STDs:vulvo-perineal condylomatosis',
    'STDs:syphilis',
    'STDs:HIV'
]

for category in nodata_categories:
    column_nodata(dane, category)
# OHE dla STDs (number)

dane = pd.concat([dane, pd.get_dummies(dane['STDs (number)'], prefix='STDs_')],axis=1)
dane.drop(['STDs (number)'],axis=1, inplace=True)
dane.describe()<jupyter_output><empty_output><jupyter_text>StandaryzacjaPozostawione po uprzedniej analizie wartości NaN w kolumnach 'Number of sexual partners', 'First sexual intercourse', 'Num of pregnancies', 'Smokes (years)', 'Smokes (packs/year)', 'Hormonal Contraceptives (years)', 'IUD (years)' wynikają z tego, że ankietowani prawdopodobnie nie odpowiedzieli na podane pytania.NaN w 'STDs: Time since first diagnosis', 'STDs: Time since last diagnosis' zaś przechowują informację o braku przeprowadzonego badania.<jupyter_code>numerical = ['Age', 'Number of sexual partners', 'First sexual intercourse', 'Num of pregnancies', 'Smokes (years)', 
            'Smokes (packs/year)', 'Hormonal Contraceptives (years)', 'IUD (years)', 'STDs: Time since first diagnosis',
            'STDs: Time since last diagnosis']
len(numerical)
dane[numerical].describe()
scaler = StandardScaler() 
dane_scaled = scaler.fit_transform(dane[numerical])
d2 = pd.DataFrame(dane_scaled, columns = numerical)
dane[numerical] = d2[numerical]
dane
dane.describe()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12784184514003294, 'fraction_numerical': 0.007248764415156508, 'mean_word_length': 5.529032258064516, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '640312', 'license': 'Apache-2.0', 'n_tokens_mistral': 1228, 'n_tokens_neox': 1140, 'n_words': 256, 'path': 'Projekty/Projekt1/Grupa3/StaronSzypulaUrbala/usuwanie_kolumn.ipynb', 'repo_name': 'niladrem/2020L-WUM', 'size': '3035'}