<jupyter_start><jupyter_text>Features can be generated from the terminal using the following command:```python keplerml.py data/filelists/ex_filelist.txt data/lightcurves/ data/output/Example_output.p ```This requires a Python 3+ to be the default version, replace `python` with an appropriate version if this is not the case, for example, use `python3.7`The above terminal command is equivalent to the following:<jupyter_code>path_to_filelist = './data/filelists/ex_filelist.txt'
path_to_fits = './data/lightcurves/'
output_file = 'data/output/Example_output.p'
features = fc.features_from_filelist(path_to_filelist,path_to_fits,output_file,fl_as_array=False,verbose=True,prime_feats=False)<jupyter_output>Reading ./data/filelists/ex_filelist.txt...
Using 47 cpus to calculate features...
Importing 247 lightcurves...
Lightcurve import took 0:00:01.129264
Processing 247 files...
247/247 completed. Time for chunk: 0:01:06.296283
Features have been calculated, total time to calculate features: 0:01:06.308606
Saving output to data/output/Example_output.p
Cleaning up...
Done.<jupyter_text>Features are returned in a Pandas DataFrame, and saved to the specified output file as a pickled dataframe, which can be read in using the pickle module<jupyter_code>features.head()
import pickle
output_file = 'data/output/Example_output.p'
with open(output_file,'rb') as f:
    feats = pickle.load(f)
feats.head()<jupyter_output><empty_output><jupyter_text>The features are optimized using the `@njit` decorator from the `numba` package. To make full use of this, the code to be optimized by numba needs to run once. This can be done manually as follows:<jupyter_code>lc_path = './data/lightcurves/kplr001026032-2011271113734_llc.fits'
lc = fc.import_lcs(lc_path)
t = lc[1]
nf = lc[2]
err = lc[3]
lc_feats = fc.feats(t,nf,err)<jupyter_output><empty_output><jupyter_text>After priming the feature calculation, the features for a filelist can be run in the same way as before.As a note, the `features_from_filelist` method will run a primer by default using the first lightcurve. Whether primed manually as above, with the default, or even run specifically without priming, runs following the first will be optimized by `numba` and be quicker.Note the drastically improved runtime following the manual priming above:<jupyter_code>path_to_filelist = './data/filelists/ex_filelist.txt'
path_to_fits = './data/lightcurves/'
output_file = 'data/output/Example_output.p'
features = fc.features_from_filelist(path_to_filelist,path_to_fits,output_file,fl_as_array=False,verbose=True,prime_feats=False)<jupyter_output>Reading ./data/filelists/ex_filelist.txt...
Using 47 cpus to calculate features...
Importing 247 lightcurves...
Lightcurve import took 0:00:01.126179
Processing 247 files...
247/247 completed. Time for chunk: 0:00:12.281421
Features have been calculated, total time to calculate features: 0:00:12.294860
Saving output to data/output/Example_output.p
Cleaning up...
Done.<jupyter_text>The file list can also be fed into the feature calculator as a list of filenames, produced however. Two examples below.<jupyter_code>path_to_filelist = './data/filelists/ex_filelist.txt'
with open(path_to_filelist,'r') as f:
    files = f.read().splitlines()
path_to_fits = './data/lightcurves/'
output_file = 'data/output/Example_output.p'
feats = fc.features_from_filelist(files,path_to_fits,output_file,fl_as_array=True,verbose=True)
import os
path_to_fits = './data/lightcurves/'
files = os.listdir('data/lightcurves')
output_file = 'data/output/Example_output.p'
feats = fc.features_from_filelist(files,path_to_fits,output_file,fl_as_array=True,verbose=True)<jupyter_output>Using 47 cpus to calculate features...
Importing 247 lightcurves...
Lightcurve import took 0:00:01.874902
Processing 247 files...
247/247 completed. Time for chunk: 0:00:14.005037
Features have been calculated, total time to calculate features: 0:00:14.058610
Saving output to data/output/Example_output.p
Cleaning up...
Done.<jupyter_text>Saving as a Cluster Outlier Object<jupyter_code>import clusterOutliers as coo
"""
!!! DOES NOT WORK FOR THIS EXAMPLE DATA
!!! CLUSTER OUTLIER OBJECT IS DESIGNED TO WORK ON LARGE DATASETS, EXAMPLE IS TOO SMALL
"""
example_coo = coo.clusterOutliers(feats=feats,fitsDir=path_to_fits,output_file='example.coo')<jupyter_output><empty_output><jupyter_text>Speed tests<jupyter_code>from datetime import datetime
start = datetime.now()
lc_path = './data/lightcurves/kplr001026032-2011271113734_llc.fits'
lc = fc.import_lcs(lc_path)
t = lc[1]
nf = lc[2]
err = lc[3]
lc_feats = fc.feats(t,nf,err)
print("Time to prime: {}".format(datetime.now()-start))
%%timeit
lc_path = './data/lightcurves/kplr001026032-2011271113734_llc.fits'
lc = fc.import_lcs(lc_path)
t = lc[1]
nf = lc[2]
err = lc[3]
lc_feats = fc.feats(t,nf,err)
%%timeit
lc_feats = fc.feats(t,nf,err)
%%timeit
lc = fc.import_lcs(lc_path)<jupyter_output>19.1 ms ± 1.24 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
{'fraction_non_alphanumeric': 0.10512562814070352, 'fraction_numerical': 0.04582914572864322, 'mean_word_length': 6.058156028368795, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '249098', 'license': 'MIT', 'n_tokens_mistral': 1888, 'n_tokens_neox': 1704, 'n_words': 463, 'path': 'Feature Calculation Example.ipynb', 'repo_name': 'd-giles/KeplerML', 'size': '4975'}