<jupyter_start><jupyter_text><jupyter_code>import io
import os
import pandas as pd
data_path = 'E:\\BaiduYunDownload\\optiondata3\\'<jupyter_output><empty_output><jupyter_text>Definitions* Underlying	The stock, index, or ETF symbol* Underlying_last	The last traded price at the time of the option quote.* Exchange	The exchange of the quote – Asterisk(*) represents a consolidated price of all exchanges and is the most common value.** Optionsymbol	The option symbol. Note that in the format starting 2010 this will be longer than 18 characters, depending on the length of the underlying. Blank This item is always blank, to preserve continuity with the older format. It is always blank. So if you are importing this into a database, either do not import this column, or make the field nullable.* Optiontype	Call or put Expiration The expiration date of the option.* Expiration date	The date of the expiration* Quotedate	The date and time of the quote. Most of the time, the time will be 4:00 PM. This only means that it is at the close, even though some options trade until 4:15 PM EST* Strike	The strike of the option* Last	The last traded price of the option which could even be from a previous day.* Bid	The bid price of the option* Ask	The ask price of the option* Volume	The number of contracts traded* Open interest	Open Interest – always a day behind. The OCC changes this number at 3:00AM every morning and the number does not change through the day* BELOW THIS LINE, THESE COLUMNS NOT CONTAINED IN BARE BONES PRODUCTS* Implied volatility	The implied volatility (a measure of the estimate of how much the price could change. A high number means that traders believe the option could make a large change)* Delta	The delta. (a measure of how much the option price would change in relation to the underlying stock price. A delta of .50 means the option would change 50 cents for every 1 dollar the stock moves)* Gamma	The gamma. (a measure of how fast the Delta will change when the stock price changes. A high number means this is a very explosive option, and could gain or loss value quickly)* Theta	The theta (a measure of how fast the option is losing value per day due to time decay. As the expiration day arrives, the theta increases)* Vega	The vega (a measure of how sensitive the option price is to a change in the implied volatility. Options that are way out of the money, or have a long time until expiration are more sensitive to a change in implied volatility)* Alias	If possible, the old name of the option. Because of the 2010 OSI Symbology, it is important to know what the old symbol name was during the 2010 switch over. If this can be determined, it will list the old name, otherwise it will display the same value as the option symbol. The Alias column has no usage outside of 2010.<jupyter_code>columns= ['UnderlyingSymbol','UnderlyingPrice','Exchange','OptionSymbol','Blank','Type','Expiration', 'DataDate','Strike','Last','Bid','Ask','Volume','OpenInterest','IV','Delta','Gamma','Theta','Vega','Alias']
print(columns)
test=pd.read_csv(data_path+"\\201801\\options_20180102.csv", header=None, 
                 names=columns)
symbols = ['ADSK', 'ANTM', 'AZPN', 'CACC', 'CHDN', 'CHE', 'CI', 'COO', 'DENN',
       'FDEF', 'FICO', 'FIS', 'FIX', 'HELE', 'HIFS', 'ICUI', 'IHC',
       'INTU', 'IT', 'JKHY', 'KAI', 'KWR', 'LFUS', 'LII', 'LION', 'LOW',
       'MCO', 'MLAB', 'MMSI', 'MOH', 'MSFT', 'MTD', 'MTN', 'NSP', 'OLBK',
       'OMCL', 'ORLY', 'PRSC', 'RDI', 'RHT', 'RMD', 'SBAC', 'SCHW', 'SCI',
       'SHEN', 'SHW', 'STE', 'TDY', 'TFX', 'TMO', 'TSS', 'ULTI', 'UNH',
       'USPH', 'VLO', 'VRSN', 'ZBRA']
df= None
for path in os.listdir(data_path):
    for file in os.listdir(data_path+'/'+path):
        print('reading file'+file)
        df_one = pd.read_csv(data_path+'/'+path+'/'+file,
                            header=None, names=columns)
        df_one = df_one[df_one['UnderlyingSymbol'].isin(symbols)]
        print(df_one.shape)
        if df is None:
            df= df_one
            print(df.shape)
            continue
        #print(df_one.head())        
        df = pd.concat([df,df_one],axis=0)
        print(df.shape)
df.shape
df.head()
df.to_csv(data_path+'/option_data_selected.csv',index = False)
test
test_aap = test[test['UnderlyingSymbol'].isin(symbols)]
test_aap<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10512997469519209, 'fraction_numerical': 0.011042097998619738, 'mean_word_length': 4.145562130177515, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '406371', 'license': 'MIT', 'n_tokens_mistral': 1340, 'n_tokens_neox': 1260, 'n_words': 584, 'path': '01_Data_Preprocessing/1_2_2_data_convert_part1.ipynb', 'repo_name': 'zjzsu2000/CMPE297_AdvanceDL_Project', 'size': '4347'}