<jupyter_start><jupyter_text>Importing Other Datatypes  1. Working with OS Package<jupyter_code>import os

wd = os.getcwd()
lis = os.listdir(wd)
print(lis)<jupyter_output>['.ipynb_checkpoints', 'datasets', 'Importing Flat Files.ipynb', 'Importing Other Datatypes.ipynb']<jupyter_text>2. Saving a serialised file using Pickle and importing itThere are a number of datatypes that cannot be saved easily to flat files, such as lists and dictionaries. If you want your files to be human readable, you may want to save them as text files in a clever manner. JSONs, which you will see in a later chapter, are appropriate for Python dictionaries.However, if you merely want to be able to import them into Python, you can serialize them. All this means is converting the object into a sequence of bytes, or a bytestream.In this exercise, you'll import the pickle package, open a previously pickled data structure from a file and load it.<jupyter_code># Import pickle package
import pickle

# Define a dictionary
dic = {'June': '69.4', 'Aug': '85', 'Airline': '8', 'Mar': '84.4'}

# Save the dictionary to a pickle file
pickle_out = open("datasets/dic.pkl","wb")
pickle.dump(dic, pickle_out)
# Open pickle file and load data: d
with open('datasets/dic.pkl', "rb") as file:
    d = pickle.load(file)

# Print d
print(d)

# Print datatype of d
print(type(d))<jupyter_output>{'June': '69.4', 'Aug': '85', 'Airline': '8', 'Mar': '84.4'}
<class 'dict'><jupyter_text>3. Importing Excel SpreadsheetsWhether you like it or not, any working data scientist will need to deal with Excel spreadsheets at some point in time. You won't always want to do so in Excel, however!Here, you'll learn how to use pandas to import Excel spreadsheets and how to list the names of the sheets in any loaded .xlsx file.Recall from the video that, given an Excel file imported into a variable spreadsheet, you can retrieve a list of the sheet names using the attribute spreadsheet.sheet_names.Specifically, you'll be loading and checking out the spreadsheet 'battledeath.xlsx', modified from the Peace Research Institute Oslo's (PRIO) dataset. This data contains age-adjusted mortality rates due to war in various countries over several years.<jupyter_code># Import pandas
import pandas as pd

# Assign spreadsheet filename: file
file = "datasets/battledeath.xlsx"

# Load spreadsheet: xl
xls = pd.ExcelFile(file)

# Print sheet names
print(xls.sheet_names)<jupyter_output>['2002', '2004']<jupyter_text>In the previous exercises, you saw that the Excel file contains two sheets, '2002' and '2004'. The next step is to import these.In this exercise, you'll learn how to import any given sheet of your loaded .xlsx file as a DataFrame. You'll be able to do so by specifying either the sheet's name or its index.The spreadsheet 'battledeath.xlsx' is already loaded as xls.<jupyter_code># Load a sheet into a DataFrame by name: df1
df1 = xls.parse("2004")

# Print the head of the DataFrame df1
print(df1.head())

# Load a sheet into a DataFrame by index: df2
df2 = xls.parse(0)

# Print the head of the DataFrame df2
print(df2.head())<jupyter_output>War(country)      2004
0  Afghanistan  9.451028
1      Albania  0.130354
2      Algeria  3.407277
3      Andorra  0.000000
4       Angola  2.597931
  War, age-adjusted mortality due to       2002
0                        Afghanistan  36.083990
1                            Albania   0.128908
2                            Algeria  18.314120
3                            Andorra   0.000000
4                             Angola  18.964560<jupyter_text>Here, you'll parse your spreadsheets and use additional arguments to skip rows, rename columns and select only particular columns.The spreadsheet 'battledeath.xlsx' is already loaded as xls.As before, you'll use the method parse(). This time, however, you'll add the additional arguments skiprows, names and usecols. These skip rows, name the columns and designate which columns to parse, respectively. All these arguments can be assigned to lists containing the specific row numbers, strings and column numbers, as appropriate.<jupyter_code># Parse the first sheet and rename the columns: df1
df1 = xls.parse(0, skiprows=[0], names=["Country", "AAM due to War (2002)"])

# Print the head of the DataFrame df1
print(df1.head())

# Parse the first column of the second sheet and rename the column: df2
df2 = xls.parse(1, usecols=[0], skiprows=[0], names=["Country"])

# Print the head of the DataFrame df2
print(df2.head())<jupyter_output>Country  AAM due to War (2002)
0              Albania               0.128908
1              Algeria              18.314120
2              Andorra               0.000000
3               Angola              18.964560
4  Antigua and Barbuda               0.000000
               Country
0              Albania
1              Algeria
2              Andorra
3               Angola
4  Antigua and Barbuda<jupyter_text>4. Importing SAS filesIn this exercise, you'll figure out how to import a SAS file as a DataFrame using SAS7BDAT and pandas.The data are adapted from the website of the undergraduate text book Principles of Econometrics by Hill, Griffiths and Lim.<jupyter_code>import matplotlib.pyplot as plt

# Import sas7bdat package
from sas7bdat import SAS7BDAT

# Save file to a DataFrame: df_sas
with SAS7BDAT('datasets/sales.sas7bdat') as file:
    df_sas = file.to_data_frame()

# Print head of DataFrame
print(df_sas.head())

# Plot histogram of DataFrame features (pandas and pyplot already imported)
pd.DataFrame.hist(df_sas[['P']])
plt.ylabel('count')
plt.show()<jupyter_output>YEAR     P           S
0  1950.0  12.9  181.899994
1  1951.0  11.9  245.000000
2  1952.0  10.7  250.199997
3  1953.0  11.3  265.899994
4  1954.0  11.2  248.500000<jupyter_text>5. Importing STATA filesThe pandas package has been imported in the environment as pd and the file disarea.dta is in your working directory. The data consist of disease extents for several diseases in various countries (more information can be found here).<jupyter_code># Load Stata file into a pandas DataFrame: df
df = pd.read_stata("datasets/disarea.dta")

# Print the head of the DataFrame df
print(df.head())

# Plot histogram of one column of the DataFrame
pd.DataFrame.hist(df[['disa10']])
plt.xlabel('Extent of disease')
plt.ylabel('Number of countries')
plt.show()<jupyter_output>wbcode               country  disa1  disa2  disa3  disa4  disa5  disa6  \
0    AFG           Afghanistan   0.00   0.00   0.76   0.73    0.0   0.00   
1    AGO                Angola   0.32   0.02   0.56   0.00    0.0   0.00   
2    ALB               Albania   0.00   0.00   0.02   0.00    0.0   0.00   
3    ARE  United Arab Emirates   0.00   0.00   0.00   0.00    0.0   0.00   
4    ARG             Argentina   0.00   0.24   0.24   0.00    0.0   0.23   

   disa7  disa8  ...  disa16  disa17  disa18  disa19  disa20  disa21  disa22  \
0   0.00    0.0  ...     0.0     0.0     0.0    0.00    0.00     0.0    0.00   
1   0.56    0.0  ...     0.0     0.4     0.0    0.61    0.00     0.0    0.99   
2   0.00    0.0  ...     0.0     0.0     0.0    0.00    0.00     0.0    0.00   
3   0.00    0.0  ...     0.0     0.0     0.0    0.00    0.00     0.0    0.00   
4   0.00    0.0  ...     0.0     0.0     0.0    0.00    0.05     0.0    0.00   

   disa23  disa24  disa25  
0    0.02    0.00    0.00  
1    0[...]<jupyter_text>6. Importing HDF5 filesIn this exercise, you'll import it using the h5py library. You'll also print out its datatype to confirm you have imported it correctly. You'll then study the structure of the file in order to see precisely what HDF groups it contains.<jupyter_code># Import packages
import h5py
import numpy as np

# Assign filename: file
file = "datasets/LIGO_data.hdf5"

# Load file: data
data = h5py.File(file, "r")

# Print the datatype of the loaded file
print(type(data))

# Print the keys of the file
for key in data.keys():
    print(key)<jupyter_output><class 'h5py._hl.files.File'>
meta
quality
strain<jupyter_text>Extracting data from your HDF5 fileIn this exercise, you'll extract some of the LIGO experiment's actual data from the HDF5 file and you'll visualize it.<jupyter_code># Get the HDF5 group: group
group = data["strain"]

# Check out keys of group
for key in group.keys():
    print(key)
# Set variable equal to time series data: strain
strain = data["strain"]["Strain"].value

# Set number of time points to sample: num_samples
num_samples = 10000

# Set time vector
time = np.arange(0, 1, 1/num_samples)

# Plot data
plt.plot(time, strain[:num_samples])
plt.xlabel('GPS Time (s)')
plt.ylabel('strain')
plt.show();<jupyter_output><empty_output><jupyter_text>7. Importing MATLAB files Loading .mat filesIn this exercise, you'll figure out how to load a MATLAB file using scipy.io.loadmat() and you'll discover what Python datatype it yields.The file 'gene_expression.mat' contains [gene expression data](https://www.mcb.ucdavis.edu/faculty-labs/albeck/workshop.htm) from the Albeck Lab at UC Davis. You can find the data and some great documentation [here](https://www.mcb.ucdavis.edu/faculty-labs/albeck/workshop.htm).<jupyter_code># Import package
import scipy.io

# Load MATLAB file: mat
mat = scipy.io.loadmat("datasets/gene_expressions.mat")

# Print the datatype type of mat
print(type(mat))<jupyter_output><class 'dict'><jupyter_text>The structure of .mat in PythonHere, you'll discover what is in the MATLAB dictionary that was loaded previously.<jupyter_code># Print the keys of the MATLAB dictionary
print(mat.keys())
# Print the type of the value corresponding to the key 'CYratioCyt'
print(type(mat["CYratioCyt"]))
# Print the shape of the value corresponding to the key 'CYratioCyt'
print(mat["CYratioCyt"].shape)
# Subset the array and plot it
data = mat['CYratioCyt'][25, 5:]
fig = plt.figure()
plt.plot(data)
plt.xlabel('time (min.)')
plt.ylabel('normalized fluorescence (measure of expression)')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08696089273147682, 'fraction_numerical': 0.05790690660500653, 'mean_word_length': 3.1588628762541804, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '462172', 'license': 'MIT', 'n_tokens_mistral': 3559, 'n_tokens_neox': 3244, 'n_words': 1229, 'path': 'Datacamp Assignments/Data Science Track/Data Importing/2. Importing Other Datatypes.ipynb', 'repo_name': 'Ali-Parandeh/Data_Science_Playground', 'size': '9947'}