<jupyter_start><jupyter_text>Homework 3 -- > **Due Friday, October 23 by 2:00 PM**. You may submit this assignment in groups of 2. Be sure to put your names above. Also, don't forget to submit your data files. The goal of this homework is to provide a realistic setting for a machine learning task. Therefore instructions will not specify the exact steps to carry out. Instead, it is part of the assignment to identify promising features, models and preprocessing methods and apply them as appropriate.The overall goal is to predict the fuel efficiency of car models based on historical data collected by the department of energy that can be found at https://www.fueleconomy.gov/feg/download.shtml. In particular, the main goal is to predict the 2018 data from the 2015-2017 data. The main performance metric is R^2 and homework grades will partially depend on your test-set score.The measure of fuel efficiency that you should predict is Combined Unrounded adjusted Fuel Economy("Comb Unrd Adj FE - Conventional Fuel"). Do not use any of the tail pipe measurement results, only features of the car provided by the manufacturer, to predict the efficiency. You should read the dataset documentation to figure out what this means -- after all, a huge part of any data scientist's or machine learning engineer's job is understanding the data and where it came from! As a hint, many fuel-related columns, such as thsoe containing 'FE' or 'EPA' are not appropriate to use. You can imagine that in many real-life scenarios these measurements are not yet available when you need to make predictions.Document your process as appropriate, in particular how and when you used the test set. The modelling process is iterative! For example, you may start preprocessing in one way, but by the time you get to training a complicated model realize that a different style of preprocessing is more appropriate, so you return to that step and modify it. However, please lay out the tasks in the below order to facilitate grading.Note: you are free to add extra code blocks as necessary.<jupyter_code># Any necessary setup code that all the below blocks will use.<jupyter_output><empty_output><jupyter_text>Task 1: Identify FeaturesRead in your data and assemble it into a dataframe or numpy arrays. One way to do this is to use Pandas' `read_excel` function to load in each year's data and concatenate dataframes together. Exactly how you do it is up to you.What features are relevant? Are there any that should be excluded because they 'leak' target information, or may not be available in a real-life prediction task?Show visualizations or statistics to support your selection.  Task 2: Preprocessing and Baseline ModelCreate a simple minimum viable model. Use an initial selection of features, doing appropriate preprocessing and cross-validating a lienar model. Feel free to exclude features or do simplified prepcoressing for this task.<jupyter_code>target = df['column name']<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.039091212829936516, 'fraction_numerical': 0.007350484463748747, 'mean_word_length': 4.859099804305283, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 10, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '663349', 'license': 'MIT', 'n_tokens_mistral': 708, 'n_tokens_neox': 667, 'n_words': 461, 'path': 'courses/comp347/f20/hwk3.ipynb', 'repo_name': 'robertutterback/robertutterback.github.io', 'size': '2993'}