<jupyter_start><jupyter_text>Your title for your regression project  Project Planning Goals(see background)...your customer is the zillow data science team.  state your goals as if you were delivering this to zillow.  They have asked for something from you (see 'background') and you are basically communicating in a more concise way, and very clearly, the goals as you understand them and as you have taken and acted upon through your research. *Project OverviewBackground:Zillow wants to improve their Zestimate. The zestimate is estimated value of a home. Zillow theorizes that there is more information to be gained to improve its existing model. Because of that, Zillow wants you to develop a model to predict the error between the Zestimate and the sales price of a home. In predicting the error, you will discover features that will help them improve the Zestimate estimate itself. Your goal of this project is to develop a linear regression model that will best predict the log error of the Zestimate. The error is the difference of the sales price and the Zestimate. The log error is computed by taking the log function of that error. You don't need to worry about the fact that the error is of a logarithmic function. It is a continuous number that represents an error rate.*  DeliverablesWhat should the zillow team expect to receive from you?  Again, as you were communicating to them, not to your instructors.  *your deliverables:*1. *A report (in the form of a presentation, both verbal and through a slides) that summarizes your findings about the drivers of the Zestimate error. This will come from the analysis you do during the exploration phase of the pipeline. In the report, you will have charts that visually tell the story of what is driving the errors.*2. *A Jupyter notebook, titled 'Regression_Proj_YourName', that contains a clearly labeled section and code clearly documented for each the stages below (project planning, data acquisition, data prep, exploration, and modeling). All of the work will take place in your jupyter notebook.*  Data Dictionary & Details**we will be using 2016 data so do NOT use properties_2017 or predictions_2017**  Data dictionary of fields you will use. Why? So that you can refer back and others can refer to the meanings as you are developing your model. This is about gaining knowledge in the domain space so that you will understand when data doesn't look right, be able to more effectively develop hypotheses, and use that domain knowledge to build a more robust model (among other reasons)Define your fields.   You may be able to to some ahead of time, but you may need to return to this section after you have explored the data and understand more about what each field means.  Also, look for a data dictionary either in the database or in the original data source (kaggle.com). You are free to use more fields than is mentioned, but I would recommend not expanding it to too many fields in the beginning as it will add complexity and you want to make sure you get an initial version completed before diving in deeper.  - logerror- bathroomcnt- bedroomcnt- calculatedfinishedsquarefeet- fullbathcnt- garagecarcnt- roomcnt- yearbuilt- taxvaluedollarcnt- taxamount  Initial Thoughts, Ideas, HypothesesBrainstorming ideas, hypotheses, related to how variables might impact or relate to each other, both within independent variables and between the independent variables and dependent variable, and also related to any ideas for new features you may have while first looking at the existing variables and challenge ahead of you.  Prepare your environmentimport libraries you will use throughout the project.  You may need to add to this as you go.  But try to keep all of your imports in this section  Data Acquisition Acquire via csv- add the path to where your files are located in your env.py file. - ensure that others can read the files based on their local env.py file. - Filter out any from predictions_2016 that do not have a logerror. - Generate a cohesive data set that includes the following fields:- `logerror`- `bathroomcnt`- `bedroomcnt`- `calculatedfinishedsquarefeet`- `fullbathcnt`- `garagecarcnt`- `roomcnt`- `yearbuilt`- `taxvaluedollarcnt`- `taxamount`- (optional) `regionidzip`recommendations for dealing with the large dataset- remove any without a logerror- sample until you have the right query (and then pull entire dataset)- export to local csv - read from local csv (sample, but larger sample than your first) - save sql query to add into to_sql() function in python.   Summarize the data. Why? So you can confirm the data look like what you would expect.- peek at a few rows- data types- summary stats- column names- number of rows and columns- ...  Data PreparationGoal: leave this section with a dataset that is ready to be analyzed.   Create a variable, `colnames`, that is a list of the column names. Why? You will likely reference this variable later.  Identify the data types of each variable. Why? You need to make sure they are what makes sense for the data and the meaning of the data that lies in that variable. If it does not, make necessary changes.  Identify the columns that have missing values and the number of missing values in each column. Why? Missing values are going to cause issues down the line so you will need to handle those appropriately. For each variable with missing values, if it makes sense to replace those missing with a 0, do so. For those where that doesn't make sense, decide if you should drop the entire observations (rows) that contain the missing values, or drop the entire variable (column) that contains the missing values.  Create a list of the independent variable names (aka attributes) and assign it to `x_vars` Why? During exploration, you will likely use this list to refer to the attribute names.  Clearly identify your dependent (target) variable. - What is the name of the variable? - Is it discrete or continuous?  View the distribution of the numeric variables- plot a histogram and box plot of each variable. - Why? To see the distribution, skewness, outliers, and unit scales. You will use this information in your decision of whether to normalize, standardize or neither.  BONUS:  Data standardization Create a new data frame that is the min-max normalization of the independent variable in the original data frame (+ the original dependent variable). You will normalize each of the numeric independent variables independently, i.e. using the min and max of each variable, not the min/max of the whole dataframe. Why? Regression is very sensitive to difference in units. It will be almost impossible to extract a meaningful linear regression model with such extreme differences in scale. For more context, see: https://medium.com/@rrfd/standardize-or-normalize-examples-in-python-e3f174b65dfcYou will use this dataframe in the future analysis.  Data ExplorationGoal is to address each of the questions you posed in your planning & brainstorming through visual or statistical analysis.When you have completed this step, you will have the findings from your analysis that will be used in your final report, the answers to your questions and your customer's questions that will reach the goal.   Split data into training and test datasets  Create a jointplot for each independent variable (normalized version) with the dependent variable. Be sure you have Pearson's r and p-value annotated on each plot.  Create a feature plot using seaborn's PairGrid() of the interaction between each variable (dependent + independent). You may want to use a normalized dataframe (if you did that) or adjust the axis scale (set to logarithmic, e.g) so you can more clearly view the interactions.  Create a heatmap of the correlation between each variable pair.  Summarize your conclusions from all of these steps.    Is the logerror significantly different for homes with 3 bedrooms vs those with 5 or more bedrooms? Run a t-test to test this difference.  Do the same for another 2 samples you are interested in comparing (e.g. those with 1 bath vs. x baths)<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.03726478907883409, 'fraction_numerical': 0.003566596974541877, 'mean_word_length': 4.528212100611829, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '41264', 'license': 'MIT', 'n_tokens_mistral': 1895, 'n_tokens_neox': 1805, 'n_words': 1282, 'path': 'backups/Project-Template.ipynb', 'repo_name': 'ecdedios/codeup-ds-projects', 'size': '8131'}