<jupyter_start><jupyter_text>{{ cookiecutter.project_name }} *Welcome to the Ryan's Codeup Data Science MVP template! The sections in this notebook are here to help you organize your data science project into a clear workflow. The text in the markdown cells is there to help you understand the goals at each stage. Follow their instructions to move your work along, and then delete the prompts when you're done.**Finish the mvp workflow and then go back and think critically about what you might of missed. The point isn't to doubt yourself; the purpose here is to check your blind spots and see if you can't find more information or insights that will help you to deliver better results.**Better yet, and this cannot be stressed enough, don't ask **yourself** these questions, but bring them up in conversation with peers, experts in other fields, or even complete strangers -- anyone with a different point of view is going to be able to help you to see what things you are taking for granted.*  Summary*What are you going to do and how are you going to do it? Write out your thoughts here in a way that you can easily explain your work to members on your team.**This is your space to put together the elevator pitch for your project, and then follow it up with a plan of attack. It's going to be short, but that doesn't mean you won't need to spend much time putting it together -- this is a process of dropping the bad ideas until you're left with something that you are confident you can work with.**Don't take shortcuts here; coming up with a useful question and a straightforward work plan is going to make the rest of your project flow much more smoothly from start to finish.**One final, and very important, point: the main reason for thinking about this stage as how you're going to describe your work to others is that you should be talking about work with others!!! The more you work on your project in isolation, the better your ideas will sound to you -- even the bad ones. Listening to your ideas spoken out loud in your own voice is an excellent sanity check, and feedback from your peers is your most valuable resource.***You don't have to answer every question, but answer each that you can now and then come back later. Also, if the question doesn't apply, remove it.**  Problem Statement*Think about the problem you are trying to solve as a business case. After you've built your model, who is going to use it and what will they do with it? Who are your customers and stakeholders and what is their need? The need is not the data they are trying to understand -- think of the need as a specific action they would like to take or a question they want to answer.**What is your model going to **do** in the real world? If you can answer this, and if you keep that answer in mind as you complete your work, you will have a much clearer view of where to go at each step.** Who are your customers?* What is the problem?* What solution do you propose?* How will you know if your work is good?*Also, think critacally here. If someone was paying you to do this work, what would they want you to build.*  Work Plan*Now that you have a clear idea of what you are trying to accomplish, you need to find a data source. Think of this as working through a maze forwards and backwards at the same time. At the start you have any number of data sets available to work with (and the whole Internet to search and scrape), and at the end is the hypothetical data set that would answer your question immediately if you had it.** What data, if you had it, would solve your problem right away?* What data do you have access to?* What additional data would be good to have?* What data would be impossible to collect?* What are the best proxies you can find for unavailable or impossible data?* What are the legal or ethical issues you might run into if you were to try to collect all of the types of data you would like to work with?*Of course you need to make a plan to turn your data into the solutions you need. Think of what type of problem this is, what models are commonly used for those types of problems, what types of data those models require and special considerations that may need to be made. Do your homework and find out what approaches other people are using on similar tasks.** What ML paradigm are you working in? (Classification, Regression, Clustering, etc)* What models are commonly used in this task?* What other solutions are being tried in this field?* What special considerations need to be taken when dealing with these models? (i.e., imbalanced classes, text preprocessing, data leakage, etc)* How will you know that your models work?* How will you recognize and diagnose cases where the predictions are incorrect?  Imports*Keep all your imports in one place. This will make it much easier to see everything that you are using*<jupyter_code>import src.acquire
import src.prepare
import src.explore
import src.model<jupyter_output><empty_output><jupyter_text>Acquire the Data Data Pipeline*This is where you describe the ETL (Extract, Transform, Load) process. Describe the actions you will need to take to obtain the data from where it is stored, and convert it into a clean format. Also be aware of how the available data is likely to change over time -- how often will it be updated, and how long will the available data be relevant?**After completing this step, be sure to edit the README data dictionary to include descriptions of where you obtained your data and what information it contains.*  Create your Pipeline Tools*Your work is going to be far less useful if anyone who wants to use it or build on it is going to have to put together a new data set from scratch. Automate everything that you possibly can. (And if you don't think it can be automated, ask your peers and check Google to see what's out there.)**The pipeline tools will connect to data sources, create local directories to store the data, download the data, clean it, reformat it, and store it. In the following step, "Run the Pipeline," you'll write a function that uses all of these tools to build your data set in one line of code.**Make sure these steps are reproducible by code. Put some thought into the directory structures and filepaths you are using to save your data, so it's easy to load files you need.*<jupyter_code># Write functions that form a data pipeline from your source to data/raw
# Once that is done move the functions to acquire.py<jupyter_output><empty_output><jupyter_text>Prove it Works*Once your tools are built (and are working reliably!!), you'll be able to work much faster if they're out of sight. Collaborators and data scientists who build on your work will also be much more productive if you provide easy access to the data. So do yourself and your team a favor, and package your pipeline tools into a single function that can easily reproduce your working data set. This is what our run function in acquire is for.**Also a helpful thing to remember is to set this function up so that when you re-run the pipeline, it checks whether the files are already available locally. This will save you lots of time downloading large files.*<jupyter_code>src.acquire.run()<jupyter_output>Acquire: downloading raw data files...
Acquire: Completed!<jupyter_text>Prepare the Data*This is the important phase and it is seen as a chore by some, even though I personally enjoy the puzzle that this stage always provides. If you tend to think of data scrubbing as a drag on your workflow, it's well worth your time to explore some new pandas functions you're unfamiliar with and speed up your workflow. Try to learn one or two new tricks with each project you work on.**The primary scientific component to scrubbing data is going to be the thought process around deciding what to do with missing data. There are frequently no right or wrong answers for what to do with missing values, so be sure to record your thoughts, and share your process with peers or collaborators to get a sanity check.*  Load (and Split) the Data*You may already have your data loaded in this notebook after the Acquire stage has been run. But you probably don't want to have to re-run your data gathering pipeline every time you try a new idea for your model. Write some code here to load your cached data set so you won't have to start from scratch.**It is especially important to go back to a clean data set with each iteration that you want to test or add to your model. Your new results will not be reproducible if this iteration is developed on data that has been transformed in any way during the previous iteration.**Lastly if possible split the data at this stage. It is always best to split the data as early as possible so that you are making you decision of preparation on a subset of the whole dataset as the model you are building will be doing on future unseen data*  Clean the Data*Make sure that your data is in a format that your models can work with, and fix any problems that need attention. This may include formatting strings, removing duplicate records, and handling missing data.**There are multiple approaches to handling missing records, including removing them, inferring the values either statistically or from other insights, or finding other data sources to fill the gaps. Be sure to note down your thought process for whatever decision you make.*<jupyter_code># Write functions that prepare the data from your data/raw and store
# the prepared data in data/
# Once that is done move the functions to prepare.py<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.03596821413634463, 'fraction_numerical': 0.0, 'mean_word_length': 4.484518348623853, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '237650', 'license': 'MIT', 'n_tokens_mistral': 2181, 'n_tokens_neox': 2101, 'n_words': 1643, 'path': '{{cookiecutter.project_directory_name}}/mvp.ipynb', 'repo_name': 'RyanMcCall/ryans_codeup_data_science_mvp', 'size': '9564'}