<jupyter_start><jupyter_text>In this tutorial, you will learn how to store and process structured data.  How to deal with a complicate data processing tasks?  Never solve problems manually The people do data have one principle:   Unless someone put the knife on your neck, don't do any  match, copy, paste and other mechanical process for each piece of data manually           When we encounter any complicate problems, we will try our best to change them into programming problems that can be solved  How should data processing tasks be decomposed? When encountering a data processing task, the first and the most important thing we should consider is:    What are the inputs and outputs of this data processing task?   More specifically, what kind of data do we have? How is the data stored? What results are we going to get? What is the form of the resulting data?   After understanding the input and output, the next step is to divide the task into several small tasks:  The output of each small task is the input of the next task  Store data as a data table In data processing task, the most commonly used way is to store data in the form of data table, that is structured data.The data stored in the form of tables can be easily processed by SQL or Python pandas and other tools with high processing efficiency.Moreover, the advantage of data tables is that each column can be added with one dimension of information.In other words, if there are multiple tables of similar structure, you can concat the tables into one table, and add a column indicating which table the data comes from  Example  Data processing task For example:Suppose there is a batch of subway travel data, the data form is as follows:In this, each row of data represents a passenger's trip, and each number represents a different subway station. A trip will pass through a series of subway stationsFrom this Metro travel data, our data processing task is:>Extract the transfer volume of each metro transfer stationThat is to say, if metro station 1 is connected to line 1, line 2 and line 3, we want to get the flow of interchange between these three lines through station 1.If we want to get all transfer volume of all transfer stations in the city, how to calculate?  A possible solution  InputThe input we already know is the data form aboveHowever, the input data is not a very perfect structured data, because the length of each row is different, and a row contains information of multiple trajectory points.The input we expect is the table data with perfect structured, like this:The advantage of data table is that we can add a column to store the information with a new dimension.We don't need to save multiple tables for each person's travel trajectory. Instead, we add a column as the trip number to distinguish different trips  OutputWhat is the output? Very simple, the transfer volume in all directions of all transfer stations in the citySo, in order to contain this information, how should we design the output data table?The information to be included is, at which station, from which line, transfer to which line, and how many people    Decompose the taskAfter understanding the input and output, the task can be decomposed into the following four tasks:>Task 1: reorganize the original input into perfect structured data to get Table 1  Task 2: Generate Table 2 from Table 1, and arrange them in order, to see which metro lines each trip has traveled through  Task 3: Generate Table 3 from Table 2, collect the transferring information, to see each trip is transferring from which line to which line at which station.   Task 4: Generate Table 4 from Table 3, aggregate    Home work  Dynamic repeatability of buslines  For a bus line, suppose that there are $n$ bus stations,then there will be $C^2_n=\frac{n(n-1)}{2}$ station pairs. If most of the station pairs of this bus line are the same as the station pairs of the other lines, then it is a huge waste of the public tansit resource, since its service can be replace by the other bus line. Here, we define the **Dynamic repeatability of buslines** as:$$D_r=\frac{\sum_{i<j}{r_{ij}}}{\frac{n(n-1)}{2}}$$  where $r_{ij}= 1$ if there are other buslines connecting station $i$ and $j$;$r_{ij}= 0$ if there is no another busline connecting station $i$ and $j$<jupyter_code>#Read data
import pandas as pd
f = open(r'data-sample/busline.csv')
busline = pd.read_csv(f)
f.close()
busline.head(5)
len(busline)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0401965163019205, 'fraction_numerical': 0.005359535506922733, 'mean_word_length': 4.584788029925187, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '96355', 'license': 'MIT', 'n_tokens_mistral': 1066, 'n_tokens_neox': 1023, 'n_words': 733, 'path': '6-structure_en.ipynb', 'repo_name': 'Jame0910/pygeo-tutorial', 'size': '4478'}