<jupyter_start><jupyter_text>Understanding how the use of righ data structure helps in Code Optimization The task we are about to perform is to find commond elements between two datapoints. > Datasets are:>> 1. all_coding_books.txt: contains the id of all the books written about coding in last two years>> 2. books_published_last_two_years.txt: contains the id of all books written in the last two years > The first easily understood method that comes to our mind will be to store the data in two lists, compare the lists using for loop<jupyter_code>#Import libraries
import time
import pandas as pd
import numpy as np
#Store both the file data to a csv as a list
with open('books_published_last_two_years.txt') as f:
    recent_books = f.read().split('\n')
    
with open('all_coding_books.txt') as f:
    coding_books = f.read().split('\n')
start = time.time()
recent_coding_books = []

for book in recent_books:
    if book in coding_books:
      recent_coding_books.append(book)
print("There are " + str(len(recent_coding_books)) + " coding books released recently")
print('Duration: {} seconds'.format(time.time() - start))<jupyter_output>There are 96 coding books released recently
Duration: 11.54421329498291 seconds<jupyter_text>Let's try using the data structure of sets instead of lists to improve the speed of search> Note: While using sets we can also use the vector function intersection to check for common elements.<jupyter_code>start = time.time()
recent_coding_books = set(recent_books).intersection(coding_books)
print("There are " + str(len(recent_coding_books)) + " coding books released recently")
print('Duration: {} seconds'.format(time.time() - start))<jupyter_output>There are 96 coding books released recently
Duration: 0.011107206344604492 seconds
{'fraction_non_alphanumeric': 0.08169014084507042, 'fraction_numerical': 0.023098591549295774, 'mean_word_length': 5.458181818181818, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '468143', 'license': 'MIT', 'n_tokens_mistral': 543, 'n_tokens_neox': 499, 'n_words': 204, 'path': 'Data_Structures_in_code_optimization_.ipynb', 'repo_name': 'LakshmiPrasannan/Data-Structures-in-Code-Optimization', 'size': '1775'}