<jupyter_start><jupyter_text>Feature engineering notebookDerive the full dataset—with hand-engineered and LDA-derived title/description features—from used laptop listings data pulled from the eBay API. Start with the abridged dataset (= no title/description features). First we get the hand-engineered features.<jupyter_code>import call_abridged_data

laptopData = call_abridged_data.read_data()
laptopData.head()
import get_hand_engineered_features

laptopData = get_hand_engineered_features.get_year_mention_features(laptopData)
laptopData.head()
laptopData = get_hand_engineered_features.get_size_mention_features(laptopData)
laptopData = get_hand_engineered_features.get_memory_mention_features(laptopData)
laptopData.head()
laptopData = get_hand_engineered_features.get_adjective_mention_features(laptopData)
laptopData = get_hand_engineered_features.get_length_features(laptopData)
laptopData.head()<jupyter_output><empty_output><jupyter_text>Apply LDA to laptop descriptions, get some topics, have columns quantifying how much a description belongs to each topic.<jupyter_code>import get_LDA_topic_features

laptopTopics = get_LDA_topic_features.LDA(laptopData)
laptopTopics.head()
laptopData["top0"] = laptopTopics["0"]
laptopData["top1"] = laptopTopics["1"]
laptopData["top2"] = laptopTopics["2"]
laptopData["top3"] = laptopTopics["3"]
laptopData["top4"] = laptopTopics["4"]
laptopData["top5"] = laptopTopics["5"]
laptopData["top6"] = laptopTopics["6"]
laptopData["top7"] = laptopTopics["7"]

laptopData.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09812782440284054, 'fraction_numerical': 0.010329244673983214, 'mean_word_length': 10.567164179104477, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '385153', 'license': 'MIT', 'n_tokens_mistral': 542, 'n_tokens_neox': 509, 'n_words': 92, 'path': 'feature_engineering/feature_engineering.ipynb', 'repo_name': 'jzymet/Uppraisal', 'size': '1549'}