<commit_before>---
layout: post
title:  "Craigslist scraping, IP bans and other adventures in programming"
date:   2017-07-13 04:55:20 +0000
---


So the current challenge at Flatiron is to make a functional website scraper, that works down to two layers deep. This is a big deal project, and is assessed. So, wanting to put my best foot forward, instead of rushing into a project, I thought for about a day on what I wanted to create.  I had been working a lot with Wikipedia on a side project and at first thought about doing that...but then I recalled there was a video lesson which had some of that in there.  So, staying away from safe waters, I started pondering some more. I had been selling some items on Craigslist (CL) and had been thinking about how price is really the key to any sale.  Too high and people scoff, too low and people immediately think there is something wrong with your listing.  But there is that sweet spot, "market value" as they say, that puts you right into the zone where you can actually make a sale.  After all, I thought, I am spending all this time looking at postings on CL and sorting through everything was a little painful. I mean, I had even listed a printer that was already there with two other listings for the exact model...but I never saw them until I had relisted the item and done a more specific search.

And then it hit me. Hey, let's have Ruby do it!
<commit_msg>Updating my blog post.<commit_after>---
layout: post
title:  "Craigslist scraping, IP bans and other adventures."
date:   2017-07-13 00:55:21 -0400
---


So the current challenge at Flatiron is to make a functional website scraper, that works down to two layers deep. This is a big deal project, and is assessed. So, wanting to put my best foot forward, instead of rushing into a project, I thought for about a day on what I wanted to create.  I had been working a lot with Wikipedia on a side project and at first thought about doing that...but then I recalled there was a video lesson which had some of that in there.  So, staying away from safe waters, I started pondering some more. I had been selling some items on Craigslist (CL) and had been thinking about how price is really the key to any sale.  Too high and people scoff, too low and people immediately think there is something wrong with your listing.  But there is that sweet spot, "market value" as they say, that puts you right into the zone where you can actually make a sale.  After all, I thought, I am spending all this time looking at postings on CL and sorting through everything was a little painful. I mean, I had even listed a printer that was already there with two other listings for the exact model...but I never saw them until I had relisted the item and done a more specific search.

And then it hit me. Hey, let's have Ruby do it!

{'fraction_non_alphanumeric': 0.04103289706402547, 'fraction_numerical': 0.012734347364697559, 'mean_word_length': 3.8341880341880343, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7396209', 'n_tokens_mistral': 723, 'n_tokens_neox': 681, 'n_words': 507}