<commit_before>---
published: false
---
## DARK WEB DRUG MARKET PREDICTION

### Objective

Train a simple machine learning model to learn about the hidden markets of the [dark web](https://en.wikipedia.org/wiki/Dark_web).

It is possible for anyone with internet access to enter dark web marketplaces and view product listings. I herein use Python software code to simulate the behavior of a human browsing these markets, collect and save interesting information from each webpage this browsing agent encounters, and finally use that information in aggregate to construct a predictive model. 

### Context
Darknet Marketplaces (cryptomarkets) are internet markets that allow anonymous buying and selling. Anonymity means that many of these markets trade illegal goods, as it is inherently difficult for law enforcement to intercept information or identify transactors. 

While [black markets](https://en.wikipedia.org/wiki/Black_market) have existed as long as regulated commerce itself, darknet markets were born somewhat recently when 4 technologies combined:

* Anonymous internet browsing (e.g. Tor and the Onion network)
* Virtual currencies (e.g. Bitcoin)
* Escrow (conditional money transfer)
* Vendor feedback systems (Amazon.com-like ratings of sellers)






- Context (Why): Define the environment in which the problem exists and set up the motivation for the research question.
- Problem (Question): Concisely describe the problem as a question that you went out and answered.
- Solution (Answer): Concisely describe the solution as an answer to the question you posed in the previous section. Be specific.
- Findings: Bulleted lists of discoveries you made along the way that interest the audience. They may be discoveries in the data, methods that did or did not work or the model performance benefits you achieved along your journey.
- Limitations: Consider where the model does not work or questions that the model does not answer. Do not shy away from these questions, defining where the model excels is more trusted if you can define where it does not excel.
- Conclusions (Why+Question+Answer): Revisit the why, research question and the answer you discovered in a tight little package that is easy to remember and repeat for yourself and others.<commit_msg>Update 2017-07-23-luther.md<commit_after>---
published: false
---
## DARK WEB DRUG MARKET PREDICTION

### Objective

Train a simple machine learning model to learn about the hidden markets of the [dark web](https://en.wikipedia.org/wiki/Dark_web).

It is possible for anyone with internet access to enter dark web marketplaces and view product listings. I herein use Python software code to simulate the behavior of a human browsing these markets, collect and save interesting information from each webpage this browsing agent encounters, and finally use that information in aggregate to construct a predictive model. 
  
    
---
    
  
### Context
Darknet Marketplaces (cryptomarkets) are internet markets that allow anonymous buying and selling. Anonymity means that many of these markets trade illegal goods, as it is inherently difficult for law enforcement to intercept information or identify transactors. 

While [black markets](https://en.wikipedia.org/wiki/Black_market) have existed as long as regulated commerce itself, darknet markets were born somewhat recently when 4 technologies combined:

* Anonymous internet browsing (e.g. Tor and the Onion network)
* Virtual currencies (e.g. Bitcoin)
* Escrow (conditional money transfer)
* Vendor feedback systems (Amazon.com-like ratings of sellers)

Total cash flow through dark web markets is hard to estimate, but indicators show it as substantial and rising. The biggest individual vendors can earn millions of dollars per year, with international reach.

### Challenge

After bragging a little too loudly in a seedy Mexican cantina about your magic data science powers of prediction, you have been kidnapped by a forward-thinking drug cartel. They want to start selling their cocaine online. They demand that you help them predict the optimal price. If they aren't convinced you've helped them maximize their profit, your life will be forfeit!


- Context (Why): Define the environment in which the problem exists and set up the motivation for the research question.
- Problem (Question): Concisely describe the problem as a question that you went out and answered.
- Solution (Answer): Concisely describe the solution as an answer to the question you posed in the previous section. Be specific.
- Findings: Bulleted lists of discoveries you made along the way that interest the audience. They may be discoveries in the data, methods that did or did not work or the model performance benefits you achieved along your journey.
- Limitations: Consider where the model does not work or questions that the model does not answer. Do not shy away from these questions, defining where the model excels is more trusted if you can define where it does not excel.
- Conclusions (Why+Question+Answer): Revisit the why, research question and the answer you discovered in a tight little package that is easy to remember and repeat for yourself and others.
{'fraction_non_alphanumeric': 0.044577383249710535, 'fraction_numerical': 0.0019297568506368198, 'mean_word_length': 4.658296943231441, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '637282', 'n_tokens_mistral': 1215, 'n_tokens_neox': 1118, 'n_words': 743}