<commit_before># stock-picker
Quick and dirty algorithms to identify which stocks might be high performing.

## Objectives
- Learn things
  - backend processing
    - cron jobs
    - offline processing
    - db
  - external apis
    - YQL
  - working in independent / limited mentoring environment
  - testing / evaluation
  - using design docs

## Overview
### Phase 1 Goals
- Create a database to store historical stock information
- Populate db with all stock symbols on Nasdaq, NYSE, American stock exchanges
- Create and use tool to populate db with historical prices
- Configure nightly cron job to run tool each night and update db with latest price data
- Create tool to compute [a single, pick one] buy/sell metric for a stock
- Create evaluation tool to evaluate metric quality based on historical data

### Phase 1 Stretch
- Create automated job to compute metric results for all stocks on daily basis
- Create tool to select highest scoring stocks and provide list of recommended stock picks

### Phase 1 Non goals
- Robotic / automatic trading
- Monitoring / alerts
- Etrade integration / oauth
- D3 / HighCharts stock data visualization
- Use React.js to create homepage
- Multiple buy/sell metrics

### System Architecture
![alt text](https://github.com/dianpan/stock-picker/app/assets/images/stock_picker.png "system")

### Component Design
- Symbol Seeder Tool
  - Should be idempotent
  - Input: CSV file containing stock symbol info
  - Output: stock symbol stored in Stock table

- Scraper #1
  - Input: stock symbols
  - Output: start date of stock, end date (if any)

- Scraper #2
  - Input: stock symbol, start date, end date or current day’s date
  - Output: stock quote data as JSON objects to store in Quotes table<commit_msg>update img path<commit_after># stock-picker
Quick and dirty algorithms to identify which stocks might be high performing.

## Objectives
- Learn things
  - backend processing
    - cron jobs
    - offline processing
    - db
  - external apis
    - YQL
  - working in independent / limited mentoring environment
  - testing / evaluation
  - using design docs

## Overview
### Phase 1 Goals
- Create a database to store historical stock information
- Populate db with all stock symbols on Nasdaq, NYSE, American stock exchanges
- Create and use tool to populate db with historical prices
- Configure nightly cron job to run tool each night and update db with latest price data
- Create tool to compute [a single, pick one] buy/sell metric for a stock
- Create evaluation tool to evaluate metric quality based on historical data

### Phase 1 Stretch
- Create automated job to compute metric results for all stocks on daily basis
- Create tool to select highest scoring stocks and provide list of recommended stock picks

### Phase 1 Non goals
- Robotic / automatic trading
- Monitoring / alerts
- Etrade integration / oauth
- D3 / HighCharts stock data visualization
- Use React.js to create homepage
- Multiple buy/sell metrics

### System Architecture
![alt text](https://raw.githubusercontent.com/dianpan/stock-picker/957d1ffa0f65b4045cd3141a04d2257a66ea11fb/app/assets/images/stock_picker.png "system")

### Component Design
- Symbol Seeder Tool
  - Should be idempotent
  - Input: CSV file containing stock symbol info
  - Output: stock symbol stored in Stock table

- Scraper #1
  - Input: stock symbols
  - Output: start date of stock, end date (if any)

- Scraper #2
  - Input: stock symbol, start date, end date or current day’s date
  - Output: stock quote data as JSON objects to store in Quotes table
{'fraction_non_alphanumeric': 0.060078607523863, 'fraction_numerical': 0.010387422796181921, 'mean_word_length': 3.621271076523995, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3967470', 'n_tokens_mistral': 966, 'n_tokens_neox': 909, 'n_words': 467}