<commit_before># bilibili-captcha

## Overview

The Python program attempts to recognize [CAPTCHA images](http://www.bilibili.com/captcha) generated by [bilibili.com](http://www.bilibili.com/).

The program is developed and tested on `Mac OS X 10.10.4` and `Windows 10` using `Python 3.4` interpreter. There might be problems if the program is run under other environments. Particularly, the program will not compile using `Python 2` because of the differences in syntax.

## Dependencies

- [matplotlib](http://matplotlib.org)
- [numpy](http://www.numpy.org)
- [pillow](https://python-pillow.github.io/)
- [requests](http://www.python-requests.org/en/latest)
- [scikit-learn](http://scikit-learn.org/stable)
- [scipy](http://www.scipy.org)
- [theano](http://deeplearning.net/software/theano)

## Usage

Clone all the files and put them under the same directory as your program, including the `dataset` folder.

## Key Files Overview

`captcha_provider.py`

This module represents the source of the CAPTCHA. An abstract base class `HttpCaptchaProvider` is defined to represent abstract CAPTCHA providers and a derived class `BilibiliCaptchaProvider` is defined to represent the specific CAPTCHA source one is attempting to recognize. If you want to define another CAPTCHA source, you can define another class that inherits the abstract base class. Two methods that must be overridden are `_get_data_from_seq` and `_is_correct_response`. `_get_data_from_seq` extracts the necessary data that need to be submitted as part of the request to verify. `_is_correct_response` checks whether the recognized character sequence is correct.

`captcha_recognizer.py`

This module is designed to recognize the images (the return value of `imread` function in `matplotlib.image`) from a particular CAPTCHA source (derived class of `HttpCaptchaProvider`). The steps to recognize an image are

1. Noise reduction

  1.1 Noise reduction by hsv (`remove_noise_with_hsv` function)

  1.2 Noise reduction by neighbors (`remove_noise_with_neighbors` function)

  After this step the remaining image is a greyscale image with slight noise.

2. Segmentation

  Components are found using the next-nearest-neighbor. (`segment_with_label` and other commands)

  After this step the image is partitioned into a number of "characters".

3. Recognition

  If the number of partitioned characters is as desired, each is recognized using the model in `captcha_learn.py`. (`recognize` function)

`captcha_learn.py`<commit_msg>Strong<commit_after># bilibili-captcha

## Overview

The Python program attempts to recognize [CAPTCHA images](http://www.bilibili.com/captcha) generated by [bilibili.com](http://www.bilibili.com/).

The program is developed and tested on `Mac OS X 10.10.4` and `Windows 10` using `Python 3.4` interpreter. There might be problems if the program is run under other environments. Particularly, the program will not compile using `Python 2` because of the differences in syntax.

## Dependencies

- [matplotlib](http://matplotlib.org)
- [numpy](http://www.numpy.org)
- [pillow](https://python-pillow.github.io/)
- [requests](http://www.python-requests.org/en/latest)
- [scikit-learn](http://scikit-learn.org/stable)
- [scipy](http://www.scipy.org)
- [theano](http://deeplearning.net/software/theano)

## Usage

Clone all the files and put them under the same directory as your program, including the `dataset` folder.

## Key Files Overview

**`captcha_provider.py`**

This module represents the source of the CAPTCHA. An abstract base class `HttpCaptchaProvider` is defined to represent abstract CAPTCHA providers and a derived class `BilibiliCaptchaProvider` is defined to represent the specific CAPTCHA source one is attempting to recognize. If you want to define another CAPTCHA source, you can define another class that inherits the abstract base class. Two methods that must be overridden are `_get_data_from_seq` and `_is_correct_response`. `_get_data_from_seq` extracts the necessary data that need to be submitted as part of the request to verify. `_is_correct_response` checks whether the recognized character sequence is correct.

**`captcha_recognizer.py`**

This module is designed to recognize the images (the return value of `imread` function in `matplotlib.image`) from a particular CAPTCHA source (derived class of `HttpCaptchaProvider`). The steps to recognize an image are

1. Noise reduction

  1.1 Noise reduction by hsv (`remove_noise_with_hsv` function)

  1.2 Noise reduction by neighbors (`remove_noise_with_neighbors` function)

  After this step the remaining image is a greyscale image with slight noise.

2. Segmentation

  Components are found using the next-nearest-neighbor. (`segment_with_label` and other commands)

  After this step the image is partitioned into a number of "characters".

3. Recognition

  If the number of partitioned characters is as desired, each is recognized using the model in `captcha_learn.py`. (`recognize` function)

**`captcha_learn.py`**
{'fraction_non_alphanumeric': 0.08408408408408409, 'fraction_numerical': 0.006806806806806807, 'mean_word_length': 4.926453143534994, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 10, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1929037', 'n_tokens_mistral': 1502, 'n_tokens_neox': 1398, 'n_words': 583}