<commit_before># SpyderWeb
## A website to compare and display http website statistics!

Developed by [<NAME>](https://github.com/615105327), [<NAME>](https://github.com/Jasmine424) and [<NAME>](https://github.com/hariharan-m) for EC601-Product Design Fall '17.

This project uses a modified version of [httpstat by reorx](https://github.com/reorx/httpstat) to compare and graph the top 10,000 websites ranked by traffic statistics.

### A brief look at the internals

Various technologies are being used in this project including
* Python
* PHP
* HTML
* CSS
* Shell Scripting
* SQL

![Block Diagram](https://github.com/hariharan-m/SpyderWeb/blob/master/Untitled%20Diagram.png "Block Diagram")

### How do you use this project?

* Clone this repo 
* Buy/Rent/Use a VPS or any hosting service which runs Linux (AWS and Google Cloud offer free 1 year trials!) to set up the following software:
  * Apache (move files to /var/www/ )
  * MySQL
* Create a Database with attributes given in the JSON file along with a column for name of the website.
* ``` chmod +x ``` the required scripts 
* Put the following entry in your crontab (change as per requirement, the following lines make the database update every 30 minutes.
```
30 * * * * /path/to/script/readfile.sh

```
And you should be up and running!

### Current Progress

* We are still working on understanding and setting up [Bootstrap] (http://getbootstrap.com/).
* We also need to implement user input and query user's website in Realtime.

Here are some pictures of the website in development.

[1](https://github.com/hariharan-m/SpyderWeb/blob/master/search%20website.png)
[2](https://github.com/hariharan-m/SpyderWeb/blob/master/homepage.png)

<commit_msg>Inline images<commit_after># SpyderWeb
## A website to compare and display http website statistics!

Developed by [<NAME>](https://github.com/615105327), [<NAME>](https://github.com/Jasmine424) and [<NAME>](https://github.com/hariharan-m) for EC601-Product Design Fall '17.

This project uses a modified version of [httpstat by reorx](https://github.com/reorx/httpstat) to compare and graph the top 10,000 websites ranked by traffic statistics.

### A brief look at the internals

Various technologies are being used in this project including
* Python
* PHP
* HTML
* CSS
* Shell Scripting
* SQL

![Block Diagram](https://github.com/hariharan-m/SpyderWeb/blob/master/Untitled%20Diagram.png "Block Diagram")

### How do you use this project?

* Clone this repo 
* Buy/Rent/Use a VPS or any hosting service which runs Linux (AWS and Google Cloud offer free 1 year trials!) to set up the following software:
  * Apache (move files to /var/www/ )
  * MySQL
* Create a Database with attributes given in the JSON file along with a column for name of the website.
* ``` chmod +x ``` the required scripts 
* Put the following entry in your crontab (change as per requirement, the following lines make the database update every 30 minutes.
```
30 * * * * /path/to/script/readfile.sh

```
And you should be up and running!

### Current Progress

* We are still working on understanding and setting up [Bootstrap] (http://getbootstrap.com/).
* We also need to implement user input and query user's website in Realtime.

Here are some pictures of the website in development.

![1](https://github.com/hariharan-m/SpyderWeb/blob/master/search%20website.png)
![2](https://github.com/hariharan-m/SpyderWeb/blob/master/homepage.png)


{'fraction_non_alphanumeric': 0.10728862973760933, 'fraction_numerical': 0.01924198250728863, 'mean_word_length': 4.377742946708464, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 14, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3414748', 'n_tokens_mistral': 1082, 'n_tokens_neox': 1019, 'n_words': 403}