<commit_before># Steam crawler

First, do consider that you should not run your own instance of the Steam crawler because there is already one being run and it doesn't give any benefit to run more instances. This code is here only for people to learn stuff.

Steam crawler is a web crawler with a goal to crawl Steam users' profiles in the Steam community and gather some data from them. This data is then analyzed and shown on the website of the one and only [Steam crawler](http://www.anttivainio.net/steambot). Visiting that website is also a good way of learning what kind of data the Steam crawler is collecting.

Steam crawler works just like any other web crawler like Google Bot or Bing Bot because it essentially is one. Steam crawler acquires its data by sending http-requests and analyzing the received html-data. Because of this, Steam crawler can only see exactly what every other crawler or human browsing the Steam community can see while not logged into Steam.

The crawler itself is a Python program but it also expects a web server to exist to which it sends the collected and analyzed data using http post-requests.

If you still feel the strongest desire to run Steam crawler even though you shouldn't, at least edit the settings and create some web sever to receive the data sent by the crawler.<commit_msg>Changed http to https in the readme<commit_after># Steam crawler

First, do consider that you should not run your own instance of the Steam crawler because there is already one being run and it doesn't give any benefit to run more instances. This code is here only for people to learn stuff.

Steam crawler is a web crawler with a goal to crawl Steam users' profiles in the Steam community and gather some data from them. This data is then analyzed and shown on the website of the one and only [Steam crawler](https://www.anttivainio.net/steambot). Visiting that website is also a good way of learning what kind of data the Steam crawler is collecting.

Steam crawler works just like any other web crawler like Google Bot or Bing Bot because it essentially is one. Steam crawler acquires its data by sending http-requests and analyzing the received html-data. Because of this, Steam crawler can only see exactly what every other crawler or human browsing the Steam community can see while not logged into Steam.

The crawler itself is a Python program but it also expects a web server to exist to which it sends the collected and analyzed data using http post-requests.

If you still feel the strongest desire to run Steam crawler even though you shouldn't, at least edit the settings and create some web sever to receive the data sent by the crawler.
{'fraction_non_alphanumeric': 0.02463605823068309, 'fraction_numerical': 0.0, 'mean_word_length': 4.349301397205589, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1478084', 'n_tokens_mistral': 621, 'n_tokens_neox': 608, 'n_words': 455}