<commit_before>UriPositionAnalysis
===================

Analysis of URI positions in RDF triples.

Description
-----------
This is a collection of python scripts and Hadoop mapreduce jobs to identify on which position in an RDF triple which URI occurs how often.

Usage
-----

# Download the an gzipped N-Quad data set [like this](http://data.dws.informatik.uni-mannheim.de/lodcloud/2014/ISWC-RDB/dump.nq.gz).
# Change the 'Config.py' entries.
# Excecute the python script 'ExtractURIs.py' in the 'python' folder.
# Use the newly generated file as input for the Hadoop mapreduce job "UriPositionAnalysisRunner" in the 'Java' folder.
Additionally:
* You can use the 'CountURIs.py" python script to do some statistical analysis like how many URIs occur how often in which position.

Todos
-----
* code is not well documented, sorry. I will change that soon.
* some variable names are not good chosen, I am aware of that.<commit_msg>added readme update<commit_after>UriPositionAnalysis
===================

Analysis of URI positions in RDF triples.

Description
-----------
This is a collection of python scripts and Hadoop mapreduce jobs to identify on which position in an RDF triple which URI occurs how often.

Usage
-----

1. Download the an gzipped N-Quad data set [like this](http://data.dws.informatik.uni-mannheim.de/lodcloud/2014/ISWC-RDB/dump.nq.gz).
2. Change the 'Config.py' entries.
3. Excecute the python script 'ExtractURIs.py' in the 'python' folder.
4. Use the newly generated file as input for the Hadoop mapreduce job "UriPositionAnalysisRunner" in the 'Java' folder.
Additionally:
* You can use the 'CountURIs.py" python script to do some statistical analysis like how many URIs occur how often in which position.

Todos
-----
* code is not well documented, sorry. I will change that soon.
* some variable names are not good chosen, I am aware of that.
{'fraction_non_alphanumeric': 0.10481283422459893, 'fraction_numerical': 0.006417112299465241, 'mean_word_length': 3.962864721485411, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6450872', 'n_tokens_mistral': 539, 'n_tokens_neox': 522, 'n_words': 247}