<commit_before>This folder contains several datasets that have been tested with JedAI. We have grouped them in two categories: 
1) those suitable for Clean-Clean ER, and 
2) those sutable for Dirty ER. 

Every dataset includes two types of files: 
1) those containing the entity profiles themselves, which are called **entity files**, and 
2) those containing the golden standard (i.e., the ground-truth with the real matches), which are called **groundtruth files**.

Note that every file that is available as a *Java Serialized Object* (**JSO**) can be read using the class DataReader.EntityReader.EntitySerializationReader for entity files or the class DataReader.GroundTruthReader.GtSerializationReader for ground-truth files. See [this class](https://github.com/scify/JedAIToolkit/blob/master/jedai-core/tests/GeneralExamples/DirtyErDatasetStatistics.java) for an example.

# Clean-Clean ER datasets

| Dataset Name | D1 Entities| D2 Entities | D1 Name-Value Pairs	| D2 Name-Value Pairs	| Duplicates | Average NVP per Entity	|
Brute-force Comparisons | File Format | Data Origin | 
|---|---| ---| ---| ---| ---|---| ---| ---| ---| 
| Restaurants | 339 | 2,256 | 1,130 | 7,519 | 89 | 3.3 | 7.64E+05 | JSO ([Rest. 1 file](cleanCleanErDatasets/restaurant1Profiles), [Rest. 2 file](cleanCleanErDatasets/restaurant2Profiles), [groundtruth file](cleanCleanErDatasets/restaurantsIdDuplicates)) | Real data |
|Abt-Buy	|1,076|	1,076|	2,568	|2,308|	1,076	|2.4|	1.16E+06|JSO ([Abt entity file](cleanCleanErDatasets/abtProfiles), [Buy entity file](cleanCleanErDatasets/buyProfiles), [groundtruth file](cleanCleanErDatasets/abtBuyIdDuplicates))|Real data |
|Amazon-Google Products|	1,354	|3,039	|5,302	|9,110	|1,104	|3.9|	4.11E+06|JSO ([Amazon entity file](cleanCleanErDatasets/amazonProfiles), [GP entity file](cleanCleanErDatasets/gpProfiles), [groundtruth file](cleanCleanErDatasets/amazonGpIdDuplicates))|Real data |
|DBLP-ACM|	2,616	|2,294	|10,464	|9,162|	2,224	|4.0|	6.00E+06|JSO ([DBLP entity file](cleanCleanErDatasets/dblpProfiles), [ACM entity file](cleanCleanErDatasets/acmProfiles), [groundtruth file](cleanCleanErDatasets/dblpAcmIdDuplicates)), CSV ([DBLP entity file](cleanCleanErDatasets/DBLP-ACM/DBLP2.csv), [ACM entity file](cleanCleanErDatasets/DBLP-ACM/ACM.csv)), XML ([DBLP entity file](cleanCleanErDatasets/DBLP2toRdf.xml), [ACM entity file](cleanCleanErDatasets/ACMtoRdf.xml)) |Real data |
|Amazon-Walmart|2,554| 22,074 | 14,143 | 114,315 | 853 | 5.2 | 5.64E+07 | JSO ([Amazon entity file](cleanCleanErDatasets/amazonProfiles2), [Walmart entity file](cleanCleanErDatasets/walmartProfiles), [groundtruth file](cleanCleanErDatasets/amazonWalmartIdDuplicates)) | Real data |
|DBLP-Scholar|	2,516	|61,353|	10,064	|198,001|	2,308|	4.0|	1.54E+08|JSO ([DBLP entity file](cleanCleanErDatasets/dblpProfiles2), [Scholar entity file](cleanCleanErDatasets/scholarProfiles), [groundtruth file](cleanCleanErDatasets/dblpScholarIdDuplicates))|Real data |
|Movies	|27,615	|23,182|	155,436	|816,009	|22,863|	5.6|	6.40E+08|JSO ([IMDB entity file](cleanCleanErDatasets/imdbProfiles), [DBPedia entity file](cleanCleanErDatasets/dbpediaProfiles.zip), [groundtruth file](cleanCleanErDatasets/moviesIdDuplicates))|Real data |
<!---|DBPedia|	1,190,733	|2,164,040	|1.69E+07	|3.50E+07	|892,586	|14.2	|2.58E+12|JSO |Real data |-->

# Dirty ER datasets

| Dataset Name | Entities | Name-Value Pairs | Duplicates | Average NVP per Entity |	Brute-force Comparisons |
File Format | Data Origin | <commit_msg>Added Large DBPedia dataset.<commit_after>This folder contains several datasets that have been tested with JedAI. We have grouped them in two categories: 
1) those suitable for Clean-Clean ER, and 
2) those sutable for Dirty ER. 

Every dataset includes two types of files: 
1) those containing the entity profiles themselves, which are called **entity files**, and 
2) those containing the golden standard (i.e., the ground-truth with the real matches), which are called **groundtruth files**.

Note that every file that is available as a *Java Serialized Object* (**JSO**) can be read using the class DataReader.EntityReader.EntitySerializationReader for entity files or the class DataReader.GroundTruthReader.GtSerializationReader for ground-truth files. See [this class](https://github.com/scify/JedAIToolkit/blob/master/jedai-core/tests/GeneralExamples/DirtyErDatasetStatistics.java) for an example.

# Clean-Clean ER datasets

| Dataset Name | D1 Entities| D2 Entities | D1 Name-Value Pairs	| D2 Name-Value Pairs	| Duplicates | Average NVP per Entity	|
Brute-force Comparisons | File Format | Data Origin | 
|---|---| ---| ---| ---| ---|---| ---| ---| ---| 
| Restaurants | 339 | 2,256 | 1,130 | 7,519 | 89 | 3.3 | 7.64E+05 | JSO ([Rest. 1 file](cleanCleanErDatasets/restaurant1Profiles), [Rest. 2 file](cleanCleanErDatasets/restaurant2Profiles), [groundtruth file](cleanCleanErDatasets/restaurantsIdDuplicates)) | Real data |
|Abt-Buy	|1,076|	1,076|	2,568	|2,308|	1,076	|2.4|	1.16E+06|JSO ([Abt entity file](cleanCleanErDatasets/abtProfiles), [Buy entity file](cleanCleanErDatasets/buyProfiles), [groundtruth file](cleanCleanErDatasets/abtBuyIdDuplicates))|Real data |
|Amazon-Google Products|	1,354	|3,039	|5,302	|9,110	|1,104	|3.9|	4.11E+06|JSO ([Amazon entity file](cleanCleanErDatasets/amazonProfiles), [GP entity file](cleanCleanErDatasets/gpProfiles), [groundtruth file](cleanCleanErDatasets/amazonGpIdDuplicates))|Real data |
|DBLP-ACM|	2,616	|2,294	|10,464	|9,162|	2,224	|4.0|	6.00E+06|JSO ([DBLP entity file](cleanCleanErDatasets/dblpProfiles), [ACM entity file](cleanCleanErDatasets/acmProfiles), [groundtruth file](cleanCleanErDatasets/dblpAcmIdDuplicates)), CSV ([DBLP entity file](cleanCleanErDatasets/DBLP-ACM/DBLP2.csv), [ACM entity file](cleanCleanErDatasets/DBLP-ACM/ACM.csv)), XML ([DBLP entity file](cleanCleanErDatasets/DBLP2toRdf.xml), [ACM entity file](cleanCleanErDatasets/ACMtoRdf.xml)) |Real data |
|Amazon-Walmart|2,554| 22,074 | 14,143 | 114,315 | 853 | 5.2 | 5.64E+07 | JSO ([Amazon entity file](cleanCleanErDatasets/amazonProfiles2), [Walmart entity file](cleanCleanErDatasets/walmartProfiles), [groundtruth file](cleanCleanErDatasets/amazonWalmartIdDuplicates)) | Real data |
|DBLP-Scholar|	2,516	|61,353|	10,064	|198,001|	2,308|	4.0|	1.54E+08|JSO ([DBLP entity file](cleanCleanErDatasets/dblpProfiles2), [Scholar entity file](cleanCleanErDatasets/scholarProfiles), [groundtruth file](cleanCleanErDatasets/dblpScholarIdDuplicates))|Real data |
|Movies	|27,615	|23,182|	155,436	|816,009	|22,863|	5.6|	6.40E+08|JSO ([IMDB entity file](cleanCleanErDatasets/imdbProfiles), [DBPedia entity file](cleanCleanErDatasets/dbpediaProfiles.zip), [groundtruth file](cleanCleanErDatasets/moviesIdDuplicates))|Real data |
|DBPedia|	1,190,733	|2,164,040	|1.69E+07	|3.50E+07	|892,586	|14.2	|2.58E+12|[JSO](https://drive.google.com/open?id=1rFAwrmlsfR86fVIkOaGXCSzbfZIZufY2) |Real data |

# Dirty ER datasets

| Dataset Name | Entities | Name-Value Pairs | Duplicates | Average NVP per Entity |	Brute-force Comparisons |
File Format | Data Origin | 
{'fraction_non_alphanumeric': 0.13415497945884686, 'fraction_numerical': 0.07281484629550927, 'mean_word_length': 6.138523761375127, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2303291', 'n_tokens_mistral': 2832, 'n_tokens_neox': 2568, 'n_words': 599}