<jupyter_start><jupyter_text>Entity Resolution Kit OverviewThe high-level methods can be accessed via a global import of the package like so:<jupyter_code>from entity_resolution import *<jupyter_output><empty_output><jupyter_text>Downloading S3 data Provided that your credentials are set you can donwload S3 data directly to your local disk. To set your credentials use `aws configure` in your terminal. The entity resolution package does not accept credentials but relies on the AWS SDK for this.<jupyter_code>s3_url = "s3://mirror-sc-identity-resolution-snowcatcloud/customer-data/enriched/stream/run=20211112/"
download_path = "/Users/Swa/temp/stuff"
download_s3_folder(s3_url, download_path)<jupyter_output><empty_output><jupyter_text>In one goThe following `demo()` method will- generate a pseudo-random graph- perform the identity resolution on it- create predictions- export the prediction to csv- open the default GraphML app to visualize the graph (with predictions)This will open the default application for GraphML files (*.graphml) and [yEd is probably the best option](https://www.yworks.com/products/yed/downloaddownload). Use the "Smart Organic Layout" and the properties mapper (Edit > Properties Mapper) to turn data values into visuals. There isa `yEdMapping.cnfx` file in the `diverse` directory you can import, it contains a set of mapping which will render something like the image below.The `df` is a dataframe and it contains rows of predictions in the shape (Device, Identity, Probability). It's saved as a CSV as well.![](../sample.png)![](../predictions.png)You can also output synthetic data to [Gephi](https://gephi.org)<jupyter_code>from entity_resolution import *
g,df = demo()
df
!ls<jupyter_output>SharedIP.png demo.ipynb<jupyter_text>VisualizationAs shown above the framework can easily generate GraphML files for visualization but you can just as easily use Gephi if you prefer. The generated `gexf` file can be opened using Gephi and some other compatible applications.<jupyter_code>from entity_resolution import *
create_synthetic_gephi_graph()<jupyter_output><empty_output><jupyter_text>Generating test graphsThis is useful for demo and test purposes.All of the methods creating synthetic data have the following tuning parameters:- **device count**: how many device nodes to create- **identity count**: how many identity nodes to create- **cookie probability**: the probability that a cookie node is shared between two devices- **ip probability**: the probability that an IP node is shared between two devices- **location probability**: the probability that an Location node is shared between two devicesThese parameters have appropriate default so you can use the methods like this<jupyter_code>g = create_synthetic_graph()<jupyter_output><empty_output><jupyter_text>When you have a graph you can infer indentities immediately like this:<jupyter_code>infer_identities(g)<jupyter_output><empty_output><jupyter_text>The ouput represents the inferred edges between devices and identities together with a probability.By default the internal id's of the nodes are returned but you can access the true unique identifiers as well:<jupyter_code>coll, ids = infer_identities(g, return_ids = True)
ids<jupyter_output><empty_output><jupyter_text>This can be saved to CSV and used for downstream tasks.  Raw formatGiven raw data (downloaded from S3 like highlighted above) in a directory you can make immediately predictions:<jupyter_code>from entity_resolution import *
raw_directory = "/Users/Swa/temp/raw"
g, df = raw_to_predictions(raw_directory)
df<jupyter_output>Reading 457 file(s) in directory '/Users/Swa/temp/raw'.<jupyter_text>The predictions are outputted and saved as CSV as well.The following parameters are defined here:- **raw_directory**: the directory where the gz-files are located. The S3 download utility described above can be used to download data from S3.- **threshold**: only predictions with a probability above this value are retained- **save_diagram**: whether the internally generated graph should be saved (as GraphML). This is only for visualization purposes.- **predictions_path**: the path where the CSV with the predictions should be saved. If not specified it will be save to `predictions.csv` in the directory where the script was ran.- **max_predictions**: specifies how many of the top predictions are retained. If for some reason you wish to convert the raw data to JSON entities for other tasks you can do it like this:<jupyter_code>from entity_resolution import *
raw_directory = "/Users/Swa/temp/raw"
json_entities_path = "/Users/Swa/temp/entities/"
raw_to_entities_json(raw_directory,json_entities_path )
!ls<jupyter_output>SharedIP.png    demo.ipynb      predictions.csv<jupyter_text>Neo4j You can make prediction directly from a Neo4j database:<jupyter_code>g, df = neo_to_predictions()
df<jupyter_output>Nodes: 1585 Edges: 1166 Components: 757 Largest component: 334
Analyzing 757 component(s). The largest contains 334 nodes.<jupyter_text>Here again, there are plenty of parameters to tune the results and you can save the resulting predictions to a CSV for other purposes. For demo purposes you can also create test databases:<jupyter_code>create_synthetic_neo("demodata")
add_predictions_to_neo("demodata")<jupyter_output>All INFERRED edges have been removed from database 'demodata'.
Nodes: 613 Edges: 664 Components: 116 Largest component: 17
Analyzing 116 component(s). The largest contains 17 nodes.
{'fraction_non_alphanumeric': 0.06989051094890511, 'fraction_numerical': 0.010583941605839416, 'mean_word_length': 6.127438231469441, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 41, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '385777', 'license': 'MIT', 'n_tokens_mistral': 1553, 'n_tokens_neox': 1464, 'n_words': 638, 'path': 'notebooks/demo.ipynb', 'repo_name': 'Orbifold/entity-resolution', 'size': '5480'}