<jupyter_start><jupyter_text>I downloaded my classifications; now what?This notebook will help you take a first look at your data, to get basic information about it like: how many classifications, how many classifiers (signed in and not signed in), etc. It uses Python 2.7.There are scripts to do this, but first let's just get a sense of what the data looks like.Before we begin, though, we will need the following packages:<jupyter_code>import sys, os
import numpy as np
import pandas as pd

print("Python version: %d.%d.%d, numpy version: %s, pandas version: %s. \nOriginally developed using Py 2.7.11, np v1.11.0, pd v0.19.2" %(sys.version_info[0], sys.version_info[1], sys.version_info[2], np.__version__, pd.__version__))
print("If these versions don't match and stuff breaks, that's probably why.")<jupyter_output>Python version: 2.7.11, numpy version: 1.11.0, pandas version: 0.19.2. 
Originally developed using Py 2.7.11, np v1.11.0, pd v0.19.2
If these versions don't match and stuff breaks, that's probably why.<jupyter_text>First, let's say your project is called "My Project". We'll make that a variable below, because any of the files we need to access (classifications file, workflow contents file, etc) will start with that name.<jupyter_code>project_name = "my-project"

classification_file = project_name + "-classifications.csv"

print(classification_file)<jupyter_output>my-project-classifications.csv<jupyter_text>Now let's read in that file, using a package called `pandas` which is designed to handle large tables.<jupyter_code>classifications_all = pd.read_csv(classification_file)
n_class = len(classifications_all)

print("File %s read with %d rows." % (classification_file, n_class))<jupyter_output>File my-project-classifications.csv read with 50000 rows.<jupyter_text>The number of rows, which we've saved as `n_class`, is the same as the total number of classifications recorded in this file.   **Note:** The more classifications in your file, the more memory it will take for your computer to work with them using `pandas`. From my experience, a few million rows isn't too big a deal as long as you have at least 8 GB of RAM. If you have a lot more, you may need something with more memory than a laptop, or you might want to use a script that doesn't try to hold them all in memory at once, or a package meant to be parallelized, like `dask`.What does each classification actually contain? Here are the column headers (*note: this example file may not have all the columns your actual data file will contain*):<jupyter_code>classifications_all.columns<jupyter_output><empty_output><jupyter_text>Each row in the file (i.e., each classification) includes: - **classification_id** - the unique ID assigned to each classification - **user_name** - the username the classifier chose when they registered on the site (this is public-facing as it's what they're identified with when they post on Talk) - **user_id** - the user's ID number in the Zooniverse database (this is not public; in the example file they've been hashed) - **user_ip** - a hashed version of the user's IP address - **workflow_id** - the ID number of the workflow this classification was recorded in - **workflow_name** - the text name of the workflow this classification was recorded in - **workflow_version** - the version number (format `major.minor`) of the workflow - **created_at** - the timestamp from when the classification was recorded - **metadata** - metadata from the classification such as browser information, operating system used - **annotations** - the actual information from the classification (answers / clicks / species identifications / etc, specific to this workflow id+version) - **subject_data** - the data on the subject that was uploaded as part of the subject upload - **subject_ids** - the unique identifier of all subjects classified in this classification (typically 1 subject) We can also quickly look at the first few rows in raw form:<jupyter_code>classifications_all.head()<jupyter_output><empty_output><jupyter_text>*Note: user IDs have been changed for this example file and are not valid Zooniverse user IDs.*Even if you ignore the classification annotations themselves (which this example file is), there's still a lot of information in this classification file. Let's find out some other basic information about the classifications.<jupyter_code>users_all = classifications_all['user_name'].unique()
n_users = len(users_all)

# if the classification is from a classifier who isn't signed in, the user_name field has "not-logged-in-[user_ip]"
is_unreg = np.array([q.startswith("not-logged-in") for q in users_all])
is_reg   = np.invert(is_unreg)

n_unreg = sum(is_unreg)
n_reg   = sum(is_reg)

print("%d classifications from %d classifiers, of which %d (%.0f percent) were signed-in and %d (%.0f percent) were not signed in.\n" % (n_class, n_users, n_reg, (float(n_reg)/float(n_users)*100.), n_unreg, (float(n_unreg)/float(n_users)*100.)))

print("Average classifications per user: %.1f" % (float(n_class)/float(n_users)))
# use created_at to print date range for classifications
print("Classifications registered between %s and %s." % (classifications_all['created_at'][classifications_all.index[0]], classifications_all['created_at'][classifications_all.index[-1]]))
# print out the classification ID of the last classification (useful in some cases)
print("Latest classification ID in this file: %d" % classifications_all['classification_id'][classifications_all.index[-1]])<jupyter_output>Latest classification ID in this file: 71523948
{'fraction_non_alphanumeric': 0.08965393580778196, 'fraction_numerical': 0.012192935269858346, 'mean_word_length': 5.470997679814385, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '656914', 'license': 'MIT', 'n_tokens_mistral': 1623, 'n_tokens_neox': 1528, 'n_words': 686, 'path': 'example_scripts/00 - First Look at Classifications (Notebook - Python 2.7).ipynb', 'repo_name': 'vrooje/Data-digging', 'size': '5577'}