<jupyter_start><jupyter_text>Welcome to source{d} jgit-spark-connector<jupyter_code>from sourced.engine import Engine
from pyspark.sql import SparkSession
from pyspark.sql.functions import *

spark = SparkSession.builder\
.master("local[*]").appName("Examples")\
.getOrCreate()

engine = Engine(spark, "/repositories", "standard")

print("%d repositories successfully loaded" % (engine.repositories.count()/2))<jupyter_output><empty_output><jupyter_text>Print the schema to show meta data from repositories<jupyter_code>engine.repositories.printSchema()<jupyter_output><empty_output><jupyter_text>Show the first 10 repositories in /repositoriesDouble the number of repositories maybe shown because of remote and local references in git<jupyter_code>engine.repositories.show()<jupyter_output><empty_output><jupyter_text>Filter for only the remote repositories Prints the full remote reference (incase of GitHub, it's the URL) of the first 10 repositories<jupyter_code>engine.repositories\
.references.filter("is_remote = true")\
.select("repository_id")\
.distinct()\
.show(10, False)<jupyter_output><empty_output><jupyter_text>Print schema of blobs in all head commits<jupyter_code>head_blobs = engine.repositories.filter("is_fork = false")\
.references.filter("is_remote = true")\
.head_ref.commits.tree_entries.blobs\
.printSchema()<jupyter_output><empty_output><jupyter_text>Get the first 50 Python blobs from head commits<jupyter_code>head_blobs = engine.repositories.filter("is_fork = false")\
.references.filter("is_remote = true")\
.head_ref.commits\
.tree_entries.blobs\
.classify_languages()\
.filter("is_binary = false")\
.filter("lang = 'Python'")\
.extract_uasts()\
.limit(50)\
.cache()<jupyter_output><empty_output><jupyter_text>Get the schema<jupyter_code>head_blobs.printSchema()<jupyter_output><empty_output><jupyter_text>Print result<jupyter_code>head_blobs.show(50)<jupyter_output><empty_output><jupyter_text>Or instead do a sampling of 5% of all the blobs in both Python & Java<jupyter_code>head_blobs = engine.repositories.filter("is_fork = false")\
.references.filter("is_remote = true")\
.head_ref.commits\
.tree_entries.blobs\
.classify_languages()\
.filter("is_binary = false")\
.filter(col("lang").isin(["Python", "Java"]))\
.sample(False, 0.05, 1024)\
.extract_uasts()\
.cache()
c = head_blobs.count()
print(c)
head_blobs.show(c)<jupyter_output><empty_output><jupyter_text>Get identifier tokens<jupyter_code>idents = head_blobs.query_uast('//*[@roleIdentifier and not(@roleIncomplete)]').cache()
idents.printSchema()
idents = idents.select("repository_id","path","result").distinct()
tokens = idents.extract_tokens().cache()
tokens.printSchema()
tokens.limit(10).show()<jupyter_output><empty_output><jupyter_text>Filter blobs based on a specific token in the blob<jupyter_code>tokens.select("path").where(array_contains("tokens", "print")).show(50, False)<jupyter_output><empty_output><jupyter_text>Top languages per number of blobs Results is only Python because there is a filter above on the blobs<jupyter_code>top_ten_langs = head_blobs.distinct()\
.groupBy("lang").agg(count("*").alias("count"))\
.orderBy("count").sort(desc("count")).limit(10)\
.show()<jupyter_output><empty_output><jupyter_text>Decode the blobs into UTF-8 string Instead of using the UAST's sometimes you want to use code as a string.<jupyter_code>decoded_blobs = engine.repositories.filter("is_fork = false").references\
.head_ref.commits\
.tree_entries.blobs\
.classify_languages()\
.filter("is_binary = false")\
.filter("lang = 'Python'")\
.sample(False, 0.2, 4214)\
.withColumn('content', decode('content', 'UTF-8'))\
.cache()
decoded_blobs.printSchema()
decoded_blobs.show()<jupyter_output><empty_output><jupyter_text>Showing commits that have a certain commit message<jupyter_code>master_commits = engine.repositories\
.references.filter("is_remote = true")\
.master_ref.commits.cache()
master_commits.printSchema()
master_commits.distinct().count()
bug_commits = master_commits\
.filter(col("message").like("%fix%"))\
.filter(col("message").like("%bug%"))\
.distinct()\
.cache()
bug_commits.count()<jupyter_output><empty_output><jupyter_text>By taking samples instead of using limit, you can change the seed and see different results. The second argument to .show() is a bool, when set to True does not truncate the resulting row.<jupyter_code>bug_commits.select("repository_id", "message")\
.sample(False, 0.05, 3421)\
.show(10,True)
bug_commits = master_commits\
.filter(col("message").like("%fix%"))\
.filter(col("message").like("%bug%"))\
.distinct()\
.cache()
bug_commits.count()<jupyter_output><empty_output><jupyter_text>By taking samples instead of using limit, you can change the seed and see different results. The second argument to .show() is a bool, when set to True does not truncate the resulting row.<jupyter_code>bug_commits.select("repository_id", "message")\
.sample(False, 0.05, 3421)\
.show(10,True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1468630219891063, 'fraction_numerical': 0.010691950776679443, 'mean_word_length': 7.6678321678321675, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 65, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '595442', 'license': 'Apache-2.0', 'n_tokens_mistral': 1769, 'n_tokens_neox': 1678, 'n_words': 339, 'path': '_examples/notebooks/Example.ipynb', 'repo_name': 'src-d/jgit-spark-connector', 'size': '4957'}