<jupyter_start><jupyter_text>**py_tree_ros library Data Collection*** I'm intersted in code using the following library "py_trees_ros" so I'll query all code files containing this string and has python file format. Similar step will be done for behaviortree.cpp library using the term:  "main_tree_to_execute"* To run the code, enter your GitHub access token. You can create one by following these [instructions](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token)<jupyter_code>def access_token(token):
    ACCESS_TOKEN = token ## enter your githun token
    g = Github(ACCESS_TOKEN)
    
    try:
        repo = g.get_repo("PyGithub/PyGithub")
        if repo.full_name == "PyGithub/PyGithub":
#         print(repo.full_name)#
            print("Access token works")
            return g
    except BadCredentialsException:
        print("Please provide a working access token")
        
## a function to query github by entering a specific string to search on files level
def query_github(g):
    g= g   
    keywords = input('Enter keyword(s)[e.g py_trees_ros, main_tree_to_execute]: ')
    keywords = [keyword.strip() for keyword in keywords.split(',')]
    query = f'"{keywords}" in:file extension:py' ### change extension to XML for "main_tree_to_execute"
    result = g.search_code(query, order='desc') 
    return result        

token = input('Enter your Github access token: ')
g = access_token(token)
result = query_github(g)
print(f'Found {result.totalCount} file(s)')<jupyter_output>Found 257 file(s)<jupyter_text>Only run below cell if you want to limit the number of returned files by Github<jupyter_code>## a function to limit the number of returned files, please only run this cell if you want to limit number of returned 
## files, otherwise ignore this cell
def limit_result_size(result, desired_size):    
    max_size = desired_size  
    if result.totalCount >= max_size:
        print(f'The number of mined GitHub files is {result.totalCount} and it was limmited to {max_size} ')
        return result[:max_size]

## change here depending on number of files that you want to download. GitHub API has a limit of maximum 1000 files
desired_size = 100
result = limit_result_size(result, desired_size)

new_repo = {}
for file in result:
    url = file.download_url 

    repo_name = file.repository.full_name.replace('/','_')
    
    if repo_name in new_repo:
        # append the new url to the existing array at this slot
        new_repo[repo_name].append(url)
    else:
        # create a new array in this slot
        new_repo[repo_name] = [url]

print("The number of repository: {}".format(len(new_repo)))
##If you want to display the repository and URL for each file uncomment below line
#new_repo
## extract wanted projects
## after manual filtering and exploring the repository a list of wanted repo was defined and passed to the "repo_name_list" variable. This can be passed as a file as well instead of a list
wanted_repo = {}

repo_name_list =['smarc-project_smarc_missions','simutisernestas_mobile_robot_project','samiamlabs_dyno','refills-project_refills_second_review','peterheim1_gizmo', 'jotix16_Robotics-Behaviour-Planning','braineniac_robotics-player','KKalem_sam_march','Taospirit_roborts_project']
for i  in range(len(repo_name_list)):
    if repo_name_list[i] in new_repo:
        # append the new url to the existing array at this slot
        wanted_repo[repo_name_list[i]] = new_repo[repo_name_list[i]]
print("The number of filtered repository: {}".format(len(wanted_repo)))<jupyter_output>The number of filtered repository: 9<jupyter_text>**Downloading Raw Data*** Downloading the raw data to calculate the base metric<jupyter_code>## downloading the matched files from above query
file_count = 0
for key,value in wanted_repo.items():
      for i in range(len(value)):
            r = requests.get(value[i])
            ## if you want to change the path where files are saved change  "/home/jovyan/src/notebooks/downloads/pytreeros" to where you want, also for XML fromat files change the re.search to xml instead of py 
            open(os.path.join("/home/jovyan/src/notebooks/downloads/pytreeros/", os.path.basename(key+'_'+(re.search('/(\w*((%20\w)*)?.py)', value[i]).group(1)))) , "wb").write(r.content)
            file_count += 1

print("Download complete for {} files".format(file_count))<jupyter_output><empty_output><jupyter_text>**BehaviorTree.CPP library Data Collection*** I'm intersted in code using the following library "BehaviorTree.CPP" so I'll query all code files containing this string "main_tree_to_execute" and has xml file format. * To run the code, enter your GitHub access token. You can create one by following these [instructions](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token)<jupyter_code>def access_token(token):
    ACCESS_TOKEN = token ## enter your githun token
    g = Github(ACCESS_TOKEN)
    
    try:
        repo = g.get_repo("PyGithub/PyGithub")
        if repo.full_name == "PyGithub/PyGithub":
#         print(repo.full_name)#
            print("Access token works")
            return g
    except BadCredentialsException:
        print("Please provide a working access token")
        
## a function to query github by entering a specific string to search on files level       
def query_github(g):
    g= g   
    keywords = input('Enter keyword(s)[e.g py_trees_ros, main_tree_to_execute]: ')
    keywords = [keyword.strip() for keyword in keywords.split(',')]
    query = f'"{keywords}" in:file extension:xml' ### change extension to py for "py_tree_ros"
    result = g.search_code(query, order='desc') 
    return result
token = input('Enter your Github access token: ')
g = access_token(token)
result = query_github(g)
print(f'Found {result.totalCount} file(s)')
def limit_result_size(result, desired_size):    
    max_size = desired_size  
    if result.totalCount >= max_size:
        print(f'The number of mined GitHub files is {result.totalCount} and it was limmited to {max_size} ')
        return result[:max_size]

desired_size = 100
result = limit_result_size(result, desired_size)
new_repo = {}
for file in result:
    url = file.download_url 

    repo_name = file.repository.full_name.replace('/','_')
    
    if repo_name in new_repo:
        # append the new url to the existing array at this slot
        new_repo[repo_name].append(url)
    else:
        # create a new array in this slot
        new_repo[repo_name] = [url]
print("The number of repository: {}".format(len(new_repo)))
##If you want to display the repository and URL for each file uncomment below line
#new_repo
## extract wanted projects
## after manual filtering and exploring the repository a list of wanted repo was defined and passed to the "repo_name_list" variable. This can be passed as a file as well instead of a list
wanted_repo = {}

repo_name_list =['ajbandera_MiRON-project', 'IntelligentRoboticsLabs_robocup2020', 'ipa-rar_pickplace', 'Adlink-ROS_BT_ros2','vislab-tecnico-lisboa_vizzy_playground','alexandrethm_mecatro-P17','ParthasarathyBana_behavior_tree_roscpp','skylerpan_neuronbot2_multibot','vislab-tecnico-lisboa_vizzy_behavior_trees','kmi-robots_hans-ros-supervisor','MROS-RobMoSys-ITP_Pilot-URJC','CARVE-ROBMOSYS_BTCompiler','julienbayle_stardust','CARVE-ROBMOSYS_carve-scenarios-config','CARVE-ROBMOSYS_Yarp-SmartSoft-Integration','MiRON-project_bundles']
for i  in range(len(repo_name_list)):
    if repo_name_list[i] in new_repo:
        # append the new url to the existing array at this slot
        wanted_repo[repo_name_list[i]] = new_repo[repo_name_list[i]]
print("The number of filtered repository: {}".format(len(wanted_repo)))<jupyter_output>The number of filtered repository: 16<jupyter_text>**Downloading Raw Data*** Downloading the raw data to calculate the base metric<jupyter_code>## downloading the matched files from above query
file_count = 0
for key,value in wanted_repo.items():
      for i in range(len(value)):
            r = requests.get(value[i])
            ## if you want to change the path where files are saved change  "/home/jovyan/src/notebooks/downloads/pytreeros" to where you want, also for XML fromat files change the re.search to xml instead of py 
            open(os.path.join("/home/jovyan/src/notebooks/downloads/behaviortreecpp/", os.path.basename(key+'_'+(re.search('/(\w*((%20\w)*)?.py)', value[i]).group(1)))) , "wb").write(r.content)
            file_count += 1

print("Download complete for {} files".format(file_count))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10308677926616192, 'fraction_numerical': 0.0043098427489807805, 'mean_word_length': 4.296730413325108, 'pattern_counts': {'":': 2, '<': 18, '<?xml version=': 0, '>': 20, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '274865', 'license': 'MIT', 'n_tokens_mistral': 2707, 'n_tokens_neox': 2619, 'n_words': 795, 'path': 'scripts/notebooks/BT-mining-script.ipynb', 'repo_name': 'RazanGhzouli/Behavior-Trees-in-Action', 'size': '8585'}