<jupyter_start><jupyter_text>What SERP features does my competitor has that I don't have?We will learn:1. How to pull data programatically from SEMRush2. How to analyze the data 3. Export the results to a CSV for further analysis Extract data from SEMRushYou can find SEMrush API reference herehttps://www.semrush.com/api-analytics/You can find your API key here https://www.semrush.com/api-use/ You can create forms to feed input into the notebook. See https://colab.research.google.com/notebooks/forms.ipynb<jupyter_code>#@title Provide input values


domain = "advanceautoparts.com" #@param {type:"string"}

csv_file = "advanceautoparts.csv" #@param {type:"string"}

key = "TYPE SEMRush API key here" #@param {type:"string"}<jupyter_output><empty_output><jupyter_text>Leverage existing codeWe will borrow my SEMrush code from this article https://searchengineland.com/brands-can-better-understand-users-on-third-party-sites-by-using-a-keyword-overlap-analysis-316157<jupyter_code>import requests
from urllib.parse import urlencode, urlparse, urlunparse, quote
import pandas as pd

#def get_seo_branded_data(brand, domain, database="us", export_columns="Ph,Po,Nq,Ur,Tg,Td,Ts", display_limit=10000, display_filter="+|Ph|Co|{brand}"):
#Found that SERP features is -> Fl in https://www.semrush.com/api-analytics/#columns
def get_serp_features(domain, database="us", export_columns="Fl", display_limit=10000):
   
  #global key
  
  url_params={"type": "domain_organic",
             "key": key,
             "display_limit": display_limit,
             "export_columns": export_columns,
             "domain": domain,
             "database": database
             }

  api_url="https://api.semrush.com/"

  qs = urlencode(url_params)

  u = urlparse(api_url)

  api_request = urlunparse((u.scheme, u.netloc, u.path, u.params, qs, u.fragment))
  
  #print(api_request)
  
  r = requests.get(api_request)
  
  if r.status_code == 200:

    results = r.text.split("\r\n") #
    headers = results[0].split(";") # save result headers to list
    table = [x.split(";") for x in results[1:]] #save columns to list of lists
    
    df = pd.DataFrame(table, columns=headers).dropna() #remove null types
    
    return df
  
  else:
    print("API call failed with code {code}".format(code=r.status_code))
    
    return None<jupyter_output><empty_output><jupyter_text>Test your changes work<jupyter_code>df = get_serp_features(domain)
df.head()<jupyter_output><empty_output><jupyter_text>Scrape the SERP Features referenceWe need to check the SERP Features reference to map the numbers to the name of the feature. See https://www.semrush.com/api-analytics/serp_featuresI used JavaScript code from https://gist.github.com/hamletbatista/b6424dac3801befbef56604a291cf2e3<jupyter_code>#https://www.semrush.com/api-analytics/#serp_features)

#First element selector
#serp_features > div.api-body-table > table > tbody > tr:nth-child(1) > td:nth-child(2)

#We change it to #serp_features > div.api-body-table > table > tbody > tr > td:nth-child(2)

serp_index=["Instant answer", "Knowledge panel", "Carousel", "Local pack", "Top stories", "Image pack", "Site links", "Reviews", "Tweet", "Video", "Featured video", "Featured Snippet", "AMP", "Image", "AdWords top", "AdWords bottom", "Shopping ads", "Hotels Pack", "Jobs search", "Featured images", "Video Carousel", "People also ask"]

example= df["SERP Features"].iloc[0]
example<jupyter_output><empty_output><jupyter_text>Map feature ids to their namesNext we want to map this list of features to their names<jupyter_code>example_list = example.split(",")
example_list
for feature in example_list:
  print(feature)
#for feature in example_list:
#  print(serp_index[feature])
for feature in example_list:
  print(serp_index[int(feature)])<jupyter_output><empty_output><jupyter_text>Generalize solutionNow we will create a function to generalize this<jupyter_code>def get_feature_names(indices):

  serp_index=["Instant answer", "Knowledge panel", "Carousel", "Local pack", "Top stories", "Image pack", "Site links", "Reviews", "Tweet", "Video", "Featured video", "Featured Snippet", "AMP", "Image", "AdWords top", "AdWords bottom", "Shopping ads", "Hotels Pack", "Jobs search", "Featured images", "Video Carousel", "People also ask"]
  
  index_list = indices.split(",")

  feature_names = list()

  for i in index_list:
      if len(i) > 0:
        feature_names.append(serp_index[int(i)])

  return feature_names
example_names = get_feature_names(example)
example_names<jupyter_output><empty_output><jupyter_text>We can convert this list back to a string<jupyter_code>",".join(example_names)<jupyter_output><empty_output><jupyter_text>Update dataframe with feature namesNow, we can add a new column to the dataframe with the names<jupyter_code>df["SERP Features"].apply(lambda x: ",".join(get_feature_names(x)) )
df["SERP Feature Names"] = df["SERP Features"].apply(lambda x: ",".join(get_feature_names(x)) )
df.head()<jupyter_output><empty_output><jupyter_text>Simplify to get only the SERP names<jupyter_code>df_consolidated = df.groupby("SERP Feature Names").count()
df_consolidated<jupyter_output><empty_output><jupyter_text>Download analysis to a CSV<jupyter_code>df_consolidated.to_csv(csv_file)
!ls
from google.colab import files

files.download(csv_file)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12713294580911308, 'fraction_numerical': 0.00881305081567598, 'mean_word_length': 5.528763769889841, 'pattern_counts': {'":': 6, '<': 41, '<?xml version=': 0, '>': 53, 'https://': 9, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '640575', 'license': 'Apache-2.0', 'n_tokens_mistral': 1843, 'n_tokens_neox': 1783, 'n_words': 445, 'path': 'SEMRush_Weekly_Wisdom_Competitor_SERP_Features.ipynb', 'repo_name': 'hamletbatista/semrush', 'size': '5333'}