<jupyter_start><jupyter_text>Self join Edinburgh Buses[Details of the database](https://sqlzoo.net/wiki/Edinburgh_Buses.) Looking at the data```stops(id, name)route(num, company, pos, stop)```<jupyter_code>import os
import pandas as pd
import findspark
os.environ['SPARK_HOME'] =  '/opt/spark'
findspark.init()

from pyspark.sql import SparkSession
sc = (SparkSession.builder.appName('app09')
      .config('spark.sql.warehouse.dir', 'hdfs://quickstart.cloudera:8020/user/hive/warehouse')
      .config('hive.metastore.uris', 'thrift://quickstart.cloudera:9083')
      .enableHiveSupport().getOrCreate())<jupyter_output><empty_output><jupyter_text>1.How many **stops** are in the database.<jupyter_code>stops = sc.read.table('sqlzoo.stops')
route = sc.read.table('sqlzoo.route')
stops.agg({'id': 'count'}).toPandas()<jupyter_output><empty_output><jupyter_text>2.Find the **id** value for the stop 'Craiglockhart'<jupyter_code>stops.filter(stops['name']=='Craiglockhart').select('id').toPandas()<jupyter_output><empty_output><jupyter_text>3.Give the **id** and the **name** for the **stops** on the '4' 'LRT' service.<jupyter_code>from pyspark.sql.functions import *
(stops.join(route, stops['id']==route['stop'], how='left')
 .filter((col('num')=='4') & (col('company')=='LRT'))
 .select('id', 'name').toPandas())<jupyter_output><empty_output><jupyter_text>4. Routes and stopsThe query shown gives the number of routes that visit either London Road (149) or Craiglockhart (53). Run the query and notice the two services that link these stops have a count of 2. Add a HAVING clause to restrict the output to these two routes.<jupyter_code>(route.filter(route['stop'].isin([149, 53]))
    .groupBy('company', 'num')
    .agg({'stop': 'count'})
    .filter("count(stop)==2")
    .toPandas())<jupyter_output><empty_output><jupyter_text>5.Execute the self join shown and observe that b.stop gives all the places you can get to from Craiglockhart, without changing routes. Change the query so that it shows the services from Craiglockhart to London Road.<jupyter_code>(route.join(route
            .withColumnRenamed('pos', 'pos2')
            .withColumnRenamed('stop', 'stop2'), 
            ['company', 'num'], how='inner')
    .filter((col('stop')==53) & (col('stop2')==149))
    .select('company', 'num', 'stop', 'stop2')
    .toPandas())<jupyter_output><empty_output><jupyter_text>6.The query shown is similar to the previous one, however by joining two copies of the **stops** table we can refer to **stops** by **name** rather than by number. Change the query so that the services between 'Craiglockhart' and 'London Road' are shown. If you are tired of these places try 'Fairmilehead' against 'Tollcross'<jupyter_code>(route.join(route
            .withColumnRenamed('pos', 'pos2')
            .withColumnRenamed('stop', 'stop2'), 
            ['company', 'num'])
    .join(stops, col('stop')==stops['id'])
    .join(stops
          .withColumnRenamed('id', 'id2')
          .withColumnRenamed('name', 'name2'), 
          col('stop2')==col('id2'))
    .filter((col('name')=="Craiglockhart") & (col('name2')=="London Road"))
    .select('company', 'num', 'name', 'name2')
    .toPandas())<jupyter_output><empty_output><jupyter_text>7. [Using a self join](https://sqlzoo.net/wiki/Using_a_self_join)Give a list of all the services which connect stops 115 and 137 ('Haymarket' and 'Leith')<jupyter_code>(route.join(route
            .withColumnRenamed('pos', 'pos2')
            .withColumnRenamed('stop', 'stop2'),
            ['company', 'num'])
    .filter((col('stop')==115) & (col('stop2')==137))
    .select('company', 'num')
    .dropDuplicates()
    .toPandas())<jupyter_output><empty_output><jupyter_text>8.Give a list of the services which connect the stops 'Craiglockhart' and 'Tollcross'<jupyter_code>(route.join(route
            .withColumnRenamed('pos', 'pos2')
            .withColumnRenamed('stop', 'stop2'),
            ['company', 'num'])
    .join(stops, col('stop')==col('id'))
    .join(stops
          .withColumnRenamed('id', 'id2')
          .withColumnRenamed('name', 'name2'), 
          col('stop2')==col('id2'))
    .filter((col('name')=="Craiglockhart") & 
            (col('name2')=="Tollcross"))
    .select('company', 'num')
    .toPandas())<jupyter_output><empty_output><jupyter_text>9.Give a distinct list of the **stops** which may be reached from 'Craiglockhart' by taking one bus, including 'Craiglockhart' itself, offered by the LRT company. Include the company and bus no. of the relevant services.<jupyter_code>(route.join(route
            .withColumnRenamed('pos', 'pos2')
            .withColumnRenamed('stop', 'stop2'),
            ['company', 'num'])
    .join(stops, col('stop')==col('id'))
    .join(stops
          .withColumnRenamed('id', 'id2')
          .withColumnRenamed('name', 'name2'), 
          col('stop2')==col('id2'))
    .filter((col('name')=="Craiglockhart") & (col('company')=="LRT"))
    .select('name2', 'company', 'num')
    .dropDuplicates()
    .toPandas())<jupyter_output><empty_output><jupyter_text>10.Find the routes involving two buses that can go from **Craiglockhart** to **Lochend**.Show the bus no. and company for the first bus, the name of the stop for the transfer,and the bus no. and company for the second bus.> _Hint_    > Self-join twice to find buses that visit Craiglockhart and Lochend, then join those on matching stops.<jupyter_code>bus1 = (route.join(route
            .withColumnRenamed('pos', 'pos2')
            .withColumnRenamed('stop', 'stop2'),
            ['company', 'num'])
    .join(stops, col('stop')==col('id'))
    .join(stops
          .withColumnRenamed('id', 'id2')
          .withColumnRenamed('name', 'name2'), 
          col('stop2')==col('id2'))
    .filter(col('name')=="Craiglockhart")
    .select('name2', 'company', 'num', 'stop2')
    .dropDuplicates())
bus2 = (route.join(route
            .withColumnRenamed('pos', 'pos2')
            .withColumnRenamed('stop', 'stop2'),
            ['company', 'num'])
    .join(stops, col('stop')==col('id'))
    .join(stops
          .withColumnRenamed('id', 'id2')
          .withColumnRenamed('name', 'name2'), 
          col('stop2')==col('id2'))
    .filter(col('name2')=="Lochend")
    .select('stop', 'company', 'num')
    .dropDuplicates())
(bus1.join(bus2
           .withColumnRenamed('company', 'company2')
           .withColumnRenamed('num', 'num2'), 
           bus1['stop2']==bus2['stop'])
    .select('num', 'company', 'name2', 'num2', 'company2')
    .toPandas())
sc.stop()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17916349809885931, 'fraction_numerical': 0.016425855513307986, 'mean_word_length': 4.093725793958172, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 47, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '526279', 'license': 'MIT', 'n_tokens_mistral': 2303, 'n_tokens_neox': 2195, 'n_words': 482, 'path': 'Spark/09 Self join.ipynb', 'repo_name': 'madlogos/sqlzoo', 'size': '6575'}