<jupyter_start><jupyter_text>Window functionsGeneral Elections were held in the UK in 2015 and 2017. Every citizen votes in a constituency. The candidate who gains the most votes becomes MP for that constituency.All these results are recorded in a table geyr	| firstName	| lastName	| constituency	| party	| votes---:|-----------|-----------|---------------|-------|------:2015	| Ian	| Murray	| S14000024	| Labour	| 192932015	| Neil	| Hay	| S14000024	| Scottish National Party	| 166562015	| Miles	| Briggs	| S14000024	| Conservative | 86262015	| Phyl	| Meyer	| S14000024	| Green	| 20902015	| Pramod	| Subbaraman	| S14000024	| Liberal Democrat	| 18232015	| Paul	| Marshall	| S14000024	| UK Independence Party	 | 6012015	| Colin	| Fox	| S14000024	| Scottish Socialist Party	| 1972017	| Ian	| MURRAY	| S14000024	| Labour	| 262692017	| Jim	| EADIE	| S14000024	| SNP	| 107552017	| <NAME>	| SMITH	| S14000024	| Conservative	| 94282017	| <NAME>	| BEAL	| S14000024	| Liberal Democrats	| 1388<jupyter_code>import os
import pandas as pd
import findspark
os.environ['SPARK_HOME'] =  '/opt/spark'
findspark.init()

from pyspark.sql import SparkSession
sc = (SparkSession.builder.appName('app09-')
      .config('spark.sql.warehouse.dir', 'hdfs://quickstart.cloudera:8020/user/hive/warehouse')
      .config('hive.metastore.uris', 'thrift://quickstart.cloudera:9083')
      .enableHiveSupport().getOrCreate())<jupyter_output><empty_output><jupyter_text>1. Warming upShow the **lastName, party** and **votes** for the **constituency** 'S14000024' in 2017.<jupyter_code>ge = sc.read.table('sqlzoo.ge')
(ge.filter((ge['constituency']=='S14000024') & (ge['yr']==2017))
    .select('lastname', 'party', 'votes')
    .toPandas())<jupyter_output><empty_output><jupyter_text>2. Who won?You can use the RANK function to see the order of the candidates. If you RANK using (ORDER BY votes DESC) then the candidate with the most votes has rank 1.f**Show the party and RANK for constituency S14000024 in 2017. List the output by party**<jupyter_code>from pyspark.sql.functions import *
from pyspark.sql import Window
(ge.filter((ge['constituency']=='S14000024') & (ge['yr']==2017))
 .select('party', 'votes')
 .withColumn('rank', rank().over(Window.orderBy(col('votes').desc())))
 .orderBy('party')
 .toPandas())<jupyter_output><empty_output><jupyter_text>3. PARTITION BYThe 2015 election is a different PARTITION to the 2017 election. We only care about the order of votes for each year.**Use PARTITION to show the ranking of each party in S14000021 in each year. Include yr, party, votes and ranking (the party with the most votes is 1).**<jupyter_code>(ge.filter(ge['constituency']=='S14000021')
 .withColumn('posn', rank().over(
     Window.partitionBy('yr').orderBy(col('votes').desc())))
 .select('yr', 'party', 'votes', 'posn')
 .orderBy('party', 'yr')
 .toPandas())<jupyter_output><empty_output><jupyter_text>4. Edinburgh ConstituencyEdinburgh constituencies are numbered S14000021 to S14000026.**Use PARTITION BY constituency to show the ranking of each party in Edinburgh in 2017. Order your results so the winners are shown first, then ordered by constituency.**<jupyter_code>(ge.filter((ge['constituency'].between('S14000021', 'S14000026')) &
       (ge['yr']==2017))
 .withColumn('posn', rank().over(
     Window.partitionBy('constituency').orderBy(col('votes').desc())))
 .select('constituency', 'party', 'votes', 'posn')
 .orderBy('posn', 'constituency')
 .toPandas())<jupyter_output><empty_output><jupyter_text>5. Winners OnlyYou can use [SELECT within SELECT](https://sqlzoo.net/wiki/SELECT_within_SELECT_Tutorial) to pick out only the winners in Edinburgh.**Show the parties that won for each Edinburgh constituency in 2017.**<jupyter_code>(ge.filter((ge['constituency'].between('S14000021', 'S14000026')) & (ge['yr']==2017))
 .withColumn('posn', rank().over(Window.partitionBy('constituency').orderBy(col('votes').desc())))
 .filter(col('posn')==1)
 .select('constituency', 'party')
 .orderBy('constituency')
 .toPandas())<jupyter_output><empty_output><jupyter_text>6. Scottish seatsYou can use **COUNT** and **GROUP BY** to see how each party did in Scotland. Scottish constituencies start with 'S'**Show how many seats for each party in Scotland in 2017.**<jupyter_code>(ge.filter((ge['constituency'].startswith('S')) & (ge['yr']==2017))
 .withColumn('posn', rank().over(
     Window.partitionBy('constituency').orderBy(col('votes').desc())))
 .filter(col('posn')==1)
 .groupBy('party')
 .count()
 .toPandas())
sc.stop()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16285211267605634, 'fraction_numerical': 0.07702464788732394, 'mean_word_length': 5.520803443328551, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '660797', 'license': 'MIT', 'n_tokens_mistral': 1789, 'n_tokens_neox': 1530, 'n_words': 388, 'path': 'Spark/09- Window function.ipynb', 'repo_name': 'madlogos/sqlzoo', 'size': '4569'}