<jupyter_start><jupyter_text>Overview of Sub QueriesLet us recap about Sub Queries.<jupyter_code>%%HTML
<iframe width="560" height="315" src="https://www.youtube.com/embed/6AOlttTRG48?rel=0&amp;controls=1&amp;showinfo=0" frameborder="0" allowfullscreen></iframe><jupyter_output><empty_output><jupyter_text>Let us start spark context for this Notebook so that we can execute the code provided. You can sign up for our [10 node state of the art cluster/labs](https://labs.itversity.com/plans) to learn Spark SQL using our unique integrated LMS.<jupyter_code>val username = System.getProperty("user.name")
import org.apache.spark.sql.SparkSession

val username = System.getProperty("user.name")
val spark = SparkSession.
    builder.
    config("spark.ui.port", "0").
    config("spark.sql.warehouse.dir", s"/user/${username}/warehouse").
    enableHiveSupport.
    appName(s"${username} | Spark SQL - Windowing Functions").
    master("yarn").
    getOrCreate<jupyter_output><empty_output><jupyter_text>If you are going to use CLIs, you can use Spark SQL using one of the 3 approaches.**Using Spark SQL**```spark2-sql \    --master yarn \    --conf spark.ui.port=0 \    --conf spark.sql.warehouse.dir=/user/${USER}/warehouse```**Using Scala**```spark2-shell \    --master yarn \    --conf spark.ui.port=0 \    --conf spark.sql.warehouse.dir=/user/${USER}/warehouse```**Using Pyspark**```pyspark2 \    --master yarn \    --conf spark.ui.port=0 \    --conf spark.sql.warehouse.dir=/user/${USER}/warehouse``` * We typically have Sub Queries in **FROM** Clause.* We need not provide alias to the Sub Queries in **FROM** Clause in Spark SQL. In earlier versions, you might have to provide alias for the Sub Query.* We use Sub Queries quite often over queries using Analytics/Windowing Functions<jupyter_code>%%sql

SELECT * FROM (SELECT current_date)
%%sql

SELECT * FROM (SELECT current_date) AS q<jupyter_output><empty_output><jupyter_text>Let us see few more examples with respect to Sub Queries.<jupyter_code>%%sql

USE itversity_retail
%%sql

SELECT * FROM (
    SELECT order_date, count(1) AS order_count
    FROM orders
    GROUP BY order_date
) q
LIMIT 10<jupyter_output><empty_output><jupyter_text>```{note}Here is an example of how we can filter based up on the derived columns using sub query. However, this can be achieved with direct query as well using `HAVING`. ```<jupyter_code>%%sql

SELECT * FROM (
    SELECT order_date, count(1) AS order_count
    FROM orders
    GROUP BY order_date
) q
WHERE q.order_count > 10
%%sql

SELECT order_date, count(1) AS order_count
FROM orders
GROUP BY order_date
    HAVING count(1) > 10<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12096168294515403, 'fraction_numerical': 0.012396694214876033, 'mean_word_length': 4.582809224318658, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 25, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '506310', 'license': 'MIT', 'n_tokens_mistral': 890, 'n_tokens_neox': 842, 'n_words': 260, 'path': '07_windowing_functions/09_overview_of_sub_queries.ipynb', 'repo_name': 'itversity/spark-sql', 'size': '2662'}