<jupyter_start><jupyter_text>Jumble Solver* See: https://en.wikipedia.org/wiki/Jumble The jumble puzzle is a common newspaper puzzle, it contains a series of anagrams that must be solved. To solve, one must solve each of the individual jumbles. The circled letters are then used to create an additional anagram to be solved. In especially difficult versions, some of the anagrams in the first set can possess multiple solutions. To get the final answer, it is important to know all possible anagrams of a given series of letters.Your challenge is to solve the five Jumble puzzles using Spark, where it makes sense to do so. You may use Scala or Python (but Python is slightly preferred). If the final puzzle has multiple possible answers, you are to include an algorithm to determine the most likely one. We have provided a dictionary where the "most common" English words are scored (1=most frequent, 9887=least frequent, 0=not scored due to infrequency of use). For each puzzle, produce the "most likely" (as you determine it) final anagram produced from solving all the other anagrams.Important Notes: Part of your task is to have this be as production ready as possible - while there are only five puzzles now, assume that there could be many more, so use Spark in the most useful way, however you don't need to spend a lot of time on tweaking the parallelization parameters. The code should be deployable and maintainable as well. Don't spend more than 24 hours to complete as much of the assignment as you can.Also included:freq_dict - keys are English Dictionary words to be used in your solving of the jumbles. Non-zero values are the frequency rankings (1=most frequent). Zero values mean that the word is too infrequent to be ranked.pictures of the jumbles we provided for you to solve. You can put these in whatever data format you'd like for your program to read inPlease send us a link to your github repository with the following:Your initial data (from the jumble pictures given)Your codeOutput from your code  Utility Functions<jupyter_code>from pyspark.sql import SparkSession
from pyspark.sql.functions import *
from pyspark.sql.types import *
import json
# The code was removed by DSX for sharing.
def parse_json(fileName):
    with open(fileName) as file:
        data = json.load(file)
        return data<jupyter_output><empty_output><jupyter_text>Dictionary Setup<jupyter_code>def create_dictionary_df(freq_data):
    schema = StructType([
        StructField("word", StringType(), False)
        , StructField("freq", IntegerType(), False)
    ])
    return spark.createDataFrame(freq_data.items(), schema)
def alter_uncommon_words_indicator(dic):
    # https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.types.IntegerType
    MAX_INT = 2**31 - 1
    
    return dic.withColumn('freq', when(dic.freq == 0, MAX_INT).otherwise(dic.freq))
def sort_letters(word):
    return ''.join(sorted(list(word)))
def create_sorted_word_key_column(dic):
    sort_letters_udf = udf(sort_letters, StringType())
    return dic.withColumn('sorted_word', sort_letters_udf(col('word')))<jupyter_output><empty_output><jupyter_text>Load External Files<jupyter_code>def load_dictionary(freqFileName):
    download_from_object_storage(freqFileName)
    freq_data = parse_json(freqFileName)
    dic = create_dictionary_df(freq_data)
    dic = alter_uncommon_words_indicator(dic)
    dic = create_sorted_word_key_column(dic)
    dic.cache()
    return dic
def load_game(gameFileName):
    download_from_object_storage(gameFileName)
    game = parse_json(gameFileName)
    return game<jupyter_output><empty_output><jupyter_text>Gameplay Functionshttps://en.wikipedia.org/wiki/Jumble_algorithm  Part 1<jupyter_code>def guess_word(dic, letters):
    word = (
        dic
        .where(col('sorted_word') == sort_letters(letters))
        .orderBy('freq')
        .select('word', 'freq')
        .first()
        .word
    )
    return word
def get_letters_by_position(word, positions): # 1-based
    letters = []
    for p in positions:
        letters.append(word[p-1])
    return letters
def play_part1(dic, game):
    guessedWords = []
    part2CircledLetters = []
    for entry in game['part1']:
        letters = entry['letters']
        guessedWord = guess_word(dic, letters)
        guessedWords.append(guessedWord)

        circledPositions = entry['circled']
        circledLetters = get_letters_by_position(guessedWord, circledPositions)
        part2CircledLetters.extend(circledLetters)
    
    return (guessedWords, part2CircledLetters)<jupyter_output><empty_output><jupyter_text>Part 2<jupyter_code>def permute_letters(letters, length):
    from itertools import permutations
    perms = list(permutations(letters, length))
    return set(map(lambda w: ''.join(w), perms))
def get_possible_word(dic, letters, length):
    candidates = permute_letters(letters, length)
    candidateWords = spark.createDataFrame(candidates, StringType())
    candidateWords = candidateWords.withColumnRenamed('value', 'word')
    possibleWord = (
        dic
        .join(candidateWords, 'word')
        .orderBy('freq')
        .first()
        )
    
    if possibleWord is None:
        possibleWord = "<! Cannot Determine Word !>"
    else:
        possibleWord = possibleWord.word
    
    return possibleWord
def remove_used_letters(letters, word):
    usedLetters = list(word)
    for c in usedLetters:
        if c in letters:
            letters.remove(c)
    return letters # for chaining
def play_part2(dic, game, circledLetters):
    guessedWords = []
    for numLetters in game['part2']:
        guessedWord = get_possible_word(dic, circledLetters, numLetters)
        guessedWords.append(guessedWord)
        circledLetters = remove_used_letters(circledLetters, guessedWord)
    
    return guessedWords<jupyter_output><empty_output><jupyter_text>Combined<jupyter_code>def play_game(dic, gameFileName):
    game = load_game(gameFileName)
    part1GuessedWords, circledLetters = play_part1(dic, game)
    part2GuessedWords = play_part2(dic, game, circledLetters)
    return {'part1':part1GuessedWords, 'part2':part2GuessedWords}<jupyter_output><empty_output><jupyter_text>Main<jupyter_code>FREQ_FILE = 'freq_dict.json'
spark = SparkSession.builder.getOrCreate()
dic = load_dictionary(FREQ_FILE)
print("Game 1: " + str(play_game(dic, 'game1.json')))
print("Game 2: " + str(play_game(dic, 'game2.json')))
print("Game 3: " + str(play_game(dic, 'game3.json')))
print("Game 4: " + str(play_game(dic, 'game4.json')))
print("Game 5: " + str(play_game(dic, 'game5.json')))<jupyter_output>Game 1: {'part1': ['gland', 'major', 'becalm', 'lawyer'], 'part2': ['and', 'well', 'jobe']}
Game 2: {'part1': ['blend', 'avoid', 'sychee', 'camera'], 'part2': ['are', 'dyad', 'iba']}
Game 3: {'part1': ['stash', 'rodeo', 'indict', 'italic'], 'part2': ['each', '<! Cannot Determine Word !>']}
Game 4: {'part1': ['dinky', 'agiel', 'encore', 'devout'], 'part2': ['addition']}
Game 5: {'part1': ['trying', 'divert', 'seaman', 'deceit', 'shadow', 'kechel'], 'part2': ['events', '<! Cannot Determine Word !>']}
{'fraction_non_alphanumeric': 0.10681658468025299, 'fraction_numerical': 0.008151791988756148, 'mean_word_length': 4.568075117370892, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '525084', 'license': 'MIT', 'n_tokens_mistral': 2280, 'n_tokens_neox': 2181, 'n_words': 656, 'path': 'Jumble Solver.ipynb', 'repo_name': 'superdude264/jumble-solver', 'size': '7115'}