<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;Making Necessary imports2&nbsp;&nbsp;Defining some additional helper functions3&nbsp;&nbsp;A paramater dictionary for the task4&nbsp;&nbsp;To Do:5&nbsp;&nbsp;How:6&nbsp;&nbsp;Now the question arises what if the the digits are misclassified?7&nbsp;&nbsp;How do we address the above issue ?8&nbsp;&nbsp;Step 1)8.0.1&nbsp;&nbsp;Creating ImageDataGenerator instance and appling it on the input image9&nbsp;&nbsp;Step 2)9.0.1&nbsp;&nbsp;Loading model10&nbsp;&nbsp;Note: From here on the next few cells show what parts of a whole process do11&nbsp;&nbsp;We will put them all together at the end12&nbsp;&nbsp;Step 3)12.0.1&nbsp;&nbsp;Resize the image to desired size and use the find_puzzle function we defined to detect puzzle outline13&nbsp;&nbsp;Step 4)13.0.1&nbsp;&nbsp;Exp :14&nbsp;&nbsp;Step 5)14.0.1&nbsp;&nbsp;Exp :15&nbsp;&nbsp;Note : It is not an issue if the puzzle was detected incorrectly in the above cell15.0.1&nbsp;&nbsp;Run the 2 following cells only if previously the puzzle was detected correctly16&nbsp;&nbsp;Step 6)16.0.1&nbsp;&nbsp;Displaying result to the user17&nbsp;&nbsp;Putting it all together17.0.1&nbsp;&nbsp;In this section, we integrate all the above mentioned parts18&nbsp;&nbsp;Thus we have successfully achieved our goal of solving a Sudoku puzzle from a given picture through OCR and Deep Learning  Making Necessary imports<jupyter_code>import cv2
import skimage
from sudoku import Sudoku
import utilities  # contains some helper functions we shall use
from skimage.segmentation import clear_border
import numpy as np
import tensorflow
from tensorflow.keras.models import load_model
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.preprocessing.image import img_to_array
from utilities.puzzle import find_puzzle
from utilities.puzzle import extract_digit
import matplotlib.pyplot as plt
%matplotlib inline
from utilities.transform import birds_eye_view
from utilities.resizer import resize
from ipywidgets import widgets<jupyter_output><empty_output><jupyter_text>Defining some additional helper functions<jupyter_code>def disp(image):
    plt.figure(figsize=(12,12))
    plt.imshow(image,cmap="gray")

def show_pre_processed_digit(image,x,y):
    image=imutils.resize(image,width=600)
    (puzzleImage,warped)=find_puzzle(image,debug=args["debug"]>0)
    startX=x*stepX
    startY=y*stepY
    endX=(x+1)*stepX
    endY=(y+1)*stepY
    cell=warped[startY:endY,startX:endX]
    digit=extract_digit(cell,debug=args["debug"]>0)
    roi=cv2.resize(digit,(28,28))
    roi=roi.astype("float")/255.0
    plt.imshow(roi,cmap="gray")
    
def predict(sm):
    sm=img_to_array(sm)
    sm=np.expand_dims(sm,axis=0)
    pred=model.predict(sm).argmax(axis=1)[0]
    print("Prediction is ",pred)<jupyter_output><empty_output><jupyter_text>A paramater dictionary for the task<jupyter_code>args={"model":"models/digit_classifier1.h5",
     "image":"images/2/1.jpg",
     "debug":-1}<jupyter_output><empty_output><jupyter_text>The debug argument can be set to any positive value if we want to view the intermediate outputs during the process  To Do:**1) Given a picture of a Sudoku puzzle, we must be able to extract the exact puzzle from the image****2) Extract out every single cell(9*9=81 cells)**                                         **3) Find out if that particular cell contains a digit or not**If yes, then extract the digit from that cell and feed it to our digit classifier modelwhich has been trained on data augmented MNIST dataset of numbers**4) Then feed the puzzle including the classified digits to the Sudoku solver from the Py Sudoku library**                                     **5) Display the solved puzzle on top of the puzzle image provided by user**  How:1) We use some basic image operations and effects, to reduce noise2) We use the (user_defined) find_puzzle function to identify the puzzle border3) We use the (user_defined) extract_digit function to extract number(if it exists) within each cell4) After extracting digits, we load the already trained keras model to classify those digits5) We also try to avoid minor misclassifications,by identifying pairs of numbers that are confusing to our modeland we try to rectify it.(Remember NO model works 100% accurately)6) We feed the above detected puzzle(with classified digits) to py Sudoku solver function7) We use OpenCV to display the solved puzzle on top of the input image   Now the question arises what if the the digits are misclassified?The user generally provides **a single picture.**But that picture might have different backgrounds, different lighting conditions,different angle etc..There are various such features which tend to make the model **misclassify digits** in it  How do we address the above issue ?We use the **ImageDataGenerator** API from tensorflow.keras to perform augmentation on the input imageThen we use multiple images from the generator and try to classify digits in the puzzle.We display the detected puzzle with the classified digits to the userBased on **user feedback**, we choose to use the current image...or to try the next image from the generator  *Note*: Make sure you set the debug paramater to a positive value only when there is a need to view outputs of intermediate steps  Step 1)  Creating ImageDataGenerator instance and appling it on the input image So,we have chosen as image to perform the task on , by including the path in the args dictionary<jupyter_code>img=cv2.imread(args["image"])
print(img.shape)
disp(img)
# Since ImageDataGenerator expects data to be in batches, we reshape the image as batch of 1 image
img=np.array([img])
print(img.shape)

# Instantiate

aug=ImageDataGenerator(rotation_range=3,
                       zoom_range=0.1,
                       width_shift_range=0.05,
                       height_shift_range=0.05,
                       shear_range=0.1,
                       brightness_range=(1.3,1.6))

# Apply to input image

aug_img=aug.flow(img)<jupyter_output>(4000, 1824, 3)
(1, 4000, 1824, 3)<jupyter_text>Step 2) Loading modelUse the load_model function of Keras to load our saved model<jupyter_code>model=load_model(args["model"])<jupyter_output><empty_output><jupyter_text>Note: From here on the next few cells show what parts of a whole process do We will put them all together at the end  Step 3) Resize the image to desired size and use the find_puzzle function we defined to detect puzzle outline Suppose we obtain an image through some augmentation<jupyter_code>for temp in range(1):
    image=next(aug_img).astype("uint8").reshape((4000,1824,3))\
    
    image=resize(image,width=600)
    
    # Using find_puzzle function
    # It returns the original image and the detected puzzle image
    
    (puzzleImage,warped)=find_puzzle(image,debug=args["debug"]>0)<jupyter_output><empty_output><jupyter_text>Step 4) So to determine the dimensions and location of each cell we define some constants<jupyter_code>stepX=warped.shape[1]//9
stepY=warped.shape[0]//9
cellLocs=[]
board=np.zeros((9,9),dtype="int")<jupyter_output><empty_output><jupyter_text>Exp : *step x and step y are constants we use for extracting cell locations while traversing the puzzle*cellLocs is a list we shall use later for displaying output to user*board is the board of numbers we feed to the Sudoku solving algorithm  Step 5)During this step we traverse through all 81 cells of the puzzle,and at each cell we perform a set of operations which will ultimately give us the detected and classified puzzle board<jupyter_code>disp(warped)
for y in range(0,9): # travel through each row
    row=[]
    for x in range(0,9): # travel through each column of that row
        startX = x * stepX
        startY = y * stepY
        endX = (x+1) * stepX
        endY = (y+1) * stepY
        
        row.append((startX,startY,endX,endY))
        
        # Now take out that particular cell from the whole image
        
        cell=warped[startY:endY,startX:endX]

        # Use the extract_digit function we defined to get the digit(if present) in that cell
        
        digit=extract_digit(cell,debug=args["debug"]>0)
        
        # If digit is present, feed it to the model for prediction
        
        if digit is not None:
            roi=cv2.resize(digit,(28,28))
            roi=roi.astype("float")/255.0
            roi=img_to_array(roi)
            # Since we trained the model on (28,28,1) images rather than (28,28)
            roi=np.expand_dims(roi,axis=0)
        
            if model.predict(roi).argmax(axis=1)[0]  in [0,6,8]:
                roi=cv2.resize(digit,(28,28))
                roi=roi.astype("float32")/255.0
                roi=cv2.medianBlur(roi,5)
                roi=img_to_array(roi)
                roi=np.expand_dims(roi,axis=0)
        
            if model.predict(roi).argmax(axis=1)[0] in [1,7]:
                roi=cv2.resize(digit,(28,28))
                roi=roi.astype("float")/255.0
                roi=cv2.GaussianBlur(roi,(5,5),2)
                roi=img_to_array(roi)
                roi=np.expand_dims(roi,axis=0)
        
        
            pred=model.predict(roi).argmax(axis=1)[0]
        
            # Update the board with the classified digit
        
            board[y,x] = pred
        
    cellLocs.append(row)<jupyter_output><empty_output><jupyter_text>Exp :* While performing classification during many testing phase,it was found that some 6s and 7s get misclassified as 8s and 1s respectively.* Lines 28-40 are to perform a bit of additonal cleaning or blurring to reduce the misclassification* These are purely based on research and muliple testings and trials.The additonal operation parameters are not arbitrary<jupyter_code>puzzle=Sudoku(3,3,board=board.tolist())
print("Sudoku board obtained from OCR..")
puzzle.show()<jupyter_output>Sudoku board obtained from OCR..
+-------+-------+-------+
|     4 |       | 6 1   |
|     9 | 8     |       |
|   3 8 |   6 1 |   7 9 |
+-------+-------+-------+
|       |     1 | 1 9   |
|       |   5   |       |
|   2 7 | 1     |       |
+-------+-------+-------+
| 5 6   | 3 2   | 8 4   |
|       |     5 | 9     |
|   9 2 |       | 3     |
+-------+-------+-------+<jupyter_text>Note : It is not an issue if the puzzle was detected incorrectly in the above cellWhile integrating all parts, we will use multiple augmented images(rather than 1)  Run the 2 following cells only if previously the puzzle was detected correctly<jupyter_code>solution=puzzle.solve()
solution.show_full()<jupyter_output>---------------------------
9x9 (3x3) SUDOKU PUZZLE
Difficulty: SOLVED
---------------------------
+-------+-------+-------+
| 7 5 4 | 2 9 3 | 6 1 8 |
| 6 1 9 | 8 4 7 | 2 3 5 |
| 2 3 8 | 5 6 1 | 4 7 9 |
+-------+-------+-------+
| 3 4 5 | 7 8 1 | 1 9 6 |
| 1 8 6 | 9 5 4 | 7 2 3 |
| 9 2 7 | 1 3 6 | 5 8 4 |
+-------+-------+-------+
| 5 6 1 | 3 2 9 | 8 4 7 |
| 8 7 3 | 4 1 5 | 9 6 2 |
| 4 9 2 | 6 7 8 | 3 5 1 |
+-------+-------+-------+<jupyter_text>Step 6) Displaying result to the user<jupyter_code>for (cellRow, boardRow) in zip(cellLocs, solution.board):
    
    # Loop over individual cells in the row
    
    for (box, digit) in zip(cellRow, boardRow):
        # Unpack the cell coordinates
        
        startX, startY, endX, endY = box
        
        # Compute coordinates of where the digit will be drawn
        # on the output puzzle image
        
        textX = int((endX - startX) * 0.33)
        textY = int((endY - startY) * -0.2)
        textX += startX
        textY += endY
        
        # Draw the result digit on the Sudoku puzzle image
        cv2.putText(puzzleImage, str(digit), (textX, textY),
        
        cv2.FONT_HERSHEY_SIMPLEX, 0.9, (0, 255, 255), 2)
# Displaythe output image

cv2.imshow("Sudoku Result", puzzleImage)
cv2.waitKey(0)<jupyter_output><empty_output><jupyter_text>Putting it all together  In this section, we integrate all the above mentioned parts **Changes made*** Here we use multiple augmented images.* First we use one image from the generator,and detect puzzle in it* We display the detected puzzle and ask the user to check * If the puzzle has been detected correctly then we proceed with solving and displaying process* If not,we take another(next) image from the generator and perform the same task* We continue this until the user accepts that the puzzle has been detected correctly<jupyter_code>def check_val(board):
    default=True
    for row in board:
        for dig in row:
            if dig is None:
                return False
    return default    
img=cv2.imread(args["image"])
img=img.reshape((1,4000,1824,3))

aug=ImageDataGenerator(rotation_range=3,zoom_range=0.1,width_shift_range=0.05,height_shift_range=0.05,shear_range=0.1,brightness_range=(1.2,1.5))
aug=aug.flow(img)
print("Loading the model..")
model=load_model(args["model"])
print("Model loaded successfully")
for attempt in range(10):
    # Get image from generator
    image=next(aug_img).astype("uint8").reshape((4000,1824,3))
    
    print("Processing image")
    
    image=resize(image,width=600)
    
    # Use the fn we defined to get images of the puzzle with outline
    
    (puzzleImage,warped)=find_puzzle(image,debug=args["debug"]>0)

    # Initialize an empty 9*9 array where we can fill the digits(as numbers)
    
    board=np.zeros((9,9),dtype="int")

# The warped image is composed of 9*9 cells
# So for one cell size we need to divide total img dimensions by 9 
    stepX=warped.shape[1]//9
    stepY=warped.shape[0]//9

    cellLocs=[] # Useful while displaying answer digits onto the question image
    for y in range(0,9):
        row=[]
        for x in range(0,9):
            startX=x*stepX
            startY=y*stepY
            endX=(x+1)*stepX
            endY=(y+1)*stepY

            row.append((startX,startY,endX,endY))

        # get that cell from the whole image
            cell=warped[startY:endY,startX:endX]
            digit=extract_digit(cell,debug=args["debug"]>0)

            if digit is not None:
                # resize the digit image to mnist type and convert to array
                roi=cv2.resize(digit,(28,28))
                roi=roi.astype("float")/255.0
                roi=img_to_array(roi)
                # SInce we trained on (28,28,1) images rather than (28,28)
                roi=np.expand_dims(roi,axis=0)
                
                if model.predict(roi).argmax(axis=1)[0]  in [0,6,8]:
                    roi=cv2.resize(digit,(28,28))
                    roi=roi.astype("float32")/255.0
                    roi=cv2.medianBlur(roi,5)
                    roi=img_to_array(roi)
                    # SInce we trained on (28,28,1) images rather than (28,28)
                    roi=np.expand_dims(roi,axis=0)
                if model.predict(roi).argmax(axis=1)[0] in [1,7]:
                    roi=cv2.resize(digit,(28,28))
                    roi=roi.astype("float")/255.0
                    roi=cv2.GaussianBlur(roi,(5,5),2)
                    roi=img_to_array(roi)
                    # SInce we trained on (28,28,1) images rather than (28,28)
                    roi=np.expand_dims(roi,axis=0)
                # Pass the digit image array through the model we trained
                pred=model.predict(roi).argmax(axis=1)[0] 
                #plt.title(pred)
                #plt.show()
                board[y,x]=pred
        cellLocs.append(row)
    puzzle=Sudoku(3,3,board=board.tolist())
    print("Sudoku board obtained from OCR..")
    puzzle.show()
    
    temp=0
    
    # Using a text widget to receive user feedback
    
    val=input("Has the puzzle been detected correctly?  y or n   ")
    if val=='y':
        solution=puzzle.solve()
        if check_val(solution.board):
            
            solution.show_full()
            
            temp+=1
            
            for (cellRow, boardRow) in zip(cellLocs, solution.board):
                # Loop over individual cells in the row
                for (box, digit) in zip(cellRow, boardRow):
                    # Unpack the cell coordinates
                    startX, startY, endX, endY = box
        
                    # Compute coordinates of where the digit will be drawn
                    # on the output puzzle image
        
                    textX = int((endX - startX) * 0.33)
                    textY = int((endY - startY) * -0.2)
                    textX += startX
                    textY += endY
        
                    # Draw the result digit on the Sudoku puzzle image
                    cv2.putText(puzzleImage, str(digit), (textX, textY),
        
                    cv2.FONT_HERSHEY_SIMPLEX, 0.9, (0, 255, 255), 2)
                # Display the output image

            cv2.imshow("Sudoku Result", puzzleImage)
            cv2.waitKey(0)
            break
        else:
            print("Invalid puzzle")
if(temp==0):
    print("Unable to detect puzzle and its elements properly..Please try feeding a similar picture again")<jupyter_output>Processing image
Sudoku board obtained from OCR..
+-------+-------+-------+
|     4 |       | 6 1   |
|     9 | 8     |       |
|   3 8 |   6 1 |   7 9 |
+-------+-------+-------+
|       |     7 | 1 9   |
|       |   5   |       |
|   2 7 | 1     |       |
+-------+-------+-------+
| 5 6   | 3 2   | 8 4   |
|       |     5 | 9     |
|   9 2 |       | 3     |
+-------+-------+-------+

Has the puzzle been detected correctly?  y or n   y

---------------------------
9x9 (3x3) SUDOKU PUZZLE
Difficulty: SOLVED
---------------------------
+-------+-------+-------+
| 7 5 4 | 2 9 3 | 6 1 8 |
| 6 1 9 | 8 7 4 | 2 3 5 |
| 2 3 8 | 5 6 1 | 4 7 9 |
+-------+-------+-------+
| 3 8 5 | 6 4 7 | 1 9 2 |
| 1 4 6 | 9 5 2 | 7 8 3 |
| 9 2 7 | 1 3 8 | 5 6 4 |
+-------+-------+-------+
| 5 6 1 | 3 2 9 | 8 4 7 |
| 8 7 3 | 4 1 5 | 9 2 6 |
| 4 9 2 | 7 8 6 | 3 5 1 |
+-------+-------+-------+
{'fraction_non_alphanumeric': 0.11781448822413215, 'fraction_numerical': 0.03700189753320683, 'mean_word_length': 2.497072599531616, 'pattern_counts': {'":': 3, '<': 45, '<?xml version=': 0, '>': 51, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '629538', 'license': 'Apache-2.0', 'n_tokens_mistral': 5960, 'n_tokens_neox': 5453, 'n_words': 1831, 'path': 'OCR_SUDOKU_SOLVER/Sudoku_Solver_OCR.ipynb', 'repo_name': 'HrithikViknesh/Projects', 'size': '17918'}