<jupyter_start><jupyter_text>GOAL:The goal of this script is to detect faces & noises in images in a quick way.To create a dataset we need to have many images as soon as possible.And to detect the air brush filter we need to use some ways.One of the quickest way is Median noise detection.Darker areas are photoshopped if the photo is not blurred or low-res.In a photoshopped face, (air brush filter to smooth) hair, eyes and lips are brighter while cheeks are dark etc.Consider this while cleaning your data.If you are not sure, check the original photo as well.If you still are not sure about it, DELETE IMAGE. USE JPG!  Imports<jupyter_code>import numpy as np
import cv2, sys, os, datetime
from PIL import Image as im
import glob
from IPython.display import Image, display
from matplotlib import pyplot as plt<jupyter_output><empty_output><jupyter_text>Face Recognition Libraries<jupyter_code>face_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml')
eye_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_eye.xml')
nose_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_mcs_nose.xml')
mouth_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_mcs_mouth.xml')<jupyter_output><empty_output><jupyter_text>Directory Path CreationEyes, Nose and Mouth are commented out. Remove comment markings if needed.outputs/currentdate/face/...<jupyter_code>current_date_path = "outputs/" + datetime.datetime.now().strftime("%H-%M")        
face_save_path = current_date_path + '/face/'
"""eye_save_path = current_date_path + '/eyes/' 
nose_save_path = current_date_path + '/nose/' 
mouth_save_path = current_date_path + '/mouth/' """

if not os.path.exists(current_date_path):
      os.makedirs(current_date_path)
if not os.path.exists(face_save_path):
      os.makedirs(face_save_path)
"""if not os.path.exists(eye_save_path):
      os.makedirs(eye_save_path)
if not os.path.exists(nose_save_path):
      os.makedirs(nose_save_path)
if not os.path.exists(mouth_save_path):
      os.makedirs(mouth_save_path)"""<jupyter_output><empty_output><jupyter_text>Noise DetectionUsing median filter it converts image located in 'path' in a noised formatThis will help put to detect photoshopped photos manually.*For further studies: Noised photo brightness can be enlightened and can be used for model creation UNCOMMENT DISPLAY ONLY IF YOU USE ONE IMAGE PER RUN<jupyter_code>def detect_noise(path):
    print("Detecting noise")
    print("-" * 30)
    img = im.open(path + '.jpg').convert("L")
    arr = np.array(img)
    removed_noise = median_filter(arr, 3) 
    s = arr - removed_noise
    img = im.fromarray(s)
    if img.mode != 'RGB':
        img = img.convert('RGB')
    img.save(path + "_noise.jpg")
    display(img)
def median_filter(data, filter_size):
    temp = []
    indexer = filter_size // 2
    data_final = []
    data_final = np.zeros((len(data),len(data[0])))
    for i in range(len(data)):

        for j in range(len(data[0])):

            for z in range(filter_size):
                if i + z - indexer < 0 or i + z - indexer > len(data) - 1:
                    for c in range(filter_size):
                        temp.append(0)
                else:
                    if j + z - indexer < 0 or j + indexer > len(data[0]) - 1:
                        temp.append(0)
                    else:
                        for k in range(filter_size):
                            temp.append(data[i + z - indexer][j + k - indexer])

            temp.sort()
            data_final[i][j] = temp[len(temp) // 2]
            temp = []
    return data_final<jupyter_output><empty_output><jupyter_text>Face Recognition using OpenCVEyes, Nose and Mouth are commented out. Resize function: Make every photo in same width and height. (Some networks need this.)To resize the original images initially: (Do this in face_recognition!!)image = Image.open('path')new_image = image.resize((400, 400))<jupyter_code>def face_recognition(image_name, image_extension):
    print("Recognizing...")
    print("-" * 30)
    img = cv2.imread(image_name + image_extension)
    ##RESIZE IMAGE HERE IF NEEDED!!
    gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
    
    plt.imshow(img)
    plt.show()
    
    faces = face_cascade.detectMultiScale(gray, 1.3, 15)
    
    count_face = 0
    
    for (x,y,w,h) in faces:              
        
        roi_gray = gray[y:y+h, x:x+w]
        roi_color = img[y:y+h, x:x+w]
        face_path = face_save_path + image_name + "_%d_face" % count_face
        cv2.imwrite(face_path + '.jpg', roi_color)
        #img = cv2.rectangle(img,(x,y),(x+w,y+h),(255,0,0),1)  
        
        detect_noise(face_path)
        
        """eyes = eye_cascade.detectMultiScale(roi_gray, 1.3, 5)
        count_eye = 0
        for (ex,ey,ew,eh) in eyes:
            crop_eye = roi_color[ey: ey + eh, ex: ex + ew]
            eye_path = eye_save_path + image_name + "_%d_eye" % count_eye
            cv2.imwrite(eye_path + '.jpg', crop_eye)
            #cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(0,255,0),2)
            count_eye += 1
            detect_noise(eye_path)
            
        noses = nose_cascade.detectMultiScale(roi_gray, 1.3, 5)
        count_nose = 0        
        for (ex,ey,ew,eh) in noses:
            crop_nose = roi_color[ey: ey + eh, ex: ex + ew]
            nose_path = nose_save_path + image_name + "_%d_nose" % count_nose
            cv2.imwrite(nose_path + '.jpg', crop_nose)
            #cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(0,0,255),2)
            count_nose += 1
            detect_noise(nose_path)

        mouths = mouth_cascade.detectMultiScale(roi_gray, 2.25, 15)
        count_mouth = 0        
        for (ex,ey,ew,eh) in mouths:
            ey = int(ey - 0.15*eh)
            crop_mouth = roi_color[ey: ey + eh, ex: ex + ew]
            mouth_path = mouth_save_path + image_name + "_%d_mouth" % count_mouth 
            cv2.imwrite(mouth_path + '.jpg', crop_mouth)
            #cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(255,255,255),2)
            count_mouth += 1
            detect_noise(mouth_path)"""
        count_face += 1
    
    #RESIZE
    """scale_percent = 40 # percent of original size
    width = int(roi_color.shape[1] * scale_percent / 100)
    height = int(roi_color.shape[0] * scale_percent / 100)
    dim = (width, height)
    # resize image
    resized = cv2.resize(roi_color, dim, interpolation = cv2.INTER_AREA)

    cv2.imshow('resized',resized)
    cv2.waitKey(0)
    cv2.destroyAllWindows()"""<jupyter_output><empty_output><jupyter_text>Load Image OR Load Image Directory Load Image Directory:    for root, dirs, files in os.walk("reference"):        for file in files:            image_name = os.path.splitext(os.path.basename(file))[0]            image_extension = os.path.splitext(os.path.basename(file))[1]            print(image_name)            face_recognition(image_name, image_extension) Load Image:    image_name = 'a' -- If the image is in a different folder than ipynb give full path!!!    image_extension = '.jpg'    face_recognition(image_name, image_extension)     Paste the necessary code piece to the cell below!<jupyter_code>import glob
if __name__ == '__main__':
    for root, dirs, files in os.walk("originals"):
        for file in files:
            image_name = os.path.splitext(os.path.basename(file))[0]
            image_extension = os.path.splitext(os.path.basename(file))[1]
            print(image_name)
            face_recognition(image_name, image_extension)
    print("DONE")<jupyter_output>test
Recognizing...
------------------------------
{'fraction_non_alphanumeric': 0.11678641410842587, 'fraction_numerical': 0.015284128020901372, 'mean_word_length': 2.8981670061099796, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '435538', 'license': 'MIT', 'n_tokens_mistral': 2676, 'n_tokens_neox': 2538, 'n_words': 609, 'path': 'RetouchDetectorWinEdition/Face_Recognition_Noise_Detection.ipynb', 'repo_name': 'ytugba/My-Otiose-Projects', 'size': '7655'}