Q:

Tips on processing a lot of images in python

I have been trying to process two huge files containing around 40000-50000 images in python. But whenever I try to convert my datasets into a numpy array I get a Memory error. I only have about 8GB RAM which isn't very much, but, because I lack experience in python, I wonder if there is any way that I can resolve this issue by using some python library I don't know about, or maybe by optimizing my code? I would like to hear your opinion on this matter.
My image processing code:
from sklearn.cluster import MiniBatchKMeans
import numpy as np
import glob
import os
from PIL import Image
from sklearn.decomposition import PCA

image_dir1 = "C:/Users/Ai/Desktop/KAGA FOLDER/C/train"
image_dir2 = "C:/Users/Ai/Desktop/KAGA FOLDER/C/test1"
Standard_size = (300,200)
pca = PCA(n_components = 10)
file_open = lambda x,y: glob.glob(os.path.join(x,y))

def matrix_image(image):
    "opens image and converts it to a m*n matrix" 
    image = Image.open(image)
    print("changing size from %s to %s" % (str(image.size), str(Standard_size)))
    image = image.resize(Standard_size)
    image = list(image.getdata())
    image = map(list,image)
    image = np.array(image)
    return image
def flatten_image(image):  
    """
    takes in a n*m numpy array and flattens it to 
    an array of the size (1,m*n)
    """
    s = image.shape[0] * image.shape[1]
    image_wide = image.reshape(1,s)
    return image_wide[0]

if __name__ == "__main__":
    train_images = file_open(image_dir1,"*.jpg")
    test_images = file_open(image_dir2,"*.jpg")
    train_set = []
    test_set = []

    "Loop over all images in files and modify them"
    train_set = [flatten_image(matrix_image(image))for image in train_images]
    test_set = [flatten_image(matrix_image(image))for image in test_images]
    train_set = np.array(train_set) #This is where the Memory Error occurs
    test_set = np.array(test_set)

Small edit: I'm using 64-bit python

A:

Assuming a 4 byte integer for each pixel, you are trying to hold about 11.2 GB of data in (4*300*200*50000 / (1024)**3). Half that for a 2 byte integer.
You have a few options: 

Reduce the number or size of images you are trying to hold in memory 
Use a file or database to hold the data instead of memory (may be too slow for some applications) 
Use the memory you have more effectively...

Instead of copying from list to numpy, which will temporarily use twice the amount of memory, as you do here:
test_set = [flatten_image(matrix_image(image))for image in test_images]
test_set = np.array(test_set)

Do this:
n = len(test_images)
test_set = numpy.zeros((n,300*200),dtype=int)
for i in range(n):
    test_set[i] = flatten_image(matrix_image(test_images[i]))

A:

Since your files are JPEGs and you have 300x200 images, for a 24-bit color image you're looking at approximately 1.4 MB per file and at least a whopping 40.2 GB overall:
In [4]: import humanize # `pip install humanize` if you need it

In [5]: humanize.naturalsize(300*200*24, binary=True)
Out[5]: '1.4 MiB'

In [6]: humanize.naturalsize(300*200*24*30000, binary=True)
Out[6]: '40.2 GiB'

If you have grayscale, you likely have 8-bit images which rings in at 13.4 GB:
In [7]: humanize.naturalsize(300*200*8, binary=True)
Out[7]: '468.8 KiB'

In [8]: humanize.naturalsize(300*200*8*30000, binary=True)
Out[8]: '13.4 GiB'

This is only for one copy too. Depending on the operations, this could get much bigger. 
Going bigger
You could always rent some time on a server with more memory.

AWS - Up to 224GB
Rackspace - Up to 120GB
DigitalOcean - Up to 96 GB
Azure - Up to 56 GB

Looking at these in terms of amount of RAM isn't the only way to think about which servers are best for your workload. There are other differences among providers including IOPS, number of cores, type of CPU, etc.
Test After you Train
After you train your model, you don't need the full set of training data. Delete what you can out of memory. Here in Python land that means not keeping references to the data. Strange beast, yes.
What this likely means is setting up your training data and creating your model within a function that only returns what you need.
Reducing your memory footprint
Let's imagine for a moment that you could store it all in memory. One improvement you can make here is to convert directly from a PIL Image to a numpy array. Existing arrays are not copied, it's a view of the original data. However, it looks like you need to flatten as well into your vector space.
image = Image.open(image)
print("changing size from %s to %s" % (str(image.size), str(Standard_size)))
image = image.resize(Standard_size)
np_image = np.asarray(image).flatten()

EDIT: Actually, this helps your code's maintainability but doesn't help performance. You do this operation on each image in a function individually. The garbage collector will toss the old stuff. Move along, nothing to see here.

