<jupyter_start><jupyter_text><jupyter_code># Last amended: 21st Feb, 2021
# Myfolder: /home/ashok/Documents/2. data_augmentation
# VM: lubuntu_deeplearning
# Ref: Page 136, Chapter 5, Deeplearning with Python by <NAME>
#      https://stackoverflow.com/questions/29864366/difference-between-function-and-generator

# Objectives:
#              a)How a generator works
#              b)Applications of a generator


# 1.0.4 Display output from multiple commands
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"<jupyter_output><empty_output><jupyter_text>Iterable and iterators
An iterable has an *\_\_iter\_\_()* method that returns an iterator
An iterator has a \_\_next\_\_() method that returns a value from container

 What is a generator?
A generator is very similar to a function
that returns an array, in that a generator has parameters, can be called, and generates a sequence of values. However, instead of building an array containing all the values and returning them all at once, a generator yields the values one at a time, which requires less memory and allows the caller to get started processing the first few values immediately. In short, a generator looks like a function but behaves like an iterator.
A generator is BOTH an iterable (so can be used in for-loop) and is also an iterator, so has a \_\_next\_\_() method.<jupyter_code># Example 1
# Objective: To get values from 1 to infinity, on demand (lazily)
#
# 2.0 
def mygenerator():
    i = 0
    # 1.2
    while True:
        i += 1
        yield i     # 'yield' returns a value
                    # Unlike in return statement
                    # function is not terminated.
                    # This statement pauses the
                    # function saving all its states
                    # and later continues from there
                    # on successive calls.

# 2.1 Using generator as iterator in a for-loop: 
for item in mygenerator():
    # 2.1
    print(item)
    # 2.2
    if item >=4:
        break
# 2.2 Generator with __iter__() method
#     Like an iterable, a generator has __iter__() method
#     but it returns the same generator ('return self')
#     and Not an iterator

# 2.2.1 Lists, tuples, dictionaries,
#       and sets are all iterable objects. 
#       They are iterable containers which 
#       you can get an iterator from.

l = [1,2,3,4]
r = l.__iter__()   # Same as iter(l)
f"Type of r is {type(r)}"
print("\n")
f"Type of r is {iter(r)}"
print("\n")
f"next() method works on an iterator: {next(r)}"
print("\n")
f"--------------"
# 2.2.2
ir = mygenerator().__iter__()
print("\n")
f"type of ir is {type(ir)}"
print("\n")
f"Output of next(ir) is: {next(ir)}"
# 2.3 Also like an iterator, 
#     it returns a value using next()
ab = mygenerator()
next(ab)
# 2.4 Or use in for loop as an iterable, as
for i in ab:
    print(i)
    if i > 20:
        break
## 3
## Example 2
# 3.1 A generator that takes an argument and
#     starts from there
def arggen(st):
    while True:
        st = st * 2
        yield st
# 3.2
t = arggen(4)
print(next(t))
print(next(t))
#################
# 4. Another example
#    https://realpython.com/introduction-to-python-generators/
#################


# 4.1 Execution begins at the start of the function
#     When calling next() the first time,
#     body and continues until the next yield statement
#     where the value to the right of the statement is returned,
#     subsequent calls to next() continue from the yield statement
#     to the end of the function, and loop around and continue from
#     the start of the function body until another yield is called.

# 4.2
def countdown(num):
    print('Starting')
    i = 0
    while num > 0:
        i = i+1          # Note that even value of 'i' will be remembered
        print(i)         #  between calls even though it is not 'yielded'
        yield num
        num -= 1
# 4.3
val = countdown(5)
val
# 4.4
print(next(val))
# 4.5
print(next(val))<jupyter_output>Starting
1
5
2
4<jupyter_text>Applications of Generator  A. In scikit-learn
Generating stratified splits of data.
`StratifiedShuffleSplit` has NO `.fit` or `.transform` method. But has `.split()` method.<jupyter_code># 5.1 Call libraries and create some data
import numpy as np
from sklearn.model_selection import StratifiedShuffleSplit
# 5.2
X = np.array([[1, 2], [3, 4], [1, 2], [3, 4], [1, 2], [3, 4]])
y = np.array([0, 0, 0, 1, 1, 1])

# 5.3
# Instantiate StratifiedShuffleSplit object
sss = StratifiedShuffleSplit(
                             n_splits=5,
                             test_size=0.5,
                             random_state=0
                             )

# 5.4 Unlike other sklearn classes, 'sss'
#     HAS NO .fit or .transform method.
#     Instead, it has .split() method.
#     Get a generator for splitting (X,y)
sp = sss.split(X,y)  # 'sp' is a generator
f"Type of sp is {type(sp)}"
# 5.5 A Generator is both an iterable ie has an __iter__()
#     method and also is an iterator, that is it has a __next__()
#     method
print(next(sp))
print(next(sp))
# 5.6 USe 'sp' in a for-loop
for train_index, test_index in sp:
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]<jupyter_output>TRAIN: [5 0 2] TEST: [4 3 1]
TRAIN: [4 1 0] TEST: [2 3 5]
TRAIN: [0 5 1] TEST: [3 4 2]<jupyter_text>B. Reading a very large file, line-by-line<jupyter_code># 6.0 Reading a very large file
#     line-by-line:
# Ref: https://realpython.com/introduction-to-python-generators/

# 6.1 Call libraries
import os
#Path on our hard-disk
#path = "C:\\Users\\Administrator\\OneDrive\\Documents\\advertising"
#os.chdir(path)
#Path on our Google drive
path = "/content/drive/MyDrive/Colab_data_files/"
os.chdir(path)
os.listdir(path)
# 6.2 
#abc = open("advertising.csv", "r")
abc = open("winequality-red.csv", "r")
# 6.2.1
type(abc)    # _io.TextIOWrapper

# 6.2.2 Read and print complete file
print(abc.read())

# 6.3 Define a function to read complete file
def csv_reader1(file_name):
    for row in open(file_name, "r"):
        print (row)

csv_reader1("winequality-red.csv")       
# 6.4 Define a function to read file, line-by-line
def csv_reader2(file_name):
    for row in open(file_name, "r"):
        yield row

# 6.5 Read line by line. It is lazy.
t = csv_reader2("winequality-red.csv").__iter__()
next(t)
next(t)<jupyter_output><empty_output><jupyter_text>C. Reading a very deep SQL database
See this [link](https://stackoverflow.com/a/23530101/3282777) in StackOverflow  D. As image generator
See this [link](https://github.com/harnalashok/deeplearning/blob/main/image_augmentation.ipynb) in github<jupyter_code>############### I am done ##########333<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11596277738010022, 'fraction_numerical': 0.025053686471009307, 'mean_word_length': 2.770102536427415, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 23, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '406650', 'license': 'Apache-2.0', 'n_tokens_mistral': 2440, 'n_tokens_neox': 2302, 'n_words': 809, 'path': 'generators in python.ipynb', 'repo_name': 'harnalashok/deeplearning', 'size': '6994'}