Have you ever had to work with a dataset or files that were so enormous that they took up all of your machine’s memory? Or perhaps you wanted to create an iterator, but the producer function was so easy that the majority of your code revolved solely around creating the iterator rather than providing the needed values? Python generators and the Python yield statement can aid in these and other situations. Python Generator functions allow you to declare a function that works as an iterator, allowing programmers to quickly, easily and cleanly create an iterator. This article will teach you how to simply generate iterations using Python generators, how it differs from iterators and standard functions, and why you should use it.
Python yield
Before understanding the concept of Python generators, let us look at what exactly does Python yield means.
The yield keyword in Python functions similarly to the return keyword, with the exception that instead of returning a value, it returns a generator object to the caller. When a function is called and the thread of execution encounters a yield keyword in the function, the function execution is terminated at that line and a generator object is returned to the caller.
Generators in Python
- Why use Generators?
Building an iterator in Python requires a significant amount of effort. We must create a class containing __iter__() and __next__() methods, as well as keep track of internal states and raise StopIteration when no values are to be returned. This is both long and contradictory. In such cases, the Python generator comes to the rescue.
- What are Python Generators?
Python Generators are a quick and easy technique to create iterators. A Python generator is a particular function by using a yield statement instead of a return statement that returns an object (iterator) over which we can loop (one value at a time). One significant advantage of the generator is that it consumes far less memory.
In Python, a generator function resembles a normal function except that it contains a yield statement rather than a return statement (it may contain a return too but the yield statement must be qualified as a generator function). Any function with the yield keyword becomes a generator and returns an iterator object that can be iterated through using a for loop.
Create Generators in Python
The yield keyword is the magic ingredient for transforming a simple function into a generator function. It’s the same as defining a regular function, but you use a yield statement instead of a return statement. The function’s state is kept by using the keyword yield, which has the following syntax:
yield [expression_list]
A function becomes a generator function if it contains at least one yield statement (it may also have other yield or return statements). Yield and return will both return a value from a function. The distinction is that a return statement closes a function completely, whereas a yield statement pauses the function while saving all of its states and then continues from there on subsequent calls.
Example
def testyield():
print('Hello')
yield 'Welcome to Python Programming'
output = testyield()
print(output)
print(next(output))
Output
<generator object testyield at 0x7f335886d580>
Hello
Welcome to Python Programming
Here’s a simple yield example. The yield keyword is used in the function testyield(), which returns the string “Welcome to Python Programming.” When the function is invoked, the output is printed, and instead of the actual value, it returns a generator object. To print the actual output, we must feed the object instance to the next() function.
Differences between Generator function and Normal function
The major difference between an ordinary function and a generator function is that the state of generator functions is maintained by utilizing the keyword yield, which works similarly to using return but with some important changes. Another distinction is that generator functions do not even execute a function; instead, they generate and return a generator object.
- One or more yield statements can be found in the generator function.
- When called, it returns an object (iterator) but does not immediately begin execution.
- Methods such as __iter__() and __next__() are automatically implemented. So we can use next to iterate through the objects ().
- When the function yields, it is paused and control is given to the caller.
- Between calls, local variables and their states are remembered.
- Finally, when the function exits, StopIteration is automatically triggered on subsequent calls.
Let us look at the below program to understand a basic difference between a normal function having a return statement and a generator function having a yield statement.
# Normal function
def normal_func():
return 'Good Morning'
#Generator function
def generator_func():
yield 'Good Morning'
print(normal_func()) # calls normal function
print(generator_func()) # calls generator function
Output
Good Morning
<generator object generator_func at 0x7f16dfde45f0>
Here’s an example that demonstrates all of the above concepts. We have numerous yield statements in a generator method called py_gen().
# Python program to illustrate simple generator function
def py_gen():
n = 10
print('Printed First')
# Generator function contains yield statements
yield n
n = n * 2
print('Printed Second')
yield n
n = n * 2
print('Printed Last')
yield n
# It returns an object but does not start execution immediately.
a = py_gen()
print(a)
# We can iterate through the items using next().
print(next(a))
# Once the function yields, the function is paused and the control is transferred to the caller.
# Local variables and theirs states are remembered between successive calls
print(next(a))
print(next(a))
# Finally, when the function terminates, StopIteration is raised automatically on further calls.
print(next(a))
Output
<generator object py_gen at 0x7f2d1bc92580>
Printed First
10
Printed Second
20
Printed Last
40
Traceback (most recent call last):
File "", line 29, in
StopIteration
Note – The value of the variable n is remembered after every yield statement. Unlike our normal function, the local variable is not destroyed when the function yields. Also, the Python generator can be iterated only once.
To restart the process, we must construct a new generator object with the syntax a = py_gen().
Another thing to keep in mind is that we can utilize generators directly with for loops. This is due to the fact that a for loop takes an iterator and iterates over it using the next() function. When StopIteration is raised, it automatically terminates.
# Python program to illustrate for loop in the generator function
def odd_num(start, end):
for i in range (start, end+1):
if i % 2 != 0:
yield(i)
print('Odd numbers are:')
# Using for loop
for num in odd_num(1, 10):
print(num)
Output
Odd numbers are:
1
3
5
7
9
Python Generators with a Loop
Typically, generator functions are implemented in the form of a loop with a suitable terminating condition. This generator function works not only with strings, but also with other types of iterables such as list, tuple, and so on.
def Squares():
i = 1
# An Infinite loop to generate squares
while True:
yield i*i
i += 1
print('Squares of numbers are:')
for num in Squares():
if num > 50:
break
print(num)
Output
Squares of numbers are:
1
4
9
16
25
36
49
Python Generator Expression
Python Generator expressions are used to create simple generators quickly and easily. Similar to how the list comprehension is used to rapidly build a list with very little code, generator expression is used to quickly create a generator with very little code.
Generator expressions generate anonymous generator functions in the same way that lambda functions do. In Python, the syntax for a generator expression is similar to that of a list comprehension. However, the square brackets have been changed with round parentheses.
The primary distinction between a list comprehension and a generator expression is that the former creates the complete list while the latter produces one item at a time.
# list comprehension - Square brackets
even_list = [num**3 for num in range(1, 5)]
print(even_list)
# generator expression - Round brackets
even_gen = (num**3 for num in range(1, 5))
print(even_gen)
# to get elements from the generator
print(next(even_gen))
print(next(even_gen))
print(next(even_gen))
print(next(even_gen))
print(next(even_gen))
Output
[1, 8, 27, 64]
<generator object <genexpr> at 0x7f26d635a5f0>
1
8
27
64
Traceback (most recent call last):
File "", line 14, in
StopIteration
As function arguments, generator expressions can be used. When used in this manner, the round parentheses can be omitted.
# Generator expression can also be used as function arguments
>>> print(sum(num**3 for num in range(1, 5)))
100
>>> print(max(num**3 for num in range(1, 5)))
64
Use of Python Generators
Generators can improve efficiency in a variety of programming contexts. There are various factors that contribute to generators being a powerful solution.
-
Easy to implement
In comparison to its iterator class counterpart, generators can be constructed in a more straightforward and concise manner. The following is an example of how to use an iterator class to implement a power of two sequences.
class PowTwo:
def __init__(self, max=0):
self.n = 0
self.max = max
def __iter__(self):
return self
def __next__(self):
if self.n > self.max:
raise StopIteration
result = 2 ** self.n
self.n += 1
return result
The above program was lengthy and complicated. Now, let’s accomplish the same thing with a generator function, whose implementation is much more concise and clearer.
def PowTwoGen(max=0):
n = 0
while n < max:
yield 2 ** n
n += 1
-
Memory Efficient
A standard function that returns a sequence will first generate the whole sequence in memory before returning the result. If the number of elements in the sequence is really huge, this is overkill. Such sequences’ generator implementation is memory friendly and preferable because it only creates one item at a time.
-
Represents Infinite Stream
Generators are ideal for representing an unending stream of data. Infinite streams cannot be stored in memory, and generators can represent an infinite stream of data because they produce only one item at a time.
-
Pipelining Generators
A set of processes can be pipelined using multiple generators. This is best demonstrated with an example.
Assume we have a generator that generates the Fibonacci numbers. We also have a generator for squaring numbers. If we wish to compute the sum of squares of numbers in the Fibonacci series, we can do it by pipelining the output of generator functions together.
def fibonacci_numbers(nums):
x, y = 0, 1
for _ in range(nums):
x, y = y, x+y
yield x
def square(nums):
for num in nums:
yield num**2
print(sum(square(fibonacci_numbers(8))))
Output
714
Frequently Asked Questions
Q1. What is a generator in Python?
Python Generators are a quick and easy technique to create iterators. A Python generator is a particular function by using a yield statement instead of a return statement that returns an object (iterator) over which we can loop (one value at a time). One significant advantage of the generator is that it consumes far less memory.
In Python, a generator function resembles a normal function except that it contains a yield statement rather than a return statement (it may contain a return too but the yield statement is a must to be qualified as a generator function). Any function with the yield keyword becomes a generator and returns an iterator object that can be iterated through using a for loop.
Q2. When should we use generators in Python?
Generators are a sophisticated tool available in Python. Generators can improve efficiency in a variety of programming contexts. Here are a few examples:
- Processing vast volumes of data/ data files is large: generators allow on-demand calculation, often known as lazy evaluation. Stream processing employs this technique.
- Piping: stacked generators can be used as pipes in the same way as Unix pipes are.
- Concurrency: Concurrency can be generated (simulated) using generators.
Leave a Reply