Python’s Map(), Filter(), and Reduce() Functions Explained
Demonstrating the building blocks of functional programming in Python with code examples, mathematics, and…emojis
At its roots, Python is an object-oriented programming language, built upon an imperative programming paradigm. When performing a sequence of statements, typically a structure such as below (for-loop) is used:
my_list= [17.43, -0.99, 6.81, -12.25, 4.52]
my_list_squared = []for i in my_list:
i_squared = i**2
my_list_squared.append(i_squared)
Each statement brings us a bit closer to the desired end result (in this case a list of squared numbers), by performing operations on the object of interest (the original list).
Alternatively, we could build a list comprehension to accomplish the same task in fewer lines of code:
my_list_squared = [i**2 for i in my_list]In both cases, the data is central, performing repeated operations on my_list to achieve the end state.
In functional programming, things work a bit differently. Unlike OOP, it relies on a declarative programming model, in which functions are at the core. Although not a functional programming language, Python does include several powerful functions to cater to this type of programming.
In functional form, using the map() function, the aforementioned procedure would look like this:
my_list_squared = list(map(lambda i: i**2, my_list))This article (very) briefly discusses the concept of functional programming and its potential benefits, and describes three key building blocks — the map(), filter() and reduce() functions — for applying functional programming principles in Python.
What is functional programming?
The quintessential property of functional programming is that it emphasizes the operations we perform, rather than the objects we perform them on. For problems where we perform many distinct operations (i.e., many functions) on a relatively limited number of objects, it may be the preferred programming paradigm.
Central in functional programming are higher-order functions. These functions take another function as input, making them powerful general purpose expressions. In mathematical terms, one could think of a derivative d/dx, taking a function f as input. This article discusses three key higher-order functions that can be used in Python: map(), filter() and reduce().
Although the preferred approach is heavily problem-dependent, functional programming has several (potential) benefits over object-oriented programming. Each function represents a standalone piece of code. It takes input, transforms it, and outputs it. This concise and modular structure often eases debugging and maintenance. Furthermore, its inherent lazy evaluation — only yielding outputs when requested — can be more memory-efficient. Finally, functional programming often eases parallelization.
Functional programming may be preferable when working with many data operations and few ‘things’ (e.g., a relatively small number of variables, sets, etc.). In contrast, OOP often works better when there are many ‘things’ but relatively few operations. Keep in mind both often get the job done, and that many programming languages incorporate elements of both.
Although gaining popularity, functional languages such as LISP or Haskell can only boast limited adoption compared to widespread OOP languages such as Python or C++. OOP languages increasingly incorporate elements of functional programming however, such that comparable structures can be applied.
Now that the fundamentals of functional programming have been introduced, it is time to move to concrete Python implementations.
Map( )
Being a higher-order function, the map function takes another function and an iterable (e.g., a list, set, tuple) as input, applies the function to the iterable, and returns an output. It’s syntax is defined as follows:
map(function, iterable)
For those mathematically inclined, it may be convenient to think in terms of mapping from a domain X to a domain Y:
f:X →Y
or alternatively:
f(x)=y, ∀x∈X
For visual learners, an emoji example of map() — credit to GlobalNerdy for the inspiration — may be more helpful:
map(cook, [🐷,🌽,🥚 ,🍠]) → [🥓,🍿,🍳,🍟]
The output y is a map object (an iterable), which we still have to convert to a set or list. To combine both steps, we can simply wrap the map statement inside, e.g., the set() function: my_set=set(map(function, iterable)).
Let’s go for a concrete example. Suppose we have a large number of strings, which we want to concatenate with “_2022”. Data science is full of such relatively simple operations that need to performed on large data sets. With map(), we could express it as
string_map_22 = map(lambda my_string: my_string + '_2022', set_of_strings)This way of working is quite powerful — with one swoop, we update the entire set! Don’t forget to convert the map object to a set though:
set_of_strings_22 = set(string_map_22)One more? Suppose we want to transform a log transformation on a list of salaries. Although working with lambda function yields concise codes, for efficiency purposes it is recommended to have a pre-defined function using def. Furthermore, we now directly wrap the map() function in a list() function:
def get_log_value(salary: int):
return np.log(salary)log_salaries = list(map(get_log_value, salaries))
Filter()
Similar to map(), the filter() higher-order function takes a function and an iterable as inputs. The function in case needs to be of a Boolean nature, returning True/False values corresponding to the filter conditions. As output, it returns a subset of the input data that meets the conditions stipulated by the function.
Mathematically, a filter operation might be loosely specified as follows:
f:X →X’, with X’⊆X
(more specifically, we might define a Boolean vector to weed out the false-values elements out of the set, but let’s keep things concise here)
Circling back to the food example, suppose we defined a filtering function non_vegan(). We could then apply the following filter:
filter(non_vegan, [🥓,🍿,🍳,🍟]) → [🥓,🍳]
Let’s check an example again. Suppose we have a DataFrame with tweets, and wish to filter the retweets (represented by ‘RT’ being the first two characters of the tweet). We can do this as follows:
def check_retweet(tweet: str):
return tweet[:2]=='RT'list_retweets = list(filter(check_retweet, tweets_df['Tweet text']))
Pretty neat, right?
Reduce
Unlike the previous two functions, reduce() must be imported from functools to work properly. It returns a single value (i.e., it reduces the input to a single element). Commonly, this would be something like the sum of all elements in a list.
In fact, the very reason reduce() is not built into Python 3, is that it was often used simply to compute sums. According to Python founder Van Rossum:
“I ended up hating reduce() because it was almost exclusively used (a) to implement sum(), or (b) to write unreadable code. So we added builtin sum() at the same time we demoted reduce() from a builtin to something in functools (which is a dumping ground for stuff I don’t really care about :-)”
The mathematical representation:
f:X →ℝ
An emoji example:
reduce(mix, [🥬,🥕,🍅,🥒]) →🥗
As mentioned, the reduce function is typically used for operations such as summing or multiplying all elements in a list. Let’s see the multiplier implementation:
from functools import reduceproduct = reduce(lambda x, y: x*y, list_integers)
Again, little code is needed to perform the operation.
For all three higher-order functions, there are more esoteric use cases (e.g., taking multiple iterators as input, nested structures), but these are beyond the scope of this introductory article.
Closing words
Functional programming puts functions — rather than objects — at the core of programming practice. Although Python is primarily an object-oriented language, three higher-order functions — map(), function(), and reduce() — go a long way in utilizing functional programming concepts in Python. Each takes a function and an iterable as input. Let’s recap once more:
- Map(): Perform the same operation on all elements in an iterable. An example is performing a log transformation on each element.
- Filter(): Filters a subset of elements that meets a certain (set of) condition(s). An example is to filter out sentences that contain a specific string.
- Reduce(): Performs an operation on an iterable, yielding a single-valued outcome. A common example is to sum all elements in a list, yielding a single number as output.
As with many problems, there is no one-size-fits-all solution. The big question is: when to use functional programming in Python? As mentioned before, there are cases where functional programming is most appropriate. However, if we are committed to Python, when concretely would we adopt the functions discussed in this article?
Truthfully, not that often. List comprehensions are often (substantially) faster and — somewhat arbitrarily — easier to read. Furthermore, NumPy operations are typically superior for large and repetitive array operations. Likely, the majority of Python programmers will never need to use the functions listed in this article (anti-climax, I know).
In the end, Python — despite offering some higher-level functions — is simply not a functional programming language. It is not designed that way. Slapping dirt tires on a race car doesn’t make it a rally car. Python does not have compilers that maximize the benefits of functional programming, and as such does not offer the same advantages as pure functional languages do.
To quote Van Rossum one last time:
“If I think of functional programming, I mostly think of languages that have incredibly powerful compilers, like Haskell. For such a compiler, the functional paradigm is useful because it opens up a vast array of possible transformations, including parallelization. But Python’s compiler has no idea what your code means, and that’s useful too. So, mostly I don’t think it makes much sense to try to add “functional” primitives to Python, because the reason those primitives work well in functional languages don’t apply to Python, and they make the code pretty unreadable for people who aren’t used to functional languages (which means most programmers).
I also don’t think that the current crop of functional languages is ready for mainstream. Admittedly I don’t know much about the field besides Haskell, but any language *less* popular than Haskell surely has very little practical value, and I haven’t heard of functional languages *more* popular than Haskell. As for Haskell, I think it’s a great proving ground for all sorts of ideas about compiler technology, but I think its “purity” will always remain in the way of adoption. Having to come to grips with Monads just isn’t worth it for most people.”
Comments