Friday, February 17, 2023

Introduction to Higher Order Functions Python

 

Introduction to Higher Order Functions

A higher order function (HOF) is a function that takes one or more functions as arguments or return a function as a result, or both. They are supported by a wide range of programming languages including the most popular ones such as Python, C++ and JavaScript.

An example of Higher Order Functions that typically built-in within programming languages are:

map fold filter Although they are simple concepts, knowing how to use them can very much improve the way you write code :). So let’s get started!

In this tutorial we’ll use Python, without any loss of generalization the same concepts apply to any other language.

1. map

The signature of map function in Python is map(function, iterable,)

map operates on iterables (lists, tuples, strings or any sequence that can be iterated on with a for loop)

map is used to apply a function to each item in the iterable, or in other words to map each item to something else.

Example: Let’s say we have this list of numbers, and we would to return zero for each negative number, and return the square of each positive number.

import math

list_of_nums = [1, 5, 10, -5, 8, -4, 3]

def my_function(x):
    if x < 0:
        return 0
    else:
        return math.pow(x, 2)

new_list_of_nums = list(map(my_function, list_of_nums))

print(new_list_of_nums)

#Output: [1, 25, 100, 0, 64, 0, 9]

2. reduce

reduce is the Pythonic version of foldl in Functional Programming, it can be used to apply operations that can accumulate the iterable, such as summing numbers, or concatenation of strings.

The reduce in Python is available in the functools module

The signature of reduce function in Python is reduce(function, iterable)

In reduce, the function we apply must have two arguments. This time we’ll use a lambda,

A lambda is an anonymous function that has the following syntax: lambda arguments : expression

In the following example, we are finding the sum of the first 100 natural numbers, the way this works is as follows it starts from the first item of the list from left to right: ((((1+2)+3)+4)+5)...+100) = 5050

import functools

list_of_nums = list(range(1, 101))

sum_of_nums = functools.reduce(lambda x, y: x + y, list_of_nums)

print(sum_of_nums)

#Output: 5050

3. filter

The filter function as the name suggests, filters an iterable based on the boolean value returned by the function we pass.

The signature of filter function in Python is filter(function, iterable)

In the following example, we are filtering the list such that lowercase characters are removed.

list_of_chars= ['A','B','c','D','e','F',]

filtered_list = list(filter(lambda x: x.isupper(), list_of_chars))

print(filtered_list)

#Output: ['A', 'B', 'D', 'F']

More Examples :


"""
Playing with Lambda Functions
From section 1.6.7 of composing programs
In general, python prefers def statements, but lambdas are useful where there is a simple nested function as an argument or return value.
"""
def compose(f, g):
return lambda x: f(g(x))
f = compose(
lambda x : x*x,
lambda y : y+1 )
result = f(11)
print(result)
#I'd like to come back and write an acutal test for this instead of a print
"""
Example of simple recursive function
"""
def sum_digits(n):
if n < 10:
return n
else:
all_but_last, last = n // 10, n % 10
return sum_digits(all_but_last) + last
summed_result = sum_digits(552)
print(summed_result)
"""
Example of mutual recursion
"""
def is_even(n):
if n == 0:
return True
else:
return is_odd(n-1)
def is_odd(n):
if n == 0:
return False
else:
return is_even(n-1)
result3 = is_even(4)
print(result3)
"""
Example of tree recursion
"""
def fib(n):
"""
Returns Nth number in fibonacci series
"""
if n == 1:
return 0
if n == 2:
return 1
else:
return fib(n-2) + fib(n-1)
result_fib = fib(6)
print(result_fib)
"""
List Comprehension
"""
def divisors(n):
return [1] + [x for x in range(2,n) if n % x == 0]
result_divisors = divisors(8)
print(result_divisors)
#keep if function
#takes an list and a filter
#So if want all posts from a user
#have a general keep if
#pass that by_user
#by user definted elsewhere -------------
'''
Lets Understaned Higher Order Function (HOF)
HOF if Function which will accept argument as function
and return function as well.
NOTE : Use Debugger to Understand the flow of code.
'''
''' Example '''
# 1
print('\nHigher Order Function\n')
def Login(func,username,password):
isValid = func(username, password)
if isValid:
return f'Welcome {username}'
else:
return 'Invalid username or password... ?'
def validate_user_data(temp_uname, temp_pass):
if temp_uname.strip()=='Mike' and temp_pass.strip()=='mikeee128':
return True
def get_user_details():
uname = str(input('Enter your username:'))
passwrd = str(input('Enter your password:'))
return uname,passwrd
if __name__=='__main__':
usrname, paswd = get_user_details()
print(f'1.{Login(validate_user_data, usrname, paswd)}')
'''
In above example i have created 3 function but if you Notice
In "Login" Function i have passed the "validate_user_data" as
an arrgument and used that function future in my code.
'''
print('#'*58)
####################################################################################################################################################################################################################################################################################################################
''' Currying '''
'''
We can use higher-order functions to convert a function that takes
multiple arguments into a chain of functions that each take a single
argument. More specifically, given a function f(x, y), we can define
a function g such that g(x)(y) is equivalent to f(x, y). Here, g is a
higher-order function that takes in a single argument x and returns
another function that takes in a single argument y. This transformation
is called currying.
'''
''' Example '''
print('\n\nCurrying\n')
# 1.
def get_1st_number(num_1):
def get_2nd_number(num_2):
return num_1+num_2
return get_2nd_number
if __name__=='__main__':
print(f'1. Addition of two Number: {get_1st_number(10)(20)}')
''' The above function is pure function because is completely depend on input.'''
# 2.
def pas_function(user_func):
def get_x(x):
def get_y(y):
return user_func(x,y)
return get_y
return get_x
def mul_function(a,b):
return a+b
pas_func = pas_function(mul_function)
print(f'2. Currying With user define or pre define function:{pas_func(2)(4)}\n')
'''
In above example you can apply any function which applicable for two arrguments such as "max","min", "pow". etc.
You can also passs user define function ..in our case i have passed "mul_function", you guys can yours
Function but make sure that function will work on 2 prameter.
'''
print('#'*58)
####################################################################################################################################################################################################################################################################################################################
''' Higher Order Function With Currying '''
print('\nHigher order fucntion with Currying\n')
#1.
def Login(func,welcom_func):
def get_username(uname):
def get_password(pas):
isValid = func(get_user_details,uname,pas)
if isValid:
return welcom_func(uname)
else:
return Invalid_user()
return get_password
return get_username
def check_valid_User(func_user_input,usernm,userpas):
tempUname,tempPass = func_user_input()
return ((tempUname.strip()==usernm) and (tempPass.strip()==userpas.strip()))
def welcome_user(uname):
return f'Welcome {uname}'
def Invalid_user():
return 'invalid username or password'
def get_user_details():
tempName = str(input('Username:'))
tempPass = str(input('Password:'))
return str(tempName).strip(), str(tempPass).strip()
login = Login(check_valid_User,welcome_user)
print(login('Mike')('Mikeee'))
print('#'*58)

Sunday, February 12, 2023

Google Colaboratory – How to Run Python Code in Your Google Drive

 The Google Colaboratory (“Colab”) is a notebook (like a Jupyter Notebook) where you can run Python code in your Google Drive.

You can write text, write code, run that code, and see the output – all in line in the same notebook.

Benefits of Google Colab

Sharing notebooks is as easy as sharing any Google document. You can also get the app and run the code from your phone.

You can use the powerful and popular Python language in your Google Drive, and the set-up will take less than five minutes.

Because Python runs on a server (and not in your local browser or on your local computer) you can easily use it to interact with an online database and analyze data in situations where you need to keep the code private.

How to Add Colab to Your Google Drive

  1. On your computer, in your Google Drive, click the “+ new” button.
us0e4vwVAXFLV1Zv_07RINIVP3uMish_sPWjumo8Y8LYBjbBqa5fq7Ioxw7KUmbIGGN18mbUcu16EhLpmWreOsqpHqnwyVt6bFvTKTg0B-FdclBXIumNAGSHm8MRQmuYKCMz7Q9_


2.  Click “more”, then click “connect more apps” at the bottom of that new menu.

kWfy4KkQcuTYxHO0CGiUVsf2PTFRrKEYQzQM1BffRAaarwnIlg3a_zgtD71_NqSzqGnvqRqfTPUi793vgPr6dzNJ6WmhHn9oPePJSaK9h1RNqR5KvwHg2UVj9sYIMBTvizjWtJ_V


3.  In the Google Workspace Marketplace, type “colab” into the search box.

dLHP0JPcfH2VIFFc_cuJeZZ7Kz5UVKGpVxAY_pQNJoGfuBw8Jg6KoLJ_UCETRmgbxmIE8A31VA3BN9KTwXD4hD6CAfHtTIKgNT-vUSVYLK8J_-I-G0YUgVklUB5zQjBKiozuloih

4.  Click to add Google Colaboratory.

BShlQ_Hnj829h2ZNUxCJTolVFTYb7EeBoV7TJyqH13pwiS6YZDX95bxVI0RC3Dqp5wl2Mo-B4r8ezHkWFOeRJxFGGuoY9eQYb2ANtEx0nPCxU9aaZQrqEJj_hbePrYTGheWjr0tM


5.  Now you have Colab in your list of available apps.

Xxwg1qNNyJQDcxrLlmOILOIGhfVQS1jQpTqjbOym7MqhCoVSdRVu_5EdVfrzgiRp70F01k0AUKZo0AQlmGrO3IgGJD-Dpx77jFhWfLXTW0m3ZfYXz5AMoK9m9BhSG-VhK9D-uX_I


How to Use Google Colab

Now when you click the “+new” button and click “more” (at the bottom of that first list) you will see “Google Colaboratory” on the next list.

Click that to open a new Colab notebook. Give your notebook a name, like you would with any Google document or spreadsheet. This notebook is in dark mode:

ttxQIV3tGb3kaiMlu-ix-J0939nbXX7Xx_Ke5UMcBnT_mRNVcNfJevAbMWm8nhIYbCM6zNjSMY_d3CwqIi-euqnf8HSAZlLG5oZST84kDnw9JpxKnLNkj-SioTtL_xhYHfiSgS1c

Type a simple command as a test. To run the code in any cell, you can click the run button on the left side of the code cell (looks like a “play” button with a triangle in a circle) or you can click [shift] + [enter]. The output will appear right below the code cell.

DCX2F45Bewre17_E27tFm0liy5l155iNB7vt4ohbFhCS7QUpwc47JHJ0ipkgJU6AcfKDcmLY8u2q8N-JHdBl1BwkTeM5-BQ250YbH-UwEKiLC8D6gjuo96vGcwwSFPJi0fxqbSkS

You can import many popular libraries without having to install them first.

Ya_VZMKr0gAEs-UJq7BZa-gnjQ3_AQSAq1YK-eCNQBMgibEeFuAl9BwYZvSrhOyC518v5bjJD9gIGs7WQoI87S3cy_cbJdzIScUvYP8pxpTHEhbbRwSLZwX5qojvn7MQPEfOu0F6

If you import a library or define a function in one cell, that will still be available in other cells for at least a few minutes. The runtime will disconnect if you go 30 minutes without running a cell or if you have the notebook open for 12 hours.  

Explore Colab

If you were following along and you now have Google Colaboratory installed, you are ready to build your own projects.

If you click on the “< >” symbol on the bottom-left side of the notebook, you will find code snippets that you can use. Google also has many resources for you, some of which are at https://colab.research.google.com/ (just close out of the pop-up window that appears).

At freeCodeCamp, we are building a curriculum to show you how to use Python to solve math problems. You now have the tools at your fingertips. Happy coding!

Friday, February 3, 2023

GCP data engineer Exam Topics

 

Section 1: Designing data processing systems

1.1 Selecting the appropriate storage technologies. Considerations include:

    ●  Mapping storage systems to business requirements

    ●  Data modeling

    ●  Trade-offs involving latency, throughput, transactions

    ●  Distributed systems

    ●  Schema design

1.2 Designing data pipelines. Considerations include:

    ●  Data publishing and visualization (e.g., BigQuery)

    ●  Batch and streaming data (e.g., Dataflow, Dataproc, Apache Beam, Apache Spark and Hadoop ecosystem, Pub/Sub, Apache Kafka)

    ●  Online (interactive) vs. batch predictions

    ●  Job automation and orchestration (e.g., Cloud Composer)

1.3 Designing a data processing solution. Considerations include:

    ●  Choice of infrastructure

    ●  System availability and fault tolerance

    ●  Use of distributed systems

    ●  Capacity planning

    ●  Hybrid cloud and edge computing

    ●  Architecture options (e.g., message brokers, message queues, middleware, service-oriented architecture, serverless functions)

    ●  At least once, in-order, and exactly once, etc., event processing

1.4 Migrating data warehousing and data processing. Considerations include:

    ●  Awareness of current state and how to migrate a design to a future state

    ●  Migrating from on-premises to cloud (Data Transfer Service, Transfer Appliance, Cloud Networking)

    ●  Validating a migration

Section 2: Building and operationalizing data processing systems

2.1 Building and operationalizing storage systems. Considerations include:

    ●  Effective use of managed services (Cloud Bigtable, Cloud Spanner, Cloud SQL, BigQuery, Cloud Storage, Datastore, Memorystore)

    ●  Storage costs and performance

    ●  Life cycle management of data

2.2 Building and operationalizing pipelines. Considerations include:

    ●  Data cleansing

    ●  Batch and streaming

    ●  Transformation

    ●  Data acquisition and import

    ●  Integrating with new data sources

2.3 Building and operationalizing processing infrastructure. Considerations include:

    ●  Provisioning resources

    ●  Monitoring pipelines

    ●  Adjusting pipelines

    ●  Testing and quality control

Section 3: Operationalizing machine learning models

3.1 Leveraging pre-built ML models as a service. Considerations include:

    ●  ML APIs (e.g., Vision API, Speech API)

    ●  Customizing ML APIs (e.g., AutoML Vision, Auto ML text)

    ●  Conversational experiences (e.g., Dialogflow)

3.2 Deploying an ML pipeline. Considerations include:

    ●  Ingesting appropriate data

    ●  Retraining of machine learning models (AI Platform Prediction and Training, BigQuery ML, Kubeflow, Spark ML)

    ●  Continuous evaluation

3.3 Choosing the appropriate training and serving infrastructure. Considerations include:

    ●  Distributed vs. single machine

    ●  Use of edge compute

    ●  Hardware accelerators (e.g., GPU, TPU)

3.4 Measuring, monitoring, and troubleshooting machine learning models. Considerations include:

    ●  Machine learning terminology (e.g., features, labels, models, regression, classification, recommendation, supervised and unsupervised learning, evaluation metrics)

    ●  Impact of dependencies of machine learning models

    ●  Common sources of error (e.g., assumptions about data)

Section 4: Ensuring solution quality

4.1 Designing for security and compliance. Considerations include:

    ●  Identity and access management (e.g., Cloud IAM)

    ●  Data security (encryption, key management)

    ●  Ensuring privacy (e.g., Data Loss Prevention API)

    ●  Legal compliance (e.g., Health Insurance Portability and Accountability Act (HIPAA), Children's Online Privacy Protection Act (COPPA), FedRAMP, General Data Protection Regulation (GDPR))

4.2 Ensuring scalability and efficiency. Considerations include:

    ●  Building and running test suites

    ●  Pipeline monitoring (e.g., Cloud Monitoring)

    ●  Assessing, troubleshooting, and improving data representations and data processing infrastructure

    ●  Resizing and autoscaling resources

4.3 Ensuring reliability and fidelity. Considerations include:

    ●  Performing data preparation and quality control (e.g., Dataprep)

    ●  Verification and monitoring

    ●  Planning, executing, and stress testing data recovery (fault tolerance, rerunning failed jobs, performing retrospective re-analysis)

    ●  Choosing between ACID, idempotent, eventually consistent requirements

4.4 Ensuring flexibility and portability. Considerations include:

    ●  Mapping to current and future business requirements

    ●  Designing for data and application portability (e.g., multicloud, data residency requirements)

    ●  Data staging, cataloging, and discovery

Deduplicating Data on the Databricks Lakehouse: Making joins, BI, and AI queries “safe by default.”

  Imagine this: your manager asks the AI analytics tool: "What were our top-selling products last quarter?" The AI generates perfe...