Introduction to Git Data Extraction and Analysis in Python

Is there anyone in the software industry who has never used or at least heard of Git?

Git is a revolutionary tool that is quite ubiquitous in software teams nowadays. This article’s purpose is not to provide an introduction to git, there are a ton of resources that can guide you through that. Its purpose is rather to analyze git relevant data in order to get important insights from those data.

Throughout this article, we are going to extract Git related data by using the Github REST API and then analyze those data by leveraging Python’s top data analysis library, Pandas as well as an interactive data visualization library that is gaining massive popularity, Plotly. We are going to take as example data the repository of Apache Spark.

Git Data Extraction

Git repositories are generally stored in source code hosting facilities. The most popular of these are Github and Bitbucket, but many others are available, like Gitea, GitLab, etc. In this article, we shall focus on Github, but the data extraction process should be similar for the other hosting facilities too.

Github provides a REST API that contains endpoints for all git related resources. In order to be able to consume Github APIs, we need to generate an access token at the Developer Settings on the Github Profile page. After having done that, we should be all set. We start our Jupyter Notebook and we begin by importing the necessary libraries:

We store in a separate file, config.pythe configuration parameters, namely the Github username and the access token that we generated earlier. The recommended way of interacting with Github is by creating a session with the API as follows:

Which are the entities related to git that may provide valuable information on how a git repository is going?

Commits are the ones that come first into mind, but there are also others like branches, pull requests, issues, contributors list, etc. Let’s say that we need to retrieve the list of commits for a given git repository. We search the Github API documentation and find the corresponding API endpoint:

GET /repos/:owner/:repo/commits

Here we need to provide as input parameters the owner and the name of the repository. We can call the above API endpoint in Python like this:

The commits variable contains the response returned from the Github API. Then we use the json() method of the json package for deserializing the above response object. However, this API call returns only 30 results, which corresponds to the number of results that are returned through a single Github API response by default. We can supply an extra parameter to our API request, per_page that allows us to increase the number of returned results up to 100, but the Apache Spark repository that we are trying to extract data from has roughly 26K commits!

No worries. The guys of Github have provided a pagination parameter, called page which combined with the per_page parameter, enable us to extract all the commits of any git repository. So now, our API request should look as follows:

We can wrap up our commits extraction process in a function, that takes as parameters the owner and the repository name and returns a list of commits:

In order to control the traversing process, within each iteration, we check the Link parameter in the headers of the response if it contains the rel="Next" attribute value, which tells us that there exists a successive page and we can continue our iteration; otherwise we stop there. In order to learn more about this approach, you can read the Traversing with Pagination guide in the Github docs.

Having extracted our commits in a list, we now can generate a Pandas Dataframe from that list of dictionaries, so we define the following function to handle that task:

The json_normalize function does properly that, it normalizes a semi-structured JSON (a list of dictionaries) into a flat table. We now invoke the above-created function by passing the necessary parameters for the Apache Spark git repository:

The same process is performed for the extraction of other resources, so I am skipping that part and you can go through it in the Github repository of this article. I have also added the possibility of storing the results in a CSV file or an SQLAlchemy supported database so that we can access those data for later analysis.

---

Data Preprocessing

So we have our commits and branches and the other resources stored in memory (or in CSV files if you like). The next step is preprocessing those data.

Inspecting the structure of the commits dataframe, we get this result:

We are going to drop the majority of these columns and more importantly, we shall generate some time-related columns that are needed for our data analysis process. The following code transforms the commit date field into a datetime field and leverages the dt accessor object for datetimelike properties of a Pandas Series:

After dropping the unnecessary columns, our commits.head method returns:

---

Git Data Analysis

Now we turn to the beautiful part of the data science field — the data analysis, and data visualization. What is the total number of contributors to the Spark repository?

A nice insight would be the distribution of commits by the hour of the day. We can calculate that metric because we already have generated the hour of the day for each commit of the repository. The Pandas code would be:

Now it is time to visualize some data. The Plotly Python library (plotly.py) is an interactive, open-source plotting library that supports over 40 unique chart types covering a wide range of statistical, financial, geographic, scientific, and 3-dimensional use-cases. Being a declarative programming library, Plotly allows us to write code describing what we want to make rather than how to make it. This reduces dramatically the time spent to build a figure and makes us focus more on presenting and interpreting the results.

The standard plotly imports along with the settings to run offline are:

Returning to our interesting metric of commits by the hour of the day, the plotly code needed for generating a bar chart would be nothing more than the following:

From the chart, we notice that the majority of contributions have been committed during the night :). Less activity is seen during the working hours of the day.

How is the Spark repository going over time? What has been its activity over the years? Let us create a time series chart and check it out.

It turns out that the Spark repository has seen its activity peek during 2015–2016. But can we prove that assumption? Of course, we can! We are going to calculate the daily average number of commits for each year in order to verify if 2015 and 2016 have been the most active years of the Spark repository.

The above chart shows it clearly that the repository has reached its apex in 2015 and the activity has fallen thereafter until 2017. Since that time we see a steady daily average number of commits per year and that constant trend is seen until the moment of this article’s writing.

Who are the top contributors to the Spark repository? Let’s find it out.

Being a very active repository, Spark has a lot of open pull requests too. As usual, pull requests are labeled by some predefined keywords. The following bar chart displays the number of pull requests for each label:

It is obvious that the most active contributors are working on SQL related functionalities. In the end, Spark is the data framework that allows SQL-like operations on multiple petabytes of data.

---

Congratulations

Photo by Ian Stauffer on Unsplash

You have reached the end of this article. In this guide we went through the following important concepts:

• Extracting data in Python via Github API
• Preprocessing git data
• Performing interactive analysis and data visualization with Pandas and Plotly

Here’s the full code for everything we ran through this article:

xhentilokaraj/git-statistics

This is a mini-project for generating statistics based on git repositories. There has been taken Github as a version…

github.com

Thanks for reading! If you want to get in touch with me, feel free to reach me on xhentilokaraj@gmail.com or my LinkedIn Profile.

Extracting Feature Importances from Scikit-Learn Pipelines

Scikit-learn pipelines provide a really simple way to chain together the preprocessing steps with the model fitting stages in machine learning development. With pipelines, you can embed these steps so that in one line of code the model will perform all necessary preprocessing steps at the same time as either fitting the model or calling predict.

There are many benefits to this besides reducing the lines of code in your project. Using the standard pipeline layouts means that it is very easy for a colleague, or your future self, to quickly understand your workflow. This in turns means that your work is more reproducible. Additionally, with pipelines, you can enforce the order in which transformations happen.

There is however one drawback in that, although scikit-learn models have the benefit of being highly explainable. Once you embed the model into a pipeline it becomes difficult to extract elements such as feature importances that make these models so interpretable.

I have been spending some time recently looking at this problem. In the following article, I am going to present a simple method I have found to extract feature importances from a pipeline using the python library ELI5.

---

Simple pipeline

In this article, I am going to be using a dataset from drivendata.org, a machine learning competition website. The dataset can be downloaded here.

First I’ll import all the libraries I am using.

import pandas as pd
import numpy as npfrom sklearn import preprocessing
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.compose import ColumnTransformer
from sklearn.metrics import f1_score
from sklearn.preprocessing import OneHotEncoder
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.metrics import classification_report
from sklearn.linear_model import LogisticRegressionimport eli5

I am then using the pandas library to read in the datasets that I have previously downloaded. The features and target labels are in separate CSV files so I am also using the pandas merge function to combine them into one data frame.

train_values = pd.read_csv('train_values.csv')
train_labels = pd.read_csv('train_labels.csv')
train_data = train_values.merge(train_labels, left_on='building_id', right_on='building_id')

If we inspect the data types we can see that there are a mixture of numerical and categorical data. We will, therefore, need to apply some preprocessing before training a model. A pipeline will, therefore, be useful for this dataset.

train_data.dtypes

Before constructing the pipeline I am dropping the ‘building_id’ column as it will not be needed for training, splitting the data into test and train sets, and defining some variables to identify the categorical and numerical columns.

train_data = train_data.drop('building_id', axis=1)numeric_features = train_data.select_dtypes(include=['int64', 'float64']).drop(['damage_grade'], axis=1).columns
categorical_features = train_data.select_dtypes(include=['object']).columns
X = train_data.drop('damage_grade', axis=1)
y = train_data['damage_grade']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

I am going to construct a simple pipeline which will chain together the preprocessing and model fitting steps. Additionally, I am going to add an imputer for any missing values. Although the dataset I am using here does not have any missing data it is sensible to add in this step. This is because in the real world if we were deploying this as a machine learning application there is a chance that new data we are trying to predict on may have missing values. It is, therefore, good practice to add this as a safety net.

The code below constructs a pipeline that imputes any missing values, applies a standard scaler to the numerical features, converts any categorical features into numerical and then fits a classifier.

numeric_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())])
categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
    ('one_hot', OneHotEncoder())])
preprocessor = ColumnTransformer(
    transformers=[
        ('num', numeric_transformer, numeric_features),
        ('cat', categorical_transformer, categorical_features)
    ])pipe = Pipeline(steps=[('preprocessor', preprocessor),
                      ('classifier',  LogisticRegression(class_weight='balanced', random_state=0))])
    
model = pipe.fit(X_train, y_train)

We can inspect the quality of the pipeline by running the below code.

target_names = y_test.unique().astype(str)
y_pred = model.predict(X_test)
print(classification_report(y_test, y_pred, target_names=target_names))

We can see that there is likely to be room for improvement in terms of the performance of the model. One area we would want to explore, besides model selection and hyperparameter optimisation would be feature engineering. However, in order to determine which new features to engineer we first need to have an understanding of which features are most predictive.

---

ELI5

It is not easy to extract feature importances from this pipeline. However, there is a python library that makes this very simple called ELI5. This library, named after the slang term “explain like I’m 5”, is a package that provides a simple way to explain and interpret machine learning models. It is compatible with most popular machine learning frameworks including scikit-learn, xgboost and keras.

The library can be installed via pip or conda.

pip install eli5conda install -c conda-forge eli5

Let's use ELI5 to extract feature importances from the pipeline.

ELI5 needs to know all feature names in order to construct feature importances. By applying one-hot encoding to the categorical variables in the pipeline we are introducing a number of new features. We therefore first need to extract these feature names and append them to the known list of numerical features. The code below uses the ‘named_steps’ function built into scikit-learn pipelines to do this.

onehot_columns = list(pipe.named_steps['preprocessor'].named_transformers_['cat'].named_steps['one_hot'].get_feature_names(input_features=categorical_features))
numeric_features_list = list(numeric_features)
numeric_features_list.extend(onehot_columns)

To extract the feature importances we then simply need to run this line of code.

eli5.explain_weights(pipe.named_steps['classifier'], top=50, feature_names=numeric_features_list)

Which gives a nicely formatted output.

---

The ELI5 library also provides the ability to explain individual predictions but this is not yet supported for pipelines. In this article, I demonstrated a simple method to extract features importances from a scikit-learn pipeline which provides a good starting point to debug and improve a machine learning model.

Thanks for reading!

ORDS Executing PL/SQL PACKAGE.PROCEDURE via REST

I don’t normally blog on a Sunday, but I’m confined to my bed/couch for the next few days, so I might as well answer another question!

This from the YouTube:
“Hi Jeff, Can you please share similar example where REST service is calling a package.procedure with IN and OUT parameters and we can pass IN parameters to get the result?”

So, absolutely you can do this, and of course I’ll show you how. My solutions require Oracle REST Data Services (ORDS).

Before I show you HOW, let me show you the WHAT (our package.)

CREATE OR REPLACE PACKAGE rest_demo_in_out IS
    PROCEDURE demo (
        x     IN    INTEGER,
        y     OUT   VARCHAR2
    );
 
END rest_demo_in_out;
/
 
CREATE OR REPLACE PACKAGE BODY rest_demo_in_out AS
 
    PROCEDURE demo (
        x     IN    INTEGER,
        y     OUT   VARCHAR2
    ) AS
    -- take in a number, and returns it as a string
    -- if we get nothing in, we return a Zero
    BEGIN
        y   := 'X has been converted to a string, :  '
             || TO_CHAR(NVL(
            x,
            0
        ) );
        NULL;
    END demo;
 
END rest_demo_in_out;

As you can see, it’s a VERY simple procedure.

Ok, now how do we make this available via HTTPS?

ORDS: Auto PLSQL

ORDS is installed. HR is REST enabled.

Now, let’s enable our package for ORDS.

But wait, what does THAT mean?

We have an AUTO feature for tables, views, and PL/SQL. For tables and views, you get a full CRUD API to the data objects. For PL/SQL, you get a POST handler to execute the stored procedure, and we automatically grab the output and format it to JSON for the response.

Right-click on the package, and select ‘Enable REST Service.’

You’ll want to alias the package, although I am not. You’ll also want to require authorization before putting this in a real application – assuming you’re not cool with ANYONE being able to execute our stored procedure.

Now we can call it.

Here’s how.

1. It’s a POST
2. We have to send the IN parameter in on the POST body, using ‘Application/JSON’ mime type.
3. The URI will be /ords/hr/rest_demo_in_out/DEMO

And…GO!

Note the RESPONSE is automatically generated based off the OUT parameter.

That solution basically required ZERO code. We told ORDS to handle our package, and it does. We just have to send the POST REQUEST. But, if we don’t like how ORDS handles the scenario, there’s not much we can do about it. Unless…unless you want to roll your OWN RESTful Service.

So let’s go do that now.

ORDS: RESTful Service

We’re going to create a custom RESTful Service, with a POST handler setup to run some PL/SQL…an anonymous block that runs our package for us.

You can find the full RESTful Service module defined below, but it looks like this –

Very basic, but a few important things not to miss!

Important things, not to miss:

1. Mime type on the Handler is set to application/json – that will let us grab the input parameter off of the POST body
2. ‘y’ is declared as a parameter for the handler anon block – that is used to pass the RESPONSE text back. I don’t have to call the variable OR the name ‘Y’, I’m just overloading it, so it’s obvious that it correlates to the ‘Y’ of the OUT parameter of the stored procedure we’re ultimately executing.
3. I don’t have to declare the Y parameter for my POST Handler block, but if I don’t, our POST response will just be a 200, without the output of the procedure attached.
Let’s run it:



Here’s also how to pass a ‘NULL’ to our Stored Procedure.

In our informal testing, it appears that the RESTful Service is executing a good bit faster than the Auto method. ORDS has more work to do on a AUTO call, whereas the RESTful SErvices are more static in nature, so this kind of makes sense to me. But you should take care to test your scenarios under load to make sure they’re adequately performant.
Also, you’ll note I’ve done all of this REST stuff with SQL Developer. You don’t HAVE to use SQL Developer. ORDS has a PL/SQL API, you can just use via and SQL*Plus if you want. I just can’t see how you’d WANT to do that.