Baselines: Training a Model using Scikit-learn

Supervised Learning (Reminder)

• Tabular data → Machine learning algorithm → ML model → new examples → predictions

To recap what we have seen so far in supervised machine learning we are given some training data.

We separate our data into features (X) and target (y), we feed it to our learning algorithm, then the learning algorithm learns some function which we call our machine learning model and we use this to predict the target on unseen test examples.

Building a simplest machine learning model using sklearn

Baseline model:

most frequent baseline: always predicts the most frequent label in the training set.

Let’s build a baseline, a simple machine learning algorithm based on simple rules.

We are going to build a most frequent baseline model which always predicts the most frequently labeled in the training set.

We make baseline models not to use for prediction purposes, but as a reference point when we are building other more sophisticated models.

Data

classification_df = pd.read_csv("data/quiz2-grade-toy-classification.csv")
classification_df.head()

	ml_experience	class_attendance	lab1	lab2	lab3	lab4	quiz1	quiz2
0	1	1	92	93	84	91	92	A+
1	1	0	94	90	80	83	91	not A+
2	0	0	78	85	83	80	80	not A+
3	0	1	91	94	92	91	89	A+
4	0	1	77	83	90	92	85	A+

Let’s take our data.

For this example, we are going to be working with the quiz2 classification data that we have seen previously.

In this dataset, we have 7 features with our target column being quiz2 which has 2 possible values; A+ or Not A+.

1. Create 𝑋 and 𝑦

𝑋 → Feature vectors
𝑦 → Target

X = classification_df.drop(columns=["quiz2"])
y = classification_df["quiz2"]

Whenever we build models, there are several important steps involved.

Our first step in building our model is splitting up our tabular data into the features and the target, also known as 𝑋 and 𝑦.

𝑋 is all of our features in our data, which we also call our feature table.
𝑦 is our target, which is what we are predicting.

For this problem, all the columns in our dataframe except quiz2 make up our 𝑋 and the quiz2 column, which is our target make up our 𝑦.

2. Create a classifier object

• import the appropriate classifier.
• Create an object of the classifier.

from sklearn.dummy import DummyClassifier

dummy_clf = DummyClassifier(strategy="most_frequent")

Our next step is creating a model object.

To make our baseline model, we need to import the necessary library.

We spoke about the Scikit Learn package in the last slide deck and we are using the same package to build our baseline model.

Here we are importing DummyClassifier() which will be used to create our baseline model.

We specify in the strategy argument most_frequent which means our model will always predict the most frequent label in the training set.

Here we are naming our model dummy_clf.

3. Fit the classifier

dummy_clf.fit(X, y)

Next, we fit the classifier.

When we call fit on our model object, the actual learning happens. In our simple model there is not much to learn and in this case, will only learn what the most frequent label is in our training data.

4. Predict the target of given examples

We can predict the target of examples by calling predict on the classifier object.

Let’s see what it predicts for a single observation first:

single_obs = X.loc[[0]]
single_obs

	ml_experience	class_attendance	lab1	lab2	lab3	lab4	quiz1
0	1	1	92	93	84	91	92

dummy_clf.predict(single_obs)

array(['not A+'], dtype='<U6')

Once we have our learned model, the next thing we can do is predict using it!

First, we will predict a single observation. We call our model dummy_clf on the object and we get a prediction of Not A+ for it.

X

	ml_experience	class_attendance	lab1	lab2	lab3	lab4	quiz1
0	1	1	92	93	84	91	92
1	1	0	94	90	80	83	91
2	0	0	78	85	83	80	80
...	...	...	...	...	...	...	...
18	1	1	91	93	90	88	82
19	0	1	77	94	87	81	89
20	1	1	96	92	92	96	87

21 rows × 7 columns

dummy_clf.predict(X)

array(['not A+', 'not A+', 'not A+', 'not A+', 'not A+', 'not A+', 'not A+', 'not A+', 'not A+', 'not A+', 'not A+', 'not A+', 'not A+', 'not A+', 'not A+', 'not A+', 'not A+', 'not A+', 'not A+', 'not A+', 'not A+'], dtype='<U6')

We can also predict on many examples.

Here, we are predicting on all of X.

And if we see the predictions for all the observations in X, the model predicts a value of not A+ for each one.

That’s because Not A+ is the most frequent label in our training set.

We will talk more about .fit() and .predict() in the next module.

5. Scoring your model

In the classification setting, .score() gives the accuracy of the model, i.e., proportion of correctly predicted observations.

Sometimes you will also see people reporting error, which is usually 1−𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦

print("The accuracy of the model on the training data:", (round(dummy_clf.score(X, y), 3)))

The accuracy of the model on the training data: 0.524

print("The error of the model on the training data:", (round(1 - dummy_clf.score(X, y), 3)))

The error of the model on the training data: 0.476

It’s at this point where we can see how well our baseline model predicts the quiz2 value.

In ML models, very often it is not possible to get 100% accuracy. How do we check how well our model is doing?

In the classification setting, score() gives the accuracy of the model, i.e., the proportion of correctly predicted examples.

Sometimes we will also see people reporting error, which is usually 1 - accuracy.

We can see that our model’s accuracy on our quiz2 problem is 0.524.

We could also say the error is 0.476.

fit and predict paradigms

The general pattern when we build ML models using sklearn:

1. Creating your 𝑋 and 𝑦 objects
2. clf = DummyClassifier() → create a model (here we are naming it clf)
   
3. clf.fit(X, y) → train the model
4. clf.score(X, y) → assess the model
5. clf.predict(Xnew) → predict on some new data using the trained model

To summarize, here are the steps we follow when building machine learning models using sklearn.

1. Create our 𝑋 and 𝑦 objects
2. Create our model object ( in this case, we created a dummy classifier)
3. Fit our model
4. Assess our model
5. Predict on new examples using this model.

Let’s apply what we learned!