Parameters and hyperparameters

• Parameters: Derived during training
• Hyperparameters: Adjustable parameters that can be set before training.

When you call fit, a bunch of values get set, like the split variables and split thresholds.

• These are called parameters.

But even before calling fit on a specific data set, we can set some “knobs” that control the learning.

• These are called hyperparameters.

classification_df = pd.read_csv("data/quiz2-grade-toy-classification.csv")
classification_df.head()

	ml_experience	class_attendance	lab1	lab2	lab3	lab4	quiz1	quiz2
0	1	1	92	93	84	91	92	A+
1	1	0	94	90	80	83	91	not A+
2	0	0	78	85	83	80	80	not A+
3	0	1	91	94	92	91	89	A+
4	0	1	77	83	90	92	85	A+

X = classification_df.drop(columns=["quiz2"])
y = classification_df["quiz2"]

Let’s first bring in the classification quiz 2 grades dataset that we’ve seen before.

We are going to set our X object and our y object.

from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier(max_depth=1)  
model.fit(X, y)

In scikit-learn, hyperparameters are set in the constructor.

max_depthis a hyperparameter that lets us decide and set the maximum depth of the decision tree.

We can set the argument max_depth=1 in our code so that it builds a decision stump.

We have control over how deep the decision tree will be.

Our decision tree here has 1 decision boundary.

For any values of lab_3 less than or equal to 83.5, our model will predict Not A+ and values greater than this will be predicted as A+.

model2 = DecisionTreeClassifier(max_depth=2)  
model2.fit(X, y)

We can do the same thing, this time using a max _depth equal to 2 and now we can see our decision tree has a depth of 2 with 2 decision boundaries; On lab3 at 83.5 and another on quiz1 at 83.5.

model3 = DecisionTreeClassifier(max_depth=3)  
model3.fit(X, y)

We can do this again for max_depth=3 with a total of 4 different splits.

model.score(X, y)

0.7619047619047619

model2.score(X, y)

0.8571428571428571

model3.score(X, y)

0.9523809523809523

model4 = DecisionTreeClassifier(max_depth=5)  
model4.fit(X, y)
model4.score(X, y)

1.0

Now we see what the trees look, let’s see how they score.

How well does our model1 score which uses hyperparameter max_depth=1? Ok, 76% that’s not too bad but what’s the score of model2 which had a max_depth to 2?

It looks like it’s increasing!

Increasing max_depth to 3, increases the model’s accuracy to 95%.

Finally, if we make a new model with max_depth=5 our model gets a score of 100%.

We see here that as max_depth increases, the accuracy of the training data does as well.

Doing this isn’t always the best idea and we’ll explain this a little bit later on.

model5 = DecisionTreeClassifier(min_samples_split=2)  
model5.fit(X, y)

Let’s explore another different hyperparameter min_samples_split.

min_samples_split sets the minimum number of samples required to split an internal node. Remember our decision boundaries?

This hyperparameter will set a minimum number of observations that need to be on either side of the boundary.

Let’s test it out first by setting min_samples_split =2.

This time we are going to look at the number of samples on each side of the boundary.

Since our dataset starts with 21 samples, we can see we have 21 samples at the root node.

This gets split up to 6 on the left branch and 15 on the right.

This means that the tree will continue to split the examples so long as there are at least 2 samples to split up.

Some of these nodes have more than the minimum split value and do not split, but that’s likely because splitting is not needed since all the values are classified to the same class.

model5.score(X, y)

1.0

What kind of score is obtained on the data the model was trained on?

It looks like the model is 100 % accurate!

model6 = DecisionTreeClassifier(min_samples_split=4) 
model6.fit(X, y)
model6.score(X,y)

0.9523809523809523

What happens when we increase this hyperparameter to 4 now?

We see that the score went from 100 percent accuracy to 95 percent.

If we look at our decision tree, we can see it’s a little bit smaller.

It now has a depth of 4 instead of 5. The bottom node on the far right originally split into 2 when we had min_samples_split of 2 but now it stops and classes both samples as A+ now.

model7 = DecisionTreeClassifier(min_samples_split=10) 
model7.fit(X, y)
model7.score(X,y)

0.9047619047619048

Let’s set min_samples_split now to 10 and see what happens.

The model’s score decreases from 95% to 90 percent and our tree looks like it’s lost a branch.

The node with 4 samples no longer is split.

So unlike max_depth, when we increase the min_samples_split hyperparameter, the score of the data the model has seen, decreases.

See this link here .

There are many other hyperparameters for decision trees that you can explore at the link here .

To summarize

• parameters are automatically learned by the algorithm during training
• hyperparameters are specified based on:
  • expert knowledge
  • heuristics, or
  • systematic/automated optimization (more on that in the upcoming modules)

Let’s summarize what we know so far:

Parameters are chosen during training and hyperparameters are specified by us before training.

They can also be chosen by automated optimization which we will cover in module 5.

Let’s apply what we learned!