Regression Measurements

from sklearn.model_selection import train_test_split

housing_df = pd.read_csv("data/housing.csv")
train_df, test_df = train_test_split(housing_df, test_size=0.1, random_state=123)

X_train = train_df.drop(columns=["median_house_value"])
y_train = train_df["median_house_value"]
X_test = test_df.drop(columns=["median_house_value"])
y_test = test_df["median_house_value"]

numeric_features = [ "longitude", "latitude",
                     "housing_median_age",
                     "households", "median_income",
                     "rooms_per_household",
                     "bedrooms_per_household",
                     "population_per_household"]
                     
categorical_features = ["ocean_proximity"]

X_train.head(3)

	longitude	latitude	housing_median_age	households	...	ocean_proximity	rooms_per_household	bedrooms_per_household	population_per_household
6051	-117.75	34.04	22.0	602.0	...	INLAND	4.897010	1.056478	4.318937
20113	-119.57	37.94	17.0	20.0	...	INLAND	17.300000	6.500000	2.550000
14289	-117.13	32.74	46.0	708.0	...	NEAR OCEAN	4.738701	1.084746	2.057910

3 rows × 9 columns

This next section involves looking at regression problems and so we are going to bring back our California housing dataset where we want to predict the median house value for different locations.

from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.compose import make_column_transformer
from sklearn.neighbors import KNeighborsRegressor

numeric_transformer = Pipeline(
    steps=[("imputer", SimpleImputer(strategy="median")), 
           ("scaler", StandardScaler())]
)

categorical_transformer = Pipeline(
    steps=[("imputer", SimpleImputer(strategy="constant", fill_value="missing")),
           ("onehot", OneHotEncoder(handle_unknown="ignore"))]
)

preprocessor = make_column_transformer(
(numeric_transformer, numeric_features),
        (categorical_transformer, categorical_features), 
    remainder='passthrough')

pipe = make_pipeline(preprocessor, KNeighborsRegressor())
pipe.fit(X_train, y_train);

We are going to bring in our previous pipelines and fit our model.

predicted_y = pipe.predict(X_train) 

predicted_y == y_train

6051     False
20113    False
14289    False
         ...  
17730    False
15725    False
19966    False
Name: median_house_value, Length: 18576, dtype: bool

y_train.values

array([113600., 137500., 170100., ..., 286200., 412500.,  59300.], shape=(18576,))

predicted_y

array([111740., 117380., 187700., ..., 271420., 265180.,  60860.], shape=(18576,))

We aren’t doing classification anymore, so we can’t just check for equality.

We need a score that reflects how right/wrong each prediction is or how close we are to the actual numeric value.

Regression measurements

The scores we are going to discuss are:

• mean squared error (MSE)
• R²
• root mean squared error (RMSE)
• MAPE

If you want to see these in more detail, you can refer to the sklearn documentation.

Mean squared error (MSE)

predicted_y

array([111740., 117380., 187700., ..., 271420., 265180.,  60860.], shape=(18576,))

np.mean((y_train - predicted_y)**2)

np.float64(2570054492.048064)

np.mean((y_train - y_train)**2)

np.float64(0.0)

Mean Squared Error is a common measure.

We calculate this by calculating the difference between the predicted and actual value, square it and sum all these values for every example in the data.

Perfect predictions would have MSE=0.

from sklearn.metrics import mean_squared_error 

mean_squared_error(y_train, predicted_y)

2570054492.048064

We can use mean_squared_error from sklearn again instead of calculating this ourselves.

If we look at MSE here, it’s huge and unreasonable.

Is this score good or bad?

Unlike classification, in regression, our target has units.

In this case, our target column is the median housing value which is in dollars.

That means that the mean squared error is in dollars².

The score also depends on the scale of the targets.

If we were working in cents instead of dollars, our MSE would be 10,000 X (100²) higher!

R² (quick notes)

Key points:

• The maximum value possible is 1 which means the model has perfect predictions.
• Negative values are very bad: “worse than baseline models such asDummyRegressor”.

from sklearn.metrics import r2_score

This is the score that sklearn uses by default when you call .score() so we’ve already seen R² in our regression problems.

You can read about it here but we are going to just give you the quick notes.

Intuition: mean squared error, but flipped where higher values mean a better measurement.

Normalized so the max is 1.

When you call fit it minimizes MSE / maximizes R² (or something like that) by default.

Just like in classification, this isn’t always what you want.

print(mean_squared_error(y_train, predicted_y))
print(mean_squared_error(predicted_y, y_train))

2570054492.048064
2570054492.048064

print(r2_score(y_train, predicted_y))
print(r2_score(predicted_y, y_train))

0.8059396097446094
0.742915970464153

We can reverse MSE but not R² (optional).

Root mean squared error (RMSE)

mean_squared_error(y_train, predicted_y)

2570054492.048064

np.sqrt(mean_squared_error(y_train, predicted_y))

np.float64(50695.704867849156)

The MSE we had before was in dollars².

A more relatable metric would be the root mean squared error, or RMSE.

This now has the units in dollars. Instead of 2 billion dollars squared our error measurement is around $50,000.

When we plot our predictions versus the examples’ actual value, we can see cases where our prediction is way off.

Under the line means we’re under-prediction, over the line means we’re over-predicting.

Question: Is an error of $30,000 acceptable?

• For a house worth $600k, it seems reasonable! That’s a 5% error.
• For a house worth $60k, that is terrible. It’s a 50% error.

MAPE - Mean Absolute Percent Error (MAPE)

percent_errors = (predicted_y - y_train)/y_train * 100.
percent_errors.head()

6051     -1.637324
20113   -14.632727
14289    10.346855
13665     6.713070
14471   -10.965854
Name: median_house_value, dtype: float64

np.abs(percent_errors).head()

6051      1.637324
20113    14.632727
14289    10.346855
13665     6.713070
14471    10.965854
Name: median_house_value, dtype: float64

100.*np.mean(np.abs((predicted_y - y_train)/y_train))

np.float64(18.19299750298522)

So, finding the percentage error may be handy. Can we compute something like that?

We can calculate a percentage error for each example. Now the errors are both positive (predict too high) and negative (predict too low).

We can look at the absolute percent error which now shows us how far off we were independent of direction.

Like MSE, we can take the average over all the examples. This is called Mean Absolute Percent Error (MAPE).

Ok, this is quite interpretable. We can see that on average, we have around 18% error in our predicted median housing valuation.

Let’s apply what we learned!