The encodeR package contains functions that allow for more elegant and informative encodings of categorical features. This package uses a large variety of R packages that need to be installed. These packages are:
Currently, this package is not available on CRAN. However, one can install the development version of this package by running:
The website for this package can be found here.
encodeR has four functions that implement different kinds of encodings for categorical features. Many of these functions take advantage of the response/target variable, y, to create more informative representations. Currently, encodeR only supports the tasks of binary classification and regression.
In most cases, preprocessing of these features is often done using a relatively simple method such as one hot encoding. While this can achieve satisfactory results, often there are better, more sparse representations of these features. These more accurate and sparse representations often come with the drawback of being tedious to implement, but encodeR allows one to easily fit such encodings all with a common interface.
One encoding method that has become popular in recent years is that of target encoding. Target encoding essentially uses the average observed response per each category to derive encodings for each category.
First, load the mtcars dataset.
suppressMessages(suppressWarnings(library(encodeR)))
suppressMessages(suppressWarnings(library(dplyr)))
mtcars_data <- mtcars
head(mtcars_data)
#> mpg cyl disp hp drat wt qsec vs am gear carb
#> Mazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4
#> Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4
#> Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1
#> Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1
#> Hornet Sportabout 18.7 8 360 175 3.15 3.440 17.02 0 0 3 2
#> Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1
Suppose the user wishes to encode the cyl
and the vs
columns, which have three (4, 6 and 8 cylinders) and two categories (0 or 1) respectively. Assume further that the user wishes to predict the variable mpg
.
Then, all that is left to do is to call the function target_encoder
:
frame_with_encodings <- target_encoder(
X_train = mtcars_data,
y = mtcars$mpg,
cat_columns = c("cyl", "vs"),
prior = 0,
objective = "regression"
)
head(frame_with_encodings$train)
#> mpg cyl disp hp drat wt qsec vs am gear carb
#> 1 21.0 19.74286 160 110 3.90 2.620 16.46 16.61667 1 4 4
#> 2 21.0 19.74286 160 110 3.90 2.875 17.02 16.61667 1 4 4
#> 3 22.8 26.66364 108 93 3.85 2.320 18.61 24.55714 1 4 1
#> 4 21.4 19.74286 258 110 3.08 3.215 19.44 24.55714 0 3 1
#> 5 18.7 15.10000 360 175 3.15 3.440 17.02 16.61667 0 3 2
#> 6 18.1 19.74286 225 105 2.76 3.460 20.22 24.55714 0 3 1
The user must specify X_train
, which is a tibble, a response variable y
which is a vector, and a character vector containing the names of the categorical columns the user wishes to encode. The user must also set the objective to “regression” here since the response variable is fully continuous.
Observe that both the cyl
and the vs
columns have been replaced with fully numeric columns. An important fact to note is that the resulting tibble is that of the original dimension - in fact, there are no additional columns added after the encoding process. This is one reason why target encoding (as well as others) are effective. They do not increase the dimension of the data, which can often lead to problems.
Either way, these new columns correspond to the learned encodings of the various categories. For example, for cyl
, the encodings are:
frame_with_encodings$train %>%
select(cyl) %>%
bind_cols(cyl_orig = mtcars_data$cyl) %>%
distinct()
#> cyl cyl_orig
#> 1 19.74286 6
#> 2 26.66364 4
#> 3 15.10000 8
The encodings for each category are fairly intuitive here. Recall that the target variable is mpg
. Thus, one can easily see that as the number of cylinders increases, the mpg decreases. This relationship is now captured in the encodings.
Often, the user splits the entire dataset into two before doing any preprocessing of the features, to prevent overfitting and to ensure an honest assessment of model performance. Preprocessing features before splitting may lead to information being leaked from the test set.
Encodings are no exception to this rule. For more common methods such as one hot encoding, users typically do not need to worry about information leakage since each row is still treated independently of all others. However, for target encoding and conjugate encoding it is vital that the dataset is split before since these methods average over multiple rows and also involve the response/target variable.
The encodeR package has been designed so that the user does not need to worry about any potential data leakage. The encodings are learned strictly on X_train
and are then joined to X_test
.
The user can easily do this by supplying an optional argument to X_test
to any of the encoding functions like so:
mtcars_train <- mtcars[1:30, ]
mtcars_test <- mtcars[31:nrow(mtcars), ]
my_encodings <- target_encoder(
X_train = mtcars_train,
X_test = mtcars_test,
y = mtcars_train$mpg,
cat_columns = c("cyl"),
prior = 0.7,
objective = "regression"
)
head(my_encodings$train)
#> mpg cyl disp hp drat wt qsec vs am gear carb
#> 1 21.0 19.78593 160 110 3.90 2.620 16.46 0 1 4 4
#> 2 21.0 19.78593 160 110 3.90 2.875 17.02 0 1 4 4
#> 3 22.8 26.73380 108 93 3.85 2.320 18.61 1 1 4 1
#> 4 21.4 19.78593 258 110 3.08 3.215 19.44 1 0 3 1
#> 5 18.7 15.36873 360 175 3.15 3.440 17.02 0 0 3 2
#> 6 18.1 19.78593 225 105 2.76 3.460 20.22 1 0 3 1
my_encodings$test
#> mpg cyl disp hp drat wt qsec vs am gear carb
#> 1 15.0 15.36873 301 335 3.54 3.57 14.6 0 1 5 8
#> 2 21.4 26.73380 121 109 4.11 2.78 18.6 1 1 4 2
Observe that the encoder function now returns a list with two tibbles labelled train
and test
. The two tibbles returned contain the preprocessed X_train
and X_test
tibbles, respectively.
Note that when using target_encoder
that if there are categories that do not appear in the training set but do appear in the test set, these categories are encoded with the group mean (see below) \(\bar x\) of the target variable.
Target encoding is highly susceptible to overfitting since the estimated relationship between the categories in the training set may not necessarily be true in general.
The prior
paramter in target_encoding
allows us to address this issue through the use of Laplace smoothing. The idea of Laplace smoothing is to use a weighted average of the mean of the response variable and the conditional mean per each category. The result is a smoothed estimate that, with a suitable prior
chosen, is less prone to overfitting. Formally, the learned encoding for the jth category \(u_{j}\) is now:
\[u_{j} = \frac{\sum_{i=1}^{N} y_{ij} + \text{prior} \times \bar x}{N + \text{prior}},\] where \(\bar x\) is the mean of the response, and \(N\) is the total number of observations belogning to category \(j\). Thus, a larger prior
parameter places more weight on the group mean \(\bar x\), whereas a lower prior fully trusts the data.
If one repeats the example above but with a prior of 5:
frame_with_encodings <- target_encoder(
X_train = mtcars_train,
y = mtcars_train$mpg,
cat_columns = c("cyl"),
prior = 5,
objective = "regression"
)
frame_with_encodings$train %>%
select(cyl) %>%
bind_cols(cyl_orig = mtcars_train$cyl) %>%
distinct()
#> cyl cyl_orig
#> 1 19.94028 6
#> 2 24.86556 4
#> 3 16.52685 8
Observe that the encodings for 6 and 8 cylinders have been dragged slightly up, whereas the encodings for 4 cylinders has been dragged down. This is because 6 and 8 cylinder vehicles have lower than average miles per gallon, whereas 4 cylinder vehicles have higher than average miles per gallon.
Another more recently popular method of encoding (especially in Kaggle competitions) is that of frequency encoding. Frequency encoding derives its encodings by using the observed frequency of each class. The intuition behind such an encoding is that those categories that appear the most should be assigned higher weight than those that do not.
The usage of the function frequency_encoder
is straightforward. Note that this function does not require a y
or objective
argument, because it does not make use of the response/target when deriving its encodings.
encodings_freq <- frequency_encoder(
X_train = mtcars_train,
cat_columns = c("cyl")
)
#> Joining, by = "cyl"
head(encodings_freq$train)
#> mpg cyl disp hp drat wt qsec vs am gear carb
#> Mazda RX4 21.0 0.2333333 160 110 3.90 2.620 16.46 0 1 4 4
#> Mazda RX4 Wag 21.0 0.2333333 160 110 3.90 2.875 17.02 0 1 4 4
#> Datsun 710 22.8 0.3333333 108 93 3.85 2.320 18.61 1 1 4 1
#> Hornet 4 Drive 21.4 0.2333333 258 110 3.08 3.215 19.44 1 0 3 1
#> Hornet Sportabout 18.7 0.4333333 360 175 3.15 3.440 17.02 0 0 3 2
#> Valiant 18.1 0.2333333 225 105 2.76 3.460 20.22 1 0 3 1
We see that the cyl
column has been replaced by the learned frequencies of each category, as expected.
Note that currently, this function does not implement Laplace smoothing like in target_encoder
. Thus, observations that do not appear in the train set but do appear in the test are given 0 frequencies.
no_8 <- mtcars_train %>%
filter(cyl != 8)
encodings_freq <- frequency_encoder(
X_train = no_8,
X_test = mtcars_test,
cat_columns = c("cyl")
)
#> Joining, by = "cyl"
#> Joining, by = "cyl"
encodings_freq$test
#> mpg cyl disp hp drat wt qsec vs am gear carb
#> Maserati Bora 15.0 0.0000000 301 335 3.54 3.57 14.6 0 1 5 8
#> Volvo 142E 21.4 0.5882353 121 109 4.11 2.78 18.6 1 1 4 2
Observe now that for the Maserati Bora, the encoding is 0 since no other cars with 8 cylinders existed in the dataset.
This method for encoding categorical features is based on a very recent paper by Slakey et al. (2019). At a very high level, this paper reworks target encoding in a full Bayesian framework so that each category in a particular column has a posterior distribution using the response as the observed data. The first k moments of the posterior distributions are then used as the encodings.
The paper only discusses the use of this method through conjugate priors for computational reasons, but in theory any arbitrary prior and likelihood can be used.
To use this method, one must specify prior parameters depending on the objective to the argument prior_params
. In the objective = "regression"
case, the list must have four named items: mu, vega, alpha, and beta, which correspond to the parameters of a Normal-Inverse-gamma distribution. Vega, alpha, and beta must all be greater than 0. mu can be any real number.
Using the function conjugate_encoder
is relatively straightforward:
prior_values <- list(mu = 0, vega = 3, alpha = 1, beta = 1)
encodings_conjugate <- conjugate_encoder(
X_train = mtcars_train,
X_test = mtcars_test,
y = mtcars_train$mpg,
prior_params = prior_values,
cat_columns = c("cyl"),
objective = "regression"
)
#> Joining, by = "cyl"
#> Joining, by = "cyl"
#> Joining, by = "cyl"
encodings_conjugate$train %>%
select(cyl_encoded_mean, cyl_encoded_var) %>%
bind_cols(cyl = mtcars_train$cyl) %>%
distinct()
#> cyl_encoded_mean cyl_encoded_var cyl
#> 1 17.94805 37.61411 6
#> 2 24.71818 86.48751 4
#> 3 13.73427 27.91508 8
As one can see, the encodings are fairly similar to that of target_encoder
but with more flexibility via. the explicit modelling of the posterior.
Note that the Normal-Inverse-gamma distribution is bivariate. In other words, we assumed that the likelihood has unknown mean and unknown variance.
In general, the Normal-Inverse-gamma distribution is parameterized such that the density is a function of two random variables, \(f(x, \sigma^2)\). Thus, the second column shown above, cyl_encoded_var
is the expected value of \(\sigma^2\).
Changing the objective to “binary” in target_encoder
makes no difference since it just calculates the mean of y, regardless if it is binary or fully continiuous. However, for conjugate_encoder
this is not true since the likelihood used is specifically chosen to handle binary outcomes (a binomial).
For classification, the user must specify a list with two named arguments for prior_params
; alpha and beta. These variables correspond to the parameters of a beta distribution.
Otherwise, the usage is exactly the same as above:
prior_values <- list(alpha = 1, beta = 1)
encodings_conjugate <- conjugate_encoder(
X_train = mtcars_train,
X_test = mtcars_test,
y = mtcars_train$vs,
prior_params = prior_values,
cat_columns = c("cyl"),
objective = "binary"
)
#> Joining, by = "cyl"
#> Joining, by = "cyl"
encodings_conjugate$train %>%
select(cyl_encoded) %>%
bind_cols(cyl = mtcars_train$cyl) %>%
distinct()
#> cyl_encoded cyl
#> 1 0.15625 6
#> 2 0.31250 4
#> 3 0.03125 8
Note that in this case, the target variable has changed to the binary variable vs
. The column cyl_encoded
is the expected value of the posterior distribution (also a beta).
encodeR would not be complete without including the most popular method for encoding categorical features. This method creates k-1 columns of 0/1 indicator variables, where k is the number of categories. This is a completely uninformative method, but the syntax remains the same for consistency.
ohe_encodings <- onehot_encoder(
X_train = mtcars_train,
X_test = mtcars_test,
cat_columns = c("cyl")
)
head(ohe_encodings$train)
#> mpg disp hp drat wt qsec vs am gear carb cyl_6 cyl_8
#> 1 21.0 160 110 3.90 2.620 16.46 0 1 4 4 1 0
#> 2 21.0 160 110 3.90 2.875 17.02 0 1 4 4 1 0
#> 3 22.8 108 93 3.85 2.320 18.61 1 1 4 1 0 0
#> 4 21.4 258 110 3.08 3.215 19.44 1 0 3 1 1 0
#> 5 18.7 360 175 3.15 3.440 17.02 0 0 3 2 0 1
#> 6 18.1 225 105 2.76 3.460 20.22 1 0 3 1 1 0
ohe_encodings$test
#> mpg disp hp drat wt qsec vs am gear carb cyl_6 cyl_8
#> 1 15.0 301 335 3.54 3.57 14.6 0 1 5 8 0 1
#> 2 21.4 121 109 4.11 2.78 18.6 1 1 4 2 0 0