Machine learning :Model Selection(Grid Search ,K fold cross vaidation)
What is grid search?
Grid search is the process of performing hyper parameter tuning in order to determine the optimal values for a given model. This is significant as the performance of the entire model is based on the hyper parameter values specified.
Why should I use it?
If you work with ML, you know what a nightmare it is to stipulate values for hyper parameters. There are libraries that have been implemented, such as GridSearchCV of the sklearn library, in order to automate this process and make life a little bit easier for ML enthusiasts.
How does it work?
Here’s a python implementation of grid search using GridSearchCV of the sklearn library.
How does it work?
Here’s a python implementation of grid search using GridSearchCV of the sklearn library.
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVRgsc = GridSearchCV(
estimator=SVR(kernel='rbf'),
param_grid={
'C': [0.1, 1, 100, 1000],
'epsilon': [0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5, 10],
'gamma': [0.0001, 0.001, 0.005, 0.1, 1, 3, 5]
},
cv=5, scoring='neg_mean_squared_error', verbose=0, n_jobs=-1)
First, we need to import GridSearchCV from the sklearn library, a machine learning library for python. The estimator parameter of GridSearchCV requires the model we are using for the hyper parameter tuning process. For this example, we are using the rbf kernel of the Support Vector Regression model(SVR). The param_grid parameter requires a list of parameters and the range of values for each parameter of the specified estimator. The most significant parameters required when working with the rbf kernel of the SVR model are c, gamma and epsilon. A list of values to choose from should be given to each hyper parameter of the model. You can change these values and experiment more to see which value ranges give better performance. A cross validation process is performed in order to determine the hyper parameter value set which provides the best accuracy levels.
grid_result = gsc.fit(X, y)
best_params = grid_result.best_params_best_svr = SVR(kernel='rbf', C=best_params["C"], epsilon=best_params["epsilon"], gamma=best_params["gamma"],
coef0=0.1, shrinking=True,
tol=0.001, cache_size=200, verbose=False, max_iter=-1)
We then use the best set of hyper parameter values chosen in the grid search, in the actual model as shown above.
K-Fold Cross Validation
Evaluating a Machine Learning model can be quite tricky. Usually, we split the data set into training and testing sets and use the training set to train the model and testing set to test the model. We then evaluate the model performance based on an error metric to determine the accuracy of the model. This method however, is not very reliable as the accuracy obtained for one test set can be very different to the accuracy obtained for a different test set. K-fold Cross Validation(CV) provides a solution to this problem by dividing the data into folds and ensuring that each fold is used as a testing set at some point. This article will explain in simple terms what K-Fold CV is and how to use the sklearn library to perform K-Fold CV.
What is K-Fold Cross Validation?
K-Fold CV is where a given data set is split into a K number of sections/folds where each fold is used as a testing set at some point. Lets take the scenario of 5-Fold cross validation(K=5). Here, the data set is split into 5 folds. In the first iteration, the first fold is used to test the model and the rest are used to train the model. In the second iteration, 2nd fold is used as the testing set while the rest serve as the training set. This process is repeated until each fold of the 5 folds have been used as the testing set.
Evaluating a ML model using K-Fold CV
Lets evaluate a simple regression model using K-Fold CV. In this example, we will be performing 10-Fold cross validation using the RBF kernel of the SVR model(refer to this article to get started with model development using ML).
- Importing libraries
First, lets import the libraries needed to perform K-Fold CV on a simple ML model.
import pandas
from sklearn.model_selection import KFold
from sklearn.preprocessing import MinMaxScaler
from sklearn.svm import SVR
import numpy as np
dataset = pandas.read_csv('housing.csv')X = dataset.iloc[:, [0, 12]]
y = dataset.iloc[:, 13]scaler = MinMaxScaler(feature_range=(0, 1))
X = scaler.fit_transform(X)This technique re-scales the data between a specified range(in this case, between 0–1), to ensure that certain features do not affect the final prediction more than the other features.
4. K-Fold CV
Now, lets get down to business.
scores = []
best_svr = SVR(kernel='rbf')
cv = KFold(n_splits=10, random_state=42, shuffle=False)
for train_index, test_index in cv.split(X):
print("Train Index: ", train_index, "\n")
print("Test Index: ", test_index)
X_train, X_test, y_train, y_test = X[train_index], X[test_index], y[train_index], y[test_index]
best_svr.fit(X_train, y_train)
scores.append(best_svr.score(X_test, y_test))We are using the RBF kernel of the SVR model, implemented using the
sklearnlibrary (the default parameter values are used as the purpose of this article is to show how K-Fold cross validation works), for the evaluation purpose of this example. First, we indicate the number of folds we want our data set to be split into. Here, we have used 10-Fold CV (n_splits=10), where the data will be split into 10 folds. We are printing out the indexes of the training and the testing sets in each iteration to clearly see the process of K-Fold CV where the training and testing set changes in each iteration.
Next, we specify the training and testing sets to be used in each iteration. For this, we use the indexes
(train_index, test_index)specified in the K-Fold CV process. Then, we train the model in each iteration using thetrain_indexof each iteration of the K-Fold process and append the error metric value to a list(scores).best_svr.fit(X_train, y_train)
scores.append(best_svr.score(X_test, y_test))The error metric computed using the
best_svr.score()function is the r2 score. Each iteration of F-Fold CV provides an r2 score. We append each score to a list and get the mean value in order to determine the overall accuracy of the model.print(np.mean(scores))Instead of this somewhat tedious method, you can use either,
cross_val_score(best_svr, X, y, cv=10)or,
cross_val_predict(best_svr, X, y, cv=10)to do the same task of 10-Fold cross validation. The first method will give you a list of r2 scores and the second will give you a list of predictions.
Comments
Post a Comment