A Beginner’s Guide to Stacking Ensemble Deep Learning Models
Most data science practitioners seek greater accuracy from their models, and neural networks regularly give us peak performance. In this article, we will see a method that allows us to increase the level of precision of our modeling procedure. This method can be viewed as a way to join multiple machine learning models. We call this method a stacking set. The main points to be discussed in the article are listed below.
- What is Stacking
- Implementing Stacking Set Deep Learning
- Data preparation
- Model preparation
- Stackable models
- Evaluation of results
Let’s start by understanding stacking.
What is Stacking?
We can think of stacking as a process of assembling multiple machine learning models. There can be different methods for ensemble models, such as bagging, amplifying and stacking is one of them. When talking about the bagging method, it works by applying multiple models with high variance and these high variances are averaged to decrease the variance. And boost the work by creating multiple incremental models to reduce bias.
Stacking is very different from these ensemble methods because this method focuses on exploring the space of different models applied to the same problem. Basically, the idea behind this method is to handle a machine learning problem using different types of models that can learn to some extent, not the whole problem space. Using these models, we can make intermediate predictions and then add a new model that can learn using the intermediate predictions.
The final model which is learned using the intermediate predictions can be viewed as stacked on top of the intermediate models. By using such methods we can improve our performance and still get a better model than the intermediate model. The picture below can explain the stacking method.
Source of images
The image above shows that a final classifier is stacked above three intermediate classifiers. In this article, we are going to see how we can do stack assembly with deep learning models. Let’s start the implementation.
Are you looking for a comprehensive repository of Python libraries used in data science, check here,
In this section, we will see how we can stack an MLP classifier on top of some neural networks. This entire section includes the use of packages from the TensorFlow, Keras, and sci-kit learning libraries. Let’s start by preparing the data for the process.
In this article, we will try to stack the classification models. For this, we will create toy data using the make_classification function provided by sklearn. Let’s see how we can do this.
from sklearn.datasets import make_classification X, y = make_classification() print(X.shape, y.shape)
Let’s separate the data set into training and test sets.
from sklearn.model_selection import train_test_split X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=0)
Let’s see what our toy data looks like.
Now, in this example stack, we are going to use neural networks as intermediate models, so we need to import some of the packages from Keras.
from tensorflow import keras from keras import layers from keras.constraints import maxnorm from keras.models import Sequential from keras.layers import Dense, Dropout, Activation, Flatten, Input
In this article, we will use the stacking function provided by scikit learn, so we need to import a wrapper provided in Keras for SciKit-Learn so that it can be compatible with scikit learn functions. Let’s import KerasClassifier.
from keras.wrappers.scikit_learn import KerasClassifier
Defining a Model Feature
def create_model (): # create model model = Sequential() model.add(Dense(20, input_dim=20, activation='relu')) model.add(Dropout(0.2)) model.add(Flatten()) optimizer= keras.optimizers.RMSprop(lr=0.001) model.add(Dense(units = 1, activation = 'sigmoid')) # Compile model model.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=[keras.metrics.AUC(), 'accuracy']) return model
In the above we defined a simple function to create models in which we added 2 dense layers and a delete and flatten function. Let’s create some classifiers using the above function.
NN_clf1=KerasClassifier(build_fn=create_model, epochs=5, batch_size= 32) NN_clf1._estimator_type = "classifier" NN_clf2=KerasClassifier(build_fn=create_model, epochs=10, batch_size= 32) NN_clf2._estimator_type = "classifier" NN_clf3=KerasClassifier(build_fn=create_model, epochs=15, batch_size= 32) NN_clf3._estimator_type = "classifier" NN_clf4=KerasClassifier(build_fn=create_model, epochs=20, batch_size= 32) NN_clf4._estimator_type = "classifier" NN_clf5=KerasClassifier(build_fn=create_model, epochs=25, batch_size= 32) NN_clf5._estimator_type = "classifier"
Here we have defined 5 models where we have changed the number of epochs, we can also use other methods to make the change.
Let’s start by combining the models defined above.
intermediate = [('NN1', NN_clf1), ('NN2', NN_clf2), ('NN3', NN_clf3), ('NN4', NN_clf4), ('NN5', NN_clf5)] intermediate
Here we can see that we have combined all the neural networks defined above. We are now ready to stack these models. In this article, we will use MLPClassifier as a final model that will learn from training intermediate neural networks.
from sklearn.ensemble import StackingClassifier from sklearn.neural_network import MLPClassifier clf = StackingClassifier(estimators=intermediate, final_estimator=MLPClassifier()) clf.fit(X_train, y_train)
Here we can see the final details of the model where it represents the estimators or the intermediate models and the final models.
Evaluation of the final model
Let’s check for accuracy.
print("Stacking model score: %.3f" % clf.score(X_test, y_test))
Here is our final clarification. Here we can see an example of deep learning model stacking. Let’s check the probability score.
Here we can see the probability that a sample is in class 0 or 1. This article represents the idea that if we apply neural networks in a stacked way, we can also use basic models like random forest and logistic regression models as intermediate or final. models.
In this article, we have discussed stacking which is a process of assembling multiple machine learning models in a stacked manner and we also see how we can stack sklearn models on top of neural networks.