Boosting your Machine Learning Models Using XGBoost

In this tutorial we’ll cover XGBoost, a machine learning algorithm that has dominated the applied machine learning space recently.

Plan of Attack

1. What is XGBoost?
2. Why would you use XGBoost?
3. Boosting Vis-a-vis Bagging
4. Applying XGBoost in Python
5. XGBoost’s Hyperparameters
6. Cross Validation when using XGBoost
7. Visualizing Feature Importance in XGBoost
8. Conclusion

What is XGBoost?

XGBoost is an open source library that provides gradient boosting for Python, Java and C++, R and Julia. In this tutorial, our focus will be on Python. Gradient Boosting is a machine learning technique for classification and regression problems that produces a prediction from an ensemble of weak decision trees.

Why would you use XGBoost?

The primary reasons you’d use this algorithm are its accuracy, efficiency, and feasibility. It’s a linear model and a tree learning algorithm that does parallel computations on a single machine. It also has extra features for doing cross validation and computing feature importance. Below are some of the main features of the model:

• Sparsity: It accepts sparse input for tree booster and linear booster.
• Customization: It supports customized objective and evaluation functions.
• DMatrix: Its optimized data structure that improves its performance and efficiency.

Boosting Vis-a-Vis Bagging

Boosting is a machine learning ensemble algorithm that reduces bias and variance that converts weak learners into strong learners. XGBoost is an example of a boosting algorithm. Bagging, on the other hand, is a technique whereby one takes random samples of data, builds learning algorithms, and takes means to find bagging probabilities.

Applying XGBoost in Python

Next let’s show how one can apply XGBoost to their machine learning models. If you don't have XGBoost installed, follow this link to install it…