top of page

XGBoost - Extreme Gradient Boosting



18 min


Model Type:


in this free machine learning lesson we discuss the a famous model XGBoost or extreme gradient boosting.  this model from a library specifically for the one model and gives us a ton of flexibility  and a large amount of additional hyperparameters when compared to sklearn's Gradient Boosting.  They use the same concept in their mathematics and in this free ML lesson in Python we discuss compare XGBoost and GradientBoosting side by side to find out which is the best ML model.

About the Model

In this lesson, our focus is on the comparison of hyperparameters between two influential algorithms: XGBoost and GradientBoosting. What makes this exploration particularly intriguing is the distinct set of hyperparameters that XGBoost brings to the table, which are not only extensive but also uniquely tailored to its architecture. On the other hand, there are common hyperparameters shared by both models, forming a bridge between their mathematical underpinnings. As we dissect these hyperparameters, we will uncover their individual significance and influence. By the end of this lesson, you will not only understand the unique features of XGBoost's hyperparameters but also appreciate the common ground they share with GradientBoosting.

Hyperparameters common to both XGBoost and Gradient Boosting (sklearn):

  1. n_estimators: This hyperparameter controls the number of boosting rounds or trees in the ensemble.

  2. learning_rate: It determines the step size at each iteration while moving toward a minimum of a loss function.

  3. max_depth: This hyperparameter sets the maximum depth of individual trees.

  4. min_samples_split: It defines the minimum number of samples required to split an internal node.

  5. min_samples_leaf: This hyperparameter specifies the minimum number of samples required to be at a leaf node.

  6. subsample: It controls the fraction of samples used for fitting the trees.

  7. loss: This determines the loss function to be optimized in the learning process (e.g., 'deviance' for Gradient Boosting).

  8. random_state: Ensures reproducibility by seeding the random number generator.

XGBoost-specific hyperparameters:

  1. booster: Specifies the type of boosting model to use, with options like 'gbtree' (tree-based models), 'gblinear' (linear models), and 'dart' (Dropouts meet Multiple Additive Regression Trees).

  2. gamma (min_split_loss): A regularization term that controls the minimum loss reduction required to make a further partition on a leaf node of the tree.

  3. lambda (reg_lambda): L2 regularization term on weights to prevent overfitting.

  4. alpha (reg_alpha): L1 regularization term on weights.

  5. tree_method: Specifies the method to use for constructing trees, including options like 'exact,' 'approx,' and 'hist.'

  6. grow_policy: It defines the method used to grow the trees, allowing options like 'depthwise' and 'lossguide.'

  7. max_leaves: Sets the maximum number of nodes to be added in the trees.

  8. min_child_weight: It's used to control the minimum sum of instance weight (hessian) needed in a child.

Free Python Code Example of XGBoost and GradientBoosting

A Little Bit more about XGBoost

XGBoost, or Extreme Gradient Boosting, is a machine learning algorithm developed by Tianqi Chen. Its history dates back to 2014 when Chen released the first version as an open-source software project. This algorithm is based on gradient boosting, which is a powerful technique for building ensemble models, where multiple weak learners (usually decision trees) are combined to create a stronger predictive model.

The primary goal of XGBoost was to address some of the limitations of traditional gradient boosting methods. It achieved this by introducing several key innovations:

  1. Regularization: XGBoost incorporates L1 and L2 regularization terms into the objective function. This helps prevent overfitting and makes the model more robust.

  2. Sparsity-Aware Split Finding: It uses an efficient algorithm to handle missing data and works well with sparse datasets.

  3. Parallel Processing: XGBoost is designed for efficiency and speed. It can take advantage of multi-core processors to train models much faster than other gradient boosting implementations.

  4. Built-in Cross-Validation: It has built-in capabilities for cross-validation, making it easier to tune hyperparameters and assess model performance.

  5. Tree Pruning: XGBoost uses a depth-first approach for tree growth and prunes branches that make no positive contribution to reducing the loss function.

  6. Gradient Boosting with Second-Order Derivatives: XGBoost is unique in that it can also compute second-order gradients, which can provide more accurate information for optimization.

XGBoost quickly gained popularity in machine learning competitions, such as those on Kaggle, due to its exceptional predictive performance and efficiency. It became a go-to algorithm for structured/tabular data, and its versatility also led to applications in natural language processing, recommendation systems, and other areas.

In 2016, XGBoost was awarded the prestigious "Test of Time" award at the ACM SIGKDD conference, recognizing its long-lasting impact and significance in the field of data mining and knowledge discovery. It has continued to evolve since its inception, with the development of distributed versions like Dask-XGBoost and support for GPUs to further enhance its capabilities.

Today, XGBoost remains a fundamental tool in the toolkit of data scientists and machine learning practitioners, showcasing how a well-designed algorithm, combined with open-source contributions and a strong community, can make a lasting impact in the field of data science.

Data Science Learning Communities

Real World Applications of XGBoost - Extreme Gradient Boosting