Book Image

Hands-On Gradient Boosting with XGBoost and scikit-learn

By : Corey Wade
Book Image

Hands-On Gradient Boosting with XGBoost and scikit-learn

By: Corey Wade

Overview of this book

XGBoost is an industry-proven, open-source software library that provides a gradient boosting framework for scaling billions of data points quickly and efficiently. The book introduces machine learning and XGBoost in scikit-learn before building up to the theory behind gradient boosting. You’ll cover decision trees and analyze bagging in the machine learning context, learning hyperparameters that extend to XGBoost along the way. You’ll build gradient boosting models from scratch and extend gradient boosting to big data while recognizing speed limitations using timers. Details in XGBoost are explored with a focus on speed enhancements and deriving parameters mathematically. With the help of detailed case studies, you’ll practice building and fine-tuning XGBoost classifiers and regressors using scikit-learn and the original Python API. You'll leverage XGBoost hyperparameters to improve scores, correct missing values, scale imbalanced datasets, and fine-tune alternative base learners. Finally, you'll apply advanced XGBoost techniques like building non-correlated ensembles, stacking models, and preparing models for industry deployment using sparse matrices, customized transformers, and pipelines. By the end of the book, you’ll be able to build high-performing machine learning models using XGBoost with minimal errors and maximum speed.

Related Content you might be interested in

Current Title:

Small book image
Hands-On Gradient Boosting with XGBoost and scikit-learn
Corey Wade
Oct, 2020 | 310 pages
Small book image
Machine Learning with LightGBM and Python
Andrich van Wyk
Sep, 2023 | 252 pages
Small book image
The Python Workshop
Dr Lau Cher Han | Graham Lee | Mario Corchero | Andrew Bird | Corey Wade
Nov, 2022 | 600 pages
Small book image
scikit-learn Cookbook
Julian Avila | Trent Hauck
Nov, 2017 | 374 pages
Table of Contents (15 chapters)
Preface
Preface
Who this book is for
To get the most out of this book
Setting up your coding environment
Download the color images
Conventions used
Get in touch
Reviews
1
Section 1: Bagging and Boosting
Section 1: Bagging and Boosting
Free Chapter
2
Chapter 1: Machine Learning Landscape
Chapter 1: Machine Learning Landscape
Previewing XGBoost
Data wrangling
Predicting regression
Predicting classification
Summary
3
Chapter 2: Decision Trees in Depth
Chapter 2: Decision Trees in Depth
Introducing decision trees with XGBoost
Exploring decision trees
Contrasting variance and bias
Tuning decision tree hyperparameters
Predicting heart disease – a case study
Summary
4
Chapter 3: Bagging with Random Forests
Chapter 3: Bagging with Random Forests
Technical requirements
Bagging ensembles
Exploring random forests
Random forest hyperparameters
Pushing random forest boundaries – case study
Summary
5
Chapter 4: From Gradient Boosting to XGBoost
Chapter 4: From Gradient Boosting to XGBoost
Technical requirements
From bagging to boosting
How gradient boosting works
Modifying gradient boosting hyperparameters
Approaching big data – gradient boosting versus XGBoost
Summary
6
Section 2: XGBoost
Section 2: XGBoost
7
Chapter 5: XGBoost Unveiled
Chapter 5: XGBoost Unveiled
Designing XGBoost
Analyzing XGBoost parameters
Building XGBoost models
Finding the Higgs boson – case study
Summary
8
Chapter 6: XGBoost Hyperparameters
Chapter 6: XGBoost Hyperparameters
Technical requirements
Preparing data and base models
Tuning XGBoost hyperparameters
Applying early stopping
Combining hyperparameters
Summary
9
Chapter 7: Discovering Exoplanets with XGBoost
Chapter 7: Discovering Exoplanets with XGBoost
Technical requirements
Searching for exoplanets
Analyzing the confusion matrix
Resampling imbalanced data
Tuning and scaling XGBClassifier
Summary
10
Section 3: Advanced XGBoost
Section 3: Advanced XGBoost
11
Chapter 8: XGBoost Alternative Base Learners
Chapter 8: XGBoost Alternative Base Learners
Technical requirements
Exploring alternative base learners
Applying gblinear
Comparing dart
Finding XGBoost random forests
Summary
12
Chapter 9: XGBoost Kaggle Masters
Chapter 9: XGBoost Kaggle Masters
Technical requirements
Exploring Kaggle competitions
Engineering new columns
Building non-correlated ensembles
Stacking models
Summary
13
Chapter 10: XGBoost Model Deployment
Chapter 10: XGBoost Model Deployment
Technical requirements
Encoding mixed data
Customizing scikit-learn transformers
Finalizing an XGBoost model
Building a machine learning pipeline
Summary
14
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Previewing XGBoost

Machine learning gained recognition with the first neural network in the 1940s, followed by the first machine learning checker champion in the 1950s. After some quiet decades, the field of machine learning took off when Deep Blue famously beat world chess champion Gary Kasparov in the 1990s. With a surge in computational power, the 1990s and early 2000s produced a plethora of academic papers revealing new machine learning algorithms such as random forests and AdaBoost.

The general idea behind boosting is to transform weak learners into strong learners by iteratively improving upon errors. The key idea behind gradient boosting is to use gradient descent to minimize the errors of the residuals. This evolutionary strand, from standard machine learning algorithms to gradient boosting, is the focus of the first four chapters of this book.

XGBoost is short for Extreme Gradient Boosting. The Extreme part refers to pushing the limits of computation to achieve gains in accuracy and speed. XGBoost's surging popularity is largely due to its unparalleled success in Kaggle competitions. In Kaggle competitions, competitors build machine learning models in attempts to make the best predictions and win lucrative cash prizes. In comparison to other models, XGBoost has been crushing the competition.

Understanding the details of XGBoost requires understanding the landscape of machine learning within the context of gradient boosting. In order to paint a full picture, we start at the beginning, with the basics of machine learning.

What is machine learning?

Machine learning is the ability of computers to learn from data. In 2020, machine learning predicts human behavior, recommends products, identifies faces, outperforms poker professionals, discovers exoplanets, identifies diseases, operates self-driving cars, personalizes the internet, and communicates directly with humans. Machine learning is leading the artificial intelligence revolution and affecting the bottom line of nearly every major corporation.

In practice, machine learning means implementing computer algorithms whose weights are adjusted when new data comes in. Machine learning algorithms learn from datasets to make predictions about species classification, the stock market, company profits, human decisions, subatomic particles, optimal traffic routes, and more.

Machine learning is the best tool at our disposal for transforming big data into accurate, actionable predictions. Machine learning, however, does not occur in a vacuum. Machine learning requires rows and columns of data.

Previous Section
End of Section 2
Next Section