What is the distinction between "bagging" and "boosting"?

The distinction between bagging and boosting is well captured in the explanation that bagging reduces variance by averaging predictions, while boosting reduces bias by sequentially improving weak models.

Bagging, short for Bootstrap Aggregating, is a technique primarily aimed at reducing the variance of a model. It works by creating multiple subsets of the training dataset through random sampling with replacement. For each of these subsets, a separate model is trained, and the final prediction is made by averaging the predictions of all the models. This averaging process helps to smooth out fluctuations specific to any individual model, thus leading to more stable and generally better predictions.

Boosting, on the other hand, is a method that focuses on improving the predictive performance by reducing bias. It does this in a sequential manner by training models one after the other. Each new model is trained to correct the errors made by the previous models. Essentially, boosting emphasizes the importance of instances that were misclassified or poorly predicted by earlier models, thereby enhancing the overall accuracy of the prediction through this iterative process.

This key difference highlights how bagging reduces overfitting and enhances model stability, while boosting seeks to create a stronger overall model by emphasizing areas of weakness in previous models, thus addressing bias.