Description about bagging and boosting wrong?

In the "Framework for Ensemble Learning", the ensemble algorithms are described as "Bag generally constructs deep trees. This construction is both time consuming and memory-intensive. This also leads to relatively slow predictions. Boost algorithms generally use very shallow trees. This construction uses relatively little time or memory. However, for effective predictions, boosted trees might need more ensemble members than bagged trees. Therefore it is not always clear which class of algorithms is superior." Should this be inverse? To my understanding, Bag is shallow and boost is deep...

Answer 1

[Akshat]

Hi @HAICHUAN TANG,

The description you provided from the "Framework for Ensemble Learning" is accurate in terms of general characteristics of bagging and boosting, but there might be some confusion regarding the depth of trees used in these methods:

• Bagging (Bootstrap Aggregating):

1. Deep Trees: Bagging typically uses deep trees, such as in the Random Forest algorithm, where each tree is grown to a large depth to capture complex patterns. This results in high variance for individual trees, which bagging mitigates by averaging their predictions.
2. Resource Intensive: The creation of deep trees is indeed time-consuming and memory-intensive, leading to slower predictions.

• Boosting:

1. Shallow Trees: Boosting often uses shallow trees, sometimes referred to as "stumps," which are trees with a small number of levels (often just one or two). The idea is to sequentially correct the errors of previous trees by focusing on difficult-to-predict instances.
2. More Trees Needed: Because each tree is weak, boosting typically requires more trees to achieve strong predictive performance.

Thus, the description correctly contrasts the two methods in terms of tree depth and resource usage.

You can refer to this link to get more information on why Bagging involves deep trees and Boosting uses shallow trees.

https://datascience.stackexchange.com/questions/61771/bagging-vs-boosting-bias-vs-variance-depth-of-trees

Hope this helps!