Ensemble learning schemes such as AdaBoost and Bagging enhance the performance of a single classifier by combining predictions from multiple classifiers of the same type. The predictions from an ensemble of diverse classifiers can be combined in related wa
Towards a Theoretical Framework for Ensemble Classification
Alexander K. Seewald
Austrian Research Institute for Artificial Intelligence
Freyung 6/VI/7, A-1010 Wien, Austria alexsee@oefai.at; alex@seewald.at
I
Abstract
Ensemble learning schemes such as AdaBoost and Bagging enhance the performance of a single clas-sifier by combining predictions from multiple clas-sifiers of the same type. The predictions from an ensemble of diverse classifiers can be combined in related ways, e.g. by voting or simply by se-Ilecting the best classifier via cross-validation - a technique widely used in machine learning. How-ever, since no ensemble scheme is always the best choice, a deeper insight into the structure of mean-ingful approaches to combine predictions is needed to achieve further progress. n this paper we offer an operational reformulation of common ensemble learning schemes - Voting, Selection by Crossvali-dation (X-Val), Grading and Bagging - as a Stacking I
scheme with appropriate parameter settings. Thus, from a theoretical point of view all these schemes can be reduced to Stacking with an appropriate combination method. This result is an important step towards a general theoretical framework for the field of ensemble learning.
1 ntroduction
We apologize that space restrictions arc forcing us to be terse. For a more detailed version, see [Seewald, 2003]. We will first explain how Stacking works in order to lay the founda-tions for our functional definitions of meta classifiers later on. Fig. 1 shows Stacking on a hypothetical dataset.
During training, all base classifiers arc evaluated via cross-validation on the original dataset. Each classifier's output is a class probability distribution for every example, see Fig. 1(b) The concatenated class probability distributions of all base classifiers, followed by the class value, forms the meta-level training set for Stacking's meta classifier, see Fig. 1(c) After training the meta classifier, the base classifiers are retrained on the complete training data.
For testing, the base classifiers are queried for their class probability distributions. These form a meta-example for the meta classifier which outputs the final class prediction.
2 Definitions
An arbitrary training dataset with n examples and k classes, and a single test instance, is given. N arbitrary base classifiers have been cross-validated on this dataset, and afterwards re-trained on the whole dataset. All base classifiers output class probability distributions, i.e. estimated probabilities for each class instead of deciding on a single class.
Then, refers to the probability given by classifier i for class j on example number k during is the class prob-ability distribution of classifier i on instance k. refers to
POSTER PAPERS


Figure 1: llustration of Stacking on a dataset with three classes (a, b and c), n examples and TV base classifiers. refers to the probability given by classifier i for class j on example number k. the class prob.dist. for classifier i on the unknown test in-stance. A fixed arbitrary order on the class values and N base classifiers is assumed. Classk is the true class for instance k.
is the attribute vector of instance k. n is the num-ber of instances. All indices are zero-based, e.g. the instance id k satisfies the equation 0 k

n — 1.
As we mentioned, we assume that all ensemble learning schemes return class probability distributions. f predictions are needed, the position of the maximum class probability in the vector - i.e. the predicted class - is easily obtained via formula (2) Trivially, Stacking with predictions can also be simulated by this variant; simply by transforming the class distributions meta-dataset to predictions via (2) prior to ap-plying the meta classifier.
We can now characterize every ensemble learning scheme by what features it extracts from the meta-dataset during training and how these features define the mapping from meta-dataset to final class probability distribution.
2.1 Voting
Voting, a straight-forward extension of voting for distribution classifiers, is the simplest case. During training, no features are extracted from the meta-dataset. n fact Voting does not even need the internal crossvalidation. During testing, Voting determines the final class probability distribution as follows.
(3)
1443
Towards a Theoretical Fr... 暂无评价 2页 免费 a general framework for ... 暂无评价 13页 免费 A classification framewo... 暂无评价 22页 免费...
Towards a Theoretical ... 暂无评价 2页 免费 A Binary Classificatio... ...A Framework for Protein Classification Anand Kumara, Barry Smithb,c Laboratory...
process classification framework_经管营销_专业资料。process classification ... Towards a theoretical ... 77页 免费 Chapter 2 What is risk... 40页 ...
A distributed SVM ensemble for image classification and annotation_互联网_IT/计算机_专业资料。 今日推荐 116份文档 2014一级建造师考试...
er. Since the Bayes classi?er gives the theoretical minimum classi?cation ...Aksoy. A Probabilistic Similarity Framework for Content-Based Image Retrieval....
GeneMining in Mayday, a feature selection framework for binary classification The microarray technology has become a standard method for broad scale gene ...
In this paper we extend previous results providing a theoretical analysis of a new Monte Carlo ensemble classifier. The framework allows us to characterize...
A classification theory is more elaborate in that it states that the ...theoretical framework, and specifically the typologies of gatekeeping mechanisms ...
Augmenting ensemble classification for word sense disambiguation with a kernel PCA model The HKUST word sense disambiguation systems benefit from a new ...
Properties of ensemble classification can be studied using the framework of ...rst, an experimental comparison between the theoretical predictions and ...
热门文档