文档下载 文档下载导航
设为首页 | 加入收藏
搜索 请输入内容:  
 导航当前位置: 文档下载 > 所有分类 > Towards a Theoretical Framework for Ensemble Classification
侵权投诉

Towards a Theoretical Framework for Ensemble Classification

Ensemble learning schemes such as AdaBoost and Bagging enhance the performance of a single classifier by combining predictions from multiple classifiers of the same type. The predictions from an ensemble of diverse classifiers can be combined in related wa

Towards a Theoretical Framework for Ensemble Classification

Alexander K. Seewald

Austrian Research Institute for Artificial Intelligence

Freyung 6/VI/7, A-1010 Wien, Austria alexsee@oefai.at; alex@seewald.at

I

Abstract

Ensemble learning schemes such as AdaBoost and Bagging enhance the performance of a single clas-sifier by combining predictions from multiple clas-sifiers of the same type. The predictions from an ensemble of diverse classifiers can be combined in related ways, e.g. by voting or simply by se-Ilecting the best classifier via cross-validation - a technique widely used in machine learning. How-ever, since no ensemble scheme is always the best choice, a deeper insight into the structure of mean-ingful approaches to combine predictions is needed to achieve further progress. n this paper we offer an operational reformulation of common ensemble learning schemes - Voting, Selection by Crossvali-dation (X-Val), Grading and Bagging - as a Stacking I

scheme with appropriate parameter settings. Thus, from a theoretical point of view all these schemes can be reduced to Stacking with an appropriate combination method. This result is an important step towards a general theoretical framework for the field of ensemble learning.

1 ntroduction

We apologize that space restrictions arc forcing us to be terse. For a more detailed version, see [Seewald, 2003]. We will first explain how Stacking works in order to lay the founda-tions for our functional definitions of meta classifiers later on. Fig. 1 shows Stacking on a hypothetical dataset.

During training, all base classifiers arc evaluated via cross-validation on the original dataset. Each classifier's output is a class probability distribution for every example, see Fig. 1(b) The concatenated class probability distributions of all base classifiers, followed by the class value, forms the meta-level training set for Stacking's meta classifier, see Fig. 1(c) After training the meta classifier, the base classifiers are retrained on the complete training data.

For testing, the base classifiers are queried for their class probability distributions. These form a meta-example for the meta classifier which outputs the final class prediction.

2 Definitions

An arbitrary training dataset with n examples and k classes, and a single test instance, is given. N arbitrary base classifiers have been cross-validated on this dataset, and afterwards re-trained on the whole dataset. All base classifiers output class probability distributions, i.e. estimated probabilities for each class instead of deciding on a single class.

Then, refers to the probability given by classifier i for class j on example number k during is the class prob-ability distribution of classifier i on instance k. refers to

POSTER PAPERS

Towards a Theoretical Framework for Ensemble Classification

Towards a Theoretical Framework for Ensemble Classification

Figure 1: llustration of Stacking on a dataset with three classes (a, b and c), n examples and TV base classifiers. refers to the probability given by classifier i for class j on example number k. the class prob.dist. for classifier i on the unknown test in-stance. A fixed arbitrary order on the class values and N base classifiers is assumed. Classk is the true class for instance k.

is the attribute vector of instance k. n is the num-ber of instances. All indices are zero-based, e.g. the instance id k satisfies the equation 0 k

Towards a Theoretical Framework for Ensemble Classification

n — 1.

As we mentioned, we assume that all ensemble learning schemes return class probability distributions. f predictions are needed, the position of the maximum class probability in the vector - i.e. the predicted class - is easily obtained via formula (2) Trivially, Stacking with predictions can also be simulated by this variant; simply by transforming the class distributions meta-dataset to predictions via (2) prior to ap-plying the meta classifier.

We can now characterize every ensemble learning scheme by what features it extracts from the meta-dataset during training and how these features define the mapping from meta-dataset to final class probability distribution.

2.1 Voting

Voting, a straight-forward extension of voting for distribution classifiers, is the simplest case. During training, no features are extracted from the meta-dataset. n fact Voting does not even need the internal crossvalidation. During testing, Voting determines the final class probability distribution as follows.

(3)

1443

第1页

你可能喜欢

热门文档

相关文档

站点地图 | 文档上传 | 侵权投诉 | 手机版
本站所有资源均来自互联网,本站只负责收集和整理,均不承担任何法律责任,如有侵权等其它行为请联系我们.
文档下载 Copyright 2013 doc.guandang.net All Rights Reserved.  -  email