Thus the MSE can be decomposed in expectation into the variance of the noiseand the MSE between the true function and the predicted values. This term canbe further composed with the same augmentation trick as above.E(fi yi)
{
2
}{(
=Efi E{yi}+E{yi}i yi=E(fi E{yi})=bias2
{
}{}+2E{(E{y} y)(f E{y})}+Var{y}+2(E{fE{y}} E{E{y}} E{yf}+E{yE{y}})
2
)}
2i
+E(E{yi} yi)
i
2
iiii
2
iii
i
ii
Note: E{fiE{yi}}=fiE{yi} since f is deterministic and E{E{z}}=z
:EE{yi}
{
2
}=E{y} since E{E{z}}=z
2i
2
:E{yifi}=fiE{yi}:E{yiE{yi}}=E{yi}
E(fi yi)
: Thus the last term in the expectation above cancels to zero.
Thus the decomposition of the MSE in expectation becomes:
E(ti yi)
{
2
}=bias
2
+Var{yi}
{
2
}=Var{noise}+bias
2
+Var{yi}
Note that the variance of the noise can not be minimized; it is independent of theneural network. Thus in order to minimize the MSE, we need to minimize boththe bias and the variance. However, this is not trivial to do this. For instance, justneglecting the input data and predicting the output somehow (e.g., just a con-stant), would definitely minimize the variance of our predictions: they would bealways the same, thus the variance would be zero—but the bias of our estimate(i.e., the amount we are off the real function) would be tremendously large. Onthe other hand, the neural network could perfectly interpolate the training data,i.e., it predict y=t for every data point. This will make the bias term vanish en-tirely, since the E(y)=f (insert this above into the squared bias term to verify this),but the variance term will become equal to the variance of the noise, which maybe significant (see also Bishop Chapter 9 and the Geman et al. Paper). In general,finding an optimal bias-variance tradeoff is hard, but acceptable solutions can befound, e.g., by means of cross validation or regularization.
2–16 Contents lists available at ScienceDirect Cognition journal homepage: www.elsevier.com/locate/COGNIT Conceptual complexity and the bias/variance trade...
对最初 SVMs 发展的驱动发生在 guises-the bias variance tradeoff、学习能力控制,过学 习等——但是基本思想是相同的。粗略的说,对于一个给定的,只有有限训练...
This paper analyzes the bias-variance tradeoff of a class of l隐藏>> Suboptimality of the Truncated SVD for Ill-Conditioned Inverse Problems Jeffrey A. ...
The Bias Variance Tradeo... 暂无评价 2页 1财富值 Individual investors ... Local Unidirectional Bias for Smooth Cutsize-Delay Tradeoff in Performance-...
对最初 SVMs 发展的驱动发生在 guises-the bias variance tradeoff、学习能力控制,过学 习等——但是基本思想是相同的。粗略的说,对于一个给定的,只有有限训练...
Achievable Regions in the Bias-Variance Plane for Parametric Estimation Problems 1 Alfred H Abstract--- In this paper we use the uniform Cramer-Rao (...
er Practical algorithms face the bias–variance tradeoff 4 Feature subset selection (2) ? Optimal feature subset de?ned with respect to the induction ...
Statistical Learning The problem which drove the initial development of SVMs occurs in several guises –the bias variance tradeoff,capacity control and over ...
A behavior typical of the bias-variance tradeoff is visible in Figure 2, where the resubstitution error and the relative over?tting rate estimate R are...
3 Fighting the bias-variance tradeoff 4 Voting (Ensemble Methods) 5 Voting (Ensemble Methods) 6 Boosting [Schapire’89] 7 Learning from weighted data ...
热门文档