In many data analysis tasks a large number of variables are being recorded or sampled. One of the first steps towards obtaining a coherent analysis is the detection of outlaying observations. Although outliers are often considered as an error or noise, the
OUTLIER DETECTION
Irad Ben-Gal
Department of Industrial Engineering
Tel-Aviv University
Ramat-Aviv, Tel-Aviv 69978, ISRAEL
Email: bengal@eng.tau.ac.il
1. Introduction: Motivation, Definitions and Applications
In many data analysis tasks a large number of variables are being recorded or sampled. One of the first steps towards obtaining a coherent analysis is the detection of outlaying observations. Although outliers are often considered as an error or noise, they may carry important
information. Detected outliers are candidates for aberrant data that may otherwise adversely lead to model misspecification, biased parameter estimation and incorrect results. It is
therefore important to identify them prior to modeling and analysis (Williams et al., 2002, Liu et al., 2004).
An exact definition of an outlier often depends on hidden assumptions regarding the data
structure and the applied detection method. Yet, some definitions are regarded general enough to cope with various types of data and methods. Hawkins (1980) defines an outlier as an observation that deviates so much from other observations as to arouse suspicion that it was generated by a different mechanism. Barnet and Lewis (1994) indicate that an outlying observation, or outlier, is one that appears to deviate markedly from other members of the sample in which it occurs, similarly, Johnson (1992) defines an outlier as an observation in a data set which appears to be inconsistent with the remainder of that set of data. Other case-specific definitions are given below.
Outlier detection methods have been suggested for numerous applications, such as credit card fraud detection, clinical trials, voting irregularity analysis, data cleansing, network intrusion, severe weather prediction, geographic information systems, athlete performance analysis, and other data-mining tasks (Hawkins 1980, Barnett and Lewis, 1994, Ruts and Rousseeuw, 1996, Fawcett and Provost, 1997, Johnson et al., 1998, Penny and Jolliffe, 2001, Acuna and Rodriguez, 2004, Lu et al., 2003).
2. Taxonomy of Outlier Detection Methods
Outlier detection methods can be divided between univariate methods, proposed in earlier works in this field, and multivariate methods that usually form most of the current body of research. Another fundamental taxonomy of outlier detection methods is between parametric (statistical) methods and nonparametric methods that are model-free (e.g., Williams et al., 2002). Statistical parametric methods either assume a known underlying distribution of the observations (e.g., Hawkins, 1980, Rousseeuw and Leory, 1987, Barnett and Lewis, 1994) or, at least, they are based on statistical estimates of unknown distribution parameters (Hadi, 1992, Caussinus and Roiz, 1990). These methods flag as outliers those observations that
deviate from the model assumptions. They are often unsuitable for high-dimensional data sets and for arbitrary data sets without prior knowledge of the underlying data distribution (Papadimitriou et al., 2002).
Within the class of non-parametric outlier detection methods one can set apart the data-mining methods, also called distance-based methods. These methods are usually based on local distance measures and are capable of handling large databases (Knorr and Ng, 1997,
NCHEN INSTITUTE FOR INFORMATICS DATABASE SYSTEMS GROUP 16th ACM SIGKDD Conference on Knowledge Discovery and Data Mining Outlier Detection Techniques Hans-Peter...
Outlier Detection A brief introduction Fuzhou University 智能技术课程---D.Y. YE ? ?What is outlier detection? Most knowledge discovery in databases focuses...
Ben-Gal I., Outlier detection, In: Maimon O. and Rockach L. (Eds.) Data Mining and Knowledge Discovery Handbook: A Complete Guide for Practitioners ...
Density-based trajectory outlier detection algorithm_材料科学_工程科技_专业资料。Journal of Systems Engineering and Electronics Vo1.24,N...
Rough Outlier Detection Based Security Risk Analysis Methodology_专业资料。呻遏 国怯 Chn mmuiaisiaConcton Ro utirD eto ugh0 l etcine B...
Figure 1 Outlier Detection Approach 3.1. Classic Outlier Classic outlier approach analyzes outlier based on transaction dataset, which consists of collections ...
Privacy-Preserving Outlier Detection Jaideep Vaidya Rutgers University 180 University Avenue Newark, NJ 07102-1803 jsvaidya@rbs.rutgers.edu Chris Clifton ...
Otherwise, it is likely an outlier. 21 Chapter 12. Outlier Analysis ? ? ? ? ? ? ? ? Outlier and Outlier Analysis Outlier Detection Methods ...
Outlier detection in astronomical data Yanxia Zhang*a , Ali Luo*a , and Yongheng Zhao*a *a National Astronomical Observatories, Chinese Academy of ...
2014Outlier detection based on granular computing and rough set theory_材料科学_工程科技_专业资料。Appl Intell DOI 10.1007/s10489-014-0591-4 Outlier ...
热门文档