文档下载 文档下载导航
设为首页 | 加入收藏
搜索 请输入内容:  
 导航当前位置: 文档下载 > 所有分类 > OUTLIER DETECTION
侵权投诉

OUTLIER DETECTION

In many data analysis tasks a large number of variables are being recorded or sampled. One of the first steps towards obtaining a coherent analysis is the detection of outlaying observations. Although outliers are often considered as an error or noise, the

OUTLIER DETECTION

Irad Ben-Gal

Department of Industrial Engineering

Tel-Aviv University

Ramat-Aviv, Tel-Aviv 69978, ISRAEL

Email: bengal@eng.tau.ac.il

1. Introduction: Motivation, Definitions and Applications

In many data analysis tasks a large number of variables are being recorded or sampled. One of the first steps towards obtaining a coherent analysis is the detection of outlaying observations. Although outliers are often considered as an error or noise, they may carry important

information. Detected outliers are candidates for aberrant data that may otherwise adversely lead to model misspecification, biased parameter estimation and incorrect results. It is

therefore important to identify them prior to modeling and analysis (Williams et al., 2002, Liu et al., 2004).

An exact definition of an outlier often depends on hidden assumptions regarding the data

structure and the applied detection method. Yet, some definitions are regarded general enough to cope with various types of data and methods. Hawkins (1980) defines an outlier as an observation that deviates so much from other observations as to arouse suspicion that it was generated by a different mechanism. Barnet and Lewis (1994) indicate that an outlying observation, or outlier, is one that appears to deviate markedly from other members of the sample in which it occurs, similarly, Johnson (1992) defines an outlier as an observation in a data set which appears to be inconsistent with the remainder of that set of data. Other case-specific definitions are given below.

Outlier detection methods have been suggested for numerous applications, such as credit card fraud detection, clinical trials, voting irregularity analysis, data cleansing, network intrusion, severe weather prediction, geographic information systems, athlete performance analysis, and other data-mining tasks (Hawkins 1980, Barnett and Lewis, 1994, Ruts and Rousseeuw, 1996, Fawcett and Provost, 1997, Johnson et al., 1998, Penny and Jolliffe, 2001, Acuna and Rodriguez, 2004, Lu et al., 2003).

2. Taxonomy of Outlier Detection Methods

Outlier detection methods can be divided between univariate methods, proposed in earlier works in this field, and multivariate methods that usually form most of the current body of research. Another fundamental taxonomy of outlier detection methods is between parametric (statistical) methods and nonparametric methods that are model-free (e.g., Williams et al., 2002). Statistical parametric methods either assume a known underlying distribution of the observations (e.g., Hawkins, 1980, Rousseeuw and Leory, 1987, Barnett and Lewis, 1994) or, at least, they are based on statistical estimates of unknown distribution parameters (Hadi, 1992, Caussinus and Roiz, 1990). These methods flag as outliers those observations that

deviate from the model assumptions. They are often unsuitable for high-dimensional data sets and for arbitrary data sets without prior knowledge of the underlying data distribution (Papadimitriou et al., 2002).

Within the class of non-parametric outlier detection methods one can set apart the data-mining methods, also called distance-based methods. These methods are usually based on local distance measures and are capable of handling large databases (Knorr and Ng, 1997,

第1页

你可能喜欢

热门文档

相关文档

  • Outlier Detection Techniques

    NCHEN INSTITUTE FOR INFORMATICS DATABASE SYSTEMS GROUP 16th ACM SIGKDD Conference on Knowledge Discovery and Data Mining Outlier Detection Techniques Hans-Peter...

  • Outlier Detection

    Outlier Detection A brief introduction Fuzhou University 智能技术课程---D.Y. YE ? ?What is outlier detection? Most knowledge discovery in databases focuses...

  • outlier detection

    Ben-Gal I., Outlier detection, In: Maimon O. and Rockach L. (Eds.) Data Mining and Knowledge Discovery Handbook: A Complete Guide for Practitioners ...

  • ...trajectory outlier detection algorithm

    Density-based trajectory outlier detection algorithm_材料科学_工程科技_专业资料。Journal of Systems Engineering and Electronics Vo1.24,N...

  • 【论文】Rough Outlier Detection Based Security Risk...

    Rough Outlier Detection Based Security Risk Analysis Methodology_专业资料。呻遏 国怯 Chn mmuiaisiaConcton Ro utirD eto ugh0 l etcine B...

  • Outlier Detection Algorithms in Data Mining

    Figure 1 Outlier Detection Approach 3.1. Classic Outlier Classic outlier approach analyzes outlier based on transaction dataset, which consists of collections ...

  • Privacy-Preserving Outlier Detection

    Privacy-Preserving Outlier Detection Jaideep Vaidya Rutgers University 180 University Avenue Newark, NJ 07102-1803 jsvaidya@rbs.rutgers.edu Chris Clifton ...

  • 12Outlier离群分析

    Otherwise, it is likely an outlier. 21 Chapter 12. Outlier Analysis ? ? ? ? ? ? ? ? Outlier and Outlier Analysis Outlier Detection Methods ...

  • Outlier detection in astronomical data

    Outlier detection in astronomical data Yanxia Zhang*a , Ali Luo*a , and Yongheng Zhao*a *a National Astronomical Observatories, Chinese Academy of ...

  • 2014Outlier detection based on granular computing a...

    2014Outlier detection based on granular computing and rough set theory_材料科学_工程科技_专业资料。Appl Intell DOI 10.1007/s10489-014-0591-4 Outlier ...

站点地图 | 文档上传 | 侵权投诉 | 手机版
本站所有资源均来自互联网,本站只负责收集和整理,均不承担任何法律责任,如有侵权等其它行为请联系我们.
文档下载 Copyright 2013 doc.guandang.net All Rights Reserved.  -  email