文档下载 文档下载导航
设为首页 | 加入收藏
搜索 请输入内容:  
 导航当前位置: 文档下载 > 所有分类 > A Unified Tagging Approach to Text Normalization
侵权投诉

A Unified Tagging Approach to Text Normalization

This paper addresses the issue of text normalization, an important yet often overlooked problem in natural language processing. By text normalization, we mean converting ‘informally inputted ’ text into the canonical form, by eliminating ‘noises’ in th

A Unified Tagging Approach to Text Normalization

Conghui Zhu Jie Tang Hang Li

Harbin Institute of Technology Department of Computer ScienceMicrosoft Research Asia

Harbin, China Tsinghua University, China Beijing, China

chzhu@mtlab.hit.edu.cn jietang@tsinghua.edu.cnhangli@microsoft.comHwee Tou Ng

Department of Computer Science

National University of Singapore, Singapore

nght@comp.nus.edu.sg

Abstract

This paper addresses the issue of text nor-malization, an important yet often over-looked problem in natural language proc-essing. By text normalization, we mean

converting ‘informally inputted’ text into

the canonical form, by eliminating ‘noises’

in the text and detecting paragraph and sen-tence boundaries in the text. Previously,

text normalization issues were often under-taken in an ad-hoc fashion or studied sepa-rately. This paper first gives a formaliza-tion of the entire problem. It then proposes

a unified tagging approach to perform the

task using Conditional Random Fields

(CRF). The paper shows that with the in-troduction of a small set of tags, most of

the text normalization tasks can be per-formed within the approach. The accuracy

of the proposed method is high, because

the subtasks of normalization are interde-pendent and should be performed together.

Experimental results on email data cleaning

show that the proposed method signifi-cantly outperforms the approach of using

cascaded models and that of employing in-dependent models.

1 Introduction More and more ‘informally inputted’ text data be-comes available to natural language processing, Tie-Jun Zhao

Harbin Institute of Technology

Harbin, China

tjzhao@mtlab.hit.edu.cn

such as raw text data in emails, newsgroups, fo-rums, and blogs. Consequently, how to effectively process the data and make it suitable for natural

language processing becomes a challenging issue.

This is because informally inputted text data is

usually very noisy and is not properly segmented.

For example, it may contain extra line breaks, extra

spaces, and extra punctuation marks; and it may

contain words badly cased. Moreover, the bounda-ries between paragraphs and the boundaries be-tween sentences are not clear.

We have examined 5,000 randomly collected

emails and found that 98.4% of the emails contain

noises (based on the definition in Section 5.1).

In order to perform high quality natural lan-guage processing, it is necessary to perform ‘nor-malization’ on informally inputted data first, spe-cifically, to remove extra line breaks, segment the

text into paragraphs, add missing spaces and miss-ing punctuation marks, eliminate extra spaces and

extra punctuation marks, delete unnecessary tokens,

correct misused punctuation marks, restore badly

cased words, correct misspelled words, and iden-tify sentence boundaries.

Traditionally, text normalization is viewed as an

engineering issue and is conducted in a more or

less ad-hoc manner. For example, it is done by us-ing rules or machine learning models at different

levels. In natural language processing, several is-sues of text normalization were studied, but were

only done separately.

This paper aims to conduct a thorough investiga-tion on the issue. First, it gives a formalization of

第1页

你可能喜欢

热门文档

相关文档

站点地图 | 文档上传 | 侵权投诉 | 手机版
本站所有资源均来自互联网,本站只负责收集和整理,均不承担任何法律责任,如有侵权等其它行为请联系我们.
文档下载 Copyright 2013 doc.guandang.net All Rights Reserved.  -  email