文档下载 文档下载导航
设为首页 | 加入收藏
搜索 请输入内容:  
 导航当前位置: 文档下载 > 所有分类 > 外语学习 > 英语学习 > Adaptive Chinese Word Segmentation1
侵权投诉

Adaptive Chinese Word Segmentation1

Adaptive Chinese Word Segmentation1

Adaptive Chinese Word Segmentation

1

Jianfeng Gao*, Andi Wu*, Mu Li*, Chang-Ning Huang*, Hongqiao Li**, Xinsong Xia$, Haowei Qin&

*

Microsoft Research. {jfgao, andiwu, muli, cnhuang}@microsoft.com **

Beijing Institute of Technology, Beijing. lhqtxm@bit.edu.cn $

Peking University, Beijing. xia_xinsong@founder.com &

Shanghai Jiaotong university, Shanghai. haoweiqin@sjtu.edu.cn

Abstract

This paper presents a Chinese word segmen-tation system which can adapt to different domains and standards. We first present a sta-tistical framework where domain-specific words are identified in a unified approach to word segmentation based on linear models. We explore several features and describe how to create training data by sampling. We then describe a transformation-based learning method used to adapt our system to different word segmentation standards. Evaluation of the proposed system on five test sets with dif-ferent standards shows that the system achieves state- of-the-art performance on all of them.

1 Introduction

Chinese word segmentation has been a long- standing research topic in Chinese language proc-essing. Recent development in this field shows that, in addition to ambiguity resolution and unknown word detection, the usefulness of a Chinese word segmenter also depends crucially on its ability to adapt to different domains of texts and different segmentation standards.

The need of adaptation involves two research issues that we will address in this paper. The first is new word detection. Different domains/applications may have different vocabularies which contain new words/terms that are not available in a general dictionary. In this paper, new words refer to OOV words other than named entities, factoids and mor-phologically derived words. These words are mostly domain specific terms (e.g. 蜂窝式 ‘cellular’) and time-sensitive political, social or cultural terms (e.g. 三通‘Three Links’, 非典 ‘SARS’).

The second issue concerns the customizable display of word segmentation. Different Chinese 1

NLP-enabled applications may have different re-quirements that call for different granularities of word segmentation. For example, speech recogni-tion systems prefer “longer words” to achieve higher accuracy whereas information retrieval systems prefer “shorter words” to obtain higher recall rates, etc. (Wu, 2003). Given a word seg-mentation specification (or standard) and/or some application data used as training data, a segmenter with customizable display should be able to provide alternative segmentation units according to the specification which is either pre-defined or implied in the data.

In this paper, we first present a statistical framework for Chinese word segmentation, where various problems of word segmentation are solved simultaneously in a unified approach. Our ap-proach is based on linear models where component models are inspired by the source-channel models of Chinese sentence generation. We then describe in detail how the new word identification (NWI) problem is handled in this framework. We explore several features and describe how to create training data by sampling. We evaluate the performance of our segmentation system using an annotated test set, where new words are simulated by sampling. We then describe a transformation-based learning (TBL, Brill, 1995) method that is used to adapt our system to different segmentation standards. We compare the adaptive system to other state-of-the-art systems using four test sets in the SIGHAN’s First Interna-tional Chinese Word Segmentation Bakeoff, each of which is constructed according to a different seg-mentation standard. The performance of our system is comparable to the best systems reported on all four test sets. It demonstrates the possibility of having a single adaptive Chinese word segmenter that is capable of supporting multiple user applica-tions.

This work was done while Hongqiao Li, Xinsong Xia and Haowei Qin were visiting Microsoft Research (MSR) Asia. We thank Xiaodan Zhu for his early contribution, and the three reviewers, one of whom alerted us the related work of (Uchimoto et al., 2001).

第1页

你可能喜欢

热门文档

相关文档

站点地图 | 文档上传 | 侵权投诉 | 手机版
本站所有资源均来自互联网,本站只负责收集和整理,均不承担任何法律责任,如有侵权等其它行为请联系我们.
文档下载 Copyright 2013 doc.guandang.net All Rights Reserved.  -  email