Adaptive Chinese Word Segmentation1
Adaptive Chinese Word Segmentation
1
Jianfeng Gao*, Andi Wu*, Mu Li*, Chang-Ning Huang*, Hongqiao Li**, Xinsong Xia$, Haowei Qin&
*
Microsoft Research. {jfgao, andiwu, muli, cnhuang}@microsoft.com **
Beijing Institute of Technology, Beijing. lhqtxm@bit.edu.cn $
Peking University, Beijing. xia_xinsong@founder.com &
Shanghai Jiaotong university, Shanghai. haoweiqin@sjtu.edu.cn
Abstract
This paper presents a Chinese word segmen-tation system which can adapt to different domains and standards. We first present a sta-tistical framework where domain-specific words are identified in a unified approach to word segmentation based on linear models. We explore several features and describe how to create training data by sampling. We then describe a transformation-based learning method used to adapt our system to different word segmentation standards. Evaluation of the proposed system on five test sets with dif-ferent standards shows that the system achieves state- of-the-art performance on all of them.
1 Introduction
Chinese word segmentation has been a long- standing research topic in Chinese language proc-essing. Recent development in this field shows that, in addition to ambiguity resolution and unknown word detection, the usefulness of a Chinese word segmenter also depends crucially on its ability to adapt to different domains of texts and different segmentation standards.
The need of adaptation involves two research issues that we will address in this paper. The first is new word detection. Different domains/applications may have different vocabularies which contain new words/terms that are not available in a general dictionary. In this paper, new words refer to OOV words other than named entities, factoids and mor-phologically derived words. These words are mostly domain specific terms (e.g. 蜂窝式 ‘cellular’) and time-sensitive political, social or cultural terms (e.g. 三通‘Three Links’, 非典 ‘SARS’).
The second issue concerns the customizable display of word segmentation. Different Chinese 1
NLP-enabled applications may have different re-quirements that call for different granularities of word segmentation. For example, speech recogni-tion systems prefer “longer words” to achieve higher accuracy whereas information retrieval systems prefer “shorter words” to obtain higher recall rates, etc. (Wu, 2003). Given a word seg-mentation specification (or standard) and/or some application data used as training data, a segmenter with customizable display should be able to provide alternative segmentation units according to the specification which is either pre-defined or implied in the data.
In this paper, we first present a statistical framework for Chinese word segmentation, where various problems of word segmentation are solved simultaneously in a unified approach. Our ap-proach is based on linear models where component models are inspired by the source-channel models of Chinese sentence generation. We then describe in detail how the new word identification (NWI) problem is handled in this framework. We explore several features and describe how to create training data by sampling. We evaluate the performance of our segmentation system using an annotated test set, where new words are simulated by sampling. We then describe a transformation-based learning (TBL, Brill, 1995) method that is used to adapt our system to different segmentation standards. We compare the adaptive system to other state-of-the-art systems using four test sets in the SIGHAN’s First Interna-tional Chinese Word Segmentation Bakeoff, each of which is constructed according to a different seg-mentation standard. The performance of our system is comparable to the best systems reported on all four test sets. It demonstrates the possibility of having a single adaptive Chinese word segmenter that is capable of supporting multiple user applica-tions.
This work was done while Hongqiao Li, Xinsong Xia and Haowei Qin were visiting Microsoft Research (MSR) Asia. We thank Xiaodan Zhu for his early contribution, and the three reviewers, one of whom alerted us the related work of (Uchimoto et al., 2001).
Adaptive Chinese Word Se... 8页 免费 中文分词入门之最大匹配法 7页 ... SYSTRAN's Chinese Word SegmentationSYSTRAN's Chinese Word Segmentation隐藏>> ...
Adaptive Chinese Word Se... 8页 免费 A Brief Introduction of ... 40... Word segmentation is the first step in Chinese information processing, and ...
31页 免费 Adaptive Chinese Word Se... 8页 免费搜你所想,读你所爱 ... Chinese Word Segmentation Liqi Gao, Zhuoran Wang IR Lab, HIT 2003-11-8...
Adaptive Chinese Word Se... 8页 免费 Novel Theory and Methods... 5页... Pruning False Unknown Words to Improve Chinese Word Segmentation During the ...
and a Support Vector Machine-based chunker for Chinese word segmentation. ... Chinese Word Segmentat... 暂无评价 16页 免费 Adaptive Chinese Word ... ...
Remove Redundancy Samples for SVM in A Chinese Word Segmentation Task 隐藏>> 维普资讯 http://www.cqvip.com Ma06Vlme3No5(eilo1)y20,ou ,.SraN.8 ...
Effective Analysis of Chinese Word-Segmentation Accuracy_专业资料。维普资讯 http://www.cqvip.com MA eyn:fcieAnayi WiiEfetv lssofChis od-Seea...
D13-1061 Deep Learning for Chinese Word Segmentation and POS Tagging_互联... Adaptive Chinese Word ... 8页 免费 A Stochastic Finite-St... 8页 ...
Introduction to CKIP Chinese word segmentation system for the first international Chinese W_畜牧兽医_农林牧渔_专业资料。Introduction to CKIP Chinese word ...
Unsupervised Training for Overlapping Ambiguity Resolution in Chinese Word Segmentation Mu Li, Jianfeng Gao, Changning Huang Microsoft Research, Asia Beijing ...
热门文档