SYSTRAN’s Chinese word segmentation is one important component of its Chinese-English machine translation system. The Chinese word segmentation module uses a rule-based approach, based on a large dictionary and fine-grained linguistic rules. It works on g
Proceedings of the Second SIGHAN Workshop on Chinese Language Processing, July 2003, pp. 180-183.
SYSTRAN’s Chinese Word Segmentation
Jin Yang
SYSTRAN Software, Inc. 9333 Genesee Ave. San Diego, CA 92121, USA
jyang@systransoft.com
Jean Senellart
SYSTRAN S.A. 1, rue du Cimetière
95230 Soisy-sous-Montmorency, France
senellart@systran.fr
Remi Zajac
SYSTRAN Software, Inc. 9333 Genesee Ave. San Diego, CA 92121, USA
zajac@systransoft.com
Abstract
SYSTRAN’s Chinese word segmentation is one important component of its Chinese-English machine translation system. The Chinese word segmentation module uses a rule-based approach, based on a large dictionary and fine-grained linguistic rules. It works on general-purpose texts from different Chinese-speaking regions, with comparable performance. SYSTRAN participated in the four open tracks in the First International Chinese Word Segmentation Bakeoff. This paper gives a general description of the segmentation module, as well as the results and analysis of its performance in the Bakeoff. 1 Introduction Chinese word segmentation is one of the pre-processing steps of the SYSTRAN Chinese-English Machine Translation (MT) system. The
development of the Chinese-English MT system
began in August 1994, and this is where the
Chinese word segmentation issue was first
addressed. The algorithm of the early version of
the segmentation module was borrowed from SYSTRAN’s Japanese segmentation module. The program ran on a large word list, which contained 600,000 entries at the time1
. The basic strategy was to list all possible matches for an entire linguistic unit, then solve the overlapping matches via linguistic rules. The development was focused on technical domains, and high accuracy was achieved after only three months of development. Since then, development has shifted to other areas
of Chinese-English MT, including the enrichment of the bi-lingual word lists with part-of-speech, syntactic and semantic features. In 2001, the development of a prototype Chinese-Japanese MT system began. Although the project only lasted for three months, some important changes were made in the segmentation convention, regarding the distinction between words and phrases2. Along with new developments of the SYSTRAN MT engine, the segmentation engine has recently been re-implemented. The dictionary and the general approach remain unchanged, but dictionary lookup and rule matching were re-implemented using finite-state technology, and linguistic rules for the segmentation module are now expressed using a context-free-based formalism, improving maintainability. The re-implementation generates multiple segmentation results with associated probabilities. This will allow for disambiguation at a later stage of the MT process, and will widen the possibility of word segmentation for other applications.
2 System Description
2.1 Segmentation Standard Our definition of words and our segmentation
conventions are based on available standards,
modified for MT purposes. The PRC standard (Liu
et al., 1993) was initially used. Sample differences
are listed as follows:
SYSTRAN NP 中华民族 中华 民族 中华人民共和国 中华 人民 共和国
31日 日 CD + M 一个 一排排 一 个 一 排 排 DI4 + CD 第一 第 一 Name 李 白 李 清照 李白 李清照
Table 1. Segmentation Divergences with the PRC Guidelines 2.2 Methodology
The SYSTRAN Chinese word segmentation module uses a rule-based approach and a large dictionary. The dictionary is derived from the
SYSTRAN’s Chinese Word ... 4页 免费 Chinese word segmentatio... 4页 ... Chinese Word Segmentation Liqi Gao, Zhuoran Wang IR Lab, HIT 2003-11-8...
Chinese word segmentatio... 4页 免费 SYSTRAN’s Chinese Word ... 4页 ... Word segmentation is the first step in Chinese information processing, and ...
including Chinese automatic word segmentation, Chinese text automatic tagging,...欧洲共同体 在美国乔治敦大学开发的机译系统 SYSTRAN 的基础上成功地进行了英、...
(http://www.chinesesoftware.com) HansVision-... SystranBox Systran在线翻译 (http://www.systran...To avoid this irrational segmentation of a ...
(http://www.chinesesoftware.com) HansVision-... SystranBox Systran在线翻译 (http://www.systran...To avoid this irrational segmentation of a ...
热门文档