文档下载 文档下载导航
设为首页 | 加入收藏
搜索 请输入内容:  
 导航当前位置: 文档下载 > 所有分类 > SYSTRAN’s Chinese Word Segmentation
侵权投诉

SYSTRAN’s Chinese Word Segmentation

SYSTRAN’s Chinese word segmentation is one important component of its Chinese-English machine translation system. The Chinese word segmentation module uses a rule-based approach, based on a large dictionary and fine-grained linguistic rules. It works on g

Proceedings of the Second SIGHAN Workshop on Chinese Language Processing, July 2003, pp. 180-183.

SYSTRAN’s Chinese Word Segmentation

Jin Yang

SYSTRAN Software, Inc. 9333 Genesee Ave. San Diego, CA 92121, USA

jyang@systransoft.com

Jean Senellart

SYSTRAN S.A. 1, rue du Cimetière

95230 Soisy-sous-Montmorency, France

senellart@systran.fr

Remi Zajac

SYSTRAN Software, Inc. 9333 Genesee Ave. San Diego, CA 92121, USA

zajac@systransoft.com

Abstract

SYSTRAN’s Chinese word segmentation is one important component of its Chinese-English machine translation system. The Chinese word segmentation module uses a rule-based approach, based on a large dictionary and fine-grained linguistic rules. It works on general-purpose texts from different Chinese-speaking regions, with comparable performance. SYSTRAN participated in the four open tracks in the First International Chinese Word Segmentation Bakeoff. This paper gives a general description of the segmentation module, as well as the results and analysis of its performance in the Bakeoff. 1 Introduction Chinese word segmentation is one of the pre-processing steps of the SYSTRAN Chinese-English Machine Translation (MT) system. The

development of the Chinese-English MT system

began in August 1994, and this is where the

Chinese word segmentation issue was first

addressed. The algorithm of the early version of

the segmentation module was borrowed from SYSTRAN’s Japanese segmentation module. The program ran on a large word list, which contained 600,000 entries at the time1

. The basic strategy was to list all possible matches for an entire linguistic unit, then solve the overlapping matches via linguistic rules. The development was focused on technical domains, and high accuracy was achieved after only three months of development. Since then, development has shifted to other areas

of Chinese-English MT, including the enrichment of the bi-lingual word lists with part-of-speech, syntactic and semantic features. In 2001, the development of a prototype Chinese-Japanese MT system began. Although the project only lasted for three months, some important changes were made in the segmentation convention, regarding the distinction between words and phrases2. Along with new developments of the SYSTRAN MT engine, the segmentation engine has recently been re-implemented. The dictionary and the general approach remain unchanged, but dictionary lookup and rule matching were re-implemented using finite-state technology, and linguistic rules for the segmentation module are now expressed using a context-free-based formalism, improving maintainability. The re-implementation generates multiple segmentation results with associated probabilities. This will allow for disambiguation at a later stage of the MT process, and will widen the possibility of word segmentation for other applications.

2 System Description

2.1 Segmentation Standard Our definition of words and our segmentation

conventions are based on available standards,

modified for MT purposes. The PRC standard (Liu

et al., 1993) was initially used. Sample differences

are listed as follows:

SYSTRAN NP 中华民族 中华 民族 中华人民共和国 中华 人民 共和国

31日 日 CD + M 一个 一排排 一 个 一 排 排 DI4 + CD 第一 第 一 Name 李 白 李 清照 李白 李清照

Table 1. Segmentation Divergences with the PRC Guidelines 2.2 Methodology

The SYSTRAN Chinese word segmentation module uses a rule-based approach and a large dictionary. The dictionary is derived from the

第1页

你可能喜欢

热门文档

相关文档

  • Chinese Word Segmentation

    SYSTRAN’s Chinese Word ... 4页 免费 Chinese word segmentatio... 4页 ... Chinese Word Segmentation Liqi Gao, Zhuoran Wang IR Lab, HIT 2003-11-8...

  • Chinese Word Segmentation at Peking University_免费...

    Chinese word segmentatio... 4页 免费 SYSTRAN’s Chinese Word ... 4页 ... Word segmentation is the first step in Chinese information processing, and ...

  • 计算机处理自然语言的探索

    including Chinese automatic word segmentation, Chinese text automatic tagging,...欧洲共同体 在美国乔治敦大学开发的机译系统 SYSTRAN 的基础上成功地进行了英、...

  • LTT-P-C3-CT3-Practice

    (http://www.chinesesoftware.com) HansVision-... SystranBox Systran在线翻译 (http://www.systran...To avoid this irrational segmentation of a ...

  • LTT-P-C3-CT3-Practice

    (http://www.chinesesoftware.com) HansVision-... SystranBox Systran在线翻译 (http://www.systran...To avoid this irrational segmentation of a ...

站点地图 | 文档上传 | 侵权投诉 | 手机版
本站所有资源均来自互联网,本站只负责收集和整理,均不承担任何法律责任,如有侵权等其它行为请联系我们.
文档下载 Copyright 2013 doc.guandang.net All Rights Reserved.  -  email