1*b1cdbd2cSJim Jagielski# 2*b1cdbd2cSJim Jagielski# Copyright (C) 2002-2003, International Business Machines Corporation and others. 3*b1cdbd2cSJim Jagielski# All Rights Reserved. 4*b1cdbd2cSJim Jagielski# 5*b1cdbd2cSJim Jagielski# file: count_word.txt 6*b1cdbd2cSJim Jagielski# 7*b1cdbd2cSJim Jagielski# ICU Word Break Rules 8*b1cdbd2cSJim Jagielski# See Unicode Standard Annex #29. 9*b1cdbd2cSJim Jagielski# These rules are based on Version 4.0.0, dated 2003-04-17 10*b1cdbd2cSJim Jagielski# 11*b1cdbd2cSJim Jagielski 12*b1cdbd2cSJim Jagielski 13*b1cdbd2cSJim Jagielski 14*b1cdbd2cSJim Jagielski#################################################################################### 15*b1cdbd2cSJim Jagielski# 16*b1cdbd2cSJim Jagielski# Character class definitions from TR 29 17*b1cdbd2cSJim Jagielski# 18*b1cdbd2cSJim Jagielski#################################################################################### 19*b1cdbd2cSJim Jagielski$Katakana = [[:Script = KATAKANA:] [:name = KATAKANA-HIRAGANA PROLONGED SOUND MARK:] 20*b1cdbd2cSJim Jagielski [:name = HALFWIDTH KATAKANA-HIRAGANA PROLONGED SOUND MARK:] 21*b1cdbd2cSJim Jagielski [:name = HALFWIDTH KATAKANA VOICED SOUND MARK:] 22*b1cdbd2cSJim Jagielski [:name = HALFWIDTH KATAKANA SEMI-VOICED SOUND MARK:]]; 23*b1cdbd2cSJim Jagielski 24*b1cdbd2cSJim Jagielski 25*b1cdbd2cSJim Jagielski$dash = \u002d; 26*b1cdbd2cSJim Jagielski 27*b1cdbd2cSJim Jagielski$ALetter = [[:Alphabetic:] [:name= HEBREW PUNCTUATION GERESH:] 28*b1cdbd2cSJim Jagielski [:P:] [:S:] [:LineBreak = Numeric:] 29*b1cdbd2cSJim Jagielski - $dash 30*b1cdbd2cSJim Jagielski - $Katakana 31*b1cdbd2cSJim Jagielski - [:Script = Thai:] 32*b1cdbd2cSJim Jagielski - [:Script = Lao:] 33*b1cdbd2cSJim Jagielski - [:Script = Hiragana:]]; 34*b1cdbd2cSJim Jagielski 35*b1cdbd2cSJim Jagielski$TheZWSP = \u200b; 36*b1cdbd2cSJim Jagielski 37*b1cdbd2cSJim Jagielski# 38*b1cdbd2cSJim Jagielski# Character Class Definitions. 39*b1cdbd2cSJim Jagielski# The names are those from TR29. 40*b1cdbd2cSJim Jagielski# 41*b1cdbd2cSJim Jagielski$CR = \u000d; 42*b1cdbd2cSJim Jagielski$LF = \u000a; 43*b1cdbd2cSJim Jagielski$Control = [[[:Zl:] [:Zp:] [:Cc:] [:Cf:]] - $TheZWSP]; 44*b1cdbd2cSJim Jagielski$Extend = [[:Grapheme_Extend = TRUE:]]; 45*b1cdbd2cSJim Jagielski 46*b1cdbd2cSJim Jagielski 47*b1cdbd2cSJim Jagielski 48*b1cdbd2cSJim Jagielski 49*b1cdbd2cSJim Jagielski#################################################################################### 50*b1cdbd2cSJim Jagielski# 51*b1cdbd2cSJim Jagielski# Word Break Rules. Definitions and Rules specific to word break begin Here. 52*b1cdbd2cSJim Jagielski# 53*b1cdbd2cSJim Jagielski#################################################################################### 54*b1cdbd2cSJim Jagielski 55*b1cdbd2cSJim Jagielski$Format = [[:Cf:] - $TheZWSP]; 56*b1cdbd2cSJim Jagielski 57*b1cdbd2cSJim Jagielski 58*b1cdbd2cSJim Jagielski 59*b1cdbd2cSJim Jagielski# Rule 3: Treat a grapheme cluster as if it were a single character. 60*b1cdbd2cSJim Jagielski# Hangul Syllables are easier to deal with here than they are in Grapheme Clusters 61*b1cdbd2cSJim Jagielski# because we don't need to find the boundaries between adjacent syllables - 62*b1cdbd2cSJim Jagielski# they won't be word boundaries. 63*b1cdbd2cSJim Jagielski# 64*b1cdbd2cSJim Jagielski 65*b1cdbd2cSJim Jagielski 66*b1cdbd2cSJim Jagielski# 67*b1cdbd2cSJim Jagielski# "Extended" definitions. Grapheme Cluster + Format Chars, treated like the base char. 68*b1cdbd2cSJim Jagielski# 69*b1cdbd2cSJim Jagielski$ALetterEx = $ALetter $Extend*; 70*b1cdbd2cSJim Jagielski$KatakanaEx = $Katakana $Extend*; 71*b1cdbd2cSJim Jagielski$FormatEx = $Format $Extend*; 72*b1cdbd2cSJim Jagielski 73*b1cdbd2cSJim Jagielski# 74*b1cdbd2cSJim Jagielski# Numbers. Rules 8, 11, 12 form the TR. 75*b1cdbd2cSJim Jagielski# 76*b1cdbd2cSJim Jagielski 77*b1cdbd2cSJim Jagielski# 78*b1cdbd2cSJim Jagielski# Words. Alpha-numerics. Rule 5, 6, 7, 9, 10 79*b1cdbd2cSJim Jagielski# - must include at least one letter. 80*b1cdbd2cSJim Jagielski# - may include both letters and numbers. 81*b1cdbd2cSJim Jagielski# - may include MideLetter, MidNumber punctuation. 82*b1cdbd2cSJim Jagielski# 83*b1cdbd2cSJim Jagielski$LetterSequence = $ALetterEx ($FormatEx* $ALetterEx)*; # rules #6, #7 84*b1cdbd2cSJim Jagielski$LetterSequence {200}; 85*b1cdbd2cSJim Jagielski 86*b1cdbd2cSJim Jagielski$ALetterEx* $dash+ {200}; 87*b1cdbd2cSJim Jagielski$ALetterEx* ($dash $LetterSequence)+ $dash* {200}; 88*b1cdbd2cSJim Jagielski 89*b1cdbd2cSJim Jagielski# 90*b1cdbd2cSJim Jagielski# Do not break between Katakana. Rule #13. 91*b1cdbd2cSJim Jagielski# 92*b1cdbd2cSJim Jagielski$KatakanaEx ($FormatEx* $KatakanaEx)* {300}; 93*b1cdbd2cSJim Jagielski[:Hiragana:] $Extend* {300}; 94*b1cdbd2cSJim Jagielski 95*b1cdbd2cSJim Jagielski# 96*b1cdbd2cSJim Jagielski# Ideographic Characters. Stand by themselves as words. 97*b1cdbd2cSJim Jagielski# Separated from the "Everything Else" rule, below, only so that they 98*b1cdbd2cSJim Jagielski# can be tagged with a return value. TODO: is this what we want? 99*b1cdbd2cSJim Jagielski# 100*b1cdbd2cSJim Jagielski# [:IDEOGRAPHIC:] $Extend* {400}; 101*b1cdbd2cSJim Jagielski 102*b1cdbd2cSJim Jagielski# 103*b1cdbd2cSJim Jagielski# Everything Else, with no tag. 104*b1cdbd2cSJim Jagielski# Non-Control chars combine with $Extend (combining) chars. 105*b1cdbd2cSJim Jagielski# Controls are do not. 106*b1cdbd2cSJim Jagielski# 107*b1cdbd2cSJim Jagielski[^$Control [:Ideographic:]] $Extend*; 108*b1cdbd2cSJim Jagielski$CR $LF; 109*b1cdbd2cSJim Jagielski 110*b1cdbd2cSJim Jagielski# 111*b1cdbd2cSJim Jagielski# Reverse Rules. Back up over any of the chars that can group together. 112*b1cdbd2cSJim Jagielski# (Reverse rules do not need to be exact; they can back up too far, 113*b1cdbd2cSJim Jagielski# but must back up at least enough, and must stop on a boundary.) 114*b1cdbd2cSJim Jagielski# 115*b1cdbd2cSJim Jagielski 116*b1cdbd2cSJim Jagielski# NonStarters are the set of all characters that can appear at the 2nd - nth position of 117*b1cdbd2cSJim Jagielski# a word. (They may also be the first.) The reverse rule skips over these, until it 118*b1cdbd2cSJim Jagielski# reaches something that can only be the start (and probably only) char in a "word". 119*b1cdbd2cSJim Jagielski# A space or punctuation meets the test. 120*b1cdbd2cSJim Jagielski# 121*b1cdbd2cSJim Jagielski$NonStarters = [$ALetter $Katakana $Extend $Format]; 122*b1cdbd2cSJim Jagielski 123*b1cdbd2cSJim Jagielski#!.*; 124*b1cdbd2cSJim Jagielski! ($NonStarters* | \n \r) .; 125*b1cdbd2cSJim Jagielski 126