Searched refs:tokenize (Results 1 – 6 of 6) sorted by relevance
153 lexer.tokenize()161 lexer.tokenize()
186 def tokenize (self): member in SrcLexer421 mclexer.tokenize()
125 mclexer.tokenize()
60 // use the standard J2SE delimiters to tokenize the string
38 // should we tokenize on "."?
174 This can be used to tokenize the string to recognize.
Completed in 27 milliseconds