|
|||||||||||
PREV NEXT | FRAMES NO FRAMES |
Packages that use TextTokenizer | |
de.fu_berlin.ties.classify.feature | This package contains classes for working with features, feature vectors and feature transformers. |
de.fu_berlin.ties.text | This package contains utility classes for working with texts. |
de.fu_berlin.ties.xml | This package contains utility classes for working with XML documents and related data. |
de.fu_berlin.ties.xml.dom | This package contains utility classes for working with DOM-like XML representations, focussing especially on dom4j. |
Uses of TextTokenizer in de.fu_berlin.ties.classify.feature |
Methods in de.fu_berlin.ties.classify.feature with parameters of type TextTokenizer | |
void |
FeatureVector.addAllTokens(CharSequence text,
TextTokenizer tokenizer)
Tokenizes a text, creating and adding a feature for each token. |
Uses of TextTokenizer in de.fu_berlin.ties.text |
Methods in de.fu_berlin.ties.text that return TextTokenizer | |
static TextTokenizer |
TokenizerFactory.createAlnumTokenizer(CharSequence text)
Static factory method to create an instance for tokenizing alphanumeric and symbol sequences and puntuation. |
static TextTokenizer |
TokenizerFactory.createCategoryTokenizer(CharSequence text)
Static factory method to create an instance for tokenizing according to Unicode categories. |
static TextTokenizer |
TokenizerFactory.createThoroughTokenizer(CharSequence text)
Static factory method to create an instance that uses the "thorough" patterns listed below. |
TextTokenizer |
TokenizerFactory.createTokenizer(CharSequence text)
Factory method to create an instance from the configured token and whitespace patterns. |
Uses of TextTokenizer in de.fu_berlin.ties.xml |
Methods in de.fu_berlin.ties.xml that return TextTokenizer | |
static TextTokenizer |
XMLTokenizerFactory.createXMLTokenizer(CharSequence text,
boolean ensureWhitespace)
Factory method to create an instance for parsing files in XML syntax. |
Uses of TextTokenizer in de.fu_berlin.ties.xml.dom |
Methods in de.fu_berlin.ties.xml.dom with parameters of type TextTokenizer | |
protected void |
TokenWalker.processCollectedText(Element element,
CharSequence collectedText,
TokenCounter tokenCounter,
TextTokenizer tokenizer,
ContextMap context)
Helper method that tokenizes the collected textual contents of an element and delegates to the token processor for each of them. |
protected void |
TokenWalker.walk(Element element,
TokenCounter tokenCounter,
TextTokenizer tokenizer,
ContextMap context)
Walks through the contents of a node, tokenizing textual contents and recursing through nested elements. |
|
|||||||||||
PREV NEXT | FRAMES NO FRAMES |