|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | |||||||||
java.lang.Objectweka.core.tokenizers.Tokenizer
weka.core.tokenizers.CharacterDelimitedTokenizer
weka.core.tokenizers.WordTokenizer
public class WordTokenizer
A simple tokenizer that is using the java.util.StringTokenizer class to tokenize the strings.
Valid options are:-delimiters <value> The delimiters to use (default ' \r\n\t.,;:'"()?!').
| Constructor Summary | |
|---|---|
WordTokenizer()
|
|
| Method Summary | |
|---|---|
java.lang.String |
getRevision()
Returns the revision string. |
java.lang.String |
globalInfo()
Returns a string describing the stemmer |
boolean |
hasMoreElements()
Tests if this enumeration contains more elements. |
static void |
main(java.lang.String[] args)
Runs the tokenizer with the given options and strings to tokenize. |
java.lang.Object |
nextElement()
Returns the next element of this enumeration if this enumeration object has at least one more element to provide. |
void |
tokenize(java.lang.String s)
Sets the string to tokenize. |
| Methods inherited from class weka.core.tokenizers.CharacterDelimitedTokenizer |
|---|
delimitersTipText, getDelimiters, getOptions, listOptions, setDelimiters, setOptions |
| Methods inherited from class weka.core.tokenizers.Tokenizer |
|---|
runTokenizer, tokenize |
| Methods inherited from class java.lang.Object |
|---|
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
| Constructor Detail |
|---|
public WordTokenizer()
| Method Detail |
|---|
public java.lang.String globalInfo()
globalInfo in class Tokenizerpublic boolean hasMoreElements()
hasMoreElements in interface java.util.EnumerationhasMoreElements in class Tokenizerpublic java.lang.Object nextElement()
nextElement in interface java.util.EnumerationnextElement in class Tokenizerpublic void tokenize(java.lang.String s)
tokenize in class Tokenizers - the string to tokenizepublic java.lang.String getRevision()
public static void main(java.lang.String[] args)
args - the commandline options and strings to tokenize
|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | |||||||||