Cross-Platform C++

ot::util
class StringTokenizer

#include "ot/util/StringTokenizer.h"

Splits a String into a sequence of delimited tokens. A StringTokenizer is constructed using a String together with a set of delimiter characters contained in another String. The StringTokenizer will split the controlled String into a sequence of tokens, each token separated by one or more characters from the set of delimiter characters.

The StringTokenizer can behave in one of three ways, depending on the values of the bReturnDelims and bReturnContiguousDelims constructor flags:

A token is thus either a sequence of consecutive characters that are not delimiters, a single delimiter character or a sequence of characters that are delimiters.

The following example splits a string containing words separated by space characters into a number of word tokens. Delimiters are not returned in this case:

    StringTokenizer tokenizer(OT_T("this is a test"));
    while (tokenizer.hasMoreTokens())
    {
        Console::cout() << tokenizer.nextToken() << endl;
    }

In common with all String handling in OpenTop, the controlled String and the set of delimiter characters may contain Unicode characters that have been encoded into a multi-character sequence. The StringTokenizer class correctly treats multi-character sequences as single Unicode characters for the purposes of comparison between characters in the controlled String and the set of delimiter characters.




Constructor/Destructor Summary
StringTokenizer(const String& str)
         Constructs a StringTokenizer with the controlled String str using a default white-space string as the delimiter.
StringTokenizer(const String& str, const String& delim, bool bReturnDelims, bool bReturnContiguousDelims)
         Constructs a StringTokenizer with the controlled String str using all the Unicode characters from delim as delimiters.

Method Summary
 bool hasMoreTokens() const
         Tests to see if more tokens exist in the controlled String.
 String nextToken()
         Returns the next token from the controlled String.
 String peekNextToken() const
         Returns the next token without advancing the position of the StringTokenizer.
 std::vector< String > toVector()
         Returns the remaining tokens as elements of a std::vector<String>.

Constructor/Destructor Detail

StringTokenizer

 StringTokenizer(const String& str)
Constructs a StringTokenizer with the controlled String str using a default white-space string as the delimiter. The default delimiter contains the white-space characters making white-space according to the description of UnicodeCharacterType::IsSpace().

Parameters:
str - The String to tokenize

StringTokenizer

 StringTokenizer(const String& str,
                 const String& delim,
                 bool bReturnDelims,
                 bool bReturnContiguousDelims)
Constructs a StringTokenizer with the controlled String str using all the Unicode characters from delim as delimiters.

Parameters:
str - The String to tokenize
delim - A String containing a set of Unicode characters to be used as token delimiters
bReturnDelims - when set to true, calls to nextToken() will return the delimiter characters as tokens in their own right. See the bReturnContiguousDelims parameter for how the delimiter tokens can be further controlled.
bReturnContiguousDelims - controls how delimiter characters are grouped into tokens. When set to true calls to nextToken() will return a maximal sequence of delimiter characters as a single token. When set to false, each delimeter character will be returned as an individual token. This parameter has no effect when the bReturnDelims parameter is set to false.

Method Detail

hasMoreTokens

bool hasMoreTokens() const
Tests to see if more tokens exist in the controlled String.

Returns:
true if a call to nextToken() will yield a non-empty String; false otherwise

nextToken

String nextToken()
Returns the next token from the controlled String. See the class description for details about how delimiters may be returned as tokens.

When all the tokens have been exhausted, an empty String is returned.

Returns:
A String containing the next token or delimiter. An empty string indicates that the end of the String has been reached.
See also:
peekNextToken()

peekNextToken

String peekNextToken() const
Returns the next token without advancing the position of the StringTokenizer. This method returns the same value as nextToken() but without advancing the position past the token.

Returns:
A String containing the next token or delimiter. An empty string indicates the end of the String has been reached.
See also:
nextToken.

toVector

std::vector< StringtoVector()
Returns the remaining tokens as elements of a std::vector<String>. On return from this function all the tokens from the controlled String will have been processed.

This function is implemented as if nextToken() is called until it returns an empty String, adding each returned token to the vector. Therefore, if the StringTokenizer is set to return delimiters as tokens, the returned vector will contain the delimiters also.

Returns:
a vector containing the remaining tokens from the controlled String.


Cross-Platform C++

Found a bug or missing feature? Please email us at support@elcel.com

Copyright © 2000-2003 ElCel Technology   Trademark Acknowledgements