Vicon Shogun banner


Returns elements of a string as "tokens".

Tokenizing a string is the process of breaking apart a string into components that are meaningful to you. For example, if whole words are the components we are concerned with in a sentence, then we would expect white-space characters to separate, or delimit, the words of the sentence. Thus, in this scenario, the sentence "Here is some text" would be tokenized into "Here", "is", "some", and "text". To tokenize a string, call strTokArray with the string to be tokenized.

The default delimiters are the white-space characters (i.e. space, tab, and new line characters), but any delimiters may be used.

To match the full delimiter string (which will almost never be done with the default delimiters), specify the -matchAll flag.

This command is similar to the strTok command, except that strTok requires multiple calls to tokenize the string, returning one token of the string with each call. Use strTok if you need to change your delimiter in the middle of tokenizing.

Functional area


Command syntax


strTokArray "string" [-delims string] [-matchAll]


strToTokenizestringyesThe string to be tokenized.


NameFlag argumentsArgument typeExclusive toComments
delims1stringOverrides the default delimiters with user provided delimiters. The default delimiters are space, tab, and new line (any white space).
matchAll0Specifies that all the delimiter characters must be matched. The default is to match any of the delimiter characters.

Return value

string array

Returns all of the "tokens" found within "string".


// Demonstrate string tokenization.
string $str = "This is some text";
string $filepath = "C:/Program Files/ShogunPost#.#/SampleFiles/Day1/Test.hdf";
string $tokens[];

// Tokenize the first string.
$tokens = `strTokArray $str`;
print $tokens;

// Now tokenize the file path. Tokenize it by
// folders, so use / as a delimiter
$tokens = `strTokArray $filepath -delims "/"`;
print $tokens;

Additional information

Related commands