uCalc API Version: 5.7.0-preview.1 Released: 7/30/2026

Warning

uCalc API Preview Release Notice:This preview documentation describes the intended behavior of the API. It is not fully accurate or complete.The current preview build contains incomplete features, unoptimized performance, and is subject to breaking changes.Use of the preview version in your production code is not recommended.

Tokens

Product: 

Class: 

Remarks

uCalc uses a tokenizer (also known as a lexer) to split raw text into meaningful units called tokens. Unlike Regex which sees a stream of characters, uCalc sees a stream of words, numbers, and symbols.

Token Categories:Every token in uCalc belongs to a specific category (defined in TokenType Enum). These categories dictate parsing behavior:

  • Generic: The default category if a token is defined without a specific type.
  • Alphanumeric: Standard identifiers and words (e.g., Variable, Function_Name).
  • Literal: Represents data with an associated type (e.g., Numbers, Quoted Strings). These allow the parser to interpret values (integer vs float) immediately.
  • Bracket / BracketMatch: Defined in pairs (e.g., ( and )). If BracketSensitive is enabled (default), uCalc enforces nesting rules automatically.
  • StatementSep: (Statement Separator) Acts as a "Stop" condition for variable captures and delimits statements (e.g., ; or \n).
  • ArgSeparator: Separates items in a list (e.g., ,).
  • MemberAccess: Used for object navigation (e.g., . in Object.Property).
  • Whitespace: Ignored by default during parsing (e.g., Space, Tab).
  • TokenTransform: Special tokens that modify interpretation, such as String Interpolation markers or Hex/Binary prefixes (0x).
  • Reducible: Tokens capable of reduction logic (See Topic 802).
  • Eof: Matches the end of the input string.
  • Empty: Not a real token in text, but returned by API calls when querying a non-existent token definition.

Core Token Concepts:

  • Category: As listed above, this defines behavior.
  • Token Name: While categories group tokens, you can also match a specific token by its unique name (e.g., _token_newline or _token_myCustomSymbol). Names always start with _token_. For details, see Matching by token name.
  • Value: The actual text content (e.g., 123, Hello, "my text").

Why Tokenization Matters:

  • Precision: Matching {@Alpha} ensures you match Apple but not the App inside Apple.
  • Speed: The parser operates on integer token IDs rather than scanning string characters repeatedly.
  • Statefulness: Literals like strings ("my string") are handled as single units, preventing "partial matches" inside quotes.

Comparative Analysis: Why uCalc?

  • vs. Regex (Character-Based):
    • Boundary Hell: In Regex, matching a word requires \bword\b. In uCalc, {@Alpha} automatically respects boundaries defined by the language's syntax, making patterns safer by default.
    • Stateful Parsing: Regex doesn't know that 123 inside a quote "123" is a string. uCalc's tokenizer handles quotes first, allowing you to distinguish {@Number} from {@String} reliably without complex look-aheads.
    • Context Awareness: Regex struggles to distinguish a parenthesis used for grouping (a+b) from a literal parenthesis in a string "(a+b)". uCalc's Literal and Bracket categories handle this distinction natively.
  • vs. String.Split - Intelligence:
    • Split(' ') breaks on spaces inside quotes ("New York" -> New, York). uCalc's tokenizer respects quotes, keeping "New York" as a single token.
    • Split(',') breaks inside function calls fn(a,b). uCalc's logic respects the nesting depth of Brackets, splitting only at the top level (depending on the uCalc.Rule.BracketSensitive configuration)

Examples

Transformer: Matching by tokens vs match by character; also whitespace sensitivity

ID: 155

See: Add(string, TokenType, string, int, RegExGrammar, int), Description = [string], Description(string), Tokens = [Tokens], Tokens, Whitespace 
				
					using uCalcSoftware;

var uc = new uCalc();
// This examples shows the default match by
// token mode, as well as how to reconfigure
// it in order to do match by character
// along with a whitespace variation

var t = uc.NewTransformer();
var txt = "This is an island test, I said.";
t.FromTo("is", "<is>");

Console.WriteLine(t.Transform(txt));
Console.WriteLine("");

t.Tokens.Description = "Match by character";
t.Tokens.Add("."); // This overrides existing tokens
t.FromTo("is", "<is>");
Console.WriteLine(t.Tokens.Description);
Console.WriteLine(t.Transform(txt));
Console.WriteLine("");

// Note: whitespace sensitivity is off by default
// Whitespace token is re-introduced
// (after being overridden in the previous Token Add())
t.Tokens.Description = "By char + whitespace ignored";
t.Tokens.Add("[\\t\\v ]+", TokenType.Whitespace);
t.FromTo("is", "<{@Self}>");
Console.WriteLine(t.Tokens.Description);
Console.WriteLine(t.Transform(txt));
				
			
This <is> an island test, I said.

Match by character
Th<is> <is> an <is>land test, I said.

By char + whitespace ignored
Th<is> <is> an <is>land test, <I s>aid.
Explore 600+ other examples
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   // This examples shows the default match by
   // token mode, as well as how to reconfigure
   // it in order to do match by character
   // along with a whitespace variation

   auto t = uc.NewTransformer();
   auto txt = "This is an island test, I said.";
   t.FromTo("is", "<is>");

   cout << t.Transform(txt) << endl;
   cout << "" << endl;

   t.Tokens().Description("Match by character");
   t.Tokens().Add("."); // This overrides existing tokens
   t.FromTo("is", "<is>");
   cout << t.Tokens().Description() << endl;
   cout << t.Transform(txt) << endl;
   cout << "" << endl;

   // Note: whitespace sensitivity is off by default
   // Whitespace token is re-introduced
   // (after being overridden in the previous Token Add())
   t.Tokens().Description("By char + whitespace ignored");
   t.Tokens().Add("[\\t\\v ]+", TokenType::Whitespace);
   t.FromTo("is", "<{@Self}>");
   cout << t.Tokens().Description() << endl;
   cout << t.Transform(txt) << endl;
}
				
			
This <is> an island test, I said.

Match by character
Th<is> <is> an <is>land test, I said.

By char + whitespace ignored
Th<is> <is> an <is>land test, <I s>aid.
Explore 600+ other examples
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      '// This examples shows the default match by
      '// token mode, as well as how to reconfigure
      '// it in order to do match by character
      '// along with a whitespace variation
      
      Dim t = uc.NewTransformer()
      Dim txt = "This is an island test, I said."
      t.FromTo("is", "<is>")
      
      Console.WriteLine(t.Transform(txt))
      Console.WriteLine("")
      
      t.Tokens.Description = "Match by character"
      t.Tokens.Add(".") '// This overrides existing tokens
      t.FromTo("is", "<is>")
      Console.WriteLine(t.Tokens.Description)
      Console.WriteLine(t.Transform(txt))
      Console.WriteLine("")
      
      '// Note: whitespace sensitivity is off by default
      '// Whitespace token is re-introduced
      '// (after being overridden in the previous Token Add())
      t.Tokens.Description = "By char + whitespace ignored"
      t.Tokens.Add("[\\t\\v ]+", TokenType.Whitespace)
      t.FromTo("is", "<{@Self}>")
      Console.WriteLine(t.Tokens.Description)
      Console.WriteLine(t.Transform(txt))
   End Sub
End Module
				
			
This <is> an island test, I said.

Match by character
Th<is> <is> an <is>land test, I said.

By char + whitespace ignored
Th<is> <is> an <is>land test, <I s>aid.
Explore 600+ other examples
Matching a certain number of tokens

ID: 200

See: Tokens 
				
					using uCalcSoftware;

var uc = new uCalc();
var t = uc.NewTransformer();
t.FromTo("is {TokenCount:4}", "is <{TokenCount}>");
Console.WriteLine("This example captures 4 tokens");
Console.WriteLine(t.Transform("This is just a small token match test"));

// Quoted text counts as one token
Console.WriteLine(t.Transform("This is a 'really really' small test with quoted text"));
				
			
This example captures 4 tokens
This is <just a small token> match test
This is <a 'really really' small test> with quoted text
Explore 600+ other examples
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   auto t = uc.NewTransformer();
   t.FromTo("is {TokenCount:4}", "is <{TokenCount}>");
   cout << "This example captures 4 tokens" << endl;
   cout << t.Transform("This is just a small token match test") << endl;

   // Quoted text counts as one token
   cout << t.Transform("This is a 'really really' small test with quoted text") << endl;
}
				
			
This example captures 4 tokens
This is <just a small token> match test
This is <a 'really really' small test> with quoted text
Explore 600+ other examples
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      Dim t = uc.NewTransformer()
      t.FromTo("is {TokenCount:4}", "is <{TokenCount}>")
      Console.WriteLine("This example captures 4 tokens")
      Console.WriteLine(t.Transform("This is just a small token match test"))
      
      '// Quoted text counts as one token
      Console.WriteLine(t.Transform("This is a 'really really' small test with quoted text"))
   End Sub
End Module
				
			
This example captures 4 tokens
This is <just a small token> match test
This is <a 'really really' small test> with quoted text
Explore 600+ other examples
Displays the complete list of default token definitions, showing their type, internal name, and regex pattern.

ID: 70

See: Count = [int], ExpressionTokens = [Tokens], IndexOf, Introduction, Matching by token name ({@Token}), Name = [string], Regex = [string], At, Tokens 
				
					using uCalcSoftware;

var uc = new uCalc();
// Lists all tokens currently defined for the expression evaluator.

Console.WriteLine($"Token Count: {uc.ExpressionTokens.Count}");
Console.WriteLine("");
Console.WriteLine("Index  Type  Name: regex");
Console.WriteLine("========================");
var Tokens = uc.ExpressionTokens;
foreach(var token in Tokens) {
   Console.Write(Tokens.IndexOf(token));
   Console.WriteLine($"  {token.Description}  {token.Name}: {token.Regex}");
}

// Note that the expression evaluator token list has a few extra tokens,
// related to hex/bin/oct notation, string interpolation, and imaginary number
// notation, which are not found in the default Transformer token list.
				
			
Token Count: 30

Index  Type  Name: regex
========================
0  generic  _token_line: .*
1  generic  _token_catchall: .
2  generic  _token_catchall_utf8_other: [\xf0-\xf7][\x80-\xbf][\x80-\xbf][\x80-\xbf]|[\xe0-\xef][\x80-\xbf][\x80-\xbf]|[\xc0-\xdf][\x80-\xbf]
3  generic  _token_punctuation: (--|\.{3}|\xE2\x80\xA6|[!"#$%&'()*+,\-./:;<=>?@\[\\\]^_`{|}~]|\xE2\x80[\x90-\x95])
4  generic  _token_quotechar: ("){3}|"|'
5  generic  _token_quotechar_single: '
6  generic  _token_quotechar_double: "
7  generic  _token_quotechar_tripledouble: """
8  memberaccess  _token_memberaccess: \.
9  generic  _token_variableargs: \.\.\.
10  reducible  _token_reducible2: [-:|+/*^&=%@!`\\<>?#$~]+
11  bracket  _token_parenthesis: \(
12  bracketclose  _token_parenthesis_close: \)
13  bracket  _token_curlybrace: \{
14  bracketclose  _token_curlybrace_close: \}
15  bracket  _token_squarebracket: \[
16  bracketclose  _token_squarebracket_close: \]
17  argseparator  _token_argseparator: ,
18  statementseparator  _token_newline: (?:\r?\n)|\r
19  statementseparator  _token_semicolon: ;
20  literal  _token_string_singlequoted: '([^']*(?:''[^']*)*)'
21  literal  _token_string_doublequoted: "([^"]*(?:""[^"]*)*)"
22  literal  _token_string_tripledoublequoted: """([\s\S]*?)"""
23  whitespace  _token_whitespace: [\t\v ]+
24  reducible  _token_reducible: [-:|+/*^&=%@!`\\<>?]+
25  literal  _token_floatnumber: [0-9]*\.?[0-9]+([eE][+-]?[0-9]+)?
26  alphanumeric  _token_alphanumeric: [a-zA-Z_][a-zA-Z0-9_]*
27  literal  _token_imaginaryunit: #i
28  tokentransform  _token_binaryhexoctalnotation: #[bho][0-9A-F]+
29  tokentransform  _token_stringinterpolationquote: \$['"]
Explore 600+ other examples
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   // Lists all tokens currently defined for the expression evaluator.

   cout << "Token Count: " << uc.ExpressionTokens().Count() << endl;
   cout << "" << endl;
   cout << "Index  Type  Name: regex" << endl;
   cout << "========================" << endl;
   auto Tokens = uc.ExpressionTokens();
   for(auto token : Tokens) {
      cout << Tokens.IndexOf(token);
      cout << "  " << token.Description() << "  " << token.Name() << ": " << token.Regex() << endl;
   }

   // Note that the expression evaluator token list has a few extra tokens,
   // related to hex/bin/oct notation, string interpolation, and imaginary number
   // notation, which are not found in the default Transformer token list.
}
				
			
Token Count: 30

Index  Type  Name: regex
========================
0  generic  _token_line: .*
1  generic  _token_catchall: .
2  generic  _token_catchall_utf8_other: [\xf0-\xf7][\x80-\xbf][\x80-\xbf][\x80-\xbf]|[\xe0-\xef][\x80-\xbf][\x80-\xbf]|[\xc0-\xdf][\x80-\xbf]
3  generic  _token_punctuation: (--|\.{3}|\xE2\x80\xA6|[!"#$%&'()*+,\-./:;<=>?@\[\\\]^_`{|}~]|\xE2\x80[\x90-\x95])
4  generic  _token_quotechar: ("){3}|"|'
5  generic  _token_quotechar_single: '
6  generic  _token_quotechar_double: "
7  generic  _token_quotechar_tripledouble: """
8  memberaccess  _token_memberaccess: \.
9  generic  _token_variableargs: \.\.\.
10  reducible  _token_reducible2: [-:|+/*^&=%@!`\\<>?#$~]+
11  bracket  _token_parenthesis: \(
12  bracketclose  _token_parenthesis_close: \)
13  bracket  _token_curlybrace: \{
14  bracketclose  _token_curlybrace_close: \}
15  bracket  _token_squarebracket: \[
16  bracketclose  _token_squarebracket_close: \]
17  argseparator  _token_argseparator: ,
18  statementseparator  _token_newline: (?:\r?\n)|\r
19  statementseparator  _token_semicolon: ;
20  literal  _token_string_singlequoted: '([^']*(?:''[^']*)*)'
21  literal  _token_string_doublequoted: "([^"]*(?:""[^"]*)*)"
22  literal  _token_string_tripledoublequoted: """([\s\S]*?)"""
23  whitespace  _token_whitespace: [\t\v ]+
24  reducible  _token_reducible: [-:|+/*^&=%@!`\\<>?]+
25  literal  _token_floatnumber: [0-9]*\.?[0-9]+([eE][+-]?[0-9]+)?
26  alphanumeric  _token_alphanumeric: [a-zA-Z_][a-zA-Z0-9_]*
27  literal  _token_imaginaryunit: #i
28  tokentransform  _token_binaryhexoctalnotation: #[bho][0-9A-F]+
29  tokentransform  _token_stringinterpolationquote: \$['"]
Explore 600+ other examples
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      '// Lists all tokens currently defined for the expression evaluator.
      
      Console.WriteLine($"Token Count: {uc.ExpressionTokens.Count}")
      Console.WriteLine("")
      Console.WriteLine("Index  Type  Name: regex")
      Console.WriteLine("========================")
      Dim Tokens = uc.ExpressionTokens
      For Each token In Tokens
         Console.Write(Tokens.IndexOf(token))
         Console.WriteLine($"  {token.Description}  {token.Name}: {token.Regex}")
      Next
      
      '// Note that the expression evaluator token list has a few extra tokens,
      '// related to hex/bin/oct notation, string interpolation, and imaginary number
      '// notation, which are not found in the default Transformer token list.
   End Sub
End Module
				
			
Token Count: 30

Index  Type  Name: regex
========================
0  generic  _token_line: .*
1  generic  _token_catchall: .
2  generic  _token_catchall_utf8_other: [\xf0-\xf7][\x80-\xbf][\x80-\xbf][\x80-\xbf]|[\xe0-\xef][\x80-\xbf][\x80-\xbf]|[\xc0-\xdf][\x80-\xbf]
3  generic  _token_punctuation: (--|\.{3}|\xE2\x80\xA6|[!"#$%&'()*+,\-./:;<=>?@\[\\\]^_`{|}~]|\xE2\x80[\x90-\x95])
4  generic  _token_quotechar: ("){3}|"|'
5  generic  _token_quotechar_single: '
6  generic  _token_quotechar_double: "
7  generic  _token_quotechar_tripledouble: """
8  memberaccess  _token_memberaccess: \.
9  generic  _token_variableargs: \.\.\.
10  reducible  _token_reducible2: [-:|+/*^&=%@!`\\<>?#$~]+
11  bracket  _token_parenthesis: \(
12  bracketclose  _token_parenthesis_close: \)
13  bracket  _token_curlybrace: \{
14  bracketclose  _token_curlybrace_close: \}
15  bracket  _token_squarebracket: \[
16  bracketclose  _token_squarebracket_close: \]
17  argseparator  _token_argseparator: ,
18  statementseparator  _token_newline: (?:\r?\n)|\r
19  statementseparator  _token_semicolon: ;
20  literal  _token_string_singlequoted: '([^']*(?:''[^']*)*)'
21  literal  _token_string_doublequoted: "([^"]*(?:""[^"]*)*)"
22  literal  _token_string_tripledoublequoted: """([\s\S]*?)"""
23  whitespace  _token_whitespace: [\t\v ]+
24  reducible  _token_reducible: [-:|+/*^&=%@!`\\<>?]+
25  literal  _token_floatnumber: [0-9]*\.?[0-9]+([eE][+-]?[0-9]+)?
26  alphanumeric  _token_alphanumeric: [a-zA-Z_][a-zA-Z0-9_]*
27  literal  _token_imaginaryunit: #i
28  tokentransform  _token_binaryhexoctalnotation: #[bho][0-9A-F]+
29  tokentransform  _token_stringinterpolationquote: \$['"]
Explore 600+ other examples
Matching tokens by category.

ID: 245

See: Tokens 
				
					using uCalcSoftware;

var uc = new uCalc();
var t = uc.NewTransformer();
// Match a word (Alpha) followed by equals and a Number (Literal)
t.FromTo("{@Alpha} = {@Number}", "Assignment Detected");

Console.WriteLine(t.Transform("x = 10")); 
				
			
Assignment Detected
Explore 600+ other examples
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   auto t = uc.NewTransformer();
   // Match a word (Alpha) followed by equals and a Number (Literal)
   t.FromTo("{@Alpha} = {@Number}", "Assignment Detected");

   cout << t.Transform("x = 10") << endl;
}
				
			
Assignment Detected
Explore 600+ other examples
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      Dim t = uc.NewTransformer()
      '// Match a word (Alpha) followed by equals and a Number (Literal)
      t.FromTo("{@Alpha} = {@Number}", "Assignment Detected")
      
      Console.WriteLine(t.Transform("x = 10"))
   End Sub
End Module
				
			
Assignment Detected
Explore 600+ other examples
Customizing token definitions (e.g., treating hyphens as part of a word).

ID: 246

See: Tokens 
				
					using uCalcSoftware;

var uc = new uCalc();
var t = uc.NewTransformer();
// Now capture it using the {@Alpha} category
t.FromTo("{@Alpha:w}", "<{w}>");

Console.WriteLine("Before:");
Console.WriteLine(t.Transform("1. Start-Up 'big ideas' well-knwon."));

// Define a new token pattern for hyphenated words
// We assign it to the 'AlphaNumeric' category so it behaves like a word
t.Tokens.Add("[a-zA-Z-]+", TokenType.AlphaNumeric);
Console.WriteLine("After:");
Console.WriteLine(t.Transform("1. Start-Up 'big ideas' well-knwon."));
				
			
Before:
1. <Start>-<Up> 'big ideas' <well>-<knwon>.
After:
1. <Start-Up> 'big ideas' <well-knwon>.
Explore 600+ other examples
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   auto t = uc.NewTransformer();
   // Now capture it using the {@Alpha} category
   t.FromTo("{@Alpha:w}", "<{w}>");

   cout << "Before:" << endl;
   cout << t.Transform("1. Start-Up 'big ideas' well-knwon.") << endl;

   // Define a new token pattern for hyphenated words
   // We assign it to the 'AlphaNumeric' category so it behaves like a word
   t.Tokens().Add("[a-zA-Z-]+", TokenType::AlphaNumeric);
   cout << "After:" << endl;
   cout << t.Transform("1. Start-Up 'big ideas' well-knwon.") << endl;
}
				
			
Before:
1. <Start>-<Up> 'big ideas' <well>-<knwon>.
After:
1. <Start-Up> 'big ideas' <well-knwon>.
Explore 600+ other examples
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      Dim t = uc.NewTransformer()
      '// Now capture it using the {@Alpha} category
      t.FromTo("{@Alpha:w}", "<{w}>")
      
      Console.WriteLine("Before:")
      Console.WriteLine(t.Transform("1. Start-Up 'big ideas' well-knwon."))
      
      '// Define a new token pattern for hyphenated words
      '// We assign it to the 'AlphaNumeric' category so it behaves like a word
      t.Tokens.Add("[a-zA-Z-]+", TokenType.AlphaNumeric)
      Console.WriteLine("After:")
      Console.WriteLine(t.Transform("1. Start-Up 'big ideas' well-knwon."))
   End Sub
End Module
				
			
Before:
1. <Start>-<Up> 'big ideas' <well>-<knwon>.
After:
1. <Start-Up> 'big ideas' <well-knwon>.
Explore 600+ other examples
Handling Brackets and Literals.

ID: 247

See: Tokens 
				
					using uCalcSoftware;

var uc = new uCalc();
var t = uc.NewTransformer();
// Capture a function call.
// {@Alpha} matches the name.
// '(' and ')' are Bracket tokens that ensure the content is captured correctly.
t.FromTo("{@Alpha:func} ( {args} )", "Call: {func} with {args}");

Console.WriteLine(t.Transform("myfunc('Hello World', (x + y) * 2 )"));
				
			
Call: myfunc with 'Hello World', (x + y) * 2
Explore 600+ other examples
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   auto t = uc.NewTransformer();
   // Capture a function call.
   // {@Alpha} matches the name.
   // '(' and ')' are Bracket tokens that ensure the content is captured correctly.
   t.FromTo("{@Alpha:func} ( {args} )", "Call: {func} with {args}");

   cout << t.Transform("myfunc('Hello World', (x + y) * 2 )") << endl;
}
				
			
Call: myfunc with 'Hello World', (x + y) * 2
Explore 600+ other examples
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      Dim t = uc.NewTransformer()
      '// Capture a function call. 
      '// {@Alpha} matches the name. 
      '// '(' and ')' are Bracket tokens that ensure the content is captured correctly.
      t.FromTo("{@Alpha:func} ( {args} )", "Call: {func} with {args}")
      
      Console.WriteLine(t.Transform("myfunc('Hello World', (x + y) * 2 )"))
   End Sub
End Module
				
			
Call: myfunc with 'Hello World', (x + y) * 2
Explore 600+ other examples