uCalc API Version: 5.7.0-preview.1 Released: 7/30/2026

Warning

uCalc API Preview Release Notice:This preview documentation describes the intended behavior of the API. It is not fully accurate or complete.The current preview build contains incomplete features, unoptimized performance, and is subject to breaking changes.Use of the preview version in your production code is not recommended.

Remove

Method

Product: 

Transformer Library

Class: 

Tokens

Dynamically removes a token definition from the collection, altering the parser's lexical rules at runtime.

Syntax

Remove(Item)

Parameters

tokenItem
Item
The `Item` object representing the token definition to remove. This is typically obtained via `ByName()` or `ByType()`.

Return

void

Remarks

🗑️ Dynamically Removing Token Definitions

The Remove method allows you to dynamically alter the behavior of the tokenizer at runtime by deleting a specific token definition. Once a token is removed, the parser will no longer recognize that pattern, which can fundamentally change how an input string is processed.

This method is a key part of uCalc's dynamic lexical analysis engine, enabling you to restrict or customize a language's syntax on the fly.

⚙️ How It Works

To remove a token, you must first get a handle to its Item object. This is typically done by querying the Tokens collection using methods like ByName() or ByType().

csharp
// Get the token for single-quoted strings
var singleQuoteToken = myTransformer.Tokens().ByName("_token_string_singlequoted");

// Remove it from the collection
myTransformer.Tokens().Remove(singleQuoteToken);

After removal, any text that previously matched the token's pattern will be tokenized by the next-highest-precedence token that matches it, often a more generic "catch-all" token.

🎯 Primary Use Cases

  • Restricting Syntax: Create a "safe" or simplified version of a language by removing tokens for features you want to disable. For example, you could remove the token for comments to disallow them.
  • Replacing a Built-in Token: The standard pattern for modifying a built-in token is to Remove the default version and then Add() a new one with a custom regex pattern. This allows you to extend the language, for instance, by allowing new characters in identifiers.
  • Resolving Ambiguity: If two token patterns are conflicting, removing one can resolve the ambiguity and ensure the desired token is matched.

💡 Why uCalc? (Comparative Analysis)

In traditional compiler development with tools like ANTLR or Flex/Bison, lexical rules are static and defined in a separate grammar file. Changing a token definition requires modifying that file, regenerating parser code, and recompiling the application. This is a compile-time, static process.

uCalc's token system is entirely dynamic and programmatic. Remove is a key part of this dynamism. It allows your application to:

  • Modify Syntax at Runtime: Your code can re-categorize or remove tokens on the fly, effectively switching between different language "dialects" without a restart.
  • Create Adaptable Parsers: Build tools that can be configured by the end-user to support different comment styles or literal formats by enabling or disabling specific tokens.

This runtime flexibility provides a significant advantage over static parsing systems. For clearing all tokens at once, see the Clear() method.

Examples

Practical: Demonstrates how removing the token for single-quoted strings causes the parser to treat the quote and its contents as individual generic tokens.

ID: 1046

See: Introduction, Remove 
				
					using uCalcSoftware;

var uc = new uCalc();
// This example removes the single-quoted string token.
var t = new uCalc.Transformer();
string txt = "This is a test, 'This is a test'";
t.FromTo("{token:1}", "<{@Self}>");

Console.WriteLine("--- Before Removing Token ---");
// Initially, 'This is a test' is treated as a single token.
Console.WriteLine(t.Transform(txt).Text);

// Now, find and remove the token definition for single-quoted strings.
var singleQuoteToken = t.Tokens.ByName("_token_string_singlequoted");
t.Tokens.Remove(singleQuoteToken);

// Re-run the transform. The text must be set again to be re-tokenized.
t.Text = txt;
Console.WriteLine("");
Console.WriteLine("--- After Removing Token ---");
// Now, the single quote is a generic token, as are the words inside it.
Console.WriteLine(t.Transform().Text);
				
			
--- Before Removing Token ---
<This> <is> <a> <test><,> <'This is a test'>

--- After Removing Token ---
<This> <is> <a> <test><,> <'><This> <is> <a> <test><'>
Explore 600+ other examples
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   // This example removes the single-quoted string token.
   uCalc::Transformer t;
   string txt = "This is a test, 'This is a test'";
   t.FromTo("{token:1}", "<{@Self}>");

   cout << "--- Before Removing Token ---" << endl;
   // Initially, 'This is a test' is treated as a single token.
   cout << t.Transform(txt).Text() << endl;

   // Now, find and remove the token definition for single-quoted strings.
   auto singleQuoteToken = t.Tokens().ByName("_token_string_singlequoted");
   t.Tokens().Remove(singleQuoteToken);

   // Re-run the transform. The text must be set again to be re-tokenized.
   t.Text(txt);
   cout << "" << endl;
   cout << "--- After Removing Token ---" << endl;
   // Now, the single quote is a generic token, as are the words inside it.
   cout << t.Transform().Text() << endl;
}
				
			
--- Before Removing Token ---
<This> <is> <a> <test><,> <'This is a test'>

--- After Removing Token ---
<This> <is> <a> <test><,> <'><This> <is> <a> <test><'>
Explore 600+ other examples
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      '// This example removes the single-quoted string token.
      Dim t As New uCalc.Transformer()
      Dim txt As String = "This is a test, 'This is a test'"
      t.FromTo("{token:1}", "<{@Self}>")
      
      Console.WriteLine("--- Before Removing Token ---")
      '// Initially, 'This is a test' is treated as a single token.
      Console.WriteLine(t.Transform(txt).Text)
      
      '// Now, find and remove the token definition for single-quoted strings.
      Dim singleQuoteToken = t.Tokens.ByName("_token_string_singlequoted")
      t.Tokens.Remove(singleQuoteToken)
      
      '// Re-run the transform. The text must be set again to be re-tokenized.
      t.Text = txt
      Console.WriteLine("")
      Console.WriteLine("--- After Removing Token ---")
      '// Now, the single quote is a generic token, as are the words inside it.
      Console.WriteLine(t.Transform().Text)
   End Sub
End Module
				
			
--- Before Removing Token ---
<This> <is> <a> <test><,> <'This is a test'>

--- After Removing Token ---
<This> <is> <a> <test><,> <'><This> <is> <a> <test><'>
Explore 600+ other examples
Internal Test: Removes the core alphanumeric token to verify that the tokenizer falls back to character-by-character tokenization.

ID: 1047

See: Remove 
				
					using uCalcSoftware;

var uc = new uCalc();
var t = new uCalc.Transformer();
string text = "word game";
t.FromTo("{token:1}", "[{@Self}]");

Console.WriteLine("--- Before Remove ---");
// By default, 'word' is a single alphanumeric token.
Console.WriteLine(t.Transform(text));

// Remove the alphanumeric token definition.
var alphaToken = t.Tokens.ByName("_token_alphanumeric");
t.Tokens.Remove(alphaToken);

// The transformer must be reset with the original text for the change to apply.
t.Text = text;

Console.WriteLine("");
Console.WriteLine("--- After Remove ---");
// Now, 'word' is no longer a single token. The fallback '.' token
// matches each character individually.
Console.WriteLine(t.Transform());
				
			
--- Before Remove ---
[word] [game]

--- After Remove ---
[w][o][r][d] [g][a][m][e]
Explore 600+ other examples
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   uCalc::Transformer t;
   string text = "word game";
   t.FromTo("{token:1}", "[{@Self}]");

   cout << "--- Before Remove ---" << endl;
   // By default, 'word' is a single alphanumeric token.
   cout << t.Transform(text) << endl;

   // Remove the alphanumeric token definition.
   auto alphaToken = t.Tokens().ByName("_token_alphanumeric");
   t.Tokens().Remove(alphaToken);

   // The transformer must be reset with the original text for the change to apply.
   t.Text(text);

   cout << "" << endl;
   cout << "--- After Remove ---" << endl;
   // Now, 'word' is no longer a single token. The fallback '.' token
   // matches each character individually.
   cout << t.Transform() << endl;
}
				
			
--- Before Remove ---
[word] [game]

--- After Remove ---
[w][o][r][d] [g][a][m][e]
Explore 600+ other examples
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      Dim t As New uCalc.Transformer()
      Dim text As String = "word game"
      t.FromTo("{token:1}", "[{@Self}]")
      
      Console.WriteLine("--- Before Remove ---")
      '// By default, 'word' is a single alphanumeric token.
      Console.WriteLine(t.Transform(text))
      
      '// Remove the alphanumeric token definition.
      Dim alphaToken = t.Tokens.ByName("_token_alphanumeric")
      t.Tokens.Remove(alphaToken)
      
      '// The transformer must be reset with the original text for the change to apply.
      t.Text = text
      
      Console.WriteLine("")
      Console.WriteLine("--- After Remove ---")
      '// Now, 'word' is no longer a single token. The fallback '.' token
      '// matches each character individually.
      Console.WriteLine(t.Transform())
   End Sub
End Module
				
			
--- Before Remove ---
[word] [game]

--- After Remove ---
[w][o][r][d] [g][a][m][e]
Explore 600+ other examples
Token Remove

ID: 169

See: Remove, ByName 
				
					using uCalcSoftware;

var uc = new uCalc();
// This example removes the single quote pattern from the list

var t = uc.NewTransformer();
var txt = "This is a test, 'This is a test'";
t.FromTo("{token:1}", "<{@Self}>");
Console.WriteLine(t.Transform(txt).Text);

// Now the ' character will no longer be special
// (see the Name() example for list of token names
t.Tokens.Remove(t.Tokens["_token_string_singlequoted"]);
Console.WriteLine(t.Transform(txt).Text);
				
			
<This> <is> <a> <test><,> <'This is a test'>
<This> <is> <a> <test><,> <'><This> <is> <a> <test><'>
Explore 600+ other examples
				
					#include <iostream>
#include "uCalc.h"

using namespace std;
using namespace uCalcSoftware;

int main() {
   uCalc uc;
   // This example removes the single quote pattern from the list

   auto t = uc.NewTransformer();
   auto txt = "This is a test, 'This is a test'";
   t.FromTo("{token:1}", "<{@Self}>");
   cout << t.Transform(txt).Text() << endl;

   // Now the ' character will no longer be special
   // (see the Name() example for list of token names
   t.Tokens().Remove(t.Tokens()["_token_string_singlequoted"]);
   cout << t.Transform(txt).Text() << endl;
}
				
			
<This> <is> <a> <test><,> <'This is a test'>
<This> <is> <a> <test><,> <'><This> <is> <a> <test><'>
Explore 600+ other examples
				
					Imports System
Imports uCalcSoftware
Public Module Program
   Public Sub Main()
      Dim uc As New uCalc()
      '// This example removes the single quote pattern from the list
      
      Dim t = uc.NewTransformer()
      Dim txt = "This is a test, 'This is a test'"
      t.FromTo("{token:1}", "<{@Self}>")
      Console.WriteLine(t.Transform(txt).Text)
      
      '// Now the ' character will no longer be special
      '// (see the Name() example for list of token names
      t.Tokens.Remove(t.Tokens("_token_string_singlequoted"))
      Console.WriteLine(t.Transform(txt).Text)
   End Sub
End Module
				
			
<This> <is> <a> <test><,> <'This is a test'>
<This> <is> <a> <test><,> <'><This> <is> <a> <test><'>
Explore 600+ other examples