[revisit]

⚙️ The Complete uCalc Processing Pipeline

Understanding how uCalc processes a string from input to output is key to leveraging its full power. It's not a single, monolithic eval() function but a sophisticated, multi-stage pipeline where each step is configurable. This allows you to intercept and modify the process at various points, enabling the creation of custom syntax, domain-specific languages (DSLs), and complex transformation logic.

This topic provides a high-level overview of the entire end-to-end pipeline.

🌊 The Processing Flow

At its core, the uCalc engine follows a five-stage process. The diagram below illustrates the path an expression takes from a raw string to a final result.

[ Expression Parser Pipeline ]

Raw String ────► [ 1. Pre-Processing ] ────► Tokenizer ────► [ 2. Token Transformation ] ────► Token Stream ────► [ 3. Syntactic Analysis (Parsing) ] ────► Executable Plan (AST) ────► [ 4. Evaluation ] ────► Raw Result ────► [ 5. Post-Processing (Formatting) ] ────► Final String

[ Transformer Pipeline ]

Raw String ────► Tokenizer ────► Token Stream ────► [ 3. Syntactic Analysis (Matching) ] ────► Match Results ────► [ 4. Replacement ] ────► Final String

Note: The Transformer follows a simplified pipeline that branches off after tokenization.

Let's break down each stage for the expression parser.

1. Stage 1: Pre-Processing (ExpressionTransformer)

Before the parser even sees the individual tokens, the entire raw string is passed to the ExpressionTransformer. This is a meta-transformer that allows you to define high-level rewrite rules.

• Purpose: To create syntactic sugar or transpile multi-word syntax into a format the standard parser can understand.
• Example: Transforming a custom 100 USD to EUR syntax into a standard function call like ConvertCurrency(100, "USD", "EUR").

2. Stage 2: Lexical Analysis & Token Transformation

A. Tokenization (Tokens)

The string (either the original or the pre-processed version) is fed into the tokenizer, also known as a lexer. The lexer uses a set of regular-expression-based rules defined in a Tokens collection to break the string into a stream of meaningful units: numbers, identifiers, operators, string literals, etc.

• Purpose: To convert a flat stream of characters into a structured stream of categorized symbols.

B. Token Transformation (TokenTransformer)

During tokenization, if the lexer finds a token that has been specially marked with TokenType::TokenTransform, it pauses and sends that single token to the TokenTransformer.

• Purpose: To implement new low-level syntax for literals. This is a surgical, high-performance transformation.
• Example: Transforming a custom C-style hexadecimal literal 0xFF into an expression the engine understands, like BaseConvert("FF", 16).

3. Stage 3: Syntactic Analysis (Parse)

The final stream of tokens is fed into the syntactic analyzer, or parser. The parser uses the grammar rules (operator precedence, function signatures) to build a hierarchical structure called an Abstract Syntax Tree (AST). This tree represents the expression's logic and order of operations.

• Purpose: To create a machine-readable, executable plan from the token stream.
• Output: This stage is represented by the Parse method and results in a compiled Expression object.

4. Stage 4: Evaluation (Evaluate / Execute)

The AST contained within the Expression object is executed. The engine walks the tree, performing the specified calculations, calling functions, and retrieving variable values.

• Purpose: To compute the final value of the expression.
• Trigger: This stage is initiated by methods like Evaluate (for getting a value) or Execute (for side effects).

5. Stage 5: Post-Processing (Format)

If the evaluation was triggered by a method that returns a string (like EvalStr or EvaluateStr), the raw string result is passed through a final, optional formatting pipeline.

• Purpose: To apply consistent, declarative formatting to output values.
• Example: Automatically formatting all Double results as currency (e.g., 123.45 becomes $123.45).
• Configuration: This stage is controlled by rules defined with the Format method.

💡 Why uCalc? (Comparative Analysis)

Most evaluation tools are a "black box". You provide a string and get a result.

• eval() in Scripting Languages: A single, non-configurable step.
• Regular Expressions: Primarily focused on a single stage: pattern matching.

uCalc's key advantage is its transparent and configurable pipeline. It gives you hooks at every major stage of processing, transforming it from a simple evaluator into a powerful language construction toolkit. You can control high-level syntax, low-level literals, and final output formatting, all within a single, cohesive engine.