Transformer: Matching by tokens vs match by character; also whitespace sensitivity
ID: 155
See: Add(string, TokenType, string, int, RegExGrammar, int), Description = [string], Description(string), Tokens = [Tokens], Tokens, Whitespace
using uCalcSoftware;
var uc = new uCalc();
// This examples shows the default match by
// token mode, as well as how to reconfigure
// it in order to do match by character
// along with a whitespace variation
var t = uc.NewTransformer();
var txt = "This is an island test, I said.";
t.FromTo("is", "<is>");
Console.WriteLine(t.Transform(txt));
Console.WriteLine("");
t.Tokens.Description = "Match by character";
t.Tokens.Add("."); // This overrides existing tokens
t.FromTo("is", "<is>");
Console.WriteLine(t.Tokens.Description);
Console.WriteLine(t.Transform(txt));
Console.WriteLine("");
// Note: whitespace sensitivity is off by default
// Whitespace token is re-introduced
// (after being overridden in the previous Token Add())
t.Tokens.Description = "By char + whitespace ignored";
t.Tokens.Add("[\\t\\v ]+", TokenType.Whitespace);
t.FromTo("is", "<{@Self}>");
Console.WriteLine(t.Tokens.Description);
Console.WriteLine(t.Transform(txt));
This <is> an island test, I said.
Match by character
Th<is> <is> an <is>land test, I said.
By char + whitespace ignored
Th<is> <is> an <is>land test, <I s>aid.
#include <iostream>
#include "uCalc.h"
using namespace std;
using namespace uCalcSoftware;
int main() {
uCalc uc;
// This examples shows the default match by
// token mode, as well as how to reconfigure
// it in order to do match by character
// along with a whitespace variation
auto t = uc.NewTransformer();
auto txt = "This is an island test, I said.";
t.FromTo("is", "<is>");
cout << t.Transform(txt) << endl;
cout << "" << endl;
t.Tokens().Description("Match by character");
t.Tokens().Add("."); // This overrides existing tokens
t.FromTo("is", "<is>");
cout << t.Tokens().Description() << endl;
cout << t.Transform(txt) << endl;
cout << "" << endl;
// Note: whitespace sensitivity is off by default
// Whitespace token is re-introduced
// (after being overridden in the previous Token Add())
t.Tokens().Description("By char + whitespace ignored");
t.Tokens().Add("[\\t\\v ]+", TokenType::Whitespace);
t.FromTo("is", "<{@Self}>");
cout << t.Tokens().Description() << endl;
cout << t.Transform(txt) << endl;
}
This <is> an island test, I said.
Match by character
Th<is> <is> an <is>land test, I said.
By char + whitespace ignored
Th<is> <is> an <is>land test, <I s>aid.
Imports System
Imports uCalcSoftware
Public Module Program
Public Sub Main()
Dim uc As New uCalc()
'// This examples shows the default match by
'// token mode, as well as how to reconfigure
'// it in order to do match by character
'// along with a whitespace variation
Dim t = uc.NewTransformer()
Dim txt = "This is an island test, I said."
t.FromTo("is", "<is>")
Console.WriteLine(t.Transform(txt))
Console.WriteLine("")
t.Tokens.Description = "Match by character"
t.Tokens.Add(".") '// This overrides existing tokens
t.FromTo("is", "<is>")
Console.WriteLine(t.Tokens.Description)
Console.WriteLine(t.Transform(txt))
Console.WriteLine("")
'// Note: whitespace sensitivity is off by default
'// Whitespace token is re-introduced
'// (after being overridden in the previous Token Add())
t.Tokens.Description = "By char + whitespace ignored"
t.Tokens.Add("[\\t\\v ]+", TokenType.Whitespace)
t.FromTo("is", "<{@Self}>")
Console.WriteLine(t.Tokens.Description)
Console.WriteLine(t.Transform(txt))
End Sub
End Module
This <is> an island test, I said.
Match by character
Th<is> <is> an <is>land test, I said.
By char + whitespace ignored
Th<is> <is> an <is>land test, <I s>aid.