Supported Characters and Normalization

When we receive an attribute value, we put it into a standard form before we use it to link events and build features. This means ABC123, abc123 and ␣abc123␣ count as the same customer token, so a customer's history isn't split across the small variations different apps and channels send.

Base standard

We use Normalization Form KD (NFKD), "Compatibility Decomposition", from Unicode Standard Annex #15: Unicode Normalization Forms. The standard describes the goal:

"When implementations keep strings in a normalized form, they can be assured that equivalent strings have a unique binary representation."

NFKD is the broadest of the four Unicode normalization forms. It splits accented letters into a base letter followed by its accent, and it folds look-alike characters into their plain equivalents. For example, full-width Ａ becomes A and the circled digit ① becomes 1.

Our process adds steps before and after NFKD, so the result is not pure NFKD. The differences are listed under Differences from the standard below.

What happens to a text value in general

Text attributes such as the customer token, username and name fields go through these steps, in order:

1. Remove every character that isn't a letter, a number, ASCII punctuation (the punctuation on a US keyboard) or a plain space. This drops tabs, line breaks, emoji, symbols such as € and any punctuation outside ASCII.
2. Encode HTML characters so the value is safe to display: ' " < > & and --.
3. Apply NFKD.
4. Convert to lowercase.
5. Tidy spaces. Runs of spaces become one space, and leading and trailing spaces are removed.

Variations by attribute

Examples

Differences from the standard

For comparison we use Unicode's own definition of a caseless, compatibility-normalised match (The Unicode Standard, section 3.13, definition D146):

"A string X is a compatibility caseless match for a string Y if and only if: NFKD(toCasefold(NFKD(toCasefold(NFD(X))))) = NFKD(toCasefold(NFKD(toCasefold(NFD(Y)))))"

Our normalization differs this in three ways.

1. We remove characters before applying NFKD. The standard never removes characters. Because our removal step runs first, it also sees characters before NFKD has had a chance to rewrite them:

• Accents sent as a separate character are removed. é can be sent as one character or as e followed by a combining accent. The standard treats these as the same text. We keep the accent in the first case and remove it in the second, so José can become either josé or jose depending on how the app or keyboard encoded it. Japanese voiced marks behave the same way (ガ or カ).
• Thai tone marks are removed. Unicode doesn't classify tone marks (่ ้ ๊ ๋), maitaikhu (็) or thanthakhat (์) as letters, so step 1 removes them. Vowel marks count as letters and are kept. No Unicode normalization form removes either.
• Characters NFKD would have converted are removed instead. The standard turns these into ordinary characters that our process would keep. We remove them first.

2. We lowercase, where the standard case-folds. Case folding is Unicode's method for matching text regardless of case. Lowercasing gives the same result for most text, with these exceptions:

3. We add steps the standard doesn't have. HTML encoding (step 2), space tidying (step 5), ignoring dots in email addresses, and the rejections below are Darwinium rules, not Unicode ones.

Some results are standard NFKD and can still be surprising:

• An accented letter is stored as the letter followed by the accent.
• Korean syllables are stored as their component letters (한 becomes ᄒ + ᅡ + ᆫ). They still display as 한, so the change only shows up when you compare the bytes.
• Thai ำ is split into ํ + า.

If you compare values exported from Darwinium with your own records, apply NFKD to your side first.

Values we reject

If a value is empty after step 1 (for example only spaces or only emoji), or looks like a payment card number, we drop that attribute. The event is still processed, the API responds with HTTP 422, and the response lists the attribute under invalid_input_attributes.

The card number check applies to values of 12 to 19 digits that start like a real card number and pass the Luhn check. If your customer tokens are plain numbers, add the attribute to confirmed_non_card_attributes in your journey to switch the check off for it.

Values sent through the API and values extracted by a journey are normalised the same way. Custom fields are the exception noted above.