Machine Learning Execution

Darwinium supports adding ML models in PMML format (Predictive Model Markup Language) directly to the Workflows of any step. PMML format permits a wide range of model types including Random Forests, Support Vector Machines and Gradient Boosting models.

The PMML is converted and compiled into Web Assembly upon journey build, to perform fast real-time execution.

Many typical models are supported via this approach including:

Regressions
Support Vector Machines
Random Forests
Gradient Boosting (including eg. LGBM)
Neural networks (when smaller, NOT deep learning)

High level steps

Have a feature.yaml file set to execute on a step
1. By specifying it on step Workflows in a Darwinium .journey.yaml file
Allow that step to trigger events and features
1. Events triggered via Darwinium API calls or Edge worker request triggers
2. Typically 2weeks+ of live or batch data, and/or well over 1,000 events
Extract the resulting dataset of Darwinium event data
1. Important column: outcome['CHAMPION'].features.general
2. Best source will be from your Darwinium analytics data sink
3. If not configured, Darwinium Portal does support a limited export to csv directly from Investigations view (right click table > Export)
Ensure a target (prediction objective) column is present in the dataset
1. That may be joined from your own outcome/feedback data
2. or formed with a Darwinium virtual column
Define a PMML pipeline
1. Preprocessing operations such as scalers
2. Choice of classifier
Fit model with target column as objective
Export trained model to PMML format
Make Darwinium execute model in real-time
1. Upload .pmml file into your Darwinium Portal Workflows instance
2. Add it to the Workflow of the step of your journey file (as Execute Model)
3. Specify output attribute; most default to probability(1)
4. Make sure to include the features as dependency
Now model will evaluate on new events and output score
1. Score will be in outcome[CHAMPION].models.score['step_name.model_name'] .
2. Can be referenced in search and rules with shorthand: modelScore('model_name')

Note: Tools to produce

Requiring only that the model be defined as a PMML file is so that execution is independent of where and how the model is trained.

The example shown here will use Python and the following packages, but countless other routes and languages are possible to output a .pmml file too.

pandas 
scikit-learn==1.7.2
sklearn2pmml 
pypmml

sklearn2pmml: Converts a PMMLPipeline model (scikit learn pipeline) to PMML format. https://github.com/jpmml/sklearn2pmml
pypmml: Read PMML file into model for inference. https://pypi.org/project/pypmml/

Basic Code Example

import pandas as pd
import json
from sklearn2pmml import sklearn2pmml, PMMLPipeline
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler
from pypmml import Model

# READ DATA
# Data source could be DataBricks/Snowflake/S3 analytics sink
# But for simple demonstration, here we assume a CSV export from Portal
df = pd.read_csv("data/export.csv", header=[0, 1])
df.columns = ['.'.join(col).strip() for col in df.columns.values]

# MAKE TARGET AND FEATURES DATAFRAME

# Assumes a binary (virtual) column called target
y = df['virtual.target']

# Features map is exported from Portal as string
# So convert back to JSON, then normalize so each feature is a column
df["outcome['CHAMPION'].features.general"] = (
    '{"' +
    df["outcome['CHAMPION'].features.general"]
        .dropna()
        .str.replace(':', '":', regex=False)
        .str.replace(', ', ', "', regex=False)
    + '}'
)

X = pd.json_normalize(
    df["outcome['CHAMPION'].features.general"]
    .map(json.loads)
).astype(float)

# MAKE PIPELINE
# PMMLPipeline is required for sklearn2pmml
# Tailor to your needs by adding preprocessing steps or changing classifier
pipeline = PMMLPipeline([
    ("scaler", StandardScaler()),
    ("classifier", RandomForestClassifier(
        n_estimators=5, max_depth=2, random_state=42
    ))
])

# TRAIN MODEL
pipeline.fit(X, y)
pipeline.target_field = "target"

# EXPORT TO PMML
sklearn2pmml(
    pipeline,
    "output.pmml",
    with_repr=True
)

# READ FROM PMML
# Read back the PMML model for inference
model = Model.load("output.pmml")
X_new = pd.DataFrame([{
    "velocity_customer": 3.0,
    "velocity_custom1": 3.0,
    "sum_payment_amount": 150.0,
    "max_payment_amount": 50.0
}])

# Run inference
predictions = model.predict(X_new)
print(predictions)

export.csv
example.feature.yaml

Specify it to run on a Darwinium step

Now the PMML model can be added directly to Darwinium step to execute in real-time.

Navigate to Workflows Tab on left hand side in Darwinium Portal instance. If can't see Workflows ensure:
1. you have edit role
2. the node you are in is not using another's repository.
Add the PMML file to the repository (can just drag, drop).
Open the journey.yaml file, and open step where model is to be run
In Workflows tab of Step, add a new Execute Code/Model/Rules, and specify path as the .pmml file
Specify target; likely probability(1). That is what will output to score.
Make sure the feature generation is a dependency of this model; makes sure they will generate first and be available to the model.
Commit and deploy for the model to for model to score events

Using the score