When using S3 to store your event data, Darwinium provides python packages to read encoded entries inside a Jupyter notebook.
Overview
Darwinium stores each event in a custom protobuf format in the S3 bucket that is configured while enabling Darwinium. Darwinium offers a python package named dwn-s3-event-transformer to allow customers to transform the objects present in s3 into a pandas dataframe. This page describes the use and access of the dwn-s3-event-transformer.
To make use of this library, one will need two components:
- The library itself
- An access token to invoke the API server
- A jupyter notebook environment that can be used to query the requisite data.
Recommended Architecture deployment of the notebook
Since the python kernel is responsible for fetching the data from the S3 bucket, the location of the kernel decides the performance of the data fetches.
The recommended deployment is envisaged as follows:
Please note that the python kernel is located within the same VPC as where the bucket is loaded. This will optimize the data fetch speeds.
If instead the kernel is present on the local users laptop (say inside a vscode editor running locally on the laptop), the performance will be degraded significantly.
Important considerations before you use the package
Please note that the intention of the package is to do a quick exploratory analysis of the data in a notebook using the same query language as that of the portal.
The following caveats need to be kept in mind when using this package:
- For PII data, the library only supports decoding the anonymized data and does not support decrypting of values.
- The speed of fetching data from S3 depends on where the host/kernel process is running. If the notebook kernel is running from within a datacenter, the access speeds will be much faster as compared to access the data in a notebook running on a local laptop.
- It is not going to be as performant as a dedicated database engine or a data warehouse/lakehouse engines as all of the columns of the data satisfying the given filter expressions will be pulled into the hosts memory where the notebook is running (even if a subset of columns are chosen to be displayed in the notebook dataframe view)
- You might be rate-limited by Darwinium backend servers if conditions for “reasonable use” are violated.
Prerequisites for using the package
One needs to have the permissions to access the API server. To get the requisite access, please follow the instructions outlined in this page:
Setting up certificates and Access
Please ensure you have access to the following artefacts before proceeding to the next section.
- The MTLS certificate file
- The CA chain certificate file
- The private key file that is to be used in conjunction with the above files.
Setting up repository access
To access our python packages, you will require an API Token. A token can be obtained by going to the Darwinium Portal and clicking Preferences:
Once in the preferences screen, click SDK Access and then click Generate Token
This will provide a username (the same as the username used to access Darwinium) and token. Note these down as they will only appear once.
Installing the library
The library can be installed by using pip.
The version of the library used in the command below should match the release of darwinium release cycles.
The command to use to install the library is as follows:
pip install --extra-index-url=https://${DARWINIUM_PYPI_USERNAME}:${DARWINIUM_PYPI_PASSWORD}@packages.darwinium.com/artifactory/api/pypi/pypi-local/simple "dwn_s3_event_transformer==1.5.21"
In the above command, DARWINIUM_PYPI_USERNAME and DARWINIUM_PYPI_PASSWORD are environment variables representing the username and password/authentication token obtained for accessing the Darwinium hosted pypi repository.
Using the library
The next few sections describe how to use the library.
Overview of the data fetch steps
The following are the steps involved in fetching the data into your notebook.
- Configure information like the bucket name, api certs etc
- Obtain an AWS s3 session
- Pass the instances created in steps 1 and 2 above into an S3DataFrame object.
- Use the S3DataFrameObject to run a filter expression and get back a pandas dataframe.
- Optionally pass parameters like the list of columns you want back. The default is to get all columns by default.
Repeat step 4 to as many use cases you want to process.
Importing the requisite package
The following are the minimal and recommended import statements into your notebook/python script.
from dwn_s3_event_transformer.s3_df_exporter import S3DataFrameConfig, S3DataFrame
import boto3 # type: ignore
from dwn_s3_event_transformer.s3_util import S3Util
One needs to import S3Util if you want to use the utility method to establish an s3 session using an AWS profile. This utility method has the added advantage of automatically refreshing the tokens in a background “thread”.
Alternately, the user can provide their own aws session handle using alternate methods available specific to their environment.
Configuring the access
To start using the reader, one needs to create an instance of S3DataFrameConfig. This is a configuration object that can be used to connect the API server and fetch the S3 object references. The name of the pydantic python class is S3DataFrameConfig. (imported in the code snippet above).
Both the API server and the s3 bucket can be configured using the code snippet shown below.
def get_config_for_exporter() -> S3DataFrameConfig:
return S3DataFrameConfig( # type: ignore
s3_bucket_name="your-org-s3-bucket-name",
node_name="your-nodename.darwinidentity.com",
org_id="org-id",
node_id="d234a875-711e-4128-acae-ae9f5812da32",
mtls_CA_file_path="path_to_cachain.pem",
mtls_cert_file_path="path_to_cert_file_cert.pem",
mtls_key_file_path="path_to_private_key.key",
mtls_key_file_password="testpass", # THIS IS OPTIONAL
api_endpoint_port=9443,
aws_profile_name="my_aws_profile_name",
aws_region="us-east-1",
aws_session_autorefresh_ttl_seconds=3600, # default
s3_get_connection_pool_size=100, # default and recommended
)
Here are the details of the config keys used in the above code snippet:
s3_bucket_name: This represents just the bucket name (and no need to provide any prefixes or suffixes like a URN). This is a mandatory config.
node_name: This is the name of your node that you are connecting to pull the data from. This is a mandatory config.
org_id: This is the organization identifier as seen in the portal. This is a mandatory config.
node_id: Also known as the instance_id, this is a unique hex representation of the node. The value of this hex identifier can be obtained from the node settings screen in the admin screens of the portal. Please note that the node_name and node_id should match as defined in the portal. This is a mandatory config.
mtls_CA_file_path: This represents the file path for the CA cert chain file on the local file system. This is referenced in the prerequites section. This is a mandatory config.
mtls_cert_file_path: This represents the file path on the local file system for the certificate used to authenticate to the API server. This is referenced in the prerequisites section.This is a mandatory config.
mtls_key_file_path: This represents the file path on the local file system for the private key used in authentication to the API server.This is a mandatory config.
password: The password for the MTLS key file.
api_end_point_port: This is default port to communicate with the API server.This is a mandatory config.
aws_profile_name: This is the profile name to be used from aws config file. This is an optional config.
aws_region: This is the aws region name.
Obtaining an AWS S3 Session
One needs to create an instance of the boto3 based session object. It is ideal that this be an auto-refreshable session as notebook sessions can run for a long time.
One can choose to create an s3 session based on their enterprise setup for AWS authentication like SSO or direct key access. Alternatively there is a helper method available if one has an AWS profile definition and the related config already setup. The following code snippet can be used to create an AWS session if the enterprise setup is not too complicated.
Please note that the mechanism to fetch the session is entirely upto the user and below is an example approach.
def get_s3_session() -> boto3.session.Session: # type: ignore
return S3Util.get_refreshable_session( # type: ignore
profile_name="your-aws-profile-name",
region_name="us-east-1",
)
Create a S3DataFrame object
Next we need to create an instance of the S3DataFrame that will be used to execute the query expressions and return back the pandas data frame.
The following code snippet shows how to create an instance of this object.
exporter : S3DataFrame = S3DataFrame(get_config_for_exporter(), get_s3_session()) # type: ignore
In this example, we have named the instance as exporter.
Crafting a query expression
One can specify a query expression to restrict the data that is being fetched back. The query expression is a string and needs to have the same expression language as one uses in the portal.
An example query expression is given below:
filter="timestamp >= '2025-01-30 00:00:00' and timestamp <= '2025-02-01 00:00:00'"
Please note that value passed to the variable called filter is the same expression you would use in the portal UI.
Making the query
The following code snippet shows how to specify the query filter
pandas_frame = await exporter.get_events_dataframe_async(filter="timestamp >= '2025-01-30 00:00:00' and timestamp <= '2025-02-01 00:00:00'", list_of_columns=None) # type: ignore
Note that the method also accepts an optional parameter called list_of_columns to restrict the result data frame to be only those columns. Note that decreasing the number of columns to return does not necessarily means a performance uplift as the filtering of the columns being displayed happens on the client side (i.e. inside the python kernel side).
Note on feature columns
All the darwinium features and the customer defined features are automatically expanded into their own columns. The name of the feature appears in between a [] prepended with the column name.
Listing all known columns
There is a utility method that returns all known columns. Example code snippet to get all known columns is as follows:
exporter.get_list_of_known_columns()