S3 Storage Config

Installation

This guide assumes a user with access to your organization's AWS console, and appropriate permissions to:

• Create S3 buckets

• Create cross-account IAM roles for read-only and write-only access to the bucket.

It also assumes a user with portal access to Darwinium with manage deployments permission on the node being setup.

Prerequisite - Node created as custom S3

BYOS3 is a node level setting decided upon first creating node. Your account will need the corresponding Node read/create permissions to perform the following actions.

Verify Existing Node is custom BYOS3

1. Navigate to Your Name (top right) > Nodes

2. Settings on node of interest > Data Storage Tab

3. In Internal Data Storage - S3 Cloud Storage if:

   1. There are configuration options eg. bucket name, the node is set up to use Custom S3.

   2. If there is no configuration options and it mentions Default S3, the node was set up as using Darwinium bucket. You will need to create a new node if wanting BYOS3.

Create New Node with custom BYOS3

(Only do this on initial setup, or when certain you want to create a new environment)
After creating, proceed with Step1 below.

1. Navigate to Your Name (top right) > Nodes

2. + Add Node

3. Node Name: name you want to give the node/environment (eg. development, test, staging, production)

4. Type: Virtual

5. S3 Type: Custom

Internal Data Storage vs. Data Sink

• The BYOS3 configuration is under Internal Data Storage - S3 Cloud Storage. This bucket that will handle storing and retrieving of PII event data for interaction with Darwinium Portal

• Data Sink is separate configuration for when a staging S3 bucket is being used for Analytics data sinks. See: Snowflake, DataBricks .

Step 1 - Obtain External IDs for Read/Write Roles

1. In the Darwinium portal, navigate to Admin > Nodes

2. Select the node you wish to configure and click

3. A dialog will appear. Select the data storage tab, and scroll to Internal Data Storage - S3 cloud storage

4. Select Cross-Account IAM role for AWS credential type

5. A section will appear with generated external ids for your S3 read-only and write only IAM roles. Write these down

Step 2 - Create your Darwinium S3 Bucket

Using the AWS console, create a new S3 bucket. Your Bucket may be customized according to your infrastructure requirements. Darwinium recommends the following default settings:

• Region: us-east-2

• Object Ownership: Bucket owner enforced

• Block All Public Access: enabled

• Bucket Versioning: enabled

• Encryption: enabled with S3 managed keys (SSE-S3) - This is strongly recommended

Write down the ARN of your S3 bucket - this will be needed in future steps.

Step 3- Create IAM write-only role

To enable Darwinium to access your Customer Hosted S3 Bucket, create AWS IAM read-only and write-only roles which Darwinium components will assume to gain access to your bucket.

This AWS-recommended role assumption process will be access-controlled by an External Id and means you do not need to provide static credentials to your AWS environment.

Create S3 Write-Only Policy

This IAM policy grants the write-only permissions that Darwinium needs to your S3 bucket.

1. Go to IAM > Policies > Create Policy

2. Edit the policy template provided below as follows: replace

   S3BUCKETARN

   with the ARN of your Customer Hosted S3 Bucket. For example,

   arn:aws:s3:::example-bucket-name

   . Then, save the edited result as the JSON policy for your new IAM policy.

{
    "Statement": [
        {
            "Action": [
                "s3:PutObject"
            ],
            "Effect": "Allow",
            "Resource": "S3BUCKETARN/*"
        },
        {
            "Action": [
                "s3:GetBucketLocation"
            ],
            "Effect": "Allow",
            "Resource": "S3BUCKETARN"
        }
    ],
    "Version": "2012-10-17"
} 

Create S3 Write-Only Role

This IAM Role will be assumed by Darwinium to acquire write-only permissions to your S3 bucket.

1. Obtain your write-only account External ID from the Darwinium Portal (per step 1). This is a secret value that is unique to you and must be stored securely.

2. Go to IAM > Roles > Create Role

3. Edit the policy template provided below as follows: replace

   EXTERNALID

   with the ExternalId that Darwinium provided.

4. And replace

   DWNACCOUNTID

   With the AccountID Darwinium will supply upon install.

5. Use the edited result as the custom trust policy for your new IAM Role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::DWNACCOUNTID:root"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "EXTERNALID"
                },
                "ArnLike": {
                    "aws:PrincipalArn": "arn:aws:iam::DWNACCOUNTID:role/crossaccountassumers/*"
                }
            }
        }
    ]
}

4. Next, on the Add Permissions menu, select the S3 Write-Only policy that you created above.

5. Choose an appropriate name for the role, and note down the role’s ARN once created. It will look like

   arn:aws:iam::123456789:role/s3writeonlyrole

Step 4 - Create IAM read-only role

Create S3 Read-Only Policy

1. Go to IAM > Policies > Create Policy

2. Edit the policy template provided below as follows: replace

   S3BUCKETARN

   with the ARN of your Customer Hosted S3 Bucket. For example,

   arn:aws:s3:::example-bucket-name

   . Then, save the edited result as the JSON policy for your new IAM policy.

{
    "Statement": [
        {
            "Action": [
                "s3:GetObjectAcl",
                "s3:GetObject",
                "s3:GetObjectAttributes"
            ],
            "Effect": "Allow",
            "Resource": "S3BUCKETARN/*"
        },
        {
            "Action": [
                "s3:GetBucketLocation"
            ],
            "Effect": "Allow",
            "Resource": "S3BUCKETARN"
        }
    ],
    "Version": "2012-10-17"
}

Create S3 Read-Only Role

This IAM Role will be assumed by Darwinium to acquire read-only permissions to your S3 bucket.

1. Obtain your Read-only External ID from the Darwinium Portal in step 1. This is a secret value that is unique to you and must be stored securely.

2. Go to IAM > Roles > Create Role

3. Edit the policy template provided below as follows: replace

   EXTERNALID

   with the ExternalId obtained in Step1. And replace

   DWNACCOUNTID

   With the AccountID Darwinium will supply upon install.

   
   Use the edited result as the custom trust policy for your new IAM Role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::DWNACCOUNTID:root"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "EXTERNALID"
                },
                "ArnLike": {
                    "aws:PrincipalArn": "arn:aws:iam::DWNACCOUNTID:role/crossaccountassumers/*"
                }
            }
        }
    ]
}

4. Next, on the Add Permissions menu, select the S3 Read-Only policy that you created above.

5. Choose an appropriate name for the role, and note down the role’s ARN once created. It will look like

   arn:aws:iam::123456789:role/s3readonlyrole

(optional) Create S3 Bucket Policy if required; CloudFront only

This section is only required if you are:

1. installing Darwinium on CloudFront
2. AND you are hosting your own PII S3 Bucket in a different AWS account to that with your CloudFront and Lambda@Edge functions.

Darwinium Lambda@Edge functions use the Lambda execution role to gain permissions to directly write to your S3 bucket. If the Lambda execution role is in a different AWS account to that containing your S3 bucket, you must configure the bucket policy on the S3 bucket itself to allow the execution role access. This is in addition to attaching the S3 write IAM policy as described above to the Lambda execution role. This assumes that you have set ‘Bucket owner enforced’ and S3 managed keys (SSE-S3) on your S3 bucket as recommended above.

Note that you should still create the main S3 read and write roles and policies in the AWS account with the S3 bucket. Also, you will need to create a duplicate of the S3 write IAM policy in your AWS account with Cloudfront, so that you can attach it to your Lambda execution role.

1. Go to your bucket in S3 > Permissions > Edit Bucket Policy

2. Edit the policy template provided below as follows: replace S3BUCKETARN with the ARN of your Customer Hosted S3 Bucket, and EXECUTIONROLEARN with the ARN of your Lambda execution role. Then, save the edited result as the bucket policy.

{
    "Statement": [
        {
            "Action": [
                "s3:PutObject"
            ],
            "Effect": "Allow",
            "Resource": "S3BUCKETARN/*",
            "Principal": {
                "AWS": "EXECUTIONROLEARN"
            }
        },
        {
            "Action": [
                "s3:GetBucketLocation"
            ],
            "Effect": "Allow",
            "Resource": "S3BUCKETARN",
            "Principal": {
                "AWS": "EXECUTIONROLEARN"
            }
        }
    ],
    "Version": "2012-10-17"
} 

Please see this AWS documentation for more information: https://repost.aws/knowledge-center/cross-account-access-s3

Step 5 - Enter S3 details into the Darwinium portal

1. Go back to the Darwinium portal where you received your external IDs earlier

2. Enter details for:

   1. The S3 bucket AWS region

   2. The S3 bucket name

   3. Write-only Role ARN

   4. Read-only ARN

3. Click "verify" to confirm that S3 credentials are working correctly. Note that your node's configuration cannot be saved until verification passes.

4. Finally click Update. This will finalise attaching the S3 bucket. You will only be able to click Update after Verification passes

Troubleshooting

If verification fails, double check:

• Region and S3Bucket name (not ARN)  match the intended bucket

• The Write and Read role ARN were supplied the correct way round

• AWS policies attached to roles; they appear under the role permissions, and that they are the right ones

• The EXTERNALID within the AWS roles match the External ID in the Portal for write/read respectively

• The DWNACCOUNTID within the AWS roles match the Darwinium AccountID provided by the Darwinium services team

Verification

If verification passes: Bucket and roles configured correctly  

The verification tool will write and leave a record in your s3 bucket to verify configuration.

This will not be automatically cleaned up since the tool should not have the permissions to do so if configured correctly, but the record can be safely deleted afterwards manually if you wish to do so.

You can confirm an object was written into the S3 bucket as part of this Verification process.

This object is not used for anything other than this verification process so can be deleted manually at this stage (but won’t cause any problems if left there).

This is sufficient verification that the S3 bucket is ready for storing events.

Event Data

The S3 bucket will now automatically populate with event data when events are processed by Darwinium.

A new folder of Event data will be created and

1. written to with event data when events processed by Darwinium

2. read from when required during user operation of the Darwinium Portal, for retrieving and showing PII event data in the clear

<orgid>/
├── <nodeinstanceid>/
│   ├── encrypted/
│   │   └── EventDetails/
│   │       └── V2/
│   │           └── <eventidprefixes>/
│   │               | <encryptedevents>
│   │               └── ....
│   ├── hashed/
│   │   └── EventDetails/
│   │       └── V2/
│   │           └── <eventidprefixes>/
│   │               | <hashedevents>
│   │               └── ....
│   └── Metadata/
│       └── schema/
│           └── dwn_proto_schema_*.json

The mechanism which generates Darwinium events is controlled under Workflows > Journey files. Either calling API steps with the EventAPI or by initiating requests to an endpoint covered by an Edge (CDN) step.

See Deployment Confirmation