What is Introspective Ingestion?

Introspective Ingestion is a process carried out by the Model-Prime Analytics Platform for each newly ingested robolog. The process gathers data from all channels of a robolog and stores it in a data lake. The channel data is made accessible as queryable tables. Along with the channel data, AP also saves a copy of the metadata linked to the robologs in the data lake.

Prerequisite

Before setting up the Analytics Platform ingestion, you must complete the Model-Prime ingestion setup.

Introspective Ingestion setup for customers using AWS

You must grant read access to the AP’s execution IAM role for the S3 bucket storing all your robologs.

• Name: ingest_analytics_platform
• Arn: arn:aws:iam::<provided-Model-Prime-account-number>:role/ingest_analytics_platform

These are the IAM policies needed.

• The action of s3:GetObject on the resource arn:aws:s3:::<your-s3-bucket>/*.
• The actions of s3:List* and s3:GetBucket* on the resource arn:aws:s3:::<your-s3-bucket>.

You should be ready to set up the permission if you are familiar with tools like Terraform, AWS CDK, or AWS CLI. The following section is for users who are using the AWS Console UI.

Configure S3 bucket permission using AWS console

Go to the S3 bucket that stores your robologs. Open the “Permissions“ tab.

Once in this tab, scroll down to the bucket policy widget and press Edit.

Once editing, append the last two statements of the following code snippet to the existing policy. Don't forget to replace <your-s3-bucket> with your S3 bucket name.

{
    "Version":"2012-10-17",
    "Statement": [
        {
            "Sid": "Grant access to Model-Prime ingest",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<provided-Model-Prime-account-number>:role/ingest"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<your-s3-bucket>/*"
        }

        ...

        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<provided-Model-Prime-account-number>:role/ingest_analytics_platform"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<your-s3-bucket>/*"
        },

        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<provided-Model-Prime-account-number>:role/ingest_analytics_platform"
            },
            "Action": [
                "s3:List*",
                "s3:GetBucket*"
            ],
            "Resource": "arn:aws:s3:::<your-s3-bucket>"
        }
    ]
}

Take note that we have set Allow for AP's ingest role to perform the s3:GetObject, s3:List* and s3:GetBucket* actions on your S3 bucket. You must use the same Principal value as shown above. The Resource value should refer to your bucket ARN which may be conveniently copied from the AWS web console.

Encrypted buckets

If you've elected to encrypt your bucket, you will need to grant Analytics Platform's IAM ingest role permission to use the bucket's KMS key. Official instructions on creating a cross-account KMS key before associating it with your bucket may be found here.

Following the official cross-account key creation instructions will result in a policy that provides access to Model-Prime's root account. You will need to edit the key policy using the editor provided during the key creation Review step, or after creating the key, by changing all references of <provided-Model-Prime-account-number>:root to <provided-Model-Prime-account-number>:role/ingest_analytics_platform.

If you've already created the key, follow these steps to edit the key policy:

1. Go to the AWS Key Management Service dashboard
2. Select Customer managed keys
3. Click on the key alias for the key associated with your ingest bucket
4. Scroll down to the Key policy section and click the Edit button
5. Ensure that the following policy statements are included:

{
  "Sid": "Allow use of the key",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::<provided-Model-Prime-account-number>:role/ingest_analytics_platform"
  },
  "Action": [
    "kms:Encrypt",
    "kms:Decrypt",
    "kms:ReEncrypt*",
    "kms:GenerateDataKey*",
    "kms:DescribeKey"
  ],
  "Resource": "*"
}

{
  "Sid": "Allow attachment of persistent resources",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::<provided-Model-Prime-account-number>:role/ingest_analytics_platform"
  },
  "Action": ["kms:CreateGrant", "kms:ListGrants", "kms:RevokeGrant"],
  "Resource": "*",
  "Condition": {
    "Bool": {
      "kms:GrantIsForAWSResource": "true"
    }
  }
}

This should conclude the S3 bucket permission setup.