Skip to main content

Automating ingestion for AWS S3

With the ingestion process having been tested manually, your workflow with Model-Prime may be optimized such that we ingest robologs whenever they are uploaded to your data store. It is assumed that all data placed in your data store is intended to be shared with Model-Prime under this configuration such that any new object added to the data store will trigger ingest. This also allows us to continually validate your data store settings in order to alert you if we ever unexpectedly lose access to the data store.

Automation managed by Model-Prime

Step 1: Register your data store​

In order to let Model-Prime handle ingestion events automatically, the data store containing robologs that you want ingested must be registered with us. An easy way to manage your registered data stores is from our web UI.

Web App UI Ingest Config Page

Click on the Add ingest store button & select the AWS S3 tab.

Web App UI Add Ingest Config

Fill in the required fields:

  • bucket: the name of your AWS S3 bucket.
  • cloud_account_id: the 12-digit AWS account number of the bucket owner.
  • log_extension: the extension of the file in which the robolog metadata (ex. ROS 2 .yaml files) or the log itself (ex. ROS 1 .bag files) can be found.
  • log_type: we currently support the ROSBag1 and ROSBag2 log types.

Step 2: Modify your AWS S3 bucket triggers​

Prerequisites​

Take note of your S3 bucket region.

Supported regions​

  • us-east-1
  • us-east-2

If you require support for a different region, send a request to support@model-prime.com, and the Model-Prime support team will enable ingestion for your selected region.

In the following example, the S3 bucket is in us-east-2.

Setting up event notifications​

  1. Navigate to your S3 bucket in the AWS dashboard, then click on the Properties tab.
  2. Scroll down to Event notifications and click the Create event notification button.
  3. You now have the option to configure this trigger. We recommend naming it something which communicates that it is going to be used to notify Model-Prime of new robolog uploads (ex. ModelPrimeIngestTrigger).
  4. Ensure that All object create events is enabled or - at the very least - make sure to enable the creation event types for which you want Model-Prime to automatically recieve events.
    Event_Type
  5. At the bottom, select Enter Lambda function ARN and enter an ARN from the following list, where the suffix of the ARN matches the region where your bucket is located:
    • arn:aws:lambda:us-east-1:<provided-Model-Prime-account-number>:function:robolog_ingest_trigger_us-east-1
    • arn:aws:lambda:us-east-2:<provided-Model-Prime-account-number>:function:robolog_ingest_trigger_us-east-2
  6. Save your changes.

Automatic ingest invocation is now set up on your bucket! Feel free to test it by uploading robologs to the bucket, and you should see the data begin to process in our web app shortly thereafter.