Skip to main content

Automating ingestion for AWS S3

With the ingestion process having been tested manually, your workflow with Model-Prime may be optimized such that we ingest robologs whenever they are uploaded to your data store. It is assumed that all data placed in your data store is intended to be shared with Model-Prime under this configuration such that any new object added to the data store will trigger ingest. This also allows us to continually validate your data store settings in order to alert you if we ever unexpectedly lose access to the data store.

Automation managed by Model-Prime

Step 1: Register your data store​

In order to let Model-Prime handle ingestion events automatically, the data store containing robologs that you want ingested must be registered with us. An easy way to manage your registered data stores is from our web UI.

Web App UI Ingest Config Page

Click on the Add ingest store button.

Web App UI Add Ingest Config

Fill in the required fields:

  • bucket: the name of your AWS S3 bucket.
  • cloud_account_id: the 12-digit AWS account number of the bucket owner.
  • log_extension: the extension of the file in which the robolog metadata (ex. ROS 2 .yaml files) or the log itself (ex. ROS 1 .bag files) can be found.
  • log_type: we currently support the ROSBag1 and ROSBag2 log types.

Alternatively, you can register your data store using the Ingest Config Swagger Page or by making the equivalent API calls from your own services.

We validate access to the data store while processing your request to POST /api/ingest/v2/stores. If the provided data store is provisioned incorrectly, this endpoint will return an error stating that it cannot access the data store. Otherwise, the endpoint will return a 200 response along with the details of the newly created ingest config in the response body.

Step 2: Modify your AWS S3 bucket triggers​

Prerequisites​

Take note of your S3 bucket region. Our onboarding team should have asked for this when discussing the initial set up of your organization, which means we should have a Lambda in that region. If your organization's S3 bucket is not in a region which is currently supported (see our supported regions below) and no one from Model-Prime has asked about your bucket region, please reach out so that we may get that set up for you.

Supported regions​

  • us-east-1
  • us-east-2

In the following example, our S3 bucket is in us-east-2 which is NOT our primary region.

Setting up event notifications​

  1. Navigate to your S3 bucket in the AWS dashboard, then click on the Properties tab.
  2. Scroll down to Event notifications and click the Create event notification button.
  3. You now have the option to configure this trigger. We recommend naming it something which communicates that it is going to be used to notify Model-Prime of new robolog uploads (ex. ModelPrimeIngestTrigger).
  4. Ensure that All object create events is enabled or - at the very least - make sure to enable the creation event types for which you want Model-Prime to automatically recieve events.
    Event_Type
  5. At the bottom, select Enter Lambda function ARN and enter an ARN from the following list, where the suffix of the ARN matches the region where your bucket is located:
    • arn:aws:lambda:us-east-1:<provided-Model-Prime-account-number>:function:robolog_ingest_trigger_us-east-1
    • arn:aws:lambda:us-east-2:<provided-Model-Prime-account-number>:function:robolog_ingest_trigger_us-east-2
  6. Save your changes.

Automatic ingest triggering is now set up on your bucket! Feel free to test it by uploading robologs to the bucket, followed by making a GET request to the /robologs endpoint.