Skip to main content

Automating ingestion for Google Drive

Model-Prime supports automated ingestion by connecting to a Shared Drive which you, the registering user, has read and write access to. Granting access in this way also grants Model-Prime with read and write permission to the Drive. This Shared Drive is where you store your robologs and supplementary files for those robologs. Once authorized, Model-Prime will be able to automatically detect when new files are uploaded by the registering user and ingest them into our system so that you may make use of the metadata found within.

Due to a technical limitation in Google Drive, Model-Prime can only receive events sent by the Google account which was used to register the Shared Drive. Model-Prime will still be able to determine that a file was uploaded by other users, but those events will not be received until the registering user uploads or changes a file

Access considerations​

Model-Prime obtains access to your Google Drive using oauth2 for a single user. This allows you to control whether Model-Prime has access your drives through Google's Third Party Connections page. Google does not provide a way to limit Model-Prime's access to specific set of folders or Drives.

For this reason, if you have sensitive data which you do not want Model-Prime to have access to, we recommend granting access by using a separate Google account which only has read and write permission to the data you wish to share with Model-Prime.

Regardless of this, Model-Prime will NEVER read files from Drives which have not been registered with the Ingest API unless a user within your organization manually triggers the ingestion process for a file.

Because of the inability to support fine-grained access to drives, and folders within drives, we recommend that organizations migrate their robologs to other supported Data Storage solutions, such as S3, which do provide the ability to limit what can be accessed

Step 1: Determine the Drive ID of your Shared Drive​

The Model-Prime platform supports the registration of Shared Drives, a type of Google Drive able to be easily shared among multiple users. You can create and access Shared Drives in the Google Drive UI by navigating to Shared Drives. This is located under My Drive in the left-hand navigation panel.

In order to tell Model-Prime which Drive(s) to ingest from, you must provide the Drive ID of the Shared Drive. To obtain the Drive ID, navigate to the root folder of a Shared Drive. Once there, the Drive ID is located in the URL of the page. For example, the Drive ID for the following URL is 0RwQmkr1qmy0DUq9PVN:

https://drive.google.com/drive/folders/0RwQmkr1qmy0DUq9PVN

IMPORTANT NOTE: The URL in Google Drive will always look similar to the one shown above, but the ID is only accurate if it is obtained from the root of a Shared Drive.

Step 2: Register your Google Drive​

The Ingest API is the entrypoint to the ingest process and also manages Model-Prime's access to Drives/Buckets. To configure your Google Drive for autmated ingest, perform the following:

  • Navigate to the Configuration page and select the Ingest Configuration tab.
  • Click on + Add ingest store.
  • Select the Google Drive tab and enter the relevant information before submitting.

Add_Ingest_Store

If the Google Account being used has previously granted Model-Prime access to its Shared Google Drives, then you are finished. Automated ingest is now set up for this newly registered Drive. Otherwise, if this is the first time that this account has been used with Model-Prime, then you must complete the next step.

Step 3: Grant Model-Prime permission to the drive​

In order to grant Model-Prime access to the Drive, you must complete an Oauth Setup process with Google. The UI or Swagger page will automatically redirect you to the necessary page where this is performed.

Google_Auth

Simply log in (if necessary) and select the desired account which has access to the Google Drive, then click Allow. This will grant Model-Prime access to the Drive and allow ingestion to occur for any new robologs which are uploaded by the registering user to it. A successful registration will redirect you back to the Model-Prime UI or Swagger page, whichever was used to register the drive.