BellaDati can import data and files from Amazon S3.

Files metadata are saved to the data set and files can be saved to the BellaDati Media gallery.

Connecting to S3

From Data Source Connectors page select S3.

BellaDati User Documentation > Connecting to Amazon S3 > image2022-12-20_12-43-31.png

Connection parameters

BellaDati User Documentation > Connecting to Amazon S3 > image2022-12-20_12-48-52.png

Provide your unique Account key

Provide your unique Account secret

Choose AWS region what has your bucket is assigned to

Provide with your Bucket name

You can select time interval Modified from-to. If data were modified in this time interval, they will be used in import

You can also use relative timing with Max file age in minutes old.

Row prefix regex defines according to what format, should rows be downloaded. For example "\.json$" will search for all files that ends with .json

Column file patterns defines according to what format, should columns be downloaded. Each column regex should be on new line. For example ^A.*\.json$ and ^B.*\.json$. This will download jsons starting with A on first column, and starting with B on second column.

S3 Functions

String getS3Text() - Gets file
String getS3Text(int columnIndex) - Gets file from selected column
JSONElement getS3JSON() - Gets json file
JSONElement getS3JSON(int columnIndex) - Gets json file from selected column
String saveS3MediaFile() - Gets picture and stores it to dataset, returns mediaId
String saveS3MediaFile(boolean skipIfExists) - Gets picture and stores it to media gallery, returns mediaId. With true parameter can skip already downloaded file
String saveS3MediaFile(int columnIndex, boolean skipIfExists) - Gets picture from selected column and stores it to media gallery, returns mediaId. With true parameter can skip already downloaded file
connectS3(String accountKey, String accountSecret, String region, String bucketName, Closure closure) - connects to S3 and sets the "client" as property to the closure
iterateS3Objects(S3Client client, String bucketName, Closure closure) - iterates objects in S3 and sets "client" and "object" as properties to the closure
deleteS3Object(S3Client client, String bucketName, String key)
deleteS3Object(S3Client client, String bucketName, S3Object object)

Example - Load data from JSON file stored in S3

Data returned from S3

S3 connector loads the list of JSON files based on the connection parameters

Getting data from loaded JSON files

In order to get the new data from the related JSON files, add a new column to the Import settings. And define the following transformation script:

getS3JSON(1)

This script will load JSON data from the related file.

Loading data from selected JSON element

For saving the data from the loaded JSON element, there should be created a new column in the import settings. The element value will be loaded by using the following transformation script (this script will load the value from JSON element "camera_id"):

getS3JSON(1).get("camera_id")

Set up Read only policy

You can set permissions for your for whole S3 or even for specific bucket.

Create your bucket with default settings
Get your ARN, this can be found under Properties of your bucket

Create new Policy under IAM. You can use this dummy policy. In Resource instead of ARN_CODE add your ARN

{
    “Version”: “2012-10-17",
    “Statement”: [
        {
            “Effect”: “Allow”,
            “Action”: [
                “s3:Get*“,
                “s3:List*“,
                “s3:Describe*“,
                “s3-object-lambda:Get*“,
                “s3-object-lambda:List*”
            ],
            “Resource”: “ARN_CODE”
        }
    ]
}

Go Next and Next
Enter name of your policy and Save it
Go to Users and search for your user
Add permissions
Select "Attach existing policies directly"
Search for name of policy set in step 5
Select it and click "Next: review" then "Add permissions"