Skip to content

S3 - DATA SINK

Description

The S3 data sink is designed to distribute data by storing it in an Amazon S3 bucket. This method is particularly useful for long-term data storage and archiving, allowing for easy access and retrieval of data. The S3 data sink supports various configuration options to customize the data distribution process, including specifying the AWS access key, secret key, region, bucket name, and the path within the bucket where the data will be stored.

Config

REQUIRED

Config Parameters

Name Description
accessKey The AWS access key for authentication. This is a required parameter.
secretKey The AWS secret key for authentication. This is a required parameter.
region The AWS region where the S3 bucket is located. This is a required parameter.
bucket The name of the S3 bucket where the data will be stored. This is a required parameter.
path The path within the S3 bucket where the data will be stored. This is an optional parameter.

Config Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
<?xml version="1.0" encoding="UTF-8"?>
<apiroConf version="1" xmlns="http://apiro.com/apiro/v1/root">
    <loadOrder>10</loadOrder>
    <dataSinks>
        <dataSink definition="S3" execPriority="10" name="ESG_DATA_OPS_S3">
            <config>
                <![CDATA[
                    {
                        "accessKey":"SYS:ESG_ACCESS_KEY",
                        "secretKey":"SYS:ESG_SECRET_KEY",
                        "region":"SYS:ESG_S3_REGION",
                        "bucket":"SYS:ESG_S3_BUCKET",
                        "path":"my/file/path"
                    }
                ]]>
            </config>
        </dataSink>
    </dataSinks>
</apiroConf>

Common Mistakes

  • Incorrect AWS Credentials: Ensure that the accessKey and secretKey parameters are correct and have the necessary permissions to write data to the specified S3 bucket.
  • Invalid S3 Bucket or Region: Verify that the bucket and region parameters are correct and point to an existing S3 bucket in the specified AWS region.
  • Path Issues: If you encounter issues with data storage, check the path parameter to ensure it is correctly formatted and points to a valid location within the S3 bucket.
  • Network Issues: If you encounter connectivity issues, check the network configuration and ensure that the AWS S3 service is accessible from the machine.