Skip to content

GCP PUBSUB - DATA SINK

Description

The GCP PUBSUB data sink is designed to distribute data to Google Cloud Pub/Sub topics. This method is particularly useful for integrating with Google Cloud services, allowing for real-time data processing and analysis. The GCP PUBSUB data sink supports various configuration options to customize the data distribution process, including specifying the project ID, topic ID, and service account key for authentication.

Config

REQUIRED

Config Parameters

The following are the configuration parameters for the GCP PUBSUB data sink:

  • projectId: The Google Cloud project ID where the Pub/Sub topic is located. This is a required parameter.
  • topicId: The name of the Pub/Sub topic to publish data to. This is a required parameter.
  • serviceAccountKey: A valid Google Cloud service account key for authenticating with Pub/Sub. This is a required parameter.
Name Description
projectId The Google Cloud project ID where the Pub/Sub topic is located. This is a required parameter.
topicId The name of the Pub/Sub topic to publish data to. This is a required parameter.
serviceAccountKey A valid Google Cloud service account key for authenticating with Pub/Sub. This is a required parameter.

Config Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<apiroConf version="1" xmlns="http://apiro.com/apiro/v1/root">
    <loadOrder>25</loadOrder>
    <dataSinks>
        <dataSink name="SECURITY_STATIC_PUBSUB" definition="GCP_PUBSUB">
            <config>
                <![CDATA[
                    {
                        "projectId":"MY_PROJECT_ID",
                        "topicId":"Security Static",
                        "serviceAccountKey":"${SYS:PUBSUB_SVC_KEY}"
                    }
            ]]>
            </config>
        </dataSink>
    </dataSinks>
</apiroConf>

Common Mistakes

  • Incorrect Project ID or Topic ID: Ensure that the projectId and topicId parameters match exactly with your Google Cloud project and Pub/Sub topic.
  • Invalid Service Account Key: The serviceAccountKey must be a valid JSON key for a service account with the necessary permissions to publish messages to the specified Pub/Sub topic.
  • Permissions Issues: Ensure that the service account associated with the serviceAccountKey has the appropriate permissions to publish messages to the specified Pub/Sub topic. This includes roles such as roles/pubsub.publisher.
  • Quota Exhaustion: Be aware of Google Cloud Pub/Sub quotas and limits. If you encounter errors related to quota exhaustion, consider adjusting your data distribution strategy or requesting a quota increase.