Skip to content

GCP PUBSUB - DATA SOURCE

Description

The GCP PUBSUB data source is designed to retrieve data from Google Cloud Pub/Sub topics. Google Cloud Pub/Sub is a fully-managed real-time messaging service that allows you to send and receive messages between independent applications. This method is particularly useful for integrating with Google Cloud services and third-party applications, enabling real-time data processing and analysis. The GCP PUBSUB data source supports various configuration options to customize the data retrieval process, including specifying the Google Cloud project ID, subscription ID, topic ID, and the MIME type of the data being transferred.

Config

REQUIRED

Config Parameters

Name Description
projectId The Google Cloud project ID where the Pub/Sub topic is located. This is a required parameter.
subscriptionId The subscription ID to listen to messages from the specified topic. This is a required parameter.
topicId The topic ID from which messages will be retrieved. This is a required parameter.
mimeType The MIME type of the data being transferred. This is an optional parameter.

Config Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
<?xml version="1.0" encoding="UTF-8"?>
<apiroConf version="1" xmlns="http://apiro.com/apiro/v1/root">
    <loadOrder>10</loadOrder>
    <dataSources>
        <dataSource definition="GCP_PUBSUB" execPriority="10" name="TEST_PUBSUB_SOURCE">
            <config>
                <![CDATA[
                    {
                        "projectId" : "your-project-id",
                        "subscriptionId" : "your-subscription-id",
                        "topicId" : "topic-id",
                        "mimeType" : "application/octet-stream"
                    }
                ]]>
            </config>
        </dataSource>
    </dataSources>
</apiroConf>

Inline Feed Source

Common Mistakes

  • Incorrect Project ID or Subscription ID: Ensure that the projectId and subscriptionId parameters are correct and correspond to an existing Google Cloud project and subscription.
  • Invalid Topic ID: Verify that the topicId parameter matches the ID of an existing Pub/Sub topic within the specified project.
  • MIME Type Issues: If you encounter issues with data transfer, check the mimeType parameter to ensure it matches the format of the data being transferred.
  • Authentication and Permissions: Ensure you have the necessary permissions and authentication credentials to access the specified Google Cloud Pub/Sub resources.