Skip to content

EMAIL - DATA SOURCE

Description

The EMAIL data source is designed to retrieve data from email attachments. This method is particularly useful for processing and analyzing data contained within email attachments, such as reports, spreadsheets, or documents. The EMAIL data source supports various configuration options to customize the data retrieval process, including specifying the email server, authentication details, and criteria for selecting specific emails and attachments.

Config

REQUIRED

Config Parameters

Name Description
url The URL of the email server or service API endpoint from which emails will be retrieved. This is a required parameter.
method The HTTP method (e.g., GET, POST) to use for the request. This is a required parameter.
headers A set of HTTP headers to include in the request, such as authentication tokens. This is an optional parameter.
requestParams A set of query parameters to include in the request, such as filters for selecting specific emails. This is an optional parameter.
substitutions A set of substitutions to apply to the request parameters, allowing for dynamic values to be inserted into the request. This is an optional parameter.
cascadeSub The name of the substitution to cascade through multiple pages of results. This is an optional parameter.
cascadeMax The maximum number of pages to cascade through when using cascading substitutions. This is an optional parameter.

Config Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
<?xml version="1.0" encoding="UTF-8"?>
<apiroConf version="1" xmlns="http://apiro.com/apiro/v1/root">
    <loadOrder>10</loadOrder>
    <dataSources>
        <dataSource definition="EMAIL" execPriority="10" name="TEST_EMAIL_SOURCE">
            <config>
                <![CDATA[
                      {
                        "host": "${SYS:APIRO_EMAIL_HOST}",
                        "port": 993,
                        "username": "${SYS:APIRO_EMAIL_USERNAME}",
                        "password": "${SYS:APIRO_EMAIL_PASSWORD}",
                        "provider": "${SYS:APIRO_EMAIL_PROVIDER}",
                        "readFolder": "INBOX",
                        "moveOlderThanDays": null,
                        "subjectPrefix": "SRMs",
                        "filePrefix": "SRM",
                        "trackMailDate": false,
                        "props": {
                            "mail.imaps.starttls.enable" : "true"
                        },
                        "transformers": [
                            {
                                "name": "UNZIP",
                                "entity": "UNZIP",
                                "config": {
                                    "fileMatch": "SRMs"
                                }
                            }
                        ]
                    }
                ]]>
            </config>
        </dataSource>
    </dataSources>
</apiroConf>

Inline Feed Source

The Email Source can also be used inline within a dataFeed config. The same parameter are used for the inline source. An example is given below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
<?xml version="1.0" encoding="UTF-8"?>
<apiroConf fragmentId="testfrag">
    <loadOrder>20</loadOrder>
    <dataFeeds>
        <dataFeed definition="EXPR_EXCEL_FEED2" name="MORNINGSTAR_INPUT_1_XLSX_EMAIL">
            <execPriority>10</execPriority>
            <enabled>true</enabled>
            <push>true</push>
            <pull>true</pull>
            <schema>MORNINGSTAR_5Y_MARKET_RETURN</schema>
            <config><![CDATA[{
  "dataSource" : {
        "entity": "EMAIL",
        "config": {
            "host": "${SYS:APIRO_EMAIL_HOST}",
            "port": 993,
            "username": "${SYS:APIRO_EMAIL_USERNAME}",
            "password": "${SYS:APIRO_EMAIL_PASSWORD}",
            "provider": "${SYS:APIRO_EMAIL_PROVIDER}",
            "readFolder": "INBOX",
            "moveOlderThanDays": null,
            "subjectPrefix": "Input_1",
            "filePrefix": "Morningstar_Input_1",
            "trackMailDate": false,
            "props": {
                "mail.imaps.starttls.enable" : "true"
            },
            "transformers": [
                {
                    "name": "UNZIP",
                    "entity": "UNZIP",
                    "config": {
                        "fileMatch": "Morningstar_Input_1"
                    }
                }
            ]
        }
    },
  "dataAtLine" : 12,
  "sheet" : "Sheet1",
  "itemLimit" : 20,
  "explicitMappings" : [
      {
          "dictionary": "SecID",
          "value": "#GRV{PAYLOAD.resolve('A')}"
      },
      {
          "dictionary": "Investment",
          "value": "#GRV{PAYLOAD.resolve('B')}"
      },
      {
          "dictionary": "APIR",
          "value": "#GRV{PAYLOAD.resolve('C')}"
      }
  ]
}]]></config>
        </dataFeed>
    </dataFeeds>
</apiroConf>

Other Uses

Email data source can also be configured to read the message content of emails. In the config below, "MESSAGE_CONTENT" is used in the config fileMatch field within config

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
<config>
    <![CDATA[
          {
            "host": "${SYS:APIRO_EMAIL_HOST}",
            "port": 993,
            "username": "${SYS:APIRO_EMAIL_USERNAME}",
            "password": "${SYS:APIRO_EMAIL_PASSWORD}",
            "provider": "${SYS:APIRO_EMAIL_PROVIDER}",
            "readFolder": "INBOX",
            "moveOlderThanDays": null,
            "subjectPrefix": "SRMs",
            "filePrefix": "SRM",
            "trackMailDate": false,
            "props": {
                "mail.imaps.starttls.enable" : "true"
            },
            "transformers": [
                {
                    "name": "UNZIP",
                    "entity": "UNZIP",
                    "config": {
                        "fileMatch": "MESSAGE_CONTENT"
                    }
                }
            ]
        }
    ]]>
</config>

Common Mistakes

  • Incorrect URL or Method: Ensure that the url and method parameters are correct and correspond to a valid email server or service API endpoint.
  • Authentication Issues: Verify that the headers parameter includes the correct authentication tokens or credentials required to access the email server or service.
  • Incorrect Request Parameters: If you encounter issues with selecting specific emails or attachments, check the requestParams for any errors or missing required fields.
  • Substitution Errors: If the data retrieval process fails due to substitution errors, ensure that the substitutions are correctly defined and that the cascading parameters (cascadeSub and cascadeMax) are properly configured.