Skip to content

Transformer - Generic Expression Data Source

Description

The Generic Expression Data Source transformer is designed to extract and process structured data from text files using regular expressions. This transformer is particularly useful for parsing and structuring unstructured data into a format that can be easily consumed by applications. It supports the extraction of various data fields such as names, addresses, and other relevant information from text documents.

Config

Parameters

Parameter Type Default Description
dataSource Object N/A Configuration for the data source, including entity type and specific settings.
listExpression String N/A Regular expression used to identify and extract data from the text.
explicitMappings Array N/A Mappings to define how extracted data fields are mapped to dictionary entries.

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
<apiroConf version="1" xmlns="http://apiro.com/apiro/v1/root">
    <loadOrder>20</loadOrder>
    <dataFeeds>
        <dataFeed definition="EXPR_REGEX" name="PEOPLE_REGEX">
            <execPriority>10</ePriority>
            <enabled>true</enabled>
            <push>false</push>
            <pull>true</pull>
            <schema>PEOPLE</schema>
            <config><![CDATA[
{
    "dataSource": {
        "entity": "GIT",
        "config": {
            "password": "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",
            "gitURL": "https://github.com/redapiro/apiro_engine_test_feeds.git",
            "branch": "rudtest",
            "pathPrefix": "/rudtest/people.txt",
            "username": "apirobot"
        }
    },
    "listExpression": "(===[^=]*)",
    "explicitMappings": [
        {
            "dictionary": "firstname",
            "value": "#{PAYLOAD.resolve('fname:\\s*([a-zA-Z]+)')}"
        },
        {
            "dictionary": "lastname",
            "value": "#{PAYLOAD.resolve('lname:\\s*([a-zA-Z]+)')}"
        },
        {
            "dictionary": "age",
            "value": "#{PAYLOAD.resolve('age:\\s*([0-9]+)')}"
        }
    ]
}
]]>
            </config>
        </dataFeed>
    </dataFeeds>
</apiroConf>

Here is a concise portion of the above example, including only the direct structure of the transformer:

1
2
3
{
  "listExpression": "(===[^=]*)"
}

Common Mistakes

  • Incorrect Regular Expression: Ensure that the listExpression is correctly defined to match the data structure in your text files.
  • Misconfigured Data Source: Verify that the dataSource configuration, including entity type and specific settings, is correctly set up to access your data source.
  • Incorrect Mappings: Check that the explicitMappings are correctly defined to map extracted data fields to dictionary entries.
  • Security Concerns: Be cautious with the use of sensitive information, such as passwords, in your configuration files.