Skip to content

FILESYSTEM - DATA SOURCE

Description

The FILESYSTEM data source is designed to retrieve data from files located on a local or network file system. This method is particularly useful for accessing and processing data stored in files, such as CSV, JSON, or XML files, without the need for a database or external service. The FILESYSTEM data source supports various configuration options to customize the data retrieval process, including specifying the base path for file storage, whether to cascade through subdirectories, and an optional path to move files after processing.

Config

REQUIRED

Config Parameters

Name Description
basePath The base path where the files are located. This is a required parameter.
cascade A boolean value indicating whether to cascade through subdirectories. This is an optional parameter.
moveToPath The path to move files to after processing. This is an optional parameter.

Config Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
<?xml version="1.0" encoding="UTF-8"?>
<apiroConf version="1" xmlns="http://apiro.com/apiro/v1/root">
    <loadOrder>10</loadOrder>
    <dataSources>
        <dataSource definition="FILESYSTEM" name="TEST_FILESYSTEM_SOURCE">
            <config>
                <![CDATA[
                    {
                          "baseDirectory":"/apiro-input/",
                          "filePrefix":"customer_a.xlsx",
                          "cascade": true,
                          "moveToPath": "/apiro-input/archive"
                    }
                ]]>
            </config>
        </dataSource>
    </dataSources>
</apiroConf>

Common Mistakes

  • Incorrect Base Path: Ensure that the basePath parameter correctly points to the directory where the files are located. The path should be absolute and accessible from the machine running the data source.
  • Cascade Configuration: If you encounter issues with file retrieval, especially when expecting files from subdirectories, verify the cascade parameter is correctly set to true if you wish to include subdirectories in the search.
  • Move Path Issues: If you have configured the moveToPath and files are not being moved as expected, ensure that the specified path exists.