Scalability

Clustering

Clustering model is sharded multi-multimaster with robust HA.

The cluster by default has 128 cluster partitions. critical jobs like feeds and distributions are distributed over the network on an embedded persistent datagrid.

Jobs have a partition code which determines the shard its assigned to and additionally to indicate some jobs are related must be done sequentially. Jobs may also indicate no affinity and they will be distributed randomly accross the cluster.

Active cluster nodes distribute responsibility for partition shards evenly amongst themselves and take responsibility for executing those partitions.  In the instance of node failure the affected shards are redistributed to other nodes and jobs that did not complete successfult are rerun. So the HA is 'at least once' processing semantics.

In a graceful node exit the transfer time is milliseconds. In a disorderly transfer such as a node failure the transfer is in the order of a few seconds. Once a job such as a feed is allocated as a cluster job, it is persisted in a data grid, and the assigned partition node will execute it to completion, even in the event of multiple node failures. A maximum retry limit may be set in the case a catastrophic system failure of some sort causes all jobs to keep rerunning, as this may result in exhorbitant access costs to some subscription services.

Clients may make use of autoscaling down to one active node and up to 128 provided its configured properly.

Extracting large data sets from databases

(eg. How does the platform load balance and perform when is fetching 1 million records from a database.) In general the platform should be viewed as being optimised for real time incremental processing of data as and when it changes. Full feed processing goes through several phases, including data sourcing. Individual data sources may make use of concurrency if they wish but generally these things are done single threaded - ie pull a file over a network etc. From there by and large data blocks are pushed through the pipeline via gated concurrency - depending on the config settings for internal threadpools. So datablock validation will be pushed through a threadpool where they are processed in parrallel 50 at a time or whatever profiling suggests is the right level to utilise the hardware. This may change depending on how IO or CPU bound the processing is. So a single feed may monopolise the resources of a single host, and you may have up to 128 active cluster nodes that can do this simultaneously. Alternatively you may run hundreds of feeds simultaneously which share the resources of one or a few nodes. A feature know as feed cascading is also available, which you can use for things like paginated rest services. This allows you to configure a feed to get the first page of a response then respawn itself to get the succeeding page till there are no more items. This can also be done with JDBC datasources. In this case the transaction unit is the single page, so a node failure at page 500 will restart at page 500. Data consistency in this case would rely on constructing the SQL request to do so. Another supported approach of that bulk processing scenario is to submit multiple feeds for portions of the total amount. In that case the one processing job could utilise the entire cluster. This is currently supported with pipeline initiated feeds. NOTE: Care should be taken if large number of violations are expected.