Incrementally add new rows without reloading the entire dataset

Suppose that you have a main table with all your data and you want to bring new records. You will need to setup a temporary table pointing to the same data source but just brings new records that are not yet on your main table.  The temporary table has to be setup on-demand to bring new records that do not exists on the main table. Something like:  >= Max([MainTable].[id]) + 1 id is a sequence. 

step by step setup procedure:

1. Add two data tables in memory on your analysis
1. destination Table contains all the data. Could be millions of records
2. source Table comes from the same source. Set on-demand parameters:
• Set a min range expression input to be : Max([destinationTable].[id]) + 1
• That will take new records
2. At this point nothing is shown, unless you refresh sourceTable with new records
3. The iron python script programmatically reloads only the sourceTable and appends those rows to the main table. 
4. To keep a fair amount of records and prevent the data canvas to have unlimited number of transformations, you can delete old records.

from Spotfire.Dxp.Data import AddRowsSettings

from Spotfire.Dxp.Data.Import import DataTableDataSource

# Parameters

source_table = sourceTable  # Temporary table for new records

dest_table = destTable      # Main table holding all data

max_ops = 4                 # Max transformations in the data canvas

# 1. Reload Source Table to Fetch New Records

source_table.ReloadAllData()

# 2. Append Rows to Destination Table

if source_table.RowCount > 0:

    # 2.1 Define the data source from Source Table

    data_source = DataTableDataSource(source_table)

    

    # 2.2 Append rows to the Destination Table

    row_settings = AddRowsSettings(dest_table, data_source)

    dest_table.AddRows(data_source, row_settings)

# 3. Remove Old Transformations from Destination Table (optional)

source_view = dest_table.GenerateSourceView()

operations = source_view.OperationsSupportingTransformations

if operations.Count > max_ops:

    first_operation = operations[1].DataOperation

    source_view.RemoveOperation(first_operation)