postgresql - Executing query in chunks on Greenplum -
i trying creating way convert bulk date queries incremental query. example, if query has condition specified
where date > now()::date - interval '365 days' , date < now()::date this fetch years data if executed today. if same query executed tomorrow, 365 days data again fetched. however, have last 364 days data previous run. want single day's data fetched , single day's data deleted system, end 365 days data better performance. data stored in separate temp table.
to achieve this, create incremental query, executed in next run. however, deleting single date data proving tricky when "date" column not feature in select clause feature in condition temp table schema not have "date" column.
so thought of executing bulk query in chunks , assign id chunk. way, can delete chunk , add chunk , other data remains unaffected.
is there way achieve same in postgres or greenplum? inbuilt functionality. went through whole documentation not find any.
also, if not, there better solution problem.
i think best handled aggregates table (i assume issue have heavy aggregates handle on lot of data). doesn't cause normalization problems (and data warehouses denormalize anyway). in regard aggregates need can stored per day able cut down 1 record per day of closed data, plus non-closed data. keeping aggregates data cannot change required avoid normal insert/update anomilies normalization prevents.
Comments
Post a Comment