I have a partitioned table that contains ~2 billion records in total. I need to get each row out, to publish to a queuing system for follow-on processing.
I was starting with
SELECT ... FROM ... ORDER BY created_at ASC LIMIT <num> OFFSET <num>. I am using a limit of 10,000. I noticed as I got further into the results (2 million rows) that the queries were taking longer to get the next 10,000 rows.
Is there a better way to get this data out? Should I perform an export and read the files from disk instead?
Thanks in advance!