2.3.2 - Manage Scheduling and Backfills using BigQuery in Kestra
Last updated Jan 30, 2025
Previous2.3.1 - Create an ETL Pipeline with GCS and BigQuery in KestraNext2.3.3 - Transform Data with dbt and BigQuery in Kestra
Last updated
Last updated Jan 30, 2025
Last updated
Youtube Video | ~5 min
Goal is to include a trigger to add the taxi datasets moving forward, and also backfilling in years past.
Working files can be found here
In Kestra: Flow 6 scheduled -> Saved -> Triggers -> backfill
Maybe it is my good-ole Alaska wifi, but I can only backfill a couple months at a time. I tried to do yellow taxi 2020 as a full year, but it timed out on me.
Take a look at the homework questions, and determine which and when you should download these files
