2.2.2 - Manage Scheduling and Backfills using Postgres in Kestra

Last updated Jan 29, 2025

Youtube Video | ~7 min

If you stopped from the last video, be sure to restart Docker and open Kestra

This video does not go through writing the script, so you will need to grab it from the repo.

❗I was having issues in the next video that talks about dbt, so I updated my docker-compose.yml. ALSO have your docker-compose.yml live outside of a sub-directory and in your main directory, because you want to run the same one in each of the upcoming videos. See those files used here

https://github.com/Tinker0425/de-zoomcamp-my-work/tree/master/module-02/video_4

I was then able to create a new server connection on pgAdmin using:

❗NOTE the difference between _and -

The full script provided, shows that the 'triggers' will be the first of each month and they two datasets will be loaded it an hour apart from each other. But, how do we backfill the data from the past? Well once we have this script saved on Kestra, we can go to the 'Triggers' tab and there is a 'Backckfill Executions' that should be highlighted in purple. Click that:

Showing what elements to fill out for Backfill execution

✅ Once your execution is complete, you should now see the months you backfilled in pgAdmin

Be sure to backfill a few months for BOTH yellow and green taxi to prepare for the next video