3.2.2 - Internals of Big Query
Last updated Feb 5, 2025
Last updated
Last updated Feb 5, 2025
Last updated
🕓 Estimated time spent on this lesson | ~15 min
Youtube Video | ~4 min
✍️ In this video we learn about the internals of BigQuery
"BigQuery and Dremel share the same underlying architecture. By incorporating columnar storage and tree architecture of Dremel, BigQuery offers unprecedented performance. But, BigQuery is much more than Dremel. Dremel is just an execution engine for the BigQuery. In fact, BigQuery service leverages Google’s innovative technologies like Borg, Colossus, Capacitor, and Jupiter. As illustrated below, a BigQuery client (typically BigQuery Web UI or bg command-line tool or REST APIs) interact with Dremel engine via a client interface. Borg - Google’s large-scale cluster management system - allocates the compute capacity for the Dremel jobs. Dremel jobs read data from Google’s Colossus file systems using Jupiter network, perform various SQL operations and return results to the client. Dremel implements a multi-level serving tree to execute queries which are covered in more detail in following sections." - https://panoply.io/data-warehouse-guide/bigquery-architecture/
"It is important to note, BigQuery architecture separates the concepts of storage (Colossus) and compute (Borg) and allows them to scale independently - a key requirement for an elastic data warehouse. This makes BigQuery more economical and scalable compared to its counterparts." - https://panoply.io/data-warehouse-guide/bigquery-architecture/
BigQuery's sandbox: Get started in the BigQuery sandbox, risk-free and at no cost.
