Skip to content

danafholmes/looker_dbt_exposure_generator

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

16 Commits
 
 
 
 
 
 
 
 

Repository files navigation

Intro

This is a project born from the 2023 Looker hackathon. It uses the Looker SDK and some metadata from dbt to automatically generate dbt Exposures.

This lets you keep your dbt exposures up to date without manually writing the YAML for them, or going through the tedious process of figuring out what models your dashboards reference. With your exposures up to date and accurately catloging the models your dashboards reference, when a model or test produces an error, you can easily answer the age-old question of 'Something in dbt broke. What dashboards is this impacting?'

Once you've updated your exposures, you might consider implementing Elementary Data to enhance your dbt instance's data observability and generate an enhanced lineage graph that lets you see what Looker Dashboards are impacted by the thing your data engineering team just broke.

Pre-Requisites/Assumptions:

  • You have a functioning dbt project, and your dbt project is used to generate the tables and views that are referenced by your Looker dashboards
  • You have dbt installed and functioning locally, or a functioning dbt cloud instance, and you can generate your dbt docs and get a hold of a catalog.json file
  • You have a Looker API key that has the permissions to retrieve dashboards, merged queries, and compile queries

Make it go:

Clone the repo to your local machine

Install requirements:

$ pip install -r requirements.txt

Create a looker.ini file

Generate a catalog.json file

$ cd your/dbt/dir
$ dbt docs generate
$ cp ./compiled/catalog.json /directory/where/you/cloned/the/repo

Generate an exposure for a dashboard:

$ python dbt_exposure_generator.py --dashboard DASHBOARD_ID

Generate an exposure for a look:

$ python dbt_exposure_generator.py --look LOOK_ID

Generate exposures for all dashboards in a folder:

$ python dbt_exposure_generator.py --folder FOLDER_ID

Generate exposures for dashboards AND looks in a folder:

$ python dbt_exposure_generator.py --folder FOLDER_ID --all

You can stack arguments, and they all accept multiple arguments too, eg:

$ python dbt_exposure_generator.py --folder FOLDER_ID1 FOLDER_ID2 --look LOOK_ID --dashboard DASHBOARD_ID456 DASHBOARD_ID789

Caveats:

  • SQL parser used is specific to BigQuery. Maybe it will work on other SQL dialects. Probably not. Wanna use it on a Looker project on a different dialect? Find a python SQL parser library for your database and figure it out

About

Project for Looker Hackathon 2023 to programatically generate dbt Exposure YAML from Looker Dashboards/Looks

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages