Use dbt2looker-bigquery to generate Looker view files automatically from dbt models in Bigquery.
This is a fork of dbt2looker that is specific to bigquery. The intention is to allow one to define most of the simple and tedious lookml settings in dbt. That way the lookml code gets less bloated, and can be more focused on advanced metrics and explores.
Want a deeper integration between dbt and your BI tool? You should also checkout Lightdash - the open source alternative to Looker
Features
-
Warehouses: BigQuery
-
Column descriptions synced to looker
-
Dimension for each column in dbt model
-
Define Dimensions define common lookml settings in dbt like label, group label, hidden
-
Opinionated Primary key automatically set the first column to be the primary key, and hide it.
-
Create explores for structs automatically generate explores for complex tables with structs and arrays in them.
-
Dimension groups for datetime/timestamp/date columns
-
Measures defined through dbt column
metadatasee below
Run dbt2looker in the root of your dbt project after compiling looker docs.
(dbt2looker-bigquery uses docs to infer types and such)
Generate Looker view files for all models:
dbt docs generate
dbt2lookerGenerate Looker view files for all models tagged prod
dbt2looker --tag prod**Generate Looker view files for all exposed models ** dbt docs - exposures
dbt2looker --exposed_onlyInstall from PyPi repository
Install from pypi into a fresh virtual environment.
# Create virtual env
python3.7 -m venv dbt2looker-venv
source dbt2looker-venv/bin/activate
# Install
pip install dbt2looker-bigquery
# Run
dbt2looker
Or if you use Poetry
# Create virtual env
poetry add dbt2looker-bigquery
# Run
dbt2looker
You can define looker measures in your dbt schema.yml files. For example:
models:
- name: pages
columns:
- name: url
description: "Page url"
- name: event_id
description: unique event id for page view
meta:
looker:
hidden: True
label: event
group_label: identifiers
value_format_name: id
looker_measures:
- type: count_distinct
sql_distinct_key: ${url}
- type: count
value_format_name: decimal_1
dbt2looker can be used in a production setting. It is a bit of a project, so if you want to try to do it, prepare to spend some time.
This explanation is based on you using some kind of pipeline tool to do ci/cd on your dbt project. I am using github actions and google cloud platform, so I will describe it as such.
after having done your continous deployment in the pipeline you can run: (assuming that the manifest.json and catalog.json is in the target folder)
dbt docs generate
dbt2looker --target-dir ./target/ --output-dir ./looker --exposures_onlyat this point you have a bunch of lookml files in a folder named /looker within your github action runner. you will now need to store these files somewhere.
example using gsutil:
gsutil -m cp -r ./looker/views/ "gs://{{dbt2looker-backend}}/${{ github.ref_name }}/"this will put the files in a google cloud storage bucket, in a folder named after the branch you deploy from. So if that branch was main, you now have a lookml state stored in a folder called /main/ in your bucket.
The next step is to create a looker repository for exposing these autogenerated files in looker. in this repo, you want a github action that syncs in the lookml files from the bucket into this project.
here is an example script: (using some ENVIRONMENT VARIABLES) (This script also allows for "locking" certain files from being synced, by adding #nosync anywhere in the file.)
- name: Sync script
id: find_script
run: |
#!/bin/bash
set -e
if [ -z "${GCS_AUTOGEN_BUCKET}" ]; then
echo "🗝️ The $ GCS_AUTOGEN_BUCKET repository variable is not set, Skipping dbt file sync. 😊"
exit 1
fi
filename="manifest.lkml"
if [ -z "${DBT_SYNC_BRANCH}" ]; then
echo "🗝️ The DBT_SYNC_BRANCH variable is not set in lkml or as a repository variable. Skipping dbt file sync. 😊"
exit 1
fi
if [ -z "${SYNC_TO_TOP_LEVEL}" ]; then
echo "Sync to top level is not set. syncing to ./views/_autogen_dbt"
destination_dir="views/_autogen_dbt"
else
echo "Sync to top level is set. syncing to ."
destination_dir="."
fi
source_dir=".tmp/_autogen_dbt"
mkdir .tmp
mkdir $source_dir
gsutil -m cp -r gs://${GCS_AUTOGEN_BUCKET}/${DBT_SYNC_BRANCH}/views/ $source_dir
copy_recursive() {
local source="$1"
local destination="$2"
local -a stack=("$source")
mkdir -p "$destination" # create destination directory if it doesn't exist
while [ ${#stack[@]} -gt 0 ]; do
local last_index=$(( ${#stack[@]} - 1 )) # Get the last index
local current="${stack[last_index]}" # Get the last item in the stack
unset stack[last_index] # Remove the last item from the stack
stack=("${stack[@]}") # Rebuild the array to maintain contiguous indexing
echo "iterating over folder $current"
for item in "$current"/*; do
if [ -d "$item" ] && [ ! -L "$item" ]; then
local rel_path="${item#$source}" # Remove source path prefix
rel_path="${rel_path#/}" # Remove leading slash
mkdir -p "$destination/$rel_path"
stack+=("$item") # Add subdirectory to the stack
elif [ -f "$item" ]; then
local rel_path="${item#$source}" # Remove source path prefix
rel_path="${rel_path#/}" # Remove leading slash
local dest_path="$(dirname "$destination/$rel_path")"
if grep -q "#nosync" "$destination/$item"; then
echo "'$item' contains '#nosync' tag and is not synced."
else
mkdir -p "$dest_path"
echo "copying $item to $dest_path"
cp -f "$item" "$dest_path/"
fi
fi
done
done
}
copy_recursive "$source_dir" "$destination_dir"
Now you have the autogenerated, autoupdating files available in looker! But how do you keep them fresh?
One option is to trigger the github action in the looker repository from the dbt repository. One way to do this is to use a PAT-token to trigger a github action from one repo to another.
Another option is to schedule the refresh of the files in the looker-repository by running the github action for syncing files on a schedule.
So how do you use this automatically refreshing repository? My recommendation is to import the project as a dependency into your main looker project, and ref the files from the autogen looker project. That way you do not end up with merge conflicts and similar issues, and you can go back to previous versions of the sync project, if a problem arises.