Use your dbt project
Load the files dbt writes, and Halo's catalog picks up your descriptions, tags, owners, keys and joins, and shows failing tests and stale sources beside the tables and datasets they affect.
A dbt project already says what discovery has to guess: what each table means, what its keys are, and how tables join. Halo reads that from the artifact files dbt Core writes, on PostgreSQL connections, and on Redshift and Dremio connections where Halo has turned them on for your organization.
Halo doesn't read dbt Cloud's API or dbt's semantic layer yet.
Load your project
Open the connection and go to its dbt tab. Loading files takes the Manage connections permission and edit access to the connection. See Roles and permissions.
By hand: choose Upload artifacts and pick files from dbt's target/ folder.
From CI: under Post from CI, enter a Token name and choose Make a token. Copy the token and the command straight away, since Halo won't show the token again. Run the command after dbt in your own pipeline:
curl -fsS https://api.usehalo.net/v1/dbt/artifacts \
-H "Authorization: Bearer $HALO_DBT_TOKEN" \
-F manifest=@target/manifest.json \
-F catalog=@target/catalog.json \
-F run_results=@target/run_results.json \
-F sources=@target/sources.jsonA token reaches only this connection and can only post these files. Revoke it when you're done with it.
Only manifest.json is needed the first time. After that, a step can post just what it wrote, such as run_results.json after dbt test, and the rest carries over from the load before.
| Limit | |
|---|---|
| One upload | Up to 1 GB, all files together |
| Loads waiting on one connection | Three. A fourth is refused until one finishes |
Each load runs as a job, so you can leave the page. Its row under Loads says what matched, how many tests ran, and what's failing or stale. Run again applies a load to the catalog again, and discovery does the same after each run, so tables it finds later pick up their model.
Which versions Halo reads
| File | dbt schema versions |
|---|---|
manifest.json | v10 to v12: dbt Core 1.6 and later, and dbt v2 |
catalog.json | v1 |
run_results.json | v4 to v6 |
sources.json | v3 |
A file outside these fails its load, and the error names the version it found and the versions Halo reads.
With dbt v2, dbt docs generate no longer writes catalog.json. Use --write-catalog on dbt compile or dbt run if you want it. Halo doesn't need it, since the manifest says where each model and source is. The catalog file only corrects that when a relation has moved.
How models find their tables
A model or source matches a catalog table by database, schema and alias, ignoring case. On Dremio, a table's schema is its full path, so the path is dbt's database then schema, and no_schema means the top of a space.
A load's row lists anything that didn't match, under didn't match a table, with the reason. Ephemeral models have no table, so they aren't listed.
What your catalog gets
| From dbt | What Halo does with it |
|---|---|
| Descriptions | Shown on tables and columns with a dbt mark. New datasets' fields inherit them |
| Tags | From tags and config.tags. They show in the catalog, and its search finds them |
| Owner | From meta.owner, or else the owner of the model's group. Shown and searched the same way |
| Keys | A column with both a unique and a not_null test counts as a key, for suggesting joins and for new datasets' count measures |
| Joins | A relationships test becomes a relationship marked dbt test. It ranks below a declared foreign key and above anything Halo inferred |
Words written by hand always win. After those come dbt's, then the database's own comment. Halo AI only proposes a description where none of them has words, and its unanswered proposals give way when dbt's arrive.
Composite keys aren't handled, because dbt's generic tests name one column.
Failing tests and stale sources
A table's page has a Lineage panel that shows:
whether its model failed to build in dbt's latest run;
its failing tests by name, and any failing upstream;
sources that are behind, meaning their freshness status isn't
pass;Built from: what it's made of;
the Datasets, Charts and Reports built on it that you can see, with a count of datasets you can't.
A dataset's page shows the same warning when a table it reads has a failing test or a failed model, or a source behind it is stale. The datasets list marks those with a failing test or failed model with a badge, such as 1 dbt test failing. A custom SQL dataset doesn't name its tables, so it doesn't get one.
When a test starts failing, the dataset's creator gets a notification, once per dataset per load. A test that was already failing doesn't notify again. To have it emailed as well, turn on dbt tests fail on a dataset of yours in your notification settings.
What changed between manifests
Each load with a new manifest compares it with the last one. Its row says what was added, removed and renamed, under Changed since the last manifest, and which datasets a removal or rename breaks, with how many charts and reports are on them.
Show Halo's reports in dbt docs
Once a project has loaded, Download halo_exposures.yml writes a file you can add to your dbt project, so its docs list the reports over each model. It only lists reports you can see, and Halo writes it only when you ask.
What Halo keeps
Halo keeps what it extracts: names, relations, descriptions, tags, owners, parents, a model's checksum, tests, and how each last ran. It doesn't keep compiled or raw SQL, or test messages, which can quote rows.
The files themselves are deleted from storage when the job ends, however it ends. The five newest loads on a connection keep their extract, so Run again has something to run from.