Discovery
Discovery reads a connection's tables and columns, finds how they relate, profiles the values, and suggests what each column means. Re-run it to see what changed.
Discovery is how Halo gets to know a connection. It makes a catalog of what's there, so that when you build a dataset, Halo can suggest joins, tell dimensions from measures, and pick sensible formats.
It runs as a background job. It takes a minute or two on a small database, and you can leave the page while it runs.
What it collects
| What | Details |
|---|---|
| Tables and views | Schema, name and kind, with any comment from your database |
| Columns | Name, type, whether it allows empty values, whether it's a primary key, and any comment |
| Size | Estimated rows and size on disk |
| Relationships | Foreign keys your database declares, and ones Halo infers |
| Profiles | For each column: distinct and empty counts, lowest and highest values, most common values and a distribution |
| Meaning | A semantic type for the column |
Where to see it
Open a connection under Data:
Tables lists what it found, with each one's description, rows, size and when it was profiled. Its search also finds dbt tags and owners.
Open a table to see its Columns, with each one's Type, Meaning, Distinct and Empty. Open a column for its profile.
Relationships lists joins between tables, and how each was found.
Discovery history lists every run.
If you use dbt, load your project too. Discovery then takes descriptions, keys and joins from it, and a table's page shows its tests and lineage.
Profiles
Discovery profiles tables, but not views, since profiling a view would run its query.
Tables over 100,000 rows are sampled, and those profiles say so: "Figures come from a sample of the table, so they're estimates." A table gets one minute. If profiling fails or runs out of time, the run carries on, and the job's steps say which table it was.
Profiles hold real values from your data, such as a column's most common values. Your access to the connection decides who can see them.
Semantic types
From a column's name, type and profile, discovery guesses what it means. Halo's apps use the guess to choose formats, map layers and how a number adds up.
| Meaning | For example |
|---|---|
| Date, Date and time | order_date, created_at |
| Currency, Percent | revenue, conversion_rate |
| Country, US state, ZIP code, DMA | Columns you can place on a map |
| Latitude, Longitude | Points on a map |
| Email, URL | contact_email, landing_page |
| Identifier | customer_id |
| Category | channel, region |
| Measure | orders |
For uploaded files, a currency or percent symbol in the values decides it. See Upload CSV and Excel files. You can change how a field is treated in the dataset: see Fields.
Relationships
The Found by column says where each one came from:
Foreign key: your database declares it.
dbt test: a
relationshipstest in your dbt project.Inferred, with a percentage: there's no declared key, but the names line up and the values overlap. At least 90% of a column's values must exist in the other column before Halo suggests it, and the percentage is how well they match.
Uploaded tables get relationships the same way. When you pick several tables for a dataset, Halo joins them along these relationships. See Build a dataset.
Running it again
Discovery runs when you choose Save and discover on a new connection, and after every file upload. To run it again, choose Run discovery on the connection's page, or from its row under Data. You'll need a role that can manage connections: Owner, Admin or Builder.
To keep the catalog up to date without thinking about it, open the connection's Settings tab, and under Scheduled discovery set Schedule to Every day or Every week. It's Off by default.
What changed
Each run compares what it found with the catalog before it, and Discovery history shows the result:
The run's status, and how many tables, columns and relationships it found.
First run, the first time.
Tables added, Tables removed, Columns added, Columns removed and Columns changed, with their names. A changed column shows its old and new types.
Or "No schema changes since the run before."
It also lists the datasets and charts each change affects, and anyone whose dataset breaks gets the notification A source change breaks a dataset.
View job opens the run's job, with its log. A run that found changes logs a warning there too.
Discovery also works as a thorough connection test, so a run updates the connection's status and server version.
Descriptions from Halo AI
After discovery, Halo AI proposes descriptions for tables and columns that have none, working from their names, types and profiles. They show on the connection's Tables tab and on each table's page, for you to Accept, Edit or Reject, or Accept all and Reject all. Nothing counts as a description until it's accepted.
This uses AI credits, and each discovery writes more for what's new. To stop it, turn off Catalog descriptions in Settings, Halo AI. See AI credits and limits.
A table or column's description comes from whichever has words first: yours, by hand or accepted from Halo AI, then dbt's, then your database's own comment. For a dataset's fields, use Describe with Halo. See Fields.
Sample rows
On a table's page, Show sample rows reads the first 100 rows the source returns, which aren't a random sample. It takes the Write custom SQL permission, and each read is recorded in the audit log.