native connector
Data warehouses · Databricks SQL
Analytics on a Databricks SQL warehouse, for the people who do not write SQL.
Knowi connects to Databricks SQL with a workspace host, a SQL warehouse ID, a personal access token, a catalog and a schema, either through the UI or through the Cloud9Agent inside your own network. You build datasets in a visual builder or in native Databricks SQL, choose per query whether it runs live or on a schedule into Knowi's own store, join the result to data that never landed in the lakehouse, and publish it as a dashboard, an alert or a widget embedded in your own product.
Databricks SQL sits under Data Warehouses in the new datasource list.
A lakehouse holds what you decided to land in it. The question someone just asked usually needs one more thing.
Six fields, all of which you already have.
If the warehouse is running, the connection is a form. Databricks SQL sits under Data Warehouses in the new datasource list, and Test Connection validates the settings before you save.
- The workspace host, plus the SQL warehouse ID that decides which warehouse runs the queries.
- A personal access token, then the catalog and schema names, both set on the datasource.
- An optional connection string carries extra url parameters such as
readTimeout=1800&connectTimeout=30in seconds. - Or no inbound route at all: the Cloud9Agent runs inside your network and pulls from Databricks SQL, on a schedule set per query with
frequencyTypeandfrequency.
Not every dashboard view has to hit the warehouse.
Every query carries an execution strategy. It is a per-query decision, not a platform setting.
- Direct execution runs on the original datasource with no storage in between, so the widget fetches data in real time. That is what you want for anything operational.
- Non-direct execution runs once or at scheduled intervals and stores the result in Knowi's own data store. The documented benefits are long-running queries and reduced load on your database.
A dashboard on a stored result therefore serves views without re-running anything against the warehouse. What that is worth is arithmetic only you can do. Large results have a separate switch: External Links disposition, which allows fetching larger data but uses cloud storage.
The half of the question that never landed in the catalog.
Support tickets sit in a SaaS tool, billing state in Postgres, product events in MongoDB, and the question someone asked needs two of those next to a table in your catalog. The usual answer is another pipeline and a wait.
Knowi joins the same or disparate SQL and NoSQL datasources into one dataset. You map the join fields between two queries, for example customer=customer, and pick the join type in the Join Builder.
- INNER JOIN is the default when a join field is present, with LEFT OUTER, FULL OUTER, RIGHT OUTER and LOOP JOIN also supported. With no join keys the datasets are blended instead. Multiple keys combine with AND.
- Joins run in memory with a disk swap for larger datasets. For joins spanning millions of records the docs point you at the agent rather than the cloud UI, and join types accept hints such as
SORTEDorMULTI HASH RIGHT. - Joins can be defined in the UI or on the Cloud9Agent, so the blend does not have to leave your network, and Cloud9QL runs as the last processing step afterwards.
Databricks SQL
PostgreSQL
MongoDB
REST APIDatabricks is rarely the only source.
Each of these is queried in its own language, then joined to the Databricks result in one dataset.
Snowflake
MySQL
Redshift
BigQuery
SparkSee every supported datasource.
Who gets to ask a question of the warehouse.
On connect, Knowi pulls a list of tables along with field samples. Every path below produces the same saved dataset.
Visual Builder
A non-SQL user drags fields or picks from drop-downs and the query is generated for them, off the tables and field samples pulled on connect.
Query Editor
Anyone who does write SQL uses the Query Editor instead, which carries a Databricks SQL language mode and Cloud9QL for post-processing.
AI Assistant
Generates a query from plain English, explains an existing one, finds issues, and answers syntax questions.
Two facts belong on the page, not in a footnote. It uses OpenAI, and per the documentation only the question is sent to the OpenAI APIs, not the data. An administrator enables the integration first, and you can supply your own API key.
Three ways teams get analytics out of Databricks SQL.
All three are defensible. They break in different places.
Where this fits, and where it does not.
Databricks SQL is one source of several, the answers people want span it and something else, and the audience is wider than the group with warehouse access. You want analytics inside your product and refresh volume controlled per query.
Everything you need is already in the catalog, everyone asking questions writes SQL, and nobody outside the data team consumes the output. There the warehouse's own query surface is doing the job and a second tool is overhead. The case for Knowi starts when the answer has to travel: to a non-technical team, to a customer inside your product, or across a boundary the lakehouse does not cross.
Databricks SQL connector questions.
What do I need to connect Knowi to Databricks SQL?
A datasource name, the Databricks SQL host, the SQL warehouse ID, a personal access token, the schema name and the catalog name. A connection string for additional url parameters is optional. Test Connection validates the settings before you save.
Does every dashboard view run a query on my Databricks SQL warehouse?
Only if you choose direct execution for that query, which fetches data in real time from the underlying datasource when the widget is displayed. Non-direct execution runs once or at scheduled intervals and stores the result in Knowi's data store, which the documentation cites as a way to handle long-running queries and reduce load on your database.
Can I join Databricks SQL data with MongoDB or a REST API in one dataset?
Yes. Knowi joins across the same or disparate SQL and NoSQL datasources into a single result. You map join fields between the two queries and pick the join type. INNER JOIN is the default when a join field is present, with LEFT OUTER, FULL OUTER, RIGHT OUTER and LOOP JOIN also supported.
Can Knowi reach a Databricks workspace that is not exposed to the internet?
Yes, through the Cloud9Agent. It runs inside your network and securely pulls data from Databricks SQL rather than requiring an inbound route, and it ships with a databricks.json datasource example and a matching query example to edit.
Can people who do not write SQL build their own Databricks queries?
Yes. Knowi pulls tables and field samples on connect, and queries are generated in a no-code environment by dragging fields or selecting from drop-downs. An AI Assistant also generates a query from plain English. It uses OpenAI, and per the documentation only the question is sent to the OpenAI APIs, not the data. An administrator enables the integration first.
How does Knowi handle large result sets from Databricks SQL?
Use External Links disposition, which allows fetching larger data but uses cloud storage. Raise readTimeout and connectTimeout in the connection string. Or run the query on a schedule with non-direct execution so the heavy read happens once rather than on every view.
Point Knowi at your SQL warehouse.
Connect through the UI with a host, warehouse ID and token, or run the Cloud9Agent inside your network. Then decide, query by query, what runs live and what runs on a schedule.
Databricks is rarely the only source. Knowi also connects Snowflake, PostgreSQL and MongoDB, and joins them to this one. See every supported datasource.