Knowi Elastic Store/Data Warehousing

Knowi provides an optional elastic store that can store and track query results. Unlike traditional warehouses that require complex ETL processes and pre-defined schema, the elastic store is is a flexible, scalable, schema-less warehouse.

This can provide a number of benefits, including:

  • Cache/persistent layer for fast analysis and visualizations
  • Leaves raw data at the source
  • Typically used to store query results, results of multi-datasource joins, etc.
  • Eliminates complex ETL processing
  • Does not require a predefined schema
  • Keeps multi-structured data intact
  • Shields your raw datasources from your BI/reporting queries
  • Build aggregations and data pipelines from datasets stored from this store

Note that this is optional. You can also use the direct query mode, that bypasses this data layer to interact with your raw datasource directly.


Data we receive is stored into a dataset based on a name (or a unique identifier). A dataset is a logical grouping of data that can seamlessly accommodate new fields when needed and can be combined from multiple datasources. A dataset is typically created from queries against a datasource, or, using our push API.


To query datasets stored within the warehouse:

  1. Create a new Knowi Warehouse Datasource (if one does not exist):
    Add Widget

  2. To query against the datasource, either:
         a. select dataset name and then enter Cloud9QL directly or,
         b. use data the Query Generator to dynamically generate the queries.
    Add Widget

  3. To set up a derived dataset from this dataset, set up the query and assign a schedule.

Triggered Datasets

Triggered datasets are dependent datasets that are auto updated when the parent dataset is modified.

For example: lets say that a query is run on the raw data spanning terabytes of data, with a query against it returning millions of records. The results of the query can be stored into the Knowi Elastic Store. Furthermore, any derived queries on the resulting dataset can be setup separately within Knowi, so that when the original query is updated, any dependent datasets will also be automatically updated.

To set up a triggered query, Use the Linked Dataset menu from the query list, define the query against the dataset and select the "Triggered Query" checkbox.

Add Widget

Cloud9 Agent

In addition to the UI based approach, you can also query Knowi using your agent. This can be used to query data in the warehouse, apply custom processors and build derived datasets. For added security, your API key needs to be pre-approved by Knowi (with an optional domain match) enable this functionality. Please Contact us to enable this.


  • Your datasets stored within Knowi can be queried and transformed.
  • Typically, used to aggregate data into another dataset.

Datasource Configuration:

Parameter Comments
name Unique Datasource Name.
datasource Set value to cloud9charts
exportApiKey Export API key from Knowi

Query Configuration:

Query Config Params Comments
entityName Dataset Name Identifier
identifier A unique identifier for the dataset. Either identifier or entityName must be specified.
dsName Name of the datasource name configured in the datasource_XXX.json file to execute the query against. Required.
c9ExportEntityName Entity Name of the dataset to pull data from.
queryStr Required. Cloud9QL query to manipulate data in Knowi. See Cloud9QL docs
frequencyType One of minutes, hours, days,weeks,months. If this is not specified, this is treated as a one time query, executed upon Cloud9Agent startup (or when the query is first saved)
frequency Indicates the frequency, if frequencyType is defined. For example, if this value is 10 and the frequencyType is minutes, the query will be executed every 10 minutes
startTime Optional, can be used to specify when the query should be run for the first time. If set, the the frequency will be determined from that time onwards. For example, is a weekly run is scheduled to start at 07/01/2014 13:30, the first run will run on 07/01 at 13:30, with the next run at the same time on 07/08/2014. The time is based on the local time of the machine running the Agent. Supported Date Formats: MM/dd/yyyy HH:mm, MM/dd/yy HH:mm, MM/dd/yyyy, MM/dd/yy, HH:mm:ss,HH:mm,mm
overrideVals This enables data storage strategies to be specified. If this is not defined, the results of the query is added to the existing dataset. To replace all data for this dataset within Knowi, specify {"replaceAll":true}. To upsert data specify "replaceValuesForKey":["fieldA","fieldB"]. This will replace all existing records in Knowi with the same fieldA and fieldB with the the current data and insert records where they are not present.

Datasource Example:


Query Example:

    "entityName":"Aggregated Data",
    "c9ExportEntityName":"Raw Data",
    "queryStr":"select * where a is not null and a < 10800 ; select sum(a) as Total A, sum(b) as Total B, \"Some Type\" as Type order by b desc",

In the example above, Raw Data dataset is queried for certain conditions, then aggregated and posted into an "Aggregated Data" dataset on a timed schedule.