ElasticSearch Datasource Integration

With Knowi you can natively connect to data in your ElasticSearch cluster, create visualizations, perform joins across multiple indexes, use Knowi?s search-based analytics feature (also known as natural language querying), and more.


  1. Connect, extract and transform data from your ElasticSearch, using one of the following options:

    a. Through our UI to connect directly.

    b. Using our Cloud9Agent. This can securely pull data inside your network. See agent configuration for more details.

  2. Visualize and Automate your Reporting instantly.

UI Based Approach


  1. Log in to Knowi and select Queries from the left sidebar.

  2. Click on New Datasource + button and select ElasticSearch from the list of datasources.

  3. After navigating to the New Datasource page, either use the pre-configured settings into Cloud9 Chart's own demo ElasticSearch database or follow the prompts and configure the following details to set up connectivity to your own ElasticSearch database:
    a. Datasource Name: Enter a name for your datasource
    b. Elasticsearch URL: Elasticsearch URL to connect to.
    c. User ID: Enter the User ID to connect
    d. Password: Enter the password to connect to the database
    e. Custom Headers: Headers for additional connection properties. f. Version: Select Elasticsearch server version

  4. Establish Network connectivity and click on the Test Connection button.

    Note: The connection validity of the network can be tested only if it has been established via Direct Connectivity or an SSH tunnel. For more information on connectivity and datasource, please refer to the documentation on- Connectivity & Datasources.

  5. Click on Save and start Querying.



Step 1: Query using a visual builder or query editor

Visual Builder: After connecting to the ElasticSearch datasource, Knowi will pull out a list of indexes along with field samples. Using these indexes, you can automatically generate queries through our visual builder in a no-code environment by either dragging and dropping fields or making your selections through the drop-down.


Tip: You can also write queries directly in the Query Editor, a versatile text editor that offers more advanced editing functionalities like ElasticSearch JSON Query, support for multiple language modes, Cloud9QL, and more.

Step 2: Define data execution strategy by using any of the following two options:

  • Direct Execution: Directly execute the Query on the original Datasource, without any storage in between. In this case, when a widget is displayed, it will fetch the data in real time from the underlying Datasource.

  • Non-Direct Execution: For non-direct queries, results will be stored in Knowi?s Elastic Store. Benefits include- long-running queries, reduced load on your database, and more.

Non-direct execution can be put into action if you choose to run the Query once or at scheduled intervals. For more information, feel free to check out this documentation- Defining Data Execution Strategy


Step 3: Click on the Preview button to analyze the results of your Query and fine-tune the desired output, if required.


The result of your Query is called Dataset. After reviewing the results, name your dataset and then hit the Create & Run button.


Cloud9Agent Configuration

As an alternative to the UI based connectivity above, you can use Cloud9Agent inside your network to pull from ElasticSearch securely. See Cloud9Agent to download your agent along with instructions to run it.


  • Pull data using SQL.
  • Execute queries on a schedule, or, one time.

The agent contains a datasource_example_elasticsearch.json and query_example_elasticsearch.json under the examples folder of the agent installation to get you started.

  • Edit those to point to your database and modify the queries to pull your data.
  • Move it into the config directory (datasource_XXX.json files first if the Agent is running).

Datasource Configuration:

Parameter Comments
name Unique Datasource Name.
datasource Set value to elasticsearch
url URL to connect to, where applicable for the datasource. Example for ElasticSearch: localhost:3306/test
userId User id to connect, where applicable.
Password Password, where applicable
userId User id to connect, where applicable.

Query Configuration:

Query Config Params Comments
entityName Dataset Name Identifier
identifier A unique identifier for the dataset. Either identifier or entityName must be specified.
dsName Name of the datasource name configured in the datasource_XXX.json file to execute the query against. Required.
queryStr ElasticSearch query to execute. Required.
frequencyType One of minutes, hours, days,weeks,months. If this is not specified, this is treated as a one time query, executed upon Cloud9Agent startup (or when the query is first saved)
frequency Indicates the frequency, if frequencyType is defined. For example, if this value is 10 and the frequencyType is minutes, the query will be executed every 10 minutes
startTime Optional, can be used to specify when the query should be run for the first time. If set, the the frequency will be determined from that time onwards. For example, is a weekly run is scheduled to start at 07/01/2014 13:30, the first run will run on 07/01 at 13:30, with the next run at the same time on 07/08/2014. The time is based on the local time of the machine running the Agent. Supported Date Formats: MM/dd/yyyy HH:mm, MM/dd/yy HH:mm, MM/dd/yyyy, MM/dd/yy, HH:mm:ss,HH:mm,mm
c9QLFilter Optional post processing of the results using Cloud9QL. Typically uncommon against SQL based datastores.
overrideVals This enables data storage strategies to be specified. If this is not defined, the results of the query is added to the existing dataset. To replace all data for this dataset within Knowi, specify {"replaceAll":true}. To upsert data specify "replaceValuesForKey":["fieldA","fieldB"]. This will replace all existing records in Knowi with the same fieldA and fieldB with the the current data and insert records where they are not present.


Datasource Example:


Query Example:

        "entityName": "ElasticSearch Demo",
        "queryStr": "{\"size\":1000,\"query\": {\"query_string\": {\"query\": \"Transactional*\"}}} ",
        "c9QLFilter": "select sum(sent) as sent, sum(opened) as Opened, date(date) as Sent Date group by date(date)",
        "dsName": "demoElasticSearch",
        "overrideVals": {
            "replaceAll": true

The first query is run every 10 minutes at the top of the hour and replaces all data for that dataset in Knowi. The second is run once a day at 07:20 AM and updates existing data with the same Type field, or inserts new records otherwise.

Query Elasticsearch Template

Knowi supports querying Elasticsearch templates by allowing the end-user queries to pass as a parameter into the search template. This prevents your query-building logic from being exposed to the end user.

Elasticsearch templates make it easy to convert the user input into Elasticsearch queries. Simply modify the template for id and parameter(s) based on your requirements and run the query against it.

Knowi automatically detects Elasticsearch templates in the code editor and allows you to query it as shown below: