asdfasdfsadf

MySQL Business Intelligence & Reporting

Knowi enables data discovery, query, aggregation, visualization and reporting automation from MySQL along with other unstructured and structured datasources.

Overview

  1. Connect, extract and transform data from your MySQL database, using one of the following options:

    • Using our Cloud9Agent. This can securely pull data inside your network. See agent configuration for more details.
    • Through our UI to connect directly.
  2. Visualize and Automate your Reporting instantly.

UI

If you are not a current Knowi user, check out our MySQL Instant Reporting page to get started.

Connecting

The following GIF image shows how to connect to MySQL.

MySQL Connect

  1. Login to Knowi and select Settings -> Datasources from the left down menu.

  2. Click on MySQL. Either follow the prompts to set up connectivity to your own MySQL database, or, use the pre-configured settings into Cloud9 Chart's own demo MySQL database.

    If you connecting through an agent, check Internal Datasource to assign it to your agent. The agent (running inside your network) will synchronize it automatically. Alternatively, configure datasource and queries directly into agent.

    When connecting from the UI directly to your MySQL database, please follow the connectivity instructions to allow Knowi to access your database.

  3. Save the Connection. Click on the "Configure Queries" link on the success bar.

Queries & Reports

  1. Set up Query to execute.

    MySql Query

    Report Name: Specify a name for the report.

    Queries can be auto-generated using our Data Discovery & Query Generator feature. You can also enter MySQL queries directly. The query results can be optionally post-processed using Cloud9QL, a SQL-like syntax.

    Data Discovery:

    i. Select a collection.

    ii. Select Metrics to track. Click on a metric to select aggregations (or date functions) on a given metric.

    iii. Select any optional dimensions to track. Dimensions are fields you want to group by.

    iv. Select any optional filters to set.

    Note that the queries are auto-generated when the Data Discovery feature is used.

    Queries:

    MySQL Query: Modify or enter SQL queries directly.

    Cloud9QL: Optional SQL-Like post processor for the data returned by the SQL query. See Cloud9QL Docs for more details.

    Click 'Preview' to see the results.

    1. Scheduling: Configure how often this should be run. Select 'None' for a one time operation. The results are stored within Knowi.

    2. Overwrite Strategy (for scheduled query runs):

      Overwrite Strategies determines how the data is stored in Knowi:

      i. If empty, data will be added on to the existing data for this dataset within Knowi. Or,

      ii. "All": Any existing data for this dataset will be replaced by this results.

      iii. One or More Field Names (Example: "A,B,C"): A new record is created where the values of the combination of the field names do not exist, and, updates current records for the field grouping where it exists. For example, if this is set to say "Date, Type", existing data with the same Date and Type values will be updated with the latest data, and new records created when they do not exist.

    3. Click on 'Save' to complete setting up the report. This also sets up this data extraction on a schedule, if configured.

    4. Click on 'Dashboards' to access dashboards. You can drag and drop the newly created report from the widget list into to the dashboard.


Cloud9Agent Configuration

As an alternative to the UI based connectivity above, you can use Cloud9Agent inside your network to pull from MySQL securely. See Cloud9Agent to download your agent along with instructions to run it.

Highlights:

  • Pull data using SQL.
  • Execute queries on a schedule, or, one time.

The agent contains a datasource_example_mysql.json and query_example_mysql.json under the examples folder of the agent installation to get you started.

  • Edit those to point to your database and modify the queries to pull your data.
  • Move it into the config directory (datasource_XXX.json files first if the Agent is running).

Datasource Configuration:

Parameter Comments
name Unique Datasource Name.
datasource Set value to mysql
url URL to connect to, where applicable for the datasource. Example for MySQL: localhost:3306/test
userId User id to connect, where applicable.
Password Password, where applicable
userId User id to connect, where applicable.

Query Configuration:

Query Config Params Comments
entityName Dataset Name Identifier
identifier A unique identifier for the dataset. Either identifier or entityName must be specified.
dsName Name of the datasource name configured in the datasource_XXX.json file to execute the query against. Required.
queryStr MySQL SQL query to execute. Required.
frequencyType One of minutes, hours, days,weeks,months. If this is not specified, this is treated as a one time query, executed upon Cloud9Agent startup (or when the query is first saved)
frequency Indicates the frequency, if frequencyType is defined. For example, if this value is 10 and the frequencyType is minutes, the query will be executed every 10 minutes
startTime Optional, can be used to specify when the query should be run for the first time. If set, the the frequency will be determined from that time onwards. For example, is a weekly run is scheduled to start at 07/01/2014 13:30, the first run will run on 07/01 at 13:30, with the next run at the same time on 07/08/2014. The time is based on the local time of the machine running the Agent. Supported Date Formats: MM/dd/yyyy HH:mm, MM/dd/yy HH:mm, MM/dd/yyyy, MM/dd/yy, HH:mm:ss,HH:mm,mm
c9QLFilter Optional post processing of the results using Cloud9QL. Typically uncommon against SQL based datastores.
overrideVals This enables data storage strategies to be specified. If this is not defined, the results of the query is added to the existing dataset. To replace all data for this dataset within Knowi, specify {"replaceAll":true}. To upsert data specify "replaceValuesForKey":["fieldA","fieldB"]. This will replace all existing records in Knowi with the same fieldA and fieldB with the the current data and insert records where they are not present.

Examples

Datasource Example:

[
  {
    "name":"demoMySQL",
    "url":"localhost:3306/test",
    "datasource":"mysql",
    "userId":"a",
    "password":"b"
  }
]

Query Examples:

[
  {
    "entityName":"Errors",
    "dsName":"demoMySQL",
    "queryStr":"select error_condition as 'Error', count 'Count' from errors",
    "frequencyType":"minute",
    "frequency":10,
    "overrideVals":{
      "replaceAll":true
    }
  },
  {
    "entityName":"Queues",
    "dsName":"demoMySQL",
    "queryStr":"select Name, size as 'Queue Size', Type from queue",
    "overrideVals":{
      "replaceValuesForKey":["Type"]
    }

  }
]

The first query is run every 10 minutes at the top of the hour and replaces all data for that dataset in Knowi. The second is run once a day at 07:20 AM and updates existing data with the same Type field, or inserts new records otherwise.

Advanced Example (multiple databases with wildcard database name matching):

The following example:

  • Connects to a set of MySQL databases using a wildcard database names.
  • Executes queries against all databases from that match group and combines the data.
  • Executes a Cloud9QL to further aggregate the data.
  • Connects and pulls data from a set of MySQL databases using a wildcard name match, to then combine and aggregate the data.
  • Resulting data from both MySQL and MySQL databases are stored into the same dataset.

Datasource:

[
  /* Wildcard token to connect to multiple databases with the same schema */
  {
    "name":"demoMySQLGroup",
    "url":"localhost:3306/app_${c9_wildcard}_somepostfix",
    "datasource":"mysql",
    "userId":"a",
    "password":"b"
  },
  {
    "name":"demoMongoGroup",
    "url":"dharma.mongohq.com:10071/cloud9${c9_wildcard}_acc",
    "datasource":"mongo",
    "userId":"x",
    "password":"y"
  }
]

Query:

[
  {
    "entityName":"Multiple Databases",
    "dsName":"demoMySQLGroup",
    /* Executes against multiple databases and combines the result */
    "queryStr":"select * from sometable",
    /* Optional c9QL that runs on the combined query results. */
    "c9QLFilter":"select count(*) as Total count, \"MySQL Counts\" as Type",
    "overrideVals":{
      "replaceValuesForKey":["Type"]
    }
  },
  {
    "entityName":"Multiple Databases",
    "dsName":"demoMongo",
    /* Runs a Mongo Query on all matched databases and combines them*/
    "queryStr":"db.pageviews.find({lastAccessTime: { $exists: true}})",
    /* Optional C9QL to Aggregate the data further*/
    "c9SQLFilter":"select count(*) as Total counts,\"Mongo Counts\" as Type",
    "overrideVals":{
      "replaceValuesForKey":["Type"]
    }
  }
]