CAN Bus Dashboards - Visualize Data Lakes [Grafana-BigQuery]

Below we provide additional details on each core element of the integration.

Tip: You may also try our step-by-step integration guide with free MF4/DBC sample data now.

Cloud Function automation

Google Cloud Functions help automate the DBC decoding of new data:

• The Cloud Function runs every time a new log file is uploaded
• It loads relevant DBC files from your input bucket to decode the raw CAN/LIN data (using our MF4 decoders)
• The resulting Parquet files are written to an output bucket
• Cloud Functions are 100% serverless - you only pay when a log file is processed
• They are fully scalable, from 1 device to 1000+ devices
• All data processing is done automatically (and instantly) upon upload

Output bucket Parquet data lake

The output bucket stores your decoded data as a 'Parquet data lake':

• The data lake consists of Parquet files
• Parquet files are grouped by device, CAN message and year/month/day
• As a result, the data lake is compact and highly efficient
• The data lake is multi-purpose and can be used directly in e.g. Python/MATLAB
• Data lake storage is 95%+ cheaper than using a database

Example: InfluxDB Cloud storage cost is ~1.44$/GB/month vs. Google Cloud bucket storage cost of ~0.023$/GB/month.

Google BigQuery interface

Grafana can visualize your Parquet data lake by using the Google BigQuery plugin:

• BigQuery provides a simple SQL interface to Parquet data lakes stored in Google buckets
• Further, it can extract results from GBs of data in seconds via parallelization
• BigQuery is serverless - you only pay for the data scanned as a result of Grafana queries
• The cost of using BigQUery is extremely low at ~5$/TB (i.e. 1-10$/year for most use cases)

See also our Grafana-Athena intro for more details on why we recommend to use Parquet data lakes as the data source.

The CANedge2/CANedge3 enables you to upload recorded log files to your own server (self-hosted or cloud) - including Amazon, Google and Azure cloud. Dashboard visualization and Parquet data lakes are core tools for enabling CAN telematics at scale - and we therefore provide step-by-step integration guides for the top 3 clouds.

Below we briefly outline the three integrations:

Grafana-Athena: Most of our users connect their CANedge2/CANedge3 to Amazon S3, which is our default recommended cloud. Here, the Grafana-Athena integration can be used. It offers the simplest deployment, widest set of functionality (e.g. automated trip summary aggregation) and most efficient use of e.g. date partitioning. Data is uploaded to S3, automatically DBC decoded by a Lambda function and output to a Parquet data lake stored in another S3 bucket. This data lake is then queried via the Amazon Athena SQL interface - e.g. by Grafana.

Grafana-BigQuery: Google Cloud offers a nearly identical experience to Amazon as it provides a native 'S3 interoperability' functionality. Further, the Grafana-BigQuery dashboard integration is nearly 1:1 identical to Grafana-Athena. Uploaded data is automatically DBC decoded via Azure Functions and output to a Parquet data lake stored in another bucket. This data lake is then queried via the BigQuery SQL interface - e.g. by Grafana.

Grafana-Synapse: Azure does not provide a native S3 interface, but you can deploy an 'S3 gateway' (Flexify) to enable the CANedge to upload data to your Azure Blob Storage container. Uploaded data is automatically DBC decoded via Azure Functions and output to a Parquet data lake stored in another container. This data lake is then queried via the Synapse SQL interface. The data can then be visualized in Grafana using the Microsoft SQL Server data source.