CAN Bus Signal Events/Alerts - Identify, Notify & Visualise

In the CANedge Intro, we describe step-by-step how you can customize the 'default' AWS Lambda function that is deployed as part of a Grafana-Athena integration.

In short, when you deploy Grafana-Athena, effectively you will use a tool called CloudFormation in AWS to auto-deploy various resources based on a JSON file provided in our guide. One of the resources is a Lambda function that will contain the CANedge MF4 decoder and a Python script. This Lambda function can be updated with your own custom script changes. In our guide we provide plug & play examples of 'custom functions' that you can use to extend the default behaviour - one of which is designed for event identification and notification.

Importantly, the event detection function is structured to ensure that you do not need to perform any custom Python coding to get started. Rather, you just need to define your 'event objects' to customize which events you wish to detect. You can add multiple event objects, which will make the script check these in sequence.

In the script you will also find pre-defined examples of event objects, e.g. to identify high-speed events, low tire pressure or active J1939 diagnostic trouble codes. The custom alert function is designed to be user-friendly, allowing you to just specify the event object details in a JSON-style form (as per below examples).

Normally, deploying custom scripted logic in Lambda functions would be quite cumbersome. This is because it can be quite complex to build your own custom Lambda function if you need to pull in various external Python libraries to work with the data.

Luckily, this is abstracted away for two reasons:

1. The use of the MF4 decoder executable within the Lambda takes care of all the MF4/DBC processing without relying on any Python dependencies, greatly simplifying the overall deployment
2. Once your data is stored as Parquet files, you can easily work with it through the use of just two popular Python libraries: Pandas and Pyarrow. While these are not by default available in the Lambda environment, their popularity ensures that well-maintained ARN layers exist, which allow you to use these libraries with minimal deployment effort

In turn, you can perform pretty much any form of statistical data manipulation within the Lambda function without the usual complexity of incorporating various libraries from scratch.

AWS Simple Notification Service is an AWS service designed to make it easy to application-to-application (A2A) or application-to-person (A2P) messages. The Lambda function deployed via the Grafana-Athena integration will have access to publish messages to a deployed SNS topic. End users can then 'subscribe' to the event SNS topic to be notified in their preferred manner (e.g. email, SMS, ...) when an event arises.

This methodology ensures that the notification aspect of the event detection script is kept simple, as all of the complexity is abstracted away through the use of AWS SNS. It also makes it very easy to e.g. add/remove subscribers over time.

You can control the frequency of MDF files being uploaded by the CANedge2/CANedge3 through the log file split size/time. For example, you may set up a CANedge3 to split files every 5 minutes. If an event arises within such a log file, you will of course face a 'delay' of up to 5 minutes due to the log file split time, plus a number of seconds for the file transfer and the Lambda/SNS execution.

In other words, you will face some delay from an event arising to the event notification being sent. However, for most use cases (like predictive maintenance), this level of delay is not a practical issue - and it will typically be negligible vs. the time it takes to react by the end user.

It may be tempting to set your file split size/time lower to minimize the delay. For example, you could in principle split files every 10 seconds. However, doing so will increase file size, data transfer costs, AWS costs - and reduce the efficiency of your Parquet data lake (due to a higher number of files). In addition, for some event types it can increase the risk of 'false negatives' as explained further below. We generally recommend using a more balanced split of 5-30 minutes for near-real time event detection use cases.

Yes, you can use the event detection Python script locally on your own PC (and in our guides we encourage you to do so during initial testing of your event objects). Of course, using a CANedge2/CANedge3 is necessary if you want to be immediately notified when an event arises in the field - and hence most of this intro takes outset in this device.

However, you can easily run the script on a locally stored Parquet data lake to detect your custom events and output this information locally. The same method is used if you e.g. wish to process a backlog of historical data (regardless of whether this originates from a CANedge1 or CANedge2/CANedge3).

Even if you use a CANedge1, however, you may want to deploy the Grafana-Athena integration by manually uploading your Parquet data lake to an S3 output bucket (using the processes described in the CANedge Intro), as doing so will allow you to deploy the event summary dashboard for easy browsing of your events. However, you can of course alternatively browse your event data in a more manual way using a local Parquet viewer like Tad, or through Python/MATLAB.