MATLAB - Analyze Your CAN Bus Data [Parquet, MDF]
Need to analyze your CAN bus data in MATLAB?
In this overview, you'll learn how to load your CAN bus data into MATLAB - either as MF4 files (MDF) or DBC decoded Parquet files. We also outline key features of MATLAB's Vehicle Network Toolbox and Predictive Maintenance Toolbox.
How to log & interpret CAN bus data?
The CAN protocol is today used in practically all automotive vehicles (cars, trucks, buses, forklifts, ...) and in many industrial machinery applications. This makes CAN data important to many use cases, such as vehicle diagnostics, fleet management, predictive maintenance, academic research and more.
Further, several 'higher layer protocols' exist based on CAN, such as J1939 (heavy duty), OBD2 (cars), CANopen (industrial), NMEA 2000 (maritime), ISOBUS (agriculture) and more.
To record CAN bus data (incl. higher layer protocols), you can use a CAN data logger. For example, the CANedge1 lets you log CAN data to an SD card, while the CANedge2/CANedge3 also enable upload of the data to your own server via WiFi/LTE.
The devices record 'raw' CAN frames, which need to be decoded to physical values for analysis - typically via a DBC file. Once the CAN bus data is DBC decoded, you can perform e.g. graphical visualization and statistical analysis.
In some cases you may simply want to open a specific log file and analyze the data via a software GUI tool. For example, you may need to analyze a specific event or issue in a vehicle.
For such use cases, various GUI software tools can be used. For the CANedge, you can for example use the free asammdf GUI - or convert & load data in your favorite GUI tools:
asammdf GUI
This is a general-purpose GUI tool that lets you load MF4 log files, DBC decode them and plot the physical values.
asammdf intro
MF4 converters
If you prefer using software like CANalyzer, PCAN-Explorer etc. you can convert your MF4 files to the relevant format.
mf4 converters intro
Why use MATLAB for CAN bus data processing?
MATLAB is popular tool for analyzing CAN bus data amongst automotive OEMs (Original Equipment Manufacturers). In particular, MATLAB is popular for large-scale automotive fleet testing, algorithm development and predictive modelling.
Looking at a typical data analytics workflow, MATLAB offers several benefits:
Load and explore CAN bus data
Load CAN data from various log file formats (Parquet, MF4, MAT, ...) and locations (local, S3, ...). Data can be loaded as e.g. single file, datastores or tall arrays - for exploration via tables, stats & plots
DBC decode and preprocess data
DBC decode raw CAN data to physical values via the Vehicle Network Toolbox. Further, easily clean and pre-process your decoded CAN bus data via smoothing, filling, outlier handling, rescaling etc.
Develop models & analyses
Datastores and tall tables let you easily switch between analyzing a single device, trip or signal - to performing statistical analysis across fleets or months of data, incl. via e.g. machine learning
Deploy & integrate in your system
MATLAB offers the expertise and tools for deploying your CAN bus analytics, from development to production - for example to implement predictive maintenance models into your fleet operations
Why use the CANedge with MATLAB?
The CANedge is ideal for OEM engineers using MATLAB:
The hardware is low cost, meaning several devices can be deployed for e.g. prototype vehicle fleet tests. At the same time, it's extremely easy to work with your CANedge data in MATLAB - either by directly loading the MF4 log files, or by connecting MATLAB to your Parquet data lake of DBC decoded data.
learn about the CANedgeHow to load your CAN bus data into MATLAB
Directly load unfinalized MF4 logs from the CANedge
With MATLAB and the Vehicle Network Toolbox, you can directly load MF4 log files from the CANedge with raw CAN/LIN data. The MF4 file can then be finalized in MATLAB via a simple function call. This approach makes it easy to load specific log files in MATLAB for e.g. vehicle diagnostics and automotive testing purposes.
Load 'finalized' MF4 files - incl. via datastores
You can also use the Vehicle Network Toolbox to load 'finalized & sorted' MF4 files. To do this, you can use our MF4 converter to perform the finalization & sorting step prior to loading your data (a process which can be easily automated). You can also load finalized MF4 files into MATLAB datastores to easily work across large numbers of log files.
docs mf4 datastores script example
Load DBC decoded MF4 files - incl. via tall arrays
The VNT also lets you load MF4 log files that have been DBC decoded to physical values outside MATLAB. The DBC decoding can be done via the asammdf GUI or automated API scripts as per our examples. In this case, you can also use tall arrays, ideal for "big data" use cases - like fleet testing across 100+ devices where the terabytes of data do not fit into memory.
MF4 tall arrays many MF4 files
Load DBC decoded Parquet data lake [recommended]
You can use our MF4 decoders to easily create a Parquet data lake of DBC decoded CAN/LIN data - which can natively be processed within MATLAB (no toolbox required). MATLAB also supports Parquet tall arrays (stored locally or on S3) for out-of-memory operations - as well as powerful time resampling functions. This is our recommended integration method.
Easily load data from S3
If you're using the CANedge2/CANedge3, you may wish to load log files directly from your S3 server. This can be done in multiple ways: Within MATLAB, you can create a datastore from an S3 bucket based on DBC decoded MF4 or Parquet files. Alternatively, you can mount your S3 bucket as a local drive using e.g. TntDrive.
Example: Building a digital twin for an EV battery pack [webinar]
In this webinar, we work with Mathworks to showcase how the CANedge can be used with MATLAB to build a digital twin for an EV battery pack. The webinar covers the following topics:
- How to log raw CAN bus data and DBC decode it
- How to collect such data via WiFi/LTE to a cloud server
- How to integrate the data with MATLAB
- How to inspect/prepare the data for analysis
- A practical MATLAB demo of digital twin modeling
The project uses 1.5 years of CAN data recorded by a CANedge from an EV heavy duty material handler. The data is DBC decoded into a Parquet data lake, stored in S3 and loaded via MATLAB's powerful Parquet support.
MATLAB's Vehicle Network Toolbox
MATLAB's Vehicle Network Toolbox offers powerful features for working with your CAN data:
MF4 SUPPORT
Easily load MF4 log files from the CANedge
DATASTORES
Load folders / buckets and work with 'datalakes'
TALL ARRAYS
Easily analyze huge datasets (beyond memory)
LOCAL / S3
Load files / datastores directly from S3 servers
DBC DECODING
Decode raw CAN bus data using DBC files incl. J1939
INTEGRATE
Easily integrate with your existing systems
The Vehicle Network Toolbox makes it easy to DBC decode raw CAN bus data to physical values. You can either extract tables containing all decoded signals - or extract only specific signals for further analysis.
You can also perform decoding of J1939 data using e.g. our J1939 DBC file (with some minor tweaks as per our script examples).
MATLAB does not currently support transport protocol decoding e.g. for J1939 multi-frame messages, NMEA 2000 fast packets or UDS (Unified Diagnostic Services). If this form of decoding is required, we recommend that you use our Python API.
Example: MF4 CAN bus tall arrays
One of the powerful features of the Vehicle Network Toolbox is ability to load MF4 log files into MDF tall arrays. Effectively, this lets you analyze an unlimited number of log files with a few lines of code - regardless of the total file size.
Further, tall arrays can be plotted directly via MATLAB's visualization tools like plot, histogram and histogram2 - enabling quick visualization of e.g. months of data.
docs script example
Case study: Truck telematics
Learn how Cummins uses the CANedge2 and CANmod.gps to collect J1939 + GPS/IMU data to their own AWS S3 server for analysis in MATLAB's Vehicle Network Toolbox.
"Excellent functionality. Outstanding support. Continuous improvements. The CANedge2 is the best you can get!"
full case studyPredictive maintenance with the CANedge and MATLAB
In recent years we've seen an increasing demand amongst our users for deploying predictive maintenance across various assets in the field - e.g. cars, trucks, machines, battery systems etc.
In this section we briefly outline how you can approach this topic using the CANedge2/CANedge3 and MATLAB.
CANedge2 for predictive maintenance
The CANedge2 is an ideal device for predictive maintenance as it lets you easily record CAN/LIN data - and automatically push this to your own S3 server for processing.
The raw CAN data is recorded to an SD card, meaning that a periodic loss of connectivity does not result in loss of data (e.g. if vehicles operate in mines, tunnels, outside of WiFi range etc). Further, data can be uploaded via stationary WiFi access points using the CANedge2 or via 3G/4G using the CANedge3 (for on-the-road upload of data). Log files can be split in configurable chunks based on size or time - e.g. in 1 MB splits, 10 second splits, 30 min splits etc.
We encounter two common ways to utilize the CANedge in predictive maintenance:
#1 Predictive model training
When training predictive models (e.g. via machine learning), it is vital that the data sets are "rich" and "complete". Here, the CANedge2 is ideally suited as you can record "everything" from 2 x CAN/LIN - sometimes comprising 4000+ frames/second. This amount of data can correspond to 2 GB+ per day - making it too costly for 3G/4G transfer, but often fine for periodic offload via WiFi routers in e.g. a garage. Via the device configuration, you can of course optimize your logging via filters, prescalers, triggers and compression.
In this type of use case, the CANedge2 may be deployed in e.g. 100-300 vehicles over a period of 6-24 months. By collecting operational data (such as speed, RPM, temperatures) along with "failures" (e.g. diagnostic trouble codes), you can collect many TB of 'failure data' directly in S3 for e.g. machine learning models
#2 Operational deployment
The CANedge2/CANedge3 can also be used in production deployments of predictive maintenance models. Here, the upload frequency is typically increased (e.g. to 1 minute or 5 minute intervals) and the incoming data is often automatically decoded, analyzed and reacted upon through event triggered scripts (using e.g. AWS Lambda functions).
Use cases often start with basic predictive models based on threshold analyses: "Does parameter X go above limit Y for Z seconds, while parameter A is below limit B?".
This type of "practical" predictive maintenance is typically a good way to start and many of our users successfully predict e.g. battery fires in forklifts, tire issues etc. via such methods.
See also our separate intro to automotive predictive maintenance and MATLAB's "event detection" MDF example.
MATLAB's Predictive Maintenance Toolbox
As use cases scale and the sophistication requirements increase, it becomes increasingly important to train and deploy more advanced predictive models. Here, tools like the MATLAB Predictive Maintenance Toolbox can be extremely useful:
ESTIMATE RUL
Estimate an asset's time to failure via Remaining Useful Life (RUL) predictions
MODELLING
Use survival, similarity, trend based or ML models to estimate the RUL
INTEGRATE MF4
Easily integrate CAN data MF4 log files via the Vehicle Network Toolbox
QUICKSTART
Take outset in existing reference models for e.g. gearboxes, batteries, motors
DETECT ISSUES
Detect system anomalies and faults via e.g. control charts or Kalman filters
DEPLOY
Deploy predictive models to edge devices or cloud e.g. via generated C/C++ code
You can learn more about the Predictive Maintenance Toolbox here. MATLAB also provides a video intro to predictive maintenance.
As indicated above, predictive maintenance for CAN bus data can sound simple enough: Evaluate signal values vs. defined threshold limits to determine if an issue is about to occur. If an issue occurs, trigger an alert.
While simple models can be set up this way, practical deployment across a fleet of vehicles or machines will typically require more advanced modelling. Even basic things like proper data preprocessing can be make-or-break to avoid countless "false alert" situations.
Here, using the CANedge2 while partnering with MATLAB provides an excellent combination of low cost hardware and access to vital support and useful tools.
See e.g. practical examples for RUL estimation in a Wind Turbine Bearing, fault detection in a DC motor or the estimation of battery degradation via Kalman filters.
FAQ
We provide basic support in regards to loading your CAN bus data into MATLAB (see our script examples). Further, we can typically answer basic questions and we're always happy to provide high level sparring on your use cases.
For more technical support on your MATLAB script, we recommend to reach out to the MATLAB support team directly, as their experts will be better suited to diagnose any issues and help you quickly progress.
You can load MF4 log files from the CANedge and replay them within the MATLAB/Simulink environment, see this article for details.
As per our Python API article, we also offer a Python API for use with the CANedge CAN bus data loggers. The API lets you do many of the same things as MATLAB and Python and our API are 100% open source and free.
However, if you're going to perform advanced analyses or if you're already familiar with MATLAB, you'll probably be better off going with MATLAB. In particular, this route has the major benefit of enabling support and expertise from the MATLAB team. When it comes to e.g. data preprocessing and predictive modelling, it's not only a matter of 'whether' something can be done in e.g. Python or MATLAB - but rather 'how' it can be done. Implementing advanced models in the best way can be hard to do on your own - and here MATLAB's team of experts can help provide outside-in sparring based on similar experiences across other automotive end users.
We also have users that start with simple use cases in Python and later on move to MATLAB. At the end of the day, you can decide on your own as the CANedge and related MF4 data is designed to be interoperable - letting you manage your data in the way you prefer.
Ready to process your CAN data in MATLAB?
Get your CANedge today!
