MF4 (MDF4) Explained - A Simple Intro [+J1939/OBD2 Data]
What is an MF4 file (aka MDF4) - and why is it key to CAN bus data logging?
The ASAM MDF (Measurement Data Format) is a binary file format for recording e.g. CAN, CAN FD and LIN bus data. Today, the MDF format is the industry standard - ensuring interoperability across many CAN tools.
The CANedge supports the latest MF4 file format - and below we list benefits, basics and software/API tools. We also provide sample MF4 data (J1939, OBD2).
What is ASAM MDF?
MDF was developed in the 90's by Vector GmbH and Robert Bosch GmbH. Initially it was proprietary, but in 2002 it was made public - and in 2009 it was standardized by ASAM. The latest version, MDF 4.11, was released in 2014.
Today, the MDF file format is the de facto industry standard - and it is used in an extensive suite of CAN software, APIs - and hardware like the CANedge1, CANedge2 and CANedge3.
6 key benefits of MF4
Logging data as MDF4 files has several major benefits:
INTEROPERABLE
MDF is open, standardized & widely supported for CAN bus logging - ensuring interoperability across many tools, software and APIs
FAST WRITING
Logging CAN data is a challenge due to increasing data rates. MDF is designed for fast writing - enabling lossless recording of e.g. CAN FD
FAST READING
The binary MDF is index based and extremely fast to read, edit or e.g. DBC convert - making it ideal for databases and big data analytics
COMPRESSIBLE
MDF 4.1 supports compression, reducing file size drastically. This improves SD capacity, server costs & 3G/4G upload costs
RAW/PHYSICAL
MDF4 supports both "raw" bus logging and "physical" values - hence you can often use the same software/ APIs
SOFTWARE/API
The format is supported by free powerful software & APIs. It's e.g. easy to DBC convert MDF4 data and export it to CSV, MATLAB, pandas
These features make MDF ideal for e.g. cloud analytics and telematics - across CAN, CAN FD, LIN, J1939, OBD2, CANopen, NMEA 2000 and more.
MF4 file structure
In this section, we provide a quick overview of the basic MDF4 file structure - with outset in a CANedge2 raw CAN bus log file.
MDF4 files are comprised of "blocks" - most of which are in practice arbitrarily ordered. Via a system of internal "links", these blocks form a hierarchy.
This structure enables more flexible log files vs. e.g. CSV - ideal for e.g. mixing CAN, CAN FD & LIN or supporting both raw/physical data.
For technical details on MDF files, expand the below:
To visualize the MDF4 hierarchy in practice, we can take outset in a raw CAN bus log file. In the screenshot, we've used Vector's free MDF Validator tool to load a raw J1939 MDF4 log file (*.MF4) from the CANedge2.
This is in general a useful tool for understanding the basic structure of MDF files - and checking their validity.
As evident from the above, there are several key MDF blocks:
| Block | Description |
|---|---|
| Identification (ID) | 64 bytes block at the beginning of an MDF. Identifies file as MDF and specifies version |
| Header (HD) | Adds info after ID (e.g. log start time). Acts as file root and links to each Data |
| File History (FH) | Describes tools involved in creating the file (incl. e.g. the CANedge FW version) |
| Data Group (DG) | Contains measurement data incl. Data Blocks and Channel Groups |
| Data (DT) | Contains records (e.g. CAN frames). The CANedge stores all raw data in one DT |
| Channel Group (CG) | Describes the layout of records. The CANedge uses 8 CGs (2 per CAN/LIN channel) |
| Channel (CN) | A Channel Group contains channels that describe the data structure and signals |
Note: MDF "channels" should not be confused with "bus channels" (the CANedge e.g. has 2 x CAN and 2 x LIN "bus channels").
If you're logging raw CAN bus data via "bus channel" 1 on the CANedge, the data will be stored in Channel Group 1 and 2. Here, CG 2 enables the use of variable data length (necessary for e.g. CAN FD).
Within CG 1, you'll see the CN CAN_DataFrame. This is split in 7 parts/sub channels:
- BusChannel: The bus channel that the data is from
- ID: The CAN ID
- IDE: The ID type (regular, extended)
- DLC: The CAN frame DLC
- DataLength: The actual data length (may differ from the DLC for CAN FD)
- DataBytes: The data bytes of the CAN frame
- Dir: The direction (received, transmitted)
The CANedge writes log files as MDF 4.11 using the 'bus logging' concept to create log files with raw CAN/LIN data.
To ensure fast writing, the CANedge appends every new CAN/LIN frame to a single Data Block. This results in an "unsorted" MDF file. To enable fast reading in e.g. your API processing you may choose to sort the MDF file before/during processing. Doing so in advance is not required (as most tools support unsorted MDFs) - but it enables fast indexing and hence faster processing.
If you sort a raw CANedge log file using e.g. the asammdf GUI/API, you'll note that the resulting MDF file is structured differently from before - see the MDF Validator example.
Instead of one Data Group, there is now a Data Group for each bus channel.
Further, there are no longer two Channel Groups per bus channel. Instead, there's a single Channel Group with two Data blocks (contained in a Data List).
This structure ensures that each Data Group contains Data blocks with records of identical layout - enabling fast index based reading.
In some cases you may want to trigger a small "sorting" script everytime a new raw log file is uploaded to your S3 server. This way you can store only sorted files on your server - enabling faster reading/processing of the files when needed.
You can also take this a step further and automatically DBC convert your raw MDF log files as they come into your server. This process will include the sorting step and enable you to directly fetch human-readable data for e.g. plots. For examples of how to sort and/or DBC convert your MDF data, see our API sample library and the examples below.
The below links can be useful in order to gain a further understanding of MDF:
- ASAM MDF Wiki - a great overview of the MDF basics
- ASAM MDF Standard - the full MDF standard provides all relevant details
Raw vs. physical (human readable) MDF4 data
When you connect the CANedge to your car, truck or other CAN/LIN application, you'll record raw bus data.
To work with the data, you'll need to convert it to human-readable data - aka "physical" or "scaled" values.
In order to convert your raw MDF files, you'll need a database of conversion rules - e.g. a DBC file (CAN database).
With a DBC, you can use e.g. asammdf (see below) to convert your raw data. The resulting MDF file contains a Data Group for each CAN ID and a Channel for each message signal.
You can then easily plot the decoded data in e.g. asammdf - or export it to csv/mat for analysis in Excel/MATLAB.
MDF4 software & API tools
So how do you actually manage your CAN bus MDF4 log files?
Below we list examples of software/APIs that you can use to manage your MDF4 data.
TIP: If you're new to MDF, we recommend the open source asammdf GUI/API and the simple MDF4 converters to get started.
asammdf (GUI)
MF4 converters
Our free open source MF4 converters let you easily convert raw CANedge data to popular formats:
- Output: ASC, TRC, CSV, ...
- Finalize & sort MDF4 files
- Drag & drop + CLI
- Easily use in scripts
- 100% open source
- Windows/Linux support
MF4 decoders
Our free open source MF4 decoders let you easily DBC decode raw CANedge data to popular formats:
- Output: DBC decoded CSV, Parquet
- Finalize & sort MDF4 files
- Drag & drop + CLI
- Easily use in scripts
- 100% open source
- Windows/Linux support
CANedge MF4 Python API
We provide an open source MF4 Python API:
- List MF4 log files (local/S3)
- Easily load MF4 log files
- Iterate or load into pandas
- DBC decoding
- Example script library
- 100% open source
- Windows/Linux support
- Dashboard integration
MDF Validator
This free MDF viewer from Vector lets you load raw/converted MDF files to view the structure and evaluate validity. Useful for understanding the MDF file.
- 100% free (requires your details)
- Windows support
- View MDF structure & meta info
- Check the validity of an MDF file
- Load raw & converted MDF files
Vector Tools (CANalyzer)
CANalyzer natively supports finalized & sorted MDF4 files.
- License based
- Comprehensive analysis tool
- Advanced graphical plots
- Load raw & converted MDF files
HBM Tools (nCode)
The nCode software is popular within equipment analysis, e.g. for durability testing and big data visualization.
- License based
- Comprehensive analysis tool
- Advanced graphical plots
- Load raw & converted MDF files
asammdf (Python API)
The API behind the asammdf GUI tool provides a full suite of all the core operations you need to process both raw and converted MDF files at scale.
- 100% free & open source
- Load raw & converted MDF files
- Concatenate, filter, resample, ...
- DBC conversion (incl. J1939, OBD2)
- Extremely fast processing
- Graphical plots
- Export (CSV, pandas, MAT, HDF5, ...)
- Easily combine with S3 boto3 API
Vehicle Network Toolbox (MATLAB API)
Other
There are many more MDF software tools available for use with the MDF file format. Examples include tools from dSpace (AutomationDesk, ControlDesk), ETAS (ASCMO, INCA) and many more.
Example: J1939 MF4 data (tractor)
The CANedge is excellent for use with heavy duty vehicles.
In particular it's ideal for vehicle fleet telematics across e.g. trucks, transit buses, excavators, tractors.
The asammdf GUI/API supports decoding J1939 data via J1939 DBC files. To test this, download the sample data below (*.MF4 & DBC) and load it in the asammdf GUI.
J1939 telematics Sample data J1939 DBC
Example: OBD2 MF4 data (Audi A4)
Another popular use case is recording OBD2 data from cars.
Most cars support OBD2 data, letting you analyze e.g. speed, fuel consumption, brake patterns and diagnostics.
A challenge with OBD2 can be in converting the raw data. However, this is easily done with asammdf and our OBD2 DBC. Try it out via the sample files below:
OBD2 logging Sample data OBD2 DBCReady to log your CAN/LIN data in the MF4 format?
Get your CANedge today!
