Read Ripple Neuroshare Format Files with J

Use J Memory Mapped Library

Neuroshare is an open file format that was adopted by a number of neural stimulation/recording hardware/software tool vendors (here a list) to save large experiment data files.

Ripple is one of these vendors. Their Trellis software supports the file format with their own tailored specifications, which can be viewed (from here).

Often a single data file can be hundreds of MBytes large, and there can be hundreds of these files saved from one experiment session.

Researches rely on MATLAB API libraries (or occasionally Python scripts) to load/process these Neuroshare data files. Ripple provides these MATLAB libraries along with their Trellis software.

On the other hand, J is a free and much light weight software tool, in comparison with MATLAB. J is able to memory map a file on disk into an array in the memory space, which makes it a perfect alternative tool to quickly examine and extract information from large data files, such as Ripple’s Neuroshare files.

Ripple NEV and NS5 Files

Data files are saved from Ripple Trellis software with extensions, such as .nev or .ns5. These file formats are defined in Ripple’s NEV file specification (so far V2.2) (download here).

To highlight the “complexity” of Neuroshare file formats

  • the data files are in binary format, with variable number of bytes encoding strings, unsigned and signed integers etc.
  • the important information such as sample rate, electrode ID are saved in header tables with different length fields.
  • raw data are saved as data packets, again with variable length, which in some cases depend on the number of recording channels.
  • data packets are time stamped, but mixed together as neural spike or stimulation events steaming in.

In short, it is difficult to extracting information from these large files, unless you read the specification very carefully.

Knowing that ambitious graduate students often like to write their own software, it is a little disappointing to see that facing with this Neuroshare file format, many have succumbed to relying on existing MATLAB function calls.

But it doesn’t have to be the case. As demonstrated in the J code below, only a few lines of code with enough magic numbers, one can extract a lot of information from the data files directly, and quickly,

J Code to Read NEV Files

For example, let’s read a sample nev file provided by Ripple (download here).

First load the J code above, saved in ns_open.ijs in J’s temp folder

load jpath '~temp/ns_open.ijs'

Again assuming Ripple’s NEV file is in the same folder. Here is the J function call to read the header information:

r =. read_nev_file jpath '~temp/sample_data_set.nev'
Image for post
Image for post

Here are the salient pieces of information from the main header:

  • This file has 427408 bytes, and the data section starts at 5616 byte mark.
  • Each data packet is 112-byte long (fixed length of all data packets in this file). You can calculate the total number of data packets = (427408–5616)/112
  • There are 165 extended headers that provide additional configuration information for each electrode ID. Since each extended header takes 32 bytes (also fixed length), and there are 165 of them in this case. It explains that the following data packets start at 5616 = 32 * 165 + 336 (main header) bytes.
  • In this case, 165 extended headers (tables) consist of 64 NEUEVWAV tables, 32 NEUEVFLT tables, 5 DIGLABEL tables and 64 NEUEVLBL tables. Eech extended header/table describes information such as bytes per sample for different electrode IDs, which is like the primary key in a database table.

The following data packets are 112-byte long for each. They are also tagged with “header” information: 4-byte timestamp, 2-byte packet ID (or electrode ID). Again this 2-byte ID is the same key in the extended header table. It links the data packets with the extended headers.

The return value from calling read_nev_file is a list of data address along side the electrode ID. These are extracted from 3766 data packets.

For example, to plot the raw waveform of electrode ID = 15, which is at address 5952.

  3{r5952 155952 plot_nev_file jpath '~temp/sample_data_set.nev'
Image for post
Image for post

J Code to Read NS5 Files

The Ripple NS5 file has a similar structure of “header + data packets”. In this case, J function read_ns_file reads the NS5 file

'base total eid'=. read_ns_file jpath '~temp/sample_data_set.ns5'
Image for post
Image for post

Because this is the raw data file, its structure is simpler than NEV files. The extended header, again one for each electrode ID, is 66-byte long. The data packets are continuous block of 2-byte per electrode/per data point.

Again some simple math to verify the numbers: 300600(data points) * 2(byte per data points) * 6 (channel) = 3607200 bytes. The additional 3507919–3607200 = 719 bytes = 6 * 66 (extended header)+ 314 (main header) + 9 (data packet header).

To plot 300600 data points on one electrode ID might be too much. Here is the J code to sub-sample and plot the data

eid_addr =: (base + 2) +2* 6 * total (] * i.@<.@%) 100
eid_addr plot_ns_file jpath '~temp/sample_data_set.ns5'
Image for post
Image for post

Summary

J’s memory map file feature allows fast access of large data files, such as these Neuroshare files.

By diving into Neuroshare file format specification, we are able to examine the raw data more closely, which is quite important, since so many neuroscience research data are saved in this file format.

Now graduate student, it’s your turn to dig into Neuroshare files with your favorite languages.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store