Reply To: .shr file format specification

I find that Signal Hound files quickly exceed the memory limits of python, especially on a PC where most python installations are still limited to 32-bit. You won’t even be able to load a 1 GB SHR file completely into python on a PC, most likely.

What I do when I need to process large files is process each spectrum sweep as it is read in. For example, if I need total power across the spectrum, I sum the power across the bins, and save just that number. Then I read the next sweep, etc. I don’t try to read the whole file into memory.

If you aren’t able to do that because you actually need to operate on the full 3D array of frequency/time/amplitude, then at a minimum I recommend doing that on Linux or Mac with a large amount of memory, and still auto-saving reasonable-sized SHR files (1 GB is probably a good start). I don’t recommend doing this on a PC with a 32-bit installation of python. To keep things even more manageable for later processing, you could decimate in frequency and/or time as the files are read, unless you really need the full time/frequency resolution that’s in the SHR file. Also, keep in mind that the latest versions of Spike support decimation when saving files. It can, for example, save the average spectrum over 1-second intervals, or the max spectrum over each 100 sweeps, or various combinations. That’s a really useful feature to avoid huge file sizes when acquiring data over long time spans.

BTW, I tried the save channelized data feature on the new versions of Spike (same option screen as decimation), but it didn’t work for me. I’m not sure what happened. I have not created any scripts that support reading SHR files when channelization was used, because I didn’t get a chance to figure out what I was doing wrong to get any valid sample files. Probably my error somewhere.

BTW, I made a version of the python script that can deal with corrupted SHR files when Signal Hound crashes during data acquisition. It’s very simple — the code just avoids reading the last 2 or 3 sweeps, which are the only ones that are corrupted/incomplete when there is a crash. The rest of the data are still fine.