I'm trying to compress some raw sensor data from older recordings that I need and don't want to remove from my Centos server. The data recorded is in a proprietary format, but for all intents and purposes we can characterize it as 306 channel PCM 32-bit audio recordings at 1000 Hz. With a few hundred lines of clear text meta data in the header of the file. Files range from 100mb to 1.9GB in directories around 7GB, processed file directories can be up to 60GB containing copies of raw data with filters applied to data.Here is the weird bit. I can compress the raw data down to 30% of original size with bzip2 and 26% of original with pxz.Similar results with lzrip using ZPAQ. But processed data PCM 32bit variable I can only shave 10 to 12 percent off. 16 bit short processed data, I can compress it to about 50% of original size. Processing data simplifies the recording data and there is less variation in the recording.Any suggestions? Any one have something similar? I want as much space saving as possible on the processed data, and plan to check the data after to make sure it has no errors. Any idea why simplified data out from processing is less compressible than raw? //edit - looking at FLAC but converting back to original format may be problematic. Not impossible .. Still looking.//solution edit. FLAC did not like 370 channels of data. But I was able to gain some compression by creating my own large dictionary and training it with segments of the large files.
Weird compression question
centos;compression;file comparison
Do you have documentation about the format? Write a small program to convert original format into a PCM wav + metadata +3d data, and back. If it's a superposition of signals on different frequencies, it should compress well with lossless audio compression algorithms, like FLAC.FLAC is like MP3: It compresses audio data by rewriting it into a different format. So it's not what I would call a wrapper (I wouldn't call any compression program a wrapper). But unlike MP3, the compressions is lossless: When you decompress, you get exactly the same data, just like in bzip2 etc. Compression ratio for FLAC on audio data is about 50%.There are various ways to store the metadata in the compressed audio, depending on the container format. It's also possible to just put all three files in an archive file, format of your chouce, though the compressed PCM data won't be compressed further.