Q:

What is the best way to compress my data in lmdb

I have a large dataset which makes my lmdb huge. For 16.000 samples my database is already 20 GB. But in total I have 800.000 images which would end up in a huge amount of data. Is there any way to compress an lmdb? Or is it better to use HDF5 files? I would like to know if anyone knows probably the best solution for this problem.

A:

If you look inside ReadImageToDatum function in io.cpp it can keep image in both compressed(jpg/png) format or raw format. To use compressed format you can compress the loaded image using cv::imencode. Now you just set the datum to the compressed data and set the encoded flag. Then you can store the datum in lmdb.

