It happens, especially if you are visiting libraries or other venues doing research.
We have all or part of our genealogy data on these drives and an unscrupulous finder may have a treasure trove of information to mine for nefarious purposes.