totally new to larger than ram datasets but I have csv files that are about 100 gb each with around 300 million rows each  (just two of them). What I am looking for is something like: don't go analyzing more than 1 terabyte of data on a laptop. Or don't analyze data that is more than 10x your ram or the wait times will prove frustrating. This is what I mean by rules of thumb. I have a mac laptop with an i5, 8gb ram, ssd is it reasonable to process this data (100gb per file and I have 2 files) with either dask or blaze in python?I have tried and could read in the csv but when doing simple trials like dropping one column or finding the length of the data frame it takes at least an hour For the drop (I gave up waiting) and half an hour for the length() (that did finish). Way too slow to be productive. I changed the data format to parquet and amazingly the data is down to just 10 gb. This is the only way they the length() command ever even finished. So my question is: are these response times normal experience in general given my hardware and csv size? Are my expectations for what dask can do way too high? Any rough guidelines on how to troubleshoot or is this clearly underpowered hardware
Need rules of thumb for out of core larger than ram dataset on a laptop
python;pandas