I have a fairly large data set ~500 million rows. The data set looks like below. Col 1 is float number, col 2 is mac id(device id)1616.93,ac:22:0b:a6:22:c32872.32,c0:bd:d1:36:bb:493314.55,d4:0b:1a:39:19:b22854.11,18:f6:43:64:81:673540.68,18:f6:43:64:81:673856.91,ac:22:0b:a6:22:c32497.93,d4:0b:1a:39:19:b2The problem requires to output the ~100,000 random samples after grouping the col 2 and then finding the largest value from col 1 for that particular group.Intermediate output would look like(group by col1 ):1616.93,ac:22:0b:a6:22:c33856.91,ac:22:0b:a6:22:c32854.11,18:f6:43:64:81:673540.68,18:f6:43:64:81:672872.32,c0:bd:d1:36:bb:493314.55,d4:0b:1a:39:19:b22497.93,d4:0b:1a:39:19:b2After this, I require largest value from every grouped col. Output will look like below:3856.91,ac:22:0b:a6:22:c33540.68,18:f6:43:64:81:672872.32,c0:bd:d1:36:bb:493314.55,d4:0b:1a:39:19:b2The final step would be to have a truly random sample and the output will look like below:3540.68,18:f6:43:64:81:672872.32,c0:bd:d1:36:bb:49Any idea, how to perform this. I have just started working on linux and have no idea on how to perform such a mammoth task. Any help would be appreciated.
Random sampling and outputting the largest value
shell script;text processing;columns;random;statistics