I type mosquitto_sub -d -t +/# from the Ubuntu terminal to access the MQTT stream.Real output from the live MQTT stream is this:Sending PINGREQReceived PINGRESPSending PINGREQReceived PINGRESPReceived PUBLISH (d0, q0, r0, m0, 'm/gf/TMX6BP/075/d/SVlts', ... (28 bytes))86,1224830,27610 27869 17565Received PUBLISH (d0, q0, r0, m0, 'm/gf/TMX6BP/075/d/status', ... (39 bytes))86,1243000,164573,-33.836465,151.051189Sending PINGREQReceived PINGRESPReceived PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/SVlts', ... (26 bytes))806,3040421,7549 7750 3904Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/status', ... (39 bytes))806,3069000,59666,-33.836465,151.051189Sending PINGREQReceived PINGRESPSending PINGREQReceived PINGRESPSending PINGREQReceived PINGRESPSending PINGREQReceived PINGRESPReceived PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/SVlts', ... (26 bytes))810,5440995,6143 7807 4076Sending PINGREQReceived PINGRESPReceived PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/status', ... (39 bytes))810,5489000,59897,-33.836465,151.051189Sending PINGREQReceived PINGRESPThere is no way of predicting when the next PUBLISH will be seen as they are only seen in the stream when the vehicle has transmission/reception from the GSM/3G towersTo filter I add mosquitto_sub -d -t +/# 2> >(grep PUBLISH) this will only allow lines with PUBLISH in it, hence the ouput is:Received PUBLISH (d0, q0, r0, m0, 'm/gf/TMX6BP/075/d/status', ... (38 bytes))86,637999,164563,-33.836465,151.051189Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/SVlts', ... (26 bytes))806,3040421,7549 7750 3904Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/status', ... (39 bytes))806,3069000,59666,-33.836465,151.051189Received PUBLISH (d0, q0, r0, m0, 'm/gf/TMX6BP/075/d/SVlts', ... (28 bytes))86,1224830,27610 27869 17565Received PUBLISH (d0, q0, r0, m0, 'm/gf/TMX6BP/075/d/status', ... (39 bytes))86,1243000,164573,-33.836465,151.051189Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/SVlts', ... (26 bytes))806,3640483,7463 7721 3933Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/status', ... (39 bytes))806,3674000,59676,-33.836465,151.051189Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/SVlts', ... (26 bytes))806,4240543,7291 7750 3933Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/status', ... (39 bytes))806,4279000,59687,-33.836465,151.051189Received PUBLISH (d0, q0, r0, m0, 'm/gf/MXE/065/d/SVlts', ... (25 bytes))455,24715,28041 28041 967How would I be able to eliminate a few of the fields and also add a time stamp everytime I receive something; I 've tried using sed but had no luck. I entered $ mosquitto_sub -d -t +/# 2< <(grep PUBLISH) 2< <(sed s/^/ date/) , $ mosquitto_sub -d -t +/# 2< <(grep PUBLISH) 2< <(sed s/^/$date`/)Q: How can I change my input to the terminal so that the output from the above live feed would be:[timestamp],m,gf,TMX6BP,075,d,status,86,637999,164563,-33.836465,151.051189[timestamp],m,NWRL,TMX,098,d,SVlts,806,3040421,7549 7750 3904[timestamp],m,NWRL,TMX,098,d,status,806,3069000,59666,-33.836465,151.051189[timestamp],m,gf,TMX6BP,075,d,SVlts,86,1224830,27610 27869 17565[timestamp],m,gf,TMX6BP,075,d,status,86,1243000,164573,-33.836465,151.051189[timestamp],m,NWRL,TMX,098,d,SVlts,806,3640483,7463 7721 3933[timestamp],m,NWRL,TMX,098,d,status,806,3674000,59676,-33.836465,151.051189[timestamp],m,NWRL,TMX,098,d,SVlts,806,4240543,7291 7750 3933[timestamp],m,NWRL,TMX,098,d,status,806,4279000,59687,-33.836465,151.051189[timestamp],m,gf,MXE,065,d,SVlts,455,24715,28041 28041 967Possible solutions (future referencing):Using the mosquitto_sub -d -t +/# 2> >(sed -n s|.*\('.*',\).*|\1|p) | sed N;s/\n/ /;s/$/ $(date)/ The output is:0 810,5440995,6143 7807 4076 Wed Feb 25 23:23:51 UTC 2015 810,5489000,59897,-33.836465,151.051189 810,6041055,7606 7693 4076 Wed Feb 25 23:23:51 UTC 2015Using the mosquitto_sub -d -t +/# 2> >(grep PUBLISH) | sed N;s/\n/ /;s/$/ $(date)/ command from the terminal the output is 817,3069000,60045,-33.836465,151.051189 609,24570,27553 27553 955 Thu Feb 26 00:06:26 UTC 2015Using the mosquitto_sub -d -t +/# 2>&1 | sed -n /PUBLISH/{s|.*\('.*',\).*|\1|;N;s/\n/ /;s/$/ $(date)/;p} The output is 'm/gf/MX3/122/d/status', 610,33000,28162,-33.836465,151.051189 Thu Feb 26 01:18:17 UTC 2015
How can I remove a carriage return, add a time stamp and ignore some data from a live MQTT feed
text processing;streaming;data
Well, rather confusing but anyway... Judging by the output ofmosquitto_sub -d -t +/# 2> >(grep PUBLISH)your app seems to output to both stderr and stdout (otherwise you should only get lines matching PUBLISH in your output). It prints the debug messages (Sending... and Received...) to stderr and the actual data (810,5440995,6143...) to stdout.  Apparently, you need comma separated values so you could try the following, if you need the timestamp from the line matching PUBLISH:mosquitto_sub -d -t +/# 2>&1 | xargs -d$'\n' -L1 sh -c 'date +%s,$0' | \sed -n /PUBLISH/{N;s|[ /]|,|g;s|^\([^,]*,\)[^']*'\([^']*\)',.*\n[^,]*,\(.*\)|\1\2,\3|;p}or, if you need the timestamp from the next line:mosquitto_sub -d -t +/# 2>&1 | xargs -d$'\n' -L1 sh -c 'date +%s,$0' | \sed -n /PUBLISH/{N;s|[ /]|,|g;s|^[^,]*,[^']*'\([^']*\)',.*\n\([^,]*,\)\(.*\)|\2\1,\3|;p}2>&1 redirects stderr to stdout, the output is then piped to xargs which passes each line as an argument to the next command sh -c 'date +%s,$0' so each line is prepended with a timestamp+comma, e.g.:[timestamp],Sending PINGREQ[timestamp],Received PINGRESP[timestamp],Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/status', ... (39 bytes))[timestamp],871,40114,4536 4536 323This is then piped to sed suppressing the automatic printing (-n).For each line matching PUBLISH, append the Next line, replace each space and / with comma then via grouping, retain only the first or second timestamp, the values in between quotes and the values after the second timestamp and finally, print the result:[timestamp],m,NWRL,TMX,098,d,status,871,40114,4536,4536,323