There is an endless data stream of XML messages (and heartbeats), that I receive via a telnet connection and through a site-to-site VPN IPsec tunnel. I'm still pondering. What is the best/most elegant solution to process the XML messages without losing any data, without redundance and with a (nearly) constant processing time? A never-ending process/script? Writing the stream in file(s) and processing it/them periodically step by step? Or something completely different?The messages usually come every few seconds. Sometimes every second. Sometimes maybe every 10 seconds. It differs but not a lot. One XML message within the stream contains 45 rows. The messages should be stored afterwards.Note: The concrete structure of the XML messages and the infrastructure of the participating systems are negligible in my opinion.
How to process an endless XML data stream
xml;data;text processing
Event stream based is your best bet! in C I will use Expat. In Java, I will use SAX parser.