[{"text_idx": "#000000", "text": "If you're inventing and pioneering, you have to be willing to be misunderstood for long periods of time"}, {"text_idx": "#000001", "text": "Apache-kafka and Atmel 328p + enc28j60"}, {"text_idx": "#000002", "text": "I put together a simple hardware"}, {"text_idx": "#000003", "text": "programmed it using C code:"}, {"text_idx": "#000004", "text": "/ Demo using DHCP and DNS to perform a web client request."}, {"text_idx": "#000005", "text": "/ ethernet interface mac address, must be unique on the LAN"}, {"text_idx": "#000006", "text": "char website[] PROGMEM = \u201cvm37.dbweb.ee\u201d;"}, {"text_idx": "#000007", "text": "/ called when the client request is complete\nSerial.print(const char*) Ethernet::buffer + off);"}, {"text_idx": "#000008", "text": "if (ether.begin(sizeof Ethernet::buffer, mymac) == 0)\nSerial.println( \u201cFailed to access Ethernet controller\u201d);"}, {"text_idx": "#000009", "text": "/ generate the header with payload \u2013 note that the stash size is used,\n/ and that a \u201cstash descriptor\u201d is passed in as argument using \u201c$H\u201d\nwebsite, PSTR(PATH), website, stash.size(), sd);"}, {"text_idx": "#000010", "text": "/ send the packet \u2013 this also releases all stash buffers once done"}, {"text_idx": "#000011", "text": "Uploaded code to MC and powered it up."}, {"text_idx": "#000012", "text": "And after it got data from DHCP I saw nice picture \u2013 it sends data into my kafka queue"}, {"text_idx": "#000013", "text": "This is simple demo about apache-storm in cluster environment. Watch it in full screen mode"}, {"text_idx": "#000014", "text": "A very simple apache-kafka cluster demo"}, {"text_idx": "#000015", "text": "Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design."}, {"text_idx": "#000016", "text": "Try this video in full screen."}, {"text_idx": "#000017", "text": "Storm has many use cases: realtime analytics, online machine learning, continuous computation, distributed RPC, ETL, and more. Storm is fast: a benchmark clocked it at over a million tuples processed per second per node. It is scalable, fault-tolerant, guarantees your data will be processed, and is easy to set up and operate"}, {"text_idx": "#000018", "text": "zookeeper on mo\u0303eldud hoidma teenuste seadistusi ja staatusi. Na\u0308iteks antud juhul on zookeeper serverites talletatud informatsioon, millised storm\u2019i workerid on olemas."}, {"text_idx": "#000019", "text": "Zookeeper teenus vo\u0303ib olla jaotunud eraldi serverite vahel, mis tagab ko\u0303rge veakindluse"}, {"text_idx": "#000020", "text": "Na\u0308iteks minu testkeskkonnas on u\u0308ks storm supervisor e worker ja hetkel on u\u0308ks topoloogia, see kajastub zookeeperis:"}, {"text_idx": "#000021", "text": "[workerbeats, errors, supervisors, storms, assignments]"}, {"text_idx": "#000022", "text": "Zookeeper vo\u0303imaldab stormi workereid du\u0308naamiliselt juurde lisada. Storm master e nimbus oskab zookeeper serverist saadud info kohaselt workereid kasutada. Na\u0308iteks, kui mo\u0303ni worker mingil po\u0303hjusel ei ole enam ka\u0308ttesaadav, siis zookeeper saab sellest teada, kuna heardbeate enam ei tule ja nimbus organiseerib voogude teekonnad ringi tekitades kadunud workeri asemel uue, eeldusel, et on kuhugile tekitada ehk on veel vabu supervisoreid."}, {"text_idx": "#000023", "text": "Storm has many use cases: realtime analytics, online machine learning, continuous computation, distributed RPC, ETL, and more. Storm is fast: a benchmark clocked it at over a million tuples processed per second per node. It is scalable, fault-tolerant, guarantees your data will be processed, and is easy to set up and operate."}, {"text_idx": "#000024", "text": "On master topoloogias, kes kordineerib, kasutades zookeeper-klastris olevat informatsiooni, storm-supervisorite to\u0308id ehk tagab voogude la\u0308bimise topoloogiast."}, {"text_idx": "#000025", "text": "Spout(id) ja/vo\u0303i Bolt(id), kes kuuluvad mingisse topoloogiasse. Vo\u0303ivad asuda u\u0308hes fu\u0308u\u0308silises serveris vo\u0303i jaotatud erinevate fu\u0308u\u0308siliste serverite vahel. Zookeeperi abil annavad nimbusele teada oma olemasolust."}, {"text_idx": "#000026", "text": "Etteruttavalt selgitan, et antud juhul on kasutusel kaks supervisor teenust, mis on erinevad ja mida on vaja lahti seletada."}, {"text_idx": "#000027", "text": "supervisor \u2013 Process Control System."}, {"text_idx": "#000028", "text": "On kasutusel, tagamaks, et teenused \u2013 nimbus, zookeeper, storm_supervisor (worker) oleksid kiirelt taastatud, kui mo\u0303ni neist peaks mingil po\u0303hjusel seiskuma."}, {"text_idx": "#000029", "text": "Hetkel on minu testkeskkonnas supervisor (mitte storm-supervisor) kontrolli all vajalikud storm teenused"}, {"text_idx": "#000030", "text": "storm-supervisor RUNNING pid 3483, uptime 2:14:55\nstorm_nimbus RUNNING pid 3765, uptime 1:44:23\nzookeeper RUNNING pid 3484, uptime 2:14:55"}, {"text_idx": "#000031", "text": "Peatades na\u0308iteks storm_nimbus protsessi 3765"}, {"text_idx": "#000032", "text": "2014-04-22 17:53:22,898 INFO success: storm_nimbus entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)"}, {"text_idx": "#000033", "text": "storm-supervisor RUNNING pid 3483, uptime 2:30:50\nstorm_nimbus RUNNING pid 4604, uptime 0:00:38\nzookeeper RUNNING pid 3484, uptime 2:30:50"}, {"text_idx": "#000034", "text": "On na\u0308ha, et just on uus protsess ka\u0308ivitatud."}, {"text_idx": "#000035", "text": "Toodangusu\u0308steemides on soovitatav jaotada storm komponendid nii, et nimbus, ui ja u\u0308ks zookeeper server on u\u0308hes masinas ja teistes asuvad zookeeper server ja storm-supervisor. Komplekte zookeeper-server ja storm-supervusor vo\u0303ib du\u0308naamiliselt hiljem lisada."}, {"text_idx": "#000036", "text": "Vahema\u0308rkusena, et tegelikult ei pea storm-supervisor ja zookeeper u\u0308hes fu\u0308u\u0308silises serveris asuma. Piisab, kui storm-supervisor teab, kus asub zookeeper server, et sinna oma staatus teatada."}, {"text_idx": "#000037", "text": "Kui mingil po\u0303hjusel peaks u\u0308ks storm-supervusor ka\u0308ttesaamatuks muutuma, siis nimbus saab sellest teada ja organiseerib topoloogia niimoodi, et voog oleks ta\u0308ielik."}, {"text_idx": "#000038", "text": "Kui mingil po\u0303hjusel peaks muutuma mitteka\u0308ttesaadavaks nimbus, siis topoloogia on terviklik ja vood ta\u0308a\u0308tavad edasi."}, {"text_idx": "#000039", "text": "Kui mingil po\u0303hjusel peaks muutuma korraga mitteka\u0308ttesaadavaks nimbus ja mo\u0303ni hetkel topoloogias aktiivselt osalev storm-supervisor, siis tekib esimene reaalne probleem. Samas ka siin ei kao voos liikuvad andmed vaid o\u0303ige seadistuse puhul iga topoloogias olev Spout registreerib voos olevate so\u0303numite mittekohalejo\u0303udmise ja kui nu\u0308u\u0308d taastatakse nimbus ja/vo\u0303i storm-supervisor, siis Spout saadab so\u0303numi uuesti."}, {"text_idx": "#000040", "text": "Kujutame ette, et meil on allpool toodud topoloogia"}, {"text_idx": "#000041", "text": "Ko\u0303ik Bolt\u2019d ja Spout\u2019d asuvad eraldi masinates ehk on srorm-supervusor + zookeeper komplektid, siis juhul, kui peaks tekkima selline olukord"}, {"text_idx": "#000042", "text": "Siis nimbus saab sellest teada, sest zookeeperisse enam heartbeate ei tule ja nimbus u\u0308ritab leida zookeeperi kaudu mo\u0303nda vaba serverit, kus on storm-supervisor."}, {"text_idx": "#000043", "text": "Kui nu\u0308u\u0308d on olemas zookeeperi kaudu nimbusele teada mo\u0303ni vaba storm-supervisor, siis topoloogia taastatakse. Kui mo\u0303nii so\u0303num ei jo\u0303udnud vahepeal kohale, kuna topoloogia ei olnud ta\u0308ielik, siis Spout on sellest teadlik ja saadab so\u0303numi uuesti."}]