a key factor to `` overlay networking '' success is that an overlay service can be quickly constructed and easily deployed and upgraded .in particular , several overlays support multicast - style data dissemination service without requiring the widespread deployment of ip multicast. however , such application - level multicast may suffer from poor performance , scale and cost problems when the delivery tree construction process ignores the topology and link characteristics of the underlying network .if an overlay is built in this way , nearby nodes in the overlay network may actually be distant from each others in the underlying network .recent proposals in multicast overlay construction [ 1 - 10 ] demonstrate the importance of exploiting underlying network topology .however , we claim that there are barriers for quality of service aspects , namely scalability and efficiency in existing topology - aware overlay multicast protocols :  1 .  although decentralized protocols have been designed to be scalable , by not relying on global knowledge , they often rely on periodic and incremental refinement processes , which induce high overhead . in these protocols such as    , nodes maintain their relative positions to the root of the delivery tree .periodically , each node tries to improve its on - tree position by finding a better parent , i.e. a non - descendant node that provides a lower delay to the root .therefore , these protocols are generally not scalable to support large multicast groups .additional overhead is incurred in case of dynamicity of either the overlay membership or the underlying network conditions .in fact , during overlay growth or membership changes , heavy control overhead is incurred due to periodical structure quality maintenance and partition repairs operations . on the other hand ,higher frequency control messages is required to map the overlay to varying network characteristics .users attending a video conferencing session or an event broadcast expect an acceptable quality as soon as they join the multicast session . since a multicast overlay delivery tree is typically studied to minimize the average incurred delay observed by the receivers , we consider that a delivery tree is `` efficient '' if the average incurred delay is less than a threshold value .however , one would expect that incremental refinements - based approaches incur a long delay before the overlay delivery tree converges to an efficient structure .  in this paper, we provide a practical solution for large - scale and efficient multicast support .first , we propose a simple and accurate location - aware process for connecting new members to the overlay network .the basic idea is to use the nodes in the already constructed overlay to suggest candidate neighbors that are close to a newcomer .the latter gradually requests the suggested nodes to refine its localization in the underlying network .this locating process does not use virtual coordinates system embedding nor fixed landmarks measurements , and aims to be accurate and scalable .second , we build a robust and scalable topology - aware clustered hierarchical overlay on the basis of the locating process .we propose proactive mechanisms to react to cluster leaders failures , and to smoothly manage overlay topology changes caused by crash scenarios or underlying network changes .scalability is achieved by drastically reducing the bandwidth requirements for overlay maintenance .robustness is obtained by mitigating the effect of dynamic environment as most changes are quickly recovered and not seen beyond clustered set of nodes .the proposed overlay multicast construction scheme , called locate , cluster and conquer ( lcc ) , has been designed to address the aforementioned quality of service issues .intuitively , running the locating process before that the node joins the overlay , and then clustering nearby nodes should allow to perform fast convergence to an efficient multicast delivery tree .furthermore , it would reduce management overhead and delivery tree changes imposed due to periodical refinements .however , these enhancements could be mitigated by the overhead of the locating process .taking into account these considerations , we evaluated the lcc scheme using two complementary evaluation methods : simulations and experimentations over the planetlab testbed .results show that lcc has low overhead upon the locating process and during the session .compared to other initially - randomly and topology - aware approaches , lcc achieves lower convergence time and performs less link adjustments rate . at the same time , it performs well in terms of data distribution efficiency even in large overlays .the remainder of this paper is structured as follows .section 2 presents the related work .section 3 provides an overview of the lcc scheme .the locating process is detailed in section 4 .then the clustering mechanism and its different components are presented in section 5 .experiments and simulations are discussed in section 6 and a comparison with various previous approaches is provided .finally , section 7 concludes the paper .there has been tremendous interest in the construction of overlays to provide application - level multicast .basically , the contributions can be categorized in two classes : overlay - router approach and p2p approach .  in the overlay - router approach such as omni  and toma , reliable servers are installed across the network to act as application - level multicast routers .the content is transmitted from the source to a set of receivers on a multicast tree consisting of the overlay servers .this approach is designed to be scalable since the receivers get the content from the application - level routers , thus alleviating bandwidth demand at the source .however , it needs dedicated infrastructure deployment and costly servers .the p2p approach requires no extra resources .several proposals have been designed to handle small groups .narada  , meshtree  , and hostcast  are examples of distributed `` mesh - first '' algorithms where nodes arrange themselves into well - connected mesh on top of which a routing protocol is run to derive a delivery tree .these protocols rely on incremental improvements over time by adding and removing mesh links based on an utility function .although these protocols offer robustness properties ( thanks to the mesh structure ) , they do not scale to large population , due to excessive overhead resulting from the improvement process .the objective of lcc is to locate the newcomer prior to joining the overlay and hence process only a few number of refinements during the multicast session .other `` tree - first '' protocols like zigzag  and nice  , are topology - aware clustering - based protocols which are designed to support wide - area size multicast for low bandwidth application. however , they do not consider individual node fan - out capability .rather , they bound the overlay fan - out using a ( global ) cluster - size parameter . in particular, since both protocols only consider latency for cluster leader selection , they may experience problems if the cluster leader has insufficient fan - out .other proposals exploit the as - level  or the router - level  underlying network topology information to build efficient overlay networks . however , these approaches assume some assistance from the ip layer ( routers sending icmp messages , or bgp information access ) , which may be problematic. lcc does not require any extra assistance from entities that do not belong to the overlay .landmark clustering is a general concept to construct topology - aware overlays .ratnasamy et al . use such an approach to build a multicast topology - aware can overlay network . prior to joining the overlay network , a newcomer has to measure its distance to each landmark .the node then orders the landmarks according to its distance measurements .the main intuition is that nodes with the same landmark ordering , are also quite likely to be close to each other topologically .an immediate issue with such a landmark - based approach is that it can be rather coarse - grained depending on the number of landmarks used and their distribution .furthermore , requiring a fixed set of landmarks known by all participating nodes renders this approach unsuitable for dynamic networks .we have designed a two - level clustered overlay multicast architecture ( lcc ) to provide scalable , efficient and robust multicast distribution service to end users .basically , the lcc overlay construction is divided into two processes : _ locating _ and _clustering_.  the * locating process * aims to direct newcomers to the `` nearest '' cluster before they receive data on the delivery tree .a newcomer initiates the locating process by sending a `` localization_request '' to a randomly selected cluster leader ( denoted by boot node ) . according to its location - information knowledge, the boot node selects a few cluster leaders ( that we will denote _ the queried nodes _ ) that it considers to be close to the newcomer .it asks them to probe the newcomer , and gets each queried node s answer .then , it suggests to the newcomer the possible closest nodes . by iteratively sending `` localization_request '' messages to the closest nodes ( called _ the requested nodes _ in the rest of the paper ) ,the newcomer is able to gradually locate nodes that are close by .each requested node uses a selection criterion to limit the number of nodes probing the newcomer , hence minimizing the locating overhead .the locating process ends by proposing one or more nearby cluster leaders .  by grouping together nodes that are close to a cluster leader, members are expected to be close to each other , which leads to low overhead of intra - cluster control messages .the * clustering process * is initiated by every node once the locating process terminates .on the basis of their locating result , nodes are partitioned into clusters of nodes .a maximum distance , , defines the interval in which other nodes are considered `` nearby '' .this interval is called the cluster leader s _ scope _ , and defines the clustering criterion . during the clustering process, a node decides at which level it will join the overlay . if it creates its own cluster , it joins the `` top - level '' topology and starts an inter - cluster mesh construction .otherwise , it becomes a cluster member and joins an intra - cluster mesh in order to derive its delivery tree .since a node could be in more than one cluster leader s scope , it could be member of more than one cluster .such nodes are called _edge nodes_. we exploit edge nodes to improve the overlay efficiency .in fact , the cluster leader is the primary responsible of connecting its cluster to the top - level overlay .nevertheless , edge nodes are also allowed to join the inter - cluster mesh at the top level .the main role of edge nodes is to allow ( if fan - out constraints are not violated ) the clusters members to derive their delivery tree considering the edge node as an alternative nearby source connected to the top - level topology .moreover , these nodes may contribute in the overlay robustness in case of cluster leaders failures .although edge nodes are attached to several meshes of different clusters , they do not receive the data several times .in fact , as each edge node derives a unique delivery tree from one of the existing intra - cluster mesh , it is then a child in this particular delivery tree . on the other hand, it could be a parent in several derived delivery trees in other clusters .a high level picture of lcc is illustrated in fig .[ clusters ] .[ htb ]   note that lcc does not specify a new tree construction protocol ; any existing protocol may be used on top of lcc . in this paper, we choose to construct the lcc overlay by running the meshtree protocol  at both the top - level and the intra - cluster level .meshtree embeds the delivery tree in a degree - bounded mesh containing many low - cost links .the constructed mesh consists then of two main components : ( i ) a backbone structure , composed of a low - cost tree and connecting nodes that are topologically close together , and ( ii ) additional links to improve the delay properties .the delivery tree is then derived from the mesh using a path - vector routing protocol .the `` flat '' meshtree first constructs a randomly connected overlay and relies on periodical adding / deleting links using a set of local rules . unlike this approach ,the lcc scheme , initially constructs location - aware overlay based on the locating and clustering processes .top - level nodes then act as particular meshtree nodes , where other clusters represent neighbors in the derived delivery tree ( see fig .[ clusters ] ) .  in order to construct an overlay spanning tree rooted at the source node , we need to consider the degree constraints . assuming that the media playback rate is and the outgoing access link capacity of any particular node , is , the total number of streams that the node can handle is = [ .the fan - out value of node represents the maximum number of connections that this node can establish with other nodes .we assume that each node can estimate its connection type ( eg adsl , 802.11 , etc . ) , relying on system and user specifications . moreover, lcc can use a history of maximum throughput of the most recent downloads , as an indication of its effective connection speed .these fan - out estimation techniques are used in order to avoid each node to measure its available bandwidth , which may involve high overhead .we also define the cluster overall capacity as , where is the number of members in the clusters .next , we detail both the locating and the clustering lcc processes .lcc adopts a nodes positioning strategy similar to meridian  to organize nodes into levels according to a distance metric . typically , the distance between two nodes is the round trip network delay .each lcc node keeps track of a fixed number of other nodes in its locating system .a locating system is a set of non overlapping and exponentially increasing levels , represented by intervals , where for and ( see fig . [ locating ] ) .[ htb ]   nodes measure the distances to the set of nodes they are aware of , and assign each node a position in the correspondent level of their locating system .for example if the measured distance satisfies , the node is positioned in the level .all considered nodes in the locating system are cluster leaders . in the following ,we describe the locating process operations .initially , a newcomer , say node , has to contact a global well - known rendezvous point to obtain the identity of a randomly selected boot node , .node measures the distance from itself to , and assigns a level in its locating system , say level .if is in s scope , i.e. , the clustering criterion is met and the locating process terminates , and sends a request to join s cluster .otherwise , sends a `` localization_request '' . upon receiving such request , the requested node simultaneously queries nodes that it considers as nearby to .these queried nodes have then to report the results back to the requested node .if a queried node is closer to the newcomer than the requested node , it is considered as a candidate .a list that identifies the set of candidate nodes is sent by the requested node to the newcomer . among this list, initiates cluster joining processes with all nodes that meet the clustering criterion . if there are no such nodes in the list , nodes in the candidate list become possible requested nodes , since re - initiates the locating process with each node in this list sorted in increasing distances .the list is updated at each response from a requested node .this procedure is repeated until the newcomer finds a cluster leader in its scope .finally , it is necessary to set a stop criterion to terminate the process within a given time period by repeating the procedure at most times .if the algorithm ends without satisfying the clustering criterion , creates its own cluster .   during the locating request ,each requested node has to query a set of nodes .it then selects among them a list of candidate nodes to send to the newcomer . in this subsection, we answer the following question : how queried nodes are chosen by the requested node?  the basic solution would be that the requested node asks all the nodes in the same level than the newcomer and in the adjacent levels , as potential queried nodes . to establish a reference, we consider this solution that we call the `` non - selective '' locating process .although it has the advantage of simplicity , this solution may induce high overhead .in fact , while being in the same or adjacent levels than the newcomer , some queried nodes should not be taken into consideration for probing the newcomer , since they may be not closer to it than the requested node .we introduce the _ selection criterion _ in order to reduce the number of useless probes during the locating process . basically , the `` selective locating '' consists in querying only specific representative nodes .nodes that are close enough to a representative node , randomly selected by the requested node , are not queried to measure their distance to : the less queried nodes , the less measurements and control overhead .closeness is defined by a distance threshold value , which is a function of the distance between the newcomer and the requested node , .if the newcomer is close to the level frontier or to the requested node , the latter should use a fine - grained selection and a small value should be used . if not , the requested node should use a greater value . in our algorithm , we choose :   nodes maintain for each level a square matrix , , representing learned distances of level s nodes to each other , and to nodes in adjacent levels and .values in are assigned as and when discovered through other nodes locating requests . if a distance is not known , it is set to a value large enough to discard the concerned node from the selection .each element in corresponds to the distance between nodes and .the row in represents the learned distances between node and other nodes in level and adjacent levels .the selection algorithm run by a requested node is presented in algorithm [ algoselection ]  [ h ] distance list of representative nodes to query     return   and can be described as follows : each requested node selects a random node , , from level or adjacent levels .if is less than the threshold value , then node is represented by .selected nodes are represented by a matrix , say , which is initially equal to . at each iteration of the selection process, is diminished by the columns of nodes in that can be represented by the selected node .the selection algorithm terminates when contains only distances of representative nodes .in this section , we describe the protocol to form and maintain clusters . in this work , we emphasize mechanisms to enhance the overlay qos by increasing scalability and robustness . in particular , we propose a proactive algorithm to manage failures of leaders , and new cluster formation afterwards .we also propose new mechanisms to smoothly manage cluster topology changes due to leadership or underlying network changes .   in early stage of the overlay formation ,new clusters are more frequently generated since few nodes exist .if the locating process ends with no leaders found in the newcomer s scope , the latter creates its proper cluster with a new cluster i d .it then contacts the closest cluster leaders that the locating process returns , to join the top - level topology .contacted cluster leaders inform their members by flooding a `` new_cluster '' message .finally , members verify if they are in the new leader s scope , i.e. if they are potential edge nodes.   a classical joining operation is initiated by a newcomer detecting cluster leaders in its scope after the locating process terminates .the newcomer sends simultaneously a `` join_request '' message to all the detected cluster leaders .the request contains its fan - out value and the set of other clusters it may belong to . upon receiving `` join_notification '' messages , it sends acknowledgement messages mentioning successfully joined clusters .  in lcc, each top - level node has two types of neighbors : nodes in its own cluster and other top - level nodes .since a cluster leader has limited available bandwidth , it should carefully set its node degree to maintain a balance between connecting to other top - level nodes for better overall performance and serving as many nodes as possible in its own cluster .if the cluster overall capacity is 1 , the cluster leader accepts the newcomer .note that the cluster overall capacity is null if all nodes are edge nodes attached to the top - level topology .so , considering the case where all of these nodes have a fan - out value of 1 , this would lead to a saturated cluster .this situation can be recovered if the cluster leader requests an edge node to leave other cluster membership to serve a newcomer .if the newcomer is accepted , the cluster leader randomly assigns it a cluster member to boot into the cluster - level mesh .the newcomer gets cluster maintenance information from the cluster members .using `` keep_alive '' messages exchanged by cluster members allows to share cluster state , and to update cluster information .information about other overlay nodes is obtained using a simple gossip style node discovery technique .basically , a node , maintains a list of known nodes in the overlay .periodically , randomly picks a node from the list , say and sends to it a randomly - constructed set of other known members .node updates its own known nodes list and replies using the same procedure .this simple informative exchange allows nodes to maintain a minimal view of the overlay membership .next , we discuss how this knowledge affects the average locating process iterations .the cluster leader election is based on the value of priority vectors ( ) used to maintain a nodes rank .basically , a is defined as : , where stands for the node s current perceived latency in the intra - cluster delivery tree , denotes the minimum distance from the node to a known foreign cluster leader , represents how long the node has stayed in the overlay , and is a boolean indicating if the current cluster leader is included in the node s scope .the priority value is computed as a linear combination of the first 4 components of with decreasing weights .these priorities are used to sort appropriate eligible nodes .nodes update periodically their .each is distributed as part of `` keep_alive '' messages .when receiving nodes , a node sorts the cluster members with increasing priorities .in fact , cluster nodes construct a proactive rescue plan , where each node maintains a * local cache * storing shared information .the local cache consists in a sorted list of nodes that are eligible to become cluster leaders . in a dynamic network environment, a cluster leader may depart unexpectedly at any time .if the leader fails , nodes will know it after a period of time as they do no more receive the `` keep_alive '' messages from the leader .meanwhile , the second node in the list becomes automatically the leader and sends out `` keep_alive '' messages .if the second also fails , the third one will stand up , etc .it is important to notice that for stability purposes , eligible nodes that win the leader election at their joining process , are maintained at a second position in the local cache , until their life time in the cluster reaches a greater value .   in this subsection, we discuss the behavior of lcc in case of cluster members migration outside their cluster due to new cluster election or underlying topology changes .we distinguish three different clustering states :  * * stabilized state * : where the cluster leader is included in the scope of each member of the cluster . ** temporary state * : where at least one node have migrated outside its original cluster . * * recovering state * : where during the temporary state , all migrated nodes know about other migrated nodes , and start to evolve towards a stabilized state .we introduce an algorithm that allows the migrated nodes to collaborate in order to create suitable new clusters after a temporary state .it is based on the nodes rank in the local cache .it consists in a recursive procedure , where a potential leader node asks subsequent nodes in the cache to join its cluster , and triggers a recovering procedure for migrated nodes that are not in its scope .basically , nodes verify at each local cache update operation whether the current cluster leader is in their scope or not .if not , they mark it as foreign cluster neighbor in their with = 1 . at eachreceived , a migrated node updates a set of other migrated nodes .the first ranked migrated node initiates the process by creating a new cluster and by sending a `` recovering_request '' to the next migrated node .the request contains identities of other nodes that are already in the node s scope .hence , each node is able , through previous received requests , to determine migrated nodes that can still be leaders ( eligible nodes ) .if the node is included in the requesting node s scope , it sends a positive _ ack _ to join its cluster and returns to a stabilized state . a node which sends a negative ack , verifies at each request if it has been contacted by all prior ranked eligible nodes in the cache . in this case, it becomes a cluster leader and initiates its proper recovering procedure by sending requests to next nodes in the local cache .the recovering algorithm terminates when contacting the last ranked migrated node .it then informs its `` new '' cluster neighbors along with its previous cluster leader of the cluster split .finally , it switches to a stabilized state and connects to the top - level topology .to evaluate and test the lcc scheme , we carried out simulations and planetlab  experiments . while the goal of simulation studies is to assess the effectiveness of proposed techniques for large scale overlays , the planetlab experiments aim to illustrate the system performance under particular real - world environments .    using the brite internet topology generator  , we simulated up to nodes networks .we used a node bandwidth reference model based on the gnutella peer bandwidth distribution observed by saroiu et al .we modeled the node join using poisson distribution with an average rate of 60 node joins per simulation tick .the duration distributions were modeled with an exponential distribution of 0.01 as parameter .we implemented lcc in a c library , and built wrappers for well - known ip - multicast applications ( vic / rat , vlc ) .we tested the lcc overlay over a set of 212 wide spread planetlab nodes .the set consists of 90 nodes in u.s , 90 nodes in europe and 32 nodes in asia .all experiments were conducted over several days in october 2005 . in this paper, we discuss a representative set of experimental results using `` planetlab1.cs.cornell.edu '' as the data source .this source generates a 560 kbits / s ( 70 kb / s ) data stream sent to all the other group members .nodes join the overlay at the average rate of one every 2 seconds .we remove the top 20% and bottom 20% of the measurement results and take the average of the remaining values . in practice , pings have been conducted using icmp echo messages , and we use 10 consecutive pings for latency measurements .we evaluate the lcc scheme in terms of ( 1 ) scalability , by studying the control overhead during both data distribution and overlay joining .we also observe the link adjustment frequency and the locating process resources usage ( time and number of nodes needed to locate a newcomer ) ; ( 2 ) efficiency , by measuring several common metrics that characterize the multicast overlay .in particular , we measure the average relative delay penalty ( ) which is defined as the average ratio between the overlay delay ( ) and the shortest path delay in the underlying network ( ) from to all other nodes : , where is the number of nodes in the overlay . by considering that the delivery tree `` converges '' or becomes `` efficient '' when the is less than a threshold value ( say 2 ), we study the convergence time .then we plot the and the stress on the link ( which represents the number of copies of an identical packet sent over a single link ) , varying the overlay size ; ( 3 ) robustness , by verifying the scheme robustness to leaders failures and its ability to recover in case of crash scenarios ; ( 4 ) locating process accuracy , by experimenting newcomers behavior during the locating process and their ability to locate their closest node in the underlying network , within a large overlay .   in order to compare lcc to initially randomly - connected overlays relying on periodic refinements , we experiment a variant of lcc , disabling the locating process and setting the scope value to zero , thus emulating meshtree behavior .we call this variant flat meshtree . in our simulations , we also compare lcc to two previously proposed multicast overlay structures : omni  as an infrastructure - based approach and zigzag  as a topology - aware hierarchical approach .   in the following , we report both simulation and experimental results .we ran simulations to evaluate the control traffic overhead in the overlay and analyzed the protocol behavior in large size overlays .we assumed a basic header size of 40 bytes per ip - packet and we measured the overall control message traffic sent and received by each node throughout a session .[ protocoloverhead ]  [ htb ]   [ htb ]   shows the average overhead per node when varying the overlay size .control overhead of lcc is lower than those of flat meshtree and zigzag , and is comparable to omni .we note that the per node overhead in lcc , for different values , is steady for small overlays .the maximum value reached for a 512 nodes overlay is 1.10 kbps for lcc with = 100ms , and control messages incur less than 2 kbps message overhead , in a 8000-nodes overlay .omni nodes obtain lower control overhead . since it is the application - level multicast servers that are in charge of managing the delivery tree , nodes in omni exchange a minimum number of control messages to join the overlay .we note that the above results include overheads due to network measurement , in particular during the locating process , as we consider these results from the instant the node contacts the rendezvous point .  to evaluate the cost of locating the closest cluster to join , we measured on the planetlab testbed the average control traffic overhead ( in kbps ) generated during overlay joining for both the non - selective and selective locating process .we observe in fig .[ overhead ] the importance of the selection criterion during the locating process .the per node overhead in the selective locating process is reasonably small with about 0.7 kbps for a 212 nodes overlay .in addition , it increases very slowly with the number of members .the locating messages are roughly 50% less frequent than those of a non - selective localization .not selecting nodes boosts the message overhead due to useless measurement operations . in this case , requested nodes would contact all the nodes in the newcomer s level and the adjacent levels .these queried nodes will also measure their distance to the newcomer , which would incrementally add network overhead .however , we note that the selective locating process may require more time to locate the newcomer . in fact , by selecting representative nodes , the newcomer may need to contact more requested nodes than the non - selective process as discussed later and shown in fig .[ hopsknownneighbors ] .[ labadjustement ] shows the lcc structure stability during membership changes . on the planetlab testbed , we start tracking the link adjustment counts right after the last node joined the overlay .results are collected at 30-second intervals .we observe that the link adjustment rate mostly falls below 5 per hour per node for the lcc overlay , whereas it stabilizes at roughly 10 per node per hour for the flat meshtree . to confirm that the lcc efficiency is robust , in case of membership frequent changes and crash scenarios , we inject a simultaneous 20-nodes failure in 7 different sites at the minute and we let them rejoin the overlay at the minute .we observe that the link adjustment activity for lcc is moderate ( mostly under 5 per hour per node ) during the membership changes . after the minute, the average link adjustment count falls around 2 per hour per node .due to randomness in initially connecting newcomers to the clusters , the link adjustment rate of meshtree is maintained at 10 .this assesses our intuition that non - initially locating schemes may require high control messages for quality maintenance and structure repairs operations .  the simulation results shown in fig .[ simadjustement ]  [ htb ]   [ htb ]   confirm the planetlab experiments conclusions .flat meshtree suffers from high adjustment rate , almost more than 20 links change per node per simulation tick . compared to zigzag , the lcc structure has a very low adjustment rate .this rate is stabilized at less than 5 adjustments per node per tick .link changes in omni are less frequent than other improvement - based overlay structures .omni nodes achieve an average link adjustment per node per tick of 12.9 , with a minimum of 0.78 .nodes in lcc ( = 20 ms ) have an average of 11.4 adjustments per node per tick with a minimum links change of 1.4 .the refinement - based approaches depend on the choice of a refinement period , say . a small value reduces the convergence time , as more adjustment procedures are processed within a short time , but may induce high overhead .a large may reduce overhead at the expense of increased convergence time . in the following experiments , we set the improvement period to 30 seconds , for each receiver , and study the convergence time metric which describes the overlay structure evolution in time . fig .[ convergence ] illustrates the convergence time , showing versus the multicast session time in both simulations ( overlay size = 2000 nodes ) and planetlab testbed .all nodes join the overlay within the first 100 seconds .[ htb ]   we observe that in lcc , rapidly decreases to a value less than 2 after the first 400 seconds , i.e. less than 14 improvement rounds per node . for flat meshtree , it takes much more time for to stabilize with almost 1300 seconds .this shows the importance of pro - actively organizing the overlay , to converge very quickly to an efficient structure .the zigzag overlay reaches an acceptable value much more quickly than meshtree . although stabilized , this value is more than 2 , which is still inefficient to consider the overlay converged .the convergence time of zigzag is around 1400 seconds when falls under 2 .the reason why zigzag does not converge quickly is that during overlay growth , several group merges and splits are imposed to not exceed the maximal group size .this may induce several improvements rounds , and link adjustments .the omni server - based structure is not affected by frequent membership changes and converges quickly , similarly to lcc .we characterize the average incurred delay observed by the receivers in a large populated overlay by observing the variation according to the overlay size in fig .[ averagerdpcomparisonvaryingoverlaysize ] .[ htb ]   [ htb ]   in flat meshtree , the increases drastically to more than 4 demonstrating that this protocol does not scale to a few thousands of nodes .nevertheless , we note that flat meshtree has lower than lcc structures in small groups ( overlay size 500 nodes ) .zigzag maintains a stable value while the overlay size is increasing but suffers relatively poor performance with 2.5 in a 3000-nodes overlay . to make it easier to read , we zoom in on a portion of the graph in fig .[ zoomaveragerdpcomparisonvaryingoverlaysize ] .we observe that the of lcc is about 60% of zigzag . for =50ms and 100 ms , values of lcc are roughly maintained at values between 1.4 and 2 for large overlays .omni has almost a constant value ( 1.82 ) and performs on average better than lcc in 12000-nodes overlay .we also note that in large overlays , for clusters defined with 10 ms as node s scope , increases to reach 3 , as nodes are more likely to be scattered .larger scopes are more efficient in this case .[ stressvsticks ]  [ htb ]   shows average physical network stress for each of the overlays , namely omni , zigzag , flat meshtree and lcc ( ) 2000 seconds after the last node joined .omni and zigzag stress values stabilize between 6.5 and 7 .the flat meshtree overlay leads to somewhat lower stress than omni and zigzag with stress highly oscillating between 4 and 5 due to random connections established by newcomers .we note finally that lcc has an impressively low stress , 2 to 3 times less than other overlays , with a steady stress value between 2.5 and 2.8 .topology information is of paramount importance in this observation , as packets sent through the top - level hierarchy are sent to the cluster leader and in some cases to potential edge nodes .our clustering process allows then to reduce the amount of redundant flows entering each network , considering clusters as `` local networks '' , and cluster leaders as `` multicast - enabled routers '' .when a non - leaf node quits , the overlay needs to be reconstructed .so , it is important that this node s children can quickly locate a new parent to resume the session . in addition, the recovery process should not result in a fan - out violation in any node . in lcc , to recover from the failure of its neighbor a node has to redirect packet requests from that neighbor to another nearby in its proper cluster .we compare lcc to the _ grandparent recovery scheme _ studied in  . in this scheme, the children of the node which departs try first to attach to their grandparent provided that the latter has enough capacity .otherwise , they are redirected to its descendants .we instruct a number of randomly selected nodes in a 5000-nodes overlay to leave the session simultaneously .then , we observe the recovery time of members , as the average time for an affected node to resume the session . results in fig . [ failurerecovery ]  [ htb ]   show that lcc always yields a smaller recovery time than the tree - based grandparent recovery scheme . on average, lcc takes 3.85 seconds to recover from failures , which is about 35% less than for the grand - parent recovery scheme .  in fig .[ recoveryleaderfailure ] ,  [ htb ]   we study the capacity of the lcc overlay to recover from cluster leader failures .each 10 seconds , a set of randomly selected cluster leaders are instructed to simultaneously leave a 5000-nodes lcc overlay ( = 50 ms ) .we observe that when 30% of cluster leaders fail simultaneously , the recovery time is almost 2 seconds .lcc is robust thanks to : 1 ) the proactive rescue plan of leaders election and 2 ) the edge nodes connected to the top - level topology , that allows to achieve data in case of leaders data disruption .to evaluate the behavior of newcomers during the locating process , we observe the average number of requested nodes contacted by a newcomer . fig .[ hopsknownneighbors ] depicts the average number of requested nodes versus the total number of known cluster leaders in each requested node s locating system .the figure plots both planetlab results and simulations of 200 newcomers running the selective locating process , once the overlay size reaches respectively 2000 , 3000 and 4000 nodes , resp .denoted by ov = 2000 , ov = 3000 and ov = 4000 in fig .[ hopsknownneighbors ] .[ htb ]   [ htb ]   we expected that the selective locating process needs more requested nodes than the non selective process . indeed , since it queried representative nodes at each iteration , it may be less accurate in one iteration , and hence requires to contact more nodes afterwards .results show however , that the selection has little impact on the locating efficiency .the selective locating process performs almost as well as the non - selective process , with a maximum of 13 requested nodes in planetlab experimentations .moreover , curves are very close when the number of known cluster leaders is large .we also observe that the locating process scales well to large overlays : in a 4000-nodes overlay , newcomers are located by contacting less than 12 requested nodes that know about only 60 nodes in their locating system .the distribution of nodes depicted in fig . [ distributionaveragehops ] shows that more than 80% of a 2000-nodes overlay are able to terminate the locating process by contacting less than 15 requested nodes . on average the locating time in the experimentis very low with a mean locating time of 3.2 seconds , a maximum of 7.2 seconds and minimum of 1.8 seconds .finally , we note that 98.4% of newcomers are able to connect to their closest cluster upon their arrival .300 seconds after the last node joins the overlay , 99.3% of nodes are connected to their closest node .this demonstrates the locating process accuracy , which is one of the reasons for the resulting promising performances of lcc .in this paper , we proposed a practical solution to enhance different qos aspects of overlays , namely scalability and efficiency .the overlay construction is initiated by a simple and scalable locating process that allows newcomers , after contacting a few nodes , to locate the closest cluster in the overlay .the locating process includes a selection algorithm to minimize measurement overhead . on the basis of the locating process, we proposed an hierarchical topology - aware overlay construction .we introduced mechanisms to pro - actively deal with leaders failures and underlying topology characteristics changes .our planetlab and simulations experiments prove that lcc incur low overhead during both localization and data distribution .compared to other enhancement - based and topology - aware approaches , lcc achieves shorter convergence time and performs less link adjustments rate . at the same time , the scheme is robust to nodes failures and performs well in terms of data distribution efficiency especially in large overlays . in conclusion ,we believe that lcc is very suitable for large - scale applications such as event broadcast for thousands of attendees . in future works, we will focus on ways to automatically tune different parameters such as nodes scope and stop criterion , through real - life tests .we will also investigate techniques to secure the overlay and detect / prevent users from cheating .

Abstract: recent proposals in multicast overlay construction have demonstrated the importance of exploiting underlying network topology . however , these topology - aware proposals often rely on incremental and periodic refinements to improve the system performance . these approaches are therefore neither scalable , as they induce high communication cost due to refinement overhead , nor efficient because long convergence time is necessary to obtain a stabilized structure . in this paper , we propose a highly scalable locating algorithm that gradually directs newcomers to their a set of their closest nodes without inducing high overhead . on the basis of this locating process , we build a robust and scalable topology - aware clustered hierarchical overlay scheme , called lcc . we conducted both simulations and planetlab experiments to evaluate the performance of lcc . results show that the locating process entails modest resources in terms of time and bandwidth . moreover , lcc demonstrates promising performance to support large scale multicast applications .


led board is a printed circuit board ( pcb ) that connects to pc via standard serial cable .led board has 8 leds , each led can be individually turned on / off from windows application ( led board control console ) .fig.1 shows simplified schematic of led board .microchip pic16f84 is responsible for receiving data from pc and controlling 8 leds .    since rs-232 and pic16f84 use different voltage levels , max232 is required for logic level conversion : rs-232 ttl . the null modem cable configuration without handshaking is used between pc and led board .as we can see from the schematic max232 is connected to microchip pic 16f84 and db9 serial connector . to provide + 5v for pic16f84 and max232 , we use 9v battery together with l7805 voltage regulator .leds are directly connected to the pic16f84 s portb i / o pins which have up to 25 ma sink / source current , this feature makes them ideal for direct led drive .4mhz crystal resonator and two 20pf capacitors ( c7 , c8 ) are used to provide external clocking ( xt mode ) .the null modem configuration uses txd and rxd lines for data interchange between pc and external device . led board uses txd to receive data , rxd is wired - up , but not used .txd is connected to max232 s r1 in ( pin 13 ) , while r1out ( pin 12 ) is directly connected to pic16f84 s ra1 ( pin 18 ) , thus ra1 is used as an input pin which receives data from pc .please see datasheets of microchip pic16f84 , max232 , l7805 for detailed specifications .+  +  + above program receives bytes and depending on bit pattern of each byte switches corresponding leds on / off . for example , if received byte is 00010100 , led # 3 and led # 5 will be switched on , while remaining leds will be off ( 1 = led on , 0= led off ) . obviously , to light up all 8 leds , pic16f84 should receive 11111111 .after serin receives a byte , it stores received byte in b0 . once byte has been received , it is written into portb with poke command , thus switching leds on / off according to above mentioned bit pattern . since we have exactly 8 leds connected to portb ( 8 pins ) , it is easy to control them all with just one byte .led board control console ( lbcc ) is a small windows application designed to control led board from pc .to switch led(s ) on / off , lbcc sends a byte with specified bit pattern into com port .fig.2 shows sample application developed with microsoft s visual c++ .( dialog based mfc application ) [ fig : graph ]   before sending data , we should setup and configure corresponding com port .winapi function _ createfile ( ) _ creates serial port , it returns handle which is later used to access serial port .the following code shows how to successfully setup com port .  `handle  comport ;   //com port handle ` + ` dcb dcb ;       //control structure for a serial device ` ` bool fsuccess ; ` +  .... comport = createfile("com1",generic_read | generic_write ,            0 ,       //comm devices must be                   // opened w / exclusive - access            null ,      //no security attributes            open_existing , //comm devices                   // must use open_existing            0 ,       // not overlapped i / o            null      //htemplate must be null                   //for comm devices            ) ;  if ( comport = = invalid_handle_value )  {    // handle the error .messagebox ( " createfile failed","error",mb_ok ) ;    return ( 1 ) ;  }   fsuccess = getcommstate(comport , & dcb ) ;   if ( !fsuccess )  {    // handle the error .    messagebox ( " getcommstate failed","error",mb_ok ) ;    return ( 2 ) ;  }   // filling in dcb structure  //  //important note :  // serial device configuration ( e.g. speed , parity ) should be matched with  //the corresponding serin mode ( see section 4 for picbasic code )   dcb.baudrate = cbr_2400 ;    // set the baud rate  dcb.bytesize = 8 ;       // data size , xmit , and rcv  dcb.parity = noparity ;     // no parity bit  dcb.stopbits = onestopbit ;   //one stop bit   fsuccess = setcommstate(comport , & dcb ) ;   if ( ! fsuccess )  {    // handle the error .    messagebox ( " setcommstate failed","error",mb_ok ) ;    return ( 3 ) ;   }   messagebox ( " com port initialized",null , mb_ok ) ; ....  com port is now initialized and ready for use .to transmit data to led board , we use winapi function _ writefile ( ) _ which writes a byte into com port . as an example , let s consider the following , suppose we want to switch on led # 3 and led # 5 while keeping remaining leds off , this can be done by sending the byte 00010100 into com port .  .... _ _ int8 combyte ;  dword nnumberofbytestowrite = 1 ; dword numberofbyteswritten ;  combyte = 20 ;  //binary 00010100 corresponds to 20 ( decimal )  writefile(comport,&combyte , nnumberofbytestowrite,&numberofbyteswritten , null ) ; ....  the first parameter in _ writefile ( ) _ is com port handle ( section 5.1 ) .suppose we need to switch on specified led without affecting other leds ( if some of them were on ( off ) , they should remain on ( off ) ) .let s also assume that the led we want to switch on was previously off . to solve this task we will use successive bitwise operations .assume that we have the following bit pattern 00010010 ( led # 2 is on , led # 5 is on ; remaining leds are off ) and we want to switch on led # 7 .let s use or mask with the original bit pattern ( 00010010 )  .... _ _ int8 ormask = 1 ; _ _ int8 combyte = 18 ;  //00010010 ....  ` ormask = ormask < < 6 ; // 00000001 ` ` 01000000 ` + ` combyte = combyte | ormask ; // 00010010 | 01000000 = 01010010 `  combyte can now be written to com port with _ writefile ( ) _ as we can see from bit pattern led # 7 is switched on , while other leds are not affected . please note that to switch on / off led , left shift should be applied times . in our case , therefore we have ormask = ormask 6 .analogously , to switch off led # 7 , xor mask should be used  .... _ _ int8 xormask = 1 ; _ _ int8 combyte = 82 ;  //01010010 ....  ` xormask = xormask < < 6 ; // 00000001 ` ` 01000000 ` + ` combyte = combyte ^ xormask ; //01010010 ^ 01000000 = 00010010 `pcb size ( approx . ) : + length : _ 112.5 mm ( 4.43 in ) _ + width : _ 57 mm ( 2.24 in ) _ +[ cols="^,^ " , ]rs-232 led board project discussed in this article shows the basics of serial communications .this project was the partial requirement for advanced digital electronics course taken at the university of oregon , eugene , usa .

Abstract: this article demonstrates how to develop a microchip pic16f84 based device that supports rs-232 interface with pc . circuit ( led board ) design and software development will be discussed . picbasic pro compiler from microengineering labs , inc . is used for pic programming . development of led board control console using c / c++ is also briefly discussed . the project requires basic work experience with microchip pics , serial communication and programming . +  +  +


european region is one of the most intensively studied areas of the earth from the point of view of regional geodynamics .there are more than 100 permanently operating gps receivers , about 10 permanent vlbi stations and more than 10 permanent slr stations .lately much attention has been devoted to comparison and combination of results obtained using different space geodesy techniques .this work is devoted to comparison of baseline length variations derived from gps and vlbi observations , continuing the cycle of works on this problem , see e.g. .it should be mentioned here that observed changes of the baseline length on the one hand are resulted by insufficient corrections for observational effects such as thermal deformations of vlbi antennas , for example , or errors in modeling of tropospheric refraction , but on the other hand they are subjected to a number of insufficiently studied or not taken into account properly geophysical effects that can result in the real changes of the baseline length , these effects may be atmospheric and snow loading , tides , postglacial rebound , and so on .it is important that the majority of these effects has both seasonal and secular components .  in this studywe have analyzed vlbi and gps observations at 6 european stations carrying out both vlbi and gps regular observations and having long enough observational history .vlbi baseline lengths were computed with the occam package using all available 24h sessions for the period of 1983.92001.5 .details of the method used can be found in .linear trend in the baseline lengths was computed for the whole period of observations and , for more accurate comparison with gps data , for the period of 1996.0 - 2001.0 .only later results are presented in this study . in compared linear trend in variations of baseline length derived from the observations over the period 1996.02001.0 with ones computed using all available vlbi sessions over a period 1990.02001.4 .differences of estimated rates are inside one sigma interval for all baselines analyzed here .  for more strict account for variation in baseline lengths due to thermal antenna deformations we used advanced model of this effect which allow to correct observed station position not only for vertical but also for horizontal displacement .at this stage of the research we used zero time delay between change of air and telescope construction temperature , because of lack of such a data for most of antennas .however , this mismodelling will effect only intra - day displacement of the telesope reference point , but not seasonal variations .it should be mentioned here that account for horizontal displacement is especially important for processing regional networks , whereas vertical displacement due to thermal deformations prevails in variations of global baselines length . in particular , errors in modelling of this effectmay be a possible reason of seasonal baseline length variations found e.g. in .variation of baseline lengths obtained from vlbi data are shown in figure [ fig : vlbi ] .unfortunately , stations crimea and yebes are not equipped with gps receiver .=   for computation of baseline lengths between european gps stations we used weekly epn solutions distributed in sinex files . however , this solutions are not suitable for immediate use in geodynamical analysis because they can not provide homogeneous long - time coordinate time series due to periodic changes in reference coordinate system and set of fiducial stations .for this reason , direct use of the euref solutions shows jumps in baseline length variations . besides , method of computation of stationcoordinates used in epn is based on using tight constrains to fiducial stations which cause a distortion of the network , i.e. fictive variations in baseline lengths ( see e.g. ) .so , variations of baseline lengths from gps data were obtained from analysis of coordinate time series for epn stations computed by the method described in .this computation is based on de - constraining of the official epn solutions with further transformation to itrf2000 .for this study we used 6-parameter helmert transformation to avoid loss of seasonal geophysical signal in baseline length . using our independent coordinate time series allows us to obtain realistic station displacement practically free of network distortion for all epn stations over the period 1996.02001.0 .variation of baseline lengths obtained from gps data are shown in figure [ fig : gps ] .unfortunately , madr coordinate time series is too short ( less than two years ) that does not allow to get reliable results .it should be mentioned that errors in gps baseline length significantly decrease during a period under investigation .e.g. , one can see abnormal trend in mate displacement in 1996 .however , that does not influence result very much due to relatively small weight of these data .=   one of the most important factor affecting variations of station coordinates derived from space geodesy observations is atmospheric loading .we investigated influence of this effect using 3-dimensional atmospheric loading time series provided by h .-scherneck .the data were averaged over a week interval corresponding to every gps week and variations of baseline lengths were computed from these weekly values .variation of baseline lengths obtained from analysis of atmospheric data are shown in figure [ fig : atmosph ] .=  it is interesting that baseline length variations contain not only seasonal but also secular component , even for short baselines , especially for continental - coastal ones , in particular baselines including wettzell stations which are often used in studies on european geodynamics , e.g. .the reason of that may be long - periodic or progressive weather and climate changes , but period of our investigation is too short to separate them .  since variations in height component of station displacement due to atmospheric loading prevail , this effect is especially significant for global baselines . for regional networkshorizontal displacements yield main contribution to variation of baseline lengths .results of computation of variations in baseline lengths are presented in [ tab : result ] .one can see that values of rates obtained from vlbi and gps observations are in good agreement for most baselines .unfortunately , it is not the case for seasonal variations .obviously , interval of investigation is too short and number of used vlbi observations is too small for many baselines .indeed , it would be important to verify our results using data obtained from other space geodesy techniques , but only mate and wett stations are equipped with slr units , and only nyal station is equipped with doris beacon ( which is explained by difficulties in collocation of doris beacon and vlbi antenna due to radio frequency interference ) ..results of analysis of variation of baseline lengths : baseline length ( l ) , km , number of epochs ( n ) processed and found in the ivs data base , linear trend ( rate ) , mm , amplitude of annual term ( as ) , mm , amplitude of semiannual term ( asa ) , mm .[ cols="^,^,^,>,>,>,^ , > , > , > , > , > , > " , ]   it is also remarkable that influence of atmospheric loading on baseline length rate is significant for many baselines .evidently , this effect must be investigated more carefully and properly accounted during geodynamical analysis .figure [ fig : lbacc ] shows dependence of error in baseline length rate on length of baseline .it is interesting that for gps data error is practically the same for all baselines unlike vlbi data .1em  further steps of our work will include new re - computation of epn coordinate time series based on new combination of individual epn analysis center solutions , reprocessing of vlbi data with new version of software , and more complete analysis of various factors effected variations of baseline lengths .analysis of variations in vertical component of station displacement is also planned .this research was partially supported by a grant of the st.petersburg scientific center of the russian academy of sciences .campbell j. the european geodetic vlbi project an overview . in : p.tomasi , f. mantovani , m. prez - torres ( eds . ) proc .14th working meeting on european vlbi for geodesy and astrometry , castel san pietro terme , 89 sep 2000 .lanotte r. , c. ferraro , a. nardi , c. sciarretta , f. vespe .the cgs vlbi eur98 geodetic solution and comparison with the cgs gps results . in : w. schlueter , h. hase ( eds . )13th working meeting on european vlbi for geodesy and astrometry , viechtach / wettzell , feb 1213 , 1999 , 101107 .malkin z. m. , n. a. panafidina , e. a. skurikhina .comparison of baseline lengths derived from gps and vlbi observations on european network .presented at the all - russian astronomy conference , st.petersburg , russia , 612 aug 2001 .panafidina n. , malkin z. on computation of an homogeneous euref coordinate time series . in : abstracts `` vistas for geodesy in the new millennium '' . presented at the iag 2001 scientific assembly ,budapest , hungary , 27 september 2001 .tornatore v. , f. sacerdote , m. crespi .comparison of deformation resulting from gps and vlbi data : problems and preliminary results . in : w. schlueter , h. hase ( eds . ) proc .13th working meeting on european vlbi for geodesy and astrometry , viechtach / wettzell , feb 1213 , 1999 , 206216 .

Abstract: results of vlbi and gps observations were analyzed with goal to investigate differences in observed baseline length derived from both techniques . vlbi coordinates for european stations were obtained from processing of all available observations collected on european and global vlbi network . advanced model for antenna thermal deformation was applied to account for change of horizontal component of baseline length . gps data were obtained from re - processing of the weekly epn ( european permanent gps network ) solutions . systematic differences between results obtained with two techniques including linear drift and seasonal effects are determined .


the emission of radio frequency ( rf ) radiation from the extended air showers ( eas ) produced by ultra high energy cosmic rays ( uhecr , i.e. , particles with energy ) impinging on the earth atmosphere has been investigated since the 60s , after askaryan proposed a coherent mechanism for the production of cherenkov radiation at mhz frequencies ( the so called askaryan effect ) . the first detection of rf radiation from eas was proved in 1965 by jelley et al . .the idea of askaryan relies on the excess of negative charges in the shower , since positrons tend to disappear through their annihilation into photons , and further electrons are created via compton scattering by photons .if the number of electrons and positrons is the same , emission would not occur because the electric field generated by opposite charges is equal ( in module ) and opposite ( in sign ) . upon certain conditions, the cherenkov yield is not proportional to the number of charged particles in the shower , but to , where is the fractional excess of charge .the quadratic dependence on ( and therefore on ) makes the askaryan effect attractive as mechanism for the detection of uhecrs .  in the following years, other mechanisms were proposed , which predict a coherent emission of rf radiation , and invoke the interaction of the shower electrons and positrons with the earth magnetic field .negative and positive charges are separated by the magnetic field , creating an electric dipole . during the motion of the dipole ,rf radiation is emitted in coherent way ( geomagnetic effect ) .alternatively , charged particles are bent because of the lorentz force and emit synchrotron radiation ( geosynchrotron effect ) .the bunch of particles in the eas arriving to the detector can be schematically depicted as a dish , with thickness of a few meters and diameter depending of the energy of the primary .as will see in the followings , the requirement that the wavelength is comparable with the dish thickness puts a frequency cutoff above which coherence is lost ( and , consequently , the emission strongly hampered ) :  mhz , where is the speed of light .the coherence condition explains why the experimental investigation is mostly concentrated in the frequency range  mhz .this picture is not complete .only very recently , it has been shown , with monte carlo simulations , that geosynchrotron radiation can be coherent above  mhz under certain conditions : since the emission occurs in air , which has a ( altitude dependent ) refractive index , a time compression of rf wavefront is produced , which causes the coherence to extend to higher frequency , even beyond the ghz .the compression takes place only inside a cone with aperture equal to the cherenkov angle , and angular spread . outside this small angular region ,coherence is lost and the rf output power falls dramatically .in this coherence regime , sometimes named geomagnetic cherenkov radiation " , the emitted electric field amplitude depends on the frequency and can be modelled , as shown by refs ., as with 300 mhz and 500 mhz . here we will not deal with such particular mechanism .the first experimental proof of the coherent cherenkov emission in laboratory controlled conditions was performed in 2001 by directing picosecond pulses of gev photons against a silica sand target . successively other tests were conducted using rock salt or ice as targets . because of the higher density of those materials , the shower dimensions are different than in air , and askaryan effect can extend beyond ghz frequencies .the coherent rf emission by charged particles in the presence of a magnetic field has been measured in controlled conditions and successfully checked versus electrodynamics simulations in ref . .beside coherent emission , a rf incoherent emission mechanism exists , that is , bremsstralhung radiation , the emission of photons during the deflection of electrons and positrons under the coulomb field of the nuclei .the emission is always present but is weak since it scales linearly with and the cross section is low .  in this paper we do not discuss the various mechanisms of production of rf radiation and their relative importance .our purpose is to explicitly show how and when coherent mechanisms appear , and to derive a general formula which involves the coherent and the incoherent regimes .we suppose that the eas is composed by charged particles divided into electrons and positrons ( ) , and indicate with the charge excess : ( ) .depending of the rf radiation mechanism , the phase of the electromagnetic wave from a single emitter ( electron or positron ) can change randomly with time or can not . for bremsstrahlung , for example ,the phase changes randomly and therefore the superposition of sources is incoherent and the emitted energy is proportional to . for synchrotron , geomagnetic , and cherenkov emission the phase of a single emitter does not change randomly with time .the superposition of emitters can be coherent or incoherent depending on the random relative position of the individual sources .  to simplify the calculations, we suppose that the electric field amplitude is unity and the fields are polarized in the same direction .  for the case of cherenkov radiation , the electron and positron electric fields have opposite sign .therefore the total electric field ( at the detector ) is where and are the phases of the rf wave emitted by the -th electron and positron , respectively . for any , and independent random variables with the same distribution , that is , . therefore their mean values , for any , are .the energy is we are interested to the mean value : since is independent of ( ) , = = = .it is reasonable to assume that the probability density is an even function between and , so . in conclusion, analogously  the energy is therefore \mean{\cos\phi}^2\ ] ]  the factor which determines the presence of the coherence is precisely .it can be easily seen in the case where the phases are uniformly distributed between and : two extreme regimes can be identified :  1 . : , which is the condition for complete incoherence .the total energy is .this is the case , for example , of the optical and uv cherenkov light emission . : total coherence . , depends on the excess of negative charges in the shower .the cherenkov angle . ]  .[ fg : systemref ]  we treat now the generic case for the cherenkov radiation where the phase is not uniformely distributed .the results can be easily adapted to the geosynchronous and the geomagnetic radiation . with reference to fig.[fg : systemref ] , we introduce a reference system with versors , where the eas axis coincides the axis . is a single emitter with cylindrical coordinates , is the cherenkov cone aperture , the angular coordinate along the emission cone .the detector d is placed on the axis at the distance : .p is the projection of s on the x - y plane .then : , .the wave vector is and the phase of the electric field in the point d is \ ] ] the phase depends also on ( fixed value ) , but since in the calculation of the energy only the phase differences enter , the term with always eliminates , so we can put to simplify the computation : \ ] ] we schematize the shower as a flat cylinder , with thickness and charge distribution along the shower axis ( ) , and radial charge distribution such as = 1 .the angular variables and are distributed uniformily between 0 and . then \big]\ ] ] the computation is straightforward in the case ( uniform distribution ) : where is the bessel function of the first kind .calling : and then   the same procedure can be applied to the geosynchrotron emission .the electric field of the synchrotron radiation is : }{r~(1-\vec{\beta}\cdot\hat{n})^3 } \bigg]_{retarded}\ ] ] where is the versor of the direction of observation , is the particle acceleration , and the electron electric charge must be considered with the sign . since positrons and electrons have opposite acceleration under magnetic field , the product is the same for the two particles , so the electric fields sum up .therefore , for geosynchrotron radiation , the eq.([eq : efield ] ) becomes and the final result is : \mean{\cos\phi}^2\ ] ] in case of total coherence , .( [ eq : wmedio ] ) and ( [ eq : wmedio2 ] ) can be summarized stating that the total power emitted by sources is obtained multiplying the power from a single source by a factor : where represents the spatial coherence factor , function of the radiation frequency and of the shower shape and dimension , and the sign depends on the emission mechanism ( for cherenkov , + for synchrotron and geomagnetic ) .the factor in eq.([eq : m ] ) determines the coherent or incoherent regime . if , the incoherence is total , and , i.e. the total emitted power is the some of the single powers . on the opposite, complete coherence occurs as , and for cherenkov emission , for geosynchrotron and geomagnetic emission , i.e. , the power is proportional to the squared number of emitters .the factorization ( ) has the physical meaning that the longitudinal and radial developments of the showers are independent of each other , which is a reasonable approximation that can be extended to the general case . ( ) , takes into account the radial emitter distribution , in the plane perpendicular to the shower axis , while ( ) takes into account the distribution along the longitudinal axis .  as a check ,note that when , and ( total spatial incoherence ) and when , and ( total spatial coherence ) .as an example of the above calculations , we apply them to the case of eas produced by cosmic rays in the atmosphere .  it is well known that a relativistic particle with velocity such that , where is the refractive index of air , radiates electromagnetic waves at the angle at the energy rate ( in si units ) : since , , and , for an eas with secondary particles the radiated power in the bandwidth centred around the frequency , neglecting coherence effets , is :   for the coherent cherenkov effect assuming the nkg radial charge distribution . ]  where is the fine - structure constant and the planck constant .when coherence occurs , the emitted power is obtained from the above equation substituting the factor with and taking into consideration the amplitude ( eq.([eq : m ] ) ) .then to calculate from eq.([eq : ar ] ) we must know the radial charge distribution .some examples can be found in the original work of askaryan .here we treat the case of the nishima , kamata , greisen ( nkg ) distribution , which describes adequately the charge distribution in an eas .the normalized density function is : where is the factorial gamma function , , and m. has been evaluated numerically and the result shown in fig.[fg : fignkg ] .the frequency cutoff , where the power is reduced by a factor of 2 , is mhz .  regarding the longitudinal coherence , in the simple case of an uniform charge distribution , we get , as shown before in eq.([eq : az ] ) , a function with the first zero occurring at mhz for m. beyond that frequency , the rf emitted power scales as .more realistic cases are discussed in ref . .we have derived a general formula for the emission of radio frequency from particle showers , which describes also the transition from the incoherent regime to the coherent one .a coherence factor is introduced , which depends on the spatial charge distribution of the radio frequency emitters .the formula applies also to dense materials , such as ice , for example , which has been proposed as target medium ( precisely , antartic ice sheet ) the for the detection of very high energy neutrinos .

Abstract: extended air showers produced by cosmic rays impinging on the earth atmosphere irradiate radio frequency radiation through different mechanisms . upon certain conditions , the emission has a coherent nature , with the consequence that the emitted power is not proportional to the energy of the primary cosmic rays , but to the energy squared . the effect was predicted in 1962 by askaryan and it is nowadays experimentally well established and exploited for the detection of ultra high energy cosmic rays .  in this paper we discuss in details the conditions for coherence , which in literature have been too often taken for granted , and calculate them analytically , finding a formulation which comprehends both the coherent and the incoherent emissions . we apply the result to the cherenkov effect , obtaining the same conclusions derived by askaryan , and to the geosynchrotron radiation .   # 1


the human auditory system gives us the extraordinary ability to converse in the midst of a noisy throng of party goers .solving this so - called _ cocktail party _problem has proven extremely challenging for computers , and separating and recognizing speech in such conditions has been the holy grail of speech processing for more than 50 years .previously , no practical method existed that could solve the problem in general conditions , especially in the case of single channel speech mixtures .this work builds upon recent advances in single - channel separation , using a method known as _ deep clustering _ . in deep clustering ,a neural network is trained to assign an embedding vector to each element of a multi - dimensional signal , such that clustering the embeddings yields a desired segmentation of the signal . in the cocktail - party problem ,the embeddings are assigned to each time - frequency ( tf ) index of the short - time fourier transform ( stft ) of the mixture of speech signals .clustering these embeddings yields an assignment of each tf bin to one of the inferred sources .these assignments are used as a masking function to extract the dominant parts of each source .preliminary work on this method produced remarkable performance , improving snr by 6 db on the task of separating two unknown speakers from a single - channel mixture .  in this paperwe present improvements and extensions that enable a leap forward in separation quality , reaching levels of improvement that were previously out of reach ( audio examples and scripts to generate the data used here are available at ) .in addition to improvements to the training procedure , we investigate the three speaker case , showing generalization between two- and three - speaker networks .  the original deep clustering system was intended to only recover a binary masks for each source , leaving recovery of the missing features to subsequent stages . in this paper, we incorporate enhancement layers to refine the signal estimate . using soft clustering, we can then train the entire system _ end - to - end _ , training jointly through the deep clustering embeddings , the clustering and enhancement stages .this allows us to directly use a signal approximation objective instead of the original mask - based deep clustering objective .prior work in this area includes auditory grouping approaches to _ computational auditory scene analysis _ ( casa ) . these methods used hand - designed features to cluster the parts of the spectrum belonging to the same source .their success was limited by the lack of a machine learning framework .such a framework was provided in subsequent work on spectral clustering , at the cost of prohibitive complexity .generative models have also been proposed , beginning with . in constrained tasks ,super - human speech separation was first achieved using factorial hmms and was extended to speaker - independent separation .variants of non - negative matrix factorization and bayesian non - parametric models have also been used .these methods suffer from computational complexity and difficulty of discriminative training .in contrast to the above , deep learning approaches have recently provided fast and accurate methods on simpler enhancement tasks .these methods treat the mask inferance as a classification problem , and hence can be discriminatively trained for accuracy without sacrificing speed. however they fail to learn in the speaker - independent case , where sources are of the same class , despite the work - around of choosing the best permutation of the network outputs during training .we call this the _ permutation problem _ : there are multiple valid output masks that differ only by a permutation of the order of the sources , so a global decision is needed to choose a permutation .deep clustering solves the permutation problem by framing mask estimation as a clustering problem . to do so, it produces an embedding for each time - frequency element in the spectrogram , such that clustering the embeddings produces the desired segmentation .the representation is thus independent of permutation of the source labels .it can also flexibly represent any number of sources , allowing the number of inferred sources to be decided at test time .below we present the deep clustering model and further investigate its capabilities . we then present extensions to allow end - to - end training for signal fidelity .the results are evaluated using an automatic speech recognition model trained on clean speech .the end - to - end signal approximation produces unprecedented performance , reducing the word error rate ( wer ) from close to 89.1% wer down to 30.8% by using the end - to - end training .this represents a major advancement towards solving the cocktail party problem .here we review the deep clustering formalism presented in .we define as a raw input signal and as a feature vector indexed by an element . in audio signals , is typically a tf index , where indexes frame of the signal , indexes frequency , and the value of the complex spectrogram at the corresponding tf bin .we assume that the tf bins can be partitioned into sets of tf bins in which each source dominates .once estimated , the partition for each source serves as a tf mask to be applied to , yielding the tf components of each source that are uncorrupted by other sources .the stft can then be inverted to obtain estimates of each isolated source .the target partition in a given mixture is represented by the indicator , mapping each element to each of components of the mixture , so that if element is in cluster .then is a binary affinity matrix that represents the cluster assignments in a permutation - independent way : if and belong to the same cluster and otherwise , and for any permutation matrix .  to estimate the partition , we seek -dimensional embeddings , parameterized by , such that clustering the embeddings yields a partition of that is close to the target . in and this work , is based on a deep neural network that is a global function of the entire input signal .each embedding has unit norm , i.e. , .we consider the embeddings to implicitly represent an estimated affinity matrix , and we optimize the embeddings such that , for an input , matches the ideal affinities .this is done by minimizing , with respect to , the training cost function summed over training examples , where is the squared frobenius norm . due to its low - rank nature , the objective and its gradient can be formulated so as to avoid operations on all pairs of elements , leading to an efficient implementation .  at test time , the embeddings are computed on the test signal , and the rows are clustered using -means .the resulting cluster assignments are used as binary masks on the complex spectrogram of the mixture , to estimate the sources .we investigated several approaches to improve performance over the baseline deep clustering method , including regularization such as drop - out , model size and shape , and training schedule .we used the same feature extraction procedure as in , with log - magnitude stft features as input , and we performed global mean - variance normalization as a pre - processing step . for all experiments we used we used rmsprop optimization with a fixed learning rate schedule , and early stopping based on cross - validation .* regularizing recurrent network units : * recurrent neural network ( rnn ) units , in particular lstm structures , have been widely adopted in many tasks such as object detection , natural language processing , machine translation , and speech recognition . herewe experiment with regularizing them using dropout .lstm nodes consist of a recurrent memory cell surrounded by gates controlling its input , output , and recurrent connections .the direct recurrent connections are element - wise and linear with weight 1 , so that with the right setting of the gates , the memory is perpetuated , and otherwise more general recurrent processing is obtained .dropout is a training regularization in which nodes are randomly set to zero . in recurrent networkthere is a concern that dropout could interfere with lstm s memorization ability ; for example , used it only on feed - forward connections , but not on the recurrent ones ._ recurrent dropout _samples the set of dropout nodes once for each sequence , and applies dropout to the same nodes at every time step for that sequence .applying recurrent dropout to the lstm memory cells recently yielded performance improvements on phoneme and speech recognition tasks with blstm acoustic models .  in this work , we sampled the dropout masks once at each time step for the forward connections , and only once for each sequence for the recurrent connections .we used the same recurrent dropout mask for each gate .  * architecture : * we investigated using deeper and wider architectures .the neural network model used in was a two layer bidirectional long short - term memory ( blstm ) network followed by a feed - forward layer to produce embeddings .we show that expanding the network size improves performance for our task .* temporal context : * during training , the utterances are divided into fixed length non - overlapping segments , and gradients are computed using shuffled mini - batches of these segments , as in .shorter segments increase the diversity within each batch , and may make an easier starting point for training , since the speech does not change as much over the segment .however , at test time , the network and clustering are given the entire utterance , so that the permutation problem can be solved globally .so we may also expect that training on longer segments would improve performance in the end .  in experiments below, we investigate training segment lengths of 100 versus 400 , and show that although the longer segments work better , pretraining with shorter segments followed by training with longer segments leads to better performance on this task .this is an example of _ curriculum learning _ , in which starting with an easier task improves learning and generalization .  * multi - speaker training : * previous experiments showed preliminary results on generalization from two speaker training to a three - speaker separation task . herewe further investigate generalization from three - speaker training to two - speaker separation , as well as multi - style training on both two and three - speaker mixtures , and show that the multi - style training can achieve the best performance on both tasks .deep clustering solves the difficult problem of segmenting the spectrogram into regions dominated by each source . it does not however solve the problem of recovering the sources in regions strongly dominated by other sources .given the segmentation , this is arguably an easier problem .we propose to use a second - stage enhancement network to obtain better source estimates , in particular for the missing regions .for each source , the enhancement network first processes the concatenation of the amplitude spectrogram of the mixture and that of the deep clustering estimate through a blstm layer and a feed - forward linear layer , to produce an output .sequence - level mean and variance normalization is applied to the input , and the network parameters are shared for all sources . a soft - maxis then used to combine the outputs across sources , forming a mask at each tf bin .this mask is applied to the mixture , yielding the final estimate . during training , we optimize the enhancement cost function where is the set of permutations on . since the enhancement network is trained to directly improve the signal reconstruction , it may improve upon deep clustering , especially in regions where the signal is dominated by other sources .in order to consider end - to - end training in the sense of jointly training the deep clustering with the enhancement stage , we need to compute gradients of the clustering step . in ,hard -means clustering was used to cluster the embeddings .the resulting binary masks can not be directly optimized to improve signal fidelity , because the optimal masks are generally continuous , and because the hard clustering is not differentiable . herewe propose a soft -means algorithm that enables us to directly optimize the estimated speech for signal fidelity .  in ,clustering was performed with equal weights on the tf embeddings , although weights were used in the training objective in order to train only on tf elements with significant energy . herewe introduce similar weights weights for each embedding to focus the clustering on tf elements with significant energy .the goal is mainly to avoid clustering silence regions , which may have noisy embeddings , and for which mask estimation errors are inconsequential .the soft weighted -means algorithm can be interpreted as a weighted expectation maximization ( em ) algorithm for a gaussian mixture model with tied circular covariances .it alternates between computing the assignment of every embedding to each centroid , and updating the centroids : where is the estimated mean of cluster , and is the estimated assignment of embedding to the cluster .the parameter controls the hardness of the clustering . as the value of increases , the algorithm approaches -means .the weights may be set in a variety of ways .a reasonable choice could be to set according to the power of the mixture in each tf bin . herewe set the weights to , except in silence tf bins where the weight is set to .silence is defined using a threshold on the energy relative to the maximum of the mixture .end - to - end training is performed by _ unfolding _ the steps of , and treating them as layers in a clustering network , according to the general framework known as _ deep unfolding _ .the gradients of each step are thus passed to the previous layers using standard back - propagation .* experimental setup : * we evaluate deep clustering on a single - channel speaker - independent speech separation task , considering mixtures of two and three speakers with all gender combinations . for two - speaker experiments , we use the corpus introduced in , derived from the wall street journal ( wsj0 ) corpus .it consists in a 30 h training set and a 10 h validation set with two - speaker mixtures generated by randomly selecting utterances by different speakers from the wsj0 training set ` si_tr_s ` , and mixing them at various signal - to - noise ratios ( snr ) randomly chosen between 0 db and 10 db .the validation set was here used to optimize some tuning parameters .the 5 h test set consists in mixtures similarly generated using utterances from 16 speakers from the wsj0 development set ` si_dt_05 ` and evaluation set ` si_et_05 ` .the speakers are different from those in our training and validation sets , leading to a speaker - independent separation task . for three - speaker experiments, we created a corpus similar to the two - speaker one , with the same amounts of data generated from the same datasets .all data were downsampled to 8 khz before processing to reduce computational and memory costs .the input features were the log spectral magnitudes of the speech mixture , computed using a short - time fourier transform ( stft ) with a 32 ms sine window and 8 ms shift .the scores are reported in terms of signal - to - distortion ratio ( sdr ) , which we define as scale - invariant snr . as oracleupper bounds on performance for our datasets , we report in table [ tab : ideal_results ] the results obtained using two types of `` ideal '' masks : the ideal binary mask ( ibm ) defined as , which leads to highest snr among all binary masks , and a `` wiener - like '' filter ( wf ) defined as , which empirically leads to good snr , with values in $ ] . here denotes the time - frequency representation of speaker .casa and previous deep clustering results are also shown for the two - speaker set ..sdr ( db ) improvements using the ideal binary mask ( ibm ) , oracle wiener - like filter ( wf ) , compared to prior methods dpcl and casa on the two- and three - speaker test sets . [cols="<,^,^,^,^",options="header " , ]   the magnitude snr is insensitive to phase estimation errors introduced by using the noisy phases for reconstruction , whereas the sdr might get worse as a result of phase errors , even if the amplitudes are accurate .speech recognition uses features based on the amplitudes , and hence the improvements in magnitude snr seem to predict the improvements in wer due to the enhancement and end - to - end training .[ fig : scatter ] shows that the sdr improvements of the end - to - end model are consistently good on nearly all of the two - speaker test mixtures .* asr performance : * we evaluated asr performance ( wer ) with gmm - based clean - speech wsj models obtained by a standard kaldi recipe .the noisy baseline result on the mixtures is 89.1 % , while the result on the clean speech is 19.9 % .the raw output from dpcl did not work well , despite good perceptual quality , possibly due to the effect of near - zero values in the masked spectrum , which is known to degrade asr performance .however , the enhancement networks significantly mitigated the degradation , and finally obtained 30.8 % with the end - to - end network .* visualization : * to gain insight into network functioning , we performed reverse correlation experiments . for each node , we average the 50-frame patches of input centered at the time when the node is active ( e.g. , the node is at 80% of its maximum value ) .[ fig : revcor ] shows a variety of interesting patterns , which seem to reflect such properties as onsets , pitch , frequency chirps , and vowel - fricative transitions .* conclusion : * we have improved and extended the deep clustering framework to perform end - to - end training for signal reconstruction quality for the first time . we show significant improvements to performance both on signal quality metrics and speech recognition error rates .

Abstract: deep clustering is a recently introduced deep learning architecture that uses discriminatively trained embeddings as the basis for clustering . it was recently applied to spectrogram segmentation , resulting in impressive results on speaker - independent multi - speaker separation . in this paper we extend the baseline system with an end - to - end signal approximation objective that greatly improves performance on a challenging speech separation . we first significantly improve upon the baseline system performance by incorporating better regularization , larger temporal context , and a deeper architecture , culminating in an overall improvement in signal to distortion ratio ( sdr ) of 10.3 db compared to the baseline of 6.0 db for two - speaker separation , as well as a 7.1 db sdr improvement for three - speaker separation . we then extend the model to incorporate an enhancement layer to refine the signal estimates , and perform end - to - end training through both the clustering and enhancement stages to maximize signal fidelity . we evaluate the results using automatic speech recognition . the new signal approximation objective , combined with end - to - end training , produces unprecedented performance , reducing the word error rate ( wer ) from 89.1% down to 30.8% . this represents a major advancement towards solving the cocktail party problem .  * index terms * : single - channel speech separation , embedding , deep learning