Failure to accurately measure the outcomes of an experiment can lead to bias and incorrect conclusions. Online controlled experiments (aka AB tests) are increasingly being used to make decisions to improve websites as well as mobile and desktop applications. We argue that loss of telemetry data (during upload or post-processing) can skew the results of experiments, leading to loss of statistical power and inaccurate or erroneous conclusions. By systematically investigating the causes of telemetry loss, we argue that it is not practical to entirely eliminate it. Consequently, experimentation systems need to be robust to its effects. Furthermore, we note that it is nontrivial to measure the absolute level of telemetry loss in an experimentation system. In this paper, we take a top-down approach towards solving this problem. We motivate the impact of loss qualitatively using experiments in real applications deployed at scale, and formalize the problem by presenting a theoretical breakdown of the bias introduced by loss. Based on this foundation, we present a general framework for quantitatively evaluating the impact of telemetry loss, and present two solutions to measure the absolute levels of loss. This framework is used by well-known applications at Microsoft, with millions of users and billions of sessions. These general principles can be adopted by any application to improve the overall trustworthiness of experimentation and data-driven decision making.
A common technique to measure packet loss in the Internet is through the use of (e.g., in the RTP protocol @cite_8 ). The sequence number is incremented for each data packet sent, allowing the receiver to detect packets that do not arrive. We apply a similar approach (referred to as the sequence method) to measure data loss, which we describe in Section . The challenges in mirroring data between unreliable endpoints have been studied in detail by the database community @cite_14 @cite_15 . While telemetry gathered from our apps do not require transaction guarantees, mobile environments have constraints in terms of network reliability and local storage. Moreover, most analysis systems do not tolerate delays of more than a day. Nevertheless, client apps can borrow ideas such as lazy replication to improve telemetry reliability, especially if the experimentation and analysis system can tolerate longer delays.
@cite_15 @cite_14 @cite_8
2111586962 2918093508
The quality of service limitation of today's Internet is a major challenge for real-time voice communications. Excessive delay, packet loss, and high delay jitter all impair the communication quality. A new receiver-based playout scheduling scheme is proposed to improve the tradeoff between buffering delay and late loss for real-time voice communication over IP networks. In this scheme the network delay is estimated from past statistics and the playout time of the voice packets is adaptively adjusted. In contrast to previous work, the adjustment is not only performed between talkspurts, but also within talkspurts in a highly dynamic way. Proper reconstruction of continuous playout speech is achieved by scaling individual voice packets using a time-scale modification technique based on the Waveform Similarity Overlap-Add (WSOLA) algorithm. Results of subjective listening tests show that this operation does not impair audio quality, since the adaptation process requires infrequent scaling of the voice packets and low playout jitter is perceptually tolerable. The same time-scale modification technique is also used to conceal packet loss at very low delay, i.e., one packet time. Simulation results based on Internet measurements show that the tradeoff between buffering delay and late loss can be improved significantly. The overall audio quality is investigated based on subjective listening tests, showing typical gains of 1 on a 5-point scale of the Mean Opinion Score. In this paper, we propose a novel transport protocol that effectively utilizes available bandwidth and diversity gains provided by heterogeneous, highly lossy paths. Our Multi-Path LOss-Tolerant (MPLOT) protocol can be used to provide significant gains in the goodput of wireless mesh networks, subject to bursty, correlated losses with average loss-rates as high as 50 , and random outage events. MPLOT makes intelligent use of erasure codes to guard against packets losses, and a Hybrid-ARQ FEC scheme to reduce packet recovery latency, where the redundancy is adaptively provisioned into both proactive and reactive FECs. MPLOT uses dynamic packet mapping based on current path characteristics, and does not require packets to be delivered in sequence to ensure reliability. We present a theoretical analysis of the different design choices of MPLOT and show that MPLOT makes an optimal trade-off between goodput and delay constraints. We test MPLOT, through simulations, under a variety of test scenarios and show that it effectively exploits path diversity in addition to aggregating path bandwidths. We also show that MPLOT is fair to single-path protocols like TCP-SACK. Supporting ultra-reliable and low-latency communications (URLLC) is one of the major goals in 5G communication systems. Previous studies focus on ensuring end-to-end delay requirement by reducing transmission delay and coding delay, and only consider reliability in data transmission. However, the reliability reflected by overall packet loss also includes other components such as queueing delay violation. Moreover, which tools are appropriate to design radio resource allocation under constraints on delay, reliability, and availability is not well understood. As a result, how to optimize resource allocation for URLLC is still unclear. In this article, we first discuss the delay and packet loss components in URLLC and the network availability for supporting the quality of service of users. Then we present tools for resource optimization in URLLC. Last, we summarize the major challenges related to resource management for URLLC, and perform a case study. Distributed processing through ad hoc and sensor networks is having a major impact on scale and applications of computing. The creation of new cyber-physical services based on wireless sensor devices relies heavily on how well communication protocols can be adapted and optimized to meet quality constraints under limited energy resources. The IEEE 802.15.4 medium access control protocol for wireless sensor networks can support energy efficient, reliable, and timely packet transmission by a parallel and distributed tuning of the medium access control parameters. Such a tuning is difficult, because simple and accurate models of the influence of these parameters on the probability of successful packet transmission, packet delay, and energy consumption are not available. Moreover, it is not clear how to adapt the parameters to the changes of the network and traffic regimes by algorithms that can run on resource-constrained devices. In this paper, a Markov chain is proposed to model these relations by simple expressions without giving up the accuracy. In contrast to previous work, the presence of limited number of retransmissions, acknowledgments, unsaturated traffic, packet size, and packet copying delay due to hardware limitations is accounted for. The model is then used to derive a distributed adaptive algorithm for minimizing the power consumption while guaranteeing a given successful packet reception probability and delay constraints in the packet transmission. The algorithm does not require any modification of the IEEE 802.15.4 medium access control and can be easily implemented on network devices. The algorithm has been experimentally implemented and evaluated on a testbed with off-the-shelf wireless sensor devices. Experimental results show that the analysis is accurate, that the proposed algorithm satisfies reliability and delay constraints, and that the approach reduces the energy consumption of the network under both stationary and transient conditions. Specifically, even if the number of devices and traffic configuration change sharply, the proposed parallel and distributed algorithm allows the system to operate close to its optimal state by estimating the busy channel and channel access probabilities. Furthermore, results indicate that the protocol reacts promptly to errors in the estimation of the number of devices and in the traffic load that can appear due to device mobility. It is also shown that the effect of imperfect channel and carrier sensing on system performance heavily depends on the traffic load and limited range of the protocol parameters. In packet audio applications, packets are buffered at a receiving site and their playout delayed in order to com- pensate for variable network delays. In this paper, we con- sider the problem of adaptively adjusting the playout delay in order to keep this delay as small as possible, while at the same time avoiding excessive "loss" due to the arrival of packets at the receiver after their playout time has al- ready passed. The contributions of this paper are twofold. First, given a trace of packet audio receptions at a receiver, we present efficient algorithms for computing a bound on the achievable performance of any playout delay adjustment algorithm. More precisely, we compute upper and lower bounds (which are shown to be tight for the range of loss and delay values of interest) on the optimum (minimum) average playout delay for a given number of packet losses (due to late arrivals) at the receiver for that trace. Second, we present a new adaptive delay adjustment algorithm that tracks the network delay of recently received packets and efficiently maintains delay percentile information. This in- formation, together with a "delay spike" detection algorithm based on (but extending) our earlier work, is used to dy- namically adjust talkspurt playout delay. We show that this algorithm outperforms existing delay adjustment algorithms over a number of measured audio delay traces and performs close to the theoretical optimum over a range of parameter values of interest. Protocols such as TCP require packets to be accepted (i.e., delivered to the receiving application) in the order they are transmitted at the sender. Packets are sometimes mis-ordered in the network. In order to deliver the arrived packets to the application in sequence, the receiver's transport layer needs to temporarily buffer out-of-order packets and resequence them as more packets arrive. Even when the application can consume the packets infinitely fast, the packets may still be delayed for resequencing. In this paper, we model packet mis-ordering by adding an IID random propagation delay to each packet and analyze the required buffer size for packet resequencing and the resequencing delay for an average packet. We demonstrate that these two quantities can be significant and show how they scale with the network bandwidth. The future wireless networks envision ultra-reliable communication with efficient use of the limited wireless channel resources. Closed-loop repetition protocols where retransmission of a packet is enabled using a feedback channel has been adopted since early days of wireless telecommunication. Protocols such as automatic repeat request (ARQ) are used in today's wireless technologies as a mean to provide the link with reduced rate of packet outage and increased average throughput. The performance of such protocols is strongly dependent to the feedback channel reliability. This paper studies the problem of feedback error and proposes a new method of acknowledging packet delivery for retransmission protocols in unreliable feedback channel conditions. The proposed method is based on backwards composite acknowledgment from multiple packets in a retransmission protocol and provides the scheduler of the wireless channel with additional parameters to configure ultra-reliable communication for a user depending on channel quality. Numerical analysis are presented which show orders of magnitude increase in reliability of the proposed method as compared to ARQ at the cost of a small increase in average experienced delay. This work studies the potential and impact of the FRANC network coding protocol for delivering high quality Dynamic Adaptive Streaming over HTTP (DASH) in wireless networks. Although DASH aims to tailor the video quality rate based on the available throughput to the destination, it relies on the TCP protocol for reliability in data delivery. TCP is known to drop its throughput performance by several fold in the presence of even 1 or 2 packet losses, which are common in wireless systems. This will force DASH to settle at a much lower video resolution, thus reducing the user's quality of experience. We show that the use of FRANC, an adaptive network coding protocol that provides both low delay and high throughput to upper layers, as a reliability mechanism for TCP can significantly increase video quality. As part of our analysis, we benchmark the performance of various TCP versions, including CUBIC, Reno, Veno, Vegas, and Westwood+, under different packet loss rates in wireless systems using a real testbed with Raspberry Pi devices. Our goal was to choose the most promising TCP version in terms of delay performance, in this case TCP Reno, and make a fair comparison between TCP running alone and using FRANC underneath for reliability. Our demonstrator with DASH in Raspberry Pi devices using the DASH benchmark, shows that the video rate delivered is 4× higher when using FRANC. Even in harsh packet loss conditions, FRANC is able to deliver higher data rates (increase 4×), while experiencing significantly shorter (decrease 10×) video lags. Video surveillance and similar real-time applications on wireless networks require increased reliability and high performance of the underlying transmission layer. Classical solutions, such as Reed-Solomon codes, increase the reliability, but typically have the negative side-effect of additional overall delays due to processing overheads. This paper describes the delay reduction achieved through online network coding approaches with a limit on the number of packets to be mixed before decoding and a systematic encoding structure. We use the inorder per packet delay as our key performance metric. This metric captures the elapsed time between (network) encoding RTP packets and completely decoding the packets in-order on the receiver side. Our solutions are implemented and evaluated on a point-to-point link between a Raspberry Pi device and a network (de)coding enabled software running on a regular PC. We find that our sliding window approach (more feedback) outperforms all other tested mechanisms in terms of per-+packet delay including the Reed-Solomon encoding employing the systematic approach, a random linear network coding approach, and our proposed on-the-fly network coding approach (which relies on less feedback).We show gains in order of magnitudes between our sliding window and the other approaches when we manage the redundancy transmission adaptively. This low per-packet delay and the inherent reliability of our schemes make these solutions particularly suitable for real-time multimedia delivery in contrast to other classical and network coding strategies. A significant number of promising wireless sensor applications for the Internet of Things (IoT) are becoming a reality. Most of these applications require a variety of heterogeneous and geographical sensor data to be delivered to users such as mobile devices, vehicles, and cloud systems for big data analysis to extract maximum value from the physical world. Data caching of popular data items in a network can significantly improve efficiency and reliability of content delivery by contributing the reduction of data access delay, bandwidth usage, and packet losses in resource-limited low-rate wireless sensor networks. In this paper, we propose an efficient cache placement protocol (BCDA) that minimizes data access cost and guarantees cache consistency in a lightweight manner. The BCDA decides where to store a copy of data content in a network using a distributed approach according to clients' access patterns, memory capacity, data size, and so on. To evaluate the feasibility and effectiveness, we implemented the BCDA in ns-2 and compared its performance with existing caching strategies. The simulation results highlight that the BCDA significantly reduces data access cost with minimizing cache management cost. When media is streamed over best-effort networks, media data is buffered at the client to protect against playout interruptions due to packet losses and random delays. While the likelihood of an interruption decreases as more data is buffered, the latency that is introduced increases. In this paper we show how adaptive media playout (AMP), the variation of the playout speed of media frames depending on channel conditions, allows the client to buffer less data, thus introducing less delay, for a given buffer underflow probability. We proceed by defining models for the streaming media system and the random, lossy, packet delivery channel. Our streaming system model buffers media at the client, and combats packet losses with deadline-constrained automatic repeat request (ARQ). For the channel, we define a two-state Markov model that features state-dependent packet loss probability. Using the models, we develop a Markov chain analysis to examine the tradeoff between buffer underflow probability and latency for AMP-augmented video streaming. The results of the analysis, verified with simulation experiments, indicate that AMP can greatly improve the tradeoff, allowing reduced latencies for a given buffer underflow probability. Client-side apps (e.g., mobile or in-browser) need cloud data to be available in a local cache, for both reads and updates. For optimal user experience and developer support, the cache should be consistent and fault-tolerant. In order to scale to high numbers of unreliable and resource-poor clients, and large database, the system needs to use resources sparingly. The SwiftCloud distributed object database is the first to provide fast reads and writes via a causally-consistent client-side local cache backed by the cloud. It is thrifty in resources and scales well, thanks to consistent versioning provided by the cloud, using small and bounded metadata. It remains available during faults, switching to a different data centre when the current one is not responsive, while maintaining its consistency guarantees. This paper presents the SwiftCloud algorithms, design, and experimental evaluation. It shows that client-side apps enjoy the high performance and availability, under the same guarantees as a remote cloud data store, at a small cost. The basic requirement of wireless healthcare monitoring systems is to send physiological signals acquired from implantable or on-body sensor nodes to a remote location. Low-power consumption is required for wireless healthcare monitoring systems since most medical sensor nodes are battery powered. The emergence of new technologies in measuring physiological signals has increased the demand for high data rate transmission systems. Ultra-wide band (UWB) is a suitable wireless technology to achieve high data rates while keeping power consumption and form factors small. Although UWB transmitters are designed based on simple techniques, UWB receivers require complex hardware and consume comparatively higher power. In order to achieve reliable low power two-way communication, a sensor node can be constructed using a UWB transmitter and a narrow band receiver. This paper proposes a new medium access control (MAC) protocol based on a dual-band physical layer technology. Co-simulation models based on MATLAB and OPNET have been developed to analyze the performance of the proposed MAC protocol. We analyzed the performance of the MAC protocol for a realistic scenario where both implantable and wearable sensor nodes are involved in the data transmission. Priority-based packet transmission techniques have been used in the MAC protocol to serve different sensors according to their QoS requirements. Analysis is done with regard to important network parameters, such as packet loss ratio, packet delay, percentage throughput, and power consumption. The next-generation Internet promises to provide a fundamental shift in the underlying architecture to support dynamic deployment of network protocols. With the introduction of programmability and dynamic protocol deployment in routers, potential vulnerabilities and attacks are expected to increase. In this paper, we consider the problem of detecting packet forwarding misbehavior in routers. Specifically, we focus on an attack scenario, where a router selectively drops packets destined for another node. Detecting such an attack is challenging since it requires differentiating malicious packet drops from congestion-based packet losses. We propose a controller-based misbehavior detection technique that effectively detects malicious routers using a hash-based delay sampling and verification. We provide a performance analysis of the detection accuracy and quantify the performance overhead of our system. Our results show that our technique provides accurate detection with low sampling rates. In this paper, we present a framework for analyzing routing performance in delay tolerant networks (DTNs). Differently from previous work, our framework is aimed at characterizing the exact distribution of relevant performance metrics, which is a substantial improvement over existing studies characterizing either the expected value of the metric, or an asymptotic approximation of the actual distribution. In particular, the considered performance metrics are packet delivery delay, and communication cost, expressed as number of copies of a packet circulating in the network at the time of delivery. Our proposed framework is based on a characterization of the routing process as a stochastic coloring process and can be applied to model performance of most stateless delay tolerant routing protocols, such as epidemic, two-hops, and spray and wait. After introducing the framework, we present examples of its application to derive the packet delivery delay and communication cost distribution of two such protocols, namely epidemic and two-hops routing. Characterizing packet delivery delay and communication cost distribution is important to investigate fundamental properties of delay tolerant networks. As an example, we show how packet delivery delay distribution can be used to estimate how epidemic routing performance changes in presence of different degrees of node cooperation within the network. More specifically, we consider fully cooperative, noncooperative, and probabilistic cooperative scenarios, and derive nearly exact expressions of the packet delivery rate (PDR) under these scenarios based on our proposed framework. The comparison of the obtained packet delivery rate estimation in the various cooperation scenarios suggests that even a modest level of node cooperation (probabilistic cooperation with a low probability of cooperation) is sufficient to achieve 2-fold performance improvement with respect to the most pessimistic scenario in which all potential forwarders drop packets. Packet loss and end-to-end delay limit delay sensitive applications over the best effort packet switched networks such as the Internet. In our previous work, we have shown that substantial reduction in packet loss can be achieved by sending packets at appropriate sending rates to a receiver from multiple senders, using disjoint paths, and by protecting packets with forward error correction. In this paper, we propose a path diversity with forward error correction (PDF) system for delay sensitive applications over the Internet in which, disjoint paths from a sender to a receiver are created using a collection of relay nodes. We propose a scalable, heuristic scheme for selecting a redundant path between a sender and a receiver, and show that substantial reduction in packet loss can be achieved by dividing packets between the default path and the redundant path. NS simulations are used to verify the effectiveness of PDF system. Legacy congestion controls including TCP and its variants are known to perform poorly over cellular networks due to highly variable capacities over short time scales, self-inflicted packet delays, and packet losses unrelated to congestion. To cope with these challenges, we present Verus, an end-to-end congestion control protocol that uses delay measurements to react quickly to the capacity changes in cellular networks without explicitly attempting to predict the cellular channel dynamics. The key idea of Verus is to continuously learn a delay profile that captures the relationship between end-to-end packet delay and outstanding window size over short epochs and uses this relationship to increment or decrement the window size based on the observed short-term packet delay variations. While the delay-based control is primarily for congestion avoidance, Verus uses standard TCP features including multiplicative decrease upon packet loss and slow start. Through a combination of simulations, empirical evaluations using cellular network traces, and real-world evaluations against standard TCP flavors and state of the art protocols like Sprout, we show that Verus outperforms these protocols in cellular channels. In comparison to TCP Cubic, Verus achieves an order of magnitude (> 10x) reduction in delay over 3G and LTE networks while achieving comparable throughput (sometimes marginally higher). In comparison to Sprout, Verus achieves up to 30 higher throughput in rapidly changing cellular networks. Future 5G systems will need to support ultra-reliable low-latency communications scenarios. From a latency-reliability viewpoint, it is inefficient to rely on average utility-based system design. Therefore, we introduce the notion of guaranteeable delay which is the average delay plus three standard deviations of the mean. We investigate the trade-off between guaranteeable delay and throughput for the point-to-point wireless erasure links with unreliable and delayed feedback, by bringing together signal flow techniques to the area of coding. We use tiny codes, i.e., sliding window by coding with just 2 packets, and design three variations of selective-repeat ARQ protocols, by building on the baseline scheme, i.e., uncoded ARQ, developed by Ausavapattanakun and Nosratinia: (i) Hybrid ARQ with soft combining at the receiver; (ii) cumulative feedback-based ARQ without rate adaptation; and (iii) coded ARQ with rate adaptation based on the cumulative feedback. Contrasting the performance of these protocols with uncoded ARQ, we demonstrate that the HARQ performs only slightly better, the cumulative feedback-based ARQ does not provide significant throughput while it has a better average delay, and the Coded ARQ can provide gains up to about 40 in terms of throughput. The Coded ARQ also provides delay guarantees, and is robust to various challenges such as imperfect and delayed feedback, burst erasures, and round-trip time fluctuations. This feature may be preferable for meeting the strict end-to-end latency and reliability requirements of the future use cases of ultra-reliable low-latency communications in 5G, such as mission-critical communications and industrial control for critical control messaging. Wireless access networks scale by replicating base stations geographically and then allowing mobile clients to seamlessly "hand off" from one station to the next as they traverse the network. However, providing the illusion of continuous connectivity requires selecting the right moment to handoff and the right base station to transfer to. Unfortunately, 802.11-based networks only attempt a handoff when a client's service degrades to a point where connectivity is threatened. Worse, the overhead of scanning for nearby base stations is routinely over 250 ms - during which incoming packets are dropped - far longer than what can be tolerated by highly interactive applications such as voice telephony. In this paper we describe SyncScan, a low-cost technique for continuously tracking nearby base stations by synchronizing short listening periods at the client with periodic transmissions from each base station. We have implemented this SyncScan algorithm using commodity 802.11 hardware and we demonstrate that it allows better handoff decisions and over an order of magnitude improvement in handoff delay. Finally, our approach only requires trivial implementation changes, is incrementally deployable and is completely backward compatible with existing 802.11 standards. Scheduling algorithms are important components in the provision of guaranteed quality of service parameters such as delay, delay jitter, packet loss rate, or throughput. The design of scheduling algorithms for mobile communication networks is especially challenging given the highly variable link error rates and capacities, and the. changing mobile station connectivity typically encountered in such networks. This article provides a survey of scheduling techniques for several types of wireless networks. Some of the challenges in designing such schedulers are first discussed. Desirable features and classifications of schedulers are then reviewed. This is followed by a discussion of several, scheduling algorithms which have been proposed for TDMA, CDMA, and multihop packet networks.