<commit_before>## Latency

There are several metrics that describe network traffic. Latency is the time it takes for a message to travel from one application to another. Latency is distinct from bandwidth, a measure of the amount of data that can be sent per time period. It's possible to have a high bandwidth, high latency connection. An example of a connection like this is a satellite in geosynchronus orbit over which we can transmit 100 MB/sec. That's high bandwidth, but at the same time it takes 250ms for a message to get from earth to the satillite and back.  That's a hard lower limit established by the speed of light; any message has to travel up to orbit at 72,000 KM and then back down. There can also be low bandwidth, low latency connection, over which we can't send very much data, but what data we do send arrives quickly.  Another metric of network performance is Packet Delay Variation (PDV), sometimes called jitter. Jitter is, loosely defined, the variance of the latency. There are multiple detailed definitions for how to measure PDV or jitter, but the "variation of the latency" term captures the essential meaning. The IETF RFC in the "further readings" section discusses the issues involved in defining a precise definition.

All three of these metrics--bandwidth, latency, and jitter--effect a networked virtual environment, and if any one of the metrics is terrible it may make the NVE implementation completely unworkable. Ask anyone who has tried to play a modern first person shooter game across a 56K dialup modem if you doubt this. Bandwidth is a classic measure, and the "Dead Reckoning: State Updates" section discusses some aspects of this problem and how to mitigate it. The problems posed by latency are less obvious. 

The latency issues are particulary relevant to military simulations. Military LVC applications often have data feeds that originate in an austere field deployment, and may disribute data via satellite links that are inherently laggy. 

Latency can create practical problems in virtual worlds, yet there are strategies to mitigate these problems.  This comes at the cost of more simulation application complexity if the programmer chooses to implement them.

### The Latency Problem

The essence of a virtual world is creating the illusion that application avatars from several hosts are interacting with each other in a shared virtual space. A fundamental problem that needs to be addressed is that it takes time for an entity state update to travel from one host to another. 

Suppose the simulation user is looking at a virtual world application and sees an entity published by another host. Is the entity really where in the world the local display says it is? It depends. 

The host publishing the entity sends out state updates with the entity's position, but it takes time for the state update to get to our host. The application that publishes the entity prepares the update message, sets the timestamp field, and then sends the message out over the local operating system's network stack. The message must travel across the network, be delivered up our local host's network stack, and be handed off to our application. There our application parses the message, does any local processing or computations, and updates the graphics display. That time delay--the latency--between the publisher sending the message and the receiver that runs the display can cause problems when attempting to maintain the illusion of a shared world. Even if two applications are running on the same host the we can still see the effects of latency.  The time necessary for a message to traverse the network in the above list of latency components is only one element that contributes to overall latency; the application still must deal with the inherent latency of creating, copying, and parsing messages.

Latency can be a problem in many aspects of a networked simulation. Broadly speaking, latency can impact any _shared, dynamic_ state in virtual worlds. The entity's location is an example of this phenomenon. Someone looking at the display of a simulator, such as Close Combat Tactical Trainer, sees multiple soldier avatars, each controlled by a different host. Suppose he looks at a particular enemy solider on the display. Is the enemy soldier where our display says it is? It could be. For example if the entity hasn't moved for the last 30 minutes it's likely that the shared, dynamic state information for the simulation is consistent between the cooperating applications.  The state updates have been sent and received long ago, and the state hasn't changed. Both applications believe the entity to be at the same location. 

On the other hand, what if the solier entity is moving? The position of the avatar we see on our screen differs from what the host controlling the avatar believes the position to be.  Without any latency compenstation techniques, the difference in entity position between the two hosts will be Entity Velocity * Latency. The two hosts will see the same entity in different positions, depending on how fast the entity is moving and what the latency is. See figure x for an illustration of this principle.

Figure X

For some applications it might not matter if the data about entity positions is not completely consistent between hosts. Even if all of them are not exactly where we think they are, the entity positions may be close enough for the purposes of the application. A map-based display for an artillery training simulator may only care if the entities are within a few meters of their displayed location. On the other hand, a virtual or augmented reality application may need the hosts to agree on the location of entities to within a few centimers. In the end, it depends on what the simulation is trying to accomplish.

If we know what direction the entity controlled by the remote host is moving and what the latency between the hosts is, then we can use this information to mitigate the latency problem. We can use Dead reckoning (sometimes called extrapolation) of the entity's position in our application to guess about where the host controlling the entity believes the entity to be. If the entity is traveling in a straight line at a constant velocity and our application is using a dead reckoning algorithm that models that behavior, our application may have correctly estimated the location.  Displays on both hosts show the entity in the essentially the same position. 

It's important to note that the application needs an estimate of what the latency is between hosts. If we receive a state update, it's helpful to know how old that state update is. Does it reflect an entity position that is 10 ms old? 500 ms? 1000 ms? The timestamp field can give the receiving application an estimate of how old the entity update is. We can use that information to dead reckon an estimate of where the entity should be at the current time. See the dead reckoning section of this document for details. 

### Dead Reckoning to Mitigate Latency Effects

Suppose we write a "Fast and Furious" themed game for drag racing. Two drivers are in a head-to-head drag race in lanes next to each other, each driving virtual vehicles. There's 200 ms of latency between the simulators. What views do the simulators have of the other vehicle? What we want to replicate is what happens in real life: Dom looks left and sees exactly where Brian is, and Brian looks right to see exactly where Dom is at the current moment.

<img src="images/deadReckoningLatency/fastFuriousDragRace.jpg">
Figure x

But suppose there's a significant amount of latency between the two hosts in the game. Let's say it's 200 ms. Now who's ahead? When Dom looks left he sees this state of the game:

<img src="images/deadReckoningLatency/domAhead.jpg">

If we relied solely on dead reckoning used as above to reduce the number of update packets sent, all the updates we received would describe the state of the other Humvee 200 ms ago. At 100 mph (and highly optimistically assuming a Humvee could get up to 100 mph) that translates into a discrepancy about 9 meters in the position of the other vehicle. The other simulator would have a similary mistaken view of us. When the crossed the finish line each might think it won the race. We need another technique to reduce the effects of latency.

In DIS this can be done by using the timestamp field in combination with the dead reckoning algorithms. Every PDU contains a timestamp field that, if the standard is being followed, includes a measure of time since the top of the hour. When we receive the entity state PDU update we can perform a relatively simple operation: based on the specified DR algorithm, extrapolate the position of the entity. If the time is synchronized between hosts using a service such as Network Time Protocol (NPT) as discussed in the <a href="Timestamps.md">Timestamps</a> section, we can compare this to our own view of the current time. This will give us an estimate of the total latency for updates. We'll then run the DR algorithm and get an estimate of the true position of the other Humvee. DIS DR algorithm five is probably a good choice; it includes both velocity and acceleration in the DR computations, and ignores angular velocity. 

Note that we are now using DR for a reason distinct from decreasing bandwidth use. It's being used to descrease the effects of latency. This isn't cost free; running the DR algorithms for both the entities being received and the entities we publish can consume some computational resources. But in the end, most modern CPUs have several cores, and these tasks can often be parallelized. Simulations are often not restrained by their ability to do computations, but rather by graphics or I/O.

### Consistency vs. Throughput

There's a fundamental tradeoff between the consistency of the shared state information and how responsive the applications are. 

Imagine two applications, A and B, that implement a networked virtual environment. Each is publishing entities that move about randomly, so dead reckoning doesn't help us, because dead reckoning depends on predictable movement. Application A will see B's entities a little time lagged, and likewise for B's view of A's entities. It takes some time period, L, to transfer entity state information between the applications.

At first glance it might appear that we can solve this problem by insisting that all applications have the same shared, dynamic state. We'll demand that A and B have exactly the same state information at the same time and implement some clever algorithms to make this happen. The distributed database application world loves this stuff, and they've developed algorithms such as distributed two-phased commits that feature atomic (all or nothing) transactions in a networked environment. The problem is that this takes time, and during that time we can't effect still more changes to the dynamic shared state. 

Let's say application A publishes an entity position update. We start our clever algorithm for consistent shared state, such as a two phase commit (see additional reading) which involves sending an update, waiting for confirmation that application B is ready to commit the update, sending a commit request, and then waiting for a acknowledgement that the commit has occured. Notice that this requires several round trips, each of time L. Suddenly our update time is several multiples of L, and our state updates aren't ocurring very quickly. What happens if the user in application A pushes a joystick to moves his entity while we are running our two-phased commit algorithm for consistent shared state with the prior position update? If we insist on completely consistent application state, our application can't update the local entity's position until our distributed update has completed or rolled back. This is very frustrating for the user; he's pushing on the joystick to move his entity, but the entity's position isn't changing. All the applications have consistent shared state, but the throughput and responsiveness is unacceptable. It gets worse if there are also applications C, D, E, and F in the virtual world, and any of the messages in the two-phase commit dance are lost in transmission. Also, notice that the coordination messages needed to coordinate a consistent state take up bandwidth. 

We may not be able to ensure a completely consistent shared dynamic state at all if we also require real time simulation. In a real time simulation the simulation has to advance at the same rate as wall clock time. If several messages need to be exchanged, with L latency each time, we may not be able to execute the distributed transactions as quickly as the state changes occur in real time.

Yet another option is to keep the authoritative game state on a central server. This is a popular technique in commercial games. If someone shoots at someone else the positions of the shooter and target and any physics involved are resolved on the central server. For various reasons (including player cheating) this can be a useful technique, but in the end it in some ways just makes the problem look different--now we have to worry about consistency between the client and the authoritative central server instead of between clients. In any event DIS made the design choice to use an entirely peer-to-peer architecture with no central server at all, so we can ignore this option in the context of DIS.

In practice, applications accept the reality that the shared states of the applications can be a little out of sync with each other in order to get acceptable responsiveness, or even a real time simulation that works at all. The lack of synchronized game state creates other problems we'll discuss later.

### How Bad is Latency?

Suppose our application is an interactive map that receives DIS ESPDUs and plots their position. A battlefield map may not be adverely affected at all if the entities are one second out of sync with the publishing application because the inconsistencies in the map display are trivial compared to the accuracy the application needs. Virtual simulations are much more sensitive to latency issues, and are the most challenging application to get right. 

The entertainment industry has many first person shooter (FPS) games that demand attention to latency details. FPS games like DOOM, Call of Duty, and Counterstrike are very popular, and players may be scattered across a continent. They have very tight, "twitch" interactions between entities, usually involving player avatars shooting at each other, and this requires low latency and high precision. Simulations with entities that move in a more stately manner, perhaps a naval ship simulator, may have game play that allows slower reaction times because the latency-induced position errors are smaller and not as relevant to the user experience. The military often focuses on training for teamwork and procedures, even in virtual worlds, rather than tasks that require fast reaction time.
<commit_msg>Placeholder<commit_after>## Latency

There are several metrics that describe network traffic. Latency is the time it takes for a message to travel from one application to another. Latency is distinct from bandwidth, a measure of the amount of data that can be sent per time period. It's possible to have a high bandwidth, high latency connection. An example of a connection like this is a satellite in geosynchronus orbit over which we can transmit 100 MB/sec. That's high bandwidth, but at the same time it takes 250ms for a message to get from earth to the satillite and back.  That's a hard lower limit established by the speed of light; any message has to travel up to orbit at 72,000 KM and then back down. There can also be low bandwidth, low latency connection, over which we can't send very much data, but what data we do send arrives quickly.  Another metric of network performance is Packet Delay Variation (PDV), sometimes called jitter. Jitter is, loosely defined, the variance of the latency. There are multiple detailed definitions for how to measure PDV or jitter, but the "variation of the latency" term captures the essential meaning. The IETF RFC in the "further readings" section discusses the issues involved in defining a precise definition.

All three of these metrics--bandwidth, latency, and jitter--effect a networked virtual environment, and if any one of the metrics is terrible it may make the NVE implementation completely unworkable. Ask anyone who has tried to play a modern first person shooter game across a 56K dialup modem if you doubt this. Bandwidth is a classic measure, and the "Dead Reckoning: State Updates" section discusses some aspects of this problem and how to mitigate it. The problems posed by latency are less obvious. 

The latency issues are particulary relevant to military simulations. Military LVC applications often have data feeds that originate in an austere field deployment, and may disribute data via satellite links that are inherently laggy. 

Latency can create practical problems in virtual worlds, yet there are strategies to mitigate these problems.  This comes at the cost of more simulation application complexity if the programmer chooses to implement them.

### The Latency Problem

The essence of a virtual world is creating the illusion that application avatars from several hosts are interacting with each other in a shared virtual space. A fundamental problem that needs to be addressed is that it takes time for an entity state update to travel from one host to another. 

Suppose the simulation user is looking at a virtual world application and sees an entity published by another host. Is the entity really where in the world the local display says it is? It depends. 

The host publishing the entity sends out state updates with the entity's position, but it takes time for the state update to get to our host. The application that publishes the entity prepares the update message, sets the timestamp field, and then sends the message out over the local operating system's network stack. The message must travel across the network, be delivered up our local host's network stack, and be handed off to our application. There our application parses the message, does any local processing or computations, and updates the graphics display. That time delay--the latency--between the publisher sending the message and the receiver that runs the display can cause problems when attempting to maintain the illusion of a shared world. Even if two applications are running on the same host the we can still see the effects of latency.  The time necessary for a message to traverse the network in the above list of latency components is only one element that contributes to overall latency; the application still must deal with the inherent latency of creating, copying, and parsing messages.

Latency can be a problem in many aspects of a networked simulation. Broadly speaking, latency can impact any _shared, dynamic_ state in virtual worlds. The entity's location is an example of this phenomenon. Someone looking at the display of a simulator, such as Close Combat Tactical Trainer, sees multiple soldier avatars, each controlled by a different host. Suppose he looks at a particular enemy solider on the display. Is the enemy soldier where our display says it is? It could be. For example if the entity hasn't moved for the last 30 minutes it's likely that the shared, dynamic state information for the simulation is consistent between the cooperating applications.  The state updates have been sent and received long ago, and the state hasn't changed. Both applications believe the entity to be at the same location. 

On the other hand, what if the solier entity is moving? The position of the avatar we see on our screen differs from what the host controlling the avatar believes the position to be.  Without any latency compenstation techniques, the difference in entity position between the two hosts will be Entity Velocity * Latency. The two hosts will see the same entity in different positions, depending on how fast the entity is moving and what the latency is. See figure x for an illustration of this principle.

Figure X

For some applications it might not matter if the data about entity positions is not completely consistent between hosts. Even if all of them are not exactly where we think they are, the entity positions may be close enough for the purposes of the application. A map-based display for an artillery training simulator may only care if the entities are within a few meters of their displayed location. On the other hand, a virtual or augmented reality application may need the hosts to agree on the location of entities to within a few centimers. In the end, it depends on what the simulation is trying to accomplish.

If we know what direction the entity controlled by the remote host is moving and what the latency between the hosts is, then we can use this information to mitigate the latency problem. We can use Dead reckoning (sometimes called extrapolation) of the entity's position in our application to guess about where the host controlling the entity believes the entity to be. If the entity is traveling in a straight line at a constant velocity and our application is using a dead reckoning algorithm that models that behavior, our application may have correctly estimated the location.  Displays on both hosts show the entity in the essentially the same position. 

It's important to note that the application needs an estimate of what the latency is between hosts. If we receive a state update, it's helpful to know how old that state update is. Does it reflect an entity position that is 10 ms old? 500 ms? 1000 ms? The timestamp field can give the receiving application an estimate of how old the entity update is. We can use that information to dead reckon an estimate of where the entity should be at the current time. See the dead reckoning section of this document for details. 

### Dead Reckoning to Mitigate Latency Effects

Suppose we write a "Fast and Furious" themed game for drag racing. Two drivers are in a head-to-head drag race in lanes next to each other, each driving virtual vehicles. There's 200 ms of latency between the simulators. What views do the simulators have of the other vehicle? What we want to replicate is what happens in real life: Dom looks left and sees exactly where Brian is, and Brian looks right to see exactly where Dom is at the current moment.

<img src="images/deadReckoningLatency/fastFuriousDragRace.jpg">
Figure x

But suppose there's a significant amount of latency between the two hosts in the game. Let's say it's 100 ms. The player controlling Dom looks left he sees this state of the game:

<img src="images/deadReckoningLatency/domAhead.jpg">

Who's really ahead?

*****

If we relied solely on dead reckoning used as above to reduce the number of update packets sent, all the updates we received would describe the state of the other Humvee 200 ms ago. At 100 mph (and highly optimistically assuming a Humvee could get up to 100 mph) that translates into a discrepancy about 9 meters in the position of the other vehicle. The other simulator would have a similary mistaken view of us. When the crossed the finish line each might think it won the race. We need another technique to reduce the effects of latency.

In DIS this can be done by using the timestamp field in combination with the dead reckoning algorithms. Every PDU contains a timestamp field that, if the standard is being followed, includes a measure of time since the top of the hour. When we receive the entity state PDU update we can perform a relatively simple operation: based on the specified DR algorithm, extrapolate the position of the entity. If the time is synchronized between hosts using a service such as Network Time Protocol (NPT) as discussed in the <a href="Timestamps.md">Timestamps</a> section, we can compare this to our own view of the current time. This will give us an estimate of the total latency for updates. We'll then run the DR algorithm and get an estimate of the true position of the other Humvee. DIS DR algorithm five is probably a good choice; it includes both velocity and acceleration in the DR computations, and ignores angular velocity. 

Note that we are now using DR for a reason distinct from decreasing bandwidth use. It's being used to descrease the effects of latency. This isn't cost free; running the DR algorithms for both the entities being received and the entities we publish can consume some computational resources. But in the end, most modern CPUs have several cores, and these tasks can often be parallelized. Simulations are often not restrained by their ability to do computations, but rather by graphics or I/O.

### Consistency vs. Throughput

There's a fundamental tradeoff between the consistency of the shared state information and how responsive the applications are. 

Imagine two applications, A and B, that implement a networked virtual environment. Each is publishing entities that move about randomly, so dead reckoning doesn't help us, because dead reckoning depends on predictable movement. Application A will see B's entities a little time lagged, and likewise for B's view of A's entities. It takes some time period, L, to transfer entity state information between the applications.

At first glance it might appear that we can solve this problem by insisting that all applications have the same shared, dynamic state. We'll demand that A and B have exactly the same state information at the same time and implement some clever algorithms to make this happen. The distributed database application world loves this stuff, and they've developed algorithms such as distributed two-phased commits that feature atomic (all or nothing) transactions in a networked environment. The problem is that this takes time, and during that time we can't effect still more changes to the dynamic shared state. 

Let's say application A publishes an entity position update. We start our clever algorithm for consistent shared state, such as a two phase commit (see additional reading) which involves sending an update, waiting for confirmation that application B is ready to commit the update, sending a commit request, and then waiting for a acknowledgement that the commit has occured. Notice that this requires several round trips, each of time L. Suddenly our update time is several multiples of L, and our state updates aren't ocurring very quickly. What happens if the user in application A pushes a joystick to moves his entity while we are running our two-phased commit algorithm for consistent shared state with the prior position update? If we insist on completely consistent application state, our application can't update the local entity's position until our distributed update has completed or rolled back. This is very frustrating for the user; he's pushing on the joystick to move his entity, but the entity's position isn't changing. All the applications have consistent shared state, but the throughput and responsiveness is unacceptable. It gets worse if there are also applications C, D, E, and F in the virtual world, and any of the messages in the two-phase commit dance are lost in transmission. Also, notice that the coordination messages needed to coordinate a consistent state take up bandwidth. 

We may not be able to ensure a completely consistent shared dynamic state at all if we also require real time simulation. In a real time simulation the simulation has to advance at the same rate as wall clock time. If several messages need to be exchanged, with L latency each time, we may not be able to execute the distributed transactions as quickly as the state changes occur in real time.

Yet another option is to keep the authoritative game state on a central server. This is a popular technique in commercial games. If someone shoots at someone else the positions of the shooter and target and any physics involved are resolved on the central server. For various reasons (including player cheating) this can be a useful technique, but in the end it in some ways just makes the problem look different--now we have to worry about consistency between the client and the authoritative central server instead of between clients. In any event DIS made the design choice to use an entirely peer-to-peer architecture with no central server at all, so we can ignore this option in the context of DIS.

In practice, applications accept the reality that the shared states of the applications can be a little out of sync with each other in order to get acceptable responsiveness, or even a real time simulation that works at all. The lack of synchronized game state creates other problems we'll discuss later.

### How Bad is Latency?

Suppose our application is an interactive map that receives DIS ESPDUs and plots their position. A battlefield map may not be adverely affected at all if the entities are one second out of sync with the publishing application because the inconsistencies in the map display are trivial compared to the accuracy the application needs. Virtual simulations are much more sensitive to latency issues, and are the most challenging application to get right. 

The entertainment industry has many first person shooter (FPS) games that demand attention to latency details. FPS games like DOOM, Call of Duty, and Counterstrike are very popular, and players may be scattered across a continent. They have very tight, "twitch" interactions between entities, usually involving player avatars shooting at each other, and this requires low latency and high precision. Simulations with entities that move in a more stately manner, perhaps a naval ship simulator, may have game play that allows slower reaction times because the latency-induced position errors are smaller and not as relevant to the user experience. The military often focuses on training for teamwork and procedures, even in virtual worlds, rather than tasks that require fast reaction time.

{'fraction_non_alphanumeric': 0.02397453327585635, 'fraction_numerical': 0.0026527837649633584, 'mean_word_length': 4.61183475995534, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3791519', 'n_tokens_mistral': 6602, 'n_tokens_neox': 6154, 'n_words': 4962}