Current State of Commercial Wearable Technology in Physical Activity Monitoring 2015-2017.

Wearable physical activity trackers are a popular and useful method to collect biometric information at rest and during exercise. The purpose of this systematic review was to summarize recent findings of wearable devices for biometric information related to steps, heart rate, and caloric expenditure for several devices that hold a large portion of the market share. Searches were conducted in both PubMed and SPORTdiscus. Filters included: humans, within the last 5 years, English, full-text, and adult 19+ years. Manuscripts were retained if they included an exercise component of 5-min or greater and had 20 or more participants. A total of 10 articles were retained for this review. Overall, wearable devices tend to underestimate energy expenditure compared to criterion laboratory measures, however at higher intensities of activity energy expenditure is underestimated. All wrist and forearm devices had a tendency to underestimate heart rate, and this error was generally greater at higher exercise intensities and those that included greater arm movement. Heart rate measurement was also typically better at rest and while exercising on a cycle ergometer compared to exercise on a treadmill or elliptical machine. Step count was underestimated at slower walking speeds and in free-living conditions, but improved accuracy at faster speeds. The majority of the studies reviewed in the present manuscript employed different methods to assess validity and reliability of wearable technology, making it difficult to compare devices. Standardized protocols would provide guidance for researchers to evaluate research-grade devices as well as commercial devices used by the lay public.


INTRODUCTION
Commercial wearable technology has seen an expansive increase in personal use as well as application in industries including medicine, healthcare, and the military. Within the area of health and fitness, wearable technology was determined to be the top worldwide fitness trend in 2016 (14), and 2017 (15). Because of this growing trend, and the ubiquity of smart-linked Additionally, there is often confusion on the part of consumers regarding which device is optimal for personal use. It is hoped that the results of this systematic review can assist consumers to make informed decisions when deciding to purchase a particular device. Toward this end, we have included evidence-based summaries of specific devices. We hope that this paper can be a resource for both researchers as well as personal consumers wishing to utilize wearable technology devices for physical activity monitoring.

Protocol
The most popular devices on the market (6) were chosen for this systematic review (12) and included: Fitbit, Garmin, Apple, Misfit, Samsung Gear, TomTom, and Lumo. The searched terms were: "(Device name) AND (validity OR validation OR validate OR comparison OR comparisons OR comparative OR reliability OR accuracy). Searches were conducted in both PubMed and SPORTdiscus. Search filters included: humans, last 5 years, English, full-text, and adult 19+ years. One researcher went through the articles listed for each device and determined if the manuscript was truly related to validation, accuracy, and reliability. Only articles related to these topics were deemed pertinent and were retained for further assessment. Table 1 shows the number of articles found and relevant for each device searched. Of these articles, those that had an exercise component of 5-min in duration or greater and had 20 participants or more were included in order to correspond with the Consumer Technology Association standard for measuring step counts on consumer wearable activity monitoring devices (4). For energy expenditure assessment, only studies that used a validated metabolic analysis system as the criterion measure were included. For step count, only studies using video or visual step count as the criterion measure were included. Studies that solely assessed sleep or exercise rehabilitation and those that were included in previous systematic reviews (7) were excluded. Two independent evaluators utilized the stated criteria, and agreed upon the final papers that were included, N = 10. Several articles included assessment of multiple devices, including devices that were not part of the original search. Data on these devices are reported in this systematic review.
International Journal of Exercise Science http://www.intjexersci.com 505 Table 1. Number of articles found in the original search and after assessment for relevance to wearable  technology, validity, accuracy, and reliability.  Device  Articles found  Pertinent articles  Fitbit  41  24  Garmin  11  11  Apple  106  9  Misfit  68  5  Samsung Gear  2  2  TomTom  4 2 Lumo 0 0

ENERGY EXPENDITURE
In general, the Fitbit family of products seem to have the higher validity compared to other wearable devices when estimating energy expenditure, and Jawbone lower (see table 2). Overall, wearable devices tend to underestimate energy expenditure compared to criterion laboratory measures (Oxycon Mobile, CosMed K4b2, or MetaMax 3B), however at higher intensities of activity energy expenditure is underestimated. Additionally, while wearable technology devices are better at estimating energy expenditure during sedentary activities, validity becomes poorer as exercise begins, and gets worse as the intensity increases. Future studies should continue to determine the optimal levels of intensity that will return acceptable validity measurements. Regarding the analysis of validity, all studies reviewed utilized the Bland-Altman procedure for determining agreement (3,13,16,17), two of four incorporated correlation techniques (Pearson Product Moment, or Rho) (3,16), and three of four reported the mean absolute percent error (3,13,17). It is recommended that future investigations utilize all three indicators of validity to allow for ease of comparison between devices. Furthermore, it is recommended that a common unit of measure be reported with respect to energy expenditure to allow comparison between studies. In this case, three of four studies reported energy expenditure in terms of overall calories expended (kcal). No study included in this current systematic review of energy expenditure estimation in wearable technology devices reported test-retest reliability measures. Future studies should include this measure so that consumers and researchers will be able to make informed decisions regarding both validity and reliability of devices.
International Journal of Exercise Science http://www.intjexersci.com 506  Each of the studies reviewed included an analysis of error (either mean percent error or absolute percent error), a correlation assessment with the criterion (either Lin's concordance, intraclass correlations, or Pearson Product Moment), and utilized the Bland-Altman method for evaluating agreement and error. Criterion assessment in the studies evaluated included an electrocardiogram (ECG) or Polar chest strap. Wrist and forearm activity monitors had a wide range of accuracy, with the Apple iWatch having the lowest mean absolute percent error (MAPE) and the Fitbit devices having the highest MAPE. The details of the study are shown in Table 3. All wrist and forearm devices had a tendency to underestimate heart rate, and this error was generally greater at higher exercise intensities and those that included greater arm movement. Heart rate measurement was also typically better at rest and while exercising on a cycle ergometer compared to exercise on a treadmill or elliptical machine. One study included a Polar chest strap as a tested device compared to an ECG, and the chest strap had the lowest MAPE and highest concordance compared to the wrist and forearm devices.
The three studies assessed used different correlation assessments and methods for evaluating error. An industry standard for reporting these two values would be useful. Of equal importance is to ensure proper wear of the devices and avoid simultaneously wearing multiple devices on one arm. Devices were worn properly and according to manufacturers' International Journal of Exercise Science http://www.intjexersci.com 508 guidelines in all three of the heart rate studies assessed in this review. Only one of the studies used continuous heart rate assessment (recorded by the devices each second) and the other two studies recorded heart rate at specific times after reaching steady state during exercise. Only the Fitbit Charge HR device was tested in both types of studies, and the results of the continuous study were less favorable. Assessing heart rate at specific intervals after reaching steady state eliminates the oscillation of heart rate with changing intensities. The response rate of the device is important to assess with changing intensities and second-by-second analysis for heart rate accuracy should be encouraged in development and evaluation of these devices. Further, this is important because devices have shown to have lag in readings of heart rate and data dropout. Gillinov et al. was the only study to address this issue, and the authors were transparent regarding which devices had errors and how many data points were missed or removed (9). While this transparency is useful, it falsely increases the accuracy of the devices by removing bad data.

STEP COUNT
Each of the studies reviewed utilized either a direct hand-tally count (1,8,13) or video recording (2,10) to serve as the criterion measure for step count. The methods used to assess validity were varied amongst the studies. Three of the five studies used MAPE (1,8,13), whereas two of the five used absolute percentage error (APE) (2, 10). Three of the five studies (1,2,8) utilized Bland-Altman plots to show 95% limits of agreement, but only one study (1) calculated correlation coefficients. All five studies utilized a treadmill protocol, with speeds ranging from 2-5 mph, while spending 3-10 minutes at each incremental speed. In addition to a treadmill protocol, three of the five studies also included an over-ground condition (1,10,13), and two studies (1, 13) examined validity in free-living conditions.
Collectively, there was wide variability and accuracy across the various physical activity monitor brands for both speed and condition, as shown in Table 4. The studies reviewed consistently demonstrated reduced validity in terms of underestimating step counts at both slower walking speeds (< 2 mph), ambulatory, and free-living conditions, but improved accuracy at faster speeds, which is consistent with previous research. The Fitbit One and Fitbit Zip consistently demonstrated MAPE < 5% and were noted by multiple studies to be the most accurate (1,10,13). Conversely, multiple studies (1,8,10) found the Nike+ FuelBand and Polar Loop to be the least accurate, with MAPE >10%.
Challenges for future research of the step count feature on activity trackers are not unlike what was observed within energy expenditure and heart rate validation studies. Whereas the treadmill protocols reviewed adhered relatively close to Consumer Technology Association (CTA) step count standards (4), over-ground and free-living conditions lack a standardized protocol, and are a notable limitation within the literature. In regards to the assessment of validity, no standardized threshold exists for what constitutes high or low MAPE, thus in the studies reviewed, a wide range of cutoff criterion for acceptable MAPE was observed. Likewise, whereas Bland-Altman plots are commonly constructed to show limits of agreement, the studies reviewed were very inconsistent in actually providing 95% limits of agreement International Journal of Exercise Science http://www.intjexersci.com 509 between the criterion measure and each respective device (mean difference ± 1.96 SD of the differences), making direct comparisons difficult.

Apple Watch
The Apple Watch has been evaluated in four recent investigations (3,8,9,16). Evidence indicates that validity to criterion measures is acceptable for heart rate and step count (see table 5). Researchers and consumers should view energy expenditure output with caution. Basis Band (B1, Peak) Two devices from the Basis brand were evaluated in recent studies (1,11,17). The devices appear to be valid for steps and heart rate, but not for energy expenditure (see Table 6). Both of these devices have been discontinued.

Fitbit Family of Devices
The Fitbit Charge HR had the greatest influence in the recent literature (3,8,11,16) and has good validity for heart rate (see table 7). Validity for energy expenditure and step count are lower than what is observed for heart rate.
The Fitbit Flex was utilized in three recent investigations that evaluated step validity (1,2,13). All studies reported MAPE, and taken together are outside of the acceptable 5-10% error for controlled or free-living investigations. However, one study reported high ICC and relatively narrow limits of agreement (1). Only one study using the Fitbit Flex determined that energy expenditure MAPE was greater than acceptable error (13).
Similarly, the Fitbit Zip was observed to have higher than acceptable step MAPE in two studies (1, 13), but a good ICC and narrow limits of agreement in the only investigation to report these values (1). Again, only one study using the Fitbit Zip determined that energy expenditure MAPE displayed greater than acceptable error (13).
A single recent investigation has evaluated the Fitbit One, and found poor MAPE values for both step and energy expenditure (13). Finally, a single investigation utilizing the Fitbit Blaze found that heart rate was valid (9).

Garmin Family of Devices
The Garmin Vivofit was evaluated in three recent studies (1,2,17). While this device appears to be valid for counting steps (see table 8), energy expenditure measurements are outside of acceptable limits. The Vivofit device itself does not provide a measure of heart rate (it can be paired with a heart rate monitor to read through the device), and as such does not have an individual assessment for this variable.
A single recent investigation evaluated the Garmin Vivosmart (8), and focused exclusively on the step count measurement. Overall, the Vivosmart can return an accurate step count at most walking speeds (see table 8).
Only one recent study evaluated the Garmin Forerunner 235 (9), and opted to evaluate heart rate as the sole dependent variable. Overall, the Forerunner 235 provides valid heart rate measurements (see table 8).

Jawbone Up24, Move
Jawbone devices were recently evaluated in four studies (1)(2)(3)13) for steps and energy expenditure (table 9). Neither device measures heart rate. The Jawbone Move device only measures steps and sleep and can be placed either on the waist or worn around the wrist. The Move is more accurate for steps when worn at waist level. The Up24 device has been discontinued by Jawbone. Misfit Family of Devices One investigation evaluated the Misfit Shine (1), and another the Misfit Flash (8). While both devices provide an estimate of energy expenditure, the studies evaluated step count only. The accuracy of step count in these devices is low (see table 10).

Polar Loop
Step count validity of the Polar Loop was evaluated in two recent studies (1,8). The device can return an estimate of energy expenditure, and must be connected to a separate heart rate monitor, however these variables were not evaluated. Average validity measurements indicate that step count validity of the Polar Loop is low (ICC = 0.460, MAPE = 15.33%, Agreement = -161.4 to 328.1).

Samsung Gear S
Two recent studies reported validity measurements in the Samsung Gear S (8,16). Energy expenditure estimates and heart rate appears to be valid (see table 11).
Step count obtained from the Samsung Gear S has acceptable ICC and MAPE, but wide limits of agreement. .

RECOMMENDATIONS
The majority of the studies reviewed in the present manuscript employed different methods to assess validity and reliability of wearable technology. This difference in protocols makes it difficult to compare devices. The Consumer Technology Association (CTA) recently published validation criteria and protocols to evaluate devices in a standardized format (4). The CTA standard is set up for laboratory-based assessment of steps only, but provides a strong basis for comparison of devices. The CTA has also published a standard for evaluating devices for sleep validity (5), and a standard for heart rate is expected to be released in 2018. More standards and protocols should be developed to include heart rate, energy expenditure, and free-living conditions. These standards would provide guidance for researchers to evaluate research-grade devices as well as commercial devices used by the lay public.
International Journal of Exercise Science http://www.intjexersci.com 514 Following these guidelines, it is recommended that exercise duration be at least 5-minutes in length in order to allow subjects to attain steady state measures. Additionally, regardless of exercise mode, it is suggested that at least two different exercise intensities be employed to allow for comparison. Furthermore, as there is a need to obtain reliability measures on devices, it is recommended that study designs be utilized to address this need. Thus future research on wearable technology devices should address both the question of validity as well as reliability. With respect to determining accuracy, it is recommended that future investigations address validity utilizing ICC, MAPE, and agreement to an established criterion measure (i.e. Bland-Altman analysis) in order to present overall evidence of validity. As the landscape of wearable technology devices is expanding, producing high-quality evidence of device accuracy and reliability will continue to be important to investigators wishing to utilize these items for research as well as general consumers that employ them for personal use.