137 hours compute hours every ~6 minutes – John O'Duinn's blog Skip to content John O'Duinn's blog Menu My “Distributed Teams” Book Jobs for Remoties Tools for Remoties Locations for Remoties Playbooks for Distributed Teams About me… Quotes 137 hours compute hours every ~6 minutes Twitter Linkedin email Reddit Followup from today’s Dev platform meeting. We talk a lot about how the number of checkins continue to rise – its always great to see developers being able to do more work – and the “Infrastructure load” series of posts is one way to measure that. However, its worth noting that the amount of work we trigger per checkin also continues to rise. Some of this is good (new builds, new test suites), but some of this is bad (consistently failing tests; obsolete tests). How many hours of builds and tests do we run per commit? In 2009, each commit triggered ~40 compute hours In Jan 2012, each commit triggered 110 compute hours In Aug 2012, each commit triggered 137 compute hours 137 hours of builds and tests per checkin is a lot – especially when we do a checkin every ~6 minutes. Put another way, that’s ~3.5 years of builds and tests being run per day. If you look at the “wait times” posts in dev.tree-management, you’ll see that we’re having a tough time keeping up. We’re busily venting excess production load over to AWS for nativefennec / B2G desktop builds / linux desktop …. and are then recycling existing hardware to help grow inhouse capacity for win32 and win64. But you can help. Please file a bug if you know of: any build or test that is being run but no longer needed OR any build or test being hidden on tbpl.m.o because it constantly fails We’ll disable the job to stop wasting CPU time, which will help everyone’s wait times. As needed, we’ll also file bugs to track fixing the problem and then reenable once the build/test is running green. While any one given test suite might not seem like a lot of time to save, we run all these suites literally thousands of times per day. Every little saving helps makes things faster for everyone! Thank you! =========================================== Full list (with duration of job in seconds) Android Armv6 mozilla-central build 1523.29213483 Android Debug mozilla-central build 1461.34090909 Android Tegra 250 mozilla-central opt test crashtest-2 1509.39215686 Android Tegra 250 mozilla-central opt test crashtest-3 1627.30769231 Android Tegra 250 mozilla-central opt test jsreftest-1 1635.7047619 Android Tegra 250 mozilla-central opt test jsreftest-2 2023.08181818 Android Tegra 250 mozilla-central opt test jsreftest-3 1498.89215686 Android Tegra 250 mozilla-central opt test mochitest-1 1723.55445545 Android Tegra 250 mozilla-central opt test mochitest-2 1433.60909091 Android Tegra 250 mozilla-central opt test mochitest-3 1533.75247525 Android Tegra 250 mozilla-central opt test mochitest-4 1311.97222222 Android Tegra 250 mozilla-central opt test mochitest-5 1332.30392157 Android Tegra 250 mozilla-central opt test mochitest-6 1373.30188679 Android Tegra 250 mozilla-central opt test mochitest-7 1373.56190476 Android Tegra 250 mozilla-central opt test mochitest-8 1691.24528302 Android Tegra 250 mozilla-central opt test reftest-1 1481.94166667 Android Tegra 250 mozilla-central opt test reftest-2 1585.47572816 Android Tegra 250 mozilla-central opt test reftest-3 1606.37864078 Android Tegra 250 mozilla-central opt test robocop 1813.01941748 Android Tegra 250 mozilla-central talos remote-tdhtml 1506.13861386 Android Tegra 250 mozilla-central talos remote-tp4m_nochrome 1592.23529412 Android Tegra 250 mozilla-central talos remote-trobocheck 1398.76146789 Android Tegra 250 mozilla-central talos remote-trobocheck2 1584.97142857 Android Tegra 250 mozilla-central talos remote-trobocheck3 1460.77669903 Android Tegra 250 mozilla-central talos remote-trobopan 1567.03883495 Android Tegra 250 mozilla-central talos remote-troboprovider 1592.09345794 Android Tegra 250 mozilla-central talos remote-ts 1761.95145631 Android Tegra 250 mozilla-central talos remote-tsvg 1490.09803922 Android XUL Tegra 250 mozilla-central opt test crashtest-2 1647.91428571 Android XUL Tegra 250 mozilla-central opt test crashtest-3 1430.54368932 Android XUL Tegra 250 mozilla-central opt test jsreftest-1 1550.04761905 Android XUL Tegra 250 mozilla-central opt test jsreftest-2 1593.39805825 Android XUL Tegra 250 mozilla-central opt test jsreftest-3 1754.47058824 Android XUL Tegra 250 mozilla-central opt test mochitest-1 1409.15686275 Android XUL Tegra 250 mozilla-central opt test mochitest-2 1521.00980392 Android XUL Tegra 250 mozilla-central opt test mochitest-3 1230.08737864 Android XUL Tegra 250 mozilla-central opt test mochitest-4 1525.06796117 Android XUL Tegra 250 mozilla-central opt test mochitest-5 1431.87254902 Android XUL Tegra 250 mozilla-central opt test mochitest-6 1476.26923077 Android XUL Tegra 250 mozilla-central opt test mochitest-7 1272.8490566 Android XUL Tegra 250 mozilla-central opt test mochitest-8 1799.10576923 Android XUL Tegra 250 mozilla-central opt test reftest-1 1569.52427184 Android XUL Tegra 250 mozilla-central opt test reftest-2 1658.66666667 Android XUL Tegra 250 mozilla-central opt test reftest-3 1551.54205607 Android XUL mozilla-central build 1277.25 Android mozilla-central build 1486.5 B2G gb_armv7a_gecko mozilla-central build 1264.95652174 B2G gb_armv7a_gecko-debug mozilla-central build 1539.45977011 B2G ics_armv7a_gecko mozilla-central build 1498.29850746 B2G ics_armv7a_gecko-debug mozilla-central build 1439.40298507 Linux mozilla-central build 1429.60227273 Linux mozilla-central leak test build 1413.39772727 Linux x86-64 mozilla-central build 1362.26136364 Linux x86-64 mozilla-central leak test build 1859.20454545 OS X 10.7 64-bit mozilla-central leak test build 1418.80681818 OS X 10.7 mozilla-central build 1324.79545455 Rev3 Fedora 12 mozilla-central debug test crashtest 1793.67857143 Rev3 Fedora 12 mozilla-central debug test jetpack 1527.98809524 Rev3 Fedora 12 mozilla-central debug test jsreftest 1477.39285714 Rev3 Fedora 12 mozilla-central debug test mochitest-other 1443.39285714 Rev3 Fedora 12 mozilla-central debug test mochitests-1/5 1704.18823529 Rev3 Fedora 12 mozilla-central debug test mochitests-2/5 1576.8452381 Rev3 Fedora 12 mozilla-central debug test mochitests-3/5 1615.10714286 Rev3 Fedora 12 mozilla-central debug test mochitests-4/5 1448.51190476 Rev3 Fedora 12 mozilla-central debug test mochitests-5/5 1751.01190476 Rev3 Fedora 12 mozilla-central debug test reftest 1603.91666667 Rev3 Fedora 12 mozilla-central debug test xpcshell 1416.51190476 Rev3 Fedora 12 mozilla-central opt test crashtest 1610.98809524 Rev3 Fedora 12 mozilla-central opt test crashtest-ipc 1559.86904762 Rev3 Fedora 12 mozilla-central opt test jetpack 1546.75 Rev3 Fedora 12 mozilla-central opt test jsreftest 1443.92857143 Rev3 Fedora 12 mozilla-central opt test mochitest-other 1358.32142857 Rev3 Fedora 12 mozilla-central opt test mochitests-1/5 1444.32941176 Rev3 Fedora 12 mozilla-central opt test mochitests-2/5 1247.67857143 Rev3 Fedora 12 mozilla-central opt test mochitests-3/5 1574.0952381 Rev3 Fedora 12 mozilla-central opt test mochitests-4/5 1452.3452381 Rev3 Fedora 12 mozilla-central opt test mochitests-5/5 1345.52380952 Rev3 Fedora 12 mozilla-central opt test peptest 1388.17857143 Rev3 Fedora 12 mozilla-central opt test reftest 1365.30952381 Rev3 Fedora 12 mozilla-central opt test reftest-ipc 1832.04761905 Rev3 Fedora 12 mozilla-central opt test reftest-no-accel 1419.16666667 Rev3 Fedora 12 mozilla-central opt test xpcshell 1645.51764706 Rev3 Fedora 12 mozilla-central talos chromer 1427.22619048 Rev3 Fedora 12 mozilla-central talos dirty 1282.8 Rev3 Fedora 12 mozilla-central talos dirtypaint 1629.80952381 Rev3 Fedora 12 mozilla-central talos dromaeo 1365.82222222 Rev3 Fedora 12 mozilla-central talos dromaeojs 1465.46153846 Rev3 Fedora 12 mozilla-central talos jstests 1725.84444444 Rev3 Fedora 12 mozilla-central talos nochromer 1409.21428571 Rev3 Fedora 12 mozilla-central talos other 1230.77380952 Rev3 Fedora 12 mozilla-central talos svgr 1486.88095238 Rev3 Fedora 12 mozilla-central talos tpn 1460.95238095 Rev3 Fedora 12×64 mozilla-central debug test crashtest 1460.81176471 Rev3 Fedora 12×64 mozilla-central debug test jetpack 1299.35294118 Rev3 Fedora 12×64 mozilla-central debug test jsreftest 1620.12790698 Rev3 Fedora 12×64 mozilla-central debug test mochitest-other 1441.22727273 Rev3 Fedora 12×64 mozilla-central debug test mochitests-1/5 1636.98888889 Rev3 Fedora 12×64 mozilla-central debug test mochitests-2/5 1292.28235294 Rev3 Fedora 12×64 mozilla-central debug test mochitests-3/5 1520.34117647 Rev3 Fedora 12×64 mozilla-central debug test mochitests-4/5 1539.4 Rev3 Fedora 12×64 mozilla-central debug test mochitests-5/5 1563.67058824 Rev3 Fedora 12×64 mozilla-central debug test reftest 1460.58823529 Rev3 Fedora 12×64 mozilla-central debug test xpcshell 1292.23529412 Rev3 Fedora 12×64 mozilla-central opt test crashtest 1539.57647059 Rev3 Fedora 12×64 mozilla-central opt test jetpack 1716.28235294 Rev3 Fedora 12×64 mozilla-central opt test jsreftest 1619.31764706 Rev3 Fedora 12×64 mozilla-central opt test mochitest-other 1523.48837209 Rev3 Fedora 12×64 mozilla-central opt test mochitests-1/5 1461.10588235 Rev3 Fedora 12×64 mozilla-central opt test mochitests-2/5 1548.6744186 Rev3 Fedora 12×64 mozilla-central opt test mochitests-3/5 1630.16470588 Rev3 Fedora 12×64 mozilla-central opt test mochitests-4/5 1540.01176471 Rev3 Fedora 12×64 mozilla-central opt test mochitests-5/5 1639.5047619 Rev3 Fedora 12×64 mozilla-central opt test peptest 1845.95294118 Rev3 Fedora 12×64 mozilla-central opt test reftest 1786.08235294 Rev3 Fedora 12×64 mozilla-central opt test xpcshell 1321.78823529 Rev3 Fedora 12×64 mozilla-central talos chromer 1399.71764706 Rev3 Fedora 12×64 mozilla-central talos dirty 1582.13333333 Rev3 Fedora 12×64 mozilla-central talos dirtypaint 1609.03529412 Rev3 Fedora 12×64 mozilla-central talos dromaeo 1793.82222222 Rev3 Fedora 12×64 mozilla-central talos dromaeojs 1266.35 Rev3 Fedora 12×64 mozilla-central talos jstests 1830.17777778 Rev3 Fedora 12×64 mozilla-central talos nochromer 1310.87058824 Rev3 Fedora 12×64 mozilla-central talos other 1684.15294118 Rev3 Fedora 12×64 mozilla-central talos svgr 1864.89411765 Rev3 Fedora 12×64 mozilla-central talos tpn 1607.82352941 Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test crashtest 1567.03960396 Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test jetpack 1587.17821782 Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test jsreftest 1711.85148515 Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test mochitest-other 1638.87128713 Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test mochitests-1/5 1424.64356436 Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test mochitests-2/5 1589.06930693 Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test mochitests-3/5 1592.65346535 Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test mochitests-4/5 1477.56435644 Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test mochitests-5/5 1662.66336634 Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test peptest 1594.99009901 Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test reftest 1704.13861386 Rev3 MacOSX Leopard 10.5.8 mozilla-central opt test xpcshell 1524.73267327 Rev3 MacOSX Leopard 10.5.8 mozilla-central talos chromer 1540.58415842 Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirty 1663.2037037 Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dirtypaint 1493.07920792 Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dromaeo 1467.05555556 Rev3 MacOSX Leopard 10.5.8 mozilla-central talos dromaeojs 1631.04255319 Rev3 MacOSX Leopard 10.5.8 mozilla-central talos jstests 1470.88888889 Rev3 MacOSX Leopard 10.5.8 mozilla-central talos nochromer 1586.46534653 Rev3 MacOSX Leopard 10.5.8 mozilla-central talos other 1580.38613861 Rev3 MacOSX Leopard 10.5.8 mozilla-central talos svgr 1610.73267327 Rev3 MacOSX Leopard 10.5.8 mozilla-central talos tpn 1768.98019802 Rev3 WINNT 5.1 mozilla-central debug test crashtest 1346.4 Rev3 WINNT 5.1 mozilla-central debug test jetpack 1568.725 Rev3 WINNT 5.1 mozilla-central debug test jsreftest 1465.05 Rev3 WINNT 5.1 mozilla-central debug test mochitest-other 1893.2625 Rev3 WINNT 5.1 mozilla-central debug test mochitests-1/5 1669.0875 Rev3 WINNT 5.1 mozilla-central debug test mochitests-2/5 1697.8875 Rev3 WINNT 5.1 mozilla-central debug test mochitests-3/5 1615.8375 Rev3 WINNT 5.1 mozilla-central debug test mochitests-4/5 1627.75 Rev3 WINNT 5.1 mozilla-central debug test mochitests-5/5 1597.425 Rev3 WINNT 5.1 mozilla-central debug test reftest 1757.05 Rev3 WINNT 5.1 mozilla-central debug test xpcshell 1720.375 Rev3 WINNT 5.1 mozilla-central opt test crashtest 1719.84146341 Rev3 WINNT 5.1 mozilla-central opt test jetpack 1640.7804878 Rev3 WINNT 5.1 mozilla-central opt test jsreftest 1725.46341463 Rev3 WINNT 5.1 mozilla-central opt test mochitest-other 1554.07317073 Rev3 WINNT 5.1 mozilla-central opt test mochitests-1/5 1405.34146341 Rev3 WINNT 5.1 mozilla-central opt test mochitests-2/5 1378.73170732 Rev3 WINNT 5.1 mozilla-central opt test mochitests-3/5 1467.54878049 Rev3 WINNT 5.1 mozilla-central opt test mochitests-4/5 1545.7195122 Rev3 WINNT 5.1 mozilla-central opt test mochitests-5/5 1382.70731707 Rev3 WINNT 5.1 mozilla-central opt test peptest 1382.92682927 Rev3 WINNT 5.1 mozilla-central opt test reftest 1650.84146341 Rev3 WINNT 5.1 mozilla-central opt test xpcshell 1608.17073171 Rev3 WINNT 5.1 mozilla-central talos chromer 1694.86585366 Rev3 WINNT 5.1 mozilla-central talos dirty 1556.30952381 Rev3 WINNT 5.1 mozilla-central talos dirtypaint 1651.80487805 Rev3 WINNT 5.1 mozilla-central talos dromaeo 1415.38095238 Rev3 WINNT 5.1 mozilla-central talos dromaeojs 1341.225 Rev3 WINNT 5.1 mozilla-central talos jstests 1612.71428571 Rev3 WINNT 5.1 mozilla-central talos nochromer 1556.0 Rev3 WINNT 5.1 mozilla-central talos other 1326.64634146 Rev3 WINNT 5.1 mozilla-central talos svgr 1524.90243902 Rev3 WINNT 5.1 mozilla-central talos tpn 1711.64634146 Rev3 WINNT 6.1 mozilla-central debug test crashtest 1308.125 Rev3 WINNT 6.1 mozilla-central debug test jetpack 1633.7875 Rev3 WINNT 6.1 mozilla-central debug test jsreftest 1833.8625 Rev3 WINNT 6.1 mozilla-central debug test mochitest-other 1391.5375 Rev3 WINNT 6.1 mozilla-central debug test mochitests-1/5 1398.2625 Rev3 WINNT 6.1 mozilla-central debug test mochitests-2/5 1269.4875 Rev3 WINNT 6.1 mozilla-central debug test mochitests-3/5 1545.325 Rev3 WINNT 6.1 mozilla-central debug test mochitests-4/5 1451.45 Rev3 WINNT 6.1 mozilla-central debug test mochitests-5/5 1265.275 Rev3 WINNT 6.1 mozilla-central debug test reftest 1292.25 Rev3 WINNT 6.1 mozilla-central debug test xpcshell 1563.7875 Rev3 WINNT 6.1 mozilla-central opt test crashtest 1645.51219512 Rev3 WINNT 6.1 mozilla-central opt test jetpack 1549.59756098 Rev3 WINNT 6.1 mozilla-central opt test jsreftest 1926.76829268 Rev3 WINNT 6.1 mozilla-central opt test mochitest-other 1845.15853659 Rev3 WINNT 6.1 mozilla-central opt test mochitests-1/5 1392.75609756 Rev3 WINNT 6.1 mozilla-central opt test mochitests-2/5 1516.2804878 Rev3 WINNT 6.1 mozilla-central opt test mochitests-3/5 1643.31707317 Rev3 WINNT 6.1 mozilla-central opt test mochitests-4/5 1658.19512195 Rev3 WINNT 6.1 mozilla-central opt test mochitests-5/5 1658.73170732 Rev3 WINNT 6.1 mozilla-central opt test peptest 1412.69512195 Rev3 WINNT 6.1 mozilla-central opt test reftest 1557.63529412 Rev3 WINNT 6.1 mozilla-central opt test reftest-no-accel 1773.42682927 Rev3 WINNT 6.1 mozilla-central opt test xpcshell 1849.8902439 Rev3 WINNT 6.1 mozilla-central talos chromer 1764.8902439 Rev3 WINNT 6.1 mozilla-central talos dirty 1440.97619048 Rev3 WINNT 6.1 mozilla-central talos dirtypaint 1697.02439024 Rev3 WINNT 6.1 mozilla-central talos dromaeo 1511.4047619 Rev3 WINNT 6.1 mozilla-central talos dromaeojs 1442.2 Rev3 WINNT 6.1 mozilla-central talos jstests 1709.88095238 Rev3 WINNT 6.1 mozilla-central talos nochromer 1299.13414634 Rev3 WINNT 6.1 mozilla-central talos other 1595.7195122 Rev3 WINNT 6.1 mozilla-central talos svgr 1865.98780488 Rev3 WINNT 6.1 mozilla-central talos tpn 1861.63414634 Rev3 WINNT 6.1 mozilla-central talos xperf 1551.48809524 Rev3 WINNT 6.1 x64 mozilla-central opt test crashtest 1598.8375 Rev3 WINNT 6.1 x64 mozilla-central opt test jetpack 1258.225 Rev3 WINNT 6.1 x64 mozilla-central opt test jsreftest 1472.35 Rev3 WINNT 6.1 x64 mozilla-central opt test mochitest-other 1490.725 Rev3 WINNT 6.1 x64 mozilla-central opt test mochitests-1/5 1404.075 Rev3 WINNT 6.1 x64 mozilla-central opt test mochitests-2/5 1376.5875 Rev3 WINNT 6.1 x64 mozilla-central opt test mochitests-3/5 1592.975 Rev3 WINNT 6.1 x64 mozilla-central opt test mochitests-4/5 1463.95 Rev3 WINNT 6.1 x64 mozilla-central opt test mochitests-5/5 1449.9 Rev3 WINNT 6.1 x64 mozilla-central opt test peptest 1529.0 Rev3 WINNT 6.1 x64 mozilla-central opt test reftest 1672.975 Rev3 WINNT 6.1 x64 mozilla-central opt test xpcshell 1316.7625 Rev3 WINNT 6.1 x64 mozilla-central talos chromer 1588.0875 Rev3 WINNT 6.1 x64 mozilla-central talos dirty 1083.07142857 Rev3 WINNT 6.1 x64 mozilla-central talos dirtypaint 1317.36842105 Rev3 WINNT 6.1 x64 mozilla-central talos dromaeo 1551.97619048 Rev3 WINNT 6.1 x64 mozilla-central talos dromaeojs 1061.57894737 Rev3 WINNT 6.1 x64 mozilla-central talos nochromer 1117.725 Rev3 WINNT 6.1 x64 mozilla-central talos other 1464.9625 Rev3 WINNT 6.1 x64 mozilla-central talos svgr 1294.5875 Rev3 WINNT 6.1 x64 mozilla-central talos tpn 1587.9875 Rev4 MacOSX Lion 10.7 mozilla-central debug test crashtest 1413.51190476 Rev4 MacOSX Lion 10.7 mozilla-central debug test jetpack 1886.20238095 Rev4 MacOSX Lion 10.7 mozilla-central debug test jsreftest 1689.70238095 Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitest-other 1419.03529412 Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitests-1/5 1367.14285714 Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitests-2/5 1447.96428571 Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitests-3/5 1306.36904762 Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitests-4/5 1560.03571429 Rev4 MacOSX Lion 10.7 mozilla-central debug test mochitests-5/5 1346.22619048 Rev4 MacOSX Lion 10.7 mozilla-central debug test reftest 1160.25 Rev4 MacOSX Lion 10.7 mozilla-central debug test xpcshell 1641.98809524 Rev4 MacOSX Lion 10.7 mozilla-central opt test crashtest 1567.85148515 Rev4 MacOSX Lion 10.7 mozilla-central opt test jetpack 1488.48514851 Rev4 MacOSX Lion 10.7 mozilla-central opt test jsreftest 1610.36633663 Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitest-other 1559.95049505 Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitests-1/5 1551.06930693 Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitests-2/5 1439.67326733 Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitests-3/5 1611.44554455 Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitests-4/5 1471.16831683 Rev4 MacOSX Lion 10.7 mozilla-central opt test mochitests-5/5 1648.7029703 Rev4 MacOSX Lion 10.7 mozilla-central opt test peptest 1444.3960396 Rev4 MacOSX Lion 10.7 mozilla-central opt test reftest 1564.81553398 Rev4 MacOSX Lion 10.7 mozilla-central opt test xpcshell 1543.82178218 Rev4 MacOSX Lion 10.7 mozilla-central talos chromer 1400.03960396 Rev4 MacOSX Lion 10.7 mozilla-central talos dirty 1479.14814815 Rev4 MacOSX Lion 10.7 mozilla-central talos dirtypaint 1492.04950495 Rev4 MacOSX Lion 10.7 mozilla-central talos dromaeo 1390.87037037 Rev4 MacOSX Lion 10.7 mozilla-central talos dromaeojs 1378.08510638 Rev4 MacOSX Lion 10.7 mozilla-central talos jstests 1653.78181818 Rev4 MacOSX Lion 10.7 mozilla-central talos nochromer 1320.79207921 Rev4 MacOSX Lion 10.7 mozilla-central talos other 1492.05940594 Rev4 MacOSX Lion 10.7 mozilla-central talos svgr 1500.45544554 Rev4 MacOSX Lion 10.7 mozilla-central talos tpn 1516.94059406 Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test crashtest 1336.52380952 Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test jetpack 1568.20238095 Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test jsreftest 1532.78823529 Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitest-other 1355.49425287 Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitests-1/5 1452.45238095 Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitests-2/5 1497.10714286 Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitests-3/5 1482.0952381 Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitests-4/5 1443.01190476 Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test mochitests-5/5 1237.79761905 Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test reftest 1343.89411765 Rev4 MacOSX Snow Leopard 10.6 mozilla-central debug test xpcshell 1545.88372093 Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test crashtest 1669.87128713 Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test jetpack 1679.46534653 Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test jsreftest 1621.41584158 Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitest-other 1568.45098039 Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitests-1/5 1644.94059406 Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitests-2/5 1851.26732673 Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitests-3/5 1618.50485437 Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitests-4/5 1634.09803922 Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test mochitests-5/5 1557.76153846 Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test peptest 1636.86138614 Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test reftest 1602.41176471 Rev4 MacOSX Snow Leopard 10.6 mozilla-central opt test xpcshell 1590.94059406 Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos chromer 1665.55882353 Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos dirty 1248.79245283 Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos dirtypaint 1670.63366337 Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos dromaeo 1626.81481481 Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos dromaeojs 1625.02083333 Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos jstests 1739.57407407 Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos nochromer 1436.52941176 Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos other 1602.79207921 Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos svgr 1676.52475248 Rev4 MacOSX Snow Leopard 10.6 mozilla-central talos tpn 1481.43564356 WINNT 5.2 mozilla-central build 1420.93258427 WINNT 5.2 mozilla-central leak test build 1364.53932584 WINNT 6.1 x86-64 mozilla-central build 1437.09090909 jetpack-mozilla-central-fedora-debug 2145.93333333 jetpack-mozilla-central-fedora-opt 1462.06896552 jetpack-mozilla-central-fedora64-debug 1806.75862069 jetpack-mozilla-central-fedora64-opt 1603.75862069 jetpack-mozilla-central-leopard-debug 1972.31034483 jetpack-mozilla-central-leopard-opt 2482.82758621 jetpack-mozilla-central-lion-debug 1446.65517241 jetpack-mozilla-central-lion-opt 1509.27586207 jetpack-mozilla-central-snowleopard-debug 1982.5862069 jetpack-mozilla-central-snowleopard-opt 1654.34482759 jetpack-mozilla-central-w764-debug 2471.62068966 jetpack-mozilla-central-w764-opt 1800.65517241 jetpack-mozilla-central-win7-debug 1794.37931034 jetpack-mozilla-central-win7-opt 1567.93103448 jetpack-mozilla-central-xp-debug 2147.37931034 jetpack-mozilla-central-xp-opt 1786.24137931 ============================================ Author JohnPosted on 21 Aug 2012 21 Aug 2012 Categories Mozilla 10 thoughts on “137 hours compute hours every ~6 minutes” Ben Hearsum says: 21 Aug 2012 at 18:46 Johnath calculated this to be a millenia per year: https://twitter.com/johnath/status/237980920102137856 Reply Havvy says: 21 Aug 2012 at 19:28 1370 years per year to be more accurate. But that assumes a constant checkin rate. Maybe some calculus could be involved to find the total for this year + estimate? Reply Benoit Jacob says: 21 Aug 2012 at 20:34 Have you considered other ways of reducing test load, such as: – running certain tests only once a day, instead on on every checkin? – optionally triggering per-checking runs of such tests when a certain directory is touched? For example, in mochitest-1, probably the most expensive mochitest is the WebGL one. But it’s useless to run it on every checkin. It’s useful to run it on every checkin that touches either content/canvas, gfx/gl, or gfx/layers; and it’s useful to run it once a day for integration testing, but it’s not useful to run it on every checkin. In this way, the load incurred by this test could be reduced by over 90% without losing coverage and with only very little loss of immediacy of detecting regressions. Reply Justin Lebar says: 21 Aug 2012 at 20:35 Like I said in the newsgroup, I think we need clear reporting of wait times (similar to the monthly infra load reports). Then we can evaluate whether we’re turning tests around within an acceptable amount of time. At the moment, I don’t even know whether tests have actually gotten slower (maybe it’s just my imagination), and if they have gotten slower, how much faster they’d need to get to match our speed from a few months ago. Reply John says: 27 Aug 2012 at 19:25 hi jlebar: We already send daily “wait times” posts to mozilla.dev.tree-mangement. There’s 3 different pools where we need to measure “waittimes”, so there’s 3 posts per day. We do need a way to get that data into something that makes trend-graph-over-time easier to see. However, meanwhile, if you just look at the subject field of the posts over time, you can see the number of jobs per day increasing, and the wait times for *some* specific OS getting worse. tc John. Reply Phil Ringnalda says: 22 Aug 2012 at 00:09 It doesn’t make a significant difference to the total, but those jetpack-mozilla-central-* jobs at the end of the list are not m-c on-push builds (the ones ending in Jetpack above are) – jetpack-* are the jetpack repo on-push builds on the separate Addon-SDK tree. Some of those number seem wildly unlikely, too: “WINNT 5.2 mozilla-central build 1420.93258427” would be 23 minutes for a Windows build, when every Windows build for the last two days took between 115 and 120 minutes. 10.6 opt mochitest-5 I noticed the other day takes between 6 and 9 minutes, but you’ve got it at nearly 26 minutes. Reply @clochix says: 22 Aug 2012 at 00:26 Chaque commit (±240 par jour) dans le dépôt de Mozilla déclenche 137h machine de compil et tests http://t.co/EvLlT3lm Reply @nyconyco says: 22 Aug 2012 at 01:55 @clochix Chaque commit (±240 par jour) dans le dépôt de Mozilla déclenche 137h machine de compil et tests http://t.co/mOmOz26K Reply And says: 22 Aug 2012 at 02:02 Couldn’t both builds and tests be run in a batch or bisect mode, so a new build would only be kicked of every hour (or such) and if a test changed status the check-in range would be bisected until the offending check-in was localized (with the triggering test run first, so the bisect can continue, while the other tests are run for completeness). Probably wouldn’t be applicable for try, where build-requests are explicit. For inbound a script could try to land the rest of the queue without the offending check-in. Reply And says: 25 Aug 2012 at 02:31 I think there is a problem with the times you have listed. They seem way to similar and it would be odd if every test suite and every build took the same amount of time. If I look in the build logs (and am interpreting them correctly) the job “WINNT 5.2 mozilla-central build” (that you list as 1420 sec) took about 3872 sec (with make build:47%, make check:20%, hg-mirror:11%, make buildsymbols:6%, make update-packaging: 4%, …, for those curious), whereas the job “jetpack-mozilla-central-w764-debug” (you list as 2471) seems to have taken just about 70sec. The same job “jetpack-mozilla-central-w764-debug” seems a little odd since it purports to run a test on a debug build for win64. But tinderbox doesn’t seem to create debug builds for win64. And if you look in the build log this job seems to fail with a file-not-found error. It also looks odd that the jetpack tests seem to be run twice (both as “jetpack-mozilla-central-…” and “Rev3 … mozilla-central … test jetpack”). Both seem to run the “run_jetpack.py” script, although with different options. I don’t think windows have the best performance when it comes to many small files. Have you tried enabling folder compression on the \talos-slave\ folder. It have helped read performance for me (on something unrelated). I assume that at least mochitests doesn’t alter the files in the test folder, so perhaps it would be an idea to not unzip the test files, but map the zip as drive Presumably “Pismo Mount File” can do this, or you could distribute the tests to the slaves as a zipped iso-file, which should be fast to unzip and multiple programs can mount. Reply Leave a Reply Cancel reply Your email address will not be published. Required fields are marked * Comment * Name * Email * Website Δ This site uses Akismet to reduce spam. Learn how your comment data is processed. Post navigation Previous Previous post: HOWTO: Use crutches Next Next post: Infrastructure load for August 2012 Search for: Search Recent Posts California’s telework dashboard wins innovation award 01 Nov 2021 California’s new Telework policy: A big step for the Future of Work 13 Oct 2021 Environmental value of scaling telework in government 30 Mar 2021 Distributed Teams (2nd edition)​ is now available! 01 Feb 2021 “A civic technologist’s practice guide” by Cyd Harrell 04 Jan 2021 Creative Commons License This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. Meta Log in Entries feed Comments feed WordPress.org Using All in one Favicon My “Distributed Teams” Book Jobs for Remoties Tools for Remoties Locations for Remoties Playbooks for Distributed Teams About me… Quotes John O'Duinn's blog Proudly powered by WordPress