dissect data at extreme scale, and throughout the years, the team’s decisions have always been driven by ease of contribution. “VisIt has an active community of contributors, and some users have relied on it for years,” says Brugger.
Consolidating the Pieces
The road to GitHub was neither straight nor narrow. VisIt consists of multiple technologies serving different purposes, collectively leaving no stone unturned in ensuring the tool’s reliability and accuracy. Each service had to be evaluated for migration potential and necessity—for example, ORNL’s Mailman email service, which is archived and searchable, is sufficient as is. According to Miller, this piecemeal situation ultimately evolved into an opportunity to consolidate. “We saw various ways to migrate, and we wanted to maintain as much history as possible,” he says.
Although it is the world’s most popular source code hosting service, GitHub is not a one-size-fits-all solution. VisIt contains 2 million lines of code and has a nearly 20-year history of versions, bugs identified and fixed, code contributions, and documentation. Harrison states, “We knew we’d have to make strategic decisions and major changes to accomplish the transition.”
Migration planning began two years ago and culminated when Harrison and Maguire wrote scripts to automatically transfer over 15,000 historical code contributions and 3,000 issue submissions. “Not everything migrated successfully the first time,” Harrison notes, referring to the ClearQuest-to-SVN migration of the mid-2000s. “The history was intact, but the details needed work. So we expected that initial migration attempts would fail at first.”
In this effort, the team’s initial SVN-to-GitHub scripting attempts resulted in all historical work being time-stamped with the same day in 2007. They kept at it, improving the import process with each iteration. “When replaying all the code changes from the past 15-plus years, we wanted to ensure the work was attributed to the correct people,” says Harrison.
Another migration challenge arose when Maguire began moving VisIt’s bug and issue history to GitHub. “The Redmine repository we’d been using was very different from GitHub,” explains Maguire. “We had to figure out how to translate from one to the other.” Redmine did not support an API for this process, so Maguire learned how to write a manual web scraper. Furthermore, GitHub’s import API was not a 1:1 match with the Redmine data, and it imposed limits on the number of queries Maguire could make. “I was flagged on a spam list and had to ask for special permission to continue,” he adds.
GitHub presented other considerations for the VisIt team. “Some of our binary files were huge, which GitHub doesn’t allow without paying for a larger bandwidth quota,” Miller states. “So we split out the big files and came up with a new development process to accommodate storage constraints.” VisIt’s size also prompted customization of continuous integration (CI), a key GitHub benefit. To prevent the test suite from timing out, the team is adjusting CI timing and building containers to run tests.
Last but not least, the team had to pick a cutover date when Lab personnel were available to assist—for example, not the same week that Computation sent more than 100 employees to the SC18 conference. A dry run was scheduled for December 2018, then VisIt version 2.13.3 debuted on GitHub in January 2019.
19 Years and Counting
Now accessible via the GitHub dashboard, VisIt’s code base is already receiving contributions from the external community. In another major scripting effort, documentation has been migrated to Sphinx, which integrates seamlessly with GitHub’s Read the Docs platform. After the team confirms the repository’s stability, Maguire will complete the bug and issue tracking migration. The VisIt website will eventually migrate as well, and the team plans to dive into GitHub’s analytics capabilities.
Version 3.0 is also on the horizon. Brugger notes, “VisIt leverages many third-party tools, all of which require maintenance. Some are more straightforward th