onnel.
“In the late 1990s, NERSC hosted most open code development for the Office of Science,” explains Miller. At the time, NERSC’s barrier to entry was lower than LLNL’s—contributors simply needed to fax an account request form—and the institution was better equipped for web service setup and open collaboration.
Harrison states, “Being open to other communities, such as users simulating tokamak fusion, has hardened our tool and enabled us to build features we wouldn’t have otherwise developed.” Miller adds, “In addition to developer contributions, VisIt also benefits from users who report issues or new use cases.” Over the following years, all VisIt development became licensed and freely available.
Eventually, the technologies supporting various development processes ran into challenges. The source code outgrew its home in ClearQuest, so the team migrated it to Subversion (SVN), to which NERSC hosted anonymous https access. However, Harrison says, “Branch development and merging became suboptimal with SVN. It could take an hour to create a branch. VisIt’s large code base was taxing on NERSC’s servers, and we saw performance issues.” As SVN started to show its age, so too did the Redmine issue tracker hosted at ORNL and the OpenOffice software used at LLNL for VisIt’s documentation. Cumulatively, these concerns provided enough motivation to migrate several of the tool’s services to the more advanced, well-established GitHub platform.
During this time, the Lab’s policies and capabilities for managing open-source software changed, too. A culture of collaboration inspires many project teams to seek feedback and ideas from the open-source community, and GitHub provides projects with greater visibility and more integrated development functionality. Currently, over 450 LLNL software repositories are available to the public. VisIt is one of the Lab’s longer-lived open-source tools, though its history with NERSC and ORNL make it a relative newcomer to GitHub.
Maintaining VisIt’s baseline health is crucial for users who dissect data at extreme scale, and throughout the years, the team’s decisions have always been driven by ease of contribution. “VisIt has an active community of contributors, and some users have relied on it for years,” says Brugger.
Consolidating the Pieces
The road to GitHub was neither straight nor narrow. VisIt consists of multiple technologies serving different purposes, collectively leaving no stone unturned in ensuring the tool’s reliability and accuracy. Each service had to be evaluated for migration potential and necessity—for example, ORNL’s Mailman email service, which is archived and searchable, is sufficient as is. According to Miller, this piecemeal situation ultimately evolved into an opportunity to consolidate. “We saw various ways to migrate, and we wanted to maintain as much history as possible,” he says.
Although it is the world’s most popular source code hosting service, GitHub is not a one-size-fits-all solution. VisIt contains 2 million lines of code and has a nearly 20-year history of versions, bugs identified and fixed, code contributions, and documentation. Harrison states, “We knew we’d have to make strategic decisions and major changes to accomplish the transition.”
Migration planning began two years ago and culminated when Harrison and Maguire wrote scripts to automatically transfer over 15,000 historical code contributions and 3,000 issue submissions. “Not everything migrated successfully the first time,” Harrison notes, referring to the ClearQuest-to-SVN migration of the mid-2000s. “The history was intact, but the details needed work. So we expected that initial migration attempts would fail at first.”
In this effort, the team’s initial SVN-to-GitHub scripting attempts resulted in all historical work being time-stamped with the same day in 2007. They kept at it, improving the import process with each iteration. “When replaying all the code changes from the past 15-plus years, we wanted to ensure the work was attributed to the correct people,” says Harrison.
Another migration challeng