I hope this isn't too general of a question; I could really use some seasoned advice.I am newly employed as the sole SW Engineer in a fairly small shop of scientists who have spent the last 10-20 years cobbling together a vast code base. (It was written in a virtually obsolete language: G2 -- think Pascal with graphics). The program itself is a physical model of a complex chemical processing plant; the team that wrote it have incredibly deep domain knowledge but little or no formal training in programming fundamentals. They've recently learned some hard lessons about the consequences of non-existant configuration management. Their maintenance efforts are also greatly hampered by the vast accumulation of undocumented sludge in the code itself. I will spare you the politics of the situation (there's always politics!), but suffice to say, there is not a consensus of opinion about what is needed for the path ahead.They have asked me to begin presenting to the team some of the principles of modern software development. They want me to introduce some of the industry-standard practices and strategies regarding coding conventions, lifecycle management, high-level design patterns, and source control. Frankly, it's a fairly daunting task and I'm not sure where to begin. Initially, I'm inclined to tutor them in some of the central concepts of The Pragmatic Programmer, or Fowler's Refactoring (Code Smells, etc). I also hope to introduce a number of Agile methodologies. But ultimately, to be effective, I think I'm going to need to hone in on 5-7 core fundamentals; in other words, what are the most important principles or practices that they can realistically start implementing that will give them the most bang for the buck. So that's my question: What would you include in your list of the most effective strategies to help straighten out the spaghetti (and prevent it in the future)?
I've inherited 200K lines of spaghetti code -- what now?
project management;agile;development process;maintenance;sdlc
ForewordThis is a daunting task indeed, and there's a lot of ground tocover. So I'm humbly suggesting this as somewhat comprehensive guidefor your team, with pointers to appropriate tools and educationalmaterial.Remember: These are guidelines, and that as such are meant toadopted, adapted, or dropped based on circumstances.Beware: Dumping all this on a team at once would most likelyfail. You should try to cherry-pick elements that would give you thebest bang-for-sweat, and introduce them slowly, one at a time.Note: not all of this applies directly to Visual ProgrammingSystems like G2. For more specific details on how to deal with these,see the Addendum section at the end.Executive Summary for the ImpatientDefine a rigid project structure, with:project templates,coding conventions,familiar build systems,and sets of usage guidelines for your infrastructure and tools.Install a good SCM and make sure they know how to use it.Point them to good IDEs for their technology, and make sure they know how to use them.Implement code quality checkers and automatic reporting in the build system.Couple the build system to continuous integration and continuous inspection systems.With the help of the above, identify code quality hotspots and refactor.Now for the long version... Caution, brace yourselves!Rigidity is (Often) GoodThis is a controversial opinion, as rigidity is often seen as a forceworking against you. It's true for some phases of some projects. Butonce you see it as a structural support, a framework that takes awaythe guesswork, it greatly reduces the amount of wasted time andeffort. Make it work for you, not against you.Rigidity = Process / Procedure.Software development needs good process and procedures for exactly thesame reasons that chemical plants or factories have manuals,procedures, drills and emergency guidelines: preventing bad outcomes,increasing predictability, maximizing productivity...Rigidity comes in moderation, though!Rigidity of the Project StructureIf each project comes with its own structure, you (and newcomers) arelost and need to pick up from scratch every time you open them. Youdon't want this in a professional software shop, and you don't wantthis in a lab either.Rigidity of the Build SystemsIf each project looks different, there's a good chance they alsobuild differently. A build shouldn't require too much research ortoo much guesswork. You want to be able to do the canonical thing andnot need to worry about specifics: configure; make install, ant,mvn install, etc...Re-using the same build system and making it evolve over the time alsoensures a consistent level of quality.You do need a quick README to point the project's specifics, andgracefully guide the user/developer/researcher, if any.This also greatly facilitates other parts of your buildinfrastructure, namely:continuous integration,continuous inspection.So keep your build (like your projects) up to date, but make itstricter over time, and more efficient at reporting violations and badpractices.Do not reinvent the wheel, and reuse what you have already done.Recommended Reading:Continuous Integration: Improving Software Quality and Reducing Risk (Duval, Matyas, Glover, 2007)Continuous Delivery: Release Sotware Releases through Build, Test and Deployment Automation (Humble, Farley, 2010)Rigidity in the Choice of Programming LanguagesYou can't expect, especially in a research environment, to have allteams (and even less all developers) use the same language andtechnology stack. However, you can identify a set of officiallysupported tools, and encourage their use. The rest, without a goodrationale, shouldn't be permitted (beyond prototyping).Keep your tech stack simple, and the maintenance and breadth ofrequired skills to a bare minimum: a strong core.Rigidity of the Coding Conventions and GuidelinesCoding conventions and guidelines are what allow you to develop bothan identity as a team, and a shared lingo. You don't want to errinto terra incognita every time you open a source file.Nonsensical rules that make life harder or forbid actions explicityto the extent that commits are refused based on single simpleviolations are a burden. However:a well thought-out ground ruleset takes away a lot of the whiningand thinking: nobody should break under no circumstances;and a set of recommended rules provide additional guidance.Personal Approach: I am aggressive when it comes to coding  conventions, some even say nazi, because I do believe in having a  lingua franca, a recognizable style for my team. When crap code  gets checked-in, it stands out like a cold sore on the face of an  Hollywood star: it triggers a review and an action automatically.  In fact, I've sometimes gone as far as to advocate the use of  pre-commit hooks to reject non-conforming commits. As mentioned, it  shouldn't be overly crazy and get in the way of productivity: it  should drive it. Introduce these slowly, especially at the  beginning. But it's way preferable over spending so much time fixing  faulty code that you can't work on real issues.Some languages even enforce this by design:Java was meant to reduce the amount of dull crap you can write withit (though no doubt many manage to do it).Python's block structure by indentation is another idea in thissense.Go, with its gofmt tool, which completely takes away any debateand effort (and ego!!) inherent to style: run gofmt beforeyou commit.Make sure that code rot cannot slip through. Codeconventions, continuous integration and continuousinspection, pair programming and code reviews are yourarsenal against this demon.Plus, as you'll see below, code is documentation, and that'sanother area where conventions encourage readability and clarity.Rigidity of the DocumentationDocumentation goes hand in hand with code. Code itself isdocumentation. But there must be clear-cut instructions on how tobuild, use, and maintain things.Using a single point of control for documentation (like a WikiWiki orDMS) is a good thing. Create spaces for projects, spaces for morerandom banter and experimentation. Have all spaces reuse common rulesand conventions. Try to make it part of the team spirit.Most of the advice applying to code and tooling also applies todocumentation.Rigidity in Code CommentsCode comments, as mentioned above, are also documentation. Developerslike to express their feelings about their code (mostly pride andfrustration, if you ask me). So it's not unusual for them to expressthese in no uncertain terms in comments (or even code), when a moreformal piece of text could have conveyed the same meaning with lessexpletives or drama. It's OK to let a few slip through for fun andhistorical reasons: it's also part of developing a teamculture. But it's very important that everybody knows what isacceptable and what isn't, and that comment noise is just that:noise.Rigidity in Commit LogsCommit logs are not an annoying and useless step of your SCM'slifecycle: you DON'T skip it to get home on time or get on with thenext task, or to catch up with the buddies who left for lunch.  Theymatter, and, like (most) good wine, the more time passes the more valuablethey become. So DO them right. I'm flabbergasted when I see co-workerswriting one-liners for giant commits, or for non-obvious hacks.Commits are done for a reason, and that reason ISN'T always clearlyexpressed by your code and the one line of commit log youentered. There's more to it than that.Each line of code has a story, and a history. The diffs can tellits history, but you have to write its story.Why did I update this line? -> Because the interface changed.Why did the interface changed? -> Because the library L1 defining it  was updated.Why was the library updated? -> Because library L2, that we need for  feature F, depended on library L1.And what's feature X? -> See task 3456 in issue tracker.It's not my SCM choice, and may not be the best one for your labeither; but Git gets this right, and tries to force you to writegood logs more than most other SCMs systems, by using short logs andlong logs. Link the task ID (yes, you need one) and a leave ageneric summary for the shortlog, and expand in the long log: writethe changeset's story.It is a log: It's here to keep track and record updates.Rule of Thumb: If you were searching for something about this  change later, is your log likely to answer your question?Projects, Documentation and Code Are ALIVEKeep them in sync, otherwise they do not form that symbiotic entityanymore. It works wonders when you have:clear commits logs in your SCM, w/ links to task IDs in yourissue tracker,where this tracker's tickets themselves link to the changesets inyour SCM (and possibly to the builds in your CI system),and a documentation system that links to all of these.Code and documentation need to be cohesive.Rigidity in TestingRules of Thumb:Any new code shall come with (at least) unit tests.Any refactored legacy code shall come with unit tests.Of course, these need:to actually test something valuable (or they are a waste of timeand energy),to be well written and commented (just like any other code you check in).They are documentation as well, and they help to outline the contractof your code. Especially if you use TDD. Even if you don't, youneed them for your peace of mind. They are your safety net when youincorporate new code (maintenance or feature) and your watchtowerto guard against code rot and environmental failures.Of course, you should go further and have integration tests, andregression tests for each reproducible bug you fix.Rigidity in the Use of the ToolsIt's OK for the occasional developer/scientist to want to try some newstatic checker on the source, generate a graph or model using another,or implement a new module using a DSL. But it's best if there's acanonical set of tools that all team members are expected to knowand use.Beyond that, let members use what they want, as long as they are ALL:productive,NOT regularly requiring assistanceNOT regularly adjusting to your general infrastructure,NOT disrupting your infrastructure (by modifying commonareas like code, build system, documentation...),NOT affecting others' work,ABLE to timely perform any task requested.If that's not the case, then enforce that they fallback to defaults.Rigidity vs Versatility, Adaptability, Prototyping and EmergenciesFlexibility can be good. Letting someone occasionally use a hack, aquick-n-dirty approach, or a favorite pet tool to get the job doneis fine. NEVER let it become a habit, and NEVER let this codebecome the actual codebase to support.Team Spirit MattersDevelop a Sense of Pride in Your CodebaseDevelop a sense of Pride in CodeUse wallboardsleader board for a continuous integration gamewallboards for issue management and defect countingUse an issue tracker / bug trackerAvoid Blame GamesDO use Continuous Integration / Continuous Inspection games: itfosters good-mannered and productive competition.DO keep track defects: it's just good house-keeping.DO identifying root causes: it's just future-proofing processes.BUT DO NOT assign blame: it's counter productive.It's About the Code, Not About the DevelopersMake developers conscious of the quality of their code, BUT make themsee the code as a detached entity and not an extension ofthemselves, which cannot be criticized.It's a paradox: you need to encourage ego-less programming for ahealthy workplace but to rely on ego for motivational purposes.From Scientist to ProgrammerPeople who do not value and take pride in code do not produce goodcode. For this property to emerge, they need to discover how valuableand fun it can be. Sheer professionalism and desire to do good is notenough: it needs passion. So you need to turn your scientists intoprogrammers (in the large sense).Someone argued in comments that after 10 to 20 years on a project andits code, anyone would feel attachment. Maybe I'm wrong but I assumethey're proud of the code's outcomes and of the work and its legacy,not of the code itself or of the act of writing it.From experience, most researchers regard coding as a necessity, or atbest as a fun distraction. They just want it to work. The ones who arealready pretty versed in it and who have an interest in programmingare a lot easier to persuade of adopting best practices and switchingtechnologies. You need to get them halfway there.Code Maintenance is Part of Research WorkNobody reads crappy research papers. That's why they arepeer-reviewed, proof-read, refined, rewritten, and approved time andtime again until deemed ready for publication. The same applies to athesis and a codebase!Make it clear that constant refactoring and refreshing of a codebaseprevents code rot and reduces technical debt, and facilitates futurere-use and adaptation of the work for other projects.Why All This??!Why do we bother with all of the above? For code quality. Or is itquality code...?These guidelines aim at driving your team towards this goal. Someaspects do it by simply showing them the way and letting them do it(which is much better) and others take them by the hand (but that'show you educate people and develop habits).How do you know when the goal is within reach?Quality is MeasurableNot always quantitatively, but it is measurable. As mentioned, youneed to develop a sense of pride in your team, and showing progressand good results is key. Measure code quality regularly and showprogress between intervals, and how it matters. Do retrospectives toreflect on what has been done, and how it made things better or worse.There are great tools for continuous inspection. Sonar beinga popular one in the Java world, but it can adapt to any technologies;and there are many others. Keep your code under the microscope andlook for these pesky annoying bugs and microbes.But What if My Code is Already Crap?All of the above is fun and cute like a trip to Never Land, but it'snot that easy to do when you already have (a pile of steamy andsmelly) crap code, and a team reluctant to change.Here's the secret: you need to start somewhere.Personal anecdote: In a project, we worked with a codebase  weighing originally 650,000+ Java LOC, 200,000+ lines of JSPs,  40,000+ JavaScript LOC, and 400+ MBs of binary dependencies.After about 18 months, it's 500,000 Java LOC (MOSTLY CLEAN),  150,000 lines of JSPs, and 38,000 JavaScript LOC, with dependencies  down to barely 100MBs (and these are not in our SCM anymore!).How did we do it? We just did all of the above. Or tried hard.It's a team effort, but we slowly inject in our process  regulations and tools to monitor the heart-rate of our product,  while hastily slashing away the fat: crap code, useless  dependencies... We didn't stop all development to do this: we have  occasional periods of relative peace and quiet where we are free to  go crazy on the codebase and tear it apart, but most of the time we  do it all by defaulting to a review and refactor mode every chance  we get: during builds, during lunch, during bug fixing sprints,  during Friday afternoons...There were some big works... Switching our build  system from a giant Ant build of 8500+ XML LOC to a  multi-module Maven build was one of them. We then had:clear-cut modules (or at least it was already a lot better, and we  still have big plans for the future),automatic dependency management (for easy maintenance and updates, and  to remove useless deps),faster, easier and reproduceable builds,daily reports on quality.Another was the injection of utility tool-belts, even though we  were trying to reduce dependencies: Google Guava and Apache Commons  slim down your code and and reduce surface for bugs in your code  a lot.We also persuaded our IT department that maybe using our new tools  (JIRA, Fisheye, Crucible, Confluence, Jenkins) was better than the  ones in place. We still needed to deal with some we despised (QC,  Sharepoint and SupportWorks...), but it was an overall improved  experience, with some more room left.And every day, there's now a trickle of between one to dozens of  commits that deal only with fixing and refactoring things. We  occasionally break stuff (you need unit tests, and you better write  them before you refactor stuff away), but overall the benefit  for our morale AND for the product has been enormous. We get there  one fraction of a code quality percentage at a time. And it's fun  to see it increase!!!Note: Again, rigidity needs to be shaken to make room for new andbetter things. In my anecdote, our IT department is partly right intrying to impose some things on us, and wrong for others. Or maybethey used to be right.  Things change. Prove that they are betterways to boost your productivity. Trial-runs and prototypes arehere for this.The Super-Secret Incremental Spaghetti Code Refactoring Cycle for Awesome Quality       +-----------------+      +-----------------+       |  A N A L Y Z E  +----->| I D E N T I F Y |       +-----------------+      +---------+-------+                ^                           |                |                           v       +--------+--------+      +-----------------+       |    C L E A N    +<-----|      F I X      |       +-----------------+      +-----------------+Once you have some quality tools at your toolbelt:Analyze your code with code quality checkers.Linters, static analyzers, or what have you.Identify your critical hotspots AND low hanging fruits.Violations have severity levels, and large classes with a large number of high-severity ones are a big red flag: as such, they appear as hot spots on radiator/heatmap types of views.Fix the hotspots first.It maximizes your impact in a short timeframe as they have the highest business value. Ideally, critical violations should dealt with as soon as they appear, as they are potential security vulnerabilities or crash causes, and present a high risk of inducing a liability (and in your case, bad performance for the lab).Clean the low level violations with automated codebase sweeps.It improves the signal-to-noise ratio so you are be able to see significant violations on your radar as they appear. There's often a large army of minor violations at first if they were never taken care of and your codebase was left loose in the wild. They do not present a real risk, but they impair the code's readability and maintainability. Fix them either as you meet them while working on a task, or by large cleaning quests with automated code sweeps if possible. Do be careful with large auto-sweeps if you don't have a good test suite and integration system. Make sure to agree with co-workers the right time to run them to minimize the annoyance.Repeat until you are satisfied. Which, ideally, you should never be, if this is still an active product: it will keep evolving.Quick Tips for Good House-KeepingWhen in hotfix-mode, based on a customer support request:It's usually a best practice to NOT go around fixing other issues,as you might introduce new ones unwillingly.Go at it SEAL-style: get in, kill the bug, get out, and ship yourpatch. It's a surgical and tactical strike.But for all other cases, if you open a file, make it your duty to:definitely: review it (take notes, file issue reports),maybe: clean it (style cleanups and minor violations),ideally: refactor it (reorganize large sections and their neigbors).Just don't get sidetracked into spending a week from file to file andending up with a massive changeset of thousands of fixes spanning multiplefeatures and modules - it makes future tracking difficult. One issue incode = one ticket in your tracker. Sometimes, a changeset can impact multipletickets; but if it happens too often, then you're probably doing something wrong.Addendum: Managing Visual Programming EnvironmentsThe Walled Gardens of Bespoke Programming SystemsMultiple programming systems, like the OP's G2, are different beasts...No Source CodeOften they do not give you access to a textual representation ofyour source code: it might be stored in a proprietary binaryformat, or maybe it does store things in text format but hidesthem away from you. Bespoke graphical programming systems areactually not uncommon in research labs, as they simplify theautomation of repetitive data processing workflows.No ToolingAside from their own, that is. You are often constrained by theirprogramming environment, their own debugger, their owninterpreter, their own documentation tools and formats. They arewalled gardens, except if they eventually capture the interestof someone motivated enough to reverse engineer their formats andbuilds external tools - if the license permits it.Lack of DocumentationQuite often, these are niche programming systems, which are usedin fairly closed environments. People who use them frequently sign NDAsand never speak about what they do. Programmingcommunities for them are rare. So resources are scarce. You'restuck with your official reference, and that's it.The ironic (and often frustrating) bit is that all the things thesesystems do could obviously be achieved by using mainstream and generalpurpose programming languages, and quite probably moreefficiently. But it requires a deeper knowledge of programming,whereas you can't expect your biologist, chemist or physicist (to namea few) to know enough about programming, and even less to have thetime (and desire) to implement (and maintain) complex systems, thatmay or may not be long-lived. For the same reason we use DSLs, we havethese bespoke programming systems.Personal Anecdote 2: Actually, I worked on one of these  myself. I didn't do the link with the OP's request, but my the  project was a set of inter-connected large pieces of  data-processing and data-storage software (primarily for  bio-informatics research, healthcare and cosmetics, but also for  business intelligence, or any domain implying the tracking of  large volumes of research data of any kind and the preparation of  data-processing workflows and ETLs). One of these applications was,  quite simply, a visual IDE that used the usual bells and whistles:  drag and drop interfaces, versioned project workspaces (using text  and XML files for metadata storage), lots of pluggable drivers to  heterogeneous datasources, and a visual canvas to design pipelines  to process data from N datasources and in the end generate M  transformed outputs, and possible shiny visualizations and complex  (and interactive) online reports. Your typical bespoke visual  programming system, suffering from a bit of NIH syndrome under the  pretense of designing a system adapted to the users' needs.And, as you would expect, it's a nice system, quite flexible for its  needs though sometimes a bit over-the-top so that you wonder why  not use command-line tools instead?, and unfortunately always  leading in medium-sized teams working on large projects to a lot of  different people using it with different best practices.Great, We're Doomed! - What Do We Do About It?Well, in the end, all of the above still holds. If you cannot extractmost of the programming from this system to use more mainstream toolsand languages, you just need to adapt it to the constraints of yoursystem.About Versioning and StorageIn the end, you can almost always version things, even with themost constrained and walled environment. Most often than not, thesesystems still come with their own versioning (which is unfortunatelyoften rather basic, and just offers to revert to previous versionswithout much visibility, just keeping previous snapshots). It's notexactly using differential changesets like your SCM of choice might,and it's probably not suited for multiple users submitting changessimultaneously.But still, if they do provide such a functionality, maybe yoursolution is to follow our beloved industry-standard guidelines above,and to transpose them to this programming system!!If the storage system is a database, it probably exposes exportfunctionalities, or can be backed-up at the file-system level. If it'susing a custom binary format, maybe you can simply try to version itwith a VCS that has good support for binary data. You won't havefine-grained control, but at least you'll have your back sort ofcovered against catastrophes and have a certain degree of disasterrecovery compliance.About TestingImplement your tests within the platform itself, and use externaltools and background jobs to set up regular backups. Quite probably,you fire up these tests the same that you would fire up the programsdeveloped with this programming system.Sure, it's a hack job and definitely not up to the standard of what iscommon for normal programming, but the idea is to adapt to thesystem while trying to maintain a semblance of professional softwaredevelopment process.The Road is Long and Steep...As always with niche environments and bespoke programming systems, andas we exposed above, you deal with strange formats, only a limited (ortotally inexistant) set of possibly clunky tools, and a void in placeof a community.The Recommendation: Try to implement the above guidelines outsideof your bespoke programming system, as much as possible. This ensuresthat you can rely on common tools, which have proper support andcommunity drive.The Workaround: When this is not an option, try to retrofit thisglobal framework into your box. The idea is to overlay thisblueprint of industry standard best practices on top of yourprogramming system, and make the best of it. The advice still applies:define structure and best practices, encourage conformance.Unfortunately, this implies that you may need to dive in and do atremendous amount of leg-work. So...Famous Last Words, and Humble Requests:Document everything you do.Share your experience.Open Source any tool your write.By doing all of this, you will:not only increase your chances of getting support from people insimilar situations,but also provide help to other people, and foster discussionaround your technology stack.Who knows, you could be at the very beginning of a new vibrantcommunity of Obscure Language X. If there are none, start one!Ask questions on Stack Overflow,Maybe even write a proposal for a new StackExchange Site in theArea 51.Maybe it's beautiful inside, but nobody has a clue so far, sohelp take down this ugly wall and let others have a peek!