Would you refactor your app as you go or focus on completing app first? Refactoring will mean progress of app app will slow down.Completing app will mean you get a possibly very hard to maintain app later on?The app is a personal project. I don't really know how to answer What drives the functionality and design, but I guess it's to solve inefficiencies in current software out there. I like minimal easy to use software too. So I am removing some features and add some that I feel will help.
Refactor or Concentrate on Completing App
programming practices;refactoring
Make it work.  Then make it fast. Finally, make it beautiful.If you have good separation between your code (presentation, business, and data layers) using interfaces, and it's not a monolithic design, then refactoring should not be that difficult.If you're having that much difficulty refactoring, that's probably a code-smell --  I suggest you look at Solid Principles