In preparation for a new release of software our company maintains, I've been working on what I believe to be a really good approach to solving our scalability issues.  I have every intention of putting together a proof of concept to validate the design on paper will actually do what I want.  When I briefed it to the team, the boss had a counter proposal, inspired in part by the way I described the problem areas.  The boss also accepted my proposal to do two proofs of concept to evaluate the alternatives.So, what's the best way to work through the proof of concept shoot out?  We have both objective and subjective criteria that we are using to evaluate the solutions.  I'd like to make sure we are comparing apples to apples with these fairly different approaches.We have requirements for throughput and size.  In short, we know we need to process a certain number of objects per second and maintain that rate for an hour.We need to evaluate scalability (both by adding more cores and by increasing the number of objects)We need to evaluate ease of development (subjective)We need to evaluate how easy it is to understand the algorithm (subjective)I have my theory about which way things will lean, but I don't want that to influence my results.  Any input on how to maintain objectivity in this process, and things I might need to consider would be greatly appreciated.
What's the best way to do a Proof of Concept shootout?
design;architecture;algorithms;evaluation
Generally what we do is...set a deadline like 1 monthuse your requirements that you listed and code it up. have some moderator who isn't writing either program control the requirements and doesn't see them think of a hidden requirement that should be doable in a short amount of time. When the rest of the team looks over the programs to evaluate them THEY have to add in the requirement to each system so they get a feel for both of them. After the team dig into the code and add in the missing requirement they should have a reasonable feel for the codebases and pick which one they would rather develop off of. then take the best remaining peices of the 'loser' and incorperate them into the winning architecture. 