Benchmarking software development partners – how to do it right

One of the most critical elements allowing to obtain all the benefits of cooperation with a software development company is choosing the right vendor to team up with. This complex process might seem time and resource-consuming, but beyond doubt it will pay off, allowing you to effectively reach your goals, especially in a highly regulated and knowledge-intensive areas. We want to show you based on our experience what should be covered to benchmark potential vendors.

Assumptions

The benchmarking process should be defined by your needs, so let’s assume that you’re considering strategic, long-term partnership rather than one-off project (thus, we ’re going to use word “partner” instead of “vendor”). We also presume that your partner-to-be would be involved in complex and demanding projects, so they must go beyond body leasing, have their own initiative, and as result provide more benefits than only team extension.

Basic partner scoring

At first, let’s start with the elements that you can verify by your own research and meetings with software development company representatives. Your potential partner should meet all the basic requirements to be involved in the following, hands-on stage. These requirements may include, but are not limited to:

Yes/No criteria:

• Geographical location – it’s worth building a list of acceptable locations i.e. taking into account time zone, data security or travel easiness
• Market knowledge and experience – you might require a number of references from their past partnerships
• Technology & competence match – if you can predict what skillsets will the cooperation require you should ensure that your potential partner would be able to provide them
• Compliance & security – if you operate in a highly regulated business area, verify if the company will be capable of meeting all the compliance and security related requirements

Numerical criteria:

• Attrition rate – <10% excellent, 10-15% very good, 15-20% good (market standard), >20% bad
• Ability to ramp up (based on organic growth capability rather than declarations) – while we can’t predict a potential number of new hires that might be required, you should estimate how the cooperation might expand and verify if the vendor will be able to recruit enough new talents. To compare this easily you can use the number of past hires to predict future capability
• The longevity of their past cooperation – ask about the average longevity of their past partnerships to rest assured they proven to be considered as long-term strategic partner rather than short term tactical support, you might even require meeting a certain benchmark such i.e. average cooperation lasting for more than 4 years

Subjective criteria

There are some criteria you can use for benchmarking, but you can’t measure them other than by subjective people’s opinion. You can still use some scale to quantify (e.g. 1-5 or strong no/no/not sure/yes/strong yes).

• Company Culture – companies with similar values and internal culture can get on the same page more easily, therefore remember to talk this matter through during the meetings
• Knowledge management and onboarding process – Experienced software development companies usually have their battle-tested methods allowing to firstly onboard and then share the knowledge fast and effectively, to start contributing as fast as possible
• Approach to people (e.g. trainings, well-being) – check if they provide employees with a chance to grow through i.e. promoting internal self-development culture and events, attending external conferences or simply securing some of their time for personal development
• Approach to clients’ (care factor) – check if you as a client will get enough “care”. We know a lot of cooperations that ended because management neglected this factor. Quality of engineers is important and usually verified while quality of management is often forgotten while it defines the care factor. For example, most of the challenges along the road require partner’s management engagement. If you’d like to have smooth management-to-management cooperation you need to take it into account

Once you have your basic partner scoring you can move forward to more hands-on benchmarking.

Proof of Concept – best way of benchmarking vendors

There’s no better way to benchmark potential partners than to see how they operate in a real-life scenario. Thus, we believe that it’s best to let them all prove their value through proof of concept – a really short project that would either confirm what they stated or show that their statements were nothing more than bragging.

How should that look like? From our experience, most such projects are small, few weeks long commitment, yet what’s important is to plug the external developers into your existing team. This way you would be able not only to verify the end results, but also your specialists would have a chance to see how the partner teams approach day-to-day work and if they are able to effectively cooperate. Once the PoC is completed, you can also finish benchmarking potential partners – for ease of reference, we will divide this process into 2 sections.

1/ Perform an anonymous survey among you developers and managers involved in the PoC

The first thing to do is to ask your employees about their opinion – after all, they are the ones who worked hand in hand with partners’ specialists during the PoC, and if you decide to kickstart the cooperation on a larger scale, they are the ones who will work alongside them. What’s important is that you should gather feedback from developers since they might notice things invisible for people who are higher in the chain of command. As for the survey, it should be built with rating scale questions to let you compare the results among potential partners. Below you can find questions that were used to benchmark Software Mind over last years:

• How rapidly and effectively they managed to onboard
• How well they understood the project and your environment
• How involved they were in project discussions and did they bring any helpful ideas
• Their level of ownership of assigned tasks
• How well they adopted your existing processes, if they were able to identify any bottlenecks and if so propose improvements
• How efficient was the communication
• The quality of the final effects (see below)
• Their ability to add any extra value you haven’t thought about (I.e. improve overall Time-To-Market or overall Cycle Time)

Finally, it’s good to include typical NPS as a part of the survey to get the overall feel of the cooperation from your employees.

2/ Test the code quality thoroughly

Very first thing to check is obvious: is the code working as expected? Best approach is to do it by automated testing scripts. Secondly, it’s worth performing code review by your experts to check code quality: is it well-designed, maintainable, follows clean code principles and so on. Finally, you can use tools for automatic code quality verification like i.e. SonarCube.

The easiest PoC alternative

If running a few week-long PoC seems like too much of an investment, you can choose an easier alternative such as a 2-day long hackathon. This wouldn’t be as comprehensive as PoC but it may be just enough to see how they approach challenges, how the skills they promised to deliver transfer to real-life action, and if their specialist can get along with your internal team.

Don’ts

Finally let’s briefly go over things you shouldn’t base you opinion on, and when a red light should start blinking.

1/ Never trust a company that promises to deliver an infinite number of specialists in no time

Hiring experienced developers keeps business owners up at night all over the world. While in countries with a wide talent pool and not so wide competition acquiring talent is more manageable, it’s still a challenging process to find and attract experienced and reliable people. Therefore, if your partner promises to get a huge team in a matter of days, that’s nothing more than a pie in the sky. This is either impossible, or these people would not be as good as they are being advertised.

2/ Don’t take the bait of “bench”

Best companies do not keep experienced developers on the so-called “bench”, waiting for an assignment. That’s not only generating extreme costs, but also best developers want to constantly move forward and face more and more challenges so in such a situation they quickly became bored and start to look for something new. Thus, while bench may look like attractive solution for short term capacity challenges, you should always consider long term effects, and be aware that if the company doesn’t have enough work for all their people, they might not only not be as good as they say, but also that if the bench would be empty and a need to ramp up occurs, they might have a problem to attract talent.

3/ Don’t put cost factors first

While beyond doubt cooperation with software development partner is a proven way to boost the cost-effectiveness of your operations, the price itself shouldn’t be the most important factor in your decision-making process. Long story short – choosing the cheapest option is likely to backfire. If you want to reap all the benefits of such cooperation, team up with a reliable partner who would be able to provide more than a number of lines of code. You would have to pick an experienced top-notch partner – and that would cost more than simple body leasing service. Thus, utterly consider what your partner-to-be might bring to the table as that’s precisely what you’re paying for – even if a day rate might fit in the upper part of industry-standard, such an investment may easily pay off in the long run.

Do it right, or you’ll do it twice

The mentioned elements are examples of what may be covered to ensure that the cooperation won’t end up losing time, money and market opportunities. Consider them as a list you may choose from or expand to fit best your particular situation. We are open to sharing our experiences so if you want to discuss your situation, fill the contact form below and we’ll do our best to help.
Or maybe you want to benchmark us? Well, that’s a challenge we will gladly accept.