You have shortlisted three agencies. Each sent a beautiful deck, a friendly account executive, and a rough estimate that somehow lands inside your budget. Then you sign, and four months later you are debugging a codebase nobody on the original sales call ever touched.
That gap between the pitch and the delivery is the single most expensive risk in a custom software vendor selection. The fix is not a longer questionnaire. It is watching the team build something real, under your conditions, before the contract is signed.
This guide gives you a structured demo day and a paid proof of concept you can run in one to three weeks, plus the scoring rubric and the questions that separate real engineering teams from good presenters.
Key takeaways
- A demo day is not a sales call. You set the problem, the timebox, and the format so you see how the vendor thinks when the answer is not pre-recorded.
- Ask for a live walkthrough of a past project's repository, tests, and deployment pipeline, not a marketing video.
- Run a small paid proof of concept with a written exit criterion. Paying for it is a feature, not a cost: it reveals how the vendor behaves when money and deadlines are real.
- Score vendors on a fixed rubric across architecture, communication, testing, and handover. Gut feel alone will not survive a procurement discussion.
- Treat the demo as a two-way evaluation. The best teams will push back on your requirements, and that is a good sign.
Why a pitch deck tells you almost nothing
Sales material is optimized for one outcome: getting you to sign. It is produced by people who may never write a line of the code you are buying. That is not dishonesty, it is just a different job function with different incentives.
What you actually need to assess is engineering behavior: how the team decomposes an ambiguous problem, how they handle a requirement they disagree with, how they test, and how they document decisions so the next engineer can pick it up. None of that fits on a slide.
In our experience, the vendors who resist a structured technical assessment are the ones with the most to hide. A confident team will welcome it, because it lets them show work the deck cannot.
Set the rules before the demo day
Send the same brief to every shortlisted vendor at least a week in advance. Consistency is what makes the comparison fair and the scores meaningful.
What to include in the brief
- A one-page problem statement drawn from your real roadmap, not a toy example.
- The format: a short live architecture discussion followed by a hands-on session.
- The timebox: typically 90 to 120 minutes total, with a hard stop.
- The people you expect to attend on their side: at least one engineer who would actually work on your project.
- The rubric you will score against, shared openly. Transparency raises the quality of the conversation.
Who should be in the room
Include your own technical decision maker, the person who owns the budget, and one engineer who will maintain the system later. The maintainer's reaction is often the most predictive signal you will get all day.
Run the custom software vendor demo in three acts
Act one: the live architecture session
Give the vendor 20 minutes to sketch an approach to your problem on a whiteboard or shared canvas. Watch how they handle ambiguity. Do they ask clarifying questions about scale, data volume, compliance, or existing systems, or do they jump straight to a stack?
Strong teams will name at least two viable approaches and explain the tradeoffs. Weak teams will present one solution with no alternatives and no stated assumptions.
Act two: the code walkthrough
Ask them to open a real project, ideally similar in domain or complexity, and walk you through it live. Request these specific things:
- The repository structure and why it is organized that way.
- One or two pull requests, including review comments and how disagreements were resolved.
- The test suite: what is covered, what is deliberately not, and how they run it.
- The deployment pipeline, from commit to production.
- How they handle secrets, environment configuration, and rollbacks.
You are not auditing every line. You are checking whether the team can explain their own decisions fluently. If they cannot navigate their own codebase without hunting, that is a signal.
Act three: the failure story
Ask each vendor to describe a project that went wrong. What broke, how did they find out, and what changed afterward? Teams that cannot produce a genuine failure story are either very new or not being candid. The quality of the answer matters more than the failure itself.
The software vendor proof of concept checklist
A demo day shows you how a team thinks. A paid proof of concept shows you how they deliver. Keep it small, timeboxed, and tied to a decision you actually need to make.
- Define one narrow slice. Pick a real integration, a data pipeline, or a single user flow. Two to four weeks of work, no more.
- Write the exit criteria in advance. For example: the service ingests a sample file, validates it, exposes one endpoint, and ships with tests and a README another engineer can follow.
- Agree on the commercial terms. A fixed fee, a fixed window, and a clear statement that the code is yours regardless of whether you continue.
- Set a working rhythm. Daily or every-other-day updates, a shared board, and one scheduled review midway so you see progress while it is still fixable.
- Evaluate the handover, not just the output. Can your engineer run it locally using only the documentation provided?
That last point is where most proofs of concept quietly fail. Working code with no handover is a demo, not a deliverable.
Score vendors on a fixed rubric
Score each vendor from one to five on the following dimensions immediately after the session, while the details are fresh. Compare totals only after everyone has been scored independently.
- Problem framing: Did they ask the right questions before proposing a solution?
- Technical depth: Could they defend architecture choices and name tradeoffs?
- Testing and quality: Is testing part of the workflow or an afterthought?
- Communication: Did they explain complex ideas without condescension or jargon padding?
- Handover readiness: Would you be comfortable taking over the code tomorrow?
- Honesty: Did they push back, flag risks, and decline to promise the impossible?
Weight the dimensions to match your situation. A regulated product should weight testing and handover heavily. A fast-moving internal tool might weight problem framing and communication higher.
Questions to ask a custom software vendor
Keep these for the end of the session, when the presentation is over and the team is tired. Tired answers are usually honest answers.
- Who exactly will work on our project, and what percentage of their time will it get?
- What happens if that person leaves mid-project?
- How do you handle a requirement we insist on that you believe is wrong?
- What does your definition of done include, and what does it exclude?
- How do you estimate, and how do you communicate when an estimate slips?
- Who owns the intellectual property, and what do we receive at handover?
- What would make you recommend we not build this at all?
That final question is the most revealing one on the list. A vendor that will tell you when a project is not worth doing is a vendor you can trust with the projects that are.
Common traps that waste the demo day
- Letting the vendor set the agenda. You end up watching a rehearsed product tour instead of a technical assessment.
- Skipping the paid proof of concept to save money. The proof of concept is the cheapest insurance you will buy in the entire project.
- Evaluating only the account team. The people in the room should be the people who will build the thing.
- Ignoring your own engineer's reaction. If your maintainer is uneasy and cannot articulate why, dig until they can.
- Using a different rubric per vendor. It feels fairer in the moment and produces an unusable comparison.
When you are ready to move from evaluation to build, it helps to work with a team that expects this level of scrutiny. Avaton builds custom software, AI systems, and blockchain products, and we are happy to be put through exactly this process. You can see how we approach delivery on our software development services page, review past engagements in our project work, or talk to our team about running a proof of concept against your own roadmap. If you want more evaluation frameworks like this one, our blog covers vendor selection, architecture, and delivery practices in depth.
Frequently Asked Questions
How long should a custom software vendor demo day take?
Plan for 90 to 120 minutes per vendor: roughly 20 minutes for a live architecture discussion, 45 minutes for a code walkthrough, and the remainder for questions and the failure story. Keep the timebox fixed across vendors so the comparison stays fair.
Should I pay for a proof of concept with a software vendor?
Yes, in most cases. A paid, timeboxed proof of concept changes the incentives: the vendor commits real engineering time, you get working code and documentation, and both sides learn how the other operates before a large contract is signed. Agree in advance that the code is yours regardless of the outcome.
What should a proof of concept deliverable include?
At minimum: working code for one narrow slice, automated tests, a README that lets your engineer run it locally, and a short written summary of decisions and known limitations. If your engineer cannot run it from the documentation alone, the handover is incomplete.
What is the most revealing question to ask a custom software vendor?
Ask what would make them recommend you not build the feature at all. Vendors who can articulate when a project is not worth doing tend to be candid about risks, estimates, and tradeoffs later, when being candid is harder.
How do I evaluate a software development agency without a technical background?
Bring your own engineer or a trusted technical advisor into the demo day and let them score the code walkthrough and handover readiness. You can still judge problem framing, communication, honesty, and how clearly the team explains its own decisions.
Cover: Photo by cottonbro studio on Pexels