User Acceptance Testing mistakes.
When it comes to Best Practices and Ultimate Success Factors, there is no better source than that of a seasoned Consultant with the voice of experience, well versed on the subject matter. Also a deep understanding of the typical obstacles and challenges. Most importantly, advice on how to overcome them for successful project completion.
One of the most crucial steps in the System Development Life Cycle (SDLC) is User Acceptance Testing (UAT). Arguably, it is still a mystery to some and an absolute failure for others.
Everything You Think You Know About UAT is WRONG
For the most part, and by that I mean 99% of the time, UAT is almost always confused with System Testing or Bug Hunting.
The fact is, User Acceptance Testing is a type of system and business objective validation, performed by the end-users, and business objective sign off by the project owner.
The purpose of User Acceptance Testing is to test use cases, that is, specific day-to-day operations in the life of an end-user and their required tasks to perform their job function as per Standard OperatingProcedures (SOPs).
What are Test Cases, and How to Write Them Like A Pro?
UAT Test Cases typically closely align with your Standard Operating Procedures (SOPs) and have subject matter experts (SMEs), operating procedure steps, input, and expected results.
Think of UAT Test Cases as cooking recipes. Imagine you are a cook preparing a meal, and your goal is to prepare a grilled cheese sandwich. The use case describes a series of written steps on how the cook would go about preparing a sandwich.
The cook would be the primary subject matter expert. The system components are a frying pan, a cooking device, bread, cheese, butter, and a spatula. The use cases would be the system’s overall operating procedures to make the grilled cheese sandwich, broken down into cases and steps. Bite-size chunks if you will. No pun intended. Such as i) assembling the sandwich, ii) preparing the cooking device, iii) operating the frying pan, and so on. The last step would be eating the sandwich. That’s certainly my favorite, with tomato soup.
For example, with Ivanti Service Manager, an enterprise software system used by Service Desks, the SME would be the Service Desk Analyst, the operating procedure taking service desk support calls, and use cases i) to look up & record caller information, ii) enter symptoms, iii) lookup relevant knowledge articles, iv) assign to 2nd or 3rd level support, v) resolve the call, and so on.
As you can see, UAT Test Cases should be simple, high level, and system access is not required to write use cases. In fact, a best practice is to discourage system access when writing UAT Test Cases. After all, a good cook doesn’t need a frying pan and a couple of slices of toast to write you a recipe. A good cook can write recipes (ingredients and steps) with his/her eyes closed.
You should consult Subject Matter Experts (SMEs) and the end-users, to ensure you have covered all the possible scenarios. In this example, this would be the chef, kitchen manager, kitchen staff. In the real world, for example, with the Ivanti ITSM Service Desk software, this could be the Service Desk Manager, Service Desk Analyst Lead, Storage Techs for Asset Scanning and Storage Management, Procurement Lead for Purchase Order related cases, and so on.
Last but not least, your focus should be on Test Case Scenarios, not features, functionality, or software. That’s System testing. We’re not testing the recipe or stove’s every button and dial. And we certainly aren’t trying to break the frying pan or stove. As much fun as that is, that’s stress testing. I know bug hunts are fun too, but sorry, that’s not UAT.
We are constructing high-level use cases and scenarios of day-to-day operations that can be used as Test Scripts by UAT Testers who will be grading the results as either Pass, Pass with Comment or Fail.
The Most Common User Acceptance Testing Mistakes …and How to Overcome Them
Many System Implementations fall short when it comes to proper UAT Testing, be it due to a lack of resources, time, or budget, but more often than not due to a shortcoming of proper UAT Action Plan, UAT Test Cases & Scripts, a lack of Best Practices, Strategy, Coordination, and Planning.
The most common mistakes are:
- Confusing UAT Testing with System Testing – System Testing is performed by the Developer or Consultant in a Development Environment. End Users perform user Acceptance Testing in a UAT Environment.
- Confusing UAT Testing with Bug Hunts – User Acceptance Testing is a validation of business requirements. The focus needs to be a day-to-day job function, not testing every single menu item, features, enhancements, looking for bugs. You might find some along the way, but that’s not the intention.
- Confusing UAT Testing with Enhancement Requests – UAT’s goal is to deliver business functionality based on the scope of work and business requirements that have already been signed off by the project sponsor, owner, and decision-makers. There is a time and place for enhancements, and that’s typically a follow up v1.1 phase to address must have enhancements. There is nothing like the real world to dictate what enhancements are needed and which ones are nice-to-have, or in some cases, are long forgotten after Go-Live, simply because the real world changes your perspective. I like to call it sandbox syndrome, that is, over-analyzing, over-designing, and losing track of the goal while playing in the development (sandbox) environment instead of focusing on business requirements.
- Not Having Use Cases – More often than not, you think you have use cases, when in fact, you have system test cases or training documentation. As outlined above, revisit your use cases and see if they pass my test. Ask one question, does this use case outline a business job function? Or is it useful for training or system testing? Or something else?
- Not Having a UAT Testing Tool – Sorry, but Excel won’t do. It’s not real-time, collaborative, and will double or triple your budget while your team’s morale takes a nose-dive. You need a proper software testing tool that allows you to pass and fail test results with sufficient supporting data. There are plenty of them on the market, or you can have one built in-house. I have developed my own for my Ivanti Professional Service engagements and can tell you that my clients love it, and my competition wishes they had this tool. Build your own or acquire one as soon as possible. Or contact me to help you along the process.
Best Practices
Now that we have covered the basics, there are some important best practices for UAT Success:
- Support Structure – Develop a support structure and identify roles & responsibilities—decision Makers, UAT Leads, SMEs, Trainers, Knowledge Managers, and so on.
- Daily Stand-Up Calls coordinated by the decision-maker and UAT Test Lead with the UAT Test Team, in this call cover the following:
- UAT Decision Tree – Create a UAT Decision Tree, as in what happens if, what to do next.
- Review Test Results, questions, unexpected behavior, workarounds, tester comments.
- Identify key areas for training (bullet points)
- Test Case Maintenance – identify new test cases or changes needed to existing test cases
- Scribe – Appoint a person to take meeting notes, and keep track of any “nice-to-haves,” “parking lot items,” “knowledge articles” needed. Typically this function falls on the Knowledge Manager/Trainer
. - Regular UAT Calls between the Decision Maker and Consultant/Developer to review the progress, test results, and discuss the UAT remediation scope.
In Summary
User Acceptance Testing should focus on Business Job-Functions and Standard Operating Procedures for end-users to perform daily job tasks. System Testing and bug hunts should be left to the Developer. The key to successful UAT Testing is having a good foundation, Use Cases, structure, and system for carrying out and grading UAT Tests. UAT Testing is not a mystery; it’s a system. One that you can learn.
Summary:
User Acceptance Testing (UAT)
One of the most crucial steps in the System Development Life Cycle (SDLC) is User Acceptance Testing (UAT). Arguably, it is still a mystery to some and an absolute failure for others. UAT is almost always confused with System Testing or Bug Hunting. User Acceptance Testing is a type of system and business objective validation, performed by the end-users, and business objective sign off by the project owner. The purpose of User Acceptance Testing is to test use cases, that is, specific day-to-day operations in the life of an end-user and their required tasks to perform their job function as per Standard OperatingProcedures (SOPs).
