Software Testing: 10 Things Most Teams Get Wrong

I’ve seen “finished” programs from developers that wouldn’t even run, because nobody had tested the code at all. As an IT veteran, I’m still astounded by that. Software testing is one of the hardest tasks in any IT department, and it’s where projects fail most often. More of what decades in IT taught me is in lessons from 40 years online.

Bad testing makes me angry, because the people who pay for it are never the ones who skipped it. The developer moves on to the next project. The users inherit the crashes, the help desk takes the calls, and somebody in operations gets dragged out of bed to clean up a failure that a test plan would have caught in an afternoon.

You must check programs and systems against both a specification and a set of standards. You can’t do this randomly. You need a solid grasp of what your systems are supposed to do. Your specification might say “a standard URL will be accepted in the address field.” Your standards might require validation of all buffers for overrun conditions, URLs in valid format, and so on. Standards apply to all testing. You tailor specifications to the specific program or system.

What is software testing really for?

The marketing department has no business running testing. Testing takes dedicated, trained, motivated people, and a company that won’t pay for them has decided to let its customers do the job instead.

A fatal mistake many companies make is shipping product copies to thousands of beta testers without clear goals, expert supervision, or consistent management, hoping to get useful feedback. Beta testing matters, but it can’t replace a professional testing team. Another truth that escapes many people: beta testing is for testing. Marketing is part of the product plan. It has no place in the testing plan.

I think unsupervised beta programs are mostly a way to get free labor and call it quality control. Thousands of people poke at the product with no goals, the reports come back as noise, and the company ships anyway because the beta happened.

What Are the Common Pitfalls in Software Testing?

The pitfalls in software testing are numerous, but a few themes keep showing up. Here are the 10 most common mistakes:

1. Testing to prove a program works: It’s natural to want your programs to work. But the purpose of testing is to test, not to validate the programmer’s ego. Testing should be thorough and ruthless. Find the deficiencies. Record them.
2. Attempting to prove a program doesn’t work: The purpose of testing is to test, not to prove anything. Follow a well-defined testing plan.
3. Using testing to prototype a product: Prototyping belongs in the analysis and design phases. It gives users a glimpse of the end product. Once design is complete, discard the prototypes.
4. Using testing to design performance: Performance goals must be defined before a project leaves the design phase. Testing validates performance against the specifications.
5. Testing without a test plan: No plan means inefficient and ineffective testing.
6. Testing without a specification: Testing ensures a system meets the specifications. Without a specification, you can’t test anything meaningful.
7. Asking developers to test their own programs: Developers make poor testers. They lack testing training and have a conflict of interest. Testers need an unbiased mind.
8. Testing without a goal: Without a clear goal, testing becomes aimless. You’ll never know when you’re done.
9. Relying solely on unsupervised beta testing: Beta testers need defined goals, supervision, and leadership to provide useful feedback.
10. Making design decisions during testing: Design decisions belong in the analysis and design phases.

The Role of Analysis and Design

Before any real testing can start, you need thorough analysis and design. When these are done right before implementation and testing, your chances of success go way up.

I once worked for a boss named Gary who ignored this rule. Gary wanted to implement a warehousing system for a client without a thorough specification. I objected. He didn’t care. His design process was to spend a couple of hours asking the client what they needed, then start coding. This led to a cycle: code something, show the client, make changes, repeat until the client gave up and called it acceptable. The project took far longer than necessary. It didn’t fully meet the client’s needs.

It was riddled with bugs. It demanded enormous support during the first couple of years.

That project is a warning, because the client paid for the shortcut for years afterward. Skipping analysis to start coding sooner feels like speed, and it comes due later as bug reports, support calls, and a customer who never quite trusts you again.

Marketing’s Role

Software testing shouldn’t be marketing’s job. But marketing should be involved in the analysis phase. A skilled analyst knows that marketing is a customer whose input during analysis is valuable. Any involvement beyond that stage can push the product in a different direction during testing and invalidate the test.

Why should specifications work as contracts?

A specification is a contract. The goal is to build something that meets the specification. This is the most effective way to deliver software that meets customer expectations, assuming solid analysis and design.

Say you’re hired to create a new warehouse system. You do your analysis, design the system, and the customer approves it. Then the customer decides to add bar coding mid-project. You must stop, assess how it affects the project (at the customer’s cost), and submit a new cost estimate and delivery date.

I’ve never had patience for teams that absorb changes like that quietly to keep the customer happy. The schedule slips anyway, the testers end up checking the product against a specification that no longer describes it, and everyone acts surprised when the release goes badly.

Maintaining Standards

Standards matter. Testing measures the implementation against both specifications and standards. Standards might cover field validation methods, buffer overflow prevention, consistent screen design, and similar concerns.

Testing confirms that the implementation matches everything in the specifications and standards. It doesn’t measure the product against customer expectations. That’s marketing’s job, and it should have been settled during analysis and design. If the specification meets customer expectations before implementation, the final product will meet those expectations. The specification represents the expectation.

Why does software testing matter to a business?

Software testing is among the most misunderstood parts of the development process, and I think companies that skimp on it are gambling with other people’s time and money.

I learned this one the hard way at a retailer, where I was in charge of production. We kept having big system failures. We had a division of labor between production, development and networking, so I finally audited all of it. The networking side was fine. The developers were working directly in production, so major untested changes went straight into the live system. One outage ran three days, when the change could have been backed out.

Edge cases are only half the job. I had a programmer who worked for me who always checked the edge cases, the things that weren’t normal. He never checked normal. We’d get the product, and it would handle every error condition fine and fail when somebody did the ordinary stuff. It used to infuriate me. I finally fired him.

Projects fail or underdeliver because of weak analysis, sloppy design, and testing that got squeezed when the schedule slipped. Write a specification, hold it as a contract, and test against it and your standards with people who have no stake in the code passing. Do that and your customers won’t be the ones finding your bugs.

Takeaways: Software testing requires a plan, clear specifications, and trained professionals. Treat specifications as contracts. Marketing belongs in the analysis phase. The goal of testing is to ensure the product matches the specifications and standards. Avoid the common pitfalls and maintain your standards throughout.

Frequently Asked Questions