Category: Approach

Toughest Testing Challenge 1.2 : Patch 1.0

There was a major bug in Toughest Testing challenge 1.2 .

Bug Details:
Management realized later that a simple module (dialogue box) like this can’t be tested forever, also they don’t have any idea about how far or how long to test this module.

Fix:
So, the management decides to give the project to  two different vendors.
First one was asked to generate  test ideas that can be executed within 15 minutes.[Time is non negotiable here]
A second vendor was asked to estimate the time required to test this module and generate test ideas for the time estimated.

What would be your test strategies if you are Vendor one?
What would be your test strategies if you are Vendor two?

And more over you should convince the client with your test strategy and estimates.

A Hint : Test Framing.

Happy Testing!

Dhanasekar S.

Norie’s Neat Nostrum :
“There’s no such thing as quick and dirty; if you want a quick job, make it a neat job” – Jerry Weinberg

How can kung fu stop something that stops kung fu?

Most software were developed by highly skilled programmers. But still none of them are bug free and many of them are very buggy.

“If builders built buildings the way programmers write programs, the first woodpecker that came along would destroy civilization.” – Jerry Weinberg

Software is a very volatile product, so it’s practically impossible to build bug free software even by the best of the programmers. So there exist testers to test the product. Testers look into a product with different perspective to inform management about the health of the product, thus helping management to take decisions better and faster. Most releases will go smooth and there will be failures in between (or vice versa 😉 ). Everything looks awesome until something goes wrong, then everything comes under the scanner. There will be a high-profile meeting to discover that THE problem was testing not done properly and THE reason was lack of time and THE solution is automation. 🙂

Now some testers who had talked about automation earlier will become automation test coders. These testers will evaluate automation tools and select the tool that has more voting in a polling conducted by ***interviews.com forum. A framework that Google shows first would be chosen as the best suited one. A PoC that automates the login page of their application. Now stage is set to automate 90%  of test cases. NOTE: 100% is not possible!!!

This is something similar to a sip test and home use test of an edible products, there was an interesting analysis about the failure of New coke in the book Blink. I too had many snap judgment failures and one among them was selecting a gaming console. In one of my earlier jobs where I was a game tester, was given an option to join either Xbox or PS2 team on my comfort with the consoles. I tried both for about ten minutes and I felt comfortable with Xbox, so moved to Xbox team. But later when I played for more than 2 hours continuously, I realized it is not comfortable at all, later found PS2 was more comfortable for longer time duration.

Without anticipating all such problems, a set of testers who were either bored by the kind of testing they do or misguided about automation testing, would become an automation engineer without much programming skills. They would then write buggy codes to find bugs in an application that was developed by skilled programmers , read the blog title again. 😀

In Kung Fu Panda II, Po finally stops the thing that stops kung Fu after finding a secret . The secret here is to invest in developing thinking, analytical skills of testers not just on automation tool.

Automation can help to solve the problem, but that is not the solution for all problems.

PS : kung fu panda 2 – An abandoned villain threatens, there exists an unknown secret, Po sets out  a mission to understand the secret to defeat the villain and bring peace. It is twice filled with attractiveness and awesomeness …

iTest

Last week I participated in a competition conducted by 99tests.com, a crowdsourced testing start-up from India.  This is my approach\experience report on how I tested and won the competition.

What were the givens?

No requirement, no design documents, no test cases, no use cases, no user stories. All I had were credentials to log in and the URL to test. So, do I need to agitate and not to start testing until I get requirements? Do I sit and write test cases? Any conventional scripted testers would’ve struck wondering what to do. So, here is a warning for all those scripted testers it looks things are changing because of the agile approach, and such crowdsourced testing services.

How did iStart?

I didn’t waste any time exploring the application. I thought the best approach with the givens (actually no givens ;-)) would be, jump straight into using the application and observing the behaviour. I thought that would help with forming test ideas and then to build from there.

Being a user of such online shopping sites helped me to frame some initial expectation. So, started with Follow the (user) Flow heuristics. I decided to create my own credentials instead of using test credentials and log in. Then observe how easy or buggy it is to find an item, add to cart, check out, paying through a third party payment gateway and choose a shipping address of my choice.

Did iFollow the flow?

Registration was successful upon giving correct details. But I didn’t get to see any error message, so decided to enter some invalid data to find the application behavior. What I saw in giving invalid input was the error messages were not user-friendly, not just in terms of the message, in terms of usability as well. Error messages were in a different page and users needing to click back button to get back to the registration page. On click on back button the entire data entered were lost. So, this made me take a deviation from my initial plans of Follow the flow. Decided to test this module thoroughly, wondering why? Read bug advocacy of this bug below.

How iDid bug advocacy?

I didn’t just log the bug with a summary line, description, steps to reproduce and screenshots. Instead, I also explained how and why this might bug the users and impact of the bug.

The user registration page plays a vital role in giving the first impression about the application behaviour. Also, users use online shopping sites mainly to save time, by displaying errors on another page and asking the user to come back to the previous screen to correct the data is actually wasting sufficient time. This advocacy helps in understanding the real impact of such bugs. So, always explain how any bugs\issues found would potentially affect the user experience.

Your bugs are your representatives. The bug logging also depends on context, if you know your developers very well and if he is sitting next to you, the bug logging or advocacy may differ from the way you log bugs in crowdsourced testing. Here you have no clue about, who is doing bug triage, developers and their understanding of the product. So always give as much detail as possible so that they can’t reject your bugs.

Always remember this  most famous movie dialogue of all time, from “The God Father” movie while logging the bugs

“I’m going to make him an offer (details) he can’t refuse (to fix).”

Did iJust log bugs?

No, I went through most of the bugs logged by other testers. I posted comments, raised questions where ever I felt the bug was really not a bug or if the priority was inflated or if the issues were duplicated. Also, I neither missed to appreciate some good bugs reported by other testers nor missed to learn from fellow testers. I was actually a little disappointed that there were a significant amount of bugs without clear description and duplicates.

What were my objectives?

Then winning the competition, my objective was to log the maximum number of valid bugs. Ended recording 50 valid bugs, maximum by any tester. Also wanted to maintain a high bug acceptance ratio, 86% of the bugs were accurate. Happy with that but still wanted to improve on acceptance ratio.

So, What iDidn’t test?

I found few even tested the Facebook ‘Like’ gadgets, Payment gateway and spell checks in the application’s blog. Though few of such bugs were valid, those were not from the application under test, so they got rejected. So,

Know your boundaries, so you do good enough testing in given time.

Ready to try some bug advocacy?

Here are a few other bugs logged by me, try to advocate for them. The application under test was an online shopping site.

  • “Similar Items” feature is missing.
  • Same book title, but displayed with a huge difference in price tag.
  • The amount should always be right justified.
  • On entering the special characters the system through an “invalid gift message.”
  • Pre-order items are shown as Available and Buy Now.
  • Is the final price displayed in product description inclusive of taxes?
  • Can’t store search results.

Happy Bug Hunting!

P.S.

Some of my favourite lessons in Bug Advocacy chapter of “Lessons Learned in Software Testing”  book

55: You are what you write.

58: Your bug report is your representative.

83: The summary line is the most important line in the bug report.

89: Use market or support data when appropriate.

101: When you decide to fight, decide to win!