Category: Testing

Toughest Testing Challenge 1.2 : Patch 1.0

There was a major bug in Toughest Testing challenge 1.2 .

Bug Details:
Management realized later that a simple module (dialogue box) like this can’t be tested forever, also they don’t have any idea about how far or how long to test this module.

Fix:
So, the management decides to give the project to  two different vendors.
First one was asked to generate  test ideas that can be executed within 15 minutes.[Time is non negotiable here]
A second vendor was asked to estimate the time required to test this module and generate test ideas for the time estimated.

What would be your test strategies if you are Vendor one?
What would be your test strategies if you are Vendor two?

And more over you should convince the client with your test strategy and estimates.

A Hint : Test Framing.

Happy Testing!

Dhanasekar S.

Norie’s Neat Nostrum :
“There’s no such thing as quick and dirty; if you want a quick job, make it a neat job” – Jerry Weinberg

How can kung fu stop something that stops kung fu?

Most software were developed by highly skilled programmers. But still none of them are bug free and many of them are very buggy.

“If builders built buildings the way programmers write programs, the first woodpecker that came along would destroy civilization.” – Jerry Weinberg

Software is a very volatile product, so it’s practically impossible to build bug free software even by the best of the programmers. So there exist testers to test the product. Testers look into a product with different perspective to inform management about the health of the product, thus helping management to take decisions better and faster. Most releases will go smooth and there will be failures in between (or vice versa 😉 ). Everything looks awesome until something goes wrong, then everything comes under the scanner. There will be a high-profile meeting to discover that THE problem was testing not done properly and THE reason was lack of time and THE solution is automation. 🙂

Now some testers who had talked about automation earlier will become automation test coders. These testers will evaluate automation tools and select the tool that has more voting in a polling conducted by ***interviews.com forum. A framework that Google shows first would be chosen as the best suited one. A PoC that automates the login page of their application. Now stage is set to automate 90%  of test cases. NOTE: 100% is not possible!!!

This is something similar to a sip test and home use test of an edible products, there was an interesting analysis about the failure of New coke in the book Blink. I too had many snap judgment failures and one among them was selecting a gaming console. In one of my earlier jobs where I was a game tester, was given an option to join either Xbox or PS2 team on my comfort with the consoles. I tried both for about ten minutes and I felt comfortable with Xbox, so moved to Xbox team. But later when I played for more than 2 hours continuously, I realized it is not comfortable at all, later found PS2 was more comfortable for longer time duration.

Without anticipating all such problems, a set of testers who were either bored by the kind of testing they do or misguided about automation testing, would become an automation engineer without much programming skills. They would then write buggy codes to find bugs in an application that was developed by skilled programmers , read the blog title again. 😀

In Kung Fu Panda II, Po finally stops the thing that stops kung Fu after finding a secret . The secret here is to invest in developing thinking, analytical skills of testers not just on automation tool.

Automation can help to solve the problem, but that is not the solution for all problems.

PS : kung fu panda 2 – An abandoned villain threatens, there exists an unknown secret, Po sets out  a mission to understand the secret to defeat the villain and bring peace. It is twice filled with attractiveness and awesomeness …

Think different but keep it simple :)

There is content below, I can read can’t you? oh!  Think different to see the content, but keep it simple 🙂

This is the most common problem we testers encounter quite often, it works in developer machine but not in test machine.

Once an interesting incident happened in one of the web application I was testing, application was working fine in developer’s environment but not in test environment.

The most likely reasons could be either  the test servers do not have the latest code or  there is conflict in environment. Quick check revealed that both the environments had the same code.

Now the focus was on to investigate the configuration of the servers.

As a tester how will  you investigate such situations?

Few tips to nail down such problems. Please share your tips as well.

1. Test in another test machine\server

2. Test in another dev machine\server

This will help to narrow down if there the problem is at client side or server side.

3. Check if  that machine  is also hosting the web server.

Time out issues and session management related issues may not be reproducible, if the machine you are testing in is a server machine.

4. Check for additional softwares installed.

Generally developer box will have lot of additional softwares installed, like debug tools, development related softwares that may prevent from the bug to be reproduced. Even the web browser in dev box will be loaded with tons of additional tools and utilities. Such additional installation may also prevent from some bugs being reproduced.

It is not enough to check if required softwares were  installed, it is equally important to make sure no additional unwanted softwares were installed.

Coming back to the issue, following those above mentioned approaches by pairing with developer helped to nail down the problem in few hours. The issue was, a bulk data extraction component used (it was ETL project) will work only in localhost, that does not have the capability to connect with external host. In the developer machine and dev server both web server and DB server was hosted  in the same machine, so that worked perfecly fine.But test environment had DB and Web server running in different machine, so this caused the problem. An important lesson learned from the incident is it is always good to have the test environment mimic the production. Keeping the test environment similar to production helped in finding the problem very earlier in the cycle.

Another good practice to follow, at any given time at least one test environment should be  in sync with production server, so any production issues reported can be easily replicated and analysed in a test environment. Having only one test environment means most time that will be updated with enhancements going on, that may also prevent from reproducing production issues because of code changes applied in the test environment.

How many of you have set up the test environments? No, I didn’t mean double clicking an installer provided by the developer. As a tester I learned a lot, by setting up test environment and maintaining it, setting up test environments will help  to understand your application better, it will help to get better test ideas, analyzing of bugs can be done  effectively .

Understanding test environment improves testing.

——————————————————————End of Post—————————————————————————–

Conference Of the Testers, By the Testers, and For the Testers

Bug DeBug Chennai Jan 29th 2011.

A Conference not by any professional\training oriented networks.

A Conference not for marketing commercial tools.

A Conference where no certificates sold.

A Conference where  no one talked from birds eye view, instead the real testers talked about ground reality.

A Conference from where  testers left  with hearts filled, not hands filled with useless pamphlets.

And most importantly the conference started on time and went as per the scheduled, isn’t this one point enough to prove this was conference with a difference ? 🙂

I traveled with Pradeep and Santhosh the previous day evening to Chennai, that was one of the most enjoyable drives I had in recent times.

Vipul Kocher, President, Indian Testing Board  started the conference with his keynote address  on “Present problems and  future solutions”, participants wouldn’t have asked for any better start. Vipul asked serious of hard questions  like “What was the last or the latest innovation that happened in testing?” He suggested testers to use heart as well along with brain  :). His key-note covered various problems and challenges faced by testers, and he credited that bugs are the only perfect being 🙂 Here is his presentation slides.

Next  was a simple but very powerful presentation on “Economical, Robust Web Automation using Sahi” by its founder Narayanan Raman. He shared the views about the current automation tools which make the testers to a tester developer. I too realised this after working in automation for a year or so. If you look at my earlier posts related to automation which recommends testers to learn DOM, HTML, Xpaths. Later realized that I shouldn’t be  a tester developer. But it seems Sahi has over come those problems of learning Xpaths and DOM, the demo showed the tool is very  simple to use, cross-platform cross browser supports is really the big advantage.

This was followed  by a talk on “Notes from a problem solving tester consultant” by the bad boy of software testing 😉 Pradeep Soundararajan, who was also the winner of  “NLTE Problem Solving Expert of 2010” :). He was at his usual best, the talk was filled with arrogance and humor. Here is his presentation.

Next presentation was “Smarter ways of doing Selenium Automation(Functional Test Automation)” by Ruturaj Doshi. unfortunately I missed this presentation, you can find the slides here.

Followed with delicious lunch.

The first session post lunch was on “The Emerging Trend of Cloud Computing and Software Testing” by Anuj Magazine. The presentation was interesting with lot of reference to history in general. He talked about emerging testing trends and its advantages and drawbacks in  virtualization and cloud computing world, slides are here.

Then it was the bug hunter, Ajay Balamurugadas, had just two slides one with a mind mapping diagram which had the essential skills needed for\to a bug hunter. He asked lots of questions and  tried to keep the session as interactive as possible. Slides are here.

The presentation sessions ended with talk on “Testing at Startups” by Praveen Singh, Founder 99 Tests. He shared his experience about the challenged faced while testing in a start-ups. He mentioned building testing skills and joining community and building authority are important for testers. Here is his presentation.

This was followed by interesting Q&A sessions,  hope that turned out to be very useful for budding testers, and the interesting questions got “Lessons Learned in Software Testing” book sponsored by Moolya Testing.  Testers were asked to share testing tips and the best tips got rewards.

Then the testers had chance to network and interact with the fellow testers. I too meet many testers, whom I knew so for only through blogs and twitter.

Hats Off and double Thumbs Up to Bharath & Chennai Software Testing Group and RIA-RUI for organise such an event without any high priority issues :-).

Every tester now proudly can say  this is my(our) conference, Thanks to the organizers and volunteers for making this possible.

Bug DeBug க்கு  ஒரு ‘ஓ’ போடு (Hip Hip Hurry to Bug DeBug)

Here are snaps from the conference

iTest

Last week I participated in a competition conducted by 99tests.com, a crowdsourced testing start-up from India.  This is my approach\experience report on how I tested and won the competition.

What were the givens?

No requirement, no design documents, no test cases, no use cases, no user stories. All I had were credentials to log in and the URL to test. So, do I need to agitate and not to start testing until I get requirements? Do I sit and write test cases? Any conventional scripted testers would’ve struck wondering what to do. So, here is a warning for all those scripted testers it looks things are changing because of the agile approach, and such crowdsourced testing services.

How did iStart?

I didn’t waste any time exploring the application. I thought the best approach with the givens (actually no givens ;-)) would be, jump straight into using the application and observing the behaviour. I thought that would help with forming test ideas and then to build from there.

Being a user of such online shopping sites helped me to frame some initial expectation. So, started with Follow the (user) Flow heuristics. I decided to create my own credentials instead of using test credentials and log in. Then observe how easy or buggy it is to find an item, add to cart, check out, paying through a third party payment gateway and choose a shipping address of my choice.

Did iFollow the flow?

Registration was successful upon giving correct details. But I didn’t get to see any error message, so decided to enter some invalid data to find the application behavior. What I saw in giving invalid input was the error messages were not user-friendly, not just in terms of the message, in terms of usability as well. Error messages were in a different page and users needing to click back button to get back to the registration page. On click on back button the entire data entered were lost. So, this made me take a deviation from my initial plans of Follow the flow. Decided to test this module thoroughly, wondering why? Read bug advocacy of this bug below.

How iDid bug advocacy?

I didn’t just log the bug with a summary line, description, steps to reproduce and screenshots. Instead, I also explained how and why this might bug the users and impact of the bug.

The user registration page plays a vital role in giving the first impression about the application behaviour. Also, users use online shopping sites mainly to save time, by displaying errors on another page and asking the user to come back to the previous screen to correct the data is actually wasting sufficient time. This advocacy helps in understanding the real impact of such bugs. So, always explain how any bugs\issues found would potentially affect the user experience.

Your bugs are your representatives. The bug logging also depends on context, if you know your developers very well and if he is sitting next to you, the bug logging or advocacy may differ from the way you log bugs in crowdsourced testing. Here you have no clue about, who is doing bug triage, developers and their understanding of the product. So always give as much detail as possible so that they can’t reject your bugs.

Always remember this  most famous movie dialogue of all time, from “The God Father” movie while logging the bugs

“I’m going to make him an offer (details) he can’t refuse (to fix).”

Did iJust log bugs?

No, I went through most of the bugs logged by other testers. I posted comments, raised questions where ever I felt the bug was really not a bug or if the priority was inflated or if the issues were duplicated. Also, I neither missed to appreciate some good bugs reported by other testers nor missed to learn from fellow testers. I was actually a little disappointed that there were a significant amount of bugs without clear description and duplicates.

What were my objectives?

Then winning the competition, my objective was to log the maximum number of valid bugs. Ended recording 50 valid bugs, maximum by any tester. Also wanted to maintain a high bug acceptance ratio, 86% of the bugs were accurate. Happy with that but still wanted to improve on acceptance ratio.

So, What iDidn’t test?

I found few even tested the Facebook ‘Like’ gadgets, Payment gateway and spell checks in the application’s blog. Though few of such bugs were valid, those were not from the application under test, so they got rejected. So,

Know your boundaries, so you do good enough testing in given time.

Ready to try some bug advocacy?

Here are a few other bugs logged by me, try to advocate for them. The application under test was an online shopping site.

  • “Similar Items” feature is missing.
  • Same book title, but displayed with a huge difference in price tag.
  • The amount should always be right justified.
  • On entering the special characters the system through an “invalid gift message.”
  • Pre-order items are shown as Available and Buy Now.
  • Is the final price displayed in product description inclusive of taxes?
  • Can’t store search results.

Happy Bug Hunting!

P.S.

Some of my favourite lessons in Bug Advocacy chapter of “Lessons Learned in Software Testing”  book

55: You are what you write.

58: Your bug report is your representative.

83: The summary line is the most important line in the bug report.

89: Use market or support data when appropriate.

101: When you decide to fight, decide to win!

Test[ing this] Case

The first thing flashes in most software testers mind immediately on hearing software testing is Test Case. Even most of us explain our work to others and even to fellow testers something like this “I understand requirements, write test cases and test the application, log the bugs and mark the test case pass or fail based on the expected results”. I too started my testing career by explaining what a test case is and SDLC model that was (never) followed to an interviewer. But soon, I realized that it is practically not possible (or I am not intelligent enough) to write most test ideas (cases) even with detailed requirements. Most test ideas don’t popup while analyzing the requirements, but while performing the real testing on the product. The test cases written with requirement documents are only based initial abstract understandings made on theory, but the real practical ideas come while testing. But even recently many people advice to write detailed test cases, that even a layperson can understand. Are they recruiting a layperson or a smart tester? Why still the main focus is with writing detailed test cases even in agile world? Looking back the water fall model, what I can think of reason for writing detailed test cases is, there was  a phase in SDLC were developers are busy writing and developing a product for months together, before delivering into testing team. So, probably some smart manager got this idea of test case writing to keep the testers engaged during that phase. Few rituals still followed without analyzing, if those suit the context. 1.  Write detailed test cases for the projects where there is good clarity on requirements. We got clear requirements so write detailed test cases. 2.  Write detailed test cases where there are no clear requirements . The requirements are not clear let use have detailed test case document. 3.  Even if the project has very tight deadline, a certain percentage of time is allocated for test case writing instead of focusing on real testing . Because Test case documents are (mis)believed to be the only Savior of testers. 4.  I’ve even seen reverse engineering of coming up with test cases from a well functioning application, with the help of help files 🙂 5.  There is something called traceability matrix, which traces all the detailed test cases back to your requirement, I think this is designed especially to say the whole world, see we’ve spent so much of man power to duplicate the requirements and here is the proof for that 6. And test cases are again considered as Savior in this job hopping era, but reality is forcing to write such detailed test cases and asking the testers to blindly execute just the documented test cases too are the also one of the reasons for job hopping. Why a similar kind of test cases is demanded everywhere for every context?Why do test case document looks similar where ever I go? 😦 . The only standard that is followed more or less correctly in testing world is test case template. The problem with such detailed test cases writing and following them is, soon the application will develop resistance against bug, because testers were focused only on what is written in the test case nothing less nothing more. Any tests executed blindly by a human just against expected result is as good as (as bad as) automated test checks. Such practices never help to spot the Black Swan* [The disproportionate role of high-impact, hard to predict, and rare events that are beyond the realm of normal expectations] you have to explore more with real application to spot the Black Swan. Testing according to James Bach Testing is the infinite process of comparing the invisible to the ambiguous so as to avoid the unthinkable happening to the anonymous. So, let me try to form a different Black Swam theory, as a tester our job is to predict and spot the Black Swan before the customers or client spot it. 🙂 Even writing test cases need exploration, but it becomes a concern when the explorations stop (or forced to stop) once test case are written. So, constantly come up with new ideas to add more value. Instead of wasting time in writing detailed test cases cheat sheets like this  Test Heuristics Cheat Sheet would cover most of the basic test cases, why to write again? If you feel You Are Not Done Yet, then check this You Are Not Done Yet. And choose SBTM or any relevant format that suits the context. Any available basic requirement document plus the above cheat sheet should be more than enough to do a good enough testing.

Testers aren’t  for documenting, they are for providing valuable information.

P.S: Can you recollect  the color of each letter of Google logo in its home page? (Of course with no doodles), the page most of  us visit every second hour. If you can’t, then you are affected with major Black Swan blindness.

Happy testing your Case!

Dhanasekar S.

* I’ve just started reading The Black Swan book by Nassim Nicholas Taleb. Yet to read it completely.

Configure, Operate, Observe and Analyze

In one of my previous post, I wrote about the below Windows  OS (XP with SP2) error message. This is one of the fascinating behaviours that took me few hours to crack down.

As a novice Window’s(OS) user.

Access is denied.

Ah!  I am the only user of my machine, and I am the one who created this folder, no one else has any rights to restrict me from accessing my own folder.

Make sure the disk is not full or write-protected.

What the hell! Don’t say all this nonsense. I didn’t protect it from writing. I can write whatever I want on the disk. How do I know the drive is full? How do I write-unprotect? Or an unwrite-protect ;)?

That the file is not currently in use

No, I am not renaming any file. I am renaming a folder, what is your problem?

User annoyed and calls customer care.

Now, as an advanced computer user.

Access is denied.

Looks like the permission settings for the folder is changed, maybe my wife? Son? Or Pet? Possible. Let me see the permission setting for the folder. Oh! This user has full access. Then, what is the problem?

Make sure the disk is not full or write-protected.

Nope, my drive has enough space, not write protected.

That the file is not currently in use

Files? I am renaming a folder. Let me see if the folder is kept open. Yes, so let me close and try. Still, I am getting the same error.

An annoyed user might solve the problem by logging off or closing all files and folder or restarting the machine, which forces him to close all the programs kept open.

As a tester, start analysing (Step 4 of testing Analyze, Step 1: Configure, Step 2: Operate, Step 3: Observe) the error message and follow the error message to see, to find the problem area.

 

Access is denied.

Does the application provide reasons to the Error? Yes it is it says

Make sure the disk is not full or write-protected.

The primary disc properties say it is either full or write-protected.

That the file is not currently in use

No, I am renaming a folder, but still, let me check if the folder is kept open? Yes.

So, let me close the folder and rename it. Even after that, I am getting the same message.

Are there any files inside the folder?

Yes, there are few.

Are they open?

Yes, a word document inside that folder is kept open.

Is that causing the problem? Let me close it and try.

At last, I can rename it, so “When a file inside a folder to be renamed is kept open, the user is not shown with an appropriate error message” the problem summary?

I have just solved the problem but haven’t got the exact steps to reproduce.

The problem was identified without knowing the proper base step up or configuration. So let me go to base state, also let me note down or take a snapshot of the current state where the problem was first spotted, including other files\folders those kept open, processes running to name a few.

So, what is the base state? I may even restart my machine or log off or close all opened files and folders.

Step 1: Configure

Let me log off and log in. Note down the entire process running (and use some tools like Systracer that takes a snapshot of your machine)

Step 2: Operate

1.    Open the Disk Drive (Say D :\)

2.    Open the folder (Say D:\Test)

3.    Open a file inside the folder (Say D:\Test\test.doc)

4.    Use back button to get back to the disk drive

5.    Try to Rename the folder

Step 3: Observe

Ah! This time I am getting a proper message.

So, what could be the problem?

Let me try changing the sequence, let me do this

1.    Open the Disk Drive (Say D :\)

2.    Open the folder. (Say D:\Test)

3.    Open a file inside the folder. (Say D:\Test\test.doc)

4.    Close the folder

5.    Reopen the folder

Now the sequence of opened windows changed, soon file first then the folder.

Nope! I got a proper message. Tried with different types of files kept open like excel, txt files, folder within folders and also be keeping few more similar file types open in same\different folders.

With no success decide to get back to the dirty environment and try again, I opened few other windows, files, browsers and application those were kept open when I first found this issue and tried all the above steps, this time I noticed, I was  able to reproduce the error once in a while but not consistently. But after more than an hour, I was not able to find the exact steps or pattern to reproduce it.

Then decided to quit and try later (Plunge and quit), but that one hour of exploration gave me confidence that it can be reproduced.

A few days later with lots of windows and browsers kept open. I tried again to reproduce this. Initial few attempts were not successful.

When I was about to rename the folder, I noticed one of my friends pinging me in IM, so I went to the browser and replied to him, then came back to folder window to rename the folder.

Whoopee! This time I got the error message I am looking for. It seems if I navigate between different windows I might be able to reproduce, but the error was not consistent. What next?? Trying out few other ways and finally was able to narrow down the steps to reproduce it consistently.

And, Here are the steps to reproduce.

1.    Open the Disk Drive (Say D :\)

2.    Open the folder (Say D:\Test)

3.    Open a file inside the folder  (Say D:\Test\test.doc)

4.    Use back button to get back to the disk drive

5. Wait (approximately) 10 seconds

6.    Try to Rename the folder

7.    Bingo!

Note to Readers:

  • Heuristics applied to solve are of James Bach’s Learning Heuristics SACKED SCOWS

Scouting Obsessively – discover the sources and tool you need
Authentic Problems – engage the mind
Cognitive Savvy – work with the rhythms of the mind
Knowledge attracts Knowledge – the more I know, the easier I learn
Experimentation – make learning vivid and direct
Disposable Time – lets me try new things (I tried this in my disposable time)
Stories – are how I make sense of things
Contrasting Ideas – lead to better ideas (Skepticism, Critical thinking, Lateral thinking, Systems thinking)
Other Minds – exercise my thinking and applaud my exploits
Words and Pictures – make a home for my thoughts
Systems Thinking – helps me tame complexity

Also, Procrastination , Plunge In and quit.

  • How to reproduce is solved. Now find why this happens?
  • Never let the user to get annoyed. As a tester, analyze every behavior of your application.
  • Read How to Investigate Intermittent Problems to learn the ability and the confidence to investigate an intermittent bug.
  • “There are no accidents” –Master Oogway, Kung Fu Panda(Movie)

FaceBook Logoff

I have this habit of logging into any shopping site like Flipkart with same ID, same time in two different browsers (Firefox and Chrome). With one of the sessions, I browse through the items, and other one to add to the shopping cart. This is very helpful to me, to do shopping faster.

I thought I would do the same with Facebook as well. I would use one of the sessions to read all the status updates and other session to chat or render the videos and watch, so I can browse through all the updates from friends in quick time. But I observed logging into second browser forces to log out from the previous browser session

Is this a problem?

I don’t think as a tester you can decide if this is a really a problem assuming either one of those scenarios as an industry standards ;-).Even if one of those scenarios is set as an industry standards it is not necessary to follow them blindly. So Face off 1 yourself as a user to analyze such scenarios.

We cannot think of everything before we start.

You don’t know what you want until you see it.

While shopping user (like me) has two sessions opened. I am done with my shopping .I do check out after adding all the items to the chart and make payment, check my mailing address,which  takes close to 5 to 10 minutes. And then I log out of this session but completely forgot about the other session opened in different browser, and if the user is using a shared computer which is very common in India. So, here it looks it is better to log off all the sessions in the machine if user log off from a session. But one way it is helpful to user if he can access more than one session at a time.

So, which is the correct behavior? It is always decided by what your users are up to. So face off like a user and analyze the observed behavior.

Even let say Flipkart decides to change this and allows only one user session similar to Facebook by maintaining the sessions in server instead of cookies. It should be communicated to user in a very positive manner by highlighting the security risk. Any change like this cannot go without educating the user. Users who are very used to such behavior may not accept those changes.

User satisfaction comes from handling change, not mitigating risk.

Wait. Did I say it is the user always? No(w), I am seeing some new terminology being used these days apart from user friendly, its advertiser friendly ;-). Sometimes you have to consider advertisers as well, so no harm in annoying user for some time.  There you should not think about user friendly close buttons.

Aware of your potential user\customer very well, to analyze what you observed is really a problem. None of the so called industry standards or testing techniques like orthogonal array, boundary value analyses or whatever it is, will help to capture or analyze such scenarios, and we can’t estimate for such analyses in advance. So

Apply your sapience and heuristics. Don’t follow rules.

“Hell, there are no rules here, we’re trying to accomplish something” ~ Thomas Edison

Note to Readers:

  • Here I have given just one possible scenario that I generally do as a user, I like you to bring\think in different perspective to analyses.
  • I had a vague idea about this log off session, was thinking on how to shape this in to an article. Finally this got this shape after  reading this wonderful post by Nathan Smith Estimation is bunk

1 –  Face off (movie) – A revolutionary medical technique allows an undercover agent to take the physical appearance of a major criminal and infiltrate his organization.