Testing != test execution

Jeff Fry on Testing

Testing includes:

  1. Building a useful mental model of the application under test, including what value it should provide and what risks could threaten that value,
  2. Designing powerful tests, using that model to investigate important questions about the application’s quality,
  3. Test execution (which might be automated entirely, partly, or not at all),
  4. Analyzing the results to determine if there’s a problem, how severe it might be, or otherwise answer stakeholder questions, and
  5. Test reporting that clearly communicates the impact of the bug, and how to quickly reproduce it.

We often talk about testing as if it’s only test execution, yet often the most interesting, challenging, skill-intensive aspects of testing are in creating a mental model that helps us understand the problem space, designing tests to quickly and effectively answer key questions, analyzing what specifically the problem is, and communicating it effectively.

View original post

Mobile App Test Coverage Model : LONG FUN CUP

Smart phones changed the way we interact with software. Once mobile apps were used to kill time but now there are apps those could save lives.

They are changing the way live, thus we need to change the way we test.

Smartphone applications are developed with immense creativity and effort. Mobile users demand a sleeker experience with applications compared to desktop users. The mind set of mobile users is very different from web or desktop users. Smartphone apps are used on the move (e.g.: while walking or using a toilet), and mobile devices have a lot to offer through hardware location tracking, gyroscopes and other integrated features.

LONG FUN CUP model helps to achieve coverage at UI  level. This is designed factoring in the new touch generation mobile devices hardware, software and users mindset.

LONG FUN CUP

Location: It’s a sin to test mobile app sitting at your desk, get out!

Location tracking is a key feature offered by mobile phones, which any app must make use of it based on its core functionality. For instance, if it’s a cab booking or food ordering app, app should be smart enough to track your current location and provided suggestions surrounding your locality. Unlike web apps, where you expect user to enter location details, here the work flow should be different. Never test such apps sitting at your desk. Check if the app asks for user permission before tracking the location, does app allow users revoke the permissions? Both Android and iOS provide options to mock your locations for testing the app using emulators and simulators.

Orientation: It’s a sin to test mobile app sitting at your desk, lie in the couch.

People can change the orientation for various reasons, any mobile app should be able to provide a consistent user experience across different orientations. Test all your screens, pop-ups, toast messages, forms in all supported orientations. There are instance where the filled in form data disappears when you change the orientation. Never test a mobile app just sitting in your desk, take it to rest rooms, lie down in a couch and use it while traveling and test if it gives the same user experience with all different orientation.

Network: It’s a sin to test mobile app sitting at your desk, switch networks.

Mobile devices supports both cellular and wi-fi, they can automatically switch between any available networks. How does your app behave when the device switches between networks? These are critical especially if your apps core purpose is to send and receive data to remote server like e-commerce, banking, broadcasting apps. Make sure all such functionality behavior is tested for all such varying network switching, network strengths.

Gestures: In mobile world, app responds to gestures not clicks.

Does your app supports all standard gestures? Is it consistent across the app? If it uses any new gestures, is it easy for the users to understand? I would recommend to play The Room game (a paid game available for both iOS and Android) to understand the unlimited potential of gestures.

Function:

Any function that defines or distinguishes the product or fulfills core requirements. Test for interactions, error handling, starting and closing of the app, file access, navigation, multimedia, and sync. Did you try tapping all GUI? Did you fill in all the input fields? Did you navigate to all the screens? Are you able to navigate back?

User Scenarios:

How easy or how hard is it to complete a task using the app? Is the first time user able to understand the work flow? Any scenario should have a story that is credible, motivating, complex and easy to evaluate. List possible users and list them the ways they might use the system to accomplish a task. Try to think about disfavored users and how they might try to exploit the app. Compare with competitor or web interface to get more real life scenarios. Refer: An Introduction to Scenario Testing

Notifications:

Notifications enable an application to inform its users that it has something for them. How does your application use notifications? How easy is to turn on or turn off your app from notifications? Does the application use local or push notifications? How does your application behave if device is in sleep mode? Does the app provide too many notifications? Test for all available types of visual notifications, sound and vibration.

Communication:

These days’ people use mobile phones for various purpose, but the primary purpose for voice calls. How does the app behave after interruptions by an incoming call or an SMS? Test for such interruptions from voice communication and all necessary functions of your app. Many times app will not be able to recover after attending a voice call for a longer duration.

Updates:

Unlike desktop mobile app gets frequent updates. Not only app updates but as a tester one has to be aware of OS updates as well. How the users are informed about updates? Does the app support the silent update? Track what changes or new features are available in the latest OS update. Analyse if the app needs any modification because of the OS updates.

Platform:

It is very critical for any mobile app tester to have a good understanding of the popular mobile platforms especially Apple and Android. Be a  fan-boy of either one of these platforms (it’s impossible to be a fan of both 😉 ) So, you know why Apple and Android does certain things in certain way. You need to be aware of the history and also the latest trends in the mobile platform. What tools are available for testing and what kind of test-ability layers are provided by the platform. Example Developer Options in Android, Instruments in Xcode.

Testers also should have good knowledge in App store Approval process, HIG, Android Design guidelines. This helps to take the app to the market very quick. It takes one to two weeks to get it approved by Apple App Store, so rejection wastes a lot of time. So, testers has to have a checklist handy to check those approval and guideline tests.

Testers should also should be aware of android fragmentation, you will never know in which device your app works and which it may fail. So, should keep an eye on the popular devices, new devices and the android OS market share to select an optimal device matrix to test.

Smart Phone Mobile App Needs Smart Testers

I am happy and proud to share that I will be presenting “Smart Phone Mobile app needs smart testers”  in CAST 2014.

Mobile applications are developed with immense creativity and effort. Mobile users demand a sleeker experience with applications compared to desktop users. End users set their expectations very high, based on their experience with state of the art iOS and Android platforms they use every day. The mindset of mobile users is very different from web or desktop users. The key aspects users expect from mobile app is speed, sleekness and social sharing (SSS). As a result, testing mobile apps must be on par with experience offered by state of the art mobile platforms.

This talk will cover

  • How to tune the tester’s mindset to model test approaches specific to smart phone apps
  • How to design tests at the UI level to find issues beyond the usual functional and non-functional testing.
  • How to  design mobile tests to uncover issues hidden under mobile UI.
  • How to design tests for  user experience.

Full CAST 2014 schedule is here

See you in New York!

HOLY COW!!!

HOLY COW!! It’s been long time since I blogged.

I was thinking for a while about what are the most important skills that help a tester to craft his own methods instead of following industry standards or best practices? This triggered while reading through one of the old posts of James Bach again recently,  especially this particular quote that grabbed my attention, “Following is for novices who are under active supervision. Instead, I craft methods on a project by project basis, and I encourage other people to do that..”

Heuristics

Extensive research is never possible in testing commercial software, due to time and budget constraints. So, how to test software quickly? Heuristics are used to speed up the process of finding an optimal solution. It’s something that helps you approach a problem and take decisions, which should be applied wisely. In order to write better on heuristics I wanted to know the Tamil (my mother tongue) meaning for heuristics. I decided to search in Google, from my knowledge and experience I believe Google would help me to get Tamil meaning of heuristics, but unfortunately it couldn’t get an answer for that. So, I decided to ask in Twitter addressing to few Tamil experts. This is heuristics, an approach to problem, but no guarantee that it solves problem.

Oracles

If heuristics help in approaching problems or take decision, Oracles help to identify problems. Some of  oracles used in testing are

1. Requirements

2. Comparing against competitors products

3. Consistency with in the products

L

“A tester is someone who knows that things can be different.” – Jerry Weinberg.

Y- Why?

“Questioning ignites thinking, leads to progress”  – Dhanasekar

Whenever I train novice testers, I provide them with few tasks, when they come back with a solution, I probe them with many questions like why was this done? why was this necessary? why do they think this is right solution? why not do it in a different way? Most times there won’t be concrete answers. My next step is to ask them to find answers for my questions. After finding answers  they would realise there were many more to learn and explore, but still many wouldn’t realise that those questions ignited their minds. The important thing is not to stop questioning. What we observe is not testing in itself, but testing exposed to our way of questioning.

Cognitive

Observation

If HOLY Sea(C) is preparation and practice for a battle, observation is the real battle. For instance, if your application sends emails, have you ever observed how it appears in the email alert message pop-ups in  many popular clients on windows platform or popular notifiers like Growl in Mac OS X? This example is out of an observation made on such email pop-up. This client we worked for, put their branding advertisements at the start of the email, so the alert pop-ups showed the first two lines of the content which is purely marketing text, but not important content. Because of which user might assume this to be spam and ignore it. This test idea would have never triggered if I missed to observe the alert pop-up. Making observations after executing tests is not end of testing, subsequent test ideas mostly result out of those observations. So, missing critical observation is missing potentially risky tests.

Toughest Testing Challenge 1.2 : Patch 1.0

There was a major bug in Toughest Testing challenge 1.2 .

Bug Details:
Management realized later that a simple module (dialogue box) like this can’t be tested forever, also they don’t have any idea about how far or how long to test this module.

Fix:
So, the management decides to give the project to  two different vendors.
First one was asked to generate  test ideas that can be executed within 15 minutes.[Time is non negotiable here]
A second vendor was asked to estimate the time required to test this module and generate test ideas for the time estimated.

What would be your test strategies if you are Vendor one?
What would be your test strategies if you are Vendor two?

And more over you should convince the client with your test strategy and estimates.

A Hint : Test Framing.

Happy Testing!

Dhanasekar S.

Norie’s Neat Nostrum :
“There’s no such thing as quick and dirty; if you want a quick job, make it a neat job” – Jerry Weinberg

How can kung fu stop something that stops kung fu?

Most software were developed by highly skilled programmers. But still none of them are bug free and many of them are very buggy.

“If builders built buildings the way programmers write programs, the first woodpecker that came along would destroy civilization.” – Jerry Weinberg

Software is a very volatile product, so it’s practically impossible to build bug free software even by the best of the programmers. So there exist testers to test the product. Testers look into a product with different perspective to inform management about the health of the product, thus helping management to take decisions better and faster. Most releases will go smooth and there will be failures in between (or vice versa 😉 ). Everything looks awesome until something goes wrong, then everything comes under the scanner. There will be a high-profile meeting to discover that THE problem was testing not done properly and THE reason was lack of time and THE solution is automation. 🙂

Now some testers who had talked about automation earlier will become automation test coders. These testers will evaluate automation tools and select the tool that has more voting in a polling conducted by ***interviews.com forum. A framework that Google shows first would be chosen as the best suited one. A PoC that automates the login page of their application. Now stage is set to automate 90%  of test cases. NOTE: 100% is not possible!!!

This is something similar to a sip test and home use test of an edible products, there was an interesting analysis about the failure of New coke in the book Blink. I too had many snap judgment failures and one among them was selecting a gaming console. In one of my earlier jobs where I was a game tester, was given an option to join either Xbox or PS2 team on my comfort with the consoles. I tried both for about ten minutes and I felt comfortable with Xbox, so moved to Xbox team. But later when I played for more than 2 hours continuously, I realized it is not comfortable at all, later found PS2 was more comfortable for longer time duration.

Without anticipating all such problems, a set of testers who were either bored by the kind of testing they do or misguided about automation testing, would become an automation engineer without much programming skills. They would then write buggy codes to find bugs in an application that was developed by skilled programmers , read the blog title again. 😀

In Kung Fu Panda II, Po finally stops the thing that stops kung Fu after finding a secret . The secret here is to invest in developing thinking, analytical skills of testers not just on automation tool.

Automation can help to solve the problem, but that is not the solution for all problems.

PS : kung fu panda 2 – An abandoned villain threatens, there exists an unknown secret, Po sets out  a mission to understand the secret to defeat the villain and bring peace. It is twice filled with attractiveness and awesomeness …

Marching to Moolya :-)

March turns out to be a wonderful month for me third time in succession.
March 2009 : Celebrating A R Rahman Oscar winning moments. He and his music keeps inspiring me.
March 2010 : I featured in James Bach’s blog, which I consider as one of the  greatest achievements in my professional career. His writings fuel the urge I have in me to learn the testing craft better.
March 2011 : The most important move in my career.  Very proudly and loudly announcing “I am joining Moolya” (read this in font size 72 ;))

The marching started when I got to know Pradeep Soundararajan through his blog. From then we had regular interactions both online and also in person. I was impressed and inspired by his passion towards software testing. One fine day Pradeep shared the news about he co-founding a testing services company along with Santhosh Tuppad. From that moment on I started dreaming that I should join them. But I was not sure what their plans were then. We used to meet now and then, to discuss and fill 😉 a lot. Later one day Pradeep asked about my interest in joining Moolya.  I immediately expressed my interest to join. Because I had already decided  to join them anytime.

It was one of the easiest plus the most important decisions I had taken so for. It was easy because I knew them very well for close to two years by now. They kept me updated about the progress they made in-terms of setting up the office. They also shared the dreams, plans and the visions they have for the future. I was so happy to be present in the inauguration function. Though I was ready to join any time, They were very clear in hiring.  They hire only if, they are very sure about keeping the employees happy both in terms of work and compensation. If you want to know more reasons why I join Moolya read further else start sending your test reports! :-). Then one fine evening I was asked to come down to their office and Pradeep said they are in a position to hire me. The moment I was waiting for!!!!

Why Moolya?
They are passionate about testing, so they care about the quality, customers, employees and also bugs :-). This is the place where you’ll get freedom to work over tracking employees by numbers and metrics. Place where you get responsibilities and opportunities to try and learn new ideas over running the same test cases again and again. Place were purpose motto is respected over business motto. And more importantly they don’t count the head, they count the brain. I love to be part of Moolya’s future success story from the start. I am thrilled and excited to work for\with such passionate and wonderful people. I thank Moolya for the faith and confidence they have in me, we will rock for sure.

Future looks promising and Beautiful!!!!

Also, I like to thank James Bach and Weekendtesting, they played a vital role in getting to know Pradeep Soundararajan and Santhosh Tuppad. I will not do justice to myself, if I miss to thank Pragmatist, the blog which I believe is the starting point for all this happenings. Dear testers please start blogging and let the world know who you are.

“Life begins at the end of your comfort zone. “- Neale Donald Walsch.

(BTW, Pradeep already promised that he will push me out of my comfort zone.)

To,
Dear Gladiators and Spartans, 🙂
Without doubt, so for this is the best team I worked with, in terms of commitment, fun and bonding. I’ll miss you all and the amazing fun we had together :-(. Thanks for all the fun, enjoyment and work we achieved together. Fifteen months with this team is the period where I laughed and enjoyed the most in my entire office life.

“But I have promises to keep, and miles to go before I sleep” – Robert Frost

Think different but keep it simple :)

There is content below, I can read can’t you? oh!  Think different to see the content, but keep it simple 🙂

This is the most common problem we testers encounter quite often, it works in developer machine but not in test machine.

Once an interesting incident happened in one of the web application I was testing, application was working fine in developer’s environment but not in test environment.

The most likely reasons could be either  the test servers do not have the latest code or  there is conflict in environment. Quick check revealed that both the environments had the same code.

Now the focus was on to investigate the configuration of the servers.

As a tester how will  you investigate such situations?

Few tips to nail down such problems. Please share your tips as well.

1. Test in another test machine\server

2. Test in another dev machine\server

This will help to narrow down if there the problem is at client side or server side.

3. Check if  that machine  is also hosting the web server.

Time out issues and session management related issues may not be reproducible, if the machine you are testing in is a server machine.

4. Check for additional softwares installed.

Generally developer box will have lot of additional softwares installed, like debug tools, development related softwares that may prevent from the bug to be reproduced. Even the web browser in dev box will be loaded with tons of additional tools and utilities. Such additional installation may also prevent from some bugs being reproduced.

It is not enough to check if required softwares were  installed, it is equally important to make sure no additional unwanted softwares were installed.

Coming back to the issue, following those above mentioned approaches by pairing with developer helped to nail down the problem in few hours. The issue was, a bulk data extraction component used (it was ETL project) will work only in localhost, that does not have the capability to connect with external host. In the developer machine and dev server both web server and DB server was hosted  in the same machine, so that worked perfecly fine.But test environment had DB and Web server running in different machine, so this caused the problem. An important lesson learned from the incident is it is always good to have the test environment mimic the production. Keeping the test environment similar to production helped in finding the problem very earlier in the cycle.

Another good practice to follow, at any given time at least one test environment should be  in sync with production server, so any production issues reported can be easily replicated and analysed in a test environment. Having only one test environment means most time that will be updated with enhancements going on, that may also prevent from reproducing production issues because of code changes applied in the test environment.

How many of you have set up the test environments? No, I didn’t mean double clicking an installer provided by the developer. As a tester I learned a lot, by setting up test environment and maintaining it, setting up test environments will help  to understand your application better, it will help to get better test ideas, analyzing of bugs can be done  effectively .

Understanding test environment improves testing.

——————————————————————End of Post—————————————————————————–

Conference Of the Testers, By the Testers, and For the Testers

Bug DeBug Chennai Jan 29th 2011.

A Conference not by any professional\training oriented networks.

A Conference not for marketing commercial tools.

A Conference where no certificates sold.

A Conference where  no one talked from birds eye view, instead the real testers talked about ground reality.

A Conference from where  testers left  with hearts filled, not hands filled with useless pamphlets.

And most importantly the conference started on time and went as per the scheduled, isn’t this one point enough to prove this was conference with a difference ? 🙂

I traveled with Pradeep and Santhosh the previous day evening to Chennai, that was one of the most enjoyable drives I had in recent times.

Vipul Kocher, President, Indian Testing Board  started the conference with his keynote address  on “Present problems and  future solutions”, participants wouldn’t have asked for any better start. Vipul asked serious of hard questions  like “What was the last or the latest innovation that happened in testing?” He suggested testers to use heart as well along with brain  :). His key-note covered various problems and challenges faced by testers, and he credited that bugs are the only perfect being 🙂 Here is his presentation slides.

Next  was a simple but very powerful presentation on “Economical, Robust Web Automation using Sahi” by its founder Narayanan Raman. He shared the views about the current automation tools which make the testers to a tester developer. I too realised this after working in automation for a year or so. If you look at my earlier posts related to automation which recommends testers to learn DOM, HTML, Xpaths. Later realized that I shouldn’t be  a tester developer. But it seems Sahi has over come those problems of learning Xpaths and DOM, the demo showed the tool is very  simple to use, cross-platform cross browser supports is really the big advantage.

This was followed  by a talk on “Notes from a problem solving tester consultant” by the bad boy of software testing 😉 Pradeep Soundararajan, who was also the winner of  “NLTE Problem Solving Expert of 2010” :). He was at his usual best, the talk was filled with arrogance and humor. Here is his presentation.

Next presentation was “Smarter ways of doing Selenium Automation(Functional Test Automation)” by Ruturaj Doshi. unfortunately I missed this presentation, you can find the slides here.

Followed with delicious lunch.

The first session post lunch was on “The Emerging Trend of Cloud Computing and Software Testing” by Anuj Magazine. The presentation was interesting with lot of reference to history in general. He talked about emerging testing trends and its advantages and drawbacks in  virtualization and cloud computing world, slides are here.

Then it was the bug hunter, Ajay Balamurugadas, had just two slides one with a mind mapping diagram which had the essential skills needed for\to a bug hunter. He asked lots of questions and  tried to keep the session as interactive as possible. Slides are here.

The presentation sessions ended with talk on “Testing at Startups” by Praveen Singh, Founder 99 Tests. He shared his experience about the challenged faced while testing in a start-ups. He mentioned building testing skills and joining community and building authority are important for testers. Here is his presentation.

This was followed by interesting Q&A sessions,  hope that turned out to be very useful for budding testers, and the interesting questions got “Lessons Learned in Software Testing” book sponsored by Moolya Testing.  Testers were asked to share testing tips and the best tips got rewards.

Then the testers had chance to network and interact with the fellow testers. I too meet many testers, whom I knew so for only through blogs and twitter.

Hats Off and double Thumbs Up to Bharath & Chennai Software Testing Group and RIA-RUI for organise such an event without any high priority issues :-).

Every tester now proudly can say  this is my(our) conference, Thanks to the organizers and volunteers for making this possible.

Bug DeBug க்கு  ஒரு ‘ஓ’ போடு (Hip Hip Hurry to Bug DeBug)

Here are snaps from the conference