Category: Testing

Bugs and Their Impacts. But Who Cares!

I’ve been using trading software for more than fifteen years. A few days ago, I logged in to buy a few stocks. After adding the stocks and filling in a few other details by navigating through a couple of screens, the app showed me the final confirmation button. I reviewed all the information and clicked ‘buy,’ expecting the trade to complete.

To my surprise, the app displayed the following error message, and the transaction was incomplete.

I read the message a few times and tried to place the order again a few more times as I neared the day’s cutoff time. Meanwhile, the stock price was increasing.

As per the message, my account was under verification for some reason. I tried to recall if I had submitted any verification recently or if there was an annual verification process. I remember that a few months back, a nominee update was made mandatory as per SEBI, and I remember submitting a form. But that process was complete, and I received acknowledgement via SMS and email. I was unsure what verification was pending for the account I had used for a long time.

Going with the error message, I frequently checked my SMS for confirmation to place the order before the day’s cutoff time. But no SMS came until the trading close time. Meanwhile, the stock price was shooting up, and I felt helpless and annoyed.

The following day, I kept an eye on my phone to get the confirmation message that never came. The afternoon came, and I lost my patience. I decided to call customer care. After waiting for a while, an executive finally picked up the call. I explained the issue instantly and politely, and she suggested I reset the password.

I was like, “WHAT?!!!” “SERIOUSLY???” 

She politely confirmed, “Yes, sir! Resetting your password will fix this issue.”

It did work!

Finally, I completed my transaction at a nearly 2.5% higher rate.

After that, I decided to dig deeper to understand why this happened. This behaviour never occurred earlier. Here is my analysis, using only my information as an end user. The issue seems to be with password reset. The application requires a password reset every six months, and I remember doing this all these years. What changed recently is that one can use FaceID to log in to the mobile app and a QR code from the mobile app to log in to the web interface, bypassing the need to type the password and probably allowing me to log in with an expired password. However, due to solid checks at the time of cash transactions and trading transactions, this gets flagged and prevents further processing.

Leaving aside those reasons, let’s look at the impact.

Impact on the user: Loss of money and time. 

Potential Impacts on the Business: 

  1. Increased expenses due to additional customer care support calls. 
  2. A decrease in the customer base due to frustrated users.
  3. Reduced transactions that might lead to the company not meeting its KPIs/OKRs.

An interesting observation is that the customer care executive instantly provided me with the solution, so the company is aware of this issue. The business decides when to fix the problem, and testers and customer service executives can only report the problem with the impact and risk of the problem and the risk of not fixing it.

As an end user, I now know what to do when I encounter this issue next time 🙂 Life moves on.

Happy Testing!

PS: While resetting the password, I found this funny error message. Are there one or two or more problems here?

Do what is right, not what is easy.

In my current role, more than twenty testers report to me. All of them were fresh college grads when I joined as project manager two years back.

They are constantly encouraged to blog, use mindmaps, share their learning and apply heuristics while testing. All important testing conferences, testing articles and testing contests are shared with the team regularly. And, I regularly discourage them doing QB certifications, even though company encourages certification.

Recently, One of the testers from my team attended an interview. The interview was for an American IT services giant. During the interview, he was asked why haven’t he done any QB certification, according to the interviewer that is just 40 simple questions to be answered and it’s easy to get certified. This guy replied if that is the case, can he become a doctor by answering 40 simple medical questions. I felt very proud of what he did and what I am doing to the young testers.

Amma.jpg.001

Oh! Curious to know the interview results?

He got selected. The trend has changed. Now, having a QB certification is too mainstream. Testers who don’t have any certification are very less and they are standing out among the crowd.

Happy Doctors’ day [ India celebrate National Doctors day on July 1st ]

NYCTesters July Meet

I was invited to do a two-hour workshop on mobile application testing at the #NYCTesters July Meet. I was thrilled and excited to speak in the gorgeous New York City.

The workshop was on Modelling Mobile Apps and the LONG FUN CUP model. So, started to shape model the session and planned to cover these during the seminar.

StoryBoard

The background of the audience was unknown, so explored the Meetup RSVP to model the audience first and then tried to structure the workshop to benefit a variety of audiences.

DinosaurWhisperer

The modelling of the audience helped to recognise that some people are new to testing. So, it became necessary to explain what is a Model and why modelling is required during testing, before the start of the workshop.

The venue was not hard to find, Google Maps helped not only to reach the place but also to explain about model &  the importance of modelling. Followed by a demonstration on how to model a mobile app & apply LONG FUN CUP test model to it. Later, the audience was given a mission to work for 40 minutes, then debrief.

Intro

Exercise

I acted as the stakeholder, and the debriefing was interactive,  participants discussed enthusiastically with lots of observations and bugs. Personally, it was a good learning for me in many ways, as I was doing it for a totally new audience. Later the discussion continued in the nearby bar with some bottles of IPA. That’s the beauty of hanging around with passionate professionals.

Now, looking forward to me full day tutorial at CAST 2015.

User’s Mistakes and his experience

Typing on the mobile device keypad is harder compared with a physical keyboard. An app that gets minimum input from users will easily win users’ heart. The reasons are the keypad is smaller & user needs to toggle between alphabets, numbers, special characters and to change the letter case. Because of this, the password setting pattern is changing after the popularity of mobile apps.

I stumbled upon a claim from a To do list mobile app, that it is the simplest To do list mobile app, so decided to give a try. The tester instinct within me never sleeps, especially if I see these two words mobile and simple. Being a big fan of Jony Ive, I always look for the simplest and cleanest of things.

In general, most apps work fine if the user makes no mistake. But, the user experience starts degrading the moment user makes mistakes. This is true for most of the apps. I make this conclusion based on the apps I test and use. My user experience test report always includes the experience, once when the user never makes any mistakes and once running the same scenarios by making all possible mistakes.

When I tried with a password that didn’t meet the app’s password rule, here is the error message I got.

Error Message

It took close to a minute to read and understand the rule, then dismiss the message to create a new password. I have already lost a minute, but haven’t completed setting my password. Now, when I started typing I was not sure and again failed to meet their policy. Unlike the web, I cannot keep the error message window and type the password. Finally, after two minutes of struggle I was able to register into the app, that claimed the simplest of all. In mobile world two minutes is a long time. Two minutes in mobile is like two hours in web (Please refer Nolan’s movies like Intersellar or Inception ;-))

For those who still think, it’s fine and the message is not so bad, users has to read and dismiss the message to enter the text, there are better solutions.

Wait!

Before scrolling down to see the embedded video, think for a while if we can provide a better solution.

This is how  Dashlane app solved it. It didn’t take more than 20 seconds to set the password. More importantly, the app avoided user from making mistakes.

This is how a Context Driven Tester thrives, always exploring, observing, relating and learning.

Also, the mobile application testing needs a different approach compared to traditional desktop softwares. My CAST 2015 tutorial on Mobile Application Testing has more such things to offer.

A Full Day Tutorial on mobile app testing & mindmaps at CAST 2015

CDT Blogpostand CAST are always close to my heart. I have had a fantastic run with the CDT community and take pride in being a part of the CDT family. I always enjoy meeting and learning from this community that does what is dependable, but not what is easy.

In CAST 2014 I gave a 40-minute lecture and met a lot of wonderful testers. This year I am taking one full day tutorial on Mobile app testing and mind maps. This tutorial was a suggestion from the master, James Bach after he sat through my talk in CAST 2014. This is what he stated after my talk “DS; this content is worth a two-day workshop.” The workshop will be a blend of my last year talk and the tutorial I conducted in 2013 EuroSTAR.

I am glad that I am capable of doing a full day tutorial in CAST 2015. This year’s theme is moving testing forward, and I believe mobile testing is a moving forward topic. This and This blog post would give an insight of what you can expect from this tutorial. Apart from this, I blog exclusively on mind maps here.

Catch you at the beer city of US 😉

Producing Software Is Not The Purpose Of Software Development

Think Different

Producing Software Is Not The Purpose Of Software Development

[Tl;Dr: All too often we can all get so wrapped-up in the joys and frustrations of creating software that we lose sight of the real purpose.]

I often invite my conference audiences to spend a few minutes discussing, amongst themselves, some question or other relevant to the session. One of the questions which I regularly pose is:

“What is the purpose of software development?”

This question tends to confound the audience for a few moments, prior to their entering into animated discussions on the subject. It always seems like a few minutes is nowhere near long enough to do the question justice.

I ask the question for a number of reasons, not least because of Dan Pink’s assertion that the three keys to intrinsic motivation are autonomy, mastery, and purpose. I rarely come across people and teams who can articulate their purpose.

View original post 476 more words

Testing != test execution

Jeff Fry on Testing

Testing includes:

  1. Building a useful mental model of the application under test, including what value it should provide and what risks could threaten that value,
  2. Designing powerful tests, using that model to investigate important questions about the application’s quality,
  3. Test execution (which might be automated entirely, partly, or not at all),
  4. Analyzing the results to determine if there’s a problem, how severe it might be, or otherwise answer stakeholder questions, and
  5. Test reporting that clearly communicates the impact of the bug, and how to quickly reproduce it.

We often talk about testing as if it’s only test execution, yet often the most interesting, challenging, skill-intensive aspects of testing are in creating a mental model that helps us understand the problem space, designing tests to quickly and effectively answer key questions, analyzing what specifically the problem is, and communicating it effectively.

View original post

Mobile App Test Coverage Model : LONG FUN CUP

Smart phones changed the way we interact with software. Once mobile apps were used to kill time but now there are apps those could save lives.

They are changing the way live, thus we need to change the way we test.

Smartphone applications are developed with immense creativity and effort. Mobile users demand a sleeker experience with applications compared to desktop users. The mind set of mobile users is very different from web or desktop users. Smartphone apps are used on the move (e.g.: while walking or using a toilet), and mobile devices have a lot to offer through hardware location tracking, gyroscopes and other integrated features.

LONG FUN CUP model helps to achieve coverage at UI  level. This is designed factoring in the new touch generation mobile devices hardware, software and users mindset.

LONG FUN CUP

Location: It’s a sin to test mobile app sitting at your desk, get out!

Location tracking is a key feature offered by mobile phones, which any app must make use of it based on its core functionality. For instance, if it’s a cab booking or food ordering app, app should be smart enough to track your current location and provided suggestions surrounding your locality. Unlike web apps, where you expect user to enter location details, here the work flow should be different. Never test such apps sitting at your desk. Check if the app asks for user permission before tracking the location, does app allow users revoke the permissions? Both Android and iOS provide options to mock your locations for testing the app using emulators and simulators.

Orientation: It’s a sin to test mobile app sitting at your desk, lie in the couch.

People can change the orientation for various reasons, any mobile app should be able to provide a consistent user experience across different orientations. Test all your screens, pop-ups, toast messages, forms in all supported orientations. There are instance where the filled in form data disappears when you change the orientation. Never test a mobile app just sitting in your desk, take it to rest rooms, lie down in a couch and use it while traveling and test if it gives the same user experience with all different orientation.

Network: It’s a sin to test mobile app sitting at your desk, switch networks.

Mobile devices supports both cellular and wi-fi, they can automatically switch between any available networks. How does your app behave when the device switches between networks? These are critical especially if your apps core purpose is to send and receive data to remote server like e-commerce, banking, broadcasting apps. Make sure all such functionality behavior is tested for all such varying network switching, network strengths.

Gestures: In mobile world, app responds to gestures not clicks.

Does your app supports all standard gestures? Is it consistent across the app? If it uses any new gestures, is it easy for the users to understand? I would recommend to play The Room game (a paid game available for both iOS and Android) to understand the unlimited potential of gestures.

Function:

Any function that defines or distinguishes the product or fulfills core requirements. Test for interactions, error handling, starting and closing of the app, file access, navigation, multimedia, and sync. Did you try tapping all GUI? Did you fill in all the input fields? Did you navigate to all the screens? Are you able to navigate back?

User Scenarios:

How easy or how hard is it to complete a task using the app? Is the first time user able to understand the work flow? Any scenario should have a story that is credible, motivating, complex and easy to evaluate. List possible users and list them the ways they might use the system to accomplish a task. Try to think about disfavored users and how they might try to exploit the app. Compare with competitor or web interface to get more real life scenarios. Refer: An Introduction to Scenario Testing

Notifications:

Notifications enable an application to inform its users that it has something for them. How does your application use notifications? How easy is to turn on or turn off your app from notifications? Does the application use local or push notifications? How does your application behave if device is in sleep mode? Does the app provide too many notifications? Test for all available types of visual notifications, sound and vibration.

Communication:

These days’ people use mobile phones for various purpose, but the primary purpose for voice calls. How does the app behave after interruptions by an incoming call or an SMS? Test for such interruptions from voice communication and all necessary functions of your app. Many times app will not be able to recover after attending a voice call for a longer duration.

Updates:

Unlike desktop mobile app gets frequent updates. Not only app updates but as a tester one has to be aware of OS updates as well. How the users are informed about updates? Does the app support the silent update? Track what changes or new features are available in the latest OS update. Analyse if the app needs any modification because of the OS updates.

Platform:

It is very critical for any mobile app tester to have a good understanding of the popular mobile platforms especially Apple and Android. Be a  fan-boy of either one of these platforms (it’s impossible to be a fan of both 😉 ) So, you know why Apple and Android does certain things in certain way. You need to be aware of the history and also the latest trends in the mobile platform. What tools are available for testing and what kind of test-ability layers are provided by the platform. Example Developer Options in Android, Instruments in Xcode.

Testers also should have good knowledge in App store Approval process, HIG, Android Design guidelines. This helps to take the app to the market very quick. It takes one to two weeks to get it approved by Apple App Store, so rejection wastes a lot of time. So, testers has to have a checklist handy to check those approval and guideline tests.

Testers should also should be aware of android fragmentation, you will never know in which device your app works and which it may fail. So, should keep an eye on the popular devices, new devices and the android OS market share to select an optimal device matrix to test.

HOLY COW!!!

HOLY COW!! It’s been long time since I blogged.

I was thinking for a while about what are the most important skills that help a tester to craft his own methods instead of following industry standards or best practices? This triggered while reading through one of the old posts of James Bach again recently,  especially this particular quote that grabbed my attention, “Following is for novices who are under active supervision. Instead, I craft methods on a project by project basis, and I encourage other people to do that..”

Heuristics

Extensive research is never possible in testing commercial software, due to time and budget constraints. So, how to test software quickly? Heuristics are used to speed up the process of finding an optimal solution. It’s something that helps you approach a problem and take decisions, which should be applied wisely. In order to write better on heuristics I wanted to know the Tamil (my mother tongue) meaning for heuristics. I decided to search in Google, from my knowledge and experience I believe Google would help me to get Tamil meaning of heuristics, but unfortunately it couldn’t get an answer for that. So, I decided to ask in Twitter addressing to few Tamil experts. This is heuristics, an approach to problem, but no guarantee that it solves problem.

Oracles

If heuristics help in approaching problems or take decision, Oracles help to identify problems. Some of  oracles used in testing are

1. Requirements

2. Comparing against competitors products

3. Consistency with in the products

L

“A tester is someone who knows that things can be different.” – Jerry Weinberg.

Y- Why?

“Questioning ignites thinking, leads to progress”  – Dhanasekar

Whenever I train novice testers, I provide them with few tasks, when they come back with a solution, I probe them with many questions like why was this done? why was this necessary? why do they think this is right solution? why not do it in a different way? Most times there won’t be concrete answers. My next step is to ask them to find answers for my questions. After finding answers  they would realise there were many more to learn and explore, but still many wouldn’t realise that those questions ignited their minds. The important thing is not to stop questioning. What we observe is not testing in itself, but testing exposed to our way of questioning.

Cognitive

Observation

If HOLY Sea(C) is preparation and practice for a battle, observation is the real battle. For instance, if your application sends emails, have you ever observed how it appears in the email alert message pop-ups in  many popular clients on windows platform or popular notifiers like Growl in Mac OS X? This example is out of an observation made on such email pop-up. This client we worked for, put their branding advertisements at the start of the email, so the alert pop-ups showed the first two lines of the content which is purely marketing text, but not important content. Because of which user might assume this to be spam and ignore it. This test idea would have never triggered if I missed to observe the alert pop-up. Making observations after executing tests is not end of testing, subsequent test ideas mostly result out of those observations. So, missing critical observation is missing potentially risky tests.