Last Friday, someone in the office came back from San Francisco with an IPAD!

Commotion followed!

The prized little gadget passed several hands in less than 2 minutes. But once it landed into the lap of a fellow QA, the testing fun begins!  Yousee we have an iPhone application in the middle of development, and what a cool way to see if the same iPhone apps will work with its big brother closely named iPad.  And no, I won’t tell you the results of of that test.

But one I thing I can tell you is this — holding an iPad for the first time is simply magical! Everything you touched moved so fast, even watching a simple YouTube video was never the same.  Surfing the web becomes a lot faster and much better!

So how does an iPad affect our job as software tester?

Here are my random buggy thoughts :

• The iPad is not a “FAD”. It is the best technological advancement in recent years, it is here to stay as it opens a new era in computing.  Technology will now be available to the average user, gone would be the days when only the geeks know how to save a file. With this in mind, we need to seriously consider usability testing.
• Accelerometer App testing. We will be testing from every angle ( portraits and landscapes) every rotation response time, without a mouse and keyboard !  It will be fun at first, but not sure how physically tiring this may be
• Memory usage and memory leaks. More apps and web pages compatible with iPad will be developed and tested. These will come in the form of digital books, games, digital newspapers, magazines, websites, email, videos, gadgets and every imaginable application that caters to every age group, gender or nationality. There are close to 200,000 apps in the Apple Store that will continue to grow exponentially with the global release of  iPad.
• Connectivity, performance, scalability and load testing would be crucial. Its portability means we can now be connected anytime, anywhere. This means faster loading web pages and apps. But how does your application scale up when the connectivity is slow or low.
• Compatibility testing. I am sure software companies would like to see their application working well with  regular desktop, laptop, netbook, other android phones, iPhone, iPod, Ipod Touch and iPad ! By the way, if you wanted to see how your website will look within iPad, you can go to this website http://ipadpeek.com/
• Security testing. We can purchase anything, anywhere as long as there is a connection. We can login into any secured site at our convenience.  Do banking,  trade stocks, pay bills etc while riding in the bus.  Securing these sites would be of great concern.

By the time you read this part, I am now convinced that testing an iPad application will really be not much different from any other product. There may be some new skills or tool sets needed, but the most important tool is always the mindset and attitude of the tester to accept challenges with enthusiasm. Almost anything can be learned if you are willing to give it a try.

Cheers!

Tess Rupprecht

Some links which you might want to check :

http://www.testiphone.com/
http://iphonetester.com/
http://furbo.org/2008/08/06/beta-testing-on-iphone-20/
http://code.google.com/p/google-toolbox-for-mac/wiki/iPhoneUnitTesting
http://www.devworld.apple.com/iphone/library/documentation/Xcode/Conceptual/iphone_development/135-Unit_Testing_Applications/unit_testing_applications.html

Having said this, I would like to share some practices I’ve learned through the years that I found effective in improving my testing skills. Who knows? They might work for you too

1. Read bugs logged by other testers.

By familiarizing yourself with known issues, you will be able to strategize your testing priorities.

2. Take the time to be aware of client issues.

By knowing client issues, you will be able to form more realistic testing scenarios.

3. Spend time doing your own research.

By getting more information about what you’re testing, you will be able to assess better how the product/feature should be tested.

4. Converse with other testers.

By talking to other testers, you will be able to get “fresh” ideas that will help add up to your testing coverage.

5. Think outside the box.

By not restricting yourself to what is written in any test case, you will be able to test more thoroughly and creatively.

If you have any questions or if you would like to share other tips please leave your comments below.

Cheers,

Gail Chua

I used to skimp on the documentation testing because I believe it detracts me from my “real” job, which is testing the program itself.

I am happy to admit that i was wrong!

Why is it important

• You will find more bugs than you expect. Surprisingly many bugs show up when a competent QA  THOROUGHLY checks the manual (or help guide) against the program. The writer looks at the program from a different angle than the developer and the QA, so the manuals will reveal different problems than the ones developers and QA look for.

• Heads up- Documentation testing doesnt always reveal sinificant new problems. QA who dont do a thorough job wont definately find many problems.

• It is an important source of real world combination test cases. One cant hope to test all the combinations of features or other options in the product, there are just too many. But one can test every combination that the manual describes as interesting and useful. Any time the manual even hints that the two aspects of the program work well together, test them together.

• Bug reports arising out of documentation testing are particularly credible. The manual is your company’s instruction to the customer on how to use the product. It is hard to dismiss a bug as “esoteric” when you report that the program failed while yo were following the manual’s instructions or suggestions, or checking one of its statements of fact. These are mainstream tets. These bugs are hard to defer- either the manual will change or the program will change.

Tips on testing the Manual

• Use the program exactly as the manual says. Enter every keystroke in every example. Customers make mistakes when they try to follow instructions. Feel free to make mistakes too.

• How does the computer respond? Bad error handling in the program will look worse when you how that it happens in response to an obvious, common mistake that several people will make when they try to follow the manual’s instructions.

• Try every suggestions, even if the suggestions are’t fully spelled out, step by step. Do what a reasonable customer would do who was trying to follow the suggestion.

• Check every statement of fact and every obvious inference from the stated facts, instructions, and suggestions. The manual is the product’s final specification, and the customer’s first place to check whether the program is working correctly.

It also pays to retest the documentation when you add a QA to the project. This keeps the manual current while the software is changing, and it educates new QA about the program.

Buggy Thoughts for Today

I strongly feel that  adding ATDD to TDD is a good testing combination. Given the right focus from all stakeholders, it can drive the entire development process to come up with quality software products whose codes are testable, reliable, stable and easy to maintain. I don’t claim to be an expert on any QA methodology but I try to justify my testing strategy by adequate research, seeking advice from the testing community (online and offline) complemented by my own experience. This will be a series of journey stories to find out if we can truly combine Test Driven Development ( TDD) and Acceptance Test Driven Development (ATDD) to come up with a successful Software Application Project using Agile.

Let the journey begins …

Journey 1 :  Thinking about  ATDD + TDD

Once upon a time, writing a Test Plan  seems an easy task if you are on the same project for the same company for a long time. You just fill up a simple template and you can come up with a completed Test Plan within a day or two.

But this simple one day activity can turn into weeks of planning when you are assigned to a new client ( or company) or a new project. Thou there may be a Test Plan template , it is not as simple as it seems. The area of the Test Plan which I found to be the hardest to come up with is the “Testing Approach and Strategy”. This is where the QA has to describe what is the best way  to test the application. Any QA with years of experience will tell you that there is no single one QA process that fits all. This is reality but misunderstood and even mocked. You need to give a serious look into a couple of factors that would affect your choice of testing strategy. This article is not about this issue so I will skip this part.

What I would want to talk about  is how I came up with a Test Strategy for one project :

• I started studying the business model and the application to be developed for about 3 days. I gather as much information from the PMs, BAs, other QAs. I read every existing artifacts.
• Then I worked with the Lead Developer alone for another 3 days, brainstorming which approach is do-able given all the resources, time and other known constraints we have. We both believed that we need to “pair” so that the testing approach we picked will be something that we both feel comfortable using. I would give much of the credit to him (SK,thank you ) for analyzing each of the testing activities I suggested and giving his technical opinion. ( Yes, I am truly lucky to work with very brilliant developers who put so much value to testing …)
• On the 7th day, I finalized the first draft of the plan with a high level Testing Approach and Strategy.

The high level strategy is like this (I copied this verbatim from my Test Plan)

• Our Team has agreed to use the Agile methodology where QA Analysts (QA) work closely with Developers, Business Analysts (BA), Project Managers (PM) and other stakeholders all throughout the project life cycle. We will implement Test Driven Development (TDD), continuous integration and pair programming.
• In addition to TDD, to ensure full unit test coverage for developed code, we will also follow the process of Acceptance Test Driven Development (ATDD) to ensure full test coverage. TDD’s approach is technology facing while ATDD is business facing. Acceptance tests are written at the beginning of an iteration and becomes an executable form of the requirements.
• Simple Process Flow of  ATDD + TDD

So why  ATDD + TDD?

• The Development Team has been using  Agile already so everyone is familiar with the iterative development life cycle.
• TDD is now a generally accepted practice so the “Test First” concept is not hard to sell to the developers.
• ATDD strategy will logically complement the PM’s practice of translating requirements into Epic Stories then User Stories.
• Our team has previously agreed that we will rely heavily on Story Acceptance Criteria to mark any task “DONE”. The Story Acceptance Criteria are converted into Automated Acceptance Tests. The story cannot be accepted as complete until the automated acceptance tests for that story pass.
• Other non technical stakeholders like the Business Analysts and the Customers are able to participate in writing automated acceptance tests minus the technical barriers. The write Tests as “Specs” in human readable language.
• The use of ATDD ensures expectations are set around testing. There is no disconnect between what the QAs think should be tested and what the Business think should be tested.
• As each automated acceptance tests are added to the continuous build, a fully automated regression suite is developed as part of the development process.
• The fast feedback mechanism is tremendously beneficial to everyone. The business has visibility into what is tested at all stages of development.

What is ATDD and other concepts

• Common names : Acceptance Test Driven Development, Story Driven Development, Behavior Driven Development, Example Driven Development
• Definition : “ATDD is an advanced extension of TDD, keeping the software’s development guided by the principles of satisfying the customer’s needs through acceptance criteria and automated testing. “ This is so far the best definition of ATDD (1)
• Format of a typical  TDD + ATDD acceptance tests:

BDD  (What is needed)

Story: [ DESCRIPTION OF STORY]
As a [ROLE]
I want [FEATURE]
So that [MOTIVATION]

ATDD  (How is the story supposed to work to arrive at the expected outcome)

Scenario: [SOME DESCRIPTION OF SCENARIO]
Given [SOME CONTEXT]
When [SOME EVENT OCCURS]
Then [ENSURE SOME DESIRED OUTCOME]

• ATDD allows us to take the customer’s stories and scenarios and add them directly into the code
• ATDD is business facing ( software that matters) while TDD is technology facing
• ATDD uses the testing direction of Outside-In
• Our software application now has more “business life” which can easily evolve and change to adapt to unknowns in the future requirements.

Challenges of ATDD + TDD to QA

I would avoid using Pros and Cons in evaluating the  ATDD + TDD approach. I’d rather combine them as “challenges” which we might face, but to which we can always find a good solution.

• Management support is essential so we need to be prepared to justify our recommendation. If management is not willing to invest on ATDD, you can still be creative. I have some thoughts on how to go about this but I would reserve that in another post.
• ATDD is somewhat a newer concept so this needs to be explained to the Team before implementation.
• It requires practice and discipline to write acceptance tests especially in a very short iteration cycle of 2 weeks or less.
• During the first few iterations, the team may suffer from poor velocity. So set your initial expectations realistically. As everyone starts getting used to writing acceptance criteria and acceptance tests, the velocity for each story should also improve.
• More tests for the developers ( and QA ) to implement and maintain.
• Automation is a must. ATDD is not as successful if it is manually implemented.
• Lack of automation tools that would fit some applications under test. The most commonly used open source tools are FIT, FITNESSE, Cucumber, NBehave, StoryQ and EasyB. For this project, I have narrowed down the choices to : NBehave and StoryQ . I will let you know in another post which tool I picked.
• Again, the selection of the automation tool is crucial . One team I know had initial success using EasyB for almost one year. Eventually, the maintenance and the load it created on build  time seemed to  offset some of the positive returns earlier. I have not used EasyB myself, so please don’t take this statement as a bad review for this free tool.
• I highly suggest that the QA works closely with the developer during the testing tool selection. Try a couple, but don’t fall in love with the initial results. Think ahead. Consider refactoring, consider integration with other project streams. Consider the maintenance overhead.

Next, please follow our journey to the bottom of ATDD + TDD

• Journey 2 : Presenting  ATDD + TDD to the entire Project Team
• Journey 3 : First Iteration Implementation of  ATDD + TDD

Thanks everyone!

Tess Rupprecht

——————

References

(1) http://testdrivendeveloper.com/2008/03/14/ATDDAcceptanceTestDrivenDevelopment.aspx

While at a client’s site, I was thrown into a schedule of endless agile meetings! Daily stand-up meetings, sit-down meetings, lunch meetings (Seriously, these meetings do really work!)

Lots of productive meetings. This is a good sign. This is one of the reasons why Agile is becoming  “the way to go“  in software development and implementation. More open communication leads to better understanding of the requirements of the product , fast resolutions of the issues resulting to more energized commitment to work as a team.

But one thing I love with Agile is how every member has embraced the idea of “less documentation in words” but more of “documentation in visuals, graphics and drawings“.  This made the white board, sketch pads, post it, markers the most loved  tools in every meeting.  Since I am very visual person, I prefer the graphical representation of stories, process, methods ( or anything you can translate to drawings).
 

 
So what has camera got to do with this post?

Since I find it hard to listen and write at the same time, I found that bringing a digital camera  in any Agile meeting is also “the way to go..”

Here’s what I do :

• I take snapshots of the whiteboard or story board
• Print the photos for easy viewing
• Upload the photos in the Project Management software

Advantages of using camera:

• Since I started this new meeting habit, I found myself more productive.
• I can participate intently in all the discussions first, then capture the diagrams later.
• There is 100% accuracy in the details compared to copying the same by hand.
• Indeed, a picture is worth a thousand words.
• And need I say, it is also more fun !!!

From that day on, my QA Tool Kit won’t be complete without a small, simple, reliable digital camera.

Don’t take my word for it, just try it !  Cheers!

Tess Rupprecht