Glossary: QA and Software Testing

Acceptance Testing: Formal testing conducted to determine whether or not a system satisfies its acceptance criteria—enables an end user to determine whether or not to accept the system.
Alpha Testing: Testing of a software product or system conducted at the developer’s site by the end user.
Ad Hoc Testing: A testing phase where the tester tries to 'break' the system by randomly trying the system's functionality. Can include negative testing as well.
Agile Testing: Testing practice which emphasize on a test-first design paradigm.
Automated Testing: That part of software testing that is assisted with software tool(s) that does not require operator input, analysis, or evaluation.
Beta Testing: Testing conducted at one or more end user sites by the end user of a delivered software product or system.
Basis Path Testing: A white box test case design technique that uses the algorithmic flow of the program to design tests.
Black Box Testing: Functional testing based on requirements with no knowledge of the internal program structure or data. Also known as closed-box testing. Black Box testing indicates whether or not a program meets required specifications by spotting faults of omission -- places where the specification is not fulfilled.
Bottom-up Testing: An integration testing technique that tests the low-level components first using test drivers for those components that have not yet been developed to call the low-level components for test.
Boundary Testing: Testing that focuses on the boundary or limit conditions of the software being tested. Stress Testing can also be considered as form of boundary testing.
Boundary Value Analysis: A test data selection technique in which values are chosen to lie along data extremes. Boundary values include maximum, mini-mum, just inside/outside boundaries, typical values, and error values.
Branch Coverage Testing: A test method satisfying coverage criteria that requires each decision point at each possible branch to be executed atleast once.
Breadth Testing: A test suite that exercises the full functionality of a product but does not test features in detail. Its in a way is used when there is no enough time to execute all the test cases
Bug: A design or implementation flaw that will result in symptoms exhibited by some module when module is subjected to an appropriate test.
Code Complete: Phase of development where functionality is implemented in entirety. Bug fixes are all that are left. All functions
found in the functional Specifications have been implemented. Code complete module may be far from release as it may have many bugs
Code Coverage: An analysis method that determines which parts of the software/code have been executed (covered) by the test case suite and which parts have not been executed and therefore may require additional attention.
Code Inspection: A formal testing technique where the programmer reviews source code with a group who ask questions analyzing the program logic, analyzing the code with respect to a checklist of historically common programming errors, and analyzing its compliance with coding standards.
Concurrency Testing: Multi-user testing geared toward determining the effects of accessing the same application code, module or database records. Identifies and measures the level of locking, deadlocking and use of single-threaded code and locking semaphores. This is one area where the cause for many bugs which were considered random can be identified.
Conformance Testing: The process of testing that an implementation conforms to the specification on which it is based. Usually applied to testing conformance to a formal standard.
Debugging: The process of finding and removing the causes of software failures. Tools used for debugging are called debuggers.
End-to-End testing: Testing a complete application environment in a situation that mimics real-world use, such as interacting with a database, using network communications, or interacting with other hardware,applications, or systems if appropriate.
Exhaustive Testing: Testing which covers all combinationsof input values and preconditions for an element of the software under test. This is practically infeasible.
Failure: The inability of a system or system component to perform a required function within specified limits. A failure may be produced when a fault is encountered.
Fault: A manifestation of an error in software. A fault, if encountered, may cause a failure.
Fault-based Testing: Testing that employs a test data selection strategy designed to generate test data capable of demonstrating the absence of a set of pre-specified faults, typically, frequently occurring faults.
Function Points: A consistent measure of software size based on user requirements. Data components include inputs, outputs, etc. Environment characteristics include data communications, performance, reusability, operational ease, etc. Weight scale: 0 = not present, 1 =minor influence, 5 = strong influence.
Functional Testing: Application of test data derived from the specified functional requirements without regard to the final program structure. Also known as black box testing.
Gorilla Testing: Testing one particular module, functionality heavily.
Gray Box Testing: A combination of Black Box and WhiteBox testing methodologies: testing a piece of software against its specification but using some knowledge of its internal workings.
Heuristics Testing: Another term for failure-directed testing.
Incremental Analysis: Incremental analysis occurs when (partial) analysis may be performed on an incomplete product to allow early feedback on the development of that product.
Infeasible Path: Program statement sequence that can never be executed. i,e the unreachable code
Inspection: A formal evaluation technique in which software requirements, design, or code are examined in detail by a person or group other than the author to detect faults, violations of development standards, and other problems. Its generic term for all inspections similar to code inspections.
Integration Testing: An orderly progression of testing in which software components or hardware components, or both, are combined and tested until the entire system has been integrated.
Intrusive Testing: Testing that collects timing and processing information during program execution that may change the behavior of the software from its behavior in a real environment. Usually involves additional code embedded in the software being tested or additional processes running concurrently with software being tested on the same platform.
Installation Testing: Confirms that the application under test recovers from expected or unexpected events without loss of data or functionality. Events can include shortage of disk space, unexpected loss of communication, or power out conditions.
IV&V: Independent verification and validation is the verification and validation of a software product by an organization that is both technically and managerially separate from the organization responsible for developing the product.
Life Cycle: The period that starts when a software product is conceived and ends when the product is no longer available for use. The software life cycle typically includes a requirements phase, design phase, implementation (code) phase, test phase, installation and checkout phase, operation and maintenance phase, and a retirement phase.
Localization Testing: This term refers to making software specifically designed for a specific locality.
Loop Testing: A white box testing technique that exercises program loops.
Manual Testing: That part of software testing that requires operator input, analysis, or evaluation.
Monkey Testing: Testing a system or an Application on the fly, i.e. just few tests here and there to ensure the system or an application does not crash out. Its a form of adhoc testing.
Mutation Testing: A method to determine test set thoroughness by measuring the extent to which a test set can discriminate the program from slight variants of the program.
Non-intrusive Testing: Testing that is transparent to the software under test; i.e., testing that does not change the timing or processing characteristics of the software under test from its behavior in a real environment. Usually involves additional hardware that collects timing or processing information and processes that information on another platform.
Negative Testing: Testing aimed at showing software does not work. Also known as "test to fail".
Path Analysis: Program analysis performed to identify all possible paths through a program, to detect incomplete paths, or to discover portions of the program that are not on any path.
Path Coverage Testing: A test method satisfying coverage criteria that each logical path through the program is tested. Paths through the program often are grouped into a finite set of classes; one path from each class is tested.
Peer Reviews: A methodical examination of software work products by the producer’s peers to identify defects and areas where changes are needed.
Path Testing: Testing wherein all paths in the program source code are tested at least once.
Performance Testing: Testing conducted to evaluate the compliance of a system or component with specified performance requirements. Often this is performed using an automated test tool to simulate large number of users. Also know as "Load Testing".
Positive Testing: Testing aimed at showing software works. Also known as "test to pass".
Proof Checker: A program that checks formal proofs of program properties for logical correctness.
Qualification Testing: Formal testing, usually conducted by the developer for the end user, to demonstrate that the software meets its specified requirements.
Random Testing: An essentially black-box testing approach in which a program is tested by randomly choosing a subset of all possible input values. The distribution may be arbitrary or may attempt to accurately reflect the distribution of inputs in the application environment.
Regression Testing: Selective retesting to detect faults introduced during modification of a system or system component, to verify that modifications have not caused unintended adverse effects, or to verify that a modified system or system component still meets its specified requirements.
Ramp Testing: Continuously raising an input signal until the system breaks down.Form of stress testing
Recovery Testing: Confirms that the program recovers from expected or unexpected events without loss of data or functionality. Events can include shortage of disk space, unexpected loss of communication, or power out conditions.
Reliability: The probability of failure-free operation for a specified period.
Run Chart: A graph of data points in chronological order used to illustrate trends or cycles of the characteristic being measured for the purpose of suggesting an assignable cause rather than random variation.
Statement Coverage Testing: A test method satisfying coverage criteria that requires each statement be executed at least once.
Static Testing: Verification performed without executing the system’s code. Also called static analysis.
Statistical Process Control: The use of statistical techniques and tools to measure an ongoing process for change or stability.
Structural Coverage: This requires that each pair of module invocations be executed at least once.
Structural Testing: A testing method where the test data is derivedsolely from the program structure.
Sanity Testing: Brief test of major functional elements of a piece of software to determine if it is basically operational.
Scalability Testing: Performance testing focused on ensuring the application under test gracefully handles increases in workload.
Security Testing: Testing which confirms that the program can restrict access to authorized personnel and that the authorized personnel can access the functions available to their security level.
Smoke Testing: A quick-and-dirty test that the major functions of a piece of software work. Originated in the hardware testing practice of turning on a new piece of hardware for the first time and considering it a success if it does not catch on fire.
Soak Testing: Running a system at high load for a prolonged period of time. For example, running several times more transactions in an entire day (or night) than would be expected in a busy day, to identify and performance problems that appear after a large number of transactions have been executed.
Software Requirements Specification: A deliverable that describes all data, functional and behavioral requirements, all constraints, and all validation requirements for software.
Software Testing: A set of activities conducted with the intent of finding errors in software.
Static Analysis: Analysis of a program carried out without executing the program.
Static Analyzer: A tool that carries out static analysis.
Static Testing: Analysis of a program carried out without executing the program.
Storage Testing: Testing that verifies the program under test stores data files in the correct directories and that it reserves sufficient space to prevent unexpected termination resulting from lack of space. This is external storage as opposed to internal storage.
Stress Testing: Testing conducted to evaluate a system or component at or beyond the limits of its specified requirements to determine the load under which it fails and how. Often this is performance testing using a very high level of simulated load.
Structural Testing: Testing based on an analysis of internal workings and structure of a piece of software.
System Testing: Testing that attempts to discover defects that are properties of the entire system rather than of its individual components.
Test Bed: 1) An environment that contains the integral hardware, instrumentation, simulators, software tools, and other support elements needed to conduct a test of a logically or physically separate component.2) A suite of test programs used in conducting the test of a component or system.
Test Development: The development of anything required to conduct testing. This may include test requirements (objectives), strategies, processes, plans, software, procedures, cases, documentation, etc.
Test Harness: A software tool that enables the testing of softwarecomponents that links test capabilities to perform specific tests, accept program inputs, simulate missing components, compare actual outputs with expected outputs to determine correctness, and report discrepancies.
Testability: The degree to which a system or component facilitates the establishment of test criteria and the performance of tests to determine whether those criteria have been met.
Testing: The process of exercising software to verify that it satisfies specified requirements and to detect errors. The process of analyzing a software item to detect the differences between existing and required conditions (that is, bugs), and to evaluate the features of the software item (Ref. IEEE Std 829). The process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component.
Test Case: Test Case is a commonly used term for a specific test. This is usually the smallest unit of testing. A Test Case will consist of information such as requirements testing, test steps, verification steps, prerequisites, outputs, test environment, etc. A set of inputs, execution preconditions, and expected outcomes developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.
Test Driven Development: Testing methodology associated with Agile Programming in which every chunk of code is covered by unit tests, which must all pass all the time, in an effort to eliminate unit-level and regression bugs during development. Practitioners of TDD write a lot of tests, i.e. an equal number of lines of test code to the size of the production code.
Test Driver: A program or test tool used to execute tests. Also known as a Test Harness.
Test Environment: The hardware and software environment in which tests will be run, and any other software with which the software under test interacts when under test including stubs and test drivers.
Test First Design: Test-first design is one of the mandatory practices of Extreme Programming (XP). It requires that programmers do not write any production code until they have first written a unit test.
Test Plan: A document describing the scope, approach, resources, and schedule of intended testing activities. It identifies test items, the features to be tested, the testing tasks, who will do each task, and any risks requiring contingency planning. Ref IEEE Std 829.
Test Procedure: A document providing detailed instructions for the execution of one or more test cases.
Test Script: Commonly used to refer to the instructions for a particular test that will be carried out by an automated test tool.
Test Specification: A document specifying the test approach for a software feature or combination or features and the inputs, predicted results and execution conditions for the associated tests.
Test Suite: A collection of tests used to validate the behavior of a product. The scope of a Test Suite varies from organization to organization. There may be several Test Suites for a particular product for example. In most cases however a Test Suite is a high level concept, grouping together hundreds or thousands of tests related by what they are intended to test.
Test Tools: Computer programs used in the testing of a system, a component of the system, or its documentation.
Thread Testing: A variation of top-down testing where the progressive integration of components follows the implementation of subsets of the requirements, as opposed to the integration of components by successively lower levels.
Top Down Testing: An approach to integration testing where the component at the top of the component hierarchy is tested first, with lower level components being simulated by stubs. Tested components are then used to test lower level components. The process is repeated until the lowest level components have been tested.
Total Quality Management: A company commitment to develop a process that achieves high quality product and customer satisfaction.
Traceability Matrix: A document showing the relationship between Test Requirements and Test Cases.
Test Objective: An identified set of software features to be measured under specified conditions by comparing actual behavior with the required behavior described in the software documentation.
Unit Testing: The testing done to show whether a unit (the smallest piece of software that can be independently compiled or assembled, loaded, and tested) satisfies its functional specification or its implemented structure matches the intended design structure.
Usability Testing: Testing the ease with which users can learn and use a product.
Use Case: The specification of tests that are conducted from the end-user perspective. Use cases tend to focus on operating software as an end-user would conduct their day-to-day activities.
V- Diagram (model): a diagram that visualizes the orderof testing activities and their corresponding phases of development
Verification: The process of determining whether or not the products of a given phase of the software development cycle meet the implementation steps and can be traced to the incoming objectives established during the previous phase. The techniques for verification are testing, inspection and reviewing.
Volume Testing: Testing which confirms that any values that may become large over time (such as accumulated counts, logs, and data files),can be accommodated by the program and will not cause the program to stop working or degrade its operation in any manner.
Validation: The process of evaluating software to determine compliance with specified requirements.
Walkthrough: Usually, a step-by-step simulation of the execution of a procedure, as when walking through code, line by line, with an imagined set of inputs. The term has been extended to the review of material that is not procedural, such as data descriptions, reference manuals, specifications, etc.
White-box Testing: Testing approaches that examine the program structure and derive test data from the program logic. This is also known as clear box testing, glass-box or open-box testing. White box testing determines if program-code structure and logic is faulty. The test is accurate only if the tester knows what the program is supposed to do. He or she can then see if the program diverges from its intended goal. White box testing does not account for errors caused by omission, and all visible code must also be readable.
Workflow Testing: Scripted end-to-end testing which duplicates specific workflows, which are expected to be utilized by the end-user.

Software Testing in Startup Companies

What makes the software testing in startup companies more difficult and challenging :

- Very low tester to developer ratio: Many startups does not like to invest much in the testing initially and hence there will be very few testers.
- No Proper Processes in Place: There will be no process and all the things will be happening in adhoc manner.
- Outsourcing the Testing: Since the startups do not invest much into the QA initially, they will go for outsourcing just before the release, but with no proper documentation it becomes difficult to handle such teams.
- New Technology : Since most startups comes up with innovative products, test cases also needs to be innovative. Testers needs to think out of box and come up with innovative ideas rather than using the age old stuff.
- Product Launch: For startups the first product is always important. If there are any major bugs than the company has to face the ire of bloggers and media.
- Hard Deadlines: Sometimes QA may end up with just one day to test because of hard deadlines and the developers working till the last day.

What software testing types can be considered?

Black box testing – This type of testing doesn’t require any knowledge of the internal design or coding. These Tests are based on the requirements and functionality.

White box testing – This kind of testing is based on the knowledge of internal logic of a particular application code. The Testing is done based on the coverage of code statements, paths, conditions.

Unit testing – the 'micro' scale of testing; this is mostly used to test the particular functions or code modules. This is typically done by the programmer and not by testers; it requires detailed knowledge of the internal program design and code. It cannot be done easily unless the application has a well-designed architecture with tight code; this type may require developing test driver modules or test harnesses.

Sanity testing or Smoke testing – This type of testing is done initially to determine if a new software version is performing well enough to accept it for a major testing effort. For example, if the new software is crashing the systems in every 5 minutes or corrupting databases, the software may not be in a 'sound’ condition to proceed for further testing in its current state.

Functional testing – This a commonly used black-box testing geared to check the functional requirements of an application; this type of testing should be done by testers.

Integration testing – This testing is combining the ‘parts’ of an application to determine if they function together correctly. The 'parts' can be code modules, individual applications, client and server applications on a network, etc. This type of testing is especially relevant to the client/server and distributed systems.

Incremental Integration testing – This is continuous testing of an application when a new functionality is added the existing ones; it checks the application functionality by verifying whether it works separately before all parts of the program are completed, in this type it will be checked whether to introduce test drivers or not; this is done by programmers or by testers.

Regression testing – This is testing the whole application again after the fixes or the modifications are done on the software. This is mostly done at the end of the Software development life cycle. Mostly Automated testing tools are used for these type of testing.

System testing – This is a type of black-box type testing that is based on overall requirements specifications; covers all combined parts of a system.

End-to-end testing – This is similar to system testing; this involves testing of a complete application environment such as interacting with a database, using network communications, or interacting with other hardware, applications and so on.

UAT ( User Acceptance Testing ) – This type of testing comes on the final stage and mostly done on the specifications of the end-user or client.

Usability testing – This testing is done to check the 'user-friendliness' of the application. This depends on the targeted end-user or customer. User interviews, surveys, video recording of user sessions, and other techniques can be used. Programmers and testers are usually not appropriate as usability testers.

Compatibility testing – Testing how well the software performs in a particular hardware, software, operating system, network etc.

Comparison testing – This is nothing comparing the software strengths and weakness with another competing product.

Mutation testing – This is another method for determining if a set of test data or test cases is useful, by purposely introducing various code changes or bugs and retesting with the original test data or cases to determine whether the 'bugs' are detected.

Basics of Automation Testing - Part II

... continued from the Basics of Automated Testing - Part I

Automated test can be broken into two big pieces:
· Running the automated tests
· Validating the results.

Running the Automated Tests

This concept is pretty basic, if you want to test the submit button a login page, you can override the system and programmatically move the mouse to a set of screen coordinates, then send a click event. There is another much trickier way to do this. You can directly call the internal API that the button click event handler calls. Calling into the API is good because it's easy. Calling an API function from your test code is a piece of cake, just add in a function call. But then you aren't actually testing the UI of your application. Sure, you can call the API for functionality testing, then every now and then click the button manually to be sure the right dialog opens.

Rationally this really should work great, but a lot of testing exists outside the rational space. There might be lots of bugs that happen when the user goes through the button instead of directly calling the API. And here's the critical part – almost all of your users will use your software through the UI, not the API. So those bugs you miss by just going through the API will be high exposure bugs. These won't happen all the time, but they're the kind of things you really don't want to miss, especially if you were counting on your automation to be testing that part of the program.

If your automation is through the API, then this way you're getting no testing coverage on your UI. And you'll have to do that by hand.

Simulating the mouse is good because it's working the UI the whole time, but it has its own set of problems. The real issue here is reliability. You have to know the coordinates that you're trying to click before hand. This is doable, but lots of things can make those coordinated change at runtime. Is the window maximized? What's the screen resolution? Is the start menu on the bottom or the left side of the screen? Did the last guy rearrange the toolbars? And what suppose the application is used by users of Arabic language, where the display is from right to left? These are all things that will change the absolute location of your UI.

The good news is there are tricks around a lot of these issues. The first key is to always run at the same screen resolution on all your automated test systems (note: there are bugs we could be missing here, but we won't worry about that now - those are beyond the scope of our automation anyway.) We also like to have our first automated test action by maximizing the program. This takes care of most of the big issues, but small things can still come up.

The really sophisticated way to handle this is to use relative positioning. If your developers are nice they can build in some test hooks for you so you can ask the application where it is. This even works for child windows, you can ask a toolbar where it is. If you know that the 'file -> new' button is always at (x, y) inside the main toolbar it doesn't matter if the application is maximized or if the last user moved all the toolbars around. Just ask the main toolbar where it is and tack on (x, y) and then click there.

So this has an advantage over just exercising the APIs since you're using the UI too, but it has a disadvantage too – it involves a lot of work.

Results Verification

So we have figured out the right way to run the tests, and we have this great test case, but after we have told the program to do stuff, we need to have a way to know if it did the right thing. This is the verification step in our automation, and every automated script needs this.

We have many options
· Verify the results manually
· Verify the results programmatically
· Use some kind of visual comparison tool.

The first method is to do it ourselves, that is by manually verifying the results and see that it meets our expectations.

The second way is of course to verify it programmatically. In this method, we can have a predefined set of expected results(baseline), which can be compared with the obtained results. The output of this would be whether a test case passed or failed. There are many ways by which we can achieve this; we can hard code the expected results in the program/script. We can also store the expected the results in a particular file or a text file or a properties file or a xml file and read the expected results from this file and compare with the obtained result.

The other way is to just grab a bitmap of the screen and save it off somewhere. Then we can use a visual comparison tool to compare it with the expected bitmap files. Using a visual comparison tool is clearly the best, but also the hardest. Here your automated test gets to a place where it wants to check the state, looks at the screen, and compares that image to a master it has stored away some where.

Basics of Automation Testing - Part I

Test automation is the use of Software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions. Over the past few years, tools that help programmers quickly create applications with graphical user interfaces have dramatically improved programmer productivity.

This has increased the pressure on testers, who are often perceived as bottlenecks to the delivery of software products. Testers are being asked to test more and more code in less and less time. Test automation is one way to do this, as manual testing is time consuming.

As and when different versions of a software are released, the new features will have to be tested manually time and again. But, now there are tools available that help the testers in the automation of the GUI which reduce the test time as well as the cost, other test automation tools support execution of performance tests.

Many test automation tools provide record and playback features that allow users to record interactively user actions and replay it back any number of times, comparing actual results to those expected.

Test Automation is an important subject since it's the most important part of any tester’s job. It's clearly not the best use of time to click on the same button looking for the same dialog box(expected result) every single day. Part of smart testing is delegating those kinds of tasks away so we can spend time on harder problems. And computers are a great place to delegate repetitive work.

That's really what automated testing is about. We try to get computers to do our job for us. One of the ways a tester describes his goal is to put himself out of a job - meaning that automate the entire job. This is, of course, unachievable so we don't worry about losing our jobs. But it's a good vision!

Our short term goal should always be to automate the parts of our job we find most annoying with the selfish idea of not having to do annoying stuff any more!!!

With people new to automated testing, that's always how we frame it. Start small, pick an easy task that you have to do all the time. Then figure out how to have a computer do it for you. This has a great effect on your work since after you get rid of the first one that will free up more of your time to automate more and more annoying, repetitive tasks. Now with all this time you can go and focus on testing more interesting parts of your software.

That last paragraph makes it sound like writing automated tests is easy, when in fact it's typically quite hard!!!

There are some fundamentally hard problems in this space. There are a lot of test tools which try to help out with these problems in different ways. Hopefully it will be valuable as a way to better understand automated testing and as a way to help choose your test tools. As a side note, implementing automated tests for a text based or API based system is really pretty easy; Let us focus on a full UI application - which is where the interesting issues are.

Automated test can be broken into two big pieces:
· Running the automated tests.
· Validating the results.

... to be continued