In uncategorized
The black magic of the "black box" technique.
Not so long ago ago we talked about how to test using boxes of different colors… No, wait. We talked about how to test using "black box", "white box" and gray box" methods. Oh, for God's sake! Black magic it is.
Seems that we need to study this issue in more detail. Let's look inside the "black box" and find out where and when it is used. It would be nice to learn more about its strengths and weaknesses.
The so-called "black-box testing" is a software testing method when the internal structure, design, and implementation of a product remains unknown to the tester. This means that when preparing test cases, the tester relies on requirements and specifications. Despite the fact that the requirements and specifications do not always exist in writing, when testing by the "black box" method, we can rely on the verbally described requirements.
So what is a "black box" according to the ISTQB terminology?
"Black box" testing is functional and non-functional testing performed without access to the internal structure of the system components. The "black box" test method is the procedure for obtaining and selecting test cases based on the analysis of the specification (functional or non-functional), components or system without reference to their internal structure.
When is the "black box" method used?
When is the "black box" method used?
A type of testing in which software and hardware components are combined and tested to evaluate the interaction between them. When using the "black box" method, the tester checks whether all components work correctly when they are integrated into a large system. After all, the normal work of each component separately cannot guarantee that they will work together throughout the project. For example, the data may not go through the interface, or the interface will not work according to the documentation. When planning such tests, experts rely on specification.
Using this method, the tester checks whether the software fulfills all the stated functions and customer requirements in full according to the documentation.
Let's say that we have an online gaming platform. Its documentation claims the possibility of simultaneous registration of 1000 users. In this case, stress testing will be a continuous stream of automated registrations (at least 1000 registrations per minute) for 12 hours.
Suppose there is an "invitation" functionality at the platform mentioned above. We need to check how much time it takes for a user to register by invitation, and also see if the system processes such requests correctly.
Using this type of testing, we can check if there are memory leaks, how fast the system is running and giving feedback if the software consumes too much traffic or creates an excessive number of connections.
After the software has been checked by testers, it is given to the customer. The customer runs "black box" acceptance tests based on expectations of functionality. As a rule, the customer determines the test mix individually. The customer also has the right to refuse acceptance (if the test results were unsatisfactory).
Held throughout the development cycle. The purpose of such testing is to check the performance of the new code and find out if it has led to bugs or breakdowns in the old functionality.
When choosing a regression test mix, the following guidelines should be used:
It is worth noting that regression testing is not always carried out only by the "black box" method. The "white box" method is also used, especially when searching for functions that are likely to be affected by changes.
When choosing a regression test mix, the following guidelines should be used:
- First, we make a special mix of tests, concerning all functions of the software.
- Then we select tests that focus on software components and functions that have been modified.
- We also need to use additional test cases and focus on the functions that are most likely to be affected by the changes.
It is worth noting that regression testing is not always carried out only by the "black box" method. The "white box" method is also used, especially when searching for functions that are likely to be affected by changes.
This testing is also carried out by the "black box" method. Almost ready software is given to those who wish to use it. This is done to identify the maximum number of errors before the software reaches the end customer.
It allows:
It allows:
- To detect unexpected errors (because beta testers think "outside the box").
- To create a wide range of testing environments, which is difficult to provide by using other methods (different operating systems, different settings, different versions of browsers).
- To reduce costs (as a rule, beta testers are not paid).
"Black box" testing techniques
Equivalence partitioning
This technique involves splitting input values into valid and invalid partitions and selecting representative values from each section as test data. It can be used to reduce the number of test cases. Suppose we have an integer variable N in the range from -99 to 99. So the positive equivalence classes are [-99, -10], [-9, -1], 0, [1, 9], [10, 99], and invalid (negative) classes are <-99,> 99, null and non-numeric strings.
Boundary value analysis
A technique that includes defining the boundaries of the input values and selecting as test data the values that are on the boundaries, and also inside and outside the boundaries. Many systems tend to behave incorrectly when reaching boundary values, therefore, the estimation of the application values boundaries is very important. When testing, use the following values: minimum, (minimum-1), maximum, (maximum + 1), and standard values. For example, in the case of -99 <= N <= 99, the set will be as follows: -100, -99, -98, -10, -9 -1, 0, 1, 9, 10, 98, 99, 100.
Transition table testing
Test scenarios are selected based on whether the state is changed correctly or incorrectly. Suppose we want to buy a movie ticket and book our seat. We choose the order form, select a seat and click the "Book" button. The seat we have chosen immediately becomes unavailable for another record since the first record led to a change in the database.
User scenario testing
Individual user scenarios are written in order to test whether the system functions correctly when processing one or another set of user actions.
This technique involves splitting input values into valid and invalid partitions and selecting representative values from each section as test data. It can be used to reduce the number of test cases. Suppose we have an integer variable N in the range from -99 to 99. So the positive equivalence classes are [-99, -10], [-9, -1], 0, [1, 9], [10, 99], and invalid (negative) classes are <-99,> 99, null and non-numeric strings.
Boundary value analysis
A technique that includes defining the boundaries of the input values and selecting as test data the values that are on the boundaries, and also inside and outside the boundaries. Many systems tend to behave incorrectly when reaching boundary values, therefore, the estimation of the application values boundaries is very important. When testing, use the following values: minimum, (minimum-1), maximum, (maximum + 1), and standard values. For example, in the case of -99 <= N <= 99, the set will be as follows: -100, -99, -98, -10, -9 -1, 0, 1, 9, 10, 98, 99, 100.
Transition table testing
Test scenarios are selected based on whether the state is changed correctly or incorrectly. Suppose we want to buy a movie ticket and book our seat. We choose the order form, select a seat and click the "Book" button. The seat we have chosen immediately becomes unavailable for another record since the first record led to a change in the database.
User scenario testing
Individual user scenarios are written in order to test whether the system functions correctly when processing one or another set of user actions.
"Black box" strengths
- Testing by the "black box" method allows you to find bugs that cannot be detected by the "white box" method. For example, the developer simply forgot to add some functionality (a perfect opening of a nightmare, isn't it?). Well, in this case, the code goes smoothly, but missing functionality is a true disaster from the point of view of the specification.
- "Black box" allows you to quickly identify bugs in the functional specifications (they describe not only the input values but also those that we have to get). If the testing result differs from what is stated in the specification, then we have a reason to chat with the analytics.
- For running the "black box" test we don't need additional qualifications. Often we use different services and applications, without really understanding them. In order to open Instagram and process a photo, we don't need to know how to implement filters. We want to open a photo, select a filter and get a perfect picture on the output. In this case, the tester needs to make sure that the user receives this most beautiful picture in accordance with the selected filter.
- Testing is running "from the user's position." The user always comes first, which means that our service should be comfortable, simple and clear.
- You can write test cases immediately after the specification is ready. This significantly reduces testing time. By the time the product is ready for testing, test cases have already been developed, and the tester can start testing immediately.
- Testing by the "black box" method allows you to find bugs that cannot be detected by the "white box" method. For example, the developer simply forgot to add some functionality (a perfect opening of a nightmare, isn't it?). Well, in this case, the code goes smoothly, but missing functionality is a true disaster from the point of view of the specification.
- "Black box" allows you to quickly identify bugs in the functional specifications (they describe not only the input values but also those that we have to get). If the testing result differs from what is stated in the specification, then we have a reason to chat with the analytics.
- For running the "black box" test we don't need additional qualifications. Often we use different services and applications, without really understanding them. In order to open Instagram and process a photo, we don't need to know how to implement filters. We want to open a photo, select a filter and get a perfect picture on the output. In this case, the tester needs to make sure that the user receives this most beautiful picture in accordance with the selected filter.
- Testing is running "from the user's position." The user always comes first, which means that our service should be comfortable, simple and clear.
- You can write test cases immediately after the specification is ready. This significantly reduces testing time. By the time the product is ready for testing, test cases have already been developed, and the tester can start testing immediately.
"Black box" weaknesses
- The main disadvantage of the black box method is that the tester can skip boundaries and transitions that are not obvious from the specification but are implemented in the code. Actually, this makes testers use the "white box" method. Here is an example. The system receives Forex currency quotes and corrects them up to 3 decimal. The system was successfully tested by the "black box" method (since no currency went beyond the respective boundaries) and performed well until the dollar / bitcoin exchange rate went beyond $ 1,000. Testing with a "white box" would reveal a bug that the currency conversion rate is limited to 3 characters.
- You can test only a small number of possible input (input) values, so many options remain unchecked.
- Tests may be needless if the developer has already tested this functionality (for example, with the help of a unit-test).
- In the absence of a clear and complete specification, it is difficult to design tests and test scenarios.
Let's sum up
The "black box" method is effective in various types of testing; but it should be remembered that some bugs cannot be found using only this method (for example, bugs in the internal structure of the code).
Carrying out the "black-box" testing helps to find out that the application does an excellent job with a large range of input data, as the test cases depend on the specification only.
Carrying out the "black-box" testing helps to find out that the application does an excellent job with a large range of input data, as the test cases depend on the specification only.
The "black box" method is beneficial if you are looking for:
- incorrectly implemented functions of the application or service
- user interface bugs
- bugs in functional specifications
To implement the most complete test, we recommend using the methods of "black" and "white" boxes simultaneously.
This will increase the coverage of possible scenarios, reduce the risk of bug missing, and improve the test results since the system will be tested using two different approaches (looking outside and inside the "box").
Stay tuned and level up! We know that testing is not magic at all, but hard study and hard work
We love what we do and do it with pleasure.
We love what we do and do it with pleasure.
QAcamp team
You may also like: