It was a bright sunny morning when I read the unbelievable news - the Mars Rover got destroyed due to a wrong calculation. The engineers who built the software did not convert metric to English system and tragedy ensued. This gives an idea of how important the practice of software testing is!
As the complexity of the software projects grow, so does the difficulty of testing those big software projects. With the growth of cloud technologies, the number of users who use a software now reaches hundreds of millions and even billions. When mistakes are made, the impact of that mistake now potentially be catastrophic. Here are a few software testing practices that I have found useful in assuring high QoS for the software and services.
Wait, AI could Test Everything!
It is true that AI agents have been making great progress in automating the process of writing and execution of software tests, it still has a long way to go. A lot of that has to do with pure combinatorics. Let’s assume we have have a program that has 10,000 functions, each of those functions has 5 input parameters and we use the AI agent to write unit tests that execute in 10 ms. The total time required to write unit tests would be 8 Hrs and that is just unit tests. Executing other types of tests would take even longer. So, sheer number and brute force execution will be too slow to be practically used in a CI/CD pipeline. We need engineers who can judiciously test the software without taking too much time1.
Types of Software Tests
Unit Test
Unit testing is the first line of defense against those creepy crawlies Once I did not add enough unit tests in my code, a senior developer asked me "How do you know that your code works?"When practicing TDD (test driven development), you would write a test for your new piece of code, run the test which will fail, and then write code until that failing test passes. The unit tests are usually white-box tests - tests that know how a unit/function operates. While building a car, a unit test would be measuring and verifying the current produced by the alternator.
Functional tests
The functional tests takes on the dependencies of external libraries and services to test the functionality of an entire computer program or module. While building a car, functional testing would be measuring the torque while running the engine with the input of gas and battery.Tools such as Sonarqube can help you run all unit and functional tests on demand or as part of the CI/CD pipeline.
Integration Tests
These are the heavyweights of the testing world. These are designed to test the software end-to-end with in a real-world environment. You can think of these as driving the automobile on a road to test how well it drives. In this test all components of the car are working in unison to produce the driving outcome. Integration tests are usually the most time-consuming tests to write and are the most difficult to maintain.
Performance Tests
So, the car drives fine in a plane obstacle-free road, but what about when the going gets tough? Performance tests are all about testing the program as a whole under load conditions. You can think of this as driving the car in with maximum load (passengers or goods).Performance tests are usually done after completing all other testing has been done and the infrastructure and code have reached a certain level of stability. Tools such as Jmeter are useful for running performance/load tests on APIs.
Monitoring
In cloud computing, the major components/modules of a piece of software is stored in the cloud and is updated frequently by a moderately large group of engineers. To ensure the QoS, the critical components of a software are now monitored continuously. The monitor is usually a program that measures the health of some service or executes a set of defined steps over and over. When the health metric of the service falls below threshold or one of those steps fail, the monitor alerts the supporting engineering team. In our car example, it would be like checking the battery power and the engine temperature while the car is being driven.Finding the right balance between effective monitoring and alerting noise will take continual testing and re-adjustment of alert threshold.
Optics
While monitors detect instantaneous failures issues, optics are a way of looking at the performance and history of the software in the real world over a longer term. Optics for a software may provide insights into its usage (growth of user base), performance (response time during peak and off-peak hours), etc. For our car example, optics would be like collecting and analyzing gas mileage data for the car over a month to verify its actual mileage. A successful optics infrastructure should contain a small set of basic reports and the ability to export collected data.
Testing in Production (TIP)
Replicating the scale and complexity of the production environment in a dev or testing environment is notoriously difficult. So, functional tests run in the dev or test environment may fail to surface the subtle bugs. To overcome this, engineers often isolate a smaller set of users within the production environment and test the software release there by running manual, automated tests and by reviewing the monitoring and optics.
Time-Tested Best Practices
Unit tests should complete in milliseconds
Code coverage should be fairly high >= 95% to be effective against regressions or bugs.
Unit tests are the best tool to avoid regressions when refactoring your legacy code.
When integrated with the build pipeline, unit tests are the best type of documentation for your code.
Incorporating the unit tests in the CI/CD pipeline usually can be helpful in preventing regressions.
Functional, Performance, and UI tests all have a significant cost of development and maintenance
Manual testing still remains useful for catching complex software solutions and services
Quantum computing may help with the execution time, but that is not commercially available yet