
Jörg “Jogi” Sievers has been working as a test automation engineer and test specialist for more than 25 years. Before that he worked for a decade in the food industry, where his eye for quality was sharpened from the start. Privately, data transmission in the pre-internet age (bulletin board systems) was his hobby, until he finally ended up in IT. Installations, migrations, support and training for medical practice software, as well as selling and setting up PCs, NeXT, Sun and Apple computers, gave him a broad IT knowledge base. This was followed by GUI test automation of StarOffice and OpenOffice.org as a desktop application, and Sun WebOffice in the web world. After that, software in the energy industry kept him busy for ten years, until he landed at the SIGNAL IDUNA Group three and a half years ago. He is a fan of business-driven test automation, for example in the context of behaviour-driven development, consumer-driven contract testing of interfaces or related approaches. IT security and privacy in the digital space have been on his agenda since the bulletin board era and still are today. For many years he has been an expert for Sahi Pro, a test tool that handles web, desktop, mobile and SAP business transaction tests.
Information about the person as of summer 2023.
Where do we stand today with testing using AI?
When we ran our software on underpowered 80386 and 80486 PCs in the pre-internet age (bulletin boards in the FIDO® and ZERBERUS® networks), which you had to dial into with modems, we already had chatbots that could create sentences in response to questions based on what they had learned. The learning space was limited to what we stored in a machine there. In that respect, the amount of learning material today has grown many times over through the last decades, and of course so has the enormous performance and the now very wide networking of the machines themselves. The idea of machine learning (ML) and of interacting with machines is even older, but back then the technical possibilities to implement it were very limited.
The hype around machine learning, like that around blockchain technology some time ago, is due to the spread of information today. Neither blockchain nor machine learning will solve all our challenges. Worse still: there will again be IT people who want to solve problems with ML where well-written algorithms would be enough but would consume much less electricity.
Artificial intelligence (AI) as we sometimes define it (à la “Terminator” movies) does not exist, and will not in the next few years either. Prof. Dr. Dagmar Monett Díaz, who holds a chair for AI at TU Berlin, said so as early as 2021 at the Agile Testing Days:
“There are no truly intelligent artifacts on the horizon yet” (source)
So we are at the beginning of a technology that we by no means master, and, as with all new technologies, one should look ahead optimistically but also think about the consequences. Risk analysis must be, and is, one of the most important building blocks a tester has to master. In the case of poorly trained neural networks, the consequences for people can be very negative, so we should not squander the trust that people [do not] have in IT [any further]. Generated false information can very quickly no longer be retrieved given today’s level of distribution. Already now we sometimes have no quick and good line of defense against identity theft or break-ins into critical infrastructure, but if this becomes even faster and more insidious through ML, we must not only see all the good but also the bad.
How will AI change software testing in the next five years?
The hype will have ebbed away and realism will have set in (cf. blockchain). Some things, such as the way we search digitally, assistance, information and prediction systems, will have been improved. That means work processes will be more automated than today, because, for example, extracting information from data stocks and putting it together contextually is something a machine will simply master better and faster than a human. People will then be able to tackle other, more complex tasks that a machine cannot perform without error without human intuition.
That was already the case when the first IT systems and robotics were used in industry: heavy, but also boring and repetitive tasks were replaced by machines and people were used for other things. If you once had to stack pallets with goods by hand, robots do that much more precisely today. Testers back then had to develop test methods for how to check the robot processes before they went to the customer.
For testers this means that we will get support with test design, test automation and evaluation, and can devote ourselves to more complex test tasks. What will be harder for us is the predictability of test results and also any reproduction of failures. It would be nice if there were new tracing technologies and test oracles for that too, so that we are not left alone with this question. But I fear that testing will, as so often, be the stepchild of the ML age, and we will then have to patch things up afterwards.
Which skills do testers need for that?
We will have to learn how to steer ML systems, for example prompt engineering will become a basic part of the role in software quality assurance. We also have to dive deeper into probability estimation and prediction in order to create methods for better interpreting the answers of ML systems. Collaboration with the coders who build these ML systems must become much stronger, because ethical, data protection or security-relevant checks already have to be agreed on at the requirements stage. Once the model is “created”, it is very hard to “test quality into it” afterwards, and we know that.
Will the role of testing still exist in 10 years?
The question seems to be asked only in computer science. Food producers would never ask whether there need to be people who ensure in some form the quality of the goods delivered. Mechanical engineers would also never get into a machine, be it an aircraft or a car, that had not been thoroughly checked beforehand.
Why computer science believes it can abolish this role will probably remain a mystery that I will take with me to my grave.
Whatever the people in this checking role will be called, they will exist, and they will have to contribute and specialize even more than today.