The speed of the first automated test execution with the TraceTronic Automotive DevOps Platform depends on the customer-specific prerequisites for an automated test process. More often than not, existing data structures and test structures need to be cleaned up and organized before anything else.

What tools are already available? Where are the requirements and test specifications managed? Is there a ticket system in place already? Which system is used for reporting? Where and how are the reports collected and evaluated? How does the work process work in general? The essential framework conditions are already defined in advance with a questionnaire like this, or a similar instrument. The test manager knows exactly how the test suite is structured. His knowledge is indispensable at this point. For, after all, the goal is to first map the existing test process and to efficiently integrate the available data into the Automotive DevOps Platform using interfaces.

Information about the test suites and test management systems is usually queried directly via API accesses to the individual tools. But script-based solutions can also be used for this. Once the content of the test-suite is defined, the test coverage can be generated automatically. This gives the testers and the test manager insight into the current status of the test activities at any given time. The TraceTronic experts adapt this precisely to suit the customer workflow.

If the test suite is designed for a new requirement, the relevant tests and the associated set of all possible parameter values (parameter spaces) will be defined. Here, it is not the concrete parameter value that is decisive, rather, the focus is on the abstract definition of the essential variables.

What is to be tested with which test cases in physical (HiL) or virtual environments (MiL, SiL) is then planned. Here, the entire team has an up-to-date overview of the test scope and the status of the test process at all times.

In the automated mode of the platform, the new planning of test scopes builds on the previously determined test results. If the release overview shows that not all the results were appropriate, then the loop is fine tuned and run through again, as many times as necessary.

With each new requirement from vehicle manufacturers, software functions are developed for ECUs. These must be verified and validated using test suites. The scope of these test suites is variable and depends on the complexity of the vehicle function. At the very least, the defined specifications of the statutory standard norms for the newly developed vehicle functions must be tested. This can be done in the form of many real and virtual driving kilometers, but also on the basis of various scenarios in which the new driving functions are tested critically and evaluated in detail.

The testers design abstract test cases and test projects in ECU-TEST for the new software functions, across platforms, i.e., for physical test environments (HiL), as well as for virtual ones (MiL, SiL). The abstraction not only ensures the reusability of the test cases but it also means that the tests can be parameterized depending on requirements. All the test cases that are created, including all the required data, are managed in a version management system (Git, SVN). The test suites are then compiled based on these.

The test automation tool that is used to develop the test cases and test suites does not necessarily have to be ECU-TEST. Any other automotive tool, such as MATLAB, AutomationDesk or vTestStudio, can also be connected. This flexibility characterizes our Automotive DevOps Platform and reflects how we work at TraceTronic – we always put customer benefits first.

During the execution of a test case, measurement data or traces are recorded. The purpose of trace analysis is to examine these in detail and with regard to different criteria. It starts with the requirements that are placed on a signal course and ends with the fully automated generation of a test report. The goal is to analyze multiple traces from different recording sources along a common, synchronized time line.

Trace analysis development is similar to test case development, both in terms of logic and in terms of sequence. The recorded signals of different tools are analyzed measuring point by measuring point, independently of the measuring grid. Meaningful diagrams help to provide a quick overview of the places where the signal characteristics deviate from the given requirements.

A trace analysis is generic, i.e., it is universally applicable and hence easy to reuse. The signals to be read are also specified only when the analysis is to be performed for a specific test case.

Strictly speaking, however, it is not just signal characteristics that can be evaluated. Image, audio and video recognition are also possible use cases for the analysis. This is possible because TRACE-CHECK, the tool for trace analysis, reads and analyzes the measurement data of all commonly used automotive formats (CSV, MDF, DAT, MAT, STI/STZ, ASC, WAV, AVI, and many more).

Regardless of the test environment (MiL, SiL, HiL or vehicle) that is used to generate the traces, generic trace analyses can be used to find and display anomalies.

The classic test case execution consists of: stimulation, recording, and evaluation of signals. Test case by test case. To avoid having to do this individually, the test cases are bundled into test suites (so-called ECU TEST projects) and executed together. However, this is not quite sufficient for optimum automation. Triggering projects one after the other in a test environment is already a big step forward. However, this does not yet make the testing very efficient. It is better to use all the available resources for the projects to be executed so as to avoid wasting time. If the distribution of test cases and projects to the free resources is also automated, it will help save more time.

TEST-GUIDE makes this possible within the Automotive DevOps Platform. For this purpose, so-called playbooks are transferred to this tool from ECU-TEST. They ensure particularly agile handling and are designed for executing test jobs in parallel and distributed across multiple physical or virtual test benches.

The location where the test execution takes place can be decided individually. Whether on HiL test benches or in the cloud (MiL, SiL) – everything is possible.

Testing and analysis belong together, like starting and braking. However, both do not necessarily have to be performed immediately one after the other. This means that the traces can also be analyzed downstream, and on any PC. Coveted test benches will thus be available sooner for testing again, and the test throughput is also increased significantly.

Recordings made during the test run and the defined trace analyses are transferred automatically to TEST-GUIDE as an analysis job. The tool then organizes the available resources for trace analysis on its own. The ResourceAdapter, which is installed on the test environments and organizes the communication with TEST-GUIDE, plays a significant role here. Thanks to this, free ECU-TEST instances on physical or virtual machines or in containerized environments are always known. The analysis jobs are then distributed to these free instances and the trace analyses are executed with TRACE-CHECK.

Does the test object meet the test specification? Is the measuring device functioning correctly? Has the correct measuring range been selected? All details of the trace analysis are presented clearly in a test report for each individual step of the analysis and then stored automatically in TEST-GUIDE.

The further the automation of test executions progresses, the higher the test throughput within 24 hours. As a result, countless test results will have to be sifted through and evaluated. This can no longer be done without tool support. TEST-GUIDE has an integrated review process for this purpose. This helps to detect and display clearly any errors and inconsistencies that may occur during vehicle software testing.

If errors are discovered during a test run, a review can be created at the push of a button. The decisive factor here is the detailed documentation of the error that has occurred. This is the only way to trace errors easily and rectify them quickly.

The best prerequisites for rapid troubleshooting exist with the connection to defect management systems, such as Jira, Redmine, HP ALM.net or Octane. This means that the errors can be handed over straight away to the Development as a ticket, and scheduled quickly into the vehicle software development process, without having to log into the system itself. In addition to this, all those who are involved will be kept informed via e-mail notification and will also have a constant overview of the processing status.

The reviews that are performed on each test case can be included in the individual test completion reports. These provide information about passed, failed and erroneous test cases.

Once the test cases for a new driving function have been executed and analyzed, the release overview in TEST-GUIDE provides information about the entire test process. Combined with the coverage, the release as a summary of the test results acts as the ideal decision-making basis for the release and for the delivery of a new software version, and provides a good overview of the initially designed test scopes.

Points that are still open can be marked out at precisely this point. Such as whether the depth of testing was sufficient, the test cases were varied enough, or the results obtained were meaningful or not. Or even how high the quality of the current software status ultimately is.

To support this assessment, an integrated final report (Excel, PDF) summarizes the determined test results and the infrastructure data specifically: comprehensively, clearly and, above all, in a manner that is transparent for everyone involved.

In addition to this, all the test results can also be fed back into the customer's own test management systems (e.g., HP-ALM, Octane, Jama, Polarion), because TEST-GUIDE can be integrated in a flexible manner with various automotive tool chains.

The lessons learned from the release and coverage will be carried over into the planning phase for subsequent iterations.

Sustainable Continuous Integration solutions are really intelligent only if every step of the test process can be traced by means of consistent monitoring. It is only then that planning, execution and analysis can intermesh systematically and run in a completely automated manner. This function is becoming increasingly important in view of the growing digitization in vehicles and the resulting increase in the number of test executions and test resources.

Especially with large numbers of test executions, it is important to have everything under control. Which test benches or test environments are connected? What is the utilization rate? What condition is the test bench in? Which test case is currently being executed on which test bench? This comprehensive monitoring is handled by the ResourceAdapter within the Automotive DevOps Platform.

This is because it automatically reports all important information pertaining to the available infrastructure to TEST-GUIDE. Therefore, TEST-GUIDE can independently optimize the resource utilization and scale it as needed. In addition, it also transmits all the other information from the test benches – such as vital data, test executions or configurations – to the central TEST-GUIDE server. There, all this data is merged and made available for various views and export options in TEST-GUIDE, with a live status of the test executions at every moment.

This effectively answers the question of how to cope with the growing number of test executions of increasingly complex vehicle software. Now is the time to rev it up – because the Automotive DevOps Platform can test on all systems, around the clock.