Automotive DevOps Platform

... tests around the clock

With the Automotive DevOps Platform, we go from the big picture to the details and unite all phases of vehicle software testing – from planning the test scopes to summarizing the test results. At the same time, continuous monitoring across all test phases always provides an overview of all activities – even with several thousand test executions per day and in different test environments.

The automotive industry has been waiting for short and dynamic development cycles with continuous updates for years now. This is due to the constantly changing requirements for solutions and technologies. This balancing act between rapid innovations and high efficiency is accomplished by means of flexible and agile development methods. We have been working in accordance with the principles of agile software development since the very beginning. Now, we are going one step further with our Automotive DevOps Platform.

We are expanding our mindset to include a whole range of technical processes that further strengthen the close interaction between development and operations. In this way, we will be automating the entire vehicle software testing process to an increasing extent, thereby also achieving process consistency within the platform.

Our focus is on the continuous integration (CI) of new and optimized functions and on establishing test automation to deliver successfully tested software versions (CD) quickly. During the entire cycle of the test automation solution, in-built monitoring ensures that every step is traceable.

With all this, our Automotive DevOps Platform is not a rigid product, but more a process that is run through again and again and improved upon constantly. This continuity at all system levels (from components to the entire vehicle system) ensures that the high-quality standards of vehicle software are met, and guarantees compliance with various standards. (e.g., ISO 26262).

We bring managers, software developers, testers and administrators to a common table to work together on one goal – delivering excellent software to our customers.

Continuous Integration (CI)

Each change to the software code is continuously integrated by the function developer (at least daily). A new software version (build) is then built and tested automatically with component testing and integration testing. In this way, the functionality of the application is constantly ensured and inconsistencies can be detected and corrected at an early stage.

Continuous Delivery (CD)

Continuous testing of newly integrated functions enables automated delivery of production-ready software versions at almost any time.

Continuous Deployment (CD)

Unlike Continuous Delivery, each software build is rolled out automatically to the production environment after successful testing. Thus, the successfully tested app change can go live within just a few minutes. This enables quick and continuous feedback from the user.

Show content

Test Planning

In the planning phase, the test scopes within a test level are defined. The basis is formed by requirements that can be read in from a wide variety of sources – via direct connection of ALM tools (e.g., Octane, IBM RQM, Polarion), a proprietary API or by importing various exchange formats (e.g., ReqIF).

The speed of the first automated test execution with the TraceTronic Automotive DevOps Platform depends on the customer-specific prerequisites for an automated test process. More often than not, existing data structures and test structures need to be cleaned up and organized before anything else.

What tools are already available? Where are the requirements and test specifications managed? Is there a ticket system in place already? Which system is used for reporting? Where and how are the reports collected and evaluated? How does the work process work in general? The essential framework conditions are already defined in advance with a questionnaire like this, or a similar instrument. The test manager knows exactly how the test suite is structured. His knowledge is indispensable at this point. For, after all, the goal is to first map the existing test process and to efficiently integrate the available data into the Automotive DevOps Platform using interfaces.

Information about the test suites and test management systems is usually queried directly via API accesses to the individual tools. But script-based solutions can also be used for this. Once the content of the test-suite is defined, the test coverage can be generated automatically. This gives the testers and the test manager insight into the current status of the test activities at any given time. The TraceTronic experts adapt this precisely to suit the customer workflow.

If the test suite is designed for a new requirement, the relevant tests and the associated set of all possible parameter values (parameter spaces) will be defined. Here, it is not the concrete parameter value that is decisive, rather, the focus is on the abstract definition of the essential variables.

What is to be tested with which test cases in physical (HiL) or virtual environments (MiL, SiL) is then planned. Here, the entire team has an up-to-date overview of the test scope and the status of the test process at all times.

In the automated mode of the platform, the new planning of test scopes builds on the previously determined test results. If the release overview shows that not all the results were appropriate, then the loop is fine tuned and run through again, as many times as necessary.

Show content

Test Development

In this step, the focus is on the development and management of requirements-based and reusable test cases for automated execution in both physical (HiL) and virtual test environments (SiL, MiL).

With each new requirement from vehicle manufacturers, software functions are developed for ECUs. These must be verified and validated using test suites. The scope of these test suites is variable and depends on the complexity of the vehicle function. At the very least, the defined specifications of the statutory standard norms for the newly developed vehicle functions must be tested. This can be done in the form of many real and virtual driving kilometers, but also on the basis of various scenarios in which the new driving functions are tested critically and evaluated in detail.

The testers design abstract test cases and test projects in ECU-TEST for the new software functions, across platforms, i.e., for physical test environments (HiL), as well as for virtual ones (MiL, SiL). The abstraction not only ensures the reusability of the test cases but it also means that the tests can be parameterized depending on requirements. All the test cases that are created, including all the required data, are managed in a version management system (Git, SVN). The test suites are then compiled based on these.

The test automation tool that is used to develop the test cases and test suites does not necessarily have to be ECU-TEST. Any other automotive tool, such as MATLAB, AutomationDesk or vTestStudio, can also be connected. This flexibility characterizes our Automotive DevOps Platform and reflects how we work at TraceTronic – we always put customer benefits first.

Show content

Trace Analysis Development

Trace analyses can be prepared for recorded measurement data, which significantly increase the test depth by means of validations and verifications of the recordings. Numerous recording formats (e.g., Matlab, ASC, MDF) are supported.

During the execution of a test case, measurement data or traces are recorded. The purpose of trace analysis is to examine these in detail and with regard to different criteria. It starts with the requirements that are placed on a signal course and ends with the fully automated generation of a test report. The goal is to analyze multiple traces from different recording sources along a common, synchronized time line.

Trace analysis development is similar to test case development, both in terms of logic and in terms of sequence. The recorded signals of different tools are analyzed measuring point by measuring point, independently of the measuring grid. Meaningful diagrams help to provide a quick overview of the places where the signal characteristics deviate from the given requirements.

A trace analysis is generic, i.e., it is universally applicable and hence easy to reuse. The signals to be read are also specified only when the analysis is to be performed for a specific test case.

Strictly speaking, however, it is not just signal characteristics that can be evaluated. Image, audio and video recognition are also possible use cases for the analysis. This is possible because TRACE-CHECK, the tool for trace analysis, reads and analyzes the measurement data of all commonly used automotive formats (CSV, MDF, DAT, MAT, STI/STZ, ASC, WAV, AVI, and many more).

Regardless of the test environment (MiL, SiL, HiL or vehicle) that is used to generate the traces, generic trace analyses can be used to find and display anomalies.

Show content

Testing

Intelligent and integrated test execution ensures that tests are automatically distributed to available test stations. At any time, the best possible utilization can be ensured and the current status of the test resources can be monitored.

The classic test case execution consists of: stimulation, recording, and evaluation of signals. Test case by test case. To avoid having to do this individually, the test cases are bundled into test suites (so-called ECU TEST projects) and executed together. However, this is not quite sufficient for optimum automation. Triggering projects one after the other in a test environment is already a big step forward. However, this does not yet make the testing very efficient. It is better to use all the available resources for the projects to be executed so as to avoid wasting time. If the distribution of test cases and projects to the free resources is also automated, it will help save more time.

TEST-GUIDE makes this possible within the Automotive DevOps Platform. For this purpose, so-called playbooks are transferred to this tool from ECU-TEST. They ensure particularly agile handling and are designed for executing test jobs in parallel and distributed across multiple physical or virtual test benches.

The location where the test execution takes place can be decided individually. Whether on HiL test benches or in the cloud (MiL, SiL) – everything is possible.

Show content

Trace Analysis

The test station-independent and downstream analysis of the measurement data – decoupled from the test case (stimulation) – enables the results to be reused and the test system to be made available earlier for further runs.

Testing and analysis belong together, like starting and braking. However, both do not necessarily have to be performed immediately one after the other. This means that the traces can also be analyzed downstream, and on any PC. Coveted test benches will thus be available sooner for testing again, and the test throughput is also increased significantly.

Recordings made during the test run and the defined trace analyses are transferred automatically to TEST-GUIDE as an analysis job. The tool then organizes the available resources for trace analysis on its own. The ResourceAdapter, which is installed on the test environments and organizes the communication with TEST-GUIDE, plays a significant role here. Thanks to this, free ECU-TEST instances on physical or virtual machines or in containerized environments are always known. The analysis jobs are then distributed to these free instances and the trace analyses are executed with TRACE-CHECK.

Does the test object meet the test specification? Is the measuring device functioning correctly? Has the correct measuring range been selected? All details of the trace analysis are presented clearly in a test report for each individual step of the analysis and then stored automatically in TEST-GUIDE.

Show content

Test Result Review

With several thousand test executions a day, it is important to have support for evaluating results and following up on anomalies. Errors that occur can be recorded and forwarded to a connected ticket system (e.g., Jira, Redmine, Octane, etc.).

The further the automation of test executions progresses, the higher the test throughput within 24 hours. As a result, countless test results will have to be sifted through and evaluated. This can no longer be done without tool support. TEST-GUIDE has an integrated review process for this purpose. This helps to detect and display clearly any errors and inconsistencies that may occur during vehicle software testing.

If errors are discovered during a test run, a review can be created at the push of a button. The decisive factor here is the detailed documentation of the error that has occurred. This is the only way to trace errors easily and rectify them quickly.

The best prerequisites for rapid troubleshooting exist with the connection to defect management systems, such as Jira, Redmine, HP ALM.net or Octane. This means that the errors can be handed over straight away to the Development as a ticket, and scheduled quickly into the vehicle software development process, without having to log into the system itself. In addition to this, all those who are involved will be kept informed via e-mail notification and will also have a constant overview of the processing status.

The reviews that are performed on each test case can be included in the individual test completion reports. These provide information about passed, failed and erroneous test cases.

Show content

Test Result Release

Release represents the conclusion of the test activities. The summary of test results in release and coverage overviews is based on the test scopes defined at the beginning. The link between planning, implementation and results forms the basis for producing informative final reports.

Once the test cases for a new driving function have been executed and analyzed, the release overview in TEST-GUIDE provides information about the entire test process. Combined with the coverage, the release as a summary of the test results acts as the ideal decision-making basis for the release and for the delivery of a new software version, and provides a good overview of the initially designed test scopes.

Points that are still open can be marked out at precisely this point. Such as whether the depth of testing was sufficient, the test cases were varied enough, or the results obtained were meaningful or not. Or even how high the quality of the current software status ultimately is.

To support this assessment, an integrated final report (Excel, PDF) summarizes the determined test results and the infrastructure data specifically: comprehensively, clearly and, above all, in a manner that is transparent for everyone involved.

In addition to this, all the test results can also be fed back into the customer's own test management systems (e.g., HP-ALM, Octane, Jama, Polarion), because TEST-GUIDE can be integrated in a flexible manner with various automotive tool chains.

The lessons learned from the release and coverage will be carried over into the planning phase for subsequent iterations.

Show content

Monitoring

Monitoring is a continuous phase that accompanies development and provides an overview of all activities from planning to the summary of test results. By means of a clear dashboard, progress can be monitored and measures can be derived to deal with problems in good time.

Sustainable Continuous Integration solutions are really intelligent only if every step of the test process can be traced by means of consistent monitoring. It is only then that planning, execution and analysis can intermesh systematically and run in a completely automated manner. This function is becoming increasingly important in view of the growing digitization in vehicles and the resulting increase in the number of test executions and test resources.

Especially with large numbers of test executions, it is important to have everything under control. Which test benches or test environments are connected? What is the utilization rate? What condition is the test bench in? Which test case is currently being executed on which test bench? This comprehensive monitoring is handled by the ResourceAdapter within the Automotive DevOps Platform.

This is because it automatically reports all important information pertaining to the available infrastructure to TEST-GUIDE. Therefore, TEST-GUIDE can independently optimize the resource utilization and scale it as needed. In addition, it also transmits all the other information from the test benches – such as vital data, test executions or configurations – to the central TEST-GUIDE server. There, all this data is merged and made available for various views and export options in TEST-GUIDE, with a live status of the test executions at every moment.

This effectively answers the question of how to cope with the growing number of test executions of increasingly complex vehicle software. Now is the time to rev it up – because the Automotive DevOps Platform can test on all systems, around the clock.

This is the place for me!

This is the solution to your problem? Then contact our sales team and arrange an appointment with us.

We were not able to answer all questions in such a short time? Then get in touch with our sales team to access the information you still need.

Contact
Contact