SYS.5 – System Qualification Test
Involved in the challenges of Automotive SPICE, particularly the system qualification test? Read on for a quick overview of key information on SYS.5, a key process from VDA Scope, including a video and our free white paper.
back to Automotive SPICE
Interested in finding out more about System Qualification Test (SYS.5), the final Automotive SPICE® test process? Our free white paper provides you with a summary of all key information, including an extract from our book on Automotive SPICE® Essentials – ideal reading for anyone new to the topic of process improvements.
Automotive SPICE® is a trademark of VDA QMC.
The System Qualification Test process in Automotive SPICE® (also known as SYS.5) helps your organization to ensure that the integrated system meets the system requirements and can be delivered to the customer.
What is the system qualification test? The expectation is that you already have system requirements, so the goal is to check against these requirements and determine if they are fully met and correctly implemented.
This process is the final stage of the testing and should ensure that the product works as intended by the developing organization and as expected by the customer. If the System Qualification Test does not work well, errors can go undetected and customer satisfaction is going to drop.
Image
The SYS.5 process under VDA Scope
System tests usually include environmental, performance and endurance tests. The test environment depends on the product. Examples are HIL, vehicle tests and climate chambers.
Since this process is carried out shortly before delivery, there are close relationships to processes such as Project Management (MAN.3), Configuration Management (SUP.8), Product Release (SPL.2) and of course System Requirements Analysis (SYS.2).
The following are the most important aspects of System Qualification Test in Automotive SPICE®.
Like all testing and supporting processes, system qualification testing requires the development and definition of a testing strategy. You may have a separate test strategy for each test level, but it is better to develop and coordinate the test strategy across all test levels. This ensures that all requirements are covered, and redundancies are avoided.
The test strategy should cover the following topics
- the test object in question
- the methods for developing test cases and test data (e.g. development of positive/negative tests, equivalence partitioning).
- a regression test strategy, which in Automotive SPICE terminology means that you define how you want to re-test after a bug fix or change request
- the test environment
- test coverage in relation to the project plan and release plan
- entry and exit criteria and test interruption criteria.
Of course, this process has a strong connection to SUP.9 Problem Resolution Management, so you can use either the test strategy or the defect management strategy on how to deal with failed tests.
Part of the strategy is also to define coverage targets based on the type of delivery. So, here is a strong link to the project plan and the demand plan.
When you document a test strategy, parts of it can be generic and covered in the internal process description and guidelines. Again, Automotive SPICE does not define how and where you document your test strategy. It can be distributed over several documents.
The test strategy should also cover your approach to test automation. Generally, it can be said that test automation is always better than manual testing, but there may be tests that are too expensive to automate or the test environment does not allow this like for instance vehicle maneuvers.
Need support with a key project? We’re your first port of call when it comes to management consulting and improvement programmes in electronics development.
Steffen Herrmann and the sales team
This process also expects a corresponding selection of test cases to be performed for the various tests. The aim and expectation is that for the different deliveries the system is properly tested on the basis of the afore mentioned test strategy.
The idea is that you can have deliveries with different expectations. A possible strategy could be that you have complete coverage of all implemented systems requirements for the important deliveries.
- For smaller deliveries, only the delta of implemented requirements since the last delivery is tested.
- For this approach, of course, the right test cases must be selected.
Another possible situation for test case selection would be the regression test, which covers change requests and/or bug fixes. Here, test cases are selected that cover the change request or bug, and the effects that they can have. That means that dependencies on requirements that can be affected by the change request or bug fix are also tested.
This process also requires that you ensure traceability between your system test cases and system requirements. Traceability can be established through hyperlinks like in DOORS, through specific traceability tools like Rectify, through traceability matrices or through other manageable means which are supported by your tool landscape.
The purpose of traceability is that it
- supports consistency checks, i.e. checking the completeness and accuracy of the coverage of system requirements
- supports the impact assessment in case of change requests or bugs
- supports the reporting of stakeholder expectations and identify which requirements have been tested, often referred to as coverage reports.
The second part of this aspect is about consistency.
Consistency can only be proven in a review where you show that you covered the system requirements completely and correctly.
If you skip this review you may have incomplete or faulty system test cases. The worst part is that you may not even notice the defects in the system qualification test because this test is performed against your system requirements. If these are faulty your test may not show false behavior.
So, this review is really important!
If you follow these points
- you will implement a powerful system qualification strategy,
- to keep control of your releases,
- and you will avoid having disappointed customers.
As part of our strategy to systematically improve development processes in the automotive electronics sector, we are also an official licensee of Automotive SPICE® – a trademark of the Association of the German Automotive Industry (VDA QMC).
- Using Automotive SPICE® to achieve the required maturity levels within your key processes in development
- Systematically improving existing workflows and methods
- Evaluating the status of your process improvements through formal assessments and gap analysis
- Fulfilling the requirements of Automotive SPICE® in harmony with SECURITY,FUNCTIONAL SAFETY and AGILE METHODS
- Training your staff and assessors
