Machine Learning Engineering meets Automotive SPICE
Vehicle development is more and more influenced by autonomous driving which, in turn, requires more and more Machine Learning. But what exactly is Machine Learning Engineering and how does it impact the process landscape(s) of carmakers and their suppliers? The answer: Different and new approaches are necessary for a Machine Learning Engineering model evaluation. In this interview, Bhaskar Vanamali and Christina Stathatou provide answers and present their approach on how to assess the Machine Learning capability and maturity.
BACK TO Automotive SPICEHello Mrs Stathatou, hello Mr Vanamali. Let us start with the first question – how do you define Machine Learning, and how do you differentiate it from Artificial Intelligence?
Christina Stathatou For me, it was easier to understand what Machine Learning is when I compared it with human learning. When a baby is born, it doesn’t have much knowledge about the world, but it has a brain so it starts to observe the environment. In the case of the Machine Learning model, the brain is the algorithm that is fed by the data provided by the environment, which is essential for developing basic skills. For example, an infant learns by observing the environment. The lady it sees every day is likely the mother, and the male person is likely the father. If it wants something, it will start crying. The infant learns some basic skills based on observation; based on getting fed information. The baby develops skills and knowledge, which are derived from the environment. And Machine Learning – in my opinion – is quite similar. A machine learns how to recognize certain things such as patterns, and hones skills based on training and data; partially from the environment, partially through the data engineers who feed it with data. And regarding the difference between Machine Learning and Artificial Intelligence: I think there are a lot of opinions about that. In our definition, Artificial Intelligence is simply the marketing term for Machine Learning. And there is not a huge difference there between the two terms. But, for sure, there are a lot of different definitions.
Bhaskar Vanamali Christina, I liked your comparison. But perhaps also one topic regarding what is the difference between Artificial Intelligence and Machine Learning. Artificial Intelligence is typically a specialized area of IT which tries to replicate human intelligence. Now, what does that mean? It is impossible at the moment with our machines, with our computers, to replicate a human brain. It’s unbelievably complex. We have more than 100 billion nerve cells in our brain and each of those has 1,000 – 10,000 connections or so-called synapses. That means, overall, even if we take the lower number, we are already talking about 100 trillion to one quadrillion connections in the brain. This is nowhere near what we can reproduce in Machine Learning. Machine Learning, on the contrary, is a subset of Artificial Intelligence that allows us to program machines to learn from data without being programmed explicitly. And just to see the comparison now, typically the connections are between 500,000 and one million connections. So, it’s nowhere near what our brain does.
The fear that some people have of Artificial Intelligence is not really based on what we can achieve. Now, what can we actually achieve with Machine Learning? We can achieve – and here Machine Learning is sometimes better than we are – recognizing patterns. That’s the real goal of Machine Learning. And in that respect sometimes machines are faster and better than we are. But we have to train them, and we have to make sure that the data set to train them is top-notch. There should not be a data bias which then would lead the model to learning the wrong patterns. Data management is an art in itself which makes model development or Machine Learning development really challenging.
Machine Learning is a big part of the future of the automotive industry.
Now we know the distinction, but how relevant do you think is Machine Learning for the automotive sector?
Vanamali The interesting part is going to be autonomous driving. There’s no question about it. We have so much data to process: by cameras, radars, and so on. We need the support of Machine Learning for the car to drive itself – without interaction by the driver or occupants of the Level 5 vehicle – which would require it to make sense of the data coming in. We probably can get near Level 3 driving now, but Level 4 and Level 5 are something in the future, even though perhaps the near future. What we can already cover though are other Machine Learning areas like warranty topics and they are already happening. We also are already developing intelligent sensors. So, it’s not only the part of autonomous driving that everybody thinks about when we talk about Machine Learning, but there is a wide variety of topics.
If we think about diagnosis, wear and tear, where Machine Learning helps us to get a better grasp of vehicle maintenance. Let’s think about a transmission system. Typically, if we talk about service, we will talk about how often the car or truck should be serviced. Just by looking at the driving patterns, if we see that there are a lot of gear changes, there will be a shorter interval to the first maintenance period. Whereas if we always drive at the same speed on a highway, long-distance, obviously there is little wear and tear, and the service intervals can be much longer. That is something where Machine Learning can recognize the driving patterns much better than just saying: ”After 80,000 kilometers you have to go to the garage and get your car checked“ or ”And oil change of your transmission system is needed“.
You’re currently designing the SPICE for Machine Learning Engineering add-on. Which institutions are you working together with and what is the current status?
Vanamali At this moment we are members of DIN and ISO working groups. DIN is the ISO-equivalent in Germany. All of these working groups are working on standards for Machine Learning. We also developed internally a plug-in which now is going to be incorporated in Automotive SPICE 4.0. We are supporting process group 13 of VDA QMC where we are improving our assessment model with OEMs and suppliers for assessing and improving Machine Learning development. We also have several customers who got wind of our plug-in and they are very interested in assessing their processes and understanding how they can achieve higher quality development for Machine Learning. They also noticed that it’s very shirt-sleeved how they develop Machine Learning models, and they did not define proper processes yet. If they have, they are struggling to find the right language to systematically approach that the processes. We receive a lot of questions regarding our plug-in. It seems we kind of hit a nerve.
Stathatou I think this is a very nice way to describe it. What Bhaskar highlights is that it’s not a topic for the long future. It’s a topic that is challenging our current development. And we are not talking about flying cars or cars which drive us around while we sleep in the back because those sound so far away for us now. What he described is the reality of today and the reality of the next few years. That is why Machine Learning for sure is already very relevant in the automotive sector.
What added value will the plug-in offer companies?
Stathatou Machine Learning is something that is already applied in many of the products that are part of the cars on the road. It’s a big part of the future of the automotive industry. The added value of the plug-in is that it addresses some topics that are special regarding Machine Learning; not related to the classic Automotive SPICE approach. We need processes for every new challenge that is happening. Every innovation eventually becomes an essential part of the development of the product. Standardization is nothing than a collection of best practices. If you account for these practices and define appropriate work products, the quality of your development will improve. That’s why we need processes for Machine Learning. We don’t need to reinvent the wheel every time that we develop a Machine Learning Engineering model.
Also, we don’t use only the classic Automotive SPICE model because there are some differences in Machine Learning development and technologies that are not covered by Automotive SPICE. A few examples: First, the very crucial role that data management plays in the context of Machine Learning. As Bhaskar mentioned before, without collecting and processing the right data, we will not be successful in developing a Machine Learning model. Cars are driving in very complex environments, so we need a whole variety of data that requires purposeful collection and appropriate analysis of different scenarios. The data must be processed and labeled as well. It is essential that the data we feed the Machine Learning algorithms is of very high quality. Having a process that controls data management in the context of Machine Learning is very important and it is not part of Automotive SPICE.
Another aspect is the training, which is the actual process of using the data to create a Machine Learning model. And also the validation and verification or, in other words, the evaluation of the Machine Learning Engineering model.
Christina Stathatou is a Senior Consultant at Kugler Maag Cie. She has been working on process improvement for nearly 10 years and has co-developed the MLE plugin. Being a provisional assessor, she has participated in Automotive SPICE assessments as a co-assessor and performed gap analyses. Statathou loves evolving and expanding her expertise in exciting new areas such as AI, data management and cybersecurity engineering.
We know some techniques for the evaluation of embedded systems already, but for Machine Learning we have different processes. Machine Learning engineers are using a lot of trial-and-error methods, they experiment a lot and, therein, have a large variety of evaluation methods. Opening the door to all those evaluation methods that are used in Machine Learning is also very important and goes beyond the scope of Automotive SPICE. Therefore, we want to provide a standard that companies can use for the development of Machine Learning models in their products. The point is also, that Machine Learning engineers don’t speak Automotive SPICE or processes. They are coming from other industries. That is also quite important for us: to grow closer to this kind of different target group like the Machine Learning and Artificial Intelligence community. So, with this plug-in, we want to give a package that associates Machine Learning terminology with Automotive SPICE. It’s a plug-in but you can also use it as a standalone in the context of Machine Learning.
Vanamali I would like to address two more aspects. Christina was talking about the development of a company that is using Machine Learning to develop certain features. But on the other hand, you also must see the OEM-side and the end-customer-side. Machine Learning Engineering is a part of the development, which is nondeterministic. You can’t know exactly how your model is going to cover the problem you’re facing. How does it recognize pedestrians? How does it recognize the lane? And because of that, concerns regarding reliability and trustworthiness are topics that I, as a driver of such a car, would have. And knowing that there is a quality standard behind it which ensures that certain steps are completed and that I am following certain processes would reduce my risk. That helps the acceptance of Machine Learning and perhaps the expectation of a certain quality in the model. And that is something which we provide with our plug-in. It’s not the only solution – there will be other elements that are required – but it’s definitely one of the stepping stones to ensure a high quality of Machine Learning Engineering models.
The other aspect is that you can check the quality of your sub-supplier. We always looked at the company itself or at the end customer, but there's also the need to examine the quality of the sub-supplier, which was the original reason for introducing Automotive SPICE. But at the same time, like Christina just said, it is also a basis for defining your internal development processes in the Machine Learning context and ensuring that you have high-quality Machine Learning software. It would work in both ways in my opinion. Similar to the original intention of Automotive SPICE.
Bhaskar Vanamali is Principal and partner at Kugler Maag Cie. He has been working on process improvement for nearly 20 years and was secretary of the working group 13 of VDA QMC. He is also a Principal assessor and trainer, and a co-author of books. With more than 160 assessments under his belt, he has trained more than 250 assessors. Vanamali's latest hobbyhorse is AI development related processes.
Need support with a key project? We’re your first port of call when it comes to management consulting and improvement programmes in electronics development.
Steffen Herrmann and the sales team
You already mentioned that some customers heard about the plug-in already. When can we expect the first practical approach?
Vanamali We already have the first requests from two companies who are very interested and who would like to have the first gap analysis. Machine Learning Engineering will be part of Automotive SPICE 4.0 which is planned to be released as a draft by end of the year. But, of course, we also have to see when we are done with all the necessary tasks. It’s not only Machine Learning Engineering, which is going into the new version, but other aspects as well.
Also, for everyone who reads this interview: It may be interesting to know that we and PG 13 are going to perform a workshop about Machine Learning Engineering at the VDA SYS in June 2022. We hope to see some of you there and are looking forward to interesting discussions!
