Thoughts on Verification: Verification in the DO-254 Process (Part 2 of 2)
In Part 2, Verilab consultants Paul Marriott and Alex Melikian continue their discussion on verification in a DO-254 quality assurance process. In this part of the conversation, they focus on the technologies, tools and methodologies that intersect DO-254 and verification. Part 1 can be viewed here.
Alex Melikian: Let’s move on to talk about some of the tools available tailored for a DO-254 process available in the industry. A name that seems to come up regularly is DOORS. What is DOORS?
Paul Marriott: DOORS is a tool, which stands for “Dynamic Object-oriented Requirements System”. It’s a tool which allows you to capture requirements and other documents, which are required for a DO-254 process, and to put in the traceability between the different levels of documentation. So every design requirement has to trace through to a design implementation; it has to trace through into a validation plan; it has to trace through into a test case. You have to describe a procedure, what you’re going to do, and what the acceptance criteria are. This ensures that every requirement in the design is actually implemented and validated.
And DOORS is a tool which allows you to manage these databases and provide the links between the different levels of documentation in a way that’s easy to see in areas where you might’ve missed out.
AM: It appears DOORS is the stalwart tool for the DO-254 certified process. So how do some of the verification tools out there that support or work with DOORS interface with it?
PM: DOORS will allow you to import and export via spreadsheets, and if you think of the way DOORS works, essentially, it’s a almost like a spreadsheet because you have a table of requirements and then a link from each requirement to another document, which may be a validation plan or a test description. And so, you could, I mean DO-254 itself says nothing about the tools that you have to use. You can achieve DO-254 certification with pencil and paper. As long as the process is repeatable and traceable and is described and audited properly, then you can do it. But of course, automation makes it much more efficient and much easier.
DOORS is not really designed to interface directly with verification tools. With modern verification, we’re using assertion-based and coverage-driven verification, and you can actually implement the validation of many of the requirements by your assertions. And so, you want to ensure that a particular assertion is tied back to a particularly requirement. Therefore, you may be able to export the set of requirements from DORS into a spreadsheet and import that into another tool, which then ties into your coverage plan, which is then linked into your simulator’s verification manager.
AM: There appears to be a lot of strong industry support behind it. Some of the major EDA tool vendors have features that interface directly to a DOORS tool, so it’s good to know that you’re not stuck with one specific vendor or tool provider.
PM: Yes, exactly. People think it’s the tools themselves which are certified, but it’s not. It’s the overall process which uses the tools. So DO-254 itself doesn’t mandate any particular tools. Of course, certain tools can make it easier to achieve the goals of DO-254, but it doesn’t mandate them.
AM: Okay. Now, the DO-254 process goes back a long way. Earlier (in Part 1), we mentioned we would talk about some of the modern verification tools and processes and how they relate to this subject. So how do things like coverage-driven testbenches or constrained random generation fit into a DO-254 process?
PM: This is one of the challenges with the DO-254 process. Because everything is requirements-based, the temptation is to write one test per requirement. And so, this leads to a very directed testing methodology, which fits in well to the traceability because you can trace one test back to one requirement. But with a modern process, in terms of verification that tends not to be the case. We tend to have fewer test cases and put in functional coverage and assertions to cover much functionality per test run. DO-254 is fine with this, as long as you can trace back those assertions and functional coverage points to actual requirements.
And the traceability tools allow you to ensure that every requirement does in fact trace down to a cover point, a test result or an assertion point.
AM: That’s good to know, especially for those who have been taking more of a directed test case suite approach to achieve their objectives, they can now be persuaded that modern verification paradigms and tools out there are still compatible with DO-254. There are other dynamics involving the management of a verification project like Agile. How does that fit into the DO-254 process?
PM: That’s a good question. Process and project management are not necessarily the same thing. The process describes things that have to be done to get you from design specification to verified implementation. Project management is the steps that you take along the way to ensure that the work gets done. Now, with DO-254 having various stages of involvement in terms of audits, it does impose a somewhat rigid structure on the order that things are done. But there’s nothing to say that along the steps of meeting those milestones that you can’t use an Agile approach.
At the end of the day, with any kind of project management, you still have to cover all of the work that has to be done to reach a certain milestone, whether it’s agile or whether it’s a standard waterfall approach or any other project management approach. And I’ve worked with companies doing DO-254 certified processes that are quite happy using a Kanban style of Agile project management to achieve the goals of the overall process.
AM: I see, thanks for clearing that up for me. So let’s set the record straight for our readers: Agile deals with project management; whereas, DO-254 is a process flow.
PM: Exactly.
AM: So does DO-254 only affect firms involved in aerospace or mission-critical products, or are there firms outside of these industries that can also adopt it?
PM: DO-254, by definition, is purely for certification of complex electronic hardware for civilian aerospace. That’s its definition. That said, it doesn’t mean that the DO-254 style of process can’t be used for other mission-critical products, and there are groups working – particularly in the automotive area – on similar processes to DO-254 to achieve the high process assurance that’s required. If you look at some of the major automotive recalls that have been done in the past few years, some of the costs of the recalls have been tens of billions of dollars because there’s been bugs found in the software or the hardware, which has caused some problems.
I mean the aim with DO-254 is to ensure that when you’re designing a Level-A criticality system it will not fail and people don’t get killed. That said, you can still have a 100 percent certified Level-A system which can still fail because somebody didn’t think of something in the requirements. So the critical aspect of any higher reliability, high assurance process is the people involved ensuring that they write good requirements in the first place. And that’s really the hardest part of any mission-critical process.
And other mission-critical areas can definitely learn from the DO-254 experience. There’s also DO-178B, which is used for the software aspects of civilian aerospace, which also has the same criticality levels as DO-254 because now with systems becoming more complex from a software point of view, you still want to have the same assurance that the software is reliable and safe.
AM: That’s a good point you made about noncritical systems taking an approach of building critical systems designs. We can all agree that though your car may have an airbag, you’d still like the brake pedal to work 100% of the time. So if a firm is not obligated to adopt a DO-254 certified process for whatever reason, what are some of the things that it can take away from the process just the same?
PM: There’s several important aspects, which I think can be taken away for everybody. The first is to have a good set of design requirements. If you have a good specification, then you have a much higher likelihood of success at producing a correctly working product, which has been essentially verified to be reliable. So the process itself ensures that the requirements are written in a formal way. And the process also describes the traceability between the requirements, the verification plan and the implementation and also ties in code coverage and other aspects to verify that what was implemented was in fact what was intended.
Code coverage tells you nothing about the quality of the code, but if you don’t have 100 percent code coverage, either you’ve missed out verifying some requirement, as requirements were implemented without being specified – or that the testing wasn’t sufficient to cover all of the implementation. So it’s very important to look at code coverage, even though it tells you nothing about the code quality. And just having the formal audit process so people sit down and actually review the work that’s been done is also important.
Because DO-254 is high assurance, they really focus on peer reviews of the requirements themselves, the verification of the requirements, and the validation to ensure that there’s no single point of failure in the entire process. This is why, in fact, there’s no DO-254 certified tools because you want to make sure that each tool is covered by another process or another tool to ensure that there’s just no one single point of failure in the entire process.
AM: That’s interesting, but I can certainly understand the thinking behind that. On that note, I believe we’ll conclude this edition of “Thoughts on Verification”. I hope our readers not only feel safer when they board an airplane, but have also learned a few things about the DO-254 process. Thanks for your time Paul.
PM: Thank you Alex for some very interesting questions.
