Thoughts on Verification: Verification in the DO-254 Process (Part 1 of 2)
After a summer hiatus, our regular “Thoughts on Verification” blog postings is back. In this edition, Verilab consultant Paul Marriott is interviewed by his colleague Alex Melikian to discuss about DO-254 processes and certification, along with how modern verification practices fit into them. Paul’s industry experience includes verification projects in the aerospace sector, necessitating strict assurance overviews such as DO-254.
In Part 1, Paul describes DO-254, his experience with projects involving it, and what implications it has on the verification process.
Alex Melikian: Hi everyone, welcome to another edition of Thoughts on Verification. I’m pleased to have Paul Marriott with me to talk about the DO-254 process and how it fits in with the verification processes. If you’re not familiar with it, not to worry, I wasn’t very familiar with it either before I started researching for this conversation. Hence, for the sake of our readers, let’s start off by asking you Paul – what is DO-254 in a nutshell?
Paul Marriott: Hi Alex, glad to be here. So: DO-254 in a nutshell is a process assurance flow for civilian aerospace design of complex electronic hardware, and complex electronic hardware basically means ASIC or complex FPGAs. It’s a design assurance process specifically for these aspects of electronic design for civilian aerospace. It is actually document number 254 published by the Radio Technical Commission for Aeronautics - a not-for-profit organization. In the United States, the Federal Aviation Authority requires a DO-254 process is followed and in Europe, the equivalent standard is EUROCAE ED-80. The FAA publishes many guidelines through their CAST papers.
AM: Okay. So we here at Verilab deal, of course, with verification. Can you tell me how some of the aspects of DO-254 generally affect the ASIC or FPGA verification process?
PM: Yes. Most important to note about DO-254 is there’s different levels of certification, depending on the criticality of the components involved. So, for example, for a Design Assurance Level-A certified system, this is a system which, if it fails, will cause death because of complete failure of the aircraft. Whereas, Level-B is a level which may cause death but may not necessarily cause death, but it’s still a critical component; and there’s a level below that called Level-C. So the idea of DO-254 is to have 100 percent confidence at Level-A that the design cannot fail and that every aspect of the design has been verified to 100 percent completeness.
This means that 100 percent code coverage has to be completed – all statements, all branches, all cases, all conditions have to be taken. And every line of design code has to be tied back to a particular requirement in the design specification and so this means that every requirement has to be implemented, and not only that, but every line of design code has to have some function. Therefore, you can’t have lines of design code which don’t have a tieback to a specific requirement.
Of course, the challenge with this is that it’s difficult to write the requirements in the first place to ensure that all of the functionality is actually described. You can write a system with a set of requirements, but you can still miss some critical functionality. Hence one of the key parts of a DO-254 process is verification of the requirements in the first place. Now, DO-254 uses the terms verification and validation rather differently than what we do as verification engineers.
So the process that we would normally call verification in DO-254 is validation where we’re validating that the implementation of the requirements actually meets the specification and that when we do a test that the performance of the design actually meets the intent of the requirement. Whereas, verification is what you do to the requirements to verify that they actually meet the intent of the design in the first place. And you have peer reviews to go through the requirements themselves and verify that they are actually functioning correctly in a proper description of what the design is.
So it’s a double process. It’s a process to verify that the requirement is correct in the first place, and then to validate that the implementation of those requirements in the design actually meets the specification that’s been written.
AM: Hmmm, interesting. Now, I do have more questions related to functional verification and other aspects of modern verification processes, but I’ll get to that later in the interview. First, lets cover the work you’ve done with DO-254 compliant verification projects. Describe what it’s like, and how did you feel it differs from verification project that is not DO-254 compliant?
PM: The first difference I noted was that the specifications were written in a much more rigorous way than the specifications we normally see in the electronic design area. Specifications for a DO-254 process tend to be written in a formal style. So a requirement would be written as, there shall be a certain function – and the word shall means a requirement that has to be implemented and verified. Whereas, unfortunately, lots of the time in verification we see specifications which are written in a more loose way, and it’s up to more the designer’s interpretation as to what the specification is supposed to do.
In theory, in a DO-254 process, you are supposed to have completed all of the specifications before you even start on the implementation and verification – though in practice this doesn’t necessarily happen. But everything has to be complete to 100 percent, so all the design requirements have to be completed; they have to trace down into implementation. They also have to trace down into a verification and validation plan. And so, every function has to have some way of verifying and validating that it’s correct. Sometimes these are by simulations; sometimes these are by inspection, and sometimes these might be actually by running a test on the target hardware.
One other difference with a DO-254 process is that you have to have independence of verification. And so, you can’t trust, necessarily, the output of one tool to be correct. You have to have a way of cross verifying that that tool is producing the correct result. So if you have a design which is synthesized into gates, you might use an equivalence checker to verify that the output of that tool is still a correct description of the RTL that went in. So you always have this chain of checks to ensure that every step of the chain is covered in more than one way.
AM: Yes, I’m starting to see the patterns and mentality behind DO-254. As verification engineers, we often obsess about coverage and particularly requirement traceability. It’s good to see that for mission critical systems, DO-254 imposes a very rigorous process in these areas.
Another remark about DO-254 that somebody looking at it from the outside would say is that it appears to be heavily oriented around auditing. Would you say this characteristic would slow the verification process down, or would it improve the overall quality through the auditing process?
PM: The auditing process in DO-254 is an audit of the process itself. People often get confused as to what DO-254 means and talk about DO-254 certified tools. In general, it’s not the tools that are certified; it’s the overall process because the idea of the process is to have high assurance that the process itself will achieve a design that’s reliable. And so, it’s not any one point in the chain which is certified; it’s the overall process. The auditing part is to ensure that the plans which are created to describe the process are actually checked to make sure that they comply with the objectives of the DO-254 process itself.
And so, there’s all these different stages of involvement called SOIs, meaning ” Stages Of Involvement “, which audit the different steps along the process from the creation of the requirements to the final implementation in hardware. There’s a document called the Plan for Hardware Aspects of Certification. This is where you set down the actual process which describes how you plan to verify and certify that the design you’ve created is actually correct. This would be audited at the first SOI to ensure that this document correctly describes a process which will achieve the assurance levels required for the criticality level of the project.
Of course, with any auditing there is an overhead, but on the flip side, it makes people sit down and go through the code that they’ve created and the different aspects of the process to ensure that it’s correct. And oftentimes in verification of design these kinds of sessions where people go through and do code reviews and documentation reviews don’t take place. So you might think you’re saving time by not doing these, but in the end if you have to do re-spins because you’ve missed out some critical aspects, then it actually takes more time.
So even though there’s an overhead in doing this extra process, if you come up with a process which is repeatable and gives you the assurance that you require, you can actually achieve the objectives with a minimal overhead in terms of time. Of course, any process that requires documentation is going to take more time than one that doesn’t.
AM: Always interesting how this proverbial balancing act in verification appears in different conversations: investing in the right amount of documentation and auditing to gain the maximum amount of quality.
(End of Part 1 of 2)
