Feed on

Archive for September, 2013

Thoughts on Verification: Verification in the DO-254 Process (Part 2 of 2)

Thursday, September 19th, 2013 by Alex Melikian

In Part 2, Verilab consultants Paul Marriott and Alex Melikian continue their discussion on verification in a DO-254 quality assurance process. In this part of the conversation, they focus on the technologies, tools and methodologies that intersect DO-254 and verification. Part 1 can be viewed here.

Alex Melikian: Let’s move on to talk about some of the tools available tailored for a DO-254 process available in the industry. A name that seems to come up regularly is DOORS. What is DOORS?

Paul Marriott: DOORS is a tool, which stands for “Dynamic Object-oriented Requirements System”. It’s a tool which allows you to capture requirements and other documents, which are required for a DO-254 process, and to put in the traceability between the different levels of documentation. So every design requirement has to trace through to a design implementation; it has to trace through into a validation plan; it has to trace through into a test case. You have to describe a procedure, what you’re going to do, and what the acceptance criteria are. This ensures that every requirement in the design is actually implemented and validated.

And DOORS is a tool which allows you to manage these databases and provide the links between the different levels of documentation in a way that’s easy to see in areas where you might’ve missed out.

AM: It appears DOORS is the stalwart tool for the DO-254 certified process. So how do some of the verification tools out there that support or work with DOORS interface with it?

PM: DOORS will allow you to import and export via spreadsheets, and if you think of the way DOORS works, essentially, it’s a almost like a spreadsheet because you have a table of requirements and then a link from each requirement to another document, which may be a validation plan or a test description. And so, you could, I mean DO-254 itself says nothing about the tools that you have to use. You can achieve DO-254 certification with pencil and paper. As long as the process is repeatable and traceable and is described and audited properly, then you can do it. But of course, automation makes it much more efficient and much easier.

DOORS is not really designed to interface directly with verification tools. With modern verification, we’re using assertion-based and coverage-driven verification, and you can actually implement the validation of many of the requirements by your assertions. And so, you want to ensure that a particular assertion is tied back to a particularly requirement. Therefore, you may be able to export the set of requirements from DORS into a spreadsheet and import that into another tool, which then ties into your coverage plan, which is then linked into your simulator’s verification manager.

AM: There appears to be a lot of strong industry support behind it. Some of the major EDA tool vendors have features that interface directly to a DOORS tool, so it’s good to know that you’re not stuck with one specific vendor or tool provider.

PM: Yes, exactly. People think it’s the tools themselves which are certified, but it’s not. It’s the overall process which uses the tools. So DO-254 itself doesn’t mandate any particular tools. Of course, certain tools can make it easier to achieve the goals of DO-254, but it doesn’t mandate them.

AM: Okay. Now, the DO-254 process goes back a long way. Earlier (in Part 1), we mentioned we would talk about some of the modern verification tools and processes and how they relate to this subject. So how do things like coverage-driven testbenches or constrained random generation fit into a DO-254 process?

PM: This is one of the challenges with the DO-254 process. Because everything is requirements-based, the temptation is to write one test per requirement. And so, this leads to a very directed testing methodology, which fits in well to the traceability because you can trace one test back to one requirement. But with a modern process, in terms of verification that tends not to be the case. We tend to have fewer test cases and put in functional coverage and assertions to cover much functionality per test run. DO-254 is fine with this, as long as you can trace back those assertions and functional coverage points to actual requirements.

And the traceability tools allow you to ensure that every requirement does in fact trace down to a cover point, a test result or an assertion point.

AM: That’s good to know, especially for those who have been taking more of a directed test case suite approach to achieve their objectives, they can now be persuaded that modern verification paradigms and tools out there are still compatible with DO-254. There are other dynamics involving the management of a verification project like Agile. How does that fit into the DO-254 process?

PM: That’s a good question. Process and project management are not necessarily the same thing. The process describes things that have to be done to get you from design specification to verified implementation. Project management is the steps that you take along the way to ensure that the work gets done. Now, with DO-254 having various stages of involvement in terms of audits, it does impose a somewhat rigid structure on the order that things are done. But there’s nothing to say that along the steps of meeting those milestones that you can’t use an Agile approach.

At the end of the day, with any kind of project management, you still have to cover all of the work that has to be done to reach a certain milestone, whether it’s agile or whether it’s a standard waterfall approach or any other project management approach. And I’ve worked with companies doing DO-254 certified processes that are quite happy using a Kanban style of Agile project management to achieve the goals of the overall process.

AM: I see, thanks for clearing that up for me. So let’s set the record straight for our readers: Agile deals with project management; whereas, DO-254 is a process flow.

PM: Exactly.

AM: So does DO-254 only affect firms involved in aerospace or mission-critical products, or are there firms outside of these industries that can also adopt it?

PM: DO-254, by definition, is purely for certification of complex electronic hardware for civilian aerospace. That’s its definition. That said, it doesn’t mean that the DO-254 style of process can’t be used for other mission-critical products, and there are groups working – particularly in the automotive area – on similar processes to DO-254 to achieve the high process assurance that’s required. If you look at some of the major automotive recalls that have been done in the past few years, some of the costs of the recalls have been tens of billions of dollars because there’s been bugs found in the software or the hardware, which has caused some problems.

I mean the aim with DO-254 is to ensure that when you’re designing a Level-A criticality system it will not fail and people don’t get killed. That said, you can still have a 100 percent certified Level-A system which can still fail because somebody didn’t think of something in the requirements. So the critical aspect of any higher reliability, high assurance process is the people involved ensuring that they write good requirements in the first place. And that’s really the hardest part of any mission-critical process.

And other mission-critical areas can definitely learn from the DO-254 experience. There’s also DO-178B, which is used for the software aspects of civilian aerospace, which also has the same criticality levels as DO-254 because now with systems becoming more complex from a software point of view, you still want to have the same assurance that the software is reliable and safe.

AM: That’s a good point you made about noncritical systems taking an approach of building critical systems designs. We can all agree that though your car may have an airbag, you’d still like the brake pedal to work 100% of the time. So if a firm is not obligated to adopt a DO-254 certified process for whatever reason, what are some of the things that it can take away from the process just the same?

PM: There’s several important aspects, which I think can be taken away for everybody. The first is to have a good set of design requirements. If you have a good specification, then you have a much higher likelihood of success at producing a correctly working product, which has been essentially verified to be reliable. So the process itself ensures that the requirements are written in a formal way. And the process also describes the traceability between the requirements, the verification plan and the implementation and also ties in code coverage and other aspects to verify that what was implemented was in fact what was intended.

Code coverage tells you nothing about the quality of the code, but if you don’t have 100 percent code coverage, either you’ve missed out verifying some requirement, as requirements were implemented without being specified – or that the testing wasn’t sufficient to cover all of the implementation. So it’s very important to look at code coverage, even though it tells you nothing about the code quality. And just having the formal audit process so people sit down and actually review the work that’s been done is also important.

Because DO-254 is high assurance, they really focus on peer reviews of the requirements themselves, the verification of the requirements, and the validation to ensure that there’s no single point of failure in the entire process. This is why, in fact, there’s no DO-254 certified tools because you want to make sure that each tool is covered by another process or another tool to ensure that there’s just no one single point of failure in the entire process.

AM: That’s interesting, but I can certainly understand the thinking behind that. On that note, I believe we’ll conclude this edition of “Thoughts on Verification”. I hope our readers not only feel safer when they board an airplane, but have also learned a few things about the DO-254 process. Thanks for your time Paul.

PM: Thank you Alex for some very interesting questions.

Thoughts on Verification: Verification in the DO-254 Process (Part 1 of 2)

Wednesday, September 11th, 2013 by Alex Melikian

After a summer hiatus, our regular “Thoughts on Verification” blog postings is back. In this edition, Verilab consultant Paul Marriott is interviewed by his colleague Alex Melikian to discuss about DO-254 processes and certification, along with how modern verification practices fit into them. Paul’s industry experience includes verification projects in the aerospace sector, necessitating strict assurance overviews such as DO-254.

In Part 1, Paul describes DO-254, his experience with projects involving it, and what implications it has on the verification process.

Alex Melikian: Hi everyone, welcome to another edition of Thoughts on Verification. I’m pleased to have Paul Marriott with me to talk about the DO-254 process and how it fits in with the verification processes. If you’re not familiar with it, not to worry, I wasn’t very familiar with it either before I started researching for this conversation. Hence, for the sake of our readers, let’s start off by asking you Paul – what is DO-254 in a nutshell?

Paul Marriott: Hi Alex, glad to be here. So: DO-254 in a nutshell is a process assurance flow for civilian aerospace design of complex electronic hardware, and complex electronic hardware basically means ASIC or complex FPGAs. It’s a design assurance process specifically for these aspects of electronic design for civilian aerospace. It is actually document number 254 published by the Radio Technical Commission for Aeronautics - a not-for-profit organization. In the United States, the Federal Aviation Authority requires a DO-254 process is followed and in Europe, the equivalent standard is EUROCAE ED-80. The FAA publishes many guidelines through their CAST papers.

AM: Okay. So we here at Verilab deal, of course, with verification. Can you tell me how some of the aspects of DO-254 generally affect the ASIC or FPGA verification process?

PM: Yes. Most important to note about DO-254 is there’s different levels of certification, depending on the criticality of the components involved. So, for example, for a Design Assurance Level-A certified system, this is a system which, if it fails, will cause death because of complete failure of the aircraft. Whereas, Level-B is a level which may cause death but may not necessarily cause death, but it’s still a critical component; and there’s a level below that called Level-C. So the idea of DO-254 is to have 100 percent confidence at Level-A that the design cannot fail and that every aspect of the design has been verified to 100 percent completeness.

This means that 100 percent code coverage has to be completed – all statements, all branches, all cases, all conditions have to be taken. And every line of design code has to be tied back to a particular requirement in the design specification and so this means that every requirement has to be implemented, and not only that, but every line of design code has to have some function. Therefore, you can’t have lines of design code which don’t have a tieback to a specific requirement.

Of course, the challenge with this is that it’s difficult to write the requirements in the first place to ensure that all of the functionality is actually described. You can write a system with a set of requirements, but you can still miss some critical functionality. Hence one of the key parts of a DO-254 process is verification of the requirements in the first place. Now, DO-254 uses the terms verification and validation rather differently than what we do as verification engineers.

So the process that we would normally call verification in DO-254 is validation where we’re validating that the implementation of the requirements actually meets the specification and that when we do a test that the performance of the design actually meets the intent of the requirement. Whereas, verification is what you do to the requirements to verify that they actually meet the intent of the design in the first place. And you have peer reviews to go through the requirements themselves and verify that they are actually functioning correctly in a proper description of what the design is.

So it’s a double process. It’s a process to verify that the requirement is correct in the first place, and then to validate that the implementation of those requirements in the design actually meets the specification that’s been written.

AM: Hmmm, interesting. Now, I do have more questions related to functional verification and other aspects of modern verification processes, but I’ll get to that later in the interview. First, lets cover the work you’ve done with DO-254 compliant verification projects. Describe what it’s like, and how did you feel it differs from verification project that is not DO-254 compliant?

PM: The first difference I noted was that the specifications were written in a much more rigorous way than the specifications we normally see in the electronic design area. Specifications for a DO-254 process tend to be written in a formal style. So a requirement would be written as, there shall be a certain function – and the word shall means a requirement that has to be implemented and verified. Whereas, unfortunately, lots of the time in verification we see specifications which are written in a more loose way, and it’s up to more the designer’s interpretation as to what the specification is supposed to do.

In theory, in a DO-254 process, you are supposed to have completed all of the specifications before you even start on the implementation and verification – though in practice this doesn’t necessarily happen. But everything has to be complete to 100 percent, so all the design requirements have to be completed; they have to trace down into implementation. They also have to trace down into a verification and validation plan. And so, every function has to have some way of verifying and validating that it’s correct. Sometimes these are by simulations; sometimes these are by inspection, and sometimes these might be actually by running a test on the target hardware.

One other difference with a DO-254 process is that you have to have independence of verification. And so, you can’t trust, necessarily, the output of one tool to be correct. You have to have a way of cross verifying that that tool is producing the correct result. So if you have a design which is synthesized into gates, you might use an equivalence checker to verify that the output of that tool is still a correct description of the RTL that went in. So you always have this chain of checks to ensure that every step of the chain is covered in more than one way.

AM: Yes, I’m starting to see the patterns and mentality behind DO-254. As verification engineers, we often obsess about coverage and particularly requirement traceability. It’s good to see that for mission critical systems, DO-254 imposes a very rigorous process in these areas.

Another remark about DO-254 that somebody looking at it from the outside would say is that it appears to be heavily oriented around auditing. Would you say this characteristic would slow the verification process down, or would it improve the overall quality through the auditing process?

PM: The auditing process in DO-254 is an audit of the process itself. People often get confused as to what DO-254 means and talk about DO-254 certified tools. In general, it’s not the tools that are certified; it’s the overall process because the idea of the process is to have high assurance that the process itself will achieve a design that’s reliable. And so, it’s not any one point in the chain which is certified; it’s the overall process. The auditing part is to ensure that the plans which are created to describe the process are actually checked to make sure that they comply with the objectives of the DO-254 process itself.

And so, there’s all these different stages of involvement called SOIs, meaning ” Stages Of Involvement “, which audit the different steps along the process from the creation of the requirements to the final implementation in hardware. There’s a document called the Plan for Hardware Aspects of Certification. This is where you set down the actual process which describes how you plan to verify and certify that the design you’ve created is actually correct. This would be audited at the first SOI to ensure that this document correctly describes a process which will achieve the assurance levels required for the criticality level of the project.

Of course, with any auditing there is an overhead, but on the flip side, it makes people sit down and go through the code that they’ve created and the different aspects of the process to ensure that it’s correct. And oftentimes in verification of design these kinds of sessions where people go through and do code reviews and documentation reviews don’t take place. So you might think you’re saving time by not doing these, but in the end if you have to do re-spins because you’ve missed out some critical aspects, then it actually takes more time.

So even though there’s an overhead in doing this extra process, if you come up with a process which is repeatable and gives you the assurance that you require, you can actually achieve the objectives with a minimal overhead in terms of time. Of course, any process that requires documentation is going to take more time than one that doesn’t.

AM: Always interesting how this proverbial balancing act in verification appears in different conversations: investing in the right amount of documentation and auditing to gain the maximum amount of quality.

(End of Part 1 of 2)

Work For Verilab