Thoughts on Verification: The Human Side of Best Practices (Part 1 of 3)
In this edition of Thoughts on Verification, Verilab consultant Alex Melikian talks with Verilab CTO Jason Sprott about the human aspects related to planning or executing a verification project. Jason was invited to DVCon 2013 to participate at one of the panels with other industry leaders and representatives covering the subject of “Best Practices”.
In Part 1, Alex and Jason discuss the concept of “ruthless prioritization” and the differences in practices between FPGA and ASIC development.
Alex Melikian: Hi, Jason. Thanks for joining me on this conversation. Today, we’re going to be talking about the ‘human’ side of best practices in verification. I emphasize ‘human’ in the title because this topic of conversation will not focus so much on tools, technical issues nor coding details. Rather, we take a closer look in the way we do the day-to-day human activities related to verification. I’m talking about things like planning, team coordination and cooperation.
So Jason, you just got back from DVCON 2013, where you were one of the panelists on the discussion of “Best Practices in Verification Planning”. Let’s get started with this conversation with one of the points you greatly emphasized at the panel you called “ruthless prioritization”. Talk about this a little bit more. How should managers or engineers execute this approach of “ruthless prioritization”?
Jason Sprott: Well, I’m glad we’re talking about the human factors because I think they play a major part in team productivity. This “ruthless prioritization”, as I call it, is something that’s difficult to automate, maybe even impossible.
There’s a lot of spin-around things like prioritization, project planning etc. Ruthless prioritization, is when you make sure that as much work as possible goes towards the features and parts of the design that really matter: the ones that the end users are going to use. These are the things that you’re taping-out and people are going to notice if they’re broken. Whereas, what we tend to do in designs, not just verification environments, is to design in a lot of things that may never be required.
There are many reasons we focus on unimportant features, but at the end of the day they can be a major distraction and don’t necessarily matter to the end result. And the problem is we’ve got to verify all those things, or at least spend time considering them. So for me, part of the planning process and the human aspect to the planning process is to ruthlessly prioritize. Not just once at the beginning but continually through the project. The aim is to ensure all the things we’re working are towards the highest priority product goals at the end of the day: the things that really matter. That’s not an easy task, but it’s worth it.
When thinking about the priorities, you have to really consider them at all stages of the development. It’s not something you just do at the very beginning. You’re continually testing: “Is this something I should be working on now?”, or “does this affect something that will definitely make it to the final product?”
If the answer is “no”, you’ve got to ruthlessly throw it out. Otherwise you may be working on stuff that doesn’t matter and you’re just burning development cycles.
AM: I can relate to that: coming from more of an FPGA background myself, I see some of the parallels in the FPGA development process. Not so much, as you said, in ruthless prioritization, but rather setting goals on how much verification you want to exert for each feature. For FPGAs, the name of the game is ‘time to market’. So you can allow yourself to make mistakes before going to the bench in the lab. You don’t have to do 100 percent coverage and test everything down to a ‘t’. Similarly, you have to set priorities.
Of course, there are some parts that are critical, and testing for 100% coverage would be beneficial. However, there are other parts where you can take the risk and aim for 80% coverage in simulation, as long as you have a good bench available in the lab carrying out the exhaustive testing. It’s counterintuitive for us verification engineers to allow a design to go into the lab with the possibility of bugs. However, by carefully allocating some of the validation effort into the lab, I think more often than not, you will achieve complete coverage without running into a bug that requires additional time to debug in the lab. Therefore the overall time would be less, than if you spent the effort to verify everything in simulation with 100% coverage. You know what they say, 80% of effort will be spent chasing the last 20% of coverage. So some time saving can be done there when dealing with an FPGA.
This decision making process of how much verification should be done in simulation also involves continual planning. This means that in mid-project it can be decided that testing a certain feature can get pushed to the lab, or conversely, it becomes necessary to simulate it with 100% coverage.
So this management approach has a lot of parallels to the “ruthless prioritization” process. Do you have any thoughts about that?
JS: Really, it depends on how rigorously you want to verify your design and on the risk you want to take. Just because something can be eyeballed on the bench, doesn’t make it the right way to verify it . I think it’s very, very important in FPGAs to understand what you’re deferring to the bench testing rather than simulation.
The things that you want to push on the bench are things that can’t be done easily in simulation, or things that make more sense for performance, or practical reasons. On the bench, controlling the design to put it in states that exercise all user scenarios can be very difficult. It can often be difficult to exactly duplicate these conditions when regression testing.
It’s just as important to plan and prioritize which features will be verified in real hardware, as it is in simulation. It’s just as easy to waste time on features that are not important to the final product on the bench. In fact, if you factor in repeatability issues, hardware setup, and phony debugging, you can end up wasting a lot of time on unimportant features.
There are also things we can also do in FPGA designs that aren’t typically possible in ASIC, such as building verification modes that can operate in simulation and on the bench, e.g. short frames. The mode is only a mode ever used for verification, and creates something short enough or concise enough to simulate, but it can also be repeated on the bench. Although typically not a real mode of operation, it does exercise areas of functionality very completely, that we would otherwise not be able to simulate if we deferred the whole thing to the bench.
So, don’t just drop stuff onto the bench assuming that the simulations will be too long. Take a more pragmatic approach and analyze what’s actually required, understand the risks you’re taking and find ways to mitigate them.
AM: Definitely agree. Your points emphasize how a lot of coordination is needed between the verification plan and the laboratory validation plan, meaning the planning of what gets tested in the lab. This is absolutely key if the strategy of splitting the burden between simulation and lab testing is to be successful in saving verification cycles.
