[NOTE: This entry was written in response to some comments posted to my previous entry "Checks or Functional Coverage?". It was only meant to be a couple of line, but got a bit out of hand ]

In my previous entry "Checks or Functional Coverage?", I made the point that checkers were more important than functional coverage, and that you had to get the checkers done first. Some of the replies said "I agree, but…" and then went on to say that we needed both. I completely agree; the ideal testbench will have checkers and functional coverage. My message here isn’t "do checks and forget about functional coverage". It’s "do the checks for the important requirements before the functional coverage for the important requirements, but do both of these before starting work on the less important requirements, because you might not end up with the time to do it all". That’s not very snappy though, so let’s go with "do the checks first".

The testbench continuum

Testbenches form a continuum, from having checks and no functional coverage at one end, to functional coverage and no checks at the other. Operating at either extreme can be “exciting”. We’ve been called in to help with many projects through our "Chip Rescue^SM" consulting that are operating at the extremes, and it’s inevitably at the wrong extreme. A testbench with checkers and no functional coverage is still useful, but a testbench with functional coverage and no checkers isn’t. This is because correctness checking is mandatory, as it tells you about the quality of the design, while functional coverage is merely extremely nice to have because it only tells you about the quality of your testbench’s stimuli generator. Ideally you’d have both, but if you are going to work at one of the ends, then at least pick the right end.

Let’s look at the FIFO example suggested by two of the commenters. If I have sufficient checkers in my design to catch an error with FIFO full handling (for example, I might have an end to end DUT checker that will flag that something has gone wrong in the design), then in a crunch, I can gain some degree of comfort by running a large bug-hunting batch-simulation. We’re not dealing with Schrödinger’s stimuli here – things will happen whether I measure them or not. I don’t know for sure if the FIFO Full condition will get hit [1] but if the project has made it to the stage where we’re in the either/or situation anyway, then this is probably my most pragmatic solution. Running a bug-hunt with a lot of checkers reduces my overall risk, because if I do happen to stimulate a bug, I will catch it.

On the other hand, if I’m in the same situation but this time have a testbench with functional coverage but no checkers, then knowing that the FIFO full situation is hit does nothing to reduce my risk. I know that the testbench stimuli generator can create that condition, but so what? I still know nothing about the design’s response. This approach hasn’t reduced my risk at all.

Therefore, you have to have the checkers, and you might as well do them first. Cover your bases.

Why operate at extremes

It’s interesting to look at why some projects operate at the extremes. One reason we’ve seen is that people simply didn’t know they were meant to put checkers in. They didn’t even realise there was a continuum. Another reason we frequently encounter is that teams will spend all of their time putting wonderful functional coverage models in place, and then run out of time because it took longer than they thought. They didn’t mean to operate at an extreme, but ended up there by mistake. When they get round to adding the checkers, there’s no time left in the schedule and they face a political problem getting more time. After all, they have been reporting excellent functional coverage, so upper management (who have dutifully read all of the slideware that Janick referred to) don’t see what the problem is.

A third reason is pressure from management who have drunk the Functional Coverage Kool-Aid. It’s well known that "You Get What you Measure", so if management only measure functional coverage, functional is all that they will get. It’s worth reflecting on one project I worked on a few years back. Company A was designing a million gate module to go into an SoC being designed by company B. Due to various project slips, company B was getting upset, and dropping words such as "lawyers" and "legal action" into conversations because company A weren’t supplying functional coverage figures. We were called in to help get functional coverage numbers. It turned out that company A had no automated checks, no verification plan and no verification team. None at all. Functional coverage was the least of their problems, but no one really cared. Functional coverage was what was being measured, so functional coverage is what they had to deliver.

A root cause of all of these is that the functional coverage message is a bit too loud these days, and the checkers message is being drowned out. Based on the material out there, it’s not obvious how new verification engineers, managers, or designers giving it a go, are meant to know about checkers.

Cheers
David

[1] In this simple case, it might be easy enough to put a functional coverage point on the condition itself, but in a larger example, that might be too time consuming. An alternative is to put a functional coverage point on the checker itself triggering, as the functional coverage is simple (the detection complexity is already in the checker), and there will normally be fewer checkers than possible functional coverage points. In this example it would tell us that the FIFO was full, but not why it became full.

This entry was posted by David Robinson on Tuesday, July 24, 2007 at 9:59 am and is filed under Methodology, News. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.