Feed on

Be afraid. Be Very Afraid

Most verification engineers burn themselves at some point by disabling a checker and then forgetting about it. There are sensible reasons for doing this; think about it. You find an RTL bug on Friday, but it doesn’t get fixed immediately. You decide to comment out the checker in order not to pollute the weekend’s regression run. The problem is that you come in on Monday morning and start debugging the new errors you have. The commented out check gets forgotten about.

Burned? I positively set myself on fire doing this on my first ever project. I spotted the commented out check 3 days before code freeze. And guess what it was masking a bug. Ouch.

I haven’t made the same mistake again. In fact, I go to excessive lengths to check that my testbench works correctly. I talk about one method in my book [1], where I create a special aspect that I load up at the start of regressions to verify that the testbench works before I run all of the other simulations. Another method I use is known as error injection (or fault injection, bug injection or mutation) where I’ll deliberately go and break the RTL and check that my testbench catches it.

The problem with this approach is that it can be manually intensive. Determining where the best place to inject a bug is, running an entire regression to see if it is caught, and repeating until you are happy you’ve done enough (and really, how do you know?) is tough.

Not any more though. I caught the Certess demo yesterday, and they seem to have solved the problem. I only saw a demo, but their solution looks pretty push button. You load the design into their tool, it runs a regression to profile your tests and work out which faults should get caught by which tests, and then it injects the faults one at a time and runs the appropriate tests. If they don’t complain about errors, then you have a problem with your testbench.

As far as I know, this is the first time that we’ve been able to measure the quality of a verification environment. So all you verification engineers, IP providers and outsourcing companies out there – be afraid. This thing will tell you what functionality your stimuli isn’t activating, what functionality it isn’t propagating, and what bugs you aren’t detecting. Your boss and customers can now find out how good a job you are really doing.


[1] D. Robinson, “Aspect-Oriented Programming with the e Verification Language: A Pragmatic Guide for Testbench Developers”

Comments are closed.

Work For Verilab