VHDL Code Coverage

We recently introduced a VHDL code coverage tool in a specific project with some fantastic results, but let me get back to that. Our good friend Anders D over at asym.dk commented on some previous Twitter comments by Michael Bolton on how to report test progress.

Basically the good question to ask is: What is the value of reporting this and that number of defects or tests not passed. Anders D have some valid points in that a simple number is much easier to communicate at the higher level. And sometimes that is what you need – something that is simple to communicate for next level management. I commented that if a number should be communicated, maybe the defect count is less interesting than a coverage number.

Always quick at the keyboard, Anders D commented on the validity of (any) test coverage number. Again some very valid points – a test coverage number is still a gross simplification of reality. Any test coverage number will still be very much dependant on how you measure. Extreme case: You can easily have 100% DUT code coverage and still have the wrong output from the DUT.

At this point I think it’s valuable to step back from the process and think about the intent. Going back to the basics:

• We have an intention =>
• so we use a process =>
• that helps produce the results

The intention

Our intention is normally to cost effectively create a great product that meets the requirements. In a high wage area, cost effective means low total cost. So that is where high quality becomes really important. The longer the product lives, the more we gain by having high (initial) quality.

The process

With the intention of producing high quality, we certainly benefit from putting processes in place that helps us keep focus on what’s important. Automated regression testing is one of those processes. Other processes include good use of reviews, and focus on fixing problems right away rather than logging bugs for later fixing (something our good friend Lars T over at zealake.com has written a nice piece about).

The process of doing automated regression testing is really two processes. One has to do with a constant effort to improve and expand the test suite to efficiently cover the parts of the product developed so far. The other is the actual execution of the regression test – wether this is done nightly, at every check-in or in some other scheme is less relevant.

Defect count vs. Coverage

The defect count comes out of the latter process – each execution of the regression test produces (among a good amount of other things) a number expressing how many tests failed.

The coverage number – expressed in some way or another – comes out of the former process. By improving the test suites, (normally) the coverage goes up. Whenever new parts, features, functions are added to the product being tested, the coverage number (normally) goes down.

Both numbers are interesting, as long as we remember that they are gross simplifications of the much more complex reality.

Reporting the coverage number keep focus on allocating time for improvement of the regression tests. The defect count focuses more on allocating time to fix the bugs. Depending on where the problems are in the team and how things flow, reporting one number or the other would convey more useful information.

I agree with Lars T in that fixing known problems should always be very high priority – so the defect count should always be close to zero. No logging of bugs to fix later.

This even holds for board level hardware development – with a little twist. Whenever a bug is found on a prototype board, you may have to develop two fixes. One fix that can be blue wired on the board, and another fix that make it into the next (and hopefully final) revision of the board. In some cases the here-and-now fix has the form of a work-around, that may also impact software, test suites etc. The best thing is to be able to quickly prototype the final fix somehow – even if not on all available prototype boards.

The Case

We recently did a project related to this exact problem set. The company had a good chunk of new FPGA code written in VHDL to do some well known algorithmic processing and various control functions. The product is a medical device with plenty possibilities to badly harm the innocent patient, if something goes wrong.

In introducing FPGA technology and trying our best to get the teams focus on quality, the ModelSim code coverage features (present in DE/SE versions, but unfortunately not in the cheap OEM versions the FPGA vendors provide) was introduced. Often new FPGA teams starts out without realizing the immense benefits of a well written testbenches for automated regression testing.

By following the code coverage numbers throughout the process, the team immediately grasped the importance and benefits of having code and testbench designed and put in use as a parallel effort. Something other teams we have worked with – even teams working on safety critical systems used in trains – have a very hard time fully committing themselves to.

As a side benefit, having test coverage measured throughout the design cycles, can make some of the formal medical device development requirements easier to meet.

So how _do_ you report test progress?

I guess it depends. It really depends on what you are trying to achieve. And by that I mean as someone observing, influencing or leading the team, you have to look at where the team need to focus. In the case with a small and fairly immature FPGA team (no offence), we had a tremendously good result by focusing on code coverage – when all your efforts had to be reduced to one word (in this case a number). In the daily work, much more detail can be added and the code coverage number becomes the “executive summary”.

Tags: code coverage, kvalitet, testbench
Kategori: Debat
Meta: ping/trackback | RSS 2.0