VHDL Code Coverage

ons mar 30, 2011 (Axcon)

We recently introduced a VHDL code coverage tool in a specific project with some fantastic results, but let me get back to that. Our good friend Anders D over at asym.dk commented on some previous Twitter comments by Michael Bolton on how to report test progress.

Photo by Flickr, Pink SherbetBasically the good question to ask is: What is the value of reporting this and that number of defects or tests not passed. Anders D have some valid points in that a simple number is much easier to communicate at the higher level. And sometimes that is what you need – something that is simple to communicate for next level management. I commented that if a number should be communicated, maybe the defect count is less interesting than a coverage number.

Always quick at the keyboard, Anders D commented on the validity of (any) test coverage number. Again some very valid points – a test coverage number is still a gross simplification of reality. Any test coverage number will still be very much dependant on how you measure. Extreme case: You can easily have 100% DUT code coverage and still have the wrong output from the DUT.

At this point I think it’s valuable to step back from the process and think about the intent. Going back to the basics:

  • We have an intention =>
  • so we use a process =>
  • that helps produce the results

The intention

Our intention is normally to cost effectively create a great product that meets the requirements. In a high wage area, cost effective means low total cost. So that is where high quality becomes really important. The longer the product lives, the more we gain by having high (initial) quality.

Remember: Quality is something we design, not a result of testing.

The process

With the intention of producing high quality, we certainly benefit from putting processes in place that helps us keep focus on what’s important. Automated regression testing is one of those processes. Other processes include good use of reviews, and focus on fixing problems right away rather than logging bugs for later fixing (something our good friend Lars T over at zealake.com has written a nice piece about).

The process of doing automated regression testing is really two processes. One has to do with a constant effort to improve and expand the test suite to efficiently cover the parts of the product developed so far. The other is the actual execution of the regression test – wether this is done nightly, at every check-in or in some other scheme is less relevant.

Defect count vs. Coverage

The defect count comes out of the latter process – each execution of the regression test produces (among a good amount of other things) a number expressing how many tests failed.

The coverage number – expressed in some way or another – comes out of the former process. By improving the test suites, (normally) the coverage goes up. Whenever new parts, features, functions are added to the product being tested, the coverage number (normally) goes down.

Both numbers are interesting, as long as we remember that they are gross simplifications of the much more complex reality.

Reporting the coverage number keep focus on allocating time for improvement of the regression tests. The defect count focuses more on allocating time to fix the bugs. Depending on where the problems are in the team and how things flow, reporting one number or the other would convey more useful information.

I agree with Lars T in that fixing known problems should always be very high priority – so the defect count should always be close to zero. No logging of bugs to fix later.

This even holds for board level hardware development – with a little twist. Whenever a bug is found on a prototype board, you may have to develop two fixes. One fix that can be blue wired on the board, and another fix that make it into the next (and hopefully final) revision of the board. In some cases the here-and-now fix has the form of a work-around, that may also impact software, test suites etc. The best thing is to be able to quickly prototype the final fix somehow – even if not on all available prototype boards.

The Case

We recently did a project related to this exact problem set. The company had a good chunk of new FPGA code written in VHDL to do some well known algorithmic processing and various control functions. The product is a medical device with plenty possibilities to badly harm the innocent patient, if something goes wrong.

In introducing FPGA technology and trying our best to get the teams focus on quality, the ModelSim code coverage features (present in DE/SE versions, but unfortunately not in the cheap OEM versions the FPGA vendors provide) was introduced. Often new FPGA teams starts out without realizing the immense benefits of a well written testbenches for automated regression testing.

By following the code coverage numbers throughout the process, the team immediately grasped the importance and benefits of having code and testbench designed and put in use as a parallel effort. Something other teams we have worked with – even teams working on safety critical systems used in trains – have a very hard time fully committing themselves to.

As a side benefit, having test coverage measured throughout the design cycles, can make some of the formal medical device development requirements easier to meet.

So how _do_ you report test progress?

I guess it depends. It really depends on what you are trying to achieve. And by that I mean as someone observing, influencing or leading the team, you have to look at where the team need to focus. In the case with a small and fairly immature FPGA team (no offence), we had a tremendously good result by focusing on code coverage – when all your efforts had to be reduced to one word (in this case a number). In the daily work, much more detail can be added and the code coverage number becomes the “executive summary”.

Kommentarer (4)

  1. Anders Enggaard added on 30. marts 2011

    I’ve always tended to say that you cannot force the quality into a product by execution of more tests. You can justify that the design quality is present by simple judgement of the growth of errors detected….

    If the errors grow rapidly at any stage when you perform the tests – then you are better of by scrapping the design and re-think the architecture and basic design. So how does this apply the the above?

    In my mind the metrics of coverage has the strength of exposing the lack of quality in the tests and in this case a poor performance of the tests will result in low or no increase of the coverage numbers – then you have to rethink your test strategy.

    In other words – coverage is no magic but can be beneficial to expose lack of quality in the tests and at the same time improve our ability to expose lack of quality in the design itself.

  2. Anders Dinsen added on 30. marts 2011

    Thanks for the response. YOU are quick at the keyboard – response next morning, that’s fast :-)

    Knowing you guys it doesn’t surprise me that you know that test is not producing quality! You’ll be surprised to find how many people actually DO think quality comes from testing. I was!

    No testing only produces /knowledge/ about quality. Quality comes from design and fixing problems.

    I know you are making products with extreme technical complexity – and you obviously know how to handle the development process. I’m not surprised you find automated regression tests are necessary.

    In my perspective, what you are doing is to do critical thinking: “What can go wrong here? Ok, I’ve got some ideas – let me check that!”

    Because things do go wrong.

    Back at the start of the second Iraq war, Donald Rumsfeld identified two kinds of lack of knowledge: Known unknowns and Unknown unknowns.

    The former are the ones we can test for (if we care – some people don’t!). But we can’t test for things we don’t know. Or can we?

    Actually, we can. It takes exploration, examination, and further exploration. It’s not unstructured, but the actual result is unpredictable.

    But even if it’s unpredictable, I have yet to see that method fail producing new and interesting knowledge. A great example happened today for me when my new Indian colleague ran a test which hit a bug which caused our test system to “melt down” completely in a very interesting way! That even took me by surprise :-)


  3. Rolf added on 30. marts 2011

    Thanks for the comments.

    Pseudo random test data, ordering etc. may be a crude – but very useful – way of testing for some types of unknown unknowns.

    Oh, no… maybe they just became known unknowns when you think about it :-)

  4. Jakob Eriksson added on 5. april 2011

    I think this is no different from how testing is done in the software world. I think in general, software guys have a lot to learn from hardware guys in taking testing seriously. Harm may come from broken products! Hardware guys probably have a lot to learn from software guys when it comes to integration of tools and sharing of knowledge. Just my 5 cents… thanks for sharing this blog post!

Hvad mener du?