In your sites stylesheet (style.css), the snippet copied below will add an underline only when a link is active or hovered (remove and/or change values to suit your site of course - the important part is in bold)
A:active { font-size: x-small; color: rgb(153,0,0); font-weight: bold; text-decoration: underline;}
A:link { font-size: x-small; color: rgb(0,0,0); font-weight: bold; text-decoration: none ;}
A:visited { font-size: x-small; color: rgb(114,114,114); font-weight: bold; text-decoration: none ;}
A:hover { font-size: x-small; color: rgb(153,0,0); font-weight: bold; text-decoration: underline;}
hth
--dlp