I like Olson too. He's a good color man. But you're comparing a game announcer to a color man, and they are completely different jobs. As an example, John Madden was a color man who teamed well with announcers like Pat Summerall and Al Michaels. It's the combination of the two that makes it work.
I learned over the years that each announcer brings something unique to the table and those that I like the most are those that created more opportunities for their color man to interject information that's interesting to the broadcast. In Harlan's case, combining with John Kuhn in Packer preseason games is a decent combination because Kuhn is pure "homer" and it's acceptable. When Harlan is doing national broadcasts, he's been teamed with Trent Green. As a team, I consider them middle of the pack. Nothing special.
Harlan's career did not start because of his father. Bob was associated with the St. Louis Cardinals baseball until 1971 I believe, when he joined the Packers in an executive role. Kevin was actually the broadcaster for Chicago Bears preseason games until 2003, and he came to the Packers because he was invited due to his ties with the organization which included being a ball boy for them as a kid. Preseason games are not broadcast with the intent of being unbiased, the teams themselves decide whom it should be not the networks, so yes, to a degree, being Bob Harlan's son did bring him back to the Packers. His roots, as a kid were there. He was already a national broadcaster for the NFL, NBA, and college basketball games. He was a well-known commodity. Actually, a pretty good choice.
What I think you don't like is his style. He's not a flamboyant announcer. He just kind of goes with the flow and lets the color man do his work. I'd bet he would have made a great combination with John Madden back in the day. Madden would have feasted on the combination.