White's use of racial stereotypes to illustrate his point was appalling, and there was a pretty big media backlash, and I'm not talking about Hollywood or traditionally liberal news outlets. Many middle-of-the-road newspapers offered up editorials slamming some of the things he said, for example. When you say things like "You can see a Hispanic person and they can put 20 or 30 people in one home", or "If you go to Japan or any Asian country, they can turn a television into a watch" to try to explain your theory on God's different gifts to different ethnic groups, I don't think it's a stretch to say most people will think that sounds racist.
In addition, his remarks in that speech regarding homosexuality were so bad,
even the Republican legislator who invited him to speak (Scott Jensen)
said he was "disappointed" by his remarks.
White also had some, er, "interesting" things to say about corporal punishment that have been thoroughly discredited by modern child psychology ("I don't need to hit my kids more than five times when I whoop them, because I don't want to go and get angry at them").
If people are inclined, take a look at the transcript of his speech here, and then draw your conclusions:
http://docs.legis.wisconsin.gov/1997/related/journals/assembly/98_04_01/_76
Do I think White was a really bad guy who acted like some sort of medieval racist? No. But he did say some racist and bigoted things that were an extension of his religious beliefs. And not all Packer fans were cool with that. But we all have our foibles, and White did do a lot of good things on and off the field, as we all know. The bottom line is, it's really what happens on the field regarding players that draw Packer fans of all stripes together, and not (sometimes especially not) what happens off.