I think they sure help Jared Allen, but he's a man of his own. Just look at his stats in KC when he had no help.
About the D being average or bottom, no doubt it'll be average/good without them. But your whole scheme is based on running the ball and great D, because you don't have a competent qb to win games for you. So, though you have a top 5/best RB in the league and a great Oline, one team cannot just run the ball if the D don't shut down the other team IMO. And say their suspension interfere with 2,3 losses. In the close contest that'll be the NFC North, 2,3 losses are the difference between being or not in the playoffs, IMO.
BTW, about the QB thing, IF #4 plays, he may help against us (and even then I doubt it), but what'll happen when he isn't motivated (by motivated read blindly enraged)?