Ditto to what Cornfed said. Could very well be the patch isn't thick enough to tightly engage the rifling. Or the rifling is so fouled it's like shooting a smooth bore and you're not getting spin to stablize the ball. If you're shooting heavy powder loads, excess powder contributes to fouling.
Conicals or miniballs might help too, but I'd give it a good scrubbing, back off the powder, and head out to the range. Wouldn't hurt to run a patch through between shots.
Good luck!
Threebeards
Seems like late muzzleloader will never get here. But hope the snow holds off until after second shotgun season. Keep 'em in the fields and out of the timber!!!