You don't have to give up any speed at all because you will probably use a lighter arrow. With all other things being equal, a 60# bow shooting 6 grains/pound will shoot about the same speed as a 70# shooting 6 grains/pound.
There are several good reasons for shooting less pounds and fewer for shooting more, although the energy is a strong selling point. Of course, you do give up some energy but 60# is plenty to zip right through them and the 50-60# bow will probably max out at 62-63 or more.