Manual J for Roanoke, VA has your summer outdoor design temperature at 91 degrees with a coincident wet bulb of 72. In layman's terms this means that someone in your area designing an HVAC system by Manual J standards will operate under the assumption that your cooling system will keep your house comfortably at 75 degrees/50% relative humidity when it is 91 degrees at 40% relative humidity outside your house (known as a "design day" in engineering parlance). For winter, this same designer will size your furnace or other heating source around a 16 degree day.
A Manual J done well accounts for how leaky your house is...which is what the blower door info should do for you. It also attempts to make an informed estimation of how much heat will enter into your house in summer or leave the house in winter during the above named "design day" conditions, and how big your HVAC needs to be to keep you comfortable during those extremes in temperature and humidity.
The only "padding", IMO that should occur arising from a thorough Manual J calculation is if the results render a cooling BTU requirement somewhere in the gray zone...IOW if the Manual J calls for 4.45 tons of cooling, a five ton unit should be chosen because there are no 4.5 ton units made. For heating, if you are talking natural gas I'd seriously look at modulating furnaces, with the "high fire" stage sized according to your Manual J results for maximum expected heating demand. If you're looking at a heat pump your designer will need to know how much backup heat you need to meet your 16 degree day because heat pumps are sized for the cooling load, not heating.
The hazard you may come across, and it is unfortunately a common occurance, is finding a residential HVAC contractor who is comfortable with Manual J's. Most do not like to deal with it, thinking either it's a waste of time and their experience is sufficient, or they may not know how to do a Manual J at all. The ones who do tend to be more expensive, but often they deliver a better installation and stand behind their work. That's what you want in the long run. You want right-sized equipment, a quality installation, and a contractor who stands behind their work. If you find this person or company and they hit well on all three points, shout their praises from the rooftops so others in your area can benefit from their expertise.
Psychrometrics: the very foundation of HVAC. A comfort troubleshooter's best friend.