Humor me and let me blab my opinions as an amateur. I am a homeowner too and must admit I would be timid about going as low as that. Would be very interested in paying a pro to repeat the load calc, and especially do the Manual S calculation to ensure the chosen equipment has enough capacity after considering latent and sensible load separately, equipment de-rating due to temperature conditions, etc.
And I would be very tempted to design the duct system for something extra-large, then 1) you could potentially use higher CFM/ton for sensible efficiency, and 2) an upgrade may be easier should you make an addition, or change your lifestyle, or want to sell to someone who thinks "old school". Keeping the AC at 4, 5 tons or whatever you end up choosing. The reason is that upsizing an AC is fairly common (tho sometimes misguided), while many duct systems are undersized, and I don't know of any substantial downside.
One last way to hedge my bets might be to choose a 2-stage AC which is much less sensitive to oversizing problems. In a weird way having 2 AC systems (most houses here do) contains some of those advantages.
Personally I have 6.5 tons for 3300 sqft in S.Texas, while my own Manual J says 4.5-5.0 is enough (your house will probably be better constructed). I *know* one of my 2 systems is well oversized, by measuring runtimes I estimate I can maintain comfort conditions with 5.5-6.0 tons. Cannot say exactly why the difference, but I have as much confidence in my own observation as I do in a mathematical model, even if that model is the venerable Manual J. In my neighborhood anyway they usually would install around 8 tons, for what that matters.
Best of luck -- Pstu