60-80 cubic feet per min.(CFM) is estimated to be adequate fresh air to purge pollutants, replace oxygen, and remove moisture for 4-5 people. Most 4,300 sqft. homes infiltrate/exfiltrate 60-80 CFM of air during the coldest windy weather because of stack effect and wind pressure. On a windless, warm day, even a leaky home does not breath. Your home has a clothes drier, kitchen hood, bath fans, water heater, space heater, central vac, and fireplace needing make-up air. A controlled amount of make-up air ventilation supports the functions of these appliances, like 60-80 CFM comes close. Operating an ERV/HRV typical homes during cold weather over ventilates . During neutral weather there is no heat transfer benefit. ERV/HRVs are great for large quantities of continuous fresh in very cold climates with expensive heat sources like electric resistance. Also ERVs in schools with high ventilation rates in humid climates have payback.
Your climate needs 135 pints of dehumidification per day to deal with the moisture from the occupants and exhaust appliances/make-up air ventilation during the wet times of the year, May-Sept. On the hot days, the a/c will remove the moisture. As you suggest, a whole house 135 pint dehumidifier would deal with the remaining moisture to maintain <50%RH to control mold/dust mites. A make-up air ventilating dehumidifier with a MERV 11 air filter will deal with the year around fresh/make-up air ventilation. Use good bath fans and a moderate sized kitchen hood. All combustion appliance except the kitchen stove should be sealed combustion including fireplaces.