Robotman, if you are setting your thermostat higher during the day when nobody is home, I would consider eliminating that. You need consistent humidity control inside your house to minimize allergen bloom. Setback strategies work against that goal. I would also consider TB's advice regarding ventilating dehumidifiers for a long term solution.
Building Physics Rule #1: Hot flows to cold.
Building Physics Rule #2: Higher air pressure moves toward lower air pressure
Building Physics Rule #3: Higher moisture concentration moves toward lower moisture concentration.