Oh boy. Do you have any natural draft appliances? Old fashioned water heater?
Be very careful depressurizing your house by 640 cfm, or at the very least remove flammables from the vicinity of your water heater!
I'm not an airflow expert but that approach sounds ok for the fan airflow. Someone more knowledgeable about that will probably share their 2c.