I've seen this problem a lot. Most of the time the problem is the metal plate at the top of the chimney is not in fact well sealed. There are big gaps that leak a lot of air. The only way to know for sure is to climb up there and take a look. Have a chimney repair or HVAC professional take a good look.