first things first
If you open a basement window, the cold air will quit coming in around the fireplace. Before you do a Vega-matic on the fireplace, fix the house. You obviously have a negative pressure problem or else the outdoor air would not be trying to get inside. Seal upper level penetrations first. Provide makeup air for all powered exhaust fans. Seal and balance ductwork. Ensure full coverage with insulation.
As for the fireplace, you can get some cold air in even if the vent termination is properly sealed to the outside wall through the seams in the DVP pipe. If this is an older unit, there would need to be a fiberglass rope gasket around the vent pipe siliconed to the rear outlet of the fireplace. If the wall over the mantel is insulated but the outer perimeter is not, I recommend eventually weatherizing these walls as JTP discussed. This entails foaming and caulking as many seams and gaps as possible. Then, fiberglass batts without Kraft paper in the stud bays. On this sheath with Thermoply with seams sealed with UL 181-B/ FX foil tape.
Around here, this work can cost 10X the price JTP quoted. Understand, he works for a major mfr.s installing distribution channel where pricing is geared towards builders and not retro. While his Division sells retail the company's pricing is low margin for the most part. To do this job properly, I would pull the fireplace from the inside, weatherize, re-install with RTV silicone on the outer pipe joints and firestops then seal it up. Note: you cannot pack insulation in the space btw the top of the Fp and the header, which is why it is useless to insulate over the mantel---it leaves a huge uninsulated hole in the thermal envelope of the home.
back from EXPO
Keep the fire inside the fireplace.
With the outside of the house being stucco, I would probably do the job from the inside. You can do a better job from the inside and drywall or cultured stone might be easier to repair than stucco.
Thanks to all of you guys, my fireplace is fixed.
The builder did it from the inside : removed the stones, the fireplace, added insulation and drywall who were totally missing and also added an air intake from the outside on the basement furnace to stop the negative pressure.
OK it is not winter yet but at least the temperature under the fireplace is the same that the house not the outside one!
Thanks again for all your help
Thanks for posting back. Its great to hear some of our advice helped someone out.
I bet it was a lot easier to convince the builder to fix it, when you had so much info about the problem.
Hopefully the builder has learned something from this as well, and will make sure it is done properly on his next house the first time around.