Depending on whether you're feeling cold air through all the vents or just some could help determine the exact location of the air leaks but the bottom line is that the cold air is coming in through leaks in the system. You didn't state as to the exact location of the vents in question, floor, wall or ceiling. Most houses leak air and the majority do so through what's called 'stack effect' of warmer air in the home rising up and finding exit points through leaks in the living envelope. As each molecule of air leaves the home, a new molecule must come in at some other location. This keeps the house in balance with the outdoors regarding atmospheric pressure. We refer to the escaping air as exfiltrating and the incoming air as infiltrating. The faster air exfiltrates, the faster it must also infiltrate. When it gets sufficient to feel the effects, it's beyond time to tighten up the envelope (the living space as opposed to an unfinished attic, as an example.
If your duct system is under the house, then the standard stack effect is at play. If, however, your ducts are overhead, that would change the troubleshooting aspects of determining the leak location(s) without testing or infrared photographs. Air can infiltrate due to high winds or more commonly as a result of forced exfiltration. The forced exfiltration is caused by powered exhausts, such as range hoods, dryer vents, bathroom vents, and leaking ducts when the blower is operating. Natural exfiltration other than small leaks is most commonly the result of fossil fuel appliances connected to a chimney, such as fire places, gas or oil water heaters and furnaces or boilers. Many heating appliances operating as non-direct vent units, also use air from the home for combustion purposes. This again results in air needing to infiltrate to compensate for that which vents up the chimney.
There a many varied solutions to curb the exchange of air in the home but the best method of identifying the losses and determining when mechanical ventilation is needed is through the use of a blower door test and ideally the added aid of an infrared photograph of the sides of your home. Once repairs are made reducing the unbridled air exchange through the house, a second blower door test should be done to be certain that the exchange rate hasn't been reduced to what could be unhealthy levels. If unhealthy rates are recorded, then mechanical ventilation would be required. That is the more ideal situation as you now have control over the exchange rate rather than just letting it run willy-nilly. Aren't you glad you asked? Isn't this fun?
If YOU want change, YOU have to first change.
If you are waiting for the 'other guy' to change first, just remember, you're the 'other guy's' other guy. To continue to expect real change when you keep acting the same way as always, is folly. Won't happen. Real change will only happen when a majority of the people change the way they vote!