I contacted the manufacturer and got this response:
With any positive pressure vent, an opening in the vent pipe can spill flue gases into the home. The test port should be sealed. Health concerns can be an issue if the oil burner combustion is not correct.
So, DavidInAustin, where's my trillion dollar lotto ticket and my name in the 2008 ballot?
After I complete a combustion test, I put a piece of foil tape over the hole. When I remove it for the following year's service, I generally see a brownish color on the inside of the tape but not black. Since there's exposed adhesive on the flue side of the tape, it will attract particulate. Get it tested, plug the hole. You've probably already figured that out by now, I'm sure.