The woodpeckers are tearing up the wood to get to the larvae.  How can I scare them off so I can treat and hopefully stop the bee damage?

There is no need to scare them off. All birds are naturally afraid of people. Woodpeckers are especially "flighty" and will quickly leave as soon as you come within their vision. That being said, I do suggest you install some MOVING SPIDERS but only after you treat the carpenter bee holes. This will insure your home doesn't sustain any more damage because the woodpeckers will no doubt stick around awhile since they now know your home is full of food.

More on woodpeckers attacking bee larvae