You won't see the worms. You have to act proactively here, and IMO/E it's really a bit too late to start. Next year as soon as you see bud starting to set start a treatment rotation of Bt spray (something like Safers Caterpillar, although I don't like whatever the green carrier is, so I use a commercial preparation) and Monterey Garden Spray with
Spinosad. Every two weeks, or if you see a lot of moths then every week. Do it in the evening so as to avoid hitting pollinators while they're actively feeding and also because that helps with controlling drift, most places the air is a bit more still just before sundown.