It took some trial and error, but I've found an organic way to keep pests off your garden without bothering the beneficial insects, like bees.
I poured some vegetable oil, chili peppers, and garlic cloves into a blender, and then applied the mixture to the leaves. (Avoid the flowers so the bees keep coming back, and avoid herbs so as not to taint their flavor.) About a week has gone by and the caterpillars that have been devouring my squash leaves have not shown back up.
I have plans to improve the method of production to make it spray-able from a spray bottle. I'll let you know how that works out.