We can have frost in this general vicinity (including where you are) in June, but normally the weather settles down around Memorial Day or so. I would go with the later date, and then pad a little if you are in an exposed spot or where cold settles (a flat expanse near hills, e.g.).
Do NOT believe anything earlier than Memorial Day! Over all, most years, you can get away with that or June 1.