Safety-Guaranteed Autonomy Under Uncertainty