Public toilets in a busy town centre car park will be closed permanently despite objections after being targeted by vandals.