BRITISH prime minister Gordon Brown was accused of being out of touch on the expenses scandal yesterday after Labour withdrew the party whip from the three MPs being prosecuted.