South Korea has said a North Korean patrol boat entered its waters around their disputed maritime border but backed off after nearly an hour following repeated warnings.