Brexit remains the biggest worry for small and medium enterprises as business become increasingly concerned about the future.