Trader in historic part of city says the area still has ‘huge potential’ but ‘is like the land that time forgot’