Firefighters have been battling a blaze at a listed 17th century building which has seen more recent use as a swingers' venue.