Men who drink heavily in their middle age speed up their rate of memory loss, a study has shown.