Concerns have been raised at the fall in the number of health and safety inspectors tasked with inspecting workplaces at a time when the number of people in work is rapidly increasing.