A man accused of killing his five children was an ex-convict whose homes were visited by social workers a dozen times in the past three years.