A former Christian Brother has told the indecent assault trial of a member of the order that he had disagreement with the accused over his relationship with young people.