Members of a rising nationalist party in Germany have elected their leadership at a convention in the city of Hannover that was overshadowed by protests.