A nationalist, anti-migration party has powered into three German state legislatures in elections held amid divisions over chancellor Angela Merkel's liberal approach to the refugee crisis.