A plan to hold a popular vote on whether Australia should allow same-sex marriage has suffered a setback after a political party announced it would not support the proposed public vote.