Tens of thousands of people will not be able to afford their own homes, or access social housing, if government policy continues, a conference on housing has been told.