Police and justice chiefs in both the Republic of Ireland and Northern Ireland have warned about the "very serious challenge" a post-Brexit border across the island could have on policing.