Controversy over the alleged shoot-to-kill policy in the North had grave implications for cross-border co-operation, while Anglo-Irish relations suffered a “serious setback”, newly released state files reveal.