An appeals court has refused to reinstate an order granting pop star Michael Jackson sole custody of his two children with ex-wife Debbie Rowe.