The union representing California state prison guards filed a formal notice that it will seek to oust Governor Arnold Schwarzenegger, a move his advisers say they are taking seriously.