Labour Party chairman Colm Keaveney has hit back at attempts by the party leadership to oust him from his position, following his vote against the budget, saying he will not engage in their “silly games”.