Xgboost prediction and how to deal with data drift

Follow the full discussion on Reddit.
I have an xgboost model that I trained on tabular data with categories (there are no numerical fields). I use pandas.get_dummies() to ohe categorical fields and feed it to xgboost model to train. Now the question is that at inference time, whenever there is new categorical value, my matrix looks different and model.predict() fails.