ClipRCNN: Tiny text-guided zero-shot object detector

Follow the full discussion on Reddit.
We propose a new tiny architecture of zero-shot object detector inspired by the classical R-CNN object detector, named ClipRCNN. How it works: 1) First, we should generate region proposals. So, as we need a class agnostic region proposals generator, we chose a classical algorithm named Selective Search instead of any modern pre-trained object detection networks. This step is similar to proposal generation in R-CNN network. 2) After, we compute CLIP loss between each proposal and all user's prompts (texts and images). 3) Last, we return top k best proposals (with minimum CLIP loss) as a prediction of the ClipRCNN model.