seemore: Implement a Vision Language Model from Scratch

Follow the full discussion on Reddit.
Hi all, I implemented a vision language model consisting of an image encoder, a multimodal projection module and a decoder language model in pure pytorch. Think of this as a simplified version of what you see in GPT-4 or Claude 3 in terms of vision capabilities demonstrated by a language model (think moondream 2 or LLaVA when it comes to Open Source Models). The name ‘seemore’ is my way of paying homage to Andrej Karpathy’s project ‘makemore’ because here I use a character level autoregressive language model much like in his nanoGPT/ makemore implementation. My goal is for this to be a hackable implementation that people use to understand how this really works and improve upon. I foresee more and more of these models coming out throughout the year.