We've unified LLMs w/ vector memory + reranking & pruning models in a single process for better performance

Follow the full discussion on Reddit.
There is a lot of latency involved shuffling data for modern/complex ML systems in production. In our experience these costs dominate end-to-end user experienced latency, rather than actual model or ANN algorithms, which unfortunately limits what is achievable for interactive applications.