Optimal Performance without Static Graphs by Fusing Tensor Operation Streams

Follow the full discussion on Reddit.
One of the most crucial aspects of current machine learning research is discovering model architectures that efficiently scale with compute resources. Transformers have emerged as the predominant architecture due to their effective utilization of contemporary hardware. However, they don't adapt their computation graphs based on the complexity of tasks, necessitating different versions for tasks of varying complexity. This approach doesn't align with the goal of having one model capable of continuous learning (lifelong learning) while remaining efficient for easy tasks. I argue that there's a pressing need for further exploration into dynamic architectures, where the computational graph adapts at runtime based on contextual cues.