Cerebras doing R&D to support fast small batch size SGD

Follow the full discussion on Reddit.
The CS-1 has 1.2 terabits of ethernet bandwidth, meaning it could load raw ImageNet (jpegs) in under 10 seconds, or 256x256 pixel ImageNet tensors in about 2 seconds. But with only 18GB of SDRAM, ImageNet won't fit in memory, so contemporary mega-sized minibatch isn't an option. That said, the hardware has amazing memory bandwidth, and native support for sparse GEMM, so there is still ample opportunity for sub-2-minute ImageNet runs.