BITCOS reaches 1.485 bits per weight for ternary LLMs

BITCOS reaches 1.485 bits per weight for ternary LLMs

BITCOS, a presence-bitmap plus sign-vector layout for ternary {-1,0,+1} weights from Evangelos Georganas, Alexander Heinecke and Pradeep Dubey, reaches 1.485 bits per weight on the sparsest of 29 tested models in a 14 September preprint, below the log2(3) ≈1.585 reference and 1.625-bit five-trit packing.

Published

Read at another depth