Adventures in Inference Time Optimizations (and cool stuff) – Hlib Avietisov blog on machine learning

Pipeline Distillation Implementation

machine-learning

distillation

import torch
import numpy as np
import matplotlib.pyplot as plt

Pipeline Distillation: Teaching Neural Networks One Layer at a Time

machine-learning

distillation

Discover how to efficiently compress neural networks by teaching them layer-by-layer, just like building with LEGO blocks.

Block-strided coalesced memory access pattern in triton

triton

Just sharing this ingenious way to create sets of block pointers.

Abusing distributional properties of 4-bit quantized weights for compression with variable length quantization

transformers

quantization

optimization

Distributional properties of 4-bit quantizations allow for variable length encoding, with and without accuracy loss (depends on what you actually want)

Unnatural abilities of context stitching

machine-learning

transformers

attention

optimization

As was described in the “You don’t need to pay that much attention” post, from slippery attention’s property to have different attention patterns for different classes of tokens, coming a rather powerful technique, which I called “Context stitching”, which allows to put context together from precomputed chunks like lego blocks. In this blog post, I elaborate how.

You don’t need to pay that much attention

machine-learning

transformers

attention

code

optimization

A sub-quadratic attention trick to enchance even pre-trained LLMs with speed and incredible abilities