In Entropy, Prefix Codes, and KL Divergence, $D_{KL}(p|q)$ is the expected excess cost of coding a $p$ sequence with a $q$ code. It is asymmetric, and the asymmetry is the subject here. Let $p$ be ...

In Entropy, Prefix Codes, and KL Divergence, a single random variable $X$ is described by a prefix code, and the optimal expected code word length is the entropy $H(X)$. With two variables, the sam...

Given a distribution $p$ over a finite alphabet, a prefix code assigns each symbol a binary code word such that no code word is a prefix of another code word. The expected code word length of any s...

Link Budget The link budget in its most general form is, [P_{RX} (\text{dBm}) = P_{TX} (\text{dBm}) + G (\text{dB}) - L (\text{dB})] Where $P_{(\cdot)}$ is power expressed in decibels-milliwatt, ...

Natural gradient descent is an optimization method that accounts for the geometry of the parameter space when updating model parameters. Unlike standard gradient descent, which treats parameter spa...

My notes taken from this Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution. import torch from matplotlib import pyplot as plt Diffusion Language Models Motivation G...

A follow up to my earlier posts on time dilation and length contraction. This post will talk about how velocities add under Special Relativity and how it differs that Newton’s laws of motion. Expe...

The following is a solution to TTM book 4, lecture 5, exercise 1: Given the metric $g_{\mu \nu}(X)$, show that the Euler-Lagrange equation, to minimize the action along a trajectory in space-ti...

This is a second post on electrical circuits and their similarity to mechanical oscillators. The previous post covers the mechanical harmonic oscillator and the LC circuit. This post focuses on the...

A weight on a spring has a mathematical similarity with an electrical circuit consisting of a capacitor and inductor (LC circuit). They are both examples of harmonic oscillators which exhibit a sin...