Lecture 08
Contents
Block symbols - combining multiple symbols
P(\sigma_i) = P(s_1) * … * P(S_n)
Assume that they are independent
1 2 3 4 5 6 7 8 | Consider S = {a,b} with P = .75, .25 :: S^2 :: aa 9/16 0 ab 3/16 11 ba 3/16 100 bb 1/16 101 |
Whilst the L values increase for L^n, we then divide by n
to get the average length
Markov Sources
A k
-memory source S is one whose symbols each depend on the previous k
.
If k=0
no symbols on any other, S is memoryless
If k=1
then S is a Markov source
p_ij = P(s_i | s_j) probability of s_i right after a s_j M = (p_ij) transition matrix - probability of getting from state s_j to state s_i
(INSERT MARKOV MODEL IN NOTES (0:41))
In a transition matrix, the sum of entries of any column in M is 1 let x_k = (s_k;m_k;e_k) –>
X_k+1 = M x_k x_k = M^k x_0
x_k = (transition matrix)^k * m_initial
Stable distribution, where Mx_0 = x_0
– don’t need to find eigenvalues nor eigenvectors
— stuff I don’t need to know — A Markov process M is in equilibrium p if p = Mp p is therefore an eigenvector for M with eigenvalue 1
M is ergodic - can get from any state j to any state i M is aperioidc gcd of cycle lengths is 1
Huffman coding for Stationary Markov sources
Consider a source S = {s_1, … s_q} with probabilities p_1, …, p_q, transition matrix M and equilibrium p
HuffE = binary HUffman code on p (ordered) Huff(i) = binary Huffman code on the i_th (ordered) column of M Huff_M: s_1 encoded by Huff_E; for i > 1, si encoded by Huff(i-1)
Average length * LE * L(1) … L_(q) –> For each, we create a new tree for the M col i values * L_M ~= p1L(1) + … + p_qL(q)
L_M <= L_E
Coding and Decoding
Huff_(i) -> Probabilities given that we just had codeword for i
- Start with Huff_E
- Find the codeword for your symbol
- Use the matrix Huff_(i) (probabilities given the previous symbol)
- Find the next codeword
To decode, still move left to right