# Compute Quantization Error

This example shows how to compute and compare the statistics of the signal quantization error when using various rounding methods. Quantization occurs when a data type cannot represent a value exactly. In these cases, the value must be rounded to the nearest value that can be represented by the data type.

First, a random signal is created that spans the range of the `quantizer` object. Next, the signal is quantized, respectively, with rounding methods `'fix'`, `'floor'`, `'ceil'`, `'nearest'`, and `'convergent'`, and the statistics of the signal are estimated.

The theoretical probability density function of the quantization error is computed with the `errpdf` function, the theoretical mean of the quantization error is computed with the `errmean` function, and the theoretical variance of the quantization error is computed with the `errvar` function.

### Create Uniformly Distributed Random Signal

Create a uniformly distributed random signal that spans the domain -1 to 1 of the fixed-point `quantizer` object `q`.

```q = quantizer([8 7]); r = realmax(q); u = r*(2*rand(50000,1) - 1); xi = linspace(-2*eps(q),2*eps(q),256);```

### Fix: Round Towards Zero

With `'fix'` rounding, the probability density function is twice as wide as the others. For this reason, the variance is four times that of the others.

```q = quantizer('fix',[8 7]); err = quantize(q,u) - u; f_t = errpdf(q,xi); mu_t = errmean(q); v_t = errvar(q); qerrordemoplot(q,f_t,xi,mu_t,v_t,err)```
```Estimated error variance (dB) = -46.8586 Theoretical error variance (dB) = -46.9154 Estimated mean = 7.788e-06 Theoretical mean = 0 ``` The theoretical variance is `eps(q)^2/3` and the theoretical mean is `0`.

### Floor: Round Towards Negative Infinity

`'floor'` rounding is often called truncation when used with integers and fixed-point numbers that are represented using two's complement notation. It is the most common rounding mode of DSP processors because it requires no hardware to implement. `'floor'` does not produce quantized values that are as close to the true values as `'round'` will, but it has the same variance. Using `'floor'`, small signals that vary in sign will be detected, whereas in `'round'` they will be lost.

```q = quantizer('floor',[8 7]); err = quantize(q,u) - u; f_t = errpdf(q,xi); mu_t = errmean(q); v_t = errvar(q); qerrordemoplot(q,f_t,xi,mu_t,v_t,err)```
```Estimated error variance (dB) = -52.9148 Theoretical error variance (dB) = -52.936 Estimated mean = -0.0038956 Theoretical mean = -0.0039062 ``` The theoretical variance is `eps(q)^2/12` and the theoretical mean is `-eps(q)/2`.

### Ceil: Round Towards Positive Infinity

```q = quantizer('ceil',[8 7]); err = quantize(q,u) - u; f_t = errpdf(q,xi); mu_t = errmean(q); v_t = errvar(q); qerrordemoplot(q,f_t,xi,mu_t,v_t,err)```
```Estimated error variance (dB) = -52.9148 Theoretical error variance (dB) = -52.936 Estimated mean = 0.0039169 Theoretical mean = 0.0039062 ``` The theoretical variance is `eps(q)^2/12` and the theoretical mean is `eps(q)/2`.

### Round: Round to Nearest; In a Tie Round to Largest Magnitude

`'round'` is more accurate than `'floor'`, but all values smaller than `eps(q)` get rounded to zero and are lost.

```q = quantizer('nearest',[8 7]); err = quantize(q,u) - u; f_t = errpdf(q,xi); mu_t = errmean(q); v_t = errvar(q); qerrordemoplot(q,f_t,xi,mu_t,v_t,err)```
```Estimated error variance (dB) = -52.9579 Theoretical error variance (dB) = -52.936 Estimated mean = -2.212e-06 Theoretical mean = 0 ``` The theoretical variance is `eps(q)^2/12` and the theoretical mean is `0`.

### Convergent: Round to Nearest; In a Tie Round to Even

`'convergent'` rounding eliminates the bias introduced by ordinary `'round'` caused by always rounding the tie in the same direction.

```q = quantizer('convergent',[8 7]); err = quantize(q,u) - u; f_t = errpdf(q,xi); mu_t = errmean(q); v_t = errvar(q); qerrordemoplot(q,f_t,xi,mu_t,v_t,err)```
```Estimated error variance (dB) = -52.9579 Theoretical error variance (dB) = -52.936 Estimated mean = -2.212e-06 Theoretical mean = 0 ``` The theoretical variance is `eps(q)^2/12` and the theoretical mean is `0`.

### Compare Nearest and Convergent Rounding

The error probability density function for convergent rounding is difficult to distinguish from that of round-to-nearest by looking at the plot.

The error probability density function of convergent is

```f(err) = 1/eps(q), for -eps(q)/2 <= err <= eps(q)/2, and 0 otherwise ```

while the error probability density function of round is

```f(err) = 1/eps(q), for -eps(q)/2 < err <= eps(q)/2, and 0 otherwise ```

The error probability density function of convergent is symmetric, while round is slightly biased towards the positive.

The only difference is the direction of rounding in a tie.

```x = (-3.5:3.5)'; [x convergent(x) nearest(x)]```
```ans = 8×3 -3.5000 -4.0000 -3.0000 -2.5000 -2.0000 -2.0000 -1.5000 -2.0000 -1.0000 -0.5000 0 0 0.5000 0 1.0000 1.5000 2.0000 2.0000 2.5000 2.0000 3.0000 3.5000 4.0000 4.0000 ```