FPGA Central

Hi all,

Suppose my input data follows Gaussian distribution with mean u and variance
sigma^2.

If I want to design 6-bit ADC... what should be the optimal step size for
this ADC?

Thanks a lot

-L

"lucy" <[email protected]> wrote in
news:cqacbp$bbc$[email protected]:

> Hi all,
>
> Suppose my input data follows Gaussian distribution with mean u and
> variance sigma^2.
>
> If I want to design 6-bit ADC... what should be the optimal step size
> for this ADC?
>
> Thanks a lot
>
> -L
>
>
>

If ADC is Analog to Digital Converter, the distribution of your data has
nothing to do with the resolution. The range of your data does, though.
You can sacrifice range for resolution, and resolution for range.

Why 6 bits?

Scott

Scott Seidman wrote:
> "lucy" <[email protected]> wrote in
> news:cqacbp$bbc$[email protected]:

>>Suppose my input data follows Gaussian distribution with mean u and
>>variance sigma^2.

>>If I want to design 6-bit ADC... what should be the optimal step size
>>for this ADC?

> If ADC is Analog to Digital Converter, the distribution of your data has
> nothing to do with the resolution. The range of your data does, though.
> You can sacrifice range for resolution, and resolution for range.

Well, Gaussian distributions have infinite range, though with
ever decreasing probability. This does sound like a homework
problem, and possibly some important information was left out.

What parameter is being optimized through step size choice?
I don't believe there is only one answer to this problem.

-- glen

glen herrmannsfeldt <[email protected]> wrote in
news:cqaejd$4gl$1@gnus01.u.wash[email protected]:

>
>
> Scott Seidman wrote:
>> "lucy" <[email protected]> wrote in
>> news:cqacbp$bbc$[email protected]:
>
>>>Suppose my input data follows Gaussian distribution with mean u and
>>>variance sigma^2.
>
>>>If I want to design 6-bit ADC... what should be the optimal step size
>>>for this ADC?
>
>> If ADC is Analog to Digital Converter, the distribution of your data
>> has nothing to do with the resolution. The range of your data does,
>> though. You can sacrifice range for resolution, and resolution for
>> range.
>
> Well, Gaussian distributions have infinite range, though with
> ever decreasing probability. This does sound like a homework
> problem, and possibly some important information was left out.
>
> What parameter is being optimized through step size choice?
> I don't believe there is only one answer to this problem.
>
> -- glen
>

Lucy isn't a student, but she does hold something near a record for thread
initiations in cssm. Frankly, she tends to use this newsgroup instead of
hauling out a textbook, or even doing a google search to try to find what
she needs to know. If I had noticed it was her post, I probably would have
skipped it.

Scott

"Scott Seidman" <[email protected]> wrote in message
news:[email protected] 1.4...
> glen herrmannsfeldt <[email protected]> wrote in
> news:cqaejd$4gl$[email protected]:
>
>>
>>
>> Scott Seidman wrote:
>>> "lucy" <[email protected]> wrote in
>>> news:cqacbp$bbc$[email protected]:
>>
>>>>Suppose my input data follows Gaussian distribution with mean u and
>>>>variance sigma^2.
>>
>>>>If I want to design 6-bit ADC... what should be the optimal step size
>>>>for this ADC?
>>
>>> If ADC is Analog to Digital Converter, the distribution of your data
>>> has nothing to do with the resolution. The range of your data does,
>>> though. You can sacrifice range for resolution, and resolution for
>>> range.
>>
>> Well, Gaussian distributions have infinite range, though with
>> ever decreasing probability. This does sound like a homework
>> problem, and possibly some important information was left out.
>>
>> What parameter is being optimized through step size choice?
>> I don't believe there is only one answer to this problem.
>>
>> -- glen
>>
>
> Lucy isn't a student, but she does hold something near a record for thread
> initiations in cssm. Frankly, she tends to use this newsgroup instead of
> hauling out a textbook, or even doing a google search to try to find what
> she needs to know. If I had noticed it was her post, I probably would
> have
> skipped it.
>
> Scott

Hi Scott,

I did not know that my posts were so disgusting... to you...

But I did try to ask questions that are not easily found solutions in
textbooks... some problems are in fact initiated by myself. I just like
problem-solving... It is hard to find guys interested in problem-solving
nearby... I thought newsgroup might be a better place... if you find my
problems are so trivial that a few googling + book hunting can work out,
please do let me know... please let me know on which book you can find the
solution to the above problem. I'd really like to know ... newsgroup serves
as pointers, right?

Hello Lucy,

Try looking in "Digital Processing of Speech Signals" by Rabiner & Schafer.
There is a bunch of info on matching the quantization to the distribution of
a signal. Basically you are maximizing the entropy. I.e gain the most info
per bit that you can.

Clay




"lucy" <[email protected]> wrote in message
news:cqacbp$bbc$[email protected]..
> Hi all,
>
> Suppose my input data follows Gaussian distribution with mean u and
> variance sigma^2.
>
> If I want to design 6-bit ADC... what should be the optimal step size for
> this ADC?
>
> Thanks a lot
>
> -L
>

"glen herrmannsfeldt" <[email protected]> wrote in message
news:cqaejd$4gl$[email protected]..
>
>
> Scott Seidman wrote:
>> "lucy" <[email protected]> wrote in
>> news:cqacbp$bbc$[email protected]:
>
>>>Suppose my input data follows Gaussian distribution with mean u and
>>>variance sigma^2.
>
>>>If I want to design 6-bit ADC... what should be the optimal step size
>>>for this ADC?
>
>> If ADC is Analog to Digital Converter, the distribution of your data has
>> nothing to do with the resolution. The range of your data does, though.
>> You can sacrifice range for resolution, and resolution for range.
>
> Well, Gaussian distributions have infinite range, though with
> ever decreasing probability. This does sound like a homework
> problem, and possibly some important information was left out.

Hello Glen,

One way is to divide the area under the bell curve into equal area
partitions. In this case 2^6 of them. The abscissal value for each of the
partitions becomes a transition point for the quantization. This maximizes
the entropy (i.e., the information) since each quantization will be equally
represented. And the entopy is maximized when each state has equal
probability. This in not unlike Huffman's idea where his coding scheme
attempts to make the length of each symbol times its frequency of occurance
be the same for all symbols.

Clay




>
> What parameter is being optimized through step size choice?
> I don't believe there is only one answer to this problem.
>
> -- glen
>

"Clay S. Turner" <[email protected]> writes:
> [...]
> One way is to divide the area under the bell curve into equal area
> partitions. In this case 2^6 of them. The abscissal value for each of the
> partitions becomes a transition point for the quantization. This maximizes
> the entropy (i.e., the information) since each quantization will be equally
> represented. And the entopy is maximized when each state has equal
> probability.

Isn't this a mapping from Gaussian to uniform? And we all know that
uniform has the greatest entropy.

Is this related to the concept of "vector quantization"?

> This in not unlike Huffman's idea where his coding scheme
> attempts to make the length of each symbol times its frequency of occurance
> be the same for all symbols.

I'm having trouble with this. Yes, I see that a Huffman code attempts to make
(symbol length)*(symbol frequency) constant over all symbols. How does change
anything about the underlying distribution, though?
--
Randy Yates
Sony Ericsson Mobile Communications
Research Triangle Park, NC, USA
[email protected], 919-472-1124

"lucy" <[email protected]> wrote in
news:cqajiv$hsm$[email protected]:

> Hi Scott,
>
> I did not know that my posts were so disgusting... to you...
>
> But I did try to ask questions that are not easily found solutions in
> textbooks... some problems are in fact initiated by myself. I just
> like problem-solving... It is hard to find guys interested in
> problem-solving nearby... I thought newsgroup might be a better
> place... if you find my problems are so trivial that a few googling +
> book hunting can work out, please do let me know... please let me
> know on which book you can find the solution to the above problem. I'd
> really like to know ... newsgroup serves as pointers, right?
>
>
>
>
>
>

lucy-

I don't find your posts disgusting, just somewhat tedious. The netscan
page lists your posts in cssm under your currently used email addy (and I
seem to remember at least one more address before the current one) for
the time period between 7/1/04 and 9/30/04. In that three-month period,
you initiated 77 threads, and your first use of that name was on 7/28!
That's more threads, by about a factor of 5, beyond your nearest
competitor. Then,

As many of those posts were requests for very basic information about GUI
programming in Matlab (which participants seemed to have taught you,
quite patiently and generously), my impression during that binge of
questioning was that you were fairly bright, but embarked on a project
in an unfamiliar environment that you didn't want to take the time to
learn. Then, the posts moved over to signal processing, where you
appeared to have absolutely no training, but you expected to learn this
stuff on cssm. Signal processing questions covered: basic fft
properties, basic sampling theorem, filter design, image processing, and
some other topics, every one of which is well covered in one of two
textbooks that you can find on just about every library's shelves. I'd
suggest "Signals and Systems" by Oppenheim and Willsky, "Digital Signal
Processing" by Oppenheim and Schaffer, and the lovely image processing
book that can be found on the mathworks site under textbooks. My
recommendations are somewhat dated, as I'm sure that more modern
references on signals and systems cover a better mix of analog and
digital techniques. Some of the docs for the Matlab toolboxes have fine
reference lists, and they don't put those list in just so endusers can
ignore them.

Personally, I'd recommend you consider taking a course on the topic, as
it will be a time saver in the long run. Help is help, but when you post
50 threads on one subject, consider going out and learning it right.
Also, a good rule of thumb is to consider spending an 45 minutes or an
hour trying to do something (like everybody who contributes to your
threads did at some time) before asking for help with it. Then, try
googling for it to see if you can help yourself without passing the hat.
In fact, if you've googled this particular question in usenet groups,
you'll find that you've already asked almost exactly the same question,
but for the more complex 2D case!

CSSM is a fine resource to get you over the rough spots, but its not the
participants' responsibility to provide you with on the job training in
areas you aren't proficient in.

Scott

Hello Randy,
comments below:

"Randy Yates" <[email protected]> wrote in message
news:[email protected]..
> "Clay S. Turner" <[email protected]> writes:
>> [...]
>> One way is to divide the area under the bell curve into equal area
>> partitions. In this case 2^6 of them. The abscissal value for each of the
>> partitions becomes a transition point for the quantization. This
>> maximizes
>> the entropy (i.e., the information) since each quantization will be
>> equally
>> represented. And the entopy is maximized when each state has equal
>> probability.
>
> Isn't this a mapping from Gaussian to uniform? And we all know that
> uniform has the greatest entropy.

Basically except for the lack of a Jacobian. And going to a form which
maximizes our entropy is the goal.

>
> Is this related to the concept of "vector quantization"?

Probably - but I havn't done much with this, so I'll have to guess here. But
since the concept of maximizing entropy is a wonderful way of handling
quantization without history, so I would be surprised if it hasn't been used
for quantizing vocoder vectors. Remember in all of this process, we are
assuming no sample to sample correlation. And in speech this is not quite
true.

>
>> This in not unlike Huffman's idea where his coding scheme
>> attempts to make the length of each symbol times its frequency of
>> occurance
>> be the same for all symbols.
>
> I'm having trouble with this. Yes, I see that a Huffman code attempts to
> make
> (symbol length)*(symbol frequency) constant over all symbols. How does
> change
> anything about the underlying distribution, though?

The idea is to try to make each symbol contribute equally to the overall
process. Imagine looking at your data after a huge number of symbols was
received. The idea is to make the info provided by each type of symbol
contribute equally. And in data compression, the idea is to find the minimum
size for all of the data for a fixed information content.

I hope this helps.

Clay


> --
> Randy Yates
> Sony Ericsson Mobile Communications
> Research Triangle Park, NC, USA
> [email protected], 919-472-1124

"Clay S. Turner" <[email protected]> writes:
> [...]
> > I'm having trouble with this. Yes, I see that a Huffman code attempts to
> > make
> > (symbol length)*(symbol frequency) constant over all symbols. How does
> > change
> > anything about the underlying distribution, though?
>
> The idea is to try to make each symbol contribute equally to the overall
> process. Imagine looking at your data after a huge number of symbols was
> received. The idea is to make the info provided by each type of symbol
> contribute equally.

But this Huffman coding won't do that. Choosing a representation for a
symbol doesn't change the probability of the symbol occurring. It
does, however, minize the average symbol rate - I certainly see
that. Perhaps I'm being blind?
--
Randy Yates
Sony Ericsson Mobile Communications
Research Triangle Park, NC, USA
[email protected], 919-472-1124

Someone wrote:
>>>I'm having trouble with this.
>>>Yes, I see that a Huffman code attempts to make
>>>(symbol length)*(symbol frequency) constant over
>>>all symbols. How does change
>>>anything about the underlying distribution, though?

> "Clay S. Turner" <[email protected]> writes:
>>The idea is to try to make each symbol contribute equally to the overall
>>process. Imagine looking at your data after a huge number of symbols was
>>received. The idea is to make the info provided by each type of symbol
>>contribute equally.

Randy Yates wrote:
> But this Huffman coding won't do that. Choosing a representation for a
> symbol doesn't change the probability of the symbol occurring. It
> does, however, minize the average symbol rate - I certainly see
> that. Perhaps I'm being blind?

The gaussian has infinite tails, so it isn't possible to cover
the range with a linear scale ADC. One could then ask for what
part of the distribution, when covered with the steps of a six
bit ADC, are the symbol probabilities most equal.

Without doing the math it isn't so obvious either way.
For the case of a very large (lim --> infinity) most of
the symbols will have almost no probability. In the
limit of very small step size, and assuming that the lower
and upper step get the tails, again most steps have almost
no probability.

So, it would seem that somewhere in between the probability
would be more equal. One should then define the appropriate
function of step size and find the minimum point.

-- glen

glen herrmannsfeldt <[email protected]> wrote in news:cqceia$n9j$1
@gnus01.u.washington.edu:

> The gaussian has infinite tails, so it isn't possible to cover
> the range with a linear scale ADC. One could then ask for what
> part of the distribution, when covered with the steps of a six
> bit ADC, are the symbol probabilities most equal.

I think the question is impossible to address in a vacuum. The theoretical
handling, while possibly interesting, must take a back seat to the
engineering concerns. Start with the questions "Why am I sampling the
signal, and what will I be doing with the signal after I sample it?"

Then, you can start talking about what happens when the signal falls
outside your ADC range. You might even choose to investigate the
possibility of a non-linear ADC. I've never heard of such a thing, but
there's no reason why it can't be implemented.

Scott

glen herrmannsfeldt wrote:

>
>
>
>
> Someone wrote:
>
>>>> I'm having trouble with this.
>
>>>>Yes, I see that a Huffman code attempts to make
>
>>>> (symbol length)*(symbol frequency) constant over
>
>>>>all symbols. How does change
>
>>>> anything about the underlying distribution, though?
>
>
>> "Clay S. Turner" <[email protected]> writes:
>>
>>> The idea is to try to make each symbol contribute equally to the
>>> overall process. Imagine looking at your data after a huge number of
>>> symbols was received. The idea is to make the info provided by each
>>> type of symbol contribute equally.
>
>
> Randy Yates wrote:
>
>> But this Huffman coding won't do that. Choosing a representation for a
>> symbol doesn't change the probability of the symbol occurring. It
>> does, however, minize the average symbol rate - I certainly see
>> that. Perhaps I'm being blind?
>
>
> The gaussian has infinite tails, so it isn't possible to cover
> the range with a linear scale ADC. One could then ask for what
> part of the distribution, when covered with the steps of a six
> bit ADC, are the symbol probabilities most equal.
>

...

For non-linear encoding, it actually is possible. One way is to divide
the cumulative distribution into six parts equal (divide the density
function into six equal areas), then have the ADC report the ***tile*
that the sample falls into. Infinite tails don't bother that scheme.

Jerry
_________________________
* Rarely the kitchen floor.
--
Engineering is the art of making what you want from things you can get.
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

"Scott Seidman" <[email protected]> wrote in message
news:[email protected] 1.4...
>
> Then, you can start talking about what happens when the signal falls
> outside your ADC range. You might even choose to investigate the
> possibility of a non-linear ADC. I've never heard of such a thing, but
> there's no reason why it can't be implemented.

Hello Scott,
Anytime you speak over a landline where the signal is carried digitally, the
speech is quantized nonlinearly. Depending on the part of the world you are
in, it will be either mu-law (North America) or A-law (pretty much the rest
of the world save for a few place like a parts of the Philliphines).

A lot of early work with nonlinear quantization was done using Laplacian and
Gamma densities as models for the speech.

Look up:

Paez, M.D., and Glisson, T.H., "Minimum Mean Squared Error Quantization in
Speech", IEEE Trans. Comm., Vol. Com-20, pp. 225-230, April 1972

and

Max, J., "Quantizing for Minimum Distortion", IRE Trans. Inform. Theory, Vol
IT-6, pp 7-12, March 1960

for more details.

Clay






>
> Scott

"Clay S. Turner" <[email protected]> wrote in
news:7%jyd.24$5y3[email protected]:

>
> "Scott Seidman" <[email protected]> wrote in message
> news:[email protected] 1.4...
>>
>> Then, you can start talking about what happens when the signal falls
>> outside your ADC range. You might even choose to investigate the
>> possibility of a non-linear ADC. I've never heard of such a thing,
>> but there's no reason why it can't be implemented.
>
> Hello Scott,
> Anytime you speak over a landline where the signal is carried
> digitally, the speech is quantized nonlinearly. Depending on the part
> of the world you are in, it will be either mu-law (North America) or
> A-law (pretty much the rest of the world save for a few place like a
> parts of the Philliphines).
>
> A lot of early work with nonlinear quantization was done using
> Laplacian and Gamma densities as models for the speech.
>
> Look up:
>
> Paez, M.D., and Glisson, T.H., "Minimum Mean Squared Error
> Quantization in Speech", IEEE Trans. Comm., Vol. Com-20, pp. 225-230,
> April 1972
>
> and
>
> Max, J., "Quantizing for Minimum Distortion", IRE Trans. Inform.
> Theory, Vol IT-6, pp 7-12, March 1960
>
> for more details.
>
> Clay
>
>
>
>
>
>
>>
>> Scott
>
>
>

Learn something new every day! My impression, though, is that the ADC is
high-bit linear, and then it's companded to a lower bit count. Is this
how it's done, or do the ADC spirits do the whole thing in one step?

Scott

Jerry Avins wrote:
> glen herrmannsfeldt wrote:

(snip)

>>The gaussian has infinite tails, so it isn't possible to cover
>>the range with a linear scale ADC. One could then ask for what
>>part of the distribution, when covered with the steps of a six
>>bit ADC, are the symbol probabilities most equal.

> For non-linear encoding, it actually is possible. One way is to divide
> the cumulative distribution into six parts equal (divide the density
> function into six equal areas), then have the ADC report the ***tile*
> that the sample falls into. Infinite tails don't bother that scheme.

To me "step size" implies linear, but I could be convinced otherwise.
I would have thought "step sizes", though, for the non-linear case.

-- glen

Scott Seidman wrote:
>>
>
> Learn something new every day! My impression, though, is that the ADC is
> high-bit linear, and then it's companded to a lower bit count. Is this
> how it's done, or do the ADC spirits do the whole thing in one step?
>
> Scott
It could be done either way. In the old days it was probably done with
256 comparators and a long string of resistors -- today it's probably
done with a 12-bit ADC and a look-up table.

--

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

"Randy Yates" <[email protected]> wrote in message
news:[email protected]..
> "Clay S. Turner" <[email protected]> writes:
>> [...]
>> The idea is to try to make each symbol contribute equally to the overall
>> process. Imagine looking at your data after a huge number of symbols was
>> received. The idea is to make the info provided by each type of symbol
>> contribute equally.
>
> But this Huffman coding won't do that. Choosing a representation for a
> symbol doesn't change the probability of the symbol occurring. It
> does, however, minize the average symbol rate - I certainly see
> that. Perhaps I'm being blind?
> --

Hello Randy,

The Huffman problem and the quantization problem are related in they both
implement methods to effectively flatten out variations. True they are
different in that the Huffman problem is trying to minimize the average
number of bits per sample and the quantization problem is trying to maximize
the information per sample. The differences arise from the constraints and
what you are starting with. In one case we have a fixed amount of
information, so how few total bits can we fit all of the info in. The
theoretical answer is the entropy, but the practical answer (comes from
requiring whole numbers of bits in a symbol is the Huffman entropy.) In the
other case we have a fixed symbol size, so how can we get the most info per
symbol. The connection is the entropy and its maximization requires a flat
distribution.

In Huffman coding we sort the symbol probabilities into descending order and
then we make multiple passes throught the list each time combining the two
lowest probilites together. So we are essentially trying to make each path's
probability be the same or at leat similar. In the quantization we are just
diving up the total probability into N equal regions.

You are not being blind, you are just asking how two seemingly different
things are related. And the relation is through a flattening out of the
probabilty function.

I hope this helps to clear some things up.

Clay

"Clay S. Turner" <[email protected]> writes:

> "Randy Yates" <[email protected]> wrote in message
> news:[email protected]..
> > "Clay S. Turner" <[email protected]> writes:
> >> [...]
> >> The idea is to try to make each symbol contribute equally to the overall
> >> process. Imagine looking at your data after a huge number of symbols was
> >> received. The idea is to make the info provided by each type of symbol
> >> contribute equally.
> >
> > But this Huffman coding won't do that. Choosing a representation for a
> > symbol doesn't change the probability of the symbol occurring. It
> > does, however, minize the average symbol rate - I certainly see
> > that. Perhaps I'm being blind?
> > --
>
> Hello Randy,
>
> The Huffman problem and the quantization problem are related in they both
> implement methods to effectively flatten out variations. True they are
> different in that the Huffman problem is trying to minimize the average
> number of bits per sample and the quantization problem is trying to maximize
> the information per sample. The differences arise from the constraints and
> what you are starting with. In one case we have a fixed amount of
> information, so how few total bits can we fit all of the info in. The
> theoretical answer is the entropy, but the practical answer (comes from
> requiring whole numbers of bits in a symbol is the Huffman entropy.) In the
> other case we have a fixed symbol size, so how can we get the most info per
> symbol. The connection is the entropy and its maximization requires a flat
> distribution.
>
> In Huffman coding we sort the symbol probabilities into descending order and
> then we make multiple passes throught the list each time combining the two
> lowest probilites together. So we are essentially trying to make each path's
> probability be the same or at leat similar. In the quantization we are just
> diving up the total probability into N equal regions.
>
> You are not being blind, you are just asking how two seemingly different
> things are related. And the relation is through a flattening out of the
> probabilty function.
>
> I hope this helps to clear some things up.
>
> Clay

Intriguing stuff! I'm not sure I follow the sorting description, but
that's OK. I need to go back and read fully and with undrstanding
the seminal Shannon papers.
--
Randy Yates
Sony Ericsson Mobile Communications
Research Triangle Park, NC, USA
[email protected], 919-472-1124

"Randy Yates" <[email protected]> wrote in message
news:xxp652u13um.fsf@usrts005.corpus[email protected]..
>
> Intriguing stuff! I'm not sure I follow the sorting description, but
> that's OK. I need to go back and read fully and with undrstanding
> the seminal Shannon papers.
> --

Hello Randy,
You will probably want to get "An Introduction to Information Theory,
Symbols, Signals, and Noise" by John Pierce. This is a wonderful book, and
it makes the material easy to grasp. Plus he throws in a lot of the
historical background, so this also aides the understanding process.

ISBN 0-486-24061-4

I got my copy a while back for 10 bucks. I have the 2nd edition from 1980.

Clay

glen herrmannsfeldt wrote:
>
>
> Jerry Avins wrote:
>
>> glen herrmannsfeldt wrote:
>
>
> (snip)
>
>>> The gaussian has infinite tails, so it isn't possible to cover
>>> the range with a linear scale ADC. One could then ask for what
>>> part of the distribution, when covered with the steps of a six
>>> bit ADC, are the symbol probabilities most equal.
>
>
>> For non-linear encoding, it actually is possible. One way is to divide
>> the cumulative distribution into six parts equal (divide the density
>> function into six equal areas), then have the ADC report the ***tile*
>> that the sample falls into. Infinite tails don't bother that scheme.
>
>
> To me "step size" implies linear, but I could be convinced otherwise.
> I would have thought "step sizes", though, for the non-linear case.
>
> -- glen

I'll buy that, but it could go either way. You could compare warehouses
by floor *areas* or floor *area*.

jerry
--
Engineering is the art of making what you want from things you can get.
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

Clay S. Turner wrote:

> "Randy Yates" <[email protected]> wrote in message
> news:[email protected]..
>
>>Intriguing stuff! I'm not sure I follow the sorting description, but
>>that's OK. I need to go back and read fully and with undrstanding
>>the seminal Shannon papers.
>>--
>
>
> Hello Randy,
> You will probably want to get "An Introduction to Information Theory,
> Symbols, Signals, and Noise" by John Pierce. This is a wonderful book, and
> it makes the material easy to grasp. Plus he throws in a lot of the
> historical background, so this also aides the understanding process.
>
> ISBN 0-486-24061-4
>
> I got my copy a while back for 10 bucks. I have the 2nd edition from 1980.
>
> Clay

http://powellbooks.com/cgi-bin/bibli...=0-486-24061-4

Easy for you; less so for me.

Jerry
--
Engineering is the art of making what you want from things you can get.
¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯

"Clay S. Turner" <[email protected]> writes:

> "Randy Yates" <[email protected]> wrote in message
> news:[email protected]..
>>
>> Intriguing stuff! I'm not sure I follow the sorting description, but
>> that's OK. I need to go back and read fully and with undrstanding
>> the seminal Shannon papers.
>> --
>
> Hello Randy,
> You will probably want to get "An Introduction to Information Theory,
> Symbols, Signals, and Noise" by John Pierce. This is a wonderful book, and
> it makes the material easy to grasp. Plus he throws in a lot of the
> historical background, so this also aides the understanding process.
>
> ISBN 0-486-24061-4
>
> I got my copy a while back for 10 bucks. I have the 2nd edition from 1980.

For $10 how can you go wrong? Thanks Clay! Merry Christmas.

--Randy

--
% Randy Yates % "Watching all the days go by...
%% Fuquay-Varina, NC % Who are you and who am I?"
%%% 919-577-9882 % 'Mission (A World Record)',
%%%% <[email protected]> % *A New World Record*, ELO
http://home.earthlink.net/~yatescr

Randy Yates wrote:

> "Clay S. Turner" <[email protected]> writes:
>
>
>>"Randy Yates" <[email protected]> wrote in message
>>news:[email protected]..
>>
>>>"Clay S. Turner" <[email protected]> writes:
>>>
>>>>[...]
>>>>The idea is to try to make each symbol contribute equally to the overall
>>>>process. Imagine looking at your data after a huge number of symbols was
>>>>received. The idea is to make the info provided by each type of symbol
>>>>contribute equally.
>>>
>>>But this Huffman coding won't do that. Choosing a representation for a
>>>symbol doesn't change the probability of the symbol occurring. It
>>>does, however, minize the average symbol rate - I certainly see
>>>that. Perhaps I'm being blind?
>>>--
>>
>>Hello Randy,
>>
>>The Huffman problem and the quantization problem are related in they both
>>implement methods to effectively flatten out variations. True they are
>>different in that the Huffman problem is trying to minimize the average
>>number of bits per sample and the quantization problem is trying to maximize
>>the information per sample. The differences arise from the constraints and
>>what you are starting with. In one case we have a fixed amount of
>>information, so how few total bits can we fit all of the info in. The
>>theoretical answer is the entropy, but the practical answer (comes from
>>requiring whole numbers of bits in a symbol is the Huffman entropy.) In the
>>other case we have a fixed symbol size, so how can we get the most info per
>>symbol. The connection is the entropy and its maximization requires a flat
>>distribution.
>>
>>In Huffman coding we sort the symbol probabilities into descending order and
>>then we make multiple passes throught the list each time combining the two
>>lowest probilites together. So we are essentially trying to make each path's
>>probability be the same or at leat similar. In the quantization we are just
>>diving up the total probability into N equal regions.
>>
>>You are not being blind, you are just asking how two seemingly different
>>things are related. And the relation is through a flattening out of the
>>probabilty function.
>>
>>I hope this helps to clear some things up.
>>
>>Clay
>
>
> Intriguing stuff! I'm not sure I follow the sorting description, but
> that's OK. I need to go back and read fully and with undrstanding
> the seminal Shannon papers.


Just a thought... don't know if it's at all relevant to the thread, but
here we go anyway:

I you do a Huffman coding, common symbols are given a longer code word.
This will minimize the average data rate. If you adjust quantization
similarly, I get the feeling this will also minimize the average
quantization error.

/Johan