Is the sum of Pi log Pi, multiplied by – 1 whereas the definition
Given by Wiener in his Cybernetics for “amount of information”
is the same sum of Pi log Pi unchanged (i.e. multiplied by +1).
(The reader should notice that p log p is necessarily negative, so
The multiplier “– 1” makes it a positive number.)
There need however be no confusion, for the basic ideas are
Identical. Both regard information as “that which removes uncer-
177
A N I N T R O D UC T I O N T O C Y B E R NE T I C S
I N CESSA N T TR AN SMI SSIO N
Tainty”, and both measure it by the amount of uncertainty it
Removes. Both further are concerned basically with the gain or
Increase in information that occurs when a message arrives— the
Absolute quantities present before or after being of minor interest.
Now it is clear that when the probabilities are well spread, as in
A of Fig. 9/14/1, the uncertainty is greater than when they are
Compact, as in B.
Thus the two measures are no more discrepant than are the two
Ways of measuring “how far is point Q to the right of point P”
Shown in Fig. 9/14/2.
Fig. 9/14/2
Fig. 9/14/1
So the receipt of a message that makes the recipient revise his esti-
Mate, of what will happen, from distribution A to distribution B,
contains a positive amount of information. Now Σp log p (where
Σ means “the sum of”), if applied to A, will give a more negative
Number than if applied to B; both will be negative but A’s will be
The larger in absolute value. Thus A might give– 20 for the sum
and B might give – 3. If we use Σp log p multiplied by plus 1 as
Amount of information to be associated with each distribution, i.e.
With each set of probabilities, then as, in general,
Gain (of anything) = Final quantity minus initial quantity
So the gain of information will be
( – 3) – (– 20)
which is + 17, a positive quantity, which is what we want. Thus,
looked at from this point of view, which is Wiener’s, Σp log p
Should be multiplied by plus 1, i.e. left unchanged; then we calcu-
Late the gain.
Shannon, however, is concerned throughout his book with the
Special case in which the received message is known with cer-
|
|
|
Tainty. So the probabilities are all zero except for a single I. Over
such a set Σp log p is just zero; so the final quantity is zero, and
The gain of information is
Initial quantity).
In other words, the information in the message, which equals the
gain in information, is Σp log p calculated over the initial distri-
Bution, multiplied by minus 1, which gives Shannon’s measure.
178
Here P and Q can be thought of as corresponding to two degrees
Of uncertainty, with more certainty to the right, and with a mes-
Sage shifting the recipient from P to Q.
The distance from P to Q can be measured in two ways, which
Are clearly equivalent. Wiener’s way is to lay the rule against P
And Q (as W in the Fig.); then the distance that Q lies to the right
Of P is given by
Q’s reading) minus (P’s reading).
Shannon’s way (S in the Fig.) is to lay the zero opposite Q, and
Then the distance that Q is to the right of P is given by
Minus (P’s reading).
There is obviously no real discrepancy between the two methods.
Channel capacity. It is necessary to distinguish two ways of
Reckoning “entropy” in relation to a Markov chain, even after the
Unit (logarithmic base) has been decided. The figure calculated in
S.9/12, from the transition probabilities, gives the entropy, or
Variety to be expected, at the next, single, step of the chain. Thus
If an unbiased coin has already given T T H H T H H H H, the
Uncertainty of what will come next amounts to I bit. The symbol
That next follows has also an uncertainty of 1 bit; and so on. So the
Chain as a whole has an uncertainty, or entropy, of 1 bit per step.
Two steps should then have an uncertainty, or variety, of 2 bits,
Дата добавления: 2019-11-16; просмотров: 210; Мы поможем в написании вашей работы! |
Мы поможем в написании ваших работ!