mutual information loss_munication

mutual information loss_munication今天挺paperreading的时候,听到了最大化互信息,还不清楚互信息是个什么东东,google了一下,从http://en.wikipedia.org/wiki/Mutual_information摘过来了:    DefinitionofmutualinformationFormally,themutualinformationoftwod

大家好,又见面了,我是你们的朋友全栈君。如果您正在找激活码,请点击查看最新教程,关注关注公众号 “全栈程序员社区” 获取激活教程,可能之前旧版本教程已经失效.最新Idea2022.1教程亲测有效,一键激活。

Jetbrains全系列IDE使用 1年只要46元 售后保障 童叟无欺

今天挺paper reading的时候,听到了最大化互信息,还不清楚互信息是个什么东东,google了一下,从

http://en.wikipedia.org/wiki/Mutual_information

摘过来了:

 

 

 

 

Definition of mutual information

Formally, the mutual information of two discrete random variables X and Y can be defined as:

 I(X;Y) = /sum_{y /in Y} /sum_{x /in X} 
                 p(x,y) /log{ /left( /frac{p(x,y)}{p_1(x)/,p_2(y)}
                              /right) }, /,/!

where p(x,y) is the joint probability distribution function of X and Y, and p1(x) and p2(y) are the marginal probability distribution functions of X and Y respectively.

In the continuous case, we replace summation by a definite double integral:

 I(X;Y) = /int_Y /int_X 
                 p(x,y) /log{ /left( /frac{p(x,y)}{p_1(x)/,p_2(y)}
                              /right) } /; dx /,dy,

where p(x,y) is now the joint probability density function of X and Y, and p1(x) and p2(y) are the marginal probability density functions of X and Y respectively.

These definitions are ambiguous because the base of the log function is not specified. To disambiguate, the function I could be parameterized as I(X,Y,b) where b is the base. Alternatively, since the most common unit of measurement of mutual information is the bit, a base of 2 could be specified.

Intuitively, mutual information measures the information that X and Y share: it measures how much knowing one of these variables reduces our uncertainty about the other. For example, if X and Y are independent, then knowing X does not give any information about Y and vice versa, so their mutual information is zero. At the other extreme, if X and Y are identical then all information conveyed by X is shared with Y: knowing X determines the value of Y and vice versa. As a result, the mutual information is the same as the uncertainty contained in Y (or X) alone, namely the entropy of Y (or X: clearly if X and Y are identical they have equal entropy).

Mutual information quantifies the dependence between the joint distribution of X and Y and what the joint distribution would be if X and Y were independent. Mutual information is a measure of dependence in the following sense: I(X; Y) = 0 if and only if X and Y are independent random variables. This is easy to see in one direction: if X and Y are independent, then p(x,y) = p(x) × p(y), and therefore:

 /log{ /left( /frac{p(x,y)}{p(x)/,p(y)} /right) } = /log 1 = 0. /,/!

Moreover, mutual information is nonnegative (i.e. I(X;Y) ≥ 0; see below) and symmetric (i.e. I(X;Y) = I(Y;X)).

[edit] Relation to other quantities

Mutual information can be equivalently expressed as


/begin{align}
I(X;Y) & {} = H(X) - H(X|Y) // 
& {} = H(Y) - H(Y|X) // 
& {} = H(X) + H(Y) - H(X,Y)
/end{align}

where H(X) and H(Y) are the marginal entropies, H(X|Y) and H(Y|X) are the conditional entropies, and H(X,Y) is the joint entropy of X and Y. Since H(X) ≥ H(X|Y), this characterization is consistent with the nonnegativity property stated above.

Intuitively, if entropy H(X) is regarded as a measure of uncertainty about a random variable, then H(X|Y) is a measure of what Y does not say about X. This is “the amount of uncertainty remaining about X after Y is known”, and thus the right side of the first of these equalities can be read as “the amount of uncertainty in X, minus the amount of uncertainty in X which remains after Y is known”, which is equivalent to “the amount of uncertainty in X which is removed by knowing Y“. This corroborates the intuitive meaning of mutual information as the amount of information (that is, reduction in uncertainty) that knowing either variable provides about the other.

Note that in the discrete case H(X|X) = 0 and therefore H(X) = I(X;X). Thus I(X;X) ≥ I(X;Y), and one can formulate the basic principle that a variable contains more information about itself than any other variable can provide.

Mutual information can also be expressed as a Kullback-Leibler divergence, of the product p(x) × p(y) of the marginal distributions of the two random variables X and Y, from p(x,y) the random variables’ joint distribution:

 I(X;Y) = D_{/mathrm{KL}}(p(x,y)/|p(x)p(y)).

Furthermore, let p(x|y) = p(x, y) / p(y). Then


/begin{align}
I(X;Y) & {} = /sum_y p(y) /sum_x p(x|y) /log_2 /frac{p(x|y)}{p(x)} //
& {} =  /sum_y p(y) /; D_{/mathrm{KL}}(p(x|y)/|p(x)) //
& {} = /mathbb{E}_Y/{D_{/mathrm{KL}}(p(x|y)/|p(x))/}.
/end{align}

Thus mutual information can thus also be understood as the expectation of the Kullback-Leibler divergence of the univariate distribution p(x) of X from the conditional distribution p(x|y) of X given Y: the more different the distributions p(x|y) and p(x), the greater the information gain.

[edit] Variations of the mutual information

Several variations on the mutual information have been proposed to suit various needs. Among these are normalized variants and generalizations to more than two variables.

[edit] Metric

Many applications require a metric, that is, a distance measure between points. The quantity

d(X,Y) = H(X,Y) − I(X;Y)

satisfies the basic properties of a metric; most importantly, the triangle inequality, but also non-negativity, indiscernability and symmetry. In addition, one also has d(X,Y) /le H(X,Y), and so

D(X,Y) = d(x,y)/H(X,Y) /le 1

The metric D is a universal metric, in that if any other distance measure places X and Y close-by, then the D will also judge them close.[1].

[edit] Conditional mutual information

Sometimes it is useful to express the mutual information of two random variables conditioned on a third.

I(X;Y|Z) = /mathbb E_Z /big(I(X;Y)|Z/big)
    = /sum_{z/in Z} /sum_{y/in Y} /sum_{x/in X}
      p_Z(z) p_{X,Y|Z}(x,y|z) /log /frac{p_{X,Y|Z}(x,y|z)}{p_{X|Z}(x|z)p_{Y|Z}(y|z)},

which can be simplified as

I(X;Y|Z) = /sum_{z/in Z} /sum_{y/in Y} /sum_{x/in X}
      p_{X,Y,Z}(x,y,z) /log /frac{p_Z(z)p_{X,Y,Z}(x,y,z)}{p_{X,Z}(x,z)p_{Y,Z}(y,z)}.

Conditioning on a third random variable may either increase or decrease the mutual information, but it is always true that

I(X;Y|Z) /ge 0

for discrete, jointly distributed random variables X, Y, Z. This result has been used as a basic building block for proving other inequalities in information theory.

[edit] Multivariate mutual information

Several generalizations of mutual information to more than two random variables have been proposed, such as total correlation and interaction information. If Shannon entropy is viewed as a signed measure in the context of information diagrams, as explained in the article Information theory and measure theory, then the only definition of multivariate mutual information that makes sense is as follows:

I(X1) = H(X1),
I(X1 | X2) = H(X1 | X2)

and for n > 1,

I(X_1;/,.../,;X_n) = I(X_1;/,.../,;X_{n-1}) - I(X_1;/,.../,;X_{n-1}|X_n),

where (as above) we define

I(X_1;/,.../,;X_{n-1}|X_n) = /mathbb E_{X_n} /big(I(X_1;/,.../,;X_{n-1})|X_n/big).

(This definition of multivariate mutual information is identical to that of interaction information except for a change in sign when the number of random variables is odd.)

[edit] Applications

Some have criticized the blind application of information diagrams used to derive the above definition, and indeed it has found rather limited practical application, since it is difficult to visualize or grasp the significance of this quantity for a large number of random variables. It can be zero, positive, or negative for any n /ge 3.

One high-dimensional generalization scheme that maximizes the mutual information between the joint distribution and other target variables is found be useful in feature selection.

[edit] Normalized variants

Normalized variants of the mutual information are provided by the coefficients of constraint (Coombs, Dawes & Tversky 1970) or uncertainty coefficient (Press & Flannery 1988)


C_{XY}=/frac{I(X;Y)}{H(Y)} ~~~~/mbox{and}~~~~ C_{YX}=/frac{I(X;Y)}{H(X)}

The two coefficients are not necessarily equal. A more useful and symmetric scaled information measure is the redundancy[citation needed]

R= /frac{I(X;Y)}{H(X)+H(Y)}

which attains a minimum of zero when the variables are independent and a maximum value of

R_{/max }=/frac{/min (H(X),H(Y))}{H(X)+H(Y)}

when one variable becomes completely redundant with the knowledge of the other. See also Redundancy (information theory). Another symmetrical measure is the symmetric uncertainty (Witten & Frank 2005), given by

U(X,Y) = 2R = 2 /frac{I(X;Y)}{H(X)+H(Y)}

which represents a weighted average of the two uncertainty coefficients (Press & Flannery 1988).

Other normalized versions are provided by the following expressions (Yao 2003, Strehl & Ghosh 2002).


/frac{I(X;Y)}{/operatorname{min}(H(X),H(Y))}, ~~~~~~~ /frac{I(X;Y)}{H(X,Y)}, ~~~~~~~ /frac{I(X;Y)}{/sqrt{H(X)H(Y)}}

The quantity

D^/prime(X,Y)=1-/frac{I(X;Y)}{/operatorname{max}(H(X),H(Y))}

is a metric, i.e. satisfies the triangle inequality, etc. The metric D^/prime is also a universal metric.[2]

[edit] Weighted variants

In the traditional formulation of the mutual information,

 I(X;Y) = /sum_{y /in Y} /sum_{x /in X} p(x,y) /log /frac{p(x,y)}{p(x)/,p(y)},

each event or object specified by (x,y) is weighted by the corresponding probability p(x,y). This assumes that all objects or events are equivalent apart from their probability of occurrence. However, in some applications it may be the case that certain objects or events are more significant than others, or that certain patterns of association are more semantically important than others.

For example, the deterministic mapping {(1,1),(2,2),(3,3)} may be viewed as stronger (by some standard) than the deterministic mapping {(1,3),(2,1),(3,2)}, although these relationships would yield the same mutual information. This is because the mutual information is not sensitive at all to any inherent ordering in the variable values (Cronbach 1954, Coombs & Dawes 1970, Lockhead 1970), and is therefore not sensitive at all to the form of the relational mapping between the associated variables. If it is desired that the former relation — showing agreement on all variable values — be judged stronger than the later relation, then it is possible to use the following weighted mutual information (Guiasu 1977)

 I(X;Y) = /sum_{y /in Y} /sum_{x /in X} w(x,y) p(x,y) /log /frac{p(x,y)}{p(x)/,p(y)},

which places a weight w(x,y) on the probability of each variable value co-occurrence, p(x,y). This allows that certain probabilities may carry more or less significance than others, thereby allowing the quantification of relevant holistic or prägnanz factors. In the above example, using larger relative weights for w(1,1), w(2,2), and w(3,3) would have the effect of assessing greater informativeness for the relation {(1,1),(2,2),(3,3)} than for the relation {(1,3),(2,1),(3,2)}, which may be desirable in some cases of pattern recognition, and the like. There has been little mathematical work done on the weighted mutual information and its properties, however.

[edit] Absolute mutual information

Using the ideas of Kolmogorov complexity, one can consider the mutual information of two sequences independent of any probability distribution:

IK(X;Y) = K(X) − K(X | Y).

To establish that this quantity is symmetric up to a logarithmic factor (I_K(X;Y) /approx I_K(Y;X)) requires the chain rule for Kolmogorov complexity (Li 1997). Approximations of this quantity via compression can be used to define a distance measure to perform a hierarchical clustering of sequences without having any domain knowledge of the sequences (Cilibrasi 2005).

[edit] Applications of mutual information

In many applications, one wants to maximize mutual information (thus increasing dependencies), which is often equivalent to minimizing conditional entropy. Examples include:

[edit] See also

[edit] References

  1. ^ Alexander Kraskov, Harald Stögbauer, Ralph G. Andrzejak, and Peter Grassberger, “Hierarchical Clustering Based on Mutual Information”, (2003) ArXiv q-bio/0311039
  2. ^ Kraskov, et al. ibid.
  • Coombs, C. H., Dawes, R. M. & Tversky, A. (1970), Mathematical Psychology: An Elementary Introduction, Prentice-Hall, Englewood Cliffs, NJ.
  • Cronbach L. J. (1954). On the non-rational application of information measures in psychology, in H Quastler, ed., Information Theory in Psychology: Problems and Methods, Free Press, Glencoe, Illinois, pp. 14—30.
  • Kenneth Ward Church and Patrick Hanks. Word association norms, mutual information, and lexicography, Proceedings of the 27th Annual Meeting of the Association for Computational Linguistics, 1989.
  • Guiasu, Silviu (1977), Information Theory with Applications, McGraw-Hill, New York.
  • Lockhead G. R. (1970). Identification and the form of multidimensional discrimination space, Journal of Experimental Psychology 85(1), 1-10.
  • Athanasios Papoulis. Probability, Random Variables, and Stochastic Processes, second edition. New York: McGraw-Hill, 1984. (See Chapter 15.)
  • Press, W. H., Flannery, B. P., Teukolsky, S. A. & Vetterling, W. T. (1988), Numerical Recipes in C: The Art of Scientific Computing, Cambridge University Press, Cambridge, p. 634
  • Witten, Ian H. & Frank, Eibe (2005), Data Mining: Practical Machine Learning Tools and Techniques, Morgan Kaufmann, Amsterdam.
  • Yao, Y. Y. (2003) Information-theoretic measures for knowledge discovery and data mining, in Entropy Measures, Maximum Entropy Principle and Emerging Applications , Karmeshu (ed.), Springer, pp. 115-136.
  • Peng, H.C., Long, F., and Ding, C., “Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 27, No. 8, pp.1226-1238, 2005. Program
  • Andre S. Ribeiro, Stuart A. Kauffman, Jason Lloyd-Price, Bjorn Samuelsson, and Joshua Socolar, (2008) “Mutual Information in Random Boolean models of regulatory networks”, Physical Review E, Vol.77, No.1. arXiv:0707.3642.
版权声明:本文内容由互联网用户自发贡献,该文观点仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 举报,一经查实,本站将立刻删除。

发布者:全栈程序员-用户IM,转载请注明出处:https://javaforall.cn/192347.html原文链接:https://javaforall.cn

【正版授权,激活自己账号】: Jetbrains全家桶Ide使用,1年售后保障,每天仅需1毛

【官方授权 正版激活】: 官方授权 正版激活 支持Jetbrains家族下所有IDE 使用个人JB账号...

(0)


相关推荐

  • 关于异步FIFO的知识点–详细代码解释(很干)[通俗易懂]

    关于异步FIFO的知识点–详细代码解释(很干)[通俗易懂]异步FIFO的读写控制详解

  • java分层打印二叉树_基于Java的二叉树层序遍历打印实现

    java分层打印二叉树_基于Java的二叉树层序遍历打印实现层序遍历的思路:若树为空,则返回空,否则从树的第一层开始,即从根节点,从上而下逐层遍历。1.二叉树层序遍历Ⅰ——剑指offer32-Ⅰ从上到下,从左到右打印二叉树,返回一维数组int[]res。classSolution{publicint[]levelOrder(TreeNoderoot){if(root==null)returnnewint[0];Queueq=…

  • shell编程100例(附PDF下载)「建议收藏」

    shell编程100例(附PDF下载)「建议收藏」1、编写helloworld脚本#!/bin/bash#编写helloworld脚本echo”HelloWorld!”2、通过位置变量创建Linux系统账户及密码#!/bin/bash#通过位置变量创建Linux系统账户及密码#$1是执行脚本的第一个参数,$2是执行脚本的第二个参数useradd”$1″echo”…

  • php源码 辅助发卡_【程序源码】PHP自动化售货发卡网源码

    php源码 辅助发卡_【程序源码】PHP自动化售货发卡网源码GitHub:https://github.com/assimon/dujiaoka安装教程:https://github.com/assimon/dujiaoka/blob/master/wikis/bt_install.md支付接口已集成支付宝当面付支付宝PC支付支付宝手机支付payjs微信扫码.Paysapi(支付宝/微信).码支付(QQ/支付宝/微信)微信企业扫码支付全网易支付支持(针对彩…

  • python hashlib_python import hashlib出现问题

    python hashlib_python import hashlib出现问题importhashlib时出现如下问题:>>>importhashlibERROR:root:codeforhashmd5wasnotfound.Traceback(mostrecentcalllast):File”/usr/local/python3.2/lib/python3.2/hashlib.py”,line141,inglobals()[__func…

  • 一个线程有几个threadlocal_thread线程

    一个线程有几个threadlocal_thread线程基本概念程序是指令的有序集合,其本身没有任何运行的含义,是一个静态的概念。而进程是程序在处理机上的一次执行过程,它是一个动态的概念。进程是由程序、数据和进程控制块三部分组成的。进程是操作系统资源分配的基本单位,而线程是任务调度和执行的基本单位。每个进程都有独立的代码和数据空间(程序上下文),程序之间的切换会有较大的开销;线程可以看做轻量级的进程,同一类线程共享代码和数据空间,每个线程都有自己…

    2022年10月23日

发表回复

您的电子邮箱地址不会被公开。

关注全栈程序员社区公众号