I am trying to determine the mutual information for a specific variable, or I guess this information gain of a specific variable against a class?  The closest thing I could find was over here, but the results didn't seem to work out (knobbed was better than bell given the example below).  So trying a different way, I am not sure if I am getting good results.  Is there any documentation on this topic?  I didn't have much luck finding any examples or anything that really stood out.I am working with the following example:   convex flat knobbed bell sunken conical1:   2764 2448     228  440      0       02:   1032  844     612   12     32       4 Rows 1 and 2 are broad and narrow respectively.  Rows represent gill-size and columns represent cap-shape.  This is taken from a mushroom dataset.  On to what I have done so far.  Attempting to determine the mutual information for gill-size given that cap-shape is convex.First, I break this down into the following transformation.  Where V1 is convex.   with.V1 without.V11:    2764       31162:    1032       1504Then I compute the probabilities    with.V1 without.V11 0.3284221  0.37024712 0.1226236  0.1787072Then I effectively run the joint entropy formula based on those probabilities      with.V1  without.V11  0.01957270 -0.018837832 -0.01820434  0.01992437Take the summation and end up with 0.002454899 for the joint entropy h(gill-size,w and w/o cape-shape).Then I compute the entropy for h(cap-shape) and h(gill-size) based on the with and without matrix above.I(gill-size|cap-shape=convex) = h(gill-size) + h(w and w/o cape-shape=x) - h(gill-size,w and w/o cape-shape)The problem I am having is that if I run the same process for all columns I end up with knobbed being almost the same as bell.  Intuitively, I would expect bell, sunken and conical to be much better than convex, flat and knobbed.  I also assume that lower is better, ie less uncertainty?  It seems like sunken and conical should effectively be 0 and bell near 0.Scores that I get are as follows (seeking the information gain for each cap-shape variable for the gill-size):convex 1.297879flat 1.276748knobbed 0.8738317bell 0.8004766sunken 0.6303651conical 0.6152766Here is the R code that I put togetherlibrary(data.table)m <- matrix(c(2764, 2448, 228, 440, 0, 0, 1032, 844, 612, 12, 32, 4), ncol=6, byrow=T)colnames(m) <- c(convex, flat, knobbed, bell, sunken, conical)rownames(m) <- c(broad, narrow)dat <- data.table(m)specificMI <- function(dat, categ) {    sel1 <- as.matrix(dat[[categ]], ncol=1)    sel2 <- rowSums(dat) - as.matrix(dat[[categ]], ncol=1)    datTable <-  data.table(with=sel1, without=(sel2))     #entropy    pa <- prop.table(rowSums(datTable))    h1 <- -sum(pa * log(pa), na.rm=T)    #entropy    pb <- prop.table(colSums(datTable))    h2 <- -sum(pb * log(pb), na.rm=T)    #joint entropy    ptable <- prop.table(datTable)    cs <- colSums(ptable)    rs <- rowSums(ptable)    mitable <- ptable * log( ptable / (rbind(cs, cs) * rs), base=2 )    mi <- sum(mitable, na.rm=T)    h1 + h2 - mi}specificMI(dat, convex)specificMI(dat, flat)specificMI(dat, knobbed)specificMI(dat, bell)specificMI(dat, sunken)specificMI(dat, conical)
Mutual information for a specific variable
feature selection;information theory;mutual information