Convert SAS NLMIXED code for zero-inflated gamma regression to R

Question: I'm trying to run a zero-inflated regression for a continuous response variable in R. I'm aware of a gamlss implementation, but I'd really like to try out this algorithm by Dale McLerran that is conceptually a bit more straightforward. Unfortunately, the code is in SAS and I'm not sure how to re-write it for something like nlme.
The code is as follows:
proc nlmixed data=mydata;
  parms b0_f=0 b1_f=0
        b0_h=0 b1_h=0
        log_theta=0;

  eta_f = b0_f + b1_f*x1 ;
  p_yEQ0 = 1 / (1 + exp(-eta_f));

  eta_h = b0_h + b1_h*x1;
  mu    = exp(eta_h);
  theta = exp(log_theta);
  r = mu/theta;

  if y=0 then
     ll = log(p_yEQ0);
  else
     ll = log(1 - p_yEQ0)
          - lgamma(theta) + (theta-1)*log(y) - theta*log(r) - y/r;

  model y ~ general(ll);
  predict (1 - p_yEQ0)*mu out=expect_zig;
  predict r out=shape;
  estimate "scale" theta;
run;

From: http://listserv.uga.edu/cgi-bin/wa?A2=ind0805A&L=sas-l&P=R20779
ADD:
Note: There are no mixed effects present here - only fixed.
The advantage to this fitting is that (even though the coefficients are the same as if you separately fit a logistic regression to P(y=0) and a gamma error regression with log link to E(y | y>0)) you can estimate the combined function E(y) which includes the zeroes. One can predict this value in SAS (with a CI) using the line predict (1 - p_yEQ0)*mu .
Further, one is able to write custom contrast statements to test the significance of predictor variables on E(y). For example, here is another version of the SAS code I have used:
proc nlmixed data=TestZIG;
      parms b0_f=0 b1_f=0 b2_f=0 b3_f=0
            b0_h=0 b1_h=0 b2_h=0 b3_h=0
            log_theta=0;

        if gifts = 1 then x1=1; else x1 =0;
        if gifts = 2 then x2=1; else x2 =0;
        if gifts = 3 then x3=1; else x3 =0;

      eta_f = b0_f + b1_f*x1 + b2_f*x2 + b3_f*x3;
      p_yEQ0 = 1 / (1 + exp(-eta_f));

      eta_h = b0_h + b1_h*x1 + b2_h*x2 + b3_h*x3;
      mu    = exp(eta_h);
      theta = exp(log_theta);
      r = mu/theta;

      if amount=0 then
         ll = log(p_yEQ0);
      else
         ll = log(1 - p_yEQ0)
              - lgamma(theta) + (theta-1)*log(amount) -                      theta*log(r) - amount/r;

      model amount ~ general(ll);
      predict (1 - p_yEQ0)*mu out=expect_zig;
      estimate "scale" theta;
    run;

Then to estimate "gift1" versus "gift2" (b1 versus b2) we can write this estimate statement:
estimate "gift1 versus gift 2"
 (1-(1 / (1 + exp(-b0_f -b1_f))))*(exp(b0_h + b1_h)) - (1-(1 / (1 + exp(-b0_f -b2_f))))*(exp(b0_h + b2_h)) ;

Can R do this?
Comment: user779747 did note in his cross posting to Rhelp that this had been posted here first. I haven't seen a specific request to post such a notice in SO, but some (most?)  of us cross-helpeRs expect it because that is the stated expectation in the R Mailing Lists.
Answer: Having spent some time on this code, it appears to me as though it basically:
1) Does a logistic regression with right hand side b0_f + b1_f*x1 andy > 0 as a target variable,
2) For those observations for which y > 0, performs a regression with right hand side b0_h + b1_h*x1, a Gamma likelihood and link=log,
3) Also estimates the shape parameter of the Gamma distribution.
It maximizes the likelihood jointly, which is nice, because you only have to make the one function call.  However, the likelihood separates anyway, so you don't get improved parameter estimates as a result.
Here is some R code which makes use of the glm function to save programming effort.  This may not be what you'd like, as it obscures the algorithm itself.  The code certainly isn't as clean as it could / should be, either.
McLerran <- function(y, x)
{
  z <- y > 0
  y.gt.0 <- y[y>0]
  x.gt.0 <- x[y>0]

  m1 <- glm(z~x, family=binomial)
  m2 <- glm(y.gt.0~x.gt.0, family=Gamma(link=log))

  list("p.ygt0"=m1,"ygt0"=m2)
}

# Sample data
x <- runif(100)
y <- rgamma(100, 3, 1)      # Not a function of x (coef. of x = 0)
b <- rbinom(100, 1, 0.5*x)  # p(y==0) is a function of x
y[b==1] <- 0

foo <- McLerran(y,x)
summary(foo$ygt0)

Call:
glm(formula = y.gt.0 ~ x.gt.0, family = Gamma(link = log))

Deviance Residuals:
     Min        1Q    Median        3Q       Max
-2.08888  -0.44446  -0.06589   0.28111   1.31066

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)   1.2033     0.1377   8.737 1.44e-12 ***
x.gt.0       -0.2440     0.2352  -1.037    0.303
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Dispersion parameter for Gamma family taken to be 0.3448334)

    Null deviance: 26.675  on 66  degrees of freedom
Residual deviance: 26.280  on 65  degrees of freedom
AIC: 256.42

Number of Fisher Scoring iterations: 6

The shape parameter for the Gamma distribution is equal to 1 / the dispersion parameter for the Gamma family.  Coefficients and other stuff which you might like to access programatically can be accessed on the individual elements of the return value list:
> coefficients(foo$p.ygt0)
(Intercept)           x
   2.140239   -2.393388

Prediction can be done using the output of the routine.  Here's some more R code that shows how to generate expected values and some other information:
# Predict expected value
predict.McLerren <- function(model, x.new)
{
  x <- as.data.frame(x.new)
  colnames(x) <- "x"
  x$x.gt.0 <- x$x

  pred.p.ygt0 <- predict(model$p.ygt0, newdata=x, type="response", se.fit=TRUE)
  pred.ygt0 <- predict(model$ygt0, newdata=x, type="response", se.fit=TRUE)

  p0 <- 1 - pred.p.ygt0$fit
  ev <- (1-p0) * pred.ygt0$fit

  se.p0 <- pred.p.ygt0$se.fit
  se.ev <- pred.ygt0$se.fit

  se.fit <- sqrt(((1-p0)*se.ev)^2 + (ev*se.p0)^2 + (se.p0*se.ev)^2)

  list("fit"=ev, "p0"=p0, "se.fit" = se.fit,
       "pred.p.ygt0"=pred.p.ygt0, "pred.ygt0"=pred.ygt0)
}

And a sample run:
> x.new <- seq(0.05,0.95,length=5)
>
> foo.pred <- predict.McLerren(foo, x.new)
> foo.pred$fit
       1        2        3        4        5
2.408946 2.333231 2.201889 2.009979 1.763201
> foo.pred$se.fit
        1         2         3         4         5
0.3409576 0.2378386 0.1753987 0.2022401 0.2785045
> foo.pred$p0
        1         2         3         4         5
0.1205351 0.1733806 0.2429933 0.3294175 0.4291541

Now for coefficient extraction and the contrasts:
coef.McLerren <- function(model)
{
  temp1 <- coefficients(model$p.ygt0)
  temp2 <- coefficients(model$ygt0)
  names(temp1) <- NULL
  names(temp2) <- NULL
  retval <- c(temp1, temp2)
  names(retval) <- c("b0.f","b1.f","b0.h","b1.h")
  retval
}

contrast.McLerren <- function(b0_f, b1_f, b2_f, b0_h, b1_h, b2_h)
{
  (1-(1 / (1 + exp(-b0_f -b1_f))))*(exp(b0_h + b1_h)) - (1-(1 / (1 + exp(-b0_f -b2_f))))*(exp(b0_h + b2_h))
}

> coef.McLerren(foo)
      b0.f       b1.f       b0.h       b1.h
 2.0819321 -1.8911883  1.0009568  0.1334845

Comment: You are correct with regard to what is happening with the "parts" (i.e logit regression for PR(y>0) and gamma regression for E(y|y>0) but it is the combined estimate (and standard errors, CI) that are of main interest - i.e E(y). Predictions of this quantity are made in the SAS code by (1 - p_yEQ0)*mu. This formulation allows you to conduct contrasts on the coefficients on this combined value.
Comment: @B_Miner - I've added some code + examples that partially address the prediction issue, thanks for pointing that out.
Comment: Is this not just separate estimates though? In SAS, NLMIXED will give the abiity to estimate the point estimate of E(y) as well as a CI (using the delta method I believe). Also, you can write user defined contrasts of the parameters as I shown above to test linear hypothesis. There must be an R alternative?
Comment: Well, yes and no.  To use the example, the returned `foo.pred$fit` gives the point estimate of E(y), but the component `foo.pred$pred.ygt0$pred` would give you E(y|y>0).  I added in the standard error calculation for y, BTW, returned as se.fit.  The coefficients can be gotten from the components by coefficients(`foo.pred$pred.ygt0`) and coefficients(`foo.pred$pred.p.ygt0`); I'll write an extraction routine and a contrast routine in a little while.
Comment: Can you please describe where this comes from:  se.fit <- sqrt(((1-p0)*se.ev)^2 + (ev*se.p0)^2 + (se.p0*se.ev)^2)
Comment: It is the formula for the std. deviation of a product of two random variables when the two variables are independent.  In this case I make use of the fact that var(1-p0) = var(p0).  The formula for the variance is (E(x)SD(y))^2 + (E(y)SD(x))^2 + (SD(x)SD(y))^2.
Comment: Thanks a lot for such a helpful answer! I haven't been here for a while and ended up solving the problem by moving to non-parametric methods altogether. Nonetheless, I'm now studying your code with great interest and will very likely use it in the future. All the best, Mike
Comment: PS. I can see that while PROC NLMIXED maximizes the likelihood jointly, in the R implementation you propose the models seem to be completely separate. You are saying however that while "the likelihood separates anyway, so you don't get improved parameter estimates as a result". Would you mind explaining this point a bit more? I suppose the joint likelihood is in the SAS code is estimated as " log(1 - p_yEQ0) - lgamma(theta) + (theta-1)*log(amount) - theta*log(r) - amount/r". What would be the equivalent way to get a single likelihood estimate for this model in R? Many thanks!


Need help for extracting columnar data

Question: I wrote the code as below its extracting all tabular data by splitting. Now i want to
extract the last column(Values) data by removing Text in the middle of the table.
    import re
    f_read = open('filename','r')
    f_write=open('filename','w')
    str='CEN/4'
    for line in f_read:
        m=re.search(str,line)
        if m:
           for line in f_read:      # loop over the rows
                if not line.strip():
                  continue
           line = line.split(' ',1)   # parse the columns
           print line
           f_write.writelines(line)

Table looks like this followed by values in each column.Now i want to remove the text and extract the values.
        S T R E S S E S   I N   Q U A D R I L A T E R A L   E L E M E N T S   ( Q U A D 4 )        OPTION = BILIN

ELEMENT              FIBER            STRESSES IN ELEMENT COORD SYSTEM         PRINCIPAL STRESSES (ZERO SHEAR)
  ID      GRID-ID   DISTANCE        NORMAL-X      NORMAL-Y      SHEAR-XY      ANGLE        MAJOR         MINOR       VON MISES

  121      3423      342             4545          45435        4345           42353     3456456       67658          3435

Comment: Why are you searching for `'CEN/4'`?
Comment: So you just want to turn the last row of that table into a list of integers?
Answer: If I've understood your question, there's several things wrong with your code. Notably, you can't read and write to same file in the manner shown. To complicate the situation, there are nestedforloops, both reading lines from the same input file.
Here's a revised version that avoids those problems by writing to a separate output file and doing everything under the control of single for loop. I also duplicated the line of data in the input file and changed all the numbers on it so they start with a 9 so there would be more than one line of data for testing purposes.
import re

with open('filename','r') as f_read, open('filename2','w') as f_write:
    for line in (line.strip() for line in f_read):
        if re.search(r'^\d+', line):  # line starts with a number?
            columnar_data = ' '.join(line.split())
            print columnar_data
            f_write.write(columnar_data+'\n')

Contents of output file (filename2):
121 3423 342 4545 45435 4345 42353 3456456 67658 3435
921 9423 942 9545 95435 9345 92353 9456456 97658 9435

Comment: its actually working fine but the above table starts from the middle of the text file so what i did is to search for a string and from that it has to start.it is reading the whole file from searched string to end of the file.Now i want to stop at certain point which doesn't have the string character.And i want to extract the last column data.
Comment: What certain point do you want to stop at? When you reach it, stop writing out data until you've read the last line, when that happens write the last line read (which until now you had been ignoring since you reached the certain point) and you're done.
Answer: I can't quite tell what you are trying to do. But if all you want to do is read in the data from the last line of that file, you can do something like this:
def read_last_line(file_path):
    f = open(file_path, 'r')
    lines = f.readlines()
    f.close()
    last_line = lines[-1].split()
    return last_line

Notice that lines[-1] is a quick way to get the last element in an array. In this case, that array is an array where each element is a line in your file. I think this may be the important trick you're looking for.
Answer: In this case, I think it is likely that in the real case there are many lines containing values (one for each of many element IDs). To output all the values only, I'd suggest something like:
import re
f_read = open('quad4.txt','r')
f_write=open('quad4.out','w')
str = 'GRID-ID'
for line in f_read:
    m=re.search(str,line)
    if m:
        break

for line in f_read:   # loop over the rows
    line = line.strip()
    if line:
        vallist = line.split()   # parse the columns
        print (vallist)
        f_write.write("%s\n" % "\t".join(vallist)

This reads the file until it finds a line containing "GRID-ID" then it starts splitting any non-empty line and writing it out in tab-delimited columns. That makes further processing easy. Alternatively, at the third-to-last line of the code above, the values are split into vallist in a convenient form for further processing. Note that in vallist the values are still strings, so if you were to do computations with them, you would need to convert them into int or float values first.

Need help for extracting columnar data


React native json stringify return my array correctly but it cannot show specific object in array

Question: When I use {alert(JSON.stringify(cart[0]))} in my react-native app it shows me the whole array of object like this:
[{
  "id": 3,
  "name": John,
  .
  .
}]

but when I use {alert(JSON.stringify(cart[0].id))} for example it returns me undefined.
Answer: The code you provided contains object with two nested arrays:
const cart = [
   [
     {
       id: 3,
       name: 'John',
       // ...
     }
   ],
]

When you try to receive value of cart[0].id you actually refer to id property of the second array. To extract the value you want use cart[0][0].id.
Comment: thanks alot....but how can i refer this items inot FlatList data?? i used to use const { cart } = this.props
Comment: According to the rules of StackOverflow you should to create new question and describe this exact problem (with FlatList) to let others answer your question. Thank you.
Comment: Also, please mark correct answers as correct (if they are) using a checkmark. https://stackoverflow.com/help/accepted-answer


Detecting raid setup in centos 5

Question: We have bougth a server off of Iweb web hosting and its supposed to have raid5 hardware installed. At this point so many things have gone wrong with IWeb we are just not sure we have our raid at all.
From ssh is there any command I could use to see what kind of raid we have setup?
Seems like mdadm is not installed and none of the other I have found.
df -h gives:
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5             1.8T  4.3G  1.7T   1% /
/dev/sda2             2.0G   36M  1.9G   2% /tmp
/dev/sda1              99M   12M   82M  13% /boot
tmpfs                 5.9G     0  5.9G   0% /dev/shm
tmpfs                 5.9G     0  5.9G   0% /usr/local/psa/handlers/before-local
tmpfs                 5.9G     0  5.9G   0% /usr/local/psa/handlers/before-queue
tmpfs                 5.9G     0  5.9G   0% /usr/local/psa/handlers/before-remote
tmpfs                 5.9G     0  5.9G   0% /usr/local/psa/handlers/info
tmpfs                 5.9G  4.0K  5.9G   1% /usr/local/psa/handlers/spool

Comment: Please consider accepting an answer if any of these have helped you resolve your question.
Answer: See: How to find RAID configuration/level and RPM speed having only remote access to the server?
Depending on the server hardware, you can try a couple of things to find the config. Start with dmidecode piped into less to obtain the server make/model information (and potentially, the serial number). For Dell, dmidecode | grep -i serial should output the service tag. You can cross reference the initial build info from Dell's support site.
For Dell and most hardware raid controllers, cat /proc/scsi/scsi should reveal the SCSI controller and/or RAID controller config.
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: TEAC     Model: DVD-ROM DV-28SW  Rev: R.2A
  Type:   CD-ROM                           ANSI  SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: SEAGATE  Model: ST9146803SS      Rev: FS62
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi2 Channel: 00 Id: 01 Lun: 00
  Vendor: SEAGATE  Model: ST9146803SS      Rev: FS62
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi2 Channel: 01 Id: 00 Lun: 00
  Vendor: Dell     Model: VIRTUAL DISK     Rev: 1028
  Type:   Direct-Access                    ANSI  SCSI revision: 05

Armed with the drive model numbers, you can google to obtain the specifications. The last entry in the output clearly shows a virtual disk comprised of the two physical disks listed above. You can get the exact controller spec from lspci -v.
Answer: mdadm is not needed if you really have hardware raid, as the OS should only see one device for every raid array in the system.
One way to detect an hardware raid controller could be to scan your boot.msg log file for scsi devices that identify themselves as RAID controllers. I.e. I have a system with a 3ware controller and find the following lines in my kernel boot log:
<6>3w-9xxx 0000:03:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
<7>3w-9xxx 0000:03:00.0: setting latency timer to 64
<6>scsi2 : 3ware 9000 Storage Controller
<4>3w-9xxx: scsi2: Found a 3ware 9000 Storage Controller at 0xba900000, IRQ: 16.
<4>3w-9xxx: scsi2: Firmware FE9X 3.08.00.016, BIOS BE9X 3.08.00.004, Ports: 16.
<5>scsi 2:0:0:0: Direct-Access     AMCC     9650SE-16M DISK  3.08 PQ: 0 ANSI: 5

Answer: If this is a virtual server of some form then no, there's no way you can know what's actually servicing your disk IO. If it's a physical server you may be able to install (or find if already installed) any disk controller tools/diags code that would be able to tell you what the disk layout is. There's certainly nothing in the 'df' you provided that would point one way or another. Ultimately it's a question best asked of them, if you get more details from them feel free to come back here and we'll see what we can do ok.
Comment: Yeah its a physical server. I'll try to ask them but as it is they have serviced us VERY badly beyond belief. Thanks for your help!
Comment: Good point regarding if the server is virtualized. +1
Answer: Try using lspci to determine if there is a hardware RAID controller. If there is a hardware controller the OS will not normally be able to see the RAID using the normal tools, however there are normally Linux based clients available for the big vendors (PECL/3com/etc).
Another useful tool to see lowlevel hardware is dmidecode. It can tell you a lot about the physical box your using.
Answer: mdadm is for Linux software RAID. Try probing the hardware with dmidecode to determine the hardware vendor for the RAID controller.  From there, you will need to use the vendor specific utility to query the RAID controller for it's configuration.
See this question for more information:
Is it possible to get an inventory of all physical hard disks on a linux server?

Detecting raid setup in centos 5


Uniform random numbers adding up to a given $L$

Question: Fix $L \in \mathbb R^{>0}$, let $x_1, x_2, x_3, \ldots$ be IID uniformly random numbers in $[0, 1]$, and
$$\mathcal N := \min \{{N \in \{1, 2, 3 , \ldots\}} \mid x_1 + x_2 + \cdots + x_N \ge L\}.$$
It's a well-known exercise that $E(\mathcal N \mid L = 1) = e$. It's not that much harder to describe the distribution for $L \le 1$ and see that
$$E\left(\mathcal N \mid 0 \le L \le 1\right) = e^L.$$
Is there a good reference that describes the distribution on $\mathcal N$ for $L > 1$, where it gets more complicated?
The reference provided here, Uspensky 1937, p. 278, doesn't appear to actually discuss the distribution of $\mathcal N$.
Answer: For $n > L$,
$$
\mathbb P(\mathcal N>n)=\mathbb P(x_1+\ldots+x_n <L) = \frac{1}{n!}\sum_{k=0}^{\lfloor L\rfloor} (-1)^k \binom{n}{k}(L-k)^n.
$$
The r.h.s. is the CDF of Irwin–Hall distribution.
For $n\leq L$, $\mathbb P(\mathcal N>n)=1$.
To find pmf of $\mathcal N$, get $\mathbb P(\mathcal N=n)=\mathbb P(\mathcal N>n-1)-\mathbb P(\mathcal N>n)$.
Comment: Not very enlightening though.


How to add additional feature/function in DataTables?

Question: I have created a basic datatable, now I want to customize it by adding filter in every column. I already got the solution from:
https://www.datatables.net/examples/api/multi_filter_select.html
screensotdatatblesite
BUT, where I should put the additional code? I have tried copy paste the code in jquery.datatables.js and also try to put in new js file, neither both of them worked.
Please kindly help me...
code that I got from datatables website:
$(document).ready(function() {
    $('#example').DataTable( {
        initComplete: function () {
            this.api().columns().every( function () {
                var column = this;
                var select = $('<select><option value=""></option></select>')
                    .appendTo( $(column.footer()).empty() )
                    .on( 'change', function () {
                        var val = $.fn.dataTable.util.escapeRegex(
                            $(this).val()
                        );

                        column
                            .search( val ? '^'+val+'$' : '', true, false )
                            .draw();
                    } );

                column.data().unique().sort().each( function ( d, j ) {
                    select.append( '<option value="'+d+'">'+d+'</option>' )
                } );
            } );
        }
    } );
} );

Comment: Please update your question with more information about what you are doing and some code samples.
Comment: hi @xjmdoo Updated! please check the screenshot and code, hopefully its clear
Comment: You can need to link a new script file or use a `
Answer: You asked a very naive question. Hopefully this helps:

You need to first add all dependencies of DataTables in your HTML file:

<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/1.10.16/css/jquery.dataTables.min.css">

Create the table in HTML with all data inside
 tag or do some scripting to insert data into table as new rows.
Once all data is present in table, you can your own custom script and add into the HTML:

urOwnScript.js can be written in two ways.
    * A. If you already have the table with data, then initialize the datatable once the page load is done.
    * B. If you are fetching data and editing html DOM with new rows, then once that operation is finished initialize datatable.
For case A, contents of urOwnScript.js can be like this:

$(document).ready(function() { // Means this is run only on page load, which means
 tag has all the data already.
    $('#YourIdOfTableTag').DataTable( {
        initComplete: function () {
            this.api().columns().every( function () {
                var column = this;
                var select = $('<select><option value=""></option></select>')
                    .appendTo( $(column.footer()).empty() )
                    .on( 'change', function () {
                        var val = $.fn.dataTable.util.escapeRegex(
                            $(this).val()
                        );

                        column
                            .search( val ? '^'+val+'$' : '', true, false )
                            .draw();
                    } );

                column.data().unique().sort().each( function ( d, j ) {
                    select.append( '<option value="'+d+'">'+d+'</option>' )
                } );
            } );
        }
    } );
} );

Comment: Your explanation is so clear! it works. I am so new in programming, I will learn more. Thanks @Vishwaraj


Artin-Wedderburn: Decomposition of a semisimple Dual Hopf algebra

Question: 1. Context
My lecture notes prove that any cocommutative finite-dimensional Hopf algebra over a field $k$ of characteristic zero is semisimple and cosemisimple.
They try to argue from there that any finite-dimensional, cocommutative Hopf algebra over a field of characteristic zero is isomorphic to a group algebra:

Since $H^*$ is semisimple, it is, as an algebra, isomorphic to $H^* \cong k \times. . . \times k$ by the Artin-Wedderburn theorem. The projection $p_i$ to the $i$-th factor is a morphism of algebras or, put differently, a grouplike element in $H^{**} \cong H$. All projections give a basis of $H$ consisting of grouplike elements. Thus $H$ is a group algebra of a finite group.

2. Question

Why does the isomorphism $H^* \cong k \times. . . \times k$ exist? Where is the Artin-Wedderburn theorem used?

The Artin-Wedderburn theorem gives an isomorphism $H^* \cong \prod M_{n_i}(D_i)$ where the $n_{i}$ are natural numbers, the $D_i$ are finite dimensional division algebras over $k$ and $M_{n_i}(D_i) $ is the algebra of $n_i \times n_i $matrices over $D_i$. If $k$ were algebraically closed we would even know that $H^* \cong \prod M_{n_i}(k)$ holds. How to proceed? I am not familiar with Artin-Wedderburn, I guess. So any hint would be appreciated.
Comment: not clear on the notation. $H$ is the Hopf algebra and $H^\ast$ is the algebra structure given by the coalgebra data on $H$?  Does $H$ being cocommutative and semisimple mean $H^\ast$ is commutative and semisimple? because that answers a lot: it woudl definitely be a product of fields, but not obviously over $k$. They could be finite extensions of $k$. Unless $k$ were algebraically closed, then it would be all true.
Comment: 1st Question: Yes, by taking the dual map of the coproduct (with suitable restriction) as the product. The unit is the dual map of the counit. 2nd: Ah, of course, thanks! Yes, then $H^*$ is indeed commutative. That solves the problem. Cosemisimple in the finite-dimensional case just means $H^*$ is semisimple (see above). Assuming that the lecturer forgot to mention that $k$ is algebraically closed, then all falls into place neatly (c.f. Corollary 2.4.2. in "Finite dimensional algebras" by Drozd and Kirichenko).
Comment: I understand the A-W theorem, but I don't understand Hopf algebras.
Answer: Since you confirmed it in the comment above:
It sounds like the intended context was an algebraically closed field, and that $H^\ast$ is a commutative semisimple ring.
Using the Artin-wedderburn theorem it is easy to see why all the matrices must have side length $1$ in order to be commutative. That makes a product of fields, each one a finite field extension of $k$.  But adding algebraiclly closed means that these fields are in fact just $k$.


Is it possible to make quantitative comparisons using income inequality metrics?

Question: I've calculated income inequality metrics (Gini/Hoover/Theil coefficients) for several populations. I know I can make claims like "this population has a higher Gini coefficient, so its distribution in this variable is more unequal than this other population."
But is it possible to make more quantifiable claims? For example, to say that "since the Gini coefficient for this population is twice as large, the variable is distributed eight times more unequally?" Or something like that?
UPDATE
The data isn't economic data - this has to do with relative numbers of contributions to a volutneer project. There are 5 datasets, with ~10,000 rows each. Each row corresponds to a single Internet user who did one or more tasks to contribute to that project. Some did just one, some did thousands.
The question is: of the five projects, which ones are such that a small core group has done a lot of tasks each, and which ones are such that many people have done a few tasks each to add up to the total?
FURTHER UPDATE
Thinking about this more, maybe the Gini coefficient doesn't have enough information to make this kind of comparison. Maybe I need to compare the actual distributions of the variable among population members, using something like a K-S test?
Comment: Could you update your question text and question with what you are trying to do? The question suggests you're interested in income, but you are looking at something different given your comment below.
Comment: You should have a look at the methods in the book: Handcock,  Morris: "Relative Distribution Methods in the Social Sciences" (Springer). Plots such as the relative distribution might be more interpretable than the better known QQplot.
Answer: The issue with this type of inter-country comparison for income is that it appears to ignore government transfers, e.g. tax rebates for lower income couples with children, socialised healthcare, etc. This means that it is very difficult to get consistent measures of income between countries, and that any relatively simplistic measure of income (e.g. earnings plus interest/dividends) won't reflect "in practice" income.
So while it looks like a coefficient of .5 shows twice as high inequality as a coefficient of 0.25, that may not be an accurate portrayal of the real levels of inequality.
Update questions: the extra information you have given is valuable and it looks like something other than an income inequality measure will be more appropriate given your data. For the task information, are the tasks the same duration, or could some tasks take longer? One would expect that longer tasks would equate to fewer tasks completed, so comparing number of tasks performed may not be a good comparator for "volunteer effort". A volunteer who completed a few long duration tasks could have contributed the same amount of effort as a volunteer who completed many short tasks. I'm not sure if this is relevant in your case.
If tasks are the same length, you could look at boxplots of number of tasks per subject (use project as the group for the plot). That will clearly show which projects are associated with more tasks.
The boxplot information should also suggest what type of test is best to use to compare the projects. It could be that an ANOVA (Analysis of Variance) is the best way forward.
Comment: Actually, this isn't economic data, it's data about the numbers of contributions to volunteer projects. The research question is: which projects come from a few volunteers who do many tasks vs. many who do only a few tasks each?

The situation is mathematically like income distribution, where "income" is actually volunteer contributions. (No judgment implied here - just a mathematical model. Also, let me know if this analogy is not accurate.)

I know I'm safe treating this as ordinal - projects with higher Ginis are more unequal - but can I say anything about relative *amounts* of inequality?
Answer: I talked to a friend of mine here. As I was starting to suspect in my edited question, you can't compare Gini coefficients in this way. The Gini coefficient is a summary statistic that erases details of the distributions. To make quantitative statements about the different levels of inequality in distributions, you have to compare the distributions themselves.
I think I can do this with a K-S test. If I run into trouble, I'll post a new question.
Comment: The K-S test has exactly the same problem you describe with the Gini coefficient: it "is a summary statistic that erases details of the distributions."  If you want to get beyond that, *draw a picture* of the distributions.  A good one would be a q-q plot.  (The K-S merely tells you the maximum deviation between that plot and a diagonal line.)


Create procedure with a insert statement in Oracle pl/sql

Question: I am trying to create procedure with this insert statement which otherwise working fine but when trying to incorporate this insert query into the procedure it giving me below error
Error(11,1): PL/SQL: SQL Statement ignored and
Error(16,17): PL/SQL: ORA-00904: "DBMS_RANDOM"."VALUE": invalid identifier

I need to run this proc for 10 months in order to replicate the data for 10 month in a loop, but I am stuck on step one, and proc is not getting compiled.
{
    create or replace PROCEDURE datareplicationProcedure
    (Start_date IN Date, End_date IN Date) AS

     BEGIN

        `insert statement`
    insert into SAM_CDS.transactions
(TENANT_CD,TRANSACTION_KEY, BATCH_ID ,ACCT_CURR_AMOUNT, BATCH_DATE_TIME,DAY_SK,WEEK_SK, MONTH_SK )
    (
    Select A.TENANT_CD, A.TRANSACTION_KEY,A.BATCH_ID,A.acct_curr_amount,A.BATCH_DATE_TIME,D.DAY_SK,D.WEEK_SK, D.MONTH_SK
    from (
    ( select TENANT_CD,'TRANSACTION_'||(DBMS_RANDOM.string('x',10))as TRANSACTION_KEY,BATCH_ID,acct_curr_amount ,
    TO_DATE( TRUNC( DBMS_RANDOM.VALUE(TO_CHAR(DATE '2021-07-01','J'),TO_CHAR(DATE '2021-07-31','J'))),'J') BATCH_DATE_TIME
    from SAM_CDS.transactions where batch_date_time
    between to_date('04-FEB-19') and to_Date('05-FEB-19')
    )A
    left outer join
    (select * from SAM_CDS.DAY where DATETIME between Start_date and End_date)D
    on D.DATETIME=A.BATCH_DATE_TIME )
);

     END datareplicationProcedure;
}

Comment: Please read [Under what circumstances may I add "urgent" or other similar phrases to my question, in order to obtain faster answers?](//meta.stackoverflow.com/q/326569) - the summary is that this is not an ideal way to address volunteers, and is probably counterproductive to obtaining answers. Please refrain from adding this to your questions.
Answer: It would appear that the owner of the procedure has not been given access to the dbms_random package via a direct grant.  If the insert statement compiles outside of a stored procedure, the grant was most likely through a role rather than through a direct grant.  Definer's rights stored procedures cannot make use of privileges granted through a role only those privileges that are granted directly to the user.  You could change the procedure to be an invoker's rights stored procedure which can use permissions granted through a role but that would require that the role was available to the caller in every session where the procedure is invoked.  More likely, you want to ask your DBA to grant access to the dbms_random to the procedure owner directly.
Comment: Thanks Justin , you are right , I asked DBA team to give permission with grant option and it worked.
Comment: Now i am facing other issue, the date i am passing in the proc , i want to use in DBAM_RANDOM function , but its giving me error of missing expression.
Comment: TO_DATE( TRUNC( DBMS_RANDOM.VALUE(TO_CHAR(DATE (Start_date),'J'),TO_CHAR(DATE (End_date),'J'))),'J')
Comment: @shweta - I don't understand what `date(start_date)` and `date(end_date)` are supposed to represent.  There is no `date` function.  You could mean `to_date` but that would only make sense if you were passing in a string.  Your procedure declaration indicates that you are passing in a date so it makes no sense to cast the parameter to the same data type.
Comment: thank you again Justin :) i was doing a silly mistake which you mentioned correctly, it accepted the parameter when i remove the date part from the function . officially this is my 1st stored proc , now i am going add a loop in this proc to run it for 10 month , wish me luck :)


Como limitar IPs que podem conetar remotamente ao mysql?

Question: Tenho um servidor centos 7 com MySQL 5.6 instalado, consigo controlar os privilégios de cada usuário/host com o GRANT.Mas minha dúvida é como permitir que somente alguns IPs conectem ao banco. Atualmente qualquer IP pode conectar, mas só os que tem algum privilegio executam algo.Alguma sugestão?
Comment: seria isso util https://unix.stackexchange.com/questions/159873/whitelist-source-ip-addresses-in-centos-7
Answer: Você precisa fazer REVOKE, veja mais detalhes no link: https://dev.mysql.com/doc/refman/5.6/en/revoke.html
REVOKE ALL PRIVILEGES ON test.* FROM 'jane'@'22.214.171.124';
Comment: Sim, já tentei com o REVOKE, mas a conexão ainda é permitida.
Answer: Consegui resolver excluindo a linha da tabela mysql.user em que o host='%' e o usuário='usuario'.Não sei se é a melhor maneira, mas funcionou, agora só conectam os usuários/hosts que tem algum privilégio.


How do I play what looks like a double legato?

Question: What is this (in the green rectangle)? To me it looks like 2 legatos. How do I play it on the violin?
It's the beginning of the 15th piece, "A-hunting we will go", from the Doflein Method, Volume III.
Comment: On guitar, I'd use pull-off and hammer on. Might that work on the same bow stroke on violin?
Comment: As a violinist I disagree; this could be intentional. I'll write more when I'm not on a phone, but the short answer is you would articulate it just like the ties in the first measure. But yes, please identify the piece and where you found it.
Answer: [Disclaimer: I'm going to give this a more thorough and serious answer than a short tune in a method book might warrant, since the same issues appear in larger pieces. I think what I'm about to say is true even of this example, though.]
For many instruments, like piano or winds, slurs are only about articulation. For bowed string instruments, though, they communicate something extra: they affect the direction that you physically move the bow. Because of that, sometimes we see slurs for one reason and sometimes for the other. Looking at the first measure, think about the "slurs" between the first two Cs (really, it's a "tie" when it's between the same pitch). For violin, the most basic meaning of slur is "keep your bow moving in the same direction as you change from this note to that." But if you took that literally, there would be nothing to mark the start of the second note; you would wind up playing a single dotted-quarter C. So instead, you articulate the second note by interrupting the bow slightly. It's up to you and your interpretation to decide how much; is it a full stop? Just a slight slow-down? How abrupt is the stop and the start? (The same idea is sometimes indicated by using no slur but printing two down-bow symbols over the first pair of notes and two up-bows over the next pair. And, although this interrupted bowing is the only reasonable explanation for these ties, it's usually notated by adding an expression mark like a dot or dash to one or both notes under the slur.)
In deciding these questions, let's figure out whether the slur is there for expressive reasons, as an articulation, or for is there to control the direction of the bow. Why would we need to control bow directions? Why care about up bows vs down bows at all? Well, it's easier to put a stronger emphasis on the down bow; our arm is moving (more or less) with the force of gravity and its own dead weight; the stroke starts closer to the frog and therefore has the physical advantage of leverage. There has been a tradition of putting the "strong" beats of the measure—especially the first, the downbeat—on a downbow. This so-called "rule of the downbow" is especially applicable in baroque music; here's a good explanation from (my friend) David Wilson.

Although he talks about Georg Muffat there, this principle of "adjust the bowing as necessary to put strong beats on downbows" is evident way before Muffat, in the earliest of violin treatises by Francesco Rognoni in 1620. Even in music from later time periods, the principle is still evident. So, for instance, it assumed that the eighth note pickup in this example is up-bow, so that the next downbeat can be down-bow. If you continue alternating bows through the passage, you arrive at the A, the third downbeat, on a downbow. If you then deleted the slur that spans "F-E-F," leaving only a two-note slur and making it match the eighth notes of the previous measure, then the final downbeat would come out up-bow.
I think we can conclude that the two-note slur is present for expressive purposes: if bow direction were not a concern, we would bow these eighths the same as in the previous measure. And the three-note slur is present for practical purposes of bow direction. So as you articulate it, you should do your best to make it sound exactly the same as if you had changed bow direction on the final F.
It's worth noting that there's no mention of a separation in the slur that starts that measure, the A to G. This makes this sustain stand out in the phrase as the only slur longer than a quarter note; in fact, the "slurs" in the first measure might be even more separated than if we had changed bow direction on every note. The rhythm of the sustains come out as, to make up a vocalization:
"di daa, di daa, di doodle di da; di doooooooeee doodle di daaa."
(By the way... is this a good bowing? You're certainly free to editorialize it as you want. And if this were a baroque text, I would delete the "hooked" slurs in the first measure and embrace a "strong-weak" bowing pattern for those quarters and eighths. But since it's a method book, and presuming you're a student, it's best to take it at face value and do what it says for the moment.)
Answer:
The two slurs in the green rectangle show two different things.
The slur from F to E is a legato slur that indicates how to phrase this bit, the other slur indicates that you continue with the bow stroke in the same direction.
It works this way:
If you start from the beginning with an up-bow stroke on the up-beat on the note G then you will arrive with an up-bow stroke on the F in the green rectangle. Here you play F-E with up-bow stroke then stop the bow and then continue with up-bow stroke on the last F. After that you will arrive with a down-bow stroke on the E that ends the phrase.
Thus the notes in the green rectangle will be played witrh a phrasing that is similar to the phrasing in the first part of bar 2 although the bow strokes are different.
¤ ¤ ¤
15th of August: Here is an elaboration inspired by a comment added by @gidds :
Comment: Is that notation specific to that edition or Method? It's a _looong_ time since I played any violin, but I don't recall seeing that use of slur marks before. (It's bad enough that they get used for ties, articulation, _and_ phrasing — a fourth meaning seems excessive!) Also, does that explain the notation in the first bar, which would otherwise be better as a couple of dotted crotchets?
Comment: @gidds see my answer
Comment: @gidds Don't forget multiplets, sometimes also marked with what looks like a slur, *usually* but not always with a number in the middle.  So five meanings?
Comment: @gidds The notation in the first bar can be misuinderstood since someone could argue that it is tie. I made an elaboration in my post in order to elabotate on the matter.


Save objects with many redundant information (with core data)

Question: I am designing an app in Objective-C that uses a core data structure.
I have the following structure :
<code>@interface classA : NSManagedObject
@property(nonatomic, strong) someType1 * property1;
..
@property(nonatomic, strong) someTypeN * propertyN;
@property(nonatomic, strong) NSSet * children;
@end
</code>
and
<code>@interface classB : classA
@property (nonatomic, strong) classA * parent;
@end
</code>
I have the following features :
1) Each object of classA will have many children in classB. (Objects in classB don't have children themselves).
2) Moreover, most objects of classB will share many properties in common with they parent (for instance, you can think that in most cases, only property1 will differ between an object of classB and the same property in its parent in classA, so for x in classB x.property2 = x.parent.property2 and so on).
3) I will only query the database through requests on object of type classA.
I am looking for a way to reduce the disk memory usage of my app by storing only the necessary properties of objects of type classB. For instance, I could keep properties of an object of classB set to nil unless it differs from the one of its parent, defining the getter of classB as :
<code>- (sometypeX*) getPropertyX {
    if (propertyX) return propertyX;
    return parent.propertyX;
}
</code>
My questions are :
1) Am I really going to gain disk memory by filling my database with nil values instead of actual values
2) Are there drawbacks to such a construction
3) Are there better ways / design patterns to deal with this issue ?
Thanks in advance for any help.
Answer: Core Data is highly optimized. It already does what you're trying to implement using faulting. From Faulting and Uniquing in the docs:

Because a fault is not realized, a managed object fault consumes less memory, and managed objects related to a fault are not required to be represented in memory at all.

To answer your questions…

1) Am I really going to gain memory by filling my database with nil values instead of actual values

If you gain any memory with this approach, it's probably insignificant.

2) Are there drawbacks to such a construction

It adds lots of complexity to your code base without much justification. It makes your code harder to read, understand, and maintain.

3) Are there better ways / design patterns to deal with this issue

Just use Core Data in a way that's easy to understand, and let it handle the optimization unless you have a clear, measurable need to optimize further.
Comment: Just to be clear I was thinking of disk memory (to store all my objects in the database), not the RAM. I think your answer deals with the latter case ?
Comment: @user3246191 I don't *think* this approach will save you any disk space. See *uniquing* on the same page. Core Data won't create duplicate records. One exception is that large `NSData` objects might be cached in virtual memory (in your app's temp directory). In this case you should write the files to disk yourself, and store the path in Core Data.
Comment: And BTW, while I'm 99% sure your approach won't give you significant memory improvements, I'm less sure about the disk space improvements. You might want to put together a sample app to measure the difference.
Comment: Ok, I don't deal with large NSData, just light properties like NSNumber or NSString, but there may be many of them. Unless somebody else answers I will do some measurements in a few days and report the results here.
Comment: @user3246191 Sounds good. I look forward to using your results!
Answer: I have made some testing following the question I had asked.
I tested the following setting : I have a classA with 11 NSString properties and children in classB. classB is a subclass of classA with the same properties and a parent property.
I created randomly between 100 and 1000 instances of classA, each of which having between 10 and 1000 children in classB. For the objects of classA, I set each string property to be a random 10 characters string.
I tested two procedures :
 - either the fields of the children of an objectA is set to nil
 - or it is set to its parent property.
And I monitored the sizes of my database files.
Unfortunately I was not able to extract a linear law from the rate of growth of the files (while this works with only one class, I couldn't figure it out with this more complicated parent/children structure). Moreover I couldn't quite understand the repartition of the sizes between the file ending with .sqlite and the one ending with .sqlite-wal.
However in all the regimes tested (where the total db size is between 1 Mb and 100 Mb) I found that I gained around a factor of 3 when using the first procedure compared to the second.
So for this regime of use not setting some properties seem to lead to a negligible gain in db size.
Please tell me if you want me to elaborate more on this answer.


Adding Parentheses to Lambda Expression

Question: I'm new to lambda calculus and was wondering if transforming the lambda expression
$v\lambda v.v$
into
$v(\lambda v.)v$
produces the same expression. Could someone help out?
Comment: Your second version is not a valid expression in the $\lambda$-calculus. $\lambda$-abstraction is over a term: given a term $v$ its $\lambda$-abstraction is ($\alpha$-equivalent to) $\lambda x . v$. With parentheses, your initial expression $v \lambda v . v$ should be $v (\lambda v . v)$.
Answer: I'm reposting Benedict Eastbaugh's comment CW:

Your second version is not a valid expression in the $λ$-calculus. $λ$-abstraction is over a term: given a term $v$ its $λ$-abstraction is ($α$-equivalent to) $λx.v$. With parentheses, your initial expression $vλv.v$ should be $v(λv.v)$.

Adding Parentheses to Lambda Expression


Zero pad Drupal 7 Form API Date Control day and month HTML select element

Question: I'm using the Form API Date controls for my form, and I am wondering if there is a way to zero pad the month and day values in the HTML select elements for values less than 10 at declaration time. I am using the following render array declarations in the page callback function.
<code>$form = array();

$form['date_from'] = array(
    '#title' => t('Date From'),
    '#type' => 'date',
);

$form['date_to'] = array(
    '#title' => t('Date To'),
    '#type' => 'date',
);
</code>
Which displays on the page as:

And creates the select element values (without zero padding):
Answer: I think you can use #date_format
<code>  $format = 'm d Y';
  $form['date_from'] = array(
     '#type' => 'date',
     '#title' => t('select a date'),
     '#date_format' => $format,
   );
</code>
More values:
http://php.net/manual/en/function.date.php


BigQueryIO.read().fromQuery performance slow

Question: One of the things I've noticed is that the performance of BigQueryIO.read().fromQuery() is quite slower than the performance of BigQueryIO.read().from() in Apache Beam. Why does this happen? And is there any way to improve it?
Answer: As discussed in this other Stack Overflow similar question (where they actually discussed about pricing and not speed), the reason why you perceive notable differences between those two BigQueryIO Reading methods is because, under the hood, they do completely different operations:

BigQueryIO.read.from() directly reads the whole table from BigQuery. This function exports the whole table to temporary files in Google Cloud Storage, where it will later be read from. This requires almost no computation, as it only performs an export job, and later Dataflow reads from GCS (not from BigQuery).
BigQueryIO.read.fromQuery() executes a query and then reads the results received after the query execution. Therefore, this function is more time-consuming, given that it requires that a query is first executed (which will incur in the corresponding economic and computational costs).

So, in short, the main difference between those two Apache Beam functions is that one reads the whole table from a BigQuery ⟷ GCS export, while the other executes a query and later reads its results. Their underlying concepts are completely different, so as an answer to your question about if there is any way to improve the performance, no, there is nothing you can do a part of designing the optimal query for your use case by following the BigQuery best practices.
Comment: Thank You... That makes sense.


Top bar too bright, my eyes are getting tired

Question: First of all, this new design is really awesome. I would like to thank all of the team member who made it possible.
I am creating this post because @Hynes suggested to New site design!
This is feedback/suggestion from user perspective on colour.
I have a little complain, due to too brightness, my eyes are getting tired and cannot stay longer at top of the site.
I wonder if there are people like me.
I am referring to this background image, it is too bright, and cannot stay at top longer. http://cdn.sstatic.net/magento/img/background-header-magento.png?v=159f37a58acc
Comment: +1. Even I feel the same. It's eye stretching.
Comment: a little red/green blind here. It is fine for me, although I'm not much at the site currently :-/
Answer: Thanks for sharing your thoughts here. Orange is one of the those tricky colors, like red, in that when you mute the saturation it can quickly become "muddied" in its appearance. That said, I still think there's something we could do here.
Below is an alternate image. Is this still too bright? I've tried to level out the "hot" red areas with a more even orange tone.

UPDATE: The header image above has replaced the previous "hot red orange" header image.
Comment: Yeah, it's more better for me now. I think it was red causing more stress to my eyes.
Comment: I reckon it also mainly depends on your screen brightness and color display. I've got three different screens, one of them is older than the two others, and the red pops up more on the old screen.


Rails: filling in model attributes during creation

Question: I have a many-to-many relationship between User and Task model.
A task can have many users, but it should keep track of its original creator (if a user creates one by @user.tasks.create). I was wondering how could I do this.
I would have to create a new column field called "creator" in the tasks table. I could then initalise a task in the following way:
@user.tasks.create(:creator=>@user)

Is there a way of not having to add the arguments since the creator will always be the user that instantiated the task.
Thanks!
EDIT
My User model has:
 has_many :taskization
 has_many :tasks, :through => :taskization

My Tasks model has:
  has_many :taskization
  has_many :users, :through => :taskization

Comment: Can you share your models code?
Comment: My models basically only have the many-to-many relation ... I have updated the question to show this. However, this is more of a general question since I have been wondering how to do this for a while - not directly application specific.
Answer: I'm inclined to put the creator attribute in the join model (Taskization). If you do that (say, with this migration),
class AddCreatorIdToTaskizations < ActiveRecord::Migration
  def change
    add_column :taskizations, :creator_id, :integer
  end
end

you can then add a callback to taskization.rb
before_create do |taskization|
  taskization.creator_id  = taskization.user_id
end

that gets you where you want. It's possible that you could do a similar callback in the Task model if you decide that's where the creator attribute belongs, but I haven't thought that through fully.
Comment: Note that Kevin's answer is spot on regarding the need to create a new association as well. Having read that, based on what you're asking I think it makes sense to put that attribute in Task model after all. To do that you'd just add the creator_id column to Tasks and set up the callback in the Task object (and set up the relationships along the lines of Kevin's answer).

The only possible advantage of setting it up in the join model would be if you wanted to get all the users working on tasks created by so-and-so. You'd have to set up yet another association to enable that.
Comment: brilliant - thanks for your answer. I think Kevin's answer is directly related to my question (correct me if I am wrong) - but from your answer I learn that the join table can keep information of each relation between Task and User - which I think I might need in the near future. Thanks a bunch!
Answer: it sounds like you're indicating that the 'original_creator' is an attribute of the Task. For each Task record, you want to keep track of which User originally created it.
So, modeling that seems like you would require both:
# return the User object of the original task creator
@original_creator_user = @task.original_creator

as well as
# get all users of this task
@users = @task.users

to work.
This requires two different relationships between Task objects and User objects.
class User < ActiveRecord::Base
  # tasks for this user
  has_many :taskization
  has_many :tasks, :through => :taskization

  # tasks this user was the original creator of
  has_many :created_tasks, :class_name => "Task"

end

class Task < ActiveRecord::Base
  # users of this task
  has_many :taskization
  has_many :users, :through => :taskization

  # user who originally created this class
  belongs_to :original_creator, :class_name => "User"

end

Notice that the 'creator' relationship is not a :through task, it's a relationship directly between the two objects.
Comment: @Kevin - no question the associations are necessary to access the creator attribute, but absent a callback, how is the value of that attribute set in the first place without calling something like `@user.tasks.create(:creator=>@user)` when the task is created? That part isn't clear to me.
Comment: Kevin, thanks for your answer. Quick question, is Steve's answer also applicable?
Comment: There are different ways to approach this, but I think if you add the associations above that should be enough. Thanks!
Comment: @SteveRowley There are a couple ways to do it. I'd probably do a call back on create that would set the value to the user that created the record.
Comment: Hi @KevinBedell, I just implemented the code above (a bit late really!). I realised that user.tasks.create doesnt set the tasks original_creator.
Comment: I have the same doubt as @SteveRowley. Could you update your answer to show how this call back on create could be implemented?


Given $form, how to programmatically generate its corresponding $form_state?

Question: The media_mover module contains this function, which builds a form and pre-populates default values:
/* step 1 */ $form = media_mover_api_configuration_base_form($configuration);

It also contains this function, which submits the form:
/* step 2 */ function media_mover_api_add_config_form_submit($form, &$form_state)

I would like to programmatically generate the form (step 1), and then programmatically submit the form (step 2).
Step 1 is not a problem; I am able to generate $form.  For step 2, I need $form_state.
Given $form, is there a way to programmatically generate its corresponding $form_state?  I am using Drupal 6.
This is a more detailed explanation of what I am trying to accomplish.
I am able to programmatically fill out and submit a particular form by laboriously setting all the field values in $form_state['values'], and then calling media_mover_api_add_config_form_submit($form, &$form_state);
However, the process would be  much simpler if I could instead use this function, which converts a $configuration object into $form, and does all the heavy lifting:
$form = media_mover_api_configuration_base_form($configuration);

This function enters a default value into each form field.
The only missing step, then, is to copy all these default field values from $form to $form_state['values']. I am wondering whether there is a function to do this.  Or is there a way to loop through all the fields in $form and extract the field-name and default value?
Answer: I've made a slight improvement to this solution. The $values array didnt respect the tree format and moves [$key][$key]['value'] to $value[$key]['value'] causing sub elements with a key of "title" to overwrite the node title:
function mymodule_get_form_state($form){
  $values = array();
  _mymodule_form_get_values($form, &$values);
  $form_state = array();
  $form_state['values'] = $values;
  return ($form_state);
}

function _mymodule_form_get_values($element, &$values) {
  foreach (element_children($element) as $key) {
    if (isset($element[$key]) && $element[$key]) {
      if (isset($element[$key]['#default_value']) && $element[$key]['#default_value']) {
        $values[$key] = $element[$key]['#default_value'];
      } else {
        // Recurse through all children elements.
        _mymodule_form_get_values($element[$key], &$values[$key]);
      }
    }
  }
}

Answer: I would wager that media_mover_api_add_config_form_submit() is a function registered as the submit function for a form. That is, it doesn't 'submit the form,' it is called when the user submits the form in their web browser.
Form API lets you register functions to handle validation and submission of forms. Generally, you don't call these; you let Drupal do it.
I suppose the obvious question is: Why do you need $form_state?
Here's a link to the Form API Quickstart Guide.
UPDATE: OK given your update....
I know very little about Media Mover, though it looks useful. Poking through the code, I suggest that if you need to save a new configuration, then you just do it: $conf = new media_mover_configuration; /* set configuration here */ $conf->save(); Or subclass the configuration if you need an encapsulated approach. You also might try asking the project's lead for guidance, and remind them to add more documentation to the code. :-)
Also, MM might support Features, if that helps you.
Comment: Yes, I believe you are right, that the intended use of media_mover_api_add_config_form_submit() is to submit the form when the user submits the form in their web browser.  I was hoping I to use the same function to programmatically submit the form.  The reason I need $form_state is that this submit function looks at $form_state, rather than $form.
Comment: Please see my clarification above, for what I am trying to accomplish.  Thank you.
Answer: FAPI is your friend and embeds this functionality.
Answer: form_state_defaults() will retrieve the default values. But what are you exactly trying to achieve? I'm guessing that at some point you want to set some values to be submitted? Do you have post data?
As paul-m I'm also guessing that media_mover_api_add_config_form_submit() is a function registered as the submit function for the form you mention. Calling this function as above won't work.
You can call the process form function though, which will build it validate and submit it if applicable.
ie: drupal_process_form($form['#form_id'], $form, $form_state);
Comment: Please see my clarification above, for what I am trying to accomplish.  Basically:  Is there a way to copy the default values from $form to $form_state['values']


Good way to use word similarity as a feature in supervised ML on text

Question: I have a pretty low N data set of small sentences tagged with a label. I would like to create a classifier on this dataset. The word choice is not very variable since the domain is pretty specific. Nonetheless, it can happen that words are misspelled or (rarer) variants that are not caught by lemmatization and stemming engines.
What are the best practices in the field to take these factors in account?
Answer: Learning your model should rely on generalization, to generalize data you need unification. If you can't stem or lemm maybe you can use Levenshtein to find nearest stem or w2v term? If you don't provide information about semantic similarity between words but noised by misspelling you could expect that your model would be much worst than model with that information. Without seeing data is hard to say what would be the best option for denoise data. However it's very important to not skip a denoising step.


How to prove these using natural deduction

Question: I'd like to prove the following logical equivalence by using natural deduction:
$$(\exists x)(p(x) \implies q) \dashv\vdash (\forall x)(p(x)) \implies q.$$
As far as I'm concerned to show that two statements are equivalent we should show proofs for both of the sides. Can anyone help. Thanks in advance
Comment: You need to adapt this to whatever inference rules you have: suppose $(\exists x)(p(x) \implies q) $ is true. Then there exists a constant $a$ such that $p(a)\to q$. (You want to prove that  $(\forall x)(p(x)) \to q$, so) assume $(\forall x)(p(x))$ is true. Then it is true in particular for $a$, that is, $p(a)$ is true. From here it's easy to conclude.
Comment: @BlediBoss Can you do either direction?
Comment: Yeah I guess so. I did the opposite direction of the given logical equivalence.
Answer: For the first direction, $(\exists x)(p(x)\implies q) \vdash (\forall x)(p(x))\implies q$, we could reason as follows (in a natural deduction system):

$(\exists x)(p(x)\implies q)\quad {\sf P}\quad$ 1
$p(a)\implies q \quad {\sf EI(1)}\quad$ 1
$(\forall x)(p(x))\quad{\sf P}\quad$ 3
$p(a)\quad {\sf UI(3)}\quad$ 3
$q \quad {\sf MP(2,4)}\quad$ 1,3
$(\forall x)(p(x))\implies q\quad {\sf CondI(3,5)}\quad$ 1

For the converse (which you state to have proven already -- this is just for completeness' sake), we could do:

$(\forall x)(p(x)) \lor \neg (\forall x)(p(x))\quad {\sf LEM}\quad$
$(\forall x)(p(x)) \implies q\quad{\sf P}\quad$ 2
$(\forall x)(p(x))\quad{\sf P}\quad$ 3
$q\quad\sf MP(2,3)\quad$ 2,3
$p(a)\quad\sf P\quad$ 5
$p(a)\implies q\quad\sf CondI(5,4)\quad$ 2,3
$(\exists x)(p(x)\implies q)\quad\sf EG(6)\quad$ 2,3
$(\forall x)(p(x)) \implies (\exists x)(p(x)\implies q)\quad \sf CondI(3,7)\quad$ 2
$\neg(\forall x)(p(x))\quad\sf P\quad$ 9
$(\exists x)(\neg p(x))\quad\sf TI(9)\quad$ 9
$\neg p(a)\quad\sf EI(10)\quad$ 9
$p(a)\quad\sf P\quad$ 12
$\bot\quad\sf BI(11,12)\quad$ 9,12
$q\quad\sf BE(13)\quad$ 9,12
$p(a)\implies q\quad\sf CondI(12,14)\quad$ 9
$(\exists x)(p(x)\implies q)\quad\sf EG(15)\quad$ 9
$\neg(\forall x)(p(x))\implies (\exists x)(p(x)\implies q)\quad\sf CondI(9,16)$
$(\exists x)(p(x)\implies q)\quad \sf DisjE(1,8,17)\quad$ 2

Where at line 10, the (somewhat tedious to prove) equivalence between $\neg(\forall x)(p(x))$ and $(\exists x)(\neg p(x))$ was used to keep the proof reasonably short.
The abbreviations used for the respective rules should be easily readable to anyone familiar with natural deduction. If someone would appreciate them being explained explicitly, please let me know in the comments.
The lines in above derivations follow the scheme:

Line number. Formula$\quad$Rule$\quad$Undischarged premises

Added: It is worthwhile to note that the invocation of the Law of Excluded Middle is used essentially in the second proof. The two propositions are inequivalent in intuitionistic logic (the logic which accepts everything from classical logic except LEM (and its sister DNE, $\neg\neg p \vdash p$)).
We also have proven the drinker paradox as a special case (with $q = (\forall x)(p(x))$). It's fun to read about.
Answer: Just for practice, here's another Fitch style proof, using proof by contradiction (DNE rather than LEM) and only fundamental rules of induction.$$\def\fitch#1#2{\quad\begin{array}{|l}#1\\\hline#2\end{array}}{\fitch{~~1.~\exists x~(p(x)\to q)\hspace{16ex}\textsf{Premise}}{\fitch{~~2.~\forall x~p(x)\hspace{19ex}\textsf{Assumption}}{\fitch{~~3.~[c]~p(c)\to q\hspace{11.5ex}\textsf{Assumption}}{~~4.~p(c)\hspace{19ex}{\forall}\textsf{Elimination 2}\\~~5.~ q\hspace{22ex}{\to}\textsf{Elimination 4,3}}\\~~6.~q\hspace{25ex}\exists\textsf{Elimination 1,3-5}}\\~~7.~(\forall x~p(x))\to q\hspace{16ex}{\to}\textsf{Introduction 2-6}}\\\exists x~(p(x)\to q)~\vdash~(\forall x~p(x))\to q\\~\\\fitch{~~8.~(\forall x~p(x))\to q\hspace{17ex}\textsf{Premise}}{\fitch{~~9.~\neg\exists x~(p(x)\to q)\hspace{12ex}\textsf{Assumption}}{\fitch{10.~\exists x~\neg p(x)\hspace{15ex}\textsf{Assumption}}{\fitch{11.~[c]~\neg p(c)\hspace{12.5ex}\textsf{Assumption}}{\fitch{12.~p(c)\hspace{13.5ex}\textsf{Assumption}}{13.~\bot\hspace{16ex}\neg\textsf{Elimination}\\14.~q\hspace{16.5ex}\textsf{Ex Falso Quodlibet}}\\15.~p(c)\to q\hspace{12ex}{\to}\textsf{Introduction}\\16.~\exists x~(p(x)\to q)\hspace{7ex}\exists\textsf{Introduction 15}\\17.~\bot\hspace{19ex}\neg\textsf{Elimination 16,9}}\\18.~\bot\hspace{22ex}\exists\textsf{Elimination 10,11-17}}\\19.~\neg\exists x~\neg p(x)\hspace{17ex}\neg\textsf{Introduction 10-18}\\\fitch{20.~[a]\hspace{22ex}\textsf{Assumption}}{\fitch{21.~\neg p(a)\hspace{15.5ex}\textsf{Assumption}}{22.~\exists x~\neg p(x)\hspace{12ex}\exists\textsf{Introduction 21}\\23.~\bot\hspace{19ex}\neg\textsf{Elimination 22,19}}\\22.~\neg\neg p(a)\hspace{17ex}\neg\textsf{Introduction 21-23}\\23.~p(a)\hspace{20ex}\neg\neg\textsf{Elimination 22}}\\24.~\forall x~p(x)\hspace{20ex}\forall\textsf{Introduction 20-23}\\25.~q\hspace{26.5ex}{\to}\textsf{Elimination 24,8}\\26.~\exists x~q\hspace{23ex}\exists\textsf{Introduction 25}\\\fitch{27.~[c]~q\hspace{20ex}\textsf{Assumption}}{\fitch{28.~p(c)\hspace{17ex}\textsf{Assumption}}{29.~q\hspace{20ex}\textsf{Reiteration 27}}\\30.~p(c)\to q\hspace{16ex}{\to}\textsf{Introduction 28-29}\\31.~\exists x~(p(c)\to q)\hspace{11ex}\exists\textsf{Introduction 30}\\32.~\bot\hspace{23ex}\neg\textsf{Elimination 31,9}}\\33.~\bot\hspace{26ex}\exists\textsf{Elimination 26,27-32}}\\34.~\neg\neg\exists x~(p(x)\to q)\hspace{14ex}{\neg}\textsf{Introduction 9-33}\\35.~\exists x~(p(x)\to q)\hspace{17ex}{\neg\neg}\textsf{Elimination 34}}\\(\forall x~p(x))\to q~\vdash~ \exists x~(p(x)\to q)}$$
Answer: Here is a similar proof using a Fitch-style proof checker:

On line 9, rather than taking the negation of the goal, I tool the negation of the material implication equivalence of the goal.  It was (perhaps) easier to reach a contradiction using that technique. Regardless, from that result on line 14, I still had to derive the desired goal.

Kevin Klement's JavaScript/PHP Fitch-style natural deduction proof editor and checker http://proofs.openlogicproject.org/


HTML form using iUI on iPad has erratic scrolling when typing in text input files

Question: I'm using the iUI framework for a site optimized for display on iPhone/iPad mobile Safari browsers. I'm having a problem when using iUI and the default theme with forms that extend beyond the height of an iPad browser window (screen height). The behavior doesn't show up on the desktop version of Safari when using an iPad user-agent and is a little hard to describe, but here goes...
I have a long form that contains lots of text fields like this:
  <form accept-charset="UTF-8" action="/customers" class="panel" enctype="multipart/form-data" id="new_customer" method="post" target="_self">
<input name="utf8" type="hidden" value="✓" /><input name="authenticity_token" type="hidden" value="rp8JH0JucnB9dfQ9TaRr2sTp2rt3Q/fKkzWm5VBB70g=" />

    <input id="fromScheduler" name="fromScheduler" type="hidden" value="0" />

Client Information

<fieldset>

    First Name
    <input autocomplete="off" id="customer_firstName" name="customer[firstName]" onBlur="checkDuplicateClient();" size="30" type="text" />

    Last Name
    <input autocomplete="off" id="customer_lastName" name="customer[lastName]" onBlur="checkDuplicateClient();" size="30" type="text" />

    Email #1
        <input autocomplete="off" id="customer_email" name="customer[email]" size="30" type="email" />

    Email #2
        <input autocomplete="off" id="customer_secondary_email" name="customer[secondary_email]" size="30" type="email" />

(All told I have 22 rows of name/value text fields in the first fieldset collection, then two more fieldsset collections of varying length, 0..n rows.)
What happens is that when clicking into the field that appears almost at the very bottom of the iPad screen without the keyboard being displayed, the keyboard pops up as expected, but the browser window scrolls to the very top of the page. Because the keyboard is displayed, this obstructs the form field being filled in. This happens for every field that appears near or past the bottom of the iPad window.
I'm using the standard iUI CSS (iui.css, iui-panel-list.css, and the default theme css) and am including the iUI javascript along with jQuery 1.5.2, jQuery UI and jQuery livequery.
Any ideas on what could be causing this? It's not just annoying, it makes the mobile version of these forms practically unusable. Any help is appreciated.
Answer: It appears that iUI calls the setOrientation() function which both adjusts the page styling based on the device orientation as well as (always) scrolling to the top. I don't believe that scrolling to the top of the page is always appropriate, so I've submitted the following patch on the iUI Google group as a suggestion. I'm posting it here too in the event it's helpful to anyone else and for general feedback as to the soundness of my approach.
--- iui.js
+++ iui.js
@@ -677,12 +677,12 @@ function orientChangeHandler()
    switch(orientation)
    {
    case 0:
-       setOrientation(portraitVal);
+       setOrientationAndScroll(portraitVal);
        break;

    case 90:
    case -90:
-       setOrientation(landscapeVal);
+       setOrientationAndScroll(landscapeVal);
        break;
    }
 }
@@ -727,6 +727,11 @@ function setOrientation(orient)
        iui.removeClass(document.body, portraitVal);
        iui.removeClass(document.body, landscapeVal);
    }
+}
+
+function setOrientationAndScroll(orient)
+{
+   setOrientation(orient);
    setTimeout(scrollTo, 100, 0, 1);
 }

@@ -844,7 +849,6 @@ function slide1(fromPage, toPage, backwards, axis, cb)
        (backwards ? fromPage : toPage).style.top = "100%";
    else
        toPage.style.left = "100%";
-
    scrollTo(0, 1);
    toPage.setAttribute("selected", "true");
    var percent = 100;


I think the problem this site has is a lack of experts

Question: There are some really good questions being asked here.
However I'm noticing that a lot of questions don't have answers, or only have one answer.
How can we attract more expert users here?
Answer: I tend not to agree with the OP.  Experts would be lured in by other experts (but here we have the "chicken-or-egg" issue), or by graduate-students-level questions -questions that are advanced and sophisticated enough, and closer to scientific papers and issues and actual research, so that they would trigger the interest of experts, mixed with a teaching desire (if any).
And this subset of questions, graduate-level ones, is the one we lack the most: We mostly have people trying to understand the economics of their life or society (essentially asking questions that are inherently very complex, and so very difficult to answer scientifically in a Q&A site), or undergraduate students asking mostly trivial theoretical questions.
What needs to be "targeted" is Master students and comparable.


Sample Size needed for study?

Question: I am coming up with a sample size for one of my studies that will use four groups. I had two questions:
1) What is the difference between using a two-group t-test of equal means with alpha = 0.01 (With four groups, six comparisions will be done) vs ANOVA with alpha = 0.05?
2) For two of my intervention groups of my study, I don't have prior information that can give me an idea of the mean nor standard deviation. What should I do considering if I choose to do an ANOVA, I need these values to estimate the sample size needed.
Thanks!
Comment: If 6 comparisons will be done, why not 0.05/6, but 0.05/5?
Comment: I don't quite understand what you are saying
Comment: There are many ways to adjust for multiple testing. One is to divide the 5% type one error rate by the number of tests. It is called Bonferronni's adjustment. In your example you only divided by 5, and that prompted me to ask that question. Perhaps I am not sure about the motivation/rationale of using 0.01.
Comment: Oh, good call! The reason for using 0.01 was to minimize error rate. Ideally I would like to use 0.05 but with 6 comparisons that's too much error. Thanks for sharing Bonferronni's adjustment. In essence, would ANOVA be better than using Bonferroni's adjustment in this case?
Answer: Question 1: It depends on what your questions are. If it's just "does any one of the means differ from the rest?" then F-test is enough. If you'd like to further examine the mean differences by pairs, then some inflation to account for multiple testing is needed. However, the p-value threshold does not need to be as low as 0.05/6.
Just an example, let's say there are four groups with means = 1, 1, 2, 2. Standard deviation (SD) are all assumed to be 1. Power 80% and alpha 5%. We would only need 48 cases to test it.
Now, change that to t-test with alpha now = 5%/6 = 0.0083, to detect a mean difference as small as 1 unit, we'll need 27 per group, 54 total. Since there should be 4 groups, sample size will become 108.
Yet, as stated above dividing the 5% by number of tests can be too harsh. A better alternative is to seek after sample size calculator that can incorporate other less extreme form of multiple test adjustment (one of such is called PASS). For instance, Tukey's adjustment would be a possible candidate. With this adjustment the total sample size for the above scenario can be reduced to 80. For more background information about sample size in multiple comparison, I'd suggest this article.
Question 2: The best approach is to carry out a very small pilot study to get that information. The second best would be to consult the investigators of this study. While we don't have the mean estimate, the actual difference still needs to be big enough for the invention to be clinically or practically meaningful. And the researchers themselves (or the end-user of this invention) are good sources of the magnitude of that meaningful difference. Standard deviation, with some assumptions, may be based on the SDs of the other groups that you do have. Depending on the nature of the intervention, you may wish to inflate the SD somewhat to include more samples, just to be on the safer side.
Comment: Thank you so much! Clear as mud! Regarding the pilot study, what would you recommend as a sample size per group? I know Julious recommends sample size of 12 per group for a pilot study - do you think that is fine?

Sample Size needed for study?


Confused about localization for affine varieties and affine schemes?

Question: I have some basic familiarity and intuition with algebraic varieties and am wanting to start on schemes. I am looking at section 2.2 of Hartshorne, in particular the definition of the sheaf of rings of $\text{spec}A$. For a ring $A$, this consists of functions,
\begin{equation}
\label{regulartionfunction}
s: U \longrightarrow \bigsqcup_{\mathfrak{p} \in U}A_{p}
\end{equation}
satisfying certain local conditions. I am trying to connect this back to the case of affine varieties where the maps are from an open set $U$ into the ground field, say $k$. Suppose $A$ is some polynomial ring, for simplicity start with $\mathbb{C}[x]$. The points of an affine variety correspond to the maximal ideals $\mathfrak{m}$, which look like $\left< x-a \right>$. My reasoning (which I think has some kind of problem), was as follows:
The prime ideals of $\mathbb{C}[x]_{\mathfrak{m}}$ are in one-one correspondence with the prime ideals of $\mathbb{C}[x]$ contained in $\mathfrak{m}$. Since the prime ideals of $\mathbb{C}[x]$ are precisely the maximal ones, then $\mathbb{C}[x]_{\mathfrak{m}}$ has precisely one maximal ideal, which is $(0)$, hence making it a field. If that is correct, then bringing this in line with above definition for schemes in the case that $A = \mathbb{C}[x]$ would result in $s$ being a map from $U$ onto a field.
However, this reasoning seems to break down for polynomial rings in more than one variable. Also, I am not even convinced my above reasoning is correct, since I only seemed to use the fact that $\mathbb{C}[x]$ was a PID, and am not sure all localizations of PIDs are fields.
So if anyone could shed some light on how the map $s$ reduces to the regular functions we are familiar with for affine varieties, and also sure up my reasoning about localizations of polynomial rings by maximal ideals I would be very appreciative.
Thanks
Luke
Comment: Apologies, the undefined references are supposed to refer to the only equation environment I've defined.
Comment: maybe these can help--https://en.wikipedia.org/wiki/Morphism_of_algebraic_varieties and https://en.wikipedia.org/wiki/Affine_variety#Structure_sheaf
Answer: Your description of the ideal structure of $\mathbb{C}[x]_\mathfrak{m}$ is incorrect. $\mathbb{C}[x]_\mathfrak{m}$ is a local ring which is not a field- it contains the ideal $\mathfrak{m}$ which is different from the zero ideal and not the whole ring. (Exercise: verify this with $\mathbb{C}[x]_{(x)}$, where every element not divisible by $x$ is a unit- show that $1\notin(x)$ and $x\notin(0)$.)
The way to recover the notion of functions with which you're familiar (target being a field) is to evaluate our function in to local rings by tensoring with the residue field at the point you're interested in. This gives more information because the local rings are larger and you can track more stuff using them.


Does the integral $\int_0^{\pi} \frac{dx}{\sin(2x)+\cos(3x)}$ exist?

Question: This link
http://www.wolframalpha.com/input/?i=integral%28x%3D0%2Cpi%2C1%2F%28sin2x%2Bcos3x%29%29
shows the visual representation of the integral
$$\int_0^{\pi} \frac{dx}{\sin(2x)+\cos(3x)}$$
Looking at the picture, I tend to believe that the integral does not exist.
The integrand contains three poles, but from this I cannot conclude that the
integral does not exist, or can I ?

How can I check, if the integral exists or not ?
Comment: Next time try to add a slash before trig functions.
Answer: Note at $x = {\pi \over 2}$, the denominator $\sin 2x + \cos 3x$ is zero. If $f(x) = \sin 2x + \cos 3x$, then $f'(x) = 2 \cos 2x - 3 \sin 3x$, so that $f'({\pi \over 2}) = 1$. Hence the function $f(x)$ behaves as ${1 \over x - {\pi \over 2}}$ near $x = {\pi \over 2}$; in particular, since $|{1 \over x - {\pi \over 2}}|$ has infinite integral in any interval centered at ${\pi \over 2}$, so does $|{1 \over \sin 2x + \cos 3x}|$. Thus in the usual sense your integral does not exist.
There are more general notions of integral where you remove intervals of the form $({\pi \over 2} - \epsilon, {\pi \over 2} + \epsilon)$ (and analogous things for the other poles), and then take limits as $\epsilon \rightarrow 0$. If you do this, sometimes you can get a finite number as your limit for integrals such as yours.
Comment: In this case that finite number would be zero, since the integrand is odd around $\pi/2$.


Thumb position when soloing (or performing scales) on electric guitar

Question: Why do so many top top electric guitar players (like Mark Knopfler and Eric Clapton) consistently hook their thumb above the neck when it goes against the conventional wisdom?
This seems especially prevalent when they are soloing (like playing scales) vs. playing rhythm (chords). Is the thumb-behind rule less clear-cut for lead/solo/scales than rhythm/chords?
I am looking for a detailed explanation of the pro's/con's of thumb hooking over, not an explanation of why thumb-behind is a "good technique". There are a million explanations of why thumb-behind is considered good. There must be at least some (perhaps minority view) school of thought behind the alternate technique as its use is pervasive.

What are the pros and cons in the context of the style of music and soloing shown in the two videos below?
How big of an impact will each way make on the ability to play solos like those?
Is this a case of where the top guys are advanced enough to be allowed to break the rules?

Here are two examples:

Mark Knopfler:

Eric Clapton:
Comment: These are all great answers. I would add that I was taught to use the thumb over to mute the fifth and sixth strings when strumming chords aren't open on those strings. It works well for me @modal.
Answer: See also: How to play power chords without fretting hand getting tired?
To me it completely goes along with conventional wisdom. The difference is what you are looking at the wisdom for.
Hooking your thumb over the top is a lot more comfortable and keeps you from dropping your wrist.
More importantly, it lets you use your thumb as leverage for bending notes. When you get to higher string gauges and/or compound bends, it helps to have the extra force from the thumb squeezing down while your fingers squeeze up.
I think you'll find most of the great rock guitarist use thumb-over when soloing, and actually are more likely to put their thumbs behind when playing chords that require the index finger on the low E string.

What are the pros and cons in the context of the style of music and soloing shown in the two videos below?

I think the pros I've outlined. One con is most guitarists can't reach the low E string with their third or fourth fingers when their thumb is on top, and the first finger may have very little power when stretched out to reach it. Another con is when you have your wrist up, you can't stretch your fingers as wide to cover as many frets, and there are some "stretch" fingerings that are difficult to impossible with a high wrist and thumb over the top.

How big of an impact will each way make on the ability to play solos like those?

Personally I can't imagine trying to play rock solos with bending without my thumb over - especially with thicker strings and/or larger bends. I pretty much have to play thumb over as much as possible or I have problems with my wrist. Beyond all of that, thumb over is much more comfortable when you wear the guitar as low as they do. If you want to play like the great rock guitarists, it does help to adopt their techniques. This is one I have recommended to every single guitar student I've ever taught.

Is this a case of where the top guys are advanced enough to be allowed to break the rules?

No, it's the opposite. These guys are experienced enough to know that the rule is actually you should have your thumb over the top as much as possible. Jimmy Page, Jimi Hendrix, and Stevie Ray Vaughan all fretted the low E string with their thumbs over the top while playing certain chords (I also do that routinely but I doubt many will be impressed by that fact).
Comment: As much a classical-technique snob I am, I still have to agree with this: bends on the upper two strings work much better if the thumb hooks around the neck.
Comment: @modal_dialog I only drop my wrist when I have to for very certain specific chords, but the way I wear my guitar and my personal body geometry mean I have to keep a high wrist or risk repetitive motion injury. If you wear your guitar higher or play seated with a highly angled neck you could even play classical style with your thumb primarily on the back of the neck. It's mostly preference, you just want your guitar position preference and your wrist geometry preference to be compatible.
Comment: Exactly what I was looking for, especially about SRV and fretting with the thumb. And the linked-to article with detailed technique is very helpful. Thanks!
Comment: Todd, saw you wrote that other answer, so can I ask a quick follow up? Is there any caveat to above if I play primarily fingerstyle (like Knopfler)? Should I still be dropping wrist when I'm focusing on chords (or Travis picking), or just flat out always keep the thumb over? Note: I won't be shredding much. My music style would range from the videos above down to more blues/folksy stuff.
Answer: The recommendation to keep your thumb behind the neck is a holdover from acoustic guitars.  Ideas from acoustic playing are still used to inform electric guitar technique despite the change in technology.
At a mechanical level, the smaller size (nut width in particular) makes it feasible to wrap your thumb around the neck (try that on a classical guitar!).  The thinner strings (compared to steel string acoustic) makes bending strings feasible.  As mentioned in other answers, wrapping the thumb provides additional leverage for doing this.  Electric guitar technique and style has evolved in relation to these technical changes of the instrument.
However, culture (and pedagogy) are slow to change, hence there are still recommendations to keep you thumb behind the neck.
Answer: As I wasn't getting any bites on this question, I kept digging and I think I've come across an explanation myself. Anyone else, please chime-in and correct me, if this isn't right (or there's more to it)...
It seems there is an explicit technique known as "The Stevie Ray Vaughan Grip". There are videos about it on YouTube (such as below), including folks selling lessons on the technique. It seems especially appropriate as they talk about fretting the 6th (and possibly 5th) string with the thumb, which looks like Mark and Eric may be doing in the videos above.
Comment: Many guitarist use the thumb to fret the two fattest strings.  Often that can be done instead of barring with your index finger.  I personally can't do this because my hands are too small and fingers too short.
Answer: Whether or not there are technical advantages to thumb-over, I think most players gravitate to it because it's the most natural and comfortable way to hold the neck, like holding a baseball bat or tennis racket. Moreover, if you like to fret chords with your thumb on the E string, your hand tends to linger in that position afterward because it's so comfortable. Even on classical, new players find themselves bringing the thumb over, even though everyone agrees it's a bad idea on classical because the neck is so wide.
You see it even more on electric because the neck is so thin, and fretting is so easy from any direction, that it doesn't really matter where you put your thumb. You could practically fret with the back of your hand and do fine. So you find a position that's comfortable and don't bother to refine it.


What kind of statistical test is appropriate for the following example?

Question: I ran an experiment varying the temperature and the quantity of chemicals added for the removal of 4 contaminants (A, B, C and D). The data obtained from this experiment was (in R):
temperature <- c(rep(20,4),rep(30,4),rep(40,4))
chemicals <- rep(seq(0,30,10),3)
contaminant_A <- c(9,22,25,28,35,50,57,78,86,94,97,100)
contaminant_B <- c(0, 0, 3, 3, 4, 5,11,11,13,18,18,21)
contaminant_C <- c(68,57,53,50,44,38,36,34,31,29,25,24)
contaminant_D <- c(39,68,35,32,53,69,69,60,35,30,43,36)

DF <- data.frame(temperature,
                 chemicals,
                 contaminant_A,
                 contaminant_B,
                 contaminant_C,
                 contaminant_D)

I'd like to use a statistical test (or some statistical tests) to answer questions like:

Is the temperature or the quantity of chemicals the most relevant variable of the process?
Are the removals of the contaminants related?

I was thinking of using a PCA, however I'm not sure.
If there is a graphical way to explain it, that would be great.
EDIT 1:
As Izy suggested, I used the GLM for all contaminats:
> glm(contaminant_A~temperature+chemicals)

Call:  glm(formula = contaminant_A ~ temperature + chemicals)

Coefficients:
(Intercept)  temperature    chemicals
   -65.1750       3.6625       0.8033

Degrees of Freedom: 11 Total (i.e. Null);  9 Residual
Null Deviance:      12030
Residual Deviance: 327.1    AIC: 81.72
> glm(contaminant_B~temperature+chemicals)

Call:  glm(formula = contaminant_B ~ temperature + chemicals)

Coefficients:
(Intercept)  temperature    chemicals
     -18.23         0.80         0.21

Degrees of Freedom: 11 Total (i.e. Null);  9 Residual
Null Deviance:      604.9
Residual Deviance: 26.77    AIC: 51.68
> glm(contaminant_C~temperature+chemicals)

Call:  glm(formula = contaminant_C ~ temperature + chemicals)

Coefficients:
(Intercept)  temperature    chemicals
    91.1250      -1.4875      -0.3833

Degrees of Freedom: 11 Total (i.e. Null);  9 Residual
Null Deviance:      2090
Residual Deviance: 99.71    AIC: 67.46
> glm(contaminant_D~temperature+chemicals)

Call:  glm(formula = contaminant_D ~ temperature + chemicals)

Coefficients:
(Intercept)  temperature    chemicals
   59.51667     -0.37500     -0.05667

Degrees of Freedom: 11 Total (i.e. Null);  9 Residual
Null Deviance:      2615
Residual Deviance: 2498     AIC: 106.1

Using contaminant A as example, can I say that the temperature is more relevant than the quantity of chemicals because the coefficient is bigger?
What value of these results informs about the relation like a R^2?
Comment: PCA $\ne$ statistical test. (And much of what I have seen published which attempts to incorporate tests into PCA is rife with violations of the i.i.d. assumption.)
Comment: I will only suggest that you use some numerical and graphical methods to get an idea which variables are associated with which others. I tried a couple of things with functions in the base of R. At first glance, it seems that regression methods may be useful. I will use Answer format to show a couple of things I tried. I don't understand the intended role of Chem, so I didn't use it.
Comment: Have you considered a general linear model (glm family in R)? e.g. glm(contaminant~temperature+chemicals). Do you expect any interaction between the effect of temperature and the effect of quantity of chemicals? Your knowledge here may be able to inform what subset of possible tests/models you might want to use. For your second question, have you checked if the removal of various contaminants is correlated, as a starting point?
Comment: @Izy first, the expected was that the increase of the temperature improve the removal, but the data didn't show this. I uses de GLM for the contaminants (look at the original question EDIT)
Comment: There is no conclusion that numbers/methods give you that you can't put meaningfully into a plot.  This is because there are 5 ways to rephrase a mathematical idea: formulae, tables, figure/graph, words, and flowchart/graph.  It is immensely valuable to double-check every conclusion you get to by plotting it.  My old retired core-comp mgr used to call them "GRC's" or Gross Reality Checks.
Answer: Not an Answer. Continuation of comment:
MAT = cbind(Temp, a, b, c, d)
cor(MAT)
           Temp          a          b           c           d
Temp  1.0000000  0.9446208  0.9199987 -0.92024370 -0.20741843
a     0.9446208  1.0000000  0.9705441 -0.95711450 -0.20573950
b     0.9199987  0.9705441  1.0000000 -0.92154735 -0.29544696
c    -0.9202437 -0.9571145 -0.9215473  1.00000000  0.07196594
d    -0.2074184 -0.2057395 -0.2954470  0.07196594  1.00000000

It seems that d isn't significantly correlated with anything else.
I think you are right that a graphical approach may be helpful, at least at the start. As suggested by correlations near $\pm 1$ above, it seems there are some nearly
linear relationships.
pairs(MAT)

Maybe based on some of this, you will want to clarify your
experiment and its goals by editing your Question (not with Comments which may be missed). Meanwhile, I will let someone who isn't booked up for the evening make
suggestions for interpreting your regression and perhaps trying
additional regression models.


Build Error "Type …. is part of the declarations of 2 modules: AppModule - Ionic

Question: I'm trying to create my APK for android and I run the following command:
<code>ionic cordova run android --prod --release
</code>
Ionic version: Ionic V3
My application is not doing lazy loading (I don't even know how to do that hahah) and I have used the ionic cli to create every page.
When I run the command above I following error:

Error: Type AutomaticPage in
  C:/Users/tamo/Desktop/dpHydmech/Hydmech/src/pages/automatic/automatic.ts
  is part of the declarations of 2 modules: AppModule in
  C:/Users/tamo/Desktop/dpHydmech/Hydmech/src/app/app.module.ts and
  AutomaticPageModule in
  C:/Users/tamo/Desktop/dpHydmech/Hydmech/src/pages/automatic/automatic.module.ts!
Please consider moving AutomaticPage in
  C:/Users/tamo/Desktop/dpHydmech/Hydmech/src/pages/automatic/automatic.ts
  to a higher module that imports AppModule in
  C:/Users/tamo/Desktop/dpHydmech/Hydmech/src/app/app.module.ts
and AutomaticPageModule in
  C:/Users/tamo/Desktop/dpHydmech/Hydmech/src/pages/automatic/automatic.module.ts.
  You can also create a new NgModule that exports and includes
  AutomaticPage in
  C:/Users/tamo/Desktop/dpHydmech/Hydmech/src/pages/automatic/automatic.ts
then import that NgModule in AppModule in
  C:/Users/tamo/Desktop/dpHydmech/Hydmech/src/app/app.module.ts and
  AutomaticPageModule in
  C:/Users/tamo/Desktop/dpHydmech/Hydmech/src/pages/automatic/automatic.module.ts.

Things I have tried:

Removing file automatic.module.ts
Remove imports everything from automatic.module.ts
Remove the AutomaticPageModule import from app.module.ts

I took a look to this:
https://forum.ionicframework.com/t/build-error-type-is-part-of-the-declarations-of-2-modules-appmodule/129767
https://forum.ionicframework.com/t/uncaught-error-type-homemainpage-is-part-of-the-declarations-of-2-modules-please-consider-moving-homemainpage-to-a-higher-module-that-imports-ionicmodule-and-appmodule/105456/10
Any suggestions on how to fix this problem?
Answer: In my opinion this is because you added automatic.ts in two modules.
So if you remove one it will be working well
Comment: Yeah I tried removing it from the automatic.module.ts and app.module.ts and no luck :(
Comment: Can you share your automatic and app module?
Comment: That worked. The reason why it didn't work for me before was because I was editing the wrong ionic project lololol

Build Error “Type …. is part of the declarations of 2 modules: AppModule - Ionic