infinity sum Bessel-Type about $\sum _{k=-\infty }^{\infty } (-i)^k \cos (k) J_k(x)$

Question: I would like to propose a problem following the sequence of series could find the general term when Co(x)^n it seem easy but i can not find
$$\sum _{k=-\infty }^{\infty } (-i)^k \cos (k) J_k(x)=\cos (x \cos (1))-i \sin (x \cos (1))$$
$$\sum _{k=-\infty }^{\infty } (-i)^k \cos ^2(k) J_k(x)=\frac{1}{2} (\cos (x)+\cos (x \cos (2))-i (\sin (x)+\sin (x \cos (2))))$$
$$\sum _{k=-\infty }^{\infty } (-i)^k \cos ^8(k) J_k(x)=\frac{1}{128} (35 \cos (x)+56 \cos (x \cos (2))+28 \cos (x \cos (4))+8 \cos (x \cos (6))+\cos (x \cos (8))-i (35 \sin (x)+56 \sin (x \cos (2))+28 \sin (x \cos (4))+8 \sin (x \cos (6))+\sin (x \cos (8))))$$
Answer: We wish to find an expression for
$$\sum^\infty_{k = -\infty} (-i)^k \cos^n (k) J_k (x),$$
where $J_k (x)$ is the Bessel function of the first kind of order $k$, $i$ is the imaginary unit, and $n \in \mathbb{N}$.
To do this we will make use of the Jacobi-Anger expansion
$$e^{i x \cos \phi} = \sum^\infty_{k = -\infty} i^k J_k (x) e^{ik \phi}.$$
As
$$\cos (k) = \frac{1}{2} (e^{ik} + e^{-ik}),$$
on applying the binomial theorem we have
$$\cos^n (k) = \left (\frac{e^{ik} + e^{-ik}}{2} \right )^n = \frac{1}{2^n} \sum^n_{m = 0} \binom{n}{m} e^{i(2m - n)k}.$$
Thus
\begin{align*}
\sum^\infty_{k = -\infty} (-i)^k \cos^n (k) J_k (x) &= \frac{1}{2^n} \sum^n_{m = 0} \binom{n}{m} \sum^\infty_{k = -\infty} (-i)^k J_k (x) e^{i(2m - n)k}\\
&= \frac{1}{2^n} \sum^n_{m = 0} \binom{n}{m} \sum^\infty_{k = -\infty} i^k (-1)^k J_k (x) e^{i(2m - n)k}\\
&= \frac{1}{2^n} \sum^n_{m = 0} \binom{n}{m} \sum^\infty_{k = -\infty} i^k J_k (-x) e^{i(2m - n)k}, \quad {\rm since} \,\, J_k(-x) = (-1)^k J_k (x)\\
&= \frac{1}{2^n} \sum^n_{m = 0} \binom{n}{m} e^{-ix \cos (2m - n)},
\end{align*}
where in the last line we have made use of the Jacobi-Anger expansion. Hence
$$\sum^\infty_{k = -\infty} (-i)^k \cos^n (k) J_k (x) = \frac{1}{2^n} \sum^n_{m = 0} \binom{n}{m} \Big{(}\cos [x \cos (2m - n)] - i \sin [x\cos (2m - n)] \Big{)},$$
and is the desired expression you seek.
Comment: very nice...omegadot


Look up formula works well except for one column

Question: I have an excel file that has been uploaded here
http://www58.zippyshare.com/v/99974349/file.html

The formula works great except for a column with descending values.
=INDEX(
  INDIRECT("'"&LOOKUP(B5,TblA)&"'!A6:A36"),
   LOOKUP(9.99999999999999E+307,
    SEARCH("-"&C8&"-","-"&INDIRECT("'"&LOOKUP(B5,TblA)&"'!C6:C36")&"-"),
    ROW(C6:C36)-ROW(C6)+1))

Let me explain the excel file.
I have one main sheet 'Report' and 4 other sheets correspond to 4 age groups. - 4.2.0 to 4.7.30, 4.8.0 to 5.1.30, 5.2.0 to 5.7.30 and 5.8.0 to 6.1.30. Depending on the Age (B5) in the sheet 'Report', I select one of the 4 sheet to pick values from. I pick the correct sheet using a Table Name TblA which contains all sheet names and is defined from A24 to B27 in the sheet 'Report'.
In the sample sheet that is uploaded, B5 contains the value 5.7 which means we have to select the sheet 5.2.0 to 5.7.30.
Now from the sheet 5.2.0 to 5.7.30, I have to seek the respective Standard Score (1st column) for every Raw Score entered in 'Report'.
Here are the steps:
A. Enter Raw scores in sheet 'Report' C7 to C15
B. Search Respective sheet depending on age (B5 cell), in our case 5.2.0 to 5.7.30 since age is  5.7
C. Populate Standard score from Raw scores by picking the corresponding column in the 4 sheets. For example, if Raw Score of Col1 is 25 (C7), then pick the Standard score of Col1 from 5.2.0 to 5.7.30 and enter in D7 and so on.
D. This way all standard scores are filled in D7 to D15.
The formula works great except for D13 in sheet 'Report' since if you observe ColD in 5.2.0 to 5.7.30, it is in descending order.
How do I change the formula to accomodate this unique column?
Answer: Well, it's not really the order that's causing the error, it's because you don't have any results! The formula you use is trying to find -159- which it cannot find at all in the age sheet. You really need something to look into ranges, so that if you have 159, it will return a positive result when you try to match against 139-160.
I have made a formula building it from smaller ones, but when assembled, the repeating units make it daunting... Also, it's an array formula, so you need to use Ctrl+Shift+Enter for it to work as intended. You can still drag the formula down.
=INDEX(
    INDIRECT("'"&LOOKUP($B$5,TblA)&"'!A6:A36"),
    IFERROR(
        MATCH(
            C7,
            INDEX(
                INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B6:J36"),
                0,
                MATCH(B7,INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B5:J5"),0)
            )*1,
            0
        ),
        MATCH(
            1,
            IF(
                1*LEFT(
                    INDEX(
                        INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B6:J36"),
                        0,
                        MATCH(B7,INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B5:J5"),0)
                    ),
                    FIND(
                        "-",
                        INDEX(
                            INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B6:J36"),
                            0,
                            MATCH(B7,INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B5:J5"),0)
                        )
                    )-1
                )<=C7,
                1,
                0
            )*
            IF(
                1*MID(
                    INDEX(
                        INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B6:J36"),
                        0,
                        MATCH(B7,INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B5:J5"),0)
                    ),
                    FIND(
                        "-",
                        INDEX(
                            INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B6:J36"),
                            0,
                            MATCH(B7,INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B5:J5"),0)
                        )
                    )+1,
                    100
                )>=C7,
                1,
                0
            )
            ,0
        )
    )
)

The single line version...
=INDEX(INDIRECT("'"&LOOKUP($B$5,TblA)&"'!A6:A36"),IFERROR(MATCH(C7,INDEX(INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B6:J36"),0,MATCH(B7,INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B5:J5"),0))*1,0),MATCH(1,IF(1*LEFT(INDEX(INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B6:J36"),0,MATCH(B7,INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B5:J5"),0)),FIND("-",INDEX(INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B6:J36"),0,MATCH(B7,INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B5:J5"),0)))-1)<=C7,1,0)*IF(1*MID(INDEX(INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B6:J36"),0,MATCH(B7,INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B5:J5"),0)),FIND("-",INDEX(INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B6:J36"),0,MATCH(B7,INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B5:J5"),0)))+1,100)>=C7,1,0),0)))

You can notice that there are some repeating blocks, namely:
INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B6:J36")

For the sheet name;
INDEX(
    INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B6:J36"),
    0,
    MATCH(B7,INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B5:J5"),0)
)

Which is a larger block to make the formula a bit more flexible (it automatically picks the correct column e.g. if you change B8 Exclusion to Col1, the formula will automatically adjust itself)
If I call the first Sheet and the second Column, it becomes much shorter and perhaps easier to understand:
=INDEX(
    Sheet,
    IFERROR(
        MATCH(
            C7,
            Column*1,
            0
        ),
        MATCH(
            1,
            IF(
                1*LEFT(
                    Column,
                    FIND(
                        "-",
                        Column
                    )-1
                )<=C7,
                1,
                0
            )*
            IF(
                1*MID(
                    Column,
                    FIND(
                        "-",
                        Column
                    )+1,
                    100
                )>=C7,
                1,
                0
            )
            ,0
        )
    )
)

Or
=INDEX(Sheet,IFERROR(MATCH(C7,Column*1,0),MATCH(1,IF(1*LEFT(Column,FIND("-",Column)-1)<=C7,1,0)*IF(1*MID(Column,FIND("-",Column)+1,100)>=C7,1,0),0)))

Disclaimer: I'm not sure if there are any way to make this even shorter, but I guess that as long as it's working right now ^^
You can download your updated sheet here.

Explanation:
As I mentioned before, the formula is based off several smaller ones and quite a few repeats of those.
INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B6:J36")

As you already know (it's a variation of a part of your own formula), this gives the area containing all the different ages. Using it and the below, we get this:
INDEX(
    INDIRECT('"&LOOKUP($B$5,TblA)&"'!B6:J36"),
    0,
    MATCH(B7,INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B5:J5"),0)
)

Into:
INDEX(
    'Sheet'!B6:J36,
    0,
    MATCH(B7,'Sheet'!B5:J5,0)
)

Index will thus look into the range 'Sheet'!B6:J36, 0 indicates it will take any column(s) and MATCH(B7,'Sheet'!B5:J5,0) returns the nth column by taking the value of B7 (in the case of your spreadsheet, Col1) and looking it into 'Sheet'!B5:J5 which gives 1. The above will thus return the range 'Sheet'!B6:B36. Let's put it in the formula:
=INDEX(
    'Sheet'!A6:A36,
    IFERROR(
        MATCH(
            C7,
            'Sheet'!B6:B36*1,
            0
        ),
        MATCH(
            1,
            IF(
                1*LEFT(
                    'Sheet'!B6:B36,
                    FIND(
                        "-",
                        'Sheet'!B6:B36
                    )-1
                )<=C7,
                1,
                0
            )*
            IF(
                1*MID(
                    'Sheet'!B6:B36,
                    FIND(
                        "-",
                        'Sheet'!B6:B36
                    )+1,
                    100
                )>=C7,
                1,
                0
            )
            ,0
        )
    )
)

This formula is itself a giant INDEX formula, with range 'Sheet'!A6:A36 and row number as the big IFERROR group. The first part of the IFERROR() gets evaluated first:
MATCH(
    C7,
    'Sheet'!B6:B36*1,
    0
)

This should be easy enough to understand. It looks for the raw score (from C7) into the range we obtained earlier, times 1 to convert everything to number (you can't look up numbers and text and expect a match). So that if there's an exact match of a number, it will return the row number of the found raw score and feed it to the INDEX(). For example, if the first row is returned, we get:
=INDEX('Sheet'!A6:A36,1)

Which is 'Sheet'!B6. If however there's no match (i.e. the raw score cannot be found), MATCH will return an error. And that's when the second part of the IFERROR comes into play:
MATCH(
    1,
    IF(
        1*LEFT(
            'Sheet'!B6:B36,
            FIND(
                "-",
                'Sheet'!B6:B36
            )-1
        )<=C7,
        1,
        0
    )*
    IF(
        1*MID(
            'Sheet'!B6:B36,
            FIND(
                "-",
                'Sheet'!B6:B36
            )+1,
            100
        )>=C7,
        1,
        0
    )
    ,0
)

This MATCH tries to find 1 within what seems to be two IFs; the first one being:
IF( 1*LEFT('Sheet'!B6:B36,FIND("-",'Sheet'!B6:B36)-1)<=C7 , 1 , 0)

FIND("-",'Sheet'!B6:B36)-1 gets the position of the last character before the - in the column 'Sheet'!B6:B36.
With those values, this FIND would return:
12-13       -> 2
145-155     -> 3
1567-1865   -> 4

The IF thus becomes:
IF( 1*LEFT('Sheet'!B6:B36,{2,3,4})<=C7 , 1 , 0)

Notice the braces here; they indicate an array and that's why this is an array formula. LEFT then extracts all the characters before the - (remember your other question, I answered with a technique very similar to this):
12-13       -> 2  ->  12
145-155     -> 3  ->  145
1567-1865   -> 4  ->  1567

Which is...
IF( 1*{12,145,1567}<=C7 , 1 , 0)

Again, 1* converts those to actual numbers because LEFT be default returns text characters. It's important here to do this because we're going to use the comparator <=, so that if the value to the left of C7 (the raw score), then the IF should return 1, else, it should return 0. Let's say that the raw score was 154. The results would be:
IF( {12,145,1567}<=154 , 1 , 0)
IF( {TRUE,TRUE,FALSE} , 1 , 0)
{1,1,0}

I just realised that the formula can be made a little shorter xD Anyway, we'll see that later. The next IF behaves in a similar fashion, but checks for the value at the right of the -:
IF( 1*MID('Sheet'!B6:B36,FIND("-",'Sheet'!B6:B36)+1,100)>=C7 , 1 , 0)

With...
                   FIND      MID('Sheet'!B6:B36, X, 100)
    12-13       -> 4     ->  13
    145-155     -> 5     ->  155
    1567-1865   -> 6     ->  1865
You can notice that this formula will stop working if you have something above 100 character long here. Anyway, the IF thus becomes:
IF( {13,155,1865}>=154 , 1 , 0)
IF( {FALSE,TRUE,TRUE} , 1 , 0)
{0,1,1}

Now that we have these, the MATCH from before becomes:
MATCH( 1 , {1,1,0}*{0,1,1} , 0)

Some simple math makes this into:
MATCH( 1 , {0,1,0} , 0)

And what is the position of the 1 in there? That's right, position 2!
Our original formula this becomes:
=INDEX( 'Sheet'!A6:A36 , IFERROR( #Error! , 2 ) )

So that if nothing was found at first, it will return an error (#N/A in this case) and instead return 2. =INDEX( 'Sheet'!A6:A36 , 2 ) gives 'Sheet'!A7.

And the slightly shorter version is:
=INDEX(INDIRECT("'"&LOOKUP($B$5,TblA)&"'!A6:A36"),IFERROR(MATCH(C7,INDEX(INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B6:J36"),0,MATCH(B7,INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B5:J5"),0))*1,0),MATCH(1,(1*LEFT(INDEX(INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B6:J36"),0,MATCH(B7,INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B5:J5"),0)),FIND("-",INDEX(INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B6:J36"),0,MATCH(B7,INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B5:J5"),0)))-1)<=C7)*(1*MID(INDEX(INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B6:J36"),0,MATCH(B7,INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B5:J5"),0)),FIND("-",INDEX(INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B6:J36"),0,MATCH(B7,INDIRECT("'"&LOOKUP($B$5,TblA)&"'!B5:J5"),0)))+1,100)>=C7),0)))

I actually removed the inner IFs, because (a>b)*(c>b) already returns 0s and 1s since TRUE multiplied by TRUE gives 1 in excel.
Comment: That's really awesome and yes its working!!! Can you explain the long version code please
Comment: @user1089173 Added an explanation which hopefully helps you understand how this works. You should still have a basic knowledge of how `INDEX`, `MATCH`, `FIND`, `LEFT`, `MID`, `INDIRECT` and `LOOKUP` work.
Answer: In Excel 2007 the formulae are returning a lot of circular reference warnings, so it may be worth adding a tag for your Excel version. Mine is Excel 2007 but with it the results you want seem achievable as below:

To shorten the formulae and reduce computation I have added in Report C5 "Table" and in D5 =VLOOKUP(B5,TblA,2,1).
I have also inserted a column immediately to the right of ColumnH ("ColD") in 5.2.0 to 5.7.30 and applied Text To Columns on Column H, with - as the delimiter.
I then applied to Report E7 and copied down to E15:
=INDEX(INDIRECT("'"&D$5&"'!A6:A36"),MATCH(C7,INDIRECT("'"&D$5&"'!"&CHAR(ROW()+59)&"6:"&CHAR(ROW()+59)&"36"),0))
and adjusted the 59s to 60s in the last three rows. Such adjustment would not be necessary if ColumnI were moved far enough to the right.
In E13 I changed the match from exact to next higher (final 0 to -1).
For Figures I cheated and changed K23 in 5.2.0 to 5.7.30 to 22 from 21-22, but such banding could, for other columns, be treated in much the same way as I did for ColD.


From 3D -normal vector to $N$ -dimensional normal -vector?

Question: I am studying Griffiths "Introduction to electrodynamism", page 7 and chat here. I want to understand the normal vector in more general situations and also here in 3D. What is wrong below? Have I misunderstood the $\nabla$ or the plane -equation or something else?
More about n -dimensional normal -vector in Wikipedia here.

I want to fast calculate and verify the normal vector with the 3D -object and then understand the case with n -dimensional object. I think the general case is with $\nabla$ so I created the implicit form for the surface i.e. $\bar F= \hat i + 2 \hat j + 3\hat k$ so $\nabla \bar F=(1,2,3)$ but manually I got $\hat n = \frac{1}{7}(6,3,2)$. Something wrong, what? Not pointing in the same direction.

Related

Gradient's Wikipedia entryhere
Comment: What is this manual method you are using?
Comment: ...with slightly different configuration, I got $\hat n=\frac{1}{7}(6,3,-2)$ but wanting to do this kind of things faster, still nothing like the $\nabla \bar F$. Is the implicit form of the plane right? Yes it should be, not seeing it yet.
Answer: If a hypersurface in $\mathbb{R}^n$ is implicitly defined by the equation $F(x_1, \dots, x_n) = d$ for some $d \in \mathbb{R}$, then you can consider it as a level curve of the function $F$. As you mention, $\nabla F$ is normal to the surface, so, provided you can determine $F$, you can find the normal vector at a point on the surface.
In your situation, the surface is defined by an equation $F(x, y, z) = d$. Furthermore, it is a plane, so $F(x, y, z) = ax + by + cz$, where $a, b, c \in \mathbb{R}$ are yet to be determined. From the figure, we know that $(1, 0, 0)$, $(0, 2, 0)$ and $(0, 0, 3)$ are on the surface, and therefore must satisfy $F(x, y, z) = d$. Substituting in each point, one by one, we get:
\begin{align*}
F(1, 0, 0) &= d \Rightarrow a(1) + b(0) + c(0) = d \Rightarrow a = d,\\
F(0, 2, 0) &= d \Rightarrow a(0) + b(2) + c(0) = d \Rightarrow 2b = d \Rightarrow b = \dfrac{1}{2}d,\\
F(0, 0, 3) &= d \Rightarrow a(0) + b(0) + c(3) = d \Rightarrow 3c = d \Rightarrow c = \dfrac{1}{3}d.
\end{align*}
Now replacing $a, b,$ and $c$ in the equation for $F$, we obtain:
$$dx + \frac{1}{2}dy + \frac{1}{3}dz = d.$$
If $d = 0$, then every element $(x, y, z) \in \mathbb{R}^3$ would satisfy the equation and hence be on the surface. As this is not the case, $d \neq 0$ so we can divide both sides of the above equation by $d$, leaving us with:
$$x + \frac{1}{2}y + \frac{1}{3}z = 1.$$
If you like, you can multiply both sides by $6$ so that all the coefficients are integers, in which case the equation is $6x + 3y + 2z = 6$. Therefore $F(x, y, z) = 6x + 3y + 2z$, so $\nabla F = (\frac{\partial F}{\partial x}, \frac{\partial F}{\partial y}, \frac{\partial F}{\partial z}) = (6, 3, 2)$ is a vector normal to the surface. As $\|\nabla F\| = \sqrt{6^2 + 3^2 + 2^2} = \sqrt{49} = 7$, $\frac{1}{\|\nabla F\|}\nabla F = \frac{1}{7}(6, 3, 2)$ is a unit normal to the surface.

Note, as this surface is a plane in $\mathbb{R}^3$, you can obtain the normal in a quicker way.
If we can find two linearly independent vectors in the plane, then their cross product will be normal to the plane. We can use the points given to find two such vectors. As $(1, 0, 0)$ and $(0, 2, 0)$ are in the plane, the vector $(0, 2, 0) - (1, 0, 0) = (-1, 2, 0)$ from $(1, 0, 0)$ to $(0, 2, 0)$ is in the plane. Likewise $(0, 3, 0) - (1, 0, 0) = (-1, 0, 3)$ is also in the plane. Clearly these two vectors are linearly independent. Therefore:
$$(-1, 2, 0) \times (-1, 0, 3) = \left| \begin{array}{ccc}
i & j & k \\
-1 & 2 & 0 \\
-1 & 0 & 3 \end{array} \right| = 6i + 3j + 2k = (6, 3, 2),$$
as before.

As for your concerns in higher dimensions, the approach I outlined is the standard approach, except that you usually know $F(x_1, \dots, x_n)$ and $d$, so all that you need to do is calculate $\nabla F = (\frac{\partial F}{\partial x_1}, \dots, \frac{\partial F}{\partial x_n})$. Most of the time $\nabla F$ will not be a constant vector as its components can be functions - in fact, the only time $\nabla F$ is constant is when your surface is a hyperplane in $\mathbb{R}^n$. Either way, $\nabla F$ is a vector which (can) vary from point to point; this is what we call a vector field. More formally, a vector field on a hypersurface $\Sigma$ is a function $\Sigma \to \mathbb{R}^n$; $\nabla F$ is such a function.
Answer: For a level surface, $F(x,y,z)=0$, the gradient $\nabla F$ gives a formula for a normal to the surface at each point.
The surface you have graphed above looks to be a plane which passes through the points: $(1,0,0)$, $(0,2,0)$, and $(0,0,3)$. An equation for this plane is $6x+3y+2z-6=0$. So if we let $F(x,y,z)=6x+3y+2z-6$ then $\nabla F=6\hat{i}+3\hat{j}+2\hat{k}$ (the components are just the coefficients of $x,y,z$). Since $\nabla F$ is constant, this plane has the same normal everywhere (in fact this happens because it is a plane).
Of course, any non-zero multiple is also a normal so any vector $\hat{n} = 6t\hat{i}+3t\hat{j}+2t\hat{k}$ where $t \not=0$ is also normal to the plane. If we normalize these vectors we are left with two choices: $\pm\frac{1}{7}\left(6\hat{i}+3\hat{j}+2\hat{k}\right)$. These are the upward and downward unit normals for this plane sometimes called "orientations" for the plane.


Testing for statistically significant difference between two groups: Frequentist vs Bayesian

Question: When testing for differences between two groups or samples, what is the best approach in concluding statistical significance?
The orthodox approach is to take a t-test or an Anova when there is more than 1 factor to consider.
I've also seen how t-tests can be considered as a linear model and can be generalized. R packages such as BEST or BRMS use a Bayesian approach such as MCMC to obtain parameter estimates. But what can you do after obtaining these estimates to say the two groups are or are not statistically significant?
Comment: A Bayesian technique is common when the mean of the population which you try to estimate is itself distributed according to some distribution which you (approximately) know. For instance you are a wine company that wishes to decide on which plot of grapes to select for the more exclusive bottling. Then you can compare two samples taken for each plot (in order to estimate the mean quality of the plots), but you may possibly include knowledge/information how the quality of the plots is usually distributed. (without such prior information Bayesian techniques are subjective and  not exact)
Answer:
When testing for differences between two groups or samples, what is the best approach in concluding statistical significance?

This is completely contextual.  If you want guaranteed frequency properties (assuming the assumptions of each procedure are not grossly violated), then I would say a Frequentist approach is best.  If you have small data and good prior information about the effects or baseline values, then I would say Bayesian methods are best.  Depends on what you want.

The orthodox approach is to take a t-test or an Anova when there is more than 1 factor to consider.

Assuming the outcome is normally distributed.  This isn't the case for binary outcomes (though I'm sure you could make a good argument for ANOVA).

But what can you do after obtaining these estimates to say the two groups are or are not statistically significant?

Ah, now this is a different question all together.  If you want to  evaluate if two groups are not different, you might be interested in a "non-inferiority" test.
Answer: A frequentist p-value and a Bayesian posterior tell you different things.

P-value: $P(\text{data} | \text{hypothesis})$ or the probability of data given a hypothesis being true (more precisely a test statistic which is a function of data but that detail doesn't really matter here).
Posterior: $P(\text{hypothesis} | \text{data})$ or the probability of an hypothesis being true given data you've observed.

Whichever is better for your situation depends on what question you're answering! My guess is that most of the time people care about the probability of some hypothesis being true given the data they've seen and so they should look at a Bayesian posterior. This guess is because it's my own experience on what I care about when I do analysis and the fact that when people misinterpret the p-value they usually misinterpret it as a Bayesian posterior (see Am I taking crazy pills here or this A/B testing tool's documentation dead wrong?).
Answer: Unfortunately, there is not a clear answer to this question.  Bayesian methods and Frequentist methods answer different questions.  They are also responding to different criteria and have different capabilities.  If the data is normally distributed, with unequal variances, and a flat prior then an analytic form exists for both the Frequentist and the Bayesian method, but the results are not the same.
Using a Frequentist method, you can use Student's distribution as your test distribution, with the appropriate test statistic.  Using a Bayesian approach, you would use the Behrens-Fisher distribution.  You have the same data but a different outcome.  Understanding that difference in outcome is key to understanding the differences.
Let us start with a simple null hypothesis of $H_0:\mu_1\ge\mu_2$ with $H_A:\mu_1<\mu_2$.  This is a one-sided test.  You would calculate your $t$ statistic and then find the area under the distribution, usually by looking it up on a table.  You are conditioning your result on the sample estimate of the variances.  You are also conditioning on the null as if perfectly true.  If it is in the rejection region, then your result is statistically significant.  Your rejection region would be chosen by choosing a value, $\alpha$.  If you follow this procedure an infinite number of times then you are guaranteed to be made of a fool of no more than $\alpha$ percent of the time.
When using Bayesian thinking, there is no null hypothesis.  Without a null to condition upon, the nature of the problem changes.  Instead of a null and an alternative, there are just two hypotheses.  If your priors are flat as assumed above, then you would calculate your test statistic, though with a different formula, and you would find the area under the curve of the Behrens-Fisher distribution.  You would have arrived at that curve by marginalizing out all possible values of $\sigma_1^2,$ and $\sigma_2^2$.  Rather than condition on the point statistic for the sample, a Bayesian method considers all possible values for both parameters and removes them through the law of total probability by summing the values weighted by their densities.  That provides a posterior probability for both regions.  As there is no $\alpha$, there is no concept of "significance."  There is no acceptance or rejection region.  There are just posterior probabilities given the data observed.
The fact that identical data does not produce an identical inference should be the warning that you are not doing the same thing with the same data.  The Frequentist statistician is asking, "if the null is true, what is the probability of observing a result as extreme or more extreme than the sample?"  The Bayesian statistician is asking, "given the data, what is the probability that $\mu_1\ge\mu_2$ versus its complement?"  The Frequentist solution guarantees an average coverage level, $\alpha,$ and also permits you to control for statistical power.  The Bayesian method is providing a direct probability statement given the data.  However, a p-value of less than five percent does not map to a ninety-five percent Bayesian probability that a hypothesis is true.  A p-value of less than five percent might easily be a posterior probability of eighty-three percent or some other value.
The p-value is based on the sample space as being random.  The Bayesian posterior is based on the parameter space being treated as random.  You cannot use p-value thinking to work out Bayesian inference.
Now let us change the problem slightly.  If the null hypothesis were $H_0:\mu_1=\mu_2$, then you are conditioning on that as if true when testing your sample.  There does not exist a Bayesian counter-part to this hypothesis.  The probability a point is true is always zero for continuous data.  The area of the hypothesis region is zero, always.  This is called a sharp null hypothesis.
Bayesian methods cannot test a sharp null.  The only workaround for this problem is to build a region of practical equivalence.   The Bayesian hypothesis would be $|\mu_1-\mu_2|<k$.  For example, if you took Indiana US quarters and Florida quarters and flipped them 100,000 times each would you treat a probability of heads of .5 as different from .5000000001?  What if you flipped them eight times?  A Bayesian method can work on "close enough" but not equal.
Now let us make the problem a bit more specific.  Let's imagine that we are concerned that the Cincinnati Red's batting average is higher than the Pittsburgh Pirate's batting average for the 2020 season.  Let's pretend that it is Spring 2020 and we have seen five weeks of games.  We have robust prior information.  We know the batting averages of all non-rookies for the previous seasons.  For rookies, except walk-ons, we have their data from their time in the farm teams, and we have prior knowledge of how rookies change between the minors and the majors.
The Frequentist would have to discard all of this information as it is not part of the sample.  The Bayesian would fully incorporate this into the prior.  It is known that the Frequentist parameter estimates would be inadmissible statistics.  The loss of information would make the Frequentist estimators noisier and the inferences less reliable.
This brings up the question of "why are you asking which team has a higher average?"  If you are gambling, then you have to use a Bayesian method because Bayesian methods are coherent.  They generate probabilities that can be used to set betting odds.  However, if your concern is scientific, then the Frequentist estimators are unbiased while the Bayesian ones are guaranteed to be biased.  Furthermore, you do not have to assume that the prior information is good information.
Are your concerns Frequentist concerns or Bayesian ones?  Answer that and you have your methodology.  If you use a Frequentist method, then you are significant if you are in a rejection region for a given $\alpha.$  If you use a Bayesian method, it is significant if you decide it is.


Interpretation R-squared and F statistics

Question: Hope I get some help
I have two predictor variables and one outcome. I have used multiple regression to get estimates. I get a value of 0.005 for adjusted Squared R. I got significant results for the model. The p-value of the F-statistic is 0.003, meaning that at least, one of the predictor variables is significantly related to the outcome variable. The coefficients table shows both predictors are significant.
I do not understand what happens given a very low value of Squared R. Which one I need to consider Squared R or the coefficient table for each predictor?
Can anyone explain it?
Comment: How many observations do you have?
Comment: thanks, Dave, 1500
Answer: R Squared (adjusted or unadjusted) can be low even with a low F statistic p-value. Consider a simple linear regression (one regressor), which has the property that the f statistic p-value equals the t statistic p-value and, providing an intercept is included, the R squared value equals the (Pearson) correlation between the dependent variable and the regressor.
If there is a lot of unexplained variation in the regression, then a plot of the independent variable against the regressor would show wide variation of points about the line. The R squared value would be low since this is the proportion of the dependent variable that is "explained", statistically at least, by the regressor.
R squared is a goodness of fit statistic, it is used to give an overall idea of the amount the independent variable is "explained" by the regressors.
Comment: Thanks, Single, I know that ( sorry). But I think the issue is related to sample size.  when the sample size is very large, any small difference would be significant.
Answer: The F stat is:
$$
F = \frac{(RSS_0 - RSS)/p}{RSS/(n - p -1)}
$$
Where $RSS_0$ is the residual sum of squares from the intercept only model and $RSS$ is the residual sum of squares from the full model. $n$ is the sample size and $p$ is the number of regressors.
Asymptotically, $RSS_0$ and $RSS$ will both grow at rate $n$, these will cancel out. This leaves the $n$ in the denominator of the denominator, meaning that $F$ will grow at rate $n$.
$R^2$ does not grow with $n$, so if the sample size is large you can get an big F-stat even with very small $R^2$.


Win 2003 IIS6 - FTP Site Configuration

Question: I'm fairly new to server administration, so please be gentle with me.
On my dedicated server, in IIS6 manager, I've just added a new site called "upload-system", and modified the FTP settings so the home directory is \upload-system\www\uploads\, instead of just \upload-system. This is all okay but have one issue:
When I connect to the server using an FTP client, for example CuteFTP, I land in my new home directory, \uploads, just like I wanted, but from there, there's nothing stopping me from going back a directory, using the up-level/back button on the FTP client. I'm actually able to go right back to the c: directory on the server. I created this FTP site so my client can FTP his photos there, I really don't want any of their office staff having access to other parts of the server when connecting via FTP.
How do I stop this from happening in IIS6 manager?
Answer: Sounds like you need to set up Isolate Users Mode
That way, users are restricted access to their own home folders, and cannot navigate out of it
Comment: That's exactly what I need to set up, thanks Mathias, much appreciated.

Win 2003 IIS6 - FTP Site Configuration


Is value in the matrix notation

Question: Given an $m\times n$ matrix $A=\begin{bmatrix} a_{1,1}&...&a_{1,n} \\ \vdots&\ddots&\vdots \\ a_{m,1}&...&a_{m,n} \\ \end{bmatrix}$, say that I wanted to describe a set generator $S(A)$ which only consists of every individual entry in A, using set builder notation. I want to write something like this...
$$S(A) = \left\{s\in\mathbb{R}\ :\ s\in A\right\}$$
But I know that '$s\in A$' is incorrect bc '$\in$' should only ever be used wrt sets; is there a common convention (perhaps another symbol) that I could use?
Answer: What is a matrix, exactly?
A good formal definition of a matrix is a function
$$A : [m] \times [n] \rightarrow \mathbb{R}$$
where $[m] = \{1,2,\cdots,m\}$ and $[n] = \{1,2,\cdots,n\}$. The codomain could be any set, but let's use $\mathbb{R}$ for sake of example.
Thus, when you see an entry $a_{i,j}$, you can essentially think of it as the output $A(i,j) \in \mathbb{R}$. In this case, you can easily see that the notation you're looking for is simply the range $ran(A)$of $A$:
$$ ran(A)  = \{A(i,j) | (i,j) \in [m]\times [n]\} = \{a_{i,j} | (i,j) \in [m] \times [n]\}$$
Comment: I strongly recommend _not_ using $\operatorname{ran}(A)$ for the set of entries of $A$ because that notation is often used for the range of $A$ considered as a linear map $\mathbb R^n \to \mathbb R^m$. You can use the other two notations on the last line, though.

Is value in the matrix notation


Passando um array via Ajax para view Django

Question: Pretendo passar um array via Ajax para a minha view do Django,mas sempre ocorre um erro

stream_or_string = stream_or_string.read()
AttributeError: 'NoneType' object has no attribute 'read'

$.ajax({
         url: 'ajax/',
         type: 'POST',
         dataType: "json",
         data: {disciplina: array_list},
         success: function(data) {
              //alert("Cadastro realizado com sucesso!");
         },
         failure: function(data) {
              //alert("Ocorreu um erro, realize a operação novamente");
         }
    }).done(function(data){
        //alert(data);
    });

view.py
for lista in serializers.deserialize("json", request.POST.get('disciplina')):
    idDisciplina = [lista]

Caso eu faço algo desse tipo
num=request.POST.get('disciplina')

Ele ainda recebe caso eu passe apenas um único valor e não um array
Deserialize usei baseado no que achei, mais não entendi bem ainda a sua funcionalidade, alguém pode ajudar ?
Answer: Primeiro você deve informar o csrf token para poder submeter via metódo 'POST' de forma segura. Segundo, quando você envia um array, django recebe a variável com colchetes no final dela, no caso seria: request.POST.get('disciplina[]').
Veja como é recebido (neste exemplo fiz um array ['a', 'b'] como exemplo:
>>> print(request.POST)
<QueryDict: {'disciplina[]': ['a', 'b'], ...>

Veja que sem colchetes a variável não existe:
>>> print(request.POST.get('disciplina'))
raise MultiValueDictKeyError(repr(key))
django.utils.datastructures.MultiValueDictKeyError: "'disciplina'"

Com colchetes:
>>> print(request.POST.get('disciplina[]'))
b

Como você pode notar, dessa forma é capturado apenas o último item do array. Para pegar todos os itens você deve usar getlist:
>>> meu_array = request.POST.getlist('disciplina[]')
>>> print(meu_array)
['a', 'b']

Para enviar com csrf, você pode adicionar o csrfmiddlewaretoken ao dicionário:

Comment: Vou estar testando e falo se funcionou, mas ainda estou com uma duvida pra q a utilização do serializers? Se pode-se usar um ex básico ajudaria mut


Proof of associativity for elliptic curve point addition: Trouble understanding lines intersecting ellipic curves lemma

Question: From Elliptic Curves Number Theory and Cryptography by Washington:
$P_K^1$ is the 1-dimensional projective space.
Lemma 2.2:
Let $G(u,v)$ be a nonzero homogeneous polynomial and let $(u_0:v_0) \in P_K^1$. Then there exists an integer $k \geq 0$ and a polynomial $H(u,v)$ with $H(u_0,v_0)\neq0$ such that $G(u,v)=(v_ou-u_0v)^kH(u,v)$.
Proof:
Suppose $v_0\neq 0$. Let $m$ be the degree of $G$. Let $g(u) = G(u, v_0)$. By factoring out as large a power of $u-\frac{u_0}{v_0}$ as possible, we can write $g(u)=(v_0u-u_0)^kh(u)$ for some $k$ and for some polynomial $h$ of degree $m-k$ with $h(u_0)\neq 0$. Let $H(u,v) = v^{m-k}h(u/v)$, so $H(u,v)$ is homogeneous of degree $m-k$. Then $G(u,v)=v^mg(u/v)=(v_0u-u_0v)^kH(u,v)$, as desired.
The parts I'm having trouble with are:
(1) If you can "factor out" polynomials, why bother with defining $g(u)$ by fixing $v_0$ at all? Wouldn't you be able to factor out $(v_0u-u_0v)$ from $G(u,v)$ and be done with it?
(2) Why does $h(u_0)\neq 0$? We factored out $u-\frac{u_0}{v_0}$ from the polynomial, which assumes that $u-\frac{u_0}{v_0} \neq 0$ or that $u\neq \frac{u_0}{v_0}$. I can't figure out why this is stated.
(3) How does this help us get the order to which a line intersects a curve at a point?
Answer: The idea behind this lemma is to look for zeros of $G(u,v)$ and see how many times it's a zero at $G(u,v)$. Given $G(u,v)$ is a homogeneous polynomial, the choice of $(u_0:v_0)$ is to represent any zero for the function. The question is, how many times is it a zero?
On the $P^k_1$ projective space, when $v_0\neq 0$, we can have any number for $u_0$, essentially making a line. The only other point is $(1:0)$ which is the point at infinity. So fixing $v_0$ doesn't limit the domain of $G(u,v)$ very much, just ignores 1 point (the point at infinity). The proof thus fixes $v_0$ and defines $g(u)=G(u,v_0)$
Now notice how if $G(u_0,v_0) \neq 0$, that the lemma is trivially true since we can make $k=0$ and $H=G$. It's really only when $G(u_0,v_0) = 0$ that we have anything interesting.
$g(u)$ is a polynomial, so we can factor out any polynomials that go in evenly. $g(u_0/v_0)=0$ implies $g(u)=(u-u_0/v_0)p(u)$ where p(u) is a polynomial. If $p(u_0/v_0)=0$, we can do the same thing and get $p(u)=(u-u_0/v_0)q(u)$ where $q(u)$ is a polynomial. $g(u)=(u-u_0/v_0)p(u)=(u-u_0/v_0)(u-u_0/v_0)q(u)=(u-u_0/v_0)^2q(u)$
We keep going until we get to the point where $h'(u_0/v_0)\neq 0$, which is $k$ times and get:
$g(u)=(u-u_0/v_0)^kh'(u)=\frac{v_0^k}{v_0^k}(u-u_0/v_0)^kh'(u)=(uv_0-u_0)^kh(u)$ where $h(u)=h'(u)/v_0^k$
$h'(u_0/v_0)\neq 0$ implies that $h(u_0/v_0)\neq 0$.
$H(u,v)=v^{m-k}h(u/v)$ means that $H(u_0/v_0)=v_0^{m-k}h(u_0/v_0)$ which since $v_0\neq 0$ and $h(u_0/v_0)\neq 0$, $H(u_0/v_0)\neq 0$.
The rest of the proof goes through.
Edit: Looks like the proof in the book was in error: http://www.math.umd.edu/~lcw/ECerrata.pdf


Standard Error of Noise Variance in Least Squares

Question: Suppose I'm doing ordinary least squares with homoskedastic errors, so something like:
$$
y = X \beta + \epsilon
$$
where $\epsilon \sim \mathcal{N}(0,\sigma^2)$.
I know how to estimate the expected value of $\sigma^2$, but I'm having trouble identifying how to compute a standard error on that estimate. Something analogous to what's shown on page 47 (pdf page 58) of this book.
Comment: @shadowtalker I guess so, but a more accurate way to say it is the sampling distribution of $\hat{\sigma}^2$ (which is random), not $\sigma^2$ (which is fixed).
Comment: What do you mean by "the expected value of $\sigma^2$"?  $\sigma^2$ is a non-random constant.  Do you try to say how to compute the standard error of the estimate of $\sigma^2$?
Comment: Very closely related questions: https://stats.stackexchange.com/q/224042/36229 and https://stats.stackexchange.com/q/20227/36229 and https://stats.stackexchange.com/q/312337/36229
Comment: @Zhanxiong it sounds like they are interested in the sampling distribution of the model variance $\sigma^2$. As in https://stats.stackexchange.com/q/121662/36229
Answer: Suppose the design matrix is $N \times (p + 1)$, I am assuming you want to calculate the variance of $\hat{\sigma}^2 = \frac{1}{N - p - 1}\epsilon'(I - H)\epsilon$, which is an unbiased estimator of $\sigma^2$, where $I$ is the order $N$ identity matrix and $H = X(X'X)^{-1}X'$.  In this thread, it has been shown that
\begin{align}
(N - p - 1)\hat{\sigma}^2/\sigma^2 \sim \chi_{N - p - 1}^2,
\end{align}
by which and the variance of a $\chi_k^2$ r.v. is $2k$ follows that
\begin{align}
\operatorname{
Var}(\hat{\sigma}^2) = \frac{\sigma^4}{(N - p - 1)^2} \times 2(N - p - 1) =
\frac{2\sigma^4}{N - p - 1}.
\end{align}
Estimate the $\sigma^4$ in the numerator by $\hat{\sigma}^4$, it follows that the standard error of $\hat{\sigma}^2$ is $\frac{\sqrt{2}\hat{\sigma}^2}{\sqrt{N - p - 1}}$.
Answer: The other answer here shows you the variance of the estimator of the error variance under the assumption that the errors are normally distributed, which is the specified form in the Gaussian regression model.  I would counsel against using this method, since it is often the case that the actual data we use in a regression model does not conform to this assumption.  In particular, it is not unusual for the residuals in regression data to exhibit leptokurtosis (or sometimes platykurtosis), contrary to the model assumptions.  Many aspects of regression analysis are robust to the normality assumption, but this aspect of the model is not robust --- it is a case where the asserted formula hinges on the Gaussian model assumption rather than being determined by actual analysis of the data used in the model.  In general, this to be avoided if we want to use robust methods.
Suppose we are willing to generalise the model slightly by no longer assuming a normal distribution for the error term (we will still assume that the errors are IID with zero mean and fixed variance).  In the generalised case we may consider error variables with kurtosis $\kappa$ (but still assuming a simple linear regression), in which case the variance of $\hat{\sigma}^2$ is actually given by:
$$\mathbb{V}(\hat{\sigma}^2) = \bigg( \kappa - \frac{n-p-4}{n-p-2} \bigg) \frac{\sigma^4}{n-p-1}.$$
In the special case of a mesokurtotic error distribution (e.g., the normal distribution) we have $\kappa = 3$ and so the variance formula reduces to the more familiar form:
$$\mathbb{V}(\hat{\sigma}^2) = \frac{2 \sigma^4}{n-p-1}.$$
Now, it is possible to estimate the kurtosis of the error distribution from the residuals in the regression model, so in principle it is possible to estimate the variance of the estimator for the error variance in a way that does not hinge on the assumption of a mesokurtic error distribution.  For example, if you have an estimator $\hat{\kappa}$ for the kurtosis of the error distribution (e.g., from a kurtosis estimator using the residuals) then you could estimate the standard error of the estimator for the error variance as:
$$\hat{\text{se}}_n = \sqrt{\hat{\mathbb{V}}(\hat{\sigma}^2)}
= \sqrt{ \hat{\kappa} - \frac{n-p-4}{n-p-2}} \times \frac{\hat{\sigma}^2}{\sqrt{n-p-1}}.$$
Assuming you use a consistent estimator $\hat{\kappa}$, this latter estimator will be a consistent estimator of the true standard deviation of the estimator of the error variance.  For more details on the moments of these types of sampling quantities you might find it useful to consult O'Neill (2014) (that paper deals with standard sampling quantities outside of regression, but its results can easily be adapted to the regression setting).
Comment: Thank you, that O'Neill review is an extremely handy reference!


problem in date column with hibernate hql query

Question: i hav a  hql query..it gets the data used between two dates..i wriiten a query like this
from com.cod.model.Billing where datecolumn between '2011-4-4' and '2011-4-20'
but i didn't get any results from this query ..i checked out this query in mysql with sql query.there it's workng fine..but if i write same query in hql it returns null results
 can any one say what's the problem in my query.thanx in advance
Comment: It is a good practice to follow up the answers the questions posted by yourself
Answer: The dates mentioned have to be of the type java.util.Date and not String


whats the difference between built in android applications and user application

Question: I am working on application which needs to auto answer the call . but due to security issues I am unable to acheive it form my app , but I can do it through android's Phone applications,
can anybody tell me that how android differentiate its built in apps and user apps..??
do it uses any certificate to diffretiate....???
Comment: Have you tried setting the proper permissions?
Comment: Paste your permission settings from you manifest file. It may be easier to debug you app then.
Comment: Hi these are the permissions that i set in manifest file.........  "android.permission.CALL_PRIVILEGED"
"android.permission.CALL_PHONE"
"android.permission.READ_PHONE_STATE"
"android.permission.MODIFY_PHONE_STATE"
Answer: Sorry, third party applications can't do this.  This isn't just a matter of permissions -- supporting such a thing requires work to provide a UI for the user to determine and manage what apps can do this, and no such thing has been written at this point.


Crucial C300 256GB SSDs in Dell M605 (or similar) Blade with Perc H700

Question: Does anyone have any experience with 3rd party SSD drives in any of the Dell Blade range?
I'm not prepared to pay dell £1000 for a 100GB SSD drive, when I can pick up 2x 256GB drives for £800!
I've read this : Dell SSD options and 3rd party SSDs in Dell hardware but it doesn't seem to mention anything about the raid controller, or if the new drives will simply slot into the SAS/SATA enclosure.
Answer: When you purchase SSDs you should be aware that there is a major technology difference between consumer grade and "enterprise" grade SSDs. I wouldn't recommend MLC for buisness production environments.
Consumer grade SSDs will use MLC (Multi-Level Cell):

Lower Write Performance
Faster wear over time

Enterprise SSDs are SLC (Single-Level Cell):

Better Write Performance
Lower wear rates

I believe the crucials are all MLC so you probably want to go a SLC SSD drive like the Intel-X25E. I can't tell you how they work in blades, but if you watch the Server Fault blog you should see how they perform in a Dell R710 in the next couple of weeks. We have an H700 controller and I believe the firmware will no longer complain about 3rd party drives -- but we will find out soon.
As far as the SATA/SAS it will depend on your backplane. The R710 will support either SATA or SAS with the same backplane as long as they are not mixed (Might be workarounds -- but not Dell recommended). I would call dell and ask about your particular blade server when it comes to this.
Comment: Really helful answer, thank you Kyle. Am I right in thinking that the wear rates simply mean that the consumer level drives will refuse to write any more data earlier than the enterprise level drives will? Or is there something more sinister/serious to bear in mind when considering using the cheaper disks in a server?
Comment: This should actually be updated... some enterprise drives (the newest Intel 710 we just put in our DB Severs are MLC). But enterprise drives seem to have less firmware bugs and I would put more trust in them. When an SSD fails it generally won't do so gracefully (in particular, it won't be predicted behind many raid cards these days). RAID will help, but personally I only really trust the best SSDs even in raid for critical servers.
Answer: I think the answer for the performance issue is twofold: First, I own one of the C300 (265 Gb) SSDs,
I use hdparm to secure-erase the drive, and I hot-plug because the unlock scripts don't seem to work with the controller? ioctl() I/O error.

(Hint don't use Windows, it is BRUTAL to the drives. It only exists to make the COMP-TIA racket more profitable.) fdisk the drive with -b 4096 with the CHS data < hdparm -i [node].
I use Mark Lord's wiper.sh that comes with the source code version of hdparm[v.9.37] that TRIMs the free space, BUT requires a partition file system, or it will abort.

THEN (3rd step) I use gparted on KNOPPIX 6.4.4 to offset the first partition by 4MB, and thereafter align by the MB, with 4MB separations, where partitions are multiples of same 4MB. (not hard at all, obviously.) I bear you witness that my MLC EXCEEDED the 355 Mb cached read spec. You will get different answers with different kernels.
   The drive does weather very well, if one is very gentle. I suppose this method is superior, because I think that simplicity is the best asset. Also make sure that your fstab reflects noatime,norelatime,discard in the mount options. This might not fly in high-security enviornments, though.


finding essential right ideals in a ring

Question: Definition: A submodule $N$ of an $R$-module $M$ is said to be essential if, $N\cap K\neq0$ for every nonzero submodule $K$ of $M$.
My Question: Let $M$ be an $R$-module. If $N$ is an essential submodule of $M$, then for any $x \in M$, $N_x=\{r\in R|rx\in N\}$ is an essential left ideal in $R$.
My attempt: Let $I$ be a nonzero left ideal of $R$, then we show that $N_x\cap I\neq 0$. If possible let $N_x\cap I\neq 0$, then ?
I an stuck in finding a contradiction. Please help me.
Comment: I reckon that $N_x$ is a **left** ideal, not a right ideal.
Comment: sorry. Now, i have corrected it.
Comment: You don't need a proof by contradiction to show this. If $I$ is a left ideal of $R$, then what can you say about the subset $Ix = \{rx \mid r\in I\} \subseteq M$?
Answer: I like to use this characterization:
$N$ is essential in $M$ iff for every $m\in M\setminus N$ there exists an $r\in R$ such that $rm\in N\setminus \{0\}$.
So wlog let $y\in R\setminus N_x$.
Then $yx\in M\setminus N$. Therefore there exists $z\in R$ such that $zyx\in N\setminus\{0\}$.  This means $zy\in N_x\setminus \{0\}$. By the characterization above, $N_x$ is essential in $R$.


What is the difference between the Romantic and post-romantic periods?

Question: This question came about when I was reading  Blair Allen Johnston's dissertation, 'Harmony and Climax in the late works of Sergei Rachmanioff'
He loosely groups the post romantic period within 1890-1940 and acknowledges the possible counterproductiveness of period groupings.
I quote:
"Romantic is characterized by chromatic expansion  and  the  development of  striking elaborations  of  linear  tonal syntax,  then  the Postromantic is characterized by exaggeration and  ultimately  fragmentation  of tonal syntax, and the juxtaposition  or superimposition of conventional functional tonal  structures  and intense chromatic and/or  modal structures that challenge and even deform the functional tonal basis.  In my view, complex interaction of  variegated  melodic-harmonic  components is one source of the continuing fascination  Postromantic  music holds."
https://deepblue.lib.umich.edu/handle/2027.42/63739
I understand this quote does answer the question that I'm asking. But I'm hoping for a clearer explanation as I don't fully understand what he's saying, in terms of chromatic expansion, elaborations of linear tonal syntax, exaggeration and fragmentation of tonal syntax and the superimposition of conventional functional tonal structures and intense chromatic and modal structures.
Comment: The except does answer the question but in a summarised way. I was hoping for a little more explanation as I don't fully understand the difference between a chromatic expansion and an intense chromatic structure. And I don't quite understand what he means when he says that the Romantic period contains 'elaboration(s) of linear tonal syntax'.
Comment: I've heard someone else have a dislike of Blair, what is the reason can I ask?
Comment: Oh, Blair's dissertation! What a great guy. But unfortunately I'm not sure I understand the question. How does his excerpt not answer the question in your title?
Comment: If you open the linked file and you search for the terms you've quoted you'll find the answers. Do you want me to copy-paste them here for you? Anyway, thank you  for this link and grace to your question I have also found and downloaded this book file: www.ssoar.info Music of the twentieth century: a study of its
elements and structure
 (Ton de Leeuw)
Comment: chromatic expansion: recontextualizing multiple tonal centers à la Wagner, Sibelius, etc. Intense chromatic structure: pantonalism and dissolution of tonal centers à la Schoenberg, Webern, Berg, etc. Let's not forget the dearth of fin du siècle genres: musique concrete, fluxism, dadaism, expressionism, early modernism and many others. Gone were the days of being transfixed my mythical beasts, magic, and fantasy that pervaded much for Romantic culture. A world shocked into sobriety by the first world war. This is the difference in periods.
Answer: Let me try to unpack the quotation a bit.
Romantic characteristics:

"Chromatic expansion": more possibilities for use of chromaticism, in context, I think this refers mostly to harmonic patterns (like chromatic chord progressions you wouldn't generally find in Bach or Mozart, for example)
"Development of striking elaborations of linear tonal syntax": "linear tonal syntax" here is likely a reference to Schenkerian theory, one of whose focuses is on trying to isolate so-called underlying "linear" voice-leading structures (e.g., simple descending lines) that are supposedly elaborated with lots of other "surface-level" notes, in a manner similar to baroque-style diminution and embellishment

The "striking elaborations" he's referring to are probably places where these "linear" structures become quite chromatic and/or incorporate extensive chromatic digressions in romantic period music.
Postromantic characteristics:

"Exaggeration and ultimately fragmentation of tonal syntax": in other words, traditional tonal progressions get used in both exaggerated ways (e.g., lots of repetitive tonal progressions) and fragmented (e.g., rather than getting full phrases of tonal harmonic progressions, you merely get fragments -- small parts of chord progressions that sound sort of like what Mozart would do, but then deviate off in an unexpected manner or get interrupted by other harmonies)
"Juxtaposition or superimposition of conventional functional tonal structures and intense chromatic and/or modal structures": this could be a reference to a lot of things, but generally it's referencing the idea that you might have something that sounds simple and harmonically straightforward (similar to Mozart) and then suddenly an intense chromatic passage follows or inserts itself, or rather than following "tonal" progressions, the music veers off into a mode "modal" harmony
"Challenge and even deform the functional tonal basis": in other words, at some points you may not know where the harmony is leading anymore, you may not end up in the key you expect; in general, the harmony becomes unpredictable compared to what you'd expect in Mozart
"Complex interaction of variegated melodic-harmonic components": more varied harmony (as noted in the points above), combined with melodic/contrapuntal driving forces that are also more varied than the typical Schenkerian reductive patterns that are often used to analyze romantic period music

(I use "Mozart" above as just an example of traditional "tonal" music that usually has very clearly defined functional harmony, with very predictable progressions and phrases, etc.)
Answer: The Main Characteristics of Romantic Music
https://www.rpfuller.com/gcse/music/romantic.html

Freedom of form and design. It was more personal and emotional.
Song-like melodies (lyrical), as well as many chromatic harmonies and
discords.
Dramatic contrasts of dynamics and pitch.
Big orchestras, due mainly to brass and the invention of the valve.
Wide variety of pieces (i.e. songs up to five hour Wagner operas)
Programme music (music that tells a story)
Shape was brought to work through the use of recurring themes.
Great technical virtuosity.
Nationalism (a reaction against German influence)
Postromantic (Blair Allen Johnston - the dissertation of your link)
Deformation and Structural Tension
Hyperdissonance: Definition and Initial Analytic Applications
Formalizing the Model: Tension Arcs
Exaggeration of Tonal Premises
Distortion of Tonal Premises
A Parenthesis: Neutralization of Tonal Premises

I don't fully understand what he's saying, in terms of chromatic expansion, elaborations of linear tonal syntax, exaggeration and fragmentation of tonal syntax and the superimposition of conventional functional tonal structures and intense chromatic and modal structures.

Let's assume that if someone will be able to find and read this dissertation he will know or understand what is meant by chromatic extension.*1)
So let's pick another example...
Blair quotes:
However, the Encyclopædia Britannica provides a
starting point:
[A] musical style typical of the last decades of the 19th century and first decades of the 20th century and characterized by exaggeration of certain elements of the musical Romanticism of the 19th century. Postromanticism exhibits extreme
largeness of scope and design, a mixture of various musical forms (e.g., opera
and symphony), and heightened contrapuntal complexity (i.e., a long or vast
array, or both, of simultaneous but independent musical lines or events). Often
Postromanticism also embraces vivid religious or mystical fervour, a sense of
longing, and a sense of the grim and the grotesque.
Exaggeration and fragmentation of tonal syntax
"Exaggeration" and "vast array…of simultaneous but independent musical lines
or events" are particularly telling. From them, I offer a more specific observation: if the
Romantic is characterized by chromatic expansion and the development of striking
elaborations of linear tonal syntax, then the Postromantic is characterized by exaggeration
and ultimately fragmentation of tonal syntax, and the juxtaposition or superimposition of
conventional functional tonal structures and intense chromatic and/or modal structures
that challenge and even deform the functional tonal basis. In my view, complex
interaction of variegated melodic-harmonic components is one source of the continuing
fascination Postromantic music holds.
A basic claim in the present document is that Rachmaninoff was a Postromantic
composer, not an anachronistic Romantic composer.
*1)
To describe the development of harmony from basic chords to diminished seventh chords, mediant, neapolitan 6th, Italian-, French- and German-Sixth until the Tristan chord would be a theme too vast to answer the chromatic extensions.
The best advice would be to listen to the music of Rachmaninoff and analyze his chords and progressions. (Just the intro of his Piano concerto will provide us a full range of interesting harmony!)


Variable instead of node name

Question: How I can use JavaScript variable instead of node name in the render to use in jsx? Something like this:
var First = React.createClass({
  render: function() {
    var nodeName = this.props.node; // this.props.node = 'Second';
    return <{nodeName}/>;
  }
});
var Second = React.creactClass({/*...*/});

Comment: Although, this smells a lot like a design smell to me. There's a good reason there isn't an easy way to do this.
Comment: It's likely not possible with JSX, but you aren't forced to use JSX to write in React.
Comment: Have you tried `return < dangerouslySetInnerHTML={{__html: nodeName}} />;`
Comment: I agree without JSX) Have this `React.createElement(eval(nodeName));` But I dont want use "eval"
Comment: I don't understand what this is trying to achieve. Why not just use a `{this.props.children}` construct and pass in the element you want as a child? This stinks of an XY Problem to me.
Comment: because I have a lot of elements of the `Second` type, and I do not know which will come `this.props.node` to the `First`
Answer: Just don't use JSX in this case.
var First = React.createClass({
  render: function() {
    var nodeName = this.props.node; // this.props.node = 'Second';
    return React.createElement(nodeName);
  }
});
var Second = React.createClass({/*...*/});

This is ultimately what JSX results in, when compiled into Vanilla JS.
Comment: `React.createElement` return node that I need, but this node does not call `Second`
Comment: What exactly are you expecting to happen? The above code does what you tried to do in the OP. By the way, if you need to pass props, you can do that as the second argument, and children either as an array in the third argument or just `3..nth` argument: `React.createElement(node, props, children)`.
Comment: its work only with "eval" - `return React.createElement(eval(nodeName));`
Comment: Can you try `console.log(nodeName)` right before the return statement. `eval` should never be necessary.

Variable instead of node name


Which test is the best I can use

Question: I want to find out whether two groups of people (with and without Autism) differ in their interpretation of ironic compliments and ironic criticisms.
To this end I did a study with (26?) persons without autism and (?) persons with autism. All participants  first had to read 10 stories, with a remark at the end that could be ironic or literal. After each story, 3 questions were asked about the story (Q1 = identification of the speakers meaning, Q2 = control question, Q3 = recognizing the speakers' emotion). So they had to fill in 30 questions.
For each question and group, I calculated ratio of correct answers to all answers from that question and group. Then I applied a Mann-Whitney U test to these ratios, separately for each question type (Q1, Q2, Q3), to see if there are differences between the 2 groups when interpreting all the Q1's, all the Q2's and all the Q3's. I am not sure if this is the correct way to analyze the data - should I do something completely different?
Comment: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
Comment: Hi Carlijn, I have edited your question quite heavily to make it more easy to see the most relevant information for potential answerers :-) I have also removed the tag "corpus-linguistics" because I don't think it is important for the answer and will deter people that are not specialist in that but could answer. Also, you did not ask about R here.
Could you please add information about the number of participants in each group, and whether all answers were answered? And try to check if you are still happy with the question in the new form (I will take the changes back otherwise).
Answer: I would do logistic regression on Q3 (which is the one you seem interested in). That would let you add covariates, which you probably should do (e.g. age, sex, education ...)

Which test is the best I can use


Nested CollectionView Cell did not appear

Question: I nested collectionView inside collectionView Cell like this:

inside first collectionView Cell, I put this code :
<code>class CaseCellView: UICollectionViewCell, UICollectionViewDelegate, UICollectionViewDataSource {
    @IBOutlet weak var lblHeader: UILabel!
    @IBOutlet weak var collectionViewItemList: UICollectionView!
    var sellingArray = ["1", "2", "3", "4", "5"]

    override func awakeFromNib(){
        super.awakeFromNib()
        let flowLayout = UICollectionViewFlowLayout()
        flowLayout.scrollDirection = .horizontal
        flowLayout.itemSize = CGSize(width: collectionViewItemList.frame.width, height: collectionViewItemList.frame.width)
        flowLayout.minimumLineSpacing = 5.0
        flowLayout.minimumInteritemSpacing = 5.0
        self.collectionViewItemList.collectionViewLayout = flowLayout
        self.collectionViewItemList?.delegate = self
        self.collectionViewItemList?.dataSource = self
        self.collectionViewItemList.reloadData()
    }

    func collectionView(_ collectionView: UICollectionView, numberOfItemsInSection section: Int) -> Int {
        return self.sellingArray.count
    }

    func collectionView(_ collectionView: UICollectionView, layout collectionViewLayout: UICollectionViewLayout, sizeForItemAt indexPath: IndexPath) -> CGSize {
        return CGSize(width: collectionViewItemList.frame.width, height: collectionViewItemList.frame.width)
    }

    func collectionView(_ collectionView: UICollectionView, layout collectionViewLayout: UICollectionViewLayout, insetForSectionAt section: Int) -> UIEdgeInsets {
        return UIEdgeInsetsMake(0, 0, 0, 0)
    }

    func collectionView(_ collectionView: UICollectionView, cellForItemAt indexPath: IndexPath) -> UICollectionViewCell {
        let cell1 = collectionView.dequeueReusableCell(withReuseIdentifier: "caseListCell", for: indexPath)as! CaseListCellView
        cell1.lblItemList.text = self.sellingArray[indexPath.item]
        cell1.lblItemList.textColor = .white
        cell1.backgroundColor = UIColor.blue

        return cell1

    }
}

class CaseListCellView: UICollectionViewCell {
    @IBOutlet weak var lblItemList: UILabel!

    override func awakeFromNib() {
        super.awakeFromNib()
    }
}
</code>
but when I run, the nested collectionView result did not appear inside the first collectionView, it just show first lblHeader from the first collectionView Cell.
How to correct my code to show nested collectionView Cell?
Answer: Try giving height width of collection cell sizeForItemAt under UICollectionViewDelegateFlowLayout. If it helps.
Comment: If you have a new question, please ask it by clicking the [Ask Question](https://stackoverflow.com/questions/ask) button. Include a link to this question if it helps provide context. - [From Review](/review/late-answers/34167193)


How to measure objects in different planes with OpenCv and a single camera

Question: I'm working on a project where I need to track markers placed in a person. The person will be walking on a treadmill. I will use a single camera for each side.
I already calibrated the cameras, but now I'm trying to understand how to solve a problem. The problem is: the person will be walking and consequently the plane of the marker will change a bit, e.g, the marker on the shoulder can get closer to the camera sometimes if the person moves a bit to the side. I can't measure distances using depth because I'm only using one camera for each side.
So, I discovered that Aruco markers could keep the same scale even if the plane changes (the distance between the camera and the object). But to use these markers I'll need to attach an Aruco marker to each marker that will be placed on the person's body. It doesn't seem to be a very "simple" solution. However, at this moment, this is the best solution that I'm thinking about.
Does anyone have another idea to overcome this problem?
Comment: what? of course you have to stick aruco markers on your subject... if you only have single cameras. those simple retroreflective balls for motion capture require a calibrated set of cameras for 3D localization of these balls.
Comment: I wrote this question to check if there was another alternative to solve this problem.
I'll use individual Aruco markers.
Answer: If the treadmill is always the same, you might use it as a "calibration object" to fix scale. Build (or find online) a 3d model for it, position it on the image in a 3d modeling tool (e.g. Blender), then work out, e.g., the position of its "walk plane" w.r.t. the camera.
Comment: the "walk plane" is not of interest though. the 3D positions of markers attached to a human _walking on the thing_ are.


ARIMA accuracy measures, rolling forecast

Question: Regarding ARIMA model selection and especially accuracy measures several questions came into my mind. To shortly summarize, in my understanding, after necessary transformations/differencing, p and q values of an ARIMA model can be estimated by minimizing AIC/AICc/BIC and after that, coefficients are calculated with least squares/max likelihood optimization ('fit' method in different frameworks). Hence, we got the model fully specified. Furthermore let's assume that residuals look like white noise, passes LB test, so that we have everything to say the selected model is the best (is it true?).
I see multiple options to check the model forecast accuracy:
Option 1: We can make train/test sets, after model fitting on train set we produce multi-step forecasts on horizon of test set (lets say h=10) and compare true values of test set and forecast MAPE/RMSE/etc.
Option 2: We can make train/test sets, after model fitting on train set we produce rolling forecasts on horizon of test set (lets say h=10) and compare true values of test set and forecast MAPE/RMSE/etc. By rolling forecast I mean first we make a one-step forecast (first value after train set), then adding that as an observation, refitting model (but no recalculate p,d,q) on 'new' train set (original train set + first forecast). Then make one-step forecast again, etc. (it is cross-validation if I am right).
My questions:

Is it 100% sure that model selected by minimizing AIC/AICc/BIC will provide the best forecasts (in terms of MAPE/RMSE/etc.)?

Why do not we select the best model by comparing MAPE/RMSE/etc. on forecasts on test set instead of minimizing AIC/AICc/BIC?

In case of option 1, differences will be definitely larger as forecast converges to the mean (for example) in case of a longer multi-step forecast. Then, why would we use Option 1 (accuracy estimation based on multi-step forecasts) instead of Option 2 (accuracy estimation based on rolling forecasts)? I think accuracy estimation based on multi-step forecasts is only required and useful if there is a demand for multi-step forecasts when using the model later. If we have to produce only one-step forecasts in the future, accuracy estimation based on rolling forecasts is more valid and interpretable. Am I right?

Sorry if some (or all) questions are dumb :/
Thank you very much!
Comment: For your 2nd bullet point, see ["AIC versus cross validation in time series: the small sample case"](https://stats.stackexchange.com/questions/139175).
Comment: start with reading this: https://otexts.com/fpp3/accuracy.html
Comment: Thanks. I went through fpp3 earlier and my Option 2 idea exactly comes from the page you have linked. Actually as I see it, simply, MAPE/RMSE/etc. measures are one kind of accuracy checks (otexts.com/fpp3/accuracy.html) while cross-validation is an other type (https://otexts.com/fpp3/tscv.html) of that. My concern is why the model-selection process (p, q) ignores cross-validation as an option and rely solely on minimalization of AIC/AICc/BIC.
Comment: https://robjhyndman.com/hyndsight/aic/ tells that: "[..] asymptotically, minimizing the AIC is equivalent to minimizing the leave-one-out cross-validation MSE for cross-sectional data, and equivalent to minimizing the out-of-sample one-step forecast MSE for time series models. This property is what makes it such an attractive criterion for use in selecting models for forecasting."

For 2nd bullet point, as Richard Hardy advised, thanks.
Answer: If I understood your main question it is about why not use cross-validation performance metrics for model selection? The short answer is that by doing so you would erase the distinction between CV and in-sample model fit metrics. Yet, effectively, many practitioners do exactly that.
The idea of CV is that you try to assess the model performance when it is applied to data that was not used to train it. The problem is that in reality you got the full data set, then you divide it into train and test. So, you actually saw all data already. we play a game of pica boo: pretend that we didn't see the test set and use it to emulate out of sample performance. However, some practitioners automate the whole process to such a degree, that they run in-sample training, then CV, get the metrics then rinse and repeat to hundreds of model speicifcations to select the best one. When they do this, what is the real distinction between CV and in-sample? None, effectively. So, you basically are left with in-sample fit.
Now, the nuance here is that with time-series models such as ARIMA, what we call in-sample metrics, such as AIC, aren't really in-sample: they're obtained from one-step ahead predictions, but it's a minor technicality for the discussion we have here


Dovecot SSL - IMAP works, SMTP fails

Question: I'm setting up a small mail server using postfix+dovecot. When running oppenssl s_client -connect mail.myserver.com:993 (IMAP), the server's certificate is displayed and I'm able to get an email client to connect fine.
However, when I change the port to 25 (SMTP), I get the error
SSL routines:ssl3_get_record:wrong version number, and no certificate is returned.
From my research, it seems that the protocols and ciphers allowed in the 10-ssl.conf file might be the issue, but I've tried so many combinations and I still get the same error. This is what they are at the moment:
ssl_protocols = TLSv1.2 TLSv1.1 TLSv1 !SSLv3 !SSLv2
ssl_cipher_list = HIGH:!SSLv2:!SSLv3:!TLSv1.0:!aNULL:!MD5

How can I get the server to return the certificate on the SMTP port?
Comment: dovecot does not do SMTP, that's the job of the mail transport agent (e.g. postfix, exim, ...). What MTA are you using?
Answer: Your Postfix + Dovecot setup has two different server software performing different roles:

Dovecot in your configuratoin is a mail storage server, responsible for IMAP. Additionally, Dovecot provides SASL authentication for Postfix. The configuration file /etc/dovecot/conf.d/10-ssl.conf is for Dovecot alone, and therefore it only affects IMAP with STARTTLS on port 143 and IMAPS on port 993.
Postfix is the mail delivery agent (MTA) responsible for SMTP, whether it's for other MTAs on TCP port 25, for message submission with STARTTLS on port 587 or SMTPS on port 465. The main configuration files for Postfix are main.cf and master.cf under /etc/postfix/.

You can only test IMAPS 993 & SMTPS 465 using oppenssl s_client -connect mail.example.com:465. For testing STARTTLS on SMTP ports 25 & 587 and IMAP port 143 you need to use:
openssl s_client -starttls smtp -connect mail.example.com:25
openssl s_client -starttls smtp -connect mail.example.com:587
openssl s_client -starttls imap -connect mail.example.com:143

There's also several websites and tools for testing, discussed in a Security Stack Exchange question. For configuring TLS in Postfix, see Postfix TLS Support. The Postfix and Dovecot SASL documentation also has an example for configuring submission. In addition, you need to add TLS parameters to your main.cf.
Answer: Port 25 do not use SSL by default. Client have to run STARTTLS command to begin the negotiations with server about cryptography. In opposite IMAPS on the port 993 begins SSL negotiations immediately on connect. You have to configure and try the SMTPS on the port 465 that acts exactly like IMAPS on 993.
Comment: Please note SMTPS on port 465 is considered deprecated
Answer: To test services, such as SMTP, that run both the clear text protocol as well as an TLS encrypted version of that protocol on the same TCP port, you need to instruct openssl to negotiate the TLS protocol upgrade with STARTTLS  and the -starttls protocol switch.
Depending on the openssl version, s_client supports a number of different protocols:

-starttls protocol
    send the protocol-specific message(s) to switch to TLS for communication.  protocol is a keyword for the intended protocol.  Currently, the only supported keywords are "smtp", "pop3", "imap",
             "ftp", "xmpp", "xmpp-server", "irc", "postgres", "lmtp", "nntp", "sieve" and "ldap".

So test your Postfix mail server with:
oppenssl s_client -starttls smtp -connect mail.example.com:25

Dovecot SSL - IMAP works, SMTP fails


Visualizing shared instances of p-values<alpha across large numbers of treatments

Question: Assume a data table that presents the p-values of a large number of independent runs of a statistical hypothesis test. Each run represents a single test with two possible hypotheses (i.e., null and alternative) on a particular dataset (or treatment or whatever you want to call it). A total of d different datasets (represented in the table as rows) and h different hypotheses (represented as columns) were evaluated to generate the table. The data table, thus, portrays the p-values of d x h different runs.
     |   h1    h2    h3    h4
----------------------------
d001 | 0.02* 0.33  0.01* 0.46
d002 | 0.14  0.25  0.03* 0.11
d003 | 0.01* 0.68  0.01* 0.04*
...
d998 | 0.02* 0.71  0.01* 0.13
d999 | 0.03* 0.29  0.02* 0.33

Since the number of datasets evaluated ranges in the hundreds, I would like to find a way to visualize shared instances of significance graphically. Specifically, I would like to highlight if any of the hypotheses (i.e., columns) share instances of p-values<alpha with other such hypotheses for the same datasets (i.e., rows). In the above example, hypotheses h1 and h3 share instances of p-values<alpha across numerous datasets.
What general type of visualization would you recommend? (I'll set up the R code myself and am just interested in the type of visualization that you would recommend.)
Comment: As a preliminary step, did you correct for multiple hypotheses? A raw p-value of 0.05 is not nearly significant when conducting thousands of hypothesis tests.
Comment: @NuclearHoagie Yes, a Bonferroni correction of the p-values is in place.
Answer: Ultimately, you want a biplot.  Briefly, a biplot rerepresents your units / rows as points in a 2-dimensional space that maximizes the amount of the original variability that can be captured.  (These are the first two principal components; for more, see: Making sense of principal component analysis, eigenvectors & eigenvalues.)  It likewise rerepresents your variables / columns as the tips of arrows in the same space on the scatterplot.  In general, the closer two points are, the more similar they are; the closer the tips of two arrows are, the more similar the variables are; and the closer a point is to a variable, the larger the proportion of its mass is associated with the variable, and vice-versa.  Be aware that a high-dimensional space is being collapsed into 2-dimensions, so there will be some error there, but that's the gist of it.  There is a lot of good information about biplots available on the site, click the tag, sort by votes and start reading from the top to learn more.
I strongly believe it is best to use the full information in data and not dichotomize (e.g., into significant vs. non-significant).  Here is a biplot of your example dataset of p-values, coded in R:
d = read.table(text="dataset    h1    h2    h3    h4
                        d001  0.02  0.33  0.01  0.46
                     ...
                        d999  0.03  0.29  0.02  0.33", header=T)
rownames(d) = d$dataset
d           = d[,-1]
d
#        h1   h2   h3   h4
# d001 0.02 0.33 0.01 0.46
# d002 0.14 0.25 0.03 0.11
# d003 0.01 0.68 0.01 0.04
# d998 0.02 0.71 0.01 0.13
# d999 0.03 0.29 0.02 0.33

windows()
  biplot(prcomp(d, scale=T))

We can see that h1 is similar to h3, d003 is similar to d998 in pattern, and that d998 has most of its mass in h2.

If you really want to know only about significance, you can conduct a correspondence analysis.  That also outputs a biplot, although by convention usually with points for both rows and columns, and the table is decomposed using a different metric than correlations (which PCA uses).  We also have several good threads on correspondence-analysis.
d2           = as.data.frame(sapply(d, function(j){  as.numeric(j<.05)  }))
rownames(d2) = rownames(d)
#      h1 h2 h3 h4
# d001  1  0  1  0
# d002  0  0  1  0
# d003  1  0  1  1
# d998  1  0  1  0
# d999  1  0  1  0
library(ca)
windows()
  plot(ca(d))

We get generally similar information here.
Comment: Thank you for taking the time to draft such a detailed answer! I will read up on biplots and correspondence analysis given your links. Much appreciated!
Comment: You're welcome, @MichaelG.


Some computers in my subnet do not recognize names of other pc's

Question: I have two subnets:

192.168.0.1-255
192.168.50.1-255

Ewerything works via VPN (because physically computer from both subnets are in different locations). We also have active directory. Now, the first subnet has all the main servers etc. Computers that are located in this subnet work just fine (i.e. they see all the pc's in the both subnets).
The problem is with second subnet. Some of the pc's (as far we didn't recognize any pattern here) have problem with recognizing pc's from first subnet by name (recognizing by ip works fine). There is totally no pattern here - we saw examples of pc's, that do recognize everything just after simple relog, but then after another relog it didn't work again.
Is it a problem with our domain/active directory? Or our DNS server? What can cause problems here?
Comment: Are they having problems with NetBIOS names (no domain), e.g. \\FileServ5, or with DNS names, e.g. "\\fileserv5.contoso.com", or both? (I would expect NetBIOS names to fail as they rely on broadcasts which should not be going between the subnets.)
Comment: @GreenstoneWalker: Yes, it's all about NetBIOS names.
Comment: @user218133 Yeah, don't use NETBIOS where you can possibly avoid it... and since you've gotten an answer below that enables you to avoid it, you should do so. :)
Answer: Configure the DNS suffix correctly on all clients and servers.
The easy way of doing this is through Group Policy

You should also configure it on your DHCP server(s) if you use DHCP, as not all clients are domain-joined in all environments. You do this by configuring DHCP option 015 and 119.
For statically adressed clients/servers (not served by DHCP) you can set this manually in the advanced TCP/IP settings on each connection
Comment: This is completely unnecessary for domain joined computers.


Running ANCOVA with categorical covariates - how to set up dataset?

Question: I am analyzing a dataset in which 80 participants rated 4 products (product A, product B, product C, product D) from -3 to 3 depending on how much they liked the product.  Each participant provided categorical demographic information such as gender, ethnicity, age group, and employment status.
I would like to run an ANCOVA with the product type as the IV, the rating number as the DV, and all of the demographic variables as categorical covariates.
Should my dataset have 320 rows with the following columns: Product, Rating, Gender, Ethnicity, AgeGroup, EmploymentStatus?  Each participant would have 4 rows such that there is a row for their product A, B, C, and D rating. I would repeat their demographic data for each row.  Or should I set up my data differently?
Additionally, I plan to use SPSS and dummy code each of the categorical covariates with C-1 groups and input these categorical covariates in the "Covariate" box and not the "Fixed Factors" box in the Analyze - Univariate - GLM tool. Would this be the right way to go about this?
Thank you kindly for any advice!
Comment: What is your research question ? Are your covariates balanced ?
Comment: Thank you for your time, Robert. My research question is - is there a difference in the mean ratings for each product (i.e. do people prefer for any of the products more than the other products). Regarding balanced covariates - I'm not exactly sure what this means, but each person provided ratings for all 4 products and there is very little missing data. The groups within each demographic categorical covariate are not equal (e.g. for ethnicity, there were more white participants than asian participants). Hope this helps clarify.
Answer: The proposed model is not an ANCOVA. ANCOVA is a model with a continuous outcome, a categorical independent variable of primary interest (main exposure), and one of more continous variables that are potential confounders or competing exposures.
The distinction isn't really important because it's just another (multivariable) regression model. So the model would look like:
<code>rating ~ product + covariates
</code>
and you would be interested in the estimates for the different levels of product, while controlling for the covariates. However I see two problems with this model. First, is it reasonable to treat rating as continuous when it appears to be ordinal with 7 levels? I would suggest proceding with it as continuous but also compare it with an ordinal model. But the main issue I see is that you have repeated measures within participants, so the ratings for one participant are more likely to be similar to each other than to those of other partipants. That is, the observations are not independent. One way to handle that is to fit a mixed effects model with random intercepts for participant.
As for the question about how to set up the data, yes, 320 with each row corresponding to one rating would be the way to go, with most software that I am aware of.


Find the energy eigenvalue from a given wave function

Question: A particle of mass $m$ is trapped in an infinitely deep one-dimensional potential well between $x = 0$ and $x = a$, and at a time $t = 0$ it is described by the wave function
$$ w(x,t=0) = \frac{1}{\sqrt{2}} \sin \left(\frac{\pi x}{a} \right) \cos\left(\frac{2\pi x}{a} \right). $$
How would I go about solving this with Schrodingers equation?
Comment: Please use MathJax and include your efforts so far.
Answer: You need to start with the eigenvalues and eigenvectors of the Schrodinger's equation. Suppose that you have calculated them, and called them $E_n$ and $\Psi_n$. In general, any solution can be written as a linear combination of the eigenvectors, so your $\Psi$ function is $$\Psi(x,0)=\frac{1}{\sqrt{2}}\sin\left(\frac{\pi x}{a}\right)\cos\left(\frac{2\pi x}{a}\right)=\sum_{n=1}^\infty c_n \Psi_n$$
To get the values of the $c_n$ coefficients, $$c_n=\int_0^a\Psi_n^*(x,0)\Psi(x,0) dx$$
The value of the energy is given by:
$$\begin{align}<E>&=\int_0^a\Psi^*(x,0)H\Psi(x,0)dx\\
&=\int_0^a\Psi^*(x,0)H\left(\sum_{n=1}^\infty c_n \Psi_n\right)dx
\\&=\int_0^a\Psi^*(x,0)\left(\sum_{n=1}^\infty c_n E_n \Psi_n\right)dx\\&=\int_0^a\left(\sum_{m=1}^\infty c_m^* \Psi_m^*\right)\left(\sum_{n=1}^\infty c_n E_n \Psi_n\right)dx\\
&=\sum_{n=1}^\infty |c_n|^2 E_n\end{align}$$
To get the last step, I used the fact that the $\Psi_n$ form an orthonormal set


VPN and Remote Desktop Stopped Working (Windows Server 2008 R2)

Question: Recently, both the VPN and remote desktop stopped working.  See below for details.
VPN:

Error 800: The remote connection was not made because the attempted VPN tunnels failed. The VPN server might be unreachable. If this connection is attempting to use an...

Remote Desktop:

Still accessible from within LAN
Gets stuck on Configuring remote session... from outside the network

From what I've gathered, every other service is working perfectly.  All ports appear to be forwarding properly, and I can access the various web interfaces that are served via that server.  This issue seems to be centered around the VPN and remote desktop feature.  I have tried restarting different services, and did a physical reboot.
Any ideas, links, etc., would be greatly appreciated.  Thanks in advance!
EDIT: The Remote Desktop functionality works when VPN is disabled.  The utter randomness of this is frustrating!
Comment: Barring any firewall changes, I would reboot your firewall and/or router.
Comment: Hi joeqwerty, No firewall changes in Windows Server, or the router.  Already reboot router and server itself.
Answer: You might check to see if any recent firewall changes have taken place.
Comment: What happens when you telnet to the RDP port from outside your network?
Comment: Good call. Telnet seems to indicate it is getting through (no connect failed message or anything like that).
Comment: But what is actually being displayed?
Comment: Your standard blinking cursor
Comment: Then something else is probably intercepting the ssh session, you might compare it to a session from within your network.
Comment: Works fine from within the network. How can I get more information on what is happening inside vs. outside?
Comment: Map out the equipment that is between the two pathing configurations and check at the additional points/equipment. It's probably your router, firewall or ISP.


Choosing best modelling practices

Question: I am modelling some behavioural data in R to determine the best parameters for explaining foraging success of marine animals. I have both fine scale parameters relating to the direct individual (e.g. dive depth, trip length) and broad-scale parameters (e.g. temperature, wind, chlorophyll a).
To avoid overfitting, I currently have 2 models, one modelling fine-scale and one modelling broad-scale predictor variables and I am using model averaging to determine the best set of predictors for each model.
My question is: Can I use the broad and the fine-scale models as initial testing of the best parameters to include, and then put all the best parameters together in a single model? I've tried this out and it provides a good model with a higher predictive capacity but would this be considered bad practice? I haven't seen this done in other work (and certainly never done it myself).
Thanks!
Answer: This method of two separate models to fight overfitting is unknown to me, and I also don't really see how it should succeed. But since you have good results, you should give it a try.
Nevertheless, I would have started with a model that contains all parameters (fine and large scale). Then, there are many methods for both selecting the appropriate parameters (random forest variable importance, ARD (e.g. relevance vector machines), spike and slab, and many more, e.g. here and here), and for fighting overfitting (e.g. regularization, Bayesian methods, or appropriate information criteria) that can be applied to this model.

Choosing best modelling practices


Causal modeling in the presence of a latent variable

Question: Suppose that four variables of $X$, $Y$, $L$, and $C$ have the following relationships in the form of directed acyclic graph.

$X$, $Y$, and $C$ are observable variables while $L$ is a latent (unobservable) variable. If one is interested in modeling the relationship between $X$ and $Y$ through, for example, regression, is it reasonable to include $C$ as a covariate and condition the $X$-$Y$ association on $C$? Alternatively, how about when one is interested in the association between $C$ and $Y$?
Answer: In your first scenario, you certainly can include $C$ on the RHS of your linear regression model, but you shouldn't need to if your DAG is correct. If your DAG is correct, then $C$ does not cause $Y$ in any way, and is not a confounder (does not set up a back door path). All of the causal effect of $X$ on $Y$ is mediated through $L:$ that's fine. There's nothing special you need to do, just regress $Y$ on $X.$
In your second scenario - finding the association between $C$ and $Y$ - you have a problem: $L$ is a confounding variable. Moreover, if you have no other variables present anywhere, then I don't see a way to estimate the causal effect of $C$ on $Y.$ $X$ is not an instrument, because if it were, it would have be pointing into $C$ and nothing else. You're not even really sure that $C$ has a causal effect on $Y.$ Technically, of course, your DAG says it doesn't have a causal effect on $Y.$ I'd say you're stuck, here.
Comment: (1) It shouldn't hurt if $C$ is included in the first model. It shouldn't change anything! (2) If the paths you specify are added to the graph, then the only change to the answer to the first scenario is that it might be a bad idea to include $C$. You might get a collider bias, even though $C$ isn't a confounder. For the second scenario, there's still no causal effect of $C$ on $Y,$ but if you wanted to regress anyway, you should include $X$ because it is a backdoor path.
Comment: Thanks! (1) For the first question, I understand that C does not have to be in the model. Would it hurt if C is included? (2) If two more paths are added in the graph (one from X to C and another from X to Y), would that change the answers for the two questions?
Answer: In the first scenario, C is a descendant of a mediator, so contrary to previous answers, yes it would be harmful to include if your interest is in estimating the total effect of X on Y. See DAG 12 here:
http://causality.cs.ucla.edu/blog/index.php/2019/08/14/a-crash-course-in-good-and-bad-control/
Agree with earlier answers that there's no available strategy to identify the effect of C on Y given this DAG is correct.
Comment: Thanks for sharing the material. Will read it soon!


Show file in browser C#

Question: I have this code:
   public void OpenFile(string fileName)
        {
            var url = Path.Combine(Server.MapPath("~/App_Data/uploads"), fileName);
            using (var fileStream =  new FileStream(url, FileMode.Open))
            {
                byte[] bytes = new byte[fileStream.Length];
                int numBytesToRead = (int)fileStream.Length;
                int numBytesRead = 0;
                fileStream.Read(bytes,numBytesRead, numBytesToRead);
            }
        }

That code is working fine, but I want to show that file in the browser, I'm executing this method on click in the name of the file, parameters are working great, which is the other code that I need to put to show the file in the browser? Mostly the files are going to be .doc and .pdf. How can I show the documents in the browser??
Comment: http://stackoverflow.com/questions/3604562/download-file-of-any-type-in-asp-net-mvc-using-fileresult
Answer: You can return a FileStreamResult from your action method. Change the method return type from void to ActionResult
public ActionResult OpenFile(string fileName)
{
   var pathToTheFile = Path.Combine(Server.MapPath("~/App_Data/uploads"), fileName);
   var fileStream = new FileStream(pathToTheFile,
                                    FileMode.Open,
                                    FileAccess.Read
                                );
   return  new FileStreamResult(fileStream,  MimeMapping.GetMimeMapping(fileName));
}

Show file in browser C#