Using .htaccess to hide the location and extension of .php file

Question: I'm trying to have all requests going to //my.domain/download?id=xx (be it http or https) be processed by /path/to/my/download.php?id=xx.
I've tried all sorts of RewriteRule rules to no avail, so there's no point in me posting anything I have, but this is a Wordpress install, so this shouldn't conflict with WP's existing rules, which are:
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

I would very much appreciate an explanation for any provided answer. Thank you!
EDIT
I already took a look at Redirect, Change URLs or Redirect HTTP to HTTPS in Apache - Everything You Ever Wanted to Know About Mod_Rewrite Rules but Were Afraid to Ask and can't seem to even pinpoint what section would cover this situation, as all that is too general. I'm looking for help wiht this specific situation.

EDIT 2
This is the code that is failing, with a 404, even if https://htaccess.madewithlove.be/ sais it shoudl all be good:
RewriteEngine on
RewriteCond %{REQUEST_URI} !^download
RewriteRule ^download(.*)$ wp-content/themes/XXX/inc/download.php$1

EDIT 3
I was having unrelated issues. For anyone else interested, see my answer below for the rules that worked.
Comment: Possible duplicate of [Redirect, Change URLs or Redirect HTTP to HTTPS in Apache - Everything You Ever Wanted to Know About Mod\_Rewrite Rules but Were Afraid to Ask](https://serverfault.com/questions/214512/redirect-change-urls-or-redirect-http-to-https-in-apache-everything-you-ever)
Answer:
I'm trying to have all requests going to //my.domain/download?id=xx (be it http or https) be processed by /path/to/my/download.php?id=xx.

For this you probably want to internally rewrite the URL rather than externally redirect the request, as in your existing answer. An internal rewrite won't change the URL in the browser's address bar.
The directives in your existing answer can also be simplified.
Try the following instead, before the existing WordPress directives, in order to rewrite any request for /download?id=N to /path/to/my/download.php?id=N, where N is 1 or more digits.
RewriteCond %{QUERY_STRING} ^id=\d+$
RewriteRule ^download$ path/to/my/download.php [L]

\d is a shorthand character class which matches any digit. The same as [0-9]. The + quantifier matches 1 or more digits, so this will fail (404) if id= (with no value) is included in the query string. Change back to * if id= should be successful. There is no need to capture this value (ie. surround the pattern in parentheses as in ([0-9]*) since you are not using this in the substitution.
The query string, ie. id=N is copied through to the substitution by default.
There is no need to repeat the RewriteEngine On directive, since this is already part of the WordPress code block.
Comment: This worked, though i had to change to `RewriteCond %{QUERY_STRING} ^id=\d+(&direct)?$` since i may sometimes have the `direct` query variable aswell. Excellent! That being said, I think you may be right on the money about me actually needing to `internally rewrite` instead of redirect, because I'm also using `mod_xsendfile`, and I have to change to `RewriteRule ^download$ path/to/my/download.php [L,R]` for it to work( as per https://stackoverflow.com/questions/20705731/mod-xsendfile-wont-work-with-cgi-and-mod-rewrite ), which exposes the actual location of my php file. How would I do that?
Answer: This is the code that got what was asked working:
RewriteEngine On
RewriteCond %{REQUEST_URI}  ^/download$
RewriteCond %{QUERY_STRING} ^id=([0-9]*)$
RewriteRule ^download(.*)$ path/to/my/download.php$1 [L,R]

Comment: You probably want a "rewrite", rather than an external "redirect"? But these directives can also be simplified... The first `RewriteCond` directive is superfluous, since you are simply repeating the check already performed by the `RewriteRule` _pattern_. The capturing group `(.*)` and corresponding backreference `$1` are also not required. I've added an answer showing these changes with more explanation.


How to get Taylor Series of $\sin \frac{x}{1-x}$

Question: I know that $\displaystyle\sin x = \sum_{k=0}^{\infty}\frac{\left( -1 \right) ^kx^{2k+1}}{(2k+1)!}$
But how to get transformation to get Series about x?
Comment: substitute $x=\frac{m}{1-m}$  into the series. That's all you need to do!
Comment: That is "all" you need to do, you say.  Indeed, you can get as many terms as you like that way.  But can you get a closed form for the terms?
Answer: comment
From Maple, I get
$$\sin\frac{x}{1-x} =
x+{x}^{2}+{\frac {5}{6}}{x}^{3}+{\frac {1}{2}}{x}^{4}+{\frac {1}{120}
}{x}^{5}-{\frac {5}{8}}{x}^{6}-{\frac {6931}{5040}}{x}^{7}-{\frac {
1591}{720}}{x}^{8}-{\frac {224179}{72576}}{x}^{9}-{\frac {31987}{8064}
}{x}^{10}-{\frac {27323293}{5702400}}{x}^{11}-{\frac {19986991}{
3628800}}{x}^{12}-{\frac {1508675351}{249080832}}{x}^{13}-{\frac {
3055361231}{479001600}}{x}^{14}-{\frac {8385570334051}{1307674368000}}
{x}^{15}-{\frac {16115825351}{2641766400}}{x}^{16}-{\frac {
1916072623603199}{355687428096000}}{x}^{17}-{\frac {88344037902079}{
20922789888000}}{x}^{18}-{\frac {12475149135090259}{4865804016353280}}
{x}^{19}+O \left( {x}^{20} \right)
$$
So, can we "recognize" these terms to guess a formula?  Can we even say that all subsequent coefficients are negative?


Can we apply here the Cayley–Hamilton theorem?

Question: We have the matrix \begin{equation*}A:=\begin{pmatrix}3 & 1 & 0 & -1& -1 \\ 0 & 2 & 0 & 0 & 0 \\ 1 & 0 & 2 & 0 & -1 \\ 0 & 0 & 0 & 2 & 0 \\ 1 & 0 & 0 & -1 & 1\end{pmatrix}\in M_5(\mathbb{R})\end{equation*}
The characteric polynomial is \begin{equation*}P_A(\lambda)=(2-\lambda)^5\end{equation*}
The eigenvalue $\lambda=2$ has the algebraic multiplicity $5$.
The eigenspace is \begin{equation*}\left \{\begin{pmatrix}e\\ 0\\ c\\ 0\\  e\end{pmatrix}:  c, e\in \mathbb{R}\right \}=\left \{e\begin{pmatrix}1\\ 0\\ 0\\ 0\\  1\end{pmatrix}+c\begin{pmatrix}0\\ 0\\ 1\\ 0\\  0\end{pmatrix}:  c, e\in \mathbb{R}\right \}\end{equation*}
So the geometric multiplicity of the eigenvalue $\lambda=2$ is $2$.
How can we calculate $(A − 2I_5)^3$ ?
Can we apply here the Cayley–Hamilton theorem?
Comment: What is $u_5$? Is it one of the standard basis vector in $M_5(\Bbb{R})$?
Comment: Oh I meant the unit matrix, I corrected it. It should be $(A − 2I_5)^3$  @AnuragA
Comment: The Cayley-Hamilton theorem tells you that $(A-2I)^5=0$; it's not difficult to show a matrix $B$ having the single eigenvalue $2$ with algebraic multiplicity $5$ and geometric multiplicity $2$, but $B^3\ne0$. In the given case, it does happen that $(A-2I)^3=0$, but not because of CH.
Comment: Why does this hold? I got stuck right now. @egreg
Comment: @MaryStar "This" what? The fact that $(A-2I)^3=0$? Because it does *for this particular matrix*. You cannot in general deduce that $(B-2I)^3=0$ from the information that the eigenvalue is unique and the eigenspace has dimension $2$.
Comment: @egreg we can deduce this kind of case by using minimal polynomial! And there can be infinitely many case of this kind, and you can create them also, for example any n×n diagonalizable matrix have only one eigenvalue a with A.M. n then clearly it is shown that (A-aI)=0 while characteristics polynomial is (A-aI)ⁿ=0.
Comment: @Subhajit Yes, of course, but computing the minimal polynomial cannot be done with CH. For the given matrix the minimal polynomial is indeed $(x-2)^3$, but you cannot deduce it just from the eigenspace.
Comment: And also it can be (x-2)⁴!
Comment: Why not? Using Jordon canonical form, it can be done,
Comment: Also along with the G.M. of eigenvalue together with AM.
Comment: And you can see it what I have done! Not only for this matrix, take any arbitrary one , and use this rule, then what will happen you can see it by yourself?
Comment: @egreg is this rule still having doubt! Please tell me , if you get something different.
Answer: Here, dimension of eigenspace corresponding to the eigenvalue 2 is 2, There are two Jordan canonical forms  possible! So, in one Jordan canonical form, there can be two blocks possible which are of order 3 and order 2 and in another of Jordon canonical form, there can also be possiblity for two blocks of order 4 and 1,
But through the first case max order of block is 3, hence there be possiblity of minimal polynomial of order 3, hence we can use Cayley Hamilton's theorem!                                        For the first case ,Jordon canonical form, $$
    \begin{pmatrix}
    2 & 1 & 0 & 0 & 0 \\
    0 & 2 & 1 & 0 & 0 \\
    0 & 0 & 2 & 0 & 0 \\
    0 & 0 & 0 & 2 & 1 \\
    0 & 0 & 0 & 0 & 2 \\
    \end{pmatrix}
$$


Properties of equality

Question:
Why is this relation transitive? $R = \{(x,y)\in\Bbb N^2\mid x + 4y = 10\}$
Comment: So you have the linear equation $y=-\frac14 x + \frac52$.  Maybe you can try visualizing it from there?
Comment: a cursory trial shows it's not reflexive.
Comment: The question was presented like this in the textbooks.. and then everything else is fine. I didn't understand how it is transitive.
Comment: The point is whenever you can find a $y$ such that $xRy$ and $yRz$, then all $x, y,z$ are forced to equal to $2$, so $xRz$ is also true.
Answer: Assume x + 4y = 10 and y + 4z = 10.  Does x + 4z = 10?
Exercise.  Prove for x,y in N, x + 4y = 10, the solutions are
(2,2), (6,1).
Thus if aRb and bRc, then b = 2.
Whence a = b = c = 2, so aRc.
Comment: N is collection of Natural Numbers.. 0 does not exist in N..  So just because it is not having a (y,z) pair.. can lead a relation to be accepted as transitive?
Comment: @zion.  Nonsequiture.  See edit
Comment: Thanks understood...

Properties of equality


Write an application that inputs three integers from the user and displays the sum, average, product, smallest and largest of the numbers

Question: I'm getting the sum, average and the product. The real difficulty I am facing is with the smallest and the largest number.
I can do it with two numbers, but three numbers is not making any sense to me. Ask me if my question isn't clear or if its not making sense.
import java.util.Scanner;

// exercise 2.17
public class ArithmeticSmallestLargest {

    public static void main(String[] args) {

        Scanner input = new Scanner(System.in);

        int num1;
        int num2;
        int num3;
        int sum;
        int average;
        int product;
        double largest

        System.out.print("Enter First Integer: ");
        num1 = input.nextInt();

        System.out.print("Enter Second Integer: ");
        num2 = input.nextInt();

        System.out.print("Enter Third Integer: ");
        num3 = input.nextInt();

        sum = num1 + num2 + num3;
        average = sum / 3;
        product = num1 * num2 * num3;

        if (largest =num1 > num2 & num2 > num3)

        System.out.println(sum);

        System.out.println(average);

        System.out.println(product);

        System.out.println("The biggest number is " + largest);
    }
}

Comment: Work on your logic to compute the largest number.
Comment: I think you would do well to read this guideline: http://meta.stackexchange.com/questions/10811/how-do-i-ask-and-answer-homework-questions
Comment: This will be closed unless you show some real effort and be clear about the actual problem that you're facing.
Comment: @ShairyarShafqat - please do provide the code you used for getting largest/smallest number for two inputs. Then we can lead you to the solution.
Comment: What's so weird about finding the largest and the smallest number out of a set of three? Your exercise simply asks you to *sort* instead of just comparing two numbers.
Comment: it is not a home work question, i am trying to learn java on my own and thought i could get some help from experienced programmers. P.S i am using the JH2P, 9th edition.
Comment: @ShairyarShafqat - please provide the code you used for finding the larges/smallest of two inputs. (you can edit your question to add more info). Then we can lead you to the solution for three.
It's not that hard - if you can compare two, why can't you compare three?
Comment: @MichałSchielmann thanks a lot buddy, I actually got it myself. Perseverance prevails. if i have any questions in the future, i will be more elaborate. Appreciate your help.
Answer: import java.util.Scanner;

// exercise 2.17
public class ArithmeticSmallestLargest {

    public static void main(String[] args) {

        Scanner input = new Scanner(System.in);

        int num1;
        int num2;
        int num3;
        int sum;
        int average;
        int product;
        int largest;
        int smallest;

        System.out.print("Enter First Integer: ");
        num1 = input.nextInt();

        System.out.print("Enter Second Integer: ");
        num2 = input.nextInt();

        System.out.print("Enter Third Integer: ");
        num3 = input.nextInt();

        sum = num1 + num2 + num3;
        average = sum / 3;
        product = num1 * num2 * num3;

        largest = num1;
        smallest = num1;

        if(num2 > largest)
            largest = num2;

        if(num3 > largest)
            largest = num3;

        if(num2 < smallest)
            smallest = num2;

        if (num3 < smallest)
            smallest = num3;

        System.out.println("The sum is " + sum);

        System.out.println("The average is " + average);

        System.out.println("The product is " + product);

        System.out.println("Largest of three integers is " + largest + " and the smallest is "+ smallest + ".");
    }
}

Comment: good for you! Don't forget to accept your answer if it solves the problem ;)
Answer: import java.util.Scanner;

// exercise 2.17
public class ArithmeticSmallestLargest {

    public static void main(String[] args) {

        Scanner input = new Scanner(System.in);

        int num1;
        int num2;
        int num3;
        int sum;
        int average;
        int product;
        int largest;
        int smallest;

        System.out.print("Enter First Integer: ");
        num1 = input.nextInt();

        System.out.print("Enter Second Integer: ");
        num2 = input.nextInt();

        System.out.print("Enter Third Integer: ");
        num3 = input.nextInt();

        sum = num1 + num2 + num3;
        average = sum / 3;
        product = num1 * num2 * num3;

        largest = num1;
        smallest = num1;

        if(num2 > largest)
            largest = num2;

        if(num3 > largest)
            largest = num3;

        if(num2 < smallest)
            smallest = num2;

        if (num3 < smallest)
            smallest = num3;

        System.out.println("The sum is " + sum);

        System.out.println("The average is " + average);

        System.out.println("The product is " + product);

        System.out.println("Largest of three integers is " + largest + " and the smallest is "+ smallest + ".");
    }
}

Answer: import java.util.Scanner;

public class ArithmeticSmallestLargest {

    public static void main(String[] args) {

        Scanner input = new Scanner(System.in);

        int num1;
        int num2;
        int num3;
        int sum;
        int average;
        int product;
        int largest;
        int smallest;

        System.out.print("Enter First Integer: ");

        num1 = input.nextInt();

        System.out.print("Enter Second Integer: ");
        num2 = input.nextInt();

        System.out.print("Enter Third Integer: ");
        num3 = input.nextInt();

        sum = num1 + num2 + num3;
        average = sum / 3;
        product = num1 * num2 * num3;
        largest = num1;
        smallest = num1;

        if(num2 > largest)
            largest = num2;

        if(num3 > largest)
            largest = num3;

        if(num2 < smallest)
            smallest = num2;

        if (num3 < smallest)
            smallest = num3;

        System.out.printf("The sum is %d%n " ,  sum);

        System.out.printf("The average is %d%n " ,  average);

        System.out.printf("The product is %d%n " , product);

        System.out.printf("Largest of three integers is %d%n " , largest);

        System.out.printf("Smallest of three integers is %d%n " , smallest);
    }
}

Answer is given below:
Enter First Integer: 60
Enter Second Integer: 90
Enter Third Integer: 30
The sum is 180
 The average is 60
 The product is 162000
 Largest of three integers is 90
 Smallest of three integers is 30
...Program finished with exit code 0
Press ENTER to exit console.


parametric survival regression and discrete time survival regression

Question: I would greatly appreciate if you could let me know how to choose among different parametric distributions including gama, Weibull, lognormal, loglogistic and etc for panel (time series cross sectional data) survival analysis or discrete time survival analysis in STATA 14.
I read these materials but they are about continuous time survival analysis:
http://spia.uga.edu/faculty_pages/rbakker/pols8501/OxfordTwoNotes.pdf
Then, I tried to calculate LR test, which is explained on page 22 of the second note, in order to calculate p_value. However, I am not sure or I don't know what to do.
Survival Distribution                     AIC          BIC       Log-Likelihood      df
Exponential-Proportional Hazard:        433.663    471.1031       -209.83151         7
Exponential-Accelerated Failure Time:   433.663    471.1031       -209.83151         7

Lognormal-Proportional Hazard:          377.6502   420.4389       -180.82508         8
Loglogistic-Proportional Hazard:        377.874    420.6627       -180.93701         8

Gama-Proportional Hazard:       cannot compute an improvement -- discontinuous region encountered
Weibull-Proportional Hazard:    cannot compute an improvement -- discontinuous region encountered

Weibull-Accelerated Failure Time:       205.8869   248.6756        -94.943472        8

Besides, I just could test PH assumption for cox model, which is not a kind of panel data.
What's more, I couldn't do what is instructed on pages 24 and 25 of Oxford second note. In fact, when I use the "predict" command, it gives me an array of continuous values even though my dependent variable is discrete.
My data set is as follows: ID represents different companies in my sample. Event shows that if the company went bankrupt or not. X1 to X5 are my independent variables.
ID  TIME    EVENT    x1      x2      x3      x4      x5
1   1        0      1.28    0.02    0.87    1.22    0.06
1   2        0      1.27    0.01    0.82    1.00    -0.01
1   3        0      1.05    -0.06   0.92    0.73    0.02
1   4        0      1.11    -0.02   0.86    0.81    0.08
1   5        1      1.22    -0.06   0.89    0.48    0.01
2   1        0      1.06    0.11    0.81    0.84    0.20
2   2        0      1.06    0.08    0.88    0.69    0.14
2   3        0      0.97    0.08    0.91    0.81    0.17
2   4        0      1.06    0.13    0.82    0.88    0.23
2   5        0      1.12    0.15    0.76    1.08    0.28
2   6        0      1.60    0.26    0.55    1.31    0.37
2   7        0      1.58    0.26    0.56    1.16    0.35
2   8        0      1.54    0.24    0.59    1.08    0.33
2   9        0      1.72    0.22    0.55    0.84    0.29
2   10       0      1.72    0.21    0.53    0.79    0.29
2   11       0      1.63    0.19    0.55    0.73    0.27
2   12       0      2.17    0.32    0.44    0.95    0.43
3   1        0      0.87    -0.03   0.79    0.61    0.00
3   2        1      0.83    -0.14   0.95    0.57    -0.02

Best regards,
Comment: Welcome to CV, ebrahimi. Unfortunately, your question is very broad, since there are many kinds of survival model, and which one is most apt will depend on the nature of one's data. Please see the [help center] for guidance on how you can improve your question (you can edit it to try and improve by clicking the "edit" link at the lower left).
Comment: Are the units of TIME the same for each company/ID? For example, does "TIME=3" mean 3 years (or months, or quarters) after the start of the company for each? If so, your data seem suitable for a Cox regression with time-dependent covariates.
Comment: @ EdM. Thanks. Yes, it represents year and it is the same for each company/ID. Besides, I used this commands using STATA 14. http://www.stata.com/new-in-stata/panel-data-survival/. However, I just tried random effects (not mixed effects). Could you please let me know what software do Cox regression with time-dependent covariates?
Comment: In your data sample, there is at most one bankruptcy event per company. Is that always the case, or do some companies undergo more than 1 bankruptcy?
Comment: Also, don't leave a space between "@" and the user name, or we won't get notified that you replied. That is, use @ebrahimi not @ ebrahimi.
Comment: @EdM. I thank you very much for your time and consideration. There is at most one bankruptcy event per company. In fact, after bankruptcy event there is no data for the bankrupt company.
Comment: @Alexis. Thanks. As you mentioned, I added some part of my data
Answer: First, with at most one bankruptcy event per company, you don't have data for a panel survival model as described on the page linked in your question. Quoting from that page:

Obviously, in survival data, we have repeated observations on the same person because we observed them over a period of time, from onset of risk until failure or the calling off of the data collection effort. Sometimes the multiple observations on a person are explicit; the data themselves contain multiple observations for some or all the individuals. That happens when covariates change over time. Other times, the multiple observations on the individuals are implicit; there is only one physical observation for each, but still that observation records a span of time.
Those kinds of repeated observations have nothing to do with panel data. Panel data arises, for instance, when individuals are from different countries and it was believed that country affects survival. In that case, in a panel-data model, there would be a random effect or, if you prefer, an unobserved latent effect for each country.
We can, however, write models in which the random effect occurs at the individual level if we have repeated failure events for them. (Emphasis added)

For example, if you were studying how quickly different people caught a common cold, which can happen often, you might include as a random effect the different tendencies for individuals to catch a cold, given the covariate values. But with only one event per individual you can't do that. What you have are data arranged in a standard format for survival analysis with time-dependent covariates. From the structure of your data you don't seem to have any random effects.
Second, unless you have a really strong reason to suspect that your survival data take a particular parametric form, you will typically be better off trying the semi-parametric approach of Cox proportional hazards regression rather than a parametric model. With Cox regression you don't need to know how the underlying hazard changes over time, removing an important assumption that you would have to make with a parametric model. Yes, you only have data for one time point per year, but that's no more incompatible with a Cox model than it would be with parametric models.
Third, you do have to be careful in the organization of the data for your model. The covariate values for any time point should represent their status just before the event. So it would be wrong to use year-end covariate values on the same row as bankruptcy events that could have occurred earlier in the year. Depending on the nature of your data you might need to reorganize the data so that covariates best represent predictors of events at the noted times.
Fourth, and perhaps most important, you might want to consider whether and how a survival model is appropriate for your study. For example, the Cox model assumes a basic shared shape of the hazard as a function of time starting from time 0. In clinical studies, for example, time 0 might be the time a patient received a particular treatment. In your case, the assumptions and implications of the assumption of a shared hazard-function shape could be quite different in 2 different scenarios: (a) if time 0 represents the time of formation of each company, or (b) time 0 represents, say, the calendar year 2000. You need to think carefully about how such assumptions correspond to what you know about the subject matter.
Comment: Thanks a lot for sharing your time and knowledge. Then, this article (https://papers.ssrn.com/sol3/papers2.cfm?abstract_id=2457917) also mentioned the use of panel logistic for bankruptcy problem so that's why I think that my data is panel data or time series cross sectional data because I have different companies during different times.
Comment: Having data on different companies during different times does not make this panel data. Re-read the quote in my answer from the paper linked in your question. The paper linked in your comment does make an important point about continuous-time versus discrete-time survival models. It might make sense to evaluate discrete-time models, but that's a separate issue from panel-data analysis. None of the parametric models you tried were discrete-time models, so far as I can tell; I have no experience with discrete-time survival models.
Comment: I greatly appreciate you for you time and consideration. With respect to the quotation, I think that it intends to distinguish between fixed-effects and random-effects because panel data could be random or fixed effects.


Entity Framework and Temporary Objects

Question: I'm new to the Entity Framework and am currently experimenting with it. I created a simple database, set up the model in VS2008, and have got the code going to query the database using the EF as well as inserting new data.
There's one thing that has me a little confused though. I have an entity (set up in my model) called Customer, and as part of the logic of my application I want to be able to create a temporary Customer object for some intermediate processing. This particular object should never actually be stored in the database. However, I noticed that as soon as I call SaveChanges() the customer is saved to the database. This isn't what I want to happen. I'd be quite happy to call AddCustomer() on the objects I do want to include - I just want to have the option to create a temporary instance for my own use.
I did discover I could call Detach() and pass in my temporary instance, which would stop it from being persisted. However I'm not sure this is the best way to do this since the temporary Customer object will have related objects, and unless I go through and detach them all I might end up in hot water.
It's possible I'm misunderstanding something about how the EF is supposed to work, or that I'm missing something obvious - I'm hoping someone can set me straight!
Thanks
John
Comment: * bump *   I have this exact same question
Answer: If you want to have a temporary instance of an entity that'll never be connected to the EF again, use this Entity Cloner for cloning the entity
If you are trying to disconnect an entity, send it over the wire some where (let us say pass it over to the client over a service, to modify it, and then again get it back), and again merge back the changes to the EF - right now this is not directly supported. How ever, you can try these solutions
Entity Bag:
EFContrib (you need PostSharp4EF)
Comment: Thanks very much for that. I'm actually creating entirely new objects rather than cloning them - ideally what I'd like to be able to do it go

Customer cust = new Customer();

without that ever being put into the EF unless I specifically say

entities.AddToCustomer(cust);

I'm not sure those links would allow me to do something like this?
Answer: Why not have another Customer class with the same fields?
Comment: That's a possibility - but then on the occasions when I do need to persist one of the temporary objects, I'd need to convert it to an EF object. Also, this gets pretty wild if I have child objects off Customer (e.g. Order) which I'm manipulating - now I'd have in-memory versions of Customer, Order, Address, etc, as well as the EF versions.

It's certainly a way around the problem - but it seems like a lot of work, and a lot of unnecessary work given that EF already made up these classes for me!

Thanks for the suggestion though.
Answer: Just ran into this problem myself with a service using EF4 - there's a simpler solution - after you create the new entity instance, call
<code>objectContext.Detach(newEntity);
</code>

Entity Framework and Temporary Objects


How to get progress update every 3 seconds in C++ in long computation

Question: I have a program that runs a computation for 10+ hours. It's an entry-by-entry based task that reads a file line by line and computes on the input. At the moment it stays silent for 10 hours before spitting out a "Time elapsed: xxx minutes" message.
I would like to get updates as I go, but I would also like to over-engineer the problem such that I get updates at regular intervals. Clearly I can do some kind of
if (++tasks_processed % 100000 == 0)
    cout << tasks_processed << " entries processed...\n";

But I expect I may improve my algorithms in the near future or simple advances in processor/disk speeds will cause my program to be spamming out a dozen of these per second in 2-3 year's time. So instead I want to be able to future proof my reporting intervals.
Now the alternative is to have some chrono based solution where I say
high_resolution_clock::time_point t_start = high_resolution_clock::now();

while (...) {
    processing...

    high_resolution_clock::time_point t_now = high_resolution_clock::now();
    auto duration = duration_cast<seconds>(t_now - t_start).count();
    if (duration >=3)
        cout << tasks_processed << " entries processed...\n";
}

But this adds a lot of overhead to a tight loop. Are there any other facilities I could make use of to achieve the desired effect?
Comment: Have one thread set a global atomic variable and a second thread that publishes progress every 3 seconds?
Comment: have hybride, check clocks every 100 000th cycle, if enough time has passed, report.
Comment: Register a callback to be invoked every 3 seconds for the O/S to notify you. The callback is a thread that you create in your own process space that does its computation quickly when triggered. That work has to get done under 3 seconds. It runs parallel to the O/S tick counter.
Comment: Are there simple standard library features that would allow me to easily set aside an asynchronous thread for doing this kind of timing?
Answer: Check this self explanatory pseudo code as a solution
void LongComputation(std::atomic<bool>& running, std::atomic<float>& progress)
{
    // do long computation
    while (running)
    {

        //update progress
    }
}

void ProgressCounter(std::atomic<bool>& running, std::atomic<float>& progress)
{
    while (running)
    {
        std::cout << progress << "\n";
        std::this_thread::sleep_for(std::chrono::seconds(3));
    }
}

int main() {

    std::atomic<bool> running{true};
    std::atomic<float> progress{0};

    std::thread t1([&running, &progress]() { LongComputation(running, progress); });
    std::thread t2([&running, &progress]() { ProgressCounter(running, progress); });

    //simulating GUI loop
    while (!getch())
    {

    }

    running = false;
    t1.join();
    t2.join();
    return 0;

}

Comment: There is a data race on `running`, switch to atomics
Comment: Thanks a lot for this. Is there some construction that would allow me to spawn an asynchronous thread from `LongComputation` so I don't have to refactor my code to wrap a master caller over both the computation and the progress counter?
Comment: @shians: You can spawn new worker threads from the main computational thread, but just pass the reference variables ( running and progress) to them as well so that you can exit cleanly from any thread when the GUI desires to terminate the program.


Workbench: Allow users to publish revisions without moderation

Question: I want to be able to moderate authors who create new nodes, but allow authors to update their existing nodes without moderation. It's not immediately obvious to me how to do this in Workbench.
It seems like I should be able to set up a Rule that will basically have
Event: After updating existing content
Condition: Content's current moderation state is published
Reaction: Set moderation state during save: Published
Something like that. I've tried a lot of combinations, but I can't get it to work.
Also, every time an author saves, it saves as a Draft, which I almost never want. Even though the Default State is set to Needs Review that only applies to new nodes. Revisions of existing nodes are always saved as drafts.
So ideally what I'm trying achieve is this

new nodes default to Needs Review
revisions of existing nodes default to Publised
Comment: This may just be a limitation of the workbench moderation module: https://drupal.org/node/1408858
Comment: Another related issue: https://drupal.org/node/1147646
Comment: The basic problem seems to be that the Workbench module is hardcoded to save new revisions as drafts, so the default state only works on a node creation. Still, it seems like I should be able to do something with Rules that would push it to the states I need in spite of the defaults.
Comment: thank you, I searched for this answer for a lot of time!!
Comment: I need the same thing but could not find any solution
Answer: This is sort of bizarre and I'd love to hear other answers, but after much trial and error with Rules, it turns out this seems to work with the following setup:
Event: After updating existing content [so it's not a new node]
Condition: Content is published [so admin has approved it at least once]
Reaction:
1. Publish Content
2. Set moderation state during save: Published
I would have thought that either #1 or #2 would have been sufficient, but strangely if they are not combined, it always reverts to draft. Also strange, the order doesn't seem to matter. I've had it working #1 then #2 and #2 then #1.
Comment: i have both #1 and #2 but it always go to draft(current)?


CSS Trebuchet MS vs. Photoshop Trebuchet MS

Question: I designed a small template which uses Trebuchet MS font (PSD). Now, if I use this font in HTML/CSS it
looks quite different. Is there any possibility to achive the same or a better quality ?
Maybe Google Webfont ? Is there a simmilar font ? Or should I use CSS @font-face?
<code>CSS:
font-family: "Trebuchet MS", sans serif;
</code>
Best regards
Answer: The basic answer is No.
You're talking about font anti-aliasing (or font smoothing).  Photoshop gives anti-aliasing in four flavors (Sharp, Crisp, Strong, Smooth).  Browsers don't do the same thing, generally the OS will control font anti-aliasing.
more on browsers and fonts: Type rendering: web browsers
Answer: No, unless you can use flash generated or image fonts, etc. You're looking at the differences in how two different beasts render fonts and it's mostly stuff outside your control.
Answer: @font-face is new and doesn't work in all browsers. In order to maintain cross browser support, you can use the JS based Cufon to deliver whatever font you want.
http://cufon.shoqolate.com/generate/
Comment: `@font-face is new and doesn't work in all browsers`...it's actually not that new and is very well supported http://caniuse.com/#feat=fontface
Comment: but I think you mean anything prior to IE4 since `@font-face` has been around since [CSS2 (1998)](http://www.zeldman.com/2009/08/17/web-fonts-and-standards/)
Comment: Ah, W3Schools. W3Schools is not W3C, despite the similar names. And in this case, they're half-right and all wrong: IE8 and earlier do support the `@font-face` rule, but they don't support all font formats.
Comment: I guess my point was legacy support, anything prior to IE 9. Cufon, being JS based, should work in any browser that's supports JS. @font-face won't.
Comment: Trebuchet is considered web safe. http://www.ampsoft.net/webdesign-l/WindowsMacFonts.html
Comment: Per W3C, it's not supported past IE9. "The @font-face at-rule allows you to define custom fonts. It was first defined in the CSS2 specification, but was removed from CSS2.1."
W3C Says "Note: Internet Explorer 8 and earlier versions, do not support the font-face rule."
Comment: @mdmullinax: Or rather it was going to be in CSS2, but pushed back due to interoperability concerns (and Microsoft generally being Microsoft).
Comment: @Brant: I find it hard to believe that W3C would say that. W3C usually makes no comments or notes about individual browsers whatsoever.


Family of generators for congruence subgroup $\Gamma_0(11)$

Question: Consider the congruence subgroup
$$\Gamma_0(11)=\left\{\begin{pmatrix}a&b\\11c&d\end{pmatrix}\in M(2,\mathbb{Z}): ad-11bc=1\right\}$$
I want to prove that the family
$$\Omega=\{-\text{Id}\}\cup\left\{\begin{pmatrix}a&b\\11c&d\end{pmatrix}\in \Gamma_0(11): c>0,\  gcd(c,6)=1\right\}$$
generates the whole group.
Here is my attempt: let's start by a generic
$$\gamma=\begin{pmatrix}a&b\\11c&d\end{pmatrix}\in\Gamma_0(11)$$
My idea is to act with matrices in $\Omega$ until I find an element of this set. If $c\leq 0$
I can act with $-\text{Id}$ to get a matrix with $c\geq 0.$ If I have $c=0$ I also have $ad=1,$ hence $a=d=\pm1.$ Using $-\text{Id}$ I can suppose $a=d=1,$ and acting on the right, for instance, with the matrix
$$\begin{pmatrix}2&-1\\11&-5\end{pmatrix}\in\Omega$$
I get
$$\begin{pmatrix}1&b\\0&1\end{pmatrix}\begin{pmatrix}2&-1\\11&-5\end{pmatrix}=\begin{pmatrix}2+11b&-1-5b\\11&-5\end{pmatrix}$$
The issue is with the other condition. Suppose $gcd(c,6)\neq1:$ then $2$ or $3$ divide $c:$ how can I  manage these cases?
EDIT
Writing $c$ in the form $c=2^m\cdot 3^n\cdot C,$ if $m,n>0$ (i.e. $6\mid c$) I can write $\gamma$ using only elements of $\Omega$ in the following way: the determinant condition implies
$$(11\cdot2^m\cdot3^n c,a)=1$$
and in particular $a$ is odd and is not a multiple of $3.$ The matrix
$$\begin{pmatrix}
   d&2^m3^n b\\
   11c& a
  \end{pmatrix}\in\Omega,$$
and
$$\begin{pmatrix}
   d&2^m3^n b\\
   11c& a
  \end{pmatrix}\begin{pmatrix}
   a&b\\
   11\cdot 2^m\cdot 3^n c& d
  \end{pmatrix}=\begin{pmatrix}
   ad+2^{m}3^n(11c+b)&bd(1+2^m3^n)\\
   11ac(2^m3^n+1)&11bc+ad
  \end{pmatrix}\in\Omega $$
Maybe I can generalize this approach even for the other cases
Answer: A calculation taking a tiny fraction of a second in Sage reveals that
$$\{ -\mathrm{Id}\} \cup \left\{\left(\begin{array}{rr}
18 & -5 \\
11 & -3
\end{array}\right), \left(\begin{array}{rr}
7 & -2 \\
11 & -3
\end{array}\right), \left(\begin{array}{rr}
8 & -3 \\
11 & -4
\end{array}\right)\right\}$$
is a generating set of $\Gamma_0(11)$. Since these all have $c = 1$ you are done.
Comment: I'm trying to avoid Sage, I'd like to find a solution by hand. I added an attempt for the case $6\mid n...$


Shutting down a Linux based NAS remotely and reliably

Question: I have a Linux based (Debian 5.0) NAS that I need to shutdown reliably.  When ssh'd into the NAS box I issue the following command:
sudo shutdown -h now

I've also tried 'poweroff' and su'ing into root and running the commands with the same results.
It shuts down all the processes and everything fine but does not always 'halt' the system.  I need it to power down into the S3 state so that I can reliably use Wake On LAN to start the unit as needed.  For what it's worth, wol works great and works every time as long as the unit is in it's S3 state.
Comment: I have seen this before on slightly older hardware.  The issue is 100% related to any ACPI/APM settings you have in BIOS, the age of the motherboard, and the quality of the BIOS as well.  Older/cheaper boards tend to have this (as ACPI and soft-poweroff were still somewhat new); newer systems have never exhibited this issue in my experience.
Comment: This is an older Intel baseboard, the D945PVS.  It's BIOS is the latest (version 0108 dated 08-17-2008) and it's configuration seems to be set correctly.  I do have a slightly newer Intel board, a 955X chipset model that I could swap it with but I was hoping to avoid that pain.  Thanks for giving me the answer I didn't want but is probably the most accurate.
Answer: Humm could be a number of things. First thing I would try is to pass either
acpi=force

or even
acpi=off

to the kernel command line to see either of them works.
Next I would try unloading the wireless drivers before you shutdown (if possible).
Answer: Try the halt -p or poweroff commands.
Answer: I finally diagnosed the problem with this particular NAS backup server.  It appears that the graphics card died at some point.  Since this unit is run remotely via ssh it's only two lifelines are it's ethernet cable and it's power cord.  Consequently, the dead graphics card went totally unnoticed.  Putting in a new graphics card has solved the problem completely.  I also did a lot of work to the unit to make sure that it has better airflow and much less heat build up since it is located in a place that is not very well ventilated and I believe that this may have been the cause of the graphics card failure.  It could have just been the age of the graphics card but since I had the unit out I decicded to update/upgrade it's thermal properties just to be on the safe side.
Comment: weird, but glad it works!
Answer: I have had similar problems, I just issue a halt instead, and it seems to work fine.


Finding the multiplicity of a fifth-degree polynomial

Question: If you have a fifth-degree polynomial equation and you know a root of this polynomial is equal to $1$, then why can you find the multiplicity of the root $1$ by dividing by $z-1$ or by differentiating the polynomial? See the image below for an example.
What is the multiplicity of the root 1?
Answer: A polynomial $p(x)$ has a root $\alpha$ of multiplicity $k$ if and only if $(z-\alpha)^k$ is a factor of $p(x)$, and $(z-\alpha)^{k+1}$ is not.
Therefore, if we start with $p(x)$, and divide by $(z-\alpha)$ a total $k$ times, until we cannot divide any more, this shows that $p(x)$ has a factor $(z-\alpha)^k$, but not $(z-\alpha)^{k+1}$. Hence, $\alpha$ is a root of multiplicity $k$.
Answer: If $p(z)$ is the fifth degree polynomial and 1 is a root, then $p(z)$ can be re-written as $p(z)=(z-1)f(z)$ where $f(z)$ is a polynomial of degree four. So, dividing $p(z)$ by $z-1$ will simply give the polynomial $f(z)$. Now 1, is a root of the equation $p(z)=0$ or $(z-1)f(z)=0$. We can check if 1 is a root of $f(z)$ and if it happens to be, then by the same arguement as before, $f(z)$ can be re-written as $f(z)=(z-1)g(z)$ for some polynomial $g(z)$ with degree 3. Combining the above we get $p(z)=(z-1)^2g(z)$ which tells us that the multiplicity of the root 1 is at least 2. And we continue the same way for $g(z)$ etc.


How did check for not nil pass

Question: I don't understand how this if statement executed as timer is nil
        if timer != nil
        {
            logInfo("Cancelling notification for op=\(op), count=\(self.retryWindow.count)")
            timer!.invalidate()

Answer: Multi-threading is a possible cause. In between your if statement and your call to invalidate, timer could have been set to nil on another thread.
Don't check against nil. Safely unwrap.
if let timer = timer {
    timer.invalidate()
}

Comment: Is this guaranteed to be thread safe? Unwrapping also might involve an `if` check somewhere, which means another thread still has chances to mess with the value.
Comment: @Cristik Excellent question. I have to admit I just assumed an `if let` was atomic (therefore thread safe) but I can't find any indication either way. I think that deserves a question of its own.
Answer: The easiest thing to do is use the power of magical optionals.. No if let or guard.. since your timer is already an optional, call the method on top of an optional, and everything should be taken care by compiler automatically.
timer?.invalidate()
timer = nil

What it will do is call the method invalidate if timer exists.
Comment: Why the downvote, I didn't understand.. Whoever down voted, you owe an explanation... Please..

How did check for not nil pass


Repeated binomial experiment with different settings, how to plot chances based on these settings?

Question: I'm using R and have a dataset which contains data for a certain experiment which is tested multiple times in environments with different  P values. The success or failure of the experiment is denoted by 1 or 0 respectively. An example below.
success   P
   1       15639
   1       18623
   1       19875
   0       12513
   0       10256
   1       12548
   1       15789
   0       12568
   0       10236
   1       15478
   0       11256
   1       12546
   0       10256
   1       14562
   0       10258
   0       11254
   1       12458
   1       13458
   0       12001
   1       14756
   0       10112
   0       11256
   1       13485
   1       12369
   0       11297
   1       12100
   1       15780
   0       11300
   0       10300
   1       12596

I now want to calculate and preferably also plot at which P value the experiment has a 50% chance of succeeding. What type of statistical measurement/plot should be used for this scenario?
Comment: Have you tried logistic regression?  If you add more data, I can show you how to do this in R.
Comment: I've not tried this yet but I'll definetly look it up now! I've expanded the dataset to around 30 examples, would that be enough or how many samples would you require?
Comment: You wrote Ph. Do you mean phosphorus, pH or something else? If pH, then the values are surprising.
Comment: May you clarify please what P column in example you have presented does means. If it is p-values then why them are not between 0 and 1?
Comment: @Bogdan I edited Ph to P because I believed that it might be phosphorus but was unlikely to be pH. It's not intended to be P-value, i.e. observed significance level.
Answer: How To Find the 50% Point
I am assuming you have your data in a dataframe or tibble named data.
The easiest way to find a point estimate for the ph value which elicits 50% success rate is to perform a logistic regression.  I won't go into the details of the model here, but you can easily read up on it.
With R...
#Rescale your data because the values are large
data = data %>% mutate(ph = PhValue/1000)

#Perform the regression
model = glm(success ~ ph, family = binomial(), data = data)

point_50p = -coef(model)['(Intercept)']/coef(model)['ph']

From the sample provided, it looks like the 50% point is approx 12,228.5.
Why Does This Work
Logistic regression models the log odds of the outcome as so
$$\log \left( \dfrac{p}{1-p} \right) = \beta_0 + \beta_1x$$
A log odds of 0 corresponds to a probability of 50%, so simply solve
$$  \beta_0 + \beta_1x = 0$$
Some simple algebra yields the solution.
$$ x = \dfrac{-\beta_0}{\beta_1} $$
Can We Do Better?
I would think so.  The problem with this solution is that we only get a point estimate.  What would be better is a confidence interval, but that presently eludes me.
Moving On; The Plot
Here is how we can plot this with R...
data %>% #Your data
  ggplot(aes(ph,success))+
  geom_jitter(height = 0.05, width = 0, alpha = 0.5)+
  geom_smooth(method = 'glm', method.args = list(family='binomial'))+
  geom_segment(aes(y = 0.5, yend = 0.5, x = 10, xend = point_50p))+
  geom_segment(aes(y = 0.5, yend = 0, x = point_50p, xend = point_50p))

Yielding

I like geom jitter because it allows me to see the density of points along the x axis, rather than obfuscating them.


Trazer valores de um select com AJAX automaticamente

Question: Comecei a usar AJAX há um tempinho e estou com uma dúvida. Possuo um <select> e estou tentando trazer o valor dele na mesma página (sem dar refresh), porém, só é exibido o valor quando eu clico no <select> novamente, para selecionar outra opção. Como faria para mostrar assim que eu selecionasse?

Vamos supor que eu seleciono o <select> com value="4", só é mostrado o "4", quando eu clico no <select> de novo...

index.php:
<select class="common_selector" id="ordenar_tabela">
     <option value="1">Lançamentos</option>
     <option value="2">Menor valor</option>
     <option value="3">Maior valor</option>
     <option value="4">A - Z</option>
     <option value="5">Z - A</option>
</select>

verificar.php:
<?php
          $query = "SELECT * FROM produtos_estoque";

          // Se o usuário quiser ordenar a tabela...
          if(isset($_POST["ordenacao"])){

          $ordenacao = $_POST["ordenacao"];
          echo $ordenacao; //Estou exibindo essa variavel na página, para ver se vai funcionar os demais códigos abaixo...

          // $query .= ($ordenacao == 1) ? " ORDER BY ID_produtos DESC" : "";
          // $query .= ($ordenacao == 2) ? " ORDER BY ID_produtos DESC" : "";
          // $query .= ($ordenacao == 3) ? " ORDER BY ID_produtos DESC" : "";
          // $query .= ($ordenacao == 4) ? ' ORDER BY produto' : "";
          // $query .= ($ordenacao == 5) ? ' ORDER BY produto DESC' : "";
        }
?>

Comment: Vou editar mostrando a `$query`, `$('.filter_data')` apenas não foi inserido na questão.
Answer: Seu problema está acontecendo pois você chama a função filter_data() quando o <select> é clicado.
Veja como chamar a função ao selecionar uma nova opção, utilizando o atributo onchange:
<select class="common_selector" id="ordenar_tabela" onchange="filter_data()">
    <option value="1">Lançamentos</option>
    <option value="2">Menor valor</option>
    <option value="3">Maior valor</option>
    <option value="4">A - Z</option>
    <option value="5">Z - A</option>
</select>

function filter_data()
{
    var action = 'fetch_data';
    var ordenacao = $('#ordenar_tabela').val();

    $.ajax({
        url:"verificar.php",
        method:"POST",
        data:{action:action, ordenacao:ordenacao},
        success:function(data){
            $('.filter_data').html(data);
        }
    });
}

Outra opção, mantendo o HTML intacto e utilizando jQuery:
<select class="common_selector" id="ordenar_tabela">
    <option value="1">Lançamentos</option>
    <option value="2">Menor valor</option>
    <option value="3">Maior valor</option>
    <option value="4">A - Z</option>
    <option value="5">Z - A</option>
</select>

function filter_data()
{
    var action = 'fetch_data';
    var ordenacao = $('#ordenar_tabela').val();

    $.ajax({
        url:"verificar.php",
        method:"POST",
        data:{action:action, ordenacao:ordenacao},
        success:function(data){
            $('.filter_data').html(data);
        }
    });
}

$(".common_selector").change(function() {
  filter_data();
});

Comment: Dei uma editada na pergunta, irei usar o `value` desse `` em uma condição em PHP, assim, mudando a consulta da minha página.
Comment: No `verificar.php` você está devolvendo o valor do POST, ou seja, o valor enviado. O código continua redundante. Caso deseje retornar os valores da query, é uma questão sobre SQL, não AJAX. Por favor, reformule sua questão para clarificar seu objetivo.
Comment: Teoricamente está funcionando, estou recebendo os valores e está sendo exibido corretamente. O problema, é que só está exibindo, quando eu clico no `` novamente, creio que isso seja como eu estou pegando o valor com `AJAX`...
Comment: Certo. Vou reformular a resposta.
Comment: Infelizmente, não funcionou, não vi nenhuma alteração...
Comment: Utilizando o atributo `onchange`, quando você selecionar uma opção no `` ele chamará a função. Seu código está utilizando o `onchange`? Confirmando, seu backend está devolvendo as informações corretamente, certo?
Comment: Vamos [continuar esta discussão no chat](https://chat.stackexchange.com/rooms/116058/discussion-between-brhaka-and-user210741).


Postfix Send many messages in one connection

Question: i have a question about postfix.
We cand send to our providers smarthost 4000 Mails per Hour.
When we send this messages, we got the failure too many connections back.
Now i want to send all theese messages in one connection.
So my question: Is it possible to send 500 Mails simultaneously in one connection to the providers smarthost?
Best regards,
Andreas
Comment: Looks like you want SMTP connection caching. See http://www.postfix.org/CONNECTION_CACHE_README.html.
Answer: With Postfix 2.2 or later, SMTP connection caching "on demand" is enabled by default, where connections will be reused "when a destination has a high volume of mail in the active queue" (what counts as "high volume" isn't discussed).
If you have a limited list of destinations which you want to keep connections open for (as per your smarthost situation) you can tell Postfix to really, really keep connections open:
<code>smtp_connection_cache_destinations = $relayhost
</code>
If your relayhost uses TLS, you're out of luck, though, because "The Postfix shared connection cache cannot be used with TLS".


Currently checked out version of a directory/file in perforce

Question: I'm trying to automate some builds for a project kept in perforce. I'd like to extract the current changelist number as a way of tagging the build version.
This command:
p4 changes -s submitted  ...  | head -1

goes very close to doing what I need, except in the case where I don't have the latest version checked out from perforce. The above command (appears to) display all the changelists, not just the ones in my check out.
Is there a way to get the changelist number of the most recently synced-to changelist?
Note: I come from more of a CVS / git background so my terminology may be off.
Answer: Try running 'p4 cstat ...'.  That'll tell you, for every changelist that affects the set of files in '...', whether you have that changelist, need it, or have it partially synced.
Answer: I agree with the "p4 cstat" approach.
I use it like:  p4 cstat| p4-version.awk
which is the following script:
 #! /usr/bin/awk -f

 # The input format is:
 # ... change 654056
 # ... status have|need|partial

 # So what we do:
 # find the first "need" or "partial"

 # and we might add +N as the number of other changes we have

 # Major is never 0. Hence End is set only once, to a non-zero.

 # Note: it's assumed that 0 is not a valid P4 changelist.
 # so  0+1-1 means: no entire changelist (i.e. the initial one)
 # has been (entirely) synced.

 # major is the last p4 CL processed
 # end   is the last CL, up to which _fully_ synced.
 # plus/minus  additional/missing CLs.

 BEGIN {end=0; contiguous=1; plus=0; minus=0}

 /^... change ([0-9])+/ {
         change=$3;
 };

 # we have 2 states: 1/ until now all have -- contiguous=1
 #                   2/ already something not "have"

 # (end=0) state 1

 /^... status have/ {
         if (!contiguous)
                  plus++;
         else
           end=change;
 };

 /^... status (partial|need)/ {
         minus++;
         if (contiguous) {
                 contiguous = 0;
         };
 }

 END {
         if (!contiguous)
                 printf "%d+%d-%d\n", end, plus, minus;
         else
                 printf "%d\n", end;
 }

Comment: Had a question about the output: you have "printf "%d+%d\n", end, plus;", what is the plus?  Otherwise, great stuff!  Thanks for sharing.
Comment: I updated the code --- version+plus-minus  means you synced since beginning till "version", plus you have other "plus" changesets synced and "minus" not (fully) synced -- hence missing.
Answer: Though I am not sure if it is applicable for this question I want to show a solution that lets you get the latest changelist up to the one you have synced without gaps (i.e. you can have synced new changelists on your client leaving out some older ones, for example if you have submitted changes yourself without previously syncing a few changes that were submitted by other people).
With the following command you list all changes not synced to your client:
p4 changes ..."#>have"

If you extract the last changelist of that list and subtract 1 you get the changelist number up to which you have synced everything without gaps:
echo $((`p4 changes ..."#>have" | tail -1 | awk '{print $2}'`-1))

Answer: To find the latest changelist on your client, use this command:
p4 changes -m 1 @name_of_your_client_here

Client names can act like labels.
Comment: It is possible that you don't have synced changelists that are older than the latest changelist that is already synced to your client.
For example if you submit a new change but didn't have your clients synced before.
You should mention that with your method.
Comment: Well, yes.  I said as much, but that wasn't the question.
Answer: I think that the following will do what you want:
p4 fstat -T headChange ... | cut -d' ' -f3 | sort -nr | head -1

p4 fstat is really nice in that it takes not only a filter (via the -F option) but a field selection parameter (using the -T option).
Comment: This appears to provide the same version as p4 changes -s submitted, i.e: the latest available, not the currently checked out version.
Comment: I must have misread the question, I'll have to check this once I get in to the office.  `p4 help fstat` lists the attributes that are available.
Answer: What you want is the 'haveRev' of your file:
rev = `p4 fstat -T haveRev <filename>|cut -d' ' -f3`

and then you can look up the changenumber like so:
p4 filelog <filename>|grep $rev|cut -d' ' -f4

Note that I used a specific file of the checked out tree.


Test for symmetry for polar graphs

Question: From a calculus book I'm reading: "Unlike the graphs of an equation in $x$ and $y$, the graph of an equation $r=f(\theta)$ can be symmetric with respect to the polar axis, the line $\theta = \pi/2$, or the pole without satisfying one of the tests for symmetry. This is true because of the many different ways of specifying a point in polar coordinates."
Why is the previous paragraph true? Why are there many ways to specify a point in polar coordinates and how does that affect the symmetry test?
Comment: The origin has infinitely many "names" $(0,\theta)$ where $\theta$ can be anything. And every point $(r,\theta)$ could also be called $(r,\theta+2n\pi)$, $n$ any integer. How that affects symmetry tests is unclear: it should not affect a properly constructed symmetry test.
Answer:
The origin has infinitely many "names" $(0,\theta)$ where $\theta$ can be anything. And every point $(r,\theta)$ could also be called $(r,\theta+2n\pi)$, $n$ any integer. How that affects symmetry tests is unclear: it should not affect a properly constructed symmetry test. –  André Nicolas on July 27, 2013, at 7:06

André has posted what I think is a satisfactory answer in the comments, hence this wiki answer.

Test for symmetry for polar graphs


How to deal with unexpected low heterogeneity in meta-analysis?

Question: I am performing a meta-analysis in R with metafor package. Primary studies report the reduction of plant species to certain amount of nitrogen fertilization. Effect size is calculated as the ratio of number of species in the treatment group divided the control for each observation.
Following Borenstein (2009), there is no reason to believe a fixed effect model should be used since these experiments are all undertaken in different ecosystems, vegetation types, soil type et cetera.
However, heterogeneity of the pooled effect sizes obtained using random models of rma() is extremely low.
Total heterogeneity is very low (I^2 = 0.00%) for regressions fitted with "DL" method:
> res_means <- rma(S_ratio, Variance, data = dat_S_ratio, method="DL")
> res_means

Random-Effects Model (k = 236; tau^2 estimator: DL)

tau^2 (estimated amount of total heterogeneity): 0 (SE = 0.0090)
tau (square root of estimated tau^2 value):      0
I^2 (total heterogeneity / total variability):   0.00%
H^2 (total variability / sampling variability):  1.00

Test for Heterogeneity:
Q(df = 235) = 170.8544, p-val = 0.9994

Model Results:

estimate       se     zval     pval    ci.lb    ci.ub
  0.7403   0.0203  36.5345   <.0001   0.7005   0.7800      ***

---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Even using REML (I read somewhere that is actually more accurate), heterogeneity is still very low (I^2 = 10.99%):
> res_means <- rma(S_ratio, Variance, data = dat_S_ratio, random=~1|PrimaryStudy, control=list(stepadj=.5))
> res_means

Random-Effects Model (k = 236; tau^2 estimator: REML)

tau^2 (estimated amount of total heterogeneity): 0.0120 (SE = 0.0090)
tau (square root of estimated tau^2 value):      0.1096
I^2 (total heterogeneity / total variability):   10.99%
H^2 (total variability / sampling variability):  1.12

Test for Heterogeneity:
Q(df = 235) = 170.8544, p-val = 0.9994

Model Results:

estimate       se     zval     pval    ci.lb    ci.ub
  0.7628   0.0221  34.4510   <.0001   0.7194   0.8062      ***

---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Subgroup analysis provide also very low values of heterogeneity when effect size is calculated for each subgroup.
Is it possible to have such unexpected value of heterogeneity?
I suspect this is due to the way I calculated the variance but I am not sure about it. Indeed variance was not extracted from primary studies but back-calculated with the following for each observation:

Variance calculation obtained with the Delta method, assuming data are following a Poisson distribution. Sst is the richness of species in the treatment group, Ssc is the richness of species at the control group
Looking at the forest plot, confidence intervals looks similar to each other in terms of range (ok, there are differences but not THAT big), although for each observation the effect size is different. Is this an explanation why heterogeneity is so low?
Does it make sense to still perform a meta-regression on observation with such low heterogeneity?
Comment: Your outcome (``S_ratio``) is a count divided by another count? Then I would suggest to log transform those values. Let's say the ratios in two studies are 20/10 and 10/20. Then the average would be 1.25, which makes little sense. Ratios of counts are not symmetric around 1. Log transforming the ratios solves this issue. Also, log transforming should make the sampling distribution more normal (an assumption underlying the model you are using). The variance is then $1/S_{st} + 1/S_{sc}$.
Comment: Since you are comparing between and within it may be that your studies have very high within variance. But follow @Wolfgang advice first.
Comment: yes. Thanks a lot for the suggestion @Wolfgang. Indeed with log transformed {S_ratio} heterogeneity is higher (I^2 = 49%, with "DL" estimator).
Comment: The Poisson distribution carries a rather stringent assumption about the variance of your measures (notably the variance exactly equals the mean). Might this bear on the assumptions you are making in your variance estimate?
Comment: @GabrieleMidolo can you do a sanity check and create a forest plot of effects and CIs before pooling them all together?
Answer: So this one does not go unanswered for ever here is @Wolfgang's comment which the OP agreed solved his problem.

Your outcome (S_ratio) is a count divided by another count? Then I would suggest to log transform those values. Let's say the ratios in two studies are 20/10 and 10/20. Then the average would be 1.25, which makes little sense. Ratios of counts are not symmetric around 1. Log transforming the ratios solves this issue. Also, log transforming should make the sampling distribution more normal (an assumption underlying the model you are using). The variance is then $\frac{1}{S_{st}} + \frac{1}{S_{sc}}$

Here, as a bonus, is a reference justifying the OP's comment about using REML to estimate $\tau^2$. It is a paper by Viechtbauer entitles "Bias and efficiency of meta--analytic variance estimators in the random--effects model" available here


How can one compute the joint probability of A and B with the information provided?

Question: Let $(A, B)$ be a discrete R.V. (vector) with the set {$(0,0),(0,1),(1,0),(1,1)$}
Their marginal distributions are $A∼Binomial(1, \frac{1}{2})$ and $B∼Bin(1, \frac{1}{2})$. Given that $P(A=B)=0.8$. Find $P(AB = 0)$.
Comment: Draw a two by two table of probabilities for each of the cases. Sum appropriate cells according to your restrictions. What is the answer you want? Read it off the table.
Comment: By symmetry, $P(A=0,B=0) = P(A=1,B=1) = 0.4$ so $P(AB=0) = 1 - P(A=1,B=1) = 1 - .4 = .6.$
Answer: You have
$
P(A=0,B=0)+P(A=0,B=1) +P(A=1,B=0)+P(A=1,B=1)=1
$
This gives you
$
P(A=0,B=1) +P(A=1,B=0) +P(A=B)=1,
$
thus using $P(A=B)=0.8$,
$$P(A=0,B=1) +P(A=1,B=0)=0.2.$$
Also $0.5=P(A=1)= P(A=1,B=1)+P(A=1,B=0)$ and $0.5=P(B=1)= P(A=0,B=1)+P(A=1,B=1)$, summing the last two equations get $1=2P(A=1,B=1)+P(A=1,B=0)+P(A=0,B=1)$, plug in the value $P(A=0,B=1) +P(A=1,B=0)=0.2$ and get $P(A=1,B=1)=0.4$. Now $$P(AB=0)=1-P(A=1,B=1)= 0.6.$$


What is the meaning of the beta for the interaction between continuous variables in a linear mixed-model?

Question: If I create a mixed-effects linear regression model similar to the following (using the lme4 package in R), where all of the fixed effect variables are continuous:
model <- lmer(Y ~ a + b + c + a*b*c + a*b + b*c + (1|randvar1) + (1|randvar2),
              data=dataset1)

summary(model) gives me the list of beta estimates. For each of a, b and c these are the expected increase in Y when that independent variable increases by 1 (assuming the other variables are unchanged).
But what does the beta estimate mean for the interaction terms between two or more continuous variables?
Answer: As you state the coefficient for a gives the expected increase in Y if you increase a by 1. Similarly for b. Suppose there is no interaction then if we simultaneously increase a and b by 1 then the expected increase in Y is simply the sum of the coefficients for a and b. If there is an interaction then the coefficient for a.b has to be added to the sum of a and b. Note this applies to all regression models it is not specific to mixed effect models nor to R.
Answer: The coefficient for the interaction of a and b is interpreted the same way in a mixed effects model like this as it is for a regular linear model.
It is the effect of each 1 unit increase in a*b.
In including this term you assume that the effect of a on Y increases linearly with b.  The interaction term reflects the slope of this linear increase.  That is for every unit increase in b, the effect of a unit increase in a goes up by the value of that coefficient.
Answer: As mentioned in the other answers, the existence of the random effects is irrelevant.
As a matter of R code, your function does not include a*c.  If that was intentional, be aware that a*b*c is automatically expanded to: a:b + a:c + b:c + a:b:c.  That is, the term will be included.  To deliberately exclude it, you should list the terms using the colon operator (:).  However, it is generally recommended not to leave out lower level terms, including lower level interaction terms, once a higher level term exists (see: Including the interaction but not the main effects in a model, and Do all interactions terms need their individual terms in regression model?).  On the other hand, if that was an unintentional omission, note that your a*b + b*c is redundant (the first term automatically expands, as just discussed).

You correctly note that "For each of a, b and c these are the expected increase in Y when that independent variable increases by 1 (assuming the other variables are unchanged)."  However, it is important to recognize that "the other variables" cannot remain "unchanged", unless the other two main effects variables are exactly $0$.  To understand this more fully, it may help to read my answer to: What does "all else equal" mean in multiple regression?  Likewise, because you have a three-way interaction term in your model, the included two-way interactions have their clearest interpretation when the variable that is included in the three-way interaction but not included in the given two-way interaction is $0$.  (For example, a:b is the interaction between a and b, when c=0.)
At this point we can start to think about the interpretation of the beta for an interaction.  As elsewhere, the standard interpretation is ceteris paribus ('all else held equal').  However, note that it is not possible to increment a:b:c (or even a:b) by $1$ without any of the other terms changing.  This makes the situation difficult.  There are some rules, of increasing complexity, for interpreting the sign of the beta as, say, superadditive ('more than the sum of its parts') or subadditive ('less than the sum of its parts'), etc., but these depend on the signs of other betas as well.  I've never met anyone who has found those rules useful.  You can compute the partial derivative and think of it as the rate of change in the slope / lower order interaction, if that helps you, but again, many people find that too much to wrap their heads around.
The best way is to try to interpret the model as a whole, and not the interaction term in isolation.  To do that, solve for the simple effects of whichever variable has precedence in your mind at several values of the other interacting variables.  You could then write out the equations of those lines, or you could plot them (cf., How to visualize a fitted multiple regression model?).
Comment: Thanks a lot for your clear explanation. I have another question regrading the interaction. If the model includes a continuous variable A (0-100) and a categorical variable B with two levels (B1, B2), if the two variables show significant interaction, how to further explore the interaction? Can I directly split the data with two groups according to the median score of variable B, and then to investigate the simple effect of B for each group, separately?
Comment: @Qian, you should ask a new question, not ask in comments.  Search the site first, though.  That's the kind of thing that's been covered in various places.


Ao fazer override de um método devo chamar a implementação da super antes ou depois do meu código?

Question: Assistindo e pesquisando alguns tutoriais na internet, me deparei com alguns código em JAVA no ANDROID, conforme abaixo:
<code>@Override
protected void onPause() {
    sensorManager.unregisterListener(this);
    super.onPause();
}

@Override
protected void onPause() {
    super.onPause();
    sensorManager.unregisterListener(this);
}
</code>
A classe MainActivity, basicamente está trabalhando com o acelerômetro para coletar e exibir os eixos X,Y e Z do dispostivo.
A minha dúvida, qual a diferença de estar acima ou abaixo do super.onPause() a liberação de registro do SensorManager?
Answer: Na maioria dos casos não faz diferença, o melhor é sempre ler a documentação do método.
A documentação de onPause() apenas refere a obrigatoriedade de chamar, não diz quando.
Em caso de dúvida, parece-me lógico que,

Nos métodos que intervêm na inicialização, como onCreate(), onStart(), onRestart() e onResume(), chamar primeiro o método da super.
Nos métodos que intervêm na finalização, como onPause(), onStop() e onDestroy(), chamar o método da super no final.
Comment: muito obrigado pela explicação.


Apache LogFormat & CustomLog

Question: I am reading a book on Logstash and the author provided a sample conf file.  I am running Ubuntu 16.04 server with Apache 2 installed.  I added the conf file to /etc/apache2/conf-available and used a2enconf to create the symlink to conf-enabled.  I see the new log file in /var/log/apache2/ but no content is added to the log file.  I've matched the permission for the other log files in this directory.  Any suggestions would be appreciated.  Below is my conf file:
<code>LogFormat "{ \
\"host\":\"webapp.local\", \
\"path\":\"/var/log/apache2/logstash_access_log\", \
\"tags\":[\"wordpress\",\"www.example.com\"], \
\"message\": \"%h %l %u %t \\"%r\\" %>s %b\", \
\"timestamp\": \"%{%Y-%m-%dT%H:%M:%S%z}t\", \
\"clientip\": \"%a\", \
\"duration\": %D, \
\"status\": %>s, \
\"request\": \"%U%q\", \
\"urlpath\": \"%U\", \
\"urlquery\": \"%q\", \
\"method\": \"%m\", \
\"bytes\": %B, \
\"vhost\": \"%v\" \
}" logstash_apache_json

CustomLog /var/log/apache2/logstash_access_log logstash_apache_json
</code>
Answer: Hopefully this helps others. The original thread is from https://stackoverflow.com/questions/525057/why-cant-i-get-apaches-customlog-directive-to-work
Workaround 1
Delete the CustomLog directive from your configuration file, if you have it set up in a separate configuration file.
Add the CustomLog directive to your site's VirtualHost entry. For me this was in /etc/apache2/sites-available/default-ssl because I'm only allowing SSL access to the Subversion repository. If you are using /etc/apache2/sites-available/default, you will want to edit that file instead (or in addition to default-ssl if you are using both).
Restart Apache:
sudo /etc/init.d/apache2 restart
Pro
You can have multiple CustomLog directives in the VirtualHost entry for your site, and they will all work.
Con
You have to move your CustomLog entry into your site's VirtualHost entry instead of having it in a separate configuration file.
Workaround 2
Comment out the CustomLog directive(s) in your site's VirtualHost entry. If you're using one of the default site configurations (default or default-ssl), there will be a CustomLog directive for the access log that you will need to comment out (yes, this turns off Apache's default access logging).
Add your CustomLog directive to the appropriate configuration file. For me this was /etc/apache2/mods-available/dav_svn.conf.
Restart Apache:
sudo /etc/init.d/apache2 restart
Pro
You can keep your CustomLog directive in a separate configuration file.
Con
This workaround has the obvious disadvantage that you have to disable Apache's default access logging, but for me I don't care that much since I'm only using the server for Subversion access.
Conclusion
Neither of these workarounds are really ideal, but so far I haven't found a way to get it to work other than than the two workarounds above. I suppose we'll have to wait for the next release of Apache for this issue to be fixed.

Apache LogFormat & CustomLog


For real variables, variance is to entropy, what the mean is to -?

Question: If $X$ is a real random variable with a pdf, variance/standard deviation is a measure of $X$'s dispersion about the pdf's central tendency, which in turn is referred to as the mean of $X$. For many, variance acts as a measure of risk and uncertainty.
Shannon entropy is a measure of the unpredictability of $X$, and is frequently compared with and does have mathematical links to variance. The estimated entropy of $X$'s pdf has been shown to scale alongside variance, in that, as $X$ is transformed by scaling it with increasing variance, entropy will correspondingly shift or grow in value, indicating more spread in the distribution like how the increasing variance indicates more dispersion. Transformations of the mean, however, will not impact the entropy rate.
What is the information analogue of the mean then?
If there is none, does this show a fundamental weakness in information theory? Is entropy and mutual information's inability to distinguish between negative and positive source data for its input probabilities make these measures blind/ignorant to 'the facts of the data', making them uninferrable with respect to one of the most important distributional concepts, central tendency/location?
Comment: @Richard The claim is not that entropy scales in direct proportion to the variance; it's a little subtler, as I explain in the linked thread.  When $X$ is scaled (say by a factor $\sigma\gt 0$), both the variance and the exponential of the entropy change in simple, predictable ways that are strictly increasing functions of the scale factor: the variance is multiplied by $\sigma^2$ and the exponential entropy $\exp(H)$ is multiplied by $\sigma.$ This means, of course, that $H$ is *shifted* by $\log\sigma,$ strongly indicating there is no analog of the mean for $H.$
Comment: Entropy does not generally scale alongside variance, because the mapping from the random phenomenon on which entropy is defined to a random variable on which variance is defined can vary a lot. I can map a coin throw to a random variable $X$ with possible values $\{0,1\}$ or $Y$ with possible values $\{0,2\}$. The variance has quadrupled when moving from $X$ to $Y$, but the entropy of the coin throw stays the same.
Comment: @Richard There is no claim made that entropy and variance are universally related. For any given values $V$ of variance and $H$ of entropy corresponding to some distribution, scaling the distribution by $\lambda\gt 0$ changes the ordered pair $(\sqrt{V},\exp(H))$ into $\lambda (\sqrt{V},\exp(H)).$ Equivalently, the ratio $\exp(H)/\sqrt{V}$ is *invariant* under scaling. That's incontroverible, clear, insightful, and useful. The difference here is that you are discussing discrete distributions--but that's not relevant to the question, *which explicitly assumes the distribution is continuous.*
Comment: is information theory lacking something crucial by not having a match for the mean
Comment: @whuber, I think my example illustrates that we cannot say anything about the relationship between entropy and variance unless we fix the mapping between the random phenomenon and the corresponding random variables. Your answer seems to apply conditionally on having fixed such a relationship.
Comment: Related: ["How to include the observed values, not just their probabilities, in information entropy?"](https://stats.stackexchange.com/questions/483535).
Comment: It seems to me that the obvious way to complete this analogy is compute a "mean" using entropy. (And how you might go about trying to do that is not obvious, at least to me, as I haven't studied the relevant material).
Comment: @whuber, I think I overlooked the last point, sorry.
Comment: from the discussion it seems that information measures are locationless
Answer: Variance is not similar to entropy at all. The variance has a scale, and entropy doesn't. Yes, the variance measures the uncertainty, but in a very different manner compared to the entropy. The variance, and especially its cousin standard deviation, reflect the absolute uncertainty. For instance, you could say that the volatility of S&P 500 index annual return is 20% in an average year. However, you can model the returns with different distributions, such as Gaussian or Student-t, then you'd get different entropies for the same reading of the volatility.
The mean also has a scale, the same scale as the dispersion (volatility). So you are looking for a metric that somehow reflects the location of the distribution but without a scale, i.e. relative to all other values. The only thing that comes to mind is skewness. It's a shape parameter, that places the location relative to the entire domain of the variable.
Comment: OK, but this is about information entropy and I am not sure that is the same as entropy in physics!
Comment: differential entropy has a scale, see https://stats.stackexchange.com/questions/415435/how-does-entropy-depend-on-location-and-scale
Comment: @kjetilbhalvorsen, in physics entropy has a unit of measure Joules per Kelvin
Comment: @kjetilbhalvorsen, Shannon entropy definition is Gibbs entropy equation multiplied by Boltzmann constant $k$ which is measured in Joules/Kelvin, see Eq 6 at https://scholar.harvard.edu/files/schwartz/files/6-entropy.pdf It's almost as he simply applied thermodynamics to signals :) $S=-k\sum_iP_i\ln P_i$
Comment: Yes, but I am unsure if that is relevant at all! As a test, do you have a physics interpretation of the formula (by user whuber) in the post I linked?
Answer: There are other measures of central tendency besides the mean. The median and mode are also measures of central tendency. Unlike the mean and median, the mode can be generalized to information theory: the value in the domain with the highest probability or probability density. So information theory does have an analog to central tendency, even if it does not have a direct analog to the mean.
The mean and variance are special-purpose tools designed for use with distributions over the real number line. As such, they incorporate assumptions intrinsic to the real number line such as the metric distance between two points. Information theory, on the other hand, constitutes a collection of general-purpose tools designed for use with distributions over arbitrary spaces. It can be applied to distributions over the real number line, but it just as well be applied to categorical data for which there is no intrinsic ordering. Inevitably, special-purpose tools will better fit the contexts for which they are designed than general-purpose tools.
So you're right that the tools provided by information theory don't necessarily stack up to the mean and variance when applied to real-valued data. But that doesn't mean that information theory is broken or incomplete.
In summary, use the tools that are best suited to the problem you are solving, and your concerns about them having relative weaknesses to each other will become moot.


Statistics and causal inference?

Question: In his 1984 paper "Statistics and Causal Inference", Paul Holland raised one of the most fundamental questions in statistics:

What can a statistical model say about
  causation?

This led to his motto:

NO CAUSATION WITHOUT MANIPULATION

which emphasized the importance of restrictions around experiments that consider causation.
Andrew Gelman makes a similar point:

"To find out what happens when you change something, it is necessary to change it."...There are things you learn from perturbing a system that you'll never find out from any amount of passive observation.

His ideas are summarized in this article.
What considerations should be made when making a causal inference from a statistical model?
Comment: To much to say. But you can read Pearl's book "Causality" (2002, but newer 2nd edition), or Hernan and Robins' book "Causal Inference" (2015, free electronic draft online if you search).
Comment: great question: see also this related question on correlation and causation http://stats.stackexchange.com/questions/534/under-what-conditions-does-correlation-imply-causation
Comment: Read this: [Experimental and Quasi-Experimental Designs for Generalized Causal Inference (Shadish, Cook, and Campbell, (2002)](http://www.amazon.com/Experimental-Quasi-Experimental-Designs-Generalized-Inference/dp/0395615569/ref=sr_1_1?ie=UTF8&s=books&qid=1284747633&sr=8-1)
Answer: This is a broad question, but given the Box, Hunter and Hunter quote is true I think what it comes down to is

The quality of the experimental design:

randomization, sample sizes, control of confounders,...

The quality of the implementation of the design:

adherance to protocol, measurement error, data handling, ...

The quality of the model to accurately reflect the design:

blocking structures are accurately represented, proper degrees of freedom are associated with effects, estimators are unbiased, ...

At the risk of stating the obvious I'll try to hit on the key points of each:

is a large sub-field of statistics, but in it's most basic form I think it comes down to the fact that when making causal inference we ideally start with identical units that are monitored in identical environments other than being assigned to a treatment.  Any systematic differences between groups after assigment are then logically attributable to the treatment (we can infer cause).  But, the world isn't that nice and units differ prior to treatment and evironments during experiments are not perfectly controlled.  So we "control what we can and randomize what we can't", which helps to insure that there won't be systematic bias due to the confounders that we controlled or randomized.  One problem is that experiments tend to be difficult (to impossible) and expensive and a large variety of designs have been developed to efficiently extract as much information as possible in as carefully controlled a setting as possible, given the costs.  Some of these are quite rigorous (e.g. in medicine the double-blind, randomized, placebo-controlled trial) and others less so (e.g. various forms of 'quasi-experiments').
is also a big issue and one that statisticians generally don't think about...though we should.  In applied statistical work I can recall incidences where 'effects' found in the data were spurious results of inconsistency of data collection or handling.  I also wonder how often information on true causal effects of interest is lost due to these issues (I believe students in the applied sciences generally have little-to-no training about ways that data can become corrupted - but I'm getting off topic here...)
is another large technical subject, and another necessary step in objective causal inference.  To a certain degree this is taken care of because the design crowd develop designs and models together (since inference from a model is the goal, the attributes of the estimators drive design). But this only gets us so far because in the 'real world' we end up analysing experimental data from non-textbook designs and then we have to think hard about things like the appropriate controls and how they should enter the model and what associated degrees of freedom should be and whether assumptions are met if if not how to adjust of violations and how robust the estimators are to any remaining violations and...

Anyway, hopefully some of the above helps in thinking about considerations in making causal inference from a model.  Did I forget anything big?
Comment: A huge plus one for point 2.  Other than going through human subjects protection training, I've never received the tiniest bit of training on data collection and storage.  Getting the data collection right is vastly more important than the analysis.
Comment: I'd love to answer too, but I'm afraid there's nothing left to add to what Kingsford said.
Answer:
What can a statistical model say about causation? What considerations should be made when making a causal inference from
  a statistical model?

The first thing to make clear is that you can't make causal inference from a purely statistical model. No statistical model can say anything about causation without causal assumptions. That is, to make causal inference you need a causal model.
Even in something considered as the gold standard, such as Randomized Control Trials (RCTs), you need to make causal assumptions to proceed. Let me make this clear. For example, suppose $Z$ is the randomization procedure, $X$ the treatment of interest and $Y$ the outcome of interest. When assuming a perfect RCT, this is what you are assuming:

In this case $P(Y|do(X)) = P(Y|X)$ so things are working well. However, suppose you have imperfect compliance resulting in a confounded relation between $X$ and $Y$. Then, now, your RCT looks like this:

You can still do an intent to treat analysis. But if you want to estimate the actual effect of $X$ things are not simple anymore. This is an instrumental variable setting, and you might be able to bound or even point identify the effect if you make some parametric assumptions.
This can get even more complicated. You may have measurement error problems,  subjects might drop the study or not follow instructions, among other issues.  You will need to make assumptions about how those things are related to procede with inference. With "purely" observational data this can be more problematic, because usually researchers will not have a good idea of the data generating process.
Hence, to draw causal inferences from models you need to judge not only its statistical assumptions, but most importantly its causal assumptions. Here are some common threats to causal analysis:

Incomplete/imprecise data
Target causal quantity of interest not well defined (What is the causal effect that you want to identify? What is the target population?)
Confounding (unobserved confounders)
Selection bias (self-selection, truncated samples)
Measurement error (that can induce confounding, not only noise)
Misspecification (e.g., wrong functional form)
External validity problems (wrong inference to target population)

Sometimes the claim of absence of these problems (or the claim to have addressed these problems) can be backed up by the design of the study itself. That's why experimental data is usually more credible. Sometimes, however, people will assume away these problems either with theory or for convenience. If the theory is soft (like in the social sciences) it will be harder to take the conclusions at face value.
Anytime you think there's an assumption that can't be backed up, you should assess  how sensitive the conclusions are to plausible violations of those assumptions --- this is usually called sensitivity analysis.
Comment: This is the best answer.
Comment: Would it be equivalent to replace the dashed bidirectional arrow with two one-directional solid arrows from an additional node?
Comment: @Taylor yes, a latent (unobserved) additional node.
Answer: In addition to the excellent answer above, there is a statistical method that can get you closer to demonstrating causality.  It is Granger Causality that demonstrates that one independent variable occurring before a dependent variable has a causal effect or not.  I introduce this method in an easy to follow presentation at the following link:
http://www.slideshare.net/gaetanlion/granger-causality-presentation
I also apply this method to testing competing macroeconomic theories:
http://www.slideshare.net/gaetanlion/economic-theory-testing-presentation
Be aware that this method is not perfect.  It just confirms that certain events occur before others and that those events appear to have a consistent directional relationship.  This seems to entail true causality but it is not always the case.  The rooster morning call does not cause the sun to rise.

Statistics and causal inference?


What's the difference between enabling Node View (and adding a variant) and creating a new Panel page?

Question: To learn more about Panels, I've been repeatedly traipsing through screencasts by Shane Thomas and Johann Falk. Right now I find myself confused about the distinction between: enabling a copy of node_view so one can add a variant for a content type, and, creating a new panel page for that content type.
Code Karate's screencast series is directed to learning about Panels, but #129 takes a side-step into creating a copy of node_view Page (through Page Manager) to create a variant, selecting by content type. I wonder why Page manager is being used for a tutorial about Panels.
I want a page for a specific content type, but I would like to use a Panel page to get there. How do I do this?
I'm also not clear on what invokes a Panel.
Note: I've read this question/answer by @Letharion about the difference between Page manager and Panels which is a bit too high-level for me at this stage.
Answer: Page Manager allows you to intercept existing paths or create new paths in the system.
As the node view path already exists, and you want to change the way it operates depending on some conditions (bundle for instance) you need to enable a copy of node view path in Page Manager, instead of creating a new one (as it already exists you cannot create a new one as it will conflict)
EXAMPLE: For all nodes of bundle type Article, when the user attempts to view such an article, you want to perform some ACTION. So you override the node view path, create a variant that filters on bundle of Article, then perform some ACTION.
The ACTION is pluggable. You can do a redirect for instance. So when the user attempts to view an Article you intercept that, look at some context on the specific article (lets say there is an Entity Reference field to a Basic Page), then redirect to that Page.
Panels has a plugin component for Page Manager. So when your condition to view an Article for instance is satisfied in Page Manager (via your variant), you can select that Panels take over the rendering of the page.
The beauty of Page Manager is that it intercepts the requested page and then makes all kinds of context on that page available, such that you can make decisions (via created variants) on what to do, and when you for instance make the decision to pass the rendering over to Panels, you have all the context available to you. You can even load more context if you wish. On a node you have limited author information available such as the Users uid, but with Page Manager you can request that it pulls in the User entity to make other fields available such as email etc. You can continue to bring in related context in this fashion: think about the case of a node that has an entity relationship to another type of node that has an author (User) that has a taxonomy reference etc. Access to more context is very helpful. Think of the difficulty with a Views Block. It only has access to the menu object's key (node id etc) and url parameters. But in Panels (via Page Manager) you can embed a View Content Pane and pass as much context to it as needed.
SO in essence, the Page Manager - Panel relationship is merely the way by which you decide what paths and what conditions on that paths must be satisfied in order for Panels to take over the rendering with access to lots of context.
Comment: When you create the variant there is an option for 'Variant Type', with default 'HTTP response code' (ie redirect, page not found, access denied). When you install Panels a new Variant type option becomes available, nl: Panel
Comment: Thanks for that explanation, I shall ponder this. How **exactly** does the designer "select that Panels take over the rendering of the page"?


Parameter estimation in a drifting normal distribution

Question: Suppose we observe $n$ pairs of points $(a_1,b_1),~(a_2,b_3)~..~(a_n,b_n)$. The underlying data generating process is known to be as follows:

$u_i \sim N(0,1)$ and $a_i \sim N(u_i,1)$
$b_i \sim N(u_i + k*a_i, 1)$

independently for each $i$ and $k$ is a constant. What is a good unbiased estimator for $k$?
Comment: Haha. It would be interesting to see how to estimate the drift in general. Say if we have $b \sim N(u+f(a), 1)$ for smooth function $f$ and we need to estimate it. Maybe your methods will come handy then?
Comment: We have $u = n_0$, $a = n_0 + n_1$, $b = n_0 + k*(n_0 + n_1)+ n_2 = (1+k)*n_0 + k*n_1 + n_2$ where $n_i$'s are all standard normals. Variance of $b$ then is $(1+k)^2+k^2+1$. Equating this with empirical estimate of variance and solving gives you a pretty good estimate for $k$. But it is not unbiased.
Comment: @bleh So simple, but so efficient... I developped formulas based on quadratic optimisations of bayesian estimators...
Comment: I tried to write a very comprehensive answer. @bleh, user363943, may I kindly ask you to help me to check that what I wrote is correct and contains no typo?
Answer: I am going to start to discuss @bleh's idea in order to explain why my solution seems more robust.

@bleh's idea:

Assume:

$N_{i,j} \sim N(\mu_j,\sigma^2_j)$ are independent random variables $\forall i,j$
$U_i=N_{i,1}$
$A_i\sim N(U_i,\sigma_2^2)\Rightarrow A_i=N_{i,1}+N_{i,2}$
$B_i \sim N(U_i+kA_i,\sigma_3^2)\Rightarrow B_i=(1+k)N_{i,1}+kN_{i,2}+N_{i,3}$

As a consequence, $B_i\sim N((1+k)\mu_1+k\mu_2+\mu_3,k^2(\sigma^2_1+\sigma^2_2)+2k\sigma^2_1+\sigma^3_1)$
It is tempting to compute an estimator $\widehat{\sigma^2_B}$ of the variance of $B$ based on the squared differences from the means $\sum_{i=1}^n\left(b_i-\mathbb{E}(B_i)\right)^2$, to equate it to $\mathbb{V}ar(B)$, and to solve for $k$.
This is a problem for 2 main reasons:

To solve this polynomial, the value of the discriminant $\Delta$ will depend on the parameters $\sigma^2_j$ as well as on the estimator $\widehat{\sigma^2_B}$. However, in some unlikely cases (ex: very small $\widehat{\sigma^2_B}$ being computed despite $\sigma^2_3$ being large), the calculation will fail (due to $\Delta<0$) because the observed values $b_i$ were not dispersed enough compared to the expected variance $\sigma^2_3$.
Even if this process worked, determining an estimator $\widehat{\sigma^2_B}$ that would allow us to find an unbiased estimator for $k$ after all this steps, including the calculations of squared deviations to a mean, and a square root, would be very difficult.

In addition, we would end up loosing some information because we would use our observations $b_i$, but not $a_i$.
Hence, I propose my solution:

Use bayesian statistics and the normal conjugate prior.

You can see here that:
If $\mu \sim N(\mu_0,\sigma^2_0)$ and $X_i\sim N(\mu,\sigma^2)$ (the expected value of $X_i$ depends on a random value $\mu$), then:

$\mu\vert x_i, i=1,...,n \sim N(\mu_0\prime, \sigma^{2}_0\prime)$, with $\mu_0\prime = \frac{1}{\frac{1}{\sigma^2_0}+\frac{n}{\sigma^2}}\left(\frac{\mu_0}{\sigma^2_0}+\frac{\sum_{i=1}^{n}x_i}{\sigma^2}\right)$ and $\sigma_0^2\prime=\frac{1}{\frac{1}{\sigma_0^2}+\frac{n}{\sigma^2}}$ (more observations of $x_i$ allow to estimate a more accurate distribution for $\mu$)
$X_{i+1} \sim N(\mu_0\prime, \sigma^{2}_0\prime + \sigma^2)$ (updated knowledge about $\mu$ allow to estimate a more accurate distribution for future observations of $X$)

Apply this with your example:

$U_i\sim N(\mu_U,\sigma_U^2)$
$A_i \sim N(U_i, \sigma_A^2)$
$B_i \sim N(U_i+kA_i, \sigma_B^2)$

So this is a "tower" of normal variables using the previous variables as conjugate prior.
The observations we have are the couples $\{a_i,b_i\}$.
Let's start by the end:
$B_i\vert A_i \sim N(U_i+kA_i, \sigma_B^2)\vert A_i$
In this "equation", we know $A_i$, but not $U_i$. But based on our bayesian knowledge, we can determine the distribution of $U_i \vert A_i$:
Based on the equations above:
$$U_i \lvert A_i \sim N\left(\tilde{\mu_i}=\frac{1}{\frac{1}{\sigma_U^2}+\frac{1}{\sigma_A^2}}\left(\frac{\mu_U}{\sigma_U^2}+\frac{A_i}{\sigma_A^2}\right),\tilde{\sigma^2_{U,i}}=\frac{1}{\frac{1}{\sigma_U^2}+\frac{1}{\sigma^2_A}}\right)$$
Hence, you will have:
$$B_i\vert A_i \sim N(kA_i+\tilde{\mu_i},\sigma^2_B+\tilde{\sigma^2_{U,i}})$$
Hence, $\mathbb{E}(B_i\vert A_i)=kA_i+\tilde{\mu_i}$, and we can propose a range of estimators $\hat{k}_i=\frac{b_i-\tilde{\mu_i}}{a_i}$. They are unbiased, and each has a variance which can be computed with the parameters above.
Since you have $i$ estimators $\hat{k}_i$ with variance $\sigma^2_{k, i}=\frac{\sigma^2_B+\tilde{\sigma^2_{U,i}}}{a_i^2}$, you can determine a linear combination of these estimators minimising the variance of the final estimator:
$\min_w w' \Sigma w$, st $w'\mathbb{1}=1$ where $\Sigma$ is the covariance matrix of the estimators $\hat{k}_i$.
Finally, the unbiased estimator with the smallest variance will be:
$$w^*=\frac{\Sigma^{-1}\mathbb{1}}{\mathbb{1}'\Sigma^{-1}\mathbb{1}}$$