Java 8e: Logic error and Array Index out of bounds Exception in a program

Question: So i have written this piece of code for a class project, and it has two issues.

When run, displays is a ArrayIndexOutOfBoundsException: 10
The program does not function properly when run (detailed below) Currently i cannot identify either the exception, or the logic error.

It may be possible that the two are related. Help is humbly requested :)
The program should take 10 input values, then separate them by weather or not they are negative, odd or even. These values should be put into arrays, and the values of the arrays should be outputted as 3 separate columns, with the number of rows limited by the length of the longest array. Currently with the input values of 10 9 8 7 6 5 4 -3 -2 -1 the output is 1 row, reading -2 -1 -1.
import java.util.Scanner;

public class APComSci9p1 {//Start program

    public static void main(String[] args) {//Start Main

    //Initilize inputs
    Scanner scnKey = new Scanner(System.in);

    //Initialize Arrays
    int[] aOriginValues,
          aEvenList,
          aOddList,
          aNegitiveList;

    aOriginValues = new int[10];
    aEvenList = new int[10];
    aOddList = new int[10];
    aNegitiveList = new int[10];

    //Initialize Array index Variables
    int iInput;
    int iListControlE = 0;
    int iListControlO = 0;
    int iListControlN = 0;

    //Start Loop
    for (iInput = 0; iInput <= 9; iInput++){
        System.out.print("Input value: ");
        aOriginValues[iInput] = scnKey.nextInt();

            //Start if/else
            if ((aOriginValues[iInput] % 2) == 0){
                aEvenList[iListControlE] = aOriginValues[iInput];
            }else {
                aOddList[iListControlO] = aOriginValues[iInput];
            }//end if/Else
            if (aOriginValues[iInput] < 0){
                aNegitiveList[iListControlN] = aOriginValues[iInput];
            }//end if
        }//End loop

    //Initialize Loop Specific variables
    int iLimit;

    //Start if/else #2
    if((aEvenList.length >= aOddList.length) && (aEvenList.length >= aNegitiveList.length)){
        iLimit = aEvenList.length;
    }else if((aOddList.length >= aEvenList.length) && (aOddList.length >= aNegitiveList.length)){
        iLimit = aOddList.length;
    }else{
        iLimit = aNegitiveList.length;
    }//End if/else #2

    System.out.println("    Even        Odd       Negitive  ");
    for (int iControl = 0; iControl <= iLimit; iControl ++){
        System.out.printf("%,12d%,11d%,12d%n", aEvenList[iControl], aOddList[iControl], aNegitiveList[iControl]);
    }

    }//End Main
}//End program

Comment: The stack trace of the exception also tells you the line number where it happens. You are trying to access index 10 in an array that does not have this index.
Comment: I realize what the error is saying, i just cannot find where it is happening. To me (a biased source i know) i cannot see where i am referencing an index of 10 in a array with 10 values.
Answer:
aEvenList[iListControlE] should actually be aEvenList[iListControlE++] similarly other arrays countor are not set correctly.
Your iLimit value is incorrectly being set.

The correct version is below,
import java.util.Scanner;

public class APComSci9p1 {// Start program

    public static void main(String[] args) {// Start Main

        // Initilize inputs
        Scanner scnKey = new Scanner(System.in);

        // Initialize Arrays
        int[] aOriginValues, aEvenList, aOddList, aNegitiveList;

        aOriginValues = new int[10];
        aEvenList = new int[10];
        aOddList = new int[10];
        aNegitiveList = new int[10];

        // Initialize Array index Variables
        int iInput;
        int iListControlE = 0;
        int iListControlO = 0;
        int iListControlN = 0;

        // Start Loop
        for (iInput = 0; iInput <= 9; iInput++) {
            System.out.print("Input value: ");
            aOriginValues[iInput] = scnKey.nextInt();

            // Start if/else
            if ((aOriginValues[iInput] % 2) == 0) {
                aEvenList[iListControlE++] = aOriginValues[iInput];
            } else {
                aOddList[iListControlO++] = aOriginValues[iInput];
            } // end if/Else
            if (aOriginValues[iInput] < 0) {
                aNegitiveList[iListControlN++] = aOriginValues[iInput];
            } // end if
        } // End loop

        // Initialize Loop Specific variables
        int iLimit;

        // Start if/else #2
        if ((iListControlE >= iListControlO)
                && (iListControlE >= iListControlN)) {
            iLimit = iListControlE;
        } else if ((iListControlO >= iListControlE)
                && (iListControlO >= iListControlN)) {
            iLimit = iListControlO;
        } else {
            iLimit = iListControlN;
        } // End if/else #2

        System.out.println("    Even        Odd       Negitive  ");
        for (int iControl = 0; iControl <= iLimit; iControl++) {
            System.out.printf("%,12d%,11d%,12d%n", aEvenList[iControl],
                    aOddList[iControl], aNegitiveList[iControl]);
        }

    }// End Main
}// End program

In general, you should learn how to debug your application.
Comment: You forgot to place `iListControlO` in place of `aOddList.length` inside `if` clause and similarly for other vars to set value of iLimit.
Comment: Thank you! Now, why do both edits work? I was unaware that adding the ++ was necessary for this problem. Similarly, what you did to redefine `iLimit` is making little sense to me right now.


Implementing raw code for back-propagation neural network

Question: I want to implement back-propagation neural network algorithm in raw code, not using any library as my interest to understand in depth operations/customization of this algorithm.
In this regard, I need to follow step-by-step simulation of back-propagation algorithm. More precisely, what code I should write in R language to each of steps in back-propagation neural network. Can anyone show me the close explanation of steps in this algorithm.
I don't need any code. Just show me what operations is done in each steps.

FYI: for reviewers, this is not my assignment. I'm doing my assignments using neuralnet library. Please leave a comment before any report.
Comment: Would a step by step derivation of a vanilla neural network help you?
Comment: @ArmenAghajanyan Can you provide any example?
Answer: If you just want to see how the internals work, the best resource for a beginner, that I have found, is this series of videos by Welch Labs.  It goes through each step of a basic feed-forward backpropogation model including the calculus steps so you understand what is happening during each step.  It is all done in Python but you can definitely convert it to R if that is your goal.  That would be a great learning exercise.


How to get aggregate volume usage in bytes of my MySQL DB's

Question: How can I get the aggregate volume usage for all of my MySQL DB's on my server?  We have 55 DB's and I am wondering if there is a nice, easy way to get this.
Answer: One of the most effective ways is to look at the raw file size..
It takes into account your indexs and data..
Which in fact is a good idea ( seeming that your index's need to be read, and hopefully all fit into memory )
du -h /var/lib/mysql/then your dbs
Are you running on myisam or innodb??
Answer: du -hcs /var/lib/mysql for a summary of size on disk for everything
du -hcs /var/lib/mysql/* this is good for myISAM engine schemas for innoDB the id_data file indicates size on disk for all innoDB files.


Is it logical to square this expression?

Question: $y={\dfrac{[3x+1]}{[4x-1]^\frac{1}{2}}}$
Is it logical to square both the numerator and denominator to cancel out the power to $\frac12$ in the denominator?
Edit: This expression is being used for a chain rule application.
Comment: It's the square root only pertinent to the denominator?
Comment: Yes. That's the way the question is set up.
Comment: If you square one side, you need to square the other side. Are you trying to differentiate?
Comment: What exactly do you want to do? Just get rid of the square root? Then yes, of course you can square the expression.
Comment: This expression is for a chain rule application. Having a square root in the denominator while trying to calculate dy/dx doesn't make the process easy. Thus, would it be reasonable to square on both sides of the fraction to make calculating dy/dx simpler?
Comment: If you want to differentiate, I believe logarithmic differentiation is the best way to proceed.
Comment: Note that $\dfrac{a^2}{b^2} = \left(\dfrac ab\right )^2$
Answer: Assuming those are meant to be normal parenthesis around the numerator and denominator, squaring both top and bottom of the fraction completely changes the equation. That is,
$y={\dfrac{3x+1}{(4x-1)^\frac{1}{2}}} \neq {\dfrac{(3x+1)^2}{\left((4x-1)^\frac{1}{2}\right)^2}}$
Edit 1:
If you did want to square both top and bottom, not that you would have to square both sides:
$y^2= \left( {\dfrac{(3x+1)}{(4x-1)^\frac{1}{2}}} \right) ^2 = {\dfrac{(3x+1)^2}{\left((4x-1)^\frac{1}{2}\right)^2}}$
Edit 2: If your main concern is differentiating, I really encourage you to follow Nilabro's advice in your comments -- logarithmic differentiation. We would have
$y={\dfrac{3x+1}{(4x-1)^\frac{1}{2}}}$
$\ln (y)= \ln \left( {\dfrac{3x+1}{(4x-1)^\frac{1}{2}}} \right)$
which by using the properties of logarithms results in
$\ln (y)= \ln (3x+1) - \dfrac{1}{2} \ln (4x-1)$
then differentiating and solving for $y'$ gives
$\dfrac{1}{y} \ y'= \dfrac{3}{3x+1} - \dfrac{1}{2} \left( \dfrac{4}{4x-1} \right)$
$ y'= y \left( \dfrac{3}{3x+1} - \dfrac{1}{2} \left( \dfrac{4}{4x-1} \right) \right)$
$ y'= \left( {\dfrac{3x+1}{(4x-1)^\frac{1}{2}}} \right) \left( \dfrac{3}{3x+1} - \dfrac{1}{2} \left( \dfrac{4}{4x-1} \right) \right)$
$y' = \dfrac{6x-5}{(4x-1)^\frac{3}{2}},$
which is the same result randomgirl came to.
Answer: You could square both sides but the problem turns into implicit problem instead of an explicit one.
$y=\frac{3x+1}{(4x-1)^\frac{1}{2}} \\\ $
We can square both sides:
$y^2=\frac{(3x+1)^2}{4x-1}$
Differentiate both sides:
$2yy'=\frac{[(3x+1)^2]'(4x-1)-[(4x-1)]'(3x+1)^2}{(4x-1)^2} \\\ $
$2 yy'=\frac{[2(3x+1) \cdot 3](4x-1)-[4](3x+1)^2}{(4x-1)^2} \\\ $
$2 yy'=\frac{6 (3x+1)(4x-1)-4(3x+1)^2}{(4x-1)^2} \\\ $
$2 yy'=(3x+1) \frac{6(4x-1)-4(3x+1)}{(4x-1)^2} \\$
Divide both sides by $2y$
$y'=\frac{3x+1}{2y} \frac{24x-6-12x-4}{(4x-1)^2} \\$
$y'=\frac{3x+1}{2y} \frac{12x-10}{(4x-1)^2} \\$
$y'=\frac{3x+1}{2} \frac{1}{y} \frac{12x-10}{(4x-1)^2} \\$
$y'=\frac{3x+1}{2} \frac{(4x-1)^\frac{1}{2}}{3x+1} \frac{12x-10}{(4x-1)^2} \\$
$y'=\frac{(4x-1)^\frac{1}{2}}{2} \frac{2(6x-5)}{(4x-1)^2} \\$
$y'=(4x-1)^\frac{1}{2} \frac{6x-5}{(4x-1)^2} \\$
$y'=\frac{6x-5}{(4x-1)^\frac{3}{2}}$


Displaying Ajax Sample Results as the Form Loads

Question: I've set up a box to display results from an Ajax form, showing water data from US Geological Survey and using a Google chart.  Works great, but I want users to see some sample results when they first load the form instead of an empty box. Not sure how to do it. Here's the code for the form:

Current River Levels

<form>
<select name="users"; onchange="showUser(this.value); drawChart(this.value)";>
<option value="01598500">North Branch at Luke, MD</option>
<option value="01603000">North Branch at Cumberland, MD</option>
<option value="01610000">Potomac River at Paw Paw, WV</option>
<option value="01613000">Potomac River at Hancock, MD</option>
<option value="01619500">Antietam Creek at Sharpsburg, MD</option>
<option value="01638500">Potomac River at Point of Rocks, MD</option>
<option value="01648000">Rock Creek at Sherrill Drive, DC</option>
<option value="01646500">Little Falls, DC</option>
</select>
</form>

##

Select a site or visit our interactive map.

The entire project is located here: http://www.riverexplorer.com/potomax/json_test_20.php
Comment: Just load some sample data on page-load ...
Answer: Trigger the onchange event when the document is ready.
$(function(){
    $('select[name="users"]').trigger('change');
 });

Comment: Thanks! Your answer led me this page, which had just what I needed...   https://api.jquery.com/change/
Comment: You are welcome. Mark the answer as correct so we can get some rep points


Poisson parameter interval estimation vs. the CLT: why not simple?

Question: I've never been much of a mathematician, and I'm now trying to catch up on some things I should have studied properly many years ago. So forgive me if the question is naive!
I'm relearning statistics, and in particular estimations and hypothesis testing. I have one doubt about the application for the central limit theorem to estimate the mean of a population. Here's my thinking:

The CLT applies to any distribution. We can estimate the mean, and the estimator itself follows a normal distribution $N(\mu,\sigma^2/n)$. This variance on the estimator is approximated by $s^2/n$ where $s$ is the standard deviation of the samples. And I can use the mean of the samples to approximate the underlying population's mean.
If I have a few samples that I believe follows a Poisson distribution, such as the 27 values $2, 4, 1, 4, 0, 1, 3, 2, 1, 4, 1, 1, 0, 1, 2, 0, 2, 2, 0, 0, 3, 1, 0, 3, 4, 3, 2$, then I can estimate its parameter as the average of these values ($\hat\lambda=
1.74$) and use the approximation of the estimator's variance to create a 95% confidence interval around that. So my interval would be $\hat\lambda \pm 1.96*s/\sqrt n$, so $1.74 \pm 1.96*0.265$, so a range from $1.22$ to $2.26$.

However I never see such a simple process applied anywhere I look. Instead I see things I haven't looked into yet such as MLE etc. Which definitely look much more complex than this. But why? Is the above incorrect? Why would the CLT fail to deliver, since I'm only looking at estimating a mean? Am I misinterpreting what the mean is, here? Or is it just that it's not very precise and the other methods give a smaller interval?
As you can see from my very unclear explanations, I'm at this stage where I understand a few things but it's quite superficial. If someone takes the time to answer, do not hesitate to do so in a "for dummies" style... I'll be super grateful.
Note: I might be particularly confused by the Poisson distribution because I'm trying to catch up on the topic from the Fundamentals of Biostatistics (Rosner) book, in which the Poisson parameter is $\mu=\lambda t$, rather than the typical $\lambda$ on its own that I see when searching the web (in which case $\mu=\lambda$). This inclusion or not of the time element as a multiplier confuses a little bit the notion of what in this is really the mean and the variance.
Thanks!
P.
Comment: Regarding $\lambda t$ vs. $\lambda$ - the distinction only matters when you have different time intervals in play. Say, you have some data that records number of events in 10-minute intervals, and some more data that records number of events in 1-hour intervals.
Comment: Nitpick - but the CLT only applies to distributions which have finite variance.
Comment: @rubikscube09 this is not nitpicking, this is actually useful. I did want to follow up with the question "when does the CLT not apply, then?" since the textbook I'm referring to did say "any distribution". But being about biostatistics, that book might live in a world where variances are always finite?
Newbie followup question then: are there cases where the data is "not distributed" at all? or is every population distributed in some way even if it follows no particular rules etc.? (thinking of the revenue per session in an A/B test! which is very far from normal or any distribution I know)
Comment: @Pierric - it depends - all data in some sense follows an unknown distribution, or if you are saying, dealing with time series data (stock returns as an obvious example), this distribution can be changing with time/what we call non-stationary.
Comment: On the other hand, the CLT is agnostic about distribution, although most versions of it assume that 1. data is identically distributed, meaning that the distribution each sample is drawn from is the same, 2. data is independent, e.g. what you get on your first draw doesn't influence what you get on the second, and 3. that the variances of each distribution is finite, which is near impossible to check empirically.
Comment: Thanks @rubikscube09 ! I would think those assumptions apply to the context I'm most often interested in (A/B test on online conversions and generated revenue -- which has nothing to do with Poisson, I'm aware, I took a detour while rereading the textbook :) )
Answer: Either of the two methods you describe is correct as far as it goes. If you do many statistical analyses of this type, on average you expect at least $95\%$ of them to correctly give you an interval containing the true mean.
(For the CLT, the story is a bit more complicated; it does not apply to any distribution. It applies to the Poisson distribution, though, and better than usual, because the Poisson distribution is approimately normal when $\lambda$ is large. So we can treat it as a way to simplify some calculations in this case.)
MLE and other Bayesian estimators are more complicated and more precise answers to more complicated questions. In particular, the posterior distribution you get from this method can be used to answer any kind of question along the lines of "what is the probability that the rate $\lambda$ satisfies $X$?"
In addition to much more complicated calculations, things are more complicated because you need more input to make it work: a prior distribution for how likely you thought values of $\lambda$ are before doing the experiment. (But you can always use an uninformative prior if you have no idea, which is what MLE does.) This, too, is valuable: it makes it easy to combine several experiments, just by taking the output of one as input into the other.
Comment: Thank you so much for your answer Misha! (I've voted it but I don't have the reputation for it to show). Regarding $\lambda$ being large, I suppose that's not considered to be the case with a value of less than 2, right? Is my range therefore unreliable, in this particular case?
Comment: Sorry, rephrased. In your case, things are fine. When $\lambda$ is large, the normal is a good approximation to a *single* Poisson variable; this doesn't affect the CLT, though we could theoretically use that to get better error bounds in the CLT.
Comment: Thanks @Misha, this makes sense!


Covariance structure with one or more random intercepts

Question: I need to choose a suitable covariance structure including one or more random intercepts:
If one factor B is nested in another factor A, one option would of course be to consider a model with random intercepts of both A and B, corresponding to (1|A) + (1|B) when using lmer in the R package lme4 or providing the argument random = ~1|A/B to the lme function in the R package nlme, but would it make any sense to consider a model with only the random intercept of B, even if B is nested in A?
Answer: Let consider 2 levels in factor A and 2 individuals in each level of factor A, so totally 4 individuals. $i$ indicates level of factor A and $j$ for B.
When the two intercepts are used, we have
$$Y_{ij} = X\beta + \gamma_i + \gamma_j + \epsilon_{ij}$$
$Var(\gamma_i) = \sigma_A^2, Var(\gamma_j) = \sigma_B^2, Var(\epsilon) = \sigma^2$
Then $$Var\left(\begin{matrix}Y_{11}\\ Y_{12}\\Y_{21}\\Y_{22}\end{matrix}\right)=
\left(\begin{matrix}\sigma^2+\sigma_A^2 + \sigma_B^2 & \sigma_A^2 + \sigma_B^2 & \sigma_A^2&\sigma_A^2 \\ \sigma_A^2 + \sigma_B^2 & \sigma^2+\sigma_A^2 + \sigma_B^2 & \sigma_A^2&\sigma_A^2 \\ \sigma_A^2 & \sigma_A^2 & \sigma^2+\sigma_A^2 + \sigma_B^2 &\sigma_A^2 + \sigma_B^2\\ \sigma_A^2 & \sigma_A^2 & \sigma_A^2 + \sigma_B^2&\sigma^2+\sigma_A^2 + \sigma_B^2  \end{matrix}\right)
$$
If intercept for factor B only, we have
$$Y_{ij} = X\beta + \gamma_j + \epsilon_{ij}$$
Then $$Var\left(\begin{matrix}Y_{11}\\ Y_{12}\\Y_{21}\\Y_{22}\end{matrix}\right)=
\left(\begin{matrix}\sigma^2 + \sigma_B^2 & \sigma_B^2 & 0&0\\ \sigma_B^2 & \sigma^2+ \sigma_B^2 & 0&0\\ 0&0& \sigma^2+ \sigma_B^2 & \sigma_B^2\\ 0 & 0&  \sigma_B^2&\sigma^2 + \sigma_B^2  \end{matrix}\right)
$$
When two intercepts are in the model, $Y$ from the same individual has covariance $\sigma_A^2+\sigma_B^2$, $Y$ from different individuals but from same level of factor A has covariance $\sigma_A^2$, which is smaller than covariance from the same individual. Generally this assumption is reasonable.
When just individual level intercept is kept in the model, the covariance between the difference individual is zero, even they come from the same level of factor A. Generally, this is not what we want.
Answer: In general I would recommend including both (1|A) and (1|B) in the model - otherwise you're not fitting the maximal model, i.e. the model that includes all terms that are statistically identifiable. If the variation in A is reasonably large and you omit the (1|A) term, the model for the random effects corresponding to the remaining (1|B) term (that they are independently drawn from a zero-centred Normal distribution) might be off.  Some reasons it might be reasonable to omit the (1|A) term:

there is something about your data that would eliminate any differences in A (not just experimental control: for example, maybe you've already subtracted the mean from the observations in each A group)
you get a singular fit, i.e. an estimate of zero variance among levels of A (in this case you'll get exactly the same answer if you drop this term)
you think there are too few levels of A to estimate the variance (in this case it would be more conservative to include A in the model as a fixed effect
if you are using data-driven model selection to decide what terms to include, and the model including (1|A) is not selected (e.g. the AIC is higher). (In general you need to be very careful when doing this kind of "sacrificial pseudoreplication" (sensu Hurlbert 1984) that you don't mess up your coverage/type-I error ...)

(Note that the levels of B need to be uniquely coded - otherwise you should specify the nested syntax (1|A/B)  (or (1|A) + (1|A:B)) in lme4 as well in nlme ...)


Certain AST nodes not possible in Parsetree.signature (mli files)

Question: I'm writing a tool that generates ml stubs from mli files. I have the meaningful mappings complete (var f : int -> float to let f _ = 0.), but I'm having a little trouble reasoning about the AST nodes for classes and modules, specifically the Pmty_* and Pcty_* nodes. These two types of nodes (given the type hierarchy) seem to most most associated with mli files. Some of the trivial members--which appear in -dparsetrees of mli files (like Pmty_ident and Pcty_constr) have obvious mappings to nodes most associated with ml files (again by the hierarchy of the signature and structure types, in the aforementioned example to Pmod_ident and Pcl_constr). However, some of the nodes don't have an obvious parallel. Specifically, I'm having trouble reasoning about:

Pmty_with and Pmty_typeof - it seems like these two can never be parsed from a valid mli file; they only occur in a structure context that has a module_type as one of its members (I've checked my suspicions against all of OCaml's parser test files)
Pcty_signature - it seems like the only case this can occur is within a Psig_class_type (which I already directly map to Pstr_class_type); I against the OCaml test files and found a Pcty_signature only in this location, but are there other valid locations for it in an mli that I am missing?
Pcty_arrow - there are no test files containing this, so I can't be sure where it is valid, but my intuition says this also goes in a class_type context within a structure only (or within one of the other Pcty_*, in which case its conversion would be handled as a child by that node's mapping from Pcty_* to Pcl_*); is this incorrect?

I'm fairly deep into this and don't completely understand what's going on with all of these advanced language features and the AST nodes representing them, so here's an attempt at a simpler explanation of my questions:
Relevant types extracted from Parsetree:
type module_type_desc =
  (* .. snip .. *)
  | Pmty_with of module_type * with_constraint list
        (* MT with ... *)
  | Pmty_typeof of module_expr
        (* module type of ME *)

type class_type_desc =
  (* .. snip .. *)
  | Pcty_signature of class_signature
        (* object ... end *)
  | Pcty_arrow of arg_label * core_type * class_type
        (* T -> CT       Simple
           ~l:T -> CT    Labelled l
           ?l:T -> CT    Optional l
         *)

I believe that Pmty_with and Pmty_typeof can only occur in ml files (and not mli files). Is this assumption correct?
Can Pcty_signature occur as a node that isn't the child of a Psig_class_type?
Can a Pcty_arrow occur in a valid mli file? Where? As a child of what?

As I mentioned before, I'm fairly confident of my handling of every besides these two (modules and classes). In case the above isn't clear, here's an annotated snippet of the code that transforms Parsetree.signature -> Parsetree.structure with all of the non-module/class stuff removed for brevity:
(* Parsetree.signature -> Parsetree.structure *)
let rec stub signature_items =
  (* Handles the module_type_desc *)
  let rec stub_module_type module_type =
    match module_type with
    | { pmty_desc = type_; pmty_attributes = attrs; _ } ->
      let expr =
        match type_ with
        | Pmty_ident ident -> Pmod_ident ident
        | Pmty_signature signatures -> Pmod_structure (stub signatures)
        | Pmty_functor (name, a, b) -> Pmod_functor (name, a, (stub_module_type b))
        (* XXX: unclear if these two can occur in an mli *)
        (* | Pmty_with (type_, constraints) -> _ TODO *)
        (* | Pmty_typeof type_ -> _ TODO *)
        | Pmty_extension ext -> Pmod_extension ext
        | Pmty_alias name -> Pmod_ident name
      in
        make_module_expr expr attrs
  in
  (* The next three functions handles the module_type for single and multiple (rec) modules *)
  let stub_module_decl module_decl =
    match module_decl with
    | { pmd_name = name; pmd_type = type_; pmd_attributes = attrs; _ } ->
      make_module_binding name (stub_module_type type_) attrs
  in
  let stub_module module_ = Pstr_module (stub_module_decl module_)
  and stub_modules modules = Pstr_recmodule (List.map stub_module_decl modules)
  and stub_include include_ =
    match include_ with
    | { pincl_mod = module_type; pincl_attributes = attrs; _ } ->
        Pstr_include (make_include_decl (stub_module_type module_type) attrs)
  in
  (* Handles classes (class_type) *)
  let stub_classes classes =
    (* Handles class_type_desc *)
    let stub_class_descr descr =
      let rec stub_class class_ =
        let stub_class_type type_ =
          match type_ with
          | Pcty_constr (ident, types) -> Pcl_constr (ident, types)
          | Pcty_signature class_ -> (* XXX: Is my below assumption true? *)
              failwith "should be covered by Psig_class_type -> Pstr_class_type"
          (* XXX: do we ever need to handle Pcty_arrow for mli files? *)
          (* | Pcty_arrow (label, a, b) -> _ *)
          | Pcty_extension ext -> Pcl_extension ext
          | Pcty_open (override, ident, class_) ->
              Pcl_open (override, ident, (stub_class class_))
        in
          match class_ with
          | { pcty_desc = type_; pcty_attributes = attrs; _ } ->
            make_class_expr (stub_class_type type_) attrs
      in
        match descr with
        | { pci_virt = virt; pci_params = params; pci_name = name;
            pci_expr = class_; pci_attributes = attrs } ->
          make_class_decl virt params name (stub_class class_) attrs
    in
      Pstr_class (List.map stub_class_descr classes)
  in
  let transform_signature signature_item =
    match signature_item with
    | { psig_desc = signature; _ } ->
      let desc =
        match signature with
        (* ... clip non-module/class stuff ... *)
        | Psig_module module_         -> stub_module module_
        | Psig_recmodule modules      -> stub_modules modules
        | Psig_include include_       -> stub_include include_
        | Psig_class classes          -> stub_classes classes
        | Psig_class_type classes     -> Pstr_class_type classes
      in
        make_str desc
  in
    List.map transform_signature signature_items

Unfortunately the module/class stuff is rather complex logic, so trimmed down there's still a lot. There are a ton of helps for creating the *_desc wrappers that encapsulate location in the file, attributes, etc., but those shouldn't be key to understanding how I'm handling modules and classes. But just for clarity, here are the types of all of the helpers:
val make_str : Parsetree.structure_item_desc -> Parsetree.structure_item

val make_module_expr :
  Parsetree.module_expr_desc -> Parsetree.attributes -> Parsetree.module_expr

val make_module_binding :
  string Asttypes.loc ->
  Parsetree.module_expr -> Parsetree.attributes -> Parsetree.module_binding

val make_include_decl :
  'a -> Parsetree.attributes -> 'a Parsetree.include_infos

val make_class_decl :
  Asttypes.virtual_flag ->
  (Parsetree.core_type * Asttypes.variance) list ->
  string Asttypes.loc ->
  'a -> Parsetree.attributes -> 'a Parsetree.class_infos

val make_class_expr :
  Parsetree.class_expr_desc -> Parsetree.attributes -> Parsetree.class_expr

Relevant docs:

parsetree.ml (better than docs, because of some comments)

Edit: As an aside, besides reading documentation on these features (which didn't yield any AST patterns I didn't already know about), I recalled that the compiled can derive the interface from the implementation ocamlc -i. I traced down the variable in the compiler (it's called print_types) that's linked to this flag and found all of its uses, but it was not immediately apparent to me where at any of its uses code is called that derives the mli file (perhaps it is done progressively with the parse, since compiling produces a cmi?). If someone with more OCaml chops or more experience with the compiler could point me to where the mli file is derived, it may be easier to reverse engineer these module and class AST nodes.
Edit 2: I am also aware of How to auto-generate stubs from mli file?, however the answer there is "do it manually," which definitely conflicts with what I'm attempting! (The answerer also claims that such a tool would be trivial, but after pouring over these AST nodes for a while, I beg to differ!)
Answer: (I did not include the -dparsetree in my answer as it is heavy and not that interesting).

I believe that Pmty_with and Pmty_typeof can only occur in ml files (and not mli files). Is this assumption correct?

module M : module type of struct type t end with type t = int

As you can see from this valid mli file, this assumption isn't correct. .mli files require the same parser as .ml files do.

Can Pcty_signature occur as a node that isn't the child of a Psig_class_type?

class type c = object inherit object method x : int end end

Yes it can. Pcty_signature can occur anywhere a class type can occur. (note that there are two Pcty_signature here, one is the child of Pctf_inherit).

a Pcty_arrow occur in a valid mli file? Where? As a child of what?

class c' : int -> object method x : int end

Yes it can! And it can occur anywhere you'd indicate a class type.
Basically, you can consider that if a constructor can happen somewhere, then all the constructors of the same type can happen there too. Any type-related constructor can be in a .mli file (and non-type related constructors can happen too through the devious module type of).
If you have questions about where those are constructed, just take a look at parser.mly. Note that the same parser is used for the two file types.
Comment: Thanks you! This is a fantastic answer! "If a constructor can happen somewhere, then all constructors of that type can happen there too" is pretty interesting. Although as an unfortunate side effect it means I have more work to do! I hope you don't mind if I use your nice contained examples as test cases. `mli` files can't have implementation though, correct? So, it seems like you couldn't have a `class_expr` anywhere in an `mli` (because that could contain a `Pcl_structure` with implementation). I think I'm still having trouble carving out the subset of cases I need to cover.
Comment: Also, given your bio blurb I'm wondering if you have any thoughts on how I've designed this thus far. I fear with the complexity needed to actually handle classes and modules it's going to get even more unwiedly (and some of the types of my `stub_*` might not work since it isn't always the case that types are synthesized to implementations). Sorry for being a bit of a leech as this is tangential to my question (and probably against some SO rules). But given your job description, it sounds like your someone who's brain I could really borrow right now!
Comment: Additionally, my plan for testing this was (in addition to hand written cases) to gather all the mli files in the ocaml repo, run my script on them, and then see if they successfully compile. I could also run `ocamlc -i` on all of the ml files and try the same on them. Presumably the compiler and stdlib should make use of most of the language features, so my hope is that this covers everything. Does this sound sane? (again sorry for being bothersome!)
Comment: @BaileyParker because of `module type of`, anything can happen in a mli.
Comment: @BaileyParker This is quite flattering. I'm not sure about what you're trying to accomplish (and why), but you should probably focus on a subset of the language first. Like, forget about classes and objects for now and have something that works when they do not appear. Also, you might want to look at `ast_invariants.ml` for the invariants that actually happen in the AST. Maybe you'd want to use `ast_iterator.mli` and `ast_helper.mli` to write your code as well.
Comment: Oh, I'll have to look more into `module type of` then. tl;dr of what I'm trying to do is make a library/script that takes an `mli` file and produces a `ml` stub  implementation (with `failwith "not implemented"`) that type checks (the ultimate reason for this is I have a OUnit test suite that I want to run `-list-test` on to get the list of tests without having a real implementation for the test to run against). `Ast_helper` looks much cleaner than the ones I've cobbled together, so I'll definitely look into using that!
Answer: This is a great game. Give me a list of AST Nodes, I'll write you a file that uses them all. :D
module K : module type of String
module M : Map.S with type key = K.t

class fakeref : K.t -> object
    method get : K.t
    method set : K.t -> unit
  end

So, to summarize: Classes can take arguments, hence Pcty_arrow. Pcty_signature can also be the child of Psig_class, as shown above. The other two are standard module constructions that can absolutely appear in .mli files.
As for how ocamlc -i works ... well, it returns the signature inferred by the typechecker. There is no single point of access to this. You can read typing/HACKING.md if you want, but beware, the rabbit hole goes very deep. That being said, I do not think this will be all that helpful to achieve your goal.
My advice would be the following: all the nodes above are fairly easy to handle, except for with_type. This one is very hard, because it basically allows to compute in signatures. Just give up on that one for now.
Also, be aware that values, modules, module types, class and class types all have different namespaces. Pmty_ident x -> Pmod_ident x is not correct.
Comment: In a signature `include X` refers to the module type `X`. In a structure, it refers to the *module* X. Modules and modules types are not the same. `include List` will not work in a signature and `include Map.S` will not work in a structure.
Comment: Thanks so much; this is really great! "I'll write you a file that uses them all" - I hope its ok if I use this as one of my test cases! I'll take your advice and have another go at everything but `with_type` first (although I don't fully understand why it is complicated, so I probably need to do more reading!). I think I see the problem you point out with  `Pmty_ident -> Pmod_ident`. But just to be clear: a module type identifier should never under any circumstance become a module identifier then?
Comment: I was hoping there would be some nice central location in the typechecker that would do `structure -> signature` (which assuming a bijection, would be easy to reverse). But given your advice, I'll stick to more understanding the syntax than blinding reversing what the typechecker does.
Comment: Also, again for clarity, is there one clear stub `ml` corresponding to the `mli` you gave? For the class it seems like `class fakeref _ = object method get = failwith "not implemented" method set _ = failwith "not implemented" end` would work. But what about for the modules?
Comment: Digging into this a bit more, I've confused myself again about `Pmty_ident -> Pmod_ident`. I believe the following transformation valid: `Psig_include (Pmty_ident) -> Pstr_include(Pmod_ident)`. This is the equivalent of `include X` in the `mli` mapping to `include X` in the `ml` right? Basically, what I'm getting at is this mapping is fine (within `stub_module_type`) as long as `stub_module_type` is only called in contexts where you need to synethsize a module (and not where you want the actual type in the `ml` like in `module M : Map.S`). Does this sound right?
Comment: Expanding upon my question immediately above, it seems like the distinction to be made is between `module_type` and `module_type_declaration` (the latter of which wraps the former). My understanding is that the aforementioned transformation is ok in the `module_type` context, but if it were wrapped in a `module_type_declaration` then it would no longer be valid. Is that what you were getting at?
Comment: Oh no, I see. This is shaping up to be a difficult corner case for me. `include List` in an `ml` has a `mli` with all of the list methods included, so reversing that is easy (I already stub all of the methods/functions). But I think if I encounter a `include Map.S` in a `mli` the only way to synthesize a `ml` matching that that `mli` is to actually import the sig of the referenced type, stub it, and then inject that stub at the location of the include . Do you see an easier way? Also, are there any cases at all where `Pmty_ident -> Pmod_ident` is valid? Or is it always wrong?
Comment: I think I have a solution to the include issue. `ocamlc -i some_module.i` will output the same signatures but with the includes inlined. So presumably if I use `Typemod` as a preprocess step to my input AST there will be no `Psig_include` nodes.
Comment: The problem is that, if you start to use the typechecker, you are going to need to care about load path to be able to find external libraries and so, and that opens up a whole new can of worm. :)


Regression - maintain the shape of the curve, but shift it above zero

Question: I'm trying to fit some data with linear regression like this:
Call:
lm(formula = time ~ nDOF:ndoms + I(nDOF^2):ndoms + I(1/nprocs^2):nDOF,
    data = dataFact)

Residuals:
      Min        1Q    Median        3Q       Max
-0.077190 -0.007059  0.000202  0.006533  0.148240

Coefficients:
                     Estimate Std. Error t value Pr(>|t|)
(Intercept)        -9.807e-03  2.123e-03  -4.619 6.57e-06 ***
nDOF:ndoms          3.441e-08  1.231e-09  27.963  < 2e-16 ***
ndoms:I(nDOF^2)     1.749e-12  8.741e-14  20.014  < 2e-16 ***
nDOF:I(1/nprocs^2)  1.103e-05  5.630e-07  19.586  < 2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.02039 on 220 degrees of freedom
Multiple R-squared:  0.9926,    Adjusted R-squared:  0.9925
F-statistic:  9857 on 3 and 220 DF,  p-value: < 2.2e-16

As we can see, the fit is pretty good, but unfortunately, the smallest predicted values tend to be negative.

Is there any way I could "shift" my model a little?

My attempt
I tried to solve the situation using generalized linear regression, like this
Call:
glm(formula = time ~ nDOF:ndoms + I(nDOF^2):ndoms + I(1/nprocs^2):nDOF,
    family = gaussian(link = "log"), data = dataFact)

Deviance Residuals:
     Min        1Q    Median        3Q       Max
-0.19161  -0.07415  -0.04443   0.01376   0.23374

Coefficients:
                     Estimate Std. Error t value Pr(>|t|)
(Intercept)        -2.550e+00  6.269e-02 -40.682  < 2e-16 ***
nDOF:ndoms          2.248e-07  1.582e-08  14.212  < 2e-16 ***
ndoms:I(nDOF^2)    -5.015e-12  9.254e-13  -5.420 1.57e-07 ***
nDOF:I(1/nprocs^2)  2.435e-05  3.958e-06   6.152 3.58e-09 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Dispersion parameter for gaussian family taken to be 0.006711178)

    Null deviance: 12.3909  on 223  degrees of freedom
Residual deviance:  1.4764  on 220  degrees of freedom
AIC: -479.25

But as we can see, the plot is completely out

What am I doing wrong here? And is there any easy way to maintain the shape of the curve and just use GLM to predict only positive numbers?

My data
nprocs,nthreads,ndoms,size,nDOF,time
1,24,288,4,375,0.0946295
2,12,288,4,375,0.0641595
3,8,288,4,375,0.0599065
4,6,288,4,375,0.0549505
6,4,288,4,375,0.0538465
8,3,288,4,375,0.05184975
12,2,288,4,375,0.051091
24,1,288,4,375,0.0534245
1,24,288,6,1029,0.301089
2,12,288,6,1029,0.205296
3,8,288,6,1029,0.1908795
4,6,288,6,1029,0.1785225
6,4,288,6,1029,0.170867
8,3,288,6,1029,0.16909225
12,2,288,6,1029,0.169228
24,1,288,6,1029,0.1673835
1,24,288,8,2187,0.68576575
2,12,288,8,2187,0.49339725
3,8,288,8,2187,0.44999925
4,6,288,8,2187,0.4184415
6,4,288,8,2187,0.4006765
8,3,288,8,2187,0.39963125
12,2,288,8,2187,0.394082
24,1,288,8,2187,0.3925375
1,24,288,10,3993,1.32663875
2,12,288,10,3993,0.94414525
3,8,288,10,3993,0.87997
4,6,288,10,3993,0.816276
6,4,288,10,3993,0.78842875
8,3,288,10,3993,0.77410125
12,2,288,10,3993,0.7764705
24,1,288,10,3993,0.7638315
1,24,288,12,6591,2.2723285
2,12,288,12,6591,1.64944125
3,8,288,12,6591,1.52428725
4,6,288,12,6591,1.40552625
6,4,288,12,6591,1.3529915
8,3,288,12,6591,1.334635
12,2,288,12,6591,1.325433
24,1,288,12,6591,1.323585
1,24,288,14,10125,3.573002
2,12,288,14,10125,2.588366
3,8,288,14,10125,2.42213025
4,6,288,14,10125,2.23624525
6,4,288,14,10125,2.14378325
8,3,288,14,10125,2.1225845
12,2,288,14,10125,2.11212025
24,1,288,14,10125,2.1014935
1,24,288,16,14739,5.3447225
2,12,288,16,14739,3.8467885
3,8,288,16,14739,3.627996
4,6,288,16,14739,3.3243295
6,4,288,16,14739,3.2092895
8,3,288,16,14739,3.16907825
12,2,288,16,14739,3.1560615
24,1,288,16,14739,3.12689
1,24,384,4,375,0.115369
2,12,384,4,375,0.0803815
3,8,384,4,375,0.0763355
4,6,384,4,375,0.07056225
6,4,384,4,375,0.0682565
8,3,384,4,375,0.07106375
12,2,384,4,375,0.06758
24,1,384,4,375,0.07009675
1,24,384,6,1029,0.3517015
2,12,384,6,1029,0.26008075
3,8,384,6,1029,0.2434895
4,6,384,6,1029,0.229354
6,4,384,6,1029,0.222431
8,3,384,6,1029,0.22455025
12,2,384,6,1029,0.22294525
24,1,384,6,1029,0.21949025
1,24,384,8,2187,0.814259
2,12,384,8,2187,0.6097535
3,8,384,8,2187,0.579098
4,6,384,8,2187,0.54244075
6,4,384,8,2187,0.5272745
8,3,384,8,2187,0.532494
12,2,384,8,2187,0.52172925
24,1,384,8,2187,0.52157525
1,24,384,10,3993,1.5572355
2,12,384,10,3993,1.197648
3,8,384,10,3993,1.1347285
4,6,384,10,3993,1.056746
6,4,384,10,3993,1.02914675
8,3,384,10,3993,1.01935525
12,2,384,10,3993,1.02304
24,1,384,10,3993,1.01973325
1,24,384,12,6591,2.6899245
2,12,384,12,6591,2.06899875
3,8,384,12,6591,1.95328225
4,6,384,12,6591,1.83718875
6,4,384,12,6591,1.79995825
8,3,384,12,6591,1.7651285
12,2,384,12,6591,1.7616405
24,1,384,12,6591,1.7832275
1,24,384,14,10125,4.24194525
2,12,384,14,10125,3.25947975
3,8,384,14,10125,3.102823
4,6,384,14,10125,2.910768
6,4,384,14,10125,2.84159925
8,3,384,14,10125,2.812863
12,2,384,14,10125,2.79452
24,1,384,14,10125,2.783541
1,24,384,16,14739,6.34359525
2,12,384,16,14739,4.87417475
3,8,384,16,14739,4.68763775
4,6,384,16,14739,4.409049
6,4,384,16,14739,4.225499
8,3,384,16,14739,4.19000525
12,2,384,16,14739,4.1685855
24,1,384,16,14739,4.13447625
1,24,576,4,375,0.149258
2,12,576,4,375,0.11306475
3,8,576,4,375,0.1106305
4,6,576,4,375,0.10277875
6,4,576,4,375,0.10034075
8,3,576,4,375,0.09992475
12,2,576,4,375,0.100055
24,1,576,4,375,0.1013415
1,24,576,6,1029,0.4631635
2,12,576,6,1029,0.363534
3,8,576,6,1029,0.346867
4,6,576,6,1029,0.33634225
6,4,576,6,1029,0.327822
8,3,576,6,1029,0.32915075
12,2,576,6,1029,0.325756
24,1,576,6,1029,0.3244965
1,24,576,8,2187,1.0745575
2,12,576,8,2187,0.8715855
3,8,576,8,2187,0.8237175
4,6,576,8,2187,0.7936485
6,4,576,8,2187,0.78037075
8,3,576,8,2187,0.78318125
12,2,576,8,2187,0.7757685
24,1,576,8,2187,0.769712
1,24,576,10,3993,2.0662435
2,12,576,10,3993,1.69664675
3,8,576,10,3993,1.61632025
4,6,576,10,3993,1.55593225
6,4,576,10,3993,1.52227825
8,3,576,10,3993,1.52255425
12,2,576,10,3993,1.51433725
24,1,576,10,3993,1.5073515
1,24,576,12,6591,3.55465175
2,12,576,12,6591,2.9219815
3,8,576,12,6591,2.813871
4,6,576,12,6591,2.688473
6,4,576,12,6591,2.642868
8,3,576,12,6591,2.62695125
12,2,576,12,6591,2.62418275
24,1,576,12,6591,2.63111275
1,24,576,14,10125,5.62003375
2,12,576,14,10125,4.66024075
3,8,576,14,10125,4.45642925
4,6,576,14,10125,4.269482
6,4,576,14,10125,4.20735775
8,3,576,14,10125,4.180191
12,2,576,14,10125,4.1697685
24,1,576,14,10125,4.17519025
1,24,576,16,14739,8.412904
2,12,576,16,14739,6.920983
3,8,576,16,14739,6.62266525
4,6,576,16,14739,6.36904225
6,4,576,16,14739,6.2505475
8,3,576,16,14739,6.21563775
12,2,576,16,14739,6.198577
24,1,576,16,14739,6.17604925
1,24,1152,4,375,0.26259325
2,12,1152,4,375,0.21321625
3,8,1152,4,375,0.20502075
4,6,1152,4,375,0.19962325
6,4,1152,4,375,0.1968235
8,3,1152,4,375,0.19961675
12,2,1152,4,375,0.20011775
24,1,1152,4,375,0.19877925
1,24,1152,6,1029,0.79873425
2,12,1152,6,1029,0.69796225
3,8,1152,6,1029,0.67075925
4,6,1152,6,1029,0.6483425
6,4,1152,6,1029,0.648511
8,3,1152,6,1029,0.64445625
12,2,1152,6,1029,0.6470985
24,1,1152,6,1029,0.63747875
1,24,1152,8,2187,1.84742675
2,12,1152,8,2187,1.624627
3,8,1152,8,2187,1.5797865
4,6,1152,8,2187,1.5516275
6,4,1152,8,2187,1.53921525
8,3,1152,8,2187,1.53861475
12,2,1152,8,2187,1.5321545
24,1,1152,8,2187,1.520184
1,24,1152,10,3993,3.5739245
2,12,1152,10,3993,3.15653825
3,8,1152,10,3993,3.120543
4,6,1152,10,3993,3.0330285
6,4,1152,10,3993,3.0156615
8,3,1152,10,3993,3.00419675
12,2,1152,10,3993,2.995779
24,1,1152,10,3993,2.9914245
1,24,1152,12,6591,6.14267725
2,12,1152,12,6591,5.5148995
3,8,1152,12,6591,5.35443675
4,6,1152,12,6591,5.25097775
6,4,1152,12,6591,5.26166925
8,3,1152,12,6591,5.1859525
12,2,1152,12,6591,5.21059225
24,1,1152,12,6591,5.17157525
1,24,1152,14,10125,9.910901
2,12,1152,14,10125,8.696456
3,8,1152,14,10125,8.519928
4,6,1152,14,10125,8.34710125
6,4,1152,14,10125,8.30145125
8,3,1152,14,10125,8.23857
12,2,1152,14,10125,8.28954975
24,1,1152,14,10125,8.22843225
1,24,1152,16,14739,14.4608565
2,12,1152,16,14739,12.976403
3,8,1152,16,14739,12.6674635
4,6,1152,16,14739,12.430683
6,4,1152,16,14739,12.319599
8,3,1152,16,14739,12.27644125
12,2,1152,16,14739,12.2994265
24,1,1152,16,14739,12.2351635

Comment: omitting main effects because of t-tests is exactly what you shouldn't have done. Now the adjusted $R^2$ is unreliable, and cross validation will **only** be reliable if, for each fold, you repeat from scratch the variable selection process. If you just omit the main effects once and for all, based on the full training set, and then you compute the cross-validation error of this selected model, the number you get is mumbo-jumbo just as the current adjusted $R^2$. It *might* be that your model is actually good, but you cannot say that based on these metrics.
Comment: PS also, omitting main effects when only interactions are significant goes against the hierarchical principle. Many good answers on this site - check [this one](http://stats.stackexchange.com/questions/27724/do-all-interactions-terms-need-their-individual-terms-in-regression-model) for example.
Comment: You can, of course, always add a constant such that your predictions are always at least 0. But then your regression will not be the one that minimizes the quadratic errors! If you need to impose boundedness on the predictions, as you already mentioned, you need to specify a different model other than linear, and this will depend on what your data is. Maybe a left-censored tobit is what you need?
Comment: In principle a log link ensures positive predictions (and in my view is a much better solution than Tobit). What's missing from this, so far as I can see, is a record of how you calculated the predicted values shown on the graph. Also, in a situation where there are several predictors an index plot is often less helpful than a plot of observed and predicted responses. Otherwise this question seems to be asking what is wrong with your R code that you don't see the results you expect, and that would be off-topic here. Please see advice in the Help Center on software-specific suggestions.
Comment: Your initial multiple regression model seems to be doing quite well **in terms of predictive accuracy**: adjusted $R^2$ is above .99 with 220 degrees of freedom. Other metrics such as cross-validation error or AIC would be more suitable to estimate generalization error, but in this case it looks like the only issue are the 6 data point with index $i>150$ where you're underestimating the response. **However**, your model is a bit weird in that only some very specific interaction terms are present, and no main effects. Did you get to this model by means of stepwise regression?  [1/2]
Comment: [1/2] If this is the case, then the adjusted $R^2$ may well be meaningless, and the model could have no predictive ability at all. If not, constrained linear regression (which doesn't mean to just add a constant so that your model is shifted away from zero) could solve your problem. Please explain how you arrived at the first model.
Comment: @DeltaIV I've ommited main effects because of t-tests - they seemed to have no singificant effect to the model. Cross-validation seems ok. I'll add more details in the evening, thank you for your response.
Answer: Given the way you derived your model (removing terms based on $t$-tests, without even taking into account the hierarchical principle), model intepretation and inference based on this model are already invalid. At this point, I don't think you would make the situation much worse it you just bounded it above zero. Something like
model <- lm(formula = time ~ nDOF:ndoms + I(nDOF^2):ndoms + I(1/nprocs^2):nDOF, data = dataFact)

dirty_trick <- function(newdata, model){
    yhat <- predict(model, newdata)
    yhat <- max(0, yhat)
}

where newdata is the dataframe containing the points where you want to make predictions. You can use the dirty_trick function to get nonnegative predictions at points newdata based on your model. You could easily modify dirty_trick to return an (invalid) upper confidence interval (the lower one is obviously 0) alongside with the point estimate, but since the confidence interval is indeed invalid, I omitted that.
I stress once again that this model is both:

hard to interpret, because, since you removed the main effects but left the interactions in, the interactions are probably "catching up" whatever influence the main effects had on the response
likely to lead to false inference, because your standard errors, confidence intervals, $p$-values, adjusted $R^2$, etc. are computed without taking into account the selection process.

Thus I strongly suggest that you use it only for prediction, and only after having tested its predictive accuracy on data not used for fitting (or using cross-validation, BUT only if you repeat the variables selection step in each fold).


how to get shopee api Authorization use php5.3

Question: i want to get shopee authorization using the source code.
The language I use is php.5.3.2.
No matter how many times I search, I only have a demo about the higher version, so I can't find the answer I want to ask.
In shopee, you can calculate the default signature sequence and partner key value as "HMAC-SHA256" and create an autorization value. How can I write it in code?
i already get partner key.. but i don't know how to get authorization.
Comment: Considering that PHP 5.3 has been unsupported for 5 years (no security updates etc), I would _really_ recommend upgrading to a supported version. When you say _"I only have a demo about the higher version"_, what exactly do you mean? If you give some examples, we might be able to help you do the same in 5.3.
Comment: The client library code I see says min code PHP7.1, which mens there is code in there stuff that requires PHP7.1. Looks like you have 2 choices. 1. Upgrade PHP 2. Dont use shopee.
Comment: Oh or 3. Rewite all the library you downloaded to work with PHP5.3
Comment: _"I am not in a situation to upgrade"_ - That's the part you should be working on instead. As mentioned, PHP 5.3 haven't even gotten any security updates in _5 years_. That's _very_ bad.
Comment: thank u for answer!! I want to upgrade too, but unfortunately I am not in a situation to upgrade. In other words, there is a demo code for the higher version on the Internet, but I can't find a demo code for my version.
Answer: In order to get shopee authorization, you need to prepare a header as mentioned below and send this header in curl headers with other required data. You can refer to the example given below.
<code>    $secretKey = 'Your Secret Key' ;
    $baseUrl = "https://partner.shopeemobile.com/api/v1/";
    $action = 'The action you want to perform';
    $apiUrl = $baseUrl.$action ;
    $parameters = 'Required Parameters' ;
    $authorisation = $apiUrl."|".json_encode($parmaeters);
    $authorisation = rawurlencode(hash_hmac('sha256', $authorisation, $secretKey,
    enter code herefalse));

    $header = array(
      'Content-Type: application/json',
      'Authorization: '.$authorisation,
    );
</code>
After preparing this, send $header in curl headers
YOUR CURL REQUEST WOULD BE LIKE THIS:
<code>    $connection = curl_init();
    curl_setopt($connection, CURLOPT_URL, $apiUrl);
    curl_setopt($connection, CURLOPT_HTTPHEADER, $header);
    curl_setopt($connection, CURLOPT_SSL_VERIFYPEER, 0);
    curl_setopt($connection, CURLOPT_SSL_VERIFYHOST, 0);
    curl_setopt($connection, CURLOPT_POST, 1);
    curl_setopt($connection, CURLOPT_POSTFIELDS, json_encode($parameters));
    curl_setopt($connection, CURLOPT_RETURNTRANSFER, 1);

    $response = curl_exec($connection);
    curl_close($connection);
</code>
Hope you get your query resolved by this.
Answer: If you can use npm package in your php app, you can use this code:
<code>import { createHmac } from 'crypto';

const timestamp = getUTCUnixTimestamps();
const baseString = `${AppID}${Apipath}${timestamp}`;
const signCode = createHmac('sha256',shopeeAppSecret).update(baseString).digest('hex');
</code>
Comment: Please add some explanation to your answer such that others can learn from it
Answer: To get the Shopee Auth code you'll need partnerID, partnerKey both can be seen on your Shopee Open Platform Console.
Next, is the API path for Authorization which is
https://partner.shopeemobile.com/api/v1/shop/auth_partner for V1
https://partner.shopeemobile.com/api/v2/shop/auth_partner for V2
both have different Auth processes
for V2
create a baseString consisting of
PartnerId,
API path (/api/v2/shop/auth_partner)
and Unix timestamp.
Create a signature using by hashing the baseString with your partnerKey.
<code>$timest = time();
$tokenBaseString = $partnerID.$path.$timest;
$sign = hash_hmac('sha256', $tokenBaseString, $partnerKey, false);
</code>
prepare the Auth URL like this
<code>$host = 'https://partner.shopeemobile.com'
$path = '/api/v2/shop/auth_partner'
$partner_id = 123456
$timestamp = 'unix_timestamp' //authorization url expires in 5 minutes(because of timestamp expiration)
$sign = 'the_generated_signature'
$returnURL = 'https://yourReturnURL.com'
</code>
Auth URL will look like this
<code>$AuthURL= $host.$path.'?partner_id='.$partner_id.'?timestamp='.$timestamp.'&sign='.$sign.'&redirect'.$returnURL;
</code>
Sample Auth URL

https://partner.shopeemobile.com/api/v2/shop/auth_partner?partner_id=123456&timestamp=1648628444&sign=xxxxxxx&redirect=https://yourReturnURL.com
Comment: I already updated my answer
Comment: What is "this"? Please add some explanation to your answer such that others can learn from it


How to prove $a^TX^{T}(XX^T)^{-2}Xa$ is increasing when the dimension of $X$ and $a$ increases?

Question: Let $X_1\in\mathbb{R}^{m\times n}$ with $n\geq m$ and $rank(X_1)=m$, $a_1\in\mathbb{R}^n$. Let $z\in\mathbb{R}^m$ and $w\in\mathbb{R}$. Define
$$
\begin{aligned}
&X_2=(X_1,z)\in\mathbb{R}^{m\times (n+1)}\
&a_2=(a_1^T,w)^T\in\mathbb{R}^{n+1}
\end{aligned}
Assume $X_2$ has full row rank. Now I want to show that
$$
a_2^TX_2^T(X_2X_2^T)^{-2}X_2a_2\geq a_1^TX_1^T(X_1X_1^T)^{-2}X_1a_1
$$
I know that, by Sherman–Morrison formula,
$$
(X_2X_2^T)^{-1}=(X_1X_1^T+zz^T)^{-1}=(X_1X_1^T)^{-1}-\frac{1}{g}(X_1X_1^T)^{-1}zz^T(X_1X_1^T)^{-1},
$$
where $g=1+z^T(X_1X_1^T)^{-1}z$. Then we can write $a_2X_2^T(X_2X_2^T)^{-2}X_2a_2$ in terms of $X_1$ and $a_1$. But then I got stuck. Is there any convenient way to proof it?
Comment: @user1551 Sorry, it's my typo. I should say $rank(X)=m$, so $XX^T$ is invertible. Then how to further prove the statement?
Comment: @user1551 Sorry for my careless editing. What I want to say is that $X_2$ still has full row rank so $X_2X_2^T$ is invertible. I think now the question should make more sense to you.
Answer: This is not true. Counterexample:
$$
X_1=I_2,\ a_1=\pmatrix{1\\ 0},\ X_2=\pmatrix{1&0&1\\ 0&1&1},\ a_2=\pmatrix{1\\ 0\\ -1}.
$$
$$
a_2^TX_2^T(X_2X_2^T)^{-2}X_2a_2 = \frac{5}{9}
< 1 = a_1^TX_1^T(X_1X_1^T)^{-2}X_1a_1.
$$


Finding all nodes not referenced by a menu entry or another node

Question: In the course of creating the content for a site, we've ended up with orphaned content nodes that are not linked to from anywhere. I'd like to find and examine these. Surprisingly, I can't find a module for this.
Tiny hypothetical example: I'm building a brochureware site. Someone wrote a "Contact" page, someone else wrote an "About Us", and someone else wrote "Directions". The person writing "Directions" forgot to put it in a menu link or add a link from "About Us" to "Directions". So "Directions" is orphaned: there are no links to it. I'd like to find all such nodes so I can figure out what to do with them.
Comment: Are you seeing this nodes when you are listing al nodes?
Comment: Yes. There's nothing special about them. They just happen to be orphaned in terms of menus and links.
Comment: Is there a reason why you cannot delete them from the from the general content page?
Comment: No, but that's not the issue. Of, say, all the Basic Pages on the content page, I want to know which ones are orphans and which ones are not. I'm going to look at the orphans and see if there's some content inside I want. If not, I'll delete them. The issue is that I can't tell which ones are orphans just by looking at the list. For instance, one of my editors may have written, say, a "Contacts" page, but we forgot to put it in a menu and so it's currently inaccessible unless you know the URL.
Comment: Are you just interested in existing modules, or would you be willing to write a custom module if no module does what you are looking for?
Comment: This is a small task right now, so I wouldn't write a module, though I know how. Just thought someone might have already done the work.  By the way, I edited the question back to a more general one. I'm not quite sure why this has required so much explanation.
Answer: I needed this for a clients request, so i decided to try and do it without any additional modules, and push my MySQL a bit. This is what i ended up with.
For clarity, this will only provide a list of NID's that DO NOT have an associated menu item.
I've also added a WHERE clause to limit content types, as some content types will naturally be excluded because of the way they was architected.
SELECT n.nid, n.type, n.title, n.status
FROM node AS n
WHERE (n.type = 'article' OR n.type = 'page')
AND n.nid NOT IN (
  SELECT REPLACE(link_path, "node/", "") AS nid
  FROM menu_links
  WHERE SUBSTRING(link_path, 6, LENGTH(link_path)) REGEXP "^[0-9]+$"
);

Comment: Nice solution! +1 for no new modules.
Answer: For links from the menu system:
Menu Node Views comes very close to what you request but needs some coding.
For links from the body (fields) of nodes:
The Search module saves references between nodes. This can be used by views to create a "backlinks" view. For Drupal 5 there was a seperate module for this with some aditional explenation, since D6 (Views 2/3) this functionallity is baked into Views Module.
Summary
views + vbo + Menu Node API + Menu Node Views ( + custom filter/patch or View with custom query )
Comment: This is helpful, though it doesn't handle links to nodes from the body (or whatever) of other nodes. I think it's as good as I'm going to get - thanks.
Comment: Oops, missed that part. Added the missing information now.
Answer: Have you tried Views? You'll need to create a view for the orphaned nodes, which would depend on your taxonomy and fields.
Comment: I'm looking for, say, Basic Page nodes that have no menu entry or are not linked to from any other node. They have no taxonomy. There's no View query that's going to find these.
Comment: I was under the impression that you could use (not) logic on Views. So you could say something like Nodes where xyz taxonomy field is empty.
Comment: These nodes have no taxonomy fields. I just want to know if they are accessible in a menu or if they are linked to from the content of some other (accessible) node.
Answer: You can accomplish the menu portion pretty easily with a custom query.
SELECT *
FROM node
WHERE node.type = 'page'
AND node.nid NOT IN (
 SELECT nid
 FROM menu_node
)


Should Jorgensen have been sent off for his DOGSO foul in Croatia-Denmark (2018 FIFA World Cup)?

Question: I am of course referring to the foul that occurred at the 113th minute of the round of 16 match between Croatia and Denmark, when Rebić would have scored an easy goal had he not been fouled in the box by Jorgensen:

As we know, a penalty kick was awarded but Jorgensen was only cautioned. Everyone seemed to agree with this decision, on the basis that Jorgensen did try to play the ball. And trust me, I believe this. The following image makes it clear that Jorgensen was attempting to play the ball:

Yet the point I'm trying to make is different. Is the fact Jorgensen was trying to get the ball enough to downgrade the red card to a simple yellow card in this case? People who agree with the referee's decision end up quoting this part of the 2018/19 Laws of the Game (Law 12, p. 108):

Where a player commits an offence against an opponent within their own penalty area which denies an opponent an obvious goal-scoring opportunity and the referee awards a penalty kick, the offender is cautioned if the offence was an attempt to play the ball; in all other circumstances (e.g. holding, pulling, pushing, no possibility to play the ball etc.) the offending player must be sent off.

But read it carefully: this paragraph is only referring to situations where an obvious goal-scoring opportunity is denied. To see why this is important, let's have a look at what the rulebook says just one page earlier:

A player, substitute or substituted player who commits any of the following offences is sent off: (...) denying a goal or an obvious goal-scoring opportunity to an opponent whose overall movement is towards the offender's goal by an offence punishable by a free kick (unless as outlined below)

As you can see, the rule is actually about two distinct situations: denying a goal OR an obvious goal-scoring opportunity. It is also clear that 'denying a goal' doesn't only occur when a player deliberately handles the ball on the goal line, because

this paragraph is specifically about offences committed against an opposing player, and
the situation where the ball is deliberately handled is already discussed in the previous bullet point, and again the Law makes a difference between denying a goal or an obvious goal-scoring opportunity ("denying the opposing team a goal or an obvious goal-scoring opportunity by deliberately handling the ball").

Now, as the first image clearly shows (and as everyone who has watched the match knows), the goal was empty (the goalkeeper, Schmeichel, was almost outside the penalty area): a tap-in would have been enough for Rebić to score a goal. And seeing how the Laws of the Game do make a distinction between

fouling an opponent and denying a goal, or
fouling an opponent and denying an obvious goal-scoring opportunity...

...well, then it looks to me that this is the clearest example of what 'denying a goal by fouling an opponent' should be.
So, why did the referee apply that special provision of Law 12 that allowed him to only give a yellow card to Jorgensen, when this exception is only valid in situations when an obvious goal-scoring opportunity is denied?
After all, that paragraph was added to Law 12 because a penalty kick effectively restores the goal-scoring opportunity that was lost, so sending off the offending player would have been too much (Law 12 FAQ #9): indeed, the chance of scoring on a penalty kick is roughly 80%, which is more or less the same chance of scoring that an attacking player has when facing only the goalkeeper during open play. But in this situation, the chance of scoring was easily 99%: the penalty kick does NOT fully make up for the previous situation, and this is why I think the Laws of the Game make a distinction between the two cases.
So, was cautioning Jorgensen a mistake under the current Laws of the Game? Or are there any other official rules/interpretations that justify a yellow card and that I'm unaware of?
Answer: This is simply an editorial oversight.
The full text of the sending-off offence you are referring up until the current (2018/19) edition of the Laws was:

denying an obvious goal-scoring opportunity to an opponent moving
  towards the opponents' goal by an offence punishable by a free kick (unless
  as outlined below)

This was changed to (emphasis added):

denying a goal or an obvious goal-scoring opportunity to an opponent whose overall movement is towards the offender's goal by an offence punishable by a free kick (unless as outlined below)

The first part of the explanatory text further on has not changed since it was introduced in the 2016/17 edition:

Where a player commits an offence against an opponent within their own penalty area which denies an opponent an obvious goal-scoring opportunity and the referee awards a penalty kick ...

If a player either denies a goal or obvious goalscoring opportunity by committing an offence punishable by a free kick as part of an attempt to play the ball, and this results in a penalty kick being awarded, the player is cautioned for unsporting behaviour instead of being sent-off. In all other cases e.g. pushing, pulling, holding, no possibility of playing the ball, deliberately handling the ball, other misconduct (i.e. entering the field of play without permission), the player is sent-off.
Answer: You can't "deny a goal" in this situation because the ball is not even close to entering it; Rebic is still regathering the ball after beating the goalkeeper and hasn't yet beaten the challenge from Jorgensen, and a shot hasn't been taken yet.
There are more than enough incidents where a player has been able to walk the ball into the six-yard box and could have walked it into the goal and still missed. To call this particular situation "denying a goal" would be wrong.
Once you accept that even a situation where the attacker is extremely likely to score is still only the denial of an obvious goal-scoring opportunity, the remainder of the analysis is moot and the answer is clear.
Jorgensen did attempt to play the ball. He fouled Rebic and therefore a DOGSO occurred. According to the laws, this is a penalty kick and a caution (yellow card). The correct decision was made.


What is this device strapped around the nut of an electric guitar?

Question: What is this device strapped around the nut of the guitar in this image?
Answer: It is a string damper - it stops open strings ringing. This can be very useful for a range of playing styles.
I tend to use a handkerchief tucked under the strings for some Satriani-style tapping melodies, as it allows me to use two hands at speed up the neck of the guitar without having to damp every note.
Comment: May I extend this saying the damper is used heavily when sweep picking as it mutes out the previous strings, stopping them ringing out.
Comment: Interesting. I'm trying to perfect my sweep picking and hadn't thought of this.
Comment: Mark J. Smith uses a plain old black sock in his [bass tapping lessons](http://www.talkingbass.net/tapping-bass-guitar-lesson-part-1/), which looks hilarious around the end of his slick Ibanez bass.


How to position before and after pseudo elements to center-right and center-left of their parent?

Question: I've been in the process of making a customise-able tooltip for Vue using SCSS, allowing a developer to specify the colour and placement of the tooltip, which is constructed from a pair of ::before and ::after pseudo-elements in a button element of variable size. The ::before acts as the tooltip's arrow, and the ::after the tooltip content.
There's four placements I want to make possible: top, bottom, left and right. The following two groups of classes apply to the top and bottom positions:
<code>    .tooltip-placement-top, .triangle-placement-top {
      left: 50%;
      transform: translate(-50%, 0);
    }

    .tooltip-placement-top {
      bottom: calc(100% + 10px);  // 3. 100% here is parent height, key point
    }

    .triangle-placement-top {
      bottom: calc(100%); // 4. no need add 5px, because transparent border also takes place
      border-top-color: var(--tooltip-border-color);
    }

    .tooltip-placement-bottom, .triangle-placement-bottom {
      left: 50%;
      transform: translate(-50%, 0);
    }

    .tooltip-placement-bottom {
      top: calc(100% + 10px);  // 3. 100% here is parent height, key point
    }

    .triangle-placement-bottom {
      top: calc(100%);
      border-bottom-color: var(--tooltip-border-color);
    }
</code>
and these result in a centred tooltip above and below the button like so:

The next stage is to position the left and right hand tooltip classes to sit aligned with the horizontal centre of the button, but conforming to the left or right edge of the button rather than the top and bottom, as seen here:

Both the button and tooltip are likely to appear as different sizes depending on their context of use, so unfortunately this needs to be a scalable solution (i.e. increasing the size of either button or tooltip won't cause them to fall out of alignment). My current attempts to adapt the code has made the tooltip appear at 100% of its own width right of the button, which as you can imagine looked terrible.
Sorry I don't have screen-caps to share, but I hope the drawings make it clear what I'm trying to achieve! If you need any additional code then I can supply it, but the question was getting quite bulky as it is.
Comment: You don't seem to be using css-positioning even though you have included this in your tagging. I'd start there.
Answer: Sorted it!
Here's what I have for my right and left classes:
<code>.tooltip-placement-left, .triangle-placement-left {
    top: 50%;
}
.tooltip-placement-left {
    right: 100%;
    transform-origin: right;
    transform: translate(-12px, -50%);
}
.triangle-placement-left {
    right: 100%;
    transform: translate(8.3px, -50%);
}
.tooltip-placement-right, .triangle-placement-right {
    top: 50%;
}
.tooltip-placement-right {
    left: 100%;
    transform-origin: left;
    transform: translate(12px, -50%);
}
.triangle-placement-right {
    left: 100%;
    transform: translate(-8.3px, -50%);
}
</code>
the exact distances in pixels depend on the dimensions of your tooltip arrow, but this is what worked for me. Hope it helps someone else!


Moment Conditions for Stochastic Discount Factor (SDF)

Question: Following extract has been taken from Computing Generalized Method of Moments and Generalized Empirical Likelihood with R, the Vignette of gmm R package.
In some cases the theory is directly based on moment conditions. When
it is the case, testing the validity of these conditions becomes a
way of testing the theory. Jagannathan et al. (2002) present
several GMM applications in finance and one of them is the stochastic
discount factor (SDF) representation of the CAPM. The general theory
implies that $\mathbb{E}\left(m_{t}R_{it}\right)=1$ for all $i$,
where $m_{t}$ is the SDF and $R_{it}$ is the gross return $\left(1+r_{it}\right)$.
It can be shown that if the CAPM holds, $m_{t}=\theta_{0}+\theta_{1}R_{mt}$
which implies the following moment conditions:
$$
\mathbb{E}\left[R_{it}\left(\theta_{0}-\theta_{1}R_{mt}\right)-1\right]=0~~\mbox{for}~~i=1,...,N.
$$
which can be tested as follows:
g5 <- function(tet, x) {
gmat <- (tet[1] + tet[2] * (1 + c(x[, 1]))) * (1 + x[, 2:6]) -
return(gmat)
}

I wonder how the moment conditions are obtained and how they are coded in R. Any help will be highly appreciated. Thanks in advance for your help.
Comment: Are you looking for an explanation of the financial model, an explanation of how the moment conditions result from that model, or just how to code conditions and estimation into R?
Comment: I'm looking for how the moment conditions resulted from the model and how they are coded in R.
Answer: How these moment conditions result from CAPM and standard SDF equations
Consider a gross return on some investment, $R_{i,t}$. The equation:
$$
E[m_tR_{i,t}]=1
$$
is one of the more fundamental equations of asset pricing. Here $m_{t}$ is called a stochastic discount factor. The theory behind an SDF is a little outside the purview of CrossValidated, if you want to learn more about it I'd suggest checking out John Cochrane's Coursera course on asset pricing, or his textbook. You could also ask about SDF's on the economics stackexchange site.
Imagine a risk free return, $R_{f,t}$, and let's look at excess returns over the risk free rate $r_{i,t}$. Then:
$$
E[m_tr_{i,t}]=E[m_tR_{i,t}]-E[m_tR_{f,t}]=1-1=0
$$
This equation can be decomposed in a different matter:
$$
E[m_tr_{i,t}]=E[m_t]E[r_{i,t}]+\text{cov}(m_t,r_{i,t})
$$
taking these last two equations together and remembering that for a regression of $r_{i,t}$ on $m_t$, the coefficient $\beta_i = \text{cov}(m_t,r_{i,t})/\text{var}(m_t)$ we have:
$$
E[r_{i,t}]=\beta_i \dfrac{\text{var}(m_t)}{E[m_t]}
$$
which follows from the definition of covariance.
The version of the CAPM you may be used to is:
$$
E[r_{i,t}]=\beta_i E[r_{m,t}]
$$
where $r_{m,t}$ is the return on the market. We can start to see connections between this and the SDF by looking at the two equations above. Let's say that $m_t=a-br_{m,t}$ will this satisfy the above? If the SDF is a linear function of $r_{m,t}$, then all $\beta$'s can be determined (up to a constant scale) by regressions of $r_{i,t}$ on $r_{m,t}$, as in this case:
$$
\text{cov}(r_{i,t},m_t)= -b\text{cov}(r_{m,t},r_{i,t})
$$
We are then left with two requirements for $b$ and $a$. First, the market return itself must be an asset with $\beta$ of 1, so that:
$$
E[r_{m,t}]=\dfrac{\text{var}(m_t)}{E[m_t]}
$$
Second,
$$
E[R_{f,t}]=\dfrac{1}{E[m_t]}
$$
this leaves us two equations in two unknowns, $a$ and $b$ so we know this SDF representation of the CAPM model will work.
Finally, since the risk free rate is a constant we can rewrite our SDF as $m_t=\theta_0-\theta_1R_{m,t}$. Substitute our new $m_t$ into the first equation and you have the equation presented in the question:
$$
E[m_tR_{i,t}]=1\Rightarrow E[(\theta_0-\theta_1R_{m,t})R_{i,t}]=1\Rightarrow E[(\theta_0-\theta_1R_{m,t})R_{i,t}-1]=0
$$
This is a moment condition: a function of the data which in expectation must be equal to zero. If the CAPM model is correct, this equation must hold for every asset in the economy.
How to program these conditions into R
Unfortunately, we do not know the expected market premium or the risk free rate, so we have to estimate $\theta_0$ and $\theta_1$ from the data (even if we did know these quantities this might still be interesting as a test of the model).
The GMM code in this package needs you to provide a function which will output the data in a form that gives the moment equation for each observation, so it can then take the expectation over your data. In otherwords, it needs a number for each observation, $e_{i,t}$:
$$
e_{i,t}=(\theta_0-\theta_1R_{m,t})R_{i,t}-1
$$
this particular setup seems to be for five test assets (five different $i$'s).
GMM will choose $\theta_0$ and $\theta_1$ such that a weighted combination of these expectational errors are minimized, and the moment conditions hold as close as possible.
The code you are given is:
g5 <- function(tet, x) {
gmat <- (tet[1] + tet[2] * (1 + c(x[, 1]))) * (1 + x[, 2:6]) -
return(gmat)
}

this implies a matrix of net returns, x, where x[,1] is the market return, and x[,2:6] are five returns on different assets. Since they are net returns they have to be converted to gross returns by adding 1 to each of them. The rest follows from the moment condition above. The output of this function will be a $T\times 5$ matrix, where $T$ is the number of time periods, and each element $e_{i,t}$ is an expectational error as above. If you took the average over these columns, you would have an (unweighted) version of the five moment conditions.
Meanwhile tet will be a 2 entry vector, containing parameter estimates, $[\theta_0,\theta_1]$. The gmm package in R then can estimate these parameters with the command:
est <- gmm(g5, x = x, t0 = rep(0, 2))

and on output you will have estimates for $\theta_0$ and $\theta_1$ as well as information about model fit.
Comment: (+1): Thanks @jayk for an excellent answer and for link to the nice reference book.


How to 'grep' a continuous log file stream with 'tailf' and when the needed string is obtained, close/break the 'tailf' automatically?

Question: into a bash script, I need to grep a contiuous log streaming and when the proper string is filtered, I need to stop the 'tailf' command to move ond with other implementations.
The common command that works is:
tailf /dir/dir/dir/server.log | grep --line-buffered "Started in"

after the "Started in" line is gathered, I need to break down the "tailf" command.
All this stuff into a bash script.
Answer: use grep -m1, it means return the first match then stop:

-m num, --max-count=num
         Stop reading the file after num matches.

tailf /dir/dir/dir/server.log | grep -m1 "Started in"

Answer: Figured out...
tailf /dir/dir/dir/server.log | while read line
do
    echo $line | grep "thing_to_grep"
        if [ "$?" -eq "0" ]; then
        echo "";echo "[ message ]";echo "";
        kill -2 -$$
    fi
done

$$ is the PID of the current shell, in this case associated to the "tailf" command.
Comment: ha, why bother to `kill` whole shell, impressive, doesn't `grep -m1` work for you? if you like *your way*, improve it with `break`: `tailf  a.log | while read line; do echo "$line" | grep "word" && break; done`, don't forget double quote.
Comment: hey there any feedback?


Complex number, absolute value and triangle inequality

Question: Hi I was wondering if the following statement can be true
$|1+a|= 1+|a|$ where $a$ is a complex number i.e $x+iy$ ?
I understand there is the triangular inequality but I was wondering if the $1$ in the absolute would simplify it ?
Comment: It is not a formal explanation but it clearly is not. Since it doesn't work for real numbers. Take a=-3 it does not work for this (2 is not equal to 4)
Comment: It is true if $a$ is a non-negative real, and every real number is a complex number.
Answer: It is true if and only if the imaginery part is zero and the real part is non negative.
Only if part: Let $a=x+yi$.
$$\sqrt{(x+1)^2+y^2}=1+\sqrt{x^2+y^2}$$
$$\implies (x+1)^2+y^2=x^2+y^2+1+2\sqrt{x^2+y^2}$$
$$\implies 2x=2\sqrt{x^2+y^2}$$
$$\implies x^2=x^2+y^2$$
$$\implies y=0$$
And for $x$ being real,
$$|x+1|=1+|x|$$
$$\implies x^2+2x+1=x^2+2|x|+1$$
$$\implies x=|x|$$
$$\implies x\geq 0$$
Comment: Are you saying it's true for $a=-1$?
Comment: Oh right, I did not read the comments carefully, I will add proof for the non-negative part.
Comment: Thanks for the details explanation @cr001:)
Answer: Assume $a=x+yi$ and $x,y\in\mathbb{R}$:
$$|1+x+yi|=\sqrt{\Re(1+x+yi)^2+\Im(1+x+yi)^2}=\sqrt{(1+x)^2+(y)^2}=\sqrt{y^2+(1+x)^2}$$


Random http 500 on wordpress

Question: My wordpressblog produces http 500s on a small percentage of the requests. It doesn't seem to be tight to special urls, but it seems to happen clustered in time. i.e there are times when it happens more often for an hour or so, and then it stops happening for a couple of days.
I no probably nobody can tell me what is wrong with this description, but can anybody give me some tips how to debug this problem?
For example is there any way to extract some logging from wordpress? So far I only have the access log which comes from the apache running wordpress. And it only shows the http 500 no hint towards the reason for them.
Answer: Do you have WP-Cache installed or some other caching plugin? I've seen WP-Cache throw 500 errors from time to time in the past, though since switching to WP-Super-Cache, they've gone away.
Comment: I do have a caching plugin installed: WP-Super-Cache. I disabled it for now, and will check if the http 500s go away. How did you identify the cache plugin being the bad guy?
Comment: Just trial and error - started disabling each plugin one by one to see who the culprit was.
Comment: Still have the problem after disabling WP-Super-Cache
Answer: It may also be a problem with the web-server. On my setup of lighttpd with only two fastcgi instances serving the requests, I do sometimes get 500 errors that go away when I increase it to three fastcgi instances.
Comment: I hope it isn't the webserver, because I don't have much control over that  :-(

Random http 500 on wordpress


Elasticsearch get results count of other indices with same query

Question: I have a cluster setup with multiple indicies, generated from a rails server. Each index is a model in my server.
Looking to search a single index, but get counts for the other indices (models) based on my query json. Is this possible?
The query im using is generated from searchkick and quite large so I'm refraining from posting here. I'm trying to figure out if aggs does what I need, something like. I'd like to refrain from doing a 2nd query if possible
POST blog_index/_search
{
  "query": { ... }
  "aggs": {
    "results_count": { ... }
  }
}
=> What I would like to receive
{
  "aggregations": {
    "results_count": {
      "blogs": 123,
      "users": 234,
    }
  }
}

Example:
If do a { "term": { "type": "funny" } } on blog_index/_search, and there are 3 users with that type, I'd like to get that count as part of my blog_index query
Comment: are you trying to query an indices and use the result to query another indices? the question is not clear. please use an example of dataset.
Comment: I'm trying to get a result count from another index based on the query I have currently.

lets say i'm doing a `match_all` on `blog_index` and there are 3 users in my `user_index`, I'd like to find out how many users there are (3)
Answer: Solution: You can aggregate on field _index.
"group_by_index": {
  "terms": {
    "field": "_index",
    "size": "30"
  }
}


Como usar setNextFocusableComponent em um JTextArea

Question: Eu usei setNextFocusableComponent em um JTextArea, porem não funciona pelo fato do TAB funcionar como um "espaço" dentro do JTextArea, como resolver isso ??
Answer: Você pode usar control+TAB para mudar o foco de uma JTextArea para outros componentes. Por padrão, o TAB insere espaços de tabulação dentro deste componente.
Mas nada impede de se alterar este comportamento padrão, adicionando um KeyAdapter para "forçar" mudar de foco ao clicar TAB. Retirei este exemplo abaixo deste site:
import javax.swing.*;
import java.awt.*;
import java.awt.event.KeyAdapter;
import java.awt.event.KeyEvent;

public class TextAreaTabMoveFocus extends JFrame {
    public TextAreaTabMoveFocus() {
        initialize();
    }

    private void initialize() {
        setSize(300, 300);
        setTitle("JTextArea TAB DEMO");
        setDefaultCloseOperation(WindowConstants.EXIT_ON_CLOSE);
        setLayout(new BorderLayout());

        JTextField textField = new JTextField();
        JPasswordField passwordField = new JPasswordField();
        final JTextArea textArea = new JTextArea();
        JScrollPane scrollPane = new JScrollPane(textArea);

        // Add key listener to change the TAB behavior in
        // JTextArea to transfer focus to other component forward
        // or backward.
        textArea.addKeyListener(new KeyAdapter() {
            @Override
            public void keyPressed(KeyEvent e) {
                if (e.getKeyCode() == KeyEvent.VK_TAB) {
                    if (e.getModifiers() > 0) {
                        textArea.transferFocusBackward();
                    } else {
                        textArea.transferFocus();
                    }
                    e.consume();
                }
            }
        });

        getContentPane().add(textField, BorderLayout.NORTH);
        getContentPane().add(scrollPane, BorderLayout.CENTER);
        getContentPane().add(passwordField, BorderLayout.SOUTH);
    }

    public static void main(String[] args) {
        SwingUtilities.invokeLater(new Runnable() {
            @Override
            public void run() {
                new TextAreaTabMoveFocus().setVisible(true);
            }
        });
    }
}

Veja funcionando:


The condition of determining a group homomorphism between two finite abelian group

Question: There is an exercise (Exercise 10.25 in Gallian's Contemporary Abstract Algebra  8/e) which asks:
How many group homomorphisms are there from $\Bbb{Z}_{20}$ onto $\Bbb{Z}_{10}$?
How many are there to $\Bbb{Z}_{10}$?
The answer is $\varphi(10)=4$ and $10$, respectively.
It is sufficient to determinet the image of the generator of $\Bbb{Z}_{20}$.
But I found an example which seems to be a homomorphism, but it is not.
The example is: $\theta:\Bbb{Z}_3\times \Bbb{Z}_2\to S_3$, $\theta(1,0)=(123)$ and $\theta(0,1)=(12)$.
This mapping preserve the operation.
But it is not well-defined.
$$(123)(12)
=\theta(1,0)\cdot \theta(0,1)
=\theta((1,0)+(0,1))
=\theta(1,1)
=\theta((0,1)+(1,0))
=\theta(0,1)\cdot \theta(1,0)
=(12)\cdot (123).$$
Is the following statement true?
Let $G\cong \Bbb{Z}_{p_1^{r_1}}\times \Bbb{Z}_{p_2^{r_2}}\times \cdots \times \Bbb{Z}_{p_s^{r_s}}$ and $H$ be two finite additive abelian groups.
If a mapping $f:G\to H$ satisfy $p_1^{r_1}f(1,0,...,0)=0_{H}$, $p_2^{r_2}f(0,1,0,...,0)=0_{H}$, ..., $p_s^{r_s}f(0,0,..., 0,1)=0_{H}$,
then $f$ must be a homomorphism from $G$ to $H$.
Remark.
There is an error in my question.
I can't just only define $f$ as a ``function''.
For example,
define $f:\Bbb{Z}_2\to \Bbb{Z}_2$ by $f(0)=f(1)=1$.
Which satisfy the condition.
But it is not a homomorphism.
Comment: This may be helpful. https://drexel28.wordpress.com/2011/11/14/homomorphisms-between-finitely-generated-abelian-groups-pt-i/
Answer: The statement is true and you could probably spend hours trying to prove it, but it's much nicer to use group presentations which give us something much more general (apologies, I will be using multiplicative notation).
You may be aware that any group $G$ can be written $\langle P|R\rangle =F(P)/\langle\langle R\rangle\rangle$ where $P\subset G$ generates $G$, $F(P)$ is the free group on $P$, and $\langle\langle R\rangle\rangle$ is the smallest normal group in $F(P)$ containing the set $R\subseteq F(P)$. $P$ is a set of generators of $G$ and $R$ is a set of relators.
In general an arbitrary map $\psi_0:P\to H$ defines a homomorphism $\psi:G\to H$ if and only if $\psi_0(r)=1_H$ for each $r\in R$.
In your question specifically $G$ is generated by $e_1,\ldots,e_s$, where $e_i$ can be thought of as an $s$-tuple with $1$ in the $i^{th}$ entry and $0$ elsewhere. One set of relators that define $G$ is $R=\{e_ie_je_i^{-1}e_j^{-1},e_i^{p_i^{r_i}}|1\le i,j\le s\}$. The $e_ie_je_i^{-1}e_j^{-1}$ just mean that $G$ is abelian and since $H$ is abelian, these certainly map under $f$ to $1_H$. The $e_i^{p_i^{r_i}}$ just mean that $e_i$ has order $p_i^{r_i}$ and by assumption, these map under $f$ to $1_H$: so  $f$ does indeed define a homomorphism.
Comment: It's actually not very hard to prove and is usually done in any introductory book / course on group presentations. $r=1_G$ in $G$ so $\psi(r)=1_H$ is immediate. For the other direction, by definition of free group $\psi_0$ defines a homomorphism $\psi_1:F(P)\to H$. The kernel  $K$ of this map contains each $r\in R$ so contains $\langle\langle R\rangle\rangle$. Therefore $H\cong F(P)/K\cong (F(P)/\langle\langle R\rangle\rangle)/(K/\langle\langle R\rangle\rangle)\cong G/K'$ so $\psi$ can be identified as the natural map $G\to G/K'$.
Comment: @RobertChamberlain This is a great answer; I would just suggest that it might be helpful in the first "direction" of the proof in the comment above, to write something like: under the presentation hom, say $\rho:F(P)\longrightarrow G$, every $r$ in $R$ is mapped to $1_G$ and so $\psi_0(r) = 1_H$ (since $\psi_0 = \psi\circ\rho$).
Comment: Thanks. Could you please tell me where did you read the lemma you use ($\psi:G\to H\Leftrightarrow \psi(r)=1_H$). It seems to be the universal property. But there is a little bit difference.


SupportMapFragment - cannot cast from Fragment to MapFragment

Question: Trying to put together a little MapFragment in an activity I'm building, but am having some trouble getting it all to work.  I know that the Maps api and Play services are both installed correctly, as I did the test tutorial and everything worked fine.
Following the documentation here, I'm running into the following problem:  In my setUpMapIfNeeded method, I can either use getFragmentManager() or getSupportFragmentManager().  When I use the getFragmentManager(), Eclipse is cool with it but when I run, I get a NoSuchMethodError saying that the method is undefined.  When I opt for getSupportFragmentManager(), Eclipse doesn't like it and gives me the error "Cannot cast from Fragment to MapFragment".  So what's the deal?  Any suggestions?
private void setUpMapIfNeeded() {
    //Do a null check to confirm that we have not already instantiated the map.
    if (mMap == null) {
        mMap = ((MapFragment) getSupportFragmentManager().findFragmentById(R.id.map)).getMap();
        if (mMap != null) {
            //do things to the map
            mMap.addMarker(new MarkerOptions().position(LOCATION).title(EXTRA_URL));
            mMap.moveCamera(CameraUpdateFactory.newLatLngZoom(LOCATION,15));
            mMap.getUiSettings().setZoomControlsEnabled(false);
        }
    }
}

Let me know if there's any other code I can provide and I'll be happy to post it quickly.
Comment: do you have the class attribute in the R.id.map?
`class="com.google.android.gms.maps.MapFragment"`
Answer:
I can either use getFragmentManager() or getSupportFragmentManager().

There should be no debate here. If getSupportFragmentManager() is available to you, then you are using the Android Support package's backport of fragments, and this is the method that you must use.

When I opt for getSupportFragmentManager(), Eclipse doesn't like it and gives me the error "Cannot cast from Fragment to MapFragment".

That is because you should not be using MapFragment. You are using the Android Support package's backport of fragments, and therefore you must use SupportMapFragment.
Comment: Sometimes silly mistakes helps others (probably newbies) to grasp a concept, so never feel too silly ;)
Comment: gosh i feel really silly now.  i was definitely aware of that, but apparently hadn't changed them all to SupportMapFragment.  I was importing MapFragment and not SupportMapFragment. Thanks!  Solved my problem.
Answer: This is what i had to do, because i was working under level 11;
    import com.google.android.gms.maps.GoogleMap;
    import com.google.android.gms.maps.SupportMapFragment;
    import android.support.v4.app.FragmentActivity;

    public class MapaActivity extends FragmentActivity {

    private GoogleMap map;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_mapa);

        map =  ((SupportMapFragment)getSupportFragmentManager().findFragmentById(R.id.mapa)).getMap();
    }
}

Answer: hey simply add v4 jar to your libs folder and add to build
then in your Fragment page just import
import android.support.v4.app.FragmentActivity;
you may able to extends FragmentActivity
public class MainActivity extends FragmentActivity implements LocationListener{
}
and it also access getSupportFragmentManager();
SupportMapFragment fm =(SupportMapFragment)getSupportFragmentManager().findFragmentById(R.id.map);
Answer: Change your dependencies to:
implementation 'com.google.android.gms:play-services-maps:16.1.0'
implementation 'com.google.android.gms:play-services-location:16.0.0'