Rather use a sample of a population or analyze a whole population but smaller?

Question: I am validating an automated surveillance program. It is about infection incidence on a hospital ward. I could easily check if the small list of infections produced by the automated surveillance includes all patients that really have an infection (in other words; calculate the positive predictive value). The list of patients not labeled as infected by the program is a whole lot longer. This brings me to two questions:

I'm not sure if I should get a sample size of the infection-negatives and check if they are indeed infection-negative. Or should I get a smaller (for example 6 month cohort with 300 patients) and check all of them?
Is combining methods even an option (check all of the positive-labeled patients and only a sample of the negative-labeled patients)?

Thanks in advance,
Suzanne
Answer: It depends on what you're trying to say about the system. Taking an unbiased sample of all patients regardless of the prediction will get you unbiased population-level statistics. You can calculate things like overall accuracy and prevalence of the predicted positive/negative classes.
If you have a rare infection, however, a small sample size may not adequately characterize the positive class. If the infection is only in 1% of cases, you'd only expect 3 cases in a cohort of 300, which is too few to give you any kind of confidence in your PPV estimate.
It is a reasonable approach to check all the predicted positives as well as a sample of the predicted negatives, as that will give you enough samples in both to get a good estimate of PPV and NPV. If you take this approach, however, you should not try to say anything about prevalence of the classes or overall accuracy, since your sample population is no longer representative of the overall population. You can still do within-class statistics (PPV/NPV), but be careful with anything that's calculated over both classes, since your sampling will have skewed the prevalence of those classes.
I think the combined approach of testing all predicted positives and a sample of predicted negatives is the way to go, particularly if your positive class is rare and would require a very large random sample to adequately characterize.
Comment: Thanks a lot for your answer, it has been very helpful.
The only thing I don't completely understand is what you mean by this: "you should not try to say anything about prevalence of the classes or overall accuracy, since your sample population is no longer representative of the overall population."
Does that mean that I won't be able to say anything about for example incidence of the infections? I understand if that is the case, that will be another study once the method is validated. As long as I can still make conclusions on test measurements such as sensitivity, specificity and NPV/PPV.
Comment: @SuzanneGreup Right, if you take all of your predicted positives and some sample of predicted negatives, you could wind up with any ratio of positives to negatives depending on how big your sample of negatives is - you can't infer anything about the natural incidence of predicted positives there, since it will depend heavily on how you sampled the negatives. You can still compute within-class statistics, but anything that relies on an unbiased sampling (like overall accuracy) won't work - you'd need to randomly sample the population without regard for their predicted status to do that.


Selecionar string aleatoriamente

Question: Preciso de um modo para criar aleatoriamente uma string entre algumas opções.
Digamos que tenho as strings "A" "B" e "C", seria possível um modo para o Java escolher aleatoriamente?
Comment: gera um numero aleatório tira o módulo e atribui o numero a uma string
Answer: Basta usar o método shuffle() disponível para uso em coleções. Este método é feito para ordenar aleatoriamente elementos de uma coleção.
public static void main(String[] args) {
    List<String> letras = Arrays.asList("A", "B", "C");
    Collections.shuffle(letras);
    System.out.println(letras);
}

Para criar a string se for poucos elementos basta fazer uma concatenação simples, caso contrário basta usar um StringBuilder (entenda porque o StringBuilder é importante quando há muitos elementos para concatenar na string).

O AP disse que interpretamos a pergunta de forma errada, então a solução é pegar a primeira letra da string gerada conforme o Bacco comentou abaixo
Ainda existe outra solução:
public static void main(String[] args) {
    String letras = "ABC";
    Random gerador = new Random();
    System.out.println(letras.charAt(gerador.nextInt(letras.length())));
}

Veja funcionando no ideone. E no repl.it. Também coloquei no GitHub para referência futura.
Comment: @Joannis tem o link com a documentação dele. Ele é melhor porque já está pronto e não precisa reinventar a roda. Como o nome diz ele embaralha aleatoriamente os elementos de uma coleção qualquer. Justamente o que o AP quer. Os criadores do Java provavelmente fizeram da melhor forma possível. A sua tem um inconveniente que adicional que é preciso mudar um código mais complexo para mudar ou adicionar um valor.
Comment: @user3511983 mas é aleatório o shuffle. Basta pegar a primeira delas, que vai ser uma aleatória depois do _shuffle_. Shuffle é "embaralhar".
Comment: como funciona esse método @bigown?
desconheço ele, porque seria melhor que minha solução?
pode explicar por favor? se quiser crio uma pergunta só pra isso!
Comment: valew, (eu n tinha visto o link para a documentação, vou dar uma olhada)
vai meu voto a favor =D
Comment: @Joannis note que a sua e a outra resposta estão corretas (pelo menos me pareceu), tanto que votei nelas.
Comment: sim, eu vi, eu e ele postamos quase ao mesmo tempo,
entendi a resposta dele também. só não tinha entendido a sua por não conhecer o suffle. (sou novo em java)
Comment: Obrigado pela ajuda @bigowmn , mas na verdade não quero reordenar, mas sim escolhe uma das 3 aleatoriamente
Comment: @user3511983 Parece que não foi o que perguntou. Note que ninguém entendeu isto. Vou colocar outra solução mas o que o Bacco disse resolve o que você quer.
Comment: O segundo parâmetro de `substring` (no segundo exemplo) não deve ser `1`, pois ele representa o final (*end index*) e não o tamanho (*length*).
Comment: @utluiz é verdade, muita linguagem na cabeça :)
Comment: Nem me fale. Cada linguagem faz substring diferente. :D
Answer: Tenta usar essa função
// Determia as letras que poderão estar presente nas chaves
String letras = "ABCDEFGHIJKLMNOPQRSTUVYWXZ";

Random random = new Random();

String armazenaChaves = "";
int index = -1;
for( int i = 0; i < 9; i++ ) {
   index = random.nextInt( letras.length() );
   armazenaChaves += letras.substring( index, index + 1 );
}
System.out.println(armazenaChaves);

Answer: Gere um numero aleatório com a função random, tire o módulo e a partir daí forme sua string.
import java.util.Random;

public class Random1 {
    public static void main(String[] args) {

        //instância um objeto da classe Random usando o
        //construtor padrão
        Random gerador = new Random();
        int x = gerador.nextInt();
        String string;
        switch (x%3){
        case 0:
            string = "A";
            break;
        case 1:
            string = "B";
            break;
        case 2:
            string = "C";
            break;
        }
        System.out.println(string);
    }
}

Answer: Para selecionar uma String entre muitas num array:
String[] opcoes = { "A", "B", "C", "D", "Z" };
String selecionada = opcoes[new Random().nextInt(opcoes.length)];

Para fazer o mesmo com uma lista:
List<String> opcoes = Arrays.asList("A", "B", "C", "D", "Z");
String selecionada = opcoes.get(new Random().nextInt(opcoes.size()));

Selecionar string aleatoriamente


Let $M$ be set of intervals $[a,b]$, $a,b \in \mathbb{N}$.

Question: Let $n$ be a natural number greater than 1. Let $M$ be set of intervals $[a,b], a,b \in\mathbb{N}, 1 \leq a < b \leq n$ such that for $I,J \in M$ either $I$ and $J$ are disjoint either one is a subset of the other one. Prove by induction that $M$ can contain at most $n-1$ such interval.
It is easy for $n=2$.
Can anyone help with step in induction?
Thanks in advance.
Comment: Hint: the minimum length of such interval is 1.
Answer: Suppose it works for n. Let $M$ be the set of intervals contained in $[1,n+1]$. If all those intervals are of shape $[a,b]$, where $b<n+1$, then all of them are contained in $[1,n]$ and we can apply i.h. to get $|M|\leq n-1 < n$. If not, there is some interval $I=[k,n+1]$ in $M$. Now we count other elements of $M$.
They can be subsets of $[k,n+1]$, there are at most $n-k+1$ of those, by i.h. ; or they can be subsets of $[1,k-1]$ (not of [1,k] because of disjoint intersection condition) and there are at most $k-2$ of them. There is another kind of those subsets- subsets containing $[k,n+1]$, but there are in 1-1 correspondence with subsets of $[1,k-1]$ (we can't have $[1,k-2]$ and $[k-2,n+1]$ at the same time).
This means we have at most $(n-k+1)+(k-2)+1=n$ intervals.


Creating dynamic forms based on the values selected

Question: I'd like to create a HTML form as follows:
<form action="<?php print($action); ?>" method="post">
Possibilites
        <select name="possibility" id="possibility">
        <?php foreach ($possibilites as $possibility): ?>
            <option value="<?php print($possibility['id']); ?>"><?php
                print($possibility['name']); ?></option>
          <?php endforeach; ?>
        </select>
        <rest>The rest of the form</rest>
</form>

When the user selects a certain value from the options menu,
the rest of the form will be generated (instead of <rest>The rest of the form</rest>)
This page accepts a lot of PHP variables that will be used in the rest of the form.
How would you generate the rest of the form based on the value selected from the options?
The rest of the form will accept lots of PHP variables in various input elements generated dynamically. The user should have the possibility to switch between various rests (<rest> ... </rest> until he submits the completed form. The code should not open security holes.
Which approach would you choose and why? (Javascript with loading partial HTML files? Building various DOM trees (with PHP variables in it) for the <rest>...</rest>s or some approach from PHP code)?
Comment: Is there a reason you wouldn't simply include all the options and then show the relevant options as the user fills out the form?  Why must you fetch them separately?  Are there a large number of subsequent options?
Comment: I could include all the options in one form, but it would be uncomfortable for the user to decide what is needed to fill in in every case. I could make disabled fields but the form would be two large, if I would include all the possibilities in one form (with lots of disabled fields).
Comment: No I'm saying hide the options until they're needed. Why is this harder than a show/hide, is what I'm asking.
Comment: @Brad Could you show some example code, to know exactly what you mean (e.g. in Fiddle)?
Comment: Using jQuery... `$('#someDivId').show()` or `$('#someDivId').hide()`.  My question for you is, what's in the rest of this form?  Is it absolutely massive?  Why are you inclined to loading it separately?
Comment: @Brad Hide wouldn't be sufficient. I need to reposition elements as well.
Comment: Whatever you do, that doesn't matter.  That's not the point.  The point is, can you send all the possible form combinations and handle this client side or not?  The question is:  How big, in bytes, are all the options of your form?  Are they small enough that they can all be loaded in one go and you can manage their presentation client-side?
Comment: @Brad Thank you, I will return to this later (I'm limited with time now)
Answer: create an additional div or any other html element where form needs to be displayed

Then in javascript
change event helps to know when any of the option is selected
$(document).ready(function() {
    $('#possibility').on('change', function() {
      var selectedVal =  $('#possibility').val();//value of the option selected
       $.ajax({
           url:'multipleForms.php',
           type:'POST',
           dataType:'json',
           data:{name:selectedVal},
           success:function(resp){
             $('.formGenerated').html(resp);
          }
       });
    });
});

// multipleForms.php
<?php
   $name = $_POST['name'];
   if($name == 'x'){
     $x = 'your form goes here';
     echo json_encode($x);
   }
   //similarly for other conditions
?>

Hope this might help you.
Answer: With jquery tho following code would be an option:
$(document).ready(function() {
    $('#selectId').on('change', function() {
        $('#restDivId').load('partialHtmlPage.php?value=' + $('#selectId').val());
    });
});

Comment: Because the load method updates the DOM you can have 1 root form. I haven't tested this but I don't think it will cause trouble
Comment: So, the partialHtmlPage should contain a root form element? Or could I have in the partial HTML page only the content of the form (with needed elements) and load it as part (children) of the form on the original page?
The problem with having two forms, is that the user would have to click two buttons to send the completed two forms. Or is it possible to have nested forms?
Answer: Building various DOM trees is the faster choice. And there is no better or more secure choice actually, both are the same if we're talking security...
Answer: Use document fragment and dom manipulation to dynamically update your markup. It is fast enough and completely client side. In respect to the use of jQuery, I would rather be framework agnostic at first unless you need some specific options provided by said frameworks.
Doc:

http://www.w3.org/TR/DOM-Level-3-Core/core.html
http://www.quirksmode.org/dom/intro.html
http://jsperf.com/jquery-dom-fragment-vs-createdocumentfragment/2
Answer: To me it would really depend on a few things:

How many additional form elements are you going to need to display?
Do you need record the steps in filling out the form on the server side in some manner?
Do you need to evaluate the data submitted in the form along with data only available on the server in order to determine what next form components to show?

If you only have a few different form options and addining in the DOM as hidden elements on page download does not add what you deem to be unacceptable incremental page download times, you may strongly consider a javascript only solution to hide/show form elements based on your desired logic.  If you answered yes to either of other items, and assuming you want to do all this without a page reload, then you would need to use AJAX methodology.


Do offsets consume degrees of freedom?

Question: In a model with some offset variables (variables with pre-determined regression coefficients), do these variables consume degrees of freedom, with regard of e.g. information criteria?
Pro: You can improve your model by adding N offsets. It would be weird, if you didn't get a penalty for that.
Con: I'm thinking of regressing GDP or GDP per capita against, say, wages. (GDP per capita would be the offsetted model.) A priori, either model can fit better. So, you would just pick the model with a higher R^2, with no penalties for degrees of freedom?
Answer: It doesn't. Very loosely speaking, degrees of freedom are about the moving parts. (If you want a more rigorous explanation, check the How to understand degrees of freedom? thread.) Offset has a constant value of a parameter, equal to 1, that is not estimated from the data. Yes, adding an offset can change estimates of other parameters of the model, and the predictions, but the offset by itself does not change if you change something about the model.
You can easily verify this by yourself by running a model without and with offset in your favorite statistical software:
<code>> summary(lm(mpg~disp, data=mtcars))

Call:
lm(formula = mpg ~ disp, data = mtcars)

Residuals:
    Min      1Q  Median      3Q     Max
-4.8922 -2.2022 -0.9631  1.6272  7.2305

Coefficients:
             Estimate Std. Error t value Pr(>|t|)
(Intercept) 29.599855   1.229720  24.070  < 2e-16 ***
disp        -0.041215   0.004712  -8.747 9.38e-10 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 3.251 on 30 degrees of freedom
Multiple R-squared:  0.7183,    Adjusted R-squared:  0.709
F-statistic: 76.51 on 1 and 30 DF,  p-value: 9.38e-10

> summary(lm(mpg~disp, data=mtcars, offset=wt))

Call:
lm(formula = mpg ~ disp, data = mtcars, offset = wt)

Residuals:
    Min      1Q  Median      3Q     Max
-5.5575 -2.3261 -0.9849  1.8896  7.4938

Coefficients:
             Estimate Std. Error t value Pr(>|t|)
(Intercept) 28.000040   1.320115  21.210  < 2e-16 ***
disp        -0.048225   0.005058  -9.534 1.37e-10 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 3.49 on 30 degrees of freedom
Multiple R-squared:  0.6904,    Adjusted R-squared:  0.6801
F-statistic: 66.91 on 1 and 30 DF,  p-value: 3.942e-09
</code>
Comment: Thanks! Then, is it cheating if you try different offsets or combinations of offsets to improve your model?
Comment: @Dr_Informatics to the same degree as changing or tuning anything else about your model. If doing too much of it, you can end up cherry-picking, or p-hacking, and the model would be useless for inference. If you care about prediction only, its the same as tuning any other parameters, you can end up overfitting to the training, or validation, data, depending on how you asess the performance of the model after the changes.
Comment: that's a very nice way to put it. P value fishing came to my mind! Basically, if you are a total purist, you should count the number of models you do, and then do a multiple testing correction. Never heard of anyone doing so, though.

Do offsets consume degrees of freedom?


Find convex envelope from the non-convex function to prove globally optimal using branch-and-bound

Question: Based on this reference branch-and-bound methods can obtain globally optimal solutions to nonlinear programming problems in which a non-convex function is to be minimized. I have a non-convex function as follows:
$$\min_{\mathbf{x}} F(\mathbf{x}) = \min_{\mathbf{x}} \sum_{i=1}^I \bigg( x_i + (1 - x_i) \Big[\big(\prod_{j=1}^J1-x_j\big)c\Big]\bigg),$$
where $c$ is constant and $\mathbf{x}$ is a vector of binary variables. I want to prove that this function can achieve the global optimal solution using branch-and-bound. To realize that, I need to find the convex envelope of the function $F^c(\mathbf{x})$. If I can find that
$F^c(\mathbf{x^*}) = F(\mathbf{x^*})$,
where $\mathbf{x}^*$ is the optimal solution, thus the function can achieve the globally optimal solution. However, I do not understand how to get the convex envelope of my function. Is there any way to prove that the convex envelope of my function has the same result as my non-convex function above when branch-and-bound algorithm is applied?
Comment: Seems be parenthesis error. Should it be after $\min_{\bf x}$ before $\sum$ ?
Comment: No reason to mess around with nonlinear models here, as you can represent that whole thing using linear constraints, by noting that the products are $0$ if any of the involved variables is $1$, and $c$ if they are all $0$. The whole thing is thus easily linearizable by introducing some extra variables and constraints
Comment: No I was not directly answering your question, but more adviced you to solve it using integer linear programming instead as the problem is easily MILP-representable, and thus solvable using branch-and-bound over linear programs (for which there is a plethora of available solvers)
Comment: @JohanLöfberg So, to get the convex hull from that function, I need to do as you suggest "linearization"?
Comment: @JohanLöfberg So is it correct that instead of finding the convex envelope, I need to find the linearization of the non-convex problem? Then, do you know how to prove that using branch-and-bound, my function can attain the globally optimal solution? I have solved the problem using neos solver and Matlab, however, I need mathematical analysis for globally optimal proof.
Comment: No. However, finding the convex envelope for your function is most likely not trivial, I have no idea how to proceed. Also, your question is a bit weird. When solving problems using b&b, you don't start by deriving the convex hull of anything. In fact, deriving the convex hull of a combinatorial object is typically as hard as solving the problem in which the object is used. Instead, you just use some outer approximation and handle the fact that it isn't a convex hull by branching. If you use the convex hull, you basically don't need b&b, roughly speaking.
Comment: I.e., what I am saying is that the easiest way to solve your problem is to perform a standard linearization of the objective, and solve using MILP.
Comment: @JohanLöfberg thanks for the explanation. Thus, to prove the globally optimal solution using the convex hull should be intractable, right? Actually, I want to find the unique feature of my non-convex problem then prove that by using b&b, I can attain the global optimum for my problem. Do you have any suggestion about the way I need to prove?May I prove using the linearization you have mentioned?
Comment: The proof is by construction if you simply perform standard linearization of binary monomials and then use a MILP solver. I have no idea what you mean when you say you want to find the unique feature of your non-convex problem. Once again, it is trivial to linearize and solve globally using a MILP solver.
Comment: @JohanLöfberg Thanks for your suggestion. I will figure out your idea to realize the proof using linearization. Is it similar to this link? https://math.stackexchange.com/questions/2875627/how-to-solve-binary-nonlinear-programming-problems/2875668#2875668. The proof of b&b which I found from the book (as I mentioned in my question above) is the standard proof, I want to prove using my non-convex problem, not the standard proof but it seems hard to do.
Answer: As a follow up to the comments where I argue that a simple way to solve the problem is to write it as a MILP instead.
Introduce a new binary decision variable $q$ representing $\prod_{j=1}^J(1-x_j)$. This can be accomplished with the constraints $q\leq 1-x_j, q\geq \sum_{j=1}^J(1-x_j)-J+1$.
Introduce yet another set of binary variables $s_i$ representing $(1-x_i)q$. This can be modelled with the constraints $s_i \leq q, s_i \leq 1-x_i, s_i \geq q + (1-x_i)-1$.
Your objective is now $\sum_{i=1}^I x_i + cs_i$, which should be minimized subject to the linear constraints above.
Comment: Thanks a lot for the answers. I will learn using your approach to prove my non-convex problem. After linearizing the problem, I would get MILP problem. Is it true that MILP is always convex?
Comment: No MILPs are not convex, by the very definition since it contains integer variables, and they have exponential worst-case complexity. MILPs are however routinely solved as it is one of the most researched and practically used problem class there is.
Comment: Clearly understand now. Thanks. One more thing, if I have $x$ is positive non-integer, $y$ is binary, thus $r = xy$ has constraints as the link here? https://orinanobworld.blogspot.com/2010/10/binary-variables-and-quadratic-terms.html
Comment: Yes, just as easy. It is a simple implication which you model using big-M
Comment: Thank you a lot
Comment: Hi Johan...I want to confirm one thing. I have found the MILP problem based on your answers. Then, as you said, my problem is still not convex even though I got MILP. If I solve this MILP using branch-and-bound, is it still valid that my MILP problem will obtain the globally optimal solution? Do you have any reference for this statement?
Comment: Your problem will never be convex as it contains binary variables. That's the whole idea with b&b. You replace the problem with an outer approximation (relax binary variables) and then branch and bound until you've effectively searched through all possible combinations and can conclude you've found the globally optimal solution. The NLP->MILP reformulation is just to get a first step to create an equivalent problem for which there are efficient (in practice, hopefully, as per theory it can never be efficient) global solvers.
Comment: Thanks. Got it. When we relax the binary variables, we will get the lower bound of the solution (if we have minimization). Can I say that every time we branch problems into subproblems (we still have continuous relaxations), we guarantee that globally optimal solution (in this case global lower bound) is achieved?
Comment: The whole idea with branch and bound is to discover globally optimal solutions, so your questions are odd.


How to execute dynamic procedure

Question: I am trying to make a dynamic procedure which populates a table within data already exist in the table. I have two variables for primary column and not primary key columns. I passed a sequence (generate.nextval) to primary key column. It will insert specific number of rows which is passed as a parameter (amount)
Create or replace procedure v_populate (table_name in varchar2, amount in number)
IS
v_pk varchar2(1000);
v_pk2 varchar2(1000);
V_dyntask LONG;
V_dyntask2 LONG;

CURSOR C1 IS
'select REPLACE(REPLACE(REPLACE(xmlagg(xmlforest(cols.column_name )
ORDER BY cols.column_name ),''</COLUMN_NAME><COLUMN_NAME>'','',''),''<COLUMN_NAME>''),''</COLUMN_NAME>'') XMLAGG  --cols.column_name
FROM all_constraints cons, all_cons_columns cols
WHERE cols.table_name ='''||table_name||'''AND cons.constraint_type NOT IN (''P'',''U'',''R'')
AND cons.constraint_name = cols.constraint_name
AND cons.owner = cols.owner
ORDER BY cols.table_name, cols.position';

CURSOR C2 IS
'select
                    REPLACE(
                      REPLACE(
                        REPLACE(xmlagg(xmlforest(cols.column_name ) ORDER BY cols.column_name )
                               ,''</COLUMN_NAME><COLUMN_NAME>'','','')
                              ,''<COLUMN_NAME>'')
                             ,''</COLUMN_NAME>'') XMLAGG --cols.column_name
               FROM all_constraints cons, all_cons_columns cols
              WHERE cols.table_name ='''||table_name||'''
                AND cons.constraint_type = ''P''
                AND cons.constraint_name = cols.constraint_name
                AND cons.owner = cols.owner
           ORDER BY cols.table_name, cols.position
              ';

BEGIN

FOR F1 in C1
LOOP
v_pk:=C1.COLUMN_NAME;
END LOOP;

FOR F2 in C2
LOOP
v_pk2:=C2.COLUMN_NAME;
END LOOP;

V_dyntask2:= 'INSERT INTO ' ||table_name|| '
                        ( '||v_pk2||' , '||v_pk||' )
                 select generate.nextval,'||v_pk||'
                   from ' ||table_name || '
                  where rownum <='||amount;
EXECUTE IMMEDIATE V_dyntask2;
COMMIT;
end;

I keep getting this error:
PLS-00103: Encountered the symbol "select REPLACE(REPLACE(REPLACE(xmlagg(xmlforest(cols.column_name" when expecting one of the following:     ( select  The symbol "select" was substituted for "select REPLACE(REPLACE(REPLACE(xmlagg(xmlforest(cols.column_name" to continue.
PLS-00103: Encountered the symbol ";" when expecting one of the following:     , * & - + / at mod remainder rem       as    from ||
Comment: New to PL/SQL? Couldn't you find something simpler than dynamic SQL? Well, you'll have to start over, creating v_dyntask2 step-by-step, testing **frequently**.
Comment: I'm not going to debug it for you. A piece of advice, though: instead of executing it, first `dbms_output.put_line(v_dyntask2)` and check whether it looks OK. Of course it doesn't - otherwise, it would work. Don't execute it until you fix the error.
Comment: I tried it now, it gives the same error. I am a bit new to plsql probably I am missing something @Littlefoot
Comment: Your code looks really strange, what do you try to achieve? The loops are useless, you just take the last row form the query. In your query your first create  XML object with `xmlagg(xmlforest(...` but then you remove the XML tags again to get plain text.
Comment: Data type `LONG` is deprecated for almost 20 years, don't use it.
Comment: I was trying to get column names written side by side. I have oracle 10g so it doesn't support LISTAGG. I wasn't that sure either about if I should use for loops. I was just trying to pass column names to variable. @Wernfried Domscheit
Answer: I don't think that you need a dynamic query for cursors.
Use the following code and let me know if it works:
CREATE OR replace PROCEDURE V_POPULATE (
    P_TABLE_NAME   IN           VARCHAR2,
    AMOUNT       IN           NUMBER
)
IS
    V_PK VARCHAR2(1000);
    V_PK2 VARCHAR2(1000);
    V_DYNTASK LONG;
    V_DYNTASK2 LONG;

BEGIN

FOR F1 in (select REPLACE(REPLACE(REPLACE(xmlagg(xmlforest(cols.column_name )
ORDER BY cols.column_name ),'</COLUMN_NAME><COLUMN_NAME>',','),'<COLUMN_NAME>'),'</COLUMN_NAME>') column_name  --cols.column_name
FROM all_constraints cons, all_cons_columns cols
WHERE cols.table_name = P_TABLE_NAME
AND cons.constraint_type NOT IN ('P','U','R')
AND cons.constraint_name = cols.constraint_name
AND cons.owner = cols.owner
ORDER BY cols.table_name, cols.position)
LOOP
v_pk:=F1.COLUMN_NAME;
END LOOP;

FOR F2 in (select
        REPLACE(
          REPLACE(
            REPLACE(xmlagg(xmlforest(cols.column_name ) ORDER BY cols.column_name )
                   ,'</COLUMN_NAME><COLUMN_NAME>',',')
                  ,'<COLUMN_NAME>')
                 ,'</COLUMN_NAME>') column_name --cols.column_name
   FROM all_constraints cons, all_cons_columns cols
  WHERE cols.table_name =P_table_name
    AND cons.constraint_type = 'P'
    AND cons.constraint_name = cols.constraint_name
    AND cons.owner = cols.owner
ORDER BY cols.table_name, cols.position)
LOOP
v_pk2:=F2.COLUMN_NAME;
END LOOP;

V_dyntask2:= 'INSERT INTO ' || P_table_name|| '
                        ( '||v_pk2||' , '||v_pk||' )
                 select generate.nextval,'||v_pk||'
                   from ' ||P_table_name || '
                  where rownum <='||amount;
EXECUTE IMMEDIATE V_dyntask2;
COMMIT;
end;

Cheers!!
Comment: Thanks for your help @Tejash. But now it gives this error: PLS-00302: component 'COLUMN_NAME' must be declared
Comment: In cursor, There was no column_name alias so it was giving the error. Updated the answer. Check now!!
Comment: Thanks @ Tejash !!! Now I can run the script without an error. I know I am asking too much but when I execute the procedure I get an error stating: PLS-00357:Table,View Or Sequence reference 'EMPLOYEES' not allowed in this context.
Answer: You don't need dynamic string for cursor and you don't need to juggle with XML. I would do it like below.
CREATE OR REPLACE PROCEDURE v_populate (table_name IN VARCHAR2, amount IN NUMBER)

    v_pk VARCHAR2(1000);
    v_pk2 VARCHAR2(1000);
    sqlstr VARCHAR2(30000);

    CURSOR C1 IS
    SELECT cols.column_name
    FROM all_constraints cons
        JOIN all_cons_columns cols USING (OWNER, constraint_name)
    WHERE cols.table_name = table_name
        AND cons.constraint_type NOT IN ('P','U','R');
    ORDER BY cols.column_name; -- For check constraints ('C') POSITION is NULL, thus 'ORDER BY POSITION' is pointless!

    CURSOR C2 IS
    SELECT cols.column_name, cols.position
    FROM all_constraints cons
        JOIN all_cons_columns cols USING (OWNER, constraint_name)
    WHERE cols.table_name = table_name
        AND cons.constraint_type = 'P';

BEGIN

    sqlstr := 'INSERT INTO ' ||table_name|| ' ( ';
    FOR aCol IN C2 LOOP
        sqlstr := sqlstr || aCol.COLUMN_NAME||',';
    END LOOP;

    FOR aCol IN C1 LOOP
        sqlstr := sqlstr || aCol.COLUMN_NAME||',';
    END LOOP;
    sqlstr := REGEXP_REPLACE(sqlstr, ',$', ') ');

    sqlstr := sqlstr || 'SELECT generate.nextval,';
    FOR aCol IN C1 LOOP
        sqlstr := sqlstr || aCol.COLUMN_NAME||',';
    END LOOP;
    sqlstr := REGEXP_REPLACE(sqlstr, ',$', ' ');
    sqlstr := sqlstr || 'FROM ' ||table_name|| ' WHERE ROWNUM <= :a';

    DBMS_OUTPUT.PUT_LINE ( sqlstr ); -- Only for testing to verify the generated command

    EXECUTE IMMEDIATE sqlstr USING amount;

END;

Be aware, the procedure will fail if your table has a composite Primary Key , i.e. Primary Key build by more then one column. It also fails if your table does not have a Primary Key at all.
Maybe you should skip the Primary Key and apply generate.nextval via Row-Level BEFORE INSERT trigger.
Note, also 17 years old Oracle 10g was already supporting modern ANSI join syntax.

How to execute dynamic procedure


Strange "less than" accent or ornament in Sullivan's "The Mikado" score

Question: In the old Kalmus handwritten conductor's score for The Mikado, a few notes in the overture had a unfamiliar notation.  For example, here's Measure 50 from the oboe part (other instruments have similar notation):

I'm reading the small vertical marks as "staccatissimo", but what's that backwards "less than" sign?  Isn't a standard "accent" written as a "greater than" sign?
Comment: If "16va" is a thing, I guess writing accent marks backwards isn't so bad.
Comment: This "backwards" symbol appears throughout the overture, but in Act I, Scene 1, he uses sf and in Act 1 scene 2 there are "normal" accents (e.g., m. 104; 2 m. after L).
Comment: I've seen the "reverse" accent in Musescore (in the articulations section) but never used it myself. Whenever I've seen and listened to misuses of it in scores in Musescore's website, I never quite figured out what it did.
Answer: I received this in a private email from Albert Bergeret, the Artistic Director of the NY G&S Players:

I had never taken notice of the mark that you mention, and I wouldn't necessarily consider such notations in a Kalmus printed score to be those of the composer, however, by my own impression of the moments noted I would suggest that it implies a leaning into a particular note during its duration rather than the accent mark which implies a hard attack followed by a diminuendo.  That's how [I] hear the places where the notation you have noted appear.

I hope this answers your question, although I must say that my response is subjective, not authoritative.
Comment: I looked for an image of Sullivan's manuscript, but I didn't find it.  think it's just a (poorly drawn) one-note crescendo.  From the clarinet part on down the score, the angle is much more acute and the lines are slightly longer, making the sign look much more like a crescendo than an accent.  In the brass parts it even extends a bit farther to the right than the note itself, which makes it look even more like a crescendo.
Comment: Though if that were the case it ought to be under the staff, not over it.  I've written to the Royal Academy of Music library to see if they can help (they hold the autograph).
Answer: It can either be a typo for an accent (to be corroborated by checking with analog passages and/or parts), or it can mean a reverse accent: a note starting softer and getting louder.  Of course that requires an instrument with continuous tone control, like a bowed string or wind instrument.
Comment: "Reverse accent"? Is that a thing? Or did he invent it? Maybe that's why he got knighted.
Comment: This would be better left as a comment, as it's speculative and nonspecific.
Comment: Reverse accents are pretty common in double bass parts I've read, especially on longer notes but sometimes fairly short ones too. You can just think of it as a quick crescendo on a note, just like a normal-direction accent can be thought of as 'make the attack louder', ie a short de-crescendo.
Answer: The Musescore desktop software program has the "reverse accent" labelled as "Fade in":


Linux software to maintain old/backup versions of directory tree

Question: I am replacing an old Linux file server serving NFS and CIFS. For the new server (still serving CIFS and NFS), I would like to have software that automatically and efficiently maintains old revisions of files in parallel trees, so that they can be accessed by users without special tools. I am looking for software that is akin to Time Machine or Flyback, but works well on a server.
The dataset is some 10000 files weighing maybe 60 GB. Changes are relatively few, usually less than 100 files changes daily.
Using LVM snapshots will not cut it, as the old revisions must reside on a separate set of disks from the live data.
Edit: To clarify: keeping old revisions is non-vital addition to the solution, so any suggestion will have to stay in the range of some hundred euros.
Comment: NetApp seems pricey from what I can find out. I'm trying to add value to the solution for the cost of a few extra disks. There seems to be some interesting ZFS alternatives, tho. Merits further investigation.
Comment: Sounds like you want a NetApp or ZFS appliance - would these make sense in your environment?
Comment: I second the NetApp suggestion.
Answer: Many people use rsync and hard links for this. Mike Rubel wrote the classic guide, Easy Automated Snapshot-Style Backups with Linux and Rsync. You could write your own script to do it or use an existing one like rsnapshot.
Comment: rsnapshot seems like a good candidate at a quick reading. I will investigate. Thx.
Comment: dirvish is a similar tool http://www.dirvish.org/
Answer: You could use FreeBSD and ZFS. ZFS is a real cool system and you can put the whole system in ZFS its a bit tricky to install but here are some informations. ZFS
In the ZFS system you can say in which interval you want a snapshot. You can configure a lot. Its possible that he can pack old versions. For example: The first 12 hours you have a snaphot every 30 minutes. Afer the 12 hours he pack it to every three hours. After a month you can set it to weekly or monthly.
Over your share you can access to a share named ".zfs" here are all snapshots with time and date. You can go to the folder and pick the file.
Comment: While the idea is tempting, I fear I would be lynched for installing FreeBSD on the common fileserver. ;)


compressed a json data, angularJs

Question: I 'm developing a web aplication with .net API and back end with angularjs restfull , problem I have big data json (4 mb) , is there a solution to reform the json data or compressed ? for optimized  the response time
Comment: Do you really need all the data? Can't you break them down into smaller pieces?
Answer: There are few ways to compress JSON data
1) Instead of Using arrays of object, try to use arrays of array
Ex:
[{"name":"qwerty","mobile":"000112233","location":"NorthPole"},{"temp":"0 deg","sunrise":"qw","sunset":"er"}]

to
[["qwerty","000112233","NorthPole"],["0","qw","er"]]

2) Remove blank space in json object
3) if you want to maintain array of objects. then reduce size of variable name:
[{"n":"qwerty","m":"000112233","l":"NorthPole"},{"t":"0 deg","r":"qw","s":"er"}]

My suggestion is not to pass large weight data in webapi. use async calls to fetch data which is required  on  demand.
Answer: Other than using some minification library in your server-side code, you could setup GZIP compression on the web server itself. All browsers today support GZIP and will send Accept-Encoding: deflate, gzip header, so the server would know to compress before sending the respone.
Here are links to setup compression on Apache and IIS:
http://httpd.apache.org/docs/2.0/mod/mod_deflate.html
https://technet.microsoft.com/en-us/library/cc771003(WS.10).aspx

compressed a json data, angularJs


Practice chord progressions along with scales?

Question: The Question
Are there any standard exercises where you play a chord progression with one hand while playing a scale with the other hand?
I'm adding the usual I–IV–I–V–I at the end of each scale (forcing myself to identify the actual chord name as I play it), but that doesn't give me any practice with the other chords (let alone any important 7th chords, where I'm very weak).  Nor does it exercise the practice of chords on one hand with melody on the other hand.
I could come up with my own chord progression, but it would be nice to have something with pedagogical value rather my meager attempts at cobbling together some chords that just rotate through the possibilities in a vaguely meaningful way. I would imagine someone has put together something using common cadences or turn-arounds that are commonly encountered, etc.
Context
After 15 years, I've finally dusted off my old piano book and am re-learning to play the piano (and my prior experience is just a single semester undergraduate course!)  I was quickly able to get back up to speed with the simple five-finger position exercises/songs from my old piano book.  To force myself to move beyond these simple/comfortable pieces, I started playing some familiar christmas carols and hymns.  I then got distracted for a few months learning about chord progressions, chord inversions, roman numeral notation, etc. and I had some fun transcribing several carols and hymns (which are written for 4 part vocal harmony - currently rather difficult for me to play) into the more simple right-hand melody + left-hand chord progression (great way to force myself to learn LilyPond).
It's clear that the only way to improve my technique is to put some serious time and effort into scales.  I've spent the past few weeks working on scales and I'm making great progress.  However, I would like to use my time practicing scales to also practice the important chord shapes and progressions in each key as well as to learn the chord progressions in terms of the actual chord names (I've been relying on roman numerals to just play chord shapes, ignoring what the chords actually are).
Comment: For the technique you want to enhance could you be more specific?  Scales and chords are great but do you have a reason for playing them?  Is it theory related, technique (speed, agility), or something else?
Comment: My motivation is largely one of skill practice: for many songs, you play chord progressions with the left hand and the melody in the right hand.  So, instead of the usual two-handed scales practiced in every key, I would like to practice chord progressions with one hand along with scales on the other hand for each key.  I guess this isn't commonly done based on the lack of answers, but I thought it would make for good skill practice.
Answer: As technical exercises, you normally do scales, arpeggios and chord progressions. If you want to do a richer combination of those, like doing different things in different hands, etc... you normally don't do exercises, but play Studies.
Czerny has hundreds of them, for different levels, and some of them might be useful for what you want. Problem is that they are many, so you have to spend some time to find the ones that are suitable to what you want to practise, and adapted to your level.
You can check your closest music shop, of check in the Petrucci Music Library, where you can download many music scores for free (and legally).
Comment: The closest things I've found to the "chord progression technical exercise" are the cadences I had to practice for Royal Conservatory of Music piano exams. I-IV-V-I was particularly emphasized.
Answer: I don't know of any standard exercises other than scales, arpeggios. What I tend to do is play regular patterns of chords on the left hand  but keeping in the scale while the other hand extemporises scale wise. Then change hands. Makes you learn the scales and the related chords and also which notes and runs work with each chord.
Patterns to try while playing sequence of three or four notes on other hand (either melody or walking bass):
Chord of each note.ie C dm em F etc
Every other note ie thirds C em G bdim dm F  etc
A pattern of inversions where just 1 or two fingers move either up or down.
ie C F (move two fingers up) C G (move two fingers down). repeated on each note of the scale.
Start off playing as chords, then arpeggios or any other regular order such as
1 5 3 5. Its amazing when playing from music how some compositions are just made up of these sequences.
These exercises teach your ear the chords that are strongly related to each other and are really useful for composing.
Try in each key then try moving from 1 key to another by using chords occurring in both keys.
Answer: A great deal of jazz utilizes the ii-V-I progression. And you can throw in a tritone substitution for color; my favorite is presently iimin11-biidom7-Imaj7. Enjoy exploring!
In the key of A, using root-6 voicings in the fifth position, the open high E string can double the fretted E on the B string and create a nice effect.


Metric circle on a sphere

Question: Consider $\mathbb S_R^2 \subseteq \mathbb R^3$ be a sphere of radius $R$. Let $N$ denote the north pole and $0 < r < R \pi$. Define the metric circle with center N to be all points, $x \in \mathbb S_R^2$ that
$$d_{\mathbb S_R^2}(x,N) = r$$
Here $d_{\mathbb S_R^2}$ is the induced metric on the sphere. Parameterize $\mathbb S_R^2$ with the usual spherical coordinates:
$$\Gamma:[0,2\pi) \times [0,\pi]\to\mathbb{R}^3 \quad \quad  (\phi, \theta)\mapsto (R\sin\theta\cos\phi,R\sin\theta\sin\phi, R\cos\theta),$$
My hunch is that the metric circle with center N is spherical circle:
$$\Gamma ( [0,2\pi) \times \{ r / R \} ) \subseteq \mathbb S_R^2$$ However, I cannot justify this claim using only elementary arguments using only elementary techniques.
Edit: Bump!
Comment: Your definition of a spherical circle is unclear.
Comment: @MoisheKohan I have now edited the post. Hopefully, it is clear now.
Comment: Hmm, to me the induced metric on sphere is the angular metric. Do you mean the chordal metric? (The restriction of the Euclidean distance.)
Comment: @MoisheKohan I think so. I am assuming the chordal metric and the induced metric are the same thing.
Comment: The answer to your question is positive and one does not need calculus to prove this, just elementary geometry. The key is that the chordal distance determines the angular distance and vice versa.
Comment: @MoisheKohan I'm bad at elementary geometry, I guess, lol. Especially with synthetic arguments. Never got around to studying them properly. If you could elaborate a bit, I'd appreciate it.
Comment: I will, later on....
Comment: Thanks a lot...
Answer: Consider sphere $S(0,R)$ of radius $R$ in the Euclidean 3-space $E^3$ and centered at the origin $0$.
There are two metrics one defines on $S(0,R)$:

The chordal metric $d(P,Q)=a=|P-Q|$, the restriction of the Euclidean distance function to the sphere.

The angular metric $\alpha=\angle(P,Q)$ which is the angle of the triangle $P0Q$ at $0$ (if you like, you can multiply it by the radius $R$).

Let us express one metric in terms of the other. For this we use the cosine formula:
$$
a^2= 2R^2 - 2R^2 \cos(\alpha)= 2R^2(1- \cos(\alpha)).
$$
Since $0\le \alpha\le \pi$, it follows that $\alpha$ determines $a$ and vice versa:
$$
\alpha= \arccos(1-\frac{a^2}{2R^2})
$$
Now, consider the circle $C(P,\alpha)\subset S(0,R)$ of radius $\alpha$ centered at $P$, where the radius of the circle is computed in terms of the angular metric.  Then a point $Q\in S(0,R)$ lies on  $C(P,\alpha)$ if and only if
$$
a:=d(P,Q)= R\sqrt{2(1-\cos(\alpha))}.
$$
In other words, $C(P,\alpha)$ is the same as the circle in $S(0,R)$ of the chordal radius $a$ and centered at $P$, answering your question. (Your question has a typo, I think, you meant $r=p$, but, regardless, your conjectural formula for the angular radius in terms of the chordal radius is wrong.)
Comment: I have corrected the typos. Thanks.

Metric circle on a sphere


How to connect an "SSLv3 only" web client to a web server that only accept TLS1.0 - TLS1.2

Question: I have a legacy embedded HTTP client (very old) that supports SSLv3, TLS1.0, TLS1.1 and TLS1.2. I would like to remove SSLv3 for security reasons. I rebuilt the openssl library and wanted to verify if this webclient does not use SSLv3 during secure HTTP connection. This client also is programmed to connect to a specific web server on the internet, which already does NOT support SSLv3.
I tried to verify if it work by using a proxy in between the embedded web client and the internet webserver.
Setup:
Webclient  <---->  Squid proxy <----> Https://www.loremipsum.com/xxx/xxx

Setting:
Webclient
    -SSLv3 only
Squid proxy:
    -Installed in windows server 2012
    -client TLS version --> TLS1.0, TLS1.1, TLS1.2 enabled
    -server TLS version --> SSLv3 Only
External web server:
    -only supports TLS1.0, TLS1.1 and TLS1.2

Expected results:
Before removing SSLv3: SSLv3 used in HTTPS connection

After removing SSLv3: Webclient cannot connect with handshake failure code in packet

Actual result:
Before removing SSLv3: **TLS1.0** is used instead of SSLv3 (this part is my problem)

After removing SSLv3: TLS1.0 used in HTTPS connection

Anyone here knows why TLS1.0 is used instead of SSLv3 during the actual test?
Or can you recommend any tool that I can use to confirm this change?
Thank you very much.
Comment: No client will prefer SSLv3 to TLS unless explicitly misconfigured to do so. Obviously you should not attempt to do this. It's not clear, therefore, that you are actually having a problem.
Comment: But you seem to have already confirmed its removal!
Comment: Hi @MichaelHampton, thank you for your response. I understand that no client nowadays will prefer SSLv3, its completely unsecure. However, as I pointed out, I just want to confirm its removal by the setup above.
Comment: Sorry for the confusion. I edited the squid proxy above, I interchanged client and server setting in OS where proxy is installed. I expect that the communication should fail given the scenario (should not use TLS1.0) since only SSLv3 was turned in in squid server side.
Answer: As why tls is used in your test: the client will go through its available ciphers and try the strongest ciphers first. So it will always prefer tls over ssl3.
If the website is visible form the internet you can use one of test services like https://testtls.com to verify which protocol is enabled/disabled.
If (as I am guessing) the service is only available internally you can use curl and specify the protocol:
<code> curl -vvv --sslv3 https://www.myhost.tld
</code>
You can go into even more detail and specify each cipher individually: https://curl.se/docs/ssl-ciphers.html
An alternative is to use openssl directly: (https://www.misterpki.com/openssl-s-client/)
<code> openssl s_client -connect www.host.tdl:443 -ssl3
</code>
Note that in both instances you need openssl/curl with ssl3 still available (newer ones do not have ssl compiled at all)
What you need to configure in squid is something like this (you might have to tweak the options depending on the openssl version):
<code>https_port ... options=SSLv3:NO_SSLv2:NO_TLS1:NO_TLS1_1:NO_TLS1_3
http_port ... ssl-bump options=NO_SSLv3:NO_SSLv2:NO_TLS1:NO_TLS1_1:NO_TLS1_3
</code>
Comment: Hi thank you for answering. "So it will always prefer tls over ssl3." --> but in proxy server, I set it to only accept SSLv3 on the server side. Why it is still choosing TLS1.0?
Comment: can your povide your squid config options where you disabled all but ssl3?
Comment: @cabrillosa : A side note, a proxy server does not pass-through the SSL connection... The connection from client to proxy and from proxy to webserver are seperate connections! So it may be, that the client opened an SSLv3 connection to the proxy, and the proxy opened an TLS connection to the webserver...

How to connect an “SSLv3 only” web client to a web server that only accept TLS1.0 - TLS1.2


Exposing message.properties to javascript?

Question: Is there an easy way to expose all translated strings to javascript? I want to be able to use translated strings in my JS files without manually building them in GSPs.
Is there an easy way to do this?
Comment: Just to add a valuable introductional link that uses the above JAWR as grails plugin:
http://mrhaki.blogspot.de/2011/11/grails-goodness-internationalize.html
Answer: The Jawr plugin (good plugin to use to optimize your JS and CSS resources anyway) can expose parts or the whole of your i18n message bundles to Javascript:
http://jawr.java.net/docs/messages_gen.html
Comment: you can use the resources plugin to minify JS. AFAIK this will be moved into core grails in version 1.4
Comment: How do you minify JS with jawr?
Comment: The link in this answer seems to be broken. This link seems to work: https://j-a-w-r.github.io/
Answer: Expose i18n/messages.properties as JSON
I18nController.groovy
package myproject

import grails.converters.JSON
import grails.plugin.springsecurity.annotation.Secured

@Secured("permitAll")
class I18nController {
  static defaultAction = "index"

  I18nService i18nService

  def index() {
    render i18nService.getAllTranslations(params, request) as JSON
  }
}

I18nService.groovy
package myproject

import grails.transaction.Transactional
import grails.web.servlet.mvc.GrailsParameterMap
import org.springframework.context.MessageSource

import javax.servlet.http.HttpServletRequest

@Transactional
class I18nService {

  MessageSource messageSource
  LocaleService localeService

  /**
   * http://www.razum.si/blog/grails-javascript-i18n-messages
   * @return
   */
  Properties getAllTranslations(GrailsParameterMap params, HttpServletRequest request) {
    Locale locale = localeService.prepareLocale(params, request)
    Properties keys = messageSource.withTraits(MessagePropertiesTrait).getMessageKeys(locale)

    //def jsKeys = keys.findAll { code, value -> ((String) code).startsWith('js.') }
    //render jsKeys as JSON

    return keys
  }
}

LocaleService.groovy
package myproject

import grails.transaction.Transactional
import grails.web.servlet.mvc.GrailsParameterMap
import org.springframework.web.servlet.support.RequestContextUtils

import javax.servlet.http.HttpServletRequest

@Transactional
class LocaleService {

  public static final String DEFAULT_LANG = 'es'
  public static final String PARAM_LANG_KEY = 'langKey'
  public static final String COOKIE_LANG_KEY = 'language'

  Locale prepareLocale(GrailsParameterMap params, HttpServletRequest request) {
    Locale locale = null
    int i = 0

    List<String> langStrings = new ArrayList<>()
    langStrings.add(params.get(PARAM_LANG_KEY) as String)
    langStrings.add(request.cookies.find { COOKIE_LANG_KEY == it.name }?.value)

    while ((locale == null) && (i < langStrings.size())) {
      String str = langStrings.get(i)
      if (str) {
        try {
          locale = new Locale(str)
        } catch (Exception ex) {
          ex.printStackTrace()
        }
      }

      i++
    }

    if (locale == null) {
      try {
        // From Request
        locale = RequestContextUtils.getLocale(request)
      } catch (Exception e) {
        // Default
        locale = new Locale(DEFAULT_LANG)
      }
    }

    return locale
  }
}

Helpers.groovy
Create this file at utils/myproject:
package myproject

import groovy.transform.CompileStatic
import groovy.transform.TypeCheckingMode

@CompileStatic(TypeCheckingMode.SKIP)
trait MessagePropertiesTrait {
  Properties getMessageKeys(Locale locale) {
    this.getMergedProperties(locale).properties
  }

  Properties getPluginMessageKeys(Locale locale) {
    this.getMergedPluginProperties(locale).properties
  }
}

UrlMappings.groovy
Add follow URL mapping:
"/i18n/$langKey?(.$format)?"(controller: 'i18n', action: 'index')

Usage
URL
http://localhost:8080/i18n # Default language
http://localhost:8080/i18n/.json # Default language

http://localhost:8080/i18n/en.json
http://localhost:8080/i18n/en

http://localhost:8080/i18n/es.json
http://localhost:8080/i18n/es

Cookie
Set a Cookie named language with the language (es, en, ...) and call http://localhost:8080/i18n or http://localhost:8080/i18n/.json


sc: spreadsheet calculator, execute with a script

Question: Suppose I have a bunch of "homegeneous" sc-files, and I need to execute one specific sequence of commands in each of them. Essentially, I need this:
for f in `ls *.sc`; do sc $f <do something>; done

In <do something>, I would like to supply a sequence of sc commands in a script. For starters, let it be something like, for the cell Z9,
assign some text string. The man-page does not feature a description of how this could be done; the online literature is scant, since sc is (unfortunately) not as popular as it (IMHO) deserves to be.
Answer: I read the docs more carefully, and understood that something of this kind can be done:
sc -v -WA0:D13 myfile.sc.
This outputs to stdin the table contained in the range A0:D13.


Find target bundle(s) for a specific entityreference field

Question: I am creating a simple administration page that shows all content types and some details about each content type field.

I am getting the list of content types with node_type_get_types()
I am getting the list of fields with field_info_instances()

When the widget module is entityreference I'd like to display the target bundle(s) of that entity reference field.
foreach(node_type_get_types() as $value) {

  $fields = field_info_instances('node', $value->type);

  // Loop through fields
  foreach($fields as $field) {

    if ($field['widget']['module'] == 'entityreference') {

      // Get the target bundle(s) for this field
    }
  }
}

Thanks in advance for any help on that!
Answer: Drupal 8 version:
$bundles = \Drupal::entityManager()
->getAllBundleInfo()['node'];
 foreach ($bundles as $id => $value) {
 $node_fields = \Drupal::entityManager()
  ->getFieldDefinitions('node', $id);
   foreach ($node_fields as $field) {
      if ($field instanceof \Drupal\field\Entity\FieldConfig) {
        if ($field->getType() == 'entity_reference') {
          $target_bundle = $field->getSettings()['handler_settings']['target_bundles'];
          kint($target_bundle);
        }
      }
    }
  }

With $target_bundle, You can do something like this as well to filter out a specific bundle:
if ($target_bundle != null && key($target_bundle) == 'mybundle'){
  kint($field->id());
}

Answer: Nevermind. I found a solution. I can get them with field_info_field():
foreach(node_type_get_types() as $value) {

  $fields = field_info_instances('node', $value->type);

  // Loop through fields
  foreach($fields as $field) {

    if ($field['widget']['module'] == 'entityreference') {

      // Get the target bundle(s) for this field
      $field_info_field = field_info_field($field['field_name']);
      $target_bundles = $field_info_field['settings']['handler_settings']['target_bundles'];
    }
  }
}

Comment: Hey, would you mind to mark your answer as the accepted answer? This would help to connect the information together.
Comment: @leymannx done!


How do I auto generate products via attribute?

Question: I have a shoe store, as you know, Drupal Commerce allow us to control each Attribute of Shoe like Color and Size.
This is my problem :
I have 2000 shoe styles, each style has sizes from 5-12 (6US to 12US)
It mean that everytime I add new style, I have to create 8 products for 8 sizes. Im using Clone Product Module, it's working very well, but how can I save my time :
It mean that when I create New Shoe Style (Node Products Display)
--> node/add/shoe-style
--> the system will auto generate 8 products (Size) for me, it depend on how many value in my attribute field (Size attribute has 8 value and system will create 8 new product), and now what we need to do is update something to each product.
Answer: Assuming you are using Drupal 7, you've got a couple of options:

Commerce Migrate will let you create products as part of a migration. It's a bit more time consuming but it's repeatable and is very powerful.
You can use Commerce Bulk Product Creation to create products based off of fields in the UI. This gives the ability to create those products once and isn't repeatable.
Comment: Commerce Bulk Product Creation is awesome, you save my day, thank you so much @nvahalik


Migration from gitlap to tuleap?

Question: Currently our company uses gitlab for repository management and issue management.
Is there a tool or an easy way to migrate (Issues, files, wiki, etc....) from gitlab to tuleap ?
Or any guides how to do it?
Answer: Dev team speaking.
There is a lot of work on going on import/export means for Tuleap but no automated stuff from gitlab right now. However, the import format is documented (would covers Issues and git ATM) so you can write your own:
https://docs.tuleap.org/administration-guide/projects-management/project-export-import.html#import-format

Migration from gitlap to tuleap?


What is the best way to visualize Likert-type dependent variables with 2x2 binary independent variables?

Question: In a between-subjects experiment, we have 2x2 conditions and several 7-point Likert dependent variables.
What would be the best way to visualize this data?
I am a statistics newbie, so please forgive my simple questions. I have, of course, searched the web and this forum. But most posts I found seem to be dealing with binary dependent variables. In contrast, we have, I think, binary independent variables.
Is it OK to treat the Likert answers as a continuous variable? I seem to recall some discussion (I forget where - maybe a textbook, maybe a paper) whether or not this is OK.
It does not seem to be a categorial variable to me. Maybe it's an ordered categorial variable?
But, more importantly, what would you suggest to plot the data?
Simple box-plots (with whiskers) don't seem to work really well, because in almost all dependent variables we had a few participants answering with the lowest, and some with the highest scores.
Also, it would be nice if one could show the answers for one dependent variable for all 4 conditions in one single plot in an intuitive way.
Comment: There are several threads related to Likert scales on this site, in particular the way they can be considered or analysed (as continuous or discrete ordered variables), depending on the question at hand. Howeber, you should defintely think about what you want to display: averaged values (in some meaningful way) across conditions or individual response distribution?
Comment: The thread just cited is more relevant than its title implies. Each pair of binary predictors defines a composite variable with 4 conditions and each 7-point scale is another variable, so many two-way displays (bar charts, mosaic plots, heat maps, etc.) are immediately available.
Comment: How many data points do you have in each of your four conditions? Can you post an example of your data (or random data of the size and structure you have)?
Comment: Thanks a lot for the comments!
Comment: Thanks a lot for the comments! We've got around 30 data points per condition. I was thinking of kind of a density or distribution plot per condition, unless you think other plots might provide more insights.  I would like to spread the two binary, independent variables along the x and y axis. So, maybe the tab plot will help. The heat map seems a bit too much aggregation: AFAIK, it can show only the mean (or median). And, yes, thanks lot, the link is helpful! it does give me some ideas.
Answer: As Nick Cox writes, it makes the most sense to facet your plot by the four conditions. Within each facet, you can do many things. Boxplots are one possibility, but they really compress your data too much, as you write.
Likert scales are indeed ordered categorical scales. You have only seven possible outcomes. This to me suggests a simple histogram or barplot with seven bars, so you would have a plot with four such histograms. Just make sure the vertical axes are of the same length so the plots are comparable. You could also add a vertical line to indicate some measure of central tendency - the mean if it makes sense (i.e., if you can meaningfull add your responses), or the median. Possibly add horizontal lines to indicate spreads, like the first and third quartile - or standard errors/confidence intervals for the central tendency.
Here is an example using simulated data in R. The vertical dashed lines indicate the means, the horizontal red lines span the first and third quartile.

R code:
<code>n_per_condition <- 100
set.seed(1)
dataset <- data.frame(condition_1=rep(c("A","B"),each=2*n_per_condition),
    condition_2=rep(c("X","Y"),times=2*n_per_condition),
    response=factor(sample(1:7,4*n_per_condition,replace=TRUE)))

plot_data <- function(condition_1,condition_2,ylim=c(0,30)) {
    index <- dataset$condition_1==condition_1 & dataset$condition_2==condition_2
    plot(c(0.5,7.5),ylim,type="n",xlab="",ylab="",xaxt="n",
        main=paste0("(",condition_1,",",condition_2,")"))
    axis(1,1:7)
    rect(xleft=-0.4+(1:7),
        ybottom=rep(0,7),
        xright=0.4+(1:7),
        ytop=as.numeric(table(dataset[index,"response"])),
        col="grey")
    abline(v=mean(as.numeric(as.character(dataset[index,"response"]))),lty=2,lwd=2)
    lines(quantile(as.numeric(as.character(dataset[index,"response"])),c(0.25,0.75)),
        rep(ylim[2],2),col="red",lwd=2)
}

opar <- par(mfrow=c(2,2),las=1,mai=c(.5,.5,.5,.1))
    plot_data("A","X")
    plot_data("B","X")
    plot_data("A","Y")
    plot_data("B","Y")
par(opar)
</code>


Residuals in a Machine Learning with forecasting

Question: I'm affraid to confuse something.
Here's my assumption :
I have a timeseries that I split in 2 parts Learning data and test data.

A residual in forecasting is the difference between an observed value
  and its forecast based on other observations: ei=yi−y^iei=yi−y^i.

I want to see how good my model is so I want t know the residuals.
But which data are my observations ? For me, it's my learning data + my test data. Because i had prediction on all my data. (On the learning data and the most important on my test data)
EDIT : So i'm calcutating the residuals on all my data.
EDIT 2 : Paper
Am I right ?
Thanks for your help.
Answer: I see two steps
1 -- MODEL SELECTION
Here we use a train and a test error to select a model.
In this contest, model refers to a the mathematical formulation of a solution (a function $\hat f (x, \theta)$) rather than a specific instance of a model where every parameter ($\theta$) has been set. Or, in other words, the performance of a method of producing a model, not of the model itself (see)
In this step. You will deal with test and train error. That is residuals on the test and train data. The analysis should take into account and discriminate between test and train.
You can read about the bias/variance tradeoff here.
I would also recommend

Hastie, Tibshirani, Friedman "Elements of Statistical Learning" 2001
This very good visual introduction: http://scott.fortmann-roe.com/docs/BiasVariance.html

In a nutshell (source):

2 -- MODEL FITTING
In this phase, after having selected a model, we fit it to the data. In case of a simple linear regression, we can find the $\theta$ that minimizes the cost function.
Here we make use of the whole dataset. We make use of the whole dataset because we have already analyzed how a model generalizes and the risk of overfitting in step 1. Now it is time to estimate $\theta$ and the more data we have, the better it is.
I believe that it is in this context that it is meaningful speaking of prediction and confidence intervals.
I mention here the two because they are often confused.

Although both are centered at $\hat y$, the prediction interval is
  wider than the confidence interval, for a given $x$ and confidence
  level. This makes sense, since the prediction interval must take
  account of the tendency of y to fluctuate from its mean value, while
  the confidence interval simply needs to account for the uncertainty in
  estimating the mean value (source)
Comment: I have an another question.
I'm trying to have a prediction interval. TO do this, i'm using this kind of formula for a 95% prediction interval :
y^t±1.96σ^
Where σ^ is an estimate of the standard deviation of the forecast distribution In forecasting.

It says after that i can use the standard deviation of the residuals.

What do you think about this ?

See EDIT 2 for the article.


Largest eigenvalue of a block diagonal matrix is an eigenvalue of the largest block?

Question: Consider this square matrix $C =  \begin{bmatrix}
A& 0   \\ 0  &B
\end{bmatrix}$, where $A$ and $B$ are also square matrices. Suppose $A$ is larger in the sense that is an $n \times n$ matrix, while $B$ is an $m \times m$ matrix, and $n > m$. Also suppose that both $A$ and $B$ are real symmetric matrices.
I know that the set of eigenvalues of $C$ is equal to the multi-set of eigenvalues of $A$ and $B$.
Question: Suppose the eigenvalues of $A$ are distinct from the eigenvalues of $B$. Must the largest eigenvalue of $C$ be an eigenvalue of $A$?
Answer: Not necessarily.  Take, for example,
$$
A = \pmatrix{1&0&0\\0&1&0\\0&0&1},\quad
B = \pmatrix{0&0\\0&2}
$$
Comment: Duh. I should have realized this. Thanks! +1 and accept.
Answer: Denote by $E(T)$ the set of eigenvalues of any matrix $T$. I don't know what you mean by multi-set, but the set of eigenvalues of $C$ is given by $E(C)=E(A)\cup E(B)$. The largest eigenvalue is just the largest element in this union. It can be either in $E(A)$ or $E(B)$. It's very easy to find examples where this eigenvalue is not an eigenvalue of $A$. Just take any $A,B$ you like, such that $\max(E(B))>\max(E(A))$.
Comment: "multi-set" means that you can repeat non-distinct elements, so that elements can have "multiplicities" within a multi-set.  So, for example, $\{1,1,2\} \neq \{1,2\}$ for multi-sets.


What's going on in the 'offsetof' macro?

Question: Visual C++ 2008 C runtime offers an operator 'offsetof', which is actually macro defined as this:
#define offsetof(s,m)   (size_t)&reinterpret_cast<const volatile char&>((((s *)0)->m))

This allows you to calculate the offset of the member variable m within the class s.
What I don't understand in this declaration is:

Why are we casting m to anything at all and then dereferencing it? Wouldn't this have worked just as well:
&(((s*)0)->m)
?
What's the reason for choosing char reference (char&) as the cast target?
Why use volatile? Is there a danger of the compiler optimizing the loading of m? If so, in what exact way could that happen?
Answer: An offset is in bytes. So to get a number expressed in bytes, you have to cast the addresses to char, because that is the same size as a byte (on this platform).
The use of volatile is perhaps a cautious step to ensure that no compiler optimisations (either that exist now or may be added in the future) will change the precise meaning of the cast.
Update:
If we look at the macro definition:
(size_t)&reinterpret_cast<const volatile char&>((((s *)0)->m))

With the cast-to-char removed it would be:
(size_t)&((((s *)0)->m))

In other words, get the address of member m in an object at address zero, which does look okay at first glance. So there must be some way that this would potentially cause a problem.
One thing that springs to mind is that the operator & may be overloaded on whatever type m happens to be. If so, this macro would be executing arbitrary code on an "artificial" object that is somewhere quite close to address zero. This would probably cause an access violation.
This kind of abuse may be outside the applicability of offsetof, which is supposed to only be used with POD types. Perhaps the idea is that it is better to return a junk value instead of crashing.
(Update 2: As Steve pointed out in the comments, there would be no similar problem with operator ->)
Comment: I don't get this. When you do s->m anywhere in your code, the assembly that is emitted adds the relevant offset to address of s and this offset is _always_ in bytes. So the '&(((s*)0)->m)' statement will _always_ give you the answer in bytes. Why the cast then? (Correct me if I'm wrong in my understanding)
Comment: The overloaded operator-> isn't a problem. To get the overloaded behaviour, you'd apply it to the instance - not the pointer.
Answer: offsetof is something to be very careful with in C++. It's a relic from C. These days we are supposed to use member pointers. That said, I believe that member pointers to data members are overdesigned and broken - I actually prefer offsetof.
Even so, offsetof is full of nasty surprises.
First, for your specific questions, I suspect the real issue is that they've adapted relative to the traditional C macro (which I thought was mandated in the C++ standard). They probably use reinterpret_cast for "it's C++!" reasons (so why the (size_t) cast?), and a char& rather than a char* to try to simplify the expression a little.
Casting to char looks redundant in this form, but probably isn't. (size_t) is not equivalent to reinterpret_cast, and if you try to cast pointers to other types into integers, you run into problems. I don't think the compiler even allows it, but to be honest, I'm suffering memory failure ATM.
The fact that char is a single byte type has some relevance in the traditional form, but that may only be why the cast is correct again. To be honest, I seem to remember casting to void*, then char*.
Incidentally, having gone to the trouble of using C++-specific stuff, they really should be using std::ptrdiff_t for the final cast.
Anyway, coming back to the nasty surprises...
VC++ and GCC probably won't use that macro. IIRC, they have a compiler intrinsic, depending on options.
The reason is to do what offsetof is intended to do, rather than what the macro does, which is reliable in C but not in C++. To understand this, consider what would happen if your struct uses multiple or virtual inheritance. In the macro, when you dereference a null pointer, you end up trying to access a virtual table pointer that isn't there at address zero, meaning that your app probably crashes.
For this reason, some compilers have an intrinsic that just uses the specified structs layout instead of trying to deduce a run-time type. But the C++ standard doesn't mandate or even suggest this - it's only there for C compatibility reasons. And you still have to be careful if you're working with class heirarchies, because as soon as you use multiple or virtual inheritance, you cannot assume that the layout of the derived class matches the layout of the base class - you have to ensure that the offset is valid for the exact run-time type, not just a particular base.
If you're working on a data structure library, maybe using single inheritance for nodes, but apps cannot see or use your nodes directly, offsetof works well. But strictly speaking, even then, there's a gotcha. If your data structure is in a template, the nodes may have fields with types from template parameters (the contained data type). If that isn't POD, technically your structs aren't POD either. And all the standard demands for offsetof is that it works for POD. In practice, it will work - your type hasn't gained a virtual table or anything just because it has a non-POD member - but you have no guarantees.
If you know the exact run-time type when you dereference using a field offset, you should be OK even with multiple and virtual inheritance, but ONLY if the compiler provides an intrinsic implementation of offsetof to derive that offset in the first place. My advice - don't do it.
Why use inheritance in a data structure library? Well, how about...
class node_base                       { ... };
class leaf_node   : public node_base  { ... };
class branch_node : public node_base  { ... };

The fields in the node_base are automatically shared (with identical layout) in both the leaf and branch, avoiding a common error in C with accidentally different node layouts.
BTW - offsetof is avoidable with this kind of stuff. Even if you are using offsetof for some jobs, node_base can still have virtual methods and therefore a virtual table, so long as it isn't needed to dereference member variables. Therefore, node_base can have pure virtual getters, setters and other methods. Normally, that's exactly what you should do. Using offsetof (or member pointers) is a complication, and should only be used as an optimisation if you know you need it. If your data structure is in a disk file, for instance, you definitely don't need it - a few virtual call overheads will be insignificant compared with the disk access overheads, so any optimisation efforts should go into minimising disk accesses.
Hmmm - went off on a bit of a tangent there. Whoops.
Answer: char is guarenteed to be the smallest number of bits the architectural can "bite" (aka byte).
All pointers are actually numbers, so cast adress 0 to that type because it's the beginning.
Take the address of member starting from 0 (resulting into 0 + location_of_m).
Cast that back to size_t.
Comment: It doesn't "cast to 0" (impossible because is `0` is not a type). It casts the value `0` to type `s`, to get a notional object of that type at memory address `0`. From that point on, your description makes sense.
Comment: My bad, English is not my primary language, so I make mistakes often :).
Answer: 1) I also do not know why it is done in this way.
2) The char type is special in two ways.
No other type has weaker alignment restrictions than the char type. This is important for reinterpret cast between pointers and between expression and reference.
It is also the only type (together with its unsigned variant) for which the specification defines behavior in case the char is used to access stored value of variables of different type. I do not know if this applies to this specific situation.
3) I think that the volatile modifier is used to ensure that no compiler optimization will result in attempt to read the memory.
Answer: 2 . What's the reason for choosing char reference (char&) as the cast target?
if type s has operator& overloaded then we can't get address using &s
so we reinterpret_cast the type s to  primitive type char because primitive type char
doesn't have operator& overloaded
now we can get address from that
if in C  then reinterpret_cast is not required
3 . Why use volatile? Is there a danger of the compiler optimizing the loading of m? If so, in what exact way could that happen?
here volatile is not relevant to compiler optimizing.
if type s have const or volatile or both qualifier(s) then
reinterpret_cast can't cast to char& because reinterpret_cast can't remove cv-qualifiers
so result is using <const volatile char&> for casting work from any combination


Uniform continuity on union of intervals

Question: Let $f$ be uniformly continuous on $I$ and uniformly continous on $J$ where $I$ and $J$ are intervals, such that $I\cap J=\emptyset$. I want to find a function such that $f$ is not uniformly continuous on $I\cup J$.
I tried the function $f(x)=0$ if $x<0$ and $f(x)=1$ if $x\ge0$. Taking the intervals $(-\infty, 0)$ and $(0,+\infty)$ it still doesn't seem to work. Any tips?
Answer: Your example is correct. The function $f$ is uniformly continuous in $I=(-\infty, 0)$ and in $J=(0,+\infty)$ because it is constant there. On the other hand $f$ is not uniformly continuous in  $I\cup J=(-\infty, 0)\cup(0,+\infty)$ because for $x_n=-1/n$ and $y_n=1/n$ we have that $y_n-x_n\to 0$ but $f(y_n)-f(x_n)=1\not \to 0$. Note that $f$ is continuous in $I\cup J$.
P.S. $\epsilon-\delta$ proof. $f$ is not uniformly continuous in $I\cup J$ iff there is  $\epsilon>0$ such that for all $\delta>0$ there are $x,y\in I\cup J$ such that  $|x-y|<\delta$ and $|f(x)-f(y)|\geq \epsilon$. Take $\epsilon=1$, $x=-\delta/3$ and $y=\delta/3$.
Comment: So is the union of two uniformly continuous disjoint intervals always continuous?
Comment: If the intervals $I$ and $J$ are open and disjoint then $f$ is continuous in $I\cup J$. Continuity is a local property. If you take $I=(-\infty, 0)$ and $J=[0,+\infty)$, then $f$  is uniformly continuous in $I$ and $J$ but it is not continuous in $I\cup J$.
Comment: Is there a way of showing that the union is not uniformly continuous with $\epsilon - \delta$ definitions? Because I cannot see it :P
Comment: $f$ is not uniformly continuous in $I\cup J$ iff there is  $\epsilon>0$ such that for all $\delta>0$ there are $x,y\in I\cup J$ such that  $|x-y|<\delta$ and $|f(x)-f(y)|\geq \epsilon$. Take $\epsilon=1$, $x=-\delta/3$ and $y=\delta/3$.


Can I calculate the integral of the O ring statistic from Ripley's K function?

Question: I need to calculate the integral of the O ring statistic, and I do not see any available package to use in Rstudio that can provide me with this value.
For example, here is the O ring statistic plot for a given landscape.

To calculate this I first generate a landscape, and then use spatialEco package to generate the graph. I also include the Ripley's K plot below.
library('spatstat')
library('ggplot2')
library('spatialEco')

set.seed(seed=24)

radiusCluster<-100
lambdaParent<-.02
lambdaDaughter<-30
hosts<-1000
randmod<-0
dim<-2000

numbparents<-rpois(1,lambdaParent*dim)

xxParent<-runif(numbparents,0+radiusCluster,dim-radiusCluster)
yyParent<-runif(numbparents,0+radiusCluster,dim-radiusCluster)

numbdaughter<-rpois(numbparents,(lambdaDaughter))
sumdaughter<-sum(numbdaughter)

thetaLandscape<-2*pi*runif(sumdaughter)

rho<-radiusCluster*sqrt(runif(sumdaughter))

xx0=rho*cos(thetaLandscape)
yy0=rho*sin(thetaLandscape)

xx<-rep(xxParent,numbdaughter)
yy<-rep(yyParent,numbdaughter)

xx<-xx+xx0

yy<-yy+yy0
cds<-data.frame(xx,yy)
is_outlier<-function(x){
  x > dim| x < 0
}
cds<-cds[!(is_outlier(cds$xx)|is_outlier(cds$yy)),]
while (nrow(cds)<hosts){
 dif<-hosts-nrow(cds)
  extraparentxx<-sample(xxParent,dif,replace = TRUE)
  extraparentyy<-sample(yyParent,dif,replace = TRUE)
  extrathetaLandscape<-2*pi*runif(dif)
  extrarho<-radiusCluster*sqrt(runif(dif))
  newextracoodsxx<-extrarho*cos(extrathetaLandscape)
  newextracoodsyy<-extrarho*sin(extrathetaLandscape)
  extraxx<-extraparentxx+newextracoodsxx
  extrayy<-extraparentyy+newextracoodsyy
  cdsextra<-data.frame(xx=extraxx,yy=extrayy)
  cds<-rbind(cds,cdsextra)
}

sampleselect<-sample(1:nrow(cds),hosts,replace=F)
cds<-cds%>%slice(sampleselect)

randfunction<-function(x){
  x<-runif(length(x),0,dim)
}
randselect<-sample(1:nrow(cds),floor(hosts*randmod),replace=F)
cds[randselect,]<-apply(cds[randselect,],1,randfunction)

landscape<-ppp(x=cds$xx,y=cds$yy,window=owin(xrange=c(0,dim),yrange=c(0,dim)))
ggplot(data.frame(landscape))+geom_point(aes(x=x,y=y))+coord_equal()+theme_minimal()

ostat<-o.ring(landscape,inhomogenous=FALSE)

plot1<-Kest(landscape)
plot(plot1)

I do not know of a method in which I can calculate the area under this curve (this was made using the "spatialEco" package).
However, I understand that the O ring statistic is $\frac{dK(r)}{dr}\frac{1}{2 \pi r}\lambda$ With r representing distance from a point, lambda being the intensity of the pattern and K representing the Ripley's K function.
My background is not in mathematics, so my mathematics is probably wrong, but the integral of the O ring statistic would be $K(r)\frac{ln(r)}{2\pi}\lambda r$
So for example, using a different package, spatstat, I can calculate the Ripley's K function for the same landscape.

I assume that the Ripley's K function part of the above equation is the difference between the edge corrected Ripley's K (such as the Isotropic line). In spatstat, it is possible to isolate the values given in the Ripley's K plot. So for example, if we use the value of Ripley's K iso at r =500, which is 968128.9, and the theoretical distribution which is 785398.2, then their difference is 182730.2.
So the integral of the O-ring statistic between 0 and 500 would be:
$182730.2 \frac{ln(500)}{2\pi}500\lambda$
I do not know how to calculate the intensity of the pattern, but is this a correct assumption? If not, what is the correct assumption and would you know how to calculate the integral of the O-ring statistic?
Many thanks for taking the time to read my question, and even more so if you have any suggestions. It is greatly appreciated.
Answer: The intensity of a point can be estimated by the number of points divided by the area of the window (survey region). In spatstat you estimate it by intensity(X) where X is the point pattern. It is not clear to me why you need the integral of the so-called O-ring statistic which is not standard in my field. We mention the statistic and its close connection to the pair correlation function in Chapter 7 of the spatstat book. You can download Chapter 7 as a free sample from the book website http://book.spatstat.org/ and I really think you could benefit from looking through this. Good luck!


Webform Module Email - Module Specific Configuration

Question: We've built a custom web form module to handle the capturing of contact information on our website. We recently came across the Webform module and are thrilled that it has so many different customization options that we're looking to transitioning over to using it across our entire site.
Before making this move we'd like to test it out using a single form on our site that captures some user data and sends it via an email submission. This data would not be stored within the Drupal database.
Our custom form module uses the Mail System and Swift Mailer modules to handle the sending of confirmation emails and we'd like to leverage the same modules when using the Webform module.
Within the Module Specific Configuration section of the Mail System module we've defined the following:
MODULE: Webform
KEY: ccpa_email_handler
FORMATTER: Webform PHP Mailer
SENDER: Webform PHP Mailer
I assume that the KEY value needs to be associated with the ID of the Email Handler that was defined for the Webform we've put built.
Unfortunately, when using the Test feature to see what the email looks doesn't seem to be working. When I submit the form I see the inline confirmation message but the email never makes it to my inbox.
I'm sure I've got something misconfigured here but I'm grasping at what it could be. I assume it has to be something with the Mail System and Swiftmailer setup so I'm hoping someone is also using this particular type of setup and could offer some insight as far as the setup and configuration.
Answer: You say that you use Swift Mailer for your current setup, but for your test setup with Webform, it seems like you use Webform PHP Mailer.
Try to use Swift Mailer instead and see if that works.
Comment: I left the FORMATTER as **Webform PHP mailer** and changed the SENDER to **Swift Mailer** which displayed the confirmation message correctly but still failed to deliver the email to my inbox.

The Swift Mailer transport settings are using PHP.

Our custom web form module uses PHP to send email so I assume I have to handle the sending of the email the same way. Do i need to create a plugin that captures the submit event for this form and then handles the sending of the email?
Comment: Try to look in the error logs to see if there are any errors.
Comment: Mails have to be set up on the webform look here for info on how to do it, https://www.ostraining.com/blog/drupal/html-emails-from-drupal-webform/.


Javascript: Factor out identical callbacks in Express app

Question: all!
Using Node/Express/Socket.IO, I wanted to refactor out identical callbacks.
I have:
io.sockets.on('connection', function(socket) {
    socket.on('find all notes' , function() { noteProvider.findAllNotes(function(err, result) {
        if(err) {
            socket.emit('err', err);
        } else {
            socket.emit('result', result);
        }
    }) });
    socket.on('find note by id', function(id) { noteProvider.findNoteById(id, function(err, result) {
        if(err) {
            socket.emit('err', err);
        } else {
            socket.emit('result', result);
        }
    }) });
}

but want rather something like:
io.sockets.on('connection', function(socket) {
    socket.on('find all notes' , function() { noteProvider.findAllNotes(callback) });
    socket.on('find note by id', function(id) { noteProvider.findNoteById(id, callback) });
}

How can I refactor out the callback? These 2 examples does not work:
Does not work 1:
var callback = function(err, result) {
    if(err) {
        socket.emit('err', err);
    } else {
        socket.emit('result', result);
    }
}

Does not work 2:
io.sockets.on('connection', function(socket) {
    socket.on('find all notes' , function() { noteProvider.findAllNotes(callback(socket, err, result)) });
    socket.on('find note by id', function(id) { noteProvider.findNoteById(id, callback(socket, err, result)) });
}

with
var callback = function(socket, err, result) {
    if(err) {
        socket.emit('err', err);
    } else {
        socket.emit('result', result);
    }
}

How can I keep my code DRY?
Frode
Answer: If you declare "does not work 1" where it has access to the socket variable it should work. That could be done like this:
io.sockets.on('connection', function(socket) {
  var callback = function(err, result) {
    if(err) {
      socket.emit('err', err);
    } else {
      socket.emit('result', result);
    }
  }

  socket.on('find all notes' , function() { noteProvider.findAllNotes(callback) });
  socket.on('find note by id', function(id) { noteProvider.findNoteById(id, callback) });
});

It seems you tried to solve that in "does not work 2", which would work if you return a function in the callback:
var callback = function(socket) {
  // Returning the callback which will be called when noteProvider is done.
  // Since it's in the closure of this function it will have a reference to socket.
  return function (err, result) {
    if(err) {
        socket.emit('err', err);
    } else {
        socket.emit('result', result);
    }
  }
}

io.sockets.on('connection', function(socket) {
  socket.on('find all notes' , function() { noteProvider.findAllNotes(callback(socket)) });
  socket.on('find note by id', function(id) { noteProvider.findNoteById(id, callback(socket)) });
}

Comment: For 1: Blush.., of course :) For 2: Aha, learned something new :) Takk, Andreas!
Comment: I'm always glad to hear when someone learned something new. Welcome to SO :)


R returns an error when I ask for "fiw %>% count(Country/Territory)"

Question: I'm trying to use "count()" to explore the variable "Count/Territory" from the data base "fiw". Why it returns an error?
I'm triying writing it like this:
fiw %>%
  count(Country/Territory)

Thank you.
Answer: The column name is unusual because of the presence of /.  Can backquote it and get it working
library(dplyr)
fiw %>%
   count(`Country/Territory`)

Comment: Now I see where was the mistake,

Thank you!
Answer: I guess the main cause of the error is the black slash /. In R to divide one column by another you can use /. For example if you need to divide the data in column A by the data in column B you can write A/B.


Delphi - Get current monitor of running process's focused window (external)

Question: So, as mentioned in the title, I'm after identifying which monitor (in multiscreen situation) is the external process (application) running on.
I know that my form can be identified by TForm.Monitor.MonitorNum property, and so far I can get the handle of the external process by using the FindWindowExtd function, which I got on the net.
Later on I check if GetForegroundWindow = FindWindowExtd('App Partial Title'), so I only do my procedures if the external application has focus.
Since I'm doing the mouse emulator, it is vital for me to also identify the screen ID, where that application has focus.
How could I do that? The web kinda left me dry on this one too today...
Here's the code for FindWindowExtd:
function FindWindowExtd(partialTitle: string): HWND;
var
  hWndTemp: hWnd;
  iLenText: Integer;
  cTitletemp: array [0..254] of Char;
  sTitleTemp: string;
begin
  hWndTemp := FindWindow(nil, nil);
  while hWndTemp <> 0 do begin
    iLenText := GetWindowText(hWndTemp, cTitletemp, 255);
    sTitleTemp := cTitletemp;
    sTitleTemp := UpperCase(copy( sTitleTemp, 1, iLenText));
    partialTitle := UpperCase(partialTitle);
    if pos( partialTitle, sTitleTemp ) <> 0 then
      Break;
    hWndTemp := GetWindow(hWndTemp, GW_HWNDNEXT);
  end;
  result := hWndTemp;
end;

Can I use some help from the handle I get out of it?
Thanks!
Comment: See MonitorFromWindow.
Comment: @SertacAkyuz Exactly what I was looking for!! Thanks for instant help!!!
Answer: Yep!
What @Sertac Akyuz said in the comment, was what I missed and most likely didn't know how to find.
The Screen.MonitorFromWindow(FindWindowExtd('AppPartialName')).MonitorNum is the answer!
Thanks for quick help!


Speeding up SQL query when sorting on foreign keys

Question: This is more of a generic SQL problem but I'm using Firebird 2.5 if anyone knows of a Firebird/Interbase specific optimization.  First, the following is a simplified example schema to illustrate the issue I'm trying to solve:
CREATE TABLE users
(
   id INTEGER PRIMARY KEY,
   name VARCHAR(16)
);

CREATE TABLE data_set
(
   id INTEGER PRIMARY KEY,
   name VARCHAR(64)
);

CREATE UNIQUE INDEX data_set_name_idx ON data_set(name);

CREATE TABLE data
(
   user_id INTEGER,
   data_set_id INTEGER,
   data BLOB,
   PRIMARY KEY(user_id, data_set_id)
);

CREATE INDEX data_user_id_idx ON data(user_id);
CREATE INDEX data_data_set_id_idx ON data(data_set_id);

The query I'm trying to run is as follows:
SELECT users.name, data_set.name, data FROM users, data_set, data
WHERE user_id=XXX AND user_id=users.id AND data_set_id=data_set.id
ORDER BY data_set.name;

With 'XXX' being filled in with the user_id I want.  So what I'm doing is selecting all the rows from the data table that are owned by a particular user and I'm sorting the results based on the data_set name.
This works as it is but the problem is the data table has over a billion rows in it and the data_set table is not small either.  The result set for a single user id may be many hundreds of millions of rows.  What happens is that in order for the ORDER BY to work the database has to create a massive amount of temporary data which is incredibly slow and uses a lot of disk space.  Without the ORDER BY it's fast but obviously not sorted like I need.
One solution would be to take the data_set.name values and just put them in a varchar column in data.  Then that could be indexed and would be quick to sort.  The problem with this approach is that it will have a lot of duplicate data and make the database absolutely massive.
Another solution would be something like an Indexed View or an indexed Computed Column.  As far as I know neither of those is supported by Firebird.
Any other ideas?
Comment: I don't have a definite answer, but I think you can play around more with your indexes and benchmark them with the query.
Comment: Why do you want to pull "hundreds of millions of rows" for a single user in the first place? It will take time not only to process the query but to transfer the data to the client.
Comment: @liggett78:  Oh, I agree it will take time to transfer it because it's a lot of data.  That's why I'm trying to speed this up as much as possible.  It doesn't need to be as slow as it is because it's wasting a lot of time creating temporary data for the sort.
Answer: This is rather speculative, but I wonder if this could be restructured as:

A cartesian product between users and data set, including the predicate on user.
Order by data set name
Joining to data

... would be more efficient, especially if you were only interested in the forst rows of the query.
In Oracle I'd think that it would not be because the nested loop join would be much less efficient than a hash join, but I'm not familiar with firebird at all I'm afraid.
Answer: Why not index data_set.name?
Also, I would skip primary key definition for facts table (data) and put two separate indexes for foreign keys, to speed up joins. (of course, indexing may have impact on insertions, if you have a large number of records to be inserted).
If you need to ensure unique constraint of the facts table, you could do it from the job transferring data into that table (I don't have details about that :).
Comment: Sorry, I forgot to include that.  Yes, data_set.name is indexed.  I'll update the example schema.  Indexing it does not fix the problem though.
Comment: You should then check the execution plan. It might be that you have to go to database specific optimization...
Comment: As far as the execution plan, the database is basically taking the entire unsorted result set and putting it in a temporary table so it can then index the ORDER BY column.  Essentially the same thing as if I added a column to *data* that contained the *data_set.name*, except just on the result set.  Due to the size of the result set this is very slow.

I'm wondering if there is some way to create an index in *data* on the foreign key reference except using the value it references (data_set.name) instead of the integer value.
Answer: Try to define an index on data_set(id, name) and experiment with it - maybe in combination with other suggestions here. You can change your existing UNIQUE index to a UNIQUE CONSTRAINT if your requirements dictate it and Firebird supports unique constraints.