Why fitting a Poisson GLM in an over dispersed dataset underestimate the standard error of the regression parameter?

Question: It is claimed by many authors that if we fit the GLM Poisson model to an over dispersed dataset of count data, the standard error of the estimated coefficients will be under-estimated.
Could you please explain me why doing like that leads to the under-estimated standard errors ? In other words, could you tell me the mechanism that makes the standard errors of the estimated coefficients under-estimated ? And they are "under-estimated" in comparison with which quantity ?
Thank you very much for your help!
Answer: As a result of the Poisson probability model, the variance and the mean of that distribution are the same. This fact entails that whenever this condition is not met, there will be a mismatch between the confidence intervals suggested by the model and what the data actually shows.
When characterizing count data, the term "overdispersed" comes when the data has a higher dispersion (or in other words, a higher variance) as compared to what would be expected under a Poisson model.
To see what this overdispersion looks like, check out this small R snippet:
library(tidyverse)

set.seed(4242)
df <-
tibble(`poisson (mean 8)` = rpois(600, 8),
       `neg_binomial (mean 8)`= rnbinom(600, mu=8, size=4),
       ) %>%
  gather(dist, val)

df %>%
  ggplot(aes(val)) +
  geom_histogram(fill='skyblue', color='black',
                 binwidth = 1 ) +
  facet_wrap(~dist, ncol=1) +
  theme_minimal()

df %>% group_by(dist) %>%
  summarise(mean = mean(val), variance = var(val))
#> # A tibble: 2 x 3
#>   dist                   mean variance
#>   <chr>                 <dbl>    <dbl>
#> 1 neg_binomial (mean 8)  7.98    23.6
#> 2 poisson (mean 8)       8.02     8.97

Created on 2021-09-20 by the reprex package (v2.0.1)
As you can see, compared to the negative binomial, the poison has much lower variance, even though both have the same mean. This capacity of the negative binomial, the capacity of adjusting to higher dispersion count data, makes it much preferable in situations of overdispersion. That is why negative binomial regression is a thing.

Now, in order to see the effect of overdispersion in the standard errors of the parameters, let's simulate a negative binomial regression with a binary variable is a "control" variable that shifts the mean by 3 units, starting from a baseline of 8.
set.seed(123)
df <-
tibble(x = rbinom(600, size=1, p=.5),
       y = rnbinom(600, mu=8 + if_else(x==1, 3, 0), size=4),
       )

df %>%
  mutate(variable = if_else(x==1, 'x = 1', 'x = 0')) %>%
  ggplot(aes(y)) +
  geom_histogram(fill='skyblue', color='black',
                 binwidth = 1 ) +
  facet_wrap(~variable, ncol=1) +
  theme_minimal()

And as we can see by the mean of each group, indeed when x = 1 the mean is greater (so is the variance as shown by the histograms, which is a characteristic of count data)
df %>%
  mutate(variable = if_else(x==1, 'x = 1', 'x = 0')) %>%
  group_by(variable) %>%
  summarise(mean = mean(y))
#> # A tibble: 2 x 2
#>   variable  mean
#>   <chr>    <dbl>
#> 1 x = 0     8.34
#> 2 x = 1    11.0

In order to show the underestimation of standard errors, lets model this negative binomial data with a Poisson glm, and afterward, with the correct model, a negative binomial
library(tidyverse)
set.seed(123)
df <-
tibble(x = rbinom(600, size=1, p=.5),
       y = rnbinom(600, mu=8 + if_else(x==1, 3, 0), size=4),
       )
glm(y~x, family = 'poisson', data = df) %>%
    summary()
#>
#> Call:
#> glm(formula = y ~ x, family = "poisson", data = df)
#>
#> Deviance Residuals:
#>     Min       1Q   Median       3Q      Max
#> -4.6970  -1.6597  -0.3155   0.8781   6.7168
#>
#> Coefficients:
#>             Estimate Std. Error z value Pr(>|z|)
#> (Intercept)  2.12091    0.01967  107.83   <2e-16 ***
#> x            0.27980    0.02645   10.58   <2e-16 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> (Dispersion parameter for poisson family taken to be 1)
#>
#>     Null deviance: 2157.7  on 599  degrees of freedom
#> Residual deviance: 2045.0  on 598  degrees of freedom
#> AIC: 4395.9
#>
#> Number of Fisher Scoring iterations: 5

MASS::glm.nb(y~x, data = df) %>% summary
#>
#> Call:
#> MASS::glm.nb(formula = y ~ x, data = df, init.theta = 3.89401969,
#>     link = log)
#>
#> Deviance Residuals:
#>     Min       1Q   Median       3Q      Max
#> -3.2348  -0.9097  -0.1631   0.4797   2.8902
#>
#> Coefficients:
#>             Estimate Std. Error z value Pr(>|z|)
#> (Intercept)  2.12091    0.03486  60.840  < 2e-16 ***
#> x            0.27980    0.04913   5.696 1.23e-08 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> (Dispersion parameter for Negative Binomial(3.894) family taken to be 1)
#>
#>     Null deviance: 657.73  on 599  degrees of freedom
#> Residual deviance: 625.25  on 598  degrees of freedom
#> AIC: 3686.3
#>
#> Number of Fisher Scoring iterations: 1
#>
#>
#>               Theta:  3.894
#>           Std. Err.:  0.319
#>
#>  2 x log-likelihood:  -3680.255

As you can see, the estimates of the parameters of both models agree. Where they don't agree is in the std error, since the Poisson model reports a lower estimate, and based on the simulation we know that it's wrong. And we know that a Poisson distribution can't handle higher values of variance (or dispersion, in the language of distributions belonging to the exponential family, used in glms).

Usually, this underestimation of standard errors comes from the notion of "if we have an actual overdispersion in our data generating process, then Poisson regression will underestimate it." One of the distributions that we know that has overdispersion is the Negative Binomial, and we shall find this overestimation.
First of all, the parametrization for the negative binomial usually used for negative binomial regression arises from the gamma-Poisson mixture interpretation, and you can learn more about it here. Under this parametrization, if $Y_i \sim NB(\mu_i, \phi)$ then:
$$
\text{E}(Y_i) = \mu_i \\
\text{Var}(Y_i) = \mu_i + \frac{\mu_i^2}{\phi}
$$
Comparing that if the Poisson case, if $Y_i \sim \text{Poisson}(\mu_i)$, we have:
$$
\text{E}(Y_i) = \mu_i \\
\text{Var}(Y_i) = \mu_i
$$
This should start giving you hints of why the estimates would be different.
Comment: I don't have much time right now to write the derivation of the standard errors for the estimates of the parameters right now. But I can later, and it will prove that for a small specific model.  But the results are connected to the difference between the variance of the two distributions.
Comment: Hi, thank you very much for your answer. however, i think that your answer does not answet the question "why the standard error of the coefficients when we fit GLM poisson to the overdispersed data are underestimated". Could you kindly add a section in your answer that answer that question please ?
Comment: Hi, thank you so much for the answer. However, it seems to me that your reasoning is based strictly on "empiric" which does not totally satisfy me. Could you kindly add some theoretical justification please ? It is nice to have your graphs but I think it will be much more persuasive if we talk about the theories. Thank you very much for your help!
Comment: +1, however I think the last sentence regarding AIC is incorrect. AIC here is computed by dropping irrelevant constants specific to the likelihood function, but these constants will differ if the likelihood function or link function used in the GLM differ. AIC as output by the GLM summary can only compare models of the same family fit on the same data.


Am I using Settings in .NET correctly?

Question: Here's what I'm doing.
I have three properties: MomsBackground, DadsBackground and ChosenBackground.
When Momsbackground is selected in the program, I set the ChosenBackground string according to what item the user has clicked (either "Mom" or "Dad").
Then on Form_Load() I use a switch case for the ChosenBackground string and according to that select This.BackgroundColor to MomsBackground or DadsBackground.
Code below: Am I using this as it was intended? Sorry, codes there now.
<code>using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;

namespace WindowsFormsApplication1
{
    public partial class Form1 : Form
    {
        public Form1()
        {
            InitializeComponent();
        }

        private void momToolStripMenuItem_Click(object sender, EventArgs e)
        {
            this.BackColor = Properties.Settings.Default.MomFormColor;
            Properties.Settings.Default.SelectedTheme = "Mom";
            Properties.Settings.Default.Save();
        }

        private void dadToolStripMenuItem_Click(object sender, EventArgs e)
        {
            this.BackColor = Properties.Settings.Default.DadFormColor;
            Properties.Settings.Default.SelectedTheme = "Dad";
            Properties.Settings.Default.Save();
        }

        private void Form1_Load(object sender, EventArgs e)
        {
            switch (Properties.Settings.Default.SelectedTheme)
            {
                case "Mom":
                    this.BackColor = Properties.Settings.Default.MomFormColor;
                    break;
                case "Dad":
                    this.BackColor = Properties.Settings.Default.DadFormColor;
                    break;
                default:
                    break;
            }
        }
    }
}
</code>
Comment: your code isn't showing up in the question.
Comment: Also, settings have nothing to do with C#. That's a .NET Framework feature. Settings work exactly the same way no matter which programming language you're using.
Comment: It directly related to C# because I'm going to be using it at runtime.
Comment: @Sergio: what @John meant was, this same question also applies with VB.Net or C++
Comment: @Sergio: right. Settings is not specific to the C# programming language. It's a feature that can be used by any .NET language.
Answer: sounds ok to me.  If in the future you have a lot more choices, you may want to store the choices and values in a db and make a db call to get the appropriate value.  Alternatively, you could store the key value pairs in the app.config and get that value from there.

Am I using Settings in .NET correctly?


Angular ngrx can I execute some code before the init action?

Question: I'm working with ngrx and I want to rehydrate it after refreshing the page but I also don't want to have my store public in the session storage, I noticed that when refreshing the page the order of events is  refresh =>ngrx init => NavigationStart => NavigationEnd, is there any way to execute code between the refresh action and the ngrx init?
Answer: The only option is to use session/local storage.
The state store is stored in memory, when the page is reset, the state is also lost.


Some calculations with polynomials in the proof of the Routh Hurwitz Test

Question: In the article with the proof of the Routh Hurwitz test I couldn't see why the following result is true:

Let $$p(s)=a_ns^n+a_{n-1}s^{n-1}+\dots+a_{1}s+a_0$$ and $$ \eta_{*} = \frac{a_n}{a_{n-1}}$$ $$g_{\eta}(s):=p(s)-\eta (a_{n-1}s^n+a_{n-3}s^{n-2}+\cdots)$$  Then one and only one zero of $g_{\eta}(s)$ tends to $\frac{-a_{n-1}}{a_n-\eta a_{n-1}}$ as $\eta \in \mathbb{R}$ approaches $\eta_{*}$ from the origin.

Notice that when $\eta=0$ then $g_{\eta}(s)=p(s)$ and when $\eta=\eta^*=\frac{ a_n}{a_{n-1}}$ then $$g(s)=p(s)-\frac{a_n}{a_{n-1}}(a_{n-1}s^n+a_{n-3}s^{n-2}+a_{n-5}s^{n-4}+\cdots)$$ a polynomial of degree $n-1$.
The name of the article is "Elementary proof of the Routh-Hurwitz test", and in the part quoted above, the author has already demonstrated that $p(s)$ and $g(s)$ have the same number of purely imaginary roots. I'd appreciate any help. Thanks in advance!
Comment: Hello, don't you mean $g_{\eta}(s):=p(s)-\eta (a_{n-1}s^{n-1}+a_{n-3}s^{n-2}+\cdots)$ instead (rather than $s^n$)?
Comment: It's with $s^n$ because $g(s)=a_{n-1}s^{n-1}+(a_{n-2}-a_{n-3}\dfrac{a_n}{a_{n-1}})s^{n-2}+a_{n-3}s^{n-3}+...$
which is a polynomial of degree $n-1$.
Comment: When $\eta=\frac{s a_n}{a_{n-1}}$, then $\eta (a_{n-1}s^n+a_{n-3}s^{n-2}+\cdots)$ is a $n+1$ degree polynomial. Hence $g_{\eta}(s)$ cannot be a polynomial of degree $n-1$.
Comment: I typed it wrong, there is no s in $\eta$, I fixed it now, it's $\eta=\eta*$.
Answer: The claim is close to the following: For $\varepsilon\approx 0$ and some polynomial $r(z)$ of degree $n-1$, the polynomial
$$
p(z)= εz^n+r(z)
$$
has exactly one root $\sim -\frac{r_0}{ε}$. The basic idea for a proof is that $p$ still has $n$ roots, $n-1$ accounted for as close to those of $r$, so the remaining one must be $\sim ε^{-1}$ for reasons of the Viete equations.
If $\zeta$ is any root of $r$, then it is an approximate root of $p$ with an exact root close-by if $ε$ is small enough, one can use the Rouché theorem.
For the last root rescale the equation using $w=εz$,
$$
ε^{n-1}p(z)=w^n+r_0w^{n-1}+εr_1w^{n-2}+...
$$
Now the later terms are the perturbation. The $n-1$ roots at zero of $0=w^n+r_0w^{n-1}$ correspond to the previously identified roots close to those of $r$. The only non-trivial root of the leading terms is $\omega=-r_0$, which again is close to a root of the full polynomial in $w$.
Thus there is exactly one root going to infinity like $-\frac{r_0}{ε}$.
The actual situation in question has $p(z)=εq(z)+r(z)$ with $\deg(p)=n$, so what counts as perturbation term gets a little more involved in the second step, but nothing much changes.


Theme user pictures in comments in drupal 7

Question: I'm trying to set a specific style for user pictures in comments in drupal 7, but can't find how to do it.
I know there is a module for this (http://drupal.org/project/imagecache_profiles) but it's really an overkill since I just want pictures to be themed in comments, according to a style I already created in Image_cache configuration.
In comment.tpl.php, there is a <?php print $picture; ?> , but it's directly the html for the image, so I can't apply the theme function of image cache, for instance theme('imagecache','size32x32', $data->users_picture);
The $author variable is also html.
Is there a way to get the theme function to work in this comment.tpl.php file ?
Is there another way to achieve this ? Thanks !
Comment: THat might be usefull
http://drupal.stackexchange.com/questions/8323/how-to-change-the-size-of-the-avatar-in-drupal-7
Answer: To help you understand the syntax, have a look at the comment.tpl.php documentation, here you will find the different variables and a link to template_preprocess_comment() where they are defined:
$variables['picture']   = theme_get_setting('toggle_comment_user_picture') ?
                          theme('user_picture', array('account' => $comment)) : '';

So there's a theme function to render this picture: theme_user_picture() which state you should have a look a the template file: user-picture.tpl.php which finally send you to template_preprocess_user_picture().
There you can override this function copying and pasting the original code and changing the bits and parts you want to suit your needs (put this code in your template.php file):
function YOURTHEME_preprocess_user_picture(&$variables) {
  //change only the relevant part and leave the rest of the code
  //For Drupal 7 imagecache is in core so I put the correct syntax
  $variables['user_picture'] = theme('image_style', array('style_name' => 'size32x32', 'path' => $filepath, 'alt' => $alt, 'title' => $alt));

  //...
}

Have a look at the $variables to find a way to target comments or add a class there and use CSS as dkiscool mentionned.
NB: if you don't like browsing all the API documentation, there's a faster way, use theme developer (see this answer).
Comment: Actually, I needed to add a few variables to get it to work:
`function mytheme_preprocess_user_picture(&$variables) {
  $account = $variables['account'];
  if (!empty($account->picture->uri)) {
        $filepath = $account->picture->uri;
  }
  if (isset($filepath)) {
      $alt = t("@user's picture", array('@user' => format_username($account)));
   $variables['user_picture'] = theme('image_style', array('style_name' => 'size32x32', 'path' => $filepath, 'alt' => $alt, 'title' => $alt));
  }
}`
Thanks again !
Comment: Thanks for this awesome answer. I don't follow how exactly you jump from file to file until this _preprocess_ser_picture, but I'm definitely going to try to understand it, to replicate it, and to try this "theme developer" tool. So thanks again for dateiling how you got this answer.
Answer: In your theme's template.php
function yourtheme_image($variables) {
        $attributes = $variables['attributes'];
        $attributes['src'] = file_create_url($variables['path']);

        foreach (array('width', 'height', 'alt', 'title') as $key) {

            if (isset($variables[$key])) {
                $attributes[$key] = $variables[$key];
            }
        }

        return '

';
}

this will place a "your_class" around all the images on your site then in your css file you can target it as
. comment  .your_class {border: solid 1px red;}

so that it targets only images on comments
Comment: Hi, Thanks for this interesting snippet, however I wanted to modify it not with css but directly at the output. But I'llo make use of your code somewhere else...


Criar cabeçalho (de linhas) da matriz no DataGridView

Question: Teria de alguma maneira no C# usar essa coluna apontada da imagem para adicionar valor nela como se fosse um "cabeçalho"?
Estou usando o DataGridView abaixo para mostrar uma matriz. O cabeçalho da matriz já está sendo mostrado nas colunas porque eu consigo usar:
gridView.Columns.Add("string", "header");

Comment: Eu teria que usar uma coluna dentro da row para ser meu cabeçalho então?
Comment: @Bart Quis dizer que você pode sim adicionar valores neste espaço.
Comment: @Bart Não precisa adicionar coluna não
Comment: @jbueno como assim?
Answer: Isso realmente é um cabeçalho da linha (row header) e é perfeitamente possível adicionar valores nele.
Só é preciso usar a propriedade HeaderCell da linha.
private void SetarCabecalhoLinha(DataGridView dgv)
{
    foreach (DataGridViewRow linha in dgv.Rows)
    {
        linha.HeaderCell.Value = (linha.Index + 1).ToString();
    }
}


Errors running builder 'AspectJ Builder' on project

Question: I am working on SpringSource Tool Suite with Spring Roo. Everything was working well up until  few days ago after I installed GWT plugin (I think this might be a problem, I am not sure ).
Every time I code something and press Save (ctrl+s), it shows a dialog box showing
<code>    Errors occurred during the build.
Errors running builder 'AspectJ Builder' on project xyz123
org/eclipse/ajdt/core/builder/AJBuilder$SourceFilesChangedVisitor

eclipse.buildId=220.127.116.11103161000-RELEASE
java.version=1.6.0_24
java.vendor=Sun Microsystems Inc.
BootLoader constants: OS=win32, ARCH=x86_64, WS=win32, NL=en_US
Framework arguments:  -product com.springsource.sts.ide
Command-line arguments:  -os win32 -ws win32 -arch x86_64 -product com.springsource.sts.ide

Error
Wed Apr 27 14:33:03 EDT 2011
Problems occurred when invoking code from plug-in: "org.eclipse.core.resources".

java.lang.ClassFormatError: org/eclipse/ajdt/core/builder/AJBuilder$SourceFilesChangedVisitor
at org.eclipse.ajdt.core.builder.AJBuilder.sourceFilesChanged(AJBuilder.java:1707)
at org.eclipse.ajdt.core.builder.AJBuilder.hasChangesAndMark(AJBuilder.java:1639)
at org.eclipse.ajdt.core.builder.AJBuilder.build(AJBuilder.java:185)
at org.eclipse.core.internal.events.BuildManager$2.run(BuildManager.java:629)
at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:172)
at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:203)
at org.eclipse.core.internal.events.BuildManager$1.run(BuildManager.java:255)
at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:258)
at org.eclipse.core.internal.events.BuildManager.basicBuildLoop(BuildManager.java:311)
at org.eclipse.core.internal.events.BuildManager.build(BuildManager.java:343)
at org.eclipse.core.internal.events.AutoBuildJob.doBuild(AutoBuildJob.java:144)
at org.eclipse.core.internal.events.AutoBuildJob.run(AutoBuildJob.java:242)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
</code>
Can anyone please help me out in solving this?
Thanks,
Comment: You can open Errors view (`Window > Show View > Other > General > Error Log`) then found there your error and choose `Copy` from context menu. If you paste the full error log into your question it will be more chances if somebody answers you.
Comment: I made the edit you suggested. Thanks.
Answer: The ClassFormatError occurs when a class file is malformed or corrupted.  See here:
http://download.oracle.com/javase/6/docs/api/java/lang/ClassFormatError.html
It could very well be that your installation of AJDT was somehow corrupted.  I would recommend trying one or all of the following:

Uninstall the Google plugin.  Does the problem go away?
Upgrade to the latest dev snapshot of AJDT.  Update site is here:  http://download.eclipse.org/tools/ajdt/36/dev/update  After upgrading does your error go away?
As a last resort, try reinstalling STS.  You can leave your workspace in tact, just unzip STS again.
Comment: Thanks a lot for your reply. This solved my problem. For benefit of community Step 2 solved it. I updated my AJDT . Step 1 didn't make any difference. Thanks again.


Oracle consulta dinâmica em colunas

Question: Estou com a seguinte situação, preciso criar um consulta dinâmica entre as colunas para montagem de um Script. Supondo que existe uma alteração nas versões onde é adicionado uma coluna extra na tabela, o Script devera reconhecer essas alterações sem ter manutenção no mesmo.
No Select deverá retornar coluna1, coluna2, coluna3, coluna 4
Tentei utilizar Listagg porém não atende, pois os valores se transformam em String.

    CREATE TABLE TESTE_STACK(
    COLUNA1 VARCHAR2(4000),
    COLUNA2 VARCHAR2(4000),
    COLUNA3 VARCHAR2(4000)
    );
ALTER TABLE TESTE_STACK add COLUNA4 VARCHAR2(4000);

SELECT Listagg(A.COLUMN_NAME, ',' ) Within GROUP(ORDER BY A.COLUMN_ID) Colunas
FROM ALL_TAB_COLUMNS A
WHERE A.OWNER = ''
AND   A.TABLE_NAME = 'TESTE_STACK';

SELECT (SELECT Listagg(A.COLUMN_NAME, ',' ) Within GROUP(ORDER BY A.COLUMN_ID) Colunas
FROM ALL_TAB_COLUMNS A
WHERE A.OWNER = ''
AND   A.TABLE_NAME = 'TESTE_STACK')
FROM TESTE_STACK A
;

Comment: "... A consulta devera retornar  'coluna1, coluna2, coluna3, coluna 4' ..." isso é uma string, o que você precisa que retorne?
Comment: Preciso que retorno em como se fosse colunas, o retorno com o Listagg já vem em String
Listagg =  'coluna1, coluna2, coluna3, coluna 4'

como se fosse colunas, ou alguma forma de concatenar com o alias
 TESTE_STACK.coluna1, TESTE_STACK.coluna2 ....
Comment: `Listagg(table_name||'.'||column_name,',')`
Comment: Isso concatena com alias, porem ainda fica em formato de String.
Preciso que retorno seja utilizado em outro Select , para definição das colunas
Comment: Por que não fazer um JOIN ?
Comment: Descreva seu problema com mais detalhes
Comment: Mas como seria?.
No cenário que estou preciso montar um Script para fazer updates nas colunas de forma de dinâmica, onde não precise dar manutenção ao realizar um add colunm na tabela
Comment: E se fizer o relacionamento, e a tabela onde terão novas colunas colocar algo como `ALIAS.*` dessa forma se tiver coluna nova ira trazer
Comment: Bom de certa forma consegui resolver a situação para o Script
Porem o retorno no Update não pega o valor da coluna
`UPDATE TESTE_STACK  SET A.COLUNA1 = A.COLUNA1;
`
`DECLARE
  Valor VARCHAR2(4000);
BEGIN
  FOR i IN (SELECT a.Column_Name FROM All_Tab_Columns a WHERE a.Owner = 'tst' AND a.Table_Name = 'TESTE_STACK')LOOP
    FOR x IN (SELECT 'A.' || i.Column_Name AS Coluna FROM Teste_Stack a)LOOP
      Valor := x.Coluna;
      Dbms_Output.Put_Line('UPDATE TESTE_STACK '||' SET '||x.Coluna||' = '||Valor||';'||Chr(10));
    END LOOP;
  END LOOP;
END;`
Comment: Coloque como resposta, e apague esse comentário
Answer: Resolvi o problema utilizando o Execute Immediet, desta forma é possível salvar o valor das colunas em uma variavel para realização do update.
Esse script é utilizando quando existe criação de colunas na tabela, de forma dinâmica ele irá reconhecer a nova coluna e seu valor
DECLARE
        VALOR VARCHAR2(4000);
        VSQL VARCHAR2(4000);
        VTABLE VARCHAR2(4000);
        BEGIN
            VTABLE := UPPER('&TABLE_NAME');
            FOR i IN (
                        SELECT TO_CHAR(a.Column_Name) COL
                        FROM All_Tab_Columns a
                        WHERE a.Owner = 'tst'
                        AND a.Table_Name = VTABLE
            )LOOP
                    VSSQL := 'SELECT '||I.COL||' FROM '||VTABLE||' ';
                    EXECUTE IMMEDIATE VSSQL INTO VALOR;
                    Dbms_Output.Put_Line('UPDATE '||VTABLE|| '
                                        ||' SET '||I.Col||' = '||Valor||';'
                                        ||Chr(10));

            END LOOP;
        END;

Oracle consulta dinâmica em colunas


How to put marchex call tracking script within conditions?

Question: I have to put the marchex call tracking script in the footer section of the site only if cookie is set,but the site is in square space and i can only use javascript/jquery . I have tried the following code but it seems not triggering, means the number changing is not working.

The actual script is :

Thank You
Answer: My solution is:

var vs_account_id = "xxxxxxxxxx";

function getCookie(cname) {
    var name = cname + "=";
    var decodedCookie = decodeURIComponent(document.cookie);
    var ca = decodedCookie.split(';');
    for(var i = 0; i <ca.length; i++) {
        var c = ca[i];
        while (c.charAt(0) == ' ') {
            c = c.substring(1);
        }
        if (c.indexOf(name) == 0) {
            return c.substring(name.length, c.length);
        }
    }
    return '';
}

window.addEventListener('load', function() {
    if(getCookie("cpc") != '' && getCookie('fb') != '') {
        var body = document.getElementsByTagName('body')[0];

        var s = document.createElement("script");
        s.type = "text/javascript";
        s.src = "https://rw1.marchex.io/euinc/number-changer.js";
        body.append(s);
    }
});

It is vanilla javascript without jQuery.


Show Progress Bar when Endless scroll is loading more data

Question: Okay so i have implemented Endless Scroll in my app using this famous android endless scroll library. I want to show the progress bar when the app gets to the bottom and loading more data and then hide it when we get data. I'm using Volley to get the data.
This is my onScroll Listener
scrollListener = new EndlessRecyclerViewScrollListener(manager) {
        @Override
        public void onLoadMore(int page, int totalItemsCount, RecyclerView view) {

            loadMore(page);
            ++page;
        }
    };

This is my loadMore function
public void loadMore(int page) {

    String url = url + page;
    StringRequest stringRequest = new StringRequest(url, new Response.Listener<String>() {
        @Override
        public void onResponse(String response) {
            ArrayList<Deal> deals = new JsonConverter<Deal>().toArrayList(response, Deal.class);
            adapter.addAll(deals);

        }
    }, new Response.ErrorListener() {
        @Override
        public void onErrorResponse(VolleyError error) {
            Log.d(TAG, error.toString());
            Toast.makeText(getApplicationContext(), "Bad Network Connection. Please Try Again", Toast.LENGTH_LONG).show();
            stopAnim();

        }
    }
    );

    int socketTimeout = 30000; // 30 seconds. You can change it
    RetryPolicy policy = new DefaultRetryPolicy(socketTimeout,
            DefaultRetryPolicy.DEFAULT_MAX_RETRIES,
            DefaultRetryPolicy.DEFAULT_BACKOFF_MULT);

    stringRequest.setRetryPolicy(policy);
    MySingleton.getInstance(getApplicationContext()).addToRequestQueue(stringRequest);
    mSwipeRefresh.setRefreshing(false);

}

So now what i want to do now is display the progress bar when more data is being loaded and hide it when its done loading.
I've tried this https://github.com/pwittchen/InfiniteScroll/blob/master/app/src/main/java/com/github/pwittchen/infinitescroll/app/MainActivity.java but didn't work.
I've also tried this https://gist.github.com/nesquena/a988aac278cff59a9a69
but wasn't getting it.
Comment: What have you tried till now to achieve what you want ?
Comment: i  tried this https://github.com/pwittchen/InfiniteScroll/blob/master/app/src/main/java/com/github/pwittchen/infinitescroll/app/MainActivity.java.
Comment: and this doing something from this https://gist.github.com/nesquena/a988aac278cff59a9a69.. but i wasn't getting the whole thing
Comment: You want to show the progress bar at the end of the endless list right ?
Comment: yes. i want to show it at the end
Comment: @MaxBillionaire I'm an author of the InfiniteScroll library. It's just an exemplary app. In real-life scenario, you need to implement the rest of the code on your own. It will be different in each use case. Library only allows to detect the moment, when user scrolled to the end of the list and get the first visible item position. It won't solve the whole problem, but it can help to create the final solution quicker.
Answer: If you are looking for code for progress dialog:
ProgressDialog progressDialog;
progressDialog = new ProgressDialog(TripletLogin.this);
            progressDialog.setTitle("Write Title here");
            progressDialog.setMessage("Loading...");

progressDialog.setProgressStyle(ProgressDialog.STYLE_SPINNER);
            progressDialog.setCancelable(false);
            progressDialog.setCanceledOnTouchOutside(false);
            progressDialog.setIndeterminate(true);
            progressDialog.show();

Comment: There is a lot of customisation that can be done to it. Let me know if it helps.
Comment: i know how to create progress dialog. but how do i show it in the recycler view when i get to the bottom of the page and more items are being loaded
Comment: Just write `progressDialog.show();` in function `loadMore(int page)` in the first line of function body.
Comment: Dont forget to `progressDialog.dismiss();` when you get in `onResponse` or inside `onErrorResponse`
Answer: Can't give you the exact source code right now, but here's how you should do it:

Add a circular ProgressBar in XML below the RecyclerView.
Set visibility to GONE
Everytime you load something, set visibility to VISIBLE until you're done loading.

Hope you can implement the rest.
Answer: Inside your onResponse() or onErrorResponse() put progressDialog.dismiss();


php headers doesn't downloads

Question: I was trying to make a "Force Download" via PHP, but all I get are strange symbols.
I tried to add the header ("application/typeofmyfile") and the header ("Content-Disposition")... and many others from many other post from here, but nothing
I've got this:
<code>$f_location = "path/myfile.nl2pkg";
$f_name = "myfile.nl2pkg";

header("Content-Type: application/octet-stream");
header("Content-Disposition: attachment; filename=".$f_name); //Here also tried basename($f_name) but nothing
header("Content-Length: ".filesize($f_location));
header("Content-Transfer-Encoding: binary");
</code>
I'm doing it like this because when an user clicks on the "download" button, with ajax I send him to the page to force the download, put some data in the db and then refresh the main page.
It's my first time working on headers, and for I what read, don't know if I'm doing something wrong.
Comment: Your code should work in a clean page.. try to run you ajax, do what you have to do, and then open a new empty page with that code and it should work.
Comment: Use header_sent to check weather or not your headers where already send http://de1.php.net/manual/en/function.headers-sent.php
Comment: Ok, for trying it, I added a new tab in the browser and put in all the variables in the header and download worked. Going to do what you ask me, and I will tell you. Gonna check that page also, thanks.
Comment: Exactly as @rennyrocha suggests. You can't download a file through an AJAX request. You have to make a normal request to the server. When you return the headers you used here the user will see a download-box and they will stay on the original page.
Comment: Hi, with Ajax I only pass to a page the variables to force the download (and id from a file and a user to add it to the db), and with thouse variables get the file. As rennyrocha said me, I've done the ajax and then open a new link with location.href and worked fine. Also will check what headers were sending before. Thanks for all.
Answer: Your Headers Code seems fine. The problem seems to be somewhere else.
To debug your Code use headers_sent to determine if headers already have been sent. You can also use headers_list to see what headers have already been sent.
<code>if (headers_sent()) {
  var_dump(headers_list());
  die('Headers have already been sent');
  exit;
}
</code>
If you use this right before you define the headers you will probably find the source of your problem.
Comment: Hi, I've done that before sending the headers, and the only one that shows me is the "Powered By PhP" header. As I said, first time working on them. Tried also to do a header_remove() but nothing, so I decided to make it as renny told me, make a page for the ajax to add to the db and another one for redirecting after ajax to the download. I would like to make it in the first page, but well, this works for the moment
Comment: if you see the `Powered By PhP` header, that means your headers have been sent. Try searching for `?> <?php` and similar problems. IT seems you do output something, even if that is just some whitespace. `header_remove` will not do the job if you also already did output some content.
Comment: Thanks, maybe a function I got that return true or false can cause the problems?
Comment: if that true or false is echoed.

php headers doesn't downloads


Surface Integral of $3z^2 d\sigma$

Question: Let $S$ be the bounded surface of the cylinder $x^2+y^2=1$ cut by the planes $z=0$ and $z=1+x$
Then how to show that the value of the surface integral $∬3z^2 d \sigma $ over $S$ is equal to $\int_{0}^{2\pi}(1+cos \theta)^3d\theta$
I tried using $\int\int_{S}g d\sigma =\int\int_{R}g \frac{|\nabla f|}{|\nabla f . p|}dA$ but I dont know what is what.
Answer: The body on which you want to carry on the surface integral can be parametrized as
$$\;r(t)=(\cos\theta,\,\sin\theta,\,t)\;,\;\;0\le \theta\le2\pi\;,\;\;0\le t\le 1+\cos\theta$$
and we thus get:
$$r_t=(0,0,1)\;,\;\;r_\theta=(-\sin\theta,\,\cos\theta,\,0)\implies$$
$$\implies r_t\times r_\theta=\begin{vmatrix}i&j&k\\0&0&1\\-\sin\theta&\cos\theta&0\end{vmatrix}=(-\cos\theta,\,-\sin\theta,\,0)\implies ||r_t\times r_\theta||=1$$
and our integral becomes
$$\int_0^1\int_0^{2\pi}\int_0^{1+\cos\theta} 3z^2dzd\theta dr=\int_0^{2\pi}(1+\cos\theta)^3d\theta =$$

Surface Integral of $3z^2 d\sigma$


Issue with an REST API- suggestions to my problem

Question: So I started a REST API, it is part of an exercise. Anyhow, I wanted to ask to the community what would you do if your request tested through Postman sent responses as null or an empty array. I use the mongo shell and it shows me there are two collections within my db. I am confused and have not really found an answer.
Ex: this is what the cmd responds

::1 - - [16/Nov/2022:11:04:14 +0000] "GET /movies HTTP/1.1" 201 2 or ::1 - - [16/Nov/2022:11:07:13 +0000] "GET
/movies/The%20Lord%20of%20Rings:%20The%20Fellowship%20of%20the%20Ring
HTTP/1.1" 200 4.

I have gone through the instructions and I do not see an error so to speak in the index file as well as the model.
Answer: It's probably an error in your GET function. No reason to have a different behaviour between MongoDB localhost & MongoDB in Atlas unless you are using completely different versions.
Maybe share the code and someone can help debug
https://community.postman.com/t/tests-for-empty-response-body/15584


How to correctly sample lines from large file when order and lines count are unknown?

Question: Input:

potentially large file with text lines
lines count is unknown, naive assumption gives range maybe more than 100K lines, but less than 400K
it's unknown if lines are sorted or not
it's fine to keep in memory at most 10.000 lines

Limitations:

read lines one by one, there is no possibility to read file at once into memory.
read file once and sample lines at the same time, it's way too long to read all lines first to get lines count.

Goal:
Sample the minimum count of lines from file to get maximum variance
Example:
Here is the file:
1,Tom,22,3000000
3,Bob,34,4000000
...
10,Tim,18,500000.05

Lines are ordered, I expect to sample
1,Tom,22,3000000
10,Tim,18,500000.05

Solution:
Does it make sense?
- read line by line
- keep tail buffer for the recent 1000 lines
- keep sample buffer for 9000 lines
- add every line with index XXX (what is the right formula here?) to sample buffer
- union sample buffer and tail buffer once end of file reached.
Answer: This sounds like the classic Reservoir Sampling use case. Reservoir sampling is a well studied family of algorithms addressing this problem. In reservoir sampling only the size of the resulting sample is kept in memory, not the entire input file.
The only difference in the problem statement from the classic algorithm is that it sounds like the intent is to preserve the input order. This can be done by saving the line number for each record and sorting prior to writing out the sample.
GNU Shuf is a popular and widely available implementation. It does not support retaining the original line order though. tsv-sample is an open-source program similar to GNU Shuf. It does support the preserving input order (disclaimer: I'm the author).
Comment: `that it sounds like the intent is to preserve the input order` - seems like poor explanation. It' a caveat, if lines are ordered, than I can't just take first records for sampling. Sample will be skewed since first lines will be similar due to sorting. I don't know in advance if lines are sorted or not.
Comment: @Capacytron - I wrote this because from the question it is not clear to me whether the output is expected to be in the original order. The example output shows the lines in the original order, but it's not clear if that is a requirement. The vanilla Reservoir Sampling algorithms I referenced do not preserve input order. Nor does it matter if the data is sorted or not, a random sample is output in all cases. GNU Shuf outputs the random sample in random order. tsv-sample has options to output in either random order or input order. Does this help?
Comment: Yes, thank you.


I.i.d. Discrete Random variables with mean 0 and variance 1

Question: Given a sequence of i.i.d. random variables $(X_i)_{i\geq 1}$ such that $\mathbb{E}(X)=0,\ \mathbb{E}(X^2)=1$ consider the sum $S_n=\sum_{k=1}^n{X_k}$. Is it true that independently of the distribution of the sequence $\mid S_n-S_{n-1}\mid=1$ for all $n\geq 2$?
Comment: The body of the Question should be as self-contained as possible, certainly  not relying on the title alone to convey crucial information.  The "i.i.d." assumption (independent, identically distributed) nature of the random variables is mentioned only in the title, which may mislead the casual Reader.
Answer: Here's a counter-example: $P(X=-\sqrt3)=1/4, P(X=1/\sqrt3)=3/4$


can we use CoreData to manage offline data when we use SqlCipher to store data in iOS

Question: can we use CoreData to manage offline data when we use SqlCipher to store data in iOS or we should directly use Sql queries to manage data?
Answer: It does not appear that there is any supported way to use SqlCipher and SQLite backed CoreData together.
It should be noted though, that current iOS devices have all of their flash storage encrypted by default.  This would provide security from offline attacks and adding a PIN to your iOS device would protect against live attacks.  Is there some specific reason you need to use SqlCipher that is not covered by whole-disk encryption?
Comment: The hardware encryption is great, but difficult to "trust", and sometimes it's not convenient to keep your device completely locked up, and you'd prefer to have, say passwords and secure credentials, in their own encrypted store, always protected from snoops and serious spooks.
Comment: Can you explain what really the flash encryption does? And can data be protected when the phone is jail broken and hackers try to fetch data ?
Comment: Please look at this http://www.medical-iphone-apps.com/blog/1300/demonstration-forensic-recovery-of-an-iphone-3gs/


OpenCL: Buffers apparently not being read in

Question: First time using OpenCL. I've put together a quick program to do vector addition from some various examples found around online. Here is the code.

     2 #define __CL_ENABLE_EXCEPTIONS
     3 #include <CL/cl2.hpp>
     4 #include <iostream>
     5 #include <vector>
     6
     7 int main() {
     8   std::cout << "Getting platforms" << std::endl;
     9   std::vector<cl::Platform> all_platforms;
    10   cl::Platform::get(&all_platforms);
    11   std::cout << all_platforms.size() << std::endl;
    12
    13   cl::Platform default_platform = all_platforms[0];
    14   std::vector<cl::Device> all_devices;
    15   default_platform.getDevices(CL_DEVICE_TYPE_ALL, &all_devices);
    16   std::cout << "Using platform: " <<
    17                default_platform.getInfo<CL_PLATFORM_NAME>() << "\n";
    18
    19   if(all_devices.size()==0){
    20       std::cout<<" No devices found. Check OpenCL installation!\n";
    21       exit(1);
    22   }
    23   cl::Device default_device=all_devices[0];
    24   std::cout<< "Using device: "<<default_device.getInfo<CL_DEVICE_NAME>()<<"\n";
    25   std::cout<< "\tType: "<<default_device.getInfo<CL_DEVICE_TYPE>()<<"\n";
    26   std::cout<< "\tMax Units: "<<default_device.getInfo<CL_DEVICE_MAX_COMPUTE_UNITS>()<<"\n";
    27   std::cout<< "\tMem Size: "<<default_device.getInfo<CL_DEVICE_GLOBAL_MEM_SIZE>()<<"\n";
    28
    29
    30   //Setting up context
    31   cl::Context context({default_device});
    32   cl::CommandQueue queue(context, default_device);
    33   cl::Program::Sources sources;
    34
    35
    36   // kernel calculates for each element C=A+B
    37   std::string kernel_code=
    38           "   void kernel simple_add(global const int* A, global const int* B, global int* C){       "
    39           "       int gid = get_global_id(0); "
    40           "       C[gid] = A[gid] + B[gid];                 "
    41           "       printf(\"%f %f %f \", A[gid], B[gid], C[gid]); "
    42           "   }                                                                               ";
    43   sources.push_back({kernel_code.c_str(), kernel_code.length()});
    44
    45   cl::Program program(context,sources);
    46   if(program.build({default_device}) != CL_SUCCESS){
    47       std::cout<<" Error building: "<<program.getBuildInfo<CL_PROGRAM_BUILD_LOG>(default_device)<<"\n";
    48       exit(1);
    49   }
    50
    51   //Create input vectors and assign values on the host:
    52   const size_t LENGTH = 10;
    53   std::vector<float> h_a(LENGTH), h_b(LENGTH), h_c(LENGTH);
    54   for (size_t i = 0; i < LENGTH; i++) {
    55     h_a[i] = rand() / (float)RAND_MAX;
    56     h_b[i] = rand() / (float)RAND_MAX;
    57     std::cout << h_a[i] << " " << h_b[i] << std::endl;
    58   }
    59
    60   cl::Buffer d_a(context, CL_MEM_READ_WRITE, sizeof(float)*LENGTH);
    61   cl::Buffer d_b(context, CL_MEM_READ_WRITE, sizeof(float)*LENGTH);
    62   cl::Buffer d_c(context, CL_MEM_READ_WRITE, sizeof(float)*LENGTH);
    63
    64   queue.enqueueWriteBuffer(d_a, CL_TRUE, 0, sizeof(float)*h_a.size(), &h_a[0]);
    65   queue.enqueueWriteBuffer(d_b, CL_TRUE, 0, sizeof(float)*h_b.size(), &h_b[0]);
    66
    67   cl::Kernel simple_add(program, "simple_add");
    68   simple_add.setArg(0, d_a);
    69   simple_add.setArg(1, d_b);
    70   simple_add.setArg(2, d_c);
    71   queue.enqueueNDRangeKernel(simple_add,cl::NullRange,cl::NDRange(LENGTH),cl::NullRange);
    72   queue.enqueueReadBuffer(d_c, CL_TRUE, 0, sizeof(float)*LENGTH, &h_c[0]);
    73
    74   for (float & ci : h_c) {
    75     std::cout << ci << std::endl;
    76   }
    77
    78   return 0;
    79 }

And the output is as follows:

    Getting platforms
    1
    Using platform: Intel(R) OpenCL HD Graphics
    Using device: Intel(R) Gen12LP HD Graphics NEO
        Type: 4
        Max Units: 96
        Mem Size: 26696065024
    0.840188 0.394383
    0.783099 0.79844
    0.911647 0.197551
    0.335223 0.76823
    0.277775 0.55397
    0.477397 0.628871
    0.364784 0.513401
    0.95223 0.916195
    0.635712 0.717297
    0.141603 0.606969
    0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 5.35055e+37
    9.89438e+37
    2.98535e+37
    3.99268e+37
    2.59261e+37
    4.96527e+37
    3.16024e+37
    1.47755e+38
    7.2566e+37
    1.43212e+37

As shown, while the kernel is running, it seems like the buffers A and B are not receiving the values from the vectors on the host.
Answer: You copy float numbers to the buffers, but the arguments for the kernel are defined as pointers to the buffers with integer numbers. So, the line 38 must be
void kernel simple_add(global const float* A, global const float* B, global float* C){


How to load Chosen jQuery plug-in

Question: Maybe I'm too ignorant, but I just can't find a way to load the Chosen plug-in into my html. I've seen previous questions and they load .js and .css files, but I don't find them in the new versions. How do I load it now?
Comment: It's called Chosen, here's the link: http://harvesthq.github.io/chosen/
Answer: Well, first you have to include the jQuery library, get it from here, then you'll have to add the chosen.js file, downloaded from here, and finally you have to turn your select HTML element into "chosen" element, your HTML code should look like this example:

This is the default usage of chosen.js, you can however specify many other option, but it is better to read the documentation.
Download the latest version of chosen.js from here.
Comment: I had already loaded jQuery and the script as you pointed, just couldn't find the chosen.js file, Thanks @bodi0

How to load Chosen jQuery plug-in


Making a GROUP_CONCAT query more efficient

Question: I have the following query. The idea is that it allows me to know what groups, and subsequently users, have access to each component_instance. I'm wondering if there is a better way to do this as the query is quite slow, but it's really handy to have these extra columns every time I deal with this table:
SELECT component_instances.*,
GROUP_CONCAT(DISTINCT IF(permissions.view, groups.id, NULL)) AS view_group_ids,
GROUP_CONCAT(DISTINCT IF(permissions.edit, groups.id, NULL)) AS edit_group_ids,
GROUP_CONCAT(DISTINCT IF(permissions.view, users.id, NULL)) AS view_user_ids,
GROUP_CONCAT(DISTINCT IF(permissions.edit, users.id, NULL)) AS edit_user_ids
FROM `component_instances`
LEFT OUTER JOIN permissions ON permissions.component_instance_id = component_instances.id
LEFT OUTER JOIN groups ON groups.id = permissions.group_id
LEFT OUTER JOIN groups_users ON groups_users.group_id = groups.id
LEFT OUTER JOIN users ON users.id = groups_users.user_id
GROUP BY component_instances.id
ORDER BY (case when component_instances.ancestry is null then 0 else 1 end), component_instances.ancestry, position

The permissions table is like so (excuse the Rails!):
create_table "permissions", :force => true do |t|
  t.integer "component_instance_id"
  t.integer "group_id"
  t.boolean "view",                  :default => false
  t.boolean "edit",                  :default => false
end

The types of permissions are edit, and view. A group can be assigned either or both. Permissions are also recursive in that if there are no group permissions on a component_instance, we'd have to check its ancestors to find the first where permissions are set (if any). This makes having the one query quite important because I can then combine this query with the selection logic that the ancestry gem provides (materialised path tree).
Update
I've since found this query benchmarks faster:
SELECT component_instances.*,
GROUP_CONCAT(DISTINCT view_groups.id) AS view_group_ids,
GROUP_CONCAT(DISTINCT edit_groups.id) AS edit_group_ids,
GROUP_CONCAT(DISTINCT view_users.id) AS view_user_ids,
GROUP_CONCAT(DISTINCT edit_users.id) AS edit_user_ids
FROM `component_instances`
LEFT OUTER JOIN permissions ON permissions.component_instance_id = component_instances.id
LEFT OUTER JOIN groups view_groups ON view_groups.id = permissions.group_id AND permissions.view = 1
LEFT OUTER JOIN groups edit_groups ON edit_groups.id = permissions.group_id AND permissions.edit = 1
LEFT OUTER JOIN groups_users view_groups_users ON view_groups_users.group_id = view_groups.id
LEFT OUTER JOIN groups_users edit_groups_users ON edit_groups_users.group_id = edit_groups.id
LEFT OUTER JOIN users view_users ON view_users.id = view_groups_users.user_id
LEFT OUTER JOIN users edit_users ON edit_users.id = edit_groups_users.user_id
GROUP BY component_instances.id
ORDER BY (case when component_instances.ancestry is null then 0 else 1 end), component_instances.ancestry, position

Here is an EXPLAIN for the query above and the table CREATE statements:
+----+-------------+---------------------+--------+-----------------------------------------------+--------------------------------------------+---------+--------------------------------------------+------+------------------------------------------------------+
| id | select_type | table               | type   | possible_keys                                 | key                                        | key_len | ref                                        | rows | Extra                                                |
+----+-------------+---------------------+--------+-----------------------------------------------+--------------------------------------------+---------+--------------------------------------------+------+------------------------------------------------------+
| 1  | SIMPLE      | component_instances | ALL    | PRIMARY,index_component_instances_on_ancestry | NULL                                       | NULL    | NULL                                       | 119  | "Using temporary; Using filesort"                    |
| 1  | SIMPLE      | permissions         | ALL    | NULL                                          | NULL                                       | NULL    | NULL                                       | 6    | "Using where; Using join buffer (Block Nested Loop)" |
| 1  | SIMPLE      | view_groups         | eq_ref | PRIMARY                                       | PRIMARY                                    | 4       | 05707d890df9347c.permissions.group_id      | 1    | "Using where; Using index"                           |
| 1  | SIMPLE      | edit_groups         | eq_ref | PRIMARY                                       | PRIMARY                                    | 4       | 05707d890df9347c.permissions.group_id      | 1    | "Using where; Using index"                           |
| 1  | SIMPLE      | view_groups_users   | ref    | index_groups_users_on_group_id_and_user_id    | index_groups_users_on_group_id_and_user_id | 5       | 05707d890df9347c.view_groups.id            | 1    | "Using index"                                        |
| 1  | SIMPLE      | edit_groups_users   | ref    | index_groups_users_on_group_id_and_user_id    | index_groups_users_on_group_id_and_user_id | 5       | 05707d890df9347c.edit_groups.id            | 1    | "Using index"                                        |
| 1  | SIMPLE      | view_users          | eq_ref | PRIMARY                                       | PRIMARY                                    | 4       | 05707d890df9347c.view_groups_users.user_id | 1    | "Using index"                                        |
| 1  | SIMPLE      | edit_users          | eq_ref | PRIMARY                                       | PRIMARY                                    | 4       | 05707d890df9347c.edit_groups_users.user_id | 1    | "Using index"                                        |
+----+-------------+---------------------+--------+-----------------------------------------------+--------------------------------------------+---------+--------------------------------------------+------+------------------------------------------------------+

CREATE TABLE `component_instances` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `visible` int(11) DEFAULT '1',
  `instance_id` int(11) DEFAULT NULL,
  `deleted_on` date DEFAULT NULL,
  `instance_type` varchar(255) DEFAULT NULL,
  `component_id` int(11) DEFAULT NULL,
  `deleted_root_item` int(11) DEFAULT NULL,
  `locked_until` datetime DEFAULT NULL,
  `theme_id` int(11) DEFAULT NULL,
  `position` int(11) DEFAULT NULL,
  `ancestry` varchar(255) DEFAULT NULL,
  `ancestry_depth` int(11) DEFAULT '0',
  `cached_name` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `index_component_instances_on_ancestry` (`ancestry`)
) ENGINE=InnoDB AUTO_INCREMENT=121 DEFAULT CHARSET=utf8

CREATE TABLE `groups` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(255) NOT NULL DEFAULT '',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=utf8

CREATE TABLE `groups_users` (
  `group_id` int(11) DEFAULT NULL,
  `user_id` int(11) DEFAULT NULL,
  KEY `index_groups_users_on_group_id_and_user_id` (`group_id`,`user_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8

CREATE TABLE `permissions` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `component_instance_id` int(11) DEFAULT NULL,
  `group_id` int(11) DEFAULT NULL,
  `view` tinyint(1) DEFAULT '0',
  `edit` tinyint(1) DEFAULT '0',
  PRIMARY KEY (`id`),
  KEY `edit_permissions_index` (`edit`,`group_id`,`component_instance_id`),
  KEY `view_permissions_index` (`view`,`group_id`,`component_instance_id`)
) ENGINE=InnoDB AUTO_INCREMENT=28 DEFAULT CHARSET=utf8

CREATE TABLE `users` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `real_name` varchar(255) DEFAULT NULL,
  `username` varchar(255) NOT NULL DEFAULT '',
  `email` varchar(255) NOT NULL DEFAULT '',
  `crypted_password` varchar(255) DEFAULT NULL,
  `administrator` int(11) NOT NULL DEFAULT '0',
  `password_salt` varchar(255) DEFAULT NULL,
  `remember_token_expires` datetime DEFAULT NULL,
  `persistence_token` varchar(255) DEFAULT NULL,
  `disabled` tinyint(1) DEFAULT NULL,
  `time_zone` varchar(255) DEFAULT NULL,
  `login_count` int(11) DEFAULT NULL,
  `failed_login_count` int(11) DEFAULT NULL,
  `last_request_at` datetime DEFAULT NULL,
  `current_login_at` datetime DEFAULT NULL,
  `last_login_at` datetime DEFAULT NULL,
  `current_login_ip` varchar(255) DEFAULT NULL,
  `last_login_ip` varchar(255) DEFAULT NULL,
  `perishable_token` varchar(255) NOT NULL DEFAULT '',
  PRIMARY KEY (`id`),
  UNIQUE KEY `index_users_on_username` (`username`),
  KEY `index_users_on_perishable_token` (`perishable_token`)
) ENGINE=InnoDB AUTO_INCREMENT=12 DEFAULT CHARSET=utf8

The ORDER BY comes from the ancestry gem but if there's a better way to do this I'd be happy to submit that as a pull request to them.
Comment: It's customary to keep all your text in the question, if I were you I would separate my each update using an `UPDATE` line and keep the stem all in the question part. It makes it much more clear to read.
Comment: Thanks Mehran, I've updated that. I initially went to answer my own question then thought to do a bounty.
Comment: Also I think if using the 2nd version you can omit the last two joins and use view_groups_users.user_id and edit_groups_users.user_id in the group_concat
Comment: Lol, you're right about the joins! I can definitely whip off the joining of the users table itself. I'll look into ancestry ordering thing according to your suggestions :)
Answer: It's almost impossible to optimize your query if we don't have your table structure and indices. Using a EXPLAIN statement is the necessary part of query optimizations.
Without the mentioned info, all I can comment on your question is that your ORDER BY part can benefit from some optimization for sure. Using any functions or statements in a condition will always result in a disaster. Also using a nullable field in an ORDER BY will also lead to problems. Perhaps the easiest way would be adding a new field to your table holding 0s and 1s instead of the current CASE statement.
Don't forget that having index on any field within a condition / order by / group by is always necessary if the number of records is considerable.
[UPDATE]
Your query is rather simple. The EXPLAIN's result shows that the only parts suitable as a candidate to be indexed are:
CREATE INDEX inx4 ON permissions (`component_instance_id`, `group_id`, `edit`, `view`);

The EXPLAIN's second line shows that there's no index of table permissions used in your query. That's because MySQL has a couple of rules when it will use indices:

Only one index of each table can be used in each one (sub-)query.
Any index could be used only if all its fields are mentioned in the query (as in conditions / order by / group by).

Considering your query, and the fact that all four fields of table permissions are mentioned, you'll need an index on all four of them or it's useless.
Yet the ORDER BY can benefit from the amendment I mentioned before.
Comment: Thanks Mehran, I've added in the extra details that you requested. I'm definitely interested in the ORDER BY statement, see the updated question. I've EXPLAINED the query in my answer above rather than the one in the question.
Answer: NULL is placed first (could use COALESCE to replace NULL with something else too instead of using an additional sort column). The second thing is reducing the joins, because the last two were on the id on which we concat.
SELECT
   component_instances.*,
   GROUP_CONCAT(DISTINCT view_groups.id) AS view_group_ids,
   GROUP_CONCAT(DISTINCT edit_groups.id) AS edit_group_ids,
   GROUP_CONCAT(DISTINCT view_groups_users.user_id) AS view_user_ids,
   GROUP_CONCAT(DISTINCT edit_groups_users.user_id) AS edit_user_ids
FROM
   `component_instances`
   LEFT OUTER JOIN permissions
      ON permissions.component_instance_id = component_instances.id
   LEFT OUTER JOIN groups view_groups
      ON view_groups.id = permissions.group_id AND permissions.view = 1
   LEFT OUTER JOIN groups edit_groups
      ON edit_groups.id = permissions.group_id AND permissions.edit = 1
   LEFT OUTER JOIN groups_users view_groups_users
      ON view_groups_users.group_id = view_groups.id
   LEFT OUTER JOIN groups_users edit_groups_users
      ON edit_groups_users.group_id = edit_groups.id
GROUP BY
   component_instances.id
ORDER BY
   component_instances.ancestry, -- MySQL was sorting the NULL values already correctly
   position
;

Comment: Thanks Maraca, sorry I accepted the other users answer first because I thought it was you! I've reversed that. You're right, NULL's are placed first. I suspect that bit of code it to support another database type perhaps as the ancestry library isn't just for MySQL. I can override that part though, so I'll do that.
Comment: Unfortunately your query above results in different results for view_user_ids and edit_user_ids vs my joins query. They both execute in about the same time, so unless you want to figure out why that may be, I'd be happy to accept the simpler answer without the sub-selects.
Comment: I think they must be. Looking at the results, it's as if the subselect is only grabbing the first group id and ignoring the rest. The extra joins definitely work as expected.
Comment: You're welcome. I just realised I had to assign the bounty separately. :)
Comment: @BrendonMuir many forget it, that's why if you accept an answer and don't assign the bounty the accepted answer will receive it anyway after the grace period. Otherwise the highest rated answer (there is a minimum score 2 or 3 I think) but only 2/3 of the bounty then, if I remember correctly. Thanks, first bounty :)
Comment: Nice to know :) and congratulations!


Why rank function is giving equal rank to all countries?

Question: I am asking a question that has been answered: Create a ranking variable with dplyr?. But for some strange reasons, the method does not work with my data. I am ranking the difference in unemployment between two period by countries.
I use this code as suggested:
df %>% mutate(rank = dense_rank(desc(difference)))

But i get 1 as the ranking for all countries. Could someone tell me what is going wrong?
Here is my data:
structure(list(cntry = structure(1:23, .Label = c("Austria",
"Belgium", "Switzerland", "Czech Republic", "Germany", "Denmark",
"Estonia", "Greece", "Spain", "Finland", "France", "Hungary",
"Ireland", "Iceland", "Italy", "Luxembourg", "Netherlands", "Norway",
"Poland", "Portugal", "Sweden", "Slovakia", "United Kingdom"), class = "factor"),
    difference = c(0.0321271618815491, -0.0251554839428438, 1.15072942999273,
    1.33128598731325, -2.26400160811014, 3.15779980836141, 6.80457896869579,
    6.70389987400804, 10.8919891165462, 0.547460084552159, 0.906834874234579,
    3.01112447330944, 8.5885631447415, 3.75206570820895, 1.58794503937105,
    0.334356006591187, 0.664766564981566, 0.0155501469693973,
    -0.984605793974606, 4.28470580541735, 1.11996749834057, 1.67278245779503,
    1.93783051552776)), row.names = c(NA, -23L), groups = structure(list(
    cntry = structure(1:23, .Label = c("Austria", "Belgium",
    "Switzerland", "Czech Republic", "Germany", "Denmark", "Estonia",
    "Greece", "Spain", "Finland", "France", "Hungary", "Ireland",
    "Iceland", "Italy", "Luxembourg", "Netherlands", "Norway",
    "Poland", "Portugal", "Sweden", "Slovakia", "United Kingdom"
    ), class = "factor"), .rows = structure(list(1L, 2L, 3L,
        4L, 5L, 6L, 7L, 8L, 9L, 10L, 11L, 12L, 13L, 14L, 15L,
        16L, 17L, 18L, 19L, 20L, 21L, 22L, 23L), ptype = integer(0), class = c("vctrs_list_of",
    "vctrs_vctr", "list"))), row.names = c(NA, 23L), class = c("tbl_df",
"tbl", "data.frame"), .drop = TRUE), class = c("grouped_df",
"tbl_df", "tbl", "data.frame"))

Answer: Based on the dput, it is a grouped dataset, i.e. grouped by 'cntry' and there is only observation for each 'cntry'.  We can ungroup and apply
library(dplyr)
df %>%
    ungroup %>%
    mutate(rank = dense_rank(desc(difference)))

Comment: beat me to it, akrun you are crazy fast sometimes.
Comment: He was so fast that stackoverflow did not let me accept his answer, only 9 minutes later


How to animate the droplets of rain in the picture?

Question: img to animate
Hey all, I was wondering if you can recommend me way to animate the droplets of rain?
Also, make it in a way without the eagle. Plus, to make droplets splash upon contact with the ground(which is the line)
I am a bit of beginner to javascript,  but finished 9 months course and reading eloquent javascript. I need some practice, so I came up with this picture (found it on youtube) as a means of practice.
Appreciate all the help.
Comment: Welcome to SO! Please, show what you have done so far and where you struggle at, providing a minimal reproducible example https://stackoverflow.com/help/minimal-reproducible-example would help you and us a lot
Answer: Considering that you are relatively new to JavaScript, this would be very hard to do. You can check out the HTML Canvas element and use JavaScript to animate using using classes and arrays for the raindrops, but without enough knowledge in JavaScript you would probably be better off using animation software
Comment: Thank you I checked out the canvas element. seems like a lot of read, but something useful.  I found some sites with ready-made animations,  but they don't seem to fit the one I intend to do. I guess I will read this through and watch a few videos on youtube before setting out to accomplish it.

I am not that new, its just that all I have is theory, I need practice, and I got stuck in code wars after just a few tasks


Which Apache module is required by Drupal system

Question: Could anyone advise which Apache module is required by Drupal system?
Comment: Found a answer here:
http://www.sitebuddy.com/apache/apache_required_modules_minimum_modules_to_run_a_drupal_site_php_mysql
Answer: The most common enabled module (or installed if not available) is mod_rewrite which helps your site have nice and clean urls. But even this isn't really required ... you can live without it but your site won't have nice links instead they will look like : www.example.com?q=node/12345. If PHP is working than that means that you have the necessary config to run Drupal.
Answer: I think you really only need (these are PHP extensions):
mbstring
curl
gd
mysql
mcrypt
jpeg
png
xml
...but not completely sure. my configuration above suits me fine. not sure if it can be of any help. If you using php as a module in apache, you can eliminate the fastcgi part.
Comment: The first items you have listed are PHP extensions, not Apache extensions, correct?
Comment: yes, that's right


Arg of $(1-i\sqrt{3})^6$. Did I do it right?

Question: Did I do it right?
$(1-i\sqrt{3})^6$
module = $\sqrt{1+3}=2$
$(1-i\sqrt{3})^6=2^6\cdot(\frac{1}{2}-\frac{\sqrt{3}}{2}i)^6$ $\implies$ Arg = $\frac{\pi}{3}$
Comment: How did you conclude that $\pi/3$ is the answer in the last step? What is the argument of $\frac{1}{2}-\frac{\sqrt{3}}{2}i$?
Answer: I'm not sure what you did, but it is wrong as $(1-\sqrt{3}\cdot i)^6=\boldsymbol{64}$ (Therefore the $\text{Arg}$ is $0$).
To solve this, let $z=(1-\sqrt{3}i)^6$. We'll turn $1-\sqrt{3}\cdot i$ into its polar form.
So, $|1-\sqrt{3}\cdot i|=\sqrt{1^2+(\sqrt{3})^2}=2$ like you've said. Also, $\text{Arg}(1-\sqrt{3}\cdot i)=-\dfrac{\pi}{3}$
Finally, $z=(2\text{cis}(-\dfrac{\pi}{3}))^6=2^6\text{cis}(-2\pi)=2^6\text{cis}(\boldsymbol{0})=2^6=64$
Comment: Yeah but I am looking for the argument of that complex number.
Comment: Please refer to the edited version (note: the argument of a real number is always $0$)
Comment: Are you saying that you do not know what the argument of 64 is?
Answer: The argument of $1- i\sqrt{3}$ is $\arctan(-\sqrt{3})= -\frac{\pi}{3}$  So the argment of $(1-i\sqrt{3})^6$ is  $6\times-\frac{\pi}{3}= -2\pi$ which is equivalent to 0.
(Edited to correct negative sign, thanks to Ripi2.)
Comment: Yoiu mean $arctan(-\sqrt 3)$, right?
Answer: $Re^{i\theta} =(1-i\sqrt{3})^6$
$\Rightarrow $$Re^{i\theta}=2^6(\frac{1} {2} - i\frac{\sqrt{3}}{2})^6$
$\Rightarrow $$Re^{i\theta}= 2^6 (e^{\frac{-i\pi}{3}}) ^{6}$
$\Rightarrow $$
Re^{i\theta}= 2^6e^{-i2\pi} $
$\Rightarrow$$ R=2^6$, $\theta=-2\pi+2k\pi$
$K=1$
So:
$R=64 $ and $ \theta=0$


Hazelcast + Hibernate query slowdown when using second level cache

Question: we are currently using in Hibernate 6.3.1 as a second level cache Hazelcast 5.3.6.
we are experiencing a significant slow down in performance when a second node joins the cluster. currently Hazelcast is configured in embedded mode (hazelcast.jcache.provider.type=member) and in hibernate config we have hibernate.cache.region.factory_class=jcache, hibernate.cache.use_second_level_cache=true and hibernate.cache.use_query_cache=true
Hazelcast is configured in this way:
    <advanced-network enabled="true">
        <join>
            <auto-detection enabled="false"/>
            <multicast enabled="true">
                <multicast-group>184.108.40.206</multicast-group>
                <multicast-port>54327</multicast-port>
                <multicast-time-to-live>32</multicast-time-to-live>
                <multicast-timeout-seconds>5</multicast-timeout-seconds>
            </multicast>
        </join>

        <member-server-socket-endpoint-config>
            <port>5701</port>
            <socket-options>
                <keep-alive>true</keep-alive>
                <tcp-no-delay>true</tcp-no-delay>
                <buffer-direct>true</buffer-direct>
            </socket-options>
        </member-server-socket-endpoint-config>
        <client-server-socket-endpoint-config>
            <port>9090</port>
            <socket-options>
                <keep-alive>true</keep-alive>
                <tcp-no-delay>true</tcp-no-delay>
                <buffer-direct>true</buffer-direct>
            </socket-options>
        </client-server-socket-endpoint-config>
    </advanced-network>

    <cache name="*">
        <statistics-enabled>false</statistics-enabled>
        <management-enabled>false</management-enabled>
        <in-memory-format>BINARY</in-memory-format>
        <expiry-policy-factory>
            <timed-expiry-policy-factory expiry-policy-type="ETERNAL"/>
        </expiry-policy-factory>
        <eviction eviction-policy="LRU" max-size-policy="ENTRY_COUNT" size="10000"/>
    </cache>

when only one server is started we see in the Hibernate statistics output for one particular query:
o.h.e.i.StatisticalLoggingSessionEventListener - Session Metrics {
    13365 nanoseconds spent acquiring 3 JDBC connections;
    9838 nanoseconds spent releasing 3 JDBC connections;
    80964 nanoseconds spent preparing 3 JDBC statements;
    1092652 nanoseconds spent executing 3 JDBC statements;
    0 nanoseconds spent executing 0 JDBC batches;
    311366 nanoseconds spent performing 2 L2C puts;
    6779427 nanoseconds spent performing 45 L2C hits;
    181972 nanoseconds spent performing 2 L2C misses;
    0 nanoseconds spent executing 0 flushes (flushing a total of 0 entities and 0 collections);
    0 nanoseconds spent executing 0 partial-flushes (flushing a total of 0 entities and 0 collections)
}

when second server joins the same query produces:
INFO  o.h.e.i.StatisticalLoggingSessionEventListener - Session Metrics {
    11922 nanoseconds spent acquiring 3 JDBC connections;
    7064 nanoseconds spent releasing 3 JDBC connections;
    69741 nanoseconds spent preparing 3 JDBC statements;
    1039659 nanoseconds spent executing 3 JDBC statements;
    0 nanoseconds spent executing 0 JDBC batches;
    1093001 nanoseconds spent performing 2 L2C puts;
    10632154 nanoseconds spent performing 45 L2C hits;
    126368 nanoseconds spent performing 2 L2C misses;
    0 nanoseconds spent executing 0 flushes (flushing a total of 0 entities and 0 collections);
    0 nanoseconds spent executing 0 partial-flushes (flushing a total of 0 entities and 0 collections)
}

L2C puts and hits taking 2 up to 3 times longer. As this is not the only query the time accumulates more and more and is then also noticeable in the UI.
Any suggestion how to speed up the L2C hits and puts?
Comment: You are seeing a jump from 0.000311366 seconds to 0.001093001 seconds, but the jump is insignificant.  With two members, half of the keys will be distributed to 2nd member, this adds the serialization and network overhead. Also in embedded mode HZ is running within the JVM of your application, using its resources. This means that anything happening in your app can affect HZ
Answer: As a distributed cache, once Hazelcast grows beyond a single node you will start to see network performance impact the time needed for gets and puts, and this will always result in performance less than what can be achieved on a purely local (single node) solution.
There are configuration options that can improve the performance, but at the cost of less consistency/accuracy.
When reading, a single-node cluster will always find the record locally (if it exists).  In a two-node cluster, we expect that 50% of gets will be satisfied locally, and 50% will require a network call.  As clusters get larger, an even larger percentage of gets will be remote.  So the performance degradation you see is a natural consequence of distributing data across the network.  With an embedded Hazelcast configuration, you can set the 'read-from-backups' flag true, and this will allow a higher percentage of calls to be resolved locally - but some of those reads may be stale values if the primary copy of the data has been modified and the modification has not yet been propagated to the backup.
When writing, on a single node configuration Hazelcast writes the value and returns to the caller.  On a multi-node configuration, Hazelcast will write first to the primary node, and then to the configured number of synchronous backup nodes, before returning to the caller - so writes will take longer as there will always be a backup copy to write (assuming default configuration).  You can improve performance by switching from synchronous to asynchronous backups, or by turning off backups altogether, but each of these changes increases the chance of data loss in the event of a node outage.  If all cached data is known to be safely stored in a system of record where it can be reloaded after a failure, then this may be an acceptable risk.


SQL Query to only select data with the maximum value of one of the column

Question: For the following data, for the same Entity and Alias, I want to select only those with maximum BatchID
Entity    Alias      Par      BatchID
35273     1186944    375000   2019022201464730000  -> I want this selected
35273     1186944    375000   2019022123151930000  -> I want this selected
35273     167        27203051 2019022201464730000
35273     167        20669929 2019022223351340000  -> I want this selected
35273     1949586    650000   2019022123151930000
35273     1949586    650000   2019022201464730000  -> I want this selected
35273     25255      18700    2019022123151930000  -> I want this selected
35273     25255      10700    2019022123151930000  -> I want this selected

This is what I want in the result:
Entity     Alias       Par        BatchID
35273      1186944     375000     2019022201464730000
35273      1186944     375000     2019022123151930000
35273      167         20669929   2019022223351340000
35273      1949586     650000     2019022201464730000
35273      25255       18700      2019022123151930000
35273      25255       10700      2019022123151930000

The following query returns all records (instead of only 6).
The result of the below query: "latest" returns 1 for all the records
select * from
(select DENSE_RANK()  over (partition by ENTITY,ALIAS order by batchid desc)
as latest,*
from test) x
where latest =1

How can I create the query to select only those with maximum BatchID ?
EDIT:
This is the table
CREATE TABLE [Test](
   [Entity] [varchar](50) NOT NULL,
   [Alias] [varchar](50) NOT NULL,
   [Par] [bigint] NOT NULL,
   [BatchID] [varchar](50) NOT NULL
) ON [PRIMARY]

Even this query returns all 1 for "latest"
select DENSE_RANK()  over (partition by ENTITY_ID,SECURITY_ALIAS order by
batchid desc) as latest,*
from test

Thank you.
Comment: Your query should work.
Comment: @AlexanderVolok no, he wants ties in the results if there are any.   I agree with Gordon, the posted query would work.   There must be something wrong with the poster's actual code that they aren't showing in their question.
Comment: Should not it be a `row_number` instead of `dense_rank`?
Comment: I'm not sure if the result are you expecting is correct, Do the first two rows should have the same batchID?
Answer: Try the below query.
select Entity, Alias, Par, MAX(BatchID) FROM TABLE_NAME
Comment: Hi, you are all correct. It looks like i had a hidden character in column Alias. I retyped the value, and now the query works. THank you so much all !
Answer: This is a bit of speculation but your table has these columns defined as strings:
[Entity] [varchar](50) NOT NULL,
[Alias] [varchar](50) NOT NULL,

I think you have hidden characters in the strings.
Here are two methods to find them:
select entity
from test
where try_convert(int, entity) is null and entity is not null;

or:
select alias
from test
where alias like '%[^0-9]%'

My guess is that you will find hidden characters and this will explain why the dense rank is always "1".  Your partitions are not combining what look like entity/alias pairs because unseen characters mean the pairs are different.
Comment: Yes, you are correct. I had a hidden character in the data. THank you
Answer: I'm not sure about the result you're expecting but you could try this:
Select Entity, Alias, max(BatchID)
FROM test T1
GROUP BY Entity, Alias
--HAVING COUNT(Entity) > 1 comment out this one if you want > 1 count rows


What is a physical interpretation of a skew symmetric bilinear form?

Question: Bilinear forms can give us a notion of distance, whether it is the typical Euclidean distance, or the spacetime interval between two events in Minkowski space.  But what about skew-symmetric bilinear forms?
Skew-symmetry means that every vector has $B(v,v)=0$.  Also, we can always find a basis such that picking any one element of that basis, say $v_i$, we get zero when applied all other basis elements, except exactly one other, $v_j$, where $B(v_i,v_j)=1$ and $B(v_j,v_i)=-1$.  Usually $B(v,w)=0$ means some kind of orthogonality or perpendicularity, like perpendicular directions in Euclidean space.   So each basis vector is "orthogonal" to all the others except one of them, where those combine together to give $+1$ or $-1$.  What's going on here?
Comment: My [very uninformed] 2 cents: I don't think that it is a good idea to use orthogonality as intuition, for precisely the first reason ($B(v,v)=0$). But I am also curious to see if there is some other interpretation.
Comment: @angryavian Orthogonality is related to skew-symmmetricity via the exp map: every orthogonal matrix is the exponential of an antisymmetric matrix.
Answer: The keywords are symplectic geometry. A symplectic manifold is a smooth manifold equipped with a non-degenerate closed $2-$form $\omega$. In each tangent space, $\omega$ is a skew-symmetric non-degenerate bilinear form.
Symplectic geometry has lots of applications in classical mechanics. I think this text by Dusa Mcduff might satisfy your curiosity.
Comment: Thanks for the link to that paper, it looks great!
Answer: Another example would be oriented intersection numbers of (say) curves on an oriented surface. Think of a torus. Let $v_1$ correspond to the "equator" and let $v_2$ correspond to a meridian (one of the circles that rotates about the axis to generate the torus). Depending on how we choose to orient everybody, you'll have $\#(v_1,v_2) = 1 = -\#(v_2,v_1)$. (The intersection number is positive if an oriented tangent vector to the first curve, followed by an oriented tangent vector to the second curve gives you the outward-pointing normal of the torus when you apply the right-hand rule.) On the other hand $\#(v_1,v_1)=0=\#(v_2,v_2)$, because you can easily "push" each one of these curves off itself.
Answer: I am not sure if this correct, hence why I am marking it community wiki.
But based on this Wikipedia article, it seems like every symplectic form induces notions of orientation and volume via its associated volume form. (Or one notion of signed volume.)
The skew-symmetry of the symplectic form certainly is reminiscent of the skew-symmetry of permutations and of alternating multilinear forms, differential forms among them.
The relationship of this to physics again comes from the geometric interpretation of phase space: basically, one way to understand Liouville's theorem -- volumes in phase space are preserved under canonical transformations. Any canonical transformation is a symplectomorphism, i.e. a map preserving symplectic structure, and thus since signed volume is derived from symplectic structure, that signed volumes (in phase space) should be preserved under maps preserving symplectic structure (canonical transformations) becomes somewhat unsurprising.
I wrote a similar answer here: https://math.stackexchange.com/a/1977701/327486


What does it mean when a function is set to zero?

Question: I am reading an article about the Reproduction numbers and I found the next phrase:

If $\mathcal{F}(x)$ is set to zero, then all eigenvalues of $Df (x)$ have
negative real parts.

and my question is what does it mean that a function is set to zero.
Answer: In various areas of mathematics, the zero set of a function is the set of all its zeros.
More precisely, if ${\displaystyle f:X\to \mathbb {R} } $ is a real-valued function (or, more generally, a function taking values in some additive group), its zero set is $ f^{-1}(0)$, the inverse image of $\{0\}$ in $X$.


jQuery - Image replacement transition [issues in Safari]

Question: Okay I have the following code:-
[SEE JSFIDDLE]
HTML:

CSS:
#header {
  width: 100%;
  background: #000000;
  height: 100px;
}

.mobile-menu {
    position: absolute;
    right: 25px;
    top: 20px;
    background: url(http://planetbounce.m360.co.uk/wp-content/themes/planetbounce/assets/img/buttons/menu-01.png);
    background-repeat: no-repeat;
    background-size: 26px !important;
    height: 26px;
    width: 26px;
    display: inline-block;
    margin: 7px 0;
    -webkit-transition-duration: 0.8s;
    -moz-transition-duration: 0.8s;
    -o-transition-duration: 0.8s;
    transition-duration: 0.8s;
}

.mobile-menu-hover {
    background: url(http://planetbounce.m360.co.uk/wp-content/themes/planetbounce/assets/img/mobile-menu-hover.png);
}

jQuery:
var imagesArray = ["http://planetbounce.m360.co.uk/wp-content/themes/planetbounce/assets/img/buttons/menu-01.png",
                   "http://planetbounce.m360.co.uk/wp-content/themes/planetbounce/assets/img/buttons/menu-02.png",
                   "http://planetbounce.m360.co.uk/wp-content/themes/planetbounce/assets/img/buttons/menu-03.png",
                   "http://planetbounce.m360.co.uk/wp-content/themes/planetbounce/assets/img/buttons/menu-04.png",
                   "http://planetbounce.m360.co.uk/wp-content/themes/planetbounce/assets/img/buttons/menu-05.png",
                   "http://planetbounce.m360.co.uk/wp-content/themes/planetbounce/assets/img/buttons/menu-06.png",
                   "http://planetbounce.m360.co.uk/wp-content/themes/planetbounce/assets/img/buttons/menu-07.png"];

function preloadImg(pictureUrls, callback) {
  var i, j, loaded = 0;
  var imagesArray = [];

  for (i = 0, j = pictureUrls.length; i < j; i++) {
    imagesArray.push(new Image());
  }
  for (i = 0, j = pictureUrls.length; i < j; i++) {
    (function (img, src) {
      img.onload = function () {
        if (++loaded == pictureUrls.length && callback) {
          callback(imagesArray);
        }
      };
      img.src = src;
    }(imagesArray[i], pictureUrls[i]));
  }
};

function changeImage(background, imagesArray, index, reverse) {
  background.css("background-image", "url('" + imagesArray[index].src + "')").fadeIn(10, function() {
    if (reverse) {
      index--;
      if (index == -1) {
        return; // stop the interval
      }
    } else {
      index++;
      if (index == imagesArray.length) {
        return; // stop the interval
      }
    }
    //Fade in the top element
    background.fadeOut(10, function () {
      //Set the background of the top element to the new background
      background.css("background-image", "url('" + imagesArray[index] + "')");
      changeImage(background, imagesArray, index, reverse);
    });
  });
}

jQuery(function () {
  /* Preload Image */
  preloadImg(imagesArray, function (imagesArray) {
    jQuery(".mobile-menu").css("background-image", "url('" + imagesArray[0].src + "')")
    jQuery('.mobile-menu').on('click', {imgs: imagesArray}, function (event) {
      var background = jQuery(".mobile-menu");
      var bi = background.css('background-image');
      var index = 0;
      var reverse = false;
      if (imagesArray[0].src != bi.replace('url("', '').replace('")', '')) {
        index = imagesArray.length - 1;
        reverse = true;
      }
      changeImage(background, event.data.imgs, index, reverse);
    });
  });
});

The Issue:
This works fine in Firefox and Chrome, it transitions between the 7 different images on click, then does the reverse on the second click (toggling).
The problem is when I try this in Safari, it basically goes through the image replacement process then reverts back to the first image for some reason and I can't figure out why?
Any ideas?
Comment: It's really strange, it starts ever from the last image and not from the first image. I think it's an issue with `on('click', {imgs: imagesArray}....` but I'm not sure
Answer: It seems to be because Safari returns the background-image without double quotes, but your replace function checks for url(", so it doesn't replace and reverse never gets true.
Instead of replacing you can check using either indexOf or matching using a RegExp, it would be safer and more straightforward. Either:
if ( bi.indexOf(imagesArray[0].src) == -1) {

or
 if (imagesArray[0].src != bi.match(/http.+png/)[0]) {

With indexOf: http://jsfiddle.net/u9ske14r/


WPF howto click out of rows in DataGrid to unselect item

Question: How can I remove selection of Row if I click outside of rows. Like in the red area of the image showing below

The related xaml shows below:
<Grid>
        <Grid.RowDefinitions>
            <RowDefinition Height="*" />
            <RowDefinition Height="auto" />
        </Grid.RowDefinitions>

        <DataGrid x:Name="ClientsList" Grid.Row="0" Grid.Column="0" CanUserAddRows="False"
                  SelectionMode="Extended" SelectionUnit="FullRow"
                  ItemsSource="{Binding ClientItems}" LostFocus="ClientsList_LostFocus" Background="Green">
        </DataGrid>

        <StackPanel Margin="2" Grid.Row="1" Grid.Column="0" Orientation="Horizontal">
            <Button x:Name="AddButton"  Margin="2,0,0,0" Content="+" Height="25" Width="40" Click="AddButton_Click"/>
            <Button x:Name="DeleteButton" Margin="2,0,0,0" Content="-" Height="25" Width="40" Click="DeleteButton_Click"/>
        </StackPanel>
    </Grid>

Answer: In you case if empty area was clicked then OriginalSource is gonna be ScrollViewer (if no additional styling applied). Try to add the following MouseUp event handler to DataGrid:
    private void OnDataGridMouseUp(object o, MouseButtonEventArgs e)
    {
        if (e.OriginalSource is ScrollViewer)
        {
            ((DataGrid) o).UnselectAll();
        }
    }

Answer: Use event LostFocus to identify when you lose focus of the DataGrid, and cancel selection of selected rows in that event.
For cancel selection:
yourDataGrid.UnselectAll()

Comment: It is not working. The red area is one part of DataGrid. I click here, it don't fire `LostFocus` event
Comment: What is this area? Kindly show me the related `xaml` code
Comment: So this is the empty grid area (in red square)?
Comment: Yes. I updated the question. I add the green background for DataGrid, This image shows more clearly


Como bloquear o acesso à um hostname

Question: Eu preciso bloquear o acesso ao site www.facebook.com de uma máquina Linux. Eu devo bloquear isso na própria máquina, isto é, eu não tenho a possibilidade de fazer isso com um Proxy.
Inicialmente eu pensei em usar o iptables, porém, como o Facebook possui IP dinâmico, e o iptables resolve o hostname somente no momento da adição da regra, o site pode ser acessado em alguns momentos (quando o IP de acesso mudar).
Existe um jeito simples para fazer isso?
Comment: Inclua o seguinte no arquivo `/etc/hosts`: `facebook.com 127.0.0.1`
Answer: Primeiro, você precisa forçar todas as requisições DNS (porta 53 UDP e TCP) a passarem pelo seu próprio resolver (por exemplo, um servidor de DNS que faça consultas recursivas), pois nada impede do usuário configurar um DNS diferente na própria máquina. Direcionando todas as requisições para o seu servidor, você terá uma facilidade maior de fazer o bloqueio. Para isto, o iptables resolve bem (redirecionando tudo que for porta 53, independente de nome ou IP).
Ex: o usuário coloca o servidor 188.8.131.52 (OpenDNS) como DNS na máquina-cliente, porém suas rotas redirecionam qualquer acesso à porta 53 para o seu próprio servidor, não permitindo que a requisição saia da rede interna. Assim que qualquer usuário digitar www.facebook.com, o seu servidor é que vai ser consultado (lembrando que tem o m.facebook.com, e mais uma série de coisas que pode ser filtrada).
Após isso,precisa configurar o seu próprio servidor DNS, como no exemplo dado pelo bfavaretto, ou customizando a configuração do DNS instalado no servidor de acordo com a documentação.

Mesmo assim, alguém poderia rodar um servidor local de DNS, que fizesse uma requisição externa em outra porta que não a 53, escapando do bloqueio (raro acontecer, mas é bom que você saiba do risco). Ou simplesmente por os IPs desejados no arquivo hosts da própria máquina.

Se for pra uso em empresas, muitas vezes a solução é colocar no contrato do trabalho dos funcionários a proibição, e uma cláusula dando ciência de que há monitoração remota das telas, e o primeiro que "escorregar", uma advertência por escrito. Assim você vai evitar um monte de problemas "infiltráveis" de maneira mais eficaz, incluindo uso pessoal indevido das máquinas, e evitando uma alegação de invasão de privacidade por parte do funcionário (desde que ele tenha ciência plena da monitoração).

Como bloquear o acesso à um hostname


What is the probability of getting FOUR OF A KIND in a $13$-card poker game?

Question: What is the probability of getting FOUR OF A KIND in a $13$-card poker game?
Here is my attempt:
The setup for the required poker hand would either be:
$$AAAABCDEFGHIJ,$$
$$AAAABBBBCDEFG,$$
or
$$AAAABBBBCCCCD,$$
where $A, B, C, D, E, F, G, H, I, J$ are distinct faces.
The total number of poker hands of type $AAAABCDEFGHIJ$ is
$${13 \choose 1}{4 \choose 4} = 13.$$
The total number of poker hands of type $AAAABBBBCDEFG$ is
$${13 \choose 2}{4 \choose 4}\cdot{_{2} P_{1}} = 156.$$
The total number of poker hands of type $AAAABBBBCCCCD$ is
$${13 \choose 3}{4 \choose 4}\cdot{_{3} P_{1}} = 858.$$
The total possible number of $13$-card poker hands from the standard deck of $52$ playing cards is
$${52 \choose 13} = 635013559600.$$
Therefore, the required probability is
$$\dfrac{1027}{635013559600} \approx 0.000000001617288299555\ldots.$$
My questions are:

(1) Is this probability computation correct?
(2) If my computation is not correct, where is/are the error(s) and what hint can you give towards rectifying that error(s)?

I am a bit unsure about my computation of the total number of poker hands of types $AAAABBBBCDEFG$ and $AAAABBBBCCCCD$, as logically these should be rarer than the poker hands of type $AAAABCDEFGHIJ$.
Comment: The number of $4$ of a kind hands is very very much undercounted in the post.
Comment: You would also need to consider hands of the form AAAABBCDEFGHI, AAAABBBCDEFGH, and the like. In other words, just because you don't have 4 aces doesn't rule out the possibility of having 1, 2, or 3 of them, or the possibility of having 4 pairs, 4 of a kind, and one additional card that doesn't match any of these.
Comment: So hands that have at least one $4$ of a kind are to be counted?  Can some of the remaining cards outside the $4$(s) of a kind form a $3$ of a kind? Or a straight? Or a royal flush?
Comment: @AndréNicolas, yes - poker hands having at least one $4$ of a kind are to be counted.  There are no restrictions to the remaining cards.
Answer: We use Inclusion/Exclusion.
Let us first count the number of hands that have $4$ Queens. The rest of the cards can be chosen in $\binom{48}{9}$ ways.
Similarly, there are $\binom{48}{9}$ hands that have $4$ Aces, and so on.
Add up. We get $\binom{13}{1}\binom{48}{9}$.
However, this double-counts, for example, the hands that have $4$ Queens and $4$ Aces. There are $\binom{44}{5}$ such hands. The $2$ kinds we have $4$ each of can be chosen in $\binom{13}{2}$ ways, so our next estimate is $\binom{13}{1}\binom{48}{9} -\binom{13}{2}\binom{44}{5}$.
However, we have taken away too much, for we have removed one too many times the $\binom{13}{3}\binom{40}{1}$ hands that have $3$ kinds  of $4$ of a kind. So our final total is
$$\binom{13}{1}\binom{48}{9} -\binom{13}{2}\binom{44}{5}+\binom{13}{3}\binom{40}{1}.$$
Finally, for the probability, divide as you did by $\binom{52}{13}$.


Redirect to the front page with a different language

Question: i have a question about redirecting to a page with a different language. I found this old post for drupal 7 but I doesn't really help me - https://www.drupal.org/node/1280468
Now we are in Drupal 8 and the redirect works like this:
<code>return new RedirectResponse(\Drupal::url('<front>'))
</code>
But how can I set the language for the redirect? In my code I have the languagecode like NZ for New Zealand.
Answer: You can try this code
<code> <?php

 use Drupal\Core\Url;
 use Symfony\Component\HttpFoundation\RedirectResponse;

 $language = Drupal::languageManager()->getLanguage('vi');
 $url = Url::fromRoute('<front>', [], ['language' => $language]);
 $response = new RedirectResponse($url->toString());
 $response->send();
</code>
Comment: It works but you can't pass 'nz', because it's not a language-code. It would be en-au for example
Answer: Within your controller method:
<code>$url = \Drupal\Core\Url::fromRoute('<front>')->toString();
return new RedirectResponse($url);
</code>
Comment: doesn't work, return current language


CUDA multi - gpu add function

Question: I followed the following example:
multi-GPU basic usage
The Code is doing summation:
The man who is answered checked as accepted, sends add function like this:
for(int i=0;i<10000;++i) {
    for(int dev=0; dev<2; dev++) {
        cudaSetDevice(dev);
        add<<<NB,NT>>>( dev_a[dev], dev_b[dev], dev_c[dev], Ns[dev] );
    }
}

In above code he added NS[dev] also to add function but, man who sends the question posts add function like this:
__global__ void add( double *a, double *b, double *c){

    int tid = threadIdx.x + blockIdx.x * blockDim.x;

    while(tid < N){
        c[tid] = a[tid] + b[tid];
        tid += blockDim.x * gridDim.x;
    }

}

What is the function of  Ns[dev] in above function. Because when i remove  Ns[dev] in below code, like this:
add<<<NB,NT>>>( dev_a[dev], dev_b[dev], dev_c[dev]);

Add function does not work. I mean it does not adds the values.
How can i use  Ns[dev] in add function here?
Answer: In the linked answer, Ns is an array specifying the amount of data which should be processed by each device. dev is the id of the current device.
You should add an argument to the kernel which should specify the length of the data being processed in the kernel.
__global__ void add( double *a, double *b, double *c, const int N)
{
    int tid = threadIdx.x + blockIdx.x * blockDim.x;

    while(tid < N){
        c[tid] = a[tid] + b[tid];
        tid += blockDim.x * gridDim.x;
    }
}

CUDA multi - gpu add function


Guideline for users to view

Question: I have a website coming up soon and the editors for this website dosen't know how to use Drupal. I have made a content guideline for users to follow(guideline for like uploading content and such), now I want this guideline to appear on the website for only certain roles to view. In my example the role editor has the only permission to view this content and no one else.
Do you guys got any clue of how to fix this and also where I should place it? On their account page or just in the navbar?
Thanks for any tips and advice
Comment: It's not clear what is the issue exactly. Also second question (where to place it) is something that is really based on preferences, not something that has a universal answer, so I suggest either paraphrasing it or taking it out.
Answer: One approach can be using content access module.So basically while creating content type you will get option to share content type with user role you want to.

This module allows you to manage permissions for content types by role
  and author. It allows you to specifiy custom view, edit and delete
  permissions for each content type. Optionally you can enable per
  content access settings, so you can customize the access for each
  content node.

Other approach does not require any module, and will be simple and out of the box.

Create a content type, lets say editor_guide.
While creating the content type opt to keep the nodes unpublished.
Now create nodes,which can be single node(copying the complete guide to one node),multiple node(dividing the guide content into chunks as separate nodes),you can also keep the guide as file field on node.
Create a view of display type page with path say editor-guide.
In views you will get option to set permission for the page, set the permission for editor, in the views filter remember to change status for content for unpublished.
Now in you nav-bar menu block add a menu item, link it to editor-guide(or whatever).

This link will be automatically visible to editor role only.

Guideline for users to view