Regression result shows small $R^2$ fit and large p value?

Question: I designed a model for my dissertation which has 7 predictors based on existing theories. After data selection, I got a sample size of 126 and I thought I was okay since the data is cross sectional. After running the regression I obtained the following result:

As you can see (i) the $R^2$ is -3.3% and p value for F test is .86 so I'm worried about the sample size being too small. (ii) all the coefficient has very large p value.
I can't find more samples as the available data is very limited and I don't have enough time to resign the dissertation. How should I interpret this result? Will conducting a failed experiment lead to a problem with the dissertation?
Comment: Could that be a duplicate?
Comment: Can you share some insight into what you are modelling?
Comment: @RichardHardy Hello, none of the observations are identical.
Comment: @Repmat Of course. The model is for testing the relationship between the average annual return per investment and cross-border capital. The followings are the main variables for my hypotheses: CB - the amount of cross-border capital, CBS - the cross-border capital share. Firm - the no. of investor, CBFrim - no. of cross-border investor.
Comment: By *duplicate* I meant that a very similar question had been asked and answered before, which means you could go and find that answer and your problem would be solved. I did not locate the duplicate myself, but I suspect it may exist. We try to avoid duplicates here at Cross Validated.
Answer: It seems to me you can make a fairly strong conclusion: The outcome you are measuring is not linearly related to the eight independent variables you selected.  You can never prove a negative, but there is no indication here that the model fits your data at all. That might be useful information.
Comment: Yes, I know fail experiment happens. If I could not fix it I would describe the characteristic of my samples and try to discuss why is my outcome so different from the others. However, given my sample is not the best, I would have to admit that I had underestimated the information availability and it is my fault that the data do not fit the model (or the other way around). Would that be a problem for my dissertation?
Comment: @Christina if there are other similar analyses available with different conclusions, could you do power calculations/simulations and see if you should be able to find anything in your sample?
Comment: @Christina . I don't see why you call it a "failure". I don't know what you measured, or what hypothesis you are testing, but you seem to have a pretty strong conclusion. You'd want to look deeper (look at residuals, etc) to make sure the findings are solid, but at first glance, you have solid negative results.
Answer: If you have anomalies in your data it can create a downwards bias in your significance tests. Additionally if you have series that are highly cross-correlated this also can have an impact on your results. If you wish you can post your data and perhaps some further inspection can be accomplished.
Answer: Very small $R^2$ and very high $p$-values for the parameters suggest that your model is poor. But, for a moment, don't look at those values, look at the standard errors and think of your parameters in terms of confidence intervals. You would calculate the 95% confidence intervals, by taking, approximately, the parameter value $\pm$ two times the errors. Look at your errors, those intervals would be huge as compared to the point estimates of the parameters, so your parameters can be literally anything and cannot be trusted. (In fact, the $p$-values for the parameters are calculated by looking at size of the parameters as compared to the magnitude of errors.)
So you cannot make any conclusions based on this data and this model. However this doesn't mean that other model wouldn't be appropriate, but that's a different story.
Answer: Well, that didn't work. Try eliminating parameters with the least contributory p(CB)=0.951 the first to go. Eventually, at least the constant should become significant, which would at least demonstrate invariant data. The other thing to try would be to look at the distribution of the $y$-values. It is entirely possible that they are not nicely distributed. In case they aren't nice, a transformation of that data by something, maybe logs, reciprocals, squares, square roots, exponentiations may produce much better regressions. That is a long list of trials because sometimes the logs or other transforms of the independent parameters is needed as well, and, for best fitting, there may be a mixture of different transforms for $y$ and $x$'s that produce best results. What that means is that if there is a functional relationship, $f(x\text{'s})$ may take any form, e.g., $f(x \text{'s})=\text{sech}(x_1^2)\space e^{x_2^{-({x_3/2})^3 }}\dots$  Although finding a functional relationship is an art form, thinking about what is happening physically can eliminate lots of elbow grease. For example, if the data is proportional type, take logs. If the data has very long right tails, try reciprocation, etc.
Comment: Thank you, transforming the data is a good advice. However, I wonder if transforming the dependent variable means R2 is less meaningful?
Comment: No, it does not mean that. Consider, for example, that if the variables are proportional type, the log transform would increase the R$^2$ value, and that would not only be more appropriate, but more indicative of the interdependent variable covariance.
Comment: Thank you that makes sense. Some of the y-values are negative or zero. Could I add a fixed value to each of the y and make it at least larger than zero before I log it? Will it affect (I) the distribution of the logged values and (ii) the significance of each variables? Someone also suggested me to leave the zero and negative values as it is and only log the positive ones.
Comment: https://robjhyndman.com/hyndsight/transformations/


How to interpret Fisher Test?

Question: How do I interpret the odds ratio for this fisher test output in R?
...below is my matrix that I input into fisher.test in R:

...and below this is the output of the test in R:
Comment: What specifically is the part you do not understand?
Comment: How to interpret the odds ratio in this case @mdewey
Answer: That p-value is for the null hypothesis of independence between the two categorical variables.  We reject the null of independence here.
For the odds ratio, if the confidence interval contains one, we fail to reject the null hypothesis of independence.
As you can see, we reject the null of independence since the confidence interval doesn't contain 1.
Update: The section below is a general description to help you understand what the odds ratio is and why a value close to 1 means we don't have evidence against independence (that is, loosely speaking, why a value close to 1 should be associated in your mind with independence)
Odds ratio $\theta$ is defined as $\theta = \frac{n_{11}/n_{12}}{n_{21}/n_{22}}=\frac{n_{11}n_{22}}{n_{12}n_{21}}$.
If the ratio of counts in row 1 is very similar to the ratio of counts in row 2, the odds ratio will be close to 1.  In the case that $\theta$ is close to one, the "row you are in" doesn't seem to affect the column count very much.  Hence, there is a (suggested) lack of association, which is to say (a suggestion of) independence.
Comment: @Jules No, your sample size is not to small and you really can make the conclusion, that "meeting" and "cfm" are not Independent.
Comment: While the proportions are not all that far from equal, at those sample sizes the null hypothesis of independence is not remotely tenable; the effect may not be all that strong (on the other hand it's not really weak either -- the second column is not close) but your last sentence is clearly at odds with the rejection of the null. There's no suggestion of independence; there's a clear negative association between cfm and meeting.
Comment: I intended for my last comments to be a general description of the odds ratio.  I'll make that clearer by editing it.
Comment: I'm a little confused about what you are suggesting. Since the odds ratio is far from 1.0, does this imply a weak association? Is my sample size too small to really make a conclusion? @Glen_b
Answer: The 95%-CI for the odds ratio spans from 0.000 to 0.6844, thus an odds ratio of 1.0 is far off and you can reject the null hypothesis, that the values in the contingency table are independent. p = 7.9e-12 means $7.9 \times 10^{-12}$, so very close to zero. Conventionally, the test result is considered 'significant' if p < 0.05. Here p is smaller than that by many magnitudes.
Comment: @jules An odds ratio of 1 means, that the odds of "meeting" are the same, no matter what "cfm" or the other way around. The stronger the odds of meeting are influenced by cfm (or the other way around) the more the odds ratio will tend towards zero or towards infinity. The important question is not, wheter 0.00000001 is in the confidence interval but that 1.000 is not within the confidence Intervall. Therefor you have proven the null hypothesis wrong and consider your test result "significant".

https://en.wikipedia.org/wiki/Odds_ratio
Comment: @jules note that the lower limit is zero because you specified a directional hypothesis and zero is the lower bound. If you had specified the other alternative the upper bound would have been infinity. Only if you ask for a two-sided interval will both bounds be estimated.
Comment: I see that 0 is the CI, but what does an odds ratio of 0 mean? @Bernhard

How to interpret Fisher Test?


If a linear operator $T^n =0$, then $T-2I$ is invertible

Question: So I know there are several duplicates to the question if a linear operator $T^n = 0$, then prove that $T-I$ is invertible. But it's set me wondering, what if it's $T-2I$? What if it's $T-aI$? I've tried approaching it along the same lines as $T-I$, but to no avail. That is, I haven't had any luck finding an inverse for the general case. Note: please don't use eigenvalues in your answers, I haven't yet studied them.
Comment: If $2=0$ in your ring, the problem is wrong. Are we supposed to assume the entries are in $\mathbb R$ or $\mathbb C$ or something?
Comment: Presumably if eigenvalues have not been studied yet one can presume real or complex...
Answer: If $T^n = 0$, then $T - aI$ will be invertible for all $a \ne 0$. This follows from the fact that $T - I$ is invertible for any such $T$. Notice that if $T^n = 0$, then
$$
\left( \frac{1}{a} T \right)^n = 0
$$
Therefore,
$$
\frac{1}{a} T - I
$$
is invertible, by the fact that we already know. Then multiplying by $a$,
$$
T - aI
$$
must also be invertible, since $a \ne 0$.
Answer: Hint: Consider the geometric series for $\dfrac{1}{z-a}$.
Answer: Suppose in your ring $u$ is a unit and you believe the theorem you referenced that "a nilpotent minus the identity is invertible".
Notice that $T/u$ is also nilpotent, and apply that theorem to get $\frac{T}{u} - I$ is invertible. But then $u(\frac{T}{u} - I)$ is invertible too.
Answer: Let $\lambda \neq 0$ and $S={1 \over \lambda} (I+({T \over \lambda}) + \cdots + ({T \over \lambda})^{n-1})$.
Note that $S(T-\lambda I) = I$


What to do with trolls?

Question: We have a recent uptick in troll activity.
The general advice is: Do not feed the troll.
Based on my own past experience this is sound advice. There is no use arguing with a troll. If he does not like what you wrote he will disagree and insult you without any reason. If you are suspicous of someone you can look at their profile. How they react to disagreement usually shows under downvoted or closed questions. To be fair, critique is upsetting to most people.
If you would save your time, stay away from questions posted by trolls.
This was not really intended as a question but it made me think of one:
Does SE have any way of dealing with trolls? E.g.: IP ban or something if user gets 100 flags?
Answer:
Does SE have any way of dealing with trolls?

Yes. Each site has a team of moderators who look after it.
If another user is doing something which you feel is disruptive, hostile, or otherwise unproductive, then flag for moderator attention. Don't engage, don't argue, just ask for help and stay out of the fray.
Alternately, calling someone out publicly on meta, regardless of whatever run ins you (or anyone else in the community) may have had with them usually leads to escalation, and a lot of gnashing of teeth for at least a couple different people. It doesn't end well. Glass houses and all that.
If you have a specific concern, please flag for moderator attention. Otherwise, please enjoy the site and get on with your day.
Answer: Even though Econ.SE is now about a year and a half old, we (the moderators) are still learning how to deal with some of these issues. But the good news is that there is a graduated range of ways that we can deal with problems and keep everything running smoothly. Here are some key steps that can be taken, in increasing order of severity:

community members (i.e. all of us) attempt to constructively engage with problems through the usual channels (comments, etc.)
persistent sources of problems are flagged for moderator attention
moderators more directly engage with problematic users to request behavior changes and issue warnings.
moderators can suspend accounts that continue to be a source of problems.

Now that this workflow is straightened-out, we should hopefully be able to deal with these issues more effectively. Thanks for everyone's patience!
The main thing for everyone to bear in mind is to try to be positive and engage constructively with people on this site, and to use flags to alert moderators of problems where necessary.

What to do with trolls?


Usar UTF-8 ou Latin1?

Question: Comecei um novo projeto e ao criar o banco de dados (MySQL), não pensei duas vezes, coloquei um CHARSET=utf8. A aplicação terá suporte ao português e inglês e os usuários deverão utilizar apenas esses dois idiomas.
Em um módulo em específico os usuários poderão redigir um procedimento. Ou seja, um texto relativamente longo, o qual vou usar um editor WYSIWYG HTML. Os usuários formatam seus textos e eu gravo o HTML no banco de dados. Para essa coluna escolhi VARCHAR(65535), assim eu usaria melhor o espaço no banco.
Porém, claro, o MySQL informou que o máximo que eu consigo no VARCHAR é 21845 por conta o UTF-8 (ocupa máximo de 3 bytes).
Pergunta: Atualmente ainda vale a pena eu utilizar Latin1, garantindo que cada caracter terá apenas 1 byte? Ou isso é obsoleto e é melhor fazer com UTF-8?
Comment: Possível duplicata de [Qual encoding escolher para um banco de dados?](https://pt.stackoverflow.com/questions/21230/qual-encoding-escolher-para-um-banco-de-dados)
Comment: Sendo a 3 anos atrás não muda, a escolha do tipo de algo é conforme a sua necessidade (a necessidade da aplicação que cria), se fosse um encoding melhor que o outro nem existiria o "pior", talvez ele já teria sido removido ou descontinuado. E mesmo que supostamente algo tivesse mudado ainda sim o correto seria você lançar uma recompensa na pergunta já existente solicitando respostas atualizadas e talvez comentar na resposta existente afirmando que algo mudou.
Comment: Se não vai atender povos não ocidentais, e não pretende encher sua aplicação de emoticons, latin-1 é uma boa pedida ainda, é mais compacto e mais performático (pela associação 1:1 entre bytes e caracteres). Se pretende internacionalizar, pode usar alguma das variantes de Unicode. Cada caso é um caso. O importante é entender que elogio cego a UTF-8 é falta de conhecer o que acontece por "baixo do capô". Unicode gasta mais espaço para ter mais caracteres. UTF-8 custa bem mais processamento do que o Unicode "cheio", mas balanceia espaço e versatilidade (só gasta espaço extra acima do char 127).
Comment: @RBZ cheguei a ler a resposta que você indicou. Porém é de 3 anos atrás, além de ter algumas informações que agora estão incorretas.
Comment: Acredito que não tenha diferença da resposta do post. Mas em todo caso, vamos aguardar alguém que possa confirmar isso ! ;]
Answer: Não existe uma resposta correta para escolha do encoding. A escolha deve ser feita de acordo com a sua necessidade. É por esse motivo que os bancos aceitam vários tipos.
Se o seu sistema não tiver chance de receber nenhum caractere especial, como no caso que você está descrevendo onde o conteúdo será sempre um HTML em que você pode, a priori, trocar todos os caracteres especiais por suas representações Unicode (i.e. &#nnnn; onde nnnn é o código unicode), então é provável que você não precise armazenar esse dado em UTF-8. Você inclusive pode ter todo o seu banco de dados como um collation UTF-8 e apenas esse campo do HTML com uma collation diferente.
Porém, muitas vezes você não tem controle sobre como o HTML será gravado no campo, você não tem um filtro para converter em casos onde o usuário colar algum caractere especial, etc. Se for esse o caso, então a melhor estratégia é usar o Unicode.
Outra questão está no fato de você escolher um campo varchar ou um campo text para guardar esse tipo de informação. Cada tipo de campo tem suas vantagens e desvantagens, principalmente se você tem alguma intenção de aplicar filtros ou ordenações sobre esse conteúdo. Os campos text também podem ser indexados, mas tem um limite (prefixo) que você deve escolher para a comparação dos caracteres. Existe também funcionalidades de FULL TEXT SEARCH no MySQL que podem ser aplicados nos dois tipos de campo.
Se apenas for uma questão de armazenar e recuperar o dado, eu indicaria o uso de um campo do tipo text em que você não se preocuparia com limitações de tamanho, no caso de você não ter esse controle do input do usuário.
Outro aspecto, é que hoje em dia a preocupação do campo ocupar 1 byte ou 2 bytes por caractere não tem muito sentido dado o custo por byte de armazenamento em disco. Apenas se você tem um sistema com uma quantidade muito grande de dados que precise replicar em várias instâncias e o custo de armazenamento do seus provedor for caro.
Se essa for sua principal preocupação e não tiver certeza se o conteúdo vai usar ou não Unicode, escolha o UTF-8. Isso irá facilitar seus scripts de banco, suas conversões quando fizer a leitura no programa e para exibir em páginas HTML.
Answer: No MySQL você tem os tipos MEDIUMTEXT (16M caracteres) e LONGTEXT (4B), então não há necessidade de se preocupar com limitações impostas pelo encoding. Padronize em UTF-8 e seja feliz :)
Comment: esses dois tipos, no caso da coluna ser opcional, elas não ocupam espaço no banco de dados desnecessariamente?
Comment: acho a resposta totalmente pertinente. Só não quero desperdiçar espaço no banco de dados.
Comment: Não, elas ocupam apenas o espaço necessário. O único "desperdício" são 3 ou 4 bytes por linha x coluna para armazenar o tamanho da string (em MEDIUMTEXT e LONGTEXT respectivamente).
Answer: Utilize apenas latin1 se tiver a certeza que não precisará de caracteres além dos que precisamos nas línguas latinas. O latin1 deixou de ser recomendado por representar carateres insuficientes (inclusive pelos próprios criadores que recomendam o utf-8 em sua substituição). Atualmente, utf-8 é o mais indicado, pois permite codificar quase todos os caracteres de todas as línguas com 1, 2 ou 3 bytes. Raramente chega aos 4 bytes.
Comment: Por que? O Latin1 tem justamente os caracteres que você diz não ter. Até a infromação sobre UTF-8 está errada.
Comment: Eu uso latin1 (na verdade, Win 1252, quando pra desktop) pra maior parte das coisas justo por ter todos os caracteres especiais que a língua portuguesa utiliza, e ser muito mais performático que UTF-8. Seria legal dar uma revisada nos conceitos e [edit] o post para dar uma melhorada.
Comment: Porquê a informação sobre o utf-8 está errada?
Comment: Por outra, acha que com 256 caracteres consegue representar tudo o que quer? Exemplo: Experimente introduzir símbolos como € (Euro) nas suas bases de dados que diz que utiliza latin1 e depois diga se o resultado é satisfatório.

Usar UTF-8 ou Latin1?


Appropriate GLM for proportion response variable

Question: I have a dataset that includes the number of apples harvested from 31 trees and the number of apples from each tree affected by different forms of pest damage. I also have insect abundance data (averaged from multiple samples throughout the season) associated with each of those trees, and I'd like to see if I can correlate abundance of certain insect taxa with pest damage types. Here's a simplified example of that data:
apple.example <- data.frame(treeID =
       paste("A", c(1:5)),
     applecount = c(1,1,13,14,5),
     chew = c(0, 0, 2, 2, 1),
     codling = c(0, 0, 8, 6, 4),
     spiders = c(1.875, 2, 1.286, 2, 1.67))

Each apple can have more than one kind of damage, but I don't have per-apple level data. Lots of trees had no apples, and of those that did there are plenty of 0% and 100% frequency of damage types.
apple.example[3:4] <- apple.example[3:4] /
              apple.example$applecount

Since I don't care about the counts of pest damage (confounded by yield) but rather their relative frequency, my initial thought was to convert these values into percentages and analyze them using a binomial GLM. When I do that, R complains that the response values are not all 0 or 1. A few posts I've seen suggest that isn't necessarily an issue, but what gives me pause is that the lsmeans predicted values are not bounded between 0 and 1 as a binomial distribution ought to be. Also, AICc model selection results are not finding relationships among the data even where I might expect one, so I'm not 100% confident this is a good way to model this data.
m1 <- glm(chew~spiders, data=apple.example,
          family="binomial")

Discussion here suggests that a beta regression is better for this kind of data, so I switched to that, incorporating the transformation here to circumvent the 0/1 error there.
y.transf.betareg <- function(y){
  n.obs <- sum(!is.na(y))
  (y * (n.obs - 1) + 0.5) / n.obs
}

m2 <- betareg(y.transf.betareg(chew) ~
        spiders, data=apple.example)

Both of these feel like bad workarounds for something that ought to be straightforward, though? The betareg method provides more resolution between models in AICc output, but neither are particularly good. I guess I'm just looking for some reassurance that one of these is a valid approach with this data, or, ideally, a more appropriate method.
Comment: The LS means will be on the same scale as the model's predictions -- which will be the logit scale, not the percentage scale. (Look at the annotations below the output which say just that.) I you add `type = "response"` to the `lsmeans` call, they will be in range.
Comment: This works great, thanks for pointing out that simple omission. Is it safe to take this as a tacit endorsement of the binomial approach for this sort of data?
Comment: I posted a corrected version of my second comment (since deleted) as an answer
Answer: First, you should incorporate the number of trials in the model. I illustrate below using what I feel is a more elegant way to obtain the fractions, and creating new variables pchew and pcodling:
> apple.example <- data.frame(treeID = paste("A", c(1:5)),
+                             applecount = c(1,1,13,14,5),
+                             chew = c(0,0,2,2,1),
+                             codling = c(0,0,8,6,4),
+                             spiders = c(1.875,2,1.286,2,1.67))
> apple.example <- transform(apple.example,
+                            pchew = chew/applecount,
+                            pcodling = codling/applecount)
> m2 <- glm(pchew ~ spiders, weights = applecount,
+           data = apple.example, family = "binomial")

> library(lsmeans)
> lsmeans(m2, "spiders", at = list(spiders = 1:3), type = "response")
 spiders      prob        SE df   asymp.LCL asymp.UCL
       1 0.1730220 0.1549970 NA 0.024423399 0.6361675
       2 0.1351918 0.0826546 NA 0.037635848 0.3845711
       3 0.1045869 0.1925788 NA 0.002070816 0.8679789

Confidence level used: 0.95
Intervals are back-transformed from the logit scale

Alternatively, you specify the same model via the success and failure counts (making the transform step unnecessary) as follows:
> m2 <- glm(cbind(chew, applecount - chew) ~ spiders,
+           family = "binomial", data = apple.example)

Comment: You're just conditioning on applecount and in principle I don't see any issue with also using it as a predictor. I'm more concerned with the solution I gave from the standpoint of it assumes no variations other than those inherent in the binomial model -- and it seems reasonable to believe that trees vary somewhat from one another. I think people fit something called an overdispersion model in such cases but I'm no expert. I hope somebody else will comment.
Comment: This is great, thanks--exactly the simple solution I was looking for. One follow up question: is it valid to use applecount as a nuisance variable in a model like the last one you suggested? When the response variable is a percent it's distanced from the absolute count but it seems sketchy for count to appear in both the predictor and response variables. There are a couple of cases where including it as a predictor changes the AICc rankings and I want to get it right.


Incremented JavaScript variable is not passing as the parameter

Question:


Lower bound for the function $f(x) = x\sin x$

Question: Consider the function $f:\mathbb{R}\to\mathbb{R}$ given by
$$f(x)=x\sin x.$$
If $x$ is sufficiently large, I mean $R>0$ sufficiently large exists such that $x$ is in the complementar of the ball $B(0, R)$, it is possible to have a lower bound for the function $f$?
Actually I think to the limit for $x\to +\infty$ fo that function which is indeterminate, so I think that it is not possible to find a lower bound, but I am not sure about that.
${\bf EDIT:}$ So, in articular, it is possible or not to have a lower bound for the integral
$$\int_0^T f(x) dx,$$
with $T>0$, isn't it?
Could someone please help me with that?
Thank you in advance!
Comment: No, $f$ is unbounded in both directions: $\sup\limits_{x\in\Bbb{R}}f(x)=\infty$ and $\inf\limits_{x\in\Bbb{R}}f(x)=-\infty$.
Comment: $f(\frac {3\pi } 2 +2n\pi) \to -\infty$ as $n \to \infty$.
Comment: "Actually I think to the limit for $x\to +\infty$ fo that function which is indeterminate, so I think that it is not possible to find a lower bound". You should have a look to the graph of this function, to see why it has no lower bound. It can help you to solve similar problem.
Answer: Consider the following sequences, where $k$ goes to infinity.
$$f(\pi/2 + k2\pi) = \pi/2 + k2\pi \rightarrow +\infty$$
$$f(-\pi/2 + k2\pi) = \pi/2 - k2\pi \rightarrow -\infty$$
You can compute the integral explicitely with an integration by part.
$$ \int_0^T x \sin(x) dx = [-x \cos(x)]_0^T - \int_0^T \cos(x) dx =
-T \cos(T) +\sin(T).$$
Here again you can find sequences for which this last quantity goes to $-\infty$ or $+\infty$.
Comment: coudy, so it can be found neither a lower bound nor an upper bound for the iintegral, isn't it?
Comment: You are correct. There is no lower bound nor upper bound for the integral.


Why may a matrix be singular or ill-conditioned with standard learning algorithm for linear classification?

Question: In the learning algorithm for linear classification by least square method, which find a weight vector $\hat w\in R^d$ and bias $\hat b\in R$ for a linear scoring function $f(x) = \hat w ^T x +\hat b$ for which may write the solution for $\hat w =[\sum^n_{i=1}(x_i-\bar x)(x_i -\bar x)^T]^{-1}\sum^n_{i=1}(x_i-\bar x)(x_i -\bar x)$, why may the matrix $\sum^n_{i=1}(x_i-\bar x)(x_i -\bar x)^T$ be singular or ill-conditioned ? What does that mean to be ill-conditioned ? why does it occurs when $n$ is less than the dimension of $x$ ?
Comment: This question seems answerable directly from definitions. A matrix is singular if it is not full rank. A matrix is ill-conditioned if it has a high condition number.
Comment: @Sycorax Yes but how may $\sum^n_{i=1}(x_i-\bar x)(x_i -\bar x)$ be singular or ill-conditioned? What issues does it imply so that the standard learning algorithm may not be computable then ?
Comment: The expression you've written in your comment is not a matrix, just a sum of scalar products, so the concepts of singularity don't apply. Did you mean to include a transpose? If so, you should consult Mark L Stone's answer.
Comment: @Sycorax, there is a missing ^T typo in the last occurrence of that in the original question.  I answered as though the typo hadn't occurred.
Comment: @Sycorax, I've taken the liberty of inserting the missing ^T in the OP's question.
Comment: @MarkL.Stone Thank you. I had mentally filled in the $~^\top$ when reading the question, but when the error was repeated in the comment I thought it best to ask for clarification.
Comment: Yes ! The $T$ is missing in my comment above !
Sorry if I am not that active, I am still trying to figure the answer out. I'm so bad and slow ind understanding maths ! I'm only a computer scientist
Answer: $\sum^n_{i=1}(x_i-\bar x)(x_i -\bar x)^T$ is a $d$ by $d$ matrix, where $x$ is a $d$ by $1$ vector.
For each $i$, $(x_i-\bar x)(x_i -\bar x)^T$ is a $d$ by $d$ matrix.  This matrix is rank one, because it is the outer product of a vector with itself.  $\sum^n_{i=1}(x_i-\bar x)(x_i -\bar x)^T$ is the sum of $n$ rank one matrices, and therefore has a rank not exceeding $n$.  If $n < d$, $\sum^n_{i=1}(x_i-\bar x)(x_i -\bar x)^T$ must therefore be singular, because it can not be of full rank k.
In a sense, singular is the most ill-conditioned a matrix can be, having condition number = $\infty$.  Even if $\sum^n_{i=1}(x_i-\bar x)(x_i -\bar x)^T$ is not singular, which can only be the case if $n \ge d$, it may still be ill-conditioned, thereby resulting in numerically unstable and very sensitive calculations when it is inverted or used in a linear system of equations. There are shrinkage estimators (search on this site and on the internet) to try to improve the conditioning of (sample) covariance matrices.  $\sum^n_{i=1}(x_i-\bar x)(x_i -\bar x)^T$ is the sample covariance matrix, other than not dividing by $n$ or $n-1$; note that dividing by $n$ or $n-1$ does not affect the rank vs not dividing at all.


Elasticity function of a given demand function

Question: $$
q=D(x)=\frac{300}{x}\ \ \ \ \ \ \ \  \text { (given ) }
$$
as we know
$$
\begin{array}{l}
E(x)=\dfrac{-\frac{d q}{d x}}{\frac{q}{x}} \\
\dfrac{d q}{d x}=\dfrac{-300}{x^{2}}
\end{array}
$$
$$
E(x)=-\left(-\frac{300}{x^{2}} \times \frac{x \times x}{300}\right)=1
$$
The demand function is given
$D(x) = 300/x$
See where am i going wrong
I used
$E(x)= ( -{\rm d}q/{\rm d}x)/(q/x)$
${\rm d}q/{\rm d}x= -300/x^2$
$E(x)= 1$
Comment: Welcome to MSE.  Please type your questions instead of posting images.  Images can't be browsed and are not accessible to those using screen readers.  If you need help formatting math on this site, here's a [tutorial](https://math.meta.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference)
Comment: Ok sure please help me out now
Answer: The formula is $$\epsilon=|\frac{dQ/Q}{dP/P}|=|\frac{dQ}{dP}\frac{P}{Q}|$$
Now
$$Q=300/P$$
$$\frac{dQ}{dP}=-300/P^2$$
$$\epsilon=|\frac{dQ}{dP}\frac{P}{Q}|=|-300/P^2\times P/Q|=|-300/PQ|=|-1|=1$$
Comment: So i did it right , you mean


Como agrupar um array de objetos em javascript (Angular)

Question: Como ler um array de objetos e retorna apenas objetos que contém o mesmo valor em vários atributos?
Fiz esse array de objetos como exemplo.
var arrayDeObjetos = [
  {"dia":1 , "desc":"Nigth","mes":1},
  {"dia":1 , "Descricao":"Brilha","mes":1},
  {"dia":2 , "desc":"Urna","mes":2},
  {"dia":3 , "Descricao":"Covas","mes":1},
  {"dia":2 , "desc":"Mario Corvas","mes":1},
  {"dia":4 , "Desc":"Estrelas","mes":2}
];

Nesse Array eu tenho vários objetos eu queria agrupar todos e um novo array que contem um objeto com  todos que contem dia = 1 e mes = 1 outro objeto que contém todos com dia = 2 e mes = 2
O resultado final tem que ser algo como
var resultado = [
  [
    {"dia":1,"mes":1,"desc":"descricao AQI"},
    {"dia":1,"mes":1,"desc":"Outra desc"}
  ],
  [
    {"dia":2,"mes":2,"desc":"outro mes"},
    {"dia":1,"mes":1,"desc":"outra coisa"}
  ]];

Comment: Há "N" formas de fazer isso, desde iterações com _forEach_ ou _for_, ou usando o método _map()_ > mais informações dele aqui: [W3Schools - Map](http://www.w3schools.com/jsref/jsref_map.asp). Ou também, utilizar uma biblioteca muito boa que irá facilitar sua vida, chama-se [LinqJS](https://github.com/mihaifm/linq).
Comment: O resultado é esse mesmo ou você quis dizer que em cada índice do array contenha os objetos com mesmo dia e mesmo mês?
Comment: Joao, no `resultado` que tens na pergunta tens `dia 1` e `mês 1` em ambas... é erro?
Comment: @Sergio foi erro de digitação meu
Comment: Ok. Podes corrigir na pergunta? eu gostava de dar uma resposta mas quero ter a certeza do resultado que esperas.
Answer: Uma maneira é utilizando o Javascript Filter para encontrar somente os valores que precisa e inserí-los em um novo array utilizando o push, veja o exemplo abaixo:

var arrayDeObjetos = [
  {"dia":1 , "desc":"Nigth","mes":1},
  {"dia":1 , "Descricao":"Brilha","mes":1},
  {"dia":2 , "desc":"Urna","mes":2},
  {"dia":3 , "Descricao":"Covas","mes":1},
  {"dia":2 , "desc":"Mario Corvas","mes":1},
  {"dia":4 , "Desc":"Estrelas","mes":2}
];

var novoArray = [];

novoArray.push(filtrarArray(arrayDeObjetos, 1, 1));
novoArray.push(filtrarArray(arrayDeObjetos, 2, 2));

console.log(novoArray);

function filtrarArray(array, dia, mes) {
  return array.filter(function(val) {
    return val.dia === dia && val.mes === mes;
  });
}
Answer: Se o array de objetos for muito extenso, você pode separar um array de objetos não duplicados mapeando os dias e meses e então filtrar com base no que foi descoberto.
var arrayDeObjetos = [
  {"dia":1 , "desc":"Nigth","mes":1},
  {"dia":1 , "Descricao":"Brilha","mes":1},
  {"dia":2 , "desc":"Urna","mes":2},
  {"dia":3 , "Descricao":"Covas","mes":1},
  {"dia":2 , "desc":"Mario Corvas","mes":1},
  {"dia":4 , "Desc":"Estrelas","mes":2}
];

Irá se tornar em objetosASeparar:
objetosASeparar = [
  {dia: 1, mes: 1},
  {dia: 2, mes: 2}
]

Então caso houver dia 3 e mês 3, dia 4 e mês 4 assim por diante, todos serão filtrados em um novo objeto.

var arrayDeObjetos = [
  {"dia":1 , "desc":"Nigth","mes":1},
  {"dia":1 , "Descricao":"Brilha","mes":1},
  {"dia":2 , "desc":"Urna","mes":2},
  {"dia":3 , "Descricao":"Covas","mes":1},
  {"dia":2 , "desc":"Mario Corvas","mes":1},
  {"dia":4 , "Desc":"Estrelas","mes":2}
];

// obter um array de objetos com dia e mes não duplicados
var objetosASeparar = [];
arrayDeObjetos.filter(item => item.dia == item.mes).map(function(item){
  if(item.dia == item.mes){
    return {
      dia: item.dia,
      mes: item.mes
      }
  }
}).forEach(function(a){
  if (objetosASeparar.length == 0) objetosASeparar.push(a);
  objetosASeparar.forEach(function(b){
    if (b.dia !== a.dia && b.mes !== a.mes)
      objetosASeparar.push(a);
  });
});

// filtrados com base nos objetos descobertos
var novoObjeto = {};
objetosASeparar.forEach(function(a,i){
    novoObjeto[i] = arrayDeObjetos.filter(b => a.dia == b.dia && a.mes == b.mes);
});

console.log(novoObjeto);
Answer: Faça
var objetos = [
   {"dia":1 , "desc":"Nigth","mes":1},
   {"dia":1 , "Descricao":"Brilha","mes":1},
   {"dia":1 , "desc":"Nigth","mes":1},
   {"dia":2 , "desc":"Urna","mes":2}
]

var meuNovoArray = [];

angular.forEach(objetos, function(objeto)){
     if(objeto.dia === 1){
         menuNovoArray.push(objeto) //adiciona o obj nesse seu novo array
     }
}

Depois basta dar um console.log(meuNovoArray) que vai dar certo.
Answer: Este tipo de operação é chamada de map reduce, no javascript todo array possui a operação reduce para estes casos. Segue 2 exemplos de como utilizá-la no seu caso.
Utilizando Objeto:
Object.values([
    {"dia":1 , "desc":"Nigth","mes":1},
    {"dia":1 , "Descricao":"Brilha","mes":1},
    {"dia":2 , "desc":"Urna","mes":2},
    {"dia":3 , "Descricao":"Covas","mes":1},
    {"dia":2 , "desc":"Mario Corvas","mes":1},
    {"dia":4 , "Desc":"Estrelas","mes":2}
  ].reduce(function(prev, item) {
    var key = item.dia * 100 + item.mes;
    prev[key] = prev[key] || [];
    prev[key].push(item);
    return prev;
  }, {})
);

Utilizando Array:
[
  {"dia":1 , "desc":"Nigth","mes":1},
  {"dia":1 , "Descricao":"Brilha","mes":1},
  {"dia":2 , "desc":"Urna","mes":2},
  {"dia":3 , "Descricao":"Covas","mes":1},
  {"dia":2 , "desc":"Mario Corvas","mes":1},
  {"dia":4 , "Desc":"Estrelas","mes":2}
]
.reduce(function(prev, current) {
    var index = prev.findIndex(function(j) { return j.dia == current.dia && j.mes == current.mes});
    if (index < 0) {
        index = prev.length;
        prev.push({
            dia: current.dia,
            mes: current.mes,
            lista: []
        });
    }
    prev[index].lista.push(current);
    return prev;
}, [])
.map(function(item){
    return item.lista;
});

Apesar de mais complexo o método utilizando array deve ser mais eficiente que o método utilizando objeto.
Observação Não indexe no array o dia e o mês, ex: array[mes*100 + dia], pois na linguagem javascript ele ira criar todas as posições no array até chegar ao valor informado.


Split the combine data(Numeric and Sting) to separate data

Question: How can i split string data 'Test123' to two columns 'Test' and '123'.
Thanks in advance.
Comment: [This question](http://stackoverflow.com/questions/11132734/getting-a-certain-part-of-the-string-in-t-sql "Getting a certain part of the string in T-SQL") is similar. Although it asks about a string where the text part has a fixed length, most suggestions disregard that and thus can be equally applied to your problem.
Answer: I have created below function for this, Please update here if there is any other simple method
<code>/*
SELECT * FROM [dbo].[SPLIT_NUMERIC_STRING] ('TEST1235745DFGBFVNYT HGEFNEYGV  @ADBSD H# CHSC5ae23674523')
*/
CREATE FUNCTION [dbo].[SPLIT_NUMERIC_STRING]
   (  @ColumnName VARCHAR(MAX)
   )
   RETURNS @TABLEOFVALUES TABLE
      (  NUMERICDATA   VARCHAR(MAX),
         STRINGDATA VARCHAR(MAX)
      )
AS
   BEGIN

        DECLARE @Column1 VARCHAR(100)
        DECLARE @Column2 VARCHAR(100)
        DECLARE @StartPosition INT
        DECLARE @String VARCHAR(100)
        DECLARE @NUMBER INT
        DECLARE @ENDPosition INT
        SET @Column1 = ''
        SET @Column2 = ''
        SET @StartPosition = 1

        SELECT @ENDPosition =  LEN(@ColumnName)

        WHILE @ENDPosition > 0
        BEGIN

            SELECT @String = SUBSTRING(@ColumnName,@StartPosition,1)
            SELECT @NUMBER = ISNUMERIC(@String)

            IF (@NUMBER = 1)
            BEGIN
             SET @Column1 = @Column1 + @String
            END
            ELSE
             SET @Column2 = @Column2 + @String

            SET @StartPosition = @StartPosition + 1
            SET @ENDPosition = @ENDPosition - 1
        END
        INSERT INTO @TABLEOFVALUES (NUMERICDATA,STRINGDATA)
        SELECT @Column1 AS NUMERICDATA,@Column2 AS STRINGDATA

      RETURN

   END
</code>
Answer: Some ideas:

build functionality so you can use Regular Expressions directly in Sql. Example: http://www.codeproject.com/Articles/42764/Regular-Expressions-in-MS-SQL-Server-2005-2008
use substring
if the length of letters vs digits is not fixed you may also use a WHILE loop for example and grab one character after another after you find a digit


Tratar array na controller Cakephp 3

Question: Entao gente eu tenho duas controllers, a produtosvendas e a vendas, em que no produtosvendas eu seleciono produto com checkbox e envio por um buttom para outra pagina, e na controller produtosvendas eu teria que tratar a array, e ela seria enviada para vendas para mostrar na tela os produtos que eu selecionei.
View da ProdutosVendas

    <?= $this->Form->create("carrinho", ["class " => "form-add", "action" => "carrinho", "controller" => "Produtos-Vendas"]) ?>
<button type="submit" class="btn-success btn">Carrinho ()</button>
<?php foreach ($produtos as $produto): ?>

        <?= $this->Number->format($produto->id) ?>
        <?= h($produto->name) ?>
        <?= h($produto->cor) ?>
        <?= h($produto->tecido) ?>
        <?= $this->Number->format($produto->estoque) ?>
        <?= $this->Number->format($produto->preco) ?>

            <button type="button" class="btn-success btn addcart">+ Adicionar</button>
            <input type="checkbox" style="" name="check[]" class="checkbox" value="<?php echo $produto['id']; ?>">

<?php endforeach; ?>
<?= $this->Form->end() ?>

Controller da ProdutoVendas
public function carrinho() {
    $check = $this->request->getData();
    if ($check == null) {
        return $this->redirect(['action' => 'index']);
    } else {
        return $this->redirect(['controller' => 'vendas', 'action' => 'index']);
    }
}

Meu problema é que eu nao sei como tratar essa array $check para que na tela vendas seja exibido o que eu selecionei
Answer: Descobri como resolver meu problema, minha view continua a mesma, mas na minha controller ficou assim:
    public function carrinho(){

    $check = $this->request->getData();
    if ($check == null) {
        return $this->redirect(['action' => 'index']);
    }else{
        session_start();
        $_SESSION['check'] = $check;
        return $this->redirect(['controller' => 'vendas', 'action' => 'index']);
    }
}

E na view que eu gostaria de exibir/passar esses dados eu somente iniciei a sessão check novamente e extrai os valores da session.

Tratar array na controller Cakephp 3


Non-zero element in exterior power

Question: Let $R=\mathbb{Z}[\sqrt{5}]$ and consider $I=(2,\, 1+\sqrt{5})$ as an $R$-module. I'm struggling to prove that the element $2\wedge (1+\sqrt{5})$ of the exterior power $\Lambda^2(I)$ is non-zero. I should construct some alternating bilinear map from $I\times I$ in $\mathbb{Z}$ (regarded in some way as an $R$-module?) not sending $(2,\, 1+\sqrt{5})$ in $0$: I'm thinking of something similar to the determinant since each element $u\in I$ can be expressed as $2a+b(1+\sqrt{5})$, but it seems it doesn't work so well... Any idea about how to do this?
See also Remark 4.7 here: https://kconrad.math.uconn.edu/blurbs/linmultialg/extmod.pdf
Comment: I may be wrong, but the fact that you look for a bilinear map $I\times I\to\Bbb{Z}$ suggests to me that the tensor product might be over $\Bbb{Z}$ as well. That's why I ask.
Comment: Ok. But $\Bbb{Z}$ is not an $R$-module in any way. You need a bilinear map $I\times I\to ?$. And I'm thinking about the appropriate $?$
Comment: @JyrkiLahtonen Of course, if we factor by $2$, we have that $-b'bc+P=b'bc+P$! Thanks
Comment: What is your definition of $\wedge^2(I)$ here? Apparently some kind of a quotient of $I\otimes_SI$, but surely the answer then depends on whether $S=R$ or $S=\Bbb{Z}$. In the latter case $I$ is a rank two free module, and $\wedge^2I$ is a free module of rank one (and the answer to your question is obvious). If $S=R$ and $\wedge^2(I)$ is to be viewed as an $R$-module, then this likely to be torsion, but may still be non-zero.
Comment: @JyrkiLahtonen $\Lambda^2(I)$ is the exterior power (see https://kconrad.math.uconn.edu/blurbs/linmultialg/extmod.pdf ) i.e. a quotient of $I\otimes I$ with the submodule generated by $a\otimes a$. I said in the question that I regarded $I$ as an $R$-module and that $\mathbb{Z}$ should be regarded as an $R$-module too. This is essentially Remark 4.7 in https://kconrad.math.uconn.edu/blurbs/linmultialg/extmod.pdf
Comment: I would try and find a bilinear map from $I\times I\to R/P$, where $P$ is the ideal generated by $2$ and $1+\sqrt{5}$. After all, recycling the argument from Conrad's example 4.5. shows that $2\wedge(1+\sqrt5)$ is annihilated by all the elements of $P$.
Comment: @JyrkiLahtonen If I define the map $\varphi: I\times I\rightarrow R/P$ to be the determinant map with respect to the $\mathbb{Z}$-basis $(2,\, 1+\sqrt{5})$ is seems that $(a'+b'\sqrt{5})(ad-bc)=a'ad-a'bc-ab'd+b'bc+P$ while $\varphi((a'+b'\sqrt{5})(2a+(1+\sqrt{5})b), 2c+(1+\sqrt{5})d))=a'ad-a'bc-ab'd-b'bc+P$, so they are not equal:, but they should be, yes?
Answer: We have an exact sequence of $R$-modules
$$K\hookrightarrow R^2\xrightarrow{[2,1+\sqrt 5]}I\to\{0\}$$
where
$$K=\{(1+\sqrt 5)b-2d,(1+\sqrt 5)d-2(b+d)):b,d\in\Bbb Z\}\subseteq I^2$$
By this, we get a commutative diagram of $R$-module homomorphisms with exact rows $\require{AMScd}$:
\begin{CD}
K\otimes_R\Lambda_R^1(R^2)@>>>\Lambda_R^2(R^2)@>>>\Lambda_R^2(I)@>>>\{0\}\\
@V\sim VV@V\sim VV@|\\
K\otimes_RR^2@>>\varphi>R@>>\psi>\Lambda_R^2(I)@>>>\{0\}
\end{CD}
where $\varphi((h,k)\otimes(u,v))=hv-ku$ and $\psi(1)=2\wedge(1+\sqrt 5)$.
If $\pi:R\to R/I$ denote the canonical projection, then $\pi\circ\varphi=0$, hence $\pi$ factors through $\psi$ giving rise to a non-zero $R$-module homomorphism $\Lambda_R^2(I)\to R/I$.

Non-zero element in exterior power


Application that modifies its underlying db schema

Question: Under what circumstances (if any) would you allow an application to change its underlying database schema? I.e. to add new column.
EDIT: A little bit of elaboration on what prompted me to ash this question. My system receives raport files from other systems and saves fetched data in "reports" table. Report includes several "parameters" which are now hard-coded: date, amount, unit_price. This parameters are saved in adequate columns in "reports" table.
We need to introduce the possibility of dynamically (via web interface) adding new parameters which can be read from report file and inserted to db.
One option is to save reports in some serialized form but there's performance issue. There will be thousands of reports each day and there will be a need to query "reports" table by parameters' values.
Answer: On first load of a new version. This is especailly relevant for systems which assign a database to each unique instance. Data writes should be marked with a system variable, and tested before update with something like
<code>show columns from test like 'new_column';
if ( ! new_column )
  alter table add column new_column after some_column
</code>
With regards to the other posts, if you are running a system with incremental versioning on the same database schemas (but different instances of those schemas) the only way to manage that change is to programmatically update the database - otherwise you'll have inconsistencies between the application versions (i.e. 1.0 and 1.1) and the schemas - it's not possible to apply the database changes across all databases at the same time in this instance.
Comment: +1 good point (though I'd try and isolate any update step from the app code itself, which is probably splitting hairs, I know!)
Answer: To put it simply, never.  Changing a database structure should be a controlled change - allowing an application (other than, for example, an application designed to upgrade a database from one version to another) is breaking this control.
Perhaps you can elaborate more on why you think this is a good idea, or what's prompted you to ask the question.
Answer: I wouldn't - ever. I know it is possible to make changes to POJOs and allow Hibernate, for example, to propogate any changes necessary to the database, but this is IMHO a bad idea, as it removes the ability to exercise proper configuration management of your schema.
EDIT: possible exception: see Andy's answer (although I would still try and separate an update step from the application itself).


Cardinality of the set of functions which holds the Equality

Question: Let $f$ be a function from $\{1,2,3, \dots ,10 \}$ to $\mathbb R$ such that
$$\bigg( \sum_{i=1}^{10}\frac{|f(i)|}{2^i}\bigg)^2 = \bigg( \sum_{i=1}^{10} |f(i)|^2 \bigg) \bigg(\sum_{i=1}^{10} \frac{1}{4^i} \bigg)$$
Let $S$ be the set of all functions which holds this Equality. Find the Cardiality of $S$
I have no idea how to start . I would be thankful if someone help me.
Answer: This is really Cauchy-Schwarz inequality, which says that
$$\left(\sum_{i=1}^n a_i b_i\right)^2 \le \left(\sum_{i=1}^n a_i^2 \right) \left( \sum_{i=1}^n b_i^2 \right)$$
and equality holds if and only if $a_i = c b_i$ for some $c\in \mathbb R$. Now in your situation, $a_i = |f(i)|$, $b_i = 1/2^i$, which means that you must have
$$ |f(i)| = c\frac{1}{2^i},\ \ i=1, 2, \cdots, 10$$
for some $c\in \mathbb R$. In particular, the cardinatlity of $S$ is that of $\mathbb R$.


Reduce rows into json key / value pair with postgres

Question: I have a postgres table something like this.
+----+-----------+----------------------+--------+
| id |    Key    |        Value         | userId |
+----+-----------+----------------------+--------+
|  1 | email     | email@example.com     |      1 |
|  2 | firstName | thomas               |      1 |
|  3 | lastName  | reggi                |      1 |
|  4 | email     | firstname.lastname@example.com |      1 |
+----+-----------+----------------------+--------+

I'm looking for a way to "reduce" this table down to a json object.
{
   "email": "firstname.lastname@example.com",
   "firstName": "thomas",
   "lastName": "reggi"
}

How close can I get to this with just using postgres?
Answer: If the table is called data, try this (jsonb_pretty is just for display purposes):
SELECT jsonb_pretty(
          jsonb_object_agg(key, value ORDER BY id)
       )
FROM data
WHERE userid = 1;

┌──────────────────────────────────────┐
│             jsonb_pretty             │
├──────────────────────────────────────┤
│ {                                   ↵│
│     "email": "firstname.lastname@example.com",↵│
│     "lastName": "reggi",            ↵│
│     "firstName": "thomas"           ↵│
│ }                                    │
└──────────────────────────────────────┘
(1 row)

This relies on the feature that jsonb does not keep duplicate keys.
It also relies on the fact that jsonb will always retain the last added key/value pair.
Comment: It can be controlled, which keys to kept: `jsonb_object_agg` (just like any other aggregate function) accepts an `ORDER BY` clause, f.ex. `jsonb_object_agg(key, value ORDER BY id)` to preserve the `key` with highest `id` (the last added key-value pair will be kept)
Comment: It [is documented](https://www.postgresql.org/docs/current/static/datatype-json.html): *If duplicate keys are specified in the input, only the last value is kept.* -- even the `json` type have similar handling: *if a JSON object within the value contains the same key more than once, all the key/value pairs are kept. (The processing functions consider the last value as the operative one.)*
Comment: Yes, but that means that you have to rely on an undocumented implementation detail in `jsonb`, namely that the last added pair will be kept. Maybe I am overly anxious. I'll add a few ideas to the answer to address the problem.
Comment: Great, thanks. I have improved the answer accorsing to that.
Answer: If you want to always have the latest value for a key, you could use a CTE and the RANK() window function:
SELECT * FROM p;
┌────┬───────────┬──────────────────────┬────────┬────────────────────────────┐
│ id │    key    │        value         │ userid │     modification_time      │
├────┼───────────┼──────────────────────┼────────┼────────────────────────────┤
│  1 │ email     │ email@example.com     │      1 │ 2016-10-05 12:53:32.936704 │
│  2 │ firstName │ thomas               │      1 │ 2016-10-05 12:53:32.936704 │
│  3 │ lastName  │ reggi                │      1 │ 2016-10-05 12:53:32.936704 │
│  4 │ email     │ email@example.com │      1 │ 2016-11-06 15:53:48.025775 │
└────┴───────────┴──────────────────────┴────────┴────────────────────────────┘
(4 rows)

WITH info_with_rank_for_user AS (
  SELECT userId,
         modification_time,
         value,
         key,
         RANK() OVER (PARTITION BY userId, key ORDER BY id DESC)
  FROM p
)
SELECT userId,
       json_object_agg(key, value),
       MAX(modification_time) AS last_settings_modification_time
FROM info_with_rank_for_user
WHERE rank = 1
GROUP BY userId
;
┌────────┬────────────────────────────────────────────────────────────────────────────────────┬─────────────────────────────────┐
│ userid │                                  json_object_agg                                   │ last_settings_modification_time │
├────────┼────────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────────┤
│      1 │ { "email" : "firstname.lastname@example.com", "firstName" : "thomas", "lastName" : "reggi" } │ 2016-11-06 15:53:48.025775      │
└────────┴────────────────────────────────────────────────────────────────────────────────────┴─────────────────────────────────┘
(1 row)

Comment: This is perfect! I have another column on my table, any idea how to get that in as well? So the columns sho up `userId, otherColumn, json_object_agg`?
Comment: @ThomasReggi: It sort of depends on what `otherColumn` represents. I added an example where it's also something you want to aggregate (here using `MAX()`). If that's not the case add it to your question, it's a bit hard to help otherwise :)


Cookie not being created (not a schoolboy error, I hope)

Question: I'm building a deeplink system into a very complex website which needs cookies and other context set-up before the deeplink will work (otherwise the page gets redirected back to the home page - not much use).
So I've added a module which provides the deeplink entry point through hook_menu:
<code>deeplink/<deeplink-parameters>
</code>
The callback builds the context, adds the cookies and then redirects to the actual required page.
So much for the theory. It doesn't work and I can't figure out why not.
In order to replicate a completely "virgin" request I've been using incognito windows, restarting the server and clearing caches before each test. In the test version I'm not redirecting to a new page, just outputting a line of text to display the page.
But even attempting to generate a simple test cookie with a long life (REQUEST_TIME + 999999) fails. No cookie created. The setcookie() function is returning TRUE (cookie created), which implies there is no problem.
The only cookies I can see are the ones created by JavaScript on the page itself, and a couple created by the Zeus wrapper (there are no Zeus rules to delete cookies on standard Drupal pages).
Just in case the cookie isn't visible on the first page I am also then moving to another page but still no cookie. Seems like it just isn't being created despite what the function says.
Additional data: There is also no PHP Session ID cookie.
Any thoughts as to why the cookies are not appearing?
EDIT: Looking at this further, I moved the setcookie() function into index.php (before the Drupal bootstrap), and it still wasn't creating the cookie.
So rather than issuing the deeplink command as above I simply went to the home page. And it's still not creating the cookie (as an anonymous user, with everything cleared before we start).
Only when I moved away and into the more complex part of the site did the test cookie appear. At the same time that the PHP Session ID cookie appeared. Is this a hidden dependency? No cookies can be created through PHP unless there's a PHP session ID?
Comment: It would help if you could show the code you are using to set the cookie ;)
Comment: setcookie('test', 'test', time() + 99999);

I spent all of yesterday working through a zillion different options and scenarios. Putting the setcookie() in various places in the code, including outside of a function so it got executed immediately.
Answer: The world has not turned upside-down.
It turned out that Zeus was indeed eating cookies of anonymous accesses, I had to go through every Zeus rule (we have a lot) until I found it.
Testing with Zeus bypassed demonstrated the cookies were created correctly.
Bloody Greek Gods, wandering in uninvited and eating all the cookies...


Xcode - Show "scene dock" - it is hidden

Question: at first, I'm not even sure how this thing is really called. On some websites it seems to be called "dock" or "scene dock", well this black bar below an view while editing the GUI in Xcode. Example Image
My problem is, that this bar appears in every tutorial and is used for different tasks (e.g. gestures), but in my Xcode-environment (4.3.2) it does not appear.
I looked all through the View-menu, I looked at the preferences, .... I just could not find any possibility, to show this "scene dock". Any Idea what I can do?
Answer:
The "scene dock" is only visible in a Storyboard, not in a single xib file. Do you have a storyboard?
If its hidden in your storyboard somehow, don't worry. You don't need it actually. The "scene dock" shows nothing else than the contents of the Document Outline (The left panel in your xib or storyboard where the First responder can be found). You can add, edit or connect all elements in the Document Outline.


How can $3^{-n}= 0$ be true for any $n$?

Question: There is an answer that has 7 up votes (so it is probably right) that says,

There are no segments of positive length left (this may be seen by,
  for instance, looking at the lengths of the segments at each
  iteration; after the $n$th iteration, the segments have length
  $3^{-n}$).

My understanding is that  for every $n \in \mathbb N : 3^{-n} \not= 0$
There may be some rule that allows us to say that $ 3^{-n} = 0$ under some condition, but it is not at all obvious to someone (like me) who has not studied math. After all topology is a very strange branch of mathematics.
Can someone explain the thinking behind this answer?
Comment: That statement applies in the limit as $n\to \infty$.  You are absolutely correct that $3^{-n}\neq0$ for any finite $n$
Comment: Suppose there would be an open interval $(a,b)$ inside the Cantor set. As $b-a>0$, for some value of $n$, $0<3^{-n} < b-a$. So already at that stage $(a,b)$ cannot be contained in one part of that stage, so some part of it will not be in $C$ at all.
Comment: @Henno Brandsma, Would it be correct to say that for every interval $(a,b) \in \mathbb C : a=b$ thus the length is zero?
Comment: If the length is positive it cannot be contained in the Cantor set
Answer: The Cantor set is the intersection of all its iterations.  What the author is trying to say is: the Cantor set contains no interval.  Why not?  Let $I$ be an interval of length $L>0$.  Then there is some $n$ such that $3^{-n}<L$.  Therefore, $I$ cannot be contained in the $n$-th iteration towards the Cantor set, and therefore cannot be contained in the Cantor set itself.
You are correct that $3^{-n}\neq 0$ for all $n$.
Answer: The point is that when constructing the Cantor set, at each step the segments have size $3^{-n}$, so at each step there are intervals of positive length. But the construction goes through all the natural numbers, so after we exhausted all the natural numbers, we end up with a set that contains no (nontrivial) intervals.
Answer: It's a limit process.  If you aren't familiar with the concept of limits, the Cantor set is not a good introduction to it, in my opinion.
It's true that $3^{-n} \ne 0$ for any $n \in \Bbb N$ (and really for any $n \in \Bbb R$).
However, the limit as $n \to +\infty$ of $3^{-n}$ is zero.  We denote this by writing $\displaystyle \lim_{n \to +\infty} 3^{-n} = 0$.
In general, $\displaystyle \lim_{n \to c} f(n)$ describes what $f(n)$ looks like or how $f(n)$ "behaves" as $n$ gets extremely close to $c$.  Or, in the case that $c$ is $+\infty$, it describes what $f(n)$ looks like or how $f(n)$ "behaves" as $n$ gets extremely huge, i.e., as we send $n$ off in the direction of $+\infty$ and never let it stop.
And in this case, we can see that as $n$ gets larger and larger, the value of $3^{-n}$ always remains positive but continues to decrease, getting smaller and smaller.  As $n$ gets infinitely larger, $3^{-n}$ gets infinitely closer to zero.  So its limit is zero, but $3^{-n}$ never takes on the value zero for any $n$.  But limits don't care about actual function values, only what the function looks like it's going to do.


escape special character (%)

Question: i have a command that displays the current name of the batch file that it's code is appended to.
echo %~n0%~x0

i want to send the command to a text file without changing its syntax.
for example;
echo %~n0%~x0 >> somefile.txt

i tried escaping the percentage sign with another percentage but it doesn't work.
echo %%~n0%%~x0

any ideas...?
Comment: As `echo %%~n0%%~x0` works in a batch file as expected (outputs `%~n0%~x0`)  you should show your _minimal_ batch file where it fails. Or at least what result you get
Answer: Try the caret instead of the %% like this ^% you might also need to escape the escape (I know weird and why .. but ..you might).
Using the CARET ^  worked for me.
echo ^%%~n0^%%~x0 >> somefile.txt

this code will produce the batch file name
echo ^%~n0^%~x0 >> somefile.txt
Comment: The caret works only directly in command prompt; in batch files you usually need to double the `%` sign...
Comment: You can use the caret, but it works the same way without the caret.
Comment: @aschipfl here is my code in the batch file , it works for me echo ^%%~n0^%%~x0 >> somefile.txt
Comment: @zask - if the answer helped you - please mark as answer.

escape special character (%)


valign attribute in style classes

Question: When we define a style class like the below in a CSS file, it is getting rendered in IE, but FF shows empty classes. vertical-align attribute works fine in both browsers. Why do we see this behaviour?
.text {
   valign: middle;
}

Comment: `valign` is not a valid CSS attribute. If a browser interprets it anyway, it's wrong
Answer: Because valign isn't valid CSS, while vertical-align is. And IE is known to ignore the rules. What more?
Comment: Thanks for your response, but why it is getting rendered in IE, does not do any validation?
Comment: Like I said, IE is known to ignore the rules. Why does it ignore the rules? Ask Microsoft.
Comment: My plan is to remove that style class from my css. Just wanted to make sure that it doesn't have any impact. Thanks a lot for you information. :)

valign attribute in style classes


Remove unused docker images

Question: Recently we found out that our Kubenetes nodes are using high disk space,after checking the nodes we found that the high disk space was due to docker images.
We know that if we run docker image prune -a command, it will remove all the unused images but it will also remove some unused images we need.
We checked if the image prune command supports removing only particular image and their tags(webapp:1, webapp:2 etc), it does not. Even though docker image ls command has this feature (docker image ls --filter reference=<image name>)
Is there a way to only prune particular images and their tags?
Answer: You can actually do this with a for loop. Until someone else tells you how to do it with Docker, you can use this;
for image in $(docker image ls --filter reference=<image_name> | awk '{if (NR>1) {print $3}}'); do docker image rm $image; done

This command simply lists the relevant containers and sends their image ids to the remove command. To test it you can change the last part like this;
for image in $(docker image ls --filter reference=<image_name> | awk '{if (NR>1) {print $3}}'); do echo $image; done

Remove unused docker images


Algoritmo O(n log n) para identificar intervalos contíguos dada uma lista de intervalos

Question: Tenho o seguinte problema: Dada uma lista com n intervalos contínuos, retorne uma lista de m intervalos contínuos tal que:

todo ponto da lista de entrada esteja contido em algum dos intervalos da segunda lista
todo ponto da lista de saída seja de pelo menos um elemento da lista de entrada
não haja interseção alguma entre os elementos da lista de saída

Exemplos de entrada e saída:
-- Entrada
1-3
2-6
7-10

-- Saída
1-6
7-10

-- Entrada
1-3
2-8
7-10

-- Saída
1-10

-- Entrada
1-3
4-7
8-10

-- Saída
1-3
4-7
8-10

A ordem da entrada e da saída são irrelevantes.
Esse problema é equivalente a descobrir quantos componentes conexos tem um grafo de intervalo, apresentado apenas os vértices/intervalos desse grafo, e qual o intervalo representado por cada componente conexo.
A estratégia que eu consegui pensar segue uma lógica quadrática. Segue um pseudo-código em Java:
List<Intervalo> entrada = ...;
ArrayList<Intervalo> saída = new ArrayList<>();

processaEntrada: for (Intervalo intervaloEntrada: entrada) {
  Integer firstNullPosition = null;
  Intervalo intervaloInserido = intervaloEntrada.copia();
  ArrayList<Integer> índicesInterseção = new ArrayList<>();
  for (int i = 0; i < saída.length(); i++) {
    Intervalo intervaloJulgamento = saída.get(i);
    if (intervaloJulgamento == null) {
      if (firstNullPosition == null) {
        firstNullPosition = i;
      }
      continue;
    }
    // como a lista  saida só contém elementos sem interseção entre si,
    // é impossível que seja possível que essa comparação e a de alguma
    // interseção com outro elemento de saida aconteçam
    if (intervaloJulgamento.contemIntervalo(intervaloInserido) {
      continue processaEntrada;
    }
    if (háInterseção(intervaloJulgamento, intervaloInserido)) {
      índicesInterseção.add(i);
    }
  }
  // se índicesInterseção for não vazio, então houve alguma interseção
  // caso contrário não houve interseção e, portanto, o intervaloInserido está
  // pronto para ser adicionado na lista
  if (índicesInterseção.isEmpty()) {
    if (firstNullPosition != null) {
      saída.set(firstNullPosition, intervaloInserido);
    } else {
      saída.add(intervaloInserido);
    }
  } else {
    Intervalo intervaloInterseções = intervaloInserido;
    int idxInserção = índicesInterseção.get(0);
    for (Integer i: índicesInterseção) {
      Intervalo interseptado = saída.get(i);
      saída.set(i, null);
      intervaloInterseções.começo = min(intervaloInterseções.começo, interseptado.começo);
      intervaloInterseções.fim = max(intervaloInterseções.fim, interseptado.fim);
    }
    saída.set(idxInserção, intervaloInterseções);
  }
}

return saída.stream().filter(Objects::nonNull).collect(Collectors.toList());

Comment: Olha Jeff, eu espero que você mesmo responda isso...
Comment: @JeffersonQuesado Dei a resposta em Java. Traduzir isso pra Lisp fica pra você (embora provavelmente vai ficar bem mais simples que no Java).
Comment: @VictorStafusa, o exemplo foi em Java, mas o problema é independente de linguagem (apesar de eu precisar resolver isso imperativamente, pontos extras para quem fizer em Lisp de modo funcional provando a complexidade). Obrigado por reparar nisso
Comment: @LINQ, na verdade eu já estou bolando uma espécie de árvore binária de busca para fazer essa função de merge mesmo =P Na verdade, como eu não domino balancear árvores, ainda vai ser uma espécie de "caso médio", mas daí seria trocar as operações de "merge" de nodos para alguma remoção com rotação da AVL ou da rubro-negra
Comment: Ah, @JeffersonQuesado Acabei de editar a minha resposta porque tinha um caso especial que dava errado (quando os intervalos apenas se encostavam, como por exemplo [1, 3] e [3, 10]). Editei ela para corrigir isso.
Comment: Sim, percebi isso quando entendi o `Evento`, mas a solução era trivial para não incomodar você com isso (no máximo fazer uma edição)
Answer: O código em Java que montei é o seguinte:
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

/**
 * @author Victor Williams Stafusa da Silva
 */
public class Main {
    public static void main(String[] args) {
        System.out.println(Intervalo.reduzir(new Intervalo(1, 3), new Intervalo(2, 6), new Intervalo(7, 10)));
        System.out.println(Intervalo.reduzir(new Intervalo(1, 3), new Intervalo(2, 8), new Intervalo(7, 10)));
        System.out.println(Intervalo.reduzir(new Intervalo(1, 3), new Intervalo(4, 7), new Intervalo(8, 10)));
        System.out.println(Intervalo.reduzir(new Intervalo(1, 3), new Intervalo(3, 10)));
    }
}

final class Intervalo {
    private final double a;
    private final double b;

    public Intervalo(double a, double b) {
        if (b < a) throw new IllegalArgumentException();
        this.a = a;
        this.b = b;
    }

    public double getA() {
       return a;
    }

    public double getB() {
       return b;
    }

    @Override
    public String toString() {
        return "[" + a + "<->" + b + "]";
    }

    public Evento getEventoA() {
        return new Evento(true, a);
    }

    public Evento getEventoB() {
        return new Evento(false, b);
    }

    @SafeVarargs
    public static List<Intervalo> reduzir(Intervalo... lista) {
        return reduzir(List.of(lista));
    }

    public static List<Intervalo> reduzir(List<Intervalo> lista) {
        List<Evento> eventos = new ArrayList<>(lista.size() * 2);
        for (Intervalo i : lista) {
            eventos.add(i.getEventoA());
            eventos.add(i.getEventoB());
        }
        Collections.sort(eventos);
        List<Intervalo> saida = new ArrayList<>(lista.size());
        int contagem = 0;
        double inicio = Double.NEGATIVE_INFINITY;
        for (Evento e : eventos) {
            if (e.isInicioIntervalo()) {
                if (contagem == 0) inicio = e.getPonto();
                contagem++;
            } else {
                contagem--;
                if (contagem == 0) saida.add(new Intervalo(inicio, e.getPonto()));
            }
        }
        return saida;
    }
}

final class Evento implements Comparable<Evento> {
    private final boolean inicioIntervalo;
    private final double ponto;

    public Evento(boolean inicioIntervalo, double ponto) {
        this.inicioIntervalo = inicioIntervalo;
        this.ponto = ponto;
    }

    public boolean isInicioIntervalo() {
        return inicioIntervalo;
    }

    public double getPonto() {
        return ponto;
    }

    @Override
    public int compareTo(Evento outro) {
        return this.ponto < outro.ponto ? -1
                : this.ponto > outro.ponto ? 1
                : this.inicioIntervalo == outro.inicioIntervalo ? 0
                : this.inicioIntervalo ? -1
                : 1;
    }
}

Vejamos qual é a complexidade do método estático reduzir:
        List<Evento> eventos = new ArrayList<>(lista.size() * 2);

Esse é O(n). Observe que dei um tamanho exato para a lista (2n) suficiente para que ela não precise ser redimensionada nunca. O método size() é O(1).
        for (Intervalo i : lista) {
            eventos.add(i.getEventoA());
            eventos.add(i.getEventoB());
        }

Os getters são O(1). O método add é O(1) se a lista não precisar ser redimensionada (e ela nunca precisará). O número de Eventos criados na lista é 2n.
        Collections.sort(eventos);

Ordenação é O(tn log n), onde t é a complexidade do método compareTo da classe Evento. Como esse método tem complexidade O(1), então isso se reduz a O(n log n). Esse é o ponto que faz o algoritmo ter tal complexidade.
        List<Intervalo> saida = new ArrayList<>(lista.size());

Novamente é O(n) e nunca precisará ser redimensionada, pois o pior caso é justamente quando nenhum dos intervalos se sobrepõe aos demais.
        int contagem = 0;
        double inicio = Double.NEGATIVE_INFINITY;

Isso daí é O(1).
        for (Evento e : eventos) {
            if (e.isInicioIntervalo()) {
                if (contagem == 0) inicio = e.getPonto();
                contagem++;
            } else {
                contagem--;
                if (contagem == 0) saida.add(new Intervalo(inicio, e.getPonto()));
            }
        }

Considerando que o número de eventos é 2n, então esse for também é O(n).
Os métodos isInicioIntervalo, getPonto, add (considerando que saida nunca precisará ser redimensionada) e o construtor de Intervalo também são todos O(1).
        return saida;

Isso é O(1).
Basicamente o algoritmo se reduz a desmanchar os intervalos e marcá-los como uma sequência ordenada de pontos, onde cada ponto correspondente ao início de um intervalo soma 1 em contagem e cada ponto equivalente ao fim subtrai 1 de contagem.
Quando contagem sai do zero, é porque estamos em algum intervalo que abriu.
Enquanto contagem for maior que zero, há intervalos abertos colidindo entre si. E o número de contagem diz exatamente quantos.
Quando contagem é reduzido para zero, significa que todos os intervalos que estavam abertos fecharam e podemos então adicionar um grande intervalo correspondendo a tudo isso, desde o ponto onde essa variável saiu do zero até o ponto em que voltou a ele.
A complexidade do algoritmo é dominada pela ordenação, que é a operação de maior complexidade (O(n log n)).
Uma vez que os elementos na lista eventos estão devidamente ordenados, isso garante que os elementos de saida também estarão corretos e devidamente ordenados.
Vale também dar uma atenção ao método compareTo:
    public int compareTo(Evento outro) {
        return this.ponto < outro.ponto ? -1
                : this.ponto > outro.ponto ? 1
                : this.inicioIntervalo == outro.inicioIntervalo ? 0
                : this.inicioIntervalo ? -1
                : 1;
    }

Ele é assim para garantir que os Eventos com menores números sempre antecederão os de maior número. E em caso de empate, que um evento de abertura de intervalo sempre tenha precedência sobre um de fechamento. Isso serve para garantir que intervalos que apenas estejam encostando (por exemplo, [1-3] e [3-10]) sejam considerados como uma colisão.
Caso seja desejado que eventos que estão apenas encostando não sejam considerados como uma colisão, basta trocar o ? -1 : 1; do final por ? 1 : -1;.
Considerando esses intervalos de entrada:
System.out.println(Intervalo.reduzir(new Intervalo(1, 3), new Intervalo(2, 6), new Intervalo(7, 10)));
System.out.println(Intervalo.reduzir(new Intervalo(1, 3), new Intervalo(2, 8), new Intervalo(7, 10)));
System.out.println(Intervalo.reduzir(new Intervalo(1, 3), new Intervalo(4, 7), new Intervalo(8, 10)));
System.out.println(Intervalo.reduzir(new Intervalo(1, 3), new Intervalo(3, 10)));

Eis a saída produzida:
[[1.0<->6.0], [7.0<->10.0]]
[[1.0<->10.0]]
[[1.0<->3.0], [4.0<->7.0], [8.0<->10.0]]
[[1.0<->10.0]]

Veja aqui funcionando no IDEONE.
Comment: Não tinha percebido que poderia definir os `Evento`s como entrada/saída e daí seria só ordenar. Estava lutando montando uma árvore e fazendo "reduções" nela.


(Basic High School Mathematics) Graphing the inverse square law

Question: I did an experiment measuring the intensity of light in relation to the distance away from a source. How would I graph the avg intensity over 1/distance squared?
Note that T1 = trial 1 etc..
It's supposed to look something like this showing exponential decay, with the values obviously being adjusted based on my results.

Any help would be much appreciated, regards.
Comment: No replies, did I violate a rule?
Answer: You will get a very different graph from the one you have drawn for intensity against distance if you plot intensity against $\cfrac 1{\text{distance}^2}$
What you seem to be trying to do is to use the numbers in your final column on the $x-$ axis, which will then run up to a maximum value of $\cfrac 1{25}=0.04$, so you might choose a scale which goes from $0$ to $0.05$.
The $y-$ axis should reflect the numbers in the fifth column.
It seems possible that you are doing an exercise to convert the kind of graph you have shown (which is equivalent to having the first column on the $x-$ axis) to a straight line.
Does this help any?


How to prove (a+b')(b+c')(c+a') = (a'+b)(b'+c)(c'+a)?

Question: Firstly I expanded both LHS and RHS as following:
LHS: (a+b')(b+c')(c+a') = ab + ac' + b'b + b'c' = ab + ac' + b'c' = abc + ac'c + b'c'c + a'ab + a'ac' + a'b'c' = abc + 0 + 0 + 0 + 0 + a'b'c' = abc + a'b'c'
RHS: (a'+b)(b'+c)(c'+a) = abc + a'b'c'. I get similar expression on RHS and LHS. This is not algebraic expression. To me it seems that both LHS and RHS are in product-of-sums form. How should this boolean equality be proved ?
Comment: _Yes_...${}{}{}$
Comment: You have already proved it by showing that both sides reduce to the same form.
Comment: A graphical method would be perhaps to generate a truth table or a Karnaugh map. I can do that, but I really want to see if boolean equality can be proved with basic axioms of boolean algebra alone.
Comment: Hi Parcly, I assumed that these are a algebraic expressions. If given expressions were in PoS form, can I still prove it with expansion ?
Comment: You've already proved it by expansion.
Comment: A PoS expression can be expanded to SoP expression. Is that statement correct ?
Answer: Here's a useful equivalence principle:
Consensus
$(P \lor Q) \land (\neg Q \lor R) \land (P \lor R) \Leftrightarrow (P \lor Q) \land (\neg Q \lor R)$
$(P \land Q) \lor (\neg Q \land R) \lor (P \land R) \Leftrightarrow (P \land Q) \lor (\neg Q \land R)$
Applied to your statement:
$$(a + b')(b+c')(c+a')= \text{ (Idempotence)}$$
$$(a + b')(b+c')(c+a')(a + b')(b+c')(c+a')= \text{(Consensus x 3)}$$
$$(a + b')(b+c')(a+c')(c+a')(a+b')(c + b')(b+c')(c+a')(b + a')= \text{(Idempotence x 3)}$$
$$(a + b')(b+c')(a+c')(c+a')(c + b')(b + a')= \text{(Commutation x 6)}$$
$$(b' + a)(c'+b)(c'+a)(a'+c)(b'+c)(a'+b)= \text{(Commutation)}$$
$$(a' + b)(b'+c)(a'+c)(c'+a)(c'+b)(b'+a)= \text{ (Idempotence x 3)}$$
$$(a' + b)(b'+c)(a'+c)(c'+a)(a' + b)(c'+b)(b'+c)(c'+a)(b'+a)= \text{ (Consensus x 3)}$$
$$(a' + b)(b'+c)(c'+a)(a' + b)(b'+c)(c'+a)= \text{ (Idempotence)}$$
$$(a' + b)(b'+c)(c'+a)$$
Comment: Thank you Bram28. Your method is appealing, and satisfying.
Comment: @Digvijay You're welcome! :)


Is a quadratic regression model valid if its y-intercept is not significant?

Question: I am testing my data with a quadratic regression model (y = ax^2 + bx +c).  The results of testing the model are:
P-value of a:< 2e-16
P-value of b:< 2e-16
P-value of c: 0.643
F(DF=138):55.58
P-value of model: < 2.2e-16
R^2: 0.4461

As the P-value of c is higher than 0.05, does that mean the quadratic model not valid? Or should I simply remove the c from the model? What does its nonsignificance imply?
Comment: This seems clear enough to me. I'm voting to leave open.
Comment: thanks @Glen_b. it's actually a species diversity vs environmental parameter analysis.
Comment: SInce that comment of mine seems to have evolved into a partial answer I'll copy it down and try to address some more of your question below
Answer: Unless you have a strong basis to remove the constant you should not generally do so (an example of a potentially reasonable reason to remove the constant is the prior consideration that at $x=0$ the response must be 0 - such as when modelling braking distance as a function of speed for cars under standardized conditions).
But even then, there may be good reasons why - within the range of the data - a model with an intercept would still be a reasonable choice. Outside a physical model like that, it's very like removing the intercept for a linear fit and it's a somewhat similar consideration to removing a non-significant main effect when an interaction term is present. In both cases the usual advice is something like "don't do it without a really good reason. If you have a really good reason then think about it very carefully, and then you still probably don't do it"
Some people advise never removing it, but I think that it's reasonable in some situations. (I wouldn't include mere non-significance among them.)
[There are many threads on site relating to both the intercept in a straight line fit and leaving out main effects when there's an interaction; it may be worth exploring via search.]
A non-significant intercept doesn't mean that the quadratic model is "not valid", it simply means that where the parabola crosses the $y$-axis (i.e. at $x=0$) the estimate of the average $y$-value is not distinguishable from 0.
from comments:

it's [...] a species diversity vs environmental parameter analysis

I assume then that you don't have a theoretical basis for thinking that the relationship must be quadratic (even though it might be a good description); in that case I'd say that there's really no reason to consider doing anything with the intercept term. It's unlikely to be worth worrying about at all (that it's not significant, I mean). I'd advise you to keep it in the model; and of itself it presents no reason to doubt the model.

Response to comment under answer:
If you have a response that cannot go negative you should generally consider models that have the same property; in those situations you should typically expect to find some particular characteristics (smaller variance when the mean gets small, for example) that would make an ordinary linear model unsuitable, and good model choice can respect the constraint on the mean as well as match such characteristics
Comment: I support this good advice. Note that for your "parameter" $x$ (statistical people really would prefer that you said _property_ or _variable_ to follow diversity $= ax^2 + bx$ it should be true that diversity is $0$ when $x = 0$, Note that a quadratic could predict negative diversity somewhere, so it sounds a dubious model without some extra device (e.g. logarithmic link in  a generalised linear model). (That is, I know of no diversity measure that is ever negative.)
Comment: Thanks. Yes, maybe some extra work is needed to add to the model. And yes, no diversity measure should be negative as far as I know.
Comment: @wwwwwwww If you have a response that cannot go negative you should generally consider models that have the same property; in those situations you should typically expect to find some particular characteristics (smaller variance when the mean gets small, for example) that would make an ordinary linear model unsuitable, and good model choice can respect the constraint on the mean as well as match such characteristics.


java local timeformat without year

Question: I like to format the local timeformat into a string without the year. At the moment I am able to show the local format containing the year:
java.text.DateFormat df = java.text.DateFormat.getDateInstance(java.text.DateFormat.SHORT);
String dateString = df.format(date);

therefore i receive an time string output like
12.03.2012
03/12/2012

for the different countries. Now i like to get a short form like
12.03.
03/12

how would i do this?
thanks for your help!
Answer: You can use SimpleDateFormat:
Date date = new Date();
java.text.SimpleDateFormat df = new java.text.SimpleDateFormat("MM/dd");
String dateString = df.format(date);

Output:
03/15

EDIT:
After researching locale formats further, and expanding on Peters answer, here's some code to demonstrator differences between toPattern() and toLocalizedPattern():
import java.text.*
import java.util.*

ArrayList<Locale> locales = new ArrayList<Locale>();
locales.add(Locale.US);
locales.add(Locale.UK);
locales.add(Locale.GERMANY);
locales.add(Locale.CHINA);

Date date = new Date();

for(Locale l : locales)
{
    SimpleDateFormat sdf = (SimpleDateFormat) SimpleDateFormat.getDateInstance(DateFormat.SHORT, l);
    String pattern = sdf.toPattern();
    String localizedPattern = sdf.toLocalizedPattern()
    println "country: " + l.getDisplayName();
    println "pattern: " + pattern;
    println "localizedPattern: " + localizedPattern;

    try {
        SimpleDateFormat temp = new SimpleDateFormat(localizedPattern, l);
        println "localized pattern re-parsed successfully"
    } catch (IllegalArgumentException e) {
        println "localized pattern re-parsed unsuccessfully: " + e.getMessage();
    }
    SimpleDateFormat df = new SimpleDateFormat(pattern, l);
    String dateString = df.format(date);
    println "resulting date: " + dateString
    String yearlessPattern = pattern.replaceAll("\\W?[Yy]+\\W?", "");
    println "yearlessPattern = " + yearlessPattern;
    SimpleDateFormat yearlessSDF = new SimpleDateFormat(yearlessPattern, l);
    println "resulting date without year: " + yearlessSDF.format(date) + "\n";
}

Produces following output:
country: English (United States)
pattern: M/d/yy
localizedPattern: M/d/yy
localized pattern re-parsed successfully
resulting date: 3/15/12
yearlessPattern = M/d
resulting date without year: 3/15

country: English (United Kingdom)
pattern: dd/MM/yy
localizedPattern: dd/MM/yy
localized pattern re-parsed successfully
resulting date: 15/03/12
yearlessPattern = dd/MM
resulting date without year: 15/03

country: German (Germany)
pattern: dd.MM.yy
localizedPattern: tt.MM.uu
localized pattern re-parsed unsuccessfully: Illegal pattern character 't'
resulting date: 15.03.12
yearlessPattern = dd.MM
resulting date without year: 15.03

country: Chinese (China)
pattern: yy-M-d
localizedPattern: aa-n-j
localized pattern re-parsed unsuccessfully: Illegal pattern character 'n'
resulting date: 12-3-15
yearlessPattern = M-d
resulting date without year: 3-15

So in conclusion, to display a localized date without a year:
String yearlessPattern = DateFormat.getDateInstance(DateFormat.SHORT).toPattern().replaceAll("\\W?[Yy]+\\W?", "");
Answer: I needed to convert date to String removing year. String should keep locale settings. Date format is of type DateFormat.LONG, not DateFormat.SHORT. E.g., full string is September 18, 2012, not 09/18/12.
My solution (based on Alex post):
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Locale;
import java.util.regex.Pattern;

public class Test {

private static void testLocaleDates() {
    String fmtString = "%-20s %-25s %-25s %-20s %-20s";
    System.out.println(String.format(fmtString, "Locale", "Complete date", "Pattern", "Yearless date", "Yearless pattern"));
    Pattern regExpPattern = Pattern.compile("Sweden|Spain|Russia|Ukraine|States|France|German|Japan|China", Pattern.CASE_INSENSITIVE);

    for (Locale locale : Locale.getAvailableLocales()) {
        boolean isPrint = regExpPattern.matcher(locale.getDisplayCountry()).find();
        if (!isPrint)
            continue;
        Date date = new Date();
        String dateTxt = DateFormat.getDateInstance(DateFormat.LONG, locale).format(date);
        SimpleDateFormat sdf = (SimpleDateFormat) SimpleDateFormat.getDateInstance(DateFormat.LONG, locale);
        String pattern = sdf.toPattern();
        // Checking 'de' we omit problems with Spain locale
        String regExpPatternTxt = pattern.contains("de") ? "[^Mm]*[Yy]+[^Mm]*" : "[^DdMm]*[Yy]+[^DdMm]*";
        String yearlessPattern = pattern.replaceAll(regExpPatternTxt, "");
        SimpleDateFormat yearlessSDF = new SimpleDateFormat(yearlessPattern, locale);

        System.out.println(String.format(fmtString, locale.getDisplayCountry(), dateTxt, pattern, yearlessSDF.format(date), yearlessPattern));
    }
}

public static void main(String[] args) {
    testLocaleDates();
}
}

Program output:
Locale            Complete date          Pattern                Yearless date     Yearless pattern

Japan             2012/09/18             yyyy/MM/dd             09/18             MM/dd

Japan             H24.09.18              Gy.MM.dd               09.18             MM.dd

United States     September 18, 2012     MMMM d, yyyy           September 18      MMMM d

Spain             18 de septiembre de 2012d' de 'MMMM' de 'yyyy  18 de septiembre  d' de 'MMMM

United States     18 de septiembre de 2012d' de 'MMMM' de 'yyyy  18 de septiembre  d' de 'MMMM

Ukraine           18 вересня 2012        d MMMM yyyy            18 вересня        d MMMM

Spain             18 / setembre / 2012   d' / 'MMMM' / 'yyyy    18 / setembre     d' / 'MMMM

Russia            18 Сентябрь 2012 г.    d MMMM yyyy 'г.'       18 Сентябрь       d MMMM

China             2012年9月18日             yyyy'年'M'月'd'日'        9月18日             M'月'd'日'

France            18 septembre 2012      d MMMM yyyy            18 septembre      d MMMM

Germany           18. September 2012     d. MMMM yyyy           18. September     d. MMMM

Sweden            den 18 september 2012  'den 'd MMMM yyyy      den 18 september  'den 'd MMMM
Comment: +1. Could you pls let me know why `pattern.contains("de") ? "[^Mm]*[Yy]+[^Mm]*" : "[^DdMm]*[Yy]+[^DdMm]` ?
Comment: @Trying Let's take 10/14/2015. US locale patttern is "MMMM d, yyyy". Without this checking we'll get "October" for US locale but we need "October 14". Because "Dd" is not listed in "excluded" of RegExp "[^DdMm]" it'll be removed. But if we will use it for Spanish locale "d' de 'MMMM' de 'yyyy", we'll get an exception!
Comment: Great but it does a bad replacement for Vietnamese resulting in yearlessPattern having an unterminated quote: dd 'tháng' M 'năm
Comment: Thanks for `DateFormat.getDateInstance(DateFormat.LONG, Locale.getDefault()).format(Date())`.
Answer: There is no predefined format to achieve this. Here is a workaround: Format the date with java.text.DateFormat.SHORT and with a custom formatter with the format yyyy. Now search the result of the former for the latter.
If the year is near the beginning, remove the next non-digit after the year, otherwise strip non-digits before the year.
But that's not perfect either, because it gives you "12.3" instead of "12.3." for German.
If you really need to get it right, look into the Java source code, specifically the package sun.text.resources in rt.jar and print all the locale dates (you can simply create a DateFormat with type SHORT for all locales).
That should give you all the data you need to create your own resource bundle with day/month format strings for all languages.
Answer: Other authors have correctly recognized that the Java platform does not manage predefined localized format patterns for a month-day-combination without year so complex workarounds like using data analysis and regexp-patterns were suggested.
I have embedded such format patterns into my library Time4J based on the newest CLDR-data. Example:
AnnualDate now = SystemClock.inLocalView().now(AnnualDate.chronology());
System.out.println("xml-schema: " + now); // --09-23

ChronoFormatter<AnnualDate> fEnglishUS =
    ChronoFormatter.ofStyle(DisplayMode.SHORT, Locale.US, AnnualDate.chronology());
ChronoFormatter<AnnualDate> fFrench =
    ChronoFormatter.ofStyle(DisplayMode.SHORT, Locale.FRENCH, AnnualDate.chronology());
ChronoFormatter<AnnualDate> fGerman =
    ChronoFormatter.ofStyle(DisplayMode.SHORT, Locale.GERMAN, AnnualDate.chronology());
System.out.println("us: " + fEnglishUS.format(now)); // 9/23
System.out.println("french:  " + fFrench.format(now)); // 23/09
System.out.println("german:  " + fGerman.format(now)); // 23.9.

Good to know: ICU4J does have this capability, too, but with a far more complex API.
Answer: You can extract the pattern and remove the year.
SimpleDateFormat df = (SimpleDateFormat)
                      DateFormat.getDateInstance(DateFormat.SHORT);
String pattern = df.toLocalizedPattern().replaceAll(".?[Yy].?", "");
System.out.println(pattern);
SimpleDateFormat mdf = new SimpleDateFormat(pattern);

prints
M/d

on my system.
Comment: @android_dev what is used for the year in Bulgarian?
Comment: Bulgarin pattern is " d.MM.yy 'г' ". The last part of the pattern is also related to year and should be removed. What we are currently planning to use is " ([^dM]*)y[^dM]* "
https://stackoverflow.com/a/46428741/1635488
Comment: As you're well experienced in Java, maybe you know a clean way to do this in Java 8? Without this kind of hacks.
Answer: java.time.MonthDay
Java offers a class to represent such a value, a month and a day-of-month without any year: java.time.MonthDay
LocalDate ld = LocalDate.parse( "2012-12-03" );  // December 3rd.
MonthDay md = MonthDay.from( ld );

ISO 8601
If you call toString, you get a string generated in the format following the ISO 8601 standard, --MM-DD. That is the Extended format, and the standard also allows for the Basic format minimizing the use of separators, --MMDD. Reference: Section 188.8.131.52 Truncated representations d) A specific day of a month in the implied year of the ISO 8601:2000 standard.
String output = md.toString();

--12-03

Custom formats
I suggest using the standard formats whenever possible. But if you must use other formats, you can specify with the DateTimeFormatter class.
DateTimeFormatter f = DateTimeFormatter.ofPattern( "MM/dd" );
String output = md.format( f );

12/03

The java.time classes provide for automatic localization of date-only values and date-time values. Unfortunately, no such support for month-day values. So you will have to define your own custom format explicitly for your desired localization. For alternatives, see the Answer by Meno Hochschild.
Comment: The format "--MM-DD" is **NOT** ISO-8601 but an invention done by XML-Schema. See also the relevant sections "188.8.131.52 Complete representations" and "18.104.22.168 Representations with reduced accuracy" in the original ISO-8601-paper. And else I must say that this post does not answer the question how to find a localized pattern without explicitly setting it.
Comment: [This final draft of ISO 8601, dated 2000-10-15, Reference number ISO/FDIS 8601:2000(E)](http://xml.coverpages.org/ISO-FDIS-8601.pdf) does indeed define a format `--MM-DD` (Extended format) and `--MMDD` (Basic format) in section *188.8.131.52 Truncated representations* under part *d) A specific day of a month in the implied year*.
Comment: Thank you very much for showing this link. However, it is an outdated ISO-paper. Probably XML-schema has taken this as base for its month-day-format, and later, the last ISO-version has now cancelled this format resp. does not mention it any longer.
Answer: As you probably realize, short formats depends not only on language but also on country. Unfortunately, there are no such patterns built-in in Java. Instead, you might need to use the ones defined by CLDR - search for "pattern" or "Md" (month / day pattern).
However, I must warn you that these patterns might be buggy. Instead you might want to move them to resource file and let the translators deal with them (appropriate comment on what it is and how to deal with it would be required.)
Answer: A note on using DateFormat.getDateInstance(DateFormat.LONG) or DateFormat.getDateInstance(DateFormat.FULL) on devices set to Spanish locale.
The format that comes out with text literals from simpleDateFormat.toLocalizedPattern().
d 'de' MMMM 'de' y

In the case I'm looking at.
(e.g. 15th January 2021 is 15 de enero de 2021 in Spanish)
Removing the y and any non word char would lead to java.lang.IllegalArgumentException: Unterminated quote exceptions (as the last ' on the last 'de' would be removed as a non word character).
So as a least worst solution, in our non-year formater, we stripped out any literals if the patern contains single quote char '.  (Using reg ex  pattern.replaceAll("'\\w+'", "")).
I have tested this with a few other locals and it seems to work.  (de, ja, pt, en)

java local timeformat without year


Uniformly distributed random variable on unit circle.

Question: Let the random vector $(x,y)$ be uniformly distributed within the unit circle $B = \{ (x,y) : x^2 + y^2 \leq 1 \}$. Then the probability density function is
$$ f(x,y) = \begin{cases} \frac{1}{\pi} \quad &(x,y) \in B \\ 0 \quad &\text{otherwise} \end{cases}$$
Now, the marginal distributions are
$$\tilde{f}(x) = \int_{\mathbb{R}} f(x,y) \, dy = \frac{2}{\pi} \sqrt{1-x^2}$$
and similar for $\tilde{f}(y)$.
Question is:
Let $y = y_0 > 0$ be fixed. What is the probability distribution function for the random variables $x^4$, $|x|$ and $-x$?
Any ideas?
Comment: It's not clear what you mean by "let $y$ be fixed". Do you mean the conditional probability?
Comment: Good question. The question is posed as above, but I guess the conditional probability is not what is meant here.  I think we are looking at the problem from the moment $y$ was fixed and now try to find out what the probbility distribution for the variables $x$, $x^4$, etc. are.
Answer: When they say:"Let $y=y_0>0$ be fixed" it is evident to me that they are interested in the conditional distribution as fixing y the support of the marginal $X$ changes.

The problem is trivial because the conditional distribution $X|Y=y$ is uniform.
$$f_{X|Y=y_0}(t)=\frac{1}{2\sqrt{1-y_0^2}}\mathbb{1}_{\Big[-\sqrt{1-y_0^2};\sqrt{1-y_0^2}\Big]}(t)$$
Clarified this, I think you will have no problems to continue by your own
Example 1:
In the case $Z=|X|y_0|$ the distribution function is
$$F_Z(z)=F_{X|Y}(z)-F_{X|Y}(-z)=\frac{z}{\sqrt{1-y_0^2}}$$
...better
$$F_Z(z)=\frac{z}{\sqrt{1-y_0^2}}\mathbb{1}_{\Big[0;\sqrt{1-y_0^2}\Big]}(z)+\mathbb{1}_{\Big[\sqrt{1-y_0^2};+\infty\Big)}(z)$$
(again an Uniform Distribution)
Example 2
$Z=X^4$
$$F_Z(z)=\mathbb{P}[X^4\leq z]=\mathbb{P}\Big[- z^{\frac{1}{4}}\leq X\leq z^{\frac{1}{4}}\Big]=F_X\Big(z^{\frac{1}{4}}\Big)-F_X\Big(-z^{\frac{1}{4}}\Big)$$
Comment: In this case I have to calculate:

$$ f(x|y=y_0) = \frac{f(x,y)}{\tilde{f}(y)}$$,

which is

$$f(x|y=y_0) = \frac{1}{2\sqrt{1-y_0^2}}$$.

But this term is $x$-independent. Wouldn't that mean, that the random variables $x^4$, $|x|$ and $-x$ have a density function of zero everywhere?
Comment: @Octavius : in the meantime you were writing the comment I edited my answer with more details. The fact that in your conditional density there is no $X$ is better for you....it's a Uniform density in the support that I showed. The X will appear as soon as you calculate the CDF
Comment: I see, so my CDF is

$$F(x) = \frac{x}{2\sqrt{1-y_0^2}} + \frac{1}{2} \quad .$$

But how do I get the CDF for the random variables $x^4$, $|x|$, $-x$? The "transformation law" for the densities would be

$$g(\tilde{x}) = \Big| \frac{\mathrm{d}h^{-1}}{\mathrm{d}\tilde{x}} \Big| \cdot f\big(h^{-1}(\tilde{x})\big)$$

where $f(x)$ is the density of the variable $X$, and $g(\tilde{x})$ is the density of the variable $\tilde{X} = h(X)$. With constant density as written above I would have density zero for the other variables.
Comment: @Octavius : ahah ... I edited the answer again with more and more details...your formula is correct but with non monotonic transformation it must be updated. in general you can pass via CDF...it is the general method. I wrote down the case of $|X|$. Reading the text you posted, they ask you the distribution function, thus my method is faster.
Comment: Understood. Thank you very much.