Rescaling exponentially distributed variables before clustering?

Question: I want to cluster data that contains binary variables, exponentially distributed (power law) variables, and normally distributed variables. I'm considering preprocessing the data in the following way and wondering whether it's reasonable.
1) shift the binary variables so that they have mean zero. no rescaling.
2) standardize the normally distributed variables, but divide by twice the standard deviation rather than by once. this is based on Gelman, A. "Scaling regression inputs by dividing by two standard deviations. Statistics in Medicine 2008 27:2865-2837. [for clarification: their point is that binary variables can be interpreted directly as indicators and can be left unscaled. But in order for numerical inputs to be interpreted in the same way, they should be divided by twice the standard deviation -- that way, both numerical and binary variables have similar standard deviations (if the binary indicators are not strongly skewed; if they are, then there is not much improvement, but also no harm).]
3) take log(x+1) [or sqrt(x)] for any exponentially distributed variable x, and standardize in the above way.
Mainly, I'm worried about (3). I'm told that it's standard practice to simply standardize exponentially distributed variables. Is it standard practice because it's correct or because it's a good enough approximation? Also, would it be better to apply k-medoid clustering rather than k-means?
Answer: 1.) Binary variables are a problem for many many algorithms. While mapping them to two numerical values $x_0,x_1\in\mathbb{R}$ works, the results are often "damaged" by not having any value inbetween; similar to the effects resulting from discrete values such as integers.
2.) Standardization is a best practise when dealing with different scales. Divding each attribute by two standard deviations instead of one standard deviation will not make any fundamental difference. It will just half all your distances.
3.) $\log(x+1)$ often works, but so does $\sqrt{x}$. Without theoretical support, either is just a guess to make things work. Standardization is still reasonable, as discussed above.
The key question you should be asking is how do I reasonably measure similarity? Don't start with how do I make k-means work on this data set. First find out what the problem is that you are trying to solve, then find the appropriate tools to do so. Don't let the tool define your problem.
Comment: Thanks for your response! I think I'm asking several questions here. On the reference mentioned above, their point is that binary variables can be interpreted directly as indicators and can be left unscaled. But in order for numerical inputs to be interpreted in the same way, they should be divided by twice the standard deviation -- that way, both numerical and binary variables have similar standard deviations (if the binary indicators are not strongly skewed; if they are, then there is not much improvement, but also no harm).
Comment: As to not letting the tool define my problem, I guess, indirectly, I'm asking what is a reasonable measure of similarity? I'll clarify above, but for variables with power law distributions...let's take for example number of days spent on x by various people: I care more about the difference between people who spent 5 and 10 days, than about those who spent 105 and 110 days. This is why log or square root seem more reasonable. But then again, I can avoid this issue with medians.
Comment: They can be interpreted as indicators, but does this imply they should have *weight* 1? Are they are important as a difference of 1 in your other attributes?
Similarity is use case dependant. 1 yard is not 1 gallon; log scale or not. You have to find a domain-specific tradeoff.
Comment: "Are they are important as a difference of 1 in your other attributes?" - that's why we standardize at all, but I see your point on the arbitrariness, even in the case of a difference of 1 in one binary variable vs another. There's some model-driven vs data-driven issue here as well. Very deep. Thanks for this
Comment: Standardization is just a heuristic, and it may work well, or be completely useless. If you have 100 binary attributes, and they are random, then giving them each a weight of 1 (or 0.5, as the standard deviation is up to 0.5) will overshadow any signal you have in the other attributes. The best strategy is to learn which attributes are important, and to which degree. In particular when they are binary, discrete, or nominal.


Exclusion-inclusion principle- task.

Question: How many such numbers in 5-digit decimal expansion such that:
(1) 3rd digit is 7 and digit 5 there is neither time or
(2) there is no number more than once.
My solution:
Let a set $A$ is a set of numbers satisfying (1),
Let a set $B$ is a set of numbers satisfying (2)
Let's calculate $$| A | = 8 \cdot 9 \cdot 9 \cdot 9$$
And now:
$$| B | = 9 \cdot 8 \cdot 7 \cdot 6 \cdot 5$$
Now, if these numbers in $A$ that satisfy $B$, de facto:
$$| A \cap B | = 8 \cdot 8 \cdot 7 \cdot 6$$
Thus, in principle, inclusions and exclusions:
$$| A \cup B | = 9 \cdot 8^3 + 9 \cdot 8 \cdot 7 \cdot 6 \cdot 5 -8 \cdot 8 \cdot 7 \cdot 6 = 17040$$
OK?
Comment: Can the first digit be $0$?
Comment: How did $8\cdot9\cdot9\cdot9$ turn into $9\cdot8^3$?
Comment: In $A\cap B$ there are $7$ choices for the first digit, since it can't be 0, 5, or 7. Then there are $7$ choices for the second digit, since it can't be 5 or 7 or repeat the first digit.
Comment: I think you got it now.
Comment: Please explain how you got $|B|=9\cdot8\cdot7\cdot6\cdot5$ instead of $9\cdot9\cdot8\cdot7\cdot6$.
Comment: The number must be 5-digit so 0 can't be the first.
In case B you right, the: $|B| = 9\cdot9\cdot 8 \cdot 7\cdot 6 $

"How did $ 8\cdot 9\cdot 9\cdot 9 $ turn into $ 9 \cdot 8^3 $"

I made mistake, to correct is $9^3 \cdot 8 $

And now is ok?
Comment: yes, you also alright. And now? :)
$ | A \cap B  | = 7 \cdot 7 \cdot 6 \cdot  5 $
Answer: Community wiki answer so the question can be marked as answered:
As noted in the comments, there are three errors in the calculation.

For $|B|$, the choice of the non-zero first digit still leaves $9$ digits to choose from, so $|B|=9\cdot9\cdot8\cdot7\cdot6$.
For $|A\cap B|$ we have to avoid both $5$ and $7$, and again $0$ in the first digit, which leaves $7\cdot7\cdot6\cdot5$ choices.
In the last equation you wrote $9\cdot8^3$ instead of $8\cdot9^3$.

So the correct total is
$$
|A\cup B|=8\cdot9^3+9\cdot9\cdot8\cdot7\cdot6-7\cdot7\cdot6\cdot5=31578\;.
$$

Exclusion-inclusion principle- task.


Calcular a soma e média de uma sequência numérica

Question: Na sequência 3,5,7,9,11,13,15. Descubra o padrão da sequência e:

a) Faz um programa para mostrar 12 valores de essa sequência iniciado por 3.
b) Mostre a soma de 'TODOS' valores (12 valores).
c) Mostre o valor médio de 'TODOS' valores (12 valores).

Comecei assim, dai me enrolei. Me ajudem.
x = 3
while x < 27:
   print(x)
   x = x + 2

Comment: Já conseguiu a letra (a), mas teve dificuldades nas outras? Se você precisa da soma de todos os valores, não bastaria você ir somando cada valor de `x`? E a média requerida em (c) é a divisão entre a soma e a quantidade de elementos.
Comment: Sim, não sei como fazer essa soma, é que sou novo em programação e to meio perdido.
Answer:
Nota: é importante salientar que existem infinitas séries numéricas que satisfazem a sequência {3,5,7,9,11,13,15,...} (ver final da resposta), então solicitar para o usuário identificar o padrão não faz sentido algum e, principalmente, se qualquer uma das séries fosse implementada, seria uma solução válida. A solução a_n = 2*n + 1 só é, possivelmente, a mais trivial delas e, provavelmente por isso, será a esperada pelo professor.

Você já fez a letra (a), mas há como melhorar. Da forma que fez, você precisa inicialmente saber que para exibir os 12 elementos solicitados a condição será x < 27, mas e se for solicitado 20, 50 ou até nem saber quantos serão? Nesses casos sua solução não funcionaria. O mais simples é você criar um contador que controle quantos números serão exibidos:
quantidade = 12
exibidos = 0
numero = 3

while exibidos < quantidade:
    print(numero)
    numero += 2
    exibidos += 1

Assim, serão mostrados os 12 números da sequência. Se alterar quantidade para 50, mostrarão os 50. Já, para a letra (b), você precisa acumular a soma em outra variável:
quantidade = 12
exibidos = 0
numero = 3
soma = 0

while exibidos < quantidade:
    print(numero)
    soma += numero
    numero += 2
    exibidos += 1

print('Soma = ', soma)

E, finalmente, a média será a divisão entre a soma e a quantidade de números:
media = soma / quantidade

Ficando:
quantidade = 12
exibidos = 0
numero = 3
soma = 0

while exibidos < quantidade:
    print(numero)
    soma += numero
    numero += 2
    exibidos += 1

print('Soma = ', soma)

media = soma / quantidade

print('Média = ', media)

Colaboração do Jefferson Quesado
Como estamos lidando com sequências numéricas, a continuidade da função geradora não é um requisito, assim, por exemplo, a sequência gerada por:

Satisfaz a sequência dada no enunciado, mas o termo seguinte seria 21 (e não 17). A sequência seria: 3, 5, 7, 9, 11, 13, 15, 21, 23, 25, ...
Colaboração do Bacco
Outra bem mais simples de entender é a sequência dada por:

Isto é, calcular o resto da divisão de 2x+1 por 17, assim a sequência gerada seria: 3, 5, 7, 9, 11, 13, 15, 0, 2, 4, 6, ..., que também satisfaz a sequência dada no enunciado, mas o termo seguinte não é o 17.
Comment: Mano obrigadão, era isso mesmo, vou dar uma estudada no código...
Answer: Sequência que deverá ser mostrada: 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25.
No lugar de while x < 27 coloque x <= 25 e criar também uma variável auxiliar para ir guardando a soma dos valores.
Exemplo: y = 0 (inicie-o com 0 para não sobrar sujeira das somas e dar valores errados).
Não sei como se faz em Phyton, fiz um exemplo usando a lógica de Java.
x = 3;
y = 0;

   while x < 27:
       print(x)
         y = x + 2;
         x++;

Comment: E qual a diferença entre a condição `x <= 25` e `x < 27`, sendo que `x` será ímpar?
Comment: Fui eu que errei, era x <= 25 mesmo ele tem que mostrar esses valores 3
5
7
9
11
13
15
17
19
21
23
25 Soma de todos esses números é 168
Comment: Ô, desculpa pelo erro! Não havia reparado nisso. Acredito que utilizando `x <= 25` pode ser de melhor entendimento (pelo menos pra mim, é. Fazem dois anos que iniciei com programação, e, quanto mais claro e óbvio cada lógica/variável que eu uso, o entendimento ao meu ponto de vista é melhor.). Você testou utilizando uma variável auxiliar?
Comment: Tente fazer uma variável auxiliar para realizar a soma dos valores e depois é só dividir por 12 (que é o tanto de elementos) para fazer sua média.
Comment: Maria Clara você poderia mostrar como é?
Comment: Eu não sei programar em Phyton. Mas vou usar a mesma lógica com JAVA, pode ser? Seria mais ou menos: `y = x + 2` aí, enquanto o `x <=25` ele vai somando (quando utilizado *x++*, ele vai acumulando até chegar o resultado que foi definido por você, que nesse caso, seria `x <= 25`). Depois que ele somar todos os valores, você printa o **Y**. Depois de printar, você faz aquele cálculo matemático: y / 2 (cálculo da média) e printa **Y** novamente.
Answer: Segue a solução:
# Geracao da sequencia contendo 12 valores
seq = list(range(3, 27, 2))

# Extrai valores da sequencia iniciados com '3'
a = [ n for n in seq if str(n)[0] == '3' ]

# Calcula soma dos valores da sequencia
b = sum(seq)

# Calcula a media dos valores da sequencia
c = sum(seq) / float(len(seq))

print(seq)
print(a)
print(b)
print(c)

Saída:
[3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25]
[3]
168
14.0

Comment: Usando `range`, não seria mais simples `list(range(3, 27, 2))`? E poderia explicar o que seria o objeto `a`?
Comment: `a`, `b` e `c` são as respostas dos respectivos itens da pergunta. `a` trata-se dos `valores da sequência iniciado por 3`.
Comment: Ah, sim, mas acredito que `seq` já seria a resposta da `a` xD
Comment: @AndersonCarlosWoss:  Ou será que *mostrar 12 valores de essa sequência iniciado por 3* não seria algo como: `print([ int('3' + str(n)) for n in seq ])` ?!?!


MEF and MVVM - way to add conventions?

Question: Is there a way to hook into the internals of CompositionContainer?
For example, let's say any time an import is called on a export ending with the word "View" I might want to inject an export ending with "ViewModel" into its DataContext.
Is there a way to do that?  I'm trying to avoid code like this in every View:
<code>    [Export("MainView")]
    public partial class MainView : Window
    {
        [ImportingConstructor]
        public MainView([Import("MainViewModel")] dynamic viewModel)
        {
            InitializeComponent();

            DataContext = viewModel;
        }
    }
</code>
Would be nice to simplify to just this, and use conventions:
<code>        [Export("MainView")]
        public partial class MainView : Window
        {
            public MainView()
            {
                InitializeComponent();
            }
        }
</code>
Comment: So far best I've come up with is to use an IViewLocator service that does this work, then always use it for importing views.
Answer: There are some folks working on this, including the Mef team apparently :)
Here is a blog post you might find interesting:
http://thecodejunkie.com/2010/03/bringing-convention-based-registration.html
And here is another slightly less detailed one:
http://blogs.msdn.com/b/hammett/archive/2011/03/08/mef-s-convention-model.aspx
Answer: I Guess you are looking for a library  or framework which can reduce your effort.
Please have a look at library WPF Application framework. I am using it very well.

MEF and MVVM - way to add conventions?


Generate download file link in PHP

Question:
Possible Duplicate:
open download dialog with php

I have a link in my page say, (Test.pdf).
When I click on that link, download dialogue box should open to download that file.
Can anyone help me in implementing this in PHP?
thanks
Comment: Here you go: http://stackoverflow.com/questions/985083/open-download-dialog-with-php http://stackoverflow.com/questions/732063/how-to-show-save-as-dialog-box-using-php-for-text-files
Comment: This isn't a dup of those questions. They want to generate a download dialogue, this one wants the link to force a download rather than opening the file in the browser. In which can the html5 download attribute can do that ``
Answer: $filename = 'Test.pdf'; // of course find the exact filename....
header('Pragma: public');
header('Expires: 0');
header('Cache-Control: must-revalidate, post-check=0, pre-check=0');
header('Cache-Control: private', false); // required for certain browsers
header('Content-Type: application/pdf');

header('Content-Disposition: attachment; filename="'. basename($filename) . '";');
header('Content-Transfer-Encoding: binary');
header('Content-Length: ' . filesize($filename));

readfile($filename);

exit;

Name the above file as download.php
HTML:
Test.pdf

That should do it.
Comment: @Caperneoignis: Might be too late, but yes it is possible. just make sure the characters you pass on the $_GET variable are URL friendly or do the encode-decode process.
Comment: the downloaded pdf file doesnt open it says the file is not decoded correctly?
Comment: @JacobRelkin Quick question. Can you pass in arguments? Like download.php?filecode=1234 and use those to fill in the header functions? or do the headers have to be called before any other call. Example: ````$filename = $_get['filecode']````
Answer: test.pdf

Comment: Simplex sigillum veri. IMO this should be the accepted answer :-)
Answer: In the case of a PDF file, most browsers are going to look for the helper (acrobat) to load it in your browser by default. You are trying to get around this default behavior is my guess.
The easiest way to do this (assuming you're on *nix box with apache) is to make an .htaccess file in the directory you want to have this result and add the line:
AddType application/octet-stream .pdf
This will cause any file with the extention .pdf to download by default. You can even have some .pdf files on the page load in the browser while others download by using the FilesMatch directive ( http://www.askapache.com/htaccess/using-filesmatch-and-files-in-htaccess.html ).
I realize your original question said "how do I do it with PHP" but I thought I'd post in case you were looking for a simpler, more elegant solution. Do keep in mind any directives you put in an .htaccess file will also affect any sub-directories below it.

Generate download file link in PHP


Qual a diferença entre ordered, unordered e sorted?

Question: Esses termos são usados em algumas estruturas de dados para definir como os elementos são inseridos e mantidos, o que quer dizer cada uma? Sorted e ordered quer dizer a mesma coisa? Unordered quer dizer que é aleatório?
Answer: Ordered
Ou ordenado, significa que a coleção de dados da estrutura está em alguma ordem. Em geral isso significa que a ordem de entrada é usada. Mas não é definido que seja exatamente assim, apenas que tem uma ordem. O termo é usado quando independentemente do valor adicionado no elemento, ele é colocado em alguma ordem.
Na maioria dos casos os mesmos dados entrados em ordem diferente produzirá um resultado diferente.
O array é o melhor exemplo deste tipo de estrutura. Mas a maioria das estruturas são ordenadas de alguma forma.
Unordered
Ou não ordenado, significa que os elementos não tem uma ordem claramente definida. Não significa que seja aleatório, e em geral você poderá listar todos os elementos e ter o mesmo resultado em todas as vezes que o fizer nos mesmos dados, mas como eles virão não tem uma definição. Não há garantias de que ordem seja esta.
Isso é detalhe de implementação. Pode ser que na próxima versão não esteja na mesma ordem. Pode até ser que na próxima execução não esteja.
Por exemplo em uma tabela de espalhamento, a função de hash não determina o índice, ela determina um código que será usado para achar um índice, o índice depende de alguns fatores, entre eles o tamanho da estrutura. Haverá uma ordem, ela não será aleatória, mas não é garantido que ela sempre será assim com o mesmo roll de dados entrado, então desconsidera-se que ela exista. Mesmo que ela permaneça igual, ela não pode ser considerada relevante como característica daquela estrutura.
As diferenças podem ocorrer por causa do compilador e versão utilizados, biblioteca, sistema operacional, hardware, configuração, ordem de entrada, volume de dados no momento e até versão do código que cria os objetos que são elementos desta coleção. Não importa o motivo, não confie na ordem que estão dispostos.
Não há garantias que seja a ordem de entrada dos valores, que a chave seja uma sequência, que seja uma ordem de acordo com seus valores ou chaves. O que pode enganar as pessoas, e por isso eu sempre digo: funcionar é diferente de estar certo.
Mesmo que observe uma certa ordem não tome como algo estável. Não confie no que você observa, nos bastidores pode ser diferente.

Existe o não ordenado aleatório também.
Uma tabela hash é o melhor exemplo, pelo menos sob certo ponto de vista.
Sorted
Ou classificado significa que há uma ordem definida por um critério previamente estabelecido baseado no valor do elemento inserido na coleção de dados. Esse valor pode ser uma chave ou o valor principal. Então toda coleção classificada também é ordenada.
Uma árvore, nas suas várias formas, é o melhor exemplo disto.
Algumas estruturas que são normalmente ordenadas podem acabar tendo seus dados classificados, mas não quer dizer que a estrutura seja considerada classificada, apenas aquela instância naquele momento tem esta característica, é circunstancial. Então para uma estrutura ser considerada sorted ela deve garantir essa condição. Em geral há uma chave que determine a classificação, desde que a chave seja considerada como parte do valor do elemento.
Um array tem uma chave classificada, que é seu índice, mas ele não faz parte do valor do elemento.
Uma tabela de banco de dados é ordenada, se ela tiver um ID como chave, e isso é quase obrigatório, pode ser considerada classificada também. Um índice é classificado, pelo menos a maioria deles.
Outras características
Independente destas formas é possível entender as coleções como mono ou multivaloradas, onde se permite que a chave ou valor possa se repetir, neste caso precisa analisar também se os repetidos possuem ordem ou não, se são classificados ou não em seu subconjunto.
Se os dados são heterogêneos como lida com a ordem ou classificação?
Certamente estou deixando passar alguma coisa.
Confusão de termos
É muito comum, eu mesmo faço isso, as pessoas usarem o termo ordenado quando na verdade querem dizer classificado. O que significa que a tag ordenação está quase sempre errada nas perguntas (por isso ela foi renomeada).
Coloquei no GitHub para referência futura.
Comment: Tenho um pouco de dificuldade de entender o conceito de *unordered*. Se uma função de hash vai determinar o índice do elemento numa array, de certa maneira ela está determinando uma ordem. Não?
Comment: @bfavaretto demorei mas editei, pena que eu esqueci algo importante que ia postar :)
Comment: Obrigado @Maniero, esclareceu minha dúvida :)


Lightbox2 doesn't show image

Question: I have Lightbox2 installed on my Drupal Project. I am trying to use lightbox2 through links, for example:

   Link

It works only when I turn on the Lite version. How can I make it work on normal one? I have read that there is a possibility of wrong path. Already checked all of the possibilities and I can confirm it's not that. Is it possible that it collide with jQuery?
Thanks.
Answer:
I think.. It will help to u.. Use views module and add image fields and rewrite the image fields. Look this image nkana-> database name.

Lightbox2 doesn't show image


Equivalent (?) definitions of $TP_2$

Question: I'm working with equivalent definitions of properties of formulas/theories, in particular I found these two definitions of $TP_2$ and I would like to prove the equivalence. A professor told me that he thinks that these definitions are not equivalent but, for my purpose, I only need one implication. Let's call $TP_2$ the first definition and $tp_2$ the second one.
Definition of $TP_2$:
A formula has the $TP_2$ if there are $(a^i_j)_{i,j < \omega}$ such that:

for all $\sigma \in \omega^\omega$
$$ \{\phi(x,a^n_{\sigma(n)}) \ : \ n < \omega\} \text{ is consistent} $$
for all $n < \omega$ and $i<j<\omega$
$$ \{\phi(x,a^n_i),\phi(x,a^n_j)\} \text{ is inconsistent}$$

A theory $T$ has $TP_2$ if some formula does.
Definition of $tp_2$:
$T$ has $tp_2$ if there are tuples $a,b$ and $(b_n^m)_{m,n<\omega}$ and $k < \omega$ such that:

for all $m< \omega$ and $n_1<n_2<...<n_k < \omega$ there is no $a_*$ such that $$ a_*b_{n_i}^m \equiv ab \text{ for all } 1 \leq i \leq k$$
for all $\sigma  \in \omega^\omega$ there is $a_*$ such that $$ a_*b_{\sigma(m)}^m \equiv ab \text{ for all } m < \omega$$

At first look it seems like an easy compactness argument, but I'm really struggling with this.
I would like to know:
(i) Is $TP_2$ equivalent to $tp_2$ for a theory?
(ii) Does $tp_2$ imply $TP_2$ for a theory? How can I show it?
EDIT : I know that "my" definition of $TP_2$ (the first one) is an instance of the definition of $k$-$TP_2$ ($k < \omega$) for $k = 2$,  obtained by replacing condition 2. with the following :
for all $n<\omega$ $$\{\phi(x,a^n_i) \ : \ i < \omega\} \text{ is $k$-inconsistent}$$
And I found out that a theory has $TP_2$ iff it has $k$-$TP_2$ for some $k \geq 2$. I think that this could be useful but I'm struggling anyway.
For $tp_2 \Rightarrow TP_2$ I've tried this way :
Consider $p(x,y) = tp(a,b)$. Now, for each $m < \omega$ and $n_1 < ... < n_k < \omega$ we have that  $$ p(x,b_{n_1}^m) \cup ... \cup p(x,b_{n_k}^m) \text{  is inconsistent}$$
Using compactness we can find a formula $\phi(x,y)$ in this type such that $$\{\phi(x,b^m_{n_i}) \ : \ 1 \leq i \leq k\} \text{ is inconsistent}$$
But this doesn't imply (at least to me) that $\{\phi(x,b^m_n) \ | \ n < \omega\}$ is $k$-inconsistent and also the same formula should satisfy that for each $m < \omega$ in order to witness $TP_2$ (iff $k$-$TP_2$). How can i proceed?
Comment: Hi @M.B. I agree with Alex (and your professor) that my definition of TP_2 is wrong. I don't have a counterexample at the moment. But I would certainly bet there is one (or, at least, if there isn't then it is for an interesting reason and not just some routine compactness). So you have my apologies for being the source of your problems. In the theorem about NTP2 in my paper, I start with the first row of the array as an indiscernible witness to k-dividing, and then subsequent rows are isomorphic copies of the first row. So the proof does provide a formula with TP2, as claimed.
Comment: Re: your second comment about getting rows of length $\lambda\geq |T|^+$, my guess is that this still isn't enough. This would get a inp-pattern of depth $\omega$, with some extra properties about various tuples having the same type. But it still seems too weak. So maybe an NTP2 non-strong theory can be developed into a counterexample. (Of course, one might not want to waste too much time on what amounts to sloppiness on my part.)
Comment: @M.B. Yes your reading of the proof is correct. If you are interested in strong theories I'd suggest [these notes by Adler](http://www.logic.univie.ac.at/~adler/docs/strong.pdf). Also [this paper by Dolich and Goodrick](https://arxiv.org/pdf/1511.08274.pdf) has some nice applications of these notions.
Comment: I agree with your professor that $\text{tp}_2$ should not imply $\text{TP}_2$. As evidence, I have an example of a theory satisfying $\text{tp}_2$, but such that no formula in $\text{tp}(ab)$ has $\text{TP}_2$. Unfortunately, in this example theory, some other formula has $\text{TP}_2$. I'll keep thinking and hopefully come up with a sharper example. On the other hand, $\text{TP}_2$ implies $\text{tp}_2$.
Comment: What if I can find a sequence $(a^i_n)_{i < \lambda, n<\omega}$ such that $\lambda \geq |T|^+$ witnessing $tp_2$ ? Would it be enough to impy $TP_2$ ?
Comment: Certainly if you can make the height and width of the array arbitrarily large (large enough to apply Erdös-Rado) then you can get $\text{TP}_2$. I'd have to think more carefully about the precise condition you suggest.
Comment: All these problems arose from my study of this article : https://arxiv.org/pdf/1505.00762.pdf. In section 7 the author defined $TP_2$ as "my" $tp_2$ and proved theorem 7.7 (simplicity and $NTP_2$ are equivalent for a certain class of theories) with that definition.
Comment: Thank you so much @GabeConant for your explanation. So,  if I understand correctly, in the theorem 7.7 we have an indiscernible sequence $(b_i)_{i<\omega}$ such that, if $p(x,y) = tp(ab/C)$, $\bigcup_{i=0}^{k-1} p(x,b_i)$ is inconsistent and so there is a formula $\phi(x,y) \in p(x,y)$ such that $\{\phi(x,b_i) \ | \ 1\leq i \leq k\}$ is inconsistent and by indiscernibility $\{\phi(x,b_i) \ | \ i < \omega\}$ is $k$-inconsistent. This formula also witnesses the $k$-inconsistency of each row since each row is  taken to be such that $b_{<\omega}^n \equiv_C b_{<\omega}$. Am I right?
Comment: The consistency of each path is proven using a lemma involving 'independent' (in the sense of the free amalgamation relation) sequences and I understood that part.
Thank you again for your explanation @GabeConant
I'm really interested in the possible counterexample from some kind of $NTP_2$ non-strong theory.
Answer: Here's an example showing that $\text{tp}_2$ does not imply $\text{TP}_2$.
Let $L = \{\leq_m\mid m\in \omega\}$, and let $T$ be the theory of independent linear orders. This is the model companion of the theory which says that each $\leq_m$ is a linear order. In the reduct to each finite sublanguage of size $n$, $T$ is the theory of the Fraïssé limit of the class of finite structures equipped with $n$ linear orders.
$T$ is $\text{NIP}$, hence $\text{NTP}_2$, i.e. no formula has $\text{TP}_2$.
Proof: $T$ has quantifier elimination, since any reduct of $T$ to a finite sublanguage has quantifier elimination (being the theory of a Fraïssé limit). So every $L$-formula is equivalent modulo $T$ to a boolean combination of atomic formulas. Every atomic formula is $\text{NIP}$ (it's an instance of equality or one of the orders $\leq_m$), and $\text{NIP}$ formulas are closed under Boolean combinations. Thus every $L$-formula is $\text{NIP}$ modulo $T$.
$T$ has $\text{tp}_2$.
Proof: Pick $a$, $b$, and $c$ such that $b<_m a <_m c$ for all $m\in \omega$. Now we pick an array $(b^m_n,c^m_n)_{m,n\in \omega}$ carefully. For each $m\in \omega$, we define the row $(b^m_n,c^m_n)_{n\in \omega}$ so that $b^m_n <_k b$ and $c<_k c^m_n$ for all $n$ and all $k\neq m$, but  $b<_m b^m_n <_m c^m_n <_m c$ for all $n$, and the $<_m$-intervals $(b^m_n,c^m_n)$ are pairwise disjoint.
Letting $p(x,b,c)$ be $\text{tp}(a/bc)$, we have $p(x,b^m_n,c^m_n)\cup p(x,b^m_{n'}c^m_{n'})$ is inconsistent for all $m\in \omega$ and $n\neq n'$, since this type says $b^m_n<_m x <_m c^m_n$ and $b^m_{n'}<_m x <_m c^m_{n'}$, but these intervals are disjoint.
But for all $\sigma\in \omega^\omega$, the type $\bigcup_{m\in \omega}p(x,b^m_{\sigma(m)},c^m_{\sigma(m)})$ is consistent. It suffices to see that what it says about each order is consistent. Fixing an order $<_k$, the type says $b^m_{\sigma(m)} <_k x <_k c^m_{\sigma(m)}$ for all $m\in \omega$. But for all $m\neq k$, we have $b^m_{\sigma(m)}<_k b <_k b^k_{\sigma(k)} <_k c^k_{\sigma(k)}<_k c <_k b^m_{\sigma(m)}$. So picking some $a_*$ in the $<_k$-interval $b^k_{\sigma(k)} <_k c^k_{\sigma(k)}$ for all $k$ works.

On the other hand, $\text{TP}_2$ implies $\text{tp}_2$.
Proof: Suppose $\varphi(x,y)$ has $\text{TP}_2$. By standard tricks, we may assume that the array $(b^i_j)_{i,j\in \omega}$ witnessing this has mutually indiscernible rows, and that the sequence of rows is indiscernible (see, for example, Lemma 7.4 here, where Casanovas calls this kind of array very indiscernible). Now we can stretch the indiscernible sequence of rows to a sequence $(b^\alpha_i)_{\alpha<\kappa,i\in \omega}$ of length $\kappa$. where $\kappa>|S_{xy}(T)|$, the number of complete types in variables $xy$ over the empty set.
By compactness, there is some $a_*$ such that $\varphi(a_*,b^\alpha_0)$ is true for all $\alpha < \kappa$. For each such $\alpha$, let $p_\alpha = \text{tp}(a_*b^\alpha_0)$. By pigeonhole, some type $p$ must appear infinitely many times. So we can refine our array back down to $(b^m_n)_{m,n\in \omega}$ and assume that $a_*b^m_0$ realizes $p$ for all $m\in \omega$. But now by mutual indiscernibility of the rows of the array, the existence of $a_*$ realizing $\bigcup_{m\in \omega} p(x,b^m_0)$ implies that for any $\sigma\in \omega^\omega$, the type $\bigcup_{m\in \omega} p(x,b^m_{\sigma(m)})$ is consistent. And on the other hand, for any fixed $m$ and any $n\neq n'$, the type $p(x,b^m_n)\cup p(x,b^m_{n'})$ is inconsistent, since this type contains both $\varphi(x,b^m_n)$ and $\varphi(x,b^m_{n'})$. So we have a witness to $\text{tp}_2$ (taking $a = a_0$, $b = b^0_0$, and $k = 2$).

Equivalent (?) definitions of $TP_2$


Erro ao retornar os registros do banco de dados

Question: Boa noite pessoal, estou com o seguinte erro ao retornar os dados do banco: "Warning: mysqli_fetch_assoc() expects parameter 1 to be mysqli_result, null given in"
Segue os fontes:
conexao.php
<?php
require 'config.php';
//conexão com BD
$conn = new mysqli($dbhost, $dbuser, $dbpassword, $dbname);
if ($conn->connect_error) {
    die ('Conexão falhou (' . $conn->connect_errno . ') '
        . $conn->connect_error);
}

if (mysqli_connect_error()) {
    die('Conexão Falhou (' . mysqli_connect_errno() . ') '
                . mysqli_connect_error());
}
?>

Fonte da tela:
<?php
session_start();
require 'database/seguranca.php';
?>

<title>SGTriagH - Especialidades</title>

<?php
  include "include/commom-css.php";
  include "css/padrao.css";

  $existe   = '';

  if(isset($_GET['valor'])) {
    if (!empty($_GET['valor'])) {
      require "database/conexao.php";
      $existe   = true;

      $campo    = $_GET['campo'];
      $condicao = $_GET['condicao'];
      $valor    = $_GET['valor'];

      $where    = '';

      if ($campo == 1) {        // Cod. Usuario
        $where = "descricao LIKE '%".$valor."%'";
      }else if ($campo == 2) {   // Cod. Pessoa
        $where = "tipo ".$condicao." '".$valor."'";
      }

      $query = "SELECT * FROM tb_especialidades WHERE $where";

      if ($valor == '') {
        $query = "SELECT * FROM tb_especialidades";
      }

      $result = mysqli_query($conn, $query);
    }
  }
?>

<?php
  include "include/Navbar.php";
?>

    <?php
      include "include/Sidebar.php";
    ?>

...

...

Especialidades

 Novo

          Campo
          <select id="campo" name="campo" onchange="verificaSelect();" class="form-control" tabindex="1" title="Selecione o campo para filtrar a consulta" autofocus>
            <option value="1">Descrição</option>
            <option value="2">Tipo</option>
          </select>

          Condição
          <select id="condicao" name="condicao" class="form-control" tabindex="2" title="Selecione a condição para filtrar" disabled>
            <option value="=">Igual</option>
            <option value="<>">Diferente</option>
            <option value=">">Maior</option>
            <option value=">=">Maior ou igual</option>
            <option value="<">Menor</option>
            <option value="<=">Menor ou igual</option>
          </select>

          Valor
          <input id="valor" name="valor" type="text" class="form-control text-uppercase" maxlength="60" tabindex="3" title="Informe o valor a ser consultado">

              Ações
              Descrição
              Tipo

            <?php
              if ($existe == '') {
                require "database/pessoas/popula_especialidade.php";
              }
              while ($popula = mysqli_fetch_assoc($result)) {
                if ($popula['tipo'] == 'M') {
                  $tipoGrid = 'Médico';
                }else{
                  $tipoGrid = 'Enfermeiro';
                }
                echo "
".
                     "".""."".
                     "	"
                     . $popula['descricao']
                     . "	"
                     . $tipoGrid
                     . "
";
              }
            ?>

<?php
    include "include/commom-js.php";
?>

Eu coloquei alguns echos para "debugar" o código e me retornar as seguintes informações:
Query: SELECT * FROM tb_especialidades WHERE tipo = 'M'
$result:
mysqli_result Object
(
   [current_field] => 0
   [field_count] => 3
   [lengths] =>
   [num_rows] => 3
   [type] => 0
)

Ou seja, está retornando os registros, mas no while, não está conseguindo ler do array
Answer: O método mysqli_fetch_assoc($result) de fazer o fetch dos dados está relacionado com o Estilo de Procedimento de acessar os resultados, enquanto que mysqli($dbhost, $dbuser, $dbpassword, $dbname) para abrir a conexão está relacionado com o Estilo Orientado a Objeto.
Sugiro escolher uma das formas e utilizar ela durante todo o processo.
Como referência, indico os exemplos em mysqli-result.fetch-assoc.php
Comment: Mas isso não responde a pergunta.


Generate entropy for PGP key

Question: I'm logged onto a VM remotely and trying to generate a 4096bit PGP key, it just hangs forever because there is no entropy and since I'm working through remote desktop it probably does not detect the mouse movement as entropy.
How can I generate some?
I tried cat /dev/urandom > /dev/null but that doesn't help.
Answer: Getting data out of /dev/random or /dev/urandom is definitely not going to help, all it will do is deplete your entropy pool, making the issue even worse. The main difference between these two files is that even when the kernel runs out of entropy urandom will keep generating random data of lesser quality, while random will block until it can gather fresh high-quality random data. PGP requires the highest possible random data to generate secure keys, so it will always use /dev/random.
If you have good random data around, or export some from another server's /dev/random, you can cat it into your server's /dev/random to get more entropy. You should never cat the same file twice into /dev/random though.
If you often find yourself running out of entropy you can also consider installing something like haveged, a daemon that re-generate entropy in the background and re-fill /dev/random as needed.
Also it can be tempting to symlink /dev/random to /dev/urandom, but this should be considered a security risk as any key generated using it may be less secure than they should. While it may help for one less critical applications, you have to consider every other possible use of /dev/random, including other users generating their own keys, CSR, etc.
Comment: @Kevin yes you are right, entropy pooling or harvesting are perhaps two distinct ways of "seeding" the PRNG. And after some testing on my BSD box, I've found out that random and urandom behave the same way, they block when they cannot generate good enough PRNG. Try running `dd if=/dev/random of=/tmp/rndtest bs=64M count=1` after a fresh boot, it took 2 consecutive runs to see the time to generate the 64MB file increase. I thought I would not see this effect with urandom as input, but FreeBSD seems to be blocking as well with it, unlike Linux.
Comment: @Kevin `/dev/random` are high quality PRNG on modern BSD and Linux, sure. But it will block if there is not enough entropy available. On the other side, `/dev/urandom` will not block if there is not enough available but its randomness quality might suffer in this case. In the details there are many subtleties between random and urandom implementations between Linux and the various BSDs, but the above should be true on all AFAIK.
Comment: On Linux you can generate more entropy by simply pinging a host (e.g. `ping 22.214.171.124`) if you own another network host try to have pings every 100ms (if your RTT is <100ms of course ). And/or use `find` to look for files on your hard disk and flush the RAM cache between each file search.
Comment: @Huygens: Open the man page I linked and Ctrl+F "kern.random.sys.seeded"; by default, `/dev/random` does not block on FreeBSD.
Comment: @Kevin I read it already because you made me doubt. But no I stay by my word. The man page state that when there is enough entropy `kern.random.sys.seeded` is equal 1 and a call to `/dev/random` is not blocking. Similar to Linux, a call to random when the pool is not empty is non-blocking. Now if the the entropy pool is too low (considered unsecure) FreeBSD set the variable to 0 which will block `/dev/random` similar to Linux again. So sorry to say it but I stay by my word and the man page you linked just confirms it.
Comment: @Huygens: The page doesn't say anything about entropy pooling.  It says "acceptably secure," which might well just mean "the seed is not some obvious value like zero or the current system time as of N seconds ago."  It does not in any way indicate the use of entropy pooling.  "Entropy harvest" could just mean "we periodically collect some entropy from the system and reseed the PRNG."
Comment: Note that on FreeBSD, `/dev/random` is a [high-quality PRNG](https://www.freebsd.org/cgi/man.cgi?query=random&sektion=4&manpath=FreeBSD+5.0-RELEASE), and should not normally block.
Comment: The random numbers produced by `/dev/urandom` are considered just as good as those of `/dev/random` these days on Linux machines. The concept that entropy can *run out* is a dated one. That's why in recent versions of FreeBSD, both interfaces are the same and just left for legacy purposes.
Answer: You can use haveged.
haveged is a daemon that generate entropy when needed.
Comment: Take care with Haveged in virtual environment. It can be suboptimal depending on the hypervisor configuration. Check this page: https://wiki.archlinux.org/index.php/Haveged#Virtual_machines
Comment: Yes apt-get install haveged
Answer: I would recommend to generate your gpg keys on your local machine which will have much better randomness than the remote one. And then migrate the keys using SSH to your remote machine.
Generating locally will be faster (more source for entropy), more secure (no one can spy on the process if your machine is not infected, better randomness).
If you still want to generate those remotely: On Linux you can generate more entropy by simply pinging a host (e.g. ping 22.214.171.124) if you own another network host try to have pings every 100ms (if your RTT is <100ms of course ). And/or use find to look for files on your hard disk and flush the RAM cache between each file search.
You could also installed haveged but read the limitations if you are running it in a virtual environment: https://wiki.archlinux.org/index.php/Haveged#Virtual_machines
Comment: To my mind, this is the right solution - +1 from me.
Answer: On Debian based systems, you can install the rng-tools package using atp-get, and then start the daemon to generate entropy:
echo HRNGDEVICE=/dev/urandom >> /etc/default/rng-tools && service rng-tools restart

On CentOS-6 servers, the rng daemon is installed as one of the base tools (at least on most systems I've worked on it is), and you can run the following command to start it, in order to generate entropy:
sed -i \'s|EXTRAOPTIONS=\"\"|EXTRAOPTIONS=\"-r /dev/urandom\"|g\' /etc/sysconfig/rngd && service rngd restart

Comment: I don't think that using urandom as a source for rngd is clever. It will help deplete the entropy available faster and once depleted it will be a biased source for entropy. So I would avoid this solution.
Answer: sudo yum install haveged && sudo systemctl start haveged

definitely works on a CentOS 7.2 VM.  Sometimes you want to create GPG keys on a vm if you're creating a bunch and want your keyring intact.
Answer: /dev/random and /dev/urandom are writable. If you write entropy to them, your entropy pool grows. cat /proc/sys/kernel/random/entropy_avail grows.
You may wirte any kind of data to /dev/urandom.# cat MyWordDocument.txt > /dev/urandom done as root! This is not a good source- but it is one source. If you write many of your documents (thousands of them) to /dev/urandom this will be a better source of entropy - but not a high quality type of entropy. If one would write .jpeg or .png or all his Data in /home to /dev/urandom it depends on the 'Mix' of the Data how good the generated entropy is.
ls -RU ~ | xargs -i cat {} > /dev/urandom will not work on bash- you need perl's <>. cp -RT ~ /dev/urandom will not work too. If you would do that in a script hourly with cron your entropy pool looks fine but it is poisoned with copies of the same thing, on and on.
If you write good entropy to /dev/random too (from your PC - generated with cat /dev/random > entropy.txt CTRL+C) you will get a good mix. But your PC is out of entropy then - dangerous on the internet. So never do this when being online.
Generate these .txt files offline and store them (I do that when my work is finished for the day). Your random seed may become poor - but if you play around with your mouse - put some CD's in your drive - play some of the build-in games (Tetris...) you generate new entropy. - The best solution is to buy a Geiger-counter and a radioactive source to do the job. Entropy sources for server farms. Up 10.000 bugs only.
So try to generate the pgp-key on a machine with good sources of entropy, like your PC in your office. Your /home on the virtual machine is empty - I think. If you got real change of data on that virtual machine, use that data in a intelligent way. Do a du -h "'/srv/Data~'" or something like that. List it with ls; cat ./* ; dd ; ... - Play around with it - and get some "good" entropy.
In professional hosting your hosting provider gives you a well tested and clean source of entropy - I hope so - do not cut this by extreme firewalling and isolating the virtual machine to that specially made source. You become a cryptographic Zombie on the internet. (Ask the hosting provider how it is done on their servers.)

Generate entropy for PGP key


Access Denied Exception message when pulling down files from google bucket

Question: I have been running a batch file to pull files from google bucket which was created by someone and had been working in the past, however, now I'm getting an error message stating
"ACCESS DENIED EXCEPTION:403 email@example.com does not have storage.objects.list access to dcct_-dcm_account870"
What can I do to resolve it?
Comment: What does this have to do with CMD - *Command Prompt (executable name cmd.exe) is the Microsoft supplied command line interpreter on OS/2, Windows CE, and all Windows operating systems (including Windows NT, Windows 2000, Windows XP, Vista, 7, 8, 10, Windows Server 2003, Server 2008, Server 2012, and Server 2016).* PLUS your question is unanswerable.
Answer: I just found out the solution to this issue.
I notice that ****@gmail.com have left  the company and i have to reconfigure the gsutil to give access to myself using the link below for previous answer
gsutil cors set command returns 403 AccessDeniedException


How to add comments to a Django post

Question: After adding this code to admin.py, I was expecting to see a Comments section under Blog on the Site Administration Page. To the best of my knowledge, I have run the appropriate migrations. I have looked at posts on Stack Overflow but still have not figured out the answer.
from django.contrib import admin

#register our models so they show up on our admin page
from .models import Post, Comment
admin.site.register(Post)

class CommentAdmin(admin.ModelAdmin):
    list_display = ('name', 'body', 'post', 'created_on', 'active')
    list_filter = ('active', 'created_on')
    search_fields = ('name', 'email', 'body')
    actions = ['approve_comments']

    def approve_comments(self, request, queryset):
        queryset.update(active=True)

Answer: Look into the current user permissions and check if you have the permission for the Comment Model.
Also you need to specify that your are using CommentAdmin for Comment model during registration ie
admin.site.register(Comment, CommentAdmin)

How to add comments to a Django post


How to notate groove/ swing/ shuffle rhythms

Question: I am trying to get better at notating songs, I am using Guitar Pro 5.
I try to notate small part of songs or riffs to learn it and getter better with it, so that I can write and notate own pieces there. But I definitely have a problem with unevem rhythms. (I don't know if theres a special name for it) For example bluesy rhythms or as a concrete example this song I gave up on yesterday (

) This example sounds easy at first, but for me it turned out a bit tricky.
I hope you understand what I mean with "odd/ uneven rhythms".
I found myself wildly trying some punctuated notes and just guessed the right notes. (Rhythm is the problem, not the actual height of the tone)
Do you have some tips for me how to approach that? Are there special approaches, techniques or best practices, or is it all just a matter of experience?
Thanks in advance
Comment: Link is dead, but no indication of what it was.
Answer: No need to bother too much with notating swing, it makes everything harder to write and to read. Just write once, above of the staff something like:

Swing [two quavers] = [a crotchet and quaver triplet]
  Shuffle [two quavers] = [a dotted quaver and a semiquaver]

With nice images instead of the […], obviously.
Comment: I've rarely seen a jazz or blues sheet written otherwise but the most basic. It  also allows musicians to play with a harder or softer shuffle if they want to. Now if your issue is with *recognizing* a shuffle instead of *writing* one, that's another discussion altogether.
Comment: So, in general you mean, most people just make a sketch of the melody in the notation and tell the musician to somehow spice it up with some shuffle/ swing/ groove/ whatever for the sake of simplicity in the notation?
Hmm, of course I accept that answer if most musicians which notate their music do it that way, but it feels a bit unsatisfying =)
Comment: I would really like to have this discussion, I think it is an interesting topic =)
But thanks for your answer =)
Answer: Usually the way I see it notated is just text that says "Swing" or "Shuffle" and then the tune notated in straight eighths.  For jazz tunes it's assumed that you're going to swing it.


What is chromatic number of the square of $C_n$?

Question: What is chromatic number of the square of $C_n$, for $n \ge 3$ ?
By "the square of $C_n$" I mean graph obtained from $C_n$ (cycle on $n$ vertices) by adding edges between all pairs of vertices with distance 2 on the cycle.
What I figured out:

chromatic number is at least 3, because there are triangles
if $3|n$ then chromatic number is 3: we color consequitive vertices on the cycle: $0, 1, 2, 0, 1, 2, \ldots $ and so on
if $4|n$ then chromatic number is at most 4: we color consequitive vertices on the cycle: $0, 1, 2, 3, 0, 1, 2, 3, \ldots $ and so on

Now I'm stuck.
Can anyone help?
Answer: Split the problem into a couple of cases. If $3 \mid n$, then as you rightly pointed out, the chromatic number is $3$ since the vertices can be colored according to the sequence $0,1,2,0,1,2,\ldots$ ($3$ colors).
Next, if $n=3k+1$ for positive integer $k$, if we want to color with $3$ colors, we are forced to use the three colors alternatively as in the first case. However, we get caught in the last vertex, proving that the chromatic number cannot be $3$. However, coloring the last vertex with a fourth color proves that the chromatic number is $4$.
Finally, if $n=3k+2$ for positive integer $k$, similar to above, the chromatic number cannot be $3$. It is clear that when $n=5$, the chromatic number must be $5$ since the graph is isomorphic to $K_5$. For $n>5$, we can color the first $8$ vertices according to the pattern $0,1,2,3,0,1,2,3$ and then color the rest of the $3(n-2)$ vertices in the pattern $0,1,2,0,1,2,\ldots$ which proves that the chromatic number is $4$. Thus, for your graph $G$, where $n \geqslant 3$, the chromatic number is as follows:
$$\chi(G)=
\begin{cases}
3 && \text{if $3 \mid n$} \\
5 && \text{if $n=5$} \\
4 && \text{otherwise}
\end{cases}$$


$\lim_{z \to z_0}\sum_{k=1}^{\infty} a_k (z-z_0)^{-k+m}= \infty$ if there exist at least one $k> m$ such that $a_k \neq 0$?

Question: Let $z_0, a_k \in \mathbb{C}$ and $m \in \mathbb{N}$. How can I formally prove that    $\lim_{z \to z_0}\sum_{k=1}^{\infty} a_k (z-z_0)^{-k+m}= \infty$ if there exist at least one $k> m$ such that $a_k \neq 0$?
Comment: you can't prove this because it's not so.
Answer: Counterexample: Take $z_0=0$. Note that $$\sin\left(\frac1z\right)=\frac1z-\frac16\frac1{z^3}+\dots .$$So that series does not tend to infinity as $z\to0$.


Strange filesystem behavior, Ubuntu 9

Question: I have two windows open on the same machine (Ubuntu 9, ia32, server).  I'll call these windows W1 and W2.
W1:
$ cd ~/test
$ ls
sample
$

In W2 I run "make" from a parent directory that recreates file test/sample:
$ make project
.
.
$ cd test
$ ls
sample
$

Now, returning to W1:
$ ls
$ cd ../test
$ ls
sample
$

In other words, after I build from another window and the file test/sample is replaced, ls shows the file as missing in the 2nd window until I cd ../test back into the directory whereupon it reappears.
I can give more details if required, but just wondering if this is a well-known behavior.
Answer: Do this experiment, and you'll see what's going on:
W1: cd /tmp
W1: mkdir foo
W1: cd foo
W1: rmdir ../foo
W1: ls
W1: touch ./bar

W2: cd /tmp
W2: mkdir foo
W2: cd foo
W2: touch bar
W2: ls

W1: ls
W1: touch bar

Basically, when the working directory a process is in is removed the system can't just randomly change its working directory for it. That would be a huge security hole. So it gives it sort of a fake working directory that can be read (giving 0 bytes) and various other operations, but cannot largely be used. It will happen to have the same name as a new directory created in the same place, but that's superficial.
If you do lsof | fgrep $$ after the directory is removed you will see an entry like this:
bash   2924   a_user   cwd   DIR   253,17   48   1327431   /tmp/joe (deleted)

Comment: you can put the shell back to where it should be via `cd "$PWD"`
Comment: Thanks for this excellent response; indeed I thought only the file was removed but you must be right that the _directory_ was removed and recreated by `make` leaving my shell stranded.  And the `lsof` illustrates your point beautifully.
Comment: One last comment: I understand that we wouldn't want a randomly-reassigned cwd as a result of a dir being deleted out from under us.  But I think an error msg like `current directory invalid` would be better than this "fake directory" solution.  ALthough I guess at this point, behavior like this would break 40 yrs of backward compatibility.
Comment: @Fixee - Well, I don't completely agree. This fake directory situation is roughly analogous to what happens when a file is deleted while some process has it open. It's not actually deleted until the process closes it. The 'current working directory' is just like a file that's still open, except that you can't create new entries in it, whereas with a regular open and deleted file you can still modify it and even append to it.
Comment: or you can use cd **$(pwd)**
Comment: @BradParks - The `pwd` command isn't going to give you the right answer. It will actually look at the current working directory and work backwards to try to figure out where you are. Try this sometime, make a directory and cd into it. Then rename it with `mv` in a different window. Then look at `echo "$PWD"` vs. `echo $(/bin/pwd)` (`pwd` itself happens to be a shell builtin that's exactly equivalent to `echo "$PWD"`).
Comment: hey! you're totally right.... I thought I'd tried it and it worked.... hmmm... I musn't have tried right ;-) Thanks for the info!
Answer: You have (resp. your shell has) a file descriptor open to a directory whose nlink count has been reduced to zero (probably — you left out the most important trace of all: from make) — but of course the object lives on as long as a reference is held:
/dev/shm$ md z
/dev/shm$ cd z
/dev/shm/z$ rd ../z
/dev/shm/z$ ls -al
total 0
/dev/shm/z$ ls -dli .
9347030 drwxr-xr-x 0 me users 40 Feb 15 21:59 .

Recreating /dev/shm/z creates a new entity.
/dev/shm/z$ md /dev/shm/z
/dev/shm/z$ ls -dli .
9347030 drwxr-xr-x 0 me users 40 Feb 15 21:59 .
/dev/shm/z$ ls -dli /dev/shm/z
9350877 drwxr-xr-x 2 me users 40 Feb 15 22:03 /dev/shm/z

All well-defined behavior.
Comment: Well, not exactly a file descriptor. Only sort of. The current working directory is sort of a descriptor-like resource a process can own.

Strange filesystem behavior, Ubuntu 9


How to register a Cocoa-Applescript application as a URL handler? (equivalent of "on open location" in Applescript Studio)

Question: Should be relatively straightforward, but I can't seem to find anything on this - I'm trying to add a URL handler to a Cocoa-Applescript application, as described here:
http://www.macosxautomation.com/applescript/linktrigger/index.html
Except that example doesn't seem to work in an Xcode/Cocoa-Applescript application.
In my AppDelegate.applescript, after applicationDidFinishLaunching_ I've got:
 on open location this_URL
   tell me to log "this_URL: " & this_URL
 end open location

And I've added all the CFBundleURLTypes/CFBundleURLschemes stuff in info.plist.
The latter seems to be working, as my app activates when I click a myAppScheme:// link.
But the log statement doesn't fire.
I also tried stuff like on handleGetURLEvent_(this_URL) but I'm kind of just guessing at this point :)
Any help much appreciated..
Comment: I have exactly the same problem. Did you ever figure this out?
Comment: Yeah, I did actually - I'll post an answer..
Answer: Solved! (The 'magic number' below is due to a bug in ASObjc stuff).
  on applicationDidFinishLaunching_(aNotification)
  -- Insert code here to initialize your application before any files are opened
      -- Register the URL Handler stuff
      tell current application's NSAppleEventManager's sharedAppleEventManager() to setEventHandler_andSelector_forEventClass_andEventID_(me, "handleGetURLEvent:", current application's kInternetEventClass, current application's kAEGetURL)

  -- all your other application startup stuff..

  end applicationDidFinishLaunching_

  -- handler that runs when the URL is clicked
  on handleGetURLEvent_(ev)
      set urlString to (ev's paramDescriptorForKeyword_(7.57935405E+8)) as string
      tell me to log "urlString: " & urlString

      -- do stuff with 'Set AppleScript's text item delimiters to "foo="', and then "&" etc, to parse your parameters out of the URL String

  end handleGetURLEvent_

And your info.plist needs to be in the form:
 <key>CFBundleIdentifier</key>
  <string>com.myappname</string>
...
  <key>CFBundleURLTypes</key>
<array>
  <dict>
  <key>CFBundleURLName</key>
  <string>My App Name Helper</string>
  <key>CFBundleURLSchemes</key>
  <array>
  <string>myappscheme</string>
  </array>
  </dict>
</array>

Which allows your app to take input from URLs in the form:
myappscheme://com.myappname/?foo=stuff&bar=otherstuff
The whole URL is passed in and and ends up in your 'urlString' variable, so you can parse it however you like. Which means you don't even necessarily have to follow the standard URL conventions like I have above, but you might find it convenient to do so, so that potentially you can support parsing let's say Facebook or Google Maps URLs in your app with a simple change of http:// to myappscheme://
Enjoy/Hope that helps.. :)
Comment: This is great! How did you figure this all out? Is there a reference somewhere that documents all this?
Answer: I don't use ApplescriptObjC, however in a normal cocoa application see this post which explains how to do it. I'd assume you need to do it like that.
Comment: Thanks and yeah stuff like that comes up in Google searches, but unfortunately I've no idea how to integrate that with an Applescript application :\
I tried adding an `on getURL_` method but that doesn't fire either - maybe I need to do the 'Register the URL handler method' bit, but no idea how to do that in Applescript..
Comment: You definitely need the register part. That's what you're missing. Unfortunately I can't help you convert that code into ApplescriptObjC terminology... it can be done though and that's what you need. So ask a question specifically about that. Good luck.
Answer: I may be running into something similar. I have an Applescript URL handler that launches but doesn't process the passed URL (just as you've described).
What I found was when another program sends a link, everything functions properly.
When I try to manually send a link via the terminal "open" command, the "on run" handler triggers rather than "on open location" - which unfortunately means I can't get the passed URL.
Comment: Could you post what you have so far? (a generalised version of, if you prefer). Quite keen to get this working so maybe I can bang on the keyboard a bit more and see if I can get a bit further. In particular curious about your on run handler and the bit where you say "What I found was when another program sends a link, everything functions properly." - sounds further than I got.. (I assume this is all in the XCode world rather than Applescript Studio btw..?)
Comment: I'm discussing it over at macscripter here: http://macscripter.net/viewtopic.php?pid=160028 . I'd post the code directly (and avoid future linkrot), but it far exceeds the character limitations. Oh, and this is straight up, plain-jane AppleScript - as ugly as that can be.
Comment: Great, thanks, I may as well try some of those handlers in Cocoa-Applescript and see if I can get any of them to fire.. incidentally I posted the Cocoa-Applescript version of the question also on Macscripter, no replies so far :) http://macscripter.net/viewtopic.php?id=40260


Finding a constant that satisfies a given inequality

Question: I have a problem where I was able to show via a series of steps, and Jensen's inequality, that
$x > y$ where $x>0$ and $y>0$. Since $x$ is always greater than $y$, is there anyway to multiply $x$ by a non-negative constant such that $x$ will always be less than $y$, i.e, I want to find a $\beta$ such that
$\beta x < y$ for $\beta >0$.
Comment: Do you mind adding your work in those steps
Comment: @Arjun They're not consequential to the question and are more in depth than just $x$ and $y$.
Answer: $\beta$ just needs to be less than $y/x$,
i.e.,
$\beta < \frac{y}{x}$


What regression model to use for judgment data?

Question: I am new to statistic modeling and I need help with choosing the right model for an analysis.
We run an experiment on word order variation in German. We presented sentences to 27 participants who had to rate the acceptability of the sentences on a 7-point scale. The sentences all contained three arguments: subject (S), direct object (DO) and indirect object (IO) with all possible their 8 permutations. They also varied with respect to whether each argument was a full noun phrase (like "the dog") or a pronoun ("he"), so that there was a total of 8 x 6 = 48 critical items. The data looks like this:
Item WordOrder PhraseTyp   Rating Participant
 1    S-IO-DO  Pro-NP-Pro    3      A
 2    S-IO-DO  NP-NP-Pro     2      A
 3    S-IO-DO  Pro-Pro-Pro   2      A
 4    S-IO-DO  NP-NP-NP      4      A
 …      …         …          …      …
 15   S-DO-IO  Pro-NP-NP     6      A
 16   S-DO-IO  NP-Pro-Pro    7      A
 …      …         …          …      …

What I would like to test is if the word order of the argument ("WordOrder") or the distribution of pronouns/nominal phrases ("PhraseType") has an effect on the rating.
I guess I can use some kind of regression modeling. I am not sure, what I should use. According to what I read, I seems as if I need some kind of mixed effects model, with random effects for participants (not sure about items). Is that correct? I am also not sure if I need linear regression or ordinal logistic regression? What is the best function to use with R? I tried the following:
glmer(Rating ~ WordOrder + PhraseType + (1|Participant),
       data = RatingData)

This gives me an warning message:
In glmer(Rating ~ WordOrder + PhraseType + (1 | Participant),
    data = RatingData): calling glmer() with family=gaussian (identity
    link) as a shortcut to lmer() is deprecated;
    please call lmer() directly

If I use this, I get the following summary of the the model:
Linear mixed model fit by REML ['lmerMod']
        Formula: Rating ~ WordOrder + PhraseType + (1 | Participant)
           Data: RatingData

    REML criterion at convergence: 4292.9

    Scaled residuals:
        Min      1Q  Median      3Q     Max
    -3.4997 -0.6709 -0.0844  0.5890  3.5763

    Random effects:
     Groups      Name        Variance Std.Dev.
     Participant (Intercept) 0.3809   0.6172
     Residual                1.7083   1.3070
    Number of obs: 1248, groups:  Participant, 26

    Fixed effects:
                          Estimate Std. Error t value
    (Intercept)            1.89503    0.18013  10.521
    WordOrderDO-S-IO       0.43269    0.12816   3.376
    WordOrderIO-DO-S      -0.06250    0.12816  -0.488
    WordOrderIO-S-DO       0.67308    0.12816   5.252
    WordOrderS-DO-IO       2.12019    0.12816  16.543
    WordOrderS-IO-DO       2.46635    0.12816  19.244
    PhraseTypeNP-NP-Pro   -0.55128    0.14799  -3.725
    PhraseTypeNP-Pro-NP    0.50000    0.14799   3.379
    PhraseTypeNP-Pro-Pro  -0.03846    0.14799  -0.260
    PhraseTypePro-NP-NP    0.98077    0.14799   6.627
    PhraseTypePro-NP-Pro  -0.30769    0.14799  -2.079
    PhraseTypePro-Pro-NP   0.42949    0.14799   2.902
    PhraseTypePro-Pro-Pro -0.40385    0.14799  -2.729

    Correlation matrix not shown by default, as p = 13 > 12.
    Use print(x, correlation=TRUE)  or
         vcov(x)     if you need it

Judging just from the intercepts, this looks as I would have expected it; e.g. having the subject at the beginning has a positive effect. (If I interpret the results correctly). What I do not really understand is why I do not get those significance stars ("**" etc.).
Is this the right approach, or should I use ordinal logistic regression? (I tried that as well but didn't get anything working).
Sorry for the longish post.
Comment: If I were you I'd prefer to treat the rating as continuous (not ordinal) and use the `lme` function from the `nlme` package.
Comment: re: the lack of "significance stars" - when you use glmer() and don't specify a family (and link function) it effectively assumes you just want linear regression and so uses lmer() instead. [lmer() doesn't give p values](https://stat.ethz.ch/pipermail/r-help/2006-May/094765.html)
Comment: @mark999 For the types of fully crossed designs common in (psycho) linguistics, `lmer` from `lme4` is much better than `lme`. Most of the published literature in (pyscho)linguistics that uses mixed models uses `lme4`, including the more tutorial-type publications.
Comment: @Livius Thanks for the comment. I tend to use `lme4` only if `nlme` doesn't have what I want (e.g. generalised linear mixed models). Nice answer.
Answer: There are several aspects to the problem, some of which you addressed explicitly in your question, some of which you didn't.
What type of scale are ratings?
Following Stevens (1946), ratings are ordinal and thus not well suited to linear regression (or its mixed-effects extensions). However, as a rule of thumb, the regression methods that can work on ordinal scales (ordinal logistic, arguably some forms of quantile regression) are more "exotic" and thus less well supported by the various software packages. When you combine "exotic" regression with "mixed-effects", the support is even worse. If you're willing to go the Bayesian route, then the wonderful MCMCglmm package does support mixed-effects ordinal logistic regression and will even output something like to p-values based on its MCMC estimates.
That said, many people in your position would just treat ratings as an interval scale and thus suitable for linear regression. It is up to you to determine which interpretation is appropriate and to make a convincing case for your interpretation to potential reviewers and other consumers of your scientific output. In the following, I will more or less assume that you went the linear regression route, although much of what I say is still applicable for ordinal logistic and other regression techniques.
lme4: Where are my p-values?
This is an R FAQ. A short summary would be

degrees of freedom aren't as trivial as you were taught when dealing with models where the different parts can partially or fully overlap (i.e. hierarchical/multilevel models with partially or fully crossed levels).

In traditional repeated-measures ANOVA (which is actually a special case of the linear mixed-effects model with a particular symmetry assumption (sphericity)) and other simpler models, it is possible to calculate the degrees of freedom analytically (i.e. there is a formula for them). However, for more general designs, this is not possible and we must approximate the degrees of freedom. There are several different ways to do this (Satterthwaite, Kenward-Roger, etc.) but all are approximations and can at times be quite computationally intensive. Without degrees of freedom, it is not possible to convert t-values to p-values, nor F-values to p-values for the ANOVA case. (After all, the denominator degrees of freedom for the F-test are the square of the degrees of freedom for the t-test.) Given all the issues with p-values and their interpretation (finding the relevant literature is left as an exercise for the reader, but the psychological science literature has taken to self-flagellation trying to come to terms with it, so it's a good place to start), why would you bother with a computationally expensive approximation?
If you absolutely must have p-values, there are several ways to compute them beyond these approximations, including likelihood-ratio tests, although those may be less interesting for your particular model.
You could also recall that the t-distribution approaches the normal distribution as its degrees of freedom approach infinity. This means that for sufficiently large degrees of freedom (which you often have for (psycho)linguistic data), you can treat t-values as z-values. The traditional 5% significance level occurs for (two-sided, which you have/want) at $z = 1.96$. To be a little bit more conservative, you can just say that you reach significance at $t = 2$ (cf. Baayen, Davidson and Bates, 2008, amongst others for a similar train of thought).
But again, I would discourage you from getting too attached to statistical significance. Think about the size of differences and how well we can see that size amongst the noise/error, and not about whether there is a difference!
How do I interpret my regression coefficients?
There are lots of good explanations of this online and in various textbooks, so I won't go into much detail here, but I think your fundamental problem is in your choice of contrasts. A linguistically motivated discussion of contrast coding (with links to other good discussions) can be found here, but you might want to look at a textbook discussion of contrast coding first (e.g. in one of Andy Field's Discovering Statistics books, which rely on slightly less mathematical intuition than other explanations I've seen). If you're looking at coefficient directly, then you should probably be using sum coding, and I would recommend using contr.Sum from the car package as it gives better names to the coefficients than the built-in contr.sum (note the capitalization).
Finally, it looks like you may actually be more interested in certain pairwise contrasts than anything else. In that case, you may want to consider the lsmeans package, as this is the most convenient way to do post-hoc pairwise comparison. For your case, this has advantages beyond being easier to interpret than regression contrasts/coefficients:

It's not sensitive to the choice of contrast coding (i.e. treatment vs. sum coding, obviously if you structure your predictors completely differently that will make a difference)
It will give you p-values for the pairwise estimates along with a multiple comparisons correction for the doing so many tests.

On the structure of you model
Your model is a good start, but based on my understanding of your experimental question, it still needs some improvement:

You have no interaction term: Surely you care about how word order and phrase type interact with each other?
Depending on what type of contrasts you're interested in, it may make sense to have several categorical predictors for word-order and phrase-type, e.g. three predictors first.arg, second.arg, third.arg (with possibilities S, DO, IO for each of them) or maybe subject, direct, indirect (with possibilities first, second, third) for each of them. Depending on the exact contrasts you set up, some things will be easier to read from the coefficients and some things will be harder. You may also get a warning about dropping columns due to a rank-deficient matrix -- this is not actually a real problem, but just means the model doesn't compute coefficients for combinations which don't occur (e.g. both subject and direct simultaneously have the value first).
You need a random-effects term for Item: this isn't like classic ANOVA-by-subjects vs. ANOVA-by-items ($F_1$, $F_2$, cf. Clark 1973), you can and should model both sources of variation (subjects/participants and items) simultaneously, otherwise you may come to the wrong conclusions (cf. Judd, Westfall and Kenny 2012).
You may want a random-effects structure that includes slopes and intercepts, but there is a lot of debate on this. See e.g. Barr, Levy and Sheepers 2013; Barr 2013; Bates, Kliegel, Vasishth and Baayen 2015/arXiv as well as the GLMM FAQ linked above.

One last thing to get started ...
A final bit of advice: it looks like you're completely new to mixed models, maybe even completely new to statistics. That's fine, we all had to start somewhere. But you might want to do some more background reading. :) Although slightly out-of-date on the software front, there is a 2008 special issue of the Journal of Memory and Language on mixed models. Other popular (book) references for this are Gelman & Hill (2006), Fox (2016 -- newest edition has much more on mixed models, but there are also several good older editions) and Pinheiro & Bates (2000). Finally, Bodo Winter has an online tutorial that many of my students have also found quite good.
Comment: Thanks a lot, @Livius! Indeed, I am a newbie; but this looks really helpful! I will work through it.


What is the validation data for the neural network?

Question: I am a beginner in this of neural networks and automatic learning and I understand that the network learns training data, but I do not understand what the validation data are.
Validation data is a set of training data?
Can you give me an example of training data vs validation data
Answer: In statistics let's say you have some dataset. You want to test whether or not your model performs well in general on this kind of data, so you tweak your model based on some of the data in your dataset and validate the model's performance on the remaining (unseen) data in that dataset. This split is referred to as a train-test split. You'd typically expect a ratio of 80:20 for training and testing data. These sets of data are not mixed - they are wholly separate chunks of your data!
But the problem with this approach is that once you test on the test set, you cannot tweak the model anymore, not without being biased towards improving your results on the test set. Maybe you tried tweaking too much for your training data and you got poor generalisation.
The solution to this problem is typically using a validation set. So now you divide your data into three chunks:

Training data (typically 60% of your data, these percentages can vary)
Validation data (typically 30% of your data)
Testing data (typically 10% of your data)

You train on the training set, get a measure for your generalisation performance on the validation set, and only when you are COMPLETELY done with model tweaking, do you then test your model on the testing data. You hide your testing data under a rock - you touch it only once and only at the very end.
It's worth stressing that this approach has some variants (k-fold cross validation), but the broad rationale for it and other techniques is that you don't want to bias your model towards the data you test it on, because otherwise you are effectively cheating and you're not really getting a true measure of generalisation.
Hope this makes things clearer - and keep in mind this is true for neural networks as well as any kind of statistical modelling!
Answer: Validation data is to "validate" that your neural network hasn't rather memorized your training data and has thus actually learned some meaningful aspects of the data so that the model can be later on used (generalized) to unseen, held-out "test dataset".


Use Angular 2 in a multiple page website

Question: How do you get from the Angular 2 quickstart app to using Angular 2 in a website?
My main problem is that the index.html in not in the route folder of the app. Also is it possible to have many pages that have angular 2 apps?
I realise that Angular 2 is designed for SPAs but the html component system must be useful in all sorts of websites.
Good page here Is AngularJS just for single-page applications (SPAs)? but no actual info on how to do it.
I would like to use a much or as little angular 2 as necessary in different pages across my website project.
A sample website using angular 2 in this way would be great.
Comment: Angular 2 is designed for you to build a Single Page Application. Can you give more detail on the problem you're having, and what you've tried already?
Answer: To answer my question (just in case anyone is interested), I just use a little TypeScript to load my html templates. Simple.
<code>public static getFileContentAsync = async (path: string): Promise<string> =>
{
    return await new Promise<string>((resolve, reject) =>
    {
        // Set up the xml http request object.
        let xhr: XMLHttpRequest = new XMLHttpRequest();
        xhr.open("GET", baseUrl + path);
        xhr.send();

        xhr.onload = () =>
        {
            resolve(xhr.response);
        };

        xhr.onerror = (errorEvent: ErrorEvent) =>
        {
            reject(errorEvent);
        };
    });
}
</code>


Oracle LogMiner (Unsupported Operation)

Question: I'm currently using Oracle 11g on RHEL5. I'm digging to see what are in the archived log files. After I query from v$logmnr_contents, I see many transactions of UNSUPPORTED operation.
What do these unsupported transactions mean?
I think that it's the cause that make my database generates lots of archived logs.
Moreover, I'm using global temporary table for generating reports. I discover that when I insert and delete from those temporary table, it also records in the archived log file.
How to do to reduce those recorded transactions?
Regards,
Sarith
Comment: Are you SURE they are global temporary tables ?
There isn't another table of the same name in a different schema, or a view or synonym.
Are you doing stuff with LOBs ?
Comment: Yes, I'm sure that they are global temporary tables in my schema.
Answer: The following operations are unsupported in Logminer, Oracle 11.

BFILE datatype
Simple and nested abstract datatypes (ADTs)
Collections  (nested tables and VARRAYs)
Object refs
Tables using table compression
SecureFiles

Oracle LogMiner (Unsupported Operation)


Proving that tensor distributes over biproduct in an additive monoidal category

Question: I'm trying to prove that the tensor product distributes over biproducts in an additive monoidal category; namely that given objects $A,B,C$, we have $A \otimes (B \oplus C) \cong (A \otimes B) \oplus (A \otimes C)$.  What I've attempted is:

Now it's clear that the arrows from $A\otimes B$ to $A \otimes B$ collapse to $\mathrm{id}_{A\otimes B}$ and similarly for $A\otimes C$.  I now want to show that $\varphi\circ\theta$ is $\mathrm{id}_{(A\otimes B)\oplus(A\otimes C)}$.  I want to invoke the uniqueness of the (co)product again but this requires a map from $(A\otimes B)$ and $(A\otimes C)$ from the bottom of the diagram up to $(A\otimes B)\oplus(A\otimes C)$ at the top.
I'd like to conclude that $p_{A\otimes B}\circ(\varphi\circ\mathrm{id}_A\otimes i_B)  = \mathrm{id}_{A\otimes B}$ implies that $(\varphi\circ\mathrm{id}_A\otimes i_B)=i_{A\otimes B}$.  This intuitively makes sense if considering a concrete example like modules or sets but seems unjustified here.  (Attempting to complete the product diagram with the diagonal composite leads to the problem that it is not a zero morphism... it is $\mathrm{id}_A \otimes 0_{bc}$.)
I found a similar proof in the setting of modules (http://unapologetic.wordpress.com/2007/04/17/more-on-tensor-products-and-direct-sums/) but unfortunately he glosses over the part where I am stuck.  At least I seem to be considering the right morphisms though.  I feel like I'm missing something obvious but I've been agonizing over this for many hours... does anyone know what to do here?
Answer: What's the definition of an additive monoidal category? Is it that tensor product distributes over addition of morphisms? If so, use the fact that a functor between additive categories preserves addition of morphisms iff it preserves biproducts (see for example this blog post).
Comment: @Omnivium: so you don't require any compatibility between the additive and tensor structures? Then there's no reason for biproducts to be preserved. I'm reasonably certain the correct definition is that tensor product distributes over addition of morphisms.
Comment: @Omnivium: for a simple counterexample, take the category of vector spaces and let $\otimes$ be the direct sum.
Comment: What I mean is a monoidal category $(C, \otimes)$ where the category C is an additive category; (http://en.wikipedia.org/wiki/Additive_category).  I believe the only property I actually need from that definition is that products and coproducts exist for any two objects, and they are always biproducts (which I denoted in my post by $\oplus$).  I'll check out that blog post though and see if there's any ideas there that help.
Comment: I agree 100% with Qiaochu here.
Comment: Haha crap, I think I even thought of a similar counterexample and dismissed it... I guess I desperately wanted it to be true for an easy proof that the multiplication $[A]\cdot[B]=[A\otimes B]$ is well defined on the split Grothendieck group (ring) of the additive category in question.  Guess it's back to the drawing board.  In any case, thanks very much for the answers!
Answer: As Qiaochu Yuan pointed out, in a monoidal category with finite biproducts, $\otimes$ doesn't always distribute over $\oplus$. If the functor $X \otimes \_\,$ has a right adjoint for every object $X$, however, then distributivity does hold (see Example 2.4.2 of Chris Heunen's thesis, Categorical Quantum Models and Logics). I don't know if this is helpful in your case, but it's pretty cool.
Unfortunately, the fact above doesn't seem to be proven in Heunen's thesis, and the proof isn't obvious to me. I've community-wiki'd this answer so that anyone who knows the proof can add it.
Comment: The proof is trivial: biproducts and binary coproducts coincide in additive categories, and left adjoints preserve all colimits.


C++ standard: return by copy to initialize a reference without RVO: is there any copy?

Question: Let's consider the next sample:
struct big_type {};

// Return by copy
auto factory() { return big_type{}; }

void any_scope_or_function() {
    big_type&& lifetime_extended = factory();
}

Under the assumption RVO is forbidden or not present at all and in any manner, will or can big_type() be copied? Or will the reference be directly bound to the temporary constructed within the return statement?
I want to be sure the big_type destructor is called only once when any_scope_or_function ends.
I use C++14, in case some behaviour has changed between standard's versions.
Comment: @Peregring-lk but for the record.. this changed in C++17 which doesn't anymore require copy elision here.
Comment: @NathanOliver I forgot to take out that compiler switch. Whoopsies.
Comment: Besides, I can't use C++17; C++14 at most.
Answer: Assuming there is no RVO/copy elison then in
auto factory() { return big_type{}; }

big_type{} is going to create a temporary big_type.  This object is then going to be used to copy initialize the object the function returns.  This means that the object you make in the function is going to be constructed and destructed.
With
big_type&& lifetime_extended = factory();

the rvalue reference will extend the lifetime of the functions return so we will see in total a default constructor call, a copy/move constructor call, and two destructor calls.
Now, if we change
auto factory() { return big_type{}; }

to
big_type factory() { return {}; }

then we are no longer creating a object in factory.  the return object is directly initialized with {} giving us in total a default constructor call and a destructor call
Answer:
Or will the reference be directly bound to the temporary constructed
  within the return statement?

No, it won't be. That's exactly what (N)RVO is about/for and you explicitly don't want that.
However, what will be attempted is using the move constructor of your big_type, which is technically not a copy. It does violate : "big_type destructor is called only once when any_scope_or_function ends" though since it'll be called twice.
GCC/Clang has a nice compiler switch : -fno-elide-constructors that disables (N)RVO which can be used for future reference. For now, here is a test that has that option turned on, showing a double destructor call.


Applying k-means over PCA

Question: I have a dataset containing 20 columns and 200 rows. This is an unlabeled dataset and I applied PCA to this dataset for dimensionality reduction. After successfully using PCA, I received a dataset with 2 columns(PC1 & PC2) and 200 rows. Then I executed the k-means algorithm on the output dataset(200x2). In the end, I got a graph plotting 5 clusters(X-axis is PC1 and Y-axis is PC2). But, how can we understand what X and Y axis on the graph represents? Also, how is it possible to figure out the understanding from this final graph when I don't have any clue about the values I obtained from PCA?
Comment: Are you sure 2 PCs suffice? How much % variance do they capture? To try to interpret the meaning of a PC you will have to use [loadings](https://stats.stackexchange.com/q/143905/3277). But a more straightforward way to understand the differences between the clusters would be to compare them on each of the 20 original features. On some, they will differ, on some, will not.
Comment: @ttnphns 2 PCs constitute more than 70% variance. So I believe that will be sufficient. As I'm working on MATLAB, how is it possible to compare PCs with each column using loading? Actually, I tried to search for an example, but I couldn't find one. Please help me with this.
Comment: [Here](https://stats.stackexchange.com/a/102999/3277) is pca steps, on iris data. Follow it and compare what you  get in Matlab. Be careful with terminology. Some programs incorrectly call eigenvector coefficients the loadings (I don't know how Matlab is at that).
Answer: Your first question: With 20 columns in your dataset, it is probably difficult to "understand" what the two first PCs "mean" unless they happen to be very easy linear combinations of only few of your features. But it doesn't hurt to check. Your PCA algorithm should give you the linear combination of the first PCs from your features.
Your second question: This method is mainly giving you the clustering, i.e. the grouping of your data. Knowing which observations "belong together" is often quite valuable. Gleaning additional information from e.g. the position of the clusters in your PCA subspace is probably rather involved.
Also note, that PCA preprocessing can change the true clustering of your data quite a bit. E.g. it could easily happen that the projection to your two-dimensional PCA subspace maps two perfectly separated clusters into a single cluster. But you probably have your reasons why you do PCA.
Comment: At the end of K-means clustering, a graph was generated representing the clusters with PC1(Principal Component) and PC2 as X & Y-axis. I was using MATLAB and was able to generate the cluster numbers corresponding to each row in the table. But, I would like to know what this clusters represent. For Eg, Is cluster 1 means Good, cluster 2 is average, etc.
Comment: @Krishna Clustering is for grouping, so you know that some observations belong together while others don't. What you seem to look for is assigning labels ("good", "average") to your data. That's called classification.
Comment: I believe working on a 2D graph with meaningful vectors, even the clusters can be grouped. So each data point in a cluster can be labeled accordingly.

Applying k-means over PCA


Open HDF5 files in Julia

Question: I have Julia 1.7.2 and HDF v"0.16.2" on a Windows 10 machine.
I ran into an issue opening an HDF5 file:
<code>using HDF5
raw_hdf_data = h5open("Data_2021-03-17_11-21-19.h5", "r")
obj = raw_hdf_data["Group"]["Data_1"]
HDF5.read(obj)
</code>
but got an error:

HDF5.API.H5Error: Error reading dataset /Flatfield/Data_1
libhdf5 Stacktrace:
[1] H5PL__find_plugin_in_path: Plugin for dynamically
loaded library/Can't open directory or file
can't open directory
⋮
[11] H5Dread: Dataset/Read failed
can't read data
Stacktrace:
[1] macro expansion
@ C:\Users\jhsu.julia\packages\HDF5\auFb7\src\api\error.jl:18 [inlined]
[2] h5d_read(dataset_id::HDF5.Dataset,
mem_type_id::HDF5.Datatype, mem_space_id::HDF5.Dataspace,
file_space_id::HDF5.Dataspace,
xfer_plist_id::HDF5.DatasetTransferProperties, buf::Matrix{Int64})
@ HDF5.API C:\Users\jhsu.julia\packages\HDF5\auFb7\src\api\functions.jl:508
[3] generic_read(::HDF5.Dataset, ::HDF5.Datatype, ::Type{Int64})
@ HDF5 C:\Users\jhsu.julia\packages\HDF5\auFb7\src\HDF5.jl:978
[4] read(obj::HDF5.Dataset)
@ HDF5 C:\Users\jhsu.julia\packages\HDF5\auFb7\src\HDF5.jl:878
[5] (::var"#66#67")(fid::HDF5.File)
@ Main .\In[68]:3
[6] h5open(::var"#66#67", ::String, ::Vararg{String, N} where N; swmr::Bool,
pv::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(),
Tuple{}}})
@ HDF5 C:\Users\jhsu.julia\packages\HDF5\auFb7\src\HDF5.jl:287
[7] h5open(::Function, ::String, ::String)
@ HDF5 C:\Users\jhsu.julia\packages\HDF5\auFb7\src\HDF5.jl:285
[8] top-level scope
@ In[68]:1
[9] eval
@ .\boot.jl:360 [inlined]
[10] include_string(mapexpr::typeof(REPL.softscope), mod::Module,
code::String, filename::String)
@ Base .\loading.jl:1116

Thanks very much!
Comment: The error message is unclear whether it can't find a plugin or just can't find the file itself. Can you do a ls on the current directory (```;ls``` or `run(\`ls\`)`) and verify that the file you are trying to open is returned in that list?
Answer: I had the same issue when I updated my HDF5 library.
Probably the file is compressed and one of the following plugins are necessary. Try:
<code>] add H5Zblosc, H5Zbzip2, H5Zlz4, H5Zzstd
using HDF5, H5Zblosc, H5Zbzip2, H5Zlz4, H5Zzstd
raw_hdf_data = h5open("Data_2021-03-17_11-21-19.h5", "r")
...
</code>
Now it should open the file.

Open HDF5 files in Julia


EXCEL SCRIPT TO COPY COLUMN TO END OF TABLE AND RENAME IT

Question: Basically we are using a powerapp to do the register. We do this once a week and it populates a column called "Attendance" with the value "Present" For each child that attends.
However before the next register happens I require the attendance column to be clear. In order to do this I would require the attendance column to be copied to a new column with "today" date".
I have managed to create a column with "today" date but need help in copying the values over. You can see the code i have used to create a column. The attendance column is column M.
function main(workbook: ExcelScript.Workbook) {
  var today = new Date();
  var dd = String(today.getDate()).padStart(2, '0');
  var mm = String(today.getMonth() + 1).padStart(2, '0'); //January is 0!
  var yyyy = today.getFullYear();
  let x = dd + '/' + mm + '/' + yyyy;

  let sh = workbook.getActiveWorksheet()
  let tbl = sh.getTable("FBRegister")
  tbl.addColumn().setName(x)
}

Answer: You can try the following script:
function main(workbook: ExcelScript.Workbook) {
    var today = new Date();
    var dd = String(today.getDate()).padStart(2, '0');
    var mm = String(today.getMonth() + 1).padStart(2, '0'); //January is 0!
    var yyyy = today.getFullYear();
    let x = dd + '/' + mm + '/' + yyyy;

    let sh = workbook.getActiveWorksheet()
    let tbl = sh.getTable("FBRegister")
    let todayColumn = tbl.addColumn(); // Save the new column
    todayColumn.setName(x); // Set the name to today's date

    // Get the attendance column, excluding the header
    let attendance = tbl.getColumn('Attendance').getRangeBetweenHeaderAndTotal();

    // Copy values from attendance column to the new column
    todayColumn.getRangeBetweenHeaderAndTotal().setValues(attendance.getValues());

    // Clear attendance column
    attendance.clear();
}

I've assumed the attendance column is named "Attendance", but if not, you can modify that line (let attendance = ... ) accordingly. Additionally, the last line of code clears the attendance column once the values have been copied over, but you can comment out/delete that line if you don't want that.
Hopefully that helps!
Comment: Thank you soo much! This has helped massively! May God Bless you!


Using Lagrange multipliers to find volume of a cone

Question: Using the method of Lagrange multipliers (or otherwise) find the maximum possible volume of a (right circular) cone which can be inscribed in a sphere of radius a.
Comment: It is custom on math.stackexchange to state what you have tried, your thoughts on the problem, or where are you stuck in the problem.
Comment: Hi! Sorry but the problem is I don't even know where to begin.
Answer: $\textit{Hint}$:  Use the formula $V=\frac{1}{3}\pi r^2h$, and slice the cone vertically down the middle to produce a triangle which contains a right triangle with sides of length $r$ and $h-a$ and a hypotenuse of length $a$.
Now use the Pythagorean Theorem to express $r^2$ in terms of $h$.


How to do «go to left/right/forward/backward» with python dronekit?

Question: I am using APM for autopiloting my hexacopter and following this tutorial. by looking at this available commands, I cannot see how one can command the drone to go to left/right/forward/backward?
Can anybody help me on this?
Answer: You need to create a  vehicle.message_factory.set_position_target_local_ned_encode.
It will require a frame of  mavutil.mavlink.MAV_FRAME_BODY_NED.
You the add the required x,y and/or z velocities (in m/s) to the message.
<code>from pymavlink import mavutil
from dronekit import connect, VehicleMode, LocationGlobalRelative
import time

def send_body_ned_velocity(velocity_x, velocity_y, velocity_z, duration=0):
    msg = vehicle.message_factory.set_position_target_local_ned_encode(
        0,       # time_boot_ms (not used)
        0, 0,    # target system, target component
        mavutil.mavlink.MAV_FRAME_BODY_NED, # frame Needs to be MAV_FRAME_BODY_NED for forward/back left/right control.
        0b0000111111000111, # type_mask
        0, 0, 0, # x, y, z positions (not used)
        velocity_x, velocity_y, velocity_z, # m/s
        0, 0, 0, # x, y, z acceleration
        0, 0)
    for x in range(0,duration):
        vehicle.send_mavlink(msg)
        time.sleep(1)

connection_string = 'tcp:192.168.1.2:5760' # Edit to suit your needs.
takeoff_alt = 10
vehicle = connect(connection_string, wait_ready=True)
while not vehicle.is_armable:
    time.sleep(1)
vehicle.mode = VehicleMode("GUIDED")
vehicle.armed = True
while not vehicle.armed:
    print('Waiting for arming...')
    time.sleep(1)
vehicle.simple_takeoff(takeoff_alt) # Take off to target altitude
while True:
    print('Altitude: %d' %  self.vehicle.location.global_relative_frame.alt)
    if vehicle.location.global_relative_frame.alt >= takeoff_alt * 0.95:
        print('REACHED TARGET ALTITUDE')
        break
    time.sleep(1)

# This is the command to move the copter 5 m/s forward for 10 sec.
velocity_x = 0
velocity_y = 5
velocity_z = 0
duration = 10
send_body_ned_velocity(velocity_x, velocity_y, velocity_z, duration)

# backwards at 5 m/s for 10 sec.
velocity_x = 0
velocity_y = -5
velocity_z = 0
duration = 10
send_body_ned_velocity(velocity_x, velocity_y, velocity_z, duration)

vehicle.mode = VehicleMode("LAND")
</code>
Have fun, and of course, observe rigorous safe guards when programming and flying UAVs. A manual mode override is a must!
Answer: Dronekit-python has non trivial APIs to command the drone in the local frame. From my personal experience, it was hard to get my head around on using these commands make my drone follow a shape locally, e.g a square or a circle.
An alternative is using FlytOS drone APIs. If you see this sample python code on github you can see how easy it is to do command the drone to go left x meters then forward y meters etc. Jon's answer does show correctly how dronekit could be used to achieve what you are trying to do, but another beginner who might be intimidated by the complex code.
Answer: I had the same problem, I hope this link helps look for the "controlling by specifying the vehicle's velocity components" section of the article here


Php date function -1 month issue

Question: I have previously searched for a way to remove a month from the current time and I came up with this line:
date("Ym",strtotime("-1 month", mktime()));

To be honest it worked until today, March 31st 2014. In my application these two operations return the same value (201403):
date("Ym",strtotime("-0 month", mktime()));
date("Ym",strtotime("-1 month", mktime()));

Can anyone explain to me why I have this error, and how can I fix it?
Thank you.
Comment: Then you should explicitly decrement the month number. What you're doing is subtract "one month" from "March 31st", which by definition is a pretty undefined operation.
Comment: What does "one month" mean exactly? What result would you expect?
Comment: Always reset your date to the first of the month before doing math like this on it
Comment: What I want to do is remove 1 month from my current date. It means that I will have for instance 201402 when we are in march and 201401 when we are in february.
Comment: Ok I will set it to first day of the month, then substract one month
Answer: By setting the date at the first day of the month before doing math on it is important! This operation allows to set the date to the first of the month :date('01-m-Y'). Here is the full solution of the problem:
date("Ym", strtotime("-1 month", strtotime(date('01-m-Y'))))

Answer: date(
    'Ym',
    mktime(1,2,3,date('m')-1,date('d'),date('Y'))
);

Comment: It looks fine to me, but I still have the same problem. Unfortunatly date('Ym', mktime(1,2,3,date('m')-1,date('d'),date('Y'))) == date('Ym', mktime(1,2,3,date('m')-0,date('d'),date('Y')))

Php date function -1 month issue