computing mean and sd of $d^\prime$ from mean and sd of hits and false alarms

Question: I'm working on a meta-analysis. Some studies report $d^\prime$, others report accuracy (hits and false alarms). I would like to transform all the data into the same measure, $d^\prime$ (also $c$). However, for many studies, I only have the mean and sd of the accuracy measures. If I compute $d^\prime$ from the mean hit and false alarms, the outcome is not the same as the mean $d^\prime$, but it is at least closely correlated with mean $d^\prime$. However, I cannot figure out how to figure out the variance of the $d^\prime$ based on the mean and sd of the accuracy data. Is this possible, or am I going to have to request individual subject data?
Comment: I take it these are all within subjects or repeated measures studies. And if so, are you more interested in individual condition variances or effect variances? And finally, what kind of measure is the preponderance of scores, proportions or d'?
Comment: They are generally within-subject studies, yes. I want the individual condition variances because most studies yield more than one comparison, and I also want to plot the d' data before showing the forest plots.
Comment: I would say it's about half and half between the two measures. But my primary hypotheses are in terms of detect ability vs criterion, which is why I prefer the SDT measures to straight accuracy.
Comment: Could you compute hit and false alarms as odd ratios? Then, you could convert OR in d...
Answer: For repeated measures designs this is unfortunately not solveable. The reason is that those hit and false alarm rates are probably correlated and probably in some unknown way. The papers will not generally report that correlation or any statistic that could be used to derive it. Therefore, the problem cannot be solved.
However, as is typical in many meta-analyses, sometimes a value can be generalized from the papers it is reported in to the ones it is not. In this case you use the mean variance from the papers with d' values and use it as the variance for one where that is not available.


Fundamental group determined by connected components

Question: Suppose $G$ is a simply connected topological group and $H$ is some subgroup. I want to prove that, if the coset space $G/H$ is connected, then
$$ \pi_1(G/H) = \pi_0(H)/\pi_0(G) $$
If $G$ is connected then I have been able to establish the result; the non-contractible loops in $G/H$ are merely the loops in $G$ that start at the identity (say) and end in some other connected component of $H$. However, if $G$ is not connected then the result is less straightforward to me. Firstly, it's crucial that $G/H$ is connected, else $H$ is allowed to sit entirely within one component of $G$, and may have fewer connected components than $G$ does. Further, it's not even clear to me how to find $\pi_0(G)$ as a subgroup of $\pi_1(H)$, let alone see whether it is normal.
EDIT: thank you everyone for the excellent comments and answers. I should make a couple of clarifications. Ordinarily, we only talk of the fundamental group of (path-)connected spaces; when I say that $G$ is simply connected, I mean that each connected component is simply connected. Furthermore, whilst I placed few restrictions on the nature of $G$ and $H$, I had in mind that both were in fact Lie groups, and that $G$ was a fibration of $H$ over $G/H$. This was anticipated by some comments and answers; I am grateful to have received  such good responses to the question which was in my head, but which I failed properly to ask!
Comment: Use the long exact sequence of the fibration $G \to G/H$.
Comment: @anomaly:   Is the quotient map always a fibration?  $H$ is not assumed to be a nice subgroup.  I'm thinking of things like $\mathbb{R}/\mathbb{Q}$.
Comment: @anomaly I think most people who see this exercise don't know the term fibration. That is the right approach, but the proof in low degrees is even more concrete.
Comment: @MikeMiller: The homotopy sequence for a fibration is a basic, very useful tool, and this time is as good as any to study it.
Comment: Where did you see this claim?  As you say, it doesn't even make sense because $\pi_0(G)$ is not naturally a subgroup of $\pi_0(H)$.  Also, "simply connected" normally includes "path connected" in its definition...
Comment: This problem makes no sense to me.
Comment: It's only a fibration if $H$ is a closed subgroup of $G$, I think?
Comment: @mikemiller:  Are you sure it's a fibration?  (I don't have a counterexample, but I think I'm only used to the "$G\rightarrow G/H$ is a fibration" when $H$ is closed in $G$.)
Comment: @JasonDeVito: I was thinking about the case where $G, H$ are Lie groups and $H\subset G$ was a nice closed subgroup. In general, weird stuff can happen with arbitrary groups (like, as you point out, $\mathbb{R}/\mathbb{Q}$).
Answer: This statement is nonsense.  As you say, there is in fact no natural way to consider $\pi_0(G)$ as a subgroup of $\pi_0(H)$ if $H$ is a subgroup of $G$.  Instead, there is a natural map $\pi_0(H)\to \pi_0(G)$ (though this map may not be injective).
What is true (assuming $H$ is a nice enough subgroup of $G$ that the quotient map $G\to G/H$ is a fibration) is that $\pi_1(G/H)$ is isomorphic to the kernel of this map $\pi_0(H)\to \pi_0(G)$.  This follows easily from what you have already done, since a loop in $G/H$ lifts to a path in $G$ which then must be contained in the identity path component $G_0\subseteq G$, so $\pi_1(G/H)$ is the same as $\pi_1(G_0/(G_0\cap H))$.  By what you have said, $\pi_1(G_0/(G_0\cap H))$ is naturally isomorphic to $\pi_0(G_0\cap H)$, which is exactly the kernel of $\pi_0(H)\to \pi_0(G)$.
[Incidentally, to me, a simply connected space must be path-connected, so $G$ must be path-connected, but your definitions may be different...]
Answer: As others have remarked, $\pi_0(G)$ is not naturally a subgroup of $\pi_0(H)$.  The following example (which meets all of they hypothesis of your problem) indicates that sometimes it's not a subgroup at all.
Let $G = S^3\times \mathbb{Z}_2\times \mathbb{Z}_2$, where $S^3$ denotes the unit quaternions and $\mathbb{Z}_2 = \{\pm 1\}$ under multiplication.  Note that $S^3$ is connected, so $\pi_0(G) = \mathbb{Z}_2\times \mathbb{Z}_2$.  Note also that $G$ is simply connected (in the sense that each connected component is simply connected).
Let $H = \langle a,b\rangle$ where $a = (i,-1,1)$ and $b = (j,1,-1)$.
Claim 1: The group $H$ is isomorphic to the quaternion group $Q_8 =\{\pm 1, \pm i, \pm j, \pm k\}$, so that, in particular, $\pi_0(H)\cong Q_8$.
Proof:  A simple calculation shows that all of the following relations hold:  $$a^4 = e, \quad a^2 = b^2, \quad b^{-1}ab = a^{-1}.$$
According to wikipedia, one presentation of the Quaternion group is $\langle a,b: a^4 = e, a^2 = b^2, b^{-1} a b = a^{-1}\rangle$, so $H$ must be a quotient of $Q_8$.  On the other hand, $\langle a\rangle$ contains $4$ elements, none of which are $b$, so $H$ consists of at least $5$ elements.  Thus, $H$ is isomorphic to $Q_8$.  $\square$
Claim 2: The space $G/H$ is connected.
Proof:  Just note that $H$ meets all the components of $G$: the elements $a,a^2, b, ab$ are in different components. $\square$
Finally, simply note that $\pi_0(H)\cong H\cong Q_8$ has no subgroups isomorphic to $\pi_0(G)\cong \mathbb{Z}_2\times \mathbb{Z}_2$.  In fact, all subgroups of order $4$ in $Q_8$ are cyclic.


Utilizar dados existentes no banco para autenticação

Question: Tenho um banco de dados com dados já existentes, com nome, CPF, RG e outros dados pessoais. Preciso implementar um sistema de login, mas queria que o usuário fosse no caso o numero de registro e a senha o CPF, por exemplo. Nas demonstrações de como fazer o login, sempre me indicam a fazer um novo model com usuário e senha. Ai pergunto, poderia usar a mesma tabela da qual já tenho os dados pessoais para fazer o login, ou teria que copiar os dados dessa tabela para um model novo, dedicado somente ao User?
E se eu quisesse deixar o usuário logar tanto com e-mail, quanto com numero de registro? Como faria essa consulta?
Comment: Dá pra fazer, mas a estrutura vai ficar estranha. Sugiro você criar uma rotina de migração de dados, onde ele alimente a nova tabela de login com os dados já existente. Se você optar por manter os dados assim, como vai fazer para o usuário alterar a senha ? O e-mail do Usuário já está presente na tabela existente?
Comment: Olá, Victor. Ainda não, os emails só estariam disponíveis depois que os usuarios atualizarem seus cadastros. O que eu ainda não consigo entender é como usar os dados da tabela presenta para autenticar. Basta copiar os dafos das colunas que eu quero usar para autenticar para a tabela criada pelo novo model?
Comment: Duvida: Na sua aplicação cada pessoa cadastrada vai ser um usuário ?
Comment: Sim, cada pessoa cadastrada é um usuário. Eles podem logar no sistema para alterar seus dados cadastrais.
Comment: Ok, nesse caso, é mais fácil você criar um campo senha, nessa mesma tabela, e joga o CPF do cara pra esse campo (como padrão), cabe a você criptografar essa senha ou não. Aproveita e cria junto o campo e-mail. Oque acha? Partindo dessa definição, qual o próximo problema ?
Comment: Era isso mesmo que eu estava pensando em fazer. Até porque, se eu copiasse os dados para um outro model, a cada associado cadastrado, eu teria que migrar os dados. Enfim, só me resta dúvidas a como permitir que o username seja tanto o registro quanto o email.
Answer: Nesse caso, é mais fácil você criar um campo senha, nessa mesma tabela, e joga o CPF do cara pra esse campo (como padrão), cabe a você criptografar essa senha ou não. Aproveita e cria junto o campo e-mail
Primeiro faz uma validação no que o cara digitou, com regex, com isso você direciona para validação do e-mail ou nome. Para a validação em Ruby, segue o link: https://stackoverflow.com/questions/22993545/ruby-email-validation-with-regex
A partir dai a validação é normal. com base noque você tem para aquele registro.


Como passar o campo de um struct para uma função numa biblioteca a parte?

Question: Olá, pessoal.
Estou com um programa que tem uma struct assim:
struct itemDeLista{
        char nomeProd[10];
        float quant;
        float vUnit;
        int item;
    };
Mas preciso contar a quantidade de letras na string nomeProd[10], como o professor pediu que nós não usássemos a string.h estou fazendo a função strlen numa biblioteca própria, estou recebendo o char pelo scanf mas quando passo para a função da biblioteca, acontece alguns erros. Vou colocar o código inteiro.
#include <stdio.h>
#include <stdlib.h>
#include "biblio.h"

struct itemDeLista{
    char nomeProd[10];
    float quant;
    float vUnit;
    int item;
};

struct itemDeLista valores;

int main(){
  int k = 0;

  scanf("%[A-Z a-z]",valores.nomeProd);
  printf("%s\n", valores.nomeProd);
  k = strcnt(valores);
  printf("%d", k);
return(0);
}

Este é o programa principal, a seguir é a biblioteca e a função pra contar a quantidade de caracteres.
#include <stdlib.h>
#include <stdio.h>

int strcnt(struct itemDeLista m);

int strcnt(struct itemDeLista m){
   int i = 0, cont = 0;
   while(1){
      if(m.nomeProd[i] != '\0'){
      cont++;
      }
    else{
        break;
    }
    i++;
  }
return(cont);
}

Agradeço muito o help, valeu!
Answer: Com a definição da struct no programa principal a biblioteca não tem informação sobre ela.
Tens de colocar a definição da struct itemDeLista num ficheiro a que ambos os "sources" tenham acesso; por exemplo: itemDeLista.h
// itemDeLista.h
#ifndef ITEMDELISTA_H
#define ITEMDELISTA_H
struct itemDeLista {
    char nomeProd[10];
    float quant;
    float vUnit;
    int item;
};
#endif

Depois incluis este header nos teus programas
#include <stdio.h>
#include <stdlib.h>
#include "biblio.h"
#include "itemDeLista.h"

struct itemDeLista valores;

int main(void) {
    int k = 0;

    scanf("%9[A-Z a-z]", valores.nomeProd);
    printf("%s\n", valores.nomeProd);
    k = strcnt(valores);
    printf("%d\n", k);
    return(0);
}

e no source da tua biblioteca
#include <stdlib.h>
#include <stdio.h>
#include "itemDeLista.h"

int strcnt(struct itemDeLista m);

int strcnt(struct itemDeLista m) {
    int i = 0, cont = 0;
    while (1) {
        if (m.nomeProd[i] != '\0') {
            cont++;
        } else {
            break;
        }
        i++;
    }
    return cont;
}

Se o header "biblio.h" já tem a definição da struct tens de inclui-lo na biblioteca.
Comment: Muito obrigado, funcionou aqui. Uma pergunta, nesse formato de header compartilhado pros dois, eu consigo: valores.quant = 15, por exemplo, nos dois arquivos?
Comment: Consegues, mas não é aconselhável usar variáveis globais. É melhor usares uma variável normal dentro do main e passar o seu endereço para outras funções.


Calculating $\lim_{x \to 0}\frac1x \int_{0}^{x}f(y)dy$ when $f$ is continuous

Question: I was trying the following problem which is :

Let $f \colon \Bbb R \to \Bbb R$ be a continuous function such that $$\lim_{x \to 0}f(x)=a .$$ Then $$\lim_{x \to 0}\frac1x \int_{0}^{x}f(y)dy $$ is which of the following:
$(A)1 \,\,(B)a \,\,\,(C)-1 \,\,\,(D)$ does not exist

My Attempt:  WLOG,I take $f(x)=a+bx$ ,then I see option (B) is the correct choice. Am I right? Is there any better alternative way to tackle the problem?
Thanks in advance for your time.
Comment: Concerning your attempt: that is certainly not WLOG.  It is with complete loss of generality when you assume that $f$ has the form $f(x) = a+bx$.  What about $f(x) = a + \sin x$?
Comment: If you take $f(x) = a+bx$, you can't say "WLOG", because nobody said that your function is differentiable, even less $C^1$; you *are* losing generality. Consider a counterexample $f(x) = a+\sqrt x$.
Comment: thanks a lot @AntonioVargas sir for pointing out the mistake in my argument.
Answer: Let $\epsilon > 0$. Choose $\delta > 0$ so that $|x| < \delta\implies |f(x) - a| < \epsilon$.  Then if $x < \delta$,
$$\left|{1\over x}\int_0^x f(t)\, dt - a\right| \le  \frac {1}{|x|}
\int_0^x |f(t) - a| \, dt \le \epsilon.$$
Consequently, the limit is $a$.
Comment: Thanks for catching the |.| I missed, TZakrevskiy.
Answer: Hint: You can use L'hopital's rule. Let

$$ F(x) = \int_{0}^{x} f(y) dy $$

and consider the limit

$$ \lim_{x \to 0} \frac{F(x)}{x}.$$
Answer:
For each interval $[0,x]$, define
$$
M_x = \sup \{f(y) : y \in [0,x]\} \text{ and } m_x = \inf\{f(y) : y\in [0,x]\}
$$
Then
$$
m_x \leq \frac{1}{x}\int_0^x f(y)dy \leq M_x
$$
Hence, by intermediate value theorem, there is a point $c_x \in [0,x]$ such that
$$
\frac{1}{x}\int_0^x f(y)dy = f(c_x)
$$
Now, $0 \leq c_x \leq x$ for all $x$, and the fact that $f$ is continuous, implies that
$$
\lim_{x\to 0} \frac{1}{x}\int_0^x f(y)dy = \lim_{x\to 0} f(c_x) = a
$$

For the rest, you are on the right track, so explain what you tried a little more, and someone here can complete your proof.
Comment: Since $f$ is continuous, $\int_0^xf(y)dy$ is differentiable. Thus using L'Hospital's rule we get $f(0)$ as the limit.
Comment: @user40615 : To prove the fundamental theorem of calculus, one usually uses the above property (the mean-value theorem for integrals)
Comment: Agree, but it is asked ''what is the limit''.
Comment: @user40615 : When one asks a question in mathematics, one usually expects justification for the answer.
Comment: Mean ''cant we use the FTC?''It is known fact why do we need prove once again?


HLS Live Stream API in Python serving files from the disk

Question: I have a camera which captures videos and then i use FFMPEG to segment those videos for HTTP Live Streaming and generate an m3u8 file and their corresponsing ts files
I have all the *.ts files and *.m3u8 files created in my local folder and i would like to serve those files to the client using an API
I have created a Flask application in Python
Flask API.
@app.route('/videos/<file_name>')
def live_stream(file_name):
    return send_from_directory("D:\\public", file_name)

HTML Code

      <video width="400" controls>
         <source src="http://127.0.0.1:5000/videos/toystory.m3u8" type="video/mp4">
      </video>

Sample m3u8 file. As you can see there is no End tag since i keep on appending new ts files to this m3u8 file to support live streaming
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:64
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:64.397667,
toystory0.ts

Couple of issues that i had:

Is there a python library which i can use which provides me a way to serve HLS stream from the files stored on disk
Even though the file is sent across as per my fiddler output i don't see the video on the browser
Comment: Any HTTP server can handle serving your HLS segments.  No special Python app needed.
Comment: For your case, [nginx](https://nginx.org/ru/docs/http/ngx_http_hls_module.html) would be a much more appropriate tool.
Answer: You need extra url like http://127.0.0.1/videos/<video_directory>/<video_segments>/ that will serve segment files. Now your server serve only index file. And you should use hls.js to playback video segments. hls.js send requests for segment files.


How do I cache an object loaded through SOAP from a third party that I use in forms?

Question: I am loading and object that I use to populate a form. Then on form submit I return some of the fields through SOAP back to the third party. On submit the form building function gets called and loads the object then the submit function gets called and loads the object again. How do I cache the object so that it is only loaded once but available to both or more function to act on it.
Answer: Would it not be possible for you to simply pass off the data you need in fields of #type => value? These are passed safely behind the scene through the entire form submission process, acting in a sense as the cache you are striving for.  Consider the following simplistic example :
$form['soap_data_email'] = array('#type' => 'value', '#value' => $soap_data_email);
$form['soap_data_name'] =  array('#type' => 'value', '#value' => $soap_data_name);

After submission, all your soap object's values would be available in the validation and submit callbacks of your form through the usual $form_state['values']['soap_data_email'] and   $form_state['values']['soap_data_name'] array.
If you have many variables this could all be factorised to be more efficient, but this should give you the basic idea.
This is the simplest way I can think of that lets you cache your data all across the form validation and submission process.
Answer: Both answers so far a very good. There is an additional method which is worth noting. static caching.
there a few places in Drupal core which use static caching and there is even a small api to support it in D7. the static keyword in PHP is a scope which is local to a function but remains between function calls.
So if you have a method to load your SOAP object you can set it as a static variable in the loading method
function my_loader($id) {
  // Initialize a static variable.
  static $loaded_items = array();
  // Check to see if object with $id has been loaded
  if (!is_set($loaded_items[$id]) P
    // code to load $soap_object;
    $loaded_items[$id] = $soap_object;
  }
  // Return the item
  return $loaded_items[$id];
}

You can see it in action in Drupal core in _menu_build_tree(). This only lasts for the duration of a request, but it can be handy with a framework where you are not sure when, or how many times a loader may be called.
protips
$trees = &drupal_static(__FUNCTION__, array());

The drupal_static() method allows for static variables to be accessible from everywhere. This stops every function which declares static variables from having a clear flag. So if you save your soap object you can clear the static cache.
The __FUNCTION__ magic word is the name of the function currently being executed.
The & before the function passes the return value by reference so that changes to $trees will be changed in the static variable in the function drupal_static()
Comment: Might be worth noting that if you use Eugene's approach (cache_get/set) it implements static caching, so you don't need to.
Comment: Although drupal_static is cool for per request caching, I am curious if this keeps values for entire form submission process.
Comment: @andy I am not sure if it does, it looks like it caches the cache controlling object statically but not the cache items themselves. I'm not sure if cache clear all would work well if it did.
Comment: @stefgosselin it depends on the process. Your method is probably best for that as you can guarantee that it will remain with the form, were as static caching will have to call soap service again if validation fails for example.
Comment: @JeremyFrench thanks for checking :) I'd read it recently in The Definitive Guide - it says _Drupal's caching functions take care of static caching for you, which is nice... when the data is fetched from the cache with a database query, it doesn't repeat this query during a page request. (See `api.drupal.org/_cache_get_object` which is called by `cache_get()`.)_ However my inspection of `_cache_get_object()` leads me to agree with you - it's the cache controller that's cached(!). Neither does DrupalDatabaseCache::getMultiple() statically cache AFAICT.
Answer: If you want to have cache that will be accessible not only in one form, but all over the site - you can use Drupal Cache API
Here is example for your case.
While creating form you save object in cache:
cache_set('my-variable_name', $object);

Then on submit handler you can load object from cache:
$object = cache_get('my-variable_name');


Xcode Organizer Metrics aren't showing three most recent releases

Question: I'm trying to find the Metrics Launch Time of my app by using Xcode Organizer but it does not show the result of the 3 most recent versions. Only for older versions.
I'm using Xcode 12.2 (12B45b), and even after upgrade to 12.3 (12C33), still doesn't work!
Does anyone run into the same problem and know how to fix it?
Answer: TLDR: Versions with limited usage will not appear in the Xcode Organizer Metrics.
We were curious why this is happening to our app aswell. We had 52 releases in 2021 and the Battery Usage section from Xcode Organiser Metrics menu shows us only 5 of them, while Scrolling section shows only 4 at the moment of writing this answer.
After doing some research and watching this WWDC20 session on Diagnose performance issues with the Xcode Organizer I have learned that apple is using some usage threshold for each version of the app to determine wether it should show up in the metrics or not. According to the linked session, in Xcode 12 they have lowered the required usage threshold by a factor of five.
Another thing is that now you will see a limited usage icon attached to versions which have passed the new threshold but their usage is still low enough so you will need to be aware of the margin or error while analysing those metrics.
So even though the threshold have been lowered, it's still not really helpful If you update the app often like we do. Which is sad since we are not really able to provide improvements to these metrics since we do not really know with what we are dealing with. Let's hope future versions of Xcode will at more information about how much usage is enough, because apparently tens of thousands of users on a version are literally not enough for it to be included on a list.
Resources:

Diagnose performance issues with the Xcode Organizer


Why is this fret buzzing on my plectrum banjo?

Question: I inherited this very used Vega VIP banjo.  I don't have an age on it, and because it was "built" by combining pieces from a few different banjos it lacks a serial number.  It plays well enough although it has a good share of scratches and nicks.  (It was a travel instrument for a professional musician from the 1950's through the 90's).
Whenever I play a chord on the fifth fret, it buzzes.  I hear the lowest-sounding string buzzing the most, and it's somewhat lessened if I bend the string towards the next string.
I've attached pictures of the fret itself and the places where the bottom string terminates.
Because it's a family instrument with a great deal of sentimental value I'd rather not trust it to just "any" shop to repair without having some idea what I'd be asking them to do.
I marked the fret in question in the picture.  It's the lowest string, far left in all pictures.
Any guesses about what to check first?
Comment: I agree, you don't want to take it just "anywhere," but you might have to find someone you trust to take a look. I imagine there could be a lot of potential causes—bridge, nut, a severely worn fret, maybe the one above standing a bit high, even maybe the neck being twisted a bit.
Comment: Maybe a picture of the fretboard, taken from above at a very small angle - in order to see if it's bended - might also lead to interesting and accurate answers!
Answer: You most likely either have a low fret on one side, or a high fret on the other.  Sometimes a fret will pop out of the fret slot a little. If you look at the edge of the frets where it meets the wood, a popped fret will have a little gap and not be flush to the fingerboard. Low frets can happen if the fret is driven in by a hit or pressure.  If you have a small straight edge, such as a metal ruler, you can place it across the frets and see if one is higher or lower than the others.
Fret buzzing and fret level issues are a common adjustment for technicians and Luthiers. A shop that offers repairs and setups, or a local Luthier should be able to take care of the problem easily.


Prove the number of roots of polynomial $Q(x)$

Question:
Let $P$ be a polynomial, it has $n$ distinct real roots and all these roots are larger than 1. Let
  $$
Q(x)=(x^2+1)P(x)P'(x)+x[(P(x))^2+(P'(x))^2].
$$
  Prove $Q(x)$ has at least $2n-1$ distinct real roots.

I start to find $Q(x)=0$ between two roots. Assume $P(x)<0$ in $(x_1,x_2)$ and $P(x_1)=P(x_2)=0$. There exists a minimal value at $\xi\in(x_1,x_2)$ s.t.
$$
P(\xi)<0, P'(\xi)=0, P''(\xi)>0.
$$
Then I get $P(\xi)+P'(\xi)<0$.
It can be deduced by contradiction that not all $x\in (x_1,x_2)$ satisfy $P(x)+P'(x)\leq 0$. Hence there exists $\eta$ s.t. $P(\eta)+P'(\eta)>0$, then there exists $\alpha$ s.t.
$$P(\alpha)+P'(\alpha)=0.$$
 $Q(\alpha)<2xP(\alpha)P'(\alpha)+x[((P(\alpha))^2+(P'(\alpha))^2]\leq 0$, i.e.
$$Q(\alpha)<0.$$
 If I have $Q(x_1)>0$ and $Q(x_2)>0$ then by mean value theorem there are two roots in $(x_1,x_2)$.
The problems of this thought are:
(1) Even I have two roots in every interval, only $2n-2$ roots be found;
(2) $Q(x_1)>0$ and $Q(x_2)>0$ may be zero and my process would be useless?
Hope someone give me an answer or a hint.
Comment: if $h(x)=f(x)g(x)$ then $deg[h(x)]=deg[f(x)]+deg[g(x)]$ and if $deg[f(x)]=n $ then $deg[f'(x)]=n-1$ since f is a polynomial?  Correct?  does this help at all?
Comment: @TZakrevskiy Thank you! I've updated.
Comment: Does your assumption implicitly imply that $P$ has degree $n$, i.e. all roots are of multiplicitly one?
Comment: @MooS I'd doubt tahat. In fact, any multiple root of $P$ is a common root of $P$ and $P'$, hence also a root of $Q$.
Comment: Neither are mentioned in the book. Hence no degree and multipilicity restriction. @MooS
Comment: @AmateurMathGuy It helps me solve the question! Thank you.
Comment: @yahoo well I can only go as far as $$\begin{align} Q(x)&=(x^2+1)P(x)P'(x)+x[(P(x))^2+(P'(x))^2] \\ deg[Q(x)]&=deg\left\lbrace(x^2+1)P(x)P'(x)+x[(P(x))^2+(P'(x))^2]\right\rbrace \\ deg[Q(x)] &\leq max\left\{deg\left[(x^2+1)P(x)P'(x)\right], deg\left[ x\cdot(P(x))^2],deg[x\cdot(P'(x))^2\right]\right\} \\ deg[Q(x)]&\leq max\{2+ n +(n-1),1+2n, 1+2(n-1)\} \\  deg[Q(x)] &\leq 2n+1\end{align}$$ So perhaps you can help me understand how I helped you
Comment: @AmateurMathGuy I've accepted an answer that gives two hints to completely sove the question. Its idea is the same as yours(use the property of polynomial's degree, and get the degree of $Q(x)$ is odd).
Comment: @yahoo ahh ok glad I can help.  Being helpful isn't a guarantee on this site.
Answer: I claim that $$H(x) := \frac{Q}{xP(x)P'(x)} = \frac{x^2+1}{x}+\frac{P^2+P'^2}{PP'}$$ has at least $2n-2$ roots in its range of definition. Then in particular $Q$ has at least $2n-2$ roots.
As you have proposed, we fix two roots $x_1 < x_2$ with no more roots in between them and some intermediate value $a$ with $P'(a)=0$ (Rolle's Theorem).
Let us assume $P(a) > 0$, the other case is dealt with the same way.
Then $P'<0$ in $(a,x_2)$.
Note that $|P(a)| > |P'(a)|$ but $|P(x_2)| \leq |P'(x_2)|$, hence at some point in $(a,x_2)$ we have $|P|=|P'|$. At this point we have $|\frac{P^2+P'^2}{PP'}| =2$ and thus $\frac{P^2+P'^2}{PP'}=-2$. We also know $\frac{x^2+1}{x}>2$ in $(a,x_2)$, because $x>1$.
Hence $H>0$ at this point and clearly $\lim\limits_{x \to a, x>a} H = -\infty = \lim\limits_{x \to x_2, x<x_2} H$, thus we obtain two roots of $H$ in $(a,x_2)$ by the intermediate value theorem.

Doing this for any two adjacent roots of $P$, this addds up to $2n-2$ roots, i.e $Q$ has at least $2n-2$ roots.
Here comes the punchline: $Q$ is a polynomial of odd degree, hence the number of real roots is odd. Thus having at least $2n-2$ real roots actually ensures having at least $2n-1$ roots.
Comment: I agree that $\bigg|\frac{P^2+P'^2}{PP'}\bigg| =2$ , but how do you see that $\frac{P^2+P'^2}{PP'}=-2$ and not $+2$ ?
Comment: $P$ is positive and $P'$ is negative in $(a,x_2)$, hence the fraction is negative.
Comment: I see now, thanks.
Answer: Ad (1): Note that $P(x)$ and $P'(x)$ have opposite signs for $x\ll 0$, hence $Q(x)<0$ for $x\ll 0$. But at the first real root $x_0$ of $P$, we have $Q(x_0)=x_0P'(x_0)^2\ge0$ because $x_0\ge1$. Thus there is an additional root of $Q$ in $(-\infty,x_0]$.
Ad (2): If $P(x_i)=0$, then $Q(x_i)=x_iP'(x_i)^2\ge 0$. Hence if both $x_i$ are only simple roots, then we obtain two distinct root insinde $(x_1,x_2)$. Whereas if $P(x_i)$ is a $k$-fold root, $k\ge 2$, then $x_i$ is a precisely $(2k-2)$-fold root of $Q$. So first of all, the $Q(x_i)$ count as distinct roots by themselves (but that''s too few). But at least we know that $Q$ has no sign change at $x_i$. Does that help?


Individually throttled torrents with Python

Question: I'm trying to write a python program that can handle seeding individually throttled torrents. Specifically, for some possibly large (up to 30 or so) simultaneously-running torrents, I would like to throttle the upload rate of each one to some individually-set values (x_1, x_2, ..., x_n) in kb/s.
I have tried using rtorrent via xmlrpc, but there seems to be a bug that prevents non-global throttles from having any effect (in fact, checking the set rate for throttles always returns "0" regardless of what it was set to). Does anyone have any recommendations for libraries or other tools I can use from within python to control torrents with this level of granularity?
Answer: It's well documented that you MUST set a global limit, since throttles share from that global bandwidth pool.
For the rest, https://code.google.com/p/pyroscope/wiki/RtXmlRpcExamples#Setting_and_checking_throttles


Why do we call the equations of least square estimation in linear regression the *normal equations*?

Question: When we want to estimate parameters of linear regression, we make normal equations as many as the linear model contain number of unknowns. Why are these equation called normal equations?
Comment: Why downvote this? It's better than someone posting their homework or test problem.
Answer: I'll give what is perhaps the most common understanding, then some additional details.
Normal is a term in geometry (Wikipedia):

In geometry, a normal is an object such as a line or vector that is perpendicular to a given object.

which in turn appears to come from a term for a carpenter's or mason's square [1]

NORM and NORMAL. According to the OED, in Latin norma could mean a square used by carpenters, masons, etc., for obtaining right angles, a right angle or a standard or pattern of practice or behaviour. These meanings are reflected in the mathematical terms based on norm and normal.

and from geometry the term moves into vector spaces.
The direct answer for "normal equations" is given here: http://mathworld.wolfram.com/NormalEquation.html

It is called a normal equation because $b-Ax$ is normal to the range of $A$.

(In the usual regression notation that's '$y-Xb$ is normal to the range of $X$')
Literally, the least squares residual is perpendicular (at right angles) to the nearest point in the space spanned by $X$.

The $y$-vector lies in $n$ dimensions. The X-matrix spans $p$ of those (or $p+1$ depending on how your notation is set up; if $X$ is of full rank, it's the number of columns of X). The least squares solution $X\hat{\beta}$ is the nearest point in that space spanned by $X$ to that $y$-vector (indeed, literally the projection of $y$ onto the space spanned by $X$). It is necessarily the case that by minimizing the sum of squares, the difference $y-X\hat{\beta}$ is orthogonal to the space spanned by $X$. (If it were not, there would be a still smaller solution.)
However, as whuber suggests in comments, it's not quite so clear cut.
Looking at [1] again:

The term NORMAL EQUATION in least squares was introduced by Gauss in 1822 [James A. Landau]. Kruskal & Stigler's "Normative Terminology" (in Stigler (1999)) consider various hypotheses about where the term came from but do not find any very satisfactory.

However, the method of normal equations is often credited to Legendre, 1805.
[1] Miller, J. (ed)
"Earliest known uses of some of the words of mathematics, N"
in Earliest known uses of some of the words of mathematics
Comment: I have always assumed this was the reason, but there are other plausible reasons, too.  The Wolfram site does not provide any references for its assertion.  Are you aware of any historical documentation about the origin of this term?
Comment: Hmm. You could be right. I'll make some edits with additional information


jsons with different fields every time

Question: I've been searching for an answer but couldn't find it.
I have a json like this:
followersperdate{
 'date2012-08-29': 16823,
 'date2012-07-09': 15571,
 'date2012-07-07': 15528,
 'date2012-06-25': 15237,
 'date2012-04-19': 13283,
 'date2012-03-16': 12999,
 etc.
}

and I want to get the values; but everytime I make the request, I recieve other dates. I don't know how to get the values.
I've tried with jsonpath but didn't work.
Comment: possible duplicate of [How to get all properties values of a Javascript Object (without knowing the keys)?](http://stackoverflow.com/questions/7306669/how-to-get-all-properties-values-of-a-javascript-object-without-knowing-the-key)
Comment: sorry for that. thanks!
Answer: This is pretty trivial using DefiantJS (http://defiantjs.com). This lib extends the global object JSON with the method "search". Using this method, you can search a JSON structure, regardless of depth for values. It returns an array with the matches (empty array if no matches were found).
To get acquainted with XPath, check out this valuable resource;
http://www.defiantjs.com/#xpath_evaluator
Check out this fiddle;
http://jsfiddle.net/hbi99/DKuAQ/
var data = {
   "date2012-08-29": 16823,
   "date2012-07-09": 15571,
   "date2012-07-07": 15528,
   "date2012-06-25": 15237,
   "date2012-04-19": 13283,
   "date2012-03-16": 12999
  },
  res = JSON.search( data, '//*' ),
  str = '';

for (var i=0; i<res.length; i++) {
   str += res[i] +'
';
}

document.getElementById('output').innerHTML = str;

jsons with different fields every time


NLS analysis in R with timeseries data to remove known effect

Question: I have a timeseries dataset collected by sensors. The data is crack displacement (disp) and temperature (temp). Thermal expansion is linear with temperature, so it causes a diurnal signature in my displacement data. I would like to remove the effect from temperature. I'm an engineer, not a statistician, so I am uncertain that I am doing it correctly.
I've used R's nls function to remove the known effect of temperature. I used a sin wave with 24-hour period. I also used a shift parameter (phi) in the function to shift the wave (because the temp sensor responds faster than the disp sensor which is mounted on concrete that takes awhile to heat and cool).
My questions are:  Am I doing this correctly? Is there a better way?
Note that I do not want to predict crack displacement, I want to remove effects of temperature and see the underlying 'unexplained' displacement.
I've developed a pseudo time series that somewhat mimics my data.
Here is my code (requires package lubridate):
# create 1 hour timeseries 1 week long
dat <- data.frame(TIMESTAMP = seq.POSIXt(from = as.POSIXct('2020-03-01'),
                                          to = as.POSIXct('2020-03-08'),
                                          by = '1 hour'))

# Add displacement data and temperature data.
# Use a sinusoidal pattern with random element
set.seed(1)
dat$disp <- sin(as.numeric(dat$TIMESTAMP)) + runif(n = nrow(dat), min = -0.2, max = 0.2)
dat$temp <- sin(as.numeric(dat$TIMESTAMP)) * runif(n = nrow(dat), min = 0, max = 5) + 20
plot(dat$TIMESTAMP, dat$disp)
plot(dat$TIMESTAMP, dat$temp)

# Add a trend to displacement - this is the unexplained displacement that I am looking for
dat$disp <- dat$disp + seq(from = 0, to = 1, length.out = nrow(dat))*rnorm(1)

# Equation for nls:
# D = b0 + b1*T + b2*sin(omega*t + phi)
    # D = predicted displacement
    # T = temperature
    # omega = angular frequency = 2*pi / 24
    # t = time of day (1...24)
    # phi = time shift between temperature and displacement
    # b0, b1, b2 are fitting parameters

# Provide starting parameters for nls()
b0 <- 1.0
b1 <- 1.0
b2 <- 1.0
phi <- 20

# create time of day (tod) series
library(lubridate)
dat$tod <- hour(dat$TIMESTAMP)

# model
fit <- nls(disp ~ (b0 + b1 * temp + b2 * sin(2 * pi * tod / 24 + phi)),
           start = list(b0 = b0, b1 = b1, b2 = b2, phi = phi),
           data = dat)

# assign modeling results (residual and fitted points) to dat
dat$res <- resid(fit)
dat$fit <- fitted(fit)

# regression on residuals to get the slope
fit2 <- lm(dat$res ~ dat$TIMESTAMP)

# plot original data
plot(dat$TIMESTAMP, dat$disp)

# plot residuals (the 'unexplained' part of displacement)
points(dat$TIMESTAMP, dat$res, col = 'red')

# plot the fitted values (the 'thermally explained' part of displacement)
points(dat$TIMESTAMP, dat$fit, col = 'green')

# add slope line through the residuals (movement not caused by thermal effects)
abline(a = fit2$coefficients[1], b = fit2$coefficients[2])
```

Answer: Your approach doesn't look like a valid one, because you "assume the conclusion" by specifying the sin time dependency. You can try one of 2 options instead:

Use Generalized Additive Model with cyclical splines from mgcv package.
Get the displacement due to temperature expansion from physics theory (temperature at time with appropriate shift) * coefficient of thermal expansion for the relevant material (concrete?) and just subtract it from actual observation

If the time lag between the temp and disp censors is non-negligible, you need to account for it with a model like disp_t = b0 + b1 * lag(temp_t, temp_disp_lag) + .... If this lag is unknown, then this is a really big problem and I am afraid there is no good solution.
Explanation:
The displacement linearly depends on temperature and on "something else":
disp_t = b0 + b1 * temp_t + u_t
You want to plot "something else" (displacement unexplained by temperature). If you somehow estimate b0 and b1, this is simply:
u_t = disp_t - b0 - b1 * temp_t
The effect that interests you is linear, so you don't need NLS. As a first idea you could try running a simple OLS:
m = lm(disp ~ temp, data = dat)
dat$dispOther = dat$disp - predict(m, newdata = dat)

However, the problem here is that u_t is autocorrelated and not iid (if it were iid, why would you want to plot it?) and even worse, it is correlated with time (that's why you want to plot it) and via time with temperature, so you have endogeneity and the OLS estimates are inconsistent. If you knew the functional form for the u_t dependency on time (u_t = f(t) + e_t, where e_t is stationary noise uncorrelated with temp_t and t), you could model and estimate b0, b1 and f parameters with an equation like this:
disp_t = b0 + b1 * temp_t + f(t) + e_t
If you assume that f(t) = sin wave with 24 hour period, you can estimate everything with nls like you did, but the result would be as good as this assumption. The assumption is likely wrong. Do you really know a priori that whatever process you are trying to measure with displacement sensors has exactly sinusoidal dependency on time besides the temperature dependency?
You can get a non-parametric estimate for the time dependency with mgcv package.
library(mgcv)
m = gam(disp ~ temp + s(t, bs="cp",...), data = dat)
dat$dispOther = dat$disp - coef(m)['(Intercept)'] - coef(m)['temp']

This would approximate the dependency of displacement on time with a smooth cyclical function (with a bit of magic). I don't remember right out of my head how to specify cyclical splines in mgcv, but you can check it here
A completely different approach would be to avoid any statistical estimations at all and get the displacement change due to temperature from physics and some reference table for relevant materials. This could be trivial or impossible depending on your specific case.
Comment: Wow, thanks, great answer. I'll start digging through and see what I come up with and will mark as correct after.
Comment: You said "Your approach doesn't look like a valid one, because you 'assume the conclusion' by specifying the sin time dependency."  What if I'm super confident that outdoor temperature, measured long term, DOES display a diurnal sinusoidal pattern? Do I need to prove this again?
Comment: If you can reasonably assume that temp is reasonably represented as sin + noise, then it helps a lot. I just thought that this is unreasonable assumption, but you have the data and you know better. You can just fit Temp ~ hour, plot the residuals and see if sin is a reasonable approximation.
Comment: Actually, the biggest question is what do you expect to find in the unexplained displacement. If this is just some occasional spikes, possibly autocorrelated, but without deterministic diurnal pattern, then just do OLS `m = lm(disp ~ temp, data = dat)` and look at the residuals, there is no need for anything more complicated. If this is some diurnal pattern, then it is pretty hard to disentangle it from temp diurnal pattern and knowing that temp has a diurnal sin pattern, but unexplained displacement doesn't have a sin pattern helps a lot and you can proceed as you have suggested.
Comment: Awesome that you are still looking at this! I am also. The `dat$disp` (the crack-width data) is from cracks in a concrete dam (or in granite, in another of my projects). They display diurnal movement highly-correlated `dat$temp` (measured with the same sensor), which, engineering-wise, makes perfect sense. The unexplained displacement would be a movement in the dam or granite caused by other-than-temperature, which would be a permanent offset in `data$disp`. It is this permanent offset that I am trying to measure.
Comment: Another difficult part to this is that the crack widths display not only hourly temperature dependence, but seasonal temperature dependence. This is because of the massive nature of the structure. E.g., on a warm sunny winter day, the sensor temperature might by relatively high, as well as the first few inches of dam face, but the massive dam structure is still cold internally. One way I get around this in one of the projects is to use only temperature data collected at 4:00 am (reducing thermal effects on the sensor from direct sunlight).
Comment: ok, so the *actual* question is totally different!You are interested in long-term (scale in days-months) persistent changes in disp. Don't worry about intraday patterns, just take the average daily disp or your 4:00am as a clever hack. The actual problem is yearly seasonality (summer => higher temp => bigger disp doesn't mean cracks). You can measure the disp~temp dependence by regressing day on previous day changes in average daily disp on changes in average daily temp. Then plot daily observations for disp - b * temp, where b = coef from the last regression = empirical disp ~ temp dependence


How do I find services?

Question: What is the best way to find right service for dependency injection? Googling is not efficient yet. For example I spent quite long time figuring out that if I need current request passed to constructor by DI I have too use: $container->get('request_stack')->getCurrentRequest()
Full code example:
/**
* {@inheritdoc}
*/
public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition) {
    return new static(
        $configuration,
        $plugin_id,
        $plugin_definition,
        $container->getParameter('serializer.formats'),
        $container->get('logger.factory')->get('job_market'),
        $container->get('current_user'),
        $container->get('request_stack')->getCurrentRequest()
    );
}

And I've found above example in drupal class Documentation
Before that I've just found this
services list
but there is no description, no examples, nothing. Just a list with possibility to search.
How do you search for right service provider?
Answer: A better way to accomplish this is to use the Drupal Console's  container:debug command. This command outputs a list of Drupal service ID's and its corresponding namespace & class.
$drupal container:debug

 Service ID                 Class Name
 class_loader               Symfony\Component\ClassLoader\ApcClassLoader
 kernel                     Drupal\Core\DrupalKernel
 service_container          Drupal\Core\DependencyInjection\Container
 cache_context.ip           Drupal\Core\Cache\Context\IpCacheContext
 cache_context.headers      Drupal\Core\Cache\Context\HeadersCacheContext
 cache_context.cookies      Drupal\Core\Cache\Context\CookiesCacheContext
 ...

This command can be employed along with grep to search for a specific class name or service ID. For example: drupal container:debug | ag 'DatabaseStorage' to find a DatabaseStorage class.
Also of interest is the PhpStorm / IntelliJ Drupal Symfony Bridge plugin if you use PHPStorm.
Comment: That's a good one but still it's missing description. So for example when you need current Request you have to use request_stack which is not so obvious. Description will definitely do the trick. For example: request_stack - allow you to get current request and do bunch of other stuff :)
Answer: If you are using drush and the devel module, you can also use 'drush devel:services'.
For example:
# drush dcs datab
- cache.backend.database
- database
- database.replica
- database.replica_kill_switch
- database_driver_uninstall_validator
- drupal.proxy_original_service.database_driver_uninstall_validator
- keyvalue.database
- keyvalue.expirable.database
- queue.database

It's not as cool as the console command because it doesn't give you the class name.
More ways to do it are also on this later question Get the list of available services

How do I find services?


Shortcut for differentiating $y=\sin^{-1}\left(\frac{a+b\cos(x)}{b+a\cos(x)}\right)$

Question: We are required to differentiate (wrt to x):
$$y=\sin^{-1}\left(\frac{a+b\cos(x)}{b+a\cos(x)}\right)$$

Background:
I know how to solve for $\frac{dy}{dx}$ manually (simple application of the chain rule of course). But I was told that there's a simpler way to solve this problem, and can't seem to find the shortcut.
For instance, if we are required to differentiate $y=\tan^{-1}\left(\frac{\sin(x)}{1+\cos(x)}\right)$, we wouldn't directly start to apply the chain rule; we would, instead, notice that arctan's argument can be simplified to $\tan\left(\frac{x}{2}\right)$, hence reducing the problem to simply differentiating $\tan^{-1}\left(\tan\left(\frac{x}{2}\right)\right)$, i.e. finding the derivative of $\frac{x}{2}$, which is simply 0.5.
{Note: Function is defined for all real nos. where $\cos(x)\neq -1$}

To reiterate, I'm just asking whether or not there exist shorter methods to solve the posed problem (involving usage of some trig identity, some substitution, etc etc).
Comment: is this $$arcsin(..)?$$
Comment: @Dr.SonnhardGraubner That's right.
Comment: @Kugelblitz I've found some simplification, but I don't know if it's faster than yours. Can I post it and get feedback anyway?
Comment: Sure! No problem
Comment: So far, the intended simplification eludes me, even after cheating: I anti-differentiated the derivative and didn't see anything particularly simple. There are indications that the substitution $a = b \cos c$ could be useful. With this, it's possible to write $$y = \frac{\pi}{2} - 2\tan^{-1}\left(\tan\frac{c}{2}\tan\frac{x}{2}\right)$$ but this won't let you avoid Chain Rule complications when differentiating. I hope that whoever told you there's a simpler way also tells you what that simpler way is; then you can come back here and tell us. :)
Answer: Let define
$$h(x)=\cos(x),$$
$$g(x)=\frac{a+bx}{b+ax},$$
$$f(x)=\arcsin(x),$$
and your function
$$y=f(g(h(x))).$$
then
$$y'=f'(g(h(x))).g'(h(x)).h'(x).$$
with
$$f'(X)=\frac{1}{\sqrt{1-X^2}},$$
$$g'(X)=(\frac{bX+a}{aX+b})'=\frac{b^2-a^2}{(aX+b)^2},$$
and of course
$$h'(x)=-\sin(x).$$
Comment: Could you please elaborate?
Comment: And this doesn't seem to be much simpler.
Comment: This is by the use of chain. OP already knows that.


Simulate a Gaussian Copula with t margins

Question: The task is the following:
Given is $Z_1,...Z_{50}$ different hypothetical assets.
Each $Z_k \sim t_3$ with standard deviation $\sigma=0.01$ and $\tau(Z_i,Z_k)=0.4$ for $j\neq k$.
I want to simulate from the distribution $(Z_1,...,Z_{50})$ assuming that it has a Gaussian Copula with $t_3$ margins.
For what I understand, from the kendals tau $\tau$ I can estimate the correlation parameter $\rho=sin(\pi\tau/2)=0.5878$. From this I can determina the covariance matrix when I do want to do the simulation. Here is the code:
require(mvtnorm)
N <- 1000
tau <- 0.4
correlation <- sin(pi*tau/2)
std <- 0.01

S <- matrix((std^2)*correlation,50,50
for (i in 1:50) {

        S[i,i] <- std^2

}

gauss <- rmvt(N, sigma = S, df = 3)
U_norm <- pnorm(gauss)
Z_gauss <- qt(U_norm,df=3)
plot(Z_gauss[,1],Z_gauss[,2])

However, this method is not correct. I cannot see any plot that resembles a gaussian copula. Any thoughts, corrections?
Comment: Could you elaborate on what you mean by "cannot see any plot that resembles a gaussian copula"?  I am puzzled by this because (1) there are no plotting commands or images in your post and (2) this copula exists in 50 dimensions.  I am also curious why in your penultimate line `pnorm(gauss)` you do not refer to `S`.  Incidentally, typos and missing commands (such as one that defines `rmvt`) in your code will make it impossible to execute.
Comment: rmvt is for generating from multidimensional t distribution. I added the library in the top. Also, added plotting code.
Comment: Great, thank you.  May I direct your attention to the one part of my comment you have not yet addressed?  It may hold the key to the problem: why are you applying `pnorm`, which is the distribution function of a *standard* Normal variate, when `gauss` contains variables whose marginal distributions are given by the *nonstandard* covariance matrix `S`?
Comment: Not sure... how should it in that case be done? I use pnorm for since I want to create a Gaussian Copula, however, i just checked that the histogram of U_norm is NOT uniformly distributed. Any help what to do?
Answer: There's an R package called "copula" that will let you do exactly this.
The process goes:

Specify a copula
Specify the population distribution, including whatever marginals you want. From the documentation: "A user-defined distribution, for example, fancy, can be used as margin provided that dfancy, pfancy, and qfancy are available."
Generate samples from that multivariate distribution.

For you, you would specify a Gaussian copula in step 1 and then say that you want t-distributed marginals in step 2.
# Step 1
#
my_copula <- normalCopula(0.8)

# Step 2
#
my_population <- mvdc(my_copula, c("t","t"),list(t=3,t=3))

# Step 3
#
my_sample <- rMvdc(1000,my_population)

Caveat: I don't have access to this package right now, so I can't swear that this will compile, though it gives the gist of what to do.
Answer: I believe the approach you're attempting to implement would proceed as follows:

Generate sample from a multivariate normal with the desired correlation matrix
transform each margin from normal to uniform (in R, pnorm is suitable), to obtain a sample from the required copula
transform each margin from uniform to the desired $t$ distribution (in R, qt will do that)
Comment: +1. Watch out, though: `qt` will produce infinite values especially with small degrees of freedom. The extreme t tails will make it practically impossible to gauge whether the resulting multivariate distribution follows a Gaussian copula. One should study the distribution of the *uniform* versions (in 2) rather than the distribution of the t variates (in 3).
Comment: @whuber the question asks for t-margins, so I don't see how to avoid attempting some transform from normal to $t_3$; of course one might try to implement a more direct approach there. However, I just generated 10 million values from a uniform and transformed via `qt` without getting a single non-finite value returned. How often are you seeing it?
Comment: It looks like making use of `lower.tail=FALSE` in `qt` with a check for being in the upper half will make it an event that would happen less than once in a googol times for each tail. When I get back I'll look at whether to lay that out in detail.
Comment: With df=3, I was seeing it relatively frequently.  I didn't check, but it was on the order of a percent or so.  I was using a minor variation of the OP's code, corrected to use the proper SD and to generate 6-D vectors rather than 50-D vectors (which are much harder to examine in detail!)
Comment: @whuber But nothing so complex is required to check the effect of `qt`  -- each marginal will be uniform and we're only transforming the margins, so we only have to check the frequency with which `qt(runif(...),3)` has a problem.
Answer: Here I will try to answer this. I am using Python code for ease.
THEORY
First, lets explicitly define your correlation matrix
$$R^{50\times50} =
\begin{bmatrix}
.01&.4 &\cdots&.4 \\
.4 & \ddots&  &.4 \\
\vdots&&\ddots &\vdots\\
.4 &\cdots &\cdots &.01
\end{bmatrix}$$
Then, since we are using a Gaussian copula to model the joint distrubtion of $z_1,...,z_{50}$, we have: $C_{\textit{Gauss}}(u_1,...,u_{50}) = Pr[U_1 \leq u_1, ..., U_{50}\leq u_{50}] = \Phi_R(x_1, ..., x_{50})$, where $\Phi_R$ is the multivariate normal CDF with correlation matrix $R$ and mean $\mathbf{0}$, $x_i$ is a Gaussian standard normal random variable, and $\Phi(x_i) = u_i$ is the standard normal CDF. Often this copula is written as
$$C_{\textit{Gauss}} (u_1,...,u_{50}) = \Phi_R(\Phi^{-1}(u_1),...,\Phi^{-1}(u_{50})),$$ where $\Phi^{-1}$ is the inverse standard normal CDF, thus $\Phi^{-1}(u_i)$ gives us a standard normal random variable $x_i$!
Lastly, we note that $u_i = F(Z_i)$, where $Z_i \sim t_3$ is your t-distributed random variable and $F$ is the CDF of your t-distrubted random variable. Thus, with a Gaussian copula, we first calculate the CDF of your random variable ($Z_i$ in this case), which gives $u_i$. We then take the inverse standard normal CDF of $u_i$, which gives us a standard normal random variable $\Phi^{-1}(u_i) = x_i \sim \mathcal{N}(0,1)$. We do this for each $Z_i$. Then after getting all the standard normal variables $x_1,...x_{50}$, we look at the CDF of their joint distribution. Because each $x_i$ is a standard normal random variables, the joint distribution of all $x_i$'s is a multivariate normal distribution! Thus, the CDF of this joint distribution is $\Phi_R$, the reason we can use a correlation matrix as our covariance matrix is because each $x_i$ has mean 0, and variance 1, so the covariance matrix is already normalized to 1!
PROCEDURE
With all of this in mind. We can now write the procedure for sampling from this copula.

Sample $u_1, ..., u_{50}$ from your Gaussian copula. $$u_1, ..., u_{50} \sim C_{\text{Gauss}}(u_1,...,u_{50}).$$ Here are the steps to do this as your copula is Gaussian
a. Since $C_{\text{Gauss}}(u_1,...,u_{50}) = \Phi_R(x_1, ..., x_{50})$, as described above, first sample $x_1,...,x_{50}$ from a joint normal multivariate distribution with mean $\mathbf{0}$ and correlation $R$
$$ x_1,...,x_{50} \sim \mathrm{Multi}(0,R).$$ There are many Python and R packages that can do this quickly and easily.
b. Once you have $x_1,...,x_{50}$, take the standard normal CDF of each $x_i$
$$ \Phi(x_1),...,\Phi(x_{50}) = u_1,...,u_{50}.$$ This will give you your uniform marginals $u_1,...,u_{50}$
Now that you have $u_1,...,u_{50}$, note for each of your t-distributed $Z_i$, that $F(Z_i) = u_i$, where $F$ is the CDF of the t-distribution. Thus to get $Z_1, ...,Z_{50}$ simply take the inverse of the CDF for each $u_i$.
         $$F^{-1}(u_1), ... , F^{-1}(u_{50}) = Z_1, ..., Z_{50}$$

Implementation
Here is the implementation of the above steps in Python
'''Simulate from a Gaussian Copula with t-margins
variance = .01
variance amongst covariates (correlation)  = .4
'''
import numpy as np
import scipy.stats as ss

n = 1 # The number of samples
p = 50 # The number of covariates
# the mean of the multivariate Gaussian CDF is zero
mu = np.zeros(p)
The correlation matrix R with R_ii = .01, R_ij = .4
correlation_matrix_r = np.zeros((p,p))
for i in range(n):
     for j in range(n):
           if (i != j):
                 correlation_matrix_r[i,j] = .4
           else:
                 correlation_matrix_r[i,j] = .01

# Step 1: sample u_1, ..., u_50 from copula

# a. Get standard normal random variables x_1,...,x_50 ~ N(0,1)
x_is = np.random.multivariate_normal(mu, correlation_matrix_r, size = (n,p))

# b. Get the uniform marginals u_1,..,u_50 from the standard normal random variables
# Do this by taking the standard normal CDF for each x_i
# ss.norm(x_is) will take the the standard normal CDF of each element in x_is
uniform_marginals_u = ss.norm(x_is)

# Step 2: The invese t-distrubtion CDF of each uniform marginal to get your samples
# F^{-1}(u_1),...,F^{-1}(u_50) = Z_1, ...,Z_50

# get t-distrubtion object from scipy, with specified parameters
t_3 = ss.t(df = 3)
t_dist_samples_z = t_3.ppf(uniform_marginals_u) #ppf is the inverse cdf

This code will give you $n$ samples of $Z_1,..Z_{50}$.
$$\begin{bmatrix}
\{(Z_1,...Z_{50})_1 \\ \vdots \\ (Z_1,...Z_{50})_n\end{bmatrix}$$
Hope this helped!


Valor random único

Question: Já vi vários posts acerca do assunto mas não é exactamente o que eu procurava.
Há um tempo atrás quando estive em estágio, disseram me para guardar alguns registos com um id random único em cada um, e disseram-me o nome daquele tipo de funções ou algo do género( infelizmente não me lembro como se chamava, dai estar aqui :d ).
Lembro que retornava um valor com letras e números, vejam o exemplo de como era retornado aquele valor, Xe123e-DE3raw-23weEa-312dWe era algo deste género.
Se alguém souber de algo parecido, ou nome deste tipo de funções etc. é de grande ajuda, obrigado
Comment: Será [UUID](https://en.wikipedia.org/wiki/Universally_unique_identifier)?
Comment: Complementando o comentário do @ramaral https://duckduckgo.com/?q=uuid
Comment: Idéia: o melhor era se o @ramaral colocasse em formato de resposta. Pq do jeito que está, esta questão aparece nas não respondidas, além dele não ganhar o crédito devido pela resposta. :)
Comment: Sim, penso que sim, mesmo que não fosse esse que usei aqui há algum tempo, esta solução resolve o meu problema, muito obrigado por responderem a este post ;) abraço
Answer: Julgo que aquilo que procura é um Universally Unique Identifier (UUID).
UUID é um número de 128 bits que na sua forma canónica é expresso por 32 dígitos hexadecimais, divididos em cinco grupos separados por hífen, com a seguinte distribuição: 8-4-4-4-12.
Ex.: 123e4567-e89b-12d3-a456-426655440000

Valor random único


When is the sign of inner products preserved?

Question: I'm interested in the following question:

Let $E$ be a real Euclidian space. What are the linear transformations $f$ of
  $E$ that preserve the sign of inner products? That is, for all vectors $x$
  and $y$,
  $$ x \cdot y > 0 \iff f(x)\cdot f(y) > 0.$$

One can solve this by using polar decomposition and diagonalization, but I am looking for a more geometric proof. Any idea?
Comment: I guess you mean $f(x)\cdot f(y) >0$? You wrote two x's instead of an x and a y...
Comment: @rschwieb: Indeed. Thank you.
Answer: If vectors begin perpendicular, they end up perpendicular.  So only rotations, reflections and multiples of those.
If $x.y=0$, then $x.-y=0$, so neither $f(x).f(y)$ nor $f(x).f(-y)$ is positive, so $f(x).f(y)=0$.
So the standard basis ends up as an orthogonal set of vectors.
Combine $f$ with a rotation/reflection $g$ that rotates multiples of $fe_i$ back to multiples of $e_i$, so that $gfe_i=\lambda_ie_i$.  That is, all the standard vectors are now eigenvectors of $gf$, with real $\lambda_i$
Let $e_i$ and $e_j$ be any two of the vectors.
$$(e_i+e_j).(e_i-e_j)=|e_i|^2-|e_j|^2=0\\
f(e_i+e_j).f(e_i-e_j)=0=|f(e_i)|^2-|f(e_j)|^2\\=|gf(e_i)|^2-|gf(e_j)|^2=\lambda_i^2-\lambda_j^2$$
So $gf$ has all eigenvalues of equal absolute value.
Apply a rotation/reflection $h$ for which $he_i=e_i$ if $\lambda_i>0$ and $he_i=-e_i$ if $\lambda_i<0$.  Then $hgfe_i=|\lambda_i|e_i$.  Then $hgf$ is a multiple of the identity.  So $f$ is a multiple of $g^{-1}h^{-1}$, that is a multiple of a rotation/reflection.
Comment: What makes you say that?
Comment: Could you please elaborate on the *"So..."*? Moreover, I don't understand if your answer includes all elements of the orthogonal group?
Comment: The transformation described in the conclusion are usually called *[conformal linear transformations](https://en.wikipedia.org/wiki/Conformal_group)*. They're the subgroup of the general linear group generated by orthogonal transformations and dilations.
Answer: The linear maps in question form a subgroup $G$ of $\mathrm{GL}(\mathbb{R}^n)$. To see that $G \subseteq \mathrm{GL}(\mathbb{R}^n)$ notice that for every $f \in G$ and $x \in \mathbb{R}^n$ with $x \neq 0$ we have $x \cdot x > 0$ and thus $f(x) \cdot f(x) > 0$, so $f$ is injective and therefore bijective. The axioms of a subgroup can easily be verified. Notice that $O(\mathbb{R}^n) \subseteq G$.
For all $f \in G$ and $x,y \in \mathbb{R}^n$ we also have
\begin{equation}
 x \cdot y < 0
 \Leftrightarrow x \cdot (-y) > 0
 \Leftrightarrow f(x) \cdot f(-y) > 0
 \Leftrightarrow f(x) \cdot f(y) < 0
\end{equation}
and therefore also $x \cdot y = 0 \Leftrightarrow f(x) \cdot f(y) = 0$.
We claim that every $f \in G$ can be written as $f = \lambda g$ with $g \in O(\mathbb{R}^n)$ and $\lambda \in \mathbb{R}^\times$. To show this we set $\lambda_i := f(e_i) \cdot f(e_i) > 0$ for $i=1,\dotsc,n$ and define $h \colon \mathbb{R}^n \to \mathbb{R}^n$ to be the linear map defined by $h(e_i) = \sqrt{\lambda_i} e_i$ for $i=1,\dotsc,n$. It is clear that $h$ is an isomorphism and we set $g := f h^{-1}$. To see that $g \in O(\mathbb{R}^n)$ notice that for all $i,j=1,\dotsc,n$
\begin{equation}
 g(e_i) \cdot g(e_j)
 = \frac{f(e_i) \cdot f(e_j)}{\sqrt{\lambda_i \lambda_j}}
 = \delta_{ij} \frac{\lambda_i}{\sqrt{\lambda_i \lambda_j}}
 = \delta_{ij}
 = e_i \cdot e_j.
\end{equation}
All that's left to show is that $\lambda_1 = \dotsc = \lambda_n$ and thus $h$ is just given by multiplication with a (non-zero) scalar. To see this first notice that $h = g^{-1} f \in G$.  For all $i,j=1,\dotsc,n$ we have $v_1 := e_i + e_j$ and $v_2 := e_i - e_j$ with $v_1 \cdot v_2 = 0$ and thus
\begin{equation}
 0
= h(v_1) \cdot h(v_2)
= (\sqrt{\lambda_i} e_i + \sqrt{\lambda_j} e_j) \cdot (\sqrt{\lambda_i} e_i - \sqrt{\lambda_j} e_j)
= \lambda_i - \lambda_j.
\end{equation}
It is now clear that $G \cong O(\mathbb{R}^n) \times \mathbb{R}^n$.


Apache - where do I get older access logs? Are they even there?

Question: As the question says, I can find access logs upto 1 month old in the /var/logs/httpd folder. Where do I get older access logs from?
Are they even maintained/saved anywhere or deleted?
Answer: Have a look at your logrotate settings for those particular logs. You'll see the time interval for it to start a new log and how many copies it must keep. Anything prior to that gets deleted, so you would need to restore them from your backups.
Comment: @lostInTransit, logrotate is not part of Apache, it's a system utility, so you'll find its settings in /etc and/or a sub-directory of /etc. On my Debian system what you're looking for is in /etc/logrotate.d/apache2.
Comment: Thanks. Where do I see these? I don't see a rotate or a logrotate in my httpd.conf
Answer: Usually they are deleted.
It all depends on the specific distribution you're using, but most use logrotate to compress and delete old log files. This configuration should be in /etc/logrotate.conf and individual files in /etc/logrotate.d


Aguardar Thread Finalizar para continuar o código - Delphi

Question: Preciso aguardar o select e o preenchimento da lista finalizar na Thread para o Result da function receber a lista carregada no metodo assincrono, é possível?
Segue o código da function:
function PreencheCampos(const uniQuery: TuniQuery) : TList<iEstoqueLote>;
var
  lista : TList<iEstoqueLote>;
begin
  try
    Application.CreateForm(TFrmLoading, FrmLoading);
    FrmLoading.Show;

    lista := TList<iEstoqueLote>.Create;

    ThreadSql := TThread.CreateAnonymousThread(procedure
    begin
      uniQuery.Open;

      if not uniQuery.IsEmpty then
        begin
          while not uniQuery.Eof do
            begin
              aEstoqueLote := TEstoqueLote.Create;
              lista.Add(aEstoqueLote);

              if UniQuery.FindField('idestoquelote') <> nil then
                lista.Items[lista.Count - 1].idestoquelote := UniQuery.FieldByName('idestoquelote').AsInteger;

              if UniQuery.FindField('idestoque') <> nil then
                lista.Items[lista.Count - 1].idestoque := UniQuery.FieldByName('idestoque').AsInteger;

              if UniQuery.FindField('idproduto') <> nil then
                lista.Items[lista.Count - 1].idproduto := UniQuery.FieldByName('idproduto').AsInteger;

              if UniQuery.FindField('idwmsmovimentoitem') <> nil then
                lista.Items[lista.Count - 1].idwmsmovimentoitem := UniQuery.FieldByName('idwmsmovimentoitem').AsInteger;

              if UniQuery.FindField('idwmsunidadearmazenagem') <> nil then
                lista.Items[lista.Count - 1].idwmsunidadearmazenagem := UniQuery.FieldByName('idwmsunidadearmazenagem').AsInteger;

              if UniQuery.FindField('ddatavalidade') <> nil then
                lista.Items[lista.Count - 1].ddatavalidade := UniQuery.FieldByName('ddatavalidade').AsDateTime;

              if UniQuery.FindField('ddatafabricacao') <> nil then
                lista.Items[lista.Count - 1].ddatafabricacao := UniQuery.FieldByName('ddatafabricacao').AsDateTime;

              if UniQuery.FindField('nquantidade') <> nil then
                lista.Items[lista.Count - 1].nquantidade := UniQuery.FieldByName('nquantidade').AsExtended;

              if UniQuery.FindField('npesoliquido') <> nil then
                lista.Items[lista.Count - 1].npesoliquido := UniQuery.FieldByName('npesoliquido').AsExtended;

              if UniQuery.FindField('npesobruto') <> nil then
                lista.Items[lista.Count - 1].npesobruto := UniQuery.FieldByName('npesobruto').AsExtended;

              if UniQuery.FindField('nvolume') <> nil then
                lista.Items[lista.Count - 1].nvolume := UniQuery.FieldByName('nvolume').AsExtended;

              if UniQuery.FindField('ssituacao') <> nil then
                lista.Items[lista.Count - 1].ssituacao := UniQuery.FieldByName('ssituacao').AsString;

              uniQuery.Next;
            end;
        end;

      ThreadSql.Synchronize(nil, procedure
      begin
        FrmLoading.Release;
        FreeAndNil(FrmLoading);
      end);
    end);

    ThreadSql.Start;

    Result := lista;

  except on e: exception do
    begin
      TFrmTipoAviso.Mensagem('Tabela EstoqueLote : '+MensagemErroBanco(E.Message),'E',[mb_OK]);
    end;
  end;
end;

Comment: O que acha de utilizar mutex ou semaforo?
Comment: @LeonardoGetulio, são componentes de terceiros para o Delphi?
Comment: Não, não. São todos built-in. Vou te mandar uns links.
Answer: Quando se cria ThreadSql do tipo CreateAnonymousThread e da um start, elas vão ser executadas imediatamente, porém, o Result será acionado imediatamente também, ou seja, o cursor vai rolar sem esperar a ThreadSql terminar.
Para isto, tente algo como:
ThreadSql.Start;
...
while not ThreadSql.Finished do
  Application.ProcessMessages;

Ou seja, enquanto ela esta ocupada, fique por ali mesmo esperando.
Answer: Segue abaixo códigos de mutex e semáforos.
O Mutex cria uma trava único em memória que pode ser validado até mesmo por outros aplicativos não-delphi. Algumas vantagens são simples implementação e verificação até mesmo por outro aplicativo, e tem a desvantagem de permitir apenas 1 executor por vez.
O Semáforo MREW por sua vez faz validação de permissão através de objeto/variável local ou global. As vantagens de usar MREW é que você pode ter inúmeras threads executando operações de leitura sem lock no destino, o lock só acontece quando uma thread abre com permissão de escrita, é tipo o gerenciador de banco de dados, quando um tá escrevendo ninguém mais pode acessar até que ele termine, quando o escritor libera o restante pode acessar simultaneamente. A maior desvantagem é a implementação "um pouco" (quase nada) onerosa.
Link e exemplo mutex: http://www.planetadelphi.com.br/dica/7251/mutex---nao-deixar-que-sua-aplicacao-seja-executada-mais-do-que-uma-vez.-
program Cobrancas;

uses
Forms,
Controls,
Windows,

{$R *.res}

var
hMutex: THandle;

begin
hMutex := CreateMutex(nil, true, PChar('[{23B1EEFF-4775-4FA9-9C5-6BFEAC48F551}]'));
if (hMutex <> 0) and (GetLastError = 0) then
begin
Application.Initialize;
Application.CreateForm(TfrmPrincipal, frmPrincipal);
Application.Run;
end
else
MensagemDeAvisoH('Já Existe Uma Instância do Aplicativo Rodando...');
end.

Exemplo semaforo MREW (multi-read exclusive-write synchronizer):
//variável global
var
    SemaforoMREW: TMultiReadExclusiveWriteSynchronizer;

//inicializador
begin
    SemaforoMREW := TMultiReadExclusiveWriteSynchronizer.Create;
end;

//thread para leitura
begin
    SemaforoMREW.BeginRead;
    try
        //... operações de leitura
    finally
        SemaforoMREW.EndRead;
    end;
end;

//thread para escrita
begin
    SemaforoMREW.BeginWrite;
    try
        //... operações de escrita
    finally
        SemaforoMREW.EndWrite;
    end;
end;

Answer: Também podia chamar um método para tratar do fim do carregamento ali antes do form.release, dentro da zona do syncronize. Acabava for ser um callback a notificar o fim da thread e até é mais performant porque nao eram necessários loops à espera que a thread terminasse


Como decompor uma Série temporal usando uma frequência de 6 meses?

Question: Tenho uma amostra de dados com o período de um ano. Eu gero a série temporal sem problemas com a frequency = 12, porém quando vou usar o decompose ele mostra a seguinte mensagem de erro:
Error in decompose(teste_compras) :
  série temporal não tem período, ou tem menos de 2

Pelo que eu entendi minha série temporal tem uma frequência de 12 meses, mas eu preciso de uma amostra com pelo menos 2 anos para ter 2 períodos. Porém, minha amostra tem apenas 12 meses e não possui mais dados anteriores.
O que eu gostaria de saber, se é possível gerar uma série temporal com frequência de 6 meses para pode gerar dois períodos de 6 meses e assim poder decompor a série. É possível?
Answer: Isso pode ser feito computacionalmente, mas não faz sentido matematicamente.
Imagine que tu tenha as temperaturas médias mensais de Porto Alegre, uma cidade com verão e inverno bem definidos. Mas tu tem apenas um ano de observações (ou seja, n=12). Sendo dados de temperatura, faz sentido imaginar que a sazonalidade destes dados é anual. Afinal, os janeiros na cidade tendem a serem quentes, enquanto os julhos tendem a serem frios.
Sazonalidade significa que o comportamento das observações tendem a se repetir depois de um período fixo de tempo. Não faz sentido, em termos de modelagem, dividir uma série como esta em dois períodos de seis meses. Afinal, uma sazonalidade de seis meses implicaria que o comportamento em janeiro e julho seriam similares; em fevereiro e agosto também; em março e setembro e assim por diante. E, ao menos para uma série de temperaturas, sabemos que o comportamento destes pares de meses são opostos: a temperatura aumenta de janeiro a fevereiro, mas baixa de julho a agosto.
O teu problema não tem solução respaldada na teoria de séries temporais incluindo sazonalidade no período. Ou mais dados devem ser coletados (e, mesmo assim, são bem mais dados: se sugerem de 50 a 100 observações para ajustar um modelo de séries temporais), ou desista da sazonalidade. Afinal, com 12 observações, uma sazonalidade de período 12 sequer vai ser detectada pela função de autocorrelação.
Comment: Entendi. Muito obrigado pela resposta.


Why logarithmic scale for hyper-parameter optimization?

Question: I'm using random search for hyper-parameter optimization of a machine learning pipeline. For example, for the C and gamma parameter it is recommended to use  logarithmically spaced values. Why should I use such values? For example, if I use logarithmic spaced values from $2^{-5}$ to $2^{15}$, then there will be many more values near to $2^{-5}$ (i.e. near zero) than near to $2^{15}$.
Comment: Why not make draws from an extreme value distribution such as Tweedie, Cauchy, and so on (see https://en.wikipedia.org/wiki/Tweedie_distribution#Examples)? This would almost certainly provide a better fit to heavy-tailed information and their optimization.
Answer: ... because logarithmic scale enables us to search a bigger space quickly. In your SVM example, we do not know the range for the hyper-parameter. So, a quicker way is trying dramatically different values, say, 1, 10, 100, 1000, which come from a logarithmic scale.
In addition, I think log scale search is the first step. Suppose that we found C=10 is better than C=1 or C=100; then we can focus on that scale to try a better value.
Another reason is for "regularization" parameters, such as C in svm. It is not too sensitive. In other words, we may not find too much difference with 10 or 15, or 20, but results would be very different from 10 to 1000. That is why we start with log search.
Comment: We use log scale for hyper-parmaeter optimization because the response function varies on a log scale. Compare a false-color plot of the hyper-parameter surface for RBF length-scale and C on log and linear scales. On a linear scale, in many problems, the plot is basically the same everywhere because the difference between C=10 and C=11 is basically zero, but the difference between C=10 and C=100 is more pronounced.
Comment: I think the first paragraph is incorrect. Suppose your budget is $k$ different values of $C$. We can search the space $[1,10^3]$ by checking $k$ points equally spaced on a linear scale. If the space is expanded to $[1,10^6]$, this is still true, we can still check $k$ values, except they'll be farther apart. But your point that the difference in performance is more important on a log scale is the heart of it: performance at $C=10\approx11$, but $C=100$ will be different.
Comment: But does this make sense for random search?
Comment: @machinery to do random search, we still need a range. Say from 1 to 1000 uniform ?
Comment: Yeah you need a range but should the range be uniform or logarithmic?


TFS 2015 - Public View of Backlog Board

Question: Is there any way to set up a public view (View Only) of the Product Backlog Board for the customer see what we are working on and so?
It would be awesome too, if the customer could vote for backlog item that we haven't start working on it.
Trying to get the same concept of https://trello.com/ in TFS
Answer: The first thing (View Only) of the Product Backlog Board for the customer is easy to achieve. You just need to set related permissions for your customer or customer group.
Such as assign the user stakeholder access which have permission  to view backlogs.

Assign Stakeholder access to those users who need to enter bugs, view
  backlogs, boards, charts, and dashboards, but who don't have a TFS
  CAL. Stakeholder access is free. Stakeholders can also view releases
  and manage release approvals. See Stakeholder access for details of
  features available to stakeholders.
Source Link: Change access levels

Then you need to deny some permissions which you don't want custom to have.
As for the second  thing, there is no this kind of feature in TFS for now. Even though  you have ability to prioritise Backlog items by dragging and dropping the item. If you need this feature, you can sumbit a feature requst in uservoice stie, TFS Admin and PM will kindly reveiw your suggestion.

TFS 2015 - Public View of Backlog Board


Adding form to .tpl

Question: This is definitely being called
function studiga_theme_theme($existing, $type, $theme, $path) {
    $items['add_property_form'] = array(
        'render element' => 'form',
        'template' => 'node--14',
        'path' => drupal_get_path('theme', 'studiga_theme') . '/templates/node',
    );

    return $items;
}

But which should set up the form fields isn't and I have no idea why
function studiga_theme_preprocess_add_property_form(&$variables) {

  // Shorten the form variable name for easier access.
  $form = $variables['form'];

  $form['first_name'] = array(
        '#type' => 'textfield',
        '#attributes' => array('placeholder' => t('First name')),
    );
    $form['last_name'] = array(
        '#type' => 'textfield',
        '#attributes' => array('placeholder' => t('Last name')),
    );
    $form['address'] = array(
        '#type' => 'textfield',
        '#attributes' => array('placeholder' => t('Address')),
    );
    $form['city'] = array(
        '#type' => 'textfield',
        '#attributes' => array('placeholder' => t('City')),
    );
    $form['state'] = array(
        '#type' => 'select',
        '#options' => array(
            'default' => 'State',
            '...' => '...',
        ),
    );
    $form['zip'] = array(
        '#type' => 'textfield',
        '#attributes' => array('placeholder' => t('Zip')),
    );
    $form['email'] = array(
        '#type' => 'textfield',
        '#attributes' => array('placeholder' => t('Email')),
    );
    $form['phone'] = array(
        '#type' => 'textfield',
        '#attributes' => array('placeholder' => t('Phone')),
    );
    $form['submit'] = array(
        '#type' => 'submit',
        '#value' => 'Submit',
    );
}

Thanks
Comment: Can you elaborate on the problem you're solving? Adding forms to templates in the theme layer is usually not the best solution: The theme layer is to control how your site looks, not how it works.
Answer: As marcvangend mentioned already, theming functions are not the right place for altering forms.
I suppose you are using Drupal 7 and like to alter a form that has already been added to the node before. In this case you should have a look into hook_form_alter() or hook_form_FORM_ID_alter():
https://api.drupal.org/api/function/hook_form_alter/7
https://api.drupal.org/api/function/hook_form_FORM_ID_alter/7
These hooks can be used within your custom module to alter forms even before theming takes place. For getting the ID of the form in question, just debug the $formid parameter of the hook_form_alter() while visiting the node.
For adding a custom form to your node you should write a custom module implementing the following hooks:
/**
 * Implements hook_block_info().
 */
function MYMODULE_block_info() {
  $block = array();
  $block['mymodule_form'] = array(
    'info' => t('My Form'),
  );
  return $block;
}

/**
 * Implements hook_block_view().
 */
function MYMODULE_block_view($delta = '') {
  $block = array();
  if ($delta == 'mymodule_form') {
    $block['content'] = drupal_get_form('MYMODULE_my_awesome_form');
    $block['subject'] = t('My Form Title');
  }
  return $block;
}

/**
 * My Form
 */
function MYMODULE_my_awesome_form($form, $form_state) {
  $form = array();

  // define your form here...

  return $form;
}

/**
 * My Form Submit Handler
 */
function MYMODULE_my_awesome_form_submit($form, $form_state) {
  // do your magic on the submitted form values in $form_state['values']
}

Then you will find the form within /admin/structure/block and can add it to the desired page region of your node. Filter the node within the block settings by choosing 'Only the listed pages' under Pages/Show block on specific pages and entering your node/14.
Comment: I just want to set up a form in node--14.tpl.php and capture what is posted, validate then write it to the database. Simple with PHP and I expect with Drupal if I knew how but I can't even display the form
Comment: For showing a new custom form on a node you should write a small module utilizing blocks.

(1) Use hook_block_info() to define a block.
(2) Use hook_block_view() to define its output.
(3) Return a custom form as output of your block.
(4) Add a submit handler for this form to use the submitted data.
(5) Add the block to your node using structure/blocks and filter to show on your node only.

It's quite an amount of code, so I will add it later to my above reply.
Comment: Please with just say a couple form fields as an example then I can just change it the fields I need as I am totally in the dark at the moment. I am a bit of a Newby
Comment: Don't worry. Noticed that already. ;) Please check my above edits. For a reference of how to define your form fields / submit buttons, I'd suggest https://api.drupal.org/api/drupal/developer!topics!forms_api_reference.html/7.
Comment: Right so I need a library called modules under the theme then another library called MYMODULE or whatever I want to call it then is it a MYMODULE.inc file under there?
Comment: You have to create a custom module located within sites/all/modules, not within your theme. For a tutorial about creating modules, please refer to https://www.drupal.org/node/1074360 or https://www.drupal.org/node/1419390.
Comment: Hmm - it shows up under modules but not under blocks
Comment: function add_property_block_info() {
  $block = array();
  $block['add_property_form'] = array(
    'info' => t('Add Property Form'),
  );
  return $block;
}
function add_property_block_view($delta = '') {
  $block = array();
  if ($delta == 'add_property_form') {
    $block['content'] = drupal_get_form('add_property_property_form');
    $block['subject'] = t('My Form Title');
  }
  return $block;
}
Comment: Did you enable the module? If so, the block should appear as "Add Property Form" within your available blocks. If you added the block hooks after you enabled the module, you might also have to clear your caches.
Comment: Thanks Can you add formatting? I want something like

Unique reference
i
Comment: The form also needs some tooltips eg                                                                                i   so can you add spans? Thanks
Comment: :) Have a look to the above mentioned Form API Reference: https://api.drupal.org/api/drupal/developer!topics!forms_api_reference.html/7. Using 'container' elements (or '#prefix' and '#suffix' attributes), you can achieve the div wrappers. Using an element's '#attributes' attribute, you can add CSS classes and placeholders.
Comment: You can also add custom HTML as your tooltips using the 'markup' element.
Comment: Ah right thanks. I knew about the css classes but the designer has used divs and spans so I thought I had better used the same. You have been a great help - thanks

Adding form to .tpl


Writing error in Beethoven's autograph of "moonlight" sonata?

Question: In Beethoven's op. 27/2 in the autograph, bar 20, in the left hand on musical beat 1, the lower tone seems to be d sharp ("dis").
https://www.beethoven.de/en/media/view/6442239777570816/scan/0
In the printed versions (i.e., Wiener Urtext) there is an e sharp ("eis").
The latter seems to be intended, since it would be an octave as an harmonic interval.
I cannot believe that Beethoven made a writing error here(?)
What is the explanation?
Comment: Side note: use IMSLP for the originals. Not that it makes a difference, but it's slightly more reliable.
Comment: @10Rep Reliable in what way exactly? The Beethoven Haus in Germany seems pretty reliable to me!
Answer: Beethoven made an error.

20.lh.1 Lower note is DD# [low D#] in all sources, a note that is harmonically absurd and was not even available on Beethoven's piano. It illustrates well the danger of placing too much faith in a composer's autograph score.

Cooper, Barry. 2007. Beethoven: The 35 Piano Sonatas, Volume 2 — Commentaries. London: ABRSM Publishing. Page 21.


facebook connect on iPhone - making calls is giving error 102 "Session key invalid or no longer valid", yet the stream dialogs still work

Question: Anyone see this?
I've a situation whereby sometimes, [session resume] returns YES.  I make a FBRequest.call, and the error delegate gets called.
The error has 102 "Session key invalid or no longer valid".
However, if within the same session I call up any FBDialogs (e.gr. to publish to a stream), these all work fine.  I don't see why they are not suffering the same error.
Any ideas?
Yes.  I know there's a forum for facebook connect, but its taking an age to get my password sent to me. (Yes...there's an irony there - why didn't they just use facebook connect for their forums...ho hum).
Comment: This doesn't answer your main question, but [session resume] only does a local check to make sure you have a saved session key and that its expiration date hasn't passed. It doesn't do any actual validation with Facebook.
Answer: <code>- (BOOL)resume {
  NSUserDefaults *defaults = [NSUserDefaults standardUserDefaults];
  FBUID uid = [[defaults objectForKey:@"FBUserId"] longLongValue];
  if (uid) {
    NSDate* expirationDate = [defaults objectForKey:@"FBSessionExpires"];
    if (!expirationDate || [expirationDate timeIntervalSinceNow] > 0) {
      _uid = uid;
      _sessionKey = [[defaults stringForKey:@"FBSessionKey"] copy];
      _sessionSecret = [[defaults stringForKey:@"FBSessionSecret"] copy];
      _expirationDate = [expirationDate retain];

      for (id<FBSessionDelegate> delegate in _delegates) {
        [delegate session:self didLogin:_uid];
      }
      return YES;
    }
  }
  return NO;
}
</code>
Here is the code of that method, I don't think that the code is trying to make the connection with facebook to test, they just get the expiration date. I think that sometimes, when you logout of facebook, the session does not expire but on server side, they delete your session


AngularJS - Manual compile vs natural compile

Question: It seems to be pretty generally accepted you shouldn't need to manually compile elements (via the $compile service) very often. Because of some dynamically driven html we have a directive that is doing a manual compile in the link function. Something like this (the actual retrieving of the dynamic html is much more involved):
<code>    link: function(scope, element, attrs) {
        element.append(someService.getDynamicHtml());
        $compile(element.contents())(scope);
    }
</code>
We've started to hit some performance issues, as there is a page that issues this directive several times, and each compile is taking 300 or so ms.
In an attempt to better understand what's going on "underneath the hood", and thus make better design decisions, my question is what is the difference between doing this kind of manual compile right in the middle of the link function, as opposed to providing the directive with a template and letting angular naturally compile it?
Here is a fiddle to demonstrate the two scenarios I'm talking about: http://jsfiddle.net/KNLea/
I think we can refactor our dynamic html generation so it constructs a string and than we can use it as the template. The above being changed to something like this:
<code>    template: someService.getDynamicHtml(),
    link: function(scope, element, attrs) {
        //no need for manual compile anymore
    }
</code>
I just want to understand what fundamental differences the two have, and thus what, if any, performance gain we can expect. Does it not make a difference at all? When angular compiles the template does it just do the same thing the $compile service does? Or is there an underlying difference in the natural compiling of the template that would make a difference in performance?
Comment: The first one does a link and then does a $compile, while the second one does them at the same time, the first one is just doing more overall.
Comment: in first case I think it's better to compile outside the DOM, then append compiled object. [Ran into this with recursive nested  list here](http://stackoverflow.com/questions/20461999/scope-of-event-arguments-in-angular-directive/20463126#20463126), was getting compiled too often and compounding event listeners
Comment: your manual compile outside DOM and using `replaceWith` http://jsfiddle.net/charlietfl/KNLea/2/
Comment: Within the link function, the children have already been compiled and linked.  If element has children, and you call $compile(element)(scope) after you've appended the HTML fragment, then the children are being compiled and linked twice.  There are no real performance gains between the two methods if you manually compile/link and avoid multiple compile/linking.  Also, moving what you can to the compile function is a good idea, especially if you are iterating over a collection and adding multiple elements to the DOM tree. Compile the element once, and link the clone multiple times.
Answer: Angular itself uses <code>$compile</code> to compile directive templates, so there is no inherent slowdown in compiling manually.
What does play an important role though, is when and how often you compile.
Angular will compile once for each directive instance and then just link as many times as needed.
You, on the other hand, make an additional compile+linking with each linking. That's a lot of overhead.
It is not quite clear how dynamic you dynamic HTML is, but if it stays the same for the lifetime of a session, there are much more efficient ways to compile.


Combinatorics deck/dice

Question:
A player has 52 cards and dice. Every time he picks a card (without seeing it) throws the dice. How many cards at least does he have to draw to make sure that discovering them there are at least 4 cards of the same suit and that the same number appears at least 3 times in the sequence of the dice throw?

I'm struggling with this problem. Can anyone help me?
Comment: If the player only had the die, or only the deck of cards, could you solve the problem?
Comment: @Arthur Yes, I suppose
Comment: And once you have the solution to both of those, can you see how to use that to solve this combined problem?
Answer: The player wants to be sure to get 3 times the same dice number and 4 times the same suit. Then, we can look for the longest series without 3 times any number nor 4 times the same suit. Next die and next card will add will add a repeated 3rd time number or a repeated 4t time suit.
With a 12 dice series we could get the numbers 1 to 6 tow times each - worst case. We can be sure that 13th die will show one of those 6 numbers that have already appeared.
Similarly, with a 12 cards series we could get at worst three card of each suit. The 13th card will be a repetition of one of those suits.
Therefore, a 13 cards and 13 dice series is enough.

Combinatorics deck/dice


Join Two Pivot Table and obtain multi value per cell in panda

Question: I have two pivot table with the same rows and columns, and I need create one table with the value separeted by comas in the cell with equal row and col.
For example
Table 1
       1    2    3    4
a     t1a1 t1a2 t1a3 t1a4
b     t1b1 t1b2 t1b3 t1b4

Table 2
       1    2    3    4
a     t2a1 t2a2 t2a3 t2a4
b     t2b1 t2b2 t2b3 t2b4

I want:
Table result
          1             2           3            4
a     (t1a1,t2a1)   (t1a2,t2a2)  (t1a3,t2a3)   (t1a4,t2a4)
b     (t1b1,t2b1)   (t1b2,t2b2)   (t1b3,t2b3)   (t1b4,t2b4)

The concat function return
        1    2    3    4   1    2    3    4
a     t1a1 t1a2 t1a3 t1a4  t2a1 t2a2 t2a3 t2a4
b     t1b1 t1b2 t1b3 t1b4  t2b1 t2b2 t2b3 t2b4

I work in python with pandas library
Thanks
Answer: You can use concanecation all DataFrames if need strings output:
df = '(' + df1 + ' , ' + df2 + ')'
#if numeric columns first cast to str
#df = '(' + df1.astype(str) + ' , ' + df2.astype(str) + ')'
print (df)
               1              2              3              4
a  (t1a1 , t2a1)  (t1a2 , t2a2)  (t1a3 , t2a3)  (t1a4 , t2a4)
b  (t1b1 , t2b1)  (t1b2 , t2b2)  (t1b3 , t2b3)  (t1b4 , t2b4)

If need tuples:
df = pd.concat([df1, df2],  keys=['a','b']) \
       .groupby(level=1).agg(lambda x: tuple(x))
print (df)
              1             2             3             4
a  (t1a1, t2a1)  (t1a2, t2a2)  (t1a3, t2a3)  (t1a4, t2a4)
b  (t1b1, t2b1)  (t1b2, t2b2)  (t1b3, t2b3)  (t1b4, t2b4)

Answer: here is one simple way
    df1 = pd.DataFrame(np.array([
    ['a','t1a1','t1a2','t1a3','t1a4'],
    ['b','t1b1','t1b2','t1b3','t1b4'],
    ['c','t1c1','t1c2','t1c3','t1c4']]),
    columns=['name', 'attr11', 'attr12', 'attr13', 'attr14'])
df2 = pd.DataFrame(np.array([
    ['a','t2a1','t2a2','t2a3','t2a4'],
    ['b','t2b1','t2b2','t2b3','t2b4'],
    ['c','t2c1','t2c2','t2c3','t2c4']]),
    columns=['name', 'attr21', 'attr22', 'attr23', 'attr24'])
df3 =pd.merge(df1,df2,on='name')

df3["attr1"] = '('+ df3['attr11']+ ',' +df3['attr21'] +')'
df3["attr2"] = '('+ df3['attr12']+ ',' +df3['attr22'] +')'
df3["attr3"] = '('+ df3['attr13']+ ',' +df3['attr23'] +')'
df3["attr4"] = '('+ df3['attr14']+ ',' +df3['attr24'] +')'
print (df3[['name','attr1','attr2','attr3','attr4',]])

Output
  name        attr1        attr2        attr3        attr4
0    a  (t1a1,t2a1)  (t1a2,t2a2)  (t1a3,t2a3)  (t1a4,t2a4)
1    b  (t1b1,t2b1)  (t1b2,t2b2)  (t1b3,t2b3)  (t1b4,t2b4)
2    c  (t1c1,t2c1)  (t1c2,t2c2)  (t1c3,t2c3)  (t1c4,t2c4)


Are "lower bound" and "upper bound" unambigous? What if one's in the negative axis?

Question: Are "lower bound" and "upper bound" unambigous? What if one's in the negative axis?
Consider e.g.
$$\{-n: n \in \mathbb{N} \}$$
This has no lower bound, if one consider lower to mean towards $- \infty$. However, what if one defined that the set is ordered so that one counts like $0,-1, -2, ...$. Then one could conceive that $0$ is the lower bound from the perspective of the index set. Not from the number greatness perspective though.
So is "lower bound" unambigous? Is my index set example "valid"? Perhaps it's not and I should rather rely on natural number axioms (Peano) and how they define order?
Answer: The definition of the concept "lower bound of a subset" involves statements about the ordering of elements. If you change the way the order is defined then you change which things are lower bounds. But there's no ambiguity in the definition.
The integers come equipped with a natural ordering defined from the Peano axioms. That's the one meant when lower bounds are discussed with no further qualification.


Como pegar apenas o valor de uma coluna no banco de dados

Question: Preciso pegar todos os registro de apenas uma coluna no banco de dados, e estou fazendo assim:
mycursor.execute("SELECT link FROM conteudo WHERE status = 'false'")

reviewlinks = mycursor.fetchall()

E ele funciona, exceto que o resultado vem dentro de (), por exemplo:
[('https://10reviewz.com/best-step-ladder-wirecutter/',), ('https://10reviewz.com/best-magnetic-stirrer-for-e-juice/',), ('https://10reviewz.com/best-luggage-cart-for-car-seat/',), ('https://10reviewz.com/best-marine-barometers/',), ('https://10reviewz.com/best-smart-scales-for-apple/',), ('https://10reviewz.com/best-golf-putters-for-alignment/',)]

Não entendo pq ele está retornando dessa maneira, pois preciso retornar apenas os links dentro do array, mas ele coloca o link dentro de () e ainda poem uma vírgula no final.
Pq isso está acontecendo ?
Answer: A definição do método fetchall é a seguinte:

O método busca todas (ou todas remanescentes) linhas do resultado de uma query e retorna uma lista de tuplas...

Haja visto que cada linha retornada pode ter mais que um campo, a vírgula do próximo elemento da tupla permanece ali mesmo que não exista (já que o conceito deste tipo de estrutura é justamente a imutabilidade).
Tendo isso em vista, considerando que cada linha retornará apenas um campo (link), você pode armazenar o primeiro valor de cada tupla (que será o link) em uma nova lista:
new_result = [f[0] for f in result]
print(new_result)

# ['https://10reviewz.com/best-step-ladder-wirecutter/',
# 'https://10reviewz.com/best-magnetic-stirrer-for-e-juice/',
# ...]

Comment: muito obrigado pelo esclarecimento e ajuda. (Y)