Trouble understanding proof of this proposition on contact type hypersurfaces

Question: I have finally started working on my thesis. I am reading the first reference book, which is Dusa McDuff, D. Salamon, Introduction to Symplectic Topology. It's a tough struggle, given my not-too-great experience with differential forms. I will recall a few concepts. A manifold $M$ is called symplectic if it is equipped with a closed non-degenerate differential 2-form $\omega$, which is also called symplectic form. A submanifold of $M$ is a subset $S\subseteq M$ such that the inclusion is an embedding, i.e. is a one-to-one immersion which is also a homeomorphism of $S$ onto its image. Alternately, it can be defined as a subset for which each point $p\in S$ has a neighborhood $U\subseteq M$ and a chart $\phi:U\to\mathbb{R}^n$ which is "adapted to $S$", that is for which $S\cap U$ maps exactly to the locus of points in $\phi(U)$ having the last $k$ coordinates all equal to 0. $\dim S$ is then $\dim M-k$. A differential 2-form is a field of bilinear and alternating forms, i.e. a map associating to each $p\in M$ a bilinear and alternating form on $T_pM$ such that locally, i.e. in a coordinate neighborhood, this filed is represented by $\sum_{i<j}a_{ij}(x^1,\dotsc,x^n)dx_i\wedge dx_j$ and $a_{ij}$ are differentiable. The exterior derivative is defined well enough on the linked Wikipedia page. That is, I'd say, just about all we need for the proof. Here are screenshots of the book containing the statement and proof of the troublesome proposition.

$\textbf{Proposition 3.58}$ Let $(M,\omega)$ be a symplectic manifold and $Q\subset M$ be a compact hypersurface. Then the following are equivalent.
$\bf (i)$ There exists a contact form $\alpha$ on $Q$ such that $d\alpha=\omega|_Q$.
$\bf (ii)$ There exists a Liouville vector field $X:U\to TM$, defined in a neighbourhood $U$ of $Q$, which is transverse to $Q$.
If these conditions are satisfied then $Q$ is said to be of $\textbf{contact type}$.
$\bf Proof:$ First assume that $\rm (ii)$ is satisfied and define $\alpha=\iota(X)\omega$. Then $$d\alpha=d\iota(X)\omega+\iota(X)d\omega={\cal L}_X\omega=\omega.$$ Since $T_qQ$ is odd-dimensional there exists a nonzero vector $Y\in T_qQ$ such that $\omega_q(Y,v)=0$ for every $v\in T_qQ$. Since $\omega$ is nondegenerate we have $\alpha_q(Y)=\omega_q(X(q),Y)\neq 0$. Hence $$\xi_q=\{v\in T_qQ|\omega_q(X(q),v)=0\}$$ is a hyperplane field on $Q$ and $Y$ is transversal to $\xi_q$. In fact $\xi_q$ is the symplectic complement of $\operatorname{span}\{X(q),Y\}$. This implies that $\omega=d\alpha$ is nondegenerate on $\xi_q$ and hence $\alpha$ restricts to a contact form on $Q$.
$\quad$ Conversly, suppose that $\alpha\in\Omega^1(Q)$ is a contact form such that $d\alpha=\omega|_Q$. Let $Y\in{\cal X}(Q)$ be the Reeb field of $\alpha$: $$\iota(Y)d\alpha=0,\qquad\alpha(Y)=1.$$ Choose a vector field $X\in{\cal X}(M)$ such that $$\omega(X,Y)=1,\qquad \omega(X,\xi)=0$$ on $Q$. (First choose any vector field $X_0$ such that $\omega(X_0,Y)=1$ on $Q$. Then for every $q\in Q$ there exists a unique vector $X_1(q)\in\xi_q$ such that $\omega(X_0+X_1,v)=0$ for every $v\in\xi_q$. Define $X=X_0+X_1$ on $Q$ and extend to a vector field on $M$.) Define $\phi:Q\times\Bbb R\to M$ by $$\phi(q,\theta)=\exp_q(\theta X(q)).$$ Then a simple calculation shows that $$\phi^{        \ast         }\omega|_{Q\times\{0\}}=d\alpha-\alpha\wedge d\theta.$$ By Moser's argument there exists a local diffeomorphism $\psi:Q\times(-\varepsilon,\varepsilon)\to M$ such that $$\psi(q,0)=q,\qquad \psi^\ast\omega=e^\theta(d\alpha-\alpha\wedge d\theta).$$ The required Liouville vector field is $\psi_\ast(\partial/\partial\theta).$ $\tag*{$\square$}$

Oh well, a hypersurface in a manifold $M$ of dimension $n$ is simply an $(n-1)$-dimensional submanifold $H\subseteq M$. I am assuming $\iota$ here is the inclusion of $Q$ into $M$, i.e. $\iota:Q\hookrightarrow M$. My questions are numerous:

How do I interpret $d\iota(X)\omega$ and $\iota(X)d\omega$? $\iota:Q\to M$, so $d\iota:T_pQ\to T_pM$, $X$ is a vector field, so $d\iota(X)$ would be a vector field. However, $X$ would have to be on $Q$ to do that, and instead it is on $U$, a neighborhood of $Q$ in $M$. And anyway, even if $d\iota(X)$ were a field on $M$, how do I multiply it with $\omega$? What does the juxtaposition of those two things mean? $\iota(X)$ may be interpreted as $\iota^\ast(X)$, which means I composte $X$ and $\iota$. And again, what is $\iota(X)$ "multiplied by" (juxtaposed to) $d\omega$?
What is that $\mathcal{L}_X\omega$?
«Since $T_qQ$ is odd-dimensional, there exists» etc. Why?
The following «Since» is also obscure.
What does it mean for $Y$ to be «transversal to $\xi_q$»?
How does one prove the existence of the Reeb field for a 1-form?
The rest of the proof is like Elvish to me.
In other words, almost evertything in this proof is incomprehensible to me. Any help is appreciated.
Comment: 1) This is the interior derivative. $\iota$ here is not the inclusion map. 2) This is the Lie derivative. 3) It can't be nondegenerate; symplectic vector spaces are even-dimensional. 4) Review the definition of nondegenerate. 5) Together, they span the tangent space. 6) Pointwise. To check that it's smooth, work in a chart.
Comment: First, your definition of symplectic is incorrect. The 2-form must be non-degenerate.  Secondly, $d\iota(X)$ is nonsense $\iota(X)$ is the contraction operation on differential forms which takes a $k$-form to a $k-1$-form by contracting by the vector field $X$. $d\iota(X)\omega=d(\iota(X)\omega)$ .It looks like you just opened Mc-S to a random page and started reading. I'd suggest going over the beginning more carefully and perhaps even finding another more basic source to learn the rudiments of differential geometry (i.e. Lee's introduction to smooth manifolds.)
Comment: @MikeMiller 1) The [interior derivative](https://en.wikipedia.org/wiki/Interior_product), OK. 2) The Lie derivative of a form? I only have that concept for vector fields, where $\mathcal{L}_X(Y)=[X,Y]$… 3) Really? Could you link me to a proof of that? A 2-form on an odd-dimensional space is always degenerate? 4) [Here it is](https://en.wikipedia.org/wiki/Degenerate_bilinear_form). But that was not the only problem here: the other problem was I didn't know of the interior derivative so the equality $\alpha_q(Y)=\dots$ was mysterious. 5) Uh-huh. Thx for this. I will reflect on this better.
Comment: @PVAL You are partly right. Currently I am reading chapter 1, but I skipped ahead when I was directed to that proposition by the book itself for the characterization of contact-type hypersurfaces. I don't need an introduction to manifolds. A course on that has dealt with that point. The problems are forms, not manifolds. I have taken no course in that. I have, however, read through the first 3-4 chapters of Manfredo P. Do Carmo. And I got the definition of symplectic wrong, will fix this in a sec.
Comment: @PVAL fixed the definition.
Comment: Here is the [Lie derivative of differential forms](https://en.wikipedia.org/wiki/Lie_derivative#The_Lie_derivative_of_differential_forms). Cartan's formula is a definition, not a theorem. That was a source of confusion for me.
Comment: But why is $d\alpha=\mathcal{L}_X(\omega)$? I mean, if $\alpha=\iota(X)\omega$, shouldn't $d\alpha$ simply be $d(\iota(X)\omega)$, without the extra term $\iota(X)d\omega$?
Comment: No, Cartan's formula is a theorem. The definition of the Lie derivative of a tensor field is, if $\phi_t$ is the flow of $X$, $\mathcal L_X T = \frac{d}{dt}\big|_{t=0} (\phi_t)^* T$. This is why we care about the Lie derivative at all. I reiterate PVAL's suggestion to look at the relevant sections of Lee's smooth manifolds book; McDuff-Salamon's book will assume you know most of it.
Comment: @MickG Lee's book has a good introduction to differential forms inside. If you know some differential topology before hand, you could probably skip to that section. Saying that there is already quite a lot of basics of the calculus of differential forms already explained earlier in Mc-S and also a more in-depth explanation of things like "Moser's argument".
Comment: Scratch my last question out. $\omega$ is closed, so the extra term is 0. And PS I am reading chapter 4 (or whatever the chapter on tangent covectors is called) of Lee.
Comment: Found [a proof of Cartan's magic formula](http://www-personal.umich.edu/~wangzuoq/437W13/Notes/Lec%2028.pdf). The only thing that still escapes me is why $\phi^\ast d=d\phi^\ast$, but Lee may enlighten me on that. And Wikipedia should mention the definition of Lie derivative of a form before stating Cartan's formula, otherwise - as happens now - the formula seems a definition rather than a theorem.
Comment: Please check out my answer below and help me answering the last two question. The reverse implication, I will deal with in a separate question after finishing Chapter 1 of McDuff-Salamon.
Answer: Let me try to sum up the comments and understand at least the first half of the proof, i.e. how we get from a Liouville field to a contact form. First of all, we need two things.
Firstly, $\iota(X)$ is not a map applied to $X$, but rather the interior product, or interior derivative, of $\omega$ and $X$. Given a field $X$, the interior derivative of a form $\omega$ with respect to $X$, or rather the interior product or antiderivative, is the form:
$$\iota_X\omega(X_2,\dotsc,X_k)=\omega(X,X_2,\dotsc,X_k),$$
where $\omega$ is a $k$-form and $\iota_X\omega$ is, therefore, a $k-1$-form. So the field takes the place of the first vector in the $k$-tuple of vectors the form is applied to. This means that for each point $p$ we have $\iota_X(p)(X_2(p),\dotsc,X_k(p))=\omega(p)(X(p),X_2(p),\dotsc,X_k(p))$ if $X_2,\dotsc,X_k$ are fields, and of course with no evaluation of $X_i$ when $X_i$ is a vector. Naturally, if $\omega$ is a 1-form, the result is a function, which is often indicated by $\langle X,\omega\rangle$ or $\langle\omega,X\rangle$.
Secondly, $\mathcal{L}_X\omega$ is the Lie derivative of $\omega$ with respect to $X$, that is:
$$\mathcal{L}_X\omega=\frac{d}{dt}\Big|_{t=0}\phi^\ast_t\omega,$$
where $\phi_t$ is the flow of the field $X$. This gives a form of the same "rank" as $\omega$, so it makes sense to compare it to $\omega$. In particular, if $\omega$ is a symplectic form (i.e. a closed nondegenerate 2-form), a field for which $\mathcal{L}_X\omega=\omega$ is called a Liouville vector field. The Lie derivative relates to the exterior derivative and interior antiderivative by Cartan's (magic) formula:
$$\mathcal{L}_X\omega=d(\iota_X\omega)+\iota_Xd\omega.$$
This is a theorem, and can be proved as follows, by the arguments I took from this pdf. Firstly, we need a lemma.
Lemma
For any forms $\omega,\eta$ and field $X$, one has:
\begin{align*}
d\mathcal{L}_X\omega={}&\mathcal{L}_Xd\omega. \tag{1} \\
\mathcal{L}_X(\omega\wedge\eta)={}&\mathcal{L}_X\omega\wedge\eta+\omega\wedge\mathcal{L}_X\eta. \tag{2}
\end{align*}
Proof.
We know that $\phi^\ast d\omega=d\phi^\ast\omega$ by Lemma 9.14 on Lee's introduction to manifolds. Let us write out number (1):
$$d\mathcal{L}_X\omega=d\left(\frac{d}{dt}\Big|_{t=0}\phi^\ast_t\omega\right)=\frac{d}{dt}\Big|_{t=0}d(\phi^\ast_t\omega),$$
since $d$ and $\frac{d}{dt}\Big|_{t=0}$ are essentially derivatives and so can be exchanged, but due to Lee's lemma $d$ swaps with the pull-back, and the result of that swapping is precisely $\mathcal{L}_Xd\omega$, proving (1). As for (2), Lemma 9.9 from Lee gives us $\phi^\ast_t(\omega\wedge\eta)=\phi^\ast_t\omega\wedge\phi^\ast_t\eta$. With this:
$$\mathcal{L}_X(\omega\wedge\eta)=\frac{d}{dt}\Big|_{t=0}\phi^\ast_t(\omega\wedge\eta)=\frac{d}{dt}\Big|_{t=0}\phi^\ast_t\omega\wedge\phi^\ast_t\eta,$$
and I bet that at this point all that is left to do is to write things in a coordinate chart and compute the derivatives, and the equality will reduce to Leibniz's rule, the well-known rule for derivatives of products. $\square$
Now we can finally proceed to prove Cartan's formula. This is done by induction. According to here:

The interior product of a vector and a 0-form is understood to be 0, since there are no $(-1)$-forms.

This makes perfect sense. THis also makes the case of 0-forms in Cartan's formula easy, since the formula reduces to simply $\mathcal{L}_Xf=\iota_Xdf$, but:
$$\iota_Xdf=df(X)=Xf=\mathcal{L}_Xf,$$
the last equality being proved here as Theorem 2.3. Getting back to Cartan's formula, now we have it is true for $k$-forms with $k=0$. Suppose it works for $(k-1)$-forms. If $\omega$ is a $k$-form, it is a sum of terms of the form $fdx^{i_1}\wedge\dotso\wedge dx^{i_k}$, so by linearity of both sides of the formula it suffices to prove the formula for $\omega=fdx^1\wedge\dotso\wedge dx^k=dx^1\wedge\omega_1$, where $\omega_1=fdx^2\wedge\dotso\wedge dx^k$. By the above lemma:
$$\mathcal{L}_X\omega=\mathcal{L}_Xdx^1\wedge\omega_1+dx^1\wedge\mathcal{L}_X\omega_1.$$
Let us take the other side of the formula. We have:
\begin{align*}
d\iota_X(dx^1\wedge\omega_1)+\iota_Xd(dx^1\wedge\omega_1)\stackrel{(1)}{=}{}&d(\iota_Xdx^1\wedge\omega_1-dx^1\wedge\iota_X\omega_1)-\iota_X(dx^1\wedge d\omega_1)={} \\
{}\stackrel{(2)}{=}{}&d(\iota_Xdx^1)\wedge\omega_1+\underline{\iota_Xdx^1\wedge d\omega_1}+dx^1\wedge d\iota_X\omega_1-{} \\
&{}+\underline{\iota_Xdx^1\wedge\omega_1}+dx^1\wedge\iota_Xd\omega_1={} \\
{}\stackrel{(3)}{=}{}&d\mathcal{L}_Xx^1\wedge\omega_1+dx^1\wedge\mathcal{L}_X\omega_1={} \\
{}\stackrel{(4)}{=}{}&\mathcal{L}_Xdx^1\wedge\omega_1+dx^1\wedge\mathcal{L}_X\omega_1
\end{align*}
Let's follow the passages one by one:

It is pretty easy to show that if $\beta$ is a $k$-form and $\gamma$ is a $h$-form then $\iota_X(\beta\wedge\gamma)=\iota_X\beta\wedge\gamma+(-1)^k\beta\wedge\iota_X\gamma$, and in our case we have $\beta=dx^1$ which is a 1-form so $(-1)^k=(-1)^1=-1$. Same line of thought for the second term, plus the fact that $d(dx^1)=0$, which follows from the general property that $d^2=0$.
Indeed, $d$ and $\iota_X$ "distribute" in the same way over a wedge product. Apply this to each term and you get to the right of (2). Remember $\iota_Xdx^1$ is a 0-form so you get a $(-1)^0=1$.
Remember $\iota_Xdx^1=Xx^1=\mathcal{L}_Xx^1$, which deals with the first term. The underlined terms cancel each other out. The wedge product is bilinear so you can collect a $dx^1$ and the other side of the wedge is $\iota_Xd\omega_1+d\iota_X\omega_1$, which by induction is $\mathcal{L}_X\omega_1$
Finally remember how, by the lemma above, $\mathcal{L}_X$ and $d$ commute, and we finish these passages, which lead to the same result as we got for $\mathcal{L}_X\omega$, proving Cartan's formula for $\omega$, which proves its validity for $k$-forms by the above arguments.

We can finally proceed to proving the first implication of the proposition:

If $X$ is a Liouville vector field defined on a neighborhood of $Q$, which is a hypersurface in $M$, and $X$ is transverse to $Q$, then there exists a contact form $\alpha$ on $Q$ such that $d\alpha=\omega|_Q$.

Proof. We set $\alpha=\iota_X\omega|_Q$. Then:
$$d\alpha=d\iota_X\omega|_Q\stackrel\ast=d\iota_X\omega+\iota_Xd\omega=\mathcal{L}_X\omega=\omega,$$
$\ast$ being because $d\omega=0$ ($\omega$ is symplectic and therefore closed, which is exactly $d\omega=0$), and the last equality being since $X$ is Liouville. So $d\alpha=\omega$. Well, $\omega|_Q$ and $\omega$ are being identified. Oh wait. $\alpha$ is on $M$, not on $Q$. Now we have to show $\alpha|_Q$ is a contact form. What does that mean? Unfortunately, my copy of McDuff is unsearchable, so I have to ask Wikipedia. This is what I understand by reading there:

A contact form is one that gives a contact structure on $Q$.
A contact structure is a distribution of contact elements which is integrable at no point.
A contact element is simply a hyperplane in the tangent space to a point $p\in Q$, which is called its contact point.
A distribution means assigning to each point a subspace of the tangent space (in this case, a contact element) in such a way that for any point $p$ in the manifold one can find a neighborhood $U$ of $p$ and $k$ (in our case, 1 less than the dimension of $Q$) linearly independent vector fields $X_1,\dotsc,X_k$ on $U$ such that for $q\in U$ we have $X_1(q),\dotsc,X_k(q)$ spanning $\Delta_q$, where $\Delta_q$ is the subspace of $T_qQ$ which our distribution assigns.
A distribution is integrable at a point $p$ if one can find a manifold $N$, which is then called an integral manifold of the distribution an $p$, and $F:N\to M$ a one-to-one immersion which sends $T_qN$ inside $\Delta_{F(q)}$ for any $q\in N$.

Now, pulling back $\omega$ to $Q$ gives a closed form, so $d\omega|_Q=0$. $Q$ is a hypersurface in a symplectic manifold, so according to this comment it cannot be symplectic, and being closed and a 2-form $\omega|_Q$ therefore can't be nondegenerate, which implies the existence of a vector $Y$ such that $\omega|_Q(Y,T_qQ)=0$ for all $q$. Let us rename the vector to $Y_q$. Assuming "transversal to $Q$" means $X(q)\not\in T_qQ$ for all $q\in Q$, then since $\omega$ is nondegenerate it is evident that $\omega(X(q),Y)$ can't be 0, otherwise that would contradict the nondegenerateness, since for $v=Y$ we would have $\omega(w,v)=0$ for all $w\in T_qM$.

If you are lost, remember we are assuming $X$ is a Liouville field in a neighborhood of the hypersurface $Q$ and transverse to its tangent space at any point, and after showing $d\alpha=\omega$, where $\alpha=\iota_X\omega$, we are attempting to show $\alpha|_Q$ gives a contact form, for the definition of which see blockquotes above.

We set:
$$\xi_q=\{v\in T_qQ:\omega(X(q),v)=0\}.$$
This is by definition contained in $T_qQ$. Now, $\omega$ with $X(q)$ fixed is a 1-form, which goes from $T_qM$ to $\mathbb{R}$. It is either 0 or surjective. Since $\omega(X(q),Y)\neq0$, it is surjective, and therefore has a hyperplane as its kernel at each point. The kernel is precisely $\xi_q$. Leaving the "symplectic complement" thing alone, this $\xi_q$ is a distribution on $Q$. Why it is not integrable at any point is probably because non-integrability is equivalent to $\alpha\wedge(d\alpha)^k\neq0$, where $(d\alpha)^k$ is the wedge product of $d\alpha$ with itself $k$ times. How this is shown still escapes me. Let us get the symplectic complement thing done. The symplectic complement of a vector space is simply, if I guess correctly, the set of vectors which give 0 if given to the symplectic form as an argument together with any vector of the subspace, i.e. the vectors $v$ such that for all $w$ in the subspace one has $\omega(w,v)=0$. We know $\omega(X(q),Y)\neq0$, and the rest is $\xi_q$, so $\xi_q$ is surely the symplectic complement of $\mathrm{span}Y$. Since $\omega(X(q),X(q))=0$ by alternatingness, adding $X(q)$ to $\mathrm{Span}Y$ and getting $\mathrm{span}(Y,X(q))$ doesn't change the fact we have a symplectic complement.
So I am left with two questions:

How can one prove symplectic (i.e. closed non-degenerate) 2-forms are only possible on even-dimensional manifolds?
How does integrability equate to that wedge product condition, or otherwise, how does one show the distribution of $\xi_q$s is non-integrable at any point of $Q$?

Update:
Question 1 is solved by this.
Update 2:
Rereading and editing this answer, I realized I have another question, so I am going to rewrite all questions.

How does non-integrability equate to that wedge product condition, or otherwise, how does one show the distribution of $\xi_q$s is non-integrable at any point of $Q$? Here is (hopefully) the answer.
How does one prove that form, $\alpha\wedge(d\alpha)^k$, is nonzero?

Update 3:
After being answered in the link above, I have an answer for question 2 as well: $d\alpha=\omega$, so $d\alpha$ is nondegenerate on $\xi_q$, which equates to $\alpha\wedge(d\alpha)^k\neq0$, and to $(\Sigma,\alpha)$ being a contact manifold, and the proof is finally complete. Now let me see how much I understand of the converse.


Linear regression: is near multicollinearity really a problem ?

Question: I have the following model for continuous variables $$Y = \beta +\beta_1 X1 + \beta_2 X2 + \beta_3 X3 + \beta_4 X4 + \beta_5 X5 + \epsilon$$
Everything works out very well, the model passes all kinds of tests, respects all assumptions but one: $X1$ and $X2$ are highly correlated (>0.9) so maybe we have multicollinearity. They have huge VIF (around 20).  When I remove $X2$ from the model, the VIF of each variable is under 4, and I obtain similar results but slightly worse. In every kind of criteria.
So if I follow theory, I should remove $X2$. But practical tests shows that the first model is slightly better (just a tiny bit). What should I do ? Why are the models behaving like this in opposition to what theory says ?
Comment: What do you mean by "better"?  What do you want to do with the model- predict y values given x values?  Do inference on the cofficients $\beta$?
Comment: Do you only care about the predictive power of your model? Or do you want to interpret the coefficient values and the effect of each predictor?
Comment: What I mean by better is that:
- Confidence interval for $\beta$'s are smaller
- The coefficient R is higher
- AIC, BIC are higher
- Errors are smaller
- MSE in k-cross validation is smaller
Comment: I am more interested in the predictive power of the model.
Answer:
There is no assumptions of uncorrelated $X$s, hence it does not violate any theoretical consideration.
However, high correlation (surely huge VIF as $20$) is important practical consideration regarding model's stability.
If you are interested in the predictive power and not worried at all about the inferential one, then it should not bother you much.
However, if you are interested in dealing with the multicolinearity with a little more sophisticated way than just dropping out $X_2$, then you can:

(i) Perform principle components regression on the subsets of $k$ best PCs.
 https://en.wikipedia.org/wiki/Principal_component_regression
(ii) Perform ridge regression:
https://en.wikipedia.org/wiki/Tikhonov_regularization


peek multiple chars from cin with putback

Question: I need to try to peek multiple characters from a std::istream (which could be std::cin), so I wrote a simple loop to call get() lots of times and then putback() lots of times:
std::vector<char> peek_many(std::istream& is, int N) {
    std::vector<char> data;
    data.reserve(N);

    for (int i = 0; i < N; ++i) {
        data.push_back(is.get());
    }

    for (int i = 0; i < N; ++i) {
        is.putback(data[N-i-1]);
    }

    return data;
}

Is this guaranteed to work on all istreams (whether I'm reading a file, cin, istringstream, etc)? If not, why not?
Comment: @RemyLebeau You can't *modify* it - but you can put the same characters back - which is what I'm trying to do.
Comment: Off topic, wouldn't it be better to return a shorter vector if `EOF` was reached, instead of returning a vector with trailing `EOF`s?
Comment: @AlanStokes: Run the example code at http://en.cppreference.com/w/cpp/io/basic_istream/putback.  The `putback()` fails on an `std::istringstream`, and is commented as such.
Comment: `std::istringstream` is input-only, it can only be read from, you can't `putback()` data into it.
Comment: @Remy Citation needed
Answer: It is only safe to assume 1 character "wide" put back buffer. You will have to add additional layer of abstraction to provide more put back space.
Alternatively, if your stream is seekable, you can remember the position before you start tentatively parsing and then seek back to the remembered position.
Comment: An extra level of buffering in your code. You read data from the stream into your own buffer of sufficient size, and then you can read/peek data from the buffer as needed without trying to put the data back into the stream.
Comment: What would such an abstraction look like?
Comment: Or you could call `[pub]setbuf` to tell the `streambuf` that you want a bigger buffer - except that sadly in general that has no useful well-defined behaviour.
Answer: putback calls sputback which only works if a putback position is available. Nothing guarantees that there is an indefinite number of such positions.
Comment: When `sputback` returns `EOF` you ran out ;-) Or you can look at `gptr()` vs `eback()`, which gives you a lower bound.
Comment: Those are `protected`.
Comment: Fair point. Assuming you're not willing/able to derive from the `streambuf` I think you're stuck with the first half of my suggestion.


Finding correlation between numeric encoded categorical variables?

Question: So I have a data set which has categorical variables. I have encoded each category from 0:x depending on the amount of categories x. I am trying to find the correlation between the variables (both numeric and categorical) to the target variable (which is also categorical, but has been coded to 0 for no and 1 for yes).
I have been able to run this code for the correlation(s) for all the variables (there are about 17) and it does seem to work, but I question whether the correlations are correct. I haven't calculated the correlation with categorical variables in my data set before, just want to ensure that I am going about this the right way:
print(df['previous'].corr(df['y']))

Any help would be great!
Answer: The trick to finding correlations within categorical variables is to dummify them. While your target variable is fine (since it i binary), the categorical variables having multiple classes need to be dummified -
pd.get_dummies(df['Categorical_Column'])

Once done, delete one column from the dummified columns and then get the correlations...
Comment: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html for get_dummies docs and https://en.wikipedia.org/wiki/Dummy_variable_(statistics) wiki link for understanding dummy vars
Comment: I am actually running an SVM algorithm on my data set, and I have manually recoded all my categorical variables based on their categories. Did I go about this wrong? Additionally, can you possible elaborate as to what these dummified variables contain? And would I do the correlation between these dummified variables and my target variable y?
Comment: Thank you. So would these dummy variables work with multi-class features? One of my categorical variables is the days of the week. So once I create these dummy variables column, I run the statistics for this dummy variables column and the target variable y?
Comment: yes you can. Just make sure you dont fall into the "dummy variable trap". Remove one column from the output dummy columns so that you ensure there is no singularity in the dataset
Comment: Okay I think I am understanding you. So I thought of creating a new dataframe containing only the dummy categorical variables and the other numeric variables and I am planning on running my SVM on this. Would this be the way to go about this?
Answer: Correlation analogue for categorical data is called cross tabulation (you get a contingency table for your data). You can do it in pandas using pandas.crosstab.
Comment: Thanks for the answer. So looking at crosstab, I would set the columns to equal all of my categorical variables and it does this encoding for me?


How to show the user badges in my tpl.php file?

Question: This maybe a dumb question but what am I doing wrong?
From here: https://www.drupal.org/node/942014
It says I have to insert:
if (module_exists('user_badges')) {
    print user_badges_for_uid($uid);
  }
When I insert that on the last line of my  node.tpl.php it just shows as Text (like the whole code shows on the website). What am I doing wrong??
Can someone please help me?
Comment: PHP needs to be wrapped in ``
Answer: The first thing that you need to do is add the php tag to the code:
<?
if (module_exists('user_badges')) {
 print user_badges_for_uid($uid);
}
?>

But for to that this code works you need to pass the uid = User Id
This code should do the job:
<?
// Getting the actual user.
global $user;

if (module_exists('user_badges')) {
 print user_badges_for_uid($user->uid);
}
?>

Comment: @GrahamMiranda if the code works for you don't forget to check the answer as accepted, this can help other users to find the correct answer, if you don't know how, please check the Tour -> https://drupal.stackexchange.com/tour
Comment: @GrahamMiranda Good luck with your projects ;-)
Comment: Hey a quick question. I've uploaded this and it works for "My Badges" do you know how can I change the code to show people's profile badges? When I put the code on their profile it shows the user who's logged in badges.
Comment: @GrahamMiranda you need to get the uid of the user profile. You can open another question about this if you don't know how.


What is the formula for the standard error of Cohen's d

Question: I found different answers to the question how to calculate the standard error (SE) of Cohen's d.
First formula is (see here, here or here):
$$ SE_d = \sqrt{\frac{n_1 + n_2}{n_1 n_2} + \frac{d^2}{2(n_1+n_2)}} $$
Second formula is (see here):
$$SE_d = \sqrt{\left(\frac{n_1 + n_2}{n_1 n_2} + \frac{d^2}{2(n_1+n_2-2)}\right) \left(\frac{n_1 + n_2}{n_1+n_2-2} \right)}$$
Third formula is a slight variation of the first one (see here in the last line of formulae):
$$ SE_d = \sqrt{\frac{n_1 + n_2}{n_1 n_2} + \frac{d^2}{2(n_1+n_2 - 2)}} $$
I know that there is some confusion on how to calculate Cohen'd itself. Cohen's d is defined as $d = \frac{\bar{x_1} - \bar{x_2}}{sd_{pooled}}$ but the pooled standard deviation is defined in two different ways, i.e. $sd_{pooled} = \sqrt{\frac{(n_1 - 1)S_1^2 + (n_2 - 1)S_2^2}{n_1 + n_2}}$ and $sd_{pooled} = \sqrt{\frac{(n_1 - 1)S_1^2 + (n_2 - 1)S_2^2}{n_1 + n_2 - 2}}$ (see here). Does the formula for SE change depending on how $sd_{pooled}$ is defined? Or, if we use always the same formula for SE of Cohen's d: Which of the fomulae above is it?
Comment: @user248711 See [this review](https://doi.org/10.20982/tqmp.14.4.p242) for an examination of all these questions.
Answer:
The statistic Cohen's d follows a scaled non-central t-distribution.
This statistic is the difference of the mean divided by an estimate of the sample standard deviation of the data:
$$d = \frac{\bar{x}_1-\bar{x}_2}{\hat{\sigma}}$$
It is used in power analysis and relates to the t-statistic (which is used in significance testing)
$$d = n^{-0.5} t $$
This factor $n$ is computed as  $n=\frac{n_1 n_2}{n_1+n_2}$
The difference is that

to compute $d$ we divide by the standard deviation to the data
and for $t$ we divide by the standard error of the means

(and these differ by a factor $\sqrt{n}$)
Confidence interval based on normal approximation of non-central t-distribution
The articles that you mention relate to the article Larry V. Hedges 1981 "Distribution Theory for Glass's Estimator of Effect Size and Related Estimators"
There they give a large sample approximation of Cohen's d as a normal distribution with the mean equal to $d$ and the variance equal to $$\frac{n_1 + n_2}{n_1n_2} + \frac{d^2}{2(n_1+n_2)}$$
These expressions stem from the mean and variance of the non-central t-distribution. For the variance we have:
$$\begin{array}{crl}
\text{Var}(t) &=& \frac{\nu(1+\mu^2)}{\nu-2} - \frac{\mu^2 \nu}{2} \left(\frac{\Gamma((\nu-1)/2)}{\Gamma(\nu/2)}\right)^2 \\ &\approx& \frac{\nu(1+\mu^2)}{\nu-2} - \frac{\mu^2 \nu}{2} \left(1- \frac{3}{4\nu-1} \right)^{-2}  \end{array} $$
Where $\nu = n_1+n_2-2$ and $\mu = d \sqrt{\frac{n_1n_2}{n_1+n_2}}$. For cohen's d this is multiplied with ${\frac{n_1+n_2}{n_1n_2}}$
$$\text{Var}(d) = \frac{n_1+n_2}{n_1n_2} \frac{\nu}{\nu-2} +  d^2 \left( \frac{\nu}{\nu-2} -\frac{1}{(1-3/(4\nu-1))^2} \right)$$
The variations in the three formula's that you mention are due to differences in simplifications like $\nu/(\nu-2) \approx 1$ or $\nu = n_1+n_2-2 \approx n_1+n_2$.
In the most simple terms
$$\frac{\nu}{\nu-2} = 1 + \frac{2}{\nu-2} \approx 1$$
and (using a Laurent Series)
$$\frac{\nu}{\nu-2} -\frac{1}{(1-3/(4\nu-1))^2} = \frac{1}{2\nu} + \frac{31}{16\nu^3} + \frac{43}{8\nu^3} + \dots \approx \frac{1}{2\nu} \approx \frac{1}{2(n_1 + n_2)} $$
Which will give
$$\text{Var}(d) \approx \frac{n_1+n_2}{n_1n_2}  +  d^2\frac{1}{2(n_1+n_2)} $$
Confidence interval based on computation
If you would like to compute the confidence interval more exactly then you could compute those values of the non-central t-distribution for which the observed statistic is an outlier.
Example code:
<code>### input: observed d and sample sizes n1 n2
d_obs = 0.1
n1 = 5
n2 = 5

### computing scale factor n and degrees of freedom
n  = n1*n2/(n1+n2)
nu = n1+n2-2

### a suitable grid 'ds' for a grid search
### based on
var_est <- n^-1 + d_obs^2/2/nu
ds <- seq(d_obs-4*var_est^0.5,d_obs+4*var_est^0.5,var_est^0.5/10^4)

### boundaries based on limits of t-distributions with ncp parameter
### for which the observed d will be in the 2.5% left or right tail
upper <- min(ds[which(pt(d_obs*sqrt(n),nu,ds*sqrt(n))<0.025)])*sqrt(n)    # t-distribution boundary
upper/sqrt(n)                                                             # scaled boundary
lower <- max(ds[which(pt(d_obs*sqrt(n),nu,ds*sqrt(n))>0.975)])*sqrt(n)
lower/sqrt(n)
</code>
Below is a situation for the case when the observed $d$ is 0.1 and the sample sizes are $n_1 = n_2 = 5$. In this case the confidence interval is
$$CI: -1.43619,1.337479$$
In the image you see how $d$ is distributed for different true values of $d$ (these distributions are scaled non-central t-distributions).
The red curve is the distribution of observed $d$ if the true value of $d$ would be equal to the upper limit of the confidence interval $1.337479$. In that case the observation of $d=0.1$ or lower would only occur in 2.5% of the cases (the red shaded area).
The blue curve is the distribution of the observed $d$ if the true value of $d$ would be equal to the lower limit of the confidence interval $-1.143619$. In that case the observation of $d=0.1$ or higher would only occur in 2.5% of the cases (the blue shaded area).
Comment: @machine *"I need to pool Cohens d and its SE with Rubins Rule after multiple imputation"* If you have the original data (absolute effect size, estimated variance and sample size) instead of only cohen's d values, then I imagine it might be better to pool the original data and from that compute a pooled value for effect size $d$.
Comment: @machine This Rubins Rule is new to me, but what I understand is that you apply it to multiple estimates derived from the *same* data (for instance multiple estimates based on different ways of imputation), and then you have an estimate of the variance due to two sources (within and between) which you can add together... But why not estimate the absolute effect size instead of cohen's d (which is a scaled effect size) in this way, and then compute cohen's d based on this pooled absolute effect size and the estimate of it's variance? I will make a question about this.
Comment: Cohen's d is $\frac{\bar{x}_1-\bar{x}_2}{\hat{\sigma}}$ which is a standardized form of the absolute effect size ${\bar{x}_1-\bar{x}_2}$
Comment: https://stats.stackexchange.com/questions/495174
Comment: @machine What sort of data imputation are you doing? You have two groups and there are missing values. Why are you using multiple imputation? Is the comparison pairwise data? Then maybe use https://onlinelibrary.wiley.com/doi/abs/10.1002/bimj.201100053 or https://www.jstor.org/stable/2335098  (and there are many other methods) Is the comparison unpaired then why use imputation to replace the missing values?
Comment: @machine I do not see what the imputing is gonna achieve. The missing data is effectively just like having a smaller sample size. With the missing data, you can still compute sample means and estimate the variance of the difference in the sample means. Possibly you are imputing to create more balanced data (e.g. to get equal class distribution in the control vs experimental groups)?
Comment: But if you are just comparing the means of two populations and it is not a pair-wise comparison, then you do not really have to resort to imputation ([you do not need equal sample sizes](https://en.wikipedia.org/wiki/Student%27s_t-test#Equal_or_unequal_sample_sizes,_similar_variances_(1/2_%3C_sX1/sX2_%3C_2))). It is for larger models that it might make sense to find a solution for the missing data by imputation.
Comment: @machine I deleted that question. I was working on an answer that simulates it (I was planning to do the following: generate two vectors, randomly delete data, then impute the data, and compute the pooled Cohen's d in two ways). But while doing that I realized that for a comparision of means in two unpaired samples it makes no sense to use imputation.
Comment: But you do not need to use imputation because you can compare samples of different sizes. (Imputation would only make sense if you are worried about unbalanced data, for instance when the demographical composition of the two groups are unequal. But still for such cases you might have better methods, like for instance mixed effects models)
Comment: *"You mean "bigger models" because of pairwise exclusion?"* With larger models I mean that some outcome variable $Y_{i}$ depends on many different variables $X_{ij}$ (for instance a [linear regression](https://en.wikipedia.org/wiki/Linear_regression) model). When some of these $X_{ij}$ are missing you may replace them in some way in order to make the entire model work. In these cases the typical models need a full data matrix, and by using imputation you do not need to remove an entire line $Y_i,X_{ij}$ because only one of the variables is missing.
Comment: @machine I have undeleted it.
Comment: @machine I am not a typical statistician, my presence here has evolved more from personal interest than from professional interest. I have studied astronomy and food technology. That is a strange mix and at the interface between the two there is a lot of applied mathematics that corresponds to both topics. So I am not a typical statistician that studied specifically mathematics or econometrics (and I guess that social science is actually a field that also makes use of a lot of statistics, it is only that not many social scientists focus on it and use it as a tool).
Comment: So at least half of my answers is based on being very good and searching. It is not out of the top of my hat.
Comment: That is really interesting. Well, I would say that on a conceptual level my statistical knowledge is quite good. But it is so frustrating to me that I have big troubles understanding equotions. Like the ones in your answer: I can not really follow what is happing. I can only compare, look for differences and implement in R. That is really annoying
Comment: Just noticed that the part about "Confidence interval based on computation" does inderictly answer the question how to get SE for d since [se= (upper CI - lower CI)/3.92](https://handbook-5-1.cochrane.org/chapter_7/7_7_3_2_obtaining_standard_deviations_from_standard_errors_and.htm). But I admit that it is not first choice to get the SE.
Comment: @machine that is not really the way to compute the SE. The SE is in the first place the standard deviation for the sampling distribution of a statistic. Often it is related to the confidence interval because confidence intervals can be derived from it (and in that case you can compute backwards the SE) but it doesn't need to be like that.


R How to rename variables in one tibble using another tibble serving as a dictionary

Question: I would like to change the names of a tibble using another tibble which work as a dictionary.
however, I don't know if the order of the table's variables are the same as the dictionary's rows.
The only thing assured is that the first variable of the names table
holds all the variable names in the data
examples:
in the simplest case (not this problem) the order of the variables is equal to the ordering of the dictionary's rows
cars <- as_tibble(mtcars)

wanted_names <- tribble(~names, ~new,
                        "mpg","arbitary1",
                        "cyl","arbitary2",
                        "disp","x",
                        "hp","y",
                        "drat","f",
                        "wt","sss",
                        "qsec","ffff",
                        "vs","wvegsb",
                        "am","dhyhjn",
                        "gear","scsssfsf",
                        "carb","arbitaryn")
names(cars)

[1] "mpg"  "cyl"  "disp" "hp"   "drat" "wt"   "qsec" "vs"   "am"   "gear" "carb"
I could just use-
names(cars)<-wanted_names$new
yet I cannot vouch for the order of wanted_names or cars, so it actually looks similar to this:
cars <- as_tibble(mtcars) %>%
  select(wt,hp,carb,everything())

wanted_names <- tribble(~names, ~new,
                        "wt","sss",
                        "qsec","ffff",
                        "vs","wvegsb",
                        "am","dhyhjn",
                        "disp","x",
                        "hp","y",
                        "drat","f",
                        "cyl","arbitary2",
                        "gear","scsssfsf",
                        "mpg","arbitary1",
                        "carb","arbitaryn")

any method that could assure correct renaming will be appreciated.
Comment: You can just join your tibbles.
`carsNew <- dplyr::left_join(cars, wanted_names, by = "names") %>% select(-names)`
Comment: thanks but the problem is that cars is in a wide format, so there is no common variable
Answer: One approach is to use match.
names(cars) <- wanted_names$new[match(names(cars),wanted_names$names)]
cars
#   arbitary1 arbitary2     x     y     f   sss  ffff wvegsb dhyhjn scsssfsf arbitaryn
#       <dbl>     <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>  <dbl>  <dbl>    <dbl>     <dbl>
# 1      21           6  160    110  3.9   2.62  16.5      0      1        4         4
# 2      21           6  160    110  3.9   2.88  17.0      0      1        4         4
# 3      22.8         4  108     93  3.85  2.32  18.6      1      1        4         1
# 4      21.4         6  258    110  3.08  3.22  19.4      1      0        3         1
# 5      18.7         8  360    175  3.15  3.44  17.0      0      0        3         2
# 6      18.1         6  225    105  2.76  3.46  20.2      1      0        3         1
# 7      14.3         8  360    245  3.21  3.57  15.8      0      0        3         4
# 8      24.4         4  147.    62  3.69  3.19  20        1      0        4         2
# 9      22.8         4  141.    95  3.92  3.15  22.9      1      0        4         2
#10      19.2         6  168.   123  3.92  3.44  18.3      1      0        4         4
# … with 22 more rows


Creating a List in Kotlin based on Conditions

Question: At the moment I have this fun:
private fun validateArguments(city: String, state: String, country: String, zip: String): List<String> {

        val list: MutableList<String> = mutableListOf()
        if (city.isNullOrBlank()) list.add("Invalid city")
        if (state.isNullOrBlank()) list.add("Invalid state")
        if (country.isNullOrBlank()) list.add("Invalid country")
        if (zip.isNullOrBlank()) list.add("Invalid zip code")

        return list.toList()
}

and I was wondering if there a more elegent way to create the list.
The final list could also be a MutableList I guess.
Comment: See https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/build-list.html
Answer: If you're using Kotlin 1.6+, there are builder APIs for lists, sets, and maps:
val list = buildList {
    if (city.isNullOrBlank()) add("Invalid city")
    if (state.isNullOrBlank()) add("Invalid state")
    if (country.isNullOrBlank()) add("Invalid country")
    if (zip.isNullOrBlank()) add("Invalid zip code")
}

This allows to limit the scope of the mutable handle to the list, and only expose the read-only interface. It also fits nicely in extracted methods.
Comment: This is gold, without this creating a list is just awkward.
Comment: buildList is no longer experimental as of Kotlin 1.6.0 !!!
Comment: Thanks, I updated my anwer!
Answer: I'm not sure there's a single, obviously better way; it's already fairly concise and clear.
But here's one alternative, using the listOfNotNull and takeIf functions to avoid an explicit temporary list:
private fun validateArguments(city: String, state: String, country: String, zip: String)
    = listOfNotNull(
        "Invalid city".takeIf{ city.isNullOrBlank() },
        "Invalid state".takeIf{ state.isNullOrBlank() },
        "Invalid country".takeIf{ country.isNullOrBlank() },
        "Invalid zip code".takeIf{ zip.isNullOrBlank() })

Here takeIf() returns null if the condition isn't met; and listOfNotNull() then removes all those nulls, leaving only the errors for those which were met.
Answer: You can't do too much to make it shorter without some crazy solutions (e.g. reflection). If all params are always strings and error messages are always very similar, you can remove some redundant code with:
@OptIn(ExperimentalStdlibApi::class)
private fun validateArguments2(city: String, state: String, country: String, zip: String) = buildList {
    for ((value, name) in listOf(
        city to "city",
        state to "state",
        country to "country",
        zip to "zip code"
    )) {
        if (value.isBlank()) {
            add("Invalid $name")
        }
    }
}

But it doesn't make too much sense if you don't have a big amount of these params.
Also, you don't need to check for nulls (isNullOrBlank()) if your params are not even nullable.
Update
Or without experimental APIs:
private fun validateArguments2(city: String, state: String, country: String, zip: String) : List<String> {
    return listOf(
        city to "city",
        state to "state",
        country to "country",
        zip to "zip code"
    ).filter { it.first.isBlank() }
        .map { "Invalid ${it.second}" }
}

But it will be more enigmatic for a reader than your original implementation, so you really need to have a reason to do this.


Sphere in a regular grid, area formula

Question: Usually, the formula for area of the sphere is $4 \pi r^2$.
My sphere is inside a regular cubic grid, with 1×1×1 unit size cells.
I'd like to know the count of the grid cells that happen to intersect with the surface of my sphere. I only need the approximate answer, for large values of $r$.
Intuitively, this going to be $4 \pi r^2 C$, where $C$ is some constant.
What's the value of $C$?
Comment: My computations show that $C \rightarrow 1.5$ when $r\rightarrow \infty$.
Comment: @Oleg567 Thanks a lot! My question is somewhat practical, so the 1.5 is good enough for me.
Comment: @Oleg567: Did you arrive at ${3\over2}$ via integral geometry or via numerical experiments?
Comment: [This person](https://math.stackexchange.com/questions/1908123/ideas-on-how-to-find-the-number-of-unit-cubes-that-intersect-ball-with-radius-r) was asking the same thing a year ago, no answers. There's a $3/2$ asymptotic factor in [this related OEIS](https://oeis.org/A085690)
Comment: @ChristianBlatter, I arrived $\frac{3}{2}$ via numerical experiments: just considered different $r$ and cubes with vertices $(a,b,c), \ldots, (a+1,b+1,c+1)$, where $a,b,c\in\mathbb{Z}$, and checked which of them intersect with the surface of the sphere.
Answer: The following heuristic makes the value $C={3\over2}$ plausible:
Consider a sphere $S_R$ of radius $R\gg1$. The lattice planes $x=j\in{\mathbb Z}$, $y=k$, or $z=l$ intersect $S_R$ in circles which together partition $S_R$ into a number of cells of various shapes. We have to estimate the number of these cells.
Looking from $(0,0,+\infty)$ down to $S_R$ we see about $\pi R^2$ unit squares. These are projections of quadrangles on $S_R$ produced by the planes $x=j$ and $y=k$, and there are another $\pi R^2$ such squares to be seen on the lower hemisphere.
The latitude circle $z=l$ has radius $r_l=\sqrt{R^2-l^2}$. It intersects about $2r_l$ planes $x=j$ and $2r_l$ planes $y=k$, generating about  $8r_l$ points of intersection. Each arc between two successive such points bisects an "$xy$-quadrangle", hence augments the number of cells by $1$. The total contribution obtained in this way is about
$$\sum_{l=-R}^R 8 r_l\doteq 4\sum_{l=-R}^R 2\sqrt{R^2-l^2}\doteq4\pi R^2\ ,$$
since the last sum is a Riemann sum for the area of a disc of radius $R$.
It follows that the total number of cells can be estimated as
$$2\pi R^2+4\pi R^2={3\over2}\cdot4\pi R^2\ .$$

Sphere in a regular grid, area formula


How can I memorize a sparse banded matrix with scipy.sparse.dia_matrix giving it arrays (diagonals) of different dimensions?

Question: I want to memorize a pentadiagonal square matrix with scipy.sparse.dia_matrix. The first and the last diagonal are far away from the main diagonal, so they contain fewer elements than the main diagonal. It seems dia_matrix accepts only diagonal of the same length of the main diagonal. I want to save memory and give it an array of different length. Is it possible? Below an example (tridiagonal):
diag1 = [88,99]
main_diag = [1,2,3,4]
diag2 = [ 101,202]

A = [ 1,   0, 88, 0
      0,   2,  0, 99
      101, 0,  3, 0
      0,  202, 0, 4 ]
A = scipy.sparse.dia_matrix(([[diag1],[main_diag],[diag2]],offsets), shape= (4,4))

But it give me an error unless I don't memorize diagonals as:
diag1 = [0,0,88,99]
main_diag = [1,2,3,4]
diag2 = [ 101,202,0,0]

But in this way it is not memory saving.
Answer: Your dense matrix (why didn't you make this cut-n-paste compatible?):
In [589]: A = np.array([[ 1,   0, 88, 0],
     ...:       [0,   2,  0, 99],
     ...:       [101, 0,  3, 0],
     ...:       [0,  202, 0, 4] ])
     ...:
In [590]: A
Out[590]:
array([[  1,   0,  88,   0],
       [  0,   2,   0,  99],
       [101,   0,   3,   0],
       [  0, 202,   0,   4]])

We can make the dia matrix directly from A:
In [591]: M = sparse.dia_matrix(A)
In [592]: M
Out[592]:
<4x4 sparse matrix of type '<class 'numpy.int64'>'
    with 8 stored elements (3 diagonals) in DIAgonal format>

dia format stores is values in two arrays, a 2d data and 1d offsets:
In [593]: M.data
Out[593]:
array([[101, 202,   0,   0],
       [  1,   2,   3,   4],
       [  0,   0,  88,  99]])
In [594]: M.offsets
Out[594]: array([-2,  0,  2], dtype=int32)

We can use the sparse.diags function to construct the dia matrix from your 3 diagonals:
In [605]: M1 = sparse.diags([diag1, main_diag, diag2],[2,0,-2], dtype=int)
In [606]: M1
Out[606]:
<4x4 sparse matrix of type '<class 'numpy.int64'>'
    with 8 stored elements (3 diagonals) in DIAgonal format>
In [607]: M1.A
Out[607]:
array([[  1,   0,  88,   0],
       [  0,   2,   0,  99],
       [101,   0,   3,   0],
       [  0, 202,   0,   4]])
In [608]: M1.data
Out[608]:
array([[  0,   0,  88,  99],
       [  1,   2,   3,   4],
       [101, 202,   0,   0]])

But the underlying storage is the same.
I don't know the details of why dia_matrix uses this format.  I suspect it has to do with balance of storage and processing costs.  To store the data as a 2d array it has to have some sort of padding.  Otherwise you have use a list of lists (or object array of arrays/lists) as in [diag1, main_diag, diag2].  Whether that uses more or less memory may depend on the size of the matrix and how far out the offsets are.
In lil format the same matrix is stored as 2 arrays of lists:
In [611]: M.tolil().data
Out[611]:
array([list([1, 88]), list([2, 99]), list([101, 3]), list([202, 4])],
      dtype=object)
In [612]: M.tolil().rows
Out[612]:
array([list([0, 2]), list([1, 3]), list([0, 2]), list([1, 3])],
      dtype=object)

and in coo:
In [615]: M.tocoo().data
Out[615]: array([101, 202,   1,   2,   3,   4,  88,  99])
In [616]: M.tocoo().row
Out[616]: array([2, 3, 0, 1, 2, 3, 0, 1], dtype=int32)
In [617]: M.tocoo().col
Out[617]: array([0, 1, 0, 1, 2, 3, 2, 3], dtype=int32)

csr, which is widely used for calculations, compresses the row array.
If you are doing all the calculations in the dense form, you might want to construct A from something like:
np.diag(main_diag,0)+np.diag(diag1,2)+np.diag(diag2,-2)

Comparing this with the sparse mediated approach:
In [623]: timeit np.diag(main_diag,0)+np.diag(diag1,2)+np.diag(diag2,-2)
27.5 µs ± 1.28 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [624]: timeit sparse.diags([diag1, main_diag, diag2],[2,0,-2], dtype=int).A
288 µs ± 13 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Comment: I see that in this way I'm not saving memory because M1.data contains 4+4+4 elemenst, not 2+4+2 elements as I want. I want to optimize memory.
Comment: But when you compare lists of list with 2d arrays, you can't just count elements.  You could save `np.hstack([diag1, main_diag, diag2])`, and then, based on shape and offsets, split that into the 3 subarrays.


Distribution of data is very separated in my graph

Question: First post here but I am quite stuck.
I plotted the date an award was given to a university, against their change in score (I calculated an index of change). It has given me this graph which i am struggling to interpret. The data points seem to be very separated out - either in the -10 - .10 region or in the -90 to -60 region.
Is there any way to test for significance, or any tests I should do next? The simple linear regression was not significant (p>0.05).
Comment: Can you describe what the "change score" is exactly? Is there any obvious differences between the two groupings - such as the lower grouping always having really low "pre" measures.
Comment: You have 2 latent clusters. It doesn't look like there is a relationship b/t X & Y w/i either cluster, but the lower cluster is more common on the right & the upper cluster is more common on the left (hence the slight downward slope).
Answer: I am suspicious of the change in score calculation as there is an obvious extreme grouping going on (and really no relationship between date and the change index).  Please explain exactly how you calculated this.


Why the cron is launched when I visit the website and why is it taking so much to complete?

Question: I have a site in development. It has Feeds module installed, with mandatoty Job Scheduler also installed. When I come back to the office to keep working on the site, it takes almost a minute for the webpage to complete. I have checked the log messages, and says that it is the Cron doing its tasks. The Feeds imports are switched off, and configured to launch only on submission (admin), and the cron is scheduled every 3 hours, in times different to those in which the problem appears, apparently it is the visit to the webpage what launches the Cron tasks.
How can I check what is causing the problem?
Answer: The cron can make different tasks such as check for updates to Drupal core and modules, removing temporary files, index content and cleaning up log files.
When you enter to your site the cron is executed and performs his tasks every X hours. You need enter to your site to activate the cron execution, if you want a different behavior you need read this: https://www.drupal.org/cron
There is no way for Drupal to run a cron job without bootstrapping. And the bootstrapping occurs when a visitor hits the site (Nothing happens on a server without someone or something running it). If you want really scheduled cron jobs, you have to set it up server side and this is certainly the best option, for this read: https://www.drupal.org/node/23714
Edit: I found now this article Why Drupal cron is not running as often as you think and how to fix it
Comment: I recieve a downvote, the downvoter can explain his point of view please? I can improve my answer. Thanks
Comment: I used to do this behaviour when doing pure PHP sites, to execute cron tasks when the user hit the page, so no need of a Linux daemon or crontab. I had completely forgotten about this, I later realized. I have tried Job Scheduler trigger, but it does not set the date correctly. I use 15 15 * * * for doing the task at 15:15 but the date is set sometime in the following months. I will have to go to Linux command.
Comment: @Cesar I edit with an aditional article. I think that you must read it.
Comment: @Cesar a pleasure help you ;-)
Answer: For Drupal to work properly, it has to run certain background tasks from time to time. These are all categorised under the heading "cron" because that's the name of the *nix service that's used for scheduling background tasks.
As @Adrian Cid Almaguer says you should read https://www.drupal.org/cron for more about this. You can schedule cron from control panel in most hosting environments.
If you DON'T schedule cron automatically, there's a feature called "poormanscron" that gets run when someone visits your site. As you've noticed this means prolonged response times for the unlucky person. So the lesson is to set up cron to run automatically. You need to configure it so the automated cron runs more frequently than the time configured under Adminstration > Configuration > System > Cron Settings, otherwise poormanscron will still run sometimes.
Answer: with elysia cron module you have full control in your admin UI, but need to add some executions to your cron tab.


Why this equation is correct? (Frenet frame)

Question: $2(T_a-T_b)•(T'_a-T'_b)=-2T_a•T'_b-2T'_a•T_b$
Is $2T_a•T'_a=0$ ?
Please explain me why this equation work.
Comment: I edited your first equation, inserting a factor of $2$ in the $T_a' \cdot T_b$ term; your original equation was false without it.  Cheers!
Answer: The equation
$(T_a - T_b) \cdot (T_a' - T_b')$
$= T_a \cdot (T_a' - T_b') - T_b \cdot (T_a' - T_b') = T_a \cdot T_a' - T_a \cdot T_b' - T_b \cdot T_a' + T_b \cdot T_b' \tag 1$
follows immediately from the ordinary distributive properties of the dot product $A \cdot B$ for any vectors $A$ and $B$.
If we understand $T_s$ to be the unit tangent vector field along some arc-length parametrized regular curve $\gamma(s)$ in the sense of Frenet-Serret,
$T_s = \gamma'(s), \tag 2$
then since
$T_s \cdot T_s = 1, \tag 3$
we find
$2T_s \cdot T_s' = T_s' \cdot T_s + T_s \cdot T_s' = 0 \tag 4$
holds for any value of $s$; taking $s = a$ and $s = b$, we have
$T_a \cdot T_a' = T_b \cdot T_b' = 0; \tag 5$
thus (1) becomes
$(T_a - T_b) \cdot (T_a' - T_b') =  - T_a \cdot T_b' - T_b \cdot T_a', \tag 6$
which we may multiply by $2$ to obtain
$2(T_a - T_b) \cdot (T_a' - T_b') =  2(- T_a \cdot T_b' - T_b \cdot T_a') = -2T_a \cdot T_b' - 2T_b \cdot T_a'. \tag 7$
Comment: Thank you!!! But the part, 2⋅′=0 holds for any value of s, is a little bit confusing. Is this because the derivative of a constant is zero? (⋅ = c then s⋅′s=0) So It doesn't matter if s is a non unit vector?
Comment: @loone96:  just to keep things straight, $s$ is not a vector at all, it is the (scalar) arc-length along the curve $\gamma(s)$; $T_s \cdot T_s' = 0$ holds as long as $T_s \cdot T_s$ is *any* constant; it doesn't have to be $1$.


Flask: model_form: AttributeError: type object 'Form' has no attribute 'query'

Question: This is the error:
File"/www/sites/blog/flask/lib/python2.7/sitepackages/flask/app.py",
   line 1701, in __call__ return self.wsgi_app(environ, start_response)

File"/www/sites/blog/flask/lib/python2.7/sitepackages/flask/app.py",
   line 1689, in wsgi_app response =self.make_response(self.handle_exception(e))

File"/www/sites/blog/flask/lib/python2.7/sitepackages/flask/app.py",
   line 1687, in wsgi_app response = self.full_dispatch_request()

File"/www/sites/blog/flask/lib/python2.7/sitepackages/flask/app.py",
   line 1360, in full_dispatch_request rv = self.handle_user_exception(e)

File"/www/sites/blog/flask/lib/python2.7/sitepackages/flask/app.py",
   line 1358, in full_dispatch_request rv = self.dispatch_request()

File"/www/sites/blog/flask/lib/python2.7/sitepackages/flask/app.py",
   line 1344, in dispatch_request return
   self.view_functions[rule.endpoint](**req.view_args)

File"/www/sites/blog/flask/lib/python2.7/sitepackages/flask/views.py",
   line 83, in view return self.dispatch_request(*args, **kwargs)

File"/www/sites/blog/flask/lib/python2.7/sitepackages/flask/views.py",
   line 150, in dispatch_request return meth(*args, **kwargs)

File"/www/sites/MyBlog/app/posts.py", line 38, in get return
   render_template('posts/detail.html', **context)

File"/www/sites/blog/flask/lib/python2.7/sitepackages/flask/templating.py",
   line 125, in render_template context, ctx.app)

File"/www/sites/blog/flask/lib/python2.7/sitepackages/flask/templating.py",
   line 107, in _render rv = template.render(context)

File"/www/sites/blog/flask/lib/python2.7/sitepackages/jinja2/environment.py",
   line 894, in render return
   self.environment.handle_exception(exc_info, True)

File"/www/sites/MyBlog/app/templates/posts/detail.html", line 9, in
   toplevel template code {% block content %}

File"/www/sites/MyBlog/app/templates/posts/detail.html", line 23, in
   block "content" {{ forms.render(form) }}

File"/www/sites/MyBlog/app/templates/_forms.html", line 13, in template
   {{ field() }}

File"/www/sites/blog/flask/lib/python2.7/sitepackages/wtforms/fields/core.py",
   line 140, in __call__ return self.widget(self, **kwargs)

File"/www/sites/blog/flask/lib/python2.7/sitepackages/wtforms/widgets/core.py",
   line 246, in __call__ for val, label, selected in field.iter_choices():

File"/www/sites/blog/flask/lib/python2.7/sitepackages/wtforms/ext/sqlalchemy/fields.py",
   line 108, in iter_choices for pk, obj in self._get_object_list():

File"/www/sites/blog/flask/lib/python2.7/sitepackages/wtforms/ext/sqlalchemy/fields.py",
   line 99, in _get_object_list Display the sourcecode for this
   frameOpen an interactive python shell in this framequery = self.query
   or self.query_factory()

File"/www/sites/blog/flask/lib/python2.7/sitepackages/wtforms/ext/sqlalchemy/orm.py",
   line 117, in <lambda> 'query_factory': lambda:
   db_session.query(foreign_model).all() AttributeError: type object
   'Form' has no attribute 'query'

This is my view.py method
class DetailView(MethodView):

  def get_context(self, slug):

    p = Post('','',slug)
    post = p.get_or_404(slug)

    form = model_form(Comment, Form, exclude=['timestamp'])(request.form)
    context = {
      "post": post,
      "form": form
    }
    return context

  def get(self, slug):
    context = self.get_context(slug)
    return render_template('posts/detail.html', **context)

  def post(self, slug):
    context = self.get_context(slug)
    form = context.get('form')

    if form.validate_on_submit():
      comment = Comment()
      form.populate_obj(comment)

      post = context.get('post')
      post.comments.append(comment)
      db.session.commit()

      return redirect(url_for('posts.detail', slug=slug))

    return render_template('posts/detail.html', **context)

and this is my template for form:
{% macro render(form) %}
<fieldset>
{% for field in form %}
{% if field.type in ['CSRFTokenField', 'HiddenField'] %}
  {{ field() }}
{% else %}

    {{ field.label }}

      {% if field.name == "body" %}
        {{ field(rows=10, cols=40) }}
      {% else %}
        {{ field() }}
      {% endif %}
      {% if field.errors or field.help_text %}

        {% if field.errors %}
          {{ field.errors|join(' ') }}
        {% else %}
          {{ field.help_text }}
        {% endif %}

      {% endif %}

{% endif %}
{% endfor %}
</fieldset>
{% endmacro %}

I am getting an error related to following code
form = model_form(Comment, Form, exclude=['timestamp'])(request.form)

and:
{{ field() }}

please suggest some solution. Thanks in advance.
Comment: Have you manged to slove this?
Answer: I will post my solution even it's not perfect.
After struggling with this, I figure out that when db_session trying to query from the db, it's fails when the foreign_model is not he model I'm editing, it's actually a relationships I have.
SOLVED BY
exclude = ['user', 'offers', 'languages', 'company']
MyForm = model_form(Driver, Form, exclude=exclude)

In my model, I had to exclude all the relationships. It works fine, I still have no idea how to edit them, but in my situations I didn't have to.
UPDATE
I managed to solve it by passing my session and not the Form like this:
MyForm = model_form(Driver, db_session=db.session)
form = MyForm(request.form, driver)


How to get items that match certain ID from a javascript object

Question: I am trying to get comments for a particular page.
So I want to get all comments where the slangID = 0Uwx4oXQVEckArRSZNerwhfDZFF3 into an array or object.
Here is my javascript object:
{
"3d0366ccf32c4459b9a38aee0dde425a": {
"comment": "This is up\nI know",
"commentID": "3d0366ccf32c4459b9a38aee0dde425a",
"slangID": "34a97a464b51d81a592cf37c8d83cc9e",
"time_date": "2016-09-15T06:15:09.969Z",
"user_id": "0Uwx4oXQVEckArRSZNerwhfDZFF3",
"username": "james"
},
"4b0839bba81943e9cb511365d2978ad6": {
"comment": "aaaaaaa",
"commentID": "4b0839bba81943e9cb511365d2978ad6",
"slangID": "34a97a464b51d81a592cf37c8d83cc9e",
"time_date": "2016-09-15T06:54:36.026Z",
"user_id": "bHDKSU6aorX9efzBCHy",
"username": "ericel123"
},
"5064603984a3a217ae02cd0dab7ede81": {
"comment": "this\n",
"commentID": "5064603984a3a217ae02cd0dab7ede81",
"slangID": "34a97a464b51d81a592cf37c8d83cc9e",
"time_date": "2016-09-15T06:37:15.113Z",
"user_id": "0Uwx4oXQVEckArRSZNer",
"username": "james"
},
"59691edb302c4d0a70d6bb860e64d4e1": {
"comment": "hello\n",
"commentID": "59691edb302c4d0a70d6bb860e64d4e1",
"slangID": "34a97a464b51d81a592cf37c8d83cc9e",
"time_date": "2016-09-15T06:36:41.308Z",
"user_id": "0Uwx4oXQVEckArRSZNer",
"username": "james"
}
}

So I can put the comments in an object called 'comments'.
I tried filter like this:
var filtered =  vm.comments.filter(function(item) {
        return item.slangID === sid;
  });
 var comments = filtered;

it didn't work out.
Comment: Have you checked [Array.prototype.filter()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/filter)?
Comment: What did you try?
Comment: So loop over, check the id, if matches, push item to an array.... (And do not put swear words in your samples....)
Comment: Your filter looks at least 90% of the way there. I think you just need to debug it.
Answer: Let's assume your object is called commentList. Just loop all the keys in the object and check if the slang matches, then push to an array.
var result = [];
for(var key in commentList){
    if(commentList[key].slangID == theSlangImLookingFor)
        result.push(commentList[key]);

}

Answer: var obj ={
"3d0366ccf32c4459b9a38aee0dde425a": {
"comment": "This is fucked up\nI know",
"commentID": "3d0366ccf32c4459b9a38aee0dde425a",
"slangID": "34a97a464b51d81a592cf37c8d83cc9e",
"time_date": "2016-09-15T06:15:09.969Z",
"user_id": "0Uwx4oXQVEckArRSZNerwhfDZFF3",
"username": "james"
},
"4b0839bba81943e9cb511365d2978ad6": {
"comment": "aaaaaaa",
"commentID": "4b0839bba81943e9cb511365d2978ad6",
"slangID": "34a97a464b51d81a592cf37c8d83cc9e",
"time_date": "2016-09-15T06:54:36.026Z",
"user_id": "bHDKSU6aorX9efzBCHy",
"username": "ericel123"
},
"5064603984a3a217ae02cd0dab7ede81": {
"comment": "fuck this\n",
"commentID": "5064603984a3a217ae02cd0dab7ede81",
"slangID": "34a97a464b51d81a592cf37c8d83cc9e",
"time_date": "2016-09-15T06:37:15.113Z",
"user_id": "0Uwx4oXQVEckArRSZNer",
"username": "james"
},
"59691edb302c4d0a70d6bb860e64d4e1": {
"comment": "hello\n",
"commentID": "59691edb302c4d0a70d6bb860e64d4e1",
"slangID": "34a97a464b51d81a592cf37c8d83cc9e",
"time_date": "2016-09-15T06:36:41.308Z",
"user_id": "0Uwx4oXQVEckArRSZNer",
"username": "james"
}
}
Object.keys(obj).filter(function(k){
  return obj[k].slangID === "34a97a464b51d81a592cf37c8d83cc9e"
});

I have used "34a97a464b51d81a592cf37c8d83cc9e" since is the data you provided, change it for "0Uwx4oXQVEckArRSZNerwhfDZFF3" in your coude
Comment: Note that this will return the IDs of the matching comments rather than the comments themselves. You'd then need to look up the comment objects by the IDs.


Methods For Measuring Non-Linear Correlation?

Question: I have been learning about standard methods in Statistics such as the Pearson's Correlation Coefficient, Spearman's Correlation and Kendall's Tau.
My understanding of this so far is that:

Pearson Correlation Coefficient measures the linear correlation between two sets of data

Spearman's Correlation measures the "monocity" between two sets of data (e.g. do they both increase and decrease at the same time?)

Kendall's Tau measures the ordinal association between two sets of data - supposedly Kendall's Tau is similar to the Spearman Correlation, but Kendall's Tau has a more logical confidence intervals.

I had the following question - can any of these methods be used for measuring a specific form of "Non Linear Correlation" between two sets of data?
For example - suppose I want to see how strongly two sets of data are correlated relative to a "second order curve" :

Is there something that could measure the "curved correlation"?
The two ideas I came up with:

Try to use some data transformations (e.g. Log) to transform one of the variables into a more linear pattern that will make it suitable for one of the above measures

Fit a polynomial regression model (of order 2) to this data and measure the MSE

But I am not sure if either of these approaches are suitable.
Comment: (1) What do you mean by "measuring"? If you want a measure of the "strength" of such a correlation, then you could indeed run a polynomial regression and report the MSE. Possibly cross-validated, otherwise if you re-ran this for higher order polynomials, you would "find" that the "second-order correlation" is smaller than the "third-order correlation" and so on. Conversely, if you want to do statistical inference, the null and alternative hypotheses will need some thinking about - are $x$ and $x^3$ for $-1<x<1$ "significantly second order correlated"? ...
Comment: ... (2) Especially for inference, the question comes up whether you want to test a *specific* polynomial correlation, or a *general* second-order polynomial, or a general polynomial of *up to* second order. Perhaps you could explain what you want to do with such a nonlinear correlation?
Comment: Another way to consider Stephan's comments is that *every* regression you could estimate for the two variables in your plot is, in a sense, a correlation measurement. Testing and comparing arbitrarily many regressions has problems with false discovery and statistical validity, so "just try stuff" isn't a great way to go about it: you need to be specific about what questions you want to ask your data & how you want to ask it. The plot you show is roughly monotonic & Spearman's correlation would characterize the extent. Lots of nonlinear functions are monotonic, so Spearman's is **an** answer.
Comment: Interesting question. Some of the trouble of defining a curved correlation will be deciding on what kind of curvature you want to measure. After all, a logarithm-type of graph has different curvature than a quadratic. Further, determining the sign will be challenging, since many curves (such as quadratics) allow for increasing and decreasing sections. I've wondered if the concavity of a parabola (up-opening vs down-opening) could be used for this, but parabolas are just one type of curve. (Maybe you can do this if you restrict to convex or concave functions.)
Answer: You may be interested in distance correlation.
distance correlation measures both linear and nonlinear association between two random variables or random vectors. This is in contrast to Pearson's correlation, which can only detect a linear association between two random variables.
https://en.wikipedia.org/wiki/Distance_correlation
Comment: [For more about distance correlation, click here](https://stats.stackexchange.com/questions/183572/understanding-distance-correlation-computations)


Como trabalhar com o @Html.DropDownList?

Question: Amigos
Após seguir o exemplo fiz o ajuste no código, não estou conseguindo enviar a informação para a ActionResult

@model ProjetoDelphiMobile.Models.cliente

@{
    ViewBag.Title = "";
}

<form>
    <fieldset data-role="controlgroup">

           Foto:

                 @Html.DropDownList("idFoto", String.Empty)

@*

*@

           Nome:
            @Html.TextBoxFor(model => model.nome, new { disabled = false })

           Nome:
            @Html.TextBoxFor(model => model.nome, new { disabled = false })

            Nome do pai:
            @Html.TextBoxFor(model => model.pai, new { disabled = false })

            Nome da Mãe:
            @Html.TextBoxFor(model => model.mae, new { disabled = false })

            Data de Nascimento:
            @Html.TextBoxFor(model => model.datanascimento, new { disabled = false })

Retornar para consulta

    </fieldset>

</form>

Na minha Views:
        public ActionResult AtualizaFoto(string caminhofoto)
    {
        int largura = 100;
        int altura  = 100;
        WebImage webImagem = new WebImage(@caminhofoto).Resize(largura, altura, false, false);
        return File(webImagem.GetBytes(), @caminhofoto);
    }

Comment: Não é mais fácil usar jQuery pra fazer isso? Você pôs a tag na pergunta mas não está usando. Por que?
Comment: @CiganoMorrisonMendez, eu coloquei a Tag porque estou usando JQuery, agora se você diz que é mais fácil então me mostra como você faria para resolver esta situação?
Comment: Tranquilo. Em qual campo fica a informação de caminho?
Comment: ViewBag.idFoto = new SelectList(dao.fotos, "idFoto", "Foto");
O idFoto recebe o caminho que é visualizado dentro do DropDownList
Answer: Em jQuery, sua função ficaria assim:

Há mais alguns ajustes que serão necessários. No Controller, por exemplo, devolva um FileResult ao invés de um ActionResult:
public FileResult AtualizaFoto(string caminhofoto)
{
    int largura = 100;
    int altura  = 100;
    var webImagem = new WebImage(@caminhofoto).Resize(largura, altura, false, false);
    return File(webImagem.GetBytes(), @caminhofoto);
}

Outra coisa é este callback, que do jeito que está não vai funcionar. O certo seria algo como:
$.post("/ConsultaCliente/AtualizaFoto", { caminhofoto: srcRecebe }).done(function (data) {
    $('#caminho').attr("src", data);
})

Comment: fiz os ajustes baseado com seu exemplo,
Você poderia me ajudar a descobri o porque não estou conseguindo enviar a informação para a ActionResult, quero retornar  imagem. agradeço
Comment: Adicionei o código completo na minha pergunta.
Comment: @itasouza Modifiquei a resposta.
Comment: Fiz a atualização da resposta. Agradeço a sua ajuda.
http://pt.stackoverflow.com/questions/59059/como-chamar-a-mesma-actionresult-mais-de-1-vez-para-atualiza-a-p%C3%A1gina


NoMethodError in Articles#show after following step 6.4 of RoR Guide Creating a Blog

Question: I'm new to RoR and I'm trying to follow the guide at http://guides.rubyonrails.org/getting_started.html
I'm currently at section 6 point 4 but I'm getting stuck here.
I'm getting the following error: NoMethodError in Articles#show
Showing /Users/riaanvanstraaten/rubyapps/blog/app/views/articles/show.html.erb where line #12 raised:

undefined method `article_comments_path' for #<#<Class:0x007fac016d6d20>:0x007fac01c05f30>
Extracted source (around line #12):
10
11
Add a comment:

12 <%= form_for([@article, @article.comments.build]) do |f| %>
13

14    <%= f.label :commenter %>

15    <%= f.text_field :commenter %>

I've double checked the code from the guide but I cannot see where I am going wrong.
I've searched for some advise and followed some suggestions with no positive result.
I've one post that is very close to my problem, but the mentioned solutions does not resolve my current problem. This is the link to the similiar post: NoMethodError in Articles#show.undefined method `article_comments_path'
Here is the code from the files used

show.html.erb

  Title:
  <%= @article.title %>

  Text:
  <%= @article.text %>

Add a comment:

<%= form_for([@article, @article.comments.build]) do |f| %>

    <%= f.label :commenter %>

    <%= f.text_field :commenter %>

    <%= f.label :body %>

    <%= f.text_area :body %>

    <%= f.submit %>

<% end %>

<%= link_to 'Back', articles_path %> |
<%= link_to 'Edit', edit_article_path(@article) %>

comments_controller.html.erb

class CommentsController < ApplicationController
  def create
    @article = Article.find(params[:article_id])
    @comment = @article.comments.create(comment_params)
    redirect_to article_path(@article)
  end

  private
    def comment_params
      params.require(:comment).permit(:commenter, :body)
    end
end

routes.rb

Rails.application.routes.draw do

    resources :articles do
        resources :commments
    end
end

Here is my 'rake routes' output
               Prefix Verb   URI Pattern                                        Controller#Action
    article_commments GET    /articles/:article_id/commments(.:format)          commments#index
                      POST   /articles/:article_id/commments(.:format)          commments#create
 new_article_commment GET    /articles/:article_id/commments/new(.:format)      commments#new
edit_article_commment GET    /articles/:article_id/commments/:id/edit(.:format) commments#edit
     article_commment GET    /articles/:article_id/commments/:id(.:format)      commments#show
                      PATCH  /articles/:article_id/commments/:id(.:format)      commments#update
                      PUT    /articles/:article_id/commments/:id(.:format)      commments#update
                      DELETE /articles/:article_id/commments/:id(.:format)      commments#destroy
             articles GET    /articles(.:format)                                articles#index
                      POST   /articles(.:format)                                articles#create
          new_article GET    /articles/new(.:format)                            articles#new
         edit_article GET    /articles/:id/edit(.:format)                       articles#edit
              article GET    /articles/:id(.:format)                            articles#show
                      PATCH  /articles/:id(.:format)                            articles#update
                      PUT    /articles/:id(.:format)                            articles#update
                      DELETE /articles/:id(.:format)                            articles#destroy

What's wrong? I copied and pasted the code from the website to see what I'm doing wrong, but I still cannot to resolve the issue.
Thanks in advance for your help!
Answer: There is a typo in your routes.rb. You have 3 m letters in word comments instead of 2 here:
Rails.application.routes.draw do
  resources :articles do
    resources :commments
  end
end


subdomain redirects not working for https

Question: I am trying to make all subdomain.example.com requests be redirected to example.com. Using nginx file in /etc/nginx/sites-enabled:
server {
        server_name ~^(?.+)\.example\.com$;
        rewrite ^ $scheme://example.com$request_uri? permanent;
}

It works perfectly for HTTP requests. So it redirects http://anySubDomain.example.com to http://example.com
but not working for https:
https://mySubDomain.example.com returns 200 ok without redirects.
So how to redirect https://anySubdomain.example.com to https://example.com?
Comment: The `server` block in your question only handles `http` connections.
Comment: Possible duplicate of [Nginx config to redirect to $URI](https://serverfault.com/questions/663091/nginx-config-to-redirect-to-uri)
Answer: The following nginx file does what I want:
server {
        listen 443;
        listen 80;
        server_name ~^(?.+)\.example\.com$;
        rewrite ^ $scheme://example.com$request_uri? permanent;
}

It redirects https://anySubdomain.example.com to https://example.com and http://anySubdomain.example.com to http://example.com


Is it possible to express a decision tree as a kernel machine?

Question: This paper argues that models trained with gradient descent like neural networks can be expressed as kernel machines with an interesting kernel function.
The kernel is
$$ K(x, x') = \int_{c(t)} \nabla_w y(x) \cdot \nabla_w y(x') \, dt$$
where $c \colon [0, 1) \rightarrow \mathbb{R}^n$ is the path taken by the weight vector $w \in \mathbb{R}^n$ during gradient descent, and $y \colon \mathbb{R}^n \rightarrow \mathbb{R}$ denotes the network output.
Is there something similar for regression (decision) trees or gradient-boosted ensembles of regression trees?  Can we write a decision tree $f$ as being a kernel regression like
$$f(x) = \sum_{i} a_i K(x, x_i) + b$$ for training points $\{x_i\}$, some kernel function $K$ and learned weights $\{a_i\}, b$?.
Answer: In Domingo's paper, there is a note stating that Theorem 1 only approximates the behavior of a gradient-descent-trained model by a kernel machine. This approximation is not the result of a finite $\epsilon$ but that these coefficients $a_i$ are effective $a_i(x)$, and since they depend on the training data, that is not a standard linear (kernel) model. Concretely, it is not possible to explain a neural network as an exact kernel machine yet, with some notable exceptions (e.g. infinite width neural networks, e.g. https://arxiv.org/abs/2111.06063). Quite a few papers comment on the strengths and weaknesses of the above-described approach, for example in https://arxiv.org/abs/2212.11826, Appendix A2 (*), and here.
To the best of my knowledge, no results have been suggesting we can express a decision tree with a kernel machine. I'm expecting that, if such a procedure exists, is not based on Domingo's Path Kernel due to the fact that its Theorem 1 heavily relies on the gradient descent training.
Out of curiosity, I'm expecting the opposite to be true: any function of fixed input size $n$ can be computed by a decision tree, if we do not limit its depth (see here).
(*) Conflict of interest: I wrote that.
Comment: Thanks for the references, will have a look!


Regex to replace parentheses with square bracket

Question: I need to convert some variables that use parentheses instead of bracket in a Javascript file. For example, MyVariable(i) should be MyVariable[i].
I use the Find and Replace tool of Visual Studio with this Regex :
MyVariable\({(.+)}\)

and replace with:
MyVariable\[\1\]

This work fine for case like :
asdas.MyVariable(i+1)
asdas.MyVariable(i)
asdas.MyVariable(i).asd
MyVariable(i+1)

But doesn't work for case like
if (parseInt(OtherObject.MyVariable(i+1).Dest.XYZ)==SINK_STATE_TYPE || OtherObject.MyVariable(i).X == "ok")

The last one will do take the first parentheses and the last one of the line and act weird. What do I need to change in the Regex to be able to make it works with line that has multiple parentheses.
Answer: Your pattern is greedy. The (.+) part matches everything, including a closing parenthesis ) character until it reaches the last ) character. To make it non-greedy you can use # instead of +, which means "match one or more occurrences of the preceding expression, and match as few characters as possible." In .NET this is usually handled by placing a ? after it, such as .+? but the Visual Studio regex flavor is different and uses the # metacharacter instead.
The updated pattern becomes:
MyVariable\({(.#)}\)

You can also use [^)]+, which matches any character that is not a ) character. When it encounters a ) it will stop matching. This alternate pattern is:
MyVariable\({([^)]+)}\)

If you're familiar with the .NET regex flavor you might be interested in checking out the Visual Studio 2010 Productivity Power Tools, which has extended Find/Replace to allow the .NET regex flavor to be used.


How to get users (entities) for a certain field value?

Question: I want to write a function that returns users which have entered a certain value to a custom user field.
Lets say there is a custom birthday field for every user. What would be the best way to get all users with a certain birthday?
I could create a DB query that searches the field_data_field_birthday table and returns the entity_id(s) (which in this case would be the users id(s)) for which the field_birthday_value matches a given value and then load the user(s).
This seems a little bit crude, are there generic functions for this use case available? In the entity API or the field API maybe?
Answer: Simple example for the use case I asked about:
<code>$query = new EntityFieldQuery();
$query->entityCondition('entity_type', 'user')
  ->fieldCondition('field_birthday', 'value', $birthday);

$result = $query->execute();
$users_ids = array_keys($result['user']);
</code>
Comment: Worked like a charm.
Comment: Please advise the namespace to include for this to avoid Fatal Error.


Linked List filling empty list

Question: struct Node
{
 int val;
 struct Node *next;
};
struct List
{
 struct Node *head;
 struct Node *tail;
 int size;
};
typedef struct List *List;

This is my node and list, I want to add each i value to the node one by one by creating a new node in every iteration, but my program crashes I don't know what is the reason
    int i;
    List origin;
    List myList = (struct List*)malloc(sizeof(struct List));
    myList->head = (struct Node*)malloc(sizeof(struct Node));
    myList->head->next = NULL;
    myList->tail = myList->head;
    origin=myList;

    for(i=0;i<5;i++)
    {
        myList->head->next = (struct Node*)malloc(sizeof(struct Node));

        myList->head->next->val = i;

        myList->head->next = myList->head->next->next;

        myList->head->next->next = NULL;

    }

Comment: Does this answer your question? [Traversing through a linked list](https://stackoverflow.com/questions/14757732/traversing-through-a-linked-list)
Comment: no it does not @DavidCullen
Comment: What is `struct ListRecord` type?
Comment: @swag2198 mistyped it should be List only
Comment: Look closely at `myList->head->next->next`. What does `myList->head->next->next` contain at this point? It's an uninitialized pointer.
Answer: Few issues:

You are not traversing the list at all.
You are mutilating the list head pointer and not updating the tail pointer.
You are using head as a dummy node. It is good practice to set its value to some special marker, say -1.
Better use a temporary temp node to traverse and newnode for inserting new nodes to avoid uninitialized pointer access and assignments.

Modified main() function:
int main()
{
    int i;
    // List origin;
    List myList = (struct List*)malloc(sizeof(struct List));
    myList->head = (struct Node*)malloc(sizeof(struct Node));
    myList->head->val = -1; /* Set a value to the dummy first node */
    myList->head->next = NULL;
    myList->tail = myList->head;
    // origin = myList;

    struct Node* temp = myList->head; /* Use a temporary pointer to traverse */
    for(i=0;i<5;i++)
    {
        struct Node* newnode = (struct Node*)malloc(sizeof(struct Node));
        newnode->val = i;
        newnode->next = NULL;

        temp->next = newnode;
        temp = newnode;         /* Update temporary head to the new node */
        myList->tail = newnode; /* Update the tail to last node */
    }

    temp = myList->head;
    while(temp != NULL)
    {
        printf("%d", temp->val);
        if(temp != myList->tail)
            printf("->");
        temp = temp->next;
    }

    return 0;
}

Result:
-1->0->1->2->3->4

Also take care in deallocating the malloced memory using free() once you are done with the list. Also I did not understand the purpose of List origin, so I commented out that part.
Answer: To find the problem, we can add some debug output:
    for(i=0;i<5;i++)
    {
        myList->head->next = (struct Node*)malloc(sizeof(struct Node));
        printf("myList->head->next = %p\n", myList->head->next);
        myList->head->next->val = i;
        printf("myList->head->next->next = %p\n", myList->head->next->next);
        myList->head->next = myList->head->next->next;
        printf("myList->head->next = %p\n", myList->head->next);
        if (NULL == myList->head->next) {
            printf("We just set our next pointer to NULL, so we should stop now.\n");
            exit(1);
        }
        myList->head->next->next = NULL;
    }

We get this output:
myList->head->next = 0x7f8062c05800
myList->head->next->next = 0x0
myList->head->next = 0x0
We just set our next pointer to NULL, so we should stop now.

I made these changes to the initialization loop:
    struct Node *current = myList->head;
    for(i=0;i<5;i++)
    {
        current->next = (struct Node*)malloc(sizeof(struct Node));
        current->next->val = i;
        current->next->next = NULL;
        current = current->next;
    }

I also added an output loop to verify that the initialization worked:
    current = myList->head->next;
    while (current) {
        printf("%p = %d\n", current, current->val);
        current = current->next;
    }

Output
0x7fced6405800 = 0
0x7fced6405810 = 1
0x7fced6405820 = 2
0x7fced6405830 = 3
0x7fced6405840 = 4

Note
You weren't doing anything with the tail pointer, so I will let you figure out how you want to handle that.

Linked List filling empty list


Maximum Green function

Question: good morning
does anyone know how to prove that
$|G(x,t)| \le \frac{1}{2}$,
where
$G(x,t)=\begin{cases}
\frac{(x-1)(t+1)}{2} \quad -1 \le t \le x \le 1\\
\frac{(x+1)(t-1)}{2} \quad -1 \le x \le t \le 1\\
\end{cases}
$
Is it possible to do it, using the Hessian method?
Thanks
Answer: Clearly $G(x,t)$ is symmetric in $x,t$ and $G(0,0)=-\frac{1}{2}$so $%
|G(0,0)|=\frac{1}{2}$. In view of the symmetry in $x,t$ we can restrict
our attention to the case $x>t$. Then
$$
G(x,t)=\frac{(x-1)(t+1)}{2}\Rightarrow \;|G(x,t)|=\frac{(1-x)(1+t)}{2}%
=F(x,t).
$$
We have
$$
F(1,t)=0,\;F(x,-1)=0,\;F(0,0)=\frac{1}{2}
$$
and around $(0,0)$ we have
\begin{eqnarray*}
t &=&-\varepsilon <0<x=+\delta \;\Rightarrow \;F(x,t)=\frac{1}{2}(1-\delta
)(1-\varepsilon )<\frac{1}{2} \\
x &=&+\delta >t=+\varepsilon >0\;\Rightarrow \;F(x,t)=\frac{1}{2}(1-\delta
)(1+\varepsilon )<\frac{1}{2} \\
t &=&-\varepsilon <x=-\delta <0\;\Rightarrow \;F(x,t)=\frac{1}{2}(1+\delta
)(1-\varepsilon )<\frac{1}{2}
\end{eqnarray*}
Thus it seems that the maximum $1/2$ is obtained in $(0,0)$. I leave the
rest to you.

Maximum Green function


Ignite thread-safe variant of java.util.List

Question: Not Sure If Ignite provides thread safe (concurrent) version of java.util.List, or is there any hack that should be done to achieve the same functionality that of concurrent List
Answer: These are distributed data structures that are supported by Apache Ignite at the moment. A distributed list is not among them. Probably you may use distributed queue or set for your use case.
Basically, you may store java.util.List as a cache entry updating the list and picking a particular element from it with IgniteCache.invoke(...) method. This EntryProcessor based approach will avoid a necessity to send the list over the wire every time you need to add a new element to it or take an existed one.


Rewrite last part of file read

Question: I see how to change the pointer in a file to read just part of the file (fseek).
Does setting the pointer for rewriting ('fwrite') part of the file work the same way?
After the 'fread' do I leave pointer alone or backup from current position to rewrite what I just read?
Note the idea is to exchange two blocks of the file for randomization.
Comment: Could you provide some feedback on my answer to your question or accept it as the correct answer?
Answer: Yes fwrite works like fread. After fread, you need to fseek to the position to write to. Here's an example:
// get temporary directory
$tmp_dir = sys_get_temp_dir() . DIRECTORY_SEPARATOR;

// get temporary file name in temporary dir
$tmp_filename = tempnam($tmp_dir, '');

// open temp file for read and write
$handle = fopen($tmp_filename, 'r+');

$str = 'the quick brown fox jumps over   the lazy dog';

echo($str . PHP_EOL);

// write string to file
$result = fwrite($handle, $str);

$pos = strpos($str, 'jumps');

// seek to jumps position
$result = fseek($handle, $pos);

// change jumps for sneaks under
$str2 = 'sneaks under';

$result = fwrite($handle, $str2);

// get file size
$file_size = filesize($tmp_filename);

// seek to file start
$result = fseek($handle, 0);

// read from file
$result = fread($handle, $file_size);

echo($result);

// close file
fclose($handle);

$ php src/fwrite.php
the quick brown fox jumps over   the lazy dog
the quick brown fox sneaks under the lazy dog

Rewrite last part of file read


Simple JavaScript/HTML Slideshow

Question: New to Javascript, but after some research it loks like this would be the best method in implementing my desired output. I'm trying to produce a slideshow of images (5 pre-selected images) that automatically change between 5 second intervals. Can anyone point towards a tutorial or guide me along in this process? Any help is very much appreciated.
Comment: Are you looking to implement this as an exercise, or is a pre-made solution in the form of (say) a jQuery plugin acceptable? There are many freely-available slideshow scripts that can do exactly what you're looking for.
Answer: Here is a Very simple code to create simple JavaScript/HTML slideshow only by using simple JavaScript and HTML codes :

Comment: the `language` attribute of the `script` tag is deprecated. I'd recommend using `type` instead.
Answer: A really nice jQuery slideshow type plugin is this http://www.devtrix.net/sliderman/
It has many different transitions between slides, and is really easy to use.
There are many out there, so a quick google search of "jQuery Slideshow" will produce hundreds of results.
Answer: Just google for javascript content sliders
Here's 350 image and content sliders:
http://www.jqueryrain.com/example/jquery-slider-slideshow/
and here's 25 more:
http://vandelaydesign.com/blog/web-development/jquery-image-galleries/.
Comment: Updated the URL's for current ones in 2015
Answer: There are thousands of examples, some very complex and sophisticated. The difficoult, rather, is to find one easy to understand for beginners.
After some research I've found this from css-tricks. (Uses jquery also).
Answer: I know this is an old post but I though I would share my tutorial, for the benefit of anyone who comes across this question in future.
Simple Slideshow
Hope this helps somebody.
Its a pretty simple and basic slideshow that is easy to build / implement.
Just In case the link becomes invalid here is the code:
The first stage is to set up the html as follows:

The Slide Title

This is the slide text

Next we will write the CSS, which is as follows:
img {
 display: block;
 width: 100%;
 height: auto;
}
p{
 background:none;
 color:#ffffff;
}
#slideShow #slideShowWindow {
 width: 650px;
 height: 450px;
 margin: 0;
 padding: 0;
 position: relative;
 overflow:hidden;
 margin-left: auto;
 margin-right:auto;
}

#slideShow #slideShowWindow .slide {
 margin: 0;
 padding: 0;
 width: 650px;
 height: 450px;
 float: left;
 position: relative;
 margin-left:auto;
 margin-right: auto;
 }

#slideshow #slideshowWindow .slide, .slideText {
    position:absolute;
    bottom:18px;
    left:0;
    width:100%;
    height:auto;
    margin:0;
    padding:0;
    color:#ffffff;
    font-family:Myriad Pro, Arial, Helvetica, sans-serif;
}
.slideText {
 background: rgba(128, 128, 128, 0.49);
}

#slideshow #slideshowWindow .slide .slideText h2,
#slideshow #slideshowWindow .slide .slideText p {
    margin:10px;
    padding:15px;
}

.slideNav {
 display: block;
 text-indent: -10000px;
 position: absolute;
 cursor: pointer;
}
#leftNav {
 left: 0;
 bottom: 0;
 width: 48px;
 height: 48px;
 background-image: url("../Images/plus_add_minus.png");
 background-repeat: no-repeat;
 z-index: 10;
}
#rightNav {
 right: 0;
 bottom: 0;
 width: 48px;
 height: 48px;
 background-image: url("../Images/plus_add_green.png");
 background-repeat: no-repeat;
 z-index: 10; }

As you can see there isn't anything exciting or complicated about this CSS.
In fact it doesn't get more basic, but I promise that's all that's needed.
Next we will create the jQuery:
$(document).ready(function () {

 var currentPosition = 0;
 var slideWidth = 650;
 var slides = $('.slide');
 var numberOfSlides = slides.length;
 var slideShowInterval;
 var speed = 3000;

 //Assign a timer, so it will run periodically
 slideShowInterval = setInterval(changePosition, speed);

 slides.wrapAll('

');

 slides.css({ 'float': 'left' });

 //set #slidesHolder width equal to the total width of all the slides
 $('#slidesHolder').css('width', slideWidth * numberOfSlides);

 $('#slideShowWindow')
 .prepend('Move Left')
 .append('Move Right');

 manageNav(currentPosition);

 //tell the buttons what to do when clicked
 $('.slideNav').bind('click', function () {

 //determine new position
 currentPosition = ($(this).attr('id') === 'rightNav')
 ? currentPosition + 1 : currentPosition - 1;

 //hide/show controls
 manageNav(currentPosition);
 clearInterval(slideShowInterval);
 slideShowInterval = setInterval(changePosition, speed);
 moveSlide();
 });

 function manageNav(position) {
 //hide left arrow if position is first slide
 if (position === 0) {
 $('#leftNav').hide();
 }
 else {
 $('#leftNav').show();
 }
 //hide right arrow is slide position is last slide
 if (position === numberOfSlides - 1) {
 $('#rightNav').hide();
 }
 else {
 $('#rightNav').show();
 }
 }

 //changePosition: this is called when the slide is moved by the timer and NOT when the next or previous buttons are clicked
 function changePosition() {
 if (currentPosition === numberOfSlides - 1) {
 currentPosition = 0;
 manageNav(currentPosition);
 } else {
 currentPosition++;
 manageNav(currentPosition);
 }
 moveSlide();
 }

 //moveSlide: this function moves the slide
 function moveSlide() {
 $('#slidesHolder').animate({ 'marginLeft': slideWidth * (-currentPosition) });
 }

});

Comment: What's with the down vote? More to the point, who thought is necessary to down vote my answer but didn't deem the situation worthy of commenting on? I have no issue with down votes but I'd like to know why, how hard is it to leave a comment? without constructive criticism how am I to improve upon my answers?
Comment: Ok thanks, I got it. It's working very nicely
Comment: how to stop slider on hover?
Comment: To stop on hover my suggestion would be to interupt the timer function on hover. So stop the timer on mouseover and restart it on mouseleave.

Simple JavaScript/HTML Slideshow