Inverse of matrix with very structured submatrix

Question: Does this matrix admit an easy analytic expression for its inverse?
$$\begin{bmatrix}
a_1 & 0 & 0 & 0 & 0 &0&\dots&0 \\
a_2 & 1 & -b & 0 & 0&0&\dots&0 \\
a_3 & 0 & 1 & -b & 0 & 0&\dots&0\\
a_4 & 0 & 0 & 1 & -b & 0 &\dots&0\\
\vdots \\
a_{N-1} & 0 & 0 & 0 & 0 & 0 & \dots & -b \\
a_N & 0 & 0 & 0 & 0 & 0 & \dots & 1
\end{bmatrix}$$
The special thing is that it has a very easy structure, apart from row one/column one.
Comment: Is the $(1,2)$ element $0$, and not $-b$?
Answer: Yes, for $a_1\ne 0$ it does. Hints:

Show that for a block-triangular matrix
$$
\begin{bmatrix}
A & 0\\B & C
\end{bmatrix}^{-1}=
\begin{bmatrix}
A^{-1} & 0\\-C^{-1}BA^{-1} & C^{-1}
\end{bmatrix}.
$$
Show that for the $n\times n$ matrix
$$
C=I-bN,
$$
where $N$ is the matrix of all zeros except of ones on the first superdiagonal, the inverse is given by
$$
C^{-1}=\sum_{k=0}^{n-1}b^kN^k.
$$
(e.g. using the fact that $\frac{1}{1-x}=1+x^2+x^3+\ldots$ and $N^n=0$).
Answer: First, left-multiply by $D=\operatorname{diag}(1/a_{1},1,1,\ldots,1)$
to get
$$
B\equiv DA=\left(\begin{array}{ccccc}
1 & 0 & 0 & 0 & 0\\
a_{2} & 1 & -b\\
a_{3} &  & 1 & -b\\
\vdots &  &  & 1 & -b\\
a_{N} &  &  &  & 1
\end{array}\right).
$$
Expanding along the first row, the determinant of $B$ is (see also this page about the continuant)
$$
\det B=\det\left(\begin{array}{ccccc}
1 & -b\\
 & 1 & -b\\
 &  & \ddots & \ddots\\
 &  &  & 1 & -b\\
 &  &  &  & 1
\end{array}\right)=1.
$$
Denoting by $C\equiv(c_{ij})$ the matrix of cofactors of $B$, we
have
$$
c_{ij}=(-1)^{i+j}\det B_{-(ij)}
$$
where $B_{-(ij)}$ is the matrix $B$ with the $i$-th row and $j$-th
column removed. Lastly,
$$
A^{-1}=B^{-1}D=\frac{1}{\det B}C^{T}D=C^{T}D
$$
and so it is just a matter of computing the form of $C$. I ran out
of time, but this hint should get you started.
Answer: If we call your matrix $A$, then
$$
A^{-1} =
\begin{pmatrix}
\frac{1}{a_1} & 0 & 0 & 0 & 0 & \dots \\
-\frac{1}{a_1b^2} \sum_{k=2}^Na_kb^k & 1 & b & b^2 & b^3 & \dots \\
-\frac{1}{a_1b^3} \sum_{k=3}^Na_kb^k & 0 & 1 & b & b^2 & \dots \\
-\frac{1}{a_1b^4} \sum_{k=4}^Na_kb^k & 0 & 0 & 1 & b & \dots \\
\vdots & \vdots & & \ddots & \ddots & \\
-\frac{a_N}{a_1} & 0 & 0 & 0 & \dots & 1
\end{pmatrix},
$$
which you can verify by a direct calculation.


How to define mapping of Geo_Points in elasticsearch?

Question: Can anybody please tell me what is the correct way of mapping a geo_point in elasticsearch, i also want to index lats and longs seperately, from below two declarations which is correct
location          : {"type":"geo_point", "lat_lon": "true"},

OR
location          : {"type":"geo_point", "lat_lon": "true"
                  lat: { type:"double"},
                  lon: { type: "double"}

},
Answer:
lat_lon: Set to true to also index the .lat and .lon as fields. Defaults to false. (link)

So you should use the first solution.
Example


Why use gradient descent with neural networks?

Question:
When training a neural network using the back-propagation algorithm, the gradient descent method is used to determine the weight updates. My question is: Rather than using gradient descent method to slowly locate the minimum point with respect to a certain weight, why don't we just set the derivative $\frac{d(\text{Error})}{dw}=0$, and find the value of weight $w$ which minimizes the error?
Also, why are we sure that the error function in back-propagation will be a minimum? Can't it turn out the error function is a maximum instead? Is there a specific property of the squashing functions that guarantees that a network with any number of hidden nodes with arbitrary weights and input vectors will always give an error function that has some minima?
Comment: @Nick Cox my apologies
Comment: All caps titles are not standard here (please look around you) and here and elsewhere widely deprecated as unwelcome SHOUTING.
Comment: FYI [why is Newton's method not widely used in machine learning?](http://stats.stackexchange.com/q/253632/12359)
Comment: It is interesting to see whenever hidden or latent variables are used in Machine Learning models, optimization (almost?) always gets non-linear, non-convex and just harder to optimize.
Answer:
Because we can't. The optimization surface $S(\mathbf{w})$ as a function of the weights $\mathbf{w}$ is nonlinear and no closed form solution exists for $\frac{d S(\mathbf{w})}{d\mathbf{w}}=0$.
Gradient descent, by definition, descends. If you reach a stationary point after descending, it has to be a (local) minimum or a saddle point, but never a local maximum.
Comment: This cannot happen, because all commonly used error functions have a strict theoretical minimum of 0. Errors can never become negative.
Comment: One other possible interpretation of 1. is "That's exactly what we do, the equation is solved using gradient descent."
Comment: there clearly is a closed form for the gradient(that's how we do gradient descent efficiently) .  The problem is no closed form root of gradient =0
Comment: If the function was concave, gradient decent would descend forever since the only way to go is downwards. Are you saying that the error surface is guaranteed to not be concave? Also, its not clear to me why the derivative of the error function would have no closed form solution. Isnt the error of the form $K-\frac{1}{1+e^{\Sigma wx}}$ where K is a constant? That function looks fairly differentiable and the resulting expression analytically solvable. Please help me clarify as there is something I clearly fail to see.
Comment: @seanv507 that's what I intended to say, sorry for the confusion. Edited my post.
Comment: Do you have a source or proof as to why no closed form exists, or a textbook that could help me understand in what cases a closed form does exist?
Answer: Regarding Marc Claesen's answer, I believe that gradient descent could stop at a local maximum in situations where you initialize to a local maximum or you just happen to end up there due to bad luck or a mistuned rate parameter.  The local maximum would have zero gradient and the algorithm would think it had converged.  This is why I often run multiple iterations from different starting points and keep track of the values along the way.
Answer: In Newton-type methods, at each step one solves $\frac{d(\text{error})}{dw}=0$ for a linearized or approximate version of the problem. Then the problem is linearized about the new point, and the process repeats until convergence. Some people have done it for neural nets, but it has the following drawbacks,

One needs to deal with second derivatives (the Hessian, specifically Hessian-vector products).
The "solve step" is very computationally expensive: in the time it takes to do a solve one could have done many gradient descent iterations.

If one uses a Krylov method for the Hessian solve, and one does not use a good preconditioner for the Hessian, then the costs roughly balance out - Newton iterations take much longer but make more progress, in such a way that the total time is roughly the same or slower than gradient descent. On the other hand, if one has a good Hessian preconditioner then Newton's method wins big-time.
That said, trust-region Newton-Krylov methods are the gold-standard in modern large-scale optimization, and I would only expect their use to increase in neural nets in the upcoming years as people want to solve larger and larger problems. (and also as more people in numerical optimization get interested in machine learning)
Comment: My point is that this multiplication by hessian and conjugate gradients was known in nnets community since say 1994. So I believe there is definitely a reason why SGD is used rather than second order methods (and I would certainly like a clear resolution of why this is)
Comment: I think you are mistaken.  People have been using nnets since the 90s, and they are well aware of second order methods. the problem is precisely that nnets are successful when there is a lot of data, which then support a lot of parameters in which case the time and memory constraints of second order methods are ineffective. see eg http://leon.bottou.org/publications/pdf/compstat-2010.pdf
Comment: @seanv507 Not really. The discussion of second order methods in that paper has a lot of flaws, in that they assume one must construct and invert the entire dense Hessian in order to use second order methods. This is simply not how it is done in modern large-scale numerical optimization. In modern second order methods one computes the action of the Hessian on vectors by solving adjoint problems, and uses them within an iterative (Krylov) solver. Generally the first inner iteration returns the gradient direction, and subsequent iterations improve it.
Comment: Although I am not a particular fan of that paper,  I don't think that is true. He has previously discussed/implemented diagonal and reduced rank  approximations of hessian. And what about pearlmutter's 1994 paper fast exact multiplication by hessian?
Comment: Right. Once you have fast Hessian applications (whether through Pearlmutter or what have you), you can do inexact Hessian solves with Krylov methods like conjugate gradient. By doing this, one effectively transfers the ill-conditioning difficulties away from the nonlinear iterative optimizer, onto the linear algebra iterative solver where one has a lot of machinery and preconditioning techniques available to deal with the problem. A good reference is the section on trust region CG-Steihaug in the classic "Numerical Optimization" by Nocedal and Wright.


Como baixar um projeto do bitbucket por git

Question: Bom dia!
Nunca tinha utilizado git nem bitbucket, queria saber como faço um check out de um projeto que está no bitbucket.
Agradeço desde já.
Comment: normalmente, `git clone remote-url-do-repo` e depois `git checkout master`; Mas esta informação vem, normalmente, no repositório que estás a querer clonar..
Comment: @MoshMage, já fiz o git clone, como faço o checkout? Me retorna o seguinte `fatal: Not a git repository (or any of the parent directories): .git`
Comment: faz `cd nome-da-pasta` e depois sim, `git checkout master`
Comment: @MoshMage, perdão. Sou novo nessa área e não consegui compreender, tens como fazer uma resposta explicando? Assim também poderia receber ela como aceita.
Answer: Como fazer git-clone de um projecto:
git clone remote-url-do-repo
cd nome-do-projecto
git fetch
git checkout master

O Primeiro passo faz a clonagem do projecto para a tua maquina, para dentro de uma pasta com o nome do projecto, a partir da pasta onde estás. Ou seja: Se estiveres em /home/username/ e o projecto se chamar xpto esse comando irá criar a pasta xpto dentro da pasta /home/username/.
O segundo comando serve para entrares dentro da pasta (cd = Change Directory) que acabas-te de clonar.
O terceiro comando (fetch) serve para o git ir buscar todos os branches, etc, etc, etc.. que existem.
O ultimo comando faz com que tu faças checkout para o branch master


Hiding referenced term name from node display

Question: I've stumbled upon a situation where I want to display a link and a custom icon for the link (I'm using paragraphs module) which will be themed correctly later, BUT. After providing a taxonomy term with a image field and making it display ONLY the image by default I'm still getting a h2 of the linked terms name.
I already tried hook_entity_view, but it doesn't have separate values for icon and the randomly displayed term name - only the term id meaning I cannot unset it at this hook.
Any suggestions/experience on how should we work with this?
Answer: In the content type or paragraph type go to Manage Display and configure the format of the term field as "Rendered entity".
Copy taxonomy-term.html.twig to the template folder of your theme and remove the linked title:
{{ name }}

taxonomy-term.html.twig

  {{ title_prefix }}
  {% if not page %}

{{ name }}

  {% endif %}
  {{ title_suffix }}
  {{ content }}

Answer: Have you checked the corresponding template?
To be sure just set debug: true in your root project services.yml file.
the inspect the HTML code with your web browser dev tool.
You should find comments like (just an example here):

Then find the corresponding template and edit it in order to hide the label.
In your case that template could be a field template or a term view mode template.


Rotas react react-router-dom

Question: Estou iniciando no react e estou tentando criar links de navegação interna. Estou na duvida se estou fazendo da forma correta.
criei o arquivo routes
import React from 'react'
import {BrowserRouter, Switch, Route} from 'react-router-dom'
import { Profile } from '../components/Profile';
import { Skills } from '../components/Skills'

export default function Routes (){
    return(
        <BrowserRouter>
            <Switch>
                <Route path="/" element={Profile} />
                <Route path="/skills" element={Skills} />
            </Switch>
        </BrowserRouter>

    );
};

Importei o arquivo Routes no index
import { Education } from '../components/Education'
import { Experience } from '../components/Experience'
import { NavigationBar } from '../components/NavigationBar'
import { Profile } from '../components/Profile'
import { Skills } from '../components/Skills'
import Routes from './routes'

export default function Home() {
  return (

        <NavigationBar />

        <Profile />

        <Skills />

        <Education />

        <Experience />

        <Contacts />

      <Routes />

  )
}

E estou tentando usar ele na minha navbar
import styles from '../styles/components/NavigationBar.module.css'
import './Skills.tsx'
import {Link} from 'react-router-dom'

export function NavigationBar() {
    return (

            <Link to="/">
Inicio
</Link>

<Link to="/skills">Habilidades</Link>

    );
}

Segue meu app
import React from 'react'
import '../styles/globals.css'

function MyApp({ Component, pageProps }) {
  return <Component {...pageProps} />
}
export default MyApp

Estou tendo o seguinte erro
Error: Invariant failed: You should not use <Link> outside a <Router>

Answer: Você vai precisar envolver as tag BrowserRouter em toda aplicação, segue exemplo abaixo.
E não precisa declarar novamente as paginas no seu Home, pois elas serão renderizadas conforme a rota:
    import { Education } from '../components/Education'
    import { Experience } from '../components/Experience'
    import { NavigationBar } from '../components/NavigationBar'
    import { Profile } from '../components/Profile'
    import { Skills } from '../components/Skills'
    import Routes from './routes'

    export default function Home() {   return (
    <BrowserRouter>

             <NavigationBar />

          <Routes />

    </BrowserRouter>   ) }

Rotas react react-router-dom


Mouse scroll wheel in Citrix

Question: One of my users is experiencing a problem whereby the mouse scroll does not work on the Citrix published application. I have tried to install mouse drivers on his local PC as well as swapping the USB with a PS/2 mouse but still no go. I have researched this issue for a few days but have not been able to find a fix.
PC is running Windows XP and ICA client is version 10
I am pretty sure its a problem local to that PC since every other user using the application published on the same server are not having any issues. What could it be?
Comment: Does that mean that the mouse wheel works fine on that computer everywhere else?  Also, what application is being published?
Comment: @Paul - Hi Paul, Yes that's correct. It works fine on that computer everywhere else. The application is a third party application that is used by about 10 other users and none of them have experienced this issue before. I escalated to the vendor and they came back saying that they have not seen it before as well. At this moment, I believe its local to that PC but I don't know what else to try... Mouse properties in Control Panel does show that the mouse has a wheel so its not like Windows is not detecting it at all
Comment: Do other users on that same PC have the same issue?  How about regular users vs admin users on that PC?  If possible, can you try just RDP to the server to run the app and see if the problem still exists?

Sorry, I'd post an answer instead of comments, but I don't have an answer yet, just more questions :).
Comment: @Paul - to answer your question, Yes, other users on the same PC have that issue and that user is a local admin of the PC. RDP to the server and running the app does not result in the scrolling issue - meaning he can scroll when using RDP. Thanks again for your comments :)
Answer: I guess I might as well start an answer for this.  Any chance you can post the Citrix ini files located at %userprofile%\application data\icaclient, particularly APPSRV.ini?  Also, do you use an ICA file to make the connection?  If you can't/would prefer not to post the files, try to compare all of them with a working computer.  These files also exist in the Citrix folder under Program Files somewhere, which you can also compare.  In case you haven't already tried it, you might want to attempt to reinstall Citrix on this client.

Mouse scroll wheel in Citrix


iphone - Displaying the videocamera as background with OpenGLES models on top

Question: Hi I'm having difficulties setting as background the Iphone's 4 videocamera and drawing simple cubes with openglES on top. What's the best way to achieve that? My project structure is simple, i have in the delegate the opengl class which draws the cubes perfectly and the controller doesn't do much.
So aparently the camera controller is the following code which gas to be set in the opengl view class, but it's not working.
<code>UIImagePickerController* imagePicker = [[UIImagePickerController alloc] init];
imagePicker.delegate = self;
imagePicker.navigationBarHidden = YES;
imagePicker.toolbarHidden = YES;
imagePicker.sourceType = UIImagePickerControllerSourceTypeCamera;
imagePicker.showsCameraControls = NO;
imagePicker.cameraOverlayView = self;

// The 54 pixel wide empty spot is filled-in by scaling the image.
// The camera view's height gets stretched from 426 pixels to 480 pixels.

float bandWidth = 54;
float screenHeight = 480;
float zoomFactor = screenHeight / (screenHeight - bandWidth);

CGAffineTransform pickerTransform = CGAffineTransformMakeScale(zoomFactor, zoomFactor);
imagePicker.cameraViewTransform = pickerTransform;

m_viewController = [[UIViewController alloc] init];
m_viewController.view = self;
[m_viewController presentModalViewController:imagePicker animated:NO];
</code>
Any tips?
Thanks
Answer: Check this example (this is exactly what you need).
Answer: You can try sample code for AR from here.
Not 100% what you looking for, but might be helpful in this situation.


How to improve a Fraud Classification Model?

Question: I built a classification model (Logistic Regression) in order to classify data in Fraud or Not Fraud.
This data is related with online CNP (Card Not Present) transactions and after choosing some parameters that seemed to be related with fraud, I tested the model.
For this I used a training set of 225000 examples, and a test set of 75000.
I conducted two different tests, in the first one I had 7 parameters and managed to obtain a 96% accuracy in classification. The problem is that the number of Fraud cases is much lower than the Not Fraud ones, and so regarding the Fraud cases I got an accuracy of only 11% while on the Not Fraud 90 and something %.
In the second test I included more parameters, making a total of 15.
Same training and test set size, and although the overall classification dropped to 92%, regarding the cases of Fraud I got an improvement to 30%, and Not Fraud still around the 90s%.
I would like to maintain the overall classification accuracy around 90%, and improve the accuracy on Fraud cases to something like 65~75%, but I can't find more parameters that seem to be relevant, to include in my model and I am stuck as other than that, no more ideas come to mind...
Can someone please give me some hints or ideas on what to try next in order to try to achieve these goals?
Also, I have another doubt. Because the values of the parameters that I am using, have a very wide range, I applied Feature Scaling and Mean Normalization over them. I have 300000 example samples (Training set - 225000 and Test set - 75000). My question is, for each of these sets should I calculate the respective Average of each column and the Max - Min, in order to convert the values to a smaller scale, or should I calculate the average and max-min, based on the whole sample (the 300000 examples)?
Answer: What proportion of your 225,000 training samples are cases of fraud? I suspect very few. This will cause problems unless you take care in how you build a classifier from the logistic regression.
Given the issue you've described, I assume you are making your classifications based on a cut-off of a probability of $0.5$ from the logistic regression. You either need to choose a more appropriate cut-off, or weight the fraud samples (a weight of $19$ would be appropriate given you have $19$ times more not fraud cases in your data set), or discard a lot of the not fraud cases so you have a balanced data set.
As for your second question, to properly assess the out of sample performance I would calculate the average/min/max of each column only using your training set. As an alternative to scaling you might consider binning the relevant variables.
Comment: Yes, in the training sample, the number of Fraud cases is much smaller than the cases of Not Fraud (like 5% Fraud, 95% Not Fraud). I thought about picking all the cases of Fraud and an equal number of Not Fraud cases, but would that make a difference, since all those cases are used in the training sample anyway?

Regarding my second question, what you are saying is to use the calculated average/min/max (based on the training set), and apply these values to the test set? Is that it?
Comment: I've edited my answer to go into a bit more detail about the issue with the unbalanced data. For the second question, yes, calculate the parameters of the transformation based on your training set and apply them to the test set
Comment: Thanks for the reply. I did as you suggested and used a Training example size with approximately the same number of positive and negative cases (5000 each). I repeated the tests, and in both cases (7 and 15 parameters), the results were very similar. Overall classify accuracy dropped to 50%, but the Fraud cases correctly classified improved to also 50%. I would like to try to apply weights, but regarding that I have this very same problem: http://stats.stackexchange.com/questions/65382/adding-weights-for-highly-skewed-data-sets-in-logistic-regression
Comment: I'm not sure what the link says about there being a problem with using weights. Either using weights or using a different cut-off should help with both your issue and the linked issue (which are the same).
Comment: Sorry, you misunderstood me. There is no problem with using weights. The problem is I don't know how to apply them in my implementation of Logistic Regression, the exactly same problem that is referred in the link.
Comment: OK, now I understand. What software are you using to fit the logistic regression?
Comment: I'm using Octave, and in my implementation I used the very same equations that the person in the other link used.
Comment: Unfortunately I'm not familiar with `Octave` but in `R` it's trivial to specify weights for each observation by setting the `weights` parameter to `glm()`. I would expect it to be equally straight forward for any statistical software. Alternatively you could avoid this issue of setting weights by focusing on the classifier cut-off instead.
Comment: Thanks for your patience @M.Berk. Could you please be more specific about what I have to do regarding the cut-off?
Comment: How are you currently performing classification? In other words, how do you go from your logistic regression, the output of which is the probability of each case being fraud, to assigning a label to each case?
Comment: I have a function _oneVsAll_ that receives an _X_ training sample and the respective _y_, the num_labels(2), and _lambda_(0.1). Based on these parameters I use the Octave function _fminunc_ and get an [all_theta]. Then in order to classify the test batch, I have a function _predictOneVsAll_ that receives all_theta and X2 (the Test sample), and I pass the max of each row from sigmoid(X2 * all_theta') to a vector _p_. Then I compare each value of _p_ with _y2_
Comment: But $p$ is a probability and $y2$ is a label (either 0 or 1) correct? Perhaps you need to explain more about what you mean by classifier accuracy in your question.
Comment: No, _p_ is the vector of predictions, based on the max of each row from sigmoid(X2 * all_theta'). _y2_ is a label. The classifier accuracy is the (mean(double(p == y2)) * 100)
Comment: What are the dimensions of X2 and all_theta? (I'm confused as to why their product would not be a vector) Isn't the output of the sigmoid function a value between 0 and 1? (I still don't get how the "max of each row from sigmoid(X2*all_theta')" gives a value of either 0 or 1, not between 0 and 1)
Answer: Depending which implementation of logistic regression you're using, scaling will probably already be done. And yes, normalizing your features is very important or the least squares cost at the heart of your algorithm will have some problems. Here's a python standard scaler if you can't find one elsewhere.
As an additional note, the overall "accuracy" isn't something you should be worried about since the recall of fraud cases is probably more important. The recall is just the number of fraud cases you accurately predict correctly divided by the number of true fraud cases.
EDIT:
@Drew75 brought up that I misunderstood how the coefficients were estimated. Scaling shouldn't matter for this particular classifier.
Comment: Is this true? Logistic regression doesn't work in the same was as a linear (ordinary least-squares) regression. As far as I know, the logit function criteria is deviance, so scaled vs unscaled shouldn't have an impact.
Comment: I thought that logistic regression still determines linear coefficients before making classifications. Least squares is usually involved in the cost function of that process, or is probably effected by scaling. That's not the only way to determine those coefficients, but I guess it might depend on he logistic regression algorithm? Damn, now I've got a question.
Comment: There are linear coefficients, but they are estimated in relation with to the link function (here logit) via an iterative process. Unliked OLS, there is no closed form solution.
Comment: My bad, post edited.


A square matrix $A$ is such that $\mathrm{rank}(A^{k}) = \mathrm{rank}(A^{k+1})$

Question:
A square matrix $A$ is such that $\mathrm{rank}(A^{k})=\mathrm{rank}(A^{k+1})$. Then $\mathrm{rank}(A^{k})=\mathrm{rank}(A^{k+i})$ for all $i$.

I think we prove this by induction on $i$.
Let $i = 2$. We show that rank $(A^{k})$ = rank $(A^{k+2})$.
Now, we know that rank $(A^{k+2})$ = rank $(A^{k+1}A)$ $\le$ min (rank$(A^{k+1})$, rank$(A)$) $\le$ rank $(A^{k+1})$ = rank $(A^{k})$.
However, I cannot proceed further.
Answer: Think geometrically. We have $A^{k+1}=A^kA$, so the image of $A^{k+1}$ is contained in the image of $A^k$. If the rank of the two are equal, that means the image is the same. Since $A^{k+1}=AA^k$, this means that $A$ is a bijection on said image.
Answer: This can be nicely shown by thinking of $A$ as a linear transformation $f \colon K^n \to K^n$ and looking at the decreasing chain $\operatorname{im} f^p \supseteq \operatorname{im} f^{p+1}$ as follows:
If $A \in \operatorname{Mat}_n(K)$ then let $f \colon K^n \to K^n$ with $f(x) = Ax$ be the associated endomorphism of $K^n$, and let $R_p := \operatorname{im} f^p$ for all $p \geq 0$.
Then
$$
    \operatorname{rank} A^p
  = \dim \operatorname{im} f^p
  = \dim R_p
  \quad
  \text{for all $p \geq 0$},
$$
as well as
$$
    f(R_p)
  = f(\operatorname{im} f^p)
  = \operatorname{im} f^{p+1}
  = R_{p+1}
  \quad
  \text{for all $p \geq 0$}.
$$
Because we have a decreasing chain
$$
            K^n
  =         R_0
  \supseteq R_1
  \supseteq R_2
  \supseteq R_3
  \supseteq \dotsb
$$
it follows from $\operatorname{rank} A^{k+1} = \operatorname{rank} A^k$ that $\dim R_{k+1} = \dim R_k$ and thus $R_{k+1} = R_k$.
It then further follows that
$$
    R_{k+2}
  = f(R_{k+1})
  = f(R_k)
  = R_{k+1},
$$
and we find inductively that $R_{k+j} = R_{k+j+1}$ for all $j \geq 0$.
So we have
$$
    R_k
  = R_{k+1}
  = R_{k+2}
  = R_{k+3}
  = \dotsb
$$
and thus $R_{k+i} = R_k$ for all $i \geq 0$.
Answer: so you know now that $rank (A^k)=rank (A^{k+1})=rank (A^{k+2})$.
you just can take $k'=k+1$  and do the same work for $A^{k'}$. You obtain $$rank (A^k)=rank (A^{k+1})=rank (A^{k'})=rank (A^{k'+1})=rank (A^{k'+2})= rank (A^{(k+1)+2}) = rank (A^{k+3}) $$ and so on


TDA api checking account balance

Question: I've been trying to use the TDA api to check my TD account balance but it can't figure it out and every time I try it gives me an error along the lines of "self not identified" here's my code
from td.client import TDClient
CLIENT_ID = 'XXXXXXXX'
REDIRECT_URI = 'XXXXXXXX'
CREDENTIALS_PATH = 'XXXXXXX'
TD_ACCOUNT = 'XXXXXXXXX'

endpoint = r"https://api.tdameritrade.com/v1/accounts"
   account_balance = TDClient.get_accounts(account = TD_ACCOUNT, fields = ['balance'])
   balance = pd.DataFrame(account_balance['balance'])

Answer: You might try it this way:
from td.client import TDClient
from pprint import pprint

CLIENT_ID = 'XXXXXXXX'
REDIRECT_URI = 'XXXXXXXX'
CREDENTIALS_PATH = 'XXXXXXX'
TD_ACCOUNT = 'XXXXXXXXX'

td_client.login()
accountinfo = td_client.get_accounts(account = 'all')
pprint(accountinfo)

Comment: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).

TDA api checking account balance


Javascript replace text between brackets with array

Question: I want to replace text inside square brackets with values of an array.
Example :

[maybe] i should [leave]
[to] help you [see]
[nothing] is [better] [than] this
[and] this is [everything] we need

will become like this

[why] i should [go]
[from] help you [run]
[that] is [nothing] [from] this
[now] this is [as] we need

Please help me.. Thank you

var arr = ['why', 'go', 'from', 'run', 'that', 'nothing', 'from', 'now', 'as']
var i = 0;
$("#text").each(function() {
  $(this).text($(this).text().replace(/\[(.+?)\]/, "["+arr[i]+"]"));
 i++;
});

[maybe] i should [leave]
[to] help you [see]
[nothing] is [better] [than] this
[and] this is [everything] we need

Answer: .each() is not called for every line in a tag. It is called for every element in a class or every element from a tag name. Read more about .each() in the documentation here.
Here is the snippet:

$(document).ready(function() {
  var arr = ['why', 'go', 'from', 'run', 'that', 'nothing', 'from', 'now', 'as']
  var text = $("#text").text();
  var count = -1;
  $("#text").text(text.replace(/\[(.*?)\]/g, function() {
    count = count + 1;
    return "["+arr[count]+"]"
  }));
});

[maybe] i should [leave]
[to] help you [see]
[nothing] is [better] [than] this
[and] this is [everything] we need

Answer: Instead of an array, you could use an object with the words to replace. If a word is found, then a corresponding value is used for replacement.

var words = { maybe: 'why', leave: 'go', to: 'from', see: 'run', nothing: 'that', better: 'nothing', than: 'from', and: 'now', everything: 'as' },
    text = document.getElementById('text');

text.innerHTML = text.innerHTML.replace(/\[([^\]\[]+)\]/g, function (_, k) {
    return '[' + words[k] + ']';
});

[maybe] i should [leave]
[to] help you [see]
[nothing] is [better] [than] this
[and] this is [everything] we need

This Version relies on order of the items and replace in the same order.

function replacer (array) {
    var i = 0;
    return function () {
        return '[' + array[i++] + ']';
    };
}

var words = ['why', 'go', 'from', 'run', 'that', 'nothing', 'from', 'now', 'as'],
    text = document.getElementById('text');

text.innerHTML = text.innerHTML.replace(/\[([^\]\[]+)\]/g, replacer(words));

[maybe] i should [leave]
[to] help you [see]
[nothing] is [better] [than] this
[and] this is [everything] we need

Answer: function change_words( replace_arr, text ){
   let i = 0;
   return text.replace( /\[(\w+)\]/g, word => `[${replace_arr[i++] || word}]` );
}
const text = "[maybe] i should [leave] [to] help you [see] [nothing] is [better] [than] this [and] this is [everything] we need"
const arr = ['why', 'go', 'from', 'run', 'that', 'nothing', 'from', 'now', 'as'];
console.log( change_words( arr, text ) )

Answer: Use String.prototype.replace() function with replacement callback:

var text = document.getElementById("text"),
    arr = ['why', 'go', 'from', 'run', 'that', 'nothing', 'from', 'now', 'as'],
    count = 0;

text.innerHTML = text.innerHTML.replace(/\[([^\]\[]+)\]/g, function (m, m1) {
    return '[' + arr[count++] + ']';
});

[maybe] i should [leave]
[to] help you [see]
[nothing] is [better] [than] this
[and] this is [everything] we need

Answer: I tried this:
$(document).ready(function(){
        arr.forEach(function(element){
        $("#text").text($("#text").text().replace(/\[.+?\]/, element ));
        });
});

It does bring the expected result without the brackets though, but when adding brackets: .replace(/\[.+?\]/, "[" + element + "]")); It doesn't work anymore. (Maybe someone could suggest something to edit this)


multicolumn index column order

Question: I've be told and read it everywhere (but no one dared to explain why) that when composing an index on multiple columns I should put the most selective column first, for performance reasons.
Why is that?
Is it a myth?
Comment: wow, so many answers to questions I dndn't make
Answer: You can omit columns from right to left when using an index, i.e. when you have an index on col_a, col_b you can use it in WHERE col_a = x but you can not use it in WHERE col_b = x.
Imagine to have a telephone book that is sorted by the first names and then by the last names.
At least in Europe and US first names have a much lower selectivity than last names, so looking up the first name wouldn't narrow the result set much, so there would still be many pages to check for the correct last name.
Comment: +1. You can still use the index if leading columns are missing, but it would be a full index scan (or an index skip scan), which is not all that efficient (could be still better than a full table scan, though).
Comment: This does not answer the part about selectivity, though.
Comment: I think at least in Europe and US first names have a much lower selectivity than last names, so an index by first name first wouldn't be of much help.
Comment: AndrewKR, true, but that is dependent on the Index being specified with the most select column left-most.  If you add that to your answer I will give you +1.
Comment: @PerformanceDBA I don't quite understand what's your point. Could you elaborate?
Comment: AndrewK. Look at Julio's answer.
Answer:
I should put the most selective column first

According to Tom, column selectivity has no performance impact for queries that use all the columns in the index (it does affect Oracle's ability to compress the index).

it is not the first thing, it is not the most important thing.  sure, it is something to consider  but it is relatively far down there in the grand scheme of things.
In certain strange, very peculiar and abnormal cases (like the above with really utterly skewed data), the selectivity could easily matter HOWEVER, they are
a) pretty rare
  b) truly dependent on the values used at runtime, as all skewed queries are
so in general, look at the questions you have, try to minimize the indexes you need based on that.
The number of distinct values in a column in a concatenated index is not relevant when considering
  the position in the index.

However, these considerations should come second when deciding on index column order. More importantly is to ensure that the index can be useful to many queries, so the column order has to reflect the use of those columns (or the lack thereof) in the where clauses of your queries (for the reason illustrated by AndreKR).

HOW YOU USE the index -- that is what is relevant when deciding.

All other things being equal, I would still put the most selective column first. It just feels right...
Update: Another quote from Tom (thanks to milan for finding it).

In Oracle 5 (yes, version 5!), there was an argument for placing the most selective columns first
  in an index.
Since then, it is not true that putting the most discriminating entries first in the index
  will make the index smaller or more efficient. It seems like it will, but it will not.
With index
  key compression, there is a compelling argument to go the other way since it can make the index
  smaller. However, it should be driven by how you use the index, as previously stated.
Comment: You have the index compression info as a little bit of a side note, but it shouldn't be ignored.  There are a lot of scenarios that compressing an index is a fabulous idea.
Comment: @Craig: I can see how column ordering would have an impact on index compression, but would that not work the other way around (low-cardinality leading columns resulting in repetitive, compressible prefixes)?
Comment: Tom said that for **Oracle 5** http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:1296165726968#59899084713981
Answer: The ordering of the columns in the index should be determined by your queries and not be any selectivity considerations.  If you have an index on (a,b,c), and most of your single column queries are against column c, followed by a, then put them in the order of c,a,b in the index definition for the best efficiency.  Oracle prefers to use the leading edge of the index for the query, but can use other columns in the index in a less efficient access path known as skip-scan.
Answer: The more selective is your index, the fastest is the research.
Simply imagine a phonebook: you can find someone mostly fast by lastname. But if you have a lot of people with the same lastname, you will last more time on looking for the person by looking at the firstname everytime.
So you have to give the most selective columns firstly to avoid as much as possible this problem.
Additionally, you should then make sure that your queries are using correctly these "selectivity criterias".
Comment: +1. That's exactly right.  Assuming that has been done, (AndrewKR) columns can be dropped from right to left.

multicolumn index column order


Shall I use weekly or monthly data for forecast?

Question: I seem not to find this in any textbooks. So I post these questions.

Is monthly data better than weekly data for forecasting?
Can there be seasonality in weekly data? Most software/methods don't  seem to find seasonality in forecasting data.
Is there a way to aggregate weekly data to monthly data?
How do methods like ARIMA/Exponential Smoothing handle seasonality with weekly data?
Answer:
Better for what? If you want weekly forecasts, use weekly data. If you want monthly forecasts, use monthly data.
Of course. Weekly data is often seasonal. If the software isn't modelling seasonality that is obviously there, then you are either using the wrong model or using the software incorrectly.
No. There isn't an integer number of weeks per month, so there is no way to reliably split the weeks that overlap month boundaries. You could apportion the week according to the number of days which fall within each month, but that doesn't take account of day-of-week effects, and with weekly data there is no way of estimating day-of-week effects.
Badly. See my blog post on this (http://robjhyndman.com/hyndsight/longseasonality/). There are several problems:

there are not an integer number of weeks in a year
exponential smoothing essentially has one parameter for each week
seasonal ARIMA models either difference away a whole year or more of data, or they regress on observations at least a year old.

A better approach is to handle the seasonality using Fourier terms (as explained in my blog post). That can be done within an exponential smoothing framework (the TBATS model) or within an ARIMA framework (a regression with ARMA errors). I've compared these models in another blog post: http://robjhyndman.com/hyndsight/forecasting-weekly-data/
Answer: Weekly forecasting is quite inadequate due to the deterministic effect of holidays and other events. Weekly data can be severely skewed by when the holiday occurs and activity before and after the holiday. Daily data analysis can provide not only good daily forecasts but good weekly forecasts. In my opinion most software errs by exclusively using autoprojective schemes when combining deterministic and autoprojective schemes should be used. Care must be taken not to assume the form of the deterministic structure and to allow the data to speak to the identification. The problem with using Fourier procedures is that it enforces a presumed structure which leads to forecasts looking like the fit but not necessarily like the data. This is easily observable by performing diagnostic checks of model residuals from these kinds of assumed procedures. If the error aren't random, the model/parameters should be questioned. If the software you are using doesn't verify the assumption of random errors to ensure that this requirement is in place by both ACF (stochastic effects) and Intervention Detection (deterministic effects) then you might be wary. We have used Fourier approaches and have been stunned by the non-independence of the error terms (residuals)
On the other hand we have found that incorporating day-of-the-week effects , changes in day-of-the-week effects, month-of-the-year effects, long-weekend effects, lead and lag structures around holidays, specific- week-of-the-month effects, specific day-of-the-month effects, changes in either levels or trends over time CAN be very useful in predicting daily, weekly and monthly aggregates.
Seldom are there any repetitious effects for a particular week in the year although we have seen some!  If you are insistent on building a weekly model identify which weeks of the year, if any, have a repetitive pattern. When encountering a year with 53 weeks some clients have eliminated one week where the response is low.
Comment: Much excellent advice as always, but this seems to presuppose that data are economic or business data. In practice some physical, chemical, biological and environmental data are collected weekly, if only as a matter of convenience or compromise (daily measurement would be too much work, monthly too coarse). The underlying processes are quite indifferent to whether _Homo sapiens_ somewhere is celebrating a holiday or it is Tuesday. Also, many economic data series are issued once-per-week and do deserve and require analysis. (The question does not stipulate what kind of data.)
Comment: If you are ONLY interested in monthly forecasta , I would probably select monthly data. If you are either interested in daily forecasts OR weekly forecasts OR creating updated monthly forecasts as you come through the month then I would strongly recommend daily data analysis.
Comment: @NickCox You are quite correct that the data that you are trying to model can often dictate the "correct granularity". I was trying to point out that pure ARIMA models are often inappropriate while deterministic structure can provide structure. If there are just a few weeks that are important then we should just indicators for those weeks while dealing with Pulses, Level Shifts, Time Trends and any needed short term ARMA. Fourier fitting ( sines/cosines and such ) might be useful for data in the physical sciences but I haven't (often) seen the proof of that.
Comment: I've often used Fourier ideas for (environmental) time series, but in contexts where neither close modelling of time series nor prediction of future events from the time series alone was a goal. Fourier models can do a good job of catching a climatically-driven seasonal signal (which can vary over time) while leaving a lot of noise. In practice this just needs a decent regression program and is not time series analysis in your style.
Comment: @irishstat Thank you for the feedback. I have daily, weekly and monthly data and have not seen any recommendation in any textbooks on which data to use. I have read somewhere that it is better to forecast using monthly data because it has structure to it. Are you suggesting that daily data is better for forecasting ?


If in some theory it is impossible to prove $X$, is it impossible to disprove $X$ too?

Question: In mathematical logic, if we can prove that it is impossible to prove a statement $X$ with some theory $T$, do we necessarily have a proof that it is impossible to disprove the statement $X$ in theory $T$?
Now, this is obviously not right, but my question is rather about how should I understand statements/results that say it is impossible to prove something:
I was reading the Wikipedia page on 'Measurable cardinals' and it is stated there that:

the existence of a measurable cardinal is not provable in ZFC.

Does it then also mean that I can't find a proof that a measurable cardinal cannot exist? Because, if I could find a proof that it doesn't exist then obviously I can't find a proof that it exists (or else ZFC is inconsistent).
That is consistent with the statement from Wikipedia:

"There isn't a proof that it exists".

So the statement from Wikipedia doesn't say that there isn't a proof that there isn't a measurable cardinal. But some common sense and natural language processing tells me that the sentence 'existence of measurable cardinal is...' accounts both statements 'it exists' and 'it doesn't exist'. I am deeply confused.
Do theorems that say it is impossible to prove something naturally mean the same as the statement is undecidable? Because they shouldn't, but they sound like they do:
If it said "It is impossible to prove it exists", it would be clear;
If it said "It is impossible to answer whether it exists or not", it would be clear again, but
when it says: "The existence is impossible to be proven" isn't so clear to me. It could mean both of previous statements and they are not equivalent.
Comment: this question might be answered better or CS SE or CSTheory SE, though I'm not sure. The current answers don't satisfy me :/
Answer: Phrase "existence of a measurable cardinal is not provable" means "there is no proof that (a measurable cardinal exists)". It claims nothing about existence of proof that (a measurable cardinal doesn't exist). For example, it is also correct to say "there is no proof that (set of all sets exists)".
But as in general we are working in a consistence theory, it's trivial that if we can disprove $X$, then we can't prove $X$. So if $X$ is disproved, we don't usually say "we can't prove $X$" - it's true, but obvious.
Sometimes we know that $X$ can't be proved, but don't know yet if it can be disproved, but I think results like this are even more rare than "$X$ can be neither proved nor disproved".
Answer: Many example of unprovable statements are of the form "The computation of this Turing machine will never stop". One way to see that this includes interesting examples is given by the Turing machine that searches for a proof of $0 = 1$ in $\mathrm{ZFC}$.
The negation of "The computation of this Turing machine will never stop" is "There is a time $t \in \mathbb{N}$ at which the computation has halted". If this is true, we can always prove it by showing the computation up to time $t$. Thus, statements of the form "The computation of this Turing machine will never stop" can be

true and we can prove it
true but unprovable
false and we can prove it

but not actually false, but we cannot prove that they are false.
Comment: Thank you for introducing this perspective. Just one question: isn't it right that the first option is impossible, "tru and we can prove it"?
Comment: @donaastor No, all three cases happen (for different TMs). For example, "while true { do nothing}" can be expressed as a TM, and it is certainly provable that it never stops.
Comment: Oh ok, I see what you meant. I misunderstood you...


How to configure NFSv4 mount so that owner of files created by root user on NFS client appear as 'root:root', rather than 'nobody:nogroup' on client?

Question: I have an Ubuntu 16.04 server on which the Nextcloud snap is installed (nextcloud.lan), and an Ubuntu 16.04 NAS configured to serve files over NFSv4 (nas.lan). I would like to mount directory /var/snap/nextcloud on nextcloud.lan via an NFS directory exported from NAS, so that all of files used by Nextcloud are stored on the NAS.
NFS authentication on the NAS is configured as default AUTH_SYS/AUTH_UNIX. Please see the following configuration files for nas.lan:
/etc/idmap.conf:
[General]

Verbosity = 0
Pipefs-Directory = /run/rpc_pipefs
# set your own domain here, if id differs from FQDN minus hostname
Domain = localdomain

/etc/exports:

/vol0/export           192.168.2.0/24(rw,fsid=0,insecure,no_subtree_check,async)
/vol0/export/nextcloud 192.168.2.0/24(rw,nohide,insecure,no_subtree_check,async,no_root_squash)

And for nextcloud.lan:
/etc/fstab:
nas:/nextcloud /mnt nfs auto 0 0
/etc/idmap.conf:
[General]

Verbosity = 0
Pipefs-Directory = /run/rpc_pipefs
# set your own domain here, if id differs from FQDN minus hostname
Domain = localdomain

Currently, when a user with a uid that exists on both nas.lan and nextcloud.lan creates a file (e.g username jacob, uid 1000) in the mounted dir on nextcloud.lan, the file is created with the appropriate owner on both systems (e.g. jacob:jacob).
However, when the root user creates files in the exported directory on nextcloud.lan, the files appear to be owned by "nobody:nogroup" in both systems. The Nextcloud snap is only able to run as the root user, and so my question is, how can I make it so that files created by root user on NFS client nextcloud.lan appear as 'root:root', rather than 'nobody:nogroup'?
I have read that NFS does some special handling around root user permissions, and does not map root user id between systems for security reasons. I am wondering if there is a way to override this?
I saw that there is one option called no_root_squash, but this has not worked for me.
I also tried setting the following in /etc/idmapd.conf on nextcloud.local, but this has also not worked for me:
[Mapping]
Nobody-User=root
Nobody-Group=root

So far, I have tried everything I can think of to map nobody:nogroup to root:root on the nextcloud.lan system, without success.
I would appreciate any insight anyone can share on how to do this. Thank you for your help.
Comment: [`no_root_squash` is **dangerous**](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security_guide/sect-security_guide-securing_nfs-do_not_use_the_no_root_squash_option).  If you don't know **exactly** what you're doing, you'll open up your network to allowing users to leverage root on one machine and get it on another. And the fact that you had to ask this question means you don't really understand it.  You'd do much better modifying whatever processing you're doing to **not** run as `root`.  **Nothing** should run as `root` outside OS-necessary processes.
Comment: @AndrewHenle Thank you for your insight on this. I agree, nothing should run as root. Unfortunately, a known shortcoming of the nextcloud snap is that it can only run as root: https://github.com/nextcloud/nextcloud-snap/issues/457
Comment: Is there a way to map the root user on nextcloud.lan to a non-root user on nas.lan?
Comment: @jbeard4 you can enforce uid/gid for user nobody (which is root if no_root_squash option is not specified) by explicitly specifying `anonuid=` and  `anongid=`
Answer: You have two entries in the export file that overlap:
/vol0/export           192.168.2.0/24(rw,fsid=0,insecure,no_subtree_check,async)
/vol0/export/nextcloud 192.168.2.0/24(rw,nohide,insecure,no_subtree_check,async,no_root_squash)

any client within subnet 192.168.2.0/24 may (and probably does) uses the first entry and ends up mapping root to user nobody. Try to narrow down the IP range:
/vol0/export  192.168.2.0/24(rw,fsid=0,insecure,no_subtree_check,async) 192.168.2.1(rw,nohide,insecure,no_subtree_check,async,no_root_squash)

where 192.168.2.1 is supposed to the IP address of nextcloud.lan.
Comment: Clients don't use export entries like that. Having exports as subordinate directories to another export does not cause any selection from the client to be made for one export or another. They can be mounted independently of one another, and the export directory is part of the client login request (rather than some kind of server or client side guesswork). The field you're talking about is for restricting access to an export based on IP, not a method of assigning exports to hosts (though in your example, it behaves somewhat similarly to the concept of assignment).
Comment: This is true for NFSv3, where an extra request sent to mound service to discover file handle based on path. However, with NFSv4 there is no concept of export path. A pseudo-fs is created which represents the virtual export tree.
Answer: I was able to achieve this with the following modification to /etc/exports:
/vol0/export           192.168.2.0/24(rw,fsid=0,insecure,no_subtree_check,async,no_root_squash)
/vol0/export/nextcloud 192.168.2.0/24(rw,nohide,insecure,no_subtree_check,async,no_root_squash)

I was then able to mount /var/snap/nextcloud via NFS to the exported directory on the nas, and everything seems to work well.
However, I agree with @AndrewHenle -- this approach seems dangerous and insecure, and it would be better to find an approach that mapped the root user on nextcloud.lan system to a non-root user on the nas.lan system. Unfortunately, the nextcloud snap needs to run as root at this time. I may spawn this off into a separate question.


How to rotate in quaternions but for 2d version for arbitrary angle?

Question: I am trying to understand the idea behind rotating in quaternions, but first I want to understand the math for 2d rotation. I saw some youtube videos, and I know that for 2D, a point in 2D can be represented like
$a + bi$ which maps to the $(x,y)$ coordinate $(a,b)$. Then lets say I want to rotate it 90 degrees. I would multiply $a + bi$ by $i$. So for example, if I had the point (3,3) so that's $3+3i$ then rotate it 90 degrees, I do
$(3+3i)i = 3i+3i^2 = 3i-3$ (since $i^2=-1$)
so that's
$-3+3i$ giving the point $(-3,3)$ which is a $90$ degree rotation of the original point $(3,3)$.
But what was not explained and what I don't get is, how to rotate by an arbitrary angle from 0 to 360? Like lets say I want to rotate by 20 or 45 degrees, what would I multiply the $a + bi$ with?
Can anyone explain?
Thanks
Answer: Consider the complex number $r=\cos \theta + i \sin \theta$, such that $|r|=1$.
For a complex number $z=x+iy$ we have:
$$
rz=(\cos \theta + i \sin \theta)(x+iy)=x\cos \theta -y\sin \theta+i(x \sin \theta+ y \cos \theta)
$$
If we interpret $z$ as the vector ${\bf z}=(x,y)^T$ we can write this result as:
$$
{\bf z'}=
r\begin{bmatrix}
\cos \theta & -\sin \theta\\
\sin \theta& \cos \theta
\end{bmatrix}
\begin{bmatrix}
x\\
y
\end{bmatrix}
$$
and we can see that the matrix represents a rotation of angle $\theta$, because the first column is the rotation of ${\bf i}=(1,0)$ and the second is the rotation of ${\bf j}=(0,1)$, that are the standard basis of the vectors in a plane.
Comment: I'm not sure I understand. Can you show an example please. So like if I had $(3,3)$ i.e. $3+3i$ and want to rotate it 20 degrees. Can you show what I would need to multiply it with?
Comment: So does it mean that first I need to convert $20$ to radians, so $q=radians(20)$. Then I do $(3+3i)(cos(q)+isin(q))$? And the result is the point where $(3,3)$ is rotated $20$ degrees?
Comment: Yes, it is correct. If you want you can also use $\cos 20°$ and $\sin 20°$, it is the same.


Instant Run is not supported on devices with API levels 20 or lower using Blue Stack emulator

Question: I have downloaded Android studio and Bluestack Emulator. However, the default device provided by Bluestack is of API 19(Samsung)
When I try to run the code there is an error message:
"Instant Run is not supported on devices with API levels 20 or lower."

How can I get a higher API emulator device or make the instant run go... ? I am just learning Android development so I am not concerned with what API I build in... I  just want the program to run so that I can move on with coding something new.
Comment: Use the official [Android Emulator](https://developer.android.com/studio/run/emulator.html) or try [Genymotion](https://www.genymotion.com/). Read FAQ and troubleshooting first if you run into any problems. Or disable Instant run since it has nothing to do with your code.
Comment: If bluestack don't provide a newer Android version then there's not a lot you can do. The best option though is to use the Official Android Emulator as part of the Android SDK
Answer: If you are using Android Studio on Mac. You can go to Preferences/Instant Run/ and disable all selections.


SQL query help - alternative to join

Question: looking to do a join where I join one table to another on for example brand and join only to the first record it comes to based on a rank I have placed on the other table. a simple example of the two tables and the output required are displayed below.
Thanks
Table 1                 Table 2
brand   serial no       brand   serial no   rank
toyota  x1              toyota  c4          1
toyota  x2              toyota  c5          2
toyota  x3              toyota  c6          3

Desired output
toyota  x1  c4
toyota  x2  c5
toyota  x3  c6

Comment: Always the same number of rows in the tables?
Comment: no there will be different number of rows in my actual dataset. I don't need it to get to the end of the list on the second table either, just take one value each but not repeating
Comment: Add one or two rows to one of the tables, and adjust the result.
Answer: You could use ROW_NUMBER in a subquery, something like this.
Sample Data;
CREATE TABLE #Table1 (Brand nvarchar(10), Serial_No nvarchar(2))
INSERT INTO #Table1 (Brand, Serial_No)
VALUES
('toyota','x1')
,('toyota','x2')
,('toyota','x3')

CREATE TABLE #Table2 (Brand nvarchar(10), Serial_No nvarchar(2), Rank_No int)
INSERT INTO #Table2 (Brand, Serial_No, Rank_No)
VALUES
('toyota','c4',1)
,('toyota','c5',2)
,('toyota','c6',3)

Query
SELECT
    t1.Brand
    ,t1.Serial_No
    ,t2.Serial_No
FROM
(SELECT
    ROW_NUMBER() OVER (ORDER BY Serial_No) Row_Num
    ,Brand
    ,Serial_No
FROM #Table1
) t1
LEFT JOIN #Table2 t2
    ON t1.Row_Num = t2.Rank_No

Output
Brand   Serial_No   Serial_No
toyota  x1          c4
toyota  x2          c5
toyota  x3          c6

Comment: *and thus by brute force you achieve the stated output*; but what is the business logic that dictates that a row number ordered by serial_no is the logical relationship.
Comment: I guess my point really is why give false hope that there is some magic formula to this mysterious relationship - that we can guess.
Comment: @Used_By_Already then I guess flag as "unclear what you are asking" (as I also have) and move on with the day?
Comment: A version of row number has helped allowed me to get the result I need. thanks for your help

SQL query help - alternative to join


How to find out the order of integration of a combination of iid random series?

Question: $\varepsilon_{1j}, \varepsilon_{2j}, \varepsilon_{3j}  ∼ iid(0; 1)$ and they are three independent processes.
$ y_{t} =x\sum_{i=1}^{t}\sum_{j=1}^{i} \varepsilon_{1j} + y\sum_{i=1}^{t} \varepsilon_{2i} +  \varepsilon_{3t}$
$x$, $y$ are non-zero constants.
How to find the order of integration $ y_{t} $?
I think it is $I(2)$. But I am not sure how to prove it. Would "constant mean and variance" for all $t$ enough to prove that the difference are stationary? If not, how to prove that the autocovariance is constant for all $t$?
Are there better ways to prove that e.g. the second difference of $ y_{t} $ is stationary?
Note: I also asked the same question here, as I am not sure which platform would be more appropriate for this question.
Comment: The statements "$y_t$ is $I(2)$" and "$y_t$ is stationary" are incompatible. If you think the process is $I(2)$, then write down the second difference $\Delta^2y_t$, which isn't very complicated. It should be relatively easy to then compute the autocovariance function of that.
Comment: @ChrisHaug I just edited the question. So I take it that autocovariance is essential? I am not sure how to compute it though. In this question: https://stats.stackexchange.com/questions/258720/show-that-a-time-series-process-is-stationary, the answer simply stated "the covariance depends on time lag only", can we always do this? Also are there other ways?
Comment: Yes, the definition of (weak) stationarity involves the requirement that the autocovariance function not depend on $t$ but only on the lag. That answer you link does not include the proof that this is true in that case, it is merely stated as if it were obvious, which is less convincing. Here it should be easy to do: first write down $z_t = \Delta^2 y_t$, then compute in turn $\text{Cov}(z_t, z_{t-1})$, then $\text{Cov}(z_t, z_{t-2})$ .. and so on. Very quickly you will see that the $\varepsilon_{i,t}$ are "slipping" away from each other and that the covariance will fall to zero quickly.
Comment: @ChrisHaug Thank you for your answer. What do you mean by ""slipping" away from each other" though? Shouldn't the covariance be the same for all lag numbers?
Answer: So, we start with the following:
$$y_{t} =x\sum_{i=1}^{t}\sum_{j=1}^{i} \varepsilon_{1,j} + y\sum_{i=1}^{t} \varepsilon_{2,i} +  \varepsilon_{3,t}$$
By definition, for $y_t$ to be $I(k)$, it must be the case that $\Delta^k y_t$ be stationary. We first compute the first difference:
\begin{align}
\Delta y_t &= x \left[ \sum_{i=1}^{t}\sum_{j=1}^{i} \varepsilon_{1,j} - \sum_{i=1}^{t-1}\sum_{j=1}^{i} \varepsilon_{1,j}\right] + y\left[\sum_{i=1}^{t} \varepsilon_{2,i} - \sum_{i=1}^{t-1} \varepsilon_{2,i}\right] + \left[\varepsilon_{3,t} - \varepsilon_{3,t-1}\right]\\
&= x \sum_{j=1}^{t} \varepsilon_{1,j} + y \varepsilon_{2,t} + \varepsilon_{3,t} - \varepsilon_{3,t-1}
\end{align}
And from there, the second:
\begin{align}
\Delta^2 y_t &= x \left[ \sum_{j=1}^{t} \varepsilon_{1,j}- \sum_{j=1}^{t-1}\varepsilon_{1,j}\right] + y\left[\varepsilon_{2,t} - \varepsilon_{2,t-1}\right] + \left[\varepsilon_{3,t} - 2\varepsilon_{3,t-1} + \varepsilon_{3,t-2}\right]\\
&= x \varepsilon_{1,t} + y\left[\varepsilon_{2,t} - \varepsilon_{2,t-1}\right] + \left[\varepsilon_{3,t} - 2\varepsilon_{3,t-1} + \varepsilon_{3,t-2}\right]
\end{align}
To show that $z_t :=\Delta^2 y_t$ is stationary, first notice that its mean is just zero (since all the $\varepsilon_{i,t}$ are mean zero), so it doesn't depend on $t$. Then compute the autocovariance function, for increasing lags. Starting at lag zero:
\begin{align}
\text{Cov}(z_t, z_{t-0}) = \text{Cov}(&x \varepsilon_{1,t} + y\left[\varepsilon_{2,t} - \varepsilon_{2,t-1}\right] + \left[\varepsilon_{3,t} - 2\varepsilon_{3,t-1} + \varepsilon_{3,t-2}\right],\\
&x \varepsilon_{1,t} + y\left[\varepsilon_{2,t} - \varepsilon_{2,t-1}\right] + \left[\varepsilon_{3,t} - 2\varepsilon_{3,t-1} + \varepsilon_{3,t-2}\right])
\end{align}
You can compute this by exploiting bilinearity and the fact that each $\varepsilon_{i,t}$ is independent from the same process at different times and from the other processes, thus eliminating most of the "cross-terms" (they are zero). You'll find that the result is a function of $x$ and $y$ but not of $t$:
$$\text{Cov}(z_t,z_t) = x^2 + 2y^2 + 6$$
Then, we do the same for lag 1:
\begin{align}
\text{Cov}(z_t, z_{t-1}) = \text{Cov}(&x \varepsilon_{1,t} + y\left[\varepsilon_{2,t} - \varepsilon_{2,t-1}\right] + \left[\varepsilon_{3,t} - 2\varepsilon_{3,t-1} + \varepsilon_{3,t-2}\right],\\
&x \varepsilon_{1,t-1} + y\left[\varepsilon_{2,t-1} - \varepsilon_{2,t-2}\right] + \left[\varepsilon_{3,t-1} - 2\varepsilon_{3,t-2} + \varepsilon_{3,t-3}\right])
\end{align}
Now compare the indices in the first line with the second line: there are far fewer "matching" terms (none from the $\varepsilon_{1,t}$), so when you expand the covariance you get even fewer terms. We get the following:
$$\text{Cov}(z_t,z_{t-1}) = -y^2 - 4$$
Then for lag 2:
\begin{align}
\text{Cov}(z_t, z_{t-2}) = \text{Cov}(&x \varepsilon_{1,t} + y\left[\varepsilon_{2,t} - \varepsilon_{2,t-1}\right] + \left[\varepsilon_{3,t} - 2\varepsilon_{3,t-1} + \varepsilon_{3,t-2}\right],\\
&x \varepsilon_{1,t-2} + y\left[\varepsilon_{2,t-2} - \varepsilon_{2,t-3}\right] + \left[\varepsilon_{3,t-2} - 2\varepsilon_{3,t-3} + \varepsilon_{3,t-4}\right])
\end{align}
Here, there is only one matching term left, $\varepsilon_{3,t-2}$, and the covariance is just 1. It's easy to see that, for all higher lags, none of the terms on the top and bottom will match, so the covariance at higher lags is zero: this is what I referred to above as "slipping away" from each other, and is typical of "MA"-type behavior.
By showing that the mean of $z_t$ is constant and the autocovariance function only depends on the lag, we have shown that $z_t$ is stationary. We can conclude therefore that $y_t$ is of order at most 2. You actually also need to show that $y_t$ itself and $\Delta y_t$ are not stationary to prove that the order of integration is 2. I'm not going to do this here but you simply need to compute the variance of each process to see this (you will see that the variance increases with $t$). This is actually where the assumption that $x$ and $y$ aren't zero comes in (because e.g. if $x=0$, then $y_t$ is actually $I(1)$, not $I(2)$).
Comment: They're all zero *except* for the covariance of one term with the same term: $\text{Cov}(\varepsilon_{i,t}, \varepsilon_{i,t}) = \text{Var}(\varepsilon_{i,t}) = 1$. That's what I mean by "matching" them.
Comment: Thank you for your answer. What exactly do you mean by "matching terms"? I think that since every process are iid and independent of each other, the covariances of all term pairs here should all be 0. Where did I get it wrong?


Routing/Networking problem with OpenVPN + Linux

Question: I have a problem where I have an office LAN and a colo network. The colo has a linux box that's acting as a router/VPN. I also have an office machine that's acting as a VPN client.
The way it looks is the following:
Office has two interfaces,
eth0 on 10.16.1.15 netmask 255.255.255.0
tun0 is a P-t-P interface that looks something like: inet addr:10.8.13.6  P-t-P:10.8.13.5
Now the linux box at the colo has no problem reaching the machine via its 10.8.13.6 address. What I'm trying to do is get it to reach the 10.16.1.0/24 network which I can't do because I don't have a subnet on the VPN that's part of 10.16.1.0. So I tried doing something such as the following:
<code># route add -host 10.16.1.5 gw 10.8.13.6
SIOCADDRT: Network is unreachable
</code>
I'd just like to be able to get a route directly to 10.16.1.5 from box at the colo(the VPN). Is what I'm trying to do rationale/possible?
Comment: It sounds like you want 1 box at the colo to be able to reach the entire office subnet - is that right?  Or do you want a full bridge (any office system able to reach any colo system and vice-versa)?
Comment: baumgart: either one would work. If the one box at the colo can reach the office subnet, or at least that one particular box -- then I can have all other clients go through the colo box to reach the office.
Comment: Does the box at 10.8.13.6 have ipv4.forwarding enabled?
Comment: 3dinfluence: It was off before but I enabled it, net.ipv4.conf.tun0.forwarding = 1
Comment: I always check that first if someone says they can connect to the VPN and can ping the other end of the tunnel but can't get to the rest of the network behind that tunnel.  I take it though that enabling that didn't solve your issue?  I'm guessing that the 10.8.13.6 needs to have a route back to your host/network in it's routing table as well.
Comment: Sorry I meant that 10.16.1.0/24 needs a route back.
Comment: Nope, I ensured that ip forwarding is enabled on both boxes. Unfortunately no go still.
Answer: Try this instead:
route add -host 10.16.1.5 gw 10.8.13.6 dev tun0
But in any case this should be handled by your vpn config. You shouldn't need to do this manually. What vpn are you using?
Answer: If you post your openvpn configurations, output of ifconfig and ip ro ( on both boxes ) i will be able to sort it out pretty quick.
The problem might be you're not pushing the right routes through the vpn, or that you misconfigured something else. There's a wide range of potential reasons, and asking them one by one and waiting for an answer would take too much time.


Projeto empty MVc. Não consigo startar

Question: Comecei a fazer um projeto do zero usando o MVC 4 do VS 2012. Como eu optei por um empty, não consigo fazer startar a view. Fiz a rota:
routes.MapRoute(
                name: "Operador",
                url: "{controller}/{action}/{id}",
                defaults: new { controller = "Operador", action = "Index", id = UrlParameter.Optional }
           );

Mas quando eu dou um F5, me dá esse erro:
Erro de Servidor no Aplicativo '/'.

Não é possível encontrar o recurso.
  Descrição: HTTP 404. O recurso que você está procurando (ou uma de suas dependências) não pôde ser removido, seu nome foi alterado ou está temporariamente indisponível. Examine o URL e certifique-se de que está digitado corretamente.

 URL solicitada: /

Informações sobre a Versão: Microsoft .NET Framework Versão:4.0.30319; Versão do ASP.NET:4.0.30319.34249

Na minha URL que é: http://localhost:60975/ eu adiciono após a barra(/) o nome da controller e continua me dando erro que não encontrou a view Index. Como eu resolvo isso?
public class OperadorController : Controller
    {
        //
        // GET: /Operador/

        public ActionResult Index()
        {
            return View();
        }

    }

View
@{
    ViewBag.Title = "Operador";
}

Operador

O Package.config
<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="AutoMapper" version="4.0.4" targetFramework="net45" />
  <package id="Microsoft.AspNet.Mvc" version="4.0.20710.0" targetFramework="net45" />
  <package id="Microsoft.AspNet.Razor" version="2.0.20710.0" targetFramework="net45" />
  <package id="Microsoft.AspNet.WebApi" version="4.0.20710.0" targetFramework="net45" />
  <package id="Microsoft.AspNet.WebApi.Client" version="4.0.20710.0" targetFramework="net45" />
  <package id="Microsoft.AspNet.WebApi.Core" version="4.0.20710.0" targetFramework="net45" />
  <package id="Microsoft.AspNet.WebApi.WebHost" version="4.0.20710.0" targetFramework="net45" />
  <package id="Microsoft.AspNet.WebPages" version="2.0.20710.0" targetFramework="net45" />
  <package id="Microsoft.Net.Http" version="2.0.20710.0" targetFramework="net45" />
  <package id="Microsoft.Web.Infrastructure" version="220.127.116.11" targetFramework="net45" />
  <package id="Newtonsoft.Json" version="4.5.6" targetFramework="net45" />
</packages>

Global.asax
// Note: For instructions on enabling IIS6 or IIS7 classic mode,
    // visit http://go.microsoft.com/?LinkId=9394801
    public class MvcApplication : System.Web.HttpApplication
    {
        protected void Application_Start()
        {
            AreaRegistration.RegisterAllAreas();

            WebApiConfig.Register(GlobalConfiguration.Configuration);
            FilterConfig.RegisterGlobalFilters(GlobalFilters.Filters);
            RouteConfig.RegisterRoutes(RouteTable.Routes);
        }
    }

O Route.config
public class RouteConfig
    {
        public static void RegisterRoutes(RouteCollection routes)
        {
        routes.IgnoreRoute("{resource}.axd/{*pathInfo}");

        routes.MapRoute(
            name: "Operador",
            url: "{controller}/{action}/{id}",
            defaults: new { controller = "Operador", action = "Index", id = UrlParameter.Optional }
       );
    }
}

Comment: `MeuController` existe?
Comment: @CiganoMorrisonMendez, sim ele existe e criei uma view para ele. Só não estou sabendo é como chamá-lo.
Comment: Esquisito. Pode colocar o Controller e a View na sua pergunta, por favor?
Comment: Editei a pergunta
Comment: Pode colocar também o `packages.config` na pergunta? Aparentemente está tudo certo com a estrutura.
Comment: Também está ok. Quero verificar o `Global.asax.cs`, se possível.
Comment: Está tudo certo. Não sei mais o que poderia ser.
Comment: @CiganoMorrisonMendez, eu clico com o botão direito em cima da minha Action e dou Go to View e me vem a seguinte mensagem: Unable to find matching view. Deletei a view e a refiz e continua a mesma coisa, ou seja, ele não está achando a view, agora não sei o porque.
Answer: Sei que já respondeu sua própria pergunta, mas queria expandir um pouco mais. Como já percebeu, o MVC usa muito convenções. Porém, há meios de fazer fora das convenções e faz o que você quiser.
No seu caso, a convenção é que tem que ter um view com o mesmo nome que a ação que está no controller. Por exemplo:
Controller.cs
public ActionResult AlgumaCoisa()
{
  return View();
}

Assim, por convenção, teria que criar um view com o mesmo nome:
AlgumaCoisa.cshtml
@{ ViewBag.Title = "Alguma Coisa" }

Alguma coisa está aqui

É bom saber também que views que existem em outras pastas também pode ser usada:
Controller.cs
public ActionResult AlgumaOutraCoisa()
{
  return View("AlgumaCoisa/AlgumaOutraCoisa");
}

E seu view:
/AlgumaCoisa/AlgumaOutraCoisa.cshtml
@{ ViewBag.Title = "Alguma outra coisa." }

Alguma outra coisa está aqui.

Comment: brazilianldsjaguar, bom dia. Então, como eu criei um projeto empty, achei que ele fosse empty mesmo. Quando eu criei a controller, ele já criou uma action chamada Index(esqueleto apenas) e no Route.config, a mesma coisa. Acho que foi isso que estava acontecendo e você me respondeu muito bem, até marquei a sua resposta por ter mais consistência e explicação plausível. Entendi o que estava acontecendo.
Comment: Valeu! O MVC já me pegou algumas vezes também.
Answer: Já sei o que houve. Não existe a view Index, logo ele quer retornar essa view. Fiz assim em minha controller e resolveu:
public ActionResult Index()
        {
            return View("Operador");
        }

Projeto empty MVc. Não consigo startar


Sorting list of lists by a third list of specified non-sorted order

Question: I have a list:
[['18411971', 'kinase_2', 36], ['75910712', 'unnamed...', 160], ...

about 60 entries long
each entry is a list with three values

I want to sort this bigger list by the first value in an order specified by another list which has them in the desired order.
Answer: The usual idiom is to sort using a key:
>>> a = [[1,2],[2,10,10],[3,4,'fred']]
>>> b = [2,1,3]
>>> sorted(a,key=lambda x: b.index(x[0]))
[[2, 10, 10], [1, 2], [3, 4, 'fred']]

This can have performance issues, though-- if the keys are hashable, this will probably be faster for long lists:
>>> order_dict = dict(zip(b, range(len(b))))
>>> sorted(a,key=lambda x: order_dict[x[0]])
[[2, 10, 10], [1, 2], [3, 4, 'fred']]

Comment: That would be a nice place for function composition: `key = b.index . itemgetter(0)` :) Unfortunately Python doesn't have it built-in :(
Answer: How about:
inputlist = [['18411971', 'kinase_2', 36], ['75910712', 'unnamed...', 160], ... # obviously not valid syntax
auxinput = aux = ['75910712', '18411971', ...] # ditto
keyed = { sublist[0]:sublist for sublist in inputlist }
result = [keyed[item] for item in auxinput]

There is no need to use sorting here. For large lists this would be faster, because it's O(n) rather than O(n * log n).
In case the keys aren't unique, it is possible to use some variant of an ordered dict (e.g. defaultdict(list) as per Niklas B's suggestion) to build the keyed representation.
Comment: +1, while this is more verbose than sorting with `auxinput.index` as the key, it's about the same amount of code than the dict-based sorting. If the entries weren't unique, one could use a `defaultdict` to build a `key->[values]` association and then use `[keyed[x] for item in auxinput for x in item]`, which would still be `O(n)` if the number of duplicates is bounded.
Comment: This assumes that the first entries are unique, but that could very well be the case.  +1.
Answer: So, if I understand you right, you have your sample input list:
a = [['18411971', 'kinase_2', 36], ['75910712', 'unnamed...', 160], ...

and you want to sort this using an extra list that mentions the order in which the first elements of the sub-lists are to occur in the output:
aux = ['75910712', '18411971', ...]

If that's right, I think the result can be achieved using something like:
sorted(a, key = lambda x: aux.index(x[0]))


How to filter attributes in the Google Cloud Platform (GCP) Logs Explorer query using RegEx

Question: Writing the query in the GCP Logs Explorer with a regular expression (RegEx) as the filter:
I need to filter the query_name for any string that has the word ¨stat" in it.
Tried it with the SQL way, and with wildcards:
logName="projects/my_project/logs/my_env-production"
labels.query_name RLIKE "stat"

logName="projects/my_project/logs/my_env-production"
labels.query_name LIKE "%stat%"

logName="projects/my_project/logs/my_env-production"
labels.query_name="%stat%"

And RegEx, just to test the "." for any letter:
logName="projects/my_project/logs/my_env-production"
labels.query_name LIKE ".stat."

The output is No data found on the left pane, and the output list is empty.
Answer: You need to use the same trick that is also used in Logs-Based Metrics or even in Grafana if you want to filter for RegEx.
logName="projects/my_project/logs/my_env-production"
labels.query_name=~".stat."

Or take:
labels.query_name!~".stat."

to avoid the RegEx.
(if you want it without RegEx, it is "=" and "!=", which is another pattern, that is why you might not find the RegEx on your own).


Distance between random variables?

Question: I am looking for distances between two random variables $X$ and $Y$, or practical estimates for measuring the distance between the i.i.d. observations $(X^1, \ldots, X^T)$ and $(Y^1,\ldots,Y^T)$.
I am aware of the divergences or statistical distances, but they focus on quantifying the dissimilarity in distribution which is sufficient when $X$ and $Y$ are independent, but fail to measure how "correlated" they are otherwise.
Any information is welcomed!
Comment: This needs to be tightened up a bit. What property of two random variables would result in zero distance?
Comment: A perfect dependence between the two random variables AND the same distribution would result in zero distance. Would you suggest any other property? I think that a reasonable distance should take into account dependence and distribution since it amounts for the whole distribution according to Sklar's theorem in copula theory.
Comment: So if we have such a distance $d(\mathbf{X,Y})$ then $d=0$ iff $X,Y$ come from the same distribution and are perfectly (linearly?) correlated?
Comment: Not necessarily linearly correlated, it could be "linearly" correlated up to some monotonous transforms or even a broader notion of "correlation".
Comment: I don't understand your criticism of divergence measures... Why doesn't mutual information provide you 'non linear'  correlation
Comment: Mutual information could help indeed. But then, one does not measure similarity in distribution, cf. Bey answer for a baseline motivation.
Answer: Here's a measure that seems to accord with your requirements for the case of monotonic relationships between $X$ and $Y$:
Let $X,Y$ be your sample vectors. Let $S(X,Y)$ be the Spearman Rank Correlation between these two vectors and let $KS(X,Y)$ be the Kolmorogov-Smirnov Statistic between the EDCF of $X$ and ECDF of $Y$.
We can construct the quantity $D(X,Y)=||S(X,Y)|-1|+KS(X,Y)$. Lets analyze the cases:

If $Y=f(X)$ where $f()$ is monotonic, then $KS(X,Y)=0 and|S(X,Y)|=1$, so the $D(X,Y)=0$.
If $X,Y$ both come from the same distribution but are uncorrelated, then $S(X,Y)\approx 0$ and $KS(X,Y)\approx 0$ for large enough samples, so $D(X,Y)\approx 1$
If $X,Y$ are neither correlated nor from the same distribution then $D(X,Y) \in [0,2]$.

So, this is more of a "coefficient" or index than a distance, but maybe itll work for you.
Comment: Thanks Bey. Actually, your idea is quite close to [this arxiv paper](http://arxiv.org/pdf/1506.00976v2.pdf). But I am not entirely satisfied with that. 1) You cannot control easily the weighting between dependence and distribution. 2) Why a convex combination of the two parts? One could have used [generalized-means](https://en.wikipedia.org/wiki/Generalized_mean)...  3) despite the two parts live in $[0,1]$, one does not control how they vary in these intervals and these two distances might be rather inhomogeneous!
Comment: @mic your need for so much control undermines your goal of having an objective metric of closeness. For example, there are weightings that will contradict your goal of having zero distance for perfect monotonic correlation. You really need to think about the specifics of what you need...what decision(s) will this metric be supporting? What are the risks of wrong decisions? There is quite a bit of literature on Statistical Decision Theory that can help turn a vague, subjective requirement into something precise.

Distance between random variables?


SQL Server Management Studio - Specify Values for Template Parameters auto-generation of "Create date"

Question: I am creating a lot of stored procedures in Microsoft SQL Server Management Studio 2008.
I am using the Query -> Specify Values for Template Parameters method.
I wonder if someone knows how to have the Author and Create date fields automatically contain my name and the current date. At the moment I have to type them in again and again.
Thanks in advance.
Answer: I don't think you can. You could write some dynamic SQL that prints your template. You can then tell it to print GETDATE(), and your name would be stored as a variable.
You would run the code then paste the result into your window as your template.
Comment: I am still a bit green when it comes to SQL but will try to implement your suggestion. Thanks a lot.
Answer: For your name, you can specify the template like below
<Author,String,Sam Leach>


Is gcc 4.9.x stable version?

Question: Hu, I was surprised at read-only mode...
Anyway, I know many program uses even-odd version numbering, like linux kernel 2.x.

[major version].[major revision].[minor revision]
If major revision is odd, it's unstable version.

But in wikipedia page, it says that gcc 4.9.0 is stable version.
Does NOT gcc uses that version numbering? But I've heard that gcc 4.7 has many bugs, so I can't make sure..
Comment: as you can see for yourself https://gcc.gnu.org/ , GCC 4.9.0 is a milestone and official release from the GCC team. Please don't refer to Wikipedia for serious stuff, it's a webpage that basically anyone can edit, pick your sources with the same care you expect from the makers of the software that you use.
Comment: Linus Torvalds seems to have complained heavily on 4.9.0 at least: http://lkml.iu.edu//hypermail/linux/kernel/1407.3/00650.html
Answer: Every released version of GCC is as stable as it can be.
C++ compilers are in a perpetual race to keep up with the progress of the ISO C++ standard. Features are continually being introduced. Currently C++11 is mostly complete and most work is going toward <code>-std=c++1y</code> mode. GCC 4.7 happened to be at the peak of C++11 feature introduction, which might explain its reputation for bugs.
To be sure, there is no advantage to running an older release version of the compiler, at least for code that is reliably compliant (portable, not depending on compiler bugs) in the first place. This holds true for any of the modern C++ compilers.
A more relevant question may be whether you want to run a prerelease version or not. Often, the "bleeding edge" contains more bugfixes than bugs.
Comment: I always want to run latest version of compiler so much that I threw my cross-built gcc 4.8.2 away as soon as 4.8.3 released and spent much time cross-building it.. After going back home, I'll do the same for 4.9.0 >o<
Answer: It makes complete sense to stick with the "latest and greatest" release when it comes to modern compilers. Although it is a fact (just not a readily acceptable one) - Compilers, like other software, are prone to bugs.
The best you can do before committing with a particular release, is take a look at the open issues / bugs reported in that release and see how they affect you. Like one of the recent bug posting 61904 for GCC 4.9.0 which is miscompiling the linux kernel !

Is gcc 4.9.x stable version?


Teaching a lefty to play drums

Question: I've been asked to teach someone to play drums. I noticed that he's predominantly left-handed. I play right-handed, and feel that for his future, he'll be better off learning with the kit set up normally.
What advantages/disadvantages will there be for him? Will it mean his potential progress will be slower, or would he eventually end up just as good as if he had a lefty kit. There seem to be a lot of good lefty drummers out there who play standard, so what are their secrets? Not seeking opinions, but experiences from players/teachers, please.
Answer: The "left-handed" musicians (and people) that I know are not 100% left handed (hence the quotes). For example, my nephew writes with his left hand but plays violin right-handed and uses a computer mouse in his right hand. Other musicians I've known both write and play "left handed".
In short, my experience is that there is an overall misconception about handedness in that it is binary and applies to everything. While most people do everything right handed, I know many people (including myself) who do some things with one apparent handedness and other things with the opposite handedness.
I disagree that drumming is ambidextrous. Hi hat and snare techniques are different, and usually they are played at the same time, so players of drum kits will have one hand trained for the hi hat and the other hand trained for the snare.
If I were in your situation, I would work with the student as early as possible in the process to get to the bottom of whether they feel at all more comfortable one way or the other. They might naturally feel more comfortable playing right handed, or they might not care one way or the other. If they feel more comfortable with a left handed setup, I suggest you may do them a great disservice by teaching them to play right handed instead.
It may seem like a lefty drummer will have a disadvantage in kit setups, and there is a little bit of truth to this. When I stage managed a festival where our target changeover time was five minutes (!!), we had a few lefty drummers come through and we knew we would not be able to switch the kit right to left in such a short amount of time. So that was inconvenient, but that is an unusual situation.
Every other situation I've been in, the drums are personal enough and setup time is reasonable enough that lefty drummers have basically had the same amount of work to do as righty drummers in terms of setup. Bass players don't seem to have a consensus on whether they generally prefer hi hat side or floor tom side of the kit, so that's not an advantage or disadvantage either.
Finally, unlike guitars and basses, there is no special equipment for left handed drummers - it's all the same gear, just set up differently. So there's no disadvantage there.
So pros of a lefty setup for lefty drummers is they are more comfortable and likely will learn faster and enjoy it more. Cons are virtually none. While pros for righty setup for a lefty drummer are virtually none and cons are the exact opposite of the pros for a lefty setup.
Find out this particular student's particular preference and honor it. If they have no preference, go righty.
Comment: Bit of feedback. Couple of years on, he's still playin g on a r.h. kit - both mine and both his are all set up r.h. He still leads with l.h. (not good), but manages feet just fine. I suggested the other day that we swapped over to l.h. spec., and he was very much against it! So, by and large, it's successful, and hopefully will set him up for playing on most kits he'll meet - i.e. r.h. kits. Drummers ought to be able to play open or closed, thus swap snare and hi-hat hands. Most I play with don't find that unusual.
Answer: As with almost all instruments,  a successful player uses both hands for a wide variety of actions.  For all practical purposes, a percussionist becomes ambidextrous.  There is no advantage to reversing a setup, and it'll be much easier for him to "sit in" if he's trained in a standard orientation.
I mean, could you imagine teaching a lefty to play piano entirely cross-handed? :-) .
With apologies to Paul and Jimi.
Comment: I disagree on the ambidexterity of kit drumming. Hi hat and snare techniques are not interchangable, and as a bit of a drummer who plays right handed, I'm completely useless sitting behind a kit that is set up lefty style.
Comment: Piano, no I couldn't. However, we're talking about something that can be, and sometimes is, moved around to accomodate. I'm trying, with the question, to find reasons why it's like it is - favouring, possibly, right-handed players, and *what* disadvantages there may be for someone who is *not* right-handed.
Comment: @ToddWilcox  I understand your point.  I misled you there -- what I meant was that a new learner can be taught to work with either a LH or RH drumset regardless of his handedness.  Absolutely true that, once you've learned, it will be insane to try to play a mirror setup.
Answer: Yeah.  Teach him to play the normal way.  Any preference otherwise will be more political than practical.
Comment: But - his 'normal way' would most likely be the opposite from a right handed person's. There must be reasons why drum kits are set up as they are. I'm trying to elicit reasons, and reasons why or not this is good for lefties.
Comment: There's a convention.  I'm not too sure there's a reason, particularly one applicable to today's playing techniques.
Comment: @Tim As a right handed drummer, I feel the dexterity demanded by popular hi hat and ride cymbal rhythms is easier to play with my more dexterous right hand. The typical snare rhythms that I play with my left hand require a lot less finesse.
Comment: @ToddWilcox  that just means you need to follow the rule about getting to Carnegie Hall  :-)

Teaching a lefty to play drums


awk array: count column return line

Question: I have tried several awk arrays but just can´t get my head around it.
Having a file with 4 colums the awk skript should count for each line how many times the value in column 2 appears.
Input File
A1^ABC^173^2012
A2^BDK^153^2013
A3^AFD^223^2012
B1^ABC^083^1999
B2^KLX^033^2012
B3^ABC^593^2003
B4^KLX^323^2001

So as result I want to print the counter for the value in column 2 at the end of each line. E.g line one has value "ABC" in column2. "ABC" appears 3 times in the entire file and is printed at the end of the line.
Output File
A1^ABC^173^2012^3
A2^BDK^153^2013^1
A3^AFD^223^2012^1
B1^ABC^083^1999^3
B2^KLX^033^2012^2
B3^ABC^593^2003^3
B4^KLX^323^2001^2

Hope anyone got an idea on how to solve it.
Cheers
Answer: Here's one way using awk:
awk 'BEGIN { FS=OFS="^" } FNR==NR { a[$2]++; next } { print $0, a[$2] }' file.txt{,}

Results:
A1^ABC^173^2012^3
A2^BDK^153^2013^1
A3^AFD^223^2012^1
B1^ABC^083^1999^3
B2^KLX^033^2012^2
B3^ABC^593^2003^3
B4^KLX^323^2001^2

Comment: @Jotne Takes the input file twice, try `echo file{,}`.
Comment: @EdMorton https://www.gnu.org/software/bash/manual/html_node/Brace-Expansion.html
Comment: @EdMorton: Thanks Ed; good question! Please see the 'portability' section [here](http://mywiki.wooledge.org/BraceExpansion).
Comment: What does the `{,}` do?
Comment: Ahh, a new one for me, thanks :) -- `{,,,}` repeat it three times
Comment: +1 for the approach, I was just going to suggest `awk -F'^' 'NR==FNR{s[$2]++;next} {print $0 FS s[$2]} ' file file`. I've never seen that `{,}` notation - is that bash-specifc?
Comment: @sudo_O Thanks. ANy idea if other shells support it or if it's just bash?
Answer: The buffered approach:
awk -F'^' -v OFS='^' '{a[NR]=$0;c[$2]++}
    END{for(i=1;i<=NR;i++){split(a[i],b);print a[i],c[b[2]]}}' file

Output:
A1^ABC^173^2012^3
A2^BDK^153^2013^1
A3^AFD^223^2012^1
B1^ABC^083^1999^3
B2^KLX^033^2012^2
B3^ABC^593^2003^3
B4^KLX^323^2001^2

To store the output in a new file use the redirection operator:
awk -F'^' -v OFS='^' '{a[NR]=$0;c[$2]++}
    END{for(i=1;i<=NR;i++){split(a[i],b);print a[i],c[b[2]]}}' file > outfile

Comment: +1 very good answer, trust me I hadn't seen this while making mine work.
Comment: works fine, many thanks. But I´m actually not getting the split and following print section. what is 'b' doing?

awk array: count column return line