Formal test for exogeneity of instruments

Question: Is there a way for me to formally test the exogeneity of my instruments? For instance, I have an endogenous variable, FDI, which I am instrumenting with "ease of doing business ratings," as a better score on ease of doing business improves the FDI inflows into an economy. How would I test that this instrument is not related to my dependent variable (patent grant) in the structural model?
Answer: If you have exactly as many instrumental variables as endogenous regressors, then there is no way to test for IV validity in a homogenous effects model.
Consider, for example the following model:
$$
Y = \alpha + \beta D + U
$$
This is a homogeneous effects model: the treatment effect is a constant $\beta$ that is the same for everyone. The two IV assumptions are relevance and exogeneity. Relevance requires that $\text{Cov}(Z,D) \neq 0$. This is directly testable. Exogeneity requires that $\text{Cov}(Z,U) = 0$. This cannot be tested. To see why suppose that $Z$ is in fact an endogenous instrument, i.e. that
Suppose that $Z$ is in fact an invalid instrument, i.e. that $\text{Cov}(Z,U) \neq 0$.
In this case the IV estimand is still perfectly well-defined, it simply doesn't equal $\beta$:
$$
  \beta_{IV} = \frac{\text{Cov}(Z,Y)}{\text{Cov}(Z,D)} = \beta + \frac{\text{Cov}(Z,U)}{\text{Cov}(Z,D)}, \quad
  \alpha_{IV} = \mathbb{E}(Y) - \beta_{IV} \mathbb{E}(D).
$$
Now, let $V$ be the IV residual: $V \equiv Y - \alpha_{IV} - \beta_{IV} D$.
Note that $V$ is only equal to $U$ if $Z$ is a valid instrument, because this is the only way that we can have $\beta_{IV} = \beta$ and $\alpha_{IV} = \alpha$.
Using our definition of $V$, we can calculate $\text{Cov}(Z,V)$ as follows:
\begin{align*}
  \text{Cov}(Z,V) &= \text{Cov}(Z, Y - \alpha_{IV} - \beta_{IV} D) = \text{Cov}(Z,Y) - \beta_{IV} \text{Cov}(Z,D) \\
  &= \text{Cov}(Z,Y) - \frac{\text{Cov}(Z,Y)}{\text{Cov}(Z,D)} \text{Cov}(Z,D) = 0.
\end{align*}
In other words, $Z$ is always perfectly uncorrelated with the IV residual $V$ by construction, regardless of whether $Z$ is correlated with the structural error $U$.
A Durbin-Hausman-Wu test checks whether the OLS and IV estimands are the same. This does not tell us whether the instrument is invalid.
When there are more instruments than endogenous regressors, an overidentifying restrictions test can be used to test the null hypothesis that both instruments are valid. The intuition is as follows. Continue to assume that $Y = \alpha + \beta D + U$ but suppose now that we have two relevant instruments $Z_1$ and $Z_2$, i.e. $\text{Cov}(Z_1, D) \neq 0$ and $\text{Cov}(Z_2,D)\neq 0$. Define two IV estimands: one that uses $Z_1$ to instrument for $D$ and another that uses $Z_2$, namely
$$
  \beta_{IV}^{(1)} \equiv \frac{\text{Cov}(Z_1,Y)}{\text{Cov}(Z_1,D)} = \beta + \frac{\text{Cov}(Z_1,U)}{\text{Cov}(Z_1,D)}
$$
and
$$
  \beta_{IV}^{(2)} \equiv \frac{\text{Cov}(Z_2,Y)}{\text{Cov}(Z_2,D)} = \beta + \frac{\text{Cov}(Z_2,U)}{\text{Cov}(Z_2,D)}.
$$
Taking differences of the two estimands, we obtain
$$
  \beta_{IV}^{(1)} - \beta_{IV}^{(2)} = \frac{\text{Cov}(Z_1,U)}{\text{Cov}(Z_1,D)} - \frac{\text{Cov}(Z_2,U)}{\text{Cov}(Z_2,D)}.
$$
If both $Z_1$ and $Z_2$ are valid instruments, then $\text{Cov}(Z_1,U) = \text{Cov}(Z_2,U) = 0$ which implies $\beta_{IV}^{(1)} - \beta_{IV}^{(2)} = 0$.
Therefore, if $\beta_{IV}^{(1)}$ and $\beta_{IV}^{(2)}$ differ then at least one of the instruments $(Z_1,Z_2)$ must be invalid.
While it is formulated in a slightly different way, a test of overidentifying restrictions exploits this basic intuition to provide a test of the joint null hypothesis that both instruments are valid: $\text{Cov}(Z_1,U) = \text{Cov}(Z_2,U) = 0$.
While this example concerns two instruments in a model with a single endogenous regressor, the same idea applies whenever there are more instruments than endogenous regressors.
In a model with heterogeneous treatment effects, the equivalent of instrument exogeneity does have testable implications even if there are as many endogenous regressors as instruments. See the following references for details:

Huber, Martin, and Giovanni Mellace. "Testing instrument validity for
LATE identification based on inequality moment constraints." Review
of Economics and Statistics 97.2 (2015): 398-411.
Mourifié, Ismael, and Yuanyuan Wan. "Testing local average treatment effect assumptions." Review of Economics and Statistics 99.2 (2017): 305-313.
Kitagawa, Toru. "A test for instrument validity." Econometrica 83.5 (2015): 2043-2063.
Answer: Hausman and Wu specifications and the test for over identification will do.
Comment: what if you have exactly identified model
Comment: @qayoomkhachoo you can't use the Overid tests if you have a just-identified model.  Read the thread [here](http://www.stata.com/statalist/archive/2009-04/msg00691.html) (links to Statalist), in particular Kit's response from 15 April 2009
Answer: You're confusing the concept of endogeneity of instrument with its independence from your outcome.
Given the equation:
$y=\beta_0+\beta_1x+\beta_2z+u$
where $y$ is the outcome, $x$ is the endogenous variable, $z$ is an instrument, and $u$ are unobservables.  Endogeneity is what happens when one or more of your right-hand-side variables is correlated with $u$, so for your instrument to be endogenous, it would have to be correlated with $u$ and not $y$.
Recall the 3 criteria for a valid instrument (Wooldridge, 2009):

$cov(z,y)=0$
$cov(z,x)≠0$
$cov(z,u)=0$

The exogeneity of the instrument criterion refers to bullet point 3 above, and an over-identified model is required to test this criterion.  The remaining 2 criteria, however, can easily be tested.  Remember that $cov(z,x)≠0$ means $z$ cannot be directly related to $y$, except through $x$.  Meaning, $z$ and $y$ are allowed to be related in bivariate relationships, but as soon as $x$ is added to the model, the relationship should be null. One easy way of testing this relationship is to fit
$y=\beta_0+\beta_1x+\beta_2z+V\beta+\epsilon$,
where $x$ is the endogenous variable and $V$ is a vector of exogenous control variables. Controlling for all model covariates, including the endogenous variable, test the coefficient of $z$. $\beta_z$ should be non-significant.
Criterion 2 (also called the Test of Instrument Relevance) can be tested in a similar way but regressing $x$ on $z$ using the same control variables.
$x=\delta_0+\delta_1z+V\delta+nu$,
The test on the coefficient of $z$ ($\delta_1$) should be statistically significant in this case.

Reference
Wooldridge JM. Introductory Econometrics: A Modern Approach. 4th ed. Mason, OH, USA: South-Western, Cengage Learning; 2009.
Comment: $Cov(z,y)=0$ is not a requirement. This is the so-called reduced form relationship. The IV estimator is $\beta_{IV}=\frac{Cov(z,y)}{Cov(z,x)}$, i.e. the reduced form coefficient divided by the first stage coefficient of $z$, so the statement of this requirement is not correct.


Load-Balancing with ARR + IIS on ports other than 80

Question: Is there any way to use ARR to re-route requests to ports other than 80/443 (the default http/https ports, respectively)? We have several web applications that listen on ports other than 80 and would like to load-balance them using ARR. Is this even possible?
Answer: You can do this - when adding a server to the farm, click the "Advanced settings..." link, in there both the HTTP and HTTPS ports are configurable to whatever you want.
In the c:\windows\system32\inetsrv\config\ApplicationHost.config file, the server entry would look something like this:
<server address="192.168.1.1" enabled="true">
    <applicationRequestRouting httpPort="8080" />
</server>

Depending on how your web application works, you may also need to turn on the "Reverse rewrite host in response headers" setting in the Proxy configuration for your farm.
Answer: Screenshot of how to do it (source):


Confusion Regarding diminished scales and how to practise them

Question: I have found that there are conflicting views on what is considered the diminished scale in jazz and when to play it.
If I have a ii-V- I in C and would like to apply a diminished scale to create a line on the V7 chord containing the chord tones of G7 (G,B,D,F) would I use the G whole/half diminished scale or the G Half-Whole diminished scale?
In order to improvise with them I would need to be able to apply them without too much thought. How should I visualise the diminished scale when I see the G-dim chord symbol? In the context of a G7 chord, I could theoretically just visualise the semitone below the G7 chord tones i.e Bb, Db, E and Gb. But I can't say if this method is a valid method of learning the diminished scale.
This confusion really rules out using the dim scale for me.
Practising the scale up and down doesn't help. I think my approach is totally wrong.
Comment: You can use the G Mixolydian scale to improvise on that very same V7 chord of C Major despite G not being a chord tone of all kinds of 3-note chords you can make with the G Mixolydian scale (Am, Dm, F, etc.). Get used to hearing nonchord tones when using a scale to improvise on a chord.
Comment: What makes you say that using any dim scale, or the notes thereof, over V or V7 in a piece?
Comment: Half whole would give you the b9 and the #9 while whole half would give you 9 and #9, miss the 3rd and give you 4.
Comment: use of the diminished scale on dominant chords is an option due to the tensions in the diminished scale which fit well with the dominant sound.
Answer: I would not use the G whole-half diminished scale to create a line on the V7 chord of a ii-V7-I chord progression in C Major. I would use the G half-whole diminished scale instead.
The G whole-half diminished scale is G-A-B♭-C-D♭-E♭-E-F♯ and sadly does not contain a B♮, unlike the V7 chord of C Major (which is a G7 chord).
The G half-whole diminished scale is G-A♭-B♭-B-C♯-D-E-F and notably contains all 4 notes of the V7 chord of C Major: G, B, D, and F.
Pick the diminished scale that contains the notes of the chord you want to play and use it. If no diminished scale contains all the notes of your chord, pick another type of scale.
Answer: In a strict sense, the diminished scale is the whole-half scale, because that's the scale you want to use over a diminished chord. It has all the notes of the diminished chord and all notes a whole step above those chord tones. Since these extra notes are a whole step above the chord tones they are no avoid notes.
If you used the half-whole scale over a diminished chord, you would also have all chord tones, but in addition you'd get four notes a half step above each chord tone. Those four notes would be dissonant in an undesirable way (i.e., they'd be avoid notes).
Having said that, it is indeed common to call both scales (whole-half and half-whole) diminished scale, so it's important to make sure which one is meant. Of course, they are modes of each other.
The half-whole scale is the one you want to use over a dominant seventh chord. It has all the chord tones and the b9, #9 and 13.
Note that the upper structure of a V7(b9) chord is just a diminished chord. E.g., G7(b9) is B diminished with a G in the bass:
G7(b9): G - B - D - F - Ab
Bdim7:      B - D - F - Ab
Over the Bdim7 chord you'd use the B whole-half scale, and over G7(b9) you'd use the G half-whole scale. This makes sense because these scales are of course identical:
B w-h: B C# D E F G Ab Bb
G h-w: G Ab Bb B C# D E F
Answer: One way to see and use a diminished scale is as a combination of two dim7 chords.
As a starting point, one way to see and use one dim7 chord is as a dominant chord, and its "mount points" are on the third and seventh of the dominant seventh. For example if you have a G7, you take its third, B, and put a dim7 there. Or you could take the G# - a semitone above the root - and place your dim7 there. It's the same thing, because dim7 is completely symmetric.
This overlaying of a dim7 on a dominant seventh is quite standard jazz stuff. For example, G7 <=> Bdim7 (or Fdim7 or G#dim7 or Ddim7) ... But if you add Gdim7 as well, you get the G half-whole diminished scale.

Gdim7 + G#dim7 = G half-whole diminished scale
... and it works as substitute for a G7 dominant chord.

It might be easier to remember, if you notice that the G half-whole diminished scale contains all of the nice colorful notes: root (G), major third (B), minor third (Bb), tritone (C#), seventh (F). And of course G# which is a colorful and bitterly melodic addition to a dominant. However it does not contain A, which sounds more like easy-listening, elevatory, mellow, soothing when added to a G. (My subjective opinion)
In addition to the dim7 chords, the half-whole diminished scale has ingredients for more weird/interesting/fun chords that can be used as a dominant. For example, G7 <=> E/G or C#/G.
Check out this incredible "E7" chord, which consists of various things taken from the E half-whole diminished scale:

It's like one of those ambiguous repeating patterns, "what do you see in this picture" ... well, depending on how you look at it, you might see Fdim7, Bb major, Bb13-9, E7+9, G7, Gdim7?
And because it's symmetric, that thing should work not only as an E7, but as a G7, Bb7 and C#7.
Answer: Using whole half scale on a V7 means your downbeats fall naturally on the the b9 and the chord tones. It also allows for an easier resolution to the tonic.


Find the parabola tangent to $4$ given lines

Question: Given $4$ lines in the Cartesian plane, defined by
$$ a_i x + b_i y + c_i = 0 , \hspace{5pt} i = 1, 2, 3, 4 $$
I want to find the parabola that is tangent to all four lines.
Here is what I did:
Any conic has the general equation
$$ A x^2 + B xy + C y^2 + D x + E y + F = 0 $$
If we define $ r = [x, y, 1]^T $ , then this equation can be put in the form
$$ r^T G r = 0 $$
where
$$ G = \begin{bmatrix} A  && \dfrac{B}{2} && \dfrac{D}{2} \\ \dfrac{B}{2} && C && \dfrac{E}{2} \\ \dfrac{D}{2} && \dfrac{E}{2} && F \end{bmatrix} $$
Since this is a parabola then
$$ A C - \left( \dfrac{B}{2} \right)^2 = 0 $$
Now for $i = 1, 2, 3, 4 $ define $N_i = [ a_i, b_i , c_i ]^T $, then it follows that
$$ N_i^T G^{-1} N_i = 0 $$
This equation is linear in the elements of $G^{-1}$.  The resulting linear system in the $6$ unknowns and $4$ equations, can be solved to obtain
$$ G^{-1} = \lambda (P_0 + \sigma P_1 ) $$
where $P_0 , P_1 $ are known by solving the system while $\lambda$ and $\sigma$ are arbitrary.  We take $ \lambda = 1 $, and note that
$$ G^{-1}_{33} = \dfrac{ \det(G22)  }{ \det(G) } $$
where $ G22 $ is the submatrix of $G$ obtained by by deleting the third column and third row.
Now since this is a parabola, as mentioned above, then $ \det(G22) = 0 $, therefore, we want $G^{-1}_{33} = 0 $.  Since $G^{-1} = P_0 + \sigma P_1 $, then $\sigma$ is given by
$$ \sigma = - \dfrac{ P_{033} }{ P_{133} } $$
Having obtained $\sigma$, we can calculate $G^{-1}$ and then invert it to obtain $G$.
From $G$, we now have all the coefficients of its general equation up to a scale.  Now it is possible after some manipulation of these coefficients to obtain the vertex and the orientation of the parabola, and, of course, its squared term multiplier.  This completes the specification of the desired parabola.
I would greatly appreciate your feedback on this method that I developed, and also other solutions that you might have to the problem.
Answer: Here's a very simple geometric construction to find the parabola tangent to four given lines.
The construction relies on a beautiful property of parabolas:

The focus of a parabola lies on the circumcircle of the triangle
formed by any three tangents to the parabola (see Appendix for a
geometric proof).

Let then $a$, $b$, $c$, $d$ be four given lines (no two of them parallel). The focus $F$ of the parabola tangent to them must lie (by the above property) on the circumcircle of the triangle formed by lines $abc$, and at the same time it must lie on the circumcircle of the triangle formed by lines $abd$. Those two circles have in common the intersection point of lines $a$ and $b$, which cannot be the focus. Hence the focus of the parabola is uniquely determined as the other intersection of the circumcircles.
To complete the construction we can then use another well-known property:

The perpendicular projection of the focus of a parabola onto any
tangent, lies on the line passing through the vertex and perpendicular
to the axis of the parabola.

If $A$ and $B$ are the projections of the focus on lines $a$ and $b$, the line through the focus perpendicular to line $AB$ is then the axis, and its intersection with $AB$ is the vertex of the parabola.
APPENDIX.
Proof of the property given at the beginning.
Let three tangents to a parabola be given, with tangency points $A$, $B$, $C$ and intersecting at $P$, $Q$, $R$ (see figure below). Let $F$ be the focus of the parabola, and $D$, $E$ the intersections of the axis of the parabola with lines $PQ$ and $QR$.
It is well known that $FA=FD$, which entails
$\angle FAQ=\angle EDQ$.
In addition, we know that (see here for a proof):

the exterior angle between any two tangents is equal to the angle
which either segment of tangent subtends at the focus,

which in our case implies $\angle AFQ=\angle DQE$.
It follows that in triangles $AFQ$, $DEQ$ we also have
$\angle FQA=\angle DEQ$.
But with an analogous reasoning we can prove that
$\angle FRB=\angle FER=\angle DEQ$. It follows that
$\angle FRP=\angle FQP$, and points $FPQR$ are concyclic as it was to be proved.


How do i download archive with static assets to my server and replace all "https://vuu.io" links with my own using wordpress?

Question: WHITE-LABEL STATIC ASSETS
All links to static content in fax report are hardcoded to Vuu.io, but you may replace them with your own URL.
In this case you'll need to download archive with static assets to your own server and replace all "https://vini.az" links.
Download static assets: https://vuu.i0/fax.zip
How can I achieve this using wordpress?
I have uploaded static assets on my server but cant find where do replace the url
Comment: Please provide enough code so others can better understand or reproduce the problem.
Answer: If you want to replace all occurrences of a certain URL in your database, I would personally use the WP CLIs search-replace command.
Example use case would be something like this:
<code>wp search-replace 'https://old-url.com' 'https://new-url.com'
</code>


Ember.Select conditional value for optionLabelPath

Question: I have a list of "Candidato" that I want to display in a dropdown box. Here's an example of a Candidato:
{"id": 3, "nombre": "Ivonne Álvarez Garcia", "sexo": 0, "confirmado": true}

Here's another one:
{"id": 1, "nombre": "Margarita Arellanes Cervantes", "sexo": 0, "confirmado": false},

In my template I have this:
{{view "select" content=model.candidatosPAN optionValuePath="content.id" optionLabelPath="content.nombre" selection=newParams.candidatoPAN}}

The thing is I want the "option label" to be variable, depending on the nombre and confirmado. So, for example, the option for Ivonne will be displayed as "Ivonne Álvarez Garcia (*)" ... because she is "confirmado". On the other hand, the option for Margarita would be: "Margarita Arellanes Cervantes" because she is not "confirmado".
What's the simplest and ember-ish way to do that?
Thanks!
Answer: One solution would be to wrap the content in a list property somewhere in your controller defining the labels, so for example:
// in your controller
candidatos: Ember.computed.map('candidatosPAN', function(candidato) {
    var suffix = candidato.get('confirmado') ? ' (*)' : '';
    return {
        id: candidato.get('id'),
        label: candidato.get('nombre') + suffix
    };
});

// in your template
{{view "select" content=candidatos optionValuePath="content.id" optionLabelPath="content.label"}}

Instead of the selection attribute you can use the value attribute to bind the selection with the id of the candidate. So instead of selecting a candidate you now select a candidate id.
Comment: Hi, I also considered that approach. However because of the way the selected option is bound (it is bound directly with the underlying model), I had to  use the original candidatosPAN I drew from the route's model.

Is there any approach that involves defining a simple extension of Ember.Select maybe?
Comment: If you want to update your model directly, you can add an observer to your controller which observes the selected id. In the observer you have to find the correct candidate and update your model with the candidate.


Monte-Carlo Simulation for Quantile Regression

Question: I am trying to perform a Monte-Carlo simulation using R. Currently I am getting stuck simulating the data. In a usual regression setting I would draw a random sample of the independent data and then compute the value of my dependent variable given a set of coefficients, set by me, and an error term. Consider for example:
$$
y_{i}=x_i'\beta+\epsilon_i
$$
In the case of OLS the coefficient is known since I would have set it in the above equation. However, I want to compare my estimate of $\beta$ for a whole range of quantiles for example $\tau = \{0.05, 0.25, 0.5, 0.75, 0.95\}$.
So my question is, how do I simulate my data and then get my true slope parameter for all $\tau$?
Comment: I don't follow what you're doing, but I'll mention that if you have $iid$ error terms, then the slope is the same at every quantile $\tau$.
Comment: My problem is essentially to get a sense of how to simulate data and then know the slope parameter for the desired quantiles. So I can compare my estimate to my true parameter.
Comment: It's going to depend on the error term. How are you simulating that?
Answer: Here is a little code to get you started. It is not a simulation study, but just one run, of course. It first illustrates that there is little difference for the slopes (as Dave already said) when errors are iid, where the second example is more meaningful with heteroskedastic errors.
library(quantreg)
n <- 1000
beta <- 2

x <- runif(n, 0, 4)
u <- rnorm(n) # homoskedastic errors
y <- x*beta + u

plot(x, y)
abline(lm(y~x), lwd=2, col="blue")
rq(y~x, seq(0.1,0.9, by=0.1))
lapply(seq(0.1, 0.9, by=0.1), function(tt) abline(rq(y~x, tau=tt)))

u <- x*rnorm(n) # heteroskedastic errors
y <- x*beta + u
plot(x, y)
abline(lm(y~x), lwd=2, col="blue")
rq(y~x, seq(0.1,0.9, by=0.1))
lapply(seq(0.1, 0.9, by=0.1), function(tt) abline(rq(y~x, tau=tt)))

Comment: Thank you for your illustrative example. So to sum it up, the coefficients differ between the quantiles if heteroskedasticity is present?
Comment: Yes, that is about it, I would say!
Answer: The true slope parameter depends on the variance function.  Here is some R code to illustrate.
n = 1000
beta0 = 2.1; beta1 = 0.7
set.seed(12345)
X = runif(n, 10, 20)
Y = beta0 + beta1*X + .4*X*rnorm(n)

beta1.90 = beta1 + qnorm(.90)*.4 # True slope of the .90 quantile function
beta1.90
library(quantreg)
fit.90 = rq(Y ~ X, tau = .90)
summary(fit.90)

Comment: Thank you. That 6th command was what I was looking for!


Run a python module in a flask HTML template

Question: I'm trying to decode an output from my database from base64 to text. here is a snippit my code:
from flask import Flask,render_template, request, json
from flask_mysqldb import MySQL
import base64

app = Flask(__name__)
app.config['MYSQL_HOST'] = 'localhost'
app.config['MYSQL_USER'] = 'example'
app.config['MYSQL_PASSWORD'] = 'example'
app.config['MYSQL_DB'] = 'products_db'

mysql = MySQL(app)

@app.route("/")
def main():
    cur = mysql.connection.cursor()
    cur.execute("select * from products;", mysql.connection.commit())
    data = cur.fetchall()
    return render_template('products.html', data=data)

my html page:
<!DOCTYPE html>

    {% for product in data %}

         {{product[0]}}
         {{product[2]}}
         {{base64.b64decode(product[3])}}

    {% endfor %}

I'm trying to decode product[3] but im failing, how to do it on flask? how can i import other modules aswell?
thank you
Comment: @CDJB I'm new to FLASK and do not understand how to implement that solution to my code. I'm still learning how FLASK works
Answer: You can apply the function to the target data value before passing the returned database contents to render_template:
@app.route("/")
def main():
   cur = mysql.connection.cursor()
   cur.execute("select * from products;", mysql.connection.commit())
   data = [[*a, base64.b64decode(b)] for *a, b in cur.fetchall()]
   return render_template('products.html', data=data)

Now, in your template:
{% for product in data %}

     {{product[0]}}
     {{product[2]}}
     {{product[3]}}

{% endfor %}

Edit: Python2 solution:
data = [i[:3]+[base64.b64decode(i[3])]+i[4:] for i in cur.fetchall()]

Comment: Thank you. it is giving me an error regarding the star before the character "a" in the data variable.
Comment: @xiholac761 No problem, please see my recent edit. `*` can be used for unpacking, and is available in Python3 only. I added a solution for Python2.
Comment: thank you so much. flask has thrown me an error:     ValueError: too many values to unpack

could it be because too many letter? what do these letters represent if i may ask?
Comment: @xiholac761 The use of multiple comma separate variables in a `for` loop is called unpacking. I have updated the Python2 solution to better reflect your database content structure.
Answer: You can't do base64 decode using jinja templating language. base64.b64decode is not supported in jinja. You may do this on server side and pass the data to template.
from flask import Flask,render_template, request, json
from flask_mysqldb import MySQL
import base64

app = Flask(__name__)
app.config['MYSQL_HOST'] = 'localhost'
app.config['MYSQL_USER'] = 'example'
app.config['MYSQL_PASSWORD'] = 'example'
app.config['MYSQL_DB'] = 'products_db'

mysql = MySQL(app)

@app.route("/")
def main():
    cur = mysql.connection.cursor()
    cur.execute("select * from products;", mysql.connection.commit())
    data = cur.fetchall()
    modified_data = []
    for item in data:
        modified_item = []
        modified_item.append(item[0])
        modified_item.append(item[1])
        modified_item.append(base64.b64decode(item[3])
        modified_data.append(modified_item)
    return render_template('products.html', data=modified_data)

template index.html
<!DOCTYPE html>

    {% for item in data %}

         {{item[0]}}
         {{item[2]}}
         {{item[3]}}

    {% endfor %}

Comment: Thanks for your help. When i run the code I get the following error:

TypeError: 'tuple' object does not support item assignment
Comment: I made some to fix that error.
Comment: Thanks for your efforts. Your code only returns one record, is there a way to display the whole table? first column: ID second column: Name third column: DECODED_BASE64('dGVzdAo=')
Comment: How many records do you have in db?
Comment: im trying to include pagination with limit. each page has 5 records. total: 468 records.
Comment: what does modified_data contain before render? I think it should have more than one record.
Comment: problem solved, thanks. any resources you recommend to learn more about flask?
Comment: Miguel made some really nice tutorial about flask https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world


How to load module include files in .test file?

Question: I'm trying to update the Drupal 7 version of the password_policy.test file that is included with the Password Policy module. When I run the tests, I'm getting "Fatal error: Call to undefined function password_policy_constraint_uppercase_validate()".
The function in question is in an include file in the constraints subdirectory, located inside the module's directory. I've tried adding to the test's setUp() function, along the lines of:
drupal_load('module', 'password_policy');
module_load_include('inc', 'password_policy', 'constraint_uppercase');

I've also tried adding the module_load_include line inside the individual test method generating the error. However, none of my attempts so far appear to be working and the error still persists.
How should I go about loading the ./constraints/constraint_uppercase.inc and other include files, so that the necessary functions are loaded prior to the tests being run?
Answer: You have a flaw in your code, module_load_include works by including files relative to the module base dir, so you should do
module_load_include('inc', 'password_policy', 'constraints /constraint_uppercase');

to include password_policy/constraints/constraint_uppercase.inc
When you do
module_load_include('inc', 'password_policy', 'constraint_uppercase');

You instead try to include password_policy/constraint_uppercase.inc which doesn't exist.
Answer: I believe you can add in the modules .info file:
files[] = constraints/includeme.inc
files[] = constraints/includeme2.inc
files[] = tests/included.test

Comment: Files added in the info file on serve the purpose of building the class registry, which allows Drupal to auto load files which contain the needed definition of a PHP class. That's why you should add the test file in the info file, as tests are classes, but this won't solve the problem of loading a class in the test itself.


Parsing HTML Data in UITableView

Question: I have an HTML Data which looks like:

Some Title

Some Title 1

Some Title 2

I'm able to show Link title in the tableView, but how can I get URL of these specific title to load a new view (which is pushed through navigation controller upon selecting table cell) ?
Here's the code I'm using to parse HTML Data:
HTMLParser * parser = [[HTMLParser alloc] initWithData:responseData error:&error];
HTMLNode * bodyNode = [parser body];
someArray = [[bodyNode findChildrenWithAttribute:@"id" matchingName:@"someID" allowPartial:NO] retain];

And this one to show the array data in the table:
- (NSInteger)tableView:(UITableView *)tableView numberOfRowsInSection:(NSInteger)section  {
    return [someArray count];
}

- (UITableViewCell *)tableView:(UITableView *)tableView cellForRowAtIndexPath:(NSIndexPath *)indexPath {

    static NSString *CellIdentifier = @"Cell";

    UITableViewCell *cell = [tableView dequeueReusableCellWithIdentifier:CellIdentifier];
    if (cell == nil) {
        cell = [[UITableViewCell alloc] initWithStyle:UITableViewCellStyleDefault reuseIdentifier:CellIdentifier];
        cell.accessoryType = UITableViewCellAccessoryDetailDisclosureButton;
    }

    HTMLNode* someNode = [someArray objectAtIndex:[indexPath row]];
    cell.textLabel.text = [NSString stringWithFormat:@"%@",[someNode allContents]];

    return cell;
}

Comment: very valuable question to ask ... i am also waiting for answers
Answer: When you get a node and want information about a child node within it, you have to continue parsing to find the child node and then get the attribute you are looking for. For example:
HTMLNode *anchorNode = [someNode findChildTag:@"a"];
NSString *anchorHref = [anchorNode getAttributeNamed:@"href"];

It might help to mention that you are looking at Ben Reeves' HTMLParser. (At least I think that is the one you are using).

Parsing HTML Data in UITableView


Android: Why is the zoom control like this?

Question:
Why does the zoom control show like this? It only happens on QVGA resolution smulator. It works perfectly on HVGA resolution.
This is the code that I used.
    ZoomControls zoomControls = (ZoomControls) findViewById(R.id.zoomcontrols);
    zoomControls.setOnZoomInClickListener(new View.OnClickListener() {
        public void onClick(View v) {
                mc.zoomIn();
        }
    });
    zoomControls.setOnZoomOutClickListener(new View.OnClickListener() {
        public void onClick(View v) {
                mc.zoomOut();
        }
    });

zoomcontrols defined in xml as:
<ZoomControls android:id="@+id/zoomcontrols"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:layout_alignParentBottom="true"
    android:layout_centerHorizontal="true"/>

Answer: Shouldn't be nothing wrong with your code. It could be a emulator bug. You should test it on a real device if you can.
It seems you are not the only one. Zoom controls on emulator running Android 1.6 with QVGA


Continuous Kolmogorov-Smirnov test?

Question: Is there something like a K-S test, but using an integral instead of a supremum?
My thinking is that $\int_{-\infty}^\infty|F(x) - G(x)|dx$, where $F$ and $G$ are CDFs, will approximate zero if they are the same distribution, and will increase as the distributions diverge (in any manner - non-parametric?). No idea what that variable would look like, though.
The K-S test seems problematic to me because with small sample sizes, noise can have large effects. By using all the data, instead of just the largest difference, some of the effect of that noise can perhaps be removed.
Comment: The integral is not invariant to monotonic transformations (so you lose the nonparametric feature).  If you apply the PIT (so you're always testing uniformity, restoring it to be nonparametric) then your integral corresponds to the Cramer-von Mises, I believe.
Comment: http://en.wikipedia.org/wiki/Probability_integral_transform
Comment: Thanks Nick, added it to the [wikipedia disambiguation page](https://en.wikipedia.org/wiki/PIT_%28disambiguation%29) :)
Comment: @Glen_b: what is PIT? I was wondering whether Cramér-von Mises was what I was talking about - I stumbled across it just after asking the question.
Comment: Apologies for the abbreviation; it occurred to me a short while ago that I should come back and mention what that was, but @NickCox is fortunately there first. [I recall asking *exactly* your question about the integral as a stats student many years ago.] ... The Anderson-Darling is one of the most common choices; it has excellent power in a wide range of situations. But I agree with the advice to consider QQplots
Comment: Just to double check: if you're estimating parameters, none of K-S, C-vM, A-D have the required null distributions.
Answer: Indeed. Check out Anderson-Darling, Cramér-von Mises tests, etc. http://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test is a start.
All these tests seem problematic to me for other reasons, not least because they don't tell you directly what is wrong. Increasingly with bigger datasets they reject hypotheses that are practically acceptable on the basis of statistically significant but practically insignificant deviations from what distribution is being fitted.
Quantile-quantile plots are a more practical technology.
Comment: Complicated territory, but if you want or need a test, go with the existing machinery; don't invent a new procedure ad hoc. (For example, successive quantiles are necessarily dependent.) Naturally, almost nothing is guaranteed to work well with small samples.
Comment: You know, I actually used QQ plots on this data, a few months ago... didn't even think about it while writing this question. I'll have to go back and have another look at it. It would be nice to get a simple statistic out of it though - I was thinking something along the lines of a linear OLS fit to the QQ-plot, but my statistical experience is minimal, and I wasn't sure how valid that would be.
Comment: Extra: The way that K-S works lends itself to criticism that it is _insensitive_ in the tails, where we should care most about being right and _unduly sensitive_ in the middles of distributions, where bad behaviour bites us less. Think of it this way: even the most outrageous outlier contributes just 1/$n$ to the CDF. See Rupert G. Miller, Beyond ANOVA, Wiley 1986/Chapman and Hall 1997 for this line.
Comment: @naught101 You're not too far off another test; in the normal case, the squared correlation in a QQplot is basically a Shapiro-Francia test (related to the Shapiro-Wilk). It won't be distribution-free, though, you'll have to get a new null distribution for every distribution you test (aside location-scale transformations). (Simulation may be required.)

Continuous Kolmogorov-Smirnov test?


Strategies for graphing distributions of log-odds estimates and the corresponding odds ratios

Question: I am currently writing up an experiment where we asked people on a 0-10 scale how much they expected three different beverages, water, decaf, and coffee, to reduce their caffeine withdrawal symptoms. Initially, using Bayesian parameter estimation, I modelled this as a within-subjects regression beverage type as the categorical predictor. I got great estimates that matched the mean differences from the descriptive stats, but the posterior predictive check of this model returned draws from the joint posterior that were outside the actual range of the data.
So to remedy this, and constrain the draws from the posterior to the actual range of the data I modeled the outcome variable as an ordinal variable, using ordered logistic regression (see this post).
I wanted to get estimates of the difference in log odds between the three beverage types. To do this I simply calculated, at each step in the markov chain, the difference in estimated log odds between each of the three pairwise comparisons of the three groups (water vs decaf, water vs coffee, decaf vs coffee). The parameter estimates from this model are expressed in log odds, which I can graph as histogram/density plots to show the mean and 95% Highest Posterior Density Interval for each pairwise comparison. However log-odds are difficult to interpret even at the best of times, let alone in an ordered logistic regression. So what I did was exponentiate each chain of log-odds estimates for each pairwise comparison to get odds ratios. These are the histograms for log-odds of the difference scores and odds ratios for each pairwise comparison

Each row graphs the histogram/density plots for each pairwise comparison as (i) log-odds (left) and (ii) odds ratios (right). Each graphs also has the mean and HPDI of each markov chain for each comparison.
To my eyes the odds ratio graphs look strange, of course this doesn't mean they're wrong, just that I am inexperienced. I have two questions

Are the mean and the 95% HPDI the right statistics to use to summarise the distribution of odds ratios and/or log-odds? Mode or mean would make little difference for the log-odds but in the odds ratio graph the mean is considerably to the right of the the peak of the density plot, due to the heavy skew in the data. Perhaps the mode would be better?
Do these graphs look ok? I understand exponentiating log-odds transforms distributions in unintuitive ways, but I don't have much experience in reporting distributions in these ways and just wondered if the right-hand column graphs look sensible, given the log odds distributions on their left?
Answer: To answer your second question first: your graphs look absolutely fine to me. Odds ratios typically have a skewed distribution - for one thing they lie in $[0, \infty)$ and the 'neutral' value is 1. That's one reason for analysing their logs, for which the range is $(-\infty, \infty)$ and which typically have a reasonably symmetric distribution.
Short answer to your first question: Take the posterior means and HPDIs of the log-odds ratios, then exponentiate them to give odds ratios for interpretation if you wish.
It makes more sense to look at geometric means rather than arithmetic means of ratio measures –  ½ and 2 are equally far from 1 on a ratio scale. The geometric mean of a set of ratios is the same as the exponential of the arithmetic mean of their logarithms (see e.g. Wikipedia). As much of statistics centres around (arithmetic) means and expectations, statistical analysis of ratios is usually conducted on their logarithms, and the results then transformed back to the ratio scale only for interpretation. From a Bayesian perspective, you'd want a Bayes estimator of a ratio measure to be based on a loss function that punishes ½ and 2 equally if the true value is 1. Working on the log scale means you can use the posterior mean or median (the posterior mode is harder to motivate).
There's a nice illustration in your case: for consistency you'd hope that the Water:Coffee odds ratio should be equal to Water:Decaf $\times$ Decaf:Coffee, implying that the log-odds ratio for Water-Coffee should equal Water-Decaf $+$ Decaf-Coffee. That doesn't work for the posterior means of the odds ratios: $0.4 \ne 3.9 \times 0.2$. It does work for the posterior means of the log-odds ratios: $-1.3 = 1.0 + (-2.3)$. (It doesn't work for the endpoints of the HDPIs because stats ain't that simple – you can't add standard errors or endpoints of uncertainty intervals even when estimates are independent, and here they're definitely not...)


Cox Regression used for business churn/ lifetime value

Question: Cox regression seems useful as a broad strokes model for measuring survival/ churn of a subscription-based product/ account lifetime estimate -- and the broad factors that might influence the survival curve (and area under the curve for estimated lifespan).
Unlike typical human survival data however where there's generally a live/ die binary ---
In business terms, unlike with an account "dying" -- the living accounts can "grow" in terms of monthly/ annual revenue.
In other words, if one were building a model to predict "lifetime value" (estimated lifespan x (value per t ) .... would one combine a linear regression model to estimate "growth/ value" over time ... to combine with 'estimated lifespan remaining' essentially?
This seems like a fairly common business problem. I've been playing around with a few things -- and the Cox portion seems straightforward. In terms of estimating lifespan based on a few key variables & current age --
The other part seems complicated. Like sure, you can do a very basic 'any account that survives grows 10% per month" -- but in reality there are many measurable variables that would impact this prediction as well.
I suppose the models wouldn't be combined per se.... there would just be a Cox model for lifespan ... a regression model (or some sort of time series) -- for estimated 'Revenue per T' ... and then essentially a cross-product of some sort of the outputs...
Answer: easiest to use a discrete time survival model, aka any probabilistic classifier (eg logistic regression or xgboost) , which predicts probably of churn in next period, given didn't churn until then.
Then you can calculate any arbitrary payoff as the sum of probability of churning at each period x payoff at each period (for example)


Notation in wikipedia article about Wick's theorem

Question: The wikipedia article about Wick's theorem can be found here.  In that article, they use a rather strange notation:
$$\hat{A}^{\bullet}\hat{B}^{\bullet}$$
the bullet superscript isn't defined in the article.  I take it that they mean that $\hat{A}$ and $\hat{B}$ are operators or wavefunctions that need to be time ordered but have not been so already.
Can anyone clear this up for me?
Also, I suppose that physics stack exchange might be another place I could ask this.
Comment: The've defined the notation at the beginning of the section "Definition of contraction". The bullet doesn't have a meaning on its own, but two bullets is the notation for Wick contraction.
Answer: I've answered my own question with the help of Adam above:
The bullet notation used in the wikipedia article is to make up for the fact that whatever math editor they're using probably doesn't have the overhead bracket notation usually used to indicate that two variables are about to undergo a Wick contraction.
Or at least the math editor doesn't handle that symbol correctly, etc.


How to quickly subset many dataframes?

Question: I have 180 DataFrame objects, each one has 3130 rows and it's about 300KB in memory.
The index is a DatetimeIndex, business days from 2000-01-03 to 2011-12-31:
from datetime import datetime
import pandas as pd
freq = pd.tseries.offsets.BDay()

index = pd.date_range(datetime(2000,1,3), datetime(2011,12,31), freq=freq)

df = pd.DataFrame(index=index)
df['A'] = 1000.0
df['B'] = 2000.0
df['C'] = 3000.0
df['D'] = 4000.0
df['E'] = 5000.0
df['F'] = True
df['G'] = 1.0
df['H'] = 100.0

I preprocess all the data taking advantage of numpy/pandas vectorization, then I have to loop through the dataframes day by day. To prevent the possibility of 'look ahead bias' and get data from the future I must be sure each day I only return a subset of my dataframes, up to that datapoint. I explain: if the current datapoint I am processing is datetime(2010,5,15) I need data from datetime(2000,1,3) to datetime(2010,5,15). You should not be able to access data more recent than datetime(2010,5,15). With this subset I'll make other computations I can't vectorize because they are path dependent.
I modified my original loop like this:
def get_data(datapoint):
    return df.loc[:datapoint]

calendar = df.index

for datapoint in calendar:
    x = get_data(datapoint)

This kind of code is painfully slow. What is my best option to improve its speed?
If I do not try to prevent the look ahead bias my production code takes about 3 minutes to run but it is too risky. With code like this it takes 13 minutes and this is unacceptable.
%%timeit
A slightly faster option is using iloc instead of loc but it is still slow:
def get_data2(datapoint):
    idx = df.index.get_loc(datapoint)
    return df.iloc[:idx]

for datapoint in calendar:
    x = get_data(datapoint)

371 ms ± 23.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
for datapoint in calendar:
    x = get_data2(datapoint)

327 ms ± 7.05 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
The original code, which was not trying to prevent the possibility of look ahead bias, simply returned the whole DataFrame when called for each datapoint. In this example is 100 time faster, real code is 4 times faster.
def get_data_no_check():
    return df

for datapoint in calendar:
    x = get_data_no_check()

2.87 ms ± 89.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Comment: Why not just use `df.loc[:datapoint]` directly, if you have a preset datapoint value ?  Why have to loop through all indices of a dataframe ?  Looping through ALL indices and each time loc up to that index, this effectively is not subsetting anything.  Is this useful ?
Comment: This is what the code that runs in 3 minutes does but it is risky. You could be tired and put a wrong value for datapoint and you'll get data from the future, which is what I am trying to prevent
Comment: Can't you just sort the dates first to always be sure that the dates after the current are all after the current one? `df.sort_values(by='date',ascending=True)`
Comment: index is already sorted
Comment: I think you can devise method to come up with a date, just call it datapoint which is a business day and verified to be a past or present date but not future.  Then just use your df slicing to get the date range you want.
Comment: When I start the loop I have all my data preprocessed. After that, for each date/datapoint I only want to get a subset of the data from the beginning to that date/datapoint. Whith this new dataframe I'll make other computations. I edited my question to better explain.
Comment: In your added example, what's the criteria such that the dates between datetime(2010,5,5) and datetime(2010,5,15) got excluded ?   No business date within this period ?
Comment: sorry a typo. right datetime is (2010,5,15)
Comment: So you have the current datapoint pre-determined somewhere else ?  Is it possible to substitute this pre-determined date to my suggested solution replacing the datetime.now() ?  By this, my code can still help determine the latest business date within that date range.
Comment: yes it is predetermined in another place. This is why, unfortunately, your suggested solution it is not useful. The real code is 5K+ lines and thhere is a mediator pattern that manages several components. Returning data preventing the look ahead bias is my bottleneck. For each datapoint I know in advance the date so I do not have to recompute the range, I already know it.
Comment: Sorry, a bit confused.  If you know the datapoints in advance, so why you can't just plug-in the datapoint to the slice `df.loc[:datapoint]`   I guessed you have to make sure each datapoint is a business date, so suggested the code.
Comment: Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/228943/discussion-between-nicola-prada-and-seabean).
Answer: See if this work for you:
datapoint_range = pd.date_range(datetime(2000,1,3), datetime.now(), freq=freq)
datapoint = datapoint_range[-1]

Logic is: replacing the ending date to be today so as to ensure not future date.  Then get the last date of the range.
Then use your df.loc[:datapoint] to get the range you want.
Comment: In my example end date is  2011-12-31. Using datetime.now() is already "future"...
Answer: I solved it like this: first I preprocess all my data in the DataFrame to take advantage of pandas vectorization then I convert it into a dict of dict and I iterate over it preventing the possibility of 'look ahead bias'. Since data are already preprocessed I can avoid the DataFrame overhead. The increase in processing speed in production code let me speechless: down from more than 30 minutes to 40 seconds!
# Convert the DataFrame into a dict of dict

for s, data in self._data.items():
    self._data[s] = data.to_dict(orient='index')

How to quickly subset many dataframes?


Validação de data com DataAnnotation

Question: Queria saber se tem como validar se a Data inserida é maior ou não que a data atual usando Annotation, ou se eu tenho que tratar no controller mesmo. Se tiver como, disponibilize um exemplo por favor.
Answer: Pode fazer isso criando uma classe que deriva de RangeAttribute:
public class DateAttribute : RangeAttribute
{
    public DateAttribute()
      : base(typeof(DateTime), DateTime.Now.ToShortDateString(), DateTime.MaxValue.ToShortDateString()) { }
}

Fonte
Comment: Eu gostei dessa resposta aqui também, mas está no SOF em Inglês. Não sei qual é a política de uso das informações da comunidade em inglês aqui no pt.SOF: http://stackoverflow.com/questions/19882296/mvc4-data-annotation-compare-two-dates
Comment: @StillBuggin Eu meti o link para a pergunta na minha resposta

Validação de data com DataAnnotation


Is it true that $A\cong B$ implies $A = B$ when $A$ and $B$ are ordered structures

Question: In Immerman's book "Descriptive complexity" he says that
$A \cong B$ implies $A = B$ when $A$ and $B$ are totally ordered structures.
See:  (Descriptive Complexity, Neil immerman)

Definition $\bf 1.21\quad$ (Isomorphism of Unordered Structures) Let $\cal A$ and $\cal B$ be structures of vocabulary $\tau=\langle R_1^{a_1},...,R_r^{a_r},c_1,...,c_2\rangle$. We say that $\cal A$ is isomorphic to $\cal B$, written, ${\cal A}\cong{\cal B}$, iff there is a map $f:|{\cal A}|\to|{\cal B}|$ with the following properties:

$f$ is $1:1$ and onto.
For every input relation symbol $R_i$ and for every $a_i$-tuple of elements of $\cal|A|$, $e_1,...,e_{a_i}$, $$\langle e_1,...,e_{a_i}\rangle\in R_i^{\cal A}\qquad\Leftrightarrow\qquad\langle f(e_1),...,f(e_{a_i})\rangle\in R_i^{\cal B}$$
For every input constant symbol $c_i$, $f(c_i^{\cal A})=c_i^{\cal B}$.

The map $f$ is called an isomorphism. $$\tag*{$\square$}$$
  $\quad$ As an example, see graphs $G$ and $H$ in Figure 1.1 which are isomorphic using the map that adds one mod five to the numbers of the vertices of $G$.
  $\quad$ Note that we have defined isomorphism so that they need only preserve the input symbols, not the ordering and other numeric relations. If we included the oredring relation then we would have $\cal A\cong B$ iff $\cal A=B$. To be completely precise, we should call the mapping $f$ defined above an "isomorphism of unordered structures" and say that $\cal A$ and $\cal B$ are "isomorphic as unordered structures". (Note also that, since "unordered string" does not make sense, neither does the concept of isomorphism for strings. By a strict interpretation of Definition 1.21, two strings would be isomorphic as unordered structures iff they had the same number of each symbol.)
  $\quad$ The following proposition is basic.

He defines the input relations as the relations that are not ordering, plus, times.
However, if we define the structures $(\{0,1\},\leq)$ and $(\{1,2\},\leq)$ are clearly isomorphic as ordered structures ($\leq$ is the usual ordering on the natural numbers). They are not equal, however. So what does he mean exactly?
Comment: In the context of descriptive complexity, "ordered structure" usually means a finite structure whose underlying set (also called its universe or its domain) is of the form $\{1,2,\dots,n\}$ for some natural number $n$ and whose basic relations include the standard linear ordering relation of natural numbers (restricted to $\{1,2,\dots,n\}$). With this understanding, Immerman's assertion is correct.  (With the usual, model-theoretic notion of "ordered structure", Immerman's statement would not be correct, because of counterexamples like yours.)
Comment: I am not sure this '*ordering*' here means partially ordering or something like that. Doesn't it say anything about it in the context?
Comment: It should be a total order.
Comment: @AndreasBlass: this is an *answer*.
Answer: At Barci's suggestion and in order not to unnecessarily leave this question unanswered, I'm repeating my comment as an answer.  In the context of descriptive complexity, "ordered structure" usually means a finite structure whose underlying set (also called its universe or its domain) is of the form $\{1,2,\dots,n\}$ for some natural number $n$, and whose basic relations include the standard ordering of the natural numbers (restricted to $\{1,2,\dots,n\}$).  With this understanding, Immerman's assertion is correct.  (With the usual, model-theoretic notion of "ordered structure, Immerman's statement would not be correct, because of counterexamples like yours.)
Addendum: Some authors prefer to use $\{0,1,\dots,n-1\}$ instead of $\{1,2,\dots,n\}$.  Either way, one of the two structures in your counterexample would not count as an ordered structure, but different authors might disagree as to which one it is.


Como posso escrever num arquivo do servidor usando FS do Node?

Question: Está rodando na minha máquina servidor o comando:
$ http-server . -p 3333

Tudo perfeito quando acesso desde um navegador Web de outra máquina (também Linux).
Porém ao rodar este script em Node, não consigo escrever num arquivo que está no servidor, nem me retorna erro. Por que o fs writeFile não funciona com um endereço IP? Há alguma solução para isso?

app.post('/formulario', (req, res) => {
 nome = req.body.nome
 idade = req.body.idade
 console.log('Nome.:', nome, '\nIdade:', Number(idade)+10)
 res.send('Nome.: '  + nome + '
Idade: ' + idade)
 fs.writeFile('http://192.168.0.113:3333/arquivoTeste.txt', 'Nome.: '+ nome, function (err) {
  if (err) throw err;
  console.log('Escrito!');
 })
 res.end()
})
Answer: Bom dia!
Para você acessar algo você deve passar o path de onde o mesmo se encontra, no seu caso você tenta acessar um ip e uma porta mas isso não te leva a diretorio nenhum dentro da maquina, por exemplo, um ftp lhe da acesso direto a uma pasta dentro da maquina atravez do ip + porta (exemplo: 192.168.0.5:21), uma outra coisa porque você não utiliza um banco de dados para poder  armazenar suas informações em vez de um arquivo?
Comment: Obrigado Gustavo. Eu consegui acessar de outra máquina o app pela porta, etx e percebi que salvava e acessava os dados perfeitamente. Supriu a necessidade. Foi algo simples por isso nao optei por um banco de dados.


Spring @Autowired annotated object value is null

Question: // My Factory class
@Component
public class UserRewardAccountValidatorFactory {

    @Autowired
    private VirginAmericaValidator virginAmericaValidator;

    private static class SingletonHolder {
        static UserRewardAccountValidatorFactory instance = new UserRewardAccountValidatorFactory();
    }

    public static UserRewardAccountValidatorFactory getInstance() {
        return SingletonHolder.instance;
    }

    private UserRewardAccountValidatorFactory() {}

    public PartnerValidator getPartnerValidator(Partner partner){
            return virginAmericaValidator;
    }
}

// My Validator class
@Service
public class VirginAmericaValidator implements PartnerValidator {

    @Override
    public void validate(String code) throws InvalidCodeException{
        //do some processing if processing fails throw exception
        if (code.equals("bad".toString())){
            throw new InvalidCodeException();
        }
    }
}

//Usage:
    PartnerValidator pv = UserRewardAccountValidatorFactory.getInstance().getPartnerValidator(partner);
    if (pv != null){
        try {
            pv.validate(userRewardAccount);
        } catch (InvalidCodeException e){
            return buildResponse(ResponseStatus.INVALID_USER_REWARD_ACCOUNT, e.getMessage());
        }
    }

My package scan level is at much higher level. Whats happening is my virginAmericaValidator is always empty. Why is @Autowired annotation not working.
Answer: Your current approach will not work with Spring as you are ultimately using new UserRewardAccountValidatorFactory to create the instance which essentially bypasses Spring context altogether. Two approaches that should possibly work are these:
a. Using a factory-method and using xml to define your bean:
<bean class="package.UserRewardAccountValidatorFactory" name="myfactory" factory-method="getInstance"/>

This will essentially return the instance that you are creating back as a Spring bean and should get autowired cleanly.
b. Using Java @Configuration based mechanism:
@Configuration
public class MyBeanConfiguration {
   @Bean
   public UserRewardAccountValidatorFactory myFactory() {
       return UserRewardAccountValidatorFactory.getInstance();
   }
}

Comment: or c) using @Configurable and AspectJ, then Spring inject even in Instances that are created by `new` (I mention this only for completes, and because to fight against this rumor that Spring can not inject in classes that are generated by `new` - because it is wrong)
Comment: If I make VirginAmericaValidator extend from another abstract class and implement interface there, will that make spring autowire for me?
Comment: No, that also will not work as the issue is still in the way `UserRewardAccountValidatorFactory` is created, not in the Validator. You should be able to get it working with one of the two options that I have mentioned. Approach that Ralph mentioned is also great, but it will require full AspectJ weaving to get it to work.


DFS synchronisation over slow internet connection

Question: I am configuring DFS replication on 2 servers connected over a VPN. The VPN is over a ADSL line with 0.06mb upload speed.
server1 - 192.168.1.2
server2 - 192.168.2.2
Both servers can see each other over the VPN by name, however if I was to set up DFS the initial sync will be moving 120GB of data, which will take far too long.
If I was to take server2 over to server1's local network, plug it in and do the initial sync over the local LAN (gigabit), then take the server back to its site, would DFS just continue to replicate the drives?
Any advice would be greatly appreciated.
Comment: if u were using dhcp for both server within domain, it could work i guess
Answer: Yes, this will work no problem.
Both DFS Namespaces and Replication are based on hostname, so as long as they can communicate and resolve that way, you're all set.
Also, using DFS for the initial sync will work but will probably take longer than pre-seeding the data manually. I'd suggest you robocopy the data  while they're on the same LAN, then let DFS-R keep them in sync. You can find more information, and a how-to guide here.


How to take TimeInput only in React-Admin?

Question: This code from React-Admin docs allows me to input Date and Time:
import { DateTimeInput } from 'react-admin';

<DateTimeInput source="published_at" />

Is there a way to input ONLY time in React-Admin?
Answer: https://mui.com/x/react-date-pickers/time-picker/
import * as React from 'react';
import TextField from '@mui/material/TextField';
import { AdapterDateFns } from '@mui/x-date-pickers/AdapterDateFns';
import { LocalizationProvider } from '@mui/x-date-pickers/LocalizationProvider';
import { TimePicker } from '@mui/x-date-pickers/TimePicker';

export default function BasicTimePicker() {
  const [value, setValue] = React.useState<Date | null>(null);

  return (
    <LocalizationProvider dateAdapter={AdapterDateFns}>
      <TimePicker
        label="Basic example"
        value={value}
        onChange={(newValue) => {
          setValue(newValue);
        }}
        renderInput={(params) => <TextField {...params} />}
      />
    </LocalizationProvider>
  );
}

Comment: Thanks for your response. I get the following error after trying the above code:
`Compiled with problems:X

ERROR in ./node_modules/date-fns/is_valid/index.js

Module build failed (from ./node_modules/source-map-loader/dist/cjs.js):
Error: ENOENT: no such file or directory, open '/Users/eshaangupta/Desktop/prototype/node_modules/date-fns/is_valid/index.js'`


Indications about the population when PCA essentially "fails" to reduce dimentionality

Question: This is more of a conceptual question rather than a methodological one I guess.
Let's assume that we have a dataset coming from a questionnaire and after some feature scaling we run a PCA to reduce the dimentionality. The results indicate that we need "a lot of" principal components (let's say close to the original number of questions) in order to capture let's say 80% of the variability. What would that indicate? What are the most probable scenarios?

The questionnaire needs smarter questions so that more info is captured by fewer questions?
The population consists of very complicated individuals?
Some questions (let's say the ordinal ones) require more levels?

Edit: Some additional details about the project
The population we are investigating are all the existing customers of a specific platform. Our random sample consists of approximately 4.2K individuals from the population who answered a questionnaire of ~80 questions (behavioural, personality, preferences etc). The objective is to i) understand the persona/behavioural groups that exist in our customer database ii) collect some golden questions to be able to classify more users afterwards without having to ask them all 80 questions again. Most of these questions are ordinal and some of them are categorical.
I've already done an initial clustering using PAM and Gower's distance but I wanted to look deeper and try more stuff. My plan is to run a Hierarchical K-means clustering after a PCA and then try some SOMs as well. My plan afterwards is to train a classification model to be able to classify the future users.
When I did the PCA in each category (I think 8 in total) I saw that for most cases the "best" PC had close to 12% variability explained which I found kind of low and it made me a bit curious. Hence the question
Comment: That simply means items hardly at all correlate. Now, what does that mean to you, a questionnaire designer/user? Do items have or have not to correlate, in your project?
Comment: @whuber makes sense. Let me edit my question and add a bit more context
Comment: If you are analyzing covariances, not correlations, uncorrelated data may still give different caliber PCs. See https://stats.stackexchange.com/q/92791/3277
Comment: Use a nonlinear dimensionality reduction technique. Using PCA is ridiculous when you have a non-Euclidian manifold.
Comment: @MatthieuBrucher, Can you elaborate? With suggestions, I mean.
Comment: If you have a non-Euclidian manifold, data that cannot be represented as a plane (even with error), then you should use something else. The typical example is Eigenfaces that is just creating wrong images compared to what can be done with an appropriate tool like autoencoders (for faces) or ISOMAP, LLE...
Comment: @ttnphns questions are meant to capture respondents' behaviours, personalities, preferences etc.  Intuitively, my guess would be that least some of these variables should correlate and I'm trying to identify the most likely scenario why this is not the case. Should we re-design the questionnaire? Is it the way it is and there is nothing I can do about it?
Comment: @MatthieuBrucher I know what you mean. However, imagine that 90% of these variables are ordinal and thus coded from -3 to 3. So it's not completely ridiculous to try PCA. My question is more conceptual as I said. Forget a bit the methodological part. Even with PCA and euclidean distances, Why is this happening you think?
Comment: If, as you write, it takes "close to the original number of questions" to account for $80\%$ of the variance, then you likely have a very small number of questions in the first place (depending on what "close to" might mean).  Given the vagueness of this assumption, as well as the complete lack of information about the questionnaire or its subjects, you are asking us to supply a great deal of speculation based on exceptionally little information.  Could you tell us what your problem *really* is?
Answer: "After some feature scaling". Maybe that is a bad choice? But you did not give any details...
You may be emphasizing variability in your data where it wasn't. Assume that all but one user chose the same answer for some question, say 2. but one funny user chose 3 for all questions. Now if you naively apply standardization first, all users will likely get a small negative value, and the one different user will get a very large other value. PCA will then preserve this variable largely unchanged - but it never was meaningful.
The other issue, in particular with PCA and standardization is resolution. Your input data only has a few levels, but that doesn't mean the "true" values shouldn't be continuous. It's best to think of these to have a high uncertainty. If the true values weren't 0 and 1, but 0.49 and 0.51 rounded, your variance would have been several times less! By PCA and these you pretend the values are of higher resolution and linear (but research has shown that even 188.8.131.52.5 questionnaire values are better not treated as numerical).


Chord analysis: b13 or #5

Question: I've come across an exercise in the Jazzology "jazz theory" book, where the chord should be identified, given the notes below:

I identified it as a B7#5#9, but the answer was B7#9b13. I'm guessing that to be a #5 the g should strictly be a double sharp f. That is a quite cumbersome notation though. My question is if I'm correct in my reasoning. I'm also interested in if the b13 is the interpretation the venerable people of this SE would make. I understand that there are voicings without the fifth, but I find it a bit strange to leave it out in an exercise.
Also, what are the implications regarding the (mis)interpretation as a #5? If it's interpreted as a #5, that would mean that the g sharp (6th) is present, while if it's a b13, f sharp (5th) is there instead. Which should have implications regarding which scales to choose to play over it.
I haven't played the chord and tried the different added tones to see what sounds good, I'm interested in understanding the exercise fully.
Comment: I think that you should add to your post that the context is jazz (right now it's a bit misleading). For example, in jazz there would be a bass note somewhere down that would justify the G natural being b13 instead of just b6.
Comment: @dtldarek: Good point, I've added this.
Answer: That's a valid voicing of both a B7#9#5 and a B7#9b13 as long as we aren't concerned with the spelling. Jazz musicians often are less concerned with spelling and more concerned with things being easy to sight read at the gig.
I (and others, but not everyone) often think of the chord nomenclature as implying things about appropriate scale choices:
<code> #5 implies a #4 (but #4 doesn't necessarily imply #5)
 b5 implies a natural 4
 b13 implies a natural 5
 #9 implies a b9
</code>
Note that Jamey Aebersold takes the #9 implications further, and says that a #9 also implies a #4 and #5.  I think that's true a lot of the time, but not always. http://www.jazzbooks.com/mm5/download/FREE-scale-syllabus.pdf
Comment: I like your answer but I would argue that b13 certainly does not imply natural 5.  b13 seems to be the standard way of describing what would classically be known as #5.  I find that b13 tends to imply #11 and no 5.
Answer: Chord nomenclature is dependent on context.
Without knowing the context, that chord could have different (and possibly equally correct labels) such as Eb+maj7(#11) (if respelled) or Gadd9(b13) to give a couple of examples.
I do not think you are correct in your reasoning - read on!
I disagree that interpreting the chord with an F## would imply a G# in any overlaid scale.  There are no G#'s in the chord, and the harmony is clearly illustrating a G natural.  Diatonically speaking, a B major (or even B melodic minor ascending) scale wouldn't make sense here.
Notationally, I'm not sure of your confusion as there is no F# in the chord.  It is a b13 (technically speaking, a b6 by the way) because the chord has been spelled with a G natural. The only reason why you would use F## as a substitute would be if the pitch is serving a specific inner-voice voice-leading function to the next chord.  Otherwise, it's just unnecessary.
To summarize, chord nomenclature does not determine what pitches are implied in scales to improvise over the chord.  Chord nomenclature should articulate the sounding pitches of the chord as clearly as possible.  In other words, you can't assign an enharmonic chord label to a chord because it implies other pitches / scales.  If a particular notation is used, it must be used to show a specific function.  If you can make a case for F## having a specific function over G natural, then great, use that notation.  Otherwise, chord names are for documentation.
Regarding the "#9", the notation is incorrect.  It should be spelled as a C##, not as a D natural. Otherwise you have a split third, in which case you'd need compelling reasoning for spelling it as such.
Comment: To be honest, to most jazz musicians the "compelling reason" you suggest in the last paragraph amounts to "That note's a sharp 9; this is jazz, don't make me read double-sharps if you don't have to." Common Practice rules of chord notation only apply up to a point in jazz, especially with a gnarly chord such as this one. (To a similar point, a jazz musician would NOT expect to see this called a b6 over a b13 -- it's just not the language.)
Comment: @NReilingh - If I were writing a jazz tune, I wouldn't write the chord with a b6 for the same reason you outlined. That said, it should be noted that it was never explicitly stated that we were talking about jazz - it merely appeared that way given the context of the OP's question. My comments about "compelling reasons" and nomenclature are intended for a broader interpretation of the question, which I thought was fair given ambiguity of the question.
Comment: Why I brought in the F# at all is due to that I interpreted the G as the fifth, and thus slipped into the track on how to voice the chord with just the name of the chord, and not the given tones. The exercise is from the Jazzology book from Hal Leonard. I thought it was a trustworthy source. The B#9b13 is in the answer key. Maybe they are trying to show that "jazz theory" isn't necessarily so strict...
Comment: I agree with you completely especially on the last paragraph. If you want a 9th spell it like a 9th not a 10th.
Comment: @MeaningfulUsername - how / why would you ever interpret "G" as a fifth above "B"?  It is *always* a sixth (a minor one in this case.)
Comment: I'm playing guitar, so when trying to figure out what chord it was, I tend to think of shapes on the fretboard. I was expecting the 5th to be part of the chord, and that is why I concluded it was a #5. This was against "formal" theory I realized, which caused me to ask this question for more information. I should have mentioned the jazz context, which I've now edited in.
Answer: B=root. D# =maj.3rd. G = b6. A = m7. D = m10. There is no b13, as G natural is too soon in the rising note list.Had the G been an F##, then B7#5 is a starter. Then the D on top, which could have been a C##, making #9, could have been B7#5#9. BUT it isn't.As is, it may be called B7b6b10, but that's hardly a common chord name.Someone goofed.
Comment: @Tim in my (limited) understanding, the difference between a 4 and an 11 is not what octave the note is played in relative to the root, but the context implication, i.e. that a 4 is usually a sus4, with the 3 omitted, while the 11 is not.
Comment: @jjmusicnotes ah, thanks for the explanation, but I don't think that's how chord extensions work. What octave a note is played in is a matter of voicing and has nothing to do with the chord identity.
Comment: @Tim let me ask you the converse question to explain my view: Do you think that a pianist, when encountering e.g. a C13, is obliged to hit a particular set of keys spanning more than one octave, but a bit less than two? That's not exactly how it works.
Comment: Could you explain what you mean by "too soon in the rising note list" ?
Comment: @microtherion - Tim means that if the G is considered a 13th, it should rightfully be on top of the chord, which it is not.
Comment: Though disagreeing with the too soon part, I agree that the D looks more like a 10b than a 9# to me (not to mention that with a 13b using a 9b would be more conventional; though I see the term conventional is a bit funny in the context of jazz). Also I think if the G is accepted as 13b, then D shouldn't be accepted as 9# either. I don't know whether the testing was carried out by a software, a textbook or a human, but I smell some inconsistency here.
Comment: @microtherion - so are you saying a 4th is the same as an 11th? a 2nd is the same as a 9th ? 'Cos they're not.
Comment: Usually, once we get above 7, the 7th is included. As in 11th is 4th +7th (4+7=11). Yes 4th is often sus 4th, in place of the 3rd, but when that actual note is an octave higher, including 7th, it doesn't clash with the 3rd underneath anyway.
Comment: I differ from this. To me it seems like a G chord in first inversion.
Comment: @NeilMeyer - with a G on the bottom, it's not going to be anything but a root. Inversions will have D,B etc in the bass. First inv. will put a B underneath.
Answer: A b13 voicing can be exactly the same in sound as a #5 voicing, but it depends on the notes you use in the construction of your voicings. The nomenclatures are different because each implies different notes which can be added to the chord. For example, in the case of a b13 voicing, you could use BOTH the natural fifth and the b13th, but in the case of a #5 voicing, the natural 5th, nor the 13th could be added but the b5 could be added.
In practice, sets of voicing such as the two mentioned above, are mixed, interchanged and blended according to aesthetic taste and the circumstances in the moment of playing.
As an aside, selecting notes for extensions of basic chord voicings (i.e. vanilla 7th chords) is intrinsically tied to scales-as scales are just horizontal representations of chords (i.e. vertical structures). Thus, #5 voicings can include notes from the wholetone scale, whilst the b13th voicings could include notes from the mixolydian (b13th) scale. Moreover, whether or not these scales were to be used for improvisation would also be an aesthetic decision.
Answer: I agree that writing a chord voicing that leaves out the 5 is a bit of a curveball.
The reason it's a b13 and not a #5 is basically because it IS a G in a B chord, and that's the "least gymnastics" way to annotate it.
Also, if there's any note you're going to leave out of a dominant chord voicing and still maintain the flavor, the 5th is the one that can go.  Leaving the 5th out basically gives the chord a bit more air and stops it from turning into a tight cluster...which has it's own uses and flavor, appropriate sometimes, undesirable other times.
Answer: I bet we could identify 20 or so "valid" spellings of that chord.  And herein lies the rub:   enharmonic spellings are the bane of existence for musicians trying to understand harmony.  A chord symbol has two purposes:  to name a particular grouping of notes such that it can be communicated to others intact, and to identify its FUNCTION within the harmony.  Enharmonic spellings can utterly obfuscate the chord's role in the harmony.  we know little about the chord's context here, but we can discern that no other inversion will be any more "lucid" than the one we've got, so "B" it is.
Answer: That looks like a G augmented chord in first inversion with a ninth added and fifth doubled.
If you are wondering if that D# is a raised fifth then I will say yes that is a G chord with an augmented fifth. Also my question will be if the other D in the chord is raised or not?
Comment: Read the whole question and the accepted answer. Note that OP writes, "My question is if I'm correct in my reasoning." Specifically, their reasoning about whether B7#5#9 is a reasonable name for the chord. Discussing a G chord does not in fact address the OP's question. The real implied question is "When do you say b13 and when do you say #5?". Again, your answer does not address this. PS: I am not the downvoter.
Comment: Again, not related to the OP's question. They didn't ask, "Is this a G or a B chord?" Notice *every other answer* addresses the idea of b13 or #5.
Comment: Could downvoters please explain there votes in a comment? My answer may be short but it still answers the question.
Comment: Well if the holy jazz book says so whom am I to question it. That is clearly not a chord built on the B. The basis for the chord is G=B=D
Comment: Without going through the exercise personally, I bet we could identify 20 or so "valid" spellings of that chord.  And herein lies the rub:   enharmonic spellings are the bane of existence for musicians trying to understand harmony.
Comment: This isn't the **first** time I have seen Neil Meyer's posts downvoted for no reason... (I've upvoted this)

Chord analysis: b13 or #5


A Maximum Likelihood Estimate of the mean for a multivariate normal distribution, the MLE of the mean is a scalar value or a vector scalar values?

Question: In the following equation, I have found the maximum likelihood estimate of the mean for a multivariate normal distribution
$\therefore \mu^*_{MLE}=\dfrac{1}{n}\sum_{i=1}^{n}x_i$
But I am wondering if the $\mu$ is a scalar value or a vector scalar values.
Comment: Is this the right way to think of what it is?
$\mu_{MLE} = \begin{bmatrix}\mu_1\\\mu_2\\\vdots\\\mu_n\end{bmatrix} =
\begin{bmatrix}\dfrac{1}{n}\sum_{i=1}^n x_1\\\dfrac{1}{n}\sum_{i=1}^n x_2\\\vdots\\\dfrac{1}{n}\sum_{i=1}^n x_i\end{bmatrix} = \begin{bmatrix}E(X_1)\\E(X_2)\\\vdots\\E(X_n)\end{bmatrix} = \begin{bmatrix}\bar{X_1} \\ \bar{X_2} \\ \vdots \\ \bar{X_n}\end{bmatrix}$
Answer: Mean of multivariate normal distribution is a vector of length equal to the number of variables. It cannot be scalar. Say you would use multivariate normal distribution to describe human weight, height and the average number of hours slept - single mean of those three variables would be meaningless.
Answer: If $x_{i}$ is vector-valued, $\mu$ is a vector (one mean for each entry of $x_{i}$). You may consider the scalar case as a special case in which $\mathrm{dim}(x_{i})=1$.


How to use WindowsIdentity.RunImpersonated(handle, action);

Question: We are trying to use RunImpersonated(handle, action); to be able to perform a REST call from a webserver but we have a hard time doing so. Project i ASP.NET Core 2.0 MVC.
We have the following general method made to establish a imp. context on behalf of the logged in wnd. user:
<code>var user = WindowsIdentity.GetCurrent();

IntPtr token = user.Token;
SafeAccessTokenHandle handle = new SafeAccessTokenHandle(token);

WindowsIdentity.RunImpersonated(handle, action);
</code>
and basically in the action we make our REST call.
Thing is that we CAN run through without any problems running locally on our dev machines but we can't do the same on the remote webserver. Hence: impersonation.
Is our approach above for the imp. part right since we can't actually se if we promote any user-credentials?
We have tried different techniques in the REST-GET impl. as well without the above. On the other hand the above call are made closer to our controller and on REST impl. not having any specifics for imp. itself.
Comment: Did you ever figure this out? We are working on something similar (making calls to an intranet API from within one of our intranet sites). We were getting the server's hostname (when returning Windows.Identity.User.Name as a string from our API). We implemented something similar to your code above and now we are getting 401s, but the correct username appears to be hitting the server... Anyways, we'd appreciate any info on the matter if you figured it out! Thanks!
Comment: I am also curious if there is any guidance in regards to this
Answer: I was concerned with some time ago. As far as I can remmember, this worked for me:

Create an asynchronous action filter:

<code> public class ImpersonationFilter : IAsyncActionFilter
    {
        public async Task OnActionExecutionAsync(
            ActionExecutingContext context,
            ActionExecutionDelegate next)
        {
            var user = (WindowsIdentity)context.HttpContext.User.Identity;

            await WindowsIdentity.RunImpersonated(user.AccessToken, async () =>
            {
                await next();
            });
        }
    }
</code>
Register it as any other filter.


How do we project a coordinate in 4-dimensions to that of a 3-dimensional coordinate?

Question: So essentially what I'm asking is this: so let's say that I have a set of points that I arrange into a sphere. Now, with those same points I arrange them into that of a hypersphere. How do I convert the coordinates of a point in that hypersphere into coordinates of where the point would be if the points were assembled into a 3d sphere instead of its 4d hypersphere form.
Essentially project a 4-d coordinate to a 3-d one assuming that a 3-d shape retains the amount of points it has when it is shaped into its 4-d version. Essentially a sphere is rearranged into a hypersphere with the same amount of points. How do I convert a 4-d coordinate on the hypersphere into a 3-d coordinate on the sphere.
I have no formal understanding of projective geometry and I'm a bit iffy with vectors. I hope this question is understandable enough. Thank you.
Comment: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Comment: Welcome to math.SE! <> To complement the existing answer, we can put a "point light source" at $(0, 0, 0, h)$ for some $h \neq 0$. A point $(x, y, z, w)$ with $w \neq h$ casts a shadow at $\frac{h}{h-w}(x, y, z)$. This projection gives a sense of "four-dimensional depth" so long as $h$ is larger than the fourth coordinate of each point being projected.
Answer: Represent a point in spherical coordinates:
$$\begin{pmatrix} x\\y\\z\\w \end{pmatrix}
= r \begin{pmatrix}
\sin\phi \sin\psi \sin\theta \\
\sin\phi \sin\psi \cos\theta \\
\sin\phi \cos\psi \\
\cos\phi\\
\end{pmatrix}$$
then you can project on the equator by letting $\phi=\pi/2$.  This gives you spherical coordinates in $(x,y,z)$:
$$\begin{pmatrix} x\\y\\z \end{pmatrix}
= r \begin{pmatrix}
\sin\psi \sin\theta \\
\sin\psi \cos\theta \\
\cos\psi \\
\end{pmatrix}$$
Notice however than when you are on a 3-sphere, i.e. $r=1$, then this will project to a 2-dimensional space, namely the 2-sphere.
If you want to retain more information and just project into a 3-dimensional space, you may want to use other projections like parallel projection.  The easiest such projection is to just "forget" one coordinate:
$$\begin{align}
\Bbb R^4 &\to \Bbb R^3 \\
(x,y,z,w) &\mapsto (x,y,z)
\end{align}$$
For more flexibility, take a basis of $\Bbb R^4$ that's ortho-normal, say $e_1, \dots, e_4$ and write the points $P$ uniquely as:
$$P=\sum_{i=1}^4 p_i e_i$$
where the $p_i$ are the coordinates with respect to the $e_i$.  Then do a parallel projection by taking $p_1, p_2, p_3$ as coordinates in $\Bbb R^3$, i.e. you just forget the 4th coordinate.  This is then a parallel projection to the 3d-space spanned by $e_1, e_2, e_3$.
Comment: Hi, thanks for answering. Just to clarify, is psi any arbitrary angle?
Comment: @Cease Desist: In the 3d-case and with the notation above, $\psi$ is sometimes called [colatitude](https://en.wikipedia.org/wiki/Colatitude) and $\theta$ is [azimuth](https://en.wikipedia.org/wiki/Azimuth).  If you prefer $\sin\psi\leftrightarrow\cos\psi$ then $\psi$ would be [latitude](https://en.wikipedia.org/wiki/Latitude).


Basis of span in $R^4$

Question: I have a set of 3 vectors in $\mathbb{R}^4$
$$v_1 = (1, 0, 1, 0)^T, v_2 = (2, h, 2, h)^T, v_3 = (1, 1+h, 1, 2h)^T$$
and I am to find the basis of the span of $v_1, v_2, v_3$
So far I have set it up in a matrix in the form of
$$\begin{bmatrix}
1 & 2 & 1 \\
0 & h & 1 + h \\
1 & 2 & 1 \\
0 & h & 2h
\end{bmatrix}
$$
and through Gaussian elimination I have come to the result
$$\begin{bmatrix}
1 & 2 & 1 \\
0 & h & 1 + h \\
0 & 0 & h - 1 \\
0 & 0 & 0
\end{bmatrix}
$$
Leading me to think that the Basis for this span is the set of the vectors $v_1$ and $v_2$ because of the linear dependence. Have I made some mistake here or completely misunderstood what the basis is?
Answer: You have to consider the following cases

for $h=1$ the first two columns vectors are liner independent thus $v_1$ and $v_2$ are a basis
for $h=0$ the first and the third columns vectors are liner independent thus  $v_1$ and $v_3$ are a basis
otherwise the first three columns vectors are liner independent thus $v_1$,$v_2$ and $v_3$ are a basis
Answer: It looks like you (also) made row operations, which are not allowed here in the following sense:
For example $e_1=\begin{bmatrix}1\\0\end{bmatrix}$ is a basis of $\Bbb{R}e_1$ but $\begin{bmatrix}1\\1\end{bmatrix}$ isn't a basis of $\Bbb{R}e_1$.
I would do the following:
$$\begin{bmatrix}
1 & 2 & 1 \\
0 & h & 1 + h \\
1 & 2 & 1 \\
0 & h & 2h
\end{bmatrix}
\rightarrow\begin{bmatrix}
1 & 0 & 0 \\
0 & h & 1 + h \\
1 & 0 & 0 \\
0 & h & 2h
\end{bmatrix}
\rightarrow\begin{bmatrix}
1 & 0 & 0 \\
0 & h & 1 - h \\
1 & 0 & 0 \\
0 & h & 0
\end{bmatrix}
$$
So you have the following three cases

$h=0$: The first and the third column form a basis
$h=1$: The first and the second column form a basis
Else: The three columns form a basis
Comment: Why should row operations not be allowed? They don't change linear relations between columns.
Comment: To find the basis vectors row operation are allowed. Of course, column space change thus the basis is given by the correponding initial vectors.
Comment: @egreg I added an example how I meant it. Of course column operations are allowed if one remembers not to pick the column vectors but instead the corresponding initial vectors, as gimusi said. Sorry for the misleading phrasing.
Comment: The OP clearly refers to $v_1$ and $v_2$.

Basis of span in $R^4$


Showing two colorbox windows side by side...or within one main frame?

Question: I have a site that I've created for some fellow musicians to learn new songs. Generally we have a link to a Youtube video of a song then a link that opens up the tab (chords, lyrics, etc) for the song. Right now I have links that open those things up individually in colorboxes. Works fine.
What I'd like to do is have one colorbox, or frame, or iframe, etc -- that opens up and in the left side I have the Youtube video and on the right side the tab colorbox ... so as I listen to the song I can scroll down at the same time and look at the tab. Here is a link to the site: http://practice.3cnorth.com/?q=all-songs
Thanks for any ideas.
ER
Comment: You can put as many information as you want in a node and display it in a colorbox popup, all together. You just need a module like https://www.drupal.org/project/colorbox_node
Answer: One simple approach would be to just create a content type that includes both items. For instance, "songs". Then, you could:

Embed each youtube video within their corresponding tabs node.
Then, you could use a module such as Display Suite to display one next to each other (E.g. video on the left column and tabs on the right column).
After this, if you want the tabs to be "scrollable", just set a fixed height in CSS and a "scrolled" overflow. E.g.
<code>.views-field-body pre {
    height: 300px;
    overflow-y: scroll;
}
</code>

That should do the trick, the colorbox will open one node, instead of 2 as currently.


Notation and terminology of definite integrals

Question: I am currently undertaking Calculus and have stumbled upon an issue regarding notation and terminology, that I can not seem to understand.
"Given the definition of the definite integral:
If f is a function on the interval [a,b] and we divide the interval into n subintervals of equal width where $x_i^*$ is the $i^{th}$ interval then the definite interval of f from a to b is:
$\int^b_a f(x)dx $ = $\lim\limits_{n\to\infty} \sum_{i=1}^n f(x_i^*)\Delta(x)$"
Now, my understanding of the definite integral is that it is the limit of the Riemann sum. However, I was doing a question in my lecture and noticed that my professor had a question that asked us to "evaluate the following definite integral:
$\int^2_0 (x^2+3x) dx$"
Calculating it is not an issue but the lead me to question what we actually consider the definite integral. Is it the limit of the Riemann sum as I mentioned or is it the notation that we use, namely $\int^b_a f(x) dx$?
Is it perhaps a clumsy way of wording a question or am I missing something?
I look forward to your responses!
Thank you in advance
Answer: You should think of
$$
\int^2_0 (x^2+3x) dx
\tag1$$
as
$$
\int_a^b f(x)\;dx
$$
where $a=0, b=2$ and $f$ is the function described by $f(x) = x^2+3x$.
Yes, $(1)$ is the limit of Riemann sums, where of course $f(x_i^* )$ is
$$
(x_i^*)^2+3x_i^*
$$
Comment: Oh ok. So even though that is what we use to denote a definite integral, we can still think of it as being the definite integral for the reasons you explained. Is this correct?
Answer:
...[this] lead me to question [of] what we actually consider [to be] the definite integral. Is it the limit of the Riemann sum as I mentioned or is it the notation that we use, namely $\displaystyle \int_{a}^{b}f(x) dx$?

I believe your confusion stems from how you are reading $$\displaystyle \int^b_a f(x)dx  = \lim\limits_{n\to\infty} \sum_{i=1}^n f(x_i^*)\Delta(x).$$ In particular, it might be better to look at the right hand side of the equals sign $$\lim\limits_{n\to\infty} \sum_{i=1}^n f(x_i^*)\Delta(x)\tag{$1$}$$ as the definition of  $$\displaystyle \int_{a}^{b}f(x) dx\tag{$2$}.$$ We use the notation $(2)$ in practice to calculate definite integrals instead of working with $(1)$ as it is much cleaner and convenient. Better yet, we have a set of rules on how to calculate definite integrals using $(2)$. In that respect if someone asked me to evaluate the definite integral $$\displaystyle \int_{0}^{2}(x^{2} + 3x)dx$$ I would use $(2)$ to do so.
Comment: Ah yes. That clears it up. Thank you so much. In Australia, we have a tendency to use the word "is" for an equality and when my lecturer kept saying that the symbol is the definite integral, I overlooked the fact that it meant equal.
Comment: @Oofy2000 - You're welcome, happy to help anytime!


Does Naive Bayes assume normality?

Question: I came across this paper about Naive Bayes that states

[Naive Bayes] is based on another common simplifying assumption: the values of numeric attributes are normally distributed within each class.

Is that true? Does Naive Bayes require that you assume continuous predictors are normal?
From this answer and my experience, I thought that you can use any distribution to describe your predictors, normal or otherwise. Am I missing something? Is the paper just badly worded, e.g. it should say "is commonly based on a normality assumption" instead of "is based on a normality assumption?"
Answer: On its own, Naive Bayes does not assume the normal distribution. The heart of Naive Bayes is the heroic conditional independence assumption:
$$P(x \mid X, C) = P(x \mid C)$$
Gaussian Naive Bayes assumes the normal distribution...
Comment: Is the paper badly worded / incorrect then? Do they just mean "Naive Bayes *often* assumes normality?"
Comment: @Qroid Without reading anything (so I could be wrong), my guess is that they mean Gaussian Naive Bayes. Often times you need to read text reasonably rather than explicitly.
Comment: Thanks. Didn't mean to nit pick. Just wondered if I understood correctly.
Comment: It perform well in case of categorical input variables compared to numerical variable(s). For numerical variable, normal distribution is assumed (bell curve, which is a strong assumption).

Does Naive Bayes assume normality?


Can results for model selection with AIC be interpretable at the population level?

Question: AIC results for model selection are dependent on the sample size. For example if I make this model with a sample size n=100:
set.seed(123)
N <- 1000
n <- 100
r2 <- .01
y <- rnorm(n)
AIC.mod0 <- AIC(glm(y ~ 1))

> AIC.mod0
[1] 268.5385

BIC.mod0 <- BIC(glm(y ~ 1))

> BIC.mod0
[1] 273.7488

res <- replicate(N, {
  x <- y * sqrt(r2) + rnorm(n, sd=sqrt(1-r2))
  list(aic = AIC(glm(y ~ x)),
       adj.r2 = summary(lm(y ~ x))$adj.r.squared)
}, simplify=F)

> summary(sapply(res, "[[", "adj.r2"))
      Min.    1st Qu.     Median       Mean    3rd Qu.       Max.
-0.0102041 -0.0076633 -0.0009994  0.0077916  0.0162644  0.1411315

> summary(sapply(res, "[[", "aic"))
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  254.3   267.9   269.6   268.7   270.3   270.5

> summary(sapply(res, "[[", "bic"))
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  262.1   275.7   277.4   276.5   278.1   278.4

We can see that adjusted $R^2$ is a bit over-corrected but is nevertheless positive. AIC is higher than in the null model so it suggests that adding the variable x as a predictor does not improve the model (the same for BIC).
But if I make the same experiment with a sample size n=1000:
set.seed(123)
N <- 1000
n <- 1000
r2 <- .01
y <- rnorm(n)
AIC.mod0 <- AIC(glm(y ~ 1))

> AIC.mod0
[1] 2824.197

BIC.mod0 <- BIC(glm(y ~ 1))

> BIC.mod0
[1] 2834.013

res <- replicate(N, {
  x <- y * sqrt(r2) + rnorm(n, sd=sqrt(1-r2))
  list(aic = AIC(glm(y ~ x)),
       bic = BIC(glm(y ~ x)),
       adj.r2 = summary(lm(y ~ x))$adj.r.squared)
}, simplify=F)

> summary(sapply(res, "[[", "adj.r2"))
     Min.   1st Qu.    Median      Mean   3rd Qu.      Max.
-0.001002  0.005103  0.008727  0.009737  0.013583  0.036300

> summary(sapply(res, "[[", "aic"))
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
   2788    2812    2816    2815    2820    2826

> summary(sapply(res, "[[", "bic"))
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
   2803    2826    2831    2830    2835    2841

Adjusted $R^2$ doesn't move a lot in mean but full model AIC is now clearly lower than null model AIC, suggesting that adding the variable x improves the model. We have the same issue for BIC.
So my question could be: is it correct to reject a more complex model based on AIC knowing that a simple sample size increase could make the more complex model more efficient?
Answer: I am not sure if the title and the body of your question are asking about the same thing, but I will try to answer in a general way to cover both.
\begin{aligned}
\text{AIC}&=-2(\text{loglik}-p) \\
&=-2(n\times \text{avg(loglik)}-p)
\end{aligned}
where

$n$ is the sample size used for estimation,
$p$ is the number of the model's degrees of freedom (a measure of model's flexibility),
$\text{loglik}$ is the log-likelihood of the sample data according to the model and
$\text{avg(loglik)}$ is the average log-likelihood per observation.

AIC allows assessing the expected log-likelihood of the model on a new data point based on a given estimation sample. (Actually, the estimate for a single new data point is $\frac{\text{AIC}}{-2n}$.) It does so by weighting fit ($\text{loglik}$) against model flexibility ($p$). As the sample size $n$ increases, average per-observation fit ($\text{avg(loglik)}$) should improve because the estimation variance should shrink. Moreover, the weight of fit ($n$) in determining the value of AIC would grow since the log-likelihood is summed over the observations. Both of these developments point in the same direction. Consequently, AIC would prefer more complex models in larger samples.
When the estimation sample is an entire finite population, there can be no new data points to be predicted, so model choice based on AIC (motivated by achieving high likelihood on a new data point) is not directly relevant.
In an infinite population, the estimation variance would be zero and only model bias (and - when it comes to forecasting - irreducible error) would be left. The likelihood would be summed over an infinite number of points ($n=\infty$), dominating the complexity penalty ($p$). Hence, the most flexible model would be chosen by AIC. (Stricly speaking, AIC might not be defined on an infinite sample, but we can still ponder how it would behave in the limit.) As such, we would not need AIC for model selection: pure model fit (the likelihood) would be enough.
Given the considerations above, I think AIC-based model selection is hard to interpret on a population level.
Comment: Thanks for the clear explanations. To take an example, in the article https://www.sciencedirect.com/science/article/pii/S2212977414000064, they used AIC to determine which variables are of interest to predict wine rating. The variables that were not part of their best model are probably less useful than the selected ones, but they could be found of interest in a larger data sample. This mean that we should re-run the procedure of model selection when building a new predictor based on other data.
Comment: @JeanPaul, thank you. Your takeaway from the article makes sense.