Modal theory and resolution

Question: Does each mode resolved to their respective roots as modes are to be seen as nothing more than another degree of the same scale/key. Or do they always resolve to the root note of the respective major or minor key? For example, If i begin with the 4th scale degree, the lydian mode of C major would the melody resolve to F or back to C?
Answer: F would be considered the "root" or tonal center, if you call the piece Lydian.  If it resolves to C then it would be an Ionian work.
Answer: Most would tend to gravitate towards the modal centre. Dorian, Lydian and Mixolydian certainly do, and obviously Aeolian does,as borne out by many works in natural minor, although sometimes helped by that raised leading note!
Locrian is the sticky one, but there aren't many examples of it, funnily enough. Having said all that, I often feel, in a modal work, that it will eventually end in the mother key (Ionian), but it needs help to stay modal by clever use of cadences and harmony.
Comment: How is Ionian the "mother key"?
Comment: I think he's referring to the Ionian mode of that pitch collection. So for D Dorian, E Phrygian, etc., C Ionian would be the "mother key" because it's the same pitch collection.
Comment: @ScottWallace - yes, Richard's correct. Not sure if there's a more apposite or even correct name for it, but all the notes from D Dorian, G Mixolydian, etc. are 'from C Ionian - the major scale notes'. Hence in some pieces, feel as though the key *could* be that of the mother key.
Comment: I understand this terminology from an historical aspect.  But I don't feel that  pieces in modes other than Ionian tend to end in Ionian.  Perhaps that's because I play a lot of modal music myself.
Comment: @ScottWallace - I didn't say they tend to end in Ionian, just that they feel (to me at least) that they *could*, due to the above reason.

Modal theory and resolution


Finding the interval of convergence using the ratio test

Question: I found that the Taylor series for $h(t)=\ln(t)$ centered at $t_0=1$ is
$$\sum^\infty_{n=0}(-1)^n\frac{1}{n+1}(t-1)^{n+1}$$
Now I want to find the interval of confidence for this series, so I use the ratio test:
\begin{align}
&\lim_{n\rightarrow\infty}\left|\frac{(-1)^{n+1}\frac{1}{n+2}(t-1)^{n+2}}{(-1)^n\frac{1}{n+1}(t-1)^{n+1}}\right|\\
=&|t-1|\lim_{n\rightarrow\infty}\left|(-1)\frac{n+1}{n+2}\right|\\
=&|t-1|\lim_{n\rightarrow\infty}|-1|\\
=&|t-1|=L
\end{align}
The series converges when $L<1$, so we get the interval $0<t<2$. However, this is incorrect. I was told that the correct interval is $0<t\leq2$.
Why is the interval inclusive of $2$?
Answer: The series converges for $|t-1| <1$ and diverges for $|t-1| >1$. For $|t-1|=1$ you have to check separately for convergence. When $t=0$ the series is divergent by comparison with the harmonic series. For $t=2$ it is convergent by the Alternating Series Test.
In general  the ratio test does not say anything about the convergence of $\sum a_n (t-c)^{n}$ when $|t-c|=L$ where $L$ is the number you get from the Ratio Test.


Limit Terminology

Question: From the $\epsilon-\delta$ definition of a limit, we can see that any limit can be broken up into two "one-sided limits". These "one-sided" limits are simple cases that arise as a consequence of the absolute value of $|x-a|$ in the formal definition of a limit.
Stated Mathematically :$$\left(\lim_{x \ \to\  a} f(x) = L\right) \Leftrightarrow \left(\lim_{x \ \to \ a^+} f(x) = L\right) \land \left(\lim_{x \ \to \ a^-} f(x) = L\right)$$
Defined Rigorously Using $\epsilon-\delta$ :
$$\left(\forall \epsilon > 0\  (\exists\ \delta > 0 : 0 <|x-a|<\delta \implies |f(x)-L|<\epsilon)\right) \Leftrightarrow \left[(\forall \epsilon > 0\  (\exists\ \delta > 0 : a <x<a +\delta \implies |f(x)-L|<\epsilon) \land \ (\forall \epsilon > 0\  (\exists\ \delta > 0 : a-\delta <x<a \implies |f(x)-L|<\epsilon)\right] $$

Is there better terminology to distinguish between limits?
But is this what the limits are formally called? Are "one-sided" limits always called/referred to as "one-sided" limits (i.e. as the "left-side limit" or "right-side limit") even in higher mathematics? Surely the concept of "one-sidedness" gets thrown out the window once you move past Real Analysis, as the only reason we can make this, ultimately false, visual analogy is due to properties of the Real Number Line?
And what do you call the original limit then, that can be broken up into two "one-sided" limits? Do you just refer to it as the "Limit as $x$ approaches $a$ of $f(x)$", the whole time? Furthermore is a one-sided limit always referred to as : "Limit as $x$ approaches $a$ from the left/right of $f(x)$"
Is the convention in terminology in higher mathematics kept constant, especially when talking about limits, throughout various areas in higher mathematics, for example going from Mathematical Analysis (Real, Complex, Functional etc.) to Topology etc?
I realize that all of this can be easily solved by writing out what is trying to be said Mathematically (as I did above, which is what I prefer to do), but let's assume for the pure terminology purposes of this question that we can't. What is the proper terminology to use in that case to refer to the different types of limits?
Comment: Very similar to a [question](http://math.stackexchange.com/questions/1760959/rigorous-definition-of-one-sided-limits) you asked only an hour ago (with much of it copy and pasted).
Comment: @Daniel In that question, I was asking about a rigorous *Mathematical definition*, of one-sided limits. Here I am asking a question purely on terminology, without any notation involved They are similar, but not the same. The only part that was copy-pasted was the LaTeX code for the opening paragraph of the post.
Comment: In higher mathematics you consider limits in spaces different than the line... In such spaces, the concept of "left/right sided" limit makes no sense.
Comment: The "left-side" and "right-side" limit are also referred to as the "limit from below" and "limit from above", respectively. One speaks of the limit "as $x$ approaches $a$ from below". Maybe not helpful, and probably you knew that already. I've not encountered limits outside of $\mathbb{R}$ yet. Sorry!
Answer: In Mathematica, Limit has an option called Direction, and "Direction->Exp[iθ] at x* indicates the direction tangent of a curve approaching the limit point x*."

Limit Terminology


Prove that the maximal eigenvalue is less than $1$

Question: Given a series of column vectors: $ x_{i} =(1,0,0,\dots,c_{i},0,0,\dots,0)\in\mathbb{R}^{N+1} $ where $ c_{i}\neq 0 $ and $ c_{i} $ is on the $ (i+1) $th coordinate of vector $ x_{i} $, $ i=1,2,...,N $. Specially, define $ x_{0} =(1,0,0,...,0)\in\mathbb{R}^{N+1} $. Define the matrix
$$ V := \sum_{i=0}^{N}x_{i}x_{i}^{\top} $$
I want to verify whether
$$ \lambda_{\max} \left( V^{-1} x_{i} x_{i}^{\top} \right) \leq 1 \, \quad\forall i=0,1,\dots,N $$
where $ \lambda_{\max}(\cdot) $ means the maximal eigenvalue of a given matrix.
For the $2$-dimensional case, it can be verified that the eigenvalues are either 0 or $ \pm 1 $. I tried some 3-dimensional cases numerically and found the eigenvalues are also either 0 or $ \pm 1 $.
Comment: Same uppercase for matrices and integers is confusing.
Comment: **Hint:** the nonzero eigenvalue of a rank-1 matrix is its trace.
Comment: @RodrigodeAzevedo Yes, I see. But here maybe I need further consider that whehter the matrix $ A_{i} $ as defined by Jean Marie is eigenvalue decomposable. If $ A_{i} $ is only singular value decomposable, knowing that the kernel is in $ N $ dimensional will still give that there are zero $ N $ singular values?
Comment: What is the rank of $A B$ if $B$ is rank-$1$?
Answer: Hint: matrix
$$A_i:=V^{-1}x_{i}x_{i}^{\top}$$
is such that $A_i u=0$ for all vectors $u$ orthogonal to $x_i \ne 0$ (a hyperplane).
Therefore, the kernel of $A_i$ in $N$ dimensional.
As a consequence all eigenvalues of $A_i$ are $0$ but one.
Then it remains to show that this eigenvalue is less than $1$.
Comment: Sorry, it is still hard for me to understand this hint. It seems that the hint does not use any information on the definition of $ x_{i} $. Btw, the $ n-1 $ is based on the implicit fact that $x_{i} $ is of dimension $ n $, right?
Comment: It is true that the fact that your $x_i$ have their specific form is not used. The only thing uses is that they aren't $0$. The $n-1$ is the dimension of a hyperplane, yes, in $\mathbb{R^n}$.
Comment: I can understand that the kernel of $ A_{i} $ is $ N $ dimensional. But how to get that the eigenvalues of $ A_{i} $ are N zeros but one? The proof of the non-zero eigenvalue should use the information in $ x_{i} $, right?
Comment: First of all: I hadn't paid attention that the ambient space is $\mathbb{R^{N+1}}$. You have understood that $n=N+1$.  Then: the fact that the kernel is N-dimensional gives you a N-dimensional basis $(u_1,... u_N)$ of the kernel, and all these are such that $A_iu_k=0 \ \iff \ A_iu_k=0u_k$, a fact that means that every $u_k$ is associated with a zero eigenvalue. Therefore eigenvalue zero has multiplicity $N$.
Comment: Thank you very much. Having known that there must exist $N$ zero eigenvalues is very helpful. I can use trace to get some information on the only one non-zero eigenvalue. Btw, given that $A_{i}$ may not be symmetric, is it possible to show that $A_{i}$ always has a eigenvalue decomposition?
Comment: $A_i$ is indeed diagonalizable. You know that the criteria for being diagonalizable is that the dimension of every eigenspace is equal to the order of multiplicity of the associated eigenvalue : this is true for eigenvalue $0$ (dimension $N$ = order of multiplicity $N$). Therefore, as it remains only one dimension for the nonzero eigenvalue, the criteria is completely fulfilled.


Interpreting 1-sample t-test results

Question: I have a question around interpretation of results of 1-sample t-test. I am using the test (with SciPy.stats.ttest_1samp function in Python) to check if the existing goal set for a productivity metric is correct. As I understand, for 1-sample t-test:
a) H0: population mean equals the specified mean value (the metric goal in this case)
b) Ha: population mean is different from the specified mean value (metric goal)
I encountered two different explanations for p-value given by the t-test. One says if p-value<alpha, then we reject H0 (https://www.graphpad.com/quickcalcs/oneSampleT1/,  and  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_1samp.html) and another which says that we fail to reject H0 (https://www.analyticsvidhya.com/blog/2019/05/statistics-t-test-introduction-r-implementation/)
What is the correct approach to follow?
Comment: useful [When to use Fisher and Neyman-Pearson framework?](https://stats.stackexchange.com/questions/23142/when-to-use-fisher-and-neyman-pearson-framework)
Comment: Your analyticsvidhya.com link does not say we fail to reject $H_0$ when $p \lt \alpha$ but when $p \gt \alpha$.  For example *"We can also verify this from the p-value, which is greater than 0.05. Therefore, we fail to reject the null hypothesis at a 95% confidence interval."*
Answer:
I encountered two different explanations for p-value given by the t-test. One says if p-value<alpha, then we reject H0 (https://www.graphpad.com/quickcalcs/oneSampleT1/,  and  https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_1samp.html) and another which says that we fail to reject H0 (https://www.analyticsvidhya.com/blog/2019/05/statistics-t-test-introduction-r-implementation/)

The difference is due to whether you compare the p-value or the t-statistic.

You have either $$\text{p-value} \leq \alpha \quad \iff \quad t \geq t_\text{critical}$$ and the hypothesis becomes rejected.
Or you have $$\text{p-value} > \alpha \quad \iff \quad t < t_\text{critical}$$  and the hypothesis becomes not rejected.

The two different explanations are not contradictory and you may have been confusing the uses of t-statistic, p-value, and the related inequality signs.
The first explanation uses the situation $\text{p-value} \leq \alpha$. The second uses the situation $t < t_\text{critical}$. So they refer to two different cases. The sign happens to be the same in the two explanations but the one explanation relates to the p-value and the other to the t-statistic.
Related: If p-values are random, why not decide using the test statistic instead?

Interpreting 1-sample t-test results


Postgresql fails on startup: "FATAL: postmaster became multithreaded during startup"

Question: Trying to start PostgreSQL, but get the following error:
$ pg_ctl -D postgres_data start
server starting
FATAL:  postmaster became multithreaded during startup
HINT:  Set the LC_ALL environment variable to a valid locale

My locale is set:
LANG="en_US.UTF-8"
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL="C"

System is MacOS 10.10.5, PostgreSQL version is 9.5.0.
I followed Michael's suggestion.  Now the locale values are:
$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL="en_US.UTF-8"

But the error remains.
I Googled the error message to see if anyone had run into this problem, but did not see a solution.
Answer: Your locale is mis-set.
You have LANG defined as en_US.UTF-8, but everything else is defined as C. These are rather different, and mixing them will cause all sorts of weirdness.
Do what the error message told you: set LC_ALL to the correct locale (and set all the rest of them too).
Comment: I know little about locales. Is the correct locale "en_US.UTF-8"?  Note that I didn't explicitly set any of those variables - that is what MacOS did.  It has a "Language and Region" system preference which apparently does not set the LC_* stuff.
Comment: That depends. Do you speak English and are in the US?
Comment: Yes - the LANG is correct.
Comment: When I run `locale` on El Capitan, they are all set correctly. If you need help with setting these, you should do so elsewhere, such as [su] or [apple.se]. That really isn't on topic here.
Answer: I have resolved the issue for My Mac running Postgres 9.5 and 9.6 by setting LC_ALL for the 'postgres' account.
See details on https://github.com/PostgresApp/PostgresApp/issues/317#issuecomment-251786830
Comment: Where is the 'postgres' account located?


split,operate and union dataframe in spark

Question: How can we split a dataframe and operate on individual split and union all the individual dataframes results back ?
Lets say i have dataframe with below columns. I need to split the dataframe based on channel and operate on individual splits which adds new column called bucket. then i need to union  back the results.
account,channel,number_of_views
The groupBy is only allowing simple aggreagted operation. On each splitted dataframe i need to do feature extraction.
currently all Feature Transformers of spark-mllib are support only single dataframe.
Comment: Can you show your code? Of course you can make an individual dataframe from a selection of columns from another
Comment: what you want in result
Answer: you can randomly split like this
val Array(training_data, validat_data, test_data) = raw_data_rating_before_spilt.randomSplit(Array(0.6,0.2,0.2))

this will create 3 df then d what you want to do then you can join or union
val finalDF = df1.join(df2, df1.col("col_name")===df2.col("col_name"))

you can also join multiple df at the same time.
this is what you want or anything else.??
Comment: Thanks For response.  Once i split dataframe how can i run a operation on each splitted dataframe on parallel fashion . I have a function which would add new column to each splitted dataframe .  I dont want to loop on arry and execute it.


Image not responsive in one content type. Can anyone help me resolve this?

Question: I'm hoping someone might be able to help me. Yesterday, i noticed that the main image on my photo gallery content stopped behaving responsively. For example: http://thesalinepost.com/photo-gallery/saline-sweeps-tecumseh-home-opening-doubleheader Now, the same image style is working correctly on article content http://thesalinepost.com/article/saline-rowers-kick-spring-season-spring-lake-michigan
And on it's working on announcement and events content.
Answer: Because you a style rule somewhere...
<code>.node-openpublish-photo-gallery .field-name-field-op-main-image {
   width: 620px;
}
</code>
They appear to be different content types...
node-openpublish-photo-gallery (not working)
node-openpublish-article (working)
Comment: THank you. I've searched CSS files can can't seem to find this rule.


onEdit trigger does not work for form edits, is there an alternative?

Question: I am trying to update another google sheet (not the one the form adds info to) when google form is submitted. I tried this with onEdit() but found out that it does not trigger for form edits.
Is there an alternative that triggers on manual edit AND form edit for google sheets?
Comment: You can read about the onFormSubmit event object [here](https://developers.google.com/apps-script/guides/triggers/events#form-submit)
Comment: You might want to take a look at [this](https://stackoverflow.com/questions/54834837/how-can-i-be-getting-multiple-unwanted-event-blocks-from-the-same-onformsubmit-t) in the event that you have any problems with unwanted triggers.
Answer: if you want your function to happen from both form submits and regular edits, add a function
<code>onSubmit(){
   onEdit();
}
</code>


Python: using either command line or import

Question: def nearbin():
    parser = argparse.ArgumentParser(description = "choose near bin")
    parser.add_argument("-bin",  type=int,action = "store", default = "10", help="bin number")
    args = parser.parse_args()
    bin = args.bin
    print bin
    return f(bin)

if __name__ == "__main__":
    nearbin()

In command line, I can use "python test.py -bin 10" to run the program
If I want to import test in another python script and use function nearbin(bin) in another script, how can I do this?
Answer: My usual approach:
def nearbin(bin):
    ...

def main(args):
    parser = argparse.ArgumentParser(...)
    ...
    args = parser.parse_args(args)

    nearbin(args.bin)

if __name__ == '__main__':
    sys.exit(main(sys.argv[1:]))

This lets you support direct command-line invocation (./script.py -bin 20), invocation with command line args (main("-bin 20".split())), and programmatic invocation (nearbin(20)).
Answer: Then you should parse the command line arguments inside the __main__ check and pass them to nearbin, something like this:
def nearbin(bin):
    print bin
    return f(bin)

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description = "choose near bin")
    parser.add_argument("-bin",  type=int,action = "store", default = "10", help="bin number")
    args = parser.parse_args()
    bin = args.bin
    nearbin(bin)


Let S be a finite linearly independent set of vectors in V, and let T be a subset of S. Show that T is a linearly independent set of vectors in V.

Question: I don't know where to start on this proof. I know that a linearly independent set is a set where c1v1+c2v2+⋯+cnvn=0 has no non-trivial solutions other than c1=c2=...=cn=0.
How can I use that to figure out this proof?
Comment: Welcome to Maths SX! Just say that he who can do the more can do the less.
Answer: Take any linear combination of elements of $T$ which turns out to be zero. Add rest of the elements of $S-T$ with coefficient 0. Now can you complete it?


Mutator funciona no tinker e não funciona na view do blade no Laravel 7.*

Question: Galera, eu consegui executar um erro meio bizarro aqui no Laravel.
Eu defini um método accessor em um modelo, testei no tinker e tudo certo, a sintaxe tá certa e funciona OK.
Porém, quando eu tento acessar esse mesmo atributo em uma blade view, eu só recebo erro atrás de erro, não consigo obter o resultado esperado!
E o mais curioso é que na blade, se eu executo um dd() no atributo, a informação que eu quero está lá!
Holder.php
public function getLastEarningAttribute()
{
    return $this->uuids->last()->earnings->last()->value;
}

home.blade.php
@foreach($holders as $holder)
{{ $holder->last_earning }}
@endforeach

A variável $holders está sendo passada normalmente pelo Controller:
public function index()
{
    $holders = Holder::all();
    return view("dashboard.holders.home", compact("holders"));
}

E mesmo assim, eu não consigo sair do erro:
ErrorException
Trying to get property 'earnings' of non-object (View: ***\resources\views\dashboard\holders\home.blade.php)

E executando o dd($holder->last_earning) no home.blade.php dentro do @foreach($holders as $holder), o resultado que eu quero é apresentado normalmente: 27.04. Assim como quando eu executo o mesmo atributo no tinker, o resultado esperado também vem.
>>> Holder::find(1)->last_earning
[!] Aliasing 'Holder' to 'App\Holder' for this Tinker session.
=> 27.04

Os relacionamentos dentro de Uuid::class e Earning::class estão todos corretos e funcionando normalmente em suas respectivas views.
O que eu estou fazendo de errado?
Answer: Acredito que em um dos holders, este trecho  $this->uuids->last() retorna null.
Seu dd() provavelmente funciona porque não deve ser o caso do primeiro registro; entretanto, algum deles deve retornar null nessa parte do código que mencionei acima e como está num foreach, quando passar por este registro vai acontecer o erro que está acontecendo.
Acho que a solução seria validar no próprio mutator se last() retorna null, seja com if ou se estiver usando o php 8, você pode usar o null safe operator.
Comment: Cara, falta de sono adequado, fiquei até 2h tentando resolver isso, tinha esquecido que uuids podia ser nulo. Muito obrigado, resolveu meu problema!


Getting error Missing required parameter for [Route: our-board.member] [URI: our-board/member/{boardMember}]

Question: I have using Laravel 10.x, I have installed Laravel Splade for SPA.
Now, I am getting an error while I am implementing Route Module Binding

web.php file
Route::group(['prefix' => 'our-board'], function() {
    Route::get('/', [HomeController::class, 'ourBoard'])->name('our-board');
    Route::get('member/{boardMember}', [HomeController::class, 'memberDetail'])->name('our-board.member');
});

in Controller
public function ourBoard(): View
{
    $data['boardMembers'] = BoardMember::select([
        'name', 'image'
    ])->get();

    return view('company.our-board', $data);
}

public function memberDetail( BoardMember $boardMember ): View
{
    $data['boardMember'] = $boardMember;
    return view('company.our-board.member', $data);
}

In view.
<x-nav-link href="{{ route('our-board.member', ['boardMember' => $boardMember]) }}" class="flex flex-column align-items-center text-decoration-none text-white">

{{ $boardMember->name }}

</x-nav-link>

I am missing something? any quick suggestion will be appreciated.
Comment: Try clearing php artisan cache:clear ,php artisan config:Cache  and then get php artisan route:list so you  can get your route list with expected parameters
Comment: [Why should I not upload images of code/data/errors?](https://meta.stackoverflow.com/questions/285551/why-should-i-not-upload-images-of-code-data-errors)
Comment: Your code looks good, Check if `$boardMember` has some value when you use it in the view
Comment: Yes, I found the issue. I forgot to add `id` in the select. :(
Answer: I have not added id in the selection.
$data['boardMembers'] = BoardMember::select([
    'id', 'name', 'image'
])->get();

If we do not add id, then the modal will bind.


Normal Bundle is a manifold in Pollack

Question: I am reading a proof in Pollack on page 71 and in one of the steps in the proof that $N(Y) = \{ (y,v) \in Y \times N_y(Y) \}$ is a manifold, he claims that
$N(U) = N(Y) \cap (U \times \mathbb{R}^M)$ where $U = \tilde{U} \cap Y = \phi^{-1}(0)$. Here $\phi: \tilde{U} \to \mathbb{R}^{codim Y}$ is a submersion and from what I understood, he is defining $Y$ to be the preimage of $0$ of some function and $\phi^{-1}$ just happens to map $0$ to an open set in $Y$.
Now what I don't understand here is the equality $$N(U) = N(Y) \cap (U \times \mathbb{R}^M)$$
I understand the inclusion $\subset $ but not the other.
Comment: You should add some more context. I believe $Y$ is embedded in $\mathbf R^M$ to begin with. Further,  $N_y(Y)$ is the orthogonal complement of the tangent space of $Y$ at $y$, where the latter has been identified as a subspace of $\mathbf R^M$.
Comment: @caffeinemachine, you are correct sorry. Also in this definition of the normal space, are we equipping the space with an inner product? There is no mention of it, but is it there?
Comment: That's the thing, in Guillemin and Pollack, all manifold's sit inside Euclidean spaces to begin with. So the tangent spaces are naturally thought of as subspaces of $\mathbf R^n$. Then using the standard inner product of $\mathbf R^n$, we can define the normal spaces.
Answer: Both sets are $\{(u,v)|u\in U \text{ and } v\in N_u(Y) \}$.
Comment: Well that is my issue, I don't see that. I only see $\subset$. I mean $N(U) = \{ (u,v) \in U \times N_u(U) \}$.
Comment: Are we saying $N_u(Y) = N_u(U)$? Is there an obvious explanation that I am not seeing?
Comment: Yes. I think we are assuming $\tilde{U}$ is open, so $U$ is automatically open in $Y$. So $N_u(Y)=N_u(U)$.
Comment: Yes $\tilde(U)$ is open by definition and hence $U$ is open in $Y$. But why does that make the normal space the same...? Sorry if this sounds stupid
Comment: Do you know why the tangent spaces are the same?
Comment: From what I understand we are taking the tangent space at the neighbhood of the point. Taking the image, this is the $T_x(U)$ and since our chart $\psi$ maps the neighbhood inside $U \subset X$, we have the first inclusion. But relabelling $U$ by $X$ we are still mapping the neighbhood into $U \subset X$. I guess that's why. But now I am confused about the next step as it claims $\ker (d\phi_y) = T_y(Y)$

Normal Bundle is a manifold in Pollack


Network stop working after adding bridge interface

Question: I'm using Fedora 25, with kernel 4.11.9, kvm and bridge-utils are installed. I'm following the instructions on http://www.linux-kvm.org/page/Networking#Public_Bridge, to get public bridge. I also have Docker installed on my machine (which creates its own bridges and virtual interfaces).
When I run ip link add br0 type bridge, everything is ok. But after that, when I run ip link set eth0 master br0 my network stop working. The interface eth0 is the interface I use to access the local network and exit to the internet.
What could be going wrong? How can I troubleshoot this?
Comment: you have to do the ip setup on br0, not eth0. you need to clear ipv4 settings from eth0 beforehand, and set br0 to use that config instead.
Comment: Worked like a charm. I've used `ip addr del 192.168.64.230/24 dev eth0` and `ip addr flush dev eth0` to clear my eth0, repeated the steps and got ir working. Could you please post your comment as the answer so that I can mark it as correct?
Answer: When you add your ethernet interface to your bridge, you need to move your ipv4 settings from your eth0 to br0 as well. So, first clear eth0 settings and then give your ipv4/6 addresses/gateway/etc to your br0.
Personally I find it convenient to use NetworkManager's GUI (or TUI) to do this task.
Answer: This how-to worked for me (using NetworkManager CLI):
https://gist.github.com/plembo/f7abd2d9b6f76e7afdece02dae7e5097
The steps would be, as root:
> nmcli con show
Ethernet 1  e923bc3e-d77f-4f68-8b33-eaa9955f437d  ethernet  eth0

Identify the ethernet connection and the physical device. From now and as an example, they were "Ethernet 1" and "eth0" respectively.
Now create a bridge network:
> nmcli con add ifname br0 type bridge con-name 'Bridge 1'

And add the physical device as slave for this new connection:
> nmcli con add type bridge-slave ifname eth0 master 'Bridge 1'

As now, the thing would show as this:
> nmcli con show
Ethernet 1         e923bc3e-d77f-4f68-8b33-eaa9955f437d  ethernet  eth0
Bridge 1           c2a4218f-70b8-488e-83a6-49530ad79d7c  bridge    br0
bridge-slave-eth0  92fab279-110a-48ec-9e55-f240353f4fb2  ethernet  --

If you use DHCP:
> nmcli con mod 'Bridge 1' ipv4.method auto

If not,
> nmcli con mod 'Bridge 1' ipv4.method manual
> nmcli con mod 'Bridge 1' ipv4.addresses 10.1.1.16/24
> nmcli con mod 'Bridge 1' ipv4.gateway 10.1.1.1
> nmcli con mod 'Bridge 1' ipv4.dns '10.1.1.1,126.96.36.199,8.8.4'

Also, STP is not needed, so disable it:
> nmcli con mod 'Bridge 1' bridge.stp no

Now turn down the ethernet connection and turn up the bridge:
> nmcli con down 'Ethernet 1'
> nmcli con up 'Bridge 1'

For now, when you restart the computer, the Ethernet 1 connection keeps binding to the eth0 device, but we don't want this. So that the bridge slave can bind to the physical device, this last have to be free, so you have to disable the autoconnect flag of the ethernet connection:
> nmcli con mod 'Ethernet 1' connection.autoconnect no

Additional steps:

In addition, you could need to open the NetworkManager GUI, and edit the bridge-slave-eth0 connection to restrict it to the eth0 physical device.
At this point, the eth0 physical device could be locked to the 'Ethernet 1' connection, and you may need to restart.


How to redirect programmatically inside a module to an external url?

Question: I am trying to redirect to a page after I unset a session.
I read this article but I am pretty new to Drupal module building.
Here this amazing guy says to do the comming thing:

If you know that the destination is an external URL then you are
  encouraged to use TrustedRedirectResponse instead:
return new RedirectResponse(\Drupal::url('<front>', [], ['absolute' => TRUE]));
  return new TrustedRedirectResponse('http://www.bbc.com/');

I don't understand how I should use this code and where to put it...
Any help would be great
This is my controller file content.
<?php

namespace Drupal\achva_sso\Controller;

use Drupal\Core\Controller\ControllerBase;

class SsoController extends ControllerBase{
    public function ssoLogout(){
        unset($_SESSION['name']);

        if(isset($_SESSION['name'])){
            $status = $_SESSION['name'] .' is logged in';
        }else{
            $status = 'is logged out';
        }
        return[
            '#type' => 'markup',
            '#markup' => '
' . $status . '
' ,
        ];

    }
}

Answer: Well if you want to redirect to an external page this can be your code:
<?php

namespace Drupal\achva_sso\Controller;

use Drupal\Core\Controller\ControllerBase;
use Drupal\Core\Routing\TrustedRedirectResponse;

class SsoController extends ControllerBase{
    public function ssoLogout(){
        unset($_SESSION['name']);

        // Redirect.
        return new TrustedRedirectResponse('http://www.bbc.com/');

    }
}

In this case, the redirection will happen when your ssoLogout() method will be called.
Comment: As the documentation states, you need to use TrustedRedirectResponse instead to redirect to external URLs.
Comment: But don't send, return the response as stated in the documentation.
Comment: @DavSev I come from update and test the answer and it works
Comment: Thanks @Kevin I thought it was to an internal URL.
Comment: @4k4 can you share the doc link? I used before the send and it works fine, jut to be updated I want to read more about this, thanks.
Comment: Thank you! but actoually I want to redirect to an external url...Is it possible to replace the `` with the wanted url?
Comment: @AdrianCidAlmaguer, I was referring to the link in the question, which returns the response. Why you shouldn't send the response is because of the http kernel interface used by Drupal (REQ->DrupalKernel->REP), see https://stackphp.com/


Can not add .gpx to Xcode to simulate location changes

Question: I can not add .gpx files to xcode. I know that xcode takes specific .gpx with , but nothing works for me.
What I've tried:
1.
<code><gpx>
<wpt lat="52.529985" lon="13.403">
</wpt>
<wpt lat="52.529976" lon="13.403069">
</wpt>
</code>
2.
Adding google routes/gps points here:
https://labs.coruscantconsulting.co.uk/garmin/gpxgmap/mobiledev.php
http://www.bikehike.co.uk/mapview.php
When I try to add .gpx to Debug/Simulate Location/Add GPX... or in Product/Scheme.../Add GPX, my .gpx files are always greyout. Any ideas?
Answer: I guess your gpx file is missing some parts, try creating a file at GPX Generator and compare it with your own.


Is accuracy a reliable evaluation metric for training a classification model?

Question: What is the problem in training on accuracy when balancing classes during training? I Understand why accuracy is not a good metric for the test data evaluation of the model, as in the test data we have no control over class imbalance. This should be the same reason why F1/Dice score is bad for evaluating classification in the test set (although currently used in most segmentation challenges -BRATS, ISLES,...-).
But I see no problem in optimizing on accuracy over training samples, for which we can control class imbalance through selective sampling.
Is there any reason not to use accuracy as the training metric, while being careful not to use it for evaluation of generalization?
Answer: The issues with accuracy are well documented.
Practically: It almost certainly does not measure what you really care about.  Accuracy assigns equal severity to all errors in classification.  This is almost always unrealistic.
Statistically: It fails to be sensitive to the probabilistic nature of classification.  It is a ridiculous delusion of grandeur to believe that a predictive model should be able to preform hard class assignments, that's not how the universe works.  All of our learnings from data are uncertain, and it is a careful scientist's job to quantify their uncertainty.
Mixing of Concerns: Assigning classes and taking actions is a business, not a statistical, problem.  A statistical model's job is to estimate predicted probabilities, which you may then use as an input to a decision rule to solve a business or scientific problem.  This has many benefits, including a clean separation of concerns, an adaptability of the model to different problem domains, and an ability to estimate the expected benefits and costs of various successes or errors in decision making.
Mathematically: The accuracy loss function is not even continuous, so in particular it is not differentiable, and very not-smooth.  Optimization techniques work best on smooth functions, so it's unclear how one would fit such a model.
Culturally: A focus on accuracy in introductory ML and statistical learning courses creates cultural problems.  Observe the backlog of questions on this site evaluating their logistic regressions by naively thresholding at a 0.5 predicted probability, getting poor results in terms of accuracy, and wondering what's wrong with their model.  Nothing is wrong with their model, logistic regression's job is to estimate conditional probabilities, which has been implicitly rejected.
None of this has anything to do with class balance.  The issue with accuracy arising from class balance is real, but secondary to deeper problems.
Answer: The accuracy measure treats all misclassifications the same. Sometimes this is fine, but other times, false positives and false negatives cannot be treated equally. For example, in the domain of medical diagnosis, false negatives are much worse than false positives, because leaving a serious disease untreated is much less desirable than some additional testing that turns out to be unnecessary. You can try to fudge the training step by skewing the class imbalance to resemble your relative misclassification cost, but it's generally better to just use a metric that lets you weight the errors differently to begin with.


Books about penalized regression

Question: I'm looking for textbooks that discuss penalized regression in a methodical manner and present various model selection methods, such as AIC, BIC, Cp as special cases of penalized regression. The textbooks i am looking for don't have to be dedicated to penalized regression, rather this subject can be discussed in a chapter or a section of a more general book. An example of the kind of discussion that I am looking for can be seen in these slides by UCSD instructor Ery Arias-Castro. Books can be in English, French or German. Thanks.
Comment: is there a problem with [ESL2](http://statweb.stanford.edu/~tibs/ElemStatLearn/)?
Answer: The Elements of Statistical Learning.
Free online.
http://statweb.stanford.edu/~tibs/ElemStatLearn/
Answer: I would recommend P.J. Green and B.W. Silverman - "Nonparametric regression and generalised linear models - a roughness penalty approach", CHapman & Hall/CRC Monographs on statistics and applied probability, volume 58.

Books about penalized regression


PDE - 1D Heat Diffusion Problem

Question: I am stuck on this problem I have to do for my PDE class.
Consider heat diffusion in a 1D object between x=0 and x=L. It can be shown that the heat flux at the surface is $-k\nabla{}T*\hat{n}$ i.e. its is proportional to the normal derivative at the surface. In our case, we specify adiabatic boundary conditions as follows : $$u_x(0,t)= 0 $$ $$ u_x(L,t)=0$$ If the initial temperature is $f(x)$ solve for the temperature as a function of x and t.
To be honest, I don't really know where to start. So far I have tried to solve the diff. eq : $$  \frac{\partial u}{\partial t} =-k\nabla{}u*\hat{n}$$ using separation of variables but I got nowhere. Could anyone help me understand how to properly start this problem ? Thank you !
Comment: What does the heat equation look like? How did you write down the equation you wrote down? In particular, why is there no time evolution in the equation you wrote down? It seems something may be wrong then!
Comment: Thank you for your answer. I wrote the equation from what I've seen in class, I corrected the LHS of the equation, I don't know why I wrote d^2u/dx^2 before.
Answer: I will omit the derivation of the heat equation, but I believe you should have
$$
\frac{\partial u}{\partial t} = k\Delta u.
$$
The idea is that for any small region $V$ we have
$$\frac{d}{dt} \int_V u = \int_{\partial V} -k\nabla u \cdot {\bf n}dS.$$
This equation is simply saying that the energy change in the region $V$ (LHS) is given by the flux through the boundary (RHS). Using the divergence theorem on the right hand side you can derive the heat equation.
Now, in 1D, on the interval $[0,L]$ we have simply
$$u_t = k u_{xx}.$$
Assume that $u(x,0)= f(x)$ and $u_x(0,t)=u_x(L,t)=0$. You can use separation of variables from here: assume that $u(x,t) = X(x)T(t)$ and see that
$$XT' = kX''T.$$
Rearranging we have
$$\frac{T'}{kT} = \frac{X''}{X} = -\lambda.$$
This gives us two ODEs:
$$T'+k\lambda T=0$$
and
$$X''+\lambda X=0.$$
Note the boundary values: $u_x(0,t) = X'(0)T(t)=0$ and $u_x(L,t)=X'(L)T(t)=0$.
If $T(t) =0 $ then $u\equiv 0$ and we only recover the trivial solution. So, we may assume that $X'(0)=X'(L)=0$. I believe the rest of the problem should be calculations from ODEs.
Comment: Just saw that! Thank you very much, very helpful and clear !

PDE - 1D Heat Diffusion Problem


The Chicken Portioning Problem

Question: I thought of this question while portioning chicken in the sandwich shop where I used to work. It may reduce to the bin-packing problem, but I am not certain. It is as follows:
You have been given a bucket of irregularly cut chicken to portion for sandwiches. You have been told that each portion must be 3.1 oz. You have used a scale and measured each piece to within 0.1 oz. So for example maybe you have a 1.1 oz piece, and a 0.7 oz piece, and a 2.3 oz piece, etc. Consider these masses to be good, known integer multiples of 0.1 oz. So you have done this and you have made a big list of all your chicken sizes (you are a very diligent employee).
Here is the question: is there any quick way to look at your list and know whether or not there is a way to portion the chicken faster than just trying every combination? You have 31 lb of chicken to portion, and there must still be living people to serve chicken sandwiches to when you are done!
Note that (in some contradiction to my last comment) you don't need to know how to portion the chicken. You just need to know if it can be done. For example, if you were given two 3.0 oz pieces and one 0.2 oz piece, you would know the total is divisible by 3.1, but you could not make two packets out of it.
Thanks,
GOT
Answer: This is not a complete answer, nor optimal, nor from a specialist. But I would like to explain what I think could be a reasonably good approach. Note that it is a constructive approach, still in the style "try out many combinations", just more efficient than a random approach. I do not know whether there is an approach that just gives you a fast yes/no answer without telling you the actual combinations.
1) Order the pieces by weight. I assume that they are all 3.1 oz or less and that you cannot cut them into smaller pieces.
2) Look for 3.1 oz pieces and put them aside (or into a sandwich).
3) Look at the biggest piece left and look for complementary pieces (of weight 3.1-x), starting from the bottom of the ordered list, and going up only until the sum is still smaller or equal than 3.1 oz. If you find a complementary piece, put them both aside. Do the same for all other pieces (of course, if you have many pieces with the same weight, you can skip the remaining ones as soon as you have a failed attempt), always starting the check for the complementary piece from the obvious place in the list (i.e.: start to look for complementary pieces from the same place in the list where you found the last successful match: smaller pieces won't do).
4) Then start again from the first piece and look for fillings with three pieces. If you have stored the information about the available sums of weights of pairs of pieces in the previous round, without repetition (but with a link to the actual pairs), and filtered out the pairs were some pieces have already been put to the side, you might be able to exploit this information to be faster: you are again looking at pairs of data, not at triples!
5) Do it again and again, always increasing the number of pieces you consider, but always comparing only the weights of single pieces with the available sums of $n-1$ weights smaller than 3.1 oz (which you have stored in the previous round)
How do you know when to stop? Well, naively, you cannot use more than 31 pieces for a sandwich. But, more efficiently, when at the beginning of a round you start comparing the biggest piece left with the smallest sum of $n-1$ weights, you can stop if the total sum is bigger than 3.1 oz.
Sorry if the explanation is not very clear. I can write down an example, if you want. Again, this is a naive way to optimize, based on ordering (so as not to check things many times) and storing data (for the same reason). I am not claiming that this is optimal, but it might be a relatively good solution. It should be doable by hand and if you know some programming and have a laptop at work, you can probably use this algorithm to get all the combinations explicitly computed in a couple of seconds.
Remark: I think that it can be shown that this greedy approach actually gives you the highest number of doable sandwiches, because at the end you are left only with small pieces, and you could not do anything meaningful with them anyway, even if they were combined into bigger pieces of the same total mass.

The Chicken Portioning Problem


How to automate calling of a file that is created daily with date stamp in file name

Question: I used logrotate to create a log file that gets rolled over every day at 03:00; so the file that is created has the following format:
userlog.%Y-%m-%d

the script then zips the file as well so the end result is userlog.%Y-%m-%d.gz .... an actual file name is userlog.2015-09-09.gz.
I am writing a shell script that will copy the file created every day and will then unzip it, run a search on the log, extract results, format them, and then email them, and then delete the copied zip file.
I can get everything to work smoothly but I cannot copy the file using the method that it was created in.
I am trying to run the following:
sudo cp userlog.%Y-%d-%m.gz /home/local/temp

but the command does not execute. My understanding is that the current date, month, and year should be substituted into the variable fields in the file name.
If you could please correct my approach and understanding of this concept. Or please let me know if you feel this should work. Or if there is an alternate approach to call a file created in this fashion then kindly advise.
Presently I am having to write the filename into the script manually every morning and I would very much like to escape this hardship.
Answer: You need a simple command substitution.
sudo cp userlog.$(date +%Y-%d-%m).gz /home/local/temp

%Y by itself is just a static string; but when passed as an argument to the date command, it specifies the four-digit year for the given date.  (Default is today's date.)
Regardless of your problems with this particular command, you should certainly not need to edit the script file every day. A very fundamental feature of scripts is the parametrization of arguments. If your current script looks like
#!/bin/bash
gzgrep -c "failure" /path/to/userlog.2015-09-11.gz
sudo cp /path/to/userlog.2015-09-11.gz /home/local/temp

then it can easily be refactored to read the date as a command-line parameter:
#!/bin/bash
: ${1?"Syntax: $0 yyyy-mm-dd"}
gzgrep -c "failure" /path/to/userlog."$1".gz
sudo cp /path/to/userlog."$1".gz /home/local/temp

If you saved this as userlog somewhere in your PATH, the solution to your original problem would now be simply
userlog $(date +%Y-%m-%d)

which of course could be saved as a script, too, or as a shell function in your personal .bash_profile or similar. But if this is mandatory daily routine, you'll probably prefer to add it as a cron job to run every morning (even if you are away). Cron will send you an email with any output. (Do notice that cron does not necessarily have the same PATH etc as your  interactive login shell, though.)
Comment: Unfortunately this did not work. I should have mentioned I am running this script in Solaris... not sure if that possibly may have an impact on the format of this command?
Comment: No, even [Solaris `date`](http://docs.oracle.com/cd/E19253-01/816-5165/6mbb0m9dc/index.html) is surprisingly standard in this regard. How does this fail? Does just `date +%Y-%m-%d` at the prompt produce the required output?


set write permissions on a file for one person everybody else just read

Question: i have a windows server 2003. What permission do i have to set to allow one person to have read write access while everybody else has just read
Comment: the only thing i am stuck with is on everybody do i deny access or just uncheck allow
Comment: Don't use deny unless you have a very rare case.
Answer: I'm going to assume you're talking about a file share, since you don't specify.
On the share ACL, set "Authenticated Users" to Full Control.
Make a group called <ShareName>_RW
Add the user that should write to this group.
Break permission inheritance for that share.
Make it so that the NTFS ACL on the share says:
Administrators - Full Control

<ShareName>_RW - Modify

Users - Read

The user that you just put in that group won't have access until he/she logs off and back on.


What exactly is the bell used in Paganini's 2nd violin concerto and how should one play it?

Question: Paganini's Violin Concerto No.2, Op.7 has a part for a "Campanello (Little Bell in F#)" in the 3rd movement, which is also the movement's namesake - "La Campanella". In a bit of research I've done, the instrument is described just as a "little bell" or sometimes as a "handbell". The Wikipedia page for Handbell discusses different types of handbells and different playing techniques. I also find it odd there is no mention there of its use in such a famous piece.
Furthermore, the IMSLP part shows the instrument plays only a C note, while it's listed as an instrument in F#.
My questions are:

What is the bell Paganini instructed?
What is the way the bell is used to produce the sound?
Does "bell in F#" mean it's a transposing instrument (like trumpet in Bb)? I thought it means it produces the tone F#, but the sheet music says C. Is it just its notation?
Answer: The third movement, "Rondo à la clochette" is in B minor.  The bell sounds the note F#.  Hear it at 0'47" in this YouTube link.  I'm not sure why Paganini (or his editor) would choose to notate it as a transposing instrument, but they did.
Comment: The bass drum part on IMSLP is also written with a treble clef on note C. Curiously, the percussion parts (timp, bass drum, bell) are not in then IMSLP score at all! The bass drum is notated the same way as the bell (5 line staff, treble clef, note C). The timp part is also treble clef, and doesn't even indicate the tuning required. The score implies the bell only plays half a dozen notes, unlike the part which keeps the bell ringer busy for the whole movement. A scholarly edition, this isn't!


My EC2 Ubuntu server was bounced after do-release-upgrade and never came back up. How can I recover?

Question: I had an Ubuntu Amazon EC2 server, running a few websites, and I left it to have an unattended do-release-upgrade. The server said that a reboot was required. I (tried to reboot) it and it never came up. Since trying to reboot, I have not succeeded in getting it to serve a webpage or shell in to it.
What are my options or next steps to recover? Is there a way to download a disk image and recover from that?
Thanks,
Comment: Have you tried simply stopping it and restarting it since it failed?
Comment: Yes; I've tried rebooting it, and nothing has worked.
Comment: advice for the future, always create an ami of your server before running any kind of update, failures are common and restoring from an ami takes seconds.
Answer: No, you can't download a disk image (at least not easily).
Here's my recommendation:

Fire up a new ubuntu server and configure it as you wish. This will become the replacement for your failed server.
Detach the disk from your old server and attach as a second disk to the new server. I cover this basic process in this answer.
Copy over any data you need from the old disk to the new one.
Detach the old disk and terminate that old instance.


Positive polynomials in $\mathbb{R}^+$

Question: Let $P\in \mathbb{R}[X]$ be a polynomial such that $P (x)\geq 0$ for all $x\geq 0$.
Show that there exists two polynomials $A, B\in \mathbb{R}[X]$  so that $$P=A^{2}+X\cdot B^{2}$$
Comment: Please always show the own thoughts. For instance try to say something in the special case of $P=x^4+x^2+1$. Else there are good chances for other downvotes. (Not mine, i type first...) It is always a good idea to mention the source, the level of the problem, the own level, one, two, or more trials... Please also use mathjax, https://math.meta.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference ...
Answer: Idea: Write $$P(x) = Q(x)+xR(x)$$
where $Q(x) ={1\over 2}(P(x)+P(-x))$ and $R(x) = {P(x)-Q(x)\over x}$ and try to prove that $Q$ and $R$ are perfect square of a real polynomials.

This is better:
1. step: Suppose all the zeroes of $P$ are real and simple (i.e. first degree). Then $$P(x) = a(x-x_1)(x-x_2)...(x-x_n)$$ Because $P(x)\geq 0$ for $x\geq 0$ they all have to be nonpositive. So $y_i = -x_i$ is positive for each $i$. So we have $$P(x) = a(x+y_1)(x+y_2)...(x+y_n)$$
Now we can prove that such polynomial is of a form we want:
Say $a>0$ and if $M = A^2+xB^2$ then $$(x+a)M = x(A+aB)^2+(aA-xB)^2$$ and we are done.

2. step Now if $P$ has also nosimple real zeroes, then we can write it as $$P= Q\cdot R^2$$
where $Q$ again has only simple real zeros, so we can use 1. step and we are done.

3. step Say $P$ has also nonreal zeros. So $P=Q\cdot R$ where $Q$ has only real zeros and $R$ only nonreal (so they are in conjugate pairs). Now we can use step 2. for $Q$ and we can prove that we can write $R$ as $C^2+D^2$. I leave that to you. Only thing is to prove now that product of polynomial $A^2 +xB^2$ and $C^2+D^2$ is again of form $A_1^2+xB_1^2$ and that I also leave to you.
Comment: Thanks Greedoid, but can you elaborate more?

Positive polynomials in $\mathbb{R}^+$


How to estimate a fixed effects regression WITH robust standard errors AND instrument variables

Question: I have been trying to estimated the stated problem, but I only succeed in parts of it. The following artificial setup is supposed to illustrate my problem in detail:
Setup the data:
library(plm)
data(Cigar)

Fixed effects AND robust standard errors  (works perfectly)
  fit <- plm(price ~ sales + pop, data=Cigar, index=c("state","year"), model="within")

> coeftest(fit, vcov.=vcovHC(fit))

t test of coefficients:

        Estimate Std. Error t value  Pr(>|t|)
sales -1.0391141  0.1671141 -6.2180 6.726e-10 ***
pop    0.0190151  0.0064447  2.9505  0.003228 **
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

New Fixed effects regression WITH Instruments: (sales is instrumented with cpi)...
fit <- plm(price ~ sales + pop | cpi + pop, data=Cigar, index=c("state","year"), model="within")

...renders usage of coeftest problematic!

coeftest(fit, vcov.=vcovHC(fit))
      Fehler in vcovG.plm(x, type = type, cluster = cluster, l = 0, inner = inner,  :
        Method not available for IV

Well the error message is obvious, but I am just wondering what other people do who face the same problem as I do?
I hope I don't miss something but there is no statistical reason why heteroskedasticity cannot apply in this setting, right?
So is there any possibility to estimate robust standard errors in this setting with the plm package or are there any R-packages which may help?
Thank you very much for your answers - any advice is appreciated!
Comment: I don't see the difference between the two `coef()` calls, so I suspect something is missing. In addition, questions about how to use software are usually off topic here (this strikes me as borderline), so you may want to try to bring the statistical issue to the fore & deemphasize the 'how do I do this in R' part.
Comment: Thanks for the quick comment. There has been a line drop during copy and paste from R, should be fixed now.
Comment: Your comment "but there is no statistical reason why heteroskedasticity cannot apply in this setting, right?" needs qualification. Do you expect the variance to be different for each cross-section? Time varying heteroskedasticity? Or do you anticipate each observation to have a different residual variance? You need an appropriate model of heteroskedasticity (and autocorrelation) that applies to your case. If it helps, note that Stata's [`xtivreg`](http://www.stata.com/manuals14/xtxtivreg.pdf) computes cluster robust standard errors when `robust` standard errors are requested (pg. 6).
Comment: The appropriate methods for robust vcovs were not implemented in plm. They are now in the development version >= 1.6-1 (see r-forge.r-project.org/R/?group_id=406).
Answer: EDIT: The methods are also in CRAN versions of plm >= 1.6-4.
The appropriate methods for robust vcovs were not implemented in plm. They are now in the development version >= 1.6-1 (see http://r-forge.r-project.org/R/?group_id=406).
library(plm)
library(lmtest)
data(Cigar)
fit <- plm(price ~ sales + pop, data=Cigar, index=c("state","year"), model="within")
fit2 <- plm(price ~ sales + pop | cpi + pop, data=Cigar, index=c("state","year"), model="within")

coeftest(fit, vcov.=vcovHC(fit))

# t test of coefficients:
#
#
#         Estimate Std. Error t value  Pr(>|t|)
# sales -1.0391141  0.1671141 -6.2180 6.726e-10 ***
# pop    0.0190151  0.0064447  2.9505  0.003228 **
# ---
# Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

coeftest(fit2, vcov.=vcovHC(fit2))

# t test of coefficients:
#
#        Estimate Std. Error t value Pr(>|t|)
# sales -6.2479556  1.9032780 -3.2827 0.001055 **
# pop   -0.0021752  0.0121773 -0.1786 0.858260
# ---
# Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1


Thread Synchronization C++

Question: I am having this weird issue with threads. On my mac with OS X this works fine but once I more it over to my desktop that is running Ubuntu, I am facing issues.
Essentially what I am doing is the following:
Function() {

   for(i = 1 to 10)
      while(array not completely changed) {
         pthread_mutex_lock(&lock);
         -- perform actions
         pthread_mutex_unlock(&unlock);
      }
   }
}

And I have two threads running this function. While it is supposed to be running in such a manner that is:
Thread 1 grabs lock
performs opperations on array
Thread 1 releases lock
Thread 2 grabs lock
performs calculations on array
Thread 2 releases lock

and so on in a back and forth pattern until the array have been completed changed but on Linux all of the calculations of Thread 1 complete and then Thread 2 starts.
So I will get:
Thread 1 grabs lock
performs opperations on array
Thread 1 releases lock
Thread 1 grabs lock
performs calculations on array
Thread 1 releases lock
Thread 1 grabs lock
performs calculations on array
Thread 1 releases lock

And so on until the array is completely changed, once I increment the for loop, then Thread 2 will perform all calculations and continue this pattern.
Can anyone explain what is going on?
Comment: Note we have `std::thread` class in the current c++ standard, that should be used preferably over `pthreads` native library API.
Comment: It's the thread scheduler that you are interested of. Check this out http://man7.org/linux/man-pages/man7/sched.7.html. Since version 3.14, Linux provides a deadline scheduling policy (SCHED_DEADLINE). You are probably interested in round-robin policy.
Comment: If you want to force a switch to a specific thread you need to manage that yourself, otherwise it is completely random which thread will get the lock (its not the order they reach the lock).
Answer: You're experiencing "starvation".  Add a small nanosleep call occasionally to give the other threads a chance to run.  Add the call outside the mutex pair (e.g. after the unlock).  Thread 1 is monopolizing things.
You may also want to consider restructuring and splitting up the critical [requires locking] vs non-critical work:
while (more) {
    lock ...
        do critical stuff ...
    unlock ...
    nanosleep ...
    do non-critical stuff
}

Comment: Where would I place this `nanosleep` within my code?
Comment: After the lock has been released but prior to reobtaining it. Putting the first thread to sleep will allow time for the second thread to acquire the lock etc.

Thread Synchronization C++


Set SoundiD xcode error - trying to play music

Question: I had this working earlier but for some reason after I've added some more code and then took some of it out, stuff seems to stop working.
The AudioServicesPlaySystemSound(soundID); is causing some error.
<+0117>  mov    %eax,-0x18(%ebp)
+0313>  test   %eax,%eax
<+0348>  test   %eax,%eax

soundID UInt32  3221214520
soundFileURLRef CFURLRef    0x75217a0
mainBundle  CFBundleRef 0x73122b0
_cmd    SEL 0x511b
self    FirstViewController *   0x7349e90
This are some of the errors I see in the stack trace.
This may seem like stupid questions but I come from Java.
What has caused this error?
Everything was playing fine before. I didn't change any of the code, why does stuff start breaking down.
How do you prevent it from happening and how do I fix?
- (IBAction) boom;  {

CFBundleRef mainBundle = CFBundleGetMainBundle();
CFURLRef soundFileURLRef;
soundFileURLRef = CFBundleCopyResourceURL(mainBundle, (CFStringRef) @"jarrvvv1", CFSTR ("mp3"), NULL);
UInt32 soundID;
AudioServicesCreateSystemSoundID(soundFileURLRef, &soundID);
AudioServicesPlaySystemSound(soundID);

}
This suggested method for some reason doesn't work.
- (IBAction) boom;  {

    UInt32 soundID;

NSURL * soundFileURL = [[NSBundle mainBundle] URLForResource:@"jarrvvv1"  withExtension:@"mp3"];
 AudioServicesCreateSystemSoundID((__bridge CFURLRef)soundFileURL, &soundID);
AudioServicesPlaySystemSound(soundID);

}
Answer: UPDATE
I just checked, AudioServicesCreateSystemSoundID does not support mp3!
From the Multimedia Programming Guide:

The AudioServicesPlaySystemSound function lets you very simply play short sound files. The simplicity carries with it a few restrictions. Your sound files must be:
No longer than 30 seconds in duration
  In linear PCM or IMA4 (IMA/ADPCM) format
  Packaged in a .caf, .aif, or .wav file

previous answer
Check that soundFileURLRef is returning non-NULL. If it's NULL the mp3 file may not have been copied to your app bundle.
p.s. you can save a little thought by doing resource loading stuff using Foundation (the objective-c API) instead of CoreFoundation:
NSURL * soundFileURL = [[NSBundle mainBundle] URLForResource:@"jarrvvv1" withExtension:@"mp3"];
NSLog(@"soundFileURL: %@\n", soundFileURL);
UInt32 soundID;
AudioServicesCreateSystemSoundID((__bridge CFURLRef)soundFileURL, &soundID);

That way ARC will take care of the memory management for you.
Comment: i took out everything inside the { } and i still gives me the same error.. check out my edited question.. @rythmicfistman
Comment: lol, you're  cool. let me check,.. [2225:207] soundFileURL: file://localhost/Users/Tahj182/Library/Application%20Support/iPhone%20Simulator/5.0/Applications/090E7639-A571-478C-AEC0-F284E9FB956D/clxxxii.app/jarrvvv1.mp3
Comment: @rythmicfistman i'm converting the sound files to .wav right now.. but why did it stop working the first time. the one i had shown in the question was playing around 3:30 o'clock today but now around 8:20 its not working.. thats crazy, java doesn't do that
Comment: you did it bro!!!!!!!!!!!!!!!  @rythmicfistman !!!!!!!!!!!!!!!!!!!!!!!!!! how can i follow you like on twitter or facebook. i dont want to bother you, but I want know important people like you. people like you are amazing, you don't even know it!
Comment: You're welcome! Twitter handle is in my bio and don't forget to mark the question as answered!
Comment: @rhythmfistman i don't have enough reputation points...  i'm about to tweet you..


gcc i686 on x86_64 platform

Question: I have some troubles to install GCC i686 on a RHEL X86_64.
Indeed, I have to build some 32bit softwares and shared libraries on this platform.
I can build these softwares and libraries on 32bit platforms (linux or windows).
My questions are at the end of this post.
My first problem was this error: (during a buil, under eclipse -helios)
In file included from /usr/include/stdlib.h:314,
                  from ../../../../../XXXX.h:19,
                  from /XXXX.c:33:
/usr/include/sys/types.h:150: error: duplicate 'unsigned'
/usr/include/sys/types.h:151: error: duplicate 'unsigned'
/usr/include/sys/types.h:151: error: duplicate 'short'
/usr/include/sys/types.h:152: error: duplicate 'unsigned'
/usr/include/sys/types.h:152: error: two or more data types in declaration specifiers
make: *** [XXXX.o] Error 1

To correct this error, I had to put the stdlib.h include before all the other files, but I have a lot of files, and sometimes this trick did not work anyway.
Moreover, I should not modify the source files.
I have exactly the same problem when I use a makefile given by a friend, to build a shared library.
This makefile works well on his platform (the same as me, RHEL 4.4.6 x86_64).
He told me the error appears because I use X86_64 lib, to build a 32bits software (or shared lib).
Here's my version of GCC :
GCC version

[root@localhost bin]# gcc -v
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr
--mandir=/usr/share/man
--infodir=/usr/share/info
--with-bugurl=http://bugzilla.redhat.com/bugzilla
--enable-bootstrap
--enable-shared
--enable-threads=posix
--enable-checking=release
--with-system-zlib
--enable-__cxa_atexit
--disable-libunwind-exceptions
--enable-gnu-unique-object
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada
--enable-java-awt=gtk
--disable-dssi
--with-java-home=/usr/lib/jvm/java-1.5.0-gcj-18.104.22.168/jre
--enable-libgcj-multifile
--enable-java-maintainer-mode
--with-ecj-jar=/usr/share/java/eclipse-ecj.jar
--disable-libjava-multilib
--with-ppl
--with-cloog
--with-tune=generic
--with-arch_32=i686
--build=x86_64-redhat-linux
thread: posix
gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC)

[root@localhost bin]# rpm -qa |grep gcc
gcc-c++-4.4.6-3.el6.x86_64
gcc-4.4.6-3.el6.x86_64
gcc-gfortran-4.4.6-3.el6.x86_64

So first, I installed glibc.i686 and libgcc.i686 packages from the RHEL DVD setup.
Now I have:
Installed packages (from redhat DVD)
[root@localhost bin]# rpm -qa |grep glibc
glibc-common-2.12-1.47.el6.x86_64
glibc-2.12-1.47.el6.x86_64
glibc-devel-2.12-1.47.el6.x86_64
glibc-devel-2.12-1.47.el6.i686
glibc-headers-2.12-1.47.el6.x86_64
glibc-2.12-1.47.el6.i686

[root@localhost bin]# rpm -qa |grep libgcc
libgcc-4.4.6-3.el6.x86_64
libgcc-4.4.6-3.el6.i686

Since GCC is x86_64, I read some documents about the cross compilation, espcially this one:
wiki.osdev.org/GCC_Criss-Compiler
So I downloaded: gcc-4.4.6.tar.gz, binutils-2.23.tar.gz, gmp-5.0.2.tar.gz, and mpfr-3.1.1.tar.gz.
I put the directories gmp-5.0.2 and mpfr-3.1.1 in gcc-4.4.6 directory (and I renamed gmp-5.0.2 to gmp, and mpfr-3.1.1 to mpfr).
I followed the wiki.osdev instructions, that is:
export PREFIX=/usr/local/cross
export TARGET=i686-elf
cd /usr/src
mkdir build-binutils build-gcc
cd /usr/src/build-binutils
../binutils-x.xx/configure --target=$TARGET --prefix=$PREFIX --disable-nls
make all
make install
cd /usr/src/build-gcc
export PATH=$PATH:$PREFIX/bin
../gcc-x.x.x/configure --target=$TARGET --prefix=$PREFIX --disable-nls \
    --enable-languages=c,c++ --without-headers
make all-gcc
make install-gcc

'make all' and 'make install' for binutils => OK
'make all-gcc'

--> 1st error: missing "mpfr.h" in "real.h". So I added mpfr.h in gcc-4.4.6/gcc and it was OK (maybe not actually ...)
--> 2nd error (the only one now):
[...]
gcc  -g -O2 -DIN_GCC -DCROSS_DIRECTORY_STRUCTURE  -W -Wall
-Wwrite-strings
-Wstrict-prototypes
-Wmissing-prototypes
-Wcast-qual
-Wold-style-definition
-Wc++-compat
-Wmissing-format-attribute
-pedantic
-Wno-long-long
-Wno-variadic-macros
-Wno-overlength-strings
-DHAVE_CONFIG_H
-o cc1-dummy c-lang.o stub-objc.o attribs.o c-errors.o c-lex.o c-pragma.o c-decl.o
c-typeck.o c-convert.o c-aux-info.o c-common.o c-opts.o c-format.o c-semantics.o
c-ppoutput.o c-cppbuiltin.o c-objc-common.o c-dump.o c-pch.o c-parser.o i386-c.o
c-gimplify.o tree-mudflap.o c-pretty-print.o c-omp.o dummy-checksum.o \
              main.o  libbackend.a ../libcpp/libcpp.a ../libdecnumber/libdecnumber.a ../libcpp/libcpp.a   ../libiberty/libiberty.a ../libdecnumber/libdecnumber.a
-L/usr/src/build-gcc/./gmp/.libs -L/usr/src/build-gcc/./gmp/_libs
-L/usr/src/build-gcc/./mpfr/.libs -L/usr/src/build-gcc/./mpfr/_libs
-lmpfr -lgmp
**/usr/bin/ld: cannot find -lmpfr
collect2: ld returned 1 exit status
make[1]: *** [cc1-dummy] Error 1
make[1]: Leaving directory `/usr/src/build-gcc/gcc'
make: *** [all-gcc] Error 2**

**Finally, my questions are :

Is this cross-compilation can resolve my problem ?
What is the good way to resolve the problem of the missing ld mpfr ?**

I made a lot of research before posting. My linux knowledges are not very good at this time.
Thank you in advance for your help.

 EDIT #1 :
I've already tried -m32 flag but the problem still here.
For example, if I run a makefile:
[root@localhost makefile]# make -f sharedLib.mak
gcc -m32 -march=i686 -O2  -Wall -I ../../sharedLib/inc/ -o XXX.o -c ../src/XXX.c
In file included from /usr/include/stdlib.h:314,
                 from ../src/XXX.c:51:
/usr/include/sys/types.h:150: error: duplicate 'unsigned'
/usr/include/sys/types.h:151: error: duplicate 'unsigned'
/usr/include/sys/types.h:151: error: duplicate 'short'
/usr/include/sys/types.h:152: error: duplicate 'unsigned'
/usr/include/sys/types.h:152: error: two or more data types in declaration specifiers
make: *** [XXX.o] Error 1

Here's XXX.c:
#include "alphabet.h"
#include "outils.h"
#include "erreur.h"
#include <string.h>
#include <stdlib.h> (line 51 error)

If a modify this way:
#include <stdlib.h>
#include "alphabet.h"
#include "outils.h"
#include "erreur.h"
#include <string.h>

Everything is OK for XXX.c but the error appears for the next source file ...
Answer: It shouldn't be necessary to compile a 32bit version of GCC; just supply the -m32 flag to generate 32bit code with the 64bit version of GCC (FAQ entry).
This question supports this: How to compile glibc 32bit on an x86_64 machine
Note that you might have to install 32bit versions and devel packages of all the dependencies of the code that you want to compile (i.e. linux header files, glibc, ...)
EDIT To debug problems like that, it's often useful to see the source which GCC tries to compile. For this, use gcc -E instead of gcc -c. That will give you the preprocessed C source plus a lot #line statements which tell what file GCC was reading to produce what follows.
Create such a file and examine it. The errors above make me wonder if you created a preprocessor macro which expands to something unexpected in stdlib.h.
Comment: Thanks again.
With gcc -E, there is no error.
Everything seems to be OK during the prepocessing stage (at the end, myLib.a is created).
Comment: When running with `-E`, the code isn't compiled. You can only use the output to understand why it fails to compile! The resulting myLib.a is probably useless.

gcc i686 on x86_64 platform


Optimize sql except query

Question: I have a query that needs to be optimized and im not sure how. This is the sql script:
declare @tempTable table(color1 int, color2 int, color3 int, color4 int, newToken uniqueidentifier default newid(), ordinal int identifier(1,1))

insert into @tempTable
select color1, color2, color3, color4
from @colorCombination
except
select c1.color as color1, c2.color as color2, c3.color as color3, c4.color as color4
from products p
inner join attributes c1 on c1.pId = p. Id and c1.type = 'primary'
inner join attributes c2 on c2.pId = p. Id and c2.type = 'secondary'
inner join attributes c3 on c3.pId = p. Id and c3.type = 'other1'
inner join attributes c4 on c4.pId = p. Id and c4.type = 'other2'
where p.category = 'furniture'

Without the except, both select statement run really fast but with the except, it took nearly 15minutes
@colorCombination has nearly 24,000 rows
The productattributes return 11,000 rows
I used the except because the need to find those 13,000 that are not in the database yet and then insert it.
This is running on sql server 2008
Is there a better way to get the missing records instead of using except?
Answer: The poor performance from so few rows is caused by the table variables and how they impact the execution plan.
This answer uses temporary tables with nonclustered indexes on (color1, color2, color3, color4). Your original table variables did not specify not null for any colors, so I will assume that some null values are acceptable. If they are not, you could drop the surrogate Id and use a clustered index on the four color columns. To allow for nulls, this also switchs back to except instead of not exists(). If null is not a factor, then not exists() should be a bit faster.
Reference concerning table variables and temp tables:

Temporary Tables in Stored Procedures - Paul White
What is the difference between a temp table and table variable in SQL Server? - Answer by Martin Smith
When should I use a table variable vs temporary table in sql server? - Answer by Martin Smith

/* step 1: existing colors in a temp table with an index */

create table #productColors (
    id int not null identity (1,1) primary key clustered
  , color1 int , color2 int , color3 int , color4 int
  );

insert into #productColors (color1, color2, color3, color4)
select distinct
    color1  = case when c.[type] = 'primary'   then c.color end
  , color2  = case when c.[type] = 'secondary' then c.color end
  , color3  = case when c.[type] = 'other1'    then c.color end
  , color4  = case when c.[type] = 'other2'    then c.color end
from from products p
  inner join attributes c
    on p.Id = c.Id
group by p.id;

create nonclustered index ix_productColors
  on #productColors (color1, color2, color3, color4);

/* step 2: color combinations in a temp table with an index */

create table #colorCombinations (
    id int not null identity (1,1) primary key clustered
  , color1 int , color2 int , color3 int , color4 int
);

insert into #colorCombinations (color1, color2, color3, color4)
select distinct color1 , color2 , color3 , color4
from @colorCombinations;

create nonclustered index ix_colorCombinations
  on #colorCombinations (color1, color2, color3, color4);

/* step 3: insert new color combinations into #tempTable */

create table #tempTable (color1 int
  , color2 int
  , color3 int
  , color4 int
  , newToken uniqueidentifier default newid()
  , ordinal int identifier(1,1)
);

insert into #tempTable(color1, color2, color3, color4)
  select color1, color2, color3, color4
  from #colorCombination
  except
  select color1, color2, color3, color4
  from #productColors

Old answer:
Using except also de-duplicates the rows in your @tempTable (which might be better off as a #temptable instead of a @TableVariable depending on how you are using it).
If you do not need to de-duplicate the rows from @tempTable, then you can use not exists() (or add distinct to the select, but it will take a performance hit):
insert into @tempTable
select color1, color2, color3, color4
from @colorCombination cc
where not exists (
  select 1
  from products p
  inner join attributes c1 on c1.pId = p. Id and c1.type = 'primary'
  inner join attributes c2 on c2.pId = p. Id and c2.type = 'secondary'
  inner join attributes c3 on c3.pId = p. Id and c3.type = 'other1'
  inner join attributes c4 on c4.pId = p. Id and c4.type = 'other2'
  where p.category = 'furniture'
    and c1.color = cc.color1
    and c2.color = cc.color2
    and c3.color = cc.color3
    and c4.color = cc.color4
)

Comment: I'd also like to inquire as to why the temp table/table variable is used at all? Unless the query results need to be used several times, I'd eliminate that part and directly use the query... This does depend on grander purpose, though.
Comment: Thanks for the quick response! I'll try it later and let you know. Thank you so much. I have edited the script and it has the columns for the temp table. After inserting in that, i need to bulk inserts the missing rows in 3 tables
Comment: i tried the not exist.. sadly it's taking 16 something minutes now
Comment: @IkraamMX Updated my answer to suggest switching to temporary tables.

Optimize sql except query


factor loadings = eigenvectors in R output?

Question: I'm trying to make sense of a principal component analysis using R (either princomp or prcomp, I get similar results) with a correlation matrix analysis.  In particular, I'm having trouble understanding the factor loadings output.
The data are in a data frame called ds.  Here is the eigenvalue/vector analysis of the correlation matrix.
> eigen(cor(ds))
$values
[1] 3.9831253 0.5483985 0.4520555 0.4191041 0.3506337 0.2466829

$vectors
           [,1]       [,2]         [,3]       [,4]       [,5]        [,6]
[1,] -0.4292649  0.2561060 -0.298728176  0.2474691  0.3939538  0.66668007
[2,] -0.3938608  0.7037562 -0.066953299  0.1803110 -0.4130921 -0.37677712
[3,] -0.4060290  0.1537541  0.365977815 -0.7055394  0.4027306 -0.13259964
[4,] -0.4166136 -0.4038936 -0.004352731  0.4760061  0.3984587 -0.52719349
[5,] -0.3951931 -0.4041873 -0.621151418 -0.3939178 -0.3732825 -0.01071668
[6,] -0.4074327 -0.2983273  0.621683956  0.1634262 -0.4624437  0.34343296

Now, when I ask for a principal component analysis, I get the following initial output.
> pca.out <- princomp(ds, cor=TRUE)
> summary(pca.out)
Importance of components:
                          Comp.1     Comp.2     Comp.3     Comp.4     Comp.5     Comp.6
Standard deviation     1.9957769 0.74053931 0.67235070 0.64738253 0.59214334 0.49667185
Proportion of Variance 0.6638542 0.09139974 0.07534258 0.06985069 0.05843896 0.04111382
Cumulative Proportion  0.6638542 0.75525396 0.83059653 0.90044722 0.95888618 1.00000000

which all makes sense, given the output of the eigen() function.
However, my understanding is that loadings are computed as the product of the eigenvector and the square root of the eigenvalue.  When I ask for the loadings from the pca, I get
Loadings:
  Comp.1 Comp.2 Comp.3 Comp.4 Comp.5 Comp.6
A -0.429  0.256 -0.299  0.247 -0.394  0.667
B -0.394  0.704         0.180  0.413 -0.377
C -0.406  0.154  0.366 -0.706 -0.403 -0.133
D -0.417 -0.404         0.476 -0.398 -0.527
E -0.395 -0.404 -0.621 -0.394  0.373
F -0.407 -0.298  0.622  0.163  0.462  0.343

               Comp.1 Comp.2 Comp.3 Comp.4 Comp.5 Comp.6
SS loadings     1.000  1.000  1.000  1.000  1.000  1.000
Proportion Var  0.167  0.167  0.167  0.167  0.167  0.167
Cumulative Var  0.167  0.333  0.500  0.667  0.833  1.000

Not only do the cumulative and proportion variance not match the initial output, but the loadings simply match the eigenvectors, with no adjustment for eigenvalue.  Furthermore, the claim that the first component captures 66% of the variance is impossible with these loading values, because every single variable in the data set (A-F) has a later component with a higher (absolute) loading.
Can someone please straighten out my confusion/error?  For the record, I'm running R version 3.3.3
Comment: ttnphns comment [here](https://stats.stackexchange.com/q/276645/67822): *R documentation labels eigenvectors "loadings" ... (to say it soft) ... recklessly. This issue was asked by users here million of time.*
Comment: @AntoniParellada - thanks.  I did actually google search for this and also looked at the suggested links that came up when posting my question.  I hope that maybe my title will make the issue easy to identify as a repeat in future.
Comment: Would this be wrong? rawLoadings <- pca_out$rotation[,1:5] %*% diag(pca_out$sdev, 5, 5)?
Comment: Yes - I believe that would work, though your code doesn't render properly in the comment.  I'll try reproducing it here from the notification I got: `rawLoadings <- pca_out$rotation[,1:5] %*% diag(pca_out$sdev, 5, 5)`  though you should have written "loadings" instead of "rotation", I think.
Answer:
However, my understanding is that loadings are computed as the product
  of the eigenvector and the square root of the eigenvalue.

I depends on definition of loading you use. In princomp loadings are simply coefficients of principal components (recall that principal components are linear combinations of original variables) that are equal to eigenvectors entries. This has one inconvenience: since variance of each PC equals corresponding eigenvaule, loadings defined this way are not correlations between PC's and original variables. Correction by square root of eigenvalue is done to standardize the variance of PC scores to 1 and therefore to allow for correlation interpretation of loadings. These standardized loadings are sometimes called loadings as well. See for example PCA function from FactoMineR package. It never uses a word loadings, it uses word coordinates for standardized loadings.

Not only do the cumulative and proportion variance not match the initial output

loadings function doesn't give you cumulative and proportion variance. It just gives you sum of squares of each PC's loadings. And this, by definition, is 1. So, you'll always see this kind of output. It sounds ridicullus but works well when you apply loadings function to Explanatory Factor Analysis. In PCA, second part of loadings output is simply useless.

the claim that the first component captures 66% of the variance is
  impossible with these loading values, because every single variable in
  the data set (A-F) has a later component with a higher (absolute)
  loading

Actually it is possible, since loadings here are just eigenvectors not standardized loadings.
Comment: +1 but when you say "standardized loadings" (e.g. in the last sentence) do you mean standardized to SS=1 or scaled to variance=eigenvalue? If the latter, it's really strange to call it "standardized" because "standardized" usually means scaling to variance=1. I'm also confused by this sentence: "Correction by square root of eigenvalue is done to standardize their variance to 1 and therefore to allow for correlation interpretation." -- corr interpretation is only possible if variance=eigenvalue not variance=1!
Comment: @Łukasz Deryło - thank you for your clear explanation.  In all the reading I've been doing it seems like "standardized loadings" gets abbreviated to just loadings.  (Given that people write that the "loadings" provide the correlations between the components and the original variables.) It seems redundant to call the eigenvectors loadings when you can simply call them eigenvectors.  However, I thank you for your explanation.
Comment: @amoeba - I believe he means the component scores are standardized to z-scores (i.e. M=0, SD=1).  It makes it possible to use the standardized loadings as the regression coefficients in a standardized regression predicting z for a measured variable from the z-scores of the components.  And in that regression equation, the correlation interpretation is possible, because the components are uncorrelated (in the initial solution or in subsequent orthogonal rotations).
Comment: @JTaylor This is exactly what I meant
Comment: Lukasz (and @JTaylor): so to be completely clear - when you say "standardized loadings" you mean "loadings corresponding to the standardized scores" i.e. loadings having sum of squares = eigenvalue? I find it confusing terminology but okay.
Comment: @amoeba This is what I meant. I agree that this terminology is a bit confusing. But whole this question is about confusing terminology :)


O que faz Python ser uma linguagem de programação multiparadigma?

Question: Aqui nessa página do Wikipedia cita alguns exemplos de linguagem multiparadigma como C++, Groovy, Oz, Ruby, Scala, Swift, Groovy e inclusive o Python. Particularmente não achei tão objetiva a explicação.
O que faz Python ser uma linguagem de programação multiparadigma?
Answer: Ela permite usar vários paradigmas :P
Sabendo que o paradigma é só um nome pomposo para o estilo, você pode programar em mais de um estilo. Se não tiver certeza absoluta do que é um paradigma e quais os principais, não deixe de ler este último link.
Como toda linguagem mainstream, Python tem o estilo básico de programação imperativa, que são a maioria do comandos, que controlam o fluxo, atribuição de variáveis, etc.
Ela tem uma influência do paradigma funcional, permitindo recursão, uso de lambda, compreensões de coleções de dados, etc. É um pouco de exagero dizer que Python é uma linguagem funcional, mas pelo marketing acaba-se usando, todas linguagens o fazem. De fato dá para adotar um pouco o estilo, mas só um pouco, bem pouco.
Além disso ela permite alguns paradigmas secundários. A orientação a objetos é o mais claro deles. Como ela permite o uso de classes com encapsulamento, herança e polimorfismo, ela se conforma com OOP (paradigma que está longe de ter a importância que muitos acreditam que ele tem, e isso não é opinião). Mas a organização do código pode seguir o paradigma procedural também, onde apenas funções simples são usadas.
Como a maioria das linguagens dinâmicas (que alguns até consideram paradigma) ela possui capacidade reflexiva (informações sobre a estrutura dos seus dados e códigos).
Uma linguagem dita multi paradigma tende a agradar mais pessoas e atende mais necessidades. E de fato tem um fundo de verdade que Python adota alguns estilos extras além do imperativo que é o seu forte.
Não se baseie na tabela comparativa da Wikipedia, tem muita coisa errada ali. Parece que foi feito por alguém que sequer entende o que sejam os paradigmas (na verdade feito por várias pessoas cada um com um background diferente, deve ter dado muita briga, foi algo criado em comitê que não tem consenso).
Coloquei no GitHub para referência futura.


rsync to unmounted webdav share

Question: if I want to transfer files to a webdav share, can I do that directly to the server that the share originates from, or does it need to be mounted on the machine that I send the rsync to?
The thing is, I sort of can't mount that webdav share anywhere, but it's accessible through a browser. That'll let me download things, but not upload via rsync. So I'm wondering, can I get rsync to put things on the webdav share while it isn't mounted anywhere?
I figure authentication might be a problem, because afaik rsync uses the ssh protocol, so how can I make it authenticate to webdav?
Answer: You can only use rsync to connect to the webdav share if you first mount the webdav share on a system that you can ssh to.  For example, in linux you can use something like davfs.  After you do that, the webdav share will appear as a regular filesystem on your system.
So for example, if you used davfs to mount the webdav share on your linux machine 'foo.example.com', you could then run rsync on another machine and connect to foo.example.com over ssh to access your files.
Comment: So once I mount that share on a machine that I can ssh into, where can I direct rsync to copy things to? It looks like it's not working like this: rsync -a /some/local/source-directory/ webdav://my.server.com/remote.php/webdav/target-directory/ , so I'm guessing it has to be: rsync -a /some/local/directory/ email@example.com://target-directory/under-local-mount-point. The problem is I don't know which local directory the share is mounted to.

rsync to unmounted webdav share


test statistics needed for data to be normalised

Question: I understand from one of the post that t-test is preferred over z-test. Based on this assumption can anyone help me answer the below queries

will t-test works better than z-test when we take sample size >=30, if not what should be ideal sample size to be taken.
Do we need to carry out hypothesis tests before carrying out prediction like regression - to ensure that samples taken are coming from the same population.  If not, when these tests are carried out.
Regression requires normality of the residuals. If the residuals are not normally distributed , do we take one more sample until residuals attain normality? Are they any other pre-requisites for regression.
Do we go with Box and whisker plot only to identify outliers.If they are outliers are we going with median and if they are no outliers we go with mean? If not, do we have any other measure
For z-test, we plot Histogram to check if the plot is normally distributed or not. Similarly do we have any plot that we can visualize for t-test. If yes, what do we need to look into the plot to make inferences
Answer: First of all: I assume you are asking about testing population mean with z-test or t-test.
Second: t-test is not 'preferred' over z-test. For z-test one have to know the population variance, for t-test it is not needed. So with know population variance you can use z-test, otherwise t-test (if all the other assumptions are met: random sample, normality of the sample mean).

No, it won't work better or worse. As sample size increased, Student's t distribution converges to normal distribution. That is why it is said for big samples you can use z-test instead of t-test even if the population variance unknown. By big sample someone means $n > 30$, someone means $n > 100$. But this is just rule of the thumb for calculations carried out on paper (more detailed tables are available for standard normal distribution, easier to use). Nowdays it doesn't matter with a computer.

Yes, linear regression have some assumptions (iid observations, exogeneity, homoscedasticity, normality, no multicollinearity, etc.), you have check them prior fitting the model.

Methods for dealing with non-normality: data transformation, reverse data transformation, applying rank based non-parametric methods, bootstrapping, trimming, winzorize, etc. Other pre-requsities: see point 2.

No, you do not use median or mean for outliers, that is a simple way to deal with missing values, not outliers. A simple way to handle outliers is to skip them, although it is not the best practice. Another common way to deal with them is winsorization.
And yes, there are other measures for outliers: e.g. Grubbs' Test, Tietjen-Moore Test, Generalized Extreme Studentized Deviate (ESD) Test, etc.

As both z-test and t-test assumes the sample mean follows normal distribution, it is common way to check the histogram. Although it is not needed that the data itself normally distributed, but in that case the assumption is met.


Open and closed sets in $\mathbb{C}$

Question: I've just started with complex analysis and have some questions about closed and open sets in the set of complex numbers. I would be glad if somebody could help me or give me a little hint how to move on.
The exercise is to determine of the following sets $M$, $N$ and $K$ are open/closed and to find their closure, interior and boundary.
1) $M=\left \{ z=x+iy \mid x\leq 1, y\leq 1 \right \}$
Here i think the set is closed, its closure is the set itself, interior is $M=\left \{ z=x+iy \mid x<1, y<1\right \}$ and the boundary $M=\left \{ z=x+iy\mid x= 1, y=1 \right \}$.
2) $N=\left \{ z\in \mathbb{C}\mid \lvert z\rvert \leq 1, Im(z)>0 \right \}$
Here i think this must be the closed half circle in the complex plane. The closure is the set itself, the interior is the open half circle $N=\left \{ z\in \mathbb{C}\mid \lvert z\rvert < 1, Im(z)>0 \right \}$, the boundary $N=\left \{ z\in \mathbb{C}\mid \lvert z\rvert =1, Im(z)>0 \right \}$.
3) $K=\left \{ z\in \mathbb{R} \mid a <z<b \right \}$
Here i am not sure at all...i've been thinking that i can consider $z$ as dense in $\mathbb{R}$, but it's not said that $z$ can be a rational number...
Thank you in advance!
Comment: I thought this is to ask if there is any closed and open sets in $C$...
Comment: 1) is not correct. The boundary is wrong. (Draw a picture, you will see that the boundary consists of two rays, not just a point.)
Answer: 1) The boundary is the union of the rays $\{1+iy, y\le1\}\cup\{x+i:x\le1\}$
You said the border was a single point.
2) The set is neither. You need to have $Im(z)\ge0$ for it to be closed. The boundary is the set you pointed united with a interval of the real line $[-1,1]$.
3) The set is neither. It is an interval on the real line. Its boundary and its closure are the set itself with the endpoints added. Its interior is empty.
Comment: Now it looks right, except for the "$z=$" which doesn't make any sense in this context. I'd just write it as $\{1+iy\mid y\le 1\}\cup\{x+i\mid x\le 1\}$, though.
Comment: Actually, 1) is not correct. (The boundary is wrong.)
Comment: Have one more go at it, this time, you read it the same way as I did at first.
Comment: Thank you for the corrections, @HenningMakholm.
Comment: thank you all for the help! I have a question about the last set: can't we consider the last set as an annulus or this will be wrong?
Comment: No it's not an annulus. An annulus is a punctured ball. It has a non empty interior.

Open and closed sets in $\mathbb{C}$


Regression Confidence

Question: I have the following challenge: I have to create a regression model with multiple independent variables. When the model is put in production, it can happen that I get completely new values in my independent variable, which my model didn't know when it was trained. The predicted value would have a very high error. In that case I would have to acquire new training data (costly) and retrain the model.
-> Are there any approaches to detect a prediction which is not reliable because my model wasn't trained for and I should better retrain the model?
Thanks!
Comment: Could you elaborate on the nature of these variables?
Answer: I might include a preprocessing step that deals with values well outside the range the model was trained on. Hopefully these are very rare (if not, then one might say you should train the model with more representative data).
You might set upper/lower bounds for each variable (e.g., +/- 3sd) and then replace them with NA to ensure the model is 'undefined' for such cases (*for models that require complete feature sets). Alternatively you could replace them with a constant or suitably large/small value, but that kind of hack isn't really ideal.
This basically amounts to outlier treatment you could also build into the training set just to keep everything kosher.
Comment: I was thinking about setting upper/lower bounds for each variable, but I thought there is a better statistical way, or some best practice behind that.
Of course in an ideal world I would use more data, but imagine that in some cases acquiring training data is very expensive, I would only want to get more training data if absolutely necessary. So I thought there is some way that tells me "Be careful the confidence of that prediction is very low, because I have never seen data in that range, better get more training data in this area"
Comment: @MikeHuber Alternatively just let the model do its thing. Even extreme values can be handled for well described models.
Comment: Okay but this is just an assumption. If new data points arrive they could have a different relationship to my target value. One idea I had is to calculate the distance every time a make a prediction to my closest training data points to get an idea if I had enough training data for the prediction. But I thought that would be a common problem with some established approaches?

Regression Confidence


Performance -- check for no rows or just perform *multiple* update statements?

Question: I have a stored procedure that updates 2 separate tables based on 1 condition.  In order to avoid a race condition, I am first recording the primary key of the rows that need to be updated in a temp table and then updating that table plus another table (foreign key reference) from that temp table.
Over 99% of the time that this is run there will be 0 rows to update.  Thus from a performance perspective I'd like to know if it would be worth checking for any rows before performing updates on no rows.
select distinct matters
into #matters_t
from mattersqdefaultprocess
where reopen is not null
   and reopen = 'Y'

update m
set status = 'Open'
from matters m
inner join #matters_t mu on mu.matters = m.matters
where m.status in ('inactive', 'closed')

update mdp
set reopen = 'N'
from mattersqdefaultprocess mdp
inner join #matters_t mu on mu.matters = mdp.matters

It seems logical to me that it would be beneficial to check for no rows if there are, say 50 update statements.  For only 2?  That is the question, I suppose.
Perhaps I should just do this:
if 0 < (select count(*)
        from mattersqdefaultprocess mdp
        where qdefprocreopen is not null
           and qdefprocreopen = 'Y')
   begin
      --do the temp table seeding and updates here
   end

Comment: Inspect the execution plan for the case when there are no rows to evaluate and compare that to the check you are considering adding. If the check takes less time to run then its beneficial.
Answer: In the general absence of any answers or discussion, I went ahead and set it up with the if statement and everything (including creating temp tables) else inside the true part of the if statement.


Disord.py - How to list all servers the bot is in?

Question: I'm trying to make it so my bot when it comes online it will check for every server the bot is on and create a folder for them in a directory but i keep getting the error list2 is not defined even though iv defined it here's the code:
import discord
from discord.ext import commands
import os

PREFIX = "$"
bot = commands.Bot(command_prefix=PREFIX, description="Hi")

list1 = os.listdir('C:/Users/User/Desktop/BOT_FOLDER')
for guild in bot.guilds:
    print(guild.name)
    list2 = guild.name

print(os.listdir('C:/Users/User/Desktop/BOT_FOLDER'))

set1 = set(list1)
set2 = set(list2)

missing = list(sorted(set1 - set2))
added = list(sorted(set2 - set1))

print('missing:', missing)
print('added:', added)

newpath = r'C:\Users\User\Desktop\BOT_FOLDER\{}'.format(added)
if not os.path.exists(newpath):
    os.makedirs(newpath)

if os.path.exists("demofile.txt"):
  os.remove('C:/Users/User/Desktop/BOT_FOLDER/{}').format(missing)
else:
  print("The file does not exist")

bot.run('BOT_TOKEN_HERE')

Comment: list2 is local variable and is only visible inside that for loop, make it global
Answer: Actually the interpreter is right! Even if it doesn't seem like it, list2 isn't defined, when you run set2 = set(list2).

Why this is the case
You're bascially calling for guild in bot.guilds right after the initialization of your bot object. However, it hasn't connnected to the API yet, which means it hasn't loaded things like bot.guilds. This attribute will be None, therefore your loop doesn't run and list2 won't get defined.

How to fix this
Wait until your bot is successfully connected to the API, and then iterate over your guilds. This can be done by using the on_ready() event. Also, you need to .append() items to a list, you can't just assign new items with the equal sign.
@bot.event
async def on_ready():
    list1 = os.listdir('C:/Users/User/Desktop/BOT_FOLDER')
    list2 = []
    for guild in bot.guilds:
        print(guild.name)
        list2.append(guild.name)

    print(os.listdir('C:/Users/User/Desktop/BOT_FOLDER'))

    set1 = set(list1)
    set2 = set(list2)

    missing = list(sorted(set1 - set2))
    added = list(sorted(set2 - set1))

    print('missing:', missing)
    print('added:', added)

    newpath = r'C:\Users\User\Desktop\BOT_FOLDER\{}'.format(added)
    if not os.path.exists(newpath):
        os.makedirs(newpath)

    if os.path.exists("demofile.txt"):
        os.remove('C:/Users/User/Desktop/BOT_FOLDER/{}').format(missing)
    else:
        print("The file does not exist")