How to start playing by ear/improvising?

Question: Up till now I've been playing music from the sheet, and eventually I've grown frustrated how after 3 years of playing I can't even play Jingle Bells if I'm asked to without figuring it out from a sheet first. I know some theory but I guess not enough because I've no idea how to even start getting myself 'unchained' from the sheet and just making music freely. I'd greatly appreciate any suggestions because music was always very important to me and now I'm beggining to hate it because I can't progress any further.
Thanks!
Answer: I started by learning to recognize intervals with some ear-training software. This sort of practice is quite frustrating at first and you'll make lots of mistakes. The error rate goes down quite gradually, but you do get better over time. It's best to do it a few minutes a day, and don't ramp up the difficulty too quickly. Singing or humming each interval before trying to recognize it really helps, so I recommend it even if you're not all that interested in being a good singer.
After some of that, you should be able to figure out the intervals for simple tunes like "Jingle Bells" that you already know how to sing. (Again, expect to make lots of mistakes and improve slowly. In a song, you will want to figure out where each note is compared to the tonic.)
The next step is to learn melodies that you don't know well from recordings of music you're interested in. It's useful to have software that lets you repeat a few bars of the song so you can work on it until you get it. Again, you want to learn to sing or hum each section before playing it, because your voice is an essential part of recognizing it.
I'm still learning this and that's as far as I got. There are some other good answers in the "Related" list on the side. For example:
What are the most effective ear training methods?
Comment: could you recommend some software tools?
Comment: Can you share that ear-training software that you have been practiced?
Answer: Sheet music is fine for those first few years, but there's nothing worse than an 'experienced' musician who can't improvise.
The quickest and most fun way to learn how to improvise is this:
1) Learn the pentatonic scale. Play it up and down and all around, all day long.
2) Learn some licks that use the pentatonic scale. There are billions and they are easy to pick up by ear. Ask if you need some help finding some.
3) Once you know about three licks in the same key, find a song that doesn't have too many chord changes and is in a key you can play. Start trying to play the pentatonic scale licks on top of the song. Try to string them together in a way that's fluid and musical. It doesn't have to be complicated. You will find that the pentatonic scale will "just work" over many pop and rock songs, but keep an ear listening for which notes do or don't work well against different chords in the song. (Later you can learn why some notes work better than others, but it's important to not get bogged down at first - play like a three-year old child would).
Once you play these licks in a variety of sequences and rhythmic locations, with occasional ventures into variations you haven't specifically practiced, then you are improvising. Come back and ask us for the next step!
Answer: People have different ways of thinking about music, so don't beat yourself up too much. I suspect there are many wonderful musicians who can't play by ear or improvise.
However, here's how I would start with the jingle bells problem. I'm assuming a piano here.

Try to play Jingle Bells monophonically, in C major using only the white keys. Do it by trial and error. If you find you need a black key, then you've started on the wrong note. Start on a different note. Think about when you're playing a C. That's the tonic. Think about how the tonic sounds; the third; the fifth -- important intervals
You can accompany yourself with the chords C, F and G. "The three chord trick". At first just play them as block chords, e.g. for the C chord, plonk down on C, E, G. Trial and error again. Hopefully you can hear when there should be a chord change. Try one of the three chords, until it sounds right. If you get bored of block chords, arpeggiate, or whatever. You can go by what sounds right, but once you've found it, reason about the chord you've chosen. Which notes in the melody match the notes in the chord? Which notes do not? Are those notes on the beat? Long or short? Are they a deliberate discord? etc.
Now try transposing it to different keys. If you've not practised any scales yet, now would be a good time to start. Learn scales in the order they're introduced in a beginner's piano book, and try to play Jingle Bells by ear in those scales in the same order.
By now, you're probably not really playing Jingle Bells by ear any more -- you're playing it by memory. Try playing folk/blues/rock/pop standards. "Go Tell Aunt Dinah". "Louie Louie" and so on. Anything that fits those three chords. One snag: you might not yet know how to tell something's going to fit the three chords. If there's someone to ask, go ahead. Otherwise, just try, and if there's a spot in the song where none of the three chords fit, ditch that song for now. Concentrate on the low hanging fruit, until you can play a three-chord song by ear, with block chords, reasonably easily.
Add in chords. Those songs ditched because they had a fourth chord -- bring them back into the mix, and try and find the right one. Stick with the simple folk/blues/rock/pop standards, you're hoping to find ones with just one new chord. In the key of C, you're most likely to next encounter an Am but it could be anything. At this simple level, there are really only 24 chords to try (all the majors, all the minors), and you can narrow that down to the seven that can be played on the white keys. It's OK to cheat and look in a book/web page, but try to learn from that for next time.

And really, it's up to you where you go from there, and it depends on the way your mind works.
You might find that you're most comfortable stringing chords together, and noodling on top using the intervals you've put into muscle memory -- not quite knowing how it will sound, but knowing it will sound OK.
You might find that you can hear something in your head, and be able to play it the way you hear it.
You might not be the type of person who naturally falls into either of these, but you'll be able to contextualise the sheet music you play ("I see, Chopin has me playing a Bb over a Gm chord because that's the third in Gm")
Answer: The two main things you need to practice are theory and ear training.
For theory I suggest you start with learning scales, intervals, chords and chord progressions.
For ear training you should practice transcribing intervals and simple rhythms first, and eventually chord progressions and complete songs.
Answer: An important way to improvise is to improvise on the melody by singing it.  Loop the melody over and over on your music player and sing with it.  Harmonize with it, figure out cute figures that fit the spaces in the music, stuff like that.
Once you can improvise when you sing, then play just the chords of the music and sing along with that, improvising as you go.
When you're comfortable with playing chords and improvising by singing, then record yourself singing with the chords.  Then loop your recording and with just your right hand, play along with yourself singing.
When that's comfortable, bring your chords back in and play the melody that you're singing.
Jazz piano players use this all the time. In jazz recordings, you can hear Thelonious Monk and Keith Jarrett singing along with themselves.  They don't sing well, but that's not the point.  The point is to really get inside the melody and play with it.
Good luck!
Comment: It always boggled my mind how out of tune master musicians can sing when they really get into it... Jarrett goes crazy if someone in the audience sneezes, but he can very well almost ruin the recordings with his noises. Anyhow, good answer, it's important to not get too stuck with theory when it comes to improvising.
Answer: I have the SAME EXACT problem: I play music by sight, not by ear.
What I usually do when I'm told to play a melody or a song, I play the note in my head. I ask myself "is this a high note, or a low note, or somewhere in-between?" Then, I would make an educated guess of which note it could possibly be, and play it. From there, I would determine if it's the right pitch or not, and keep playing each note that it could possibly be until I get the note. For the next note, I ask myself "is this note lower, higher, or a repeated note?" I try each pitch, one by one, making educated guesses, until I get the note.
I know that this process can be VERY time-consuming, but it does help. After doing this many times, and I know this from experience, the process gets faster. Always remember to take your time, and don't listen to other people. It's not about how fast you can do it, it's all about getting the right pitches and the final product. The more you do it, the easier and FASTER you can do it.
Hope this helped!
Comment: I'd say you've nailed it -this is the very basic stuff people need to know when learning to play by ear. I have the exact opposite problem: I can only play by ear (but quite good at it)- and this is how I started :-)
Answer: I have the exact opposite problem: I can only play by ear. Actually for me it's not a problem because the kind of thing I play (rock / pop guitar) is generally played by ear anyway.
One axiom which helped me enormously when starting on this journey (given I was new to music generally):

The notes are all on the instrument somewhere!

Your'e already aware of that, of course, but for one given note there are only 12 others to choose from before the pattern repeats (per octave) so choosing the next note for a melody is kind of a limited choice.
When I started learning guitar, I found a song or two where the chords are picked on the strings individually. Using a bit of logic like "They played it on a guitar, ands it's a simple song, so the notes are on here somewhere and they're easy to find", I managed to find what was being played, and picked out the basics of a chord. To play other chords, I applied the same thing just at a different starting point. On guitar, moving a chord about is easy: Just move your starting point up/down the fret until it's changed the right amount. (That assumes no open strings). To play other chord shapes, find the same sequence of notes but starting on another string.
Then there's the melody: I think the answer from thatweirdpandanextdoor covers that very well so I shan't add to it.
For gluing it all together, I found a few other tunes which have 'known obvious intervals' (having sussed them on guitar) which if I hum the tune to myself, I know this part is a 5th or that part is a minor 3rd, or whatever. It's then possible to compare that to a song I'm trying to learn to see any similarities.
I've managed to learn popular tunes, some quite intricate, and improvise quite successfully without paying much attention to music theory. It's more that I've established for myself a system of how things seem to fit together, then I look at places like the magnificent StackExchange and find that others have already sussed it all out and have got names for it. I guess that won't be the same for you as you can already read music (which, from my point of view, seems like an amazing skill).
Finally: I can understand your frustraton. Learning to play (by ear) such that I can fumble a tune out without much/any practice was probably the best thing I've ever done- I hope it works out well for you !
Answer: An easy way of learning to improvise is this:

Learn a scale such as the C major scale or A minor pentatonic scale (can be used to play the blues)
Find a backing track in this key (try YouTube) such a C major or blues in A
Play the scale over the backing track and pay attention to what sounds good

If you do this, congratulations you have done your first solo!!!
If you do this after time you will be able to pick out melodies such as twinkle twinkle or happy birthday with relative ease.
So lets go through a couple of examples using a blues scale, and a major scale over a blues backing track, a pop backing track and a jazz backing track (don't let the word jazz scare you for this one, it's worth trying).
Blues solo:

Use the A minor pentatonic scale (minor or major will both work, just use your ears and try both)
Find a blues backing track in  A (minor or major will both work, just use your ears and try both)
Play the scale over the backing track

Jazz solo:

Use the C major scale
Find a jazz backing track in C (major)
Play the scale over the backing track

Pop solo:

Use the C major scale
Find a backing track in C (major)
Play the scale over the backing track

Try all these and maybe mix them up!!!
Once you have done this, you can start advancing your music theory by reading or just trying out different scales over different backing tracks and using your ears to see what you like!
Good luck.
Additional notes:

I like to record myself so I can listen back to it and find the bits I liked best and write them down or something
For pentatonic scales, it doesn't matter if you use the minor or major pentatonic scale over a major or minor progression
Try the different scales over the different backing tracks and see if you like the sound of any of them. A minor pentatonic, C major pentatonic and C major scales will be able to be used over the examples.
If you play the piano or something and can accompany yourself then you can learn the chord progressions that the backing tracks use to accompany yourself
And the key to ALL of this is: What matters is if it sounds good or bad to yours ears!!!


Laravel multitenancy com um BD só para usuários

Question: Tenho pesquisado sobre Saas e Multi-tenancy mas tem pouco material em português. Vi que é possível fazer um banco de dados para cada empresa, mas a minha dúvida é:
Da pra fazer um sistema web único para todos os clientes(usando PHP+Laravel+MySQL) com um Banco de dados pra cada empresa MAS fazer um Banco de dados só de "empresas"(contendo uma tabela "users" com os campos de empresa, usuário, senha, e bd_empresa) usando ESSE BD de empresa somente para verificar o login e se os dados estiverem certos logar na aplicação usando o BD daquela empresa?
Não sei se consegui deixar claro, espero que sim.
Comment: Sua pergunta parece bastante clara, mas também muito ampla. Não sei até onde a comunidade poderá de fato ajudar nesse caso. De modo geral acredito que a sua proposta é viável sim. Na prática podem haver alguns desafios (acho que mais conceituais do que técnicos) que deverão ser tratados especifica e individualmente.
Comment: Dá sim, eu não uso e nem tenho conhecimento em frameworks PHP, faço tudo na unha, em PDO, mas no meu sistema é basicamente assim, cada empresa tem um banco de dados específico, e compartilham de um banco de dados comum, no banco específico, contem as tabelas de cada uma, e no banco em comum contem informações de acesso público e exclusivo do sistema, tanto para login, quanto para exatamente da mesma forma buscar o banco de cada uma.
No momento em que a empresa executa o login, faz a busca pelo BD dela, e essa informação é passada para o metodo construtor da classe
Answer: Tem como fazer sim e tem varias abordagens.
1 -  Usar a biblioteca multi-tenant
https://github.com/hyn/multi-tenant
2 - Fazer um clase que vai servir de middleware e sempre fazer as requisições no banco da empresa correta.
3 - Distribuir a aplicação e deixar um serviço responsável por gerenciar os bancos de dados e suas consultas
de qualquer forma vai muito do que você precisa fazer
Answer: Eu tenho dois sistemas SaaS desenvolvidos em Laravel, e utilizo um único banco de dados.
Para a separação dos dados por empresa, todas as tabelas possuem a chave estrangeira referenciando a tabela de empresa contratante.
Para auxiliar o código, na verdade de forma bem transparente, utilizo este pacote que usa traits para filtrar sempre pela empresa que o usuário autenticado pertence:
https://github.com/torzer/awesome-landlord
Existem ainda outras alternativas que trabalham com a solução de vários bancos de dados, uma por tenant, e também que permitem segregar e gerenciar os tenants por subdomínio, as mais robustas que analisei para usar estas abordagens, em relação à estabilidade e contínuo desenvolvimento são:

Laravel Tenancy (antigo Hyn Tenancy) - https://tenancy.dev/
Spatie Laravel Multitenancy - https://github.com/spatie/laravel-multitenancy
Tenancy for Laravel - https://tenancyforlaravel.com/


What is the difference between belief propagation and loopy belief propagation?

Question: Belief propagation (BP) is an algorithm (or a family of algorithms) that can be used to perform inference on graphical models (e.g. a Bayesian network). BP can produce exact results on cycle-free graphs (or trees). BP is a message passing algorithm: messages are iteratively passed between nodes of the graph (or tree). The marginal probability of a random variable (or node) is then estimated from these messages.
In the case the graph contains loops (or cycles), the algorithm is called loopy belief propagation and it is not an exact inference algorithm (that is, it doesn't produce exact results).
What is the difference between belief propagation and loopy belief propagation, in terms of operations that they perform in order e.g. to find the marginal probability of a random variable? How do BP and loopy BP differ?
Answer: The main difference I think isn't so much in terms of how BP operates vs How Loopy BP operates (although there are small differences). It is in the fact that once the graphs start having cycles, the complexity of the problem being solved jumps from the class P (considered easy to solve form a computational point of view) to the class NP (considered hard to solve - see the P vs. NP problem for further details). That's why Loopy BP isn't an exact inference algorithm: Finding an exact solution is very difficult (can take exponential time).
You might want to check out Generalized Belief Propagation and Survey Propagation which also address graphs with cycles.
Answer: The key differences in the implementation of the algorithm / in the passing of the messages are as follows:

messages in BP are passed only once; in contrast, messages in loopy BP (LBP) are passed iteratively until "convergence is reached" (convergence means that the belief does not change anymore from one iteration to the other).
in a Bayesian Network, messages in BP are passed from "leaves" to "roots", i.e. upwards in the Bayesian Network; in contrast, messages in LBP don't need to be passed in a particular order.

Other than that, the equations to calculate the belief and the messages are completely identical.
Comment: I was looking for some answers to the same question, lucky to see this post. Can you provide any links to numerical examples of belief propagation and loopy belief propagation?
Comment: I describe LBP and BP in one of my papers, which also comes with an R implementation: https://arxiv.org/pdf/2112.09217.pdf


Direct sum of two vector spaces

Question: Can someone point out where is the fallacy in my proof below?
Claim: If $U_1,U_2,W$ are subspaces of a vector space $V$ and  $V = U_1\bigoplus W = U_2 \bigoplus W $, then $U_1 = U_2$
My counter example: Let $V = R^2$, the 2-dimensional space.
$W$: the x-axis, $(y=0)$.
$U_1$: the y-axis, $(x=0)$.
$U_2$: the straight line $x=y$.
Then
$(1): V = W + U_1$, because coordinate wise sum of $(x,0)$ and $(0,y)$ will span whole of $R^2, (x,y)$,
$(2):$ any $v\in V$ can be uniquely represented by a unique element in $W$ and an unique element in $U_1$. $(x,y)=(x,0)+(0,y)$
So $V=W \bigoplus U_1$
Also,
$(1): V = W + U_2$, because coordinate wise sum of $(x,0)$ and $(y,y)$ will span whole of $R^2, (x,y)$,
$(2):$ any $v\in V$ can be uniquely represented by a unique element in $W$ and an unique element in $U_2$. $(x,y)=((x-y)+y,y)=((x-y),0)+(y,y)$
So $V=W \bigoplus U_2$
But, as we see, $U_1 \neq U_2$
My proof: $V=U_1\bigoplus W\Rightarrow \forall v\in V, \exists$ a unique $u\in U_1$ and $w\in W$, such that $v=u+w$.
Let us consider a specific $v_1=u_1+w_1......(1)$
$V=U_2\bigoplus W\Rightarrow \forall v\in V, \exists $ unique $u\in U_2, w\in W$, such that $v=u+w$.
With respect to the specific $v_1$ and $w_1$ in equation $(1), \exists$ a unique $u_2\in U_2$, satisfying $v_1=u_2+w_1.............(2)$
Comparing equations $(1)$ and $(2)$, we get $$u_1+w_1=u_2+w_1$$
Adding $(-1)w$ to both sides, we get
$$u_1 + w_1 -w_1= u_2+w_1-w_1$$
$$\Rightarrow u_1+0 = u_2 +0$$
$$\Rightarrow u_1 = u_2.........(3)$$
Now, for any $u_1\in U_1, \exists $ unique $v_1, w_1$ and $u_2$, satisfying equation $(1)$ and $(2)$. And $u_1=u_2\in U_2$. In other words, $u_1\in U_1\Rightarrow u_1\in U_2$. Therefore $U_1\subseteq U_2.......(4)$
Similarly, for any $u_2\in U_2$, from equations $(1),(2) $ and $(3)$, we can show $u_2\in U_2 \Rightarrow u_2\in U_1$. Therefore $U_2\subseteq U_1.......(5)$.
Hence, from equations $(4)$ and $(5)$, we conclude $U_1=U_2$.
Answer:
Comparing equations (1) and (2), we get u1+w1=u2+w1

There's no reason to say that the $w$ you get from $U_1 \oplus W$ is the same as the one you get from $U_2 \oplus W$.

Direct sum of two vector spaces


Giving permissions to Virtual Service Accounts on domain controllers

Question: The service I'm implementing will run on a domain controller, so I'd like it to have minimal privileges.
Ideally, it would simply run as Local Service. However, it needs to be able to:

monitor performance counters (be a member of Performance Monitor Users)
manage performance counters, logs and alerts (be a member of Performance Log Users)
read event logs (be a member of Event Log Readers)

Adding Local Service to these groups is obviously not an good approach.
Running the service as the Virtual Service Account generated for it will allow it to access the network with the computer's identity which is also undesirable. So I'd like to run it as Local Service with non-zero SID type, thereby passing onto it privileges given to the VSA.
I'm having trouble adding the service's VSA to the groups stated above. I suspect it is because the VSA is local (and exists only within the domain controller), while the groups are domain groups. Is it possible?
Group Managed Service Accounts may prove to be useful (replacing the VSA), be they need to be created manually.
What's the correct approach to set up a service to run with only the specified permissions, while the deployment has no prerequisites (no GMSA creation)?
Answers specific to specified groups are also welcome.
Answer: On a domain controller, you don't have a "local accounts" database, and you don't have local server groups.
Local groups are called "Domain Local" and can be used anywhere within the domain (in keeping with domain rules on how groups can be nested).  AD is the DC's "local account database".
The best solution here is to NOT run it on the DC itself.  The SYSTEM context on a DC has full access to everything in AD.  That's a possible security breech waiting to happen.


Affine vs Projective curves?

Question: Let $\mathbb A^n_k$ be affine $n$-space and let $\mathbb P^n_k$ be the projective $n$-space.
An affine curve is a set of the form $$C_f(k):=\{(a_1, \ldots, a_n)\in\mathbb A^n_k: f(a_1, \ldots, a_n)=0\}$$ where $f\in k[x_1, \ldots, x_n]$ is non-constant.
A projective curve is a set of the form $$P_f(k):=\{(a_1:....: a_{n+1}): f(a_1, \ldots, a_{n+1})=0\}$$ where $f\in k[x_1, \ldots, x_{n+1}]$ is non-constant.
Every $f\in k[x_1, \ldots, x_n]$ produces an homogeneous polynomial ${}^hf\in k[x_1, \ldots, x_{n+1}]$ via $${}^h f(x_1, \ldots, x_{n+1}):=x_{n+1}^{\textrm{deg}(f)}\cdot f(x_1/x_{n+1}, \ldots, x_n/x_{n+1}).$$ Conversely, every homogeneous $f\in k[x_1, \ldots, x_{n+1}]$ induces a polynomial ${}^d f\in k[x_1, \ldots, x_n]$ via $${}^d f(x_1, \ldots, x_n):=f(x_1, \ldots, x_n, 1).$$
There are two natural questions:
1) Given $f\in k[x_1, \ldots, x_{n}]$, how does $C_f(k)$ relate to $P_{{}^h f}(k)$?
2) Given $f\in k[x_1, \ldots, x_{n+1}]$ homogeneous, how does $P_f(k)$ relate to $C_{{}^d f}(k)$?
I presume it works as follows:
There is an inclusion $$\imath: \mathbb A^n_k\hookrightarrow \mathbb P^n_{k}$$ given by $$\imath(a_1, \ldots, a_n):=(a_1: \ldots: a_n: 1).$$
Then:
Answer to 1) $\imath(C_f(k))=\imath(\mathbb A^n_k)\cap P_{{}^h f}(k)$. In particular, there is an identification $$C_f(k)\simeq \imath(\mathbb A^n_k)\cap P_{{}^h f}(k).$$
Answer to 2) I conjecture that $$P_f(k)\simeq C_{{}^d f}(k)\sqcup \{\mathcal{O}\}$$ where $\mathcal{O}\not\in C_{{}^d f}(k)$.
Are my answers to 1 and 2 correct?
Maybe my answer to 2) doesn't work always. I guess there must be a condition we must impose on $z_{n+1}$. In that case, what would it be the correct answer to 2)?
Thanks.
Comment: For it to be a (union of affine ie. irreducible) curves you need $n=2$, for $n=3$ it is a surface. And in general your $\{O\}$ is finitely many points, not only one (the remaining points of the projective completion of $f(x,y) = 0$ with $f(x,y) \in k[x,y]$ are the roots $(0,z_j)$ of $g(w,z)=w^a z^b f(1/w,1/z)$)
Comment: Thanks a lot =)
Comment: Do you have any reference which discuss that @reuns?
Comment: For $k$ an algebraically closed field and $(f(X,Y))$ a non-zero prime ideal of $k[X,Y]$ then $V((f))=\{ (x,y) \in k^2,f(x,y) = 0\}$ is an affine curve, you can take $x \in k$ as you want there are always finitely many solutions for $y$. If $(F(X,Y,Z))$ is a non-zero prime ideal of $k[X,Y,Z]$ then $V((F))$ is a surface, you can take $x,y \in k^2$ as you want there are always finitely many solutions for $z$. It corresponds to the number of transcendental elements needed to generated the function field $Frac(k[X,Y,Z]/(F(X,Y,Z)))$
Comment: @reuns Can I say that $P_f(k)\simeq C_{{}^d f}(k)\sqcup P_{f_0}(k)$ where $f_0(x_1, \ldots, x_n):=f(x_1, \ldots, x_n, 0)$? Or even better, $P_f(k)\simeq C_{{}^d f}(k)\sqcup C_{{}^df_p}(k)$ where ${}^d f_p$ is the homogeneous component of degree $p$ of ${}^d$ (where I'm supposing $\textrm{deg}(f)=p$)?
Answer: Subtle issues arise when you do this kind of homogenizing and dehomogenizing. It's not so straightforward. In my opinion the best introductory explanation of all this can be found in the brilliant book Ideals, Varieties, and Algorithms. Go through chapter $8$ section $2$.
Comment: Thanks, I'll check it out?

Affine vs Projective curves?


Android: Generating Elliptic Curve KeypPair

Question: I am trying to implement key generation based on Eliptic Curves Diffie Hellman. Now I wanted to specify my Eliptic Curve by myself, so I took the parameters  from here.
The code which I wrote to implement this:
public void createKey(){

        // base point (generator???)
        BigInteger x = new BigInteger("2fe13c0537bbc11acaa07d793de4e6d5e5c94eee8", 16);
        BigInteger y = new BigInteger("289070fb05d38ff58321f2e800536d538ccdaa3d9", 16);

        // the order of generator
        BigInteger n = new BigInteger("5846006549323611672814741753598448348329118574063", 10);

        // curves coefficients
        BigInteger ab = new BigInteger("1", 2);

        // curves cofactor
        BigInteger h = new BigInteger("2", 10);

        // exponents of the equotation
        int[] ks = {7, 6, 3};

        ECFieldF2m ecField = new ECFieldF2m(163, ks);
        // Elliptic curve
        EllipticCurve ec = new EllipticCurve(ecField, ab, ab);

        // GENERATOR POINT
        ECPoint g = new ECPoint(x, y);

        // Parameter specs?
        ECParameterSpec ecps = new ECParameterSpec(ec , g , n, h.intValue());

    try {
        // get keypair
        KeyPairGenerator kg = KeyPairGenerator.getInstance("ECDSA");
        kg.initialize(ecps, new SecureRandom());
        KeyPair kp = kg.generateKeyPair();
        Log.d("SECLIENT"+type,kp.getPublic().toString());
        Log.d("SECLIENT"+type,kp.getPrivate().toString());
    }catch (Exception e){
        e.printStackTrace();
    }
}

At this point, the code compiles. However it fails in the try_catch block where it has to generate the KeyPair.
The Error Message is:
java.lang.RuntimeException: Unable to create EC KeyFactory: unhandled field class java.security.spec.ECFieldF2m
Does anyone have this problem? How can I solve this?
Comment: Have you tried with an Fp curve? Or is that outside requirements? Oh, and welcome to SO, hell of a question to start off with.
Answer: I didn't come across the same exact problem you are having butsince you are using EC in Android and having a weird RuntimeException, I'll leave you my 2 cents. Hope it helps.
I had another weird error (OEM related)
java.lang.RuntimeException: error:0f06707b:elliptic curve routines:EC_GROUP_new_by_curve_name:UNKNOWN_GROUP

while trying to use
KeyAgreement.getInstance("ECDH);

What happened to me was that I was adding SpongyCastle as a SecurityProvider, doing this:
Security.addProvider(new BouncyCastleProvider());

And for some devices everything worked as expected but in others some device (namely LGE Nexus 5) I got the exact same exception.
What fixed it for me was changing the way I add SpongyCastle and doing:
Security.insertProviderAt(new BouncyCastleProvider(), 1);

Comment: Also, Why do you say you are trying out Diffie-Hellman? You are using `"ECDSA"` which is for Digital Signatures. Diffie-Hellman is for secure key exchanges.


Steps to debugging a PCI-Cardbus bridge on Linux

Question: I have a PCI to PCMCIA adapter card that's installed in a Dell server running Linux 2.6. The documentation indicates that it works with both 3.3V and 5V cards, and with both 16-bit PCMCIA and Cardbus. When I insert a 16-bit PCMCIA card, the card is recognized, appears in /sys/bus/pcmcia, and everything works. When I insert a Cardbus card, dmesg reports "pccard: CardBus card inserted into slot 0" and nothing else happens. The card isn't visible in lspci or /sys/bus/pci, the driver isn't loaded, and there are no resource allocation errors or any kind of weirdness in dmesg, just nothing.
pccardctl info shows this:
PRODID_1=""
PRODID_2=""
PRODID_3=""
PRODID_4=""
MANFID=0000,0000
FUNCID=255

but as I understand it, this is to be expected for Cardbus cards in 2.6? I think?
The adapter is a "Creative I/O" model, which is a no-name, but the chip is a Ricoh RL5c475 (PCI ID 1180:0475), which I thought was supposed to work quite well in Linux. The problem is the same on 2.6.18 and 2.6.26 (both Debian builds). I can try 2.6.30 from backports or 2.6.31 custom built but only if there's some good reason to expect an improvement.
Does anyone have suggestions for next steps, or should I just try replacing the adapter?
Additional info: the card works when I install the adapter in a Windows machine. Different hardware, though -- I don't have a PE2850 with windows.
Answer: Problem Resolved
The problem was a PCI bus overlap issue. The BIOS-provided PCI bus numbering didn't leave any space for the Cardbus bus on the other side of the bridge to receive a bus number. This was indicated by the message "Upper limit for fixing this bridge's parent bridge: #0a" in dmesg (unfortunately it wasn't a very insistent message!). I was able to solve the problem by booting linux with the option pci=assign-busses.
Comment: Nasty - how did you figure it out?
Comment: using `lspci -H1 -M` which was a command I found on another website. It maps the PCI bus directly, and it had a more forceful warning about the bus overlap that made me realize yes, that was the problem.


How to format 'node reference' field values from the Views module

Question: How can I format the output of a 'node reference' field that is generated using a View?
Currently it's formatted as (See 2nd graphic): "FirstName - Lastname Pagetitle"
I'm trying to change that to: "Firstname Lastname"
Where do I modify this fields format?
Here is my current view:

Here is the 'node reference' output:
Answer: Interesting. Initially it would appear that the "Pagetitle" is being stored in the Last Name field. However assuming that it's not, you can use the devel_themer module to find out what and where is theming that field's output. The culpript could very likely be inside one of your own custom Views templates or theme function override. If not, devel_themer should point you in the right direction.
Comment: Thank you for this.  devel_themer is an incredible tool!  You absolutely nailed the issue as well. :)  Thank you!


How to set default parameters for a graphical device?

Question: I like to read white on black. So, in R I would do something along the lines of:
par (bg = "black")
par (fg = "ivory1")

I would like these options to be set by default. However, one does not simply write these lines in .Rprofile because, as I understand, at the time it gets executed, the graphical device is not yet initialized. Rather, as suggested in another answer, one should re-assign options()$device to include the necessary option setting. I did not have success in that.

This is what I tried:
~/.Rprofile
f_device <- options()$device

blackdevice <- function (...) {

    f_device(...)

    par (bg       = "black")
    par (fg       = "ivory1")
}

options (device = blackdevice)

The idea here is to save the original device function to another variable, and then call it from my new device function. What I get is:
Error in f_device(...) : could not find function "f_device"

— At the time I run plot (something).

Another idea I had is to go like that:
~/.Rprofile
.First <- function () {

    options(f_device = options()$device)

    blackdevice <- function (...) {

        options()$f_device(...)

        par (bg       = "black")
        par (fg       = "ivory1")
    }

    options (device = blackdevice)
}

— Assigning the original device someplace else in options. But this leads to:
Error in (function (...)  : attempt to apply non-function

I'm out of ideas. Can you help me figure this out?
Answer: One solution seems to be to define a 'hook' that is called when a new plot is created. From the documentation of plot.new:

There are two hooks called '"before.plot.new"' and '"plot.new"'  (see
  'setHook') called immediately before and after advancing the  frame.
  The latter is used in the testing code to annotate the new  page. The
  hook function(s) are called with no argument.  (If the  value is a
  character string, 'get' is called on it from within the  'graphics'
  namespace.)

The following seems to work:
setHook("before.plot.new", function(...) {
  par(bg = "black",
    fg = "ivory1",
    col.axis = "ivory1",
    col.lab = "ivory1",
    col.main = "ivory1",
    col.sub = "ivory1")
})

Comment: Is there a particular reason you choose to call `par` 6 times instead of one time with all arguments?
Comment: @r2evans No, not really, I copied the first two from the question; added the others later to get the axes and labels right. Fixed it now. Thanks
Answer: Your problem here is that the device option is not yet set at the time .Rprofile is processed.
Therefore, options()$device is NULL at that point, which leads to the error you are observing.
The .First() function also gets executed before the packages from options("defaultPackages") are attached, so that gives the same error.
If you want to put something in .Rprofile, it would have to be like:
setHook(
    packageEvent("grDevices", "onLoad"),
    function(...) {
        f_device <- getOption("device")

        blackdevice <- function(...) {
            f_device(...)
            par(bg = "black")
            par(fg = "ivory1")
        }

        options(device = blackdevice)
    }
)

Note that this is a solution for the standard R GUI, but not for e.g. RStudio, which sets its own "RStudioGD" device afterwards (and requires that grDevices is already loaded).
(This also explains the awkward approach (from the question you linked) of re-defining "RStudioGD" in .Rprofile to get it ahead of "tools:rstudio" on the search path)


Como permitir que o usuário adicione uma nova opção num ?

Question: Preciso montar um select onde o usuário possa adicionar novas opções, então preciso que o select permita digitação e um botão ao lado dele faria chamaria a função para incluir na lista.
Como posso fazer isso?
Tenho apenas o select básico com o link que deveria funcionar como botão:
<select class="tags" name="abrangencia" style="width: 300px;">
    <option value="">opt 1</option>
    <option value="">opt 2</option>
    <option value="">opt 3</option>
    <option value="">opt 4</option>
</select>

Comment: Relacionado: https://pt.stackoverflow.com/questions/92187/como-adicionar-option-para-um-select-por-jquery-javascript
Answer: Podes fazer assim, eu aqui verifico se existe alguma coisa digitada para que não adicione uma <option> sem texto:

var input_ele = document.getElementById('new-opt');
var add_btn = document.getElementById('add-btn');
var sel_opts = document.getElementById('sel-opts');
var input_val;
add_btn.addEventListener('click', function() {
  input_val = input_ele.value;
  if(input_val.trim() != '') {
    sel_opts.innerHTML += '<option>' +input_val+ '</option>';
    input_ele.value = '';
  }
});
<input id="new-opt"/>
<button id="add-btn">Adicionar opção</button>
<select id="sel-opts">
  <option>Eu já existo</option>
</select>

Com datalist ficaria:

var input_ele = document.getElementById('new-opt');
var add_btn = document.getElementById('add-btn');
var sel_opts = document.getElementById('sel-opts');
var input_val;
add_btn.addEventListener('click', function() {
  input_val = input_ele.value;
  if(input_val.trim() != '') {
    sel_opts.innerHTML += '<option>' +input_val+ '</option>';
    input_ele.value = '';
  }
});
<input list="sel-opts" id="new-opt"/>
<button id="add-btn">Adicionar opção</button>
<datalist id="sel-opts">
  <option>Eu já existo</option>
</datalist>

Atenção, como explicado no link essa ultima alternativa não funciona em IE <= v9 nem no safari
Comment: muito bom! mas existe algum jeito de tornar a própria combo box digitável sem precisar de outro input do tipo text?
Comment: @rayraisemeup, o que é o combo box, podes mostrar que elemento html é esse sff
Comment: combo box é o mesmo que .. a caixa de seleção com uma seta que ao clicar mostre as opções. Tem como tornar esse elemento digitável?
Comment: Acrescentei uma alternativa @rayraisemeup, pode ser isso?
Comment: Sim! Funciona muito bem, obrigado!


How to diagonalize a transformation from a $2\times 2$ matrix to a $2 \times 2$ matrix?

Question: Say $V = M_{2 \times 2}(\mathbb{R})$, and
$$T\left(\begin{bmatrix} a & b \\ c & d \end{bmatrix}\right) = \begin{bmatrix} c & d \\ a & b \end{bmatrix}$$
How to find the eigenvalues of T and a ordered basis $B$ for $V$ that $[V]_B$ is a diagonal matrix?
My approach is to find the matrix representation of this transfromation, and find the characteristic polynomial. However, I am stuck in finding such a matrix representation.
I guess the matrix would be $4 \times 4$, but it does not make sense for $M_{4 \times 4}(\mathbb{R}) \times M_{2 \times 2}(\mathbb{R}) = M_{2 \times 2}(\mathbb{R})$, if we follow the "definition" of a transformation matrix?
Comment: You should forget about the matrix structure of $V$, and treat it just as any 4-dimensional real vector space. In fact, perhaps it makes more sense to think of $V$ as $\mathbb R^4$. Using your notation, the "equation" $M_{4\times 4}(\mathbb R) \times \mathbb R^4 = \mathbb R^4$ should make more sense. Edit: but, in this case, it might be easier to just *guess* four linearly independent eigenvectors, since the definition of $T$ is sufficiently simple that you could probably do that.
Comment: NB - in this particular case, things are simple enough for you to not even need much machinery - $ \begin{bmatrix} c & d \\ a & b \end{bmatrix} = \lambda \begin{bmatrix} a & b \\c & d  \end{bmatrix}$ immediately gives $\lambda = \pm 1$, since $a = \lambda c = \lambda ( \lambda a)$. Next just squinting at the problem a bit suggests the eigenvectors $\begin{bmatrix} 1 & 0 \\ 1 & 0 \end{bmatrix}, \begin{bmatrix} 1 & 0 \\ -1 & 0 \end{bmatrix}$ and the corresponding ones with the first and second columns switched.
Comment: @stochasticboy321 Thank you, now I see $\lambda = \pm 1$. Can you explain a little bit more how to get the eigenvectors? My intuition stays at finding a kernel of $T - \lambda Id$.
Answer: The operator $\;T\;$ simply changes rows, so;
$$\begin{pmatrix}c&d\\a&b\end{pmatrix}=\color{red}T\begin{pmatrix}a&b\\c&d\end{pmatrix}=\color{red}{\begin{pmatrix}0&1\\1&0\end{pmatrix}}\begin{pmatrix}a&b\\c&d\end{pmatrix}$$
Thus, this operator's action on some basis (say the usual one) of $\;M_{2\times2}(\Bbb R)\;$ isw pretty simple:
$$\begin{align*}&\begin{pmatrix}0&1\\1&0\end{pmatrix}\begin{pmatrix}1&0\\0&0\end{pmatrix}=\begin{pmatrix}0&0\\1&0\end{pmatrix}\\{}\\
&\begin{pmatrix}0&1\\1&0\end{pmatrix}\begin{pmatrix}0&1\\0&0\end{pmatrix}=\begin{pmatrix}0&0\\0&1\end{pmatrix}\\{}\\
&\begin{pmatrix}0&1\\1&0\end{pmatrix}\begin{pmatrix}0&0\\1&0\end{pmatrix}=\begin{pmatrix}1&0\\0&0\end{pmatrix}\\{}\\
&\begin{pmatrix}0&1\\1&0\end{pmatrix}\begin{pmatrix}0&0\\0&1\end{pmatrix}=\begin{pmatrix}0&1\\0&0\end{pmatrix}\end{align*}$$
and thus
$$[T]=\begin{pmatrix}0&0&1&0\\0&0&0&1\\1&0&0&0\\0&1&0&0\end{pmatrix}$$
This is a symmetric matrix and thus diagonalizable. Check it characteristic polynomial is
$$(x^2-1)^2=(x-1)^2(x+1)^2\;,\;\;\text{with eigenvalues}\;\;\pm1$$
each of geometric multiplicity two, and thus its JCF is
$$[T]_J=\begin{pmatrix}1&0&0&0\\0&1&0&0\\0&0&\!-1&0\\0&0&0&\!-1\end{pmatrix}$$
Comment: @JayWong Not exactly: you have to pick a basis, apply the operator to each vector in int and write each result as a linear combination of the basis, and then take the transpose of the coefficients matrix. In this particular case things are very simple...
Comment: @JayWong Exactly...and again: in **this particular case** it doesn't matter the coefficients matrix or its transpose as that's a *symmetric* matrix.
Comment: Thank you! This answer helps a lot. So to find a matrix representation, I just need to think the result of $T(e_1) = \begin{pmatrix}0&0\\1&0\end{pmatrix}$ as $ \begin{pmatrix}0 \\ 0\\1 \\0\end{pmatrix}$, then put it as one column of $[T]$?
Comment: Ok I see now. In this case $\begin{pmatrix}0 & 0\\1 &0\end{pmatrix} = 0e_1 + 0e_2 + 1e_3 + 0e_4$, so take the ranspose of those coefficients we then can get $\begin{pmatrix}0 \\ 0\\1 \\0\end{pmatrix}$


Is there a way to get two songs on a single page using lilypond-book?

Question: I am currently using Frescobaldi to edit my lilypond files. I have several short songs (a half-page or less) that I would like to combine into a set with more than one song on each page.  Currently, the only solution I've found is to use \book part{}, but that creates a single pdf with one song per page. Right now, in a seven-song set, everyone has to layout seven sheets of paper across the table to see everything and it would be nice if I could reduce it to three pages or so.
Answer: You can just put several scores below each other. Each of them can have an own \header. You cannot use title or composer here, but you can use the piece option to give each of these scores a title.
\version "2.19.83"

\header {
  title = "A couple of songs"
  tagline = ##f
}

\score {

  \header {
    piece = \markup { \vspace #'2 \fill-line { \bold \huge "The first song" } }
  }

  \new Staff \with {
    instrumentName = "Soprano"
  } \relative c' {
    c4 d e f g1 \bar "|."
  }

}

\score {

  \header {
    piece = \markup { \vspace #'2 \fill-line { \bold \huge "The second song" } }
  }

  \new Staff \with {
    instrumentName = "Soprano"
  } \relative c' {
    c4 d e f g1 \bar "|."
  }

}

If you need every song to have a title and a composer, it may be easier to just set print-all-headers = ##t in the \paper block (see this link).
\version "2.19.83"

\paper {
  print-all-headers = ##t
}

\header {
  tagline = ##f
}

\score {

  \header {
    title = "The first song"
    composer = "C. Omposer"
  }

  \new Staff \with {
    instrumentName = "Soprano"
  } \relative c' {
    c4 d e f g1 \bar "|."
  }

}

\score {

  \header {
    title = "The second song"
    composer = "C. Omposer"
  }

  \new Staff \with {
    instrumentName = "Soprano"
  } \relative c' {
    c4 d e f g1 \bar "|."
  }

}

Comment: Adding the `print-all-headers = ##t` and stacking scores was exactly what I needed. I'm using different songs, not just different sections, so I was looking to keep the details (i.e., title, composer, meter, etc.).


Probability of passing an exam given two different grading schemes

Question: I had a discussion with a peer about the probability of passing an exam given two different grading schemes and I'm not sure I believe what my peer stated.
They stated that given an exam with 10 questions where a passing grade is 5 correct questions and an exam with 6 questions where a passing grade is 3 questions, it would be better to take the exam with 10 questions as there is an increased probability of passing. I didn't buy the argument as it seems that the exams are equivalent, i.e., you need a 50% to pass either. However my peer was adamant about the their point. Can anyone clarify this?
Comment: My guess is your friends intuition is something like: on a 10 question exam, I stand a better chance of being familiar with at least half of the topics and hence of getting those questions right; on a 6 question exam there is less of a topic spread which risks not having enough familiari
Comment: @TokenToucan I think that may be right. Is there a way to quantify this idea?
Answer: The rightness or wrongness of your peer's statement depends on the probability of success of answering each question correctly.
If you assume that the test is a set of $2N$ true/false questions, with N correct answers required to pass, where your probability of answering any question is $p$, then the probability $P$ of passing the test is such that:
for $p<0.5$, $P$ falls monotonically with increasing N and in the limit of $N {\rightarrow} {\infty}$, $P {\rightarrow} 0$, so it will always be preferential to choose the test with the least number of questions.
for $p=0.5$ the probability of passing still falls with increasing N (but now asymptotes to 0.5), $N {\rightarrow} {\infty}$, $P {\rightarrow} 0.5$, so still choose the test with the least number of questions.
for $0.5<p<2/3$ the probability of passing initially falls with increasing N, but then increases with larger N and in the limit $N {\rightarrow} {\infty}$, $P {\rightarrow} 1.0$, so your choice would depend on the maximum number of questions. For example, if $p=0.51$ then sitting a test with $N\simeq570$ questions is marginally better than sitting a test with $N=2$ questions.
for $p>2/3$ the probability of passing increases monotonically with increasing N, and in the limit $N {\rightarrow} {\infty}$, $P {\rightarrow} 1.0$, so you should always choose the test with the most questions.
In your example, choosing either a 6 question or a 10 question test, your probability of success will be approximately equal if $p\simeq0.564$ (in that case $P\simeq0.7674$), it would be better to do the 6 question test if $p<0.564$, but you should choose the 10 question test if $p>0.564$.
Answer: Your friend is incorrect, assuming you're flipping a coin to determine whether you're right or wrong. You've both forgotten that a score of 0 is possible, which means the chance of passing isn't 50% for either.
On the test with 10 questions, 6 of 11 possible scores pass. On the 6 question test, 4/7 scores pass. If you're tossing coins, then the probabilities of passing are
$$ \frac{1}{2^6}\sum_0^3 {6 \choose k} $$ or $$ \frac{1}{2^{10}}\sum_0^6 {10 \choose k} $$
For the six and 10 question tests respectively. That's 65% for the 6 question test, and 62% for the 10 question test.
It's better to take the 6-question exam if you truly think each question is a coin toss as to whether you're right.
Comment: Not all scores are equally likely. Your method would say a probability of $2/3$ for passing a two question exam when it is actually $3/4$. The difference is one can get just 1 question right in two different ways.
Comment: A good point, if we're using coin flips. Then the correct answer uses binomial coefficients. Will edit.
Comment: Edited, thanks for the correction.
Answer: Answer: it depends, but for a true/false test, your peer is incorrect.
Getting 5 right and 5 wrong is how to order RRRRRWWWWW where R is right and W is wrong. There are $\binom{10}{5} = 252$ ways. Similarly for 6 right, 7 right, etc.
This is $\sum_{i=0}^{5} \binom{10}{i}$, which equals $638$. Dividing by $2^{10}$ gets us approx. $0.623$.
Now, getting 3 right and 3 wrong is ordering RRRWWW. We want to find $\sum_{i=0}^{3} \binom{6}{i}$, which comes out to $42$. Dividing by $2^6$ gets us approx. $0.652$.
$0.623 < 0.652$, so your peer is incorrect.


Change java process description in windows task manager

Question: Is there a command line argument to set the title that JVM reports to the windows task manager? All my java processes only show up as 'javaw.exe' with description 'Java(TM) Platform SE binary'. It would be great, if I could set some -param="This is eclipse" e.g. so that I can easily tell them apart. It would prevent me from frequently killing my IDE instead of, e.g., tomcat.
Answer: You could hack the JVM binary or u could implement a small binary app to invoke the JVM rather than a batch file; beyond this there's not really a pure java way to do it!
Answer: The process in the description column of the Windows task manager is taken from the product name in the version resource that is statically compiled into the executable. An executable has no way to set this description as runtime.
You only option is to create a native wrapper executable that creates a version resource with a configurable name.
exe4j does that, for example. Disclaimer: My company develops exe4j.
Comment: What a pity! Windows is missing a real essential here…


Como enviar várias strings por url no Android studio?

Question: Tenho dados para enviar para um servidor web, o servidor recebe as strings seguindo esse padrão:
http://teste.com/{id}/{latitude}/{longitude}
Como faço para enviar esses dados ocupando esses espaços de uma só vez pelo Android studio?
Answer: Simples:
String url = "http://teste.com/pagina?id=" + id + "&latitude=" + latitude + "&longitude=" + longitude;

Depois do lado do servidor:
$id = $_GET['id'];
$latitude = $_GET['latitude'];
$longitude = $_GET['longitude'];

Esta página do android developer ensina a fazer algo semelhante usando volley, mas eu faria isso usando POST em vez de GET por questões de segurança, já que é enviado pela url pode ter seus dados alterados, mais informações página w3schools.com
Answer: Recomendo muito o uso do retrofit para consumir APIs
interface TesteService{
    @GET("{id}/{latitude}/{longitude}"
    Call<Objeto> getObject(@Path("id") String id,
                           @Path("latitude") String latitude,
                           @Path("longitude") String longitude);
}

public class TesteRetrofit{
    Retrofit retrofit = new Retrofit.Builder()
        .baseUrl("http://teste.com/")
        .build();
    TesteService tService = retrofit.create(TesteService.class);

    public Objeto pegarObjeto(){
     return tService.getObject("123","20°34'07.2\"S","43°48'44.6\"W").execute().body();
    }
}

Esse é um exemplo básico de código, sugiro da uma lida na documentação para saber mais.
Esse código vai lançar a excepção I/O on UIThread se executado na Thread principal do Android


How to put each string that a for loop outputs in a different variable in VB.NET?

Question: so as the title states i want to know how I could have each of the strings given by a for loop that runs three times into a different variable for each.
It's meant to display a list of cards that are randomly generated and I want to get only the cards' values in the variable.
Console.WriteLine("You drew: ")
For count As Integer = 1 To 3
     Console.WriteLine("A " & GetCardValue() & GetSuit())
Next
Thank you.
Comment: Why not use [Array's](https://www.tutorialspoint.com/vb.net/vb.net_arrays.htm) for your problem?
Answer: You can use a List(Of T) to store a variable number of items or an array to store fixed number of items. (Showing both in the same example, assuming that the card value is an Integer)
Dim valuesArray(3) As Integer
Dim valuesList As New List(Of Integer)

Console.WriteLine("You drew: ")
For count As Integer = 1 To 3
    Dim cardValue = GetCardValue()
    valuesArray(count) = cardValue
    valuesList.Add(cardValue)
    Console.WriteLine("A " & cardValue & GetSuit())
Next

Now, you can access the values with
Console.WriteLine(valuesArray(1))
Console.WriteLine(valuesArray(2))
Console.WriteLine(valuesArray(3))

or
Console.WriteLine(valuesList(0))
Console.WriteLine(valuesList(1))
Console.WriteLine(valuesList(2))

Both, array and list, start numbering the items with 0. Note that the array has a lenght of 4. We did not use valuesArray(0).
You can also use for-each
For Each item In valuesArray
    Console.WriteLine(item)
Next

or
For Each item In valuesList
    Console.WriteLine(item)
Next

The iteration variable item is automatically declared and is limited to the for each loop. You can also specify the type explicitly For Each item As Integer In valuesArray


Bars or Boxplot

Question: For my thesis work I need to plot data as dry mass content, Brix, sugar content,   etc., of 12 genotypes of onions. My question is, what is convenient to use,  barplots or boxplots? Personally I like boxplots more because I see the distribution but I do not want to misuse this.
Here are two examples:
Comment: http://onlinelibrary.wiley.com/doi/10.1002/sim.6549/abstract. It depends on what you want to show. You could also consider a stacked bar chart.
Comment: The plots are misleading because the y-axis does not have the same scale (ylim=c(0,20). I would use Boxplots.
Answer: Boxplots are highly preferable in general. If you have few enough individual data points, you might even consider superimposing them on the boxplots. (You do need to consider appropriate limits for the y-axis in any case. One might argue that your second example, with y-axis starting at 9, overemphasizes the differences among genotypes.)
Edward Tufte, a renowned expert on display of information, recommends maximizing the data/ink ratio. If a plot has a lot of ink that can be removed without losing information about the data, then the data/ink ratio is too low. Barplots almost always have a very low data/ink ratio.
See Tufte's website for further information and discussion.

Bars or Boxplot


Inclusion of Age^2 changes results completely

Question: I'm running a diff-in-diff with model with years of education as an outcome variable.
$Y_{it}=\beta_0+\beta_1Treat_i+\gamma_t+\beta_2Treat_{i}*Post_t+X'\gamma+\epsilon_{it}$
The treatment dummy is based on a reform which had an age cutoff, and is therefore by construction negatively correlated with age. The sample runs from 2010-2017 meaning that the oldest individuals are only 38 in the last observed year.
When I include Age as a control variable $\beta_2$ is almost completely unchanged. However, once i include Age^2 $\beta_2$ changes sign and becomes negative. Any suggestions on why this might be?
Also I'm aware that fitting a non-linear relationship is often plausible. E.g  years of education, employment, wages etc. tend to exhibit decreasing marginal effects. Is this also necessary in a "young" sample since one might expect the turning point to be outside the range of the data?
Answer: You would need to do some explorative data analysis to answer this. Look at the leverages (your linear regression tool should be able to produce diagnostic plots). If there are a few very old individuals, their squared age would dominate their expected response. Suppose, for example, age^2 has a negative coefficient. A very old individual therefore has a very low expected response, and since very old individuals are not treated, this would lower the estimates treatment effect and might well cause it to be negative.
You might also want to carry out an analysis on the subset of individuals who were eligible for treatment (i.e. only young individuals).


Intermintent slow MySQL Connections

Question: We keep getting errors like this on our PHP sites.

Can't connect to MySQL server on '192.168.100.85' (4)

web is the web server 192.168.1.116
mysql is the mysql server 192.168.100.85
So I built a script on the web server to make 10,000 mysql connections in a row and time them. It will "sometimes" produce the error. Most times everything runs fine and there is an average time of 5-10 ms for the call to mysql_connect
Some serious Googling showed that the (4) error is due to the connection being cut off by the timeout.

web# grep mysql.connect_timeout /etc/php.ini
  mysql.connect_timeout = 1

So I modified the timeout in the script to 30 to see if it would work. The connection errors went away, but occasionally the connection would take 5 seconds.
After more Googling and some tcpdump I found that occasionally when the MySQL server is doing its reverse lookup of the IP the DNS server would fail to respond. So after 5 seconds it would give up and allow the connection.
I have since added skip-name-resolve to the server. But this did not solve the problem.
Now my test would show slow connections taking 3-4.5 seconds instead of the set 5 with the DNS issue.
So I ran my tests again with tcpdump running on both ends.

web# tcpdump -n -s 65535 -w web3-$(date +"%F_%H-%M-%S").pcap host 192.168.100.85 and port 3306
  mysql# tcpdump -n -s 65535 -w master1-$(date +"%F_%H-%M-%S").pcap host 192.168.1.116 and port 3306

Here is the packets from the relevant slow connection.
Packets from web:
No.     Time                       Source                Destination           Protocol Info
13312 2010-10-13 10:01:01.201965 192.168.1.116         192.168.100.85        TCP      41560 > mysql [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=904829062 TSER=0 WS=2
13316 2010-10-13 10:01:04.201577 192.168.1.116         192.168.100.85        TCP      41560 > mysql [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=904832062 TSER=0 WS=2
13317 2010-10-13 10:01:04.204837 192.168.100.85        192.168.1.116         TCP      mysql > 41560 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1380 TSV=562240314 TSER=904832062 WS=7
13318 2010-10-13 10:01:04.204853 192.168.1.116         192.168.100.85        TCP      41560 > mysql [ACK] Seq=1 Ack=1 Win=5840 Len=0 TSV=904832065 TSER=562240314
13319 2010-10-13 10:01:04.205886 192.168.100.85        192.168.1.116         MySQL    Server Greeting proto=10 version=5.0.77-log
13320 2010-10-13 10:01:04.205899 192.168.1.116         192.168.100.85        TCP      41560 > mysql [ACK] Seq=1 Ack=61 Win=5840 Len=0 TSV=904832066 TSER=562240316
13321 2010-10-13 10:01:04.205959 192.168.1.116         192.168.100.85        MySQL    Login Request userexample
13322 2010-10-13 10:01:04.206800 192.168.100.85        192.168.1.116         TCP      mysql > 41560 [ACK] Seq=61 Ack=71 Win=5888 Len=0 TSV=562240317 TSER=904832066
13323 2010-10-13 10:01:04.206874 192.168.100.85        192.168.1.116         MySQL    Response OK
13324 2010-10-13 10:01:04.208823 192.168.1.116         192.168.100.85        MySQL    Request Quit
13325 2010-10-13 10:01:04.208839 192.168.1.116         192.168.100.85        TCP      41560 > mysql [FIN, ACK] Seq=76 Ack=72 Win=5840 Len=0 TSV=904832069 TSER=562240317
13326 2010-10-13 10:01:04.210422 192.168.100.85        192.168.1.116         TCP      mysql > 41560 [FIN, ACK] Seq=72 Ack=76 Win=5888 Len=0 TSV=562240320 TSER=904832069
13327 2010-10-13 10:01:04.210437 192.168.1.116         192.168.100.85        TCP      41560 > mysql [ACK] Seq=77 Ack=73 Win=5840 Len=0 TSV=904832071 TSER=562240320
13328 2010-10-13 10:01:04.210567 192.168.100.85        192.168.1.116         TCP      mysql > 41560 [ACK] Seq=73 Ack=77 Win=5888 Len=0 TSV=562240320 TSER=904832069

Packets from mysql:
No.     Time                       Source                Destination           Protocol Info
13315 2010-10-13 10:01:04.204817 192.168.1.116         192.168.100.85        TCP      41560 > mysql [SYN] Seq=0 Win=5840 Len=0 MSS=1380 TSV=904832062 TSER=0 WS=2
13316 2010-10-13 10:01:04.204836 192.168.100.85        192.168.1.116         TCP      mysql > 41560 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=562240314 TSER=904832062 WS=7
13317 2010-10-13 10:01:04.206611 192.168.1.116         192.168.100.85        TCP      41560 > mysql [ACK] Seq=1 Ack=1 Win=5840 Len=0 TSV=904832065 TSER=562240314
13318 2010-10-13 10:01:04.206808 192.168.100.85        192.168.1.116         MySQL    Server Greeting proto=10 version=5.0.77-log
13319 2010-10-13 10:01:04.207658 192.168.1.116         192.168.100.85        TCP      41560 > mysql [ACK] Seq=1 Ack=61 Win=5840 Len=0 TSV=904832066 TSER=562240316
13320 2010-10-13 10:01:04.207815 192.168.1.116         192.168.100.85        MySQL    Login Request user=example
13321 2010-10-13 10:01:04.207872 192.168.100.85        192.168.1.116         TCP      mysql > 41560 [ACK] Seq=61 Ack=71 Win=5888 Len=0 TSV=562240317 TSER=904832066
13322 2010-10-13 10:01:04.207910 192.168.100.85        192.168.1.116         MySQL    Response OK
13323 2010-10-13 10:01:04.210817 192.168.1.116         192.168.100.85        MySQL    Request Quit
13324 2010-10-13 10:01:04.210849 192.168.100.85        192.168.1.116         TCP      mysql > 41560 [FIN, ACK] Seq=72 Ack=76 Win=5888 Len=0 TSV=562240320 TSER=904832069
13325 2010-10-13 10:01:04.211632 192.168.1.116         192.168.100.85        TCP      41560 > mysql [FIN, ACK] Seq=76 Ack=72 Win=5840 Len=0 TSV=904832069 TSER=562240317
13326 2010-10-13 10:01:04.211640 192.168.100.85        192.168.1.116         TCP      mysql > 41560 [ACK] Seq=73 Ack=77 Win=5888 Len=0 TSV=562240320 TSER=904832069
13327 2010-10-13 10:01:04.213243 192.168.1.116         192.168.100.85        TCP      41560 > mysql [ACK] Seq=77 Ack=73 Win=5840 Len=0 TSV=904832071 TSER=562240320

As you can see, web re-sent the initial ACK after 3 seconds of no response. But mysql never even saw that initial packet.
I also tried to run a ping flood to check for dropped packets. If you leave it running long enough you will get dropped packets.
web3# ping -f 192.168.100.85
PING 192.168.100.85 (192.168.100.85) 56(84) bytes of data.
....................................................................
--- 192.168.100.85 ping statistics ---
38253 packets transmitted, 38185 received, 0% packet loss, time 460851ms
rtt min/avg/max/mdev = 0.880/3.430/66.904/8.015 ms, pipe 7, ipg/ewma 12.047/1.378 ms

This issue is very intermittent but is happening all the time. I do understand that simply increasing our timeout would help greatly, but I'd rather have the connects always be fast and never have a 3-4 second delay in serving a page.
I contacted our hosting provider and they say its likely a result of the bad interaction between Nagle's Algorithm and Delayed ACK. http://www.stuartcheshire.org/papers/NagleDelayedAck/
It seems to me like packets are being dropped. Any ideas on a way I could better prove it to the hosting guys? Ping dropping 60 out of 38,000 packets does seem kind of minor. Is this something I should just live with?
Thank you for your time in looking into this!
Answer: I wouldn't try it with a ping flood, but I would set up a continous stream and let it run for, say, 2 minutes. That's a more valid test of network "loss" - sometimes ping -f can overrun particular pieces of equipment and give unreliable results.
Comment: Marking this as answer since its the only one. We ended up upgrading a piece of networking gear between the two servers and have had much better performance since.

Intermintent slow MySQL Connections


How do I find out which system.info my site is using?

Question: I have 3  sites using the same host. I have updated them from 7.89 to 7.91 however only one site shows as using 7.91. I have hunted my host files for a system.info with 7.89 without any luck.
Where do I find which system.info files are being used by my sites?
Answer: The Drupal 7 Core version is in includes/bootstrap.inc on line 10, not in a .info file. Example:
define('VERSION', '7.91');

Comment: See my comment below. That is the system.info that my sites should be using (I think).
Answer: The system.info file is simply the file containing information for the System module. As such, there should not be more than a system.info file in a Drupal site, except in the case you copied the Drupal core files more than once.
The system.info file doesn't contain any release value. It just contains a reference to a constant defined in another file.
name = System
description = Handles general site configuration for administrators.
package = Core
version = VERSION
core = 7.x
files[] = system.archiver.inc
files[] = system.mail.inc
files[] = system.queue.inc
files[] = system.tar.inc
files[] = system.updater.inc
files[] = system.test
required = TRUE
configure = admin/config/system

VERSION is defined in the includes/bootstrap.inc file. If that file has been updated, it's probable the Drupal core release isn't reported correctly.
Comment: This is what is in my system.info in the /public_html/modules/system folder. name = System
description = Handles general site configuration for administrators.
package = Core
version = VERSION
core = 7.x
files[] = system.archiver.inc
files[] = system.mail.inc
files[] = system.queue.inc
files[] = system.tar.inc
files[] = system.updater.inc
files[] = system.test
required = TRUE
configure = admin/config/system

; Information added by Drupal.org packaging script on 2022-07-20
version = "7.91"
project = "drupal"
datestamp = "1658334256"
Comment: This is VERSION in bootstrap.inc       /**
 * The current system version.
 */
define('VERSION', '7.91');


How to preprocess view field?

Question: I have view with one date field. Now I want to preprocess this date field(lets say I want to change the date format).
Twig file override : views-view-field--field_date.html.twig
Prepocess code I have tried but kint is showing me only DEPTH TOO GREAT value.
function my_sub_theme_preprocess_views_view_field__field_date(&$variables) {
    kint($variables['view']);
}

Please let me know how to preprocess this field. Is it possible to change date format in twig itself?
Answer: The output of views field templates is flat markup. You can't change a render array like you can in field templates when rendering entities in view modes.
So if you want to change the output you have to start from scratch with the values that are in the database. In views field templates you can use the getValue() method from the field variable. As parameter you have to provide row:
{{ field.value(row) }}

The result is a single value or an array for multivalue fields.
Example
views-view-field--created.html.twig
The raw value from created is a unix timestamp
Timestamp: {{ field.value(row) }}

You can use the timestamp for example to reformat the date with the date filter from twig:
Date: {{ field.value(row)|date("m/d/Y") }}

Answer: Yes. A date format can be changed from TWIG template itself. Look for more documentation on Date in Twig here - http://twig.sensiolabs.org/doc/2.x/filters/date.html
About DEPTH TOO GREAT of Twig. It's coming due to Kint having setting of less maxLevel than what is there in the variable you are trying to dump.
You can change DEPTH in same function itself by writing something like -
$_kintSettings['maxLevels'] = 10;

How to preprocess view field?


Get local variable from ssh call in shell

Question: I have a script called global.sh which copies a shell script called my_local_script.sh on a remote machine.
scp my_local_script.sh host@user_a:remote_directory

Then i connect to that ssh host and execute it like this
ssh host@user "
                cd remote_directory
                ./my_local_script.sh
              "

So far so good and everything is ok, the problem is that i want to get a local variable inside my_local_script.sh to global.sh. I tried doing this
ssh host@user "
                cd remote_directory
                my_arg=$(./my_local_script.sh)
              "

But this does not work and the even the call of my local script does not work properly anymore. Any suggestions how can i overcome this issue.
PS : Copying the variable to a temporary file is not an option. I am looking for a shell solution for this if possible.
Answer: You are using a command substitution, which is necessary, but it is in the wrong place. You should place it around the entire SSH command:
my_var=$(ssh host@user "cd remote_directory; ./my_local_script.sh")

This is assuming that the variable you want to capture is being outputted by my_local_script eg by echo or printf.


Expected value for a continuous random variable

Question: Let's fix the generic probability space $(\Omega,\Sigma,P)$, and consider the generic random variable $X\colon \Omega \to \mathbb{R}$.
The expected value of $X$ (supposed to exist) is then defined as $E(X):=\int_{\Omega} X\,\text{d}P \in [-\infty,+\infty]$, namely using the Lebesgue integral of $X$ with respect to the probability measure $P\colon \Sigma \to [0,1]$.
When $X$ is a continuous variable, namely when there exists $f_X\colon \mathbb{R} \to [0,+\infty)$ such that $\forall \,x \in \mathbb{R} \quad F_X(x):=P(X \le x)=\int_{-\infty}^x f_X(t)\,\text{d}t$, we have also that $E(X)=\int_{-\infty}^{+\infty} xf_X(x)\,\text{d}x$.
My question is: how can I show that the general definition of $E(X)$ reduces to the last one for continuous random variables? Can you suggest me a textbook or a source where I can find some help?
P.S.
I have found here Wikipedia, pushforward measure that $\int_{\Omega} X\,\text{d}P=\int_{\mathbb{R}} \text{d}\mu_X$, where $\mu_X\colon \mathcal{B}(\mathbb{R}) \to [0,1] \mid \mu_X(A)=P(X \in A)$ is the pushforward measure of $X$, with $\mathcal{B}(\mathbb{R})$ the Borel $\sigma$-algebra over $\mathbb{R}$.
I know also that $F_X'=f_X$ and that $F_X(x)=\mu_X((-\infty,x])$, so maybe $\text{d}\mu_X=\text{d}F_X=f_X\,\text{d}x$, but I seriously think this is totally wrong.
Thank you for your time and help!
Comment: https://math.stackexchange.com/q/570348/321264, https://math.stackexchange.com/q/3285425/321264
Comment: Wondering why would anyone accept a non-answer.
Comment: I'm studyng probability theory at a pretty elementary level, so I'm trying to avoid measure theory every time I can. I thought that the definition of expected value in its general form must use Lebesgue integral. However, I discovered that it is also possible to give a decent definition only using cumulative distribution function. That helped me a lot. Again, no one else answered my question.
Comment: No one else answered your question perhaps because it was already briefly answered in the links above.
Comment: Yes, but I can't accept a comment, only an answer. The above comment was very useful to me, and in fact I upvoted it. However, even the only given answer has been useful, even if in a different way. However, I agree with you that it doesn't addresses my original question. Am I wrong? However, if my acceptance violates the rules of the site I will remove it.
Comment: A little precisation: with the expression "I can't accept a comment, only an answer", I mean that this site doesn't allow me to accept a comment. I don't mean that I cannot be satisfied by a comment. Just to make it clear. However, thank you very much for the link above!
Answer:
how can I show that the general definition of $E(X)$ reduces to the last one for continuous random variables? Can you suggest me a textbook or a source where I can find some help?

This is the general definition of Expectation taken from Mood Graybill Boes
Comment: I don't see that this answer addresses the question. The question was how to prove that $$ \int_\Omega X\, \mathrm{d}P = \int_{-\infty}^{+\infty} xf_X(x)\,\mathrm{d}x. $$
Comment: Wait, I can see that here we are not using Lebesgue integral with respect to a generic measure...Ok I'm pretty confused now...


Centering a widget inside another widget in Qt

Question: I'm trying to center a QWidget localized inside a QTabWidget. When one resizes the inner QWidget I want it to be placed in the center always.
How can I do this?
Comment: Add spacers on all four sides around it in the layout
Comment: @NicolasHolthaus there are no layouts that support such combination. It is enough to add 2 spacers and set widget size policy of rest dimension to maximum. Or combine vertical + horizontal layout with 4 spaces, as you suggested :)
Comment: @SaZ There is a grid layout in Qt, which provide that feature.
Comment: Are you creating the widget from code or from Qt designer?
Comment: I have solved this problem by using layout alignment. Now my widget is aligned but resizing is working no more. It has a fixed size of minimum size which I have defined as 600X600. How can I enable risizing again?
Comment: Set an appropriate size policy, then. Most likely you want `MinimumExpanding` in both directions.
Answer: 1/ Create a layout in your QTabWidget: a QBoxLayout for example.
2/ Set your QWidget in it, with the desired alignment.
QTabWidget* tabwid = new QTabWidget(this);
QBoxLayout* layout = new QBoxLayout(QBoxLayout::LeftToRight, tabwid);
//if you want to set margins : layout->setContentsMargins(5, 5, 5, 5);
QWidget* widget = new QWidget();
layout->addWidget(widget, Qt::AlignCenter); // center alignment

Comment: The `addWidget()` syntax is not correct, since the second argument of [`addWidget()`](https://doc.qt.io/qt-5/qboxlayout.html#addWidget) is the `stretch`, not the alignment. It should be `layout->addWidget(widget, 0, Qt::AlignCenter);`


What's the most idiomatic way to to remove collection elements with values containing one or more of a given collection of words?

Question: Supposing I wanted to remove the list elements mentioning the animals in the banned-from-house collection:
(def list (atom [{:animal "a quick happy brown fox that rocks!"}
                 {:animal "a quick happy brown hamster that rocks!"}
                 {:animal "a quick happy brown bird that rocks!"}
                 {:animal "a quick happy brown dog and fox that rock!"}
                 {:animal "a quick happy brown fish that rocks!"}]))

(def banned-from-house (atom ["fox" "bird"]))

What would be the most idiomatic way to go about that?
Also, what would be a better title for this question? (I struggle in discussing clojure code)
Comment: First, get rid of those atoms :)
Answer: Let's build this step by step.
First, let's test if a String mentions some animal name, using clojure.string/includes?.
(defn mentions-animal? [s animal]
  (clojure.string/includes? s animal))

(mentions-animal?
  "a quick happy brown fox that rocks!"
  "fox")
=> true
(mentions-animal?
  "a quick happy brown fox that rocks!"
  "dog")
=> false

Second, let's test if a string mentions some of a seq of animal names, using clojure.core/some.
(defn mentions-any? [s animals]
  (some #(mentions-animal? s %) animals))

(mentions-any?
  "a quick happy brown fox that rocks!"
  #{"fox" "dog"})
=> true
(mentions-any?
  "a quick happy brown fox that rocks!"
  #{"cat" "dog"})
=> nil

Next, extend this logic to animal maps instead of strings.
(defn animal-mentions-any?
  [a animals]
  (mentions-any? (:animal a) animals))

Finally, implement the filtering logic using clojure.core/remove:
(defn remove-banned-animals
  [animals-list banned-animals]
  (remove #(animal-mentions-any? % banned-animals) animals-list))

(remove-banned-animals
  [{:animal "a quick happy brown fox that rocks!"}
   {:animal "a quick happy brown hamster that rocks!"}
   {:animal "a quick happy brown bird that rocks!"}
   {:animal "a quick happy brown dog and fox that rock!"}
   {:animal "a quick happy brown fish that rocks!"}]
  ["fox" "bird"])
=> ({:animal "a quick happy brown hamster that rocks!"} {:animal "a quick happy brown fish that rocks!"})

Comment: @leetwinski I agree, but even regexps have shortcomings (pluralization etc.), in which case the solution would be to use an NLP library. It seemed to me the focus of the question was code organization, which is why I opted for a simplistic solution.
Comment: This is awesome! Thanks so much. I was a bit stumped about how to do it with this brevity as I didn't know about 'includes?'. The step-wise explanation is also great.
Comment: just one note: since we want to classify data by some word inclusion, it is probably better to use regexp instead of `includes?` , (something like `#"\bfox\b"`), to avoid problems with words, including your target ones (like `"a quick happy brown dog dancing foxtrot"`)


demeaning or differentiation in fixed-effect equation

Question: I have panel data (with short time-dimension $T>2$) and I consider a simple model of the form:
$$y_{i,t} = x_{i,t} \beta + c_i + \epsilon_{i,t}$$
where $E(x_{i,s}\epsilon_{i,t})=0$ and $\epsilon_{i,t}$ are iid and of mean 0.
To get rid  of the $c_i$ and to estimate the $\beta$, one can consider in particular:

either first differentiating succesive observations and then performing an OLS regression: $\Delta y_{i,t} = (\Delta x_{i,t}) \beta +\Delta \epsilon_{i,t}$
or demeaning and then performing an OLS regression: $y_{i,t} - \bar{y_{i,t}} = (x_{i,t} - \bar{x_{i,t}}) \beta +(\epsilon_{i,t}-\bar{\epsilon_{i,t}})$

Please note that the demeaning seems to be prefered (it is the "fixed-effect estimator"), and it can be written (for instance here*) that "The [first difference] estimator is slightly less efficient than the [demeaning], as $\Delta \epsilon$ exhibits serial correlation, even if $\epsilon_{i,t}$'s are uncorrelated." For instance demeaning is used in the Stata command xtreg, fe.
My first question is: why should the demeaning approach should be prefered to the differentiating approach?
As a temptative answer, it is indeed easy to see that $\Delta \epsilon_{i,t}$ is correlated with $\Delta \epsilon_{i,t+1}$ as both terms share $\epsilon_{i,t}$. It comes: $cov(\Delta \epsilon_{i,t+1},\Delta \epsilon_{i,t}) = - \sigma_\epsilon<0$
However, both $(\epsilon_{i,t}-\bar{\epsilon_{i,t}})$ and $(\epsilon_{i,t+1}-\bar{\epsilon_{i,t}})$ also share the terme $\bar{\epsilon_{i,t}}$ (and this term includes also $\frac{\epsilon_{i,t}}{T}$ and $\frac{\epsilon_{i,t+1}}{T}$). It comes: $cov(\epsilon_{i,t}-\bar{\epsilon_{i,t}},\epsilon_{i,t+1}-\bar{\epsilon_{i,t}}) = - \frac{2}{T}(1-\frac{1}{T}) \sigma_\epsilon^2<0$
Hence, my understanding is that there is also some correlation after applying demeaning, but less than when applying first-differencing. This is why it is better.
My second (highly related) questions are: am I making a mistake above? If not, what is the impact of these correlations on the estimates of the $\beta$? Thank you in advance!
*see http://statmath.wu.ac.at/~hauser/LVs/FinEtricsQF/FEtrics_Chp5.pdf in page 25 for demeaning, and in page 28 for first differentiating.
Comment: Bounty expires in 3 days, and there is still no answer. If someone has a suggestion to make the question more understandable/readable/attractive, please make it in comments! thank you in advance. Regards.
Comment: I guess there are rather few people who are active on Cross Validated at the moment and who work on panel data. You might have better luck on Economics Stack Exchange (though I am little familiar with the current situation there).
Comment: Demeaning (fixed effects) does not dominate first differences. The assumptions under which they identify $\beta$ are different (though the difference is minor and irrelevant in practice except in the case of a dynamic regression model), and the assumptions under which either efficient (smallest asymptotic variance) are very different. A very detailed discussion developing these conclusions can be found in Chapters 10 and 11 in Wooldridge's Econometric Analysis of Cross Section and Panel Data, if you can't get that I might be able to write this up later.
Comment: @CloseToC : Thank you (really!) for the reference!!! Section 10.7.1 of "Wooldridge's Econometric Analysis of Cross Section and Panel Data" is indeed: Comparison of Fixed Estimators - FE vs Demeaning. If I do understand him correctly, he notably states that demeaning is more efficient when $\epsilon_{i,t}$ are i.i.d., but the differencing is more efficient when $\epsilon_{i,t}$ follows a random walk. However, note that I restricted the question to the first case.
Answer: Both the first difference transformation and the within transformation generate serial correlation in the estimated residuals even when it is not there in the underlying data generating process. For $T > 2$, the first difference estimator is less efficient than the within if the error term is really iid. For $T = 2$ the first differences estimator and the within estimator are equivalent.
The first difference estimator requires a slightly weaker assumption than strict exogeneity, namely that the transitory component of the error term is uncorrelated with the conditioning variables in the current period, as well as one period before and one period after (rather than all of them). This will not matter in most economic contexts.
Comment: Thanks you for your answer that appears correct to me. I awarded the bounty to @IMA as I think her/his answer helps me better to understand what is happening.
Answer: Why is it more efficient?
Let $e$ denote the transformed error from either transformation. We have that
$$V(\hat{\beta}|X)=(X'X)^{-1}X'V(e|X)X(X'X)^{-1}$$
So looking at $V(e|X)$, we check diagonal and off-diagonal elements. For diagonal elements, we have $2\sigma^2$ for the first-difference, and for the within transform $\sigma^2+V(\overline{\epsilon}(T))$ where the latter term goes towards zero with at a rate of $T^2$.
For the off-diagonal elements, you have already checked that the within transform becomes more efficient with $T$ for all elements.
Check for example $T=3$, where the estimators differ. On the diagonal, you have for the first-difference $2\sigma^2$. But for the within estimator, the diagonal comes out as $\sigma^2+\frac{1}{4}(2\sigma^2)=\frac{3}{2}\sigma^2$ which is less than $2\sigma^2$.
You can do the same for other elements and for larger $T$.
In that sense, the estimator is more efficient, it provides smaller standard errors for the same data generating process.
Edit: You can calculate this out (I think the exact formula is not so important in this case), but say you will get something like
$$V(\hat{\beta}|x)=\frac{\sum_{t,h}x_{t}x_h \sigma_{t,h}}{2\sum_t x_t^2}$$ and then obviously that thing is declining in $T$ for the within estimator, meaning it makes use of the additional data we have if $T>2$ whereas the difference estimator is constant and thereby not efficient over $T$
Comment: First, thanks for this answer!
Comment: Second (and not very important), I get that $\sigma^2+V(\overline{\epsilon}(T)) = \sigma^2 - \frac{\sigma^2}{T}$. So I would say that the convergence rate is $\frac{1}{T}$ rather than $\frac{1}{T^2}$. Am I correct?
Comment: Third, regarding the off-diagonal elements, I realized by reading your answer that some are zero for the first difference while they are non-zero for the demeaning (for instance for $cov(e_t,e_{t-2})$. It is 0 for the first difference, but $-\frac{2}{T}(1-\frac{1}{T})\sigma$ for the demeaning).)
Comment: I award the bounty as I think your answer is the one that brings me a very good understanding of how to deal with the problem. Please do not hesitate to look at (and answer) my above comments. Thank you in advance
Comment: Hey, sorry for answering late. Second, yes, take all my calculations with a grain of salt, I did not double check ;-) Point is that there is convergence in this element of the cov-matrix.
Third, you are correct about zero elements, and so its not cut-and dry for any sample size. But efficiency in this case means that the variance converges for the within estimator over T and you can do the same calculations for other configurations such as with more regressors, a panel etc. But I think the idea is quite clear - something dependent on an average will converge with T, the other thing not
Comment: Edit: Also as noted above, this all depends on how the within estimator behaves compared to the FD estimator. In your case, what I wrote is true, but it doesn't hold for every distribution (see Woolridge).


Is there any way to reset a node's properties?

Question: I would like to know if there is a way to reset a node's properties/fields?For example node's body, title,attachments, so as to re write the contents of the page.
Comment: you mean editing it?  surfing over to `www.jonssite.com/node/####/edit` where `####` is the nid will do that.
Comment: No Just for display.
Comment: can you give a more concrete example of what you are trying to do?
Answer: In your theme folder you can override the default output using custom template files.
Let's say you want to modify the output of a standard 'Page' (the machine name for this content type is 'page' -- you can check it in /?q=admin/structure/types).
Go to your theme folder (under /sites/all/themes) and add/modify 'node--page.tpl.php' file.
A standard node template file will have something like:

<?php print render($title_prefix); ?>
<?php if (!$page): ?>
<h2<?php print $title_attributes; ?>>
  <a href="<?php print $node_url; ?>"><?php print $title; ?>

<?php endif; ?>
<?php print render($title_suffix); ?>

This piece of code is used to show the title of the current content. You can delete (or comment) the entire code and your title won't appear when you display this page.
To hide the entire content (including custom fields attached to this content type) simply comment (or delete) this line:

print render($content);

After that, you can add your custom code to display anything in place of the original title (or content) for this page.
You can find an extensive tutorial on working with custom themes here: Theming Guide


Convert into multiple data ranges

Question: I have employee data link below

This is one example in the file a have multiple lines. Initial input can have more than 2 lines for each employee but convert logic will be the same.
Thank you for help!
Answer: After setting up some test data:
DECLARE @EmployeeUtil TABLE (
    Employee_ID INT,
    [Date] DATE,
    Previous_FTE DECIMAL(3,2),
    FTE DECIMAL(3,2)
)

INSERT INTO @EmployeeUtil
SELECT 1, '2016-03-24', 1, 0.67
UNION
SELECT 1, '2016-04-24', 0.67, 0.8
UNION
SELECT 1, '2016-06-01', 0.8, 1
UNION
SELECT 2, '2016-01-01', 1, 0.5
UNION
SELECT 2, '2016-02-03', 0.5, 0.6
UNION
SELECT 2, '2016-03-25', 0.6, 0.8
UNION
SELECT 2, '2016-04-20', 0.8, 0.25
UNION
SELECT 2, '2016-05-01', 0.25, 0.4
UNION
SELECT 2, '2016-06-01', 0.4, 0.8
UNION
SELECT 2, '2016-07-01', 0.8, 1

I would use Window Functions to achieve the desired result:
    SELECT  Employee_ID,
            [Date] AS [Start_date],
            LEAD([Date]) OVER (PARTITION BY Employee_ID ORDER BY [DATE]) AS [End Date],
            FTE
    FROM @EmployeeUtil
UNION
    SELECT  e1.Employee_ID,
            NULL AS [Start_date],
            FIRST_VALUE(e1.[Date]) OVER (PARTITION BY e1.Employee_ID ORDER BY e1.[DATE]) AS [End Date],
            FIRST_VALUE(e1.Previous_FTE) OVER (PARTITION BY e1.Employee_ID ORDER BY e1.[DATE]) AS FTE
    FROM @EmployeeUtil e1
ORDER BY 1, 2, 3

The SELECT in the first part of the UNION fetches all of the ranges of dates with the appropriate start and end dates and FTE values for that date range, except for the unbounded period prior to the earliest date recorded for each employee.
The SELECT in the second part of the UNION deals with this, creating an extra row in the results for each Employee to show their FTE value prior to the earliest date.

Convert into multiple data ranges


Swagger - Change project directory

Question: I created new project (nodeJS) inside root directory inside my workspace and after some time I wanted to separate client and server side of project. To do that I created two folders (server and client) inside root dir of project and moved all files and directories to appropriate folder.
After that I can't start server because it says that Swagger can't find swagger.yaml file (I moved it to server folder)
<code>{ Error: ENOENT: no such file or directory, open 'C:\Users\User\Documents\movieCollection\api\swagger\swagger.yaml'
    at Error (native)
    at Object.fs.openSync (fs.js:634:18)
    at Object.fs.readFileSync (fs.js:502:33)
    at C:\Users\User\AppData\Roaming\npm\node_modules\swagger\lib\commands\project\project.js:283:44
    at findProjectFile (C:\Users\User\AppData\Roaming\npm\node_modules\swagger\lib\commands\project\project.js:308:14)
    at readProject (C:\Users\User\AppData\Roaming\npm\node_modules\swagger\lib\commands\project\project.js:268:3)
    at Command.edit (C:\Users\User\AppData\Roaming\npm\node_modules\swagger\lib\commands\project\project.js:240:3)
    at Command.<anonymous> (C:\Users\User\AppData\Roaming\npm\node_modules\swagger\lib\util\cli.js:167:27)
    at Command.listener (C:\Users\User\AppData\Roaming\npm\node_modules\swagger\node_modules\commander\index.js:301:8)
    at emitTwo (events.js:106:13)
  errno: -4058,
  code: 'ENOENT',
  syscall: 'open',
  path: 'C:\\Users\\User\\Documents\\movieCollection\\api\\swagger\\swagger.yaml' }
</code>
I tried to edit default.yaml file inside config folder of node server but problem still persists.
Is there way to point swagger project configuration to new laction (directory-> server folder) ?
default.yaml:
<code># swagger configuration file

# values in the swagger hash are system configuration for swagger-node
swagger:

  fittingsDirs: [ api/fittings ]
  defaultPipe: null
  swaggerControllerPipe: swagger_controllers  # defines the standard processing pipe for controllers

  # values defined in the bagpipes key are the bagpipes pipes and fittings definitions
  # (see https://github.com/apigee-127/bagpipes)
  bagpipes:
_router:
  name: swagger_router
  mockMode: false
  mockControllersDirs: [ ./server/api/mocks ]
  controllersDirs: [ ./server/api/controllers ]

_swagger_validate:
  name: swagger_validator
  validateResponse: true

# pipe for all swagger-node controllers
swagger_controllers:
  - onError: json_error_handler
  - cors
  - swagger_security
  - _swagger_validate
  - express_compatibility
  - _router

# pipe to serve swagger (endpoint is in swagger.yaml)
swagger_raw:
  name: swagger_raw

# any other values in this file are just loaded into the config for application access...
</code>
EDIT: Added app.js
app.js:
<code>'use strict';

var SwaggerExpress = require('swagger-express-mw');
var express = require("express");
var config = require('config');
var cors = require('cors');
var bodyParser = require('body-parser');
var mongoose = require('mongoose');
var passport = require('passport');
var flash = require('connect-flash');
var session = require('express-session');
var routes = require('./src/routes');
var routingPath = '../server';
var app = express();

var spec = fs.readFileSync('../server/api/swagger.yaml', 'utf8');
var swaggerDoc = jsyaml.safeLoad(spec);
// Enable CORS
app.use(cors());

// Bootstrap routes
app.use(routes);

// Static files
app.use('/', express.static(__dirname + '/../public'));
module.exports = app; // for testing

var config = {
  appRoot: __dirname   // required config
};
swaggerExpress.runner.swagger.basePath = routingPath;

SwaggerExpress.create(config, function(err, swaggerExpress) {
  if (err) { throw err; }

  // install middleware
  swaggerExpress.register(app);

  var port = process.env.PORT || 10010;
  app.listen(port);

    console.log('Server started: http://127.0.0.1:' + port );

});
</code>
Comment: Could you figure this out?
Answer: <code>'use strict';

import errors from './components/errors';
import path from 'path';
var swaggerUi = require('swagger-ui-express');
var swaggerDocument = require('./swagger.js');

export default function(app) {
    // Insert routes below

    app.use('/api-docs', swaggerUi.serve, swaggerUi.setup(swaggerDocument));
}

added this in route file...
</code>

Swagger - Change project directory


Keyman Android custom layout

Question: I have created an app with Android Studio that needs a custom keyboard with only a few letters and numbers.  I can create alternate keys in the Keyman development IDE but I can't find how to change what is displayed on the virtual keyboard buttons.
Thanks in advance
Sam
Answer: You'll first need to add the "Touch-Optimised Keyboard" feature in the Details tab in the Keyman Developer IDE.

Once you have added this, the Touch Layout tab will be added which allows you to change the layout for Android (and iOS) devices.

Keyman Android custom layout


jQuery - Need to remove the value on mouse leave but not on mouse enter

Question: I am using jquery.rateit.min.js to make a star rating, I need to consider below 2 conditons:

On mouse leave the value(coming as rating good, average etc.) should be remove.
After clicking on star and than removing the mouse the value should not be remove.

Below is the code:
<input type="range" min="0" max="5" value="0" step="1" id="backing1">

Comment: put your code in [jsfiddle](http://jsfiddle.net).
Answer: Just use a flag for it:
var hasBeenClicked = false;

$('.stars').on('mouseleave', function(e) {
    if(hasBeenClicked) {
        //let the stars value
    } else {
        //reset the stars
    }
});

$('.stars .star').on('click', function() {
    //set the rating
    hasBeenClicked = true;
});

Comment: Thanks Jonas, really appreciated, i am new to jquery it would be good if you write full code for me as i am unable to write the correct code.
Comment: In order to write complete code, we'll need more information such as "where is the value saved, an input field?". Best way is for you to create a JSFiddle so we can correct your code there.
Comment: Clicking the reset button (the no entry icon) of the RateIt plugin is not covered by this logic. You would need to set `hasBeenClicked` to `false` in that case.
Answer: Is this what you are looking for? All I have done is bind a function to the mouseleave event.
$('#rateitHover').bind('mouseleave', function() {
    var v = $('#rateitHover').rateit('value');
    if (v == 0) {
        $('.tooltip').html('');
    } else {
        $('.tooltip').html('R: ' + tooltipvalues[v - 1]);
    }
});

Here is a JSFiddle to see it in action: http://jsfiddle.net/9xrkr/
Comment: Yes this is what i need there is just one problem. When i select the star and mouse over on another stars its value doesn't remove on mouse leave.
Comment: But you said "After clicking on star and than removing the mouse the value should not be remove."
Comment: Yes.."After clicking on star and than removing the mouse the value should not be remove." but what happen when we clicked on star and at same time hover to another stars than only the click star value should be appear on mouse leave..
Comment: I've updated my answer. Now the tooltip is always "R: something" when you have a star clicked.
Comment: @user1798659 Good to hear. Please do not forget to mark your question as Answered.
Comment: @vegemite4me...I again need your help. now i want to use this star rating(input) several times in a page obviously with different ids. Please let me know how i do this..http://jsfiddle.net/9xrkr/