Find the origin of folk

Question: The origins of a specific piece of music can be difficult to find. Especially when there are multiple variations in multiple languages. Folk music is often old and translated many times.
Sometimes part of the lyrics or the title give away the origin. For instance, The Irish Rover is probably of Irish origin. It's even easier on titles like Dúlamán and Eilean Mo Chridhe (respectively Irish and Scottish).
If a location is mentioned, there's a fair chance the language spoken in that location is the language of origin (for example, The Red Rose Cafe was originally written in Dutch and Maid of Culmore in Irish). However, this can result false positives. My love's in Germany is Scottish, not German. Even the language of the title can be misleading. Que Sera, Sera originated in the USA, but the title is French.
To make things even harder, titles are often translated along with the lyrics. The German Alle, die mit uns auf Kaperfahrt fahren probably has a Flemish origin (Al die willen te kapren varen) but originated around Dunkirk (which is located in northern France). Garten Eden is actually a translated version of Scarborough Fair.

Now we've established finding the origins is hard, what are often used tactics to find them with relatively high accuracy? Wikipedia knows much, but not everything and can't always be trusted. A large, historic database would be perfect (it would be great if the ICTM will launch something like that), but I can't imagine something like that actually exists. I'm not afraid of doing the hard work, I just don't know where to start.

How does one find the origin of old, often translated music?

The question is specifically about folk.
Comment: Que sera, sera - more likely Spanish or Italian.
Comment: And according to Wikipedia, it is ungrammatical in all three languages. :)
Comment: I'd think that for a song with a title like "My love's in X", you could be confident that it's written in somewhere other that "X".
Comment: An excellent book about the origins of old jazz, blues and also pop music is "It's One For The Money" by Clinton Heylin. He shows how unscrupulous people claimed credit for songs that had been around for decades, or were folk songs, and published them claiming it is their copyright. Ostensibly it is about the history of copyright but is a wonderful history of popular music. Be warned though, after reading it you could become misanthropic.
Answer: Many of the old traditional folk songs originated before the advent of recorded music.  Typically folk songs were composed for the enjoyment of friends and family and neighbors with no thought of profiting commercially, thus there was no desire or need to copyright the songs or even write them down.
Many folk songs became popular and were transmitted orally through the process of someone hearing the song and then remembering it as best they could and performing it from memory for their friends and family and neighbors.
Often each new iteration contained a change from the original due to errant memory or personal or cultural preference or embellishments or additions.  So each subsequent version and translation was slightly different from the original.
Because there was no copyright process involved and often the songs were not even written and not recorded, many are of unknown origin.  It can be impossible to definitively determine the true origin of many folk songs and any attempt to do so is based purely on speculation.
I read a book by Ted Anthony called "Chasing The Rising Sun" Chasing The Rising Sun on Amazon which chronicles the author's attempt to determine the origin of the popular song "House of The Rising Sun" which became a hit for The Animals when they recorded their version of it.  After many years of traveling all over America and even England (starting with meeting and interviewing Eric Burdon of The Animals who ironically was living in New Orleans), Anthony was never even able to definitively determine the country of origin (early versions omitted mention of New Orleans) of that song, much less the author and never could find a house in New Orleans called "The Rising Sun".
There are many folk songs for which the origin was buried with the original composer and those friends and family members who heard the original debut performance. So the answer to your question is, you can attempt to trace the origins in the manner of a genealogist, but for many songs, there are no "known" origins.
Answer: Often it's a case of chasing through libraries, recordings, and memories. The Library of Congress has a large amount of old sheet music.
https://www.loc.gov/collections/historic-sheet-music/about-this-collection/
A source for early tangos is Todotango.com.
https://www.todotango.com/english/
You can search for melodies or lyrics (mostly melodies.) There are some melodic indexes online. (I don't have a like for any of these. Most are for classical music.)
There are also some older collections (hymnals, early English song, ...) that may link up to a tune of interest.
I did chase a couple of pieces I used to play but neither were folksongs. One was "Orange Blossom Special"; it was written by Ernest Rouse in 1938 (I thought it was older.) The other was "Red River Valley" which learned as a song about the Red River between Texas and Oklahoma. I heard it all over China where it refers to a river in Yunnan, I think. The earliest I found on the net was that a song called something like "Red River Valley" in the late 1800s; a similar song was called "Bright Mohawk Valley" published in New York with different attributions.
Supposedly "Las Mañanitas" was supposed to be composed by Alfonso Esparza Oteo, but I had thought it was earlier.
It's like a lot of historical research. One just chases a melody down every musical rabbit hole and see what's therer.

Find the origin of folk


Project Euler #329 (Prime Frog) - Stochastic independence

Question:
Susan has a prime frog.Her frog is jumping around over 500 squares numbered 1 to 500. He can only jump one square to the left or to the right, with equal probability, and he cannot jump outside the range [1;500].(if it lands at either end, it automatically jumps to the only available square on the next move.)
When he is on a square with a prime number on it, he croaks 'P' (PRIME) with probability 2/3 or 'N' (NOT PRIME) with probability 1/3 just before jumping to the next square.When he is on a square with a number on it that is not a prime he croaks 'P' with probability 1/3 or 'N' with probability 2/3 just before jumping to the next square.
Given that the frog's starting position is random with the same probability for every square, and given that she listens to his first 15 croaks, what is the probability that she hears the sequence PPPPNNPPPNPPNPN?
Give your answer as a fraction p/q in reduced form.

My idea is this:
First calculate $P_n$ = $P\text{("frog sits on a prime after }n\text{ jumps")}$, which can be done by some basic combinatorics + evaluating the "ribbon". I got
\begin{equation}
P_0 = 19/100 = 95/500, P_1 = 24/125, P_2 = 571/3000, \dots
\end{equation}
which should be ok.
Next, for $n=0,\dots,14$ calculate
\begin{equation}
Q_n = P_n \cdot P(\text{"sits on $P$ and croaks }X_n\text{"}) + (1 - P_n) \cdot P(\text{"sits on $N$ and croaks }X_n\text{"})
\end{equation}
where $(X_n)_{n=0\dots14} = PPPPNNPPPNPPNPN$ is our croaking sequence and
\begin{equation}
  P(\text{"sits on $X$ and croaks }Y\text{"})=\begin{cases}
    1/3, & \text{if $X\dot{\lor}Y$}.\\
    2/3, & \text{else}.
  \end{cases}
\end{equation}
That is, $Q_n$ should exactly be $P(\text{"}n\text{th croak equals }X_n\text{"})$ and the product of the $Q_n$ should be the solution. But it's not, so where am I wrong? Are the $Q_n$ not stochastically independent? Whether the frog croaked $P$ or $N$ after $n$ steps should not affect the probability for him to croak $X_{n+1}$ next!?
My results are
\begin{equation}
119/300\text{ for $X = P$ (should be correct)}\\
17731/112500\text{ for $X = PP$ (should actually be $173/1125$ I think)}\\
\dots
\end{equation}
Comment: This whole discussion is kind of frustrating. I was expecting insights on a plain mathematical problem. What I got so far is **don't cheat, do it by yourself** and **what's the point of thinking if we have fast machines** (no offense guys)
Comment: What you got is "This is a programming problem that (according to the people here) **does not have a mathematical solution**". You simply picked the wrong problem in the hopes to apply some math. It seems to be of no use here.
Comment: I would write a program which computes probabilities of the given sequence of croaks for each of 500 starting points and take average.
Comment: Brute force would take like 8 Mio. steps, I'm looking for something a bit more efficient ;-)
Comment: I'm voting to close this question because it is a contest question.
Comment: And it seems to be much more complicated to tweak the naive approach, caching f.e.
Comment: @heropup What kind of contest you're talking about? It's a simple question by a simple guy who likes to get his head around some neat little problem. No fame or money included... so why the downvote?
Comment: @DonFuchs The point of Project Euler is to **learn how to solve the problems yourself,** not to have other people solve them for you.  Asking for outside help just so that you can increase your rank on the site is discouraged.
Comment: @heropup I tried it and I'm stuck. I may try the rest of my life and fail, or I may as well ask for **some advice** (nothing more) and learn something new. That's the point of ProjectEuler and every other maths problem. But today all has to be about ranks and credit of course.
Comment: I doubt that computationally efficient solution exists. Are you sure that the problem is not a plain programming problem?
Comment: Usually it's applying math to reduce a problem from "takes years to run when programming the naive approach" to "takes minutes to run when progtamming using ther math". This one seems to actually already be in the form that the naive brute force approch is doable. $500\times2^{15}\approx 16.3$ million steps is "nothing" for Euler problems and also nothing for modern computers. The high number of correct solutions seems to support that this is an easy problem.
Comment: @Ingix $16.3$ million may be doable, but that's not the point. I do not care about efficiency to reduce execution time, it's all about improving my programming skills as well as applying my (obviously inadequate) maths skills. And I think this is also in the spirit of PE to even tackle the simplest of questions with brains rather than brute force.
Comment: Right. Now a few people have told you that they don't see a way to significantly reduce the problem. I tried to point where your errors are in the my answer below. Maybe there is a way to attack the problem mathematically without brute force? I don't see it, a few others here don't see it. Maybe someone will see it and make an answer: Great, then we all will have learned something. In  the mean time, **my** opinion is that this is a problem that is easily solvable with brute force and no such matematical simplification exist. The 16 mio. steps just confirm the brute force approach works.
Comment: There are lots of other Euler problems that actually need to apply some maths to reduce the problem to something that can be done with brute force or rather an algoritm that has an acceptable running time. Again, it seems you have by bad luck picked the wrong problem to apply the skills you want to apply.
Comment: @Ingix How can a number problem (solved by a computer) not have a mathematical solution?? It's only maths inside, the computer is only doing maths. You can find highly sophisticated maths inside the dumbest of mathematical (or programming) problems.
Answer: I checked your $P_n$ calculation for $n=0,1$, so let's assume it's correct. Your main error is that the events "frog sits on a prime after n jumps" and therefore the event $Q_n$ describes are highly correlated for consecutive $n$.
After all, if the frog is on a prime number after n steps, this is highly likely an odd number. After one more jump it is then higly likely on an even number, which in turn is highly likely not $2$, so not a prime number. So the frog being on a prime before and after a jump is highly unlikely (just $2 \leftrightarrow 3$)
You approach might work if it was an "odd/even" Frog, that croaks differently based on if the number it's on is odd/even. Then you know for a fact that the Frog will alternate beween an odd and an even postition. But with prime numbers and their irregurities, you cannot hope to describe the differences between how prime numbers are distributed near one prime and another.
Comment: But now you have, the frog did croak previously, so the probability that it was on certain numbers is a posteriori different from what you assumed when calculating P_n. You can check this for just the croak sequence "P", for example. After the initial "P" croak, the probability  for frog to be after his first jump in any of the 500 possible postions is no longer the same as you assumed, because the frog was mor likely to have started from a prime than you assumed.
Comment: I agree that the $P_n$ events are highly correlated. But for the $Q_n$ events, is it not like throwing a dice: no matter how many 1s I've thrown before, the prop. to throw 1 next will always be 1/6.
No matter how often the frog has croaked P in the past, the prop. of the next P should only depend on the frogs position
Comment: To be more precise: that the frog did just croak P does not say anything about his position; he could have sit on a prime or not.
Comment: Exactly. But still you repeatedly use events like "frog sits on a prime after n jumps" when you have no idea what that probability is under the condition that the frog did the croaks he made before. What you are calculating is probabilities for a game where after each croak the frog is not jumping left or right, but randomly placed on primes or not primes according to your probability $P_n$. This is only the correct probability if you have no further information. ...
Comment: Thanks, I got it... so the $Q_n$ are independent, but my calculation of $Q_n$ is wrong as $P_n$ is not independent of the previous croaks...?! That's a start, I got to think about it.


Summation Skip Notation

Question: The following notation means to sum 1 to N:
$$\sum_{n=1}^N n$$
Is there a notation to not increment by one for each step, but, say, 10?
Answer: As long as you have a constant skip size, you can handle it by multiplying the subscript by the skip size wherever it appears inside the summation: in your example you'd get
$$\sum_{n=1}^N10n\;.$$
Of course that gives you the sum of all $N$ values from $10$ through $10N$. If you wanted just the sum of those multiples of $10$ that are no bigger than $N$, you'd have to adjust the upper limit:
$$\sum_{n=1}^{\lfloor N/10\rfloor}10n\;.$$
If you wanted to start with $n=1$ and increase in steps of $10$, for a total of $N$ terms, it would be
$$\sum_{n=0}^{N-1}(1+10n)\;.$$
And if you wanted to start with $n=1$ and increase in steps of $10$ up through a maximum value of $N$, it would be
$$\sum_{n=0}^{\left\lfloor\frac{N-1}{10}\right\rfloor}(1+10n)\;.$$
Answer: You could just write:
$$
\sum_{k = 1}^{N/10} 10 k
$$
Another option is to write conditions under the sum sign:
$$
\sum_{\substack{1 \le n \le N \\ 10 \mid n}} n
$$
Answer: Has anyone considered using programming notation? For example for an increment by 3
$$\sum_{\substack{n=0 \\ n+=3}}^{\inf} x$$
I think even better would be to put the increment in the middle of the sum (start under, increment middle, end top), but latex lacks support of that without making it look like the contents of the sum.
Comment: It is interesting that since I program a lot this is how I tend to think of the problem as well.

Summation Skip Notation


Switching file system from private to public makes all inline image paths broken

Question: I built a site with private file system initially, but now want to switch to public so that I can use performance enhancements like css and js minification as they are not available for private file systems.
The problem is that switching to public changes the file path and breaks all the inline image paths. Any ideas how to do this properly?
Just as an example, with private file system paths show as
mysite.com/system/files/myfile.jpg
this is what was stored by the body field for inline images. These change to
mysite.com/sites/default..../myfile.jpg
Answer: There's no miracle for that, you've to replace all path in the database. Fortunately MySQL has a replace function that will be useful for you.
First, you've to find every table/field that can contain inline image, if you're using Drupal 6, you'll look at node and node_revision for Drupal 7 it's more something like field_data_body and field_revision_body.
And finally you have to apply this query (for Drupal 7)
UPDATE field_data_body SET body_value = REPLACE(body_value, 'system/files/', 'files/');

Perhaps you should customize a query a little, but the main idea is there.
Comment: Appreciated!  I just found this as well http://www.drupalace.com/questions/how-to-fix-broken-image-paths-when-changing-download-method-from-private-to-public and http://drupal.org/node/258099
Comment: isn't it 'body_value' not 'value_body'?
Comment: @cybercampbell you're right, I edited my answer. Thanks for the notice.


Contradiction with Banach Fixed Point Theorem

Question: I am trying to find the fixed point of the function $g(x) = e^{-x}$. Wolfram Alpha tells me that this fixed point is approximately $x \approx 0,567$. However, if I apply the Banach fixed point theorem, I can prove that $g(x)$ has a fixed point in the interval $[2,\infty)$. I reasoned as follows:
Banach fixed point theorem:
Let $X$ be a Banach space, $D \subseteq X$ a closed interval and $T:D \rightarrow D$ a contraction, which means that $T$ is Lipschitz continuous with Lipschitz constant $L < 1$:
\begin{equation}
\Vert T(u) - T(v) \Vert_X \leq L \Vert u-v \Vert_X \text{ } \forall u,v \in D.
\end{equation}
Now $X = (\mathbb{R}, \Vert \cdot \Vert_1)$ is a Banach space and $D = [2,\infty)$ is a closed interval in $X$. Now, I show that $g(x)$ is a contraction: without loss of generality, assume $x < y$. Then
\begin{equation}
    \Vert g_1(x) - g_1(y) \Vert_1 = |e^{-x} - e^{-y}| = e^{-x} - e^{-y} = e^{-x}(1-e^{x-y}).
\end{equation}
Since $e^a \geq 1+a \text{ } \forall a \in \mathbb{R}$, I obtain
\begin{equation}
\begin{split}
    \Vert g_1(x) - g_1(y) \Vert \leq e^{-x}(1-(1+x-y)) = e^{-x}(y-x) \\
    \leq e^{-2}(y-x) = e^{-2}|x-y| = e^{-2} \Vert x-y \Vert_1 = L \Vert x-y \Vert_1,
\end{split}
\end{equation}
where $L = e^{-2}$ < 1.
So the conditions of the Banach fixed point theorem are satisfied and $g(x)$ has a fixed point in the interval $[2, \infty)$. However... This is not true!
Can anyone tell me what is going wrong here?
Thank you very much in advance!
Comment: The important line in the theorem is $T : D \to D$.  This is the hypothesis you are missing.
Answer: $e^{-x}$ does not map $[2,\infty)$ into itself.
Comment: @Acccumulation - the phrase "map into" does not, and should not, mean "injection". "into" in standard English usage indicates inclusion, not uniqueness. Attaching such a specialized meaning to it in Mathematics simply invites confusion to readers. And while I obviously cannot say that no author has used such a thing, I personally am not familiar with any text where "map into" is given such a meaning.
Comment: @Accumulation - So there is one person who is willing to make such an association in a forum post. However, I still say this is far from conventional. The texts I am familiar with do not use "into" to mean "injection", but simply when the function is not known to be surjective.
Comment: Regardless, the main point of this answer is that the range of $e^{-x}$ is not in $[2,\infty)$.  Whether "into" means "one-to-one" or not, $e^{-x}$ does not map $[2,\infty)$ into itself because the range is disjoint from the domain.
Comment: @Acccumulation I think I have uncommonly seen "into" and "onto" used to mean "injective" and "surjective", respectively. But in that sense, and including in the linked mathforum thread, I think they are always used like adjectives, which is not generally the case in English. Used as a preposition, I would have to assume the normal English meaning.
Comment: The phrase "map into" is often used to denote injections (and "map onto" used to denote surjections), and this is injective.
Comment: In standard mathematical usage, "into" does not mean one-to-one, but "injective" does. I suspect that perceived similarity between the words "into" and "injective" is behind the error in the mathforum post linked in the comment of @PaulSinclair
Comment: @PaulSinclair "An "into" or "one-to-one" function, also called an "injection" (which,
as you can guess, is French for "throwing into") moves the domain A
INTO B" http://mathforum.org/library/drmath/view/52454.html
Comment: @Acccumulation while the function is injective (which is not needed for the Banach theorem), the "itself" part is not fulfilled.
Answer: $e^{-2}$ is not in the interval $[2,\infty)$, so the image of the interval is not contained within the interval; i.e. $g$ does not map the interval to itself. In fact, the interval and its image are disjoint, so there can't possibly be a fixed point. One way of thinking of the Banach fixed-point theorem is that if you have an interval that is mapped to itself, then you can find a a sub-interval that is mapped to that sub-interval, and a sub-sub-interval of that sub-interval that is mapped to that sub-sub-interval, and so on, and the limit of that process is a single point that's mapped to itself. Once you get an interval that isn't mapped to itself, though, that interval doesn't necessarily have a fixed point, even though the space as a whole does. If you don't check that an interval is mapped to itself, you would find that the BFPT requires every interval to contain a fixed point, which is clearly absurd.
Comment: @FedericoPoloni You're right; the article on the BFPT says that L has to be in the interval [0,1), which I missed.


Visual Studio 2017 Solution Explorer is empty

Question: When I open an existing VS2015 solution file in VS2017, Solution Explorer is empty. I've tried to close and reopen Visual Studio several times, but the same thing happens every time. How to fix this?
Answer: This should fix it:

Close Visual Studio
Open Developer Command Prompt for VS 2017 (from Windows start menu)
Run devenv /updateconfiguration
Reopen Visual Studio
Comment: This did not fix the issue for me. I am opening a VS2015 solution from the pinned items in the Windows 10 task bar. Very disappointed with VS2017 so far.
Comment: @VinnyBrown I've seen another suggestion to delete .suo file(s). I've not tried that myself, so if you try it and it works, you can post a second answer.
Comment: Eivind Gussiås Løkseth  Nope, that didn't fix it. Seems this might be a problem with a corrupted MEF Cache (according to their documentation). I guess I'll have to wait for a fix. I can get the solution to open sometimes. Unfortunately, the same methods don't seem to work consistently. Sometimes I'll use the Open File icon, other times I'll open it from File/Open. Pretty disappointing from such a usually solid development tool.
Comment: This did not work for me either.  Double clicking on solution or opening from "most recent" list do not work.  Opening from File menu does work.
Comment: devenv /updateconfiguration worked for me.  My VS2017 version is VisualStudio/15.0.0+26228.9. .NET is 4.6.01586.
Comment: this worked for me without having to delete my *.suo file.
Answer: I'm experiencing the same issue. VS goes through all the steps when loading a solution, including "Preparing Solution", but then the Solution Explorer just comes up empty. So far I haven't found a solution. I tried deleting .suo files, and I tried cleaning the solution.
Like Eivind, I need to open the solution from the file menu. Fortunately, that has always worked for me so far.
Comment: The solution I gave Mar 8th worked for me. Tip if you've still got this issue: There's a small update available today. Maybe it'll fix it?
Answer: Today's update for VS2017 fixed it for me. Also see https://developercommunity.visualstudio.com/content/problem/24529/solution-doesnt-open-in-vs-2017.html
Answer: I faced this type of issue when opening project on visual studio 2017.
If you have connected with TFS and I believed that your connection has been lost.
Re-connect and close the project and then reopen using visual studio.
Hope this will helpfull!
Answer: On the Notification Area there is a message for Visual Studio 2017 updates. Click on that and this fixed my problem of empty Solution Explorer when u click on a 2015 sln file.


how to change ion-content bgcolor style except card content

Question: Hi I use ionic cdn in my web page
https://vipage.vivbook.com/lianaox/
Current site doesn't have ion-content tag.
The first problem is the screen is not scroll able.
I got that it is because of I didn't use ion-content
I add ion-content but the problem is how to use this animated css to the background of the page (not background of the card!)?
ion-content{

    background: linear-gradient(-45deg, #ee7752, #e73c7e, #23a6d5, #23d5ab);
    background-size: 400% 400%;
    animation: gradient 15s ease infinite;
}

@keyframes gradient {
    0% {
        background-position: 0% 50%;
    }
    50% {
        background-position: 100% 50%;
    }
    100% {
        background-position: 0% 50%;
    }
}

And is there anyway to enable scrollbar without using ion-content?
Comment: anyone knows???
Answer: If you're using Ionic 4@5.
ion-content {
    --background: *whatever you want here*
}

Or,
ion-content {
    --background: transparent;
    background: linear-gradient(*whatever you want here*);
}


Metric spaces - sets proof

Question: Let $(X,\rho)$ be a metric space and $A\subset X$. Prove that $\partial A = \overline A \cap \overline{X \setminus A} $.
I have no idea how to prove that. Please help.
Comment: Not every question including the word "set" is a question in set theory.
Comment: What definition of $\partial A$ are you using?  Have you solved other problems where you have to show that two sets are equal?
Comment: How is teh boundary defined?  As the closure less the interior?
Comment: C'est vrai, Asaf.
Comment: My definition is that $\partial A$ is the set of all points all of whose neighborhoods meet both $A$ and $A^c$.
Comment: Yes, I have boundary defined as closure less the interior.
Comment: @user74200 I hope my answer is helpful.  It is hard to give an optimal answer when I don't know what you've proven before, e.g. if you know that $\partial A = \partial (X \setminus A)$.  Please comment my answer if anything is unclear or if you have any further difficulty.
Comment: Or you can do that by the definition of the boundary of a set
Answer: To show two sets are equal, first assume we have an element of the first set, and show it is in the second set.  Then, assume we have an element of the second set, and show it is in the first.
Proof outline:

Let $x \in \partial A$ arbitrary.
Show that $x \in \overline{A}$ (this should be easy from your definition).
Then, show that $x \in \overline{X \setminus A}$ using the fact that $\partial A = \partial (X \setminus A)$.
Let $y \in \overline{A} \cap \overline{X \setminus A}$ arbitrary.
Show that $y \in \partial A$ by showing that $y \in \overline{A}$ but $y \not \in \text{Int } A$.
Comment: "using the fact that $\partial A=\partial(X\setminus A)$": Also proving that fact, of course:)
Comment: @JonasMeyer certainly :)  See also my comment on the question itself.

Metric spaces - sets proof


Geodesics Examples

Question: Can someone provide me exemples of connected Riemannian manifolds containing two points through each there are : (i) infinitely many geodesics (up to reparametrization)  and   (ii) no geodesics.
Thank you
Comment: For infinitely-many, if I understood you well, $S^2$ could work.
Answer: Ok, so for (i) we can consider $S^{2}$ and for (ii) we can consider $\mathbb{R}^{n}\setminus\lbrace 0\rbrace$ (with Euclidian metric).
Thanks anyway and have a nice weekend :)

Geodesics Examples


How to order data in sqlalchemy by list

Question: I have list of ID from external postgresql database.
A = [1,2,3,4,5,6,7,98,0]

I would to do query to database using SQLAlchemy, but I would to sort data in postgresql by A list.
I have read a lot of documentation but cannot find any suggestions how to do that.
So, in final I would to have:
results = session.query(user).limit(20).offset(10).order_by(A)

Cheers
UPDATE:
I found solution, it's not so good as I expected, but works well. Anyway if you know better solution, just let me know!
ids = ','.join([str(i) for i in A])
results = session.query(user).filter(user.id.in_(A)).\
limit(20).offset(10).\
order_by(" position(CONCAT(',',users.id::text,',') in ',{0},'.format(ids)")

Comment: BTW. I know that postgresql supports "join values" http://stackoverflow.com/questions/866465/sql-order-by-the-in-value-list, but cannot figure out how to implement this with sqlalchemy
Comment: I am curious about the reason for such ordering, would you mind elaborating more?
Answer: If you do not necessarily need to do this in SQL, you could simply sort the returned list of objects directly in python.
Example (using python's sorted function)
results = session.query(user).filter(user.id.in_(A)).all()
results = sorted(results, key=lambda o: A.index(o.id))

Answer: Here's the example of sorting by using SQLAlchemy case expression.
from sqlalchemy import case

from core.orm import session
from core.orm import models

query = session.query(models.Model)

ids_list = [1, 2, 4, 3, 5]

# order query by index of id in `id_list`

id_ordering = case(
    {_id: index for index, _id in enumerate(ids_list)},
    value=models.Model.id
)

# order
query = query.order_by(id_ordering)

Comment: Internally, that syntax compose this SQL sentence `ORDER BY CASE Table.column WHEN index THEN _id`.

Was helpful for me.
Answer: You can try:
results = session.query(user).limit(20).offset(10).order_by(*A)

Answer: Another way would be to create another helper table which contains order positions for each user.id, join on it, and order:
A = [1,2,3,4,5,6,7,98,0]
stmt = " UNION ALL ".join( 'SELECT {0} AS user_id, {1} AS order_id'.format(uid, row)
        for row, uid in enumerate(A))
ordq = text(stmt).columns(user_id=Integer, order_id=Integer).alias("t")
results = session.query(user).join(ordq, user.id == ordq.c.user_id).order_by(ordq.c.order_id).all()

I cannot judge whether this is better compared to your version, but it should at least be non-RDBMS specific.
Comment: Well i did think about this way, but probably it should be very slow, as you have to do one more operation in database and one more join instead of just one query.
Comment: no need for an operation in the database as you do this in-memory (as code shows). Would it be slow or not - very much depends on the length of this select statement.

How to order data in sqlalchemy by list


Changing a variable value every 5 seconds and return

Question: Why this script does not work correctly (freezes sometimes)?
var period = 600;

function Boom(){
    var timeBoom = window.setInterval(function() {
        if (period > 300) {
            period = 300;
            setTimeout(function () {
                period = 600;
            }, 1000);
        } else {
            period = 600;
        }
    }, 5000);
}

function Shake() {
    this.draw = function() {
        setTimeout(function () {
            Boom()
        }, 5000)
    };
}

I need to just every 5 seconds the function Boom() was called, but the variable should be changed back again after executing (var period = 600).
Comment: sooo, you are calling a delayed function and make execute every 5 seconds but make her change variables after 1 second each time its being executed? what exactly are you trying to do man...
Comment: You're creating timeouts inside intervals?
Comment: I need a loop in which the function Boom() is called every 5 seconds and changing variable (var period = 300 / 600).
Comment: i see. i have corrected your code, see my answer :)
Comment: Thnx. But then always be the same value (period = 300). The variable should be changed back again after executing Boom() (var period = 600).
Answer: John Resig -- creator of jQuery -- has a very nice piece on javascript timers.
Basically, becuse javascript is single-threaded, setInterval is not guaranteed to fire on every interval -- it will miss a beat if the js engine is busy at a particular cycle.
setTimeout will always call, but there could be an extra delay if another piece of code is queued up already when the time expires -- that is, the wait is potentially longer than the number of milliseconds you passed in as the second argument.
I'm guessing some of your setTimeout code might be getting in the way of one of the  setInterval loops, but that's just a guess.
If that's the issue, try naming the anonymous fn you passed into setInterval, and then pass it into setTimeout instead, with 5000 ms wait, and another call (recursive) to the same function at when the interval expires.
Answer: ok mate, basically you need to get rid of all the un needed timing stuff.
first of all, a brief explanation:
setTimeout (func, x) will execute function func after x milliseconds.
setInterval (func, x) will execute function func Each x milliseconds.
so you only need to set interval one time, and let it work. i have corrected your code:
var period = 600;

function Boom() {

    if (period != 300) {
        period = 300;
    } else {
        period = 600;
    }

}

function Shake() {
    this.draw = function () {
        var time = new Date().getTime();
        var shakeX = (Math.sin(time * 2.0 * Math.PI / period) + 0);
        this.x = shakeX;
        var shakeY = (Math.sin(time * 2.0 * Math.PI / period) + 0);
        this.y = shakeY;
        this.context.drawImage(image, 0, 0);
        this.context.translate(this.x, this.y);
        setInterval(Boom, 5000)
    };
}

Comment: I think you meant `setInterval(Boom, 5000)`
Comment: And the initial setInterval can just reference Boom without wrapping it in another needless function.
Comment: oh right, i didnt even notice it was wrapped. i corrected the answer, thanks :)
Comment: Thnx. But then always be the same value (period = 300). The variable should be changed back again after executing (var period = 600).
Comment: it will alternate between 300 and 600 every 5 seconds. if this is not what you wanted, can you please be more specific ?


Is there any way to accelerate tensorflow mobile on Android?

Question: any ideas is OK,cpu usage,tensorflow settings,image preprocess,etc.BTW,not tf lite,and my model is ssd-mobilenet
Answer: Would you consider to switch to TensorFlow Lite? It is the recommended approach to run TensorFlow models on mobile devices. It supports kernels which are optimized for ARM CPU and quantization.
See also Google's blog post for running SSD:
https://ai.googleblog.com/2018/07/accelerated-training-and-inference-with.html


Bulk emails - what to do with "over quota / mailbox full" undeliverable emails?

Question: We send out newsletters / bulk mail (double opt-in of course). We automatically delete hard bounces (mailbox unavailable, user unknown, host unknown...) right away from our mailing lists.
But a lot of "Undelivered" mails are generated because the user's mailbox is full / the user is over quota. So the email address still exists, but the mailbox is full.
So the two most common scenarios are probably
a) the user is still using his account and will delete some emails in the near future so he's not over quota any more and can receive emails again.
b) the user abandoned his account (for whatever reason), it ran full and that's that. It will never be used again...
So the first scenario tells me to NOT delete the email address from our mailing list, the second scenario tells me I should delete it. Of course, I don't know.
What's the best strategy in terms of working on the "reputation of my mail server / IP address"? Should I try to send for the next few days / couple of weeks and if it's still not working, should I treat it as b)?
I would assume that most ESP would delete inactive / full email addresses after a certain time so they will become hard bounces anyways? Or would an ESP discourage sending mails to a full mailbox too often?
Comment: Most commercial ESPs (in Poland) I've worked with rejects an e-mail immediately after establishing that e-mail box is full. I don't see anything wrong with it from postmaster, sysadmin and user point of view.
Comment: rejecting the email is fine. I mean, should I try send the next newsletter to a full mailbox or should I remove this email address from the mailing list so I won't be sending emails again to this address?
Comment: @Perrschy Have you considered some exponential back-off strategy with a notification included in the next successfully delivered mail indicating how many mails were not delivered due to mailbox full, and how the user can get them retransmitted - should they wish?
Answer: Your mail server should retry when mail delivery is deferred.  Do use a less aggressive retry policy such as after 1, 3, 6, 12, and 24 hours before failing.  You could also retry after 2 and 4 days. If the newsletters are time sensitive you may want an even shorter retry period.   You are unlikely to get penalized for retrying after a temporary failure, such as mailbox full.  You may be penalized for and aggressive retry policy.  I've seen servers retry several times a minute until they succeed.
Some organizations will pass temporary failures to a second server dedicated to handling retries.  If you do this, don't continue passing it to different servers.
If you send out mailings more than once a month, consider temporarily suspending addresses that bounce from the retry server.  Tacking how long an address is bouncing due to long term temporary failures (delivery success rate).  Consider dropping addresses that have failed for a few months.
You will accumulate dead addresses as not all organizations will bounce dead addresses.   Consider sending messages asking the users to opt-in again every year or so.  While you may not want to drop users immediately, you may want to drop them if they haven't responded for two or more years.
EDIT: Overquota/mailbox full is not the only reason you will get messages deferred.  You should retry all deferrals on a reasonable schedule as listed above.  The server will defer connections if it is to busy.  Greylists will defer messages from new servers that don't are not whitelisted.  My server also defers messages from servers that fail various policy checks: SPF, rDNS (IP address), rDNS (HELO/EHLO if not matching IP address) etc.  My greylisting is per network block, but my policy listing is done by server.  My policy list would apply to Microsoft controlled services if they weren't whitelisted.  Otherwise my policy lists have been catching spam bots, and junk mailers.
Comment: This probably makes most sense (also thanks to kasperd's comment): Track the not delivered due to mailbox full and implement a retry-strategy!
Answer: RFC1893 - Enhanced Mail System Status Codes:
<code>3.3 Mailbox Status
(...)

   X.2.2   Mailbox full

      The mailbox is full because the user has exceeded a
      per-mailbox administrative quota or physical capacity.  The
      general semantics implies that the recipient can delete
      messages to make more space available.  This code should be
      used as a persistent transient failure.
</code>


Understanding the Differences in VBScript/Javascript Regex to Solve SubMatch Issue

Question: I have a regex pattern that works perfectly in Python and various other languages, but is failing to capture the sub matches I need for my implementation in a VBScript regex (the engine of which is apparently almost identical to JavaScript). The pattern in question is as follows:
"Sincerely,[\s\n]+([\w\.]+)\s+(\w+)\s+(.+)[\s\n]+(\d+\s.+)[\s\n]+(.+)"

An example test case is as follows:
email received 3/30/17:

Dear Sir,

Hello

Sincerely,

Mr. Robert Thomas
1104 Madison Avenue
New York, NY 10021

email received 3/30/17:

Dear Sir,

Hello

Sincerely,

Ms. Angela Carraway
402 Arlington Drive
Concord, MA 01742

The objective is for a global regex that extracts 5 subgroups out of this example match after a variable keyword which here is "Sincerely,". The subgroups should be Ms. (1st subgroup), Angela (second subgroup), Carraway (third subgroup), 402 Arlington Drive (fourth subgroup), Concord, MA 01742 (fifth subgroup). In Python, it matches the 5 groups perfectly in a Regex tester, yet for VBScript (the JavaScript engine) it matches the entire string as a match, but with no subgroups at all. Therefore when I call the sub matches in an Excel VBA macro to write to a cell, I get all of the text jumbled up into a couple cells. What am I doing wrong? Is there some character that I am missing that is disabling capturing subgroups? If so what is the critical difference between these two engines so that I can avoid this in the future and how could one fix this pattern in this test case? I've tried reading about the differences online, yet everything said seems to be only small differences that should cause the issue I am having. Any help would be greatly appreciated because I cannot seem to isolate the difference/problem. Thank you!
Edit:
The following is the VBA code that utilizes the regex:
Sub regex()
    Dim docxinput As String
    Dim keyword As Variant
    Dim patterninput As Variant
    Dim pattern As String
    Dim regex As New RegExp

    docxinput = Application.GetOpenFilename(Title:="Step #1: Enter Word Document Input File Name")
        Dim wrdApp As Word.Application
        Dim wrdDoc As Word.Document
        Dim strInput As String

        Set wrdApp = CreateObject("Word.Application")
        wrdApp.Visible = False

        Set wrdDoc = wrdApp.Documents.Open(docxinput)
        strInput = wrdDoc.Range.Text

        Debug.Print (strInput)
        wrdDoc.Close 0
        Set wrdDoc = Nothing
        wrdApp.Quit
        Set wrdApp = Nothing

    pattern = "Sincerely,[\s\n]+([\w\.]+)\s+(\w+)\s+(.+)[\s\n]+(\d+\s.+)[\s\n]+(.+)"

    Dim objMatches As MatchCollection

    With regex
        .Global = True
        .MultiLine = True
        .IgnoreCase = False
        .pattern = pattern
    End With

    Set objMatches = regex.Execute(strInput)
    Dim row As Variant

    Dim SubMatches As Variant
    row = 2
    For Each SubMatches In objMatches
        Cells(row, 1).Value = objMatches(0).SubMatches(0)
        Cells(row, 2).Value = objMatches(0).SubMatches(1)
        Cells(row, 3).Value = objMatches(0).SubMatches(2)
        Cells(row, 4).Value = objMatches(0).SubMatches(3)
        Cells(row, 5).Value = objMatches(0).SubMatches(4)
        row = row + 1
    Next
End Sub

This is a picture of the results. As you can see, The first two subgroups work but then the regex (or at least I think) runs into  grouping error and dumps almost of the other content into the next column. It then moves onto the fourth column, running into errors there as well. Is this an issue with the code iterating or the regex itself. I have tried to troubleshoot the code and cannot find reasons why it cannot break the text up correctly other than the regex being at fault. Any thoughts?
The picture:
Comment: Could you post the VBA code that you're using the execute the regex and retrieve the submatches?
Comment: @RichHolton I have made edits to the original post that shows the code and further commentary along with a picture of the results I get when running the VBA. Any thoughts?
Comment: I suspect that your text has something other than \n separating some of the lines, so the third group (.*) captures too much, which then throws the rest off. Can you check to see?
Comment: @RichHolton I think you re right as with the modified code form Plirkee below, it works great, yet this Word doc part causes to err. I think possibly the strInput = wrdDoc.Range.Text does something that adds or subtracts characters causing the regex to fail. Is there a more elegant way to perform a regex on the text without running the associated risks of mashing it into on string?
Comment: I *think* that Word uses \r for paragraph marks, and \n for line breaks. So try replacing [\s\n] with [\s\r\n].
Answer: Your regex should run with VBA with no problems...
(tested it here)
To get the desired group in vba take a look here how-to-use-regular-expressions-regex-in-microsoft-excel-both-in-cell-and-loops.
Edit:
For the following input:
email received 3/30/17:

Dear Sir,

Hello

Sincerely,

Mr. Robert Thomas
1104 Madison Avenue
New York, NY 10021

email received 3/30/17:

Dear Sir,

Hello

Sincerely,

Ms. Angela Carraway
402 Arlington Drive
Concord, MA 01742

which was put inside the cell A1
and the vba code:
(note that I had to alter your for each loop - so that this would work for multiple matches)
Sub myregex()
    Dim keyword As Variant
    Dim patterninput As Variant
    Dim pattern As String
    Dim regex As New RegExp

    Set Myrange = ActiveSheet.Range("A1:A1")
   For Each C In Myrange
   strInput = C.Value
   strPattern = "Sincerely,[\s\n]+([\w\.]+)\s+(\w+)\s+(.+)[\s\n]+(\d+\s.+)[\s\n]+(.+)"

     With regex
                .Global = True
                .MultiLine = True
                .IgnoreCase = False
                .pattern = strPattern
            End With
            If regex.Test(strInput) Then
                 Set objMatches = regex.Execute(strInput)
                 row = 2
                 For Each SubMatches In objMatches
                 Cells(row, 1).Value = objMatches(row - 2).SubMatches(0)
                 Cells(row, 2).Value = objMatches(row - 2).SubMatches(1)
                 Cells(row, 3).Value = objMatches(row - 2).SubMatches(2)
                 Cells(row, 4).Value = objMatches(row - 2).SubMatches(3)
                 Cells(row, 5).Value = objMatches(row - 2).SubMatches(4)
                 row = row + 1
                Next
            Else
                C.Offset(0, 1) = "(Not matched)"
            End If

    Next
End Sub

I got the following result:
     A      B       C           D                    E
  2  Mr.    Robert  Thomas      1104 Madison Avenue  New York, NY 10021
  3  Ms.    Angela  Carraway    402 Arlington Drive  Concord, MA 01742

Conclusion:
   Everything works as expected.
Comment: I tried that tester too and it yields everything as one match with no subgroups. Is that normal? I need the 5 captured subgroups within that group if that makes sense. I have also edited my original post to show more detail of the issue I having with what I presume is the regex pattern.
Comment: @J.Squillaro So, in that tester (my first link) if you choose `split lists` tab you will see your 5 groups there. For how to get each group in vba - look at my second link.
Comment: @J.Squillaro your `vba` script worked for me fine in my excel (of course after some adaptions - I didn't use word document but an excel cell with one string `Dear Sir,

Hello

Sincerely,

Mr. Robert Thomas
1104 Madison Avenue
New York, NY 10021` - however regex part worked -  excel 2010, vb script regular expressions 5.5  )
Comment: @J.Squillaro take a look at my test results (edited answer)
Comment: This works perfect under the conditions if put it into Cell A1! I think that through all of this, the Word doc part of the code is the culprit as if I do your code with the my Word doc input, I get my same issue. So perhaps, as Rich Holton was thinking above, strInput = wrdDoc.Range.Text concatenates the text poorly causing the regex to fail. Is there any way, in your experience, one can just open the file and perform a regex on it as in Python without having to combine lines together and put it into a string? I think that's the final issue here.
Comment: @J.Squillaro I think that  now we have narrowed down the problem space.  However, this, in my opinion, is a separate issue - and has little to do with the original question (the way it was presented). The original question has been answered....


Laravel Model Binding missing relationships

Question: I'm trying to convert to Model binding on a Model built on customer orders. My route:
Route::model('order', 'App\Models\Order');
Route::resource('orders', 'OrderController');

This allows me to pull up an Order to edit through my controller (also grabbing statuses to populate a table and passing the logged in user):
public function index(Order $order)
 {
  $orders = $order->get();
  return view('orders.index', compact('orders'));
}

My orders.index will display $order->id properly but when I try to loop through the Actions, which is connected by a hasMany relationship, nothing shows. Or I try to show $order->user->firstname which belongs to User by user_id.
@foreach( $order->actions as $action )
  {{ $action->type->type }}
@endforeach

From my Action model:
public function order()
{
    return $this->belongsTo('\App\Models\Order', 'order_id');
}

From my Order model:
public function actions()
{
    return $this->hasMany('\App\Models\Action', 'order_id');
}

Here's an excerpt from a dump of the Order:
`Illuminate\Database\Eloquent\Collection Object
(
    [items:protected] => Array
        (
            [0] => App\Models\Order Object
                (
                    [table:protected] => orders
                    [timestamps] => 1
                    [dates:protected] => Array
                        (
                            [0] => deleted_at
                        )

                    [connection:protected] =>
                    [primaryKey:protected] => id
                    [perPage:protected] => 15
                    [incrementing] => 1
                    [attributes:protected] => Array
                        (
                            [id] => 1
                            [created_at] => 2015-03-16 23:42:45
                            [updated_at] => 2015-03-19 04:37:53
                            [deleted_at] =>
                            [user_id] => 16
                            [status_id] => 5
                            [address_id] => 5
                            [datetime_pickup_actual] =>
                            [datetime_delivery_actual] =>
                            [datetime_pickup_requested] => 2015-03-20 17:00:00
                            [datetime_delivery_requested] => 2015-03-21 17:00:00
                            [hold] => 0
                            [weight] => 20
                        )

                    [original:protected] => Array
                        (
                            [id] => 1
                            [created_at] => 2015-03-16 23:42:45
                            [updated_at] => 2015-03-19 04:37:53
                            [deleted_at] =>
                            [user_id] => 16
                            [status_id] => 5
                            [address_id] => 5
                            [datetime_pickup_actual] =>
                            [datetime_delivery_actual] =>
                            [datetime_pickup_requested] => 2015-03-20 17:00:00
                            [datetime_delivery_requested] => 2015-03-21 17:00:00
                            [hold] => 0
                            [weight] => 20
                        )

                    [relations:protected] => Array
                        (
                        )

                    [hidden:protected] => Array
                        (
                        )

                    [visible:protected] => Array
                        (
                        )

                    [appends:protected] => Array
                        (
                        )

                    [fillable:protected] => Array
                        (
                        )

                    [guarded:protected] => Array
                        (
                            [0] => *
                        )

                    [casts:protected] => Array
                        (
                        )

                    [touches:protected] => Array
                        (
                        )

                    [observables:protected] => Array
                        (
                        )

                    [with:protected] => Array
                        (
                        )

                    [morphClass:protected] =>
                    [exists] => 1
                    [forceDeleting:protected] =>
                )`

Answer: As you compact orders in your controller you should use orders in your Blade template:
@foreach( $orders->actions as $action )
  {{ $action->type->type }}
@endforeach

You should be also aware that Order in your index method in controller have nothing in common with Route Model Binding. For index it won't be used at all.
For other methods (show/edit/delete) it won't work because you make a binding with wrong name. It should be:
Route::model('orders', 'App\Models\Order');

and not
Route::model('order', 'App\Models\Order');

Answer: Model name (MoneyEntry)
Changed from
Route::resource('moneyentries', 'MoneyEntryController');

to
Route::resource('moneyEntries', 'MoneyEntryController');

Did a trick.


Please tell me there's a better way to set the ScrolledText view window

Question: I have this sample code to explore the inner workings of ScrolledText in an attempt to discover why:

The scrollbar flickers when I set the view window after replacing the text contents.
If the text slightly exceeds the view window, the view window is set to a value other than what I specify. Note that this is not caused by line rounding since the value is off by multiple lines.
If the text greatly exceeds the view window, while the view window is initially set to the value I specify, it is then overridden twice. The final time causes the view window jumps to a very incorrect position. Presumably causing [1].

#!/usr/bin/env python3

import tkinter as tk

# This test program is a stand-in for ScrolledText, so I can examine
# the effects of 'yscrollcommand'.
#from tkinter import scrolledtext

root = tk.Tk()

frame = tk.Frame(root)
frame.pack()

text_out = tk.Text(frame, state=tk.DISABLED)
text_out.pack(side=tk.LEFT)

def from_scollbar_yview(*args, **kwargs):
    print("\ntext_out.yview: ", args, kwargs)
    return text_out.yview(*args, **kwargs)

scrollbar = tk.Scrollbar(frame, command=from_scollbar_yview)
scrollbar.pack(side=tk.LEFT, fill='y')

button = tk.Button(root, text="copy")
button.pack()

text_in = tk.Text(root)
text_in.pack()

def from_text_set(*args, **kwargs):
    print("\nscrollbar.set: ", args, kwargs)
    print("... and yview is: ", text_out.yview())
    return scrollbar.set(*args, **kwargs)

text_out.configure(yscrollcommand = from_text_set)

def on_click(event):
    y1 = text_out.yview()
    print("\ncopy start: ", y1)
    s = text_in.get("1.0","end-1c")

    # Attempted work-around. If the explanations below had been
    # correct this, though extremely hacky, would have worked.
    #yc = text_out.cget("yscrollcommand")
    #i = 0
    #def yc_ignore_n(*args):
    #    nonlocal i
    #    print("called yscrollcommand: ", i, *args)
    #    if i >= 1:
    #        print("setting yview[0]: ", y2[0])
    #        text_out.yview("moveto", y2[0])
    #        text_out.configure(yscrollcommand = yc)
    #        print("yscrollcommand restored!!")
    #    i += 1
    #    return True
    #if y1[0] > 0 or y1[1] < 1:
    #    print("implementing workaround")
    #    text_out.configure(yscrollcommand = yc_ignore_n)

    print(f"copying text: {len(s)} chars")
    text_out.configure(state=tk.NORMAL)
    # I now think these explanations are incorrect:
    #
    # If the text height changes, 'delete' will queue a
    # 'yscrollcommand' at the *front*. This *will* mess
    # with the 'yview'.
    text_out.delete(1.0, "end")
    # If the text height changes, 'insert' will queue a
    # 'yscrollcommand' at the *front* (before 'delete').
    text_out.insert("end-1c", s)
    text_out.configure(state=tk.DISABLED)

    print("setting yview[0]: ", 0.25)
    text_out.yview("moveto", 0.25)
    # When the contents only slightly exceed the size of the
    # text, 'yview' is set to an incorrect value. Why?
    y2 = text_out.yview()

    print("copy stop:", y2)

button.bind("<Button-1>", on_click)

root.mainloop()

Notice the workaround commented-out. It only works for [3], so go ahead and paste in a very long string. I'm using this Lorem Ipsum generator.
Here's the output for a very long string, [3]. I'm trying to set the top of the view window at 25% but I end up at 15%. The comments in my code explain why this is happening.
...
copy start:  (0.0, 1.0)
copying text: 2623 chars
setting yview[0]:  0.25
copy stop: (0.25, 1.0)

scrollbar.set:  ('0.25', '1.0') {}
... and yview is:  (0.25, 1.0)

scrollbar.set:  ('0.15625', '0.75625') {}
... and yview is:  (0.15625, 0.75625)

Here's the output for a slightly long string, [2].
copy start:  (0.1724137931034483, 1.0)
copying text: 2569 chars
setting yview[0]:  0.25
copy stop: (0.35135135135135137, 1.0)

scrollbar.set:  ('0.35135135135135137', '1.0') {}
... and yview is:  (0.35135135135135137, 1.0)

While I now think the explanation in the code comments is incorrect, I am certain that they are caused by yscrollcommand being called excessively. I can't completely disable yscrollcommand without breaking scrollbars. I can't be certain a callback to set the correct yview will run after all queued yscollcommand, nor will it fix the flickering [1].
Comment: What exactly is your question?
Comment: How to set the `ScrolledText` view window.
Comment: You might try including a `wrap = "none"` when declaring each `Text` widget. This is necessary if you use scrollbars, otherwise `wrap char` (the default) will control text layout!
Comment: Which version of Tk? The `text` widget was rewritten substantially in 8.5 to have a more sophisticated content measuring model.
Comment: @DonalFellows: 8.6
Answer: The first yscrolledcommand is queued for either state change (normal/disabled). The second from insert (not delete) is triggered when there is line wrapping outside the text view. Possibly to adjust the view position to any line wrapping. Since the adjustment causes the scrollbar to shrink, that implies it initially considers wrapped lines as a single line, then considers wrapped lines as multiple lines. Normally 25% is 25% but since the view can't move during the adjustment, and there's now more content, the top position percentage must be reduced. This is what causes the flickering.
For partially-obscured wrapped lines this only triggers at the bottom.
The initial configuration of the window view only considers wrapped lines that are visible and does not cause any flickering. For partially-obscured wrapped lines this only considers lines at the top. In their handling of wrapped lines, the initial and final view configurations are exact inverses.
Anyway I want to demonstrate the Tk is doing it completely wrong. Let's say I have 20 non-wrapped lines, a single lined wrapped into 20 lines, and 20 non-wrapped lines. I want to set the top of the scroll window at 25% which means the wrapped line will be completely visible.
At 25%, this is the first line that should be visible:
(12+20*2)*.25
>>> 13.0

However Tk shows a portion of the 11th line. What does this correspond to?
(1+20*2)*.25
>>> 10.25

10.25/(12+20*2)
>>> 0.1971153846153846

And that's what Tk adjusts to top of my yview to: roughy 19%.
Now let's say I have 40 non-wrapped lines and a single line wrapped into 20 lines. I want to set the top of the scroll window at 25% which means no wrapped lines will be visible.
I'd show the calculations, but they're pretty much exactly the same. The only difference is that the scrollbar will flicker. Instead:
copy start:  (0.0, 0.46153846153846156)
copying text: 1025 chars
setting yview[0]:  0.25
copy stop: (0.25, 0.8353658536585366)

scrollbar.set:  ('0.25', '0.8353658536585366') {}
... and yview is:  (0.25, 0.8353658536585366)

scrollbar.set:  ('0.1971153846153846', '0.6586538461538461') {}
... and yview is:  (0.1971153846153846, 0.6586538461538461)

So how do we solve this? You can try to undo Tk's math, working backwards to get the value that will be transformed into the value you want. However, you will still get flickering. The best solution is to wrap the lines yourself (wrap = "none"). For that, you need the window size. While window size is specified in terms of font size at creation, that is not maintained internally. All you can get is the window size in pixels using winfo_width. Luckily Font.measure provides a pixel width of a given string for the instantiated font.
The only question is it worth it? You'll have to bind to <Configure> to rewrap every time the widget is resized. That also means deciding how to adjust the scrollbar and which view to show.
Comment: You'll only get flickering in Tk if you've done something horribly wrong, like having several callbacks fight over what to do in response to an event. This is because Tk postpones its redraws until the event processing system is quiescent.
Comment: @DonalFellows You've seen the example in the question, has something horribly wrong been done? There are several callbacks fighting over the scrollbar position.


Adding a wildcard (*) to a PHP if statement

Question: I'm trying to update the "$searchValue" below to include a wildcard  before & after so if the string in "$searchValue" is found anywhere in the $label it will work. Any ideas on how to do that?
        foreach($json as $key=>$value){

            foreach($value as $item)
            {
                if($item->$label == "$searchValue")
                {
                    echo $item->title.'
';
                }
            }
        }

Thanks for your help!
Comment: What you're actually trying to do is determine whether the search value appears as a substring of the values you're checking against, which you can do with [`strpos`](http://php.net/strpos).
Answer: foreach($json as $key=>$value){

        foreach($value as $item)
        {
            if(strpos($item->$label, $searchValue) !== false)
            {
                echo $item->title.'
';
            }
        }
    }


Normal Operators: Numerical Range

Question: Disclaimer:
As I realized in the comments that this works for normal operators I decided to modify this question. Besides, I got the proof now - thanks to T.A.E.!
Prove that for normal operators the spectrum is contained in the closure of the numerical range:
$$\sigma(N)\subseteq\overline{\mathcal{W}(N)}$$
Answer: If the distance from $\lambda$ to $\mathcal{W}(A)$ is $d > 0$, then
$$
              |((A-\lambda I)\phi,\phi)| \ge d\|\phi\|^{2},\;\;\; \phi \in \mathcal{D}(A)\\
               \implies d\|\phi\| \le \|(A-\lambda I)\phi\|,\;\;\; \phi \in\mathcal{D}(A).
$$
If $(A\phi,\phi)=(\phi,A^{\star}\phi)$ for $\phi$ in a core domain of $A^{\star}$, then you also get
$$
              d\|\phi\| \le \|(A^{\star}-\overline{\lambda}I)\phi\|,\;\;\;\phi \in \mathcal{D}(A^{\star}).
$$
So, in this case $\mathcal{N}(A^{\star}-\overline{\lambda}I)=\{0\}$. Using the adjoint relation, $\mathcal{R}(A-\lambda I)^{\perp}=\mathcal{N}(A^{\star}-\overline{\lambda}I)$, it follows that $(A-\lambda I)$ has dense range and bounded inverse. So, under these assumptions, $\lambda \in \rho(A)$.
Comment: Assume $\|Ax\| \ge d\|x\|$ for some $d > 0$ and for all $x$ in a core $C$ for $A$. To say that $C$ is a core for $A$ means that $\{ (x,Ax)\in X\times X : x \in C\}$ is dense in the graph of $A$. So, if $x \in \mathcal{D}(A)$, there exists $\{ c_{n} \}\subset C$ such that $c_{n}\rightarrow x$ and $Ac_{n}\rightarrow Ax$. So $\|Ax\|=\lim_{n}\|Ac_{n}\|\ge \lim_{n}d\|c_{n}\|=d\|x\|$.
Comment: What if $\mathcal{D}(A)\cap\mathcal{D}(A^*)=\{0\}$?
Comment: I cannot think of an argument that will work in that case. There's not much to work with. Assuming that some subspace of $\mathcal{D}(A)$ is a core for $\mathcal{D}(A^{\star})$ is a stretch, too. I'd say that the argument almost requires a bounded $A$.
Comment: Hmm I was hoping to expand the proof from Werner's book to closed ones with dense range but I thought so too that this probably won't work - to bad...
Comment: Btw your argument fails for nonreal lambda since the numerical range is not closed under conjugation in general. You will note this if you go through the proof for the self adjoint case where (there you'll have to consider two separate cases for lambda).
Comment: All I am assuming is that $|(A\phi,\phi)-\lambda(\phi,\phi)| \ge d(\phi,\phi)$ for all $\phi \in\mathcal{D}(A)$. If $\phi$ is also in the domain of $\mathcal{D}(A^{\star})$, then you automatically get $|(A^{\star}\phi,\phi)-\overline{\lambda}(\phi,\phi)|\ge d(\phi,\phi)$. Then if some some subspace of $\mathcal{D}(A)$ is a core for $A^{\star}$, you get the desired result for $A^{\star}$ that $\mathcal{N}(A^{\star}-\overline{\lambda}I)=\{0\}$.
Comment: But how can you deduce injectivity from dense subspaces alone? In general for discontinuous a.k.a. unbounded operators this will be wrong...
Comment: Ok cool (if I did not any mistake) this actually works for all normal operators. Hope you don't mind if I merge your proof together with the assumption that the operator is normal...
Answer: Disclaimer: This prove is mostly due to T.A.E. - greatest regards to him!

Preparation:
(This is where real work has to be done.)
The distance of $\lambda$ to the numerical range gives us the estimate:
$$d(\lambda,\mathcal{W})\leq|\langle \hat{x},(N-\lambda)\hat{x}\rangle|\leq\|(N-\lambda)\hat{x}\|\implies \|(N-\lambda)x\|\geq d(\lambda,\mathcal{W})\|x\|$$
Moreover since $N$ is normal and densely defined we have $N^*N=NN^*$ as a really nontrivial result (see proposition 4.17 b in [1]). Also since $N$ is normal it is closed and since it is densely defined $\mathcal{D}(NN^*)$ is a core for $N^*$ (see proposition 4.11 a in [1]). We thus have:
$$d(\lambda,\mathcal{W})\leq|\langle \hat{c},(N-\lambda)\hat{c}\rangle|=|\langle (N^*-\overline{\lambda})\hat{c},\hat{c}\rangle|\leq\|(N^*-\overline{\lambda})\hat{c}\|\quad \hat{c}\in\mathcal{D}(N^*N)=\mathcal{D}(NN^*)\implies\|(N^*-\overline{\lambda})y\|=\lim_n\|(N^*-\overline{\lambda})c_n\|\geq\lim_n d(\lambda,\mathcal{W})\|c_n\|=d(\lambda,\mathcal{W})\|y\|$$
Main Work:
(The prove will be now an easy consequence.)
Assume $\lambda\notin\overline{\mathcal{W}}$.
Since $\lambda$ has positive distance to the numerical range $(N-\lambda)$ will be bounded below by the first estimate given in the preparation and due to this especially injective. Moreover since $N$ is normal it is closed so as well $(N-\lambda)$ and $(N-\lambda)^{-1}$. Thus by the closed graph theorem applied to $(N-\lambda)^{-1}$ the range of $(N-\lambda)$ is closed.
On the other hand we have that the $(N^*-\overline{\lambda})$ is bounded below as well due to the second estimate given in the preparation. But therefore the kernel is zero only and thus we have:
$$\overline{\mathcal{R}(N-\lambda)}=\mathcal{N}(N^*-\overline{\lambda})^\perp=\{0\}^\perp=Y$$
That is the range of $(N-\lambda)$ is dense.
Collecting all together $(N-\lambda)$ is injective surjective and bounded below.
So $\lambda$ was in the resolvent set as was to be shown.

Reference: [1]: German version of Weidmann's 'Lineare Operatoren in Hilberträumen'

Normal Operators: Numerical Range


Comparing residuals in two regression models

Question: Let us have two linear models. Let $\alpha_i$ be real numbers same in both.
$$
LM1:  Y=\alpha_0 + \alpha_1X_1+\alpha_2X_2 + \varepsilon
$$
$$
LM2:  Y^\prime=\alpha_0 + \alpha_1X^\prime_1+\alpha_2X^\prime_2 + \varepsilon`
$$
The difference between them is only in the fact that $X_i, X^\prime_i$ are different. Now, I want to test a hypothesis that the noise distribution remains the same $H_0: \varepsilon\overset{D}{=}\varepsilon`$ (we assume noise has a zero mean and finite variance). We have available random samples, from the first one of length $n$: $(Y_1, X_1)^\top, \dots, (Y_{n}, X_{n})^\top$, and from the second one with length $m$:  $(Y`_1, X`_1)^\top, \dots, (Y`_{m}, X`_{m})^\top$.
My approach:  Based on these samples, we estimate $\hat{\alpha_i},\hat{\alpha_i}^\prime $ where e.g. $\hat{\alpha_i}^\prime$ is the estimation using LSE from the LM2.
Now, compute the residuals. So $\hat{\varepsilon}=Y-\hat{\alpha_0}-\hat{\alpha_1}X_1 - \hat{\alpha_2}X_2$ and similarly for the second model $\hat{\varepsilon}^\prime$.
Asymptotically  $\hat{\varepsilon}\overset{D}{=}\hat{\varepsilon}^\prime$, as $n,m\to\infty$ because estimates of the coefficients are consistent. BUT, for finite $n$, we did not converge yet to these limits. If I do e.g. KStest ($ks.test(rez1, rez2)$), it will reject this hypothesis too often even if it is true. Is it a correct test at least asymptotically? Do you know about some other test comparing these distributions with a proper $p-$ value?
My interest is more academic and I want a test that provably works.
Here is also some idea of the sample code I used
\documentclass{article}
\usepackage{listings}
\begin{document}

\begin{verbatim}

n=1e+04
f <- function(X1, X2){return(-1 + 1X1 + 2X2)}
X11=rnorm(n, 0,1);X12=rnorm(n, -1, 2)
Y1= f(X11, X12) + rLaplace(n)
fit=lm(Y1~X11+X12, data= data.frame(Y1, X11, X12))
rez1=fit$residuals

X21=rexp(n, 1);X22=rexp(n,2)
Y2= f(X21, X22) + rLaplace(n)
fit2=lm(Y2~X21+X22, data= data.frame(Y2, X21, X22))
rez2=fit2$residuals
ks.test(rez1, rez2))
\end{verbatim}

\end{document}

Comment: I am sceptical of your statement about the KS test.  How have you tried to apply it?  How did you determine its false positive rate in this situation? The concern here is whether you truly want to test for identity of *distribution* or only want to compare some properties of the distributions, such as their variances.
Comment: Thanks for your input. Ok, here is what I did: generate $Y=1+10*X_1+20*X_2+\varepsilon$ for some random $X_1, X_2, \varepsilon$. Compute $rez1=lm(Y\sim X_1+X_2)\$residuals$.  Do the same thing with different $X_1, X_2$ and finally test $ks.test(rez1, rez2)$. This will reject more than once in every 20generations.
Comment: Thank you: I see what you mean.  There are several problematic aspects of this approach that make *any* distributional test inapplicable.  First is that the residuals within one model are not independent; the second is that they are guaranteed to have a mean of zero.  The first becomes a non-issue as the dataset grows large, but the second is critical.  If instead of doing any regression you simply were to generate the $X_i$ independently with a common continuous distribution and run `ks.test(x1-mean(x1), x2-mean(x2))` you would experience identical problems.
Comment: The crux of the matter is that by forcing the means to be equal, you make the two sets of values look more similar to each other than would occur in two independent random samples from a common distribution.  What ultimately rescues us is that it's neither necessary nor advisable to conduct a formal distributional test of residuals, making this problem purely academic (although certainly an interesting one).
Comment: I see and fully agree with the mean problem. I added the fact that we assume the mean to be zero. It is interesting that because of this, residuals will be centred around zero "more" than actual starting random samples.

Considering your first problem- yes, residuals are dependent but with decreasing dependence as $n\to\infty$. But I don't know if it means that the test will be bad or not (asymptotically).

And yes, it is a purely academic problem, I am only trying to prove that it works (and I am using it as a part of a different research problem), or find some better test.
Comment: The assumption makes no difference. In any least squares model that includes an intercept, the mean of the residuals is guaranteed to be zero by construction.
Comment: Yes, I see (although if the noises differ only as a shift, we wouldn't observe this in residuals (it would only change $\alpha_0$)). But anyway, I don't see why is it a bad thing- it should "correctly not reject" more often, but asymptotically it is still neglicable.
Comment: Can we assume that $\epsilon$ and $\epsilon^\prime$ are both Gaussian? If not, can we assume they're from the same (location-scale) family of distributions?
Comment: We can not assume Gaussianity, unfortunately. But most of the time, we are interested in results when they are the same (which we assume under the null hypothesis). Under alternative, they can be basically anything with finite variance
Comment: So $\epsilon$ and $\epsilon^\prime$ can have any distributions? They don't have to be in the same family (i.e. both t-distributions, cauchy distributions, gamma distributions)? We could have, for example, $\epsilon \sim \Gamma(\alpha, \beta)$ and $\epsilon^\prime \sim Cauchy(\eta, \gamma)$?
Comment: hmm. I guess we can assume continuity and full support on R. And not Cauchy(1,1) because it does not have finite variance.
Comment: But no, actually in the research, they are used to model "hidden covariates", where we do not really impose any restrictions on the distribution of covariates. But if it helps, I am definitely interested even if they lie in the same family (and even on X you can assume something "reasonable" if you want), it would still help
Answer:
I can replicate the problem from the question with the code below. The discrepancy between the distribution of the test statistic and the Kolmogorov distribution can be changed by manipulating the fit. If we use an intercept lm(Y~1+X1+X2)$residuals then the difference is largest.

Without intercept, the difference is only small

With intercept, the difference is large

The case with only an intercept
We get the discrepancy already when we use only an intercept, ie. if we would model lm(Y~1)$residuals.
The discrepancy due to the intercept is easier to understand. The sampling of residuals is not independently drawing from a distribution for the residuals. Instead, the residuals will be correlated (the same is true for the case without intercept, but less strong).
This is most easy to see for the case of a sample of size 2 that is being modeled by only an intercept. The residuals will equal $\pm |x_1-x_2|$ and are symmetric.
More generally, for larger $n$, the residuals can be equated as
$$r_i = \epsilon_i - \frac{1}{n} \sum_{j=1}^n \epsilon_j $$
This makes the residuals multivariate normal distributed, if the $\epsilon$ are normal distributed. If we consider $\epsilon \sim N(0,1)$ then the covariance matrix for the $r_i$ is
$$
\Sigma_{ij} = \begin{cases} \frac{(n-1)^2}{n^2}  & \quad \text{if $i=j$}\\
 -\frac{n-1}{n^2}  & \quad \text{if $i \neq j$}
\end{cases}
$$
We can obtain the same result by sampling from a multivariate normal distribution with this correlation. In the code below we did this with the function sim2 which produces the following image
Intuitively, what's happening?
The Kolmogornov Smirnoff test compares the deviation of the empirical distribution function. Let's focus on this distribution in a single point, at zero.
The error terms, if they are distributed as a standard normal distribution, then they will have fifty-fifty probability to be above or below the point zero. The empirical distribution in the point zero will follow a binomial distribution with $n$ trials and $p=0.5$.
The residual terms will be different. The residual terms are transformed error terms.
The fit with an intercept will shift the terms up or down. If there are many error terms above zero then a mean above zero is more likely and a shift down will be more likely. If there are many error terms below zero then a mean below zero is more likely and a shift up will be more likely.
So in comparison to the error terms, the residual terms will have a smaller variation for the empirical distribution in the point zero.
R-code
### function to simulate data
### and compute test statistic for KS test
sim = function(n = 100, plot = "n") {
  X1 = rnorm(n)
  X2 = rnorm(n)
  Y = rnorm(n)
  R1 = lm(Y~0+X1+X2)$residuals

  # use these alternatives below to get the clearest discrepancy
  #R1 = lm(Y~1+X1+X2)$residuals
  #R1 = lm(Y~1)$residuals

  X1 = rnorm(n)
  X2 = rnorm(n)
  Y = rnorm(n)
  R2 = lm(Y~0+X1+X2)$residuals

  # use these alternatives below to get the clearest discrepancy
  #R2 = lm(Y~1+X1+X2)$residuals
  #R2 = lm(Y~1)$residuals

  if (plot == "y") {
    plot(R1[order(R1)],1:n, type ="l")
    lines(R2[order(R2)],1:n, col = 2)
  }
  test = ks.test(R1,R2)
  return(test$statistic)
}

### function to simulate data using multivariate normal
sim2 = function(n = 100, plot = "n") {
  ### create covariance matrix for mvrnorm distribution
  Sig = matrix(-(n-1)/n^2,n,n)
  diag(Sig) <- rep((n-1)^2/n^2,n)

  R1 = MASS::mvrnorm(1, mu = rep(0,n), Sigma = Sig)
  R2 = MASS::mvrnorm(1, mu = rep(0,n), Sigma = Sig)

  if (plot == "y") {
    plot(R1[order(R1)],1:n, type ="l")
    lines(R2[order(R2)],1:n, col = 2)
  }
  test = ks.test(R1,R2)
  return(test$statistic)
}

### perform 10^3 simulations and plot histogram
set.seed(1)
n = 600
D = replicate(10^4,sim(n))

br = seq(0,max(D)+d,1/n)*sqrt(n/2)
hist(sqrt(n/2)*D, breaks = br, freq = 0)

### function to compute kolmogorov distribution
pkolm = function(x) {
  k = 1:100
  sqrt(2*pi)/x*sum( exp(-(2*k-1)^2*pi^2/(8*x^2)) )
}
pkolm = Vectorize(pkolm)

### add kolmogorov distribution to histogram
P = c(pkolm(br))
P[1] = 0
points(br[-1]-br[2]/2, diff(P)/br[2])

Comment: Sorry for my late response, I had a "no internet" holiday. And thanks for the response. As for the context: I agree with what you wrote. Sure, residuals are dependent with correlation going to 0 as $n\to\infty$, and will be shifted in some direction (even in only the intercept case), but again, consistent as $n\to\infty$. However, I don't really understand the the conclusion- so is testing by ks.test(rez1,rez2) wrong? You say that residuals will have smaller variation as the error terms. That should even shift for more stronger test statistic, shouldn't it?
Comment: @AlbertParadek Yes, the conclusion is that it is wrong.
Comment: Would it help If I compute an estimate for the distribution in the intuitive example case, which considered only the statistic for the difference between the empirical distributions in a single point on the distribution curve rather than the entire curve (the latter, entire curve/distribution comparison, is the KS test)? This example case is easier because the statistic in the ideal situation follows a binomial distribution. But, the discrepancy is very similar as what we would get with the KS test.
Comment: I am sorry, but I am still trying to understand what you are doing- I feel like I don't really comprehend the point. So you are taking only the case when $Y=c+\epsilon$ and comparing $Y-\hat{c}$?
Comment: @AlbertParadek I am comparing two samples of residuals as if the individual residuals in a sample are iid distributed. But, because this independency condition is violated (which happens especially due to the intercept) we get a discrepancy when we compute the KS statistic. What I explain in the answer is not for the KS statistic, which is the largest difference between the two empirical distributions, but for the simpler difference between the two empirical distributions in the point zero (which doesn't follow the Kolmogornov distribution but the simpler binomial distribution).
Comment: @albertparadek *"So you are taking only the case when $Y=c+\epsilon$ and comparing $Y−\hat{c}$?"* Yes, this simpler situation is sufficient to replicate the problem from your question. Effectively we have two samples of random numbers $\epsilon_i$ from which we compute residuals $r_i = \epsilon_i - \bar{\epsilon}_i$. The KS statistic and the Kolmogorov distribution work for the comparison of the empirical distributions based on the $\epsilon_i$, but not for the empirical distributions based on the $r_i$.


How do I include a JavaScript library without using a module?

Question: Note: I'm very new at using Drupal, so I may be missing something.
I've searched high and low, but I can't seem to find an answer to this.
I've got a custom Drupal theme based on Zen, and I want to include a jQuery plugin (specifically, Lightbox) into a page that needs it (I'm using Panels and Views to create custom landing pages). Frankly, I'd be happy even if I had to include it on the entire site.
I looked at JavaScript Libraries Manager, jQuery Plugins Module, but these didn't seem to be able to include CSS or images, which Lightbox requires.
I tried placed the script and CSS/image files in my theme's folder, and then using the following in mytheme_preprocess_page() in my theme's template.php:
$options = array('group' => JS_THEME);
drupal_add_js(drupal_get_path('theme', 'mytheme') . "/js/jquery.lightbox.js', $options);

This loads the JavaScript file OK, but it doesn't function correctly because it's looking for the images at the web root / (and I didn't include the CSS file, anyway).
I saw an example of using hook_library(), and this looks promising as it allows specification of CSS files as well, but I'm not sure where/how to include this setup code.
Also, I realize that there is a Lightbox module, but because the way that my content types and views are done, I can't add rel="lightbox" to the anchors...
Answer: It doesn't sound like you need to hook the theme system or deploy a module, why not try including the paths to Light Box's dependencies in one or more of your template files?
Better yet add the CSS and JS file paths (relative to the theme folder) to your theme's info file.
Stylesheets
Scripts
Comment: I'm having a funny problem: the CSS file worked fine when adding it to the .info file, but the script isn't being added correctly when I reference it like `scripts[] = js/jquery.lightbox.js`. The strange thing is that if I rename the script to just `script.js` and update the .info file accordingly, the script gets loaded correctly...
Comment: My guess, off hand, is that whatever parses the info file doesn't understand two periods in the js file name.  Maybe try renaming the file and specifying jquery_lightbox.js and see if the file is resolved that way.


Simple chord progression question

Question: I am learning the piano online.
I am trying to learn chord progressions.

In this simple example the chord progression is supposed to be C F G C (I, IV, V, I).
However, in the third part (G) the first note is D.
Can someone explain why that part is in G?
Comment: @No'amNewman It's not: the third chord has a G in the bass, but a D in the melody. This sounds harmoniously, because D is one of the notes of the major chord on G (G B D).
Comment: Without looking at the music, I would say that the third chord is a G with D in the bass, i.e. second inversion.
Comment: I wrote 'without looking at the music'. Looking at the music, it's clear that the D is in the melody.
Comment: Here is a detailed answer on which melody notes work well with a given chord [http://music.stackexchange.com/a/30595/16897](link)
Answer: The melody of a song, that is being played on top of a chord, can be any of the notes of the chord (but is not limited to those1 !).
The part you mentioned has G as a chord. The notes that consist the G chord are G,B,D; you can play any of those notes in the melody and they will sound nice. Τhe order in which they are played doesn't affect the harmony in the least. You can play the notes in any way you want (you can also omit some of them, like the last bar of your example).
The given melody starts off with the third note of the chord, D, then goes to G and then to B. It plays all the three notes of the G chord.
1 For example, you can see on the F chord that there are the notes E & D that don't belong  to the chord. These notes are used to pass from one note of the chord to some other. They are called Passing Tones.

Simple chord progression question


Closed form for integral $\int_{0}^{\pi} \left[1 - r \cos\left(\phi\right)\right]^{-n} \phi \,{\rm d}\phi$

Question: Is there a closed form for $$I_n =\int_{0}^{\pi} \frac{\phi}{(1 - r \cos\phi)^n} \,{\rm d}\phi $$ for $\left\vert\,r\,\right\vert < 1$ real and $n > 0$
integer ?
The solution to this integral would give a closed form solution for this integral, which describes the interaction energy of vector resonant relaxation in astrophysical dynamics.
Using Mathematica and analytic methods I have derived the following result for $n=\{1,2,3,4,5,6\}$:
$$\tag{1} I_n = -\frac{a_n r}{s^{2n-2}} + \frac{b_n}{s^{2n-1}}\left[\chi_2(q) +\left(-\frac{c_n}{b_n}s + {\rm arctanh}(q) \right){\rm arctanh}(r) \right]$$
where $s=\sqrt{1-r^2}$, $q=\sqrt{(1-r)/(1+r)}$, $\chi_2(x)$ is the Legendre chi function, $a_n$, $b_n$, and $c_n$ are constants given by
\begin{align}
a_1 &= 0, \quad b_1 = 4, \qquad c_1 = 0,\\
a_2 &= 0, \quad b_2 = 4, \qquad c_2 = 2,\\
a_3 &= 1, \quad b_3 = 4 + 2r^2, \qquad c_3 = 3,\\
a_4 &= \frac{7}{3}, \quad b_4 = 4 + 6r^2, \qquad c_4= \frac{11}{3} + \frac{4}{3}r^2,\\
a_5 &= \frac{23}{6}+\frac{11}{12}r^2, \quad b_5 = 4 + 12 r^2 + \frac{3}{2}r^4,
\qquad c_5=\frac{25}{6} + \frac{55}{12}r^2,\\
a_6 &=\frac{163}{30} + \frac{47}{12}r^2, \quad b_6 = 4 + 20 r^2 + \frac{15}{2}r^4,
\qquad c_6=\frac{137}{30} + \frac{607}{60}r^2 + \frac{16}{15}r^4.
\end{align}
Is the integral $I_n$ in the general closed form given by (1) for all $n$? If so, what are the constants $a_n$, $b_n$, and $c_n$?
Comment: @AhaanRungta: I did not mean it to. I said Please and Thanks. What would have been a more polite way?
Comment: @robjohn The same thing could have been mentioned in a more polite manner. There's nothing wrong in your statement but it came off a bit rude. No offense!
Comment: a.) Please X. Thanks.

b.) Could you please x so that Y? Edited this time. Thank you. <- which do you think sounds better?
Comment: @CarlMummert: I added the context to the question.
Comment: Please [don't use displaystyle in the title](http://meta.math.stackexchange.com/questions/3135/why-no-use-displaystyle-in-titles). Thanks.
Comment: @robjohn No problem, if the equation didn't show up in the title for some reason, I have added it to the main body.
Comment: Hypergeometric functions first come into my mind...
Comment: @AhaanRungta: The link provided a much more comprehensive reason why. I didn't think I needed to repeat it.
Comment: @bkocsis: Don't worry; it wasn't your doing. Someone else edited the title to use displaystyle.
Comment: @sos440: It is no need to call for the hypergeometric functions as $n$ in this question is an integer.
Comment: You could improve this question by including more context. In particular, what is the interest in this integral? There is a general description about how to write questions on this site at http://meta.math.stackexchange.com/questions/9959/how-to-ask-a-good-question
Comment: This integral should has closed form. You can just apply the [Weierstrass substitution](http://en.wikipedia.org/wiki/Tangent_half-angle_substitution), the only thing is how the tedious rational function integral you should face.
Comment: Needs more detail to be considered an answer and not just a comment, I think.
Comment: While not a complete answer, it is a useful comment. The substitution $t=\tan(x/2)$, $x=2{\rm arctan\,}t$, gives
$$(1−r)^{−n}\int_0^{\infty}\frac{ 4 (1+t^2)^{n−1} {\rm arctan\,}t}{(1+q^2t^2)^n} dt$$
where $q=(1+r)/(1−r)$. While Mathematica 9.0 cannot solve this for any arbitrary positive integer $n$, it does produce closed form solutions for $n=\{1,2,3,4,5,6\}$ using ${\rm arctanh\,}(q)$, ${\rm LerchPhi\,}(q^2,2,k/2)$, and ${\rm PolyLog\,}(2,\pm q)$ functions where $k=\{1,3,\dots,2n−1\}$. Perhaps there is a closed form solution for arbitrary $n$ using these functions.
Comment: The same question at MO: http://mathoverflow.net/questions/150539/closed-form-for-int-0-pi-frac-phi1-r-cos-phin-d-phi
Answer: First lets make the integral from $-\pi/2$ to $\pi/2$. Set $x = \phi-\pi/2$. We then get
$$I = \int_{-\pi/2}^{\pi/2}(1+r\sin(x))^{-n}(x+\pi/2)dx$$
Now recall that
$$(1+r\sin(x))^{-n} = \sum_{k=0}^{\infty} (-1)^k \dbinom{n+k}k r^k \sin^k(x)$$
Hence,
$$I = \sum_{k=0}^{\infty}(-r)^k \dbinom{n+k}k \int_{-\pi/2}^{\pi/2} (x+\pi/2)\sin^k(x)dx$$
Now from here, we can obtain
$$\int_{-\pi/2}^{\pi/2} \sin^k(x) dx \text{ and }\int_{-\pi/2}^{\pi/2} x\sin^k(x) dx$$ Lets call them $I_k$ and $J_k$ respectively. Hence,
$$I = \dfrac{\pi}2\sum_{k=0}^{\infty}r^{2k} \dbinom{n+2k}{2k} I_{2k} - \sum_{k=0}^{\infty}r^{2k+1} \dbinom{n+2k+1}{2k+1} J_{2k+1} \tag{$\star$}$$

Expression for $J_{2k+1}$ and convergence of the above summation:
We have
$$J_{2k+1} = -\int_{-\pi/2}^{\pi/2}x \sin^{2k}(x) d(\cos(x)) = \int_{-\pi/2}^{\pi/2} \cos(x) \sin^{2k}(x) dx + 2k \int_{-\pi/2}^{\pi/2} x \cos^2(x) \sin^{2k-1}(x) dx$$
$$\int_{-\pi/2}^{\pi/2} \cos(x) \sin^{2k}(x) dx = \int_{-1}^1 t^{2k} dt = \dfrac2{2k+1}$$
$$\int_{-\pi/2}^{\pi/2} x \cos^2(x) \sin^{2k-1}(x) dx = J_{2k-1} - J_{2k+1}$$
Hence,
$$(2k+1)J_{2k+1} = \dfrac2{2k+1} + 2k J_{2k-1}$$
Using this recurrence you can obtain $J_{2k+1}$. Also, it is easy to show that $\left \vert J_{2k+1} \right \vert \leq 2$ using induction.
Similarly, $I_{2k} = \dfrac1{4^k} \dbinom{2k}k \pi \leq \pi$.
Now, for $\vert r \vert < 1$, we have
$\displaystyle \sum_{k=0}^{\infty} \dbinom{n+2k}{2k} r^{2k}$ and $\displaystyle \sum_{k=0}^{\infty} \dbinom{n+2k+1}{2k+1} r^{2k}$ converges.
This ensures $\star$ makes perfect sense.
Comment: @bkocsis Also, can you remove the down-vote. My answer is a perfectly valid answer.
Comment: I was looking for a closed form expression, i.e. a finite sum of known functions. See edited version of the question.
Comment: @bkocsis I don't understand what you mean by closed form. The above is a closed form in my opinion. If you get the answer as $\sin(k)$ or $\cos(k)$ or $\Gamma(k)$ or $\arctan(k)$, etc, do you think that is closed form? All these are infinite series and not a closed form. By truncating the above sum, you can get an arbitrary accurate answer.
Comment: @bkocsis You are showing a wiki article to mean what a closed form is? I assume that you are interested in calculating the value of the integral for a fixed $r$ and $n$. If so, the infinite series above is a perfectly valid answer.
Comment: The beauty of closed form expressions is that the parameter dependencies become tractable, and that they are often calculable quickly. Most CPUs and GPUs can calculate many transcendental functions extremely efficiently. Closed form results can speed up simulations very significantly. If you can show that your sums converge for $|r|<1$ and give $J_{2k+1}$ specifically, I can remove the down vote.
Comment: @bkocsis Added. Also, if you need, I can send you a robust implementation accurate up-to $15$ digits.
Answer: The integral can be evaluated by realizing that it is a derivative with respect to $R=1/r$, i.e.
$$I_n =\int_{0}^{\pi} \frac{\phi}{(1 - r \cos\phi)^n} \,{\rm d}\phi = R^n \int_{0}^{\pi} \frac{\phi}{(R - \cos\phi)^n} \,{\rm d}\phi = \frac{(-1)^{n-1} R^n}{(n-1)!} \frac{d^{n-1}}{dR^{n-1}}\int_{0}^{\pi} \frac{\phi}{R - \cos\phi} \,{\rm d}\phi. $$
This integral can be evaluated with a Weierstrass substitution $t = \tan (\phi/2)$,
$$F(R) = \int_{0}^{\pi} \frac{\phi}{R - \cos\phi} \,{\rm d}\phi
= \frac{4r}{1+r}\int_0^{\infty} \frac{\arctan t}{q^2 + t^2} dt
= \frac{4}{\sqrt{R^2-1}}\int_0^{\pi/2} \arctan \left(q \tan y \right) dy
=  \frac{4}{\sqrt{R^2-1}}\left[\chi_2(q) - {\rm arctanh}(q) \ln(q) \right]
=\frac{4}{\sqrt{R^2-1}}\left[\chi_2(q) +\frac{1}{2} {\rm arccosh}(R) {\rm arctanh}(R) \right],$$
where $q=\sqrt{(1-r)/(1+r)}$ and $\chi_2(x)$ is the Legendre chi function. We can now derive the integral by taking derivatives of this function
$$I_n =  \frac{(-1)^{n-1} R^n}{(n-1)!} \frac{d^{n-1}}{dR^{n-1}}F(R).$$
Note that ${\rm arccosh\,}R=2{\rm arctanh\,}q$, ${\rm arccoth\,}R=\ln q$, and
$$\frac{d}{dx}\chi_2(x) = \frac{{\rm arctanh}(x)}{x} \quad{\rm so\quad} \frac{d}{dR}\chi_2(q) = \frac{{\rm arctanh}(q)}{R^2 - 1} = \frac{1}{2}\frac{{\rm arccosh}(R)}{R^2 - 1} $$
$$\frac{d}{dR} {\rm arccosh}(R) = \frac{1}{\sqrt{R^2 - 1}}  \quad{\rm and\quad} \frac{d}{dR} {\rm arccoth}(R) = \frac{-1}{R^2 - 1}
= \frac{1}{2}\left(\frac{1}{R+1} - \frac{1}{R-1}\right).$$
The derivatives for $n\geq 3$ simplify to
$$I_n = \frac{R^n}{S^{2n-2}}\left\{ \frac{4 A_{n-1}}{S}\left[\chi_2(q) - {\rm arctanh(q)\ln q}\right]\right. \\
\left.+2\sum_{k=0}^{n-2} \frac{A_k A_{n-k-2}}{k+1} \ln q
+ \sum_{j=0}^{n-3}\sum_{k=0}^{n-3-j}\frac{A_k A_{n-3-j-k}}{n-1-k} \frac{(R-1)^{1+j}-(R+1)^{1+j}}{1+j}
 \right\} $$
where $S=\sqrt{R^2-1}$ and $A_n$ is a polynomial of $R$ defined as
$$A_n = \frac{(-1)^n}{n!}\left(R^2-1\right)^{n+(1/2)}\frac{d^{n}}{dR^{n}} \left(R^2-1\right)^{-1/2}\\
=\left(\frac{R+1}{4}\right)^n\sum_{s=0}^{n}\frac{(2s)!}{(s!)^2}\frac{[2(n-s)]!}{[(n-s)!]^2}q^{2s}.$$
The coefficients are related to Legendre polynomials $P_{2n}(0)=(-1)^n(2n)!/[4^n (n!)^2]$.
The result simplifies to
$$
I_n=\left(\frac{R}{R-1}\right)^n \frac{q^2}{4^{n-2}}\left\{
a(q) [\chi_2(q) - {\rm arctanh(q)}\ln q]
+b(q)\ln q + c(q)\right\}$$
where
$$
a(q)=\sum_{K=0}^{n-1}q^{2K-1}
\frac{(2K)!}{(K!)^2} \frac{[2(n-1-K)]!}{[(n-1-K)!]^2}\,,
$$
$$
b(q)=\sum_{K=0}^{n-2}q^{2K}
\sum_{s=0}^{K}\frac{(2s)!}{(s!)^2}\frac{[2(K-s)]!}{[(K-s)!]^2}
\sum_{j=K}^{n-2}\frac{[2(j-K)]!}{[(j-K)!])^2}\frac{[2(n-2-j)]!}{[(n-2-j)!]^2}\frac{2}{j-s+1}\,,
$$
$$
c(q)=\sum_{K=0}^{n-3}q^{2K+2}
\sum_{J=0}^{K} \frac{4^{K-J+1}}{1+K-J} \sum_{s=0}^{J}\frac{(2s)!}{(s!)^2}\frac{[2(J-s)]!}{[(J-s)!]^2}\\
\quad\times
\sum_{j=K}^{n-3}\frac{[2(j-K)]!}{[(j-K)!])^2}\frac{[2(n-3-j)]!}{[(n-3-j)!]^2}\frac{1}{n-1-j+s+K-J}\\
\quad+\sum_{K=0}^{n-3}q^{2K}
\sum_{J=K}^{L-3} \frac{-4^{J-K+1}}{1+J-K}\sum_{s=0}^{K}\frac{(2s)!}{(s!)^2}\frac{[2(K-s)]!}{[(K-s)!]^2}\\
\quad\times
\sum_{j=J}^{n-3}\frac{[2(j-J)]!}{[(j-J)!])^2}\frac{[2(n-3-j)]!}{[(n-3-j)!]^2}\frac{1}{n-1-j+s+J-K}\,.$$
Answer: Not a full answer, but maybe interesting:
$I_{n}(r)$ obeys a recursion relation:
$$
I_{n+1}(r)=I_{n}(r)+\frac{r}{n} \frac{\partial}{\partial r}I_{n}(r)
$$
A large $n$ approximation is:
$$
I_{n}(r)\stackrel{n \rightarrow \infty}{\sim} \int_{0}^{\pi}\phi \exp(n\, r \cos \phi)\mathrm{d}\phi,
$$
which shows that for large $n$ the integral depends only on the product $n\,r$. Interestingly, also the approximation seems not to be solvable in closed form.
For the approximation I used:
$$
\left(1 - \frac{z}{n}\right)^{n} \stackrel{n \rightarrow \infty}{\sim} e^{- z}
$$


Gaussian Processes Probabilistic Least Squares Classifier - Leave one Out Cross-Validation Means, Variances Shape?

Question: I am currently working on implementing some of the algorithms covered in Rasmussen and Williams' book, and stuck on a particular part in chapter 5 (link to the chapter here).
In particular, on page 117, in equation 5.12, there seem to be major explanations missing there, the most important of which being that one would expect both the LOO-CV mean and variance at a single training point to be a single value - the point I'm stuck on/need help clarifying.
EDIT: I should clarify that below, the subscript $_{i}$ indicates that it is the $i^{th}$ row that is removed before inversion, and subscript $_{ii}$ that the $i^{th}$ row and column are removed before inversion. The syntax is misleading, but can be found appropriately in some papers, as well as directly implied by the preceding and subsequent paragraph where these equations are (by way of explanations of efficiency of inverting the covariance matrix over and over).
However,
$$\mu_i = y_i - \frac{[K^{-1}y]_i}{[K^{-1}]_{ii}}$$
is, to my understanding, an $m \times m$ matrix, where $m = n-1$, and $n$ is the size of the square covariance matrix. The same goes for the other equation,
$$\sigma_i^2 = \frac{1}{[K^{-1}]_{ii}} $$
which also produces an $m \times m$ matrix as opposed to a single value.
I'm sure that I have misunderstood something (possibly quite fundamental/basic), or am missing some implied knowledge of some sort that would solve my supposed problem.
Any help greatly appreciated!
Answer: On the previous page, they state that the subscript $-i$ refers to the $i$th row being removed before inversion. I believe that in their notation $a_{i}$ refers to the $i$th element of vector $a$, and likewise, $A_{ii}$ refers to the $i$th element on the diagonal of matrix $A$.
So for the mean:
$y_i$ is a scalar. $[K^{-1}\mathbf{y}]$ is a vector, and thus $[K^{-1}\mathbf{y}]_i$ is the $i$th element of that vector. $K^{-1}_{ii}$ is a scalar.
Comment: Thanks Kevin. I upvoted the response but haven't hit 15 rep yet sp it doesn't show D:. I guessed as much/originally implemented it that way, and was misled by a particular paper. Is it implied the *y* in those omit the item at *i*, with the same applying to the original training data before calculating _K inverse_?


How to append url using rewrite?

Question: When user visited my website www.abc.com, I would like to make it to be www.abc.com\xx\yy\zz.html. However if user visited anything other than www.abc.com, the url should remain the same.
Expected:
www.abc.com => www.abc.com\xx\yy\zz.html
www.abc.com\xx => www.abc.com\xx
www.abc.com?hello=123 => www.abc.com?hello=123
I know I should use rewrite and redirect to achieve this but I'm not sure how to conditionally check for above situation
Comment: Can you check for `^$` ? I'm looking [here](https://www.nginx.com/blog/creating-nginx-rewrite-rules/) but unsure of exact implementation. Something along the lines of `rewrite ^$ /xx/yy/zz.html` maybe?
Answer: How about:
rewrite ^/$ /xx/yy/zz.html;

This should match the root URL and rewrite to /xx/yy/zz.html.

How to append url using rewrite?


Easy way to see that $(x^2 + 5x + 4)(x^2 + 5x + 6) - 48 = (x^2 + 5x + 12)(x^2 + 5x - 2)$?

Question: As the title suggests, is there an easy way to see that$$(x^2 + 5x + 4)(x^2 + 5x + 6) - 48 = (x^2 + 5x + 12)(x^2 + 5x - 2)$$that doesn't require expanding in full? Is there a trick?
Answer: $$(x^2 + 5x + 5-1)(x^2 + 5x + 5+1) - 48 = (x^2 + 5x + 5)^2-7^2$$
Comment: Great catch! Upvoted.
Comment: This is what was intended. Wonder what makes the admin not accept it till now @EmperorConcerto?
Answer: For $a=x^2+5x+4$ we get to factor $a(a+2)-48=a^2+2a-48=(a+8)(a-6)$.
Answer: If $x^2 + 5x+12 = 0$ then
$$(x^2 + 5x + 4)(x^2 + 5x + 6) - 48 = (-8)(-6)-48 = 0.$$
If $x^2 + 5x-2 = 0$ then
$$(x^2 + 5x + 4)(x^2 + 5x + 6) - 48 = 6\cdot 8-48 = 0.$$
Therefore LHS and RHS have the same zeroes. Since RHS has no double roots and both sides are polynomials of degree $4$, we conclude LHS = RHS.
Comment: You also need to see that they both have leading coefficient one
Comment: @HagenvonEitzen Sure, that is evident.
Answer: For me it's just Vieta's formulas (or comparing coefficients as if $x^2+5x$ were a single variable):
$(x^2 + 5x + \color{red}4)(x^2 + 5x + \color{red}6) \color{red}{-48} = (x^2 + 5x + \color{red}{12})(x^2 + 5x \color{red}{-2})$?
$4+6=12+(-2)$, and $4\cdot 6 - 48 = 12 \cdot (-2)$, so yes.