The theory behind western pentatonic scales

Question: I'm curious about a more in-depth explanation as to why the major and minor pentatonic scales are prevalent in the western music tradition (Yes, i make a claim here that some might consider debatable. Please hear me out).
I did a brief googling and the most common explanations were: "because it's easy" or "because it sounds good", or the likes. It's not that i disagree, but i feel that there probably is more to the theory than that.
For instance, why are the fourth and seventh notes omitted in a major pentatonic scale (in reference to a major diatonic scale)? It makes me wonder since both the fourth and the fifth are perfect intervals, but only the latter is kept. Is it simply because it goes along the vein of a musical tradition revolving around the I-V relationship?
I'm assuming one could argue that the seventh is left out because of the dissonant quality in relation to the root, no?
Could anyone shed some more light on why this is? I like to hear some of your thoughts on this!
Comment: Related (YouTube video of Bobby McFerrin demonstrating universal human understanding of the pentatonic scales): https://youtu.be/Hodp2esSV9E
Comment: @NeilMeyer and marhsalcraft, there are [many traditions](https://en.wikipedia.org/wiki/Pentatonic_scale#Pervasiveness) that use pentatonic scales, including the music of ancient Greece. I don't think it's a narrowly limited or recent phenomenon.
Comment: @leftaroundabout but linguistics research has shown that these people do not intuitively understand fiveness. There's no evidence that the visual cortex can learn to identify groups of five in the absence of an iterative process to identify that number (to train the visual cortex, if you will). Humans also seem to be able to grasp heptatonic scales without a theoretical basis, so I don't see anything special in that regard about pentatonic scales.  There's no need to have a theoretical basis for a scale to use it, regardless of the number of notes it contains.
Comment: @leftaroundabout there are languages that lack words for specific numbers, and native speakers of those languages seem not to know the number five.
Comment: Pentatonic is basically a modern invention, not very prevalent at all.
Comment: @Todd Wilcox That was so awesome. Makes you wonder whether those notes are known a priori, or learned.
Comment: @Erik the pentatonic is mathematically so simple that it's hard to see how anybody could "not know it", any more than you could possibly "not know the number 5": even somebody who'd never heard the word "five" could still identify that number of items.
Comment: I think the origins go back to native American Indians. Their flutes were in pentatonic, maybe they just made them even intervals. Then likely adapted to western music.
Comment: Outside of that I don't think there is any significance to it.
Comment: This question appears to be two questions. One, what are the tonal qualities of the pentatonic?  Two, why is the pentatonic scale popular?  Suggest that the question be split into two questions so that the two questions can be more clearly answered.
Comment: @marshalcraft so Africans and east Asians got their pentatonic scales from the Americas?
Comment: @phoog that was kind of my point: even those speakers who don't have a _word_ for the number five still intuitively understand five-ness. That's few enough so basically the visual cortex already "counts" it as a pattern without any of the iterative process that we use to count things (which does become necessary for higher numbers). Likewise, humans seem to grasp pentatonics without any theoretical justification whatsoever.
Comment: @phoog "linguistics research has shown..." [citation needed]. If you mean the original Sapir-Whorf studies, those are flawed in many ways. Ok, yes, for the cortex speculation I indeed don't have any evidence whatsoever, but I personally find five to be indeed about the limit of objects I can immediately quantify in any picture, without needing to actually count. — As for heptatonic scales: these aren't so widespread. Of course you find diatonic modes everywhere where European culture has left its mark, but for the very different Arabic or Indonesian scales the number 7 seems to be coincidence.
Comment: @leftaroundabout I think I was thinking of something from Steven Pinker's *The Stuff of Thought,* but I can't find my copy right now.  There's also the work of Peter Gordon and a paper co-authored by Pinker and Ray Jackendoff, both cited in chapter 5 of Christine Kenneally's *The First Word.* Gordon's work in particular supports my statement. And if Arabic and Indonesian scales having seven notes is coincidental, doesn't that strengthen my hypothesis that it's possible to use scales without having a theoretical basis for them?  At least, it does not weaken it.
Answer: A common "academic" description of these two pentatonic scales is anhemitonic. Major and minor scales are hemitonic, meaning they contain at least one half step (or "hemitone"). Meanwhile, these pentatonic scales are anhemitonic because they do not contain any half steps.
If you think about the C-major collection:

     C D E F G A B C

We see half steps between the third and fourth degrees (E and F) and the seventh and final degrees (B and C). So we simply remove the offenders; by removing scale-degrees 4 and 7 (F and B), we create an anhemitonic scale of C D E G A C.
As for the minor pentatonic, let's just rotate that early collection to start on A instead of C, giving us an A-minor collection:

     A B C D E F G A

We again have half steps between B and C and E and F, so we again remove B and F (although this time they're scale-degrees 2 and 6) to create the A C D E G A minor pentatonic collection.

We can get a bit more advanced here and look at the interval-class content of these two (same) collections. Without getting too specific, when we track all of the different intervals between every single pitch, we see that this particular anhemitonic collection is made up of 4 perfect fifths/fourths, 3 major seconds, 2 minor thirds, and one major third. Note that there are precisely zero half steps (as expected) or tritones. From a set-theory standpoint, the interval vector is <032140>; Hanson's chemical analysis would be p4mn2s3.
Comment: You may want to point out what intervals you are left with when you take out those scale degrees. The [Hanson analysis](https://music.stackexchange.com/questions/29997/what-is-the-hanson-system-used-for) of the set yields p4m2ns3 which lacks intervals which most would consider dissonant.
Comment: IMO it's quite backwards to use set theory to derive anything about how tonal Western music has evolved – except perhaps Jazz-descended styles, but certainly not something as basic as pentatonic scales. Most Western music is _not_ sensibly constructed _starting_ from the 12-edo set, although that set of notes can be used to _play_ almost all Western music (or, _approximate_, as I prefer to call it).
Comment: @Richard yep. My hand writing is terrible and I screwed up a m and n.
Comment: @Tim Your guess is as good as mine; I'm not sure why it isn't "ansemitonic." (Although, now that I type it out, it does look a little too much like "antisemitic.")
Comment: J'ew know, it's a little bit close...
Comment: @Fefo I approached my answer this way because that's how the OP phrased their question, that's all.
Comment: @Dom Interesting point; I'll make an edit. (Though it would be p4mn2s3, no?)
Comment: Thank you both for clarifying. I think Dom is elaborating on what i had in mind, i.e. prescriptive theories that gives further explanations beyond staments like "i enjoy the way it sounds".
Comment: If a half-step is a 'hemitone' (which is technically word-wise fine), why is it normally known as a 'semitone'?
Comment: @leftaroundabout- yep.  As you explain in your answer below, at least Western (and arguably, many other traditions, such as Indian) scales were historically derived from the harmonic series.  Whole steps and half steps are a consequence of what steps we chose to use, not the other way around.
Comment: Interesting answer but doesn't answer the question about why the pentatonic scale is prevalent in Western music.
Comment: @pro Yes, I decided to forego that question because it seemed subsidiary to his actual question (i.e., it's not the question in the title).
Comment: With regards to 'A more "proper" term for these two pentatonic scales is anhemitonic.', I think "anhemitonic" would be one of the properties, but it doesn't really define what we call "the pentatonic(s)" as other pitch collections (such as the 5 tones in "9" chord, that's a "dominant" with an added major 9th) are also anhemitonic when laid in the form of a scale. In other words, "anhemitonic" would be "necessary but not sufficient".
Comment: @AlexLopez Fair point. I'll edit accordingly, thanks!
Comment: I don't see why you would want to explain something complete on its own like a pentatonic as part of something more specific and complex as the diatonic scale. How come the first instruments used pentatonic scales before the invention of the diatonic one?
Answer: The major pentatonic scale is the richest scale that can be rendered above a Ⅰ pedal bass using only consonances (in a sense of "consonance" which I'll elaborate). For example on C,

C: fundamental/octave, obviously stable.
D: pure Pythagorean (9:4) ninth, not a consonance in the usual sense but actually a perfectly smooth interval as long as the bass and melody are far enough apart.
E: major tenth, extremely consonant if rendered in just intonation (5:2).
G: fifth/tritave 3:1, easily pure.
A: major thirteenth/sixth. This one actually works particularly good in the same octave as the bass note, namely as the 5:3 major sixth.
c: yet another octave, duh.

So: those have all frequency ratios with numerator smaller than 10 and denominator <5. In fact, if you list all fractions >1 fulfilling that condition, you find almost only notes belonging to the major pentatonic, and only a few out of scale:

1⁄1 — C — fundamental
5⁄4 — E — third
4⁄3 — F — fourth
3⁄2 — G — fifth
5⁄3 — A — sixth
7⁄4 — B₇♭ — septimal seventh
2⁄1 — c — octave
9⁄4 — d — ninth
7⁄3 — e₇♭ — subminor tenth
5⁄2 — e — tenth
8⁄3 — f — eleventh
3⁄1 — g — twelfth
7⁄2 — b₇♭ — septimal fourteenth
4⁄1 — c' — 2 octaves
9⁄2 — d' — 8ve+9th
5⁄1 — e' — 8ve+10th
6⁄1 — g' — 8ve2+5th
7⁄1 — b₇♭' 8ve2+₇7th
8⁄1 — c'' — 3 octaves
9⁄1 — d'' — 8ve2+9th

In particular, the range from G to e (which is quite a typical domain for a pentatonic melody) is exactly the pentatonic scale if you leave out those foreign 7-limit intervals. The F would of course be a rather natural candidate too, but observe that in my system it only occurs in the low octaves, so it's a more natural choice for the bass to jump to (which indeed it often does in folk tunes) rather than the melody.
The septimal seventh is somewhat disqualified in Western music because it can't be approximated by many instruments. This is quite different in many styles around the world.
You may ask what's special about "numerator <10 and denominator <5". Well, those exact numbers are certainly a bit arbitrary, but actually you will find that the ninth overtone is about the highest that can be reliably played as a flageolett on string instruments. (I believe most brass instruments can also play the 9th overtone fairly well but get into trouble if trying to go much higher, but not sure, I don't play any brass.) Also, three octaves are quite a typical range  "in which most music happens", traditionally – anything lower or higher is more of a sonic-effect thing.
That's not to say that anything which can't be written as a <10 fraction is necessarily immediately much more consonant. In particular, the 15:8 major seventh is actually a good contender to the 9:8 ninth – in particular, the major seventh chord sounds very sweet. But that relies on the proxy-quality of the chord construction: the major seventh can readily be constructed as a third over the fifth in a normal major chord. This kind of construction doesn't really work for a more intuitive, improvisational, single-voice-over-pedal approach.
As for intervals like the 7-limit sevenths, and also slightly higher limits like 11 or 13: those are found in some music, in particular Indian, Persian and Arabic music. The problem with these is that if you actually devise tuning systems that allow rendering them on fixed-frequency instruments, you pretty much have to commit to one single key, else it gets very complicated. Western music evolved to also embrace modulations. Those can be nicely incorporated if you only consider the 5-limit intervals, which is probably the reason why we have never found much use of the 7-limit intervals. Except we have, actually: the harmonic seventh is all over Barbershop singing, and the subminor third and 7:5 or 11:8 tritones are arguably the prototypical blue note intervals. Just, you don't get to write down such intervals exactly in the Western system, so they remain considered as more of an embellishment.
Comment: It should perhaps be mentioned that while this pentatonic scale consists of intervals which are all consonant with the tonic C (if you consider 9:8 consonant with 8:8, which is not unreasonable), the interval between the D (9:8) and A (5:3) is the classic wolf fifth, 40:27.
Comment: @ScottWallace indeed. Which adds to the verdict that the pentatonic is mostly a "monophonic scale", great for freely developing a single voice over a static accompaniment but not all that useful for writing counterpoint.
Comment: yep.  Of course, you can tune your pentatonic scale to equal (or meantone, etc) temperament and smear that wolf over a wide area, but then you are left with no advantage over heptatonicity as far as sound goes, and impoverished in the number of scale steps you have for making effective counterpoint.
Comment: Don't otonal fractions have a denominator being a power of 2?
Comment: @awelotta good point! I actually meant, all fractions >1, not all otonal ones. (Though that would take care of the Fs, but TBH I find otonal vs utonal a bit arbitrary.)
Answer: Probably it has to do with the Pythagorean tuning as it was used in the development of medieval music.
The tuning is based on the 3:2 ratio (or the perfect fifth), undoubtedly consonant for human ears. The process of stacking fifths upon a fundamental frequency four times generates the scale:
C G D' A' E''
Which can be rearranged with octave equivalency to form a pentatonic scale:
C D E G A
It works well for the first five notes. Maybe the third would sound considerably higher to our equal-tempered ears, but it was probably fine before equal-temperament was invented (which was not long ago).
The real problem with Pythagorean tuning is that the nice consonance we get for the first five notes is lost for the next two notes (which would make the diatonic scale):
C G D' A' E'' B'' F''' or C D E F G A B
The relationship of 3:2 between B and F is actually consonant, but from C to F we hear a tritone (as in C - F#), as the fourth is too high (even higher than the third).
So it was probably more common to use pentatonic scales with instruments tuned in Pythagorean method during medieval times, as all intervals would sound consonant.
This might have a direct relationship with Renascence music, as they tend to consider octaves, fifths, sixths and thirds as consonances, but seconds, fourths and sevenths as dissonances. Renascence modality was the basis for baroque's tonality and so on.
Answer: You are operating under a standard-harmony bias. The pentatonic scale is not a major scale with notes missing it is simply a scale with 5 notes, Penta means 5. It is useful for modern music because its wide intervals make for greater harmonic leeway, but on the other hand, its bare-bones harmony makes it in one sense naked. This is why it often has notes added to it.
It is relevant to the modern genres because, with each new passing of the musical era in the west, there comes a need for the theorist to reinvent the wheel. Now with the pentatonic scales limits in mind, we should still respect it for the big paradigm shift it enabled.
Although I mainly operate under the guise of standard harmony I must say the paradigm shift in music in the 20th century, that was enabled by this scale was a rather brilliant. Jazz, Rock, Blues... all these things take from this piece of harmony. We should all be glad it exists.
Comment: There is nothing "modern" about the pentatonic scale. Western Baroque and Classical music are the exception, rather than the norm, in their avoidance of pentatonic harmonies.
Comment: @marshalcraft Pentatonic scales existed in the majority of musical cultures, not just in the Americas: https://en.m.wikipedia.org/wiki/Pentatonic_scale#Pervasiveness Even in Western classical music, pentatonic scales aren't particularly related to "contemporary America".
Comment: I like neils answer, @Neil Meyer wasn't native Americans who developed it. Probably just put four evenly spaced wholes for da fingers, so gives 5 notes if you refer the workings of a flute. Then westerners came along, and probably used it for inspiration, to give a new feel that described contemporary America and what better than the music of the Indians they took it from.
Comment: @marshalcraft  _what better than the music of the Indians they took it from_ - Source? Are you saying the African slaves adopated their work songs and chants from the native americans?! AFAIK, It's quite well documented that the pentatonic was absorbed into modern culture through African Americans. IMO the evidence indicates the opposite: traditional American music from whites, although they had more contact with the N-A's than blacks, tends to avoid/minimize pentatonic scales and harmony. Traditional country music, bluegrass etc use decidedly Major/European oriented structures and harmonies.
Answer: If you start at C and go up in 5ths and stop after 5 notes (before the M7 interval), that is all of the notes of the pentatonic scale out of order.  C>G>D>A>E
5ths are the most pleasing interval (other than the octave) so you just have one pleasing interval and then it's most pleasing interval and so on til you get 5 notes. Then you can reduce the above notes down by octaves so all notes are within the roots octave range so you get a scale C D E G A (C). That's why it sounds so freakin good, because there isn't an unpleasing note in the bunch.
Comment: The phenomenon happens on *any* consecutive five. Take D A E B F#. In 'alphabetic' order, D F# E A B - D maj. pent. So it's not that remarkable. A triton needs two notes, and you didn't lose F#, you lost the F-B/ B-F, which *is* the tritone out of C maj., thus ending up with C pent. maj. pent.
Comment: You are all considering equal-tuning system to explain something that it's not necessarily based on it. The idea that the fifth from B is F# exists on our newly developed temperament. See my answer.
Comment: You've lost the tritone, that's all. This explanation has been aired many times, and is just co-incidental.
Comment: "You've lost the tritone, that's all." What about the other 6 lost ones?
Comment: You quoted C,D,E,G,A. You must have lost the others somewhere!! They were never in the equation you came up with.
Comment: Yes thank you sir! I will clarify. I intentionally lost the tritone (F#) , including all the other notes but more specifically the 4th (F) and the 7th (B) that was asked about because I stopped after 5 notes since it is a pentatonic scale. They are in the equation, which is the glorious circle of 5ths. C > G > D > A > E > B > F# > C# > G#/Ab > Eb > Bb > F > (C). The beauty is that is is the first 5.
Comment: Thanks for pointing out the beauty of it working no matter where you start on the circle of fifths. I started on C to stay within the context of the question asked. I must disagree about it being unremarkable because I'm certain that any other person reading this, that was previously unaware, would be rejoicing in their new understanding and finding it quite remarkable for it's pure simplicity in how it relates to the almighty Circle of 5ths, which is rooted in Western music tradition.
Answer: I am no expert but I'm surprised no one has mentioned this....
Most articles I have read explain that pentatonic scales are elementary starting scales for students to quickly adapt to any instrument.
Once the student has acquired skillful pentatonic scales, they can complete their training with incorporating the seven note scales.
I believe this method for education purposes would keep their student attentive and practicing, without initially feeling lost.
Comment: Which articles? Could you provide any links?
Comment: https://www.ncbi.nlm. didactic.gov/pmc/articles/PMC2779864/#!po=1.04167

See pentatonic scale's last paragraph figure 1
Comment: The article I listed above mentions didactic...
Answer: Besides analyzing the theory in an abstract way, it may be useful in this case to take a look at history.

Western classical music doesn't really use pentatonic scales at all.
Current Western pop music use of pentatonic scales derives from rock music.
Rock music's use of pentatonic scales derives from blues.

In other words, so far as Western music is concerned, today's trend of massive use of pentatonic scales in popular music actually originated with the music of American slaves, which lead to various forms of blues (beginning of 20th century) , which in turn heavily influenced rock 'n' roll (50s etc.), which led to the rock music of the 60s and especially the 70s. As rock spread everywhere, pop absorbed more and more elements from it. And one of the key elements that were passed on through all these eras was the use pentatonic scales, especially in instrumental solos and bass lines, and to a slightly lesser extent, song melodies.
Because of this direct line from slaves' songs to modern rock and pop music, I feel that any theoretical analysis of pentatonic scales in themselves runs the risk of being sterile and arbitrary, because it is completely separate from the context in which the tradition was born and grown.
Outside of the Western world, pentatonic scales have a lot of history. Chinese classical music, in particular, is almost completely based on pentatonic scales, and has been so possibly for thousands of years.


mysql Creating a temporary table not working...undeclared variable error...whats wrong with my code?

Question: When I run the code below inside phpmyadmin I get an error "Undefined Variable tmpportfolio"
what is the problem?
SQL:
CREATE TEMPORARY TABLE tmpportfolio (p_name VARCHAR(255),portfolio_id INT UNSIGNED NOT NULL);

SELECT p_name, portfolio_id INTO tmpportfolio
    FROM portfolio
    WHERE parent_portfolio_id IS NULL
    AND portfolio_id NOT IN
        (
           SELECT x.parent_portfolio_id
           FROM portfolio x
           WHERE x.parent_portfolio_id IS NOT NULL
        )
    AND account_id = 1

SELECT * FROM tmpportfolio;

Answer: SELECT field INTO variable loads the value of the field into a variable. This is why it's complaining about the variable not being defined.
What you need to do is use an INSERT INTO table (colummn) SELECT field statement instead. For instance:
INSERT INTO tmpportfolio (p_name, portfolio_id)
SELECT p_name, portfolio_id
FROM portfolio
WHERE parent_portfolio_id IS NULL
AND portfolio_id NOT IN
    (
       SELECT x.parent_portfolio_id
       FROM portfolio x
       WHERE x.parent_portfolio_id IS NOT NULL
    )
AND account_id = 1

Answer: You cannot use INTO to insert records into a table in MySql (in SQL Server, you do use INTO that way). In MySql, INTO is used to put a value into a variable.
By doing the SELECT fields INTO tmpportfolio statement, MySql thinks that tmpportfolio is now a variable and it cannot find the variable in the SELECT * ... The fix is to change the insert-records sql to something like:
INSERT INTO tmpportfolio(p_name,portfolio_id) SELECT ....;
Answer: Just join first two SQL statement in one CREATE TABLE ... SELECT statment:
CREATE TABLE tmpportfolio SELECT p_name, portfolio_id FROM portfolio
    WHERE parent_portfolio_id IS NULL
    AND portfolio_id NOT IN
        (
           SELECT x.parent_portfolio_id
           FROM portfolio x
           WHERE x.parent_portfolio_id IS NOT NULL
        )
    AND account_id = 1

SELECT * FROM tmpportfolio;

Answer: I can't speak for mysql sepcifically but usually once you have created a table you insert into it it not use selct into which creates the table (and which can't since the table already exists.)
Answer: The confusion seems to arise from W3Schools explaining how to do "select into table" without mentioning it's specific to some (non-Mysql) implementations. A page on dev.mysql specifically denies this possibility on Mysql, by saying:

MySQL Server doesn't support the SELECT ... INTO TABLE Sybase SQL extension. Instead, MySQL Server supports the INSERT INTO ... SELECT standard SQL syntax, which is basically the same thing.


Rate of convergence of random variables for weak convergence

Question: Suppose $X_{n}$ be a sequence of random variable that converges to $X$ in distribution. How can we define the rate of convergence? What would be the reference?
Comment: Again, since you are asking for a definition and a reference I have added both those tags.
Answer: If
$\alpha_n \stackrel{\rightarrow}{_D} \alpha$,  s.t.
$\alpha_n \stackrel{_D}{\approx} \alpha + \epsilon(n)$,
then the rate of convergence is the convergence rate of $\epsilon(n)$. Eg, in CLT type convergence, the error would be of order $n^{-\frac{1}{2}}$ and the measures would converge at such rate.


Openpyxl: set background color to a row and column Attribute Error

Question: having looked at a few examples over here I tried to set background color to an entire row and column. I have done
 import openpyxl
 from openpyxl.styles import PatternFill
 wb = openpyxl.load_workbook(self.inputfile)
 ws = wb.active
 ws['A1'].fill = PatternFill(bgColor="FFC7CE", fill_type = "solid")`

I get an Attribute error if I do `ws[1].fill =PatternFill(bgColor="FFC7CE", fill_type = "solid")
The above code fills a single cell(A1). But how do I go forward if I want to fill an entire row(1), and an entire column(A).
Comment: This is covered in the documentation: https://openpyxl.readthedocs.io/en/latest/styles.html#applying-styles
Comment: @CharlieClark Following the documentation the following does not work for me: row = ws.row_dimensions[1]
row.font = Font(bold=True), am I doing anything wrong?
Answer: Iterates all columns, starting at the Column specified in the min_col=1 argument.
Ends after one row, as the row arguments min_row=1 and max_row=1 are equal.
Arguments min_row/max_row can point to any row, even also outside data.
  for rows in ws.iter_rows(min_row=1, max_row=1, min_col=1):
    for cell in rows:
      cell.fill = PatternFill(bgColor="FFC7CE", fill_type = "solid")

For entire Column, use:
iter_cols(min_col=None, max_col=None, min_row=None, max_row=None)

If you only  give min_* attribute values, max row/column are used.
Tested with Python:3.4.2 - openpyxl:2.4.1
Answer: Slight modification on the accepted answer (white background):
   for rows in sheet.iter_rows(min_row=1, max_row=40, min_col=1, max_col=40):
    for cell in rows:
        cell.fill = PatternFill(start_color='00FFFFFF', end_color='00FFFFFF', fill_type="solid")

This provides a white fill on the sheet starting at row 1 column 1 and going to row 40 and column 40.
Answer: I tried the accepted answer but row background was always black, So i have to change the PatternFill to following.
  cell.fill = PatternFill(start_color='FFC7CE', end_color='FFC7CE', fill_type = "solid")


Using pyplot.close() crashes the flask app

Question: I'm making a web app with Flask which displays different pyplot charts on a single webpage. Following is the code which I'm using to create different plots and then return them as a base64 encoded png image.
import pandas as pd
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
import matplotlib.pyplot as plt
import io
import base64

def simple_plot():
    img = io.BytesIO()
    np.random.seed(196801)
    x = np.random.rand(50)
    y = np.random.rand(50)
    plt.plot(x, y)
    plt.savefig(img, format='png')
    img.seek(0)
    lp = base64.b64encode(img.getvalue()).decode()
    plt.close()
    return lp

def scatter_plot():
    img = io.BytesIO()
    np.random.seed(19680801)
    x = np.random.rand(30)
    y = np.random.rand(30)
    colors = np.random.rand(30)
    area = np.pi * (15 * np.random.rand(30)) ** 2
    plt.scatter(x, y, s=area, c=colors, alpha=0.5)
    plt.savefig(img, format='png')
    img.seek(0)
    sp = base64.b64encode(img.getvalue()).decode()
    plt.close()
    return sp

def linear_reg_plot():
    data = pd.read_csv("data/clean_df.csv")
    lm = LinearRegression()
    img = io.BytesIO()
    X = data[['highway-mpg']]
    Y = data['price']
    lm.fit(X, Y)
    predict = np.around(np.array(lm.predict(X)))
    intercept = lm.intercept_
    slope = lm.coef_
    plt.plot(X, predict)
    plt.savefig(img, format='png')
    img.seek(0)
    plot = base64.b64encode(img.getvalue()).decode()
    plt.close()
    x = data['highway-mpg'].values.tolist()
    y = data['price'].values.tolist()
    result = {
        'X': x[:50],
        'Y': y[:50],
        'Predicted': predict[:50],
        'intercept': intercept,
        'slope': slope,
        'msqe': mean_squared_error(y, predict),
        'r2': lm.score(X, predict),
        'chart': plot
    }
    return result

If I don't use the plt.close() it causes all of the graphs to become same (same sort of data plotted on all of them). But after using he plt.close() it displays the correct plots but now I am facing this error which is causing the flask app to crash as well as there's a message box saying that Python is crashed.
Following the error shown by the debugger:
127.0.0.1 - - [05/Jul/2018 14:55:04] "GET /dashboard HTTP/1.1" 200 -
Exception ignored in: <bound method Image.__del__ of <tkinter.PhotoImage object at 0x000000644962F0F0>>
Traceback (most recent call last):
  File "E:\ProgramData\Anaconda3\Lib\tkinter\__init__.py", line 3501, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <bound method Image.__del__ of <tkinter.PhotoImage object at 0x0000006449638048>>
Traceback (most recent call last):
  File "E:\ProgramData\Anaconda3\Lib\tkinter\__init__.py", line 3501, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <bound method Image.__del__ of <tkinter.PhotoImage object at 0x00000064495AAA20>>
Traceback (most recent call last):
  File "E:\ProgramData\Anaconda3\Lib\tkinter\__init__.py", line 3501, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <bound method Image.__del__ of <tkinter.PhotoImage object at 0x00000064495D5C50>>
Traceback (most recent call last):
  File "E:\ProgramData\Anaconda3\Lib\tkinter\__init__.py", line 3501, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <bound method Image.__del__ of <tkinter.PhotoImage object at 0x00000064495B61D0>>
Traceback (most recent call last):
  File "E:\ProgramData\Anaconda3\Lib\tkinter\__init__.py", line 3501, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <bound method Image.__del__ of <tkinter.PhotoImage object at 0x0000006448E629E8>>
Traceback (most recent call last):
  File "E:\ProgramData\Anaconda3\Lib\tkinter\__init__.py", line 3501, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <bound method Image.__del__ of <tkinter.PhotoImage object at 0x0000006448EC8940>>
Traceback (most recent call last):
  File "E:\ProgramData\Anaconda3\Lib\tkinter\__init__.py", line 3501, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <bound method Image.__del__ of <tkinter.PhotoImage object at 0x0000006448E69160>>
Traceback (most recent call last):
  File "E:\ProgramData\Anaconda3\Lib\tkinter\__init__.py", line 3501, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Tcl_AsyncDelete: async handler deleted by the wrong thread

The below picture is shows the plots being displayed as well as the error:
Python crash error
Answer: Right before the line
import matplotlib.pyplot as plt

add
import matplotlib
matplotlib.use('agg')

Currently, matplotlib is accessing the "tkagg" backend that connects to the GUI event loop and that causes unexpected behaviour. The plain "agg" backend does not connect to the GUI at all.
Comment: Thanks I'll try it out.
Comment: Cool, glad to know :-)
Comment: Thanks, good answer. This was also the solution for:
`Exception ignored in: >
Traceback (most recent call last):
  File "/usr/lib/python3.6/tkinter/__init__.py", line 3507, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Tcl_AsyncDelete: async handler deleted by the wrong thread
Aborted (core dumped)`


OpenVPN working - but how to route all traffic down it?

Question: I've setup a static one-to-one OpenVPN connection fine. However I can't seem to get the client to route all traffic via the VPN device. Here's my configs:
Client config:
remote 89.21.xx.xx
dev tun
ifconfig 10.8.0.2 10.8.0.1
secret static.key
comp-lzo
keepalive 10 60
ping-timer-rem
persist-tun
persist-key

Server config:
dev tun
ifconfig 10.8.0.1 10.8.0.2
push "redirect-gateway def1"
secret static.key
comp-lzo
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
verb 5

From all the research I've done the server config line push "redirect-gateway def1" should make the clients route all traffic via the VPN. However it doesn't have this effect. I can ping/ssh/http to the server on 10.8.0.1, and the routing table of the client looks like below, but normal traffic is taking the normally (unsecure) route:
root@t42:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.8.0.1        0.0.0.0         255.255.255.255 UH    0      0        0 tun0
192.168.69.0    0.0.0.0         255.255.255.0   U     2      0        0 wlan0
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 wlan0
0.0.0.0         192.168.69.1    0.0.0.0         UG    0      0        0 wlan0

Answer: You need to have "pull" in your client config in order to be able to "push" settings from the server.
Once you add that, it should work as you expect.
(if you use the "client" directive, it basically expands to "pull" and "tls-client")
Comment: I tried the "pull" flag (also played with the "client" flag). However it seemed it didn't like the static setup, and wanted a proper CA based setup. However with that now in place it works. Cheers.
Answer: You can either run route add default gw 10.8.0.1 (either by hand or automatically when the tunnel comes up); that'll add the route.  Alternately, I've got push "route 0.0.0.0 0.0.0.0" in the OpenVPN config that does a default route for me, and that works, too.
Comment: I swear I tried manually adding the route, but with no luck. I shall try again this evening.
Answer: The def1 argument alters the behaviour of redirect-gateway slightly. It is akin to:
route add 0.0.0.0/1 gw <vpn_endpoint>

This ensures that the VPN route takes preference over but doesn't remove the existing default route.
As for debugging the issue I would suggest:

Take a look at jamesb's very good point.
Increase the client verbosity. It should indicate whether the directive is coming down from the server and whether there was a problem running the route command.
Try placing the directive in the client's configuration to test whether it works without push.
Try dropping the def1 argument and have it replace the default route to see whether it makes any difference to the behaviour.
Comment: Thanks for the help. It seems when I switched from a static to CA setup it all worked fine.
Comment: I know this is a really old answer, but I've just come across this and it looks wrong to me.  Doesn't 0.0.0.0/1 match all addresses from 0.0.0.0 to 127.255.255.255 and leave everything else to the default route?  So half the internet goes over your VPN and half over your default route...


Symmetry assumption in the Wilcoxon-Mann-Whitney test

Question: The Wilcoxon-Mann-Whitney test requires that two distributions are symmetrical
How can I check this assumption by using a hypothesis test and how apply it in R?
If this assumption is not met what test should I use rather than the Wilcoxon-Mann-Whitney test?
Comment: [This post](https://stats.stackexchange.com/questions/553416/assumptions-of-mann-whitney-test-for-at-least-ordinal-data) has extensive discussion of the various assumptions of the MWU test and I think is relevant here.
Answer:
The Wilcoxon-Mann-Whitney test requires that two distributions are symmetrical

No, it doesn't require symmetry of both distributions.
(What makes you think this is necessary?)
It requires exchangeability of the ranks under H0 (and not under H1); the most typical way to get that would be if the two distributions had the same shape when H0 is true. They don't have to have the same shape when its false.

How can I check this assumption by using a hypothesis test

Even if it were to be a necessary assumption, testing it on the samples is not especially relevant, since you'd only require it under H0, which you have no good reason to think is true; indeed with an equality null, it's almost certainly false, so the assumption applies under a situation which the data don't typically tell you much about.

Edit: Let me give an explicit example.
Illustrative example
Imagine for the sake of argument that I have two samples, one notionally a sort of 'control' group and one a sort of 'treatment' group. There's a claim that the treatment performed on the treatment group is highly valuable (has a substantive effect on the measure of interest) but we don't really think the treatment does anything at all.
Now imagine that each population has a beta distribution (under both H0 and H1). If the treatment does nothing, the parameters of the distribution would be the same (so that indeed the Wilcoxon-Mann-Whitney would have exchangeable ranks under H0).
If the treatment does something that corresponds to the claim we would expect the parameters to change.
For example, if the alternative is true we could see something like this:

while if the treatment did nothing at all, we would expect to see the black distribution for both populations.
If $H_1$ were true and we had samples from these two distributions, checking the assumption of the same shape under $H_0$ by looking at the data would be completely misleading $-$ in a reasonable-sized sample we would be highly likely to conclude that the shapes differ.
If we then conclude that there's a problem with the test, we are making a grave error. We're actually in a situation that the test is designed for!
Comment: Agreed, naturally. Another way to explain this is that WMW works with the ranks and there is absolutely no information in the ranks about symmetry or asymmetry. You can easily get the same ranks from quite different distribution shapes.
Comment: @Nick Nevertheless there is information about asymmetry in the original data.  Some investigation of that is worthwhile in any case: if the test leads you to reject the null hypothesis, consider whether the explanation might be asymmetry rather than a true difference in location; and when it does not lead you to reject the null, consider whether an asymmetry might be making the test less powerful than expected.
Comment: @whuber Indeed, and as almost always we agree.  Either way being aware of asymmetry and its consequences is essential too.
Comment: I agree with the answer, however the H0 should at least be relevant for the situation in question; why would anybody want to test it otherwise? I'm not saying it should literally be true, but it should at least potentially fit the data well enough to not be rejected! It's about compatibility, not truth, and therefore I disagree with "the data don't typically tell you much about it" - it's the very point of the test that they do.
Comment: @Glen_b I found this assumption when I researched about this test on google (If the test is used as a test of dominance, it has no distributional assumptions. If it used to compare medians, the two distributions must be identical apart from their locations. If it used to compare means, those two distributions must also be symmetrical.) and I want to use this test to compare means.
Comment: Harold Jeffreys in his often difficult but always brilliant book _Theory of Probability_ (utterly mistitled) has a key section on the dire influence of _wishful thinking_ in statistical analysis. Wanting WMW to be a test of means can be a prime example of wishful thinking.  What is the real scientific question here? Why not answer it quantitatively with say a bootstrapped confidence interval for the difference in means?
Comment: @ChristianHennig Hi Christian, thanks for your comment. I have added an illustrative example of the point I am making at the end to clarify the situation for H0 and a potential one for H1 and more discussion of why looking at the data may then not be relevant for the assumptions under H0. I suspect we don't disagree about much that's substantive but if you have a problem with what's there, please let me know.
Comment: @Statisticalscientist I fear I disagree with several parts of what you found. See, for example the illustration I added at the end for my response to Christian. That example comes from a sequence of alternatives that have a difference in mean (e.g. under the null, we have a beta($\alpha_0,\beta$) and and under the alternative we have beta($\alpha_i,\beta$) for some $\alpha_i>\alpha_0$; indeed, consider ...
Comment: ... a sequence of possible $\alpha$s, $\alpha_0<\alpha_1<\alpha_2<....$. each alternative in the sequences has a larger mean than the ones before it (with lower index). The shapes are all different from the others, and only one of them is symmetric. Clearly, then, the test is sensitive to changes in mean in cases where you don't necessarily have symmetry nor identical shapes in the population distributions the samples were drawn from.
Comment: @Glen_b Fine addition, I like it. I have learnt as alternative of the $P=Q$ null hypothesis $P\prec Q$ (stochastic ordering), which holds in your example, and as you correctly say, doesn't require equality of distributional shapes. Harder situations to discuss are those where $P\neq Q$, but where none of these is stochastically larger than the other. It is not always clear in these cases whether the test *should* reject $H_0$ or not, and I still think it's valuable to diagnose differences in distributional shape to know what's going on.
Comment: I think we're essentially on the same page then.


What type of pairwise test does emmeans run?

Question: Can someone please explain what kind of test emmeans runs when using the pairs function (is it a paired t-test between levels)?
I have a linear mixed effects model with two fixed effects (A, B) and one random effect (C). I fit this with the lme function from the nlme package:
mod = lme(val~ A*B, random = ~1|C, data = df)

For each level of A, I want to perform pairwise comparisons for levels of B to check which one dominates. When I run pairs
mod.emm = emmeans(mod, ~A*B)
pairs(mod.emm, simple = "B")

Does this function take my random effect C into account, the way a paired t-test would?
Thanks!
Answer: Here is an illustration of how the model determines the right test. First, create a toy data set and run both a pooled and a paired t test:
y = c(7,6,9,3,2,6)

t.test(y[1:3], y[4:6], var.equal = TRUE)
##
##  Two Sample t-test
##
## data:  y[1:3] and y[4:6]
## t = 2.4597, df = 4, p-value = 0.06972
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -0.4722133  7.8055467
## sample estimates:
## mean of x mean of y
##  7.333333  3.666667

t.test(y[1:3], y[4:6], paired = TRUE)
##
##  Paired t-test
##
## data:  y[1:3] and y[4:6]
## t = 11, df = 2, p-value = 0.008163
## alternative hypothesis: true mean difference is not equal to 0
## 95 percent confidence interval:
##  2.232449 5.100884
## sample estimates:
## mean difference
##        3.666667

Now, I'm going to create two experimental setups:
foo1 = data.frame(trt = factor(rep(c("A", "B"), each = 3)),
                  subj = factor(1:6))
foo1
##   trt subj
## 1   A    1
## 2   A    2
## 3   A    3
## 4   B    4
## 5   B    5
## 6   B    6

foo2 = foo1; foo2$subj = factor(c(1:3,1:3))
foo2
##   trt subj
## 1   A    1
## 2   A    2
## 3   A    3
## 4   B    1
## 5   B    2
## 6   B    3

foo1 has 6 different subjects, each observed once, while foo2 has 3 subjects, each observed twice. Now with each dataset, I will fit mixed models with exactly the same specs:
library(nlme)
mod1 = lme(y ~ trt, random = ~1|subj, data = foo1)
mod2 = lme(y ~ trt, random = ~1|subj, data = foo2)

Now let's test the treatment comparison for each model, again with exactly the same call:
library(emmeans)
pairs(emmeans(mod1, "trt"))
##  contrast estimate   SE df t.ratio p.value
##  A - B        3.67 1.49  4   2.460  0.0697
##
## Degrees-of-freedom method: containment

pairs(emmeans(mod2, "trt"))
##  contrast estimate    SE df t.ratio p.value
##  A - B        3.67 0.333  2  11.000  0.0082
##
## Degrees-of-freedom method: containment

Created on 2023-11-06 with reprex v2.0.2
Note that the t test and P value for foo1 is exactly the same as the pooled test; and that these results for foo2 are exactly the same as the paired test. This works because the lme function knows what to do with the relationship between subjects and treatments, and that is what determines the correct test.
If you had three or more treatments in foo1 or foo2, you would obtain several tests, one for each pairwise comparison. And none of them would be the same as either the pooled or paired t tests. That's because the models use all of the data, while the pooled and paired tests use only the data for two treatments at a time. So already we are in a situation where we've moved past those simple hand calculations.
Answer: Using the model, it figures out what linear combination $c$ of regression coefficients b are needed to estimate the quantities $\theta$ in question. Then it computes that estimate using the regression formula $\hat\theta = c^Tb$ , and its standard error using $SE(\hat\theta) = \sqrt{c^TVc}$ where $V$ is the variance-covariance matrix of $b$, typically obtained from the model via the vcov() function. Then it obtains the $t$ ratio $t = \hat\theta / SE(\hat\theta)$.
When you have fitted a mixed model especially,  is not a simple hand-computation formula that you'll find in a basic textbook; that's why you need computer packages to fit the model and obtain these results.
Answer: I think there might be some confusion coming from two types of "pairs" here.
The "pairs" you seem to be thinking about are pairs of observations within individuals. In your mixed model, pairs of observations within individuals are effectively taken into account by the random intercepts. Ideally, modeling with random intercepts in that situation improves the precision of coefficient estimates, similarly to the way a paired t-test can. See this page for discussion.
The pairs() function in emmeans evaluates pairs of estimated marginal means (EMMs), which are predictions from the model. That can be done for any model type supported by emmeans, whether the model involved random effects or not. The pairs() function just evaluates the differences between all pairs of EMMs that meet the specifications you have set. See this vignette section.
The pairs() function thus doesn't by itself consider paired observations within individuals. It depends on the mixed model to have taken such within-individual pairing into account. It then uses the fixed-effect coefficient estimates and their covariances to get the differences and associated standard errors for all pairwise comparisons of the EMMs that you specify, as the package's author explains in his answer.


Is a semigroup with left identity und unique left inverse a group?

Question: Let $(G, \cdot)$ be a semigroup (i.e. a set with an associative binary operation) and fix some $e\in G$. If
1) $\forall g\in G: e\cdot g=g$ (left identity),
2) $\forall g\in G~ \exists g^{-1}\in G: g\cdot g^{-1}=e$ (right inverse),
3) $\forall g,h\in G: g\cdot h=e\Rightarrow h=g^{-1}, h\cdot g^{-1}=e\Rightarrow h=g$ (unique inverse),
must $(G, \cdot)$ be a group?
I know that a left idenitity and right inverse don't necessarily give a group, and that a unique left identity and unique right inverse give a group. Yet this question I have no clue how to attempt.
Comment: Axiom 2-3 is awkwardly written, since right inverse is not unique. If 2 is existence of a right inverse, then it has no reason to be written $g^{-1}$. If you mean you fix such a function, it should be part of the axiom: you have a function $g\mapsto g^{-1}$ satisfying axioms 2,3.
Answer: A semigroup with a left identity and unique right inverses is not necessarily a group.  Here is an example:
Let $G$ be any set with more than one element and call one of the elements $e$.  Define a multiplication on $G$ by $x\cdot y = y$.  This is associative, since $x\cdot(y\cdot z) = (x\cdot y)\cdot z = z$.  For all $x\in G$ we have $e\cdot x = x$, so $e$ is a left identity (as is every other element of $G$).  Also, $x\cdot y = e$ if and only if $y = e$, so each $x$ has a unique right inverse, namely $e$.  But $(G,\cdot)$ is not a group, since there is no right identity.
EDIT:  Looking closely at your axiom $3$, you seem to be requiring something more than unique right inverses.  It seems that you also want an element to be the right inverse of at most one element; equivalently, if an element has a left inverse, it must be unique.  In this case, $(G, \cdot)$ must indeed be a group.
To prove this, it suffices to show that a right inverse is also a left inverse.  Choose any $x\in G$ and consider the product $x'\cdot x\cdot x'\cdot x''$, where $x'$ denotes a right inverse.  On the one hand, $$x'\cdot (x\cdot x')\cdot x'' = x'\cdot e \cdot x'' = x'\cdot x'' = e.$$  On the other, $x'\cdot x\cdot (x'\cdot x'') = x'\cdot x\cdot e$, which must also be $e$.  Since the right inverse of $x'$ is unique, $x'' = x\cdot e$.  It follows that
$$
x''\cdot x' = (x\cdot e)\cdot x'= x\cdot(e\cdot x') = x\cdot x' = e.
$$
Hence $x'$ is a right inverse for both $x$ and $x''$, so $x = x''$.  Finally, $x'\cdot x = x'\cdot x'' = e$, so the right inverse of $x$ is also its left inverse.


Methods for allowing administrators to set desktop software's per user configuration

Question: Edit: Hello serverfault, they didn't like my question on programmers and migrated it here.
In short my questions is: How do you (administrators) prefer to control settings for third party, off the shelf, desktop software?
My application needs a server name and a database name.  I also have some offline data storage that is occasionally redirected from the default location in My Documents.
Right now users need to open the my software and type in these values.  I would like to provide administrators with a way to automatically set these for users while still letting users self configure in smaller organisations.

I have a desktop software application that stores its configuration information in a file located in the user's AppData folder.  Defaults are created by the application if the configuration file does not exist.  Some of the settings like registration codes and server/database name can't be set by default.
I'm looking for methods to allow and IT department to push out default settings to my application.  Right now I am considering having my application read two configuration files.  One that would be in a fixed location copied by the administrator.  This would be manually added (or probably pushed out by some other tool) to either the install directory or All Users or other public location.  Any settings in that file would override the per user configuration settings.
I know that other applications have different ways for administrators to control per user settings using SMS or GPO.  But not being an administrator myself I have never used any of those tools.
Are there any best practices for allowing administrators to override default settings for desktop software?
I need this two work in two different situations.  One is laptop users who are connected to the network when software is installed but often disconnected when running the software - one user per computer.  The other is Citrix based systems where clients are always connected but multiple users per computer.  Some of the settings are global - i.e. the server name, while others are user specific.  Things like redirecting the default save location to a per user network share.
Comment: Why not create a GUI administration page?  Maybe part of the application or on the web/intranet.  There admins can update the change-able parameters.  The file focused "push" updates seems overly complex.
Comment: I have a GUI setup page. I want to automatically configure the app without end user input. I am saving the config file in the user profile to ensure access. That makes it hard for admins to make changes. In some cases the profile does not even exist when the admin is setting up a system (the user has not logged in yet). If I go look for a central config file on a server how do I know the server name/path? If I move the config file to a computer wide location then users could not change it. I also don't want to force admins to open a GUI on 60 different computers to key in the same settings.
Comment: My personal opinion is to use a Group Policy. They're not that bad to setup once you get used to them.
Comment: You can pull the configuration path from a centralized and update able location.  For example, a web server.  Then use this location to to download the settings on first launch.
Comment: @Rachel - What does Group Policy actually do though?  As a programmer how do I read what they set?  Does it just push out registry entries?  I've never found any good references for programmers who want to use group policy settings in their apps.
Comment: @P.Brian.Mackey - We sell off the shelf software so the problem is figuring out the server name.  I have considered hosting configuration files for our clients on my server.  But then anyone could get to another company's configuration files unless I have some type of encryption involved.  Even then I would need to use something like domain name or user entered key to get their specific configuration.  At least I think so - I'm hoping someone has a solution to these issues. :-)
Answer: I think you need to do some requirements gathering.  Determine what the clients need and are willing to pay for.  See if it makes sense to host the files yourself, tell the clients to self-host and provide a solution that they can install or some other solution that is requirements specific.
Once you have requirements in place we will be better prepared to help you with the technical portion.
Whatever the implementation turns out to be, you should do something to centralize and automate the process.
Answer: My own software always saves its settings in the user profile. Each user can therefore create their own configuration. If I wish to create a default config for new users I create it the way I want and then simply copy the settings (registry or ini file) to the default users's profile. That frofile is used as the a new user's profile when they first log on. It can also be propagated via Group Policies if that is ever necessary. Anything more complex than that is just not worth the effort for the dev, the admin or the users.


Erro no valor de parseInt do JavaScript com número muito grande

Question: Ao converter um número em formato de string decimal para o valor numérico, recebo o número errado:
parseInt("36428971654192132")
// 36428971654192130

Answer: De acordo com Javascript.info, o tipo number não consegue representar corretamente números inteiros maiores que 2^53-1, ou seja, 9,007,199,254,740,991.
Comparando:

9007199254740991
36428971654192132

Tente usar BigInt:
BigInt("36428971654192132");


Non-empty intersection of equivalence classes.

Question: I'm having troubles with the following exercise about equivalence classes on a defined set.
Let $R$ be an equivalence relation on a set $A$. Given $a,b \in A$ prove the following statements are equivalent (logic equivalence):

$[a] \cap [b] \neq \emptyset $
$a \in [b]$
$[a] \subseteq [b]$

Where the notation $[a]$ means the equivalence class on the set $A$ under the equivalence relation $R$.
Edit: I started considering the definition of an equivalence class:
$$[a] = \{b\in A:aRb\} $$
And then applying the definition of the intersection:
$$[a] \cap [b] = \{a,b \in A:aRb \land bRa\}$$
However, I don't know if I applied the definition correctly and I don't know how to continue.
Can you help me out? Thanks in advance.
Answer: The way you've written $[a] \cap [b]$ is incorrect; you first treat $a$ and $b$ as bound variables (i.e. elements of $A$ that were already specified), and then later as free variables (i.e. variables into which you can substitute any element of $A$); I'm no expert on this but I'm quite sure that the way you've intercharged these two notions is syntactically incorrect (at any rate it confuses the reader). If this doesn't make any sense you could consider reading up on this a bit on wikipedia, or at least try to understand what the trouble is with your definition.
Anyway, the correct definition would be $[a] \cap [b] = \{c \in A : aRc \mbox{ } \land \mbox{ }bRc \}$. (can you see why?)
For (1) $\Rightarrow$ (2) use the above definition and the fact that $R$ is transitive; you can prove (2) $\Rightarrow$ (3) again by using the transitive property (and of course the definition of equivalence classes); and (3) $\Rightarrow$ (1) is trivial (or rather it's just very basic set theory).
Hope that helps, and if you have any questions remaining I'll try to answer them!


SVG animation on I.E 11

Question: I have an svg animation on an angular 8 app,an svg circle whick working fine on chrome and mozilla but dosen't work on I.E 11. Any ideas how to make it work there too?How can i change my css to work on I.E 11 too? Thanks for your attention. I'm looking forward to your reply.
I have add meta for IE on my header:
<meta http-equiv="X-UA-Compatible" content="IE=edge">

Live example

   #countdown {
      position: relative;
      margin: auto;
      height: 40px;
      width: 40px;
      text-align: center;
    }

    #countdown-number {
      color: #1C5BA2;
      display: inline-block;
      line-height: 40px;
    }

    svg {
      position: absolute;
      top: 0;
      right: 0;
      width: 40px;
      height: 40px;
      transform: rotateY(-180deg) rotateZ(-90deg);
    }

    svg circle {
      stroke-dasharray: 113px;
      stroke-dashoffset: 0px;
      stroke-linecap: round;
      stroke-width: 2px;
      stroke: #1C5BA2;
      fill: none;
      -webkit-animation: countdown 31s linear forwards;
      animation: countdown 31s linear forwards;
    }

    @keyframes countdown {
      from {
        stroke-dashoffset: 0px;
      }
      to {
        stroke-dashoffset: 113px;
      }
    }
<link href="https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css" rel="stylesheet"/>

          <svg>
            <circle r="18" cx="20" cy="20"></circle>
          </svg>

my index

<!doctype html>

 <meta charset="utf-8">
 <title>App Name</title>
 <base href="/">

  <meta name="viewport" content="width=device-width, initial-scale=1">
  <meta http-equiv="X-UA-Compatible" content="IE=edge" />

 <app-root></app-root>

Comment: Does this answer your question? [SVG animation is not working on IE11](https://stackoverflow.com/questions/33812303/svg-animation-is-not-working-on-ie11)
Answer: The IE browser does not support CSS transforms on SVG elements, more detail information, please check this article.
As a workaround, I suggest you could try to display the circle using div, instead of SVG, please refer to the following samples: CSS3 Circle loader One Time around and CSS circle loader
Code as below:

The result like this:

Besides, there have another choice, we could make a .gif image to show the SVG animation, then, use  tag to display it. Like this:
The result like this:
Answer: Because of document mode
<meta http-equiv="X-UA-Compatible" content="IE=edge">

Comment: Sorry i didn't mention it on my post. I am using this meta.
Comment: np, what about `doctype`? can you share the beginning of the html file?

SVG animation on I.E 11


SLURM: Elapsed time does not return day?

Question: Following link https://slurm.schedmd.com/sacct.html defines Elapsed time as follows:

Elapsed The jobs elapsed time. The format of this fields output is as
  follows: [DD-[HH:]]MM:SS

Following code returns:
jobId = 1;
sacct -j $jobId --format="Elapsed" | tail -n1 | head -n1

Output: 00:10:11
I won't able to geth the day(DD), which should be 00 on this example. I was hoping to get an output like this: 00:00:10:11.
[Q] What should I do to see also the used day on Elapsed time output? or day does not exist and hour keep increment after, it covered 24 hours.
Thank you for your valuable time and help.
Answer: It seems that the days will only be displayed for jobs that have been running for more than 24 hours.
On our cluster here, I have two jobs, one of which have been running for just over 24 hours (id 319) and one that has been running for just a few minutes (336):
$ sacct -j 319 --format="Elapsed" | tail -n1 | head -n1
1-00:26:08
$ sacct -j 336 --format="Elapsed" | tail -n1 | head -n1
  00:08:39
$

This has been tested on SLURM version 17.02.2


If a function is measurable with respect to Prog then it is progressively measurable.

Question: I have come across a statement I am not sure how to prove.
Assume you have a filtered probability space $(\Omega, \mathcal{F},\{\mathcal{F}_t\},P)$. A stochastic process $X_t$ is progressively measurable if for every $t\ge 0$, the function $f(\omega,s) $from $\Omega\times[0,t]$ to $\mathbb{R}$ given by $f(\omega,s)=X_s(\omega)$, is $\mathcal{F}_t\otimes\mathcal{B}([0,t])$ measurable.
Prog is the smallest sigma-algebra on $\Omega\times[0,\infty)$ such that all progressively measurable processes are Prog-measurable.
Obviously we have that every progressively measurable process is Prog-measurable. However, assume you have a function $g: \Omega\times[0,\infty)\rightarrow \mathbb{R}$, and assume that it is Prog-measurable, then it is stated that the stochastic process $Y_t$ defined by $Y_t(\omega)=g(\omega,t)$ is progressively measurable.
Do you know how to show this?
Attempt:
My only idea is using something like the Doob-Dynkin lemma. The Doob-Dynkin lemma states that if a function $f$ is $\sigma(h)$ measurable then there exists a borel function k, such that $f=k(h).$ The problem I have is that Prog is not generated by a single progressively measurable process it is generated by an infinite amount, so I don't think I can use the Doob-Dynkin lemma directly? But if I can prove that if $g$ is Prog-measurable, then it is some kind of limit or some kind of Borel function of the progressively measurable processes, then it will also be progressively measurable. Is there a way to do this?
Answer: This is a consequence of the functional form of the monotone class theorem as found, for example, at https://almostsuremath.com/2019/10/27/the-functional-monotone-class-theorem/ (see Theorem 1 there).
The key observation is that Prog is generated by an algebra of bounded process that contains the constant processes and is stable under bounded pointwise convergence.
Comment: Yes, this is what I had in mind. As you see, the conclusion is quite general, depending only on certain linear "closure" properties. That is, the same argument works to reach the same conclusion for optional processes, predictable process, ...
Comment: Thank you, do you mind checking if this is the argument: Let $H$ be the vector space of bounded progressively measurable functions, $H_+$ is closed under increasing limits, $H$ is also closed under multiplication. $H$ contains the constant function. Then $H$ contains all the functions which are measurable with respect to $\sigma(H)=Prog$. The argument is a little unusual because usually the space which is closed under multiplication is smaller than $H$.
Comment: Thank you, but what do you mean that we can use the same argument for optional and predictable processes? Left- or right-continuity isn't necessarily preserved under pointwise limits?
Comment: @user394334: Ack!  Good point.  Strike that comment.
Answer: There is no need for the monotone class theorem here.  It is sometimes useful to believe that since $\sigma$-algebras are collections of sets, anything that is true about them should follow from operations on measurable sets without reference to measurable functions.  (Remember that a measurable function is something like a concatenation of measurable sets.)  Hence here it is helpful to replace "if $g$ is $\text{Prog}$-measurable, then..." with the corresponding statement for sets "if $A$ is $\text{Prog}$-measurable, then..."
In this case, it boils down to: $A \in \text{Prog}$ only if the process $\chi_{A}$ is progressively measurable.  (Note that, by the definition of $\text{Prog}$, we already know that the "if" version of this statement is true.)
Define $\mathcal{P}$ by
\begin{equation*}
\mathcal{P} = \{A \subseteq \Omega \times [0,\infty) \, \mid \, \chi_{A} \, \, \text{is progressively measurable}\}.
\end{equation*}
It is quick to show that $\mathcal{P}$ is a $\sigma$-algebra since $\chi_{\bigcup_{n = 1}^{\infty} A_{n}} = \sum_{n = 1}^{\infty} \chi_{A_{n}}$ if $\{A_{n}\}_{n \in \mathbb{N}}$ is pairwise disjoint, and $\chi_{A^{c}} = 1 - \chi_{A}$.
Since every progressively measurable process is $\text{Prog}$-measurable (by definition), the inclusion $\mathcal{P} \subseteq \text{Prog}$ is immediate.  We need to show that $\text{Prog} \subseteq \mathcal{P}$.  Since $\text{Prog}$ is on the left-hand side, this suggests using the minimality property of $\text{Prog}$.  That is, we only need to show that if $X$ is progressively measurable, then $X$ is $\mathcal{P}$-measurable.
Here is where we get back to the point: $\sigma$-algebras are about sets.  Fix $c \in \mathbb{R}$.  To show that $X$ is $\mathcal{P}$-measurable, we only need to show that $A(c) = \{X \leq c\} \in \mathcal{P}$ holds.  Looking at the definition of $\mathcal{P}$ suggests studying the "process" $\chi_{A(c)}$.
Fix $t \geq 0$.  If we define $f^{A(c),t} : \Omega \times [0,t] \to \mathbb{R}$ by $f^{A(c),t}(\omega,s) = \chi_{A(c)}(\omega,s)$, then the progressive measurability of $X$ implies that
\begin{equation*}
\{f^{A(c),t} = 1\} = \{(\omega,s) \in \Omega \times [0,t] \, \mid \, X^{\omega}_{s} \leq c\} \in \mathcal{F}_{t} \otimes \mathcal{B}([0,t]).
\end{equation*}
Taking the complement, $\{f^{A(c),t} = 0\} \in \mathcal{F}_{t} \otimes \mathcal{B}([0,t])$.  Since $f^{A(c),t}$ takes values in $\{0,1\}$, this proves it is $\mathcal{F}_{t} \otimes [0,t]$-measurable.
We showed that if $X$ is progressively measurable, then $\chi_{A(c)}$ is progressively measurable for all $c \in \mathbb{R}$ --- this fact is really why $\text{Prog}$ completely characterizes progressively measurable processes.  Hence, by definition of $\mathcal{P}$, we find that $X$ is $\mathcal{P}$-measurable if it is progressively measurable.  But $\text{Prog}$ is the minimal $\sigma$-algebra with this property so $\text{Prog} \subseteq \mathcal{P}$.
Just to reiterate: $X$ is progressively measurable if and only if the "derived sets" $\{A(c)\}_{c \in \mathbb{R}}$ given by $A(c) = \{X \leq c\}$ are progressively measurable.  Thus, progressive measurability, at some level, is a question about subsets of $\Omega \times [0,\infty)$ --- so the fact that there is a $\sigma$-algebra $\text{Prog}$ lying behind it is no surprise.  In particular, note that $X$ is progressively measurable if and only if $X$ is $\text{Prog}$-measurable.  (Further, the definition of $\mathcal{P}$ above gives a succinct, explicit definition of $\text{Prog}$ that does not involve the sometimes-unwieldy "minimal $\sigma$-algebra" concept.)


What is the solfa method to learn intervals?

Question: I am working on recognizing intervals. I am reading online material to determine which technique would be the most efficient.
I came across this article:

The second method is "solfa". This is the "do re mi" approach made
famous by the childrens song in The Sound of Music. But don't let that
fool you. Solfa (also called "solfege") is probably the most powerful
and versatile framework there is for learning relative pitch. There's
a bit of a learning curve but if you're truly dedicated to developing
your musicianship solfa is a great choice.

I do not know what this method is. I looked online, and found the song.
However, I do not understand what method of learning this is.
Can anyone explain?
Comment: Does this answer your question? [What exactly is the "tonic sol–fa" system, and how is it different from solfège?](https://music.stackexchange.com/questions/58015/what-exactly-is-the-tonic-sol-fa-system-and-how-is-it-different-from-solf%c3%a8ge)
Comment: @aparente001 I was educated in the US, but I have the impression that the numeric system is less common than movable do.  Numbers are inconvenient since one of them has more than one syllable.  How do you express chromatic alterations?
Comment: @phoog - Once you are comfortable enough with sightsinging to need a lot of chromatics, you'll have internalized the numbers and their interrelationships, and you won't need to explicitly sing numbers any more.  However, in the transition, one could add "sharp" or "flat" if the note is long enough, but just think it, if not.  But please take that with a grain of salt because (a) I studied sightsinging such a long time ago, and (b) I never *taught* it. It is certainly fair for the chart of advantages and disadvantages to have "accidentals could be a challenge" on the line for the number system.
Comment: Hello, no because the other link answers in the context of notation, whereas I am asking about the *ear training* method. They are two different things (though probably related).
Comment: The method is explained a bit more on [this page](https://www.musical-u.com/learn/series/starting-solfa/) which you can reach by clicking the do-re-mi image (not very obvious, IMHO).
Comment: Usually sightsinging is taught in the US using 1, 2, 3, etc., in place of do, re, mi, etc.  I don't know where you live, but communicating with others would be easier if you follow the number trend.  Example -- You can learn the perfect fourth by getting used to singing (or imagining) the sequence "5 - 1." / If you want to communicate with international musicians, many of them will use the fixed do system (where C is ALWAYS called "do" regardless of what key you're in), so learning solfege with the movable do could make communicating with the international folks confusing.
Comment: @phoog - I'd imagine 'se'n' would suffice, wouldn't it?
Comment: @Tim yes, but it's inconvenient since it may not be strongly linked in the mind with 7.  In mine it tends to evoke "send" with a missing "d."
Comment: @phoog - I used "sev" but you can use whatever you want.  You could even borrow "si" for that one note.  And phoog, if solfege works well for you, that's great!  OP is just getting started, and I see our role here as laying out options and providing information about advantages and disadvantages of each approach.
Comment: @aparente001 "whatever I want" tends to be "ti" ;-) along with "do" through "la."  My comment, really, was just to question whether numbers really are more prevalent in the US.  I'm all for giving new students the opportunity to pick the options that work best for them.
Comment: @phoog - I can't say what the frequencies are, seems like an interesting question.  I learned numbers in my sightsinging but I honestly don't remember if that was at my community college or at Indiana U.  But it was a long time ago so things could be completely different now.
Comment: @aparente001 I also went to IU but did not study sight singing there beyond a couple of very brief and incidental forays into fixed do and Guidonian solmization.  I'm still curious how you express chromatic alterations with numbers 1 through 7.
Answer: The solfege idea is that each note in a major scale has its own name - it could just as easily have a number, but here it has a name.
It uses the movable do system, whereby do is always regarded as the root note. So do = 1 or C (in key C), re = 2 or D, mi = 3 or E, fa = 4 or F, sol = G or 5, la = 6 or A, and ti (sometimes si) = B or 7. Each name is designated to a particular number note, whatever the key is.
We then learn to sing each letter name using do as the base point, and the intervals become apparent as we learn what each note sounds like against it. Thus, for example, do re mi do, do re mi do would be the solfege equivalent of Frere Jacques.
In France, it's commonplace to sing in such a way, except that it's complicated (or not!) by the fact that there, fixed do is used. Thus any key other than C will have sharps or flats involved with the notes - although that system doesn't lend itself to interval training much at all - only in key C!
Listening to Doe a Deer, later in the song, the solfege names are sung as a sort of counterpoint to the tune.
So, yes, it can be part of interval learning, but in somewhat a roundabout way, maybe not the easiest, as it doesn't acually teach intervals per se. Unless you go to the trouble of working out each name against its particular partner's for that interval - like mi>sol = m3, sol>do = P5.
Comment: "In France, it's commonplace to sing in such a way": indeed, it's the primary way to name pitches in every context, so symphonies and the like aren't in D major or B minor but in ré majeur and si mineur.  But (strictly speaking) it's not "fixed do" because they still use "ut" instead of "do" :-)
Comment: @phoog - not in my experience - never heard 'ut' in several years - always 'do'. Fixed ut? (Ut fixe)?
Comment: Well truth be told I don't know what they traditionally use for solfege; maybe it is actually "do."  To my surprise, "do" is also used to denote keys, but Google's N-gram viewer suggests that "ut" was more common until very recently indeed: https://books.google.com/ngrams/graph?content=do+mineur%2Cut+mineur%2Cdo+majeur%2Cut+majeur&year_start=1800&year_end=2019&corpus=30&smoothing=3
Comment: @phoog - just spoken to head of a school of music: ut for instruments and symphonies, etc., do is easier to sing, therefore do wins there. Keys and notes definitely called do.
Comment: How is a key different from a symphony?  Are you saying that a French theorist would report the key of Bizet's "symphonie en ut majeur" as "do majeur"?  If you are, you would be in agreement with [French Wikipedia](https://fr.wikipedia.org/wiki/Symphonie_en_ut_majeur_de_Bizet).  How bizarre.
Comment: @phoog - re-read the comment. I said instruments and symphonies use ut, while keys and notes use do.
Comment: @phoog - I thought *ut* was used in German -- wasn't aware it was also used in French.
Answer: Tim's answer is great. Just to fill in the "why" of moveable-do: it doesn't teach intervals in the abstract (M3, P4), but it does teach "tonicity." It teaches the dominant feel of scale degree 5, the leading tone of ^7, etc. IMO this is even more useful for the end goal of "singing in tune" then being able to string together a series of decontextualized intervals. To compare it to someone navigating a course, you hold the tonic and dominant in your head and orient other pitches in relation to them, like someone navigating with a compass, instead of someone whose directions are "turn 45 degrees left, turn 18 degrees right," and have no fixed reference against which to check their accuracy.
Also, a bit of trivia: Where do these nonsense syllables come from? Ultimately, from the lyrics to a song: a song used, much like "Do a Deer", as a mnemonic to remember scalar steps. A medieval choir director took a song in which each line started one note higher, and used the first syllable of the Latin word that started each line as a mnemonic for that note of the scale.
Comment: "it doesn't teach intervals in the abstract": mi-sol is always going to be a minor third, isn't it?
Comment: @phoog True, but it gets a different "label" than re-fa. If we pay attention to the intervals along the way we wind up "incidentally" learning them (and, IMO, getting a better contextual understanding on which to "hang" them), but it's not a system for "learning a minor third," decontextualized, in the abstract. And IMO it's better than such a thing.
Comment: I see what you mean, and I agree.  (Of course, re-fa and mi-sol are likely to differ by a syntonic comma.)


Server config for PHP/Apache2/MySQL for 250,000 hits per day?

Question: Can anyone recommend a config, or better yet point me to a full case study for this web setup?
Basically we have a php site getting about 100,000 hits per day.  It's already getting connection timeouts etc. when trying to serve this load on a VPS, due to hitting CPU capacity.  I'd like to spec something that will allow it to scale to about 250,000 hits per day.
Currently the server is:

a single-core 1Ghz burstable VPS (from a 2.6Ghz dual core Xeon)
2GB burstable RAM (again, I find burstable very vague)
32MB per php process, running expression engine (as far as I know) or something similarly heavy
quite image-intensive graphical pages
occasional behind-the-scenes work on some pages, like xmlrpc to third-party servers
plain apache with modphp (so static files like images also need full php processes -- this can't be changed for now)

So far, CPU seems to be the main bottle neck, with some apache process using around 30% CPU during requests.  Some other apache processes seem much better at only 1% CPU; I assume these are requests for static files.
So.  We want to get away from the VPS onto some sort of dedicated or virtual dedicated box with non-burst (measurable, guaranteed) performance.  It has to be able to handle currnet traffic and at least double that in future.
Any recommendations or studies you can point me to?
Are there any tools around which can give you an estimated server spec based on parameters like memory per apache process, percentage of requests that are dynamic, whether PHP code is heavy CMS stuff or lightweight hello-world stuff, etc?
Answer: I doubt there are any useful case studies on this matter as performance will always vary enormously based on the complexity of the application - the only way to get close to a fit is to measure and extrapolate.
You don't mention OS but I'd setup a series of performance monitors capturing threads, cpu loads, memory stats, disk and network data etc. also make sure you capture the number of hits you're dealing with. From this you should be able to determine the min, average and max resource usage for all these components based on actual seen load - once you have these stats (mhz/10k-users, mem/10k-users etc) you should be able to quickly extrapolate the performance you need. Make sure to plan for around 20% additional headroom even when at your peak expected load.
I know it's a lot of spreadsheet work but it's the only way you'll feel confident - well there's always the 'throw tons of hardware at the problem' method which works too but costs :)
Best of luck and come back with some data later if you need help understanding it.
Answer: Just add nginx in proxy mode in front of apache, and set it up to server really static data without touching apache. You will fell instant speed boost.
Then I would switch apache2 into preforked mode with like 4-16 childs.
Background work like external connections really hurts, I would rewrite that code to make these things async.
Finally, you might be out of bandwidth on master node (i.e. server holding all VPSses). Ask you hosting company if they have 1Gbit connection on their master node, and if it's just 100Mbit - you might want to change hosting.


If you have a really big N, is using a regular vs. an m-n bootstrap ever a good idea?

Question: I know that the regular bootstrap (assuming that the necessary assumptions are valid) is more efficient than the m-n bootstrap.
However, supposing that you have access to really large data (such as a 1% sample of Google's daily searches), does it ever make sense to use the regular bootstrap whose assumptions may be violated?
Comment: @user68 and dchandler - please do join in the discussion over on meta: http://meta.economics.stackexchange.com/questions/40/where-stats-questions-belong
Comment: Judging from this [article](http://www.google.com/url?sa=t&source=web&cd=2&ved=0CCoQFjAB&url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.127.8759%26rep%3Drep1%26type%3Dpdf&rct=j&q=m%20out%20of%20n%20bootstrap%20method&ei=XtOaTp2PKI7n-gbGsNjaBQ&usg=AFQjCNGPHDUEkmzl4HktI6rkzcGkKI76gg&cad=rja) this questions is more suitable for mathoverflow.net. As far as I understand the answer is yes. Figure out whether the conditions are violated and if you need efficiency go with normal bootstrap. If you only care about consistency use m out of n.
Comment: @dchandler, if the issue is computational vs. statistical (i.e., being unable to efficiently sample $10^9$ observations with replacement), you might want to state so. Michael Chernick gave an exhaustive answer to the statistical question, but it may not answer the computational question.
Answer: As the author of texts on the bootstrap, I think I can provide a meaningful answer.
There are very few assumptions to violate with the regular bootstrap.  The issue with the bootstrap is whether or not it provides consistent estimates for the particular problem.  When the ordinary (i.e., the usual $n$-out-of-$n$) bootstrap is consistent, $m$-out-of-$n$ is less efficient and there is no reason to use it over the ordinary bootstrap.
The $m$-out-of-$n$ bootstrap comes in handy for the few exceptional cases where the ordinary bootstrap fails.  In cases such as estimating extremes the ordinary bootstrap fails, but $m$-out-of-$n$ has been shown to be consistent as long as m goes to infinity at a slower rate than $n$.  Then $m$-out-of-$n$ should be used.
A practical question might be what to take for $m$.  I would suggest something simple like square root of $n$.
Comment: I have taken the liberty of softening the first couple sentences a bit and reformatting some things. Please review and make any changes you'd like, or, even, feel free to rollback if you prefer the original version.


Como buscar informações do usuário logado sem ser pelo controller?

Question: Estou utilizando o spring security para fazer o login do meu site. Está funcionando, consigo cadastrar novos usuários, logar e deslogar.
Tenho uma navbar que, quando o usuário não está logado, aparece a opção de "Logar ou cadastrar", e quando ele está logado aparece a opção "Sair".
Porém eu gostaria que exibisse o nome do usuário logado do lado do link de logout, e não estou conseguindo fazer isso:
SecurityConfig:
@Override
protected void configure(AuthenticationManagerBuilder auth) throws Exception {
    auth
        .jdbcAuthentication()
        .usersByUsernameQuery(userQuery)
        .authoritiesByUsernameQuery(roleQuery)
        .dataSource(dataSource)
        .passwordEncoder(passwordEncoder);
}

@Override
protected void configure(HttpSecurity http) throws Exception {
    http
        .authorizeRequests()
        .antMatchers("/**").permitAll()
        .anyRequest()
            .authenticated()
                .and().csrf().disable()
            .formLogin()
                .loginPage("/login").failureUrl("/login?error=true").defaultSuccessUrl("/")
                .usernameParameter("email").passwordParameter("senha")
            .and().logout()
                .logoutRequestMatcher(new AntPathRequestMatcher("/logout")).logoutSuccessUrl("/home");
}

Navbar:

    <form class="form-inline">
        <input class="form-control mr-sm-2 searchNav" type="search" placeholder="Pesquisar">
        <button class="btn btn-outline-light my-2 my-sm-0" type="submit">Pesquisar</button>
    </form>

Entre  |  Cadastre-se

Sair

Como esse navbar é um fragmento, eu gostaria de saber como buscar as informações pelo thymeleaf, para não precisar ficar repetindo código buscando informações do usuário logado em vários controllers.
Answer: Você pode recuperar os dados de autenticação através do SecurityContextHolder e SecurityContext:
Authentication authentication = SecurityContextHolder.getContext().getAuthentication();

Com o objeto Authentication, você pode recuperar os dados do Principal autenticado através do getPrincipal(). Existem outros métodos para recuperar authorities, credenciais, verificar se ele está logado...
Detalhes podem ser vistos aqui, aqui e aqui
Comment: Esse comando consigo executar no thymeleaf?
Comment: Sim, você pode criar um service que contenha este código e referenciá-lo em seu fragment


OrientDB: Find Connected Components Values during the visit

Question: I have schema with 3 main classes: Transaction , Address and ValueTx(Edge).
I am trying to find connected components within a range of time.
Now I am doing this query based on this one ( OrientDB: connected components OSQL query) :
SELECT distinct(traversedElement(0)) from ( TRAVERSE both('ValueTx') from (select * from Transaction where height >= 402041 and height <= 402044))

And this returns the rid of the 'head' of each trasversal and from it doing another DFS I can get every node and edge of the connected component I want to search about.
How can I, using the query above, also get the number of the transactions within the connected component and also the sum of their values? (The value of a tx is a property of the class Transaction)
I want to do something like:
SELECT distinct(traversedElement(0)) as head, count(Transaction), sum(valueTot) from ( TRAVERSE both('ValueTx') from (select * from Transaction where height >= 402041 and height <= 402044)) group by head

But of course is not working. I get only one row with the last head and the sum of all the transactions.
Thanks in advance.
Edit:
This is an example of what I'm looking for:
Connected Transactions
Every transaction there is within the same range of height:
Using my query ( the first one in my post) I get the rid of the first node of each group of transaction that are linked through several addresses.
example:
#15:27
#15:28
#15:30
#15:34
#15:35
#15:36
#15:37
#15:41
#15:47
#15:53

What I'm trying to get is a list of every first node with the total number of transactions (not addresses only the transaction) of the group it belongs to and the sum of the value of every Transaction (stored in valueTot inside the class transaction.
Edit2:
 This is the dataset where I am making the tests:
The main problem is that I have a lot of data and the approach I was trying before (from every rid I make a different sql query) it's quite slow, I hope there is a faster way.
Edit3:
This is an updated sample db: Download
(note, it's way larger than the other)
select head, sum(valueTot) as valueTot, count(*) as numTx,sum(miner) as minerCount from (SELECT *,traversedElement(0) as head from ( TRAVERSE both('ValueTx') from (select * from Transaction where height >= 0 and height <= 110000 ) while ( @class = 'Address' or (@class = 'Transaction' and height >= 0 and height <= 110000 )) ) where @class = 'Transaction' ) group by head

This query on my system takes around one minute, also if I limit the result set, so I think the problem maybe in the internal query that selects the transactions that isn't using the indexes... Do you have any idea?
Comment: Can you post a little example of your graph ?
Comment: I posted an update with a picture hoping it is more clear, let me know if you still need clarifications.
Comment: Hi, could you add the schema (Classes with properties) as I can reproduce your environment and do some tests? Tnx
Comment: or if you can share an export of a small test db it would be perfect
Comment: @IvanMainetti I shared my test db, I hope you can help, thanks
Answer: You can use this query
select @rid, $a[0].sum as sumValueTot ,$a[0].count as countTransaction from Transaction
let $a = ( select sum(valueTot),count(*) from (TRAVERSE both('ValueTx') from $parent.$current) where @class="Transaction")
where height >= 402041 and height <= 402044

Hope it helps.
Comment: Thanks for your time, sadly the query it's really slow and at the end the results are not the expected one: I get some repeated results(sum and count) with different rids.
But the main issue it's the speed, it's way slower (5-10 times) than just doing a single query for every group.
Comment: So after studying better the results of the query (that takes around 9 minutes on my dataset of around 40k edges) I saw that I just get the result for every Transaction of the db (around 4k). I didn't manage to fix it.  Maybe also for this reason is really slow?
I just assumed that if listing the origin of every group is really fast, also counting them would, am I wrong?
Comment: @FrancescoMoca I think that the solution of Ivan Mainetti is correct,
to increase the speed you can put an index(SB-Tree) on the field height
Answer: is this what are you looking for?
select head, sum(valueTot), count(*) from (SELECT *,traversedElement(0) as head from ( TRAVERSE both('ValueTx') from (select * from Transaction where height >= 402041 and height <= 402044)) where @class = "Transaction") group by head

Comment: Sadly after adding more datas I realized my idea was wrong :/.
The traverse of course isn't stopping on the transaction outside the given range of course, so the result is very different from what I need.
I think I need to use the while clause on the trasversal but I can't figure out how.
When the visit meets an address it should check all the next transaction and only continue the visit on the transactions where the height is inside the given range),
So I should have a while in the traverse checking for the range  and return only the correct transactions.
 Any Idea?
Comment: Hi @FrancescoMoca could you update the testDb so I can make some tests? Tnx
Comment: Sorry for now answering before but i think i solved my problem:
I am adding " while ( @class = 'Address' or (@class = 'Transaction' and height >=0 and height <=5000 )" at the end of the traverse.
Comment: Now I am using this query:
`select head, sum(valueTot) as valueTot, count(*),sum(miner) as minerCount from (SELECT *,traversedElement(0) as head from ( TRAVERSE both('ValueTx') from (select * from Transaction where height >= 0 and height <= 5000) while ( @class = 'Address' or (@class = 'Transaction' and height >=0 and height <=5000 ))  )  where @class = 'Transaction'  ) group by head `
But it's really slow with a lot of transactions.
I tried to use an index like @Alessandro suggested but I don't think it helped. What can i do to be sure that is using the Transaction.height index?
Thanks
Comment: Hi, if you update the shared test DB I'll make some testing
Comment: BTW, I'm trying with the current db (the one you linked before) and I'm having different results using this new query instead of the old one
Comment: Yeah, the last one gives me the results I was looking for.
The problem that with around 1M vertices is really slow, I updated my question, hope you can help :)
Comment: Hi, I'm trying your query `select head, sum(valueTot) as valueTot, count(*),sum(miner) as minerCount from (SELECT *,traversedElement(0) as head from ( TRAVERSE both('ValueTx') from (select * from Transaction where height >= 0 and height <= 5000) while ( @class = 'Address' or (@class = 'Transaction' and height >=0 and height <=5000 )) ) where @class = 'Transaction' ) group by head` on your updated DB. It's taking about 3seconds to me resulting 4756 items
Comment: what if you try with a larger one ? `select head, sum(valueTot) as valueTot, count(*) as numTx,sum(miner) as minerCount from (SELECT *,traversedElement(0) as head from ( TRAVERSE both('ValueTx') from (select * from Transaction where height >= 0 and height <= 110000 ) while ( @class = 'Address' or (@class = 'Transaction' and height >= 0 and height <= 110000 )) ) where @class = 'Transaction' ) group by head
`


How to tell someone that we can't imply an equation to another equation as to when multiplying $x$ both sides?

Question: So my friends and I was trying to tell someone that this implication is wrong, he (that person) said that these equations are the same:
$$9x = 1 \Rightarrow 3x^2 = \frac x3$$
He confused why $9x = 1 \Rightarrow x={1/9, 0}$ isn't true. You see that the second one of the solutions is $x=0$, that obviously multiplying both sides with $0$ isn't wise. I was trying to explain that by fundamental theorem of algebra the first equation only has $1$ solution, but he's too obtuse to accept that and I gave up. In my real analysis class, we also had someone that doing this he thought it was legal.
If I don't mistakenly remember, it's about solving absolute equations, or whatever you call it, he suggested multiplying both sides with $0$ and that led him into wrong answers different set of solutions. My answer, however, avoiding multiplying both sides with $0$ and using basic axioms led me into correct answer which is, in my opinion, multiplying both sides with an unknown variable is really dangerous. It's really shameful I can't remember the exact problem to illustrate this.
Anyway, related to the question, how to explain this problem?
Comment: Let me put it this way: If a person posts on Facebook that he believes $x=0$ is a solution to $9x=1$, nothing that anyone on math.stackexchange writes will make him understand he's wrong.
Comment: The most straightforward way to describe this problem is that for any functions of a real variable (or a variable of any field) $f(x) = g(x)$ is equivalent to $xf(x) = xg(x)$ **or** $x = 0$. If you drop the "or $x = 0$" your statements are no longer logically equivalent.
Comment: Wait a second. Are you saying your friend believes $x=0$ is a solution to $9x=1$? If that is so, I have no idea how to help. If a person thinks $9x=1$ is true for $x=0$ there is a more fundamental issue here. -- If, on the other hand, your friend accepts that $x=0$ is *not* a solution to $9x=1$, but is a solution to $3x^2=x/3$, and they are only confused about why it is a solution to one but not the other, then one can start explaining ...
Comment: @TorstenSchoeneberg it's the second issue I think. but I'm also not sure whether they believe the first issue that you mentioned.
Comment: You should clarify that first. As said, unless a person sees clearly that $x=0$ is a solution to the second equation but not to the first (even if they are puzzled *why*), any argument involving algebra or logic is totally in vain.
Comment: @TorstenSchoeneberg I've re-read his question. It was actually posted on Facebook. He really thinks both equations are exactly the same, that's why he thinks $x=0$ must be true.
Answer: It is true that $9x=1\implies 3x^2=\frac{x}{3}$. Since $3x^2=\frac{x}{3}\iff x\in\{0,\frac{1}{9}\}$, it is also true that $9x=1\implies x\in\{0,\frac{1}{9}\}$.
However, I believe you're friend might think that this implies that $x=0$ is somehow a solution to $9x=1$. However, that is a logically flawed statement. What we would need to do is substitute $x=0$ and $x=\frac{1}{9}$ independently to determine which ones were actual solutions. And we come up with that $x=\frac{1}{9}$ is the only solution to $9x=1$.
Answer: Does ${0\times 3 = 0\times 5 = 0}$ imply ${3 = 5}$? No, since we would be dividing by $0$. Consider
$$
x\times (3x) = x\times \left(\frac{1}{3}\right)
$$
Just like our obviously wrong ${3=5}$ , if ${x=0}$ we cannot conclude ${3x = \frac{1}{3}}$.


JOIN em postgres não traz todos os dados desejados

Question: Estou tentando fazer um JOIN entre a tabela A e B, e quando faço:
SELECT * FROM tabela_A JOIN tabela_B ON tabela_a.id=tabela_b.id

Dá certo, tenho todos os dados de ambas as tabelas reunidos.
Todavia, quando filtro colunas especificas da tabela A, exemplo:
SELECT coluna_a FROM tabela_A JOIN tabela_B ON tabela_a.id=tabela_b.id

Os únicos dados retornados são essa coluna_a da tabela_A.
Testei com várias tabelas e com outros tipos de JOIN tipo Left, right, inner, e ainda sim o único retorno que tenho é a coluna filtrada.
Como faço para ter a coluna específica da tabela A + toda a tabela B?
Answer: Uma maneira de resolver o problema é declarando um alias para cada tabela e depois exibindo as colunas desejadas. Isso é possível da seguinte forma:
SELECT b.*, a.coluna_especifica FROM tabela_1 a JOIN tabela_2 b ON a.id=b.id

Nesta consulta eu desejo exibir todas as colunas da tabela_2 (e dei um
alias chamado b para esta tabela) e desejo apenas  exibir uma
coluna específica da tabela_1 (que dei um alias chamado a para
esta tabela).

Desta forma um nome temporário pode ser dado a uma tabela para fazer referências a dados derivados das tabelas no resto da consulta. Isto é chamado de alias.
Existem duas sintaxes possíveis para declarar um alias de tabela (abaixo um destaque da sintaxe no FROM:
FROM table_reference AS alias

ou
FROM table_reference alias

Para o caso desta resposta escolhi utilizei a segunda sintaxe exibida por ser mais concisa.
Para mais exemplos de uso consulte a documentação.