Enharmonic key signatures in Bach's Well Tempered Clavier

Question: I have been listening to Bach's Well Tempered Clavier Book 1 and have noticed something intriguing. I have two (related) questions about it. The work consists of 24 preludes and fugues in each major and minor keys. So, for instance, it begins with:
No. 1: Prelude and Fugue in C major, BWV 846;
No. 2: Prelude and Fugue in C minor, BWV 847.
But in number 8 the prelude is in E♭ minor and the fugue is in D♯ minor:
No. 8: Prelude in E♭ minor and Fugue in D♯ minor, BWV 853.
Since E♭ and D♯ are enharmonic and are the same key in the keyboard, I wonder what is the harmonic justification for this. Why didn't he just name both E♭ or both D♯?
Later, following his G minor prelude and fugue (no. 16), he writes his A♭ major prelude and fugue and, after it, the prelude and fugue are in G♯ minor, not in A♭ major:
No. 17: Prelude and Fugue in A♭ major, BWV 862;
No. 18: Prelude and Fugue in G♯ minor, BWV 863.
Again, why is this so? Why not write both in the same key?
Comment: IIRC, the D# minor fugue already existed in a d minor version before the entire collection was compiled, and presumably Bach preferred re-copying it into D# rather than Eb because this means only adjusting the accidentals rather than the accidentals *and* all notes.
Comment: @PedroG.Mattos consider the C sharp diminished seventh chord in D minor: C#, E, G, and Bb. To transpose that to D sharp minor (Cx, E#, G#, B), you change the key signature, copy the noteheads to the same places on the staff, and modify all of the accidentals.  To transpose to E flat minor (D, F, Ab, Cb), you change the key signature, move all the noteheads up from a line to the next higher space or from a space to the next higher line, and adjust the accidentals.  In practice, the need to of noteheads to a different position on the staff results in a significant increase in copying errors.
Comment: A little confusion here with enharmonic minors, then enharmonic major/minor. Two different questions/answers.
Comment: Why would it be easier to transpose to D♯ instead of transposing it to E♭?
Comment: Thanks for the example, this is very clear now.
Answer:
Why didn't he just name both E♭ or both D♯?

Why not both in D# major seems clear to me as D# major would need 9 sharps. Eb minor has 6 flats like D# minor has 6 sharps. As usual Bach's pieces are extending to the dominant and subdominant what would imply additional sharps for the secondary dominants: e.g. F7 in measure 9 would be a E#7, the major 3rd of V/iv, which is G in Eb7 would be a double sharp F => Fx (##), also the secondary vii dim7  chords would need more sharps. So the reason to decide for one or the other key is to find in the intension to avoid additional accidentals, as a natural resolved sign is easier for reading.
Comment: 1. I didn't understand that OP was asking only about the minor variants.
2. Bach wrote  the 3rd prelude in  C#-major (7 sharps) instead of Db (5 flats) this was in opposite of my advise and is rather not congruent with my argumentation concerning the secondary dominants! (I hope they won't down vote it now ;) 3. May be Bach wrote the latter prelude in C# that the students learn to read sharp keys? :) One point that we can exclude is that Bach thought the key of  C# would be another character than Db. (joke) But I'm sure some interprets will swear they hear it different looking at all the #.
Comment: What's the reason for your first sentence? There is no D# major, unless you meant a modulation during the piece itself.
Comment: I just got confused by the mention of D# major, especially since it's the first sentence, since it doesn't come up anywhere else, either in your answer or the question.
Comment: Thanks for the answer! This seems to be the way to understand it, but it is still not clear to me... I have to think about it for a while.
Comment: I think I should add some examples what exactly means secondary dominants ... or do you understand this?
Comment: I don't, that would help.
Comment: @AlbrechtHügli actually I remember seeing at least one DWC edition having both the C# version of the 3rd prelude as well as a version re-written as Db by the editor (maybe it was Edition Peters - if I remember I'll check when I get home).
Comment: @Simo Kivistö: yes, I know, I've  found it too in the IMSLP just some minutes ago. I used to transpose it spontaneous when playing in Db and also reading in C (simply ignoring the sharps). Interesting point: I've read today that it had been originally written in C-major for W.F. Bach.
Comment: @ Pedro G. Mattos: I'm planning to explain the advantage of notation  in  the equivalent flat key for reading by the example of the prelude in C#-major. But meanwhile you could study here on this SE some answers to   questions about "secondary dominants" for better understanding.
Answer: The reason may be that, at least in the first book of the 48, Bach was deliberately pushing the notation of tonal music to its limits instead of taking the easier options.
The E flat minor prelude and D sharp minor fugue are a good example. Because of the differing modulation schemes, the prelude contains some important chords of F flat major, while the fugue has some of E sharp major. In fact writing the prelude in D sharp and the fugue in E flat would have been "easier to read" if Bach had intended the music to be "easy to read". (When I first learned that prelude and fugue, as a teenager, my teacher gave me the task of writing out the music in the "opposite" keys by hand. That was before personal computers and notation software which could do the job with a couple of mouse clicks existed!)
The same may be true of the prelude and fugue in C sharp major, which arguably would be easier to read in D flat.
Some of the other pieces in Book I were original written specifically for educational purposes. For example the first prelude, in C major, was originally an improvisation exercise on the given chord sequence, written for Bach's son Wilhelm Friedemann, and the version in the WTC is Bach's "solution manual" to the exercise.
WTC Book II was compiled about 20 years after Book I, and Bach may have lost interest in such overt educational objectives by that time.
Comment: (When I first learned that prelude and fugue, as a teenager, my teacher gave me the task of writing out the music in the "opposite" keys by hand.) That's what I use to do when practicing the prelude in C#-major, or even ignore all sharps and play it in C, so I can better understand what's happening harmonically)
Comment: Sources, please.
Answer: There are a few leading factors which have a bearing:

pedagogical / theoretical
convenience / historiography
musical references

The first is well covered by other answers: that there was a didactic element to Book I of Das Wohltemperirte Clavier. By having this key, the most remote of the minor keys from the non-accidental A minor, represented by two different notations, it cemented the full circularity of the 12-tone system, both in terms of understanding the equivalence of both Eb minor and D# minor for the keyboard, but also how the idea of temperament worked to bring both to the same note.
Ledbetter's comment that the equivalence of Eb and D# minors was well-known among theorists and composers stands, although its use of Rameau's Traité de l'Harmonie does not seem to be correct (Rameau only gives Eb minor in his Supplément, at least in the original printing).
The second is because of strong historiographical evidence for the fugue of BWV 853 being originally in D minor. According to Ledbetter, the "most obvious pointer" is the scale in the soprano voice of bar 15 which in D minor would have gone up to the note c''', the top note of the keyboard that Bach was writing for, but now has to jump down to c#'' instead.
Ledbetter also gives the example of some stray accidentals from the "original" notation, B naturals instead of B sharps, E flat for E natural. However, the older convention of "sharp/natural" = go a semitone up, "flat" = go a semitone down still holds in many Bach manuscripts and printed editions from this time in the 18th century.
The factor of musical reference is more arcane but also very interesting. The prelude is in the style of a tombeau, making reference to the French Baroque school, with their highly dotted rhythms and tirata scale passages. Many recordings make use of notes inégales. The abundance of flats in the key signature is also part of this reference: one can think of Froberger's Tombeau in C minor or that of Louis Couperin, or how Bach himself uses the dissonance of the extreme flat keys in the Passions at the death of Christ.
On the other side, the fugue is one of the stile antico fugues and Ledbetter says "it is the only piece in Book I to have a substantial element of the modal tradition". Its subject is very clearly based in Dorian mode, and it has a very strict treatment of dissonance while being rhythmically very loose (so much so that the subject itself mutates quite a bit, again a stile antico feature). Hence it hearkens back to a D dorian modal palette, and by simply transposing up from D minor, it refreshes the "old" into something "new" while still making its roots clear.
Answer: Bach was simply completing his task of writing a movement in every key. It would be unlike Bach to leave any out.
Bach's new keyboard tuning/temperament allowed him for the first time to move through all of the keys pleasingly. The most accurate recreation of this tuning is the Lehman Tuning. Here is a demonstration

. Each key had its own character and would lend itself to open or closed intervals. Bach composed for each key noting its particular characteristics; for example the opening of the first prelude in C, It consists of many open intervals due to the tuning being based on perfect fifths.
Bach would have also taken into consideration the modulations into relevant keys. Without equal temperament, The 8th Fugue would sound like a different key due to its modulations.
Comment: This doesn't make sense as an explanation of choosing to notate something in one key rather than another. Why is there a video in the answer?
Comment: You mention equal temperament, but I thought that well-tempered was meant to refer to some other system such as some Werckmeister system that would also allow all keys to sound reasonable. Indeed, your video says equal temperament was a 20C development. You also speak of "Bach's new tuning" as though he had invented it, which I did not think was the case.
Answer: You must see Bach for what he was. He was a pioneer in seeing what could be done. He got to the point where he was a master in counter-point and he wanted to see how far he could take it. These pieces were an exploration in what the art of fugue sounded like in all the different keys, this was his treatise into how keyboard players should determine a suitable key for there musical ideas and how the harmonical possibilities were influenced by that choice, all that and they sounded pretty good too.


Is it possible to perform a regression analysis without a dependent variable?

Question: I have a data set with 5 independent variables. Is it possible to do a regression analysis without the presence of a dependent variable?
> names(bb)
[1] "BB1"  "BB2"  "BB3" "BB4"   "BB5"

> x1<-bb$BB1
> x2<-bb$BB2
> x3<-bb$BB3
> x4<-bb$BB4
> x5<-bb$BB5

> OLS<-lm(y~x1+x2+x3+x4+x5, data = bb)

> summary(OLS)

Call:
  lm(formula = y ~ x1 + x2 + x3 + x4 + x5, data = bb)

Residuals:
  Min      1Q  Median      3Q     Max
-168.79  -74.18  -19.31   53.45  222.43

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept) 151.2558    69.6001   2.173   0.0325 *
x1            0.3859     1.4976   0.258   0.7972
x2            0.2746     0.1940   1.416   0.1605
x3           -2.6899     2.5056  -1.074   0.2861
x4           -2.2511     1.1785  -1.910   0.0595 .
x5           -0.3369     1.6231  -0.208   0.8360
---
  Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 95.18 on 85 degrees of freedom
Multiple R-squared:  0.09186,   Adjusted R-squared:  0.03844
F-statistic:  1.72 on 5 and 85 DF,  p-value: 0.1388

Comment: What do you mean? Why do you need that? What do you think will happen when you don't have a dependent variable in regression framework? Could you give us more information about your analysis?
Comment: I think this should be left open (see my answer) but am not sure if it belongs here on CV or on StackOverflow.
Answer: No, it isn't possible.  And R gives an error message when you try.  What appears to have happened in your case is that y was defined somewhere else and you forgot that it was there.
E.g.
set.seed(1075)
x1 <- rnorm(100)
x2 <- rnorm(100)

df1 <- as.data.frame(cbind(x1,x2))

m1 <- lm(y~x1+x2, data = df1)

gives an error. But if we add
y <- rnorm(100)

m1 <- lm(y~x1+x2, data = df1)

there is no error.  This might be regarded as good or bad behavior by R, but it has nothing to do with regression.
Answer: One constructive way to understand the question is to interpret it as asking for a way to find a nontrivial linear combination of variables that is as close to zero as possible.
We have to impose some kind of nontriviality condition, for otherwise the solution given by all zero coefficients is always available--and useless.  A reasonable condition to impose is to set some kind of lower bound on the overall size of the coefficients.
For instance, suppose we were to require the sum of squares of coefficients to be no less than unity.  Letting $X$ be, as usual, the $n\times p$ matrix of $n$ observations of $p$ variables (perhaps centered, perhaps normalized), the problem can be formulated
$$\text{Minimize } ||X\beta||^2 \text{ subject to } ||\beta||^2 = 1.$$
This problem was solved in my answer at https://stats.stackexchange.com/a/74328/919: $X\beta$ must be a principal component of $X$ with a smallest eigenvalue.
Such a "regression," as indicated in that thread, finds a way to express any one of the $p$ variables as a linear combination of the remaining $p-1$ variables, perhaps with some error.  However, it does not single out any of the variables as a "response" and thereby avoids the asymmetry imposed by Least Squares regression.  Geometrically, it amounts to finding the best-fitting $p-1$-dimensional hyperplane to the $p$-dimensional point cloud determined by the observations (in the least squares sense).

Here, to illustrate, is a "no-response regression" of $100$ observations of $p=2$ variables, displayed as a point cloud.  The model matrix was centered.  The solution is shown as the red line.  For reference, shown in blue in the background is the least squares fit of the model $Y \sim X$.  Unlike the "no-response" fit, it exhibits regression to the mean due to the asymmetrical treatment of $X$ and $Y$.
Answer: In short, No. Regression analysis is supervised in nature by definition (and by implementation as well). To help you understand why, let me describe, in simple words, what regression analysis does.
A regression analysis is essentially finding coefficients for variables in a function. We do not know what is this "function" beforehand, and therefore we try to find the function by guessing it. The way we find a close match with the actual function is by minimising the difference with actual value (i.e. the known value of dependent variable) and what we are getting as the result of our guess. Therefore, we have to have a way to know this difference i.e. error. Without dependent variable, this is impossible. Therefore, a regression analysis is not possible without dependent variable.
If you do not want to use dependent variable, can you elaborate what you are attempting to achieve? There are many unsupervised algorithms (i.e. to work without dependent variable). But a bit more details is needed before such an algorithm can be suggested.
Answer: I am assuming you mean a regression that would use no explanatory variables, which seems to be the {x1,...,x5} in your set.
This would mean creating a model of the type $\hat y = \alpha$, which means your response variable $Y$ is not dependent on the values of any other variable (including itself); all predicted values $\hat y$ will be the same, $\alpha$. The observed values $y$ will thus equal $\alpha + \varepsilon$, where $\varepsilon$ is the residual error of your model.
This may sound like a silly regression model, but it is essentially what happens when you use a summary statistic like the mean: you're representing all values of a dataset by one single number. Statistical procedures such as ANOVA compare this simple model $\hat y = \bar y$ with a more complex model such as $\hat y = \alpha + X\beta$ in order to measure the gain in error reduction provided by the more complex model.
Comment: You seem to have misinterpreted "dependent" as "independent".
Answer: I am convinced that a valid interpretation of the question above can be:
"I have a cloud of dots resulting from a dataset of n observations and 5 dimensions.
How can I fit a line through the dots?"
Under this interpretation, the answer is a straight line whose position is identified by the mean values of the dimensions.
Indeed, none of the coefficients in the code snippet provided with the question have a p-value over 0.05, besides the Intercept.
Comment: The "mean values of the dimensions" identifies a *point,* not a line!
Comment: This is basically equivalent to @whuber's answer.
Comment: @Whuber: Y = mean(x) is a straight line, even though mean(x) is a point.

BenBolker: Do you mean equivalent to Waldir's answer?
Comment: "Y = mean(x)" makes no sense at all.  Could you clarify?
Comment: Yea, you're right, it should have been Y = sum(yi) / count(yi), with i being each observation. I believe the user has a cloud of dots, and is trying to approximate it with a line.


Finding the Upper and Lower Bounds of Possible $x$ values

Question: Given that $n$ is an odd semiprime (not a square of primes), $a=\operatorname{ceil}(\sqrt{n})$ and $b=a^2-n$. We also have a function, $f(x)$, such that $$f(x)=\frac{b-x^2}{2x-2a}$$ For each $n$, there is one trivial $x$ value that is equal to $$x=a-1$$ For each $n$, there is also one non-trivial positive $x$ value that is an integer (not including $0$) that also makes $f(k)$ a positive integer (including $0$). We'll call this $k$. What I am asking is this: what are the tightest bounds that this integer $k$ must be within? The ones I have is: $$-1+\sqrt{1+b+2a}\leq k\leq a-2$$ I don't know if those are the tightest. Also, what I want to know is what are the bounds that $f(k)$ must be within? All bounds must be in terms of $a,b$ and $n$.
Answer: The initial premise is wrong. In fact, there are precisely eight integers $k$ such that $f(k)$ is an integer (including $a-1$).
For this $k$ we have $2(k-a)\mid b-k^2$. But we have $b-k^2\equiv b-a^2=-n\pmod{k-a}$, therefore $k-a\mid n$. If we write $n=pq$ with $p,q$ odd primes, then we have $k-a=\pm 1,\pm p,\pm q,\pm pq$. And, indeed, any of those values can appear.
Let $k$ be any of the numbers $a\pm 1,a\pm p,a\pm q,a\pm pq$. For each of those $k-a$ is odd and, by going with the previous calculation backwards, $k-a\mid b-k^2$. Further, $k$ has parity opposite to that of $a$ (if $a$ is even, $k$ is odd and vice versa), and the same is true of $b$, so $b-k^2$ is even. It follows that $2(k-a)\mid b-k^2$, and hence $f(k)=\frac{b-k^2}{2k-2a}$ is an integer.
Edit: Let me assume $p<q$. Then of the eight values of $k$ above all are positive, except for $a-q,a-pq$. To get $f(k)$ positive, we must either have $k>a$ and $b\geq k^2$ or $k<a$ and $b\leq k^2$. The former is impossible, since it implies $b\geq (a+1)^2$, and this is easily seen to never hold - indeed, $b\leq 2a-1$.
Therefore we have $k<a$. This implies that $k=a-1$ or $k=a-p$, so $a-p$ is necessarily the value you seek for. In terms of $n$, you can't say anything more about $p$ than $3\leq p\leq\sqrt{n+1}-1$ (which happens when $q=p+2$), so $a-\sqrt{n+1}+1\leq k=a-p\leq a-3$. If you want a bound in terms of just $a,b$ you get $a-\sqrt{a^2-b+1}+1\leq k\leq a-3$ and those bounds are optimal.
Comment: Whoops meant to say that $f(k)$ is positive.
Comment: $k$ must also be positive.
Comment: Since $b=0$, $f(1)=0$ which is not positive.
Comment: $0$ is usually (in the English-speaking countries at least) *not* considered to be positive nor negative. Rather, it is nonnegative and nonpositive.
Comment: @QuoteDave In that case, $k$ may not exist, take e.g. $n=15$.
Comment: Actually it does: $x=1$
Comment: Well, technically, I consider $0$ positive, but I'll adjust the parameters.
Comment: Yeah, so I adjusted my question.
Comment: Thank you, but those bounds are slightly worse than mine


WinUI 3 fill width/height

Question: So I'm trying a simple chat application in which I have some tabs on the top and each tab should have a RichTextBlock for output, a ListView at the right of the RichTextBlock for info and a RichTextBox for input. This is all contained in a Page to be displayed under a NavigationView.
I want the chat output to take full width of the window and most of the height, and the chat input to take full width and the remaining of the height.
Now matter what I 'm trying, I can't make the sizes work.
Results: The row heights in the grid are ignored. The RichTextBlock only grows in height where there's text in it (while I want it full height even without text), and the chat input width isn't expanded.
<DataTemplate x:Key="Template_Tab" x:DataType="local:Item">
    <TabViewItem  HeaderTemplate="{StaticResource Template_TabHeader}" IsClosable="True" CloseRequested="Close_Requested" >
        <Grid Margin="10,10,10,10" >
            <Grid.RowDefinitions>
                <RowDefinition Height="12*"></RowDefinition>
                <RowDefinition Height="*"></RowDefinition>
                <RowDefinition Height="*"></RowDefinition>
            </Grid.RowDefinitions>

            <SplitView x:Name="splitView" Grid.Row="0"
                           IsPaneOpen="{x:Bind IsOpenChannels}" OpenPaneLength="256" DisplayMode="Inline" PanePlacement="Right">
                <SplitView.Pane>
                    <Grid>
                        <Grid.RowDefinitions>
                            <RowDefinition Height="Auto"/>
                            <RowDefinition Height="*"/>
                            <RowDefinition Height="Auto"/>
                        </Grid.RowDefinitions>
                        <InfoBar Grid.Row="0" IsOpen="True" Severity="Informational" Title="" CloseButtonClick="Cmd_CloseList"/>
                        <ListView  Grid.Row="1" ItemsSource="{x:Bind Channels,Mode=OneWay}" x:Name="List_Channels"  SelectionMode="Multiple" ItemTemplate="{StaticResource Template_Channel}"
                                   RightTapped="Channels_RightTap" DoubleTapped="Channels_DoubleTap"
                                   />
                    </Grid>
                </SplitView.Pane>

                <ScrollViewer Name="Scroll"
                      Margin="0"
                      VerticalAlignment="Top"
                      VerticalScrollBarVisibility="Visible"
                      HorizontalScrollBarVisibility="Disabled">
                    <RichTextBlock x:Name="Output" Margin="0,0,0,10" IsColorFontEnabled="True" IsTextScaleFactorEnabled="True" IsTextSelectionEnabled="True" >
                    </RichTextBlock>
                </ScrollViewer>

            </SplitView>

            <Line Stroke="SteelBlue" Grid.Row="1" x:Name="LineSep1"
                      X1="0" Y1="0"
                      X2="{x:Bind ChatWidth}" Y2="0"
                      Margin="0,10,0,10"
                      StrokeThickness="3"/>

            <StackPanel x:Name="InputSP" Margin="0,10,10,0" Orientation="Horizontal" Grid.Row="2" MaxHeight="70" >
                <Button x:Name="ib1" >
                    <FontIcon Glyph="" />
                    <Button.Flyout>
                        <MenuFlyout>
                            <MenuFlyoutItem Text="Reset"/>
                            <MenuFlyoutSeparator/>
                            <ToggleMenuFlyoutItem Text="Repeat" IsChecked="True"/>
                            <ToggleMenuFlyoutItem Text="Shuffle" IsChecked="True"/>
                        </MenuFlyout>
                    </Button.Flyout>
                </Button>
                <RichEditBox x:Name="Input"  KeyDown="KD"  AcceptsReturn="False"></RichEditBox>
            </StackPanel>
        </Grid>
    </TabViewItem>
</DataTemplate>

Basically, my question is how to use the Grid to force width and height of elements fully in the contained Page.
In plain Win32 of course I would handle WM_SIZE, take the rect of the window with GetClientRect and adjust each control's position with SetWindowPos.
Answer: Try these:
First, make sure your TabView's VerticalAlignment is set to Stretch.
Then, adjust the RowDefinitions on the root Grid of the TabViewItem. You are assigning too much space for the "Line" and the "Input" section.
<Grid.RowDefinitions>
    <RowDefinition Height="*"></RowDefinition>
    <RowDefinition Height="Auto"></RowDefinition>
    <RowDefinition Height="Auto"></RowDefinition>
</Grid.RowDefinitions>

This way, the 2nd and 3rd rows will take the minimum height, and the 1st row, which is the "Output" row, will take the rest.
You should also try setting each ScrollViewer's [Horizontal/Vertical](Content)Aligment to Stretch.
Comment: I pasted your code on a blank WinUI 3 app and at least the `ScrollViewer` is stretched. Since the `RichTextBlock` doesn't have `Background` or `Border`s, I guess that should do.

WinUI 3 fill width/height


Configurar tabela HTML, usando a estrutura 'for' do python django

Question: Estou fazendo uma tabela através de 3 listas geradas na views, mas não consigo deixar a tabela configurada corretamento no html, seriam 3 colunas e 60 linhas, mas elas não ficam ajustadas corretamente, segue meu código HTML:

                Horario
                Ligado
                Desligado

            {% for x in teste4 %}

                {{ x }}

            {% endfor %}

            {% for y in tabela1 %}

                {{ y }}

            {% endfor %}
            {% for z in tabela2 %}

                {{ z }}

            {% endfor %}

Gera esse resultado(diminui o tamanho para caber na imagem)

E o código que gera essa tabela:
    x = PrettyTable(["Horario", "Ligado", "Desligado"])
    x.align["Horario"] = "2"
    x.align["Ligado"] = "2"
    x.align["Desligado"] = "r"
    x.padding_width = 1
    z=0
    while z < len(teste4):
        x.add_row([teste4[z], tabela1[z], tabela2[z]])
        z = z + 1
    print (x)

EDIT
Conforme falado nos comentarios alterei a views para passar um método e a tabela gerada tambem deu errado, segue código e imagem
    x = PrettyTable(["Horario", "Ligado", "Desligado"])
    x.align["Horario"] = "2"
    x.align["Ligado"] = "2"
    x.align["Desligado"] = "r"
    x.padding_width = 1
    z=0
    while z < len(teste4):
        x.add_row([teste4[z], tabela1[z], tabela2[z]])
        z = z + 1
    print (x)
    passar = x.get_html_string()

com o html ficando Tabela:{{ passar }}
Comment: A estrutura da sua tabela no HTML está bem errada. Você está inserindo elementos `` dentro de elementos ``, que já estão dentro de ``. Na tabela, você sempre define uma linha por vez, ou seja, para cada `` você terá um ou mais `` e este elemento **não pode** ter `` dentro.
Comment: Sim eu ja fiz com a estrutura certa, mas o resultado sai pior do que esta na pergunta, por incrivel que pareça com essa estrutura é onde eu consegui o resultado mais parecido com oq eu quero, mas arrumei a pergunta com a estrutura certa
Comment: Como comentado na sua outra pergunta, basta utilizar o método `get_html_string` do `PrettyTable`.
Comment: Até agora nao entendi como usar esse metodo
Comment: Ele retorna uma *string* com todo o código HTML da sua tabela, é só exibir.
Comment: Por curiosidade, por que possui duas contas, [esta](https://pt.stackoverflow.com/users/118738/guilherme) e [esta](https://pt.stackoverflow.com/users/141733/guilherme), distintas no site? Criou uma nova por engano?
Comment: tentei aqui e retornou errado, vou editar a pergunta com o metodo e seu retorno, sobre as duas contas, eu uso uma em cada pc, quando criei um nao lembrava que tinha no outro, fazi tempo que nao usava
Comment: E se você fizer `{{ passar|safe }}`?
Comment: opa, ai deu certo, muito obrigado, só uma duvida, tem como eu deixar os campos da tabela mais espaçados entre eles?
Answer: Como já citado, o método mais fácil é utilizar get_html_string do PrettyTable, pois ele já retorna, como string, todo o código HTML da sua tabela.
Porém, como padrão, o Django escapa os valores injetados nos templates como forma de prevenir alguns tipos de ataque, como XSS. Para conseguir exibir corretamente a sua tabela, será necessário indicar que o valor injetado é seguro fazendo, no seu template:
{{ passar|safe }}

Para mais detalhes do filtro safe, veja a documentação.


Event handler not triggering when user created using SCIM/REST APIs for UDF

Question: While creating user via OIM API--> My Event handler gets Triggered
While creating user via OIM GUI--> My Event handler gets Triggered
While creating user via OIM SCIM API--> My Event handler is NOT getting Triggered...
Any clues?
Comment: How is your handler configured? Is it conditional?
Answer:
Using GET schemas you will get to under which Schema the attribute would belong.(In my case I had two custom attribute one is text field and other was of type lookup.While using GET schemas I only to got to know schema for the text field as it returned nothing for the lookup field.
"urn:ietf:params:scim:schemas:extension:oracle:2.0:OIG:User").
Take metadata export of the file User.xml .Search for the attribute name in that file and under <scim> tag enter the following entry in <value>
<name>scim</name>
<value>urn:ietf:params:scim:schemas:extension:oracle:2.0:OIG:User:<attrName></value>
<category>properties</category>

Then import the User.xml file

Restart OIM server. Then test the scenario, event handler will get triggered.
Although for the lookup field I didn't get it any under any schema but when I used the same schema as that of text field it started working fine.


Laravel helpers render view display empty

Question: I have this global helper for displaying error page
app/Helpers/ErrorPage.php
function error_page($txt,$url=false,$button_txt='Go back'){
    if( !$url ){
        $url = url('/');
    }
    $data = [
        'page' => 'error page',
        'text' => $txt,
        'button_url' => $url,
        'button_text' => $button_txt
    ];

    return view('utility.error',$data);
}

then add it to autoload
"autoload": {
    "files": [
        "app/Helpers/ErrorPage.php"
    ]
},

then did composer dump-autoload
then in my controller, call
error_page('This account is already verified!');

and it display empty unless if I do echo view('utility.error',$data); from the error_page function, the view renders the page with unacceptable abnormalities on the layout.
any help, ideas is greatly appreciated.
Comment: You need to return the view from your controller. `return error_page('This account is already verified!'); `
Comment: Your error_page function just returns a view. It doesn't return it to the controller or stop the request processing. Where is the code where you call `error_page()`?
Comment: @bassxzero in my controller
Comment: @bassxzero did test it via route display, same result
Comment: ah yes, its woking now. can you post your answer so I can accept it
Answer: The controller expects you to return a view, string, etc. for it to send back as a response.
You are creating the view in your error_page() function, but you aren't using it.
This is how it should be used in your controller.
return error_page('This account is already verified!');


Countable Subsets of Irrational Sets

Question: If A is an uncountable subset of the reals such that all elements of A are irrational, is there a countable subset Z of A such that for all x,y in A, there is a z in Z such that x>z>y?
In general, I want to claim that for any uncountable subset B of the reals, there is always a countable subset Y such that $\forall b,b' \in B$ such that $b>b'$, $\exists y\in Y$ such that $b>y>b'$.
Comment: No, there isn't. For instance, let $A$ be the set of all irrational numbers $x$ such that $x^2>=2$. Now take $x=+\sqrt 2, y=-\sqrt 2$. (By the way, you forgot to say that $x>y$.)
Comment: Not true, as others have pointed out, but a slight change makes it true and more interesting: $\forall b,b' \in B \setminus Y ....$
Answer: As noted in the comments, this is not always possible, because $A$ can contain points that adjacent in $A$. However, it cannot contain too many of them.
Let $A\subseteq\Bbb R$ be uncountable. Let
$$S=\left\{x\in A:\exists y\in A\big(x<y\text{ and }(x,y)\cap A=\varnothing\big)\right\}\,,$$
the set of elements of $A$ that have immediate successors in $A$. For $s\in S$ let $s^+$ be the immediate successor of $s$ in $S$. The intervals $(s,s^+)$ for $s\in S$ must contain distinct rationals, so $S$ is countable.
$\Bbb R$ is hereditarily separable, so $A$ has a countable subset $D$ that is dense in $A$ in the subspace topology on $A$. Suppose that $x,y\in A$ and $x<y$. If $(x,y)\cap A=\varnothing$, then $x\in S$, and $y=s^+$. Otherwise, $(x,y)\cap A$ is a non-empty open subset of $A$ in the subspace topology on $A$, so there is a $z\in D$ such that $x<z<y$.
This is the best that you can do along these lines unless you begin by requiring that $A$ be densely ordered.
Comment: Thank you. I am specifically trying to show that if I have an asymmetric and negatively transitive binary relation >' on an uncountable set X and there exists a function u: X -> R such that for x,y in X, x>'y iff u(x) > u(y), then there must exist a countable order-dense subset Z of X, so that for all x,y in X with x>'y, there exists a z in Z such that x>'z>'y. My concern was if the range of u was an uncountable set of irrationals from R. So now I'm not sure.
Comment: @N.C.: Am I correct in thinking that a relation $\succ$ is negatively transitive if $x\not\succ y$ and $y\not\succ z$ imply that $x\not\succ z$?
Comment: That's correct. And this is equivalent to saying that if $x\succ y$, then for all $z\in X,$ either $x\succ z$ or $z\succ y$.
Comment: @N.C.: Okay; I'll try to give it some thought.
Comment: Thanks a lot. Much appreciated.
Comment: @N.C. I don't understand your initial comment: if $x$ and $y$ are successors in $X$, then regardless of what $Z$ we pick there is no $z\in Z$ between $x$ and $y$. Maybe you want to say "for $x,y$ in $Z$" instead of "for $x,y$ in $X$?"
Comment: @NoahSchweber I never said they were successors.

Countable Subsets of Irrational Sets


logarithm different bases comparison

Question: In my math textbook, I came across this question which I do not know how to answer. For a variable n, where n>1, how would you prove that $\log_n (n+1) > \log_{n+1} (n+2)$?
Thank you.
Comment: What I would do is take the limit of $\log_n(n+1)$ as n tends to $\infty$ and try making something out of that. If the limit converges to $0$, then the inequality is correct, if it diverges to $\infty$, then the inequality is wrong.
Comment: I wonder if it is true that $\log_x(x+1)$ is a strictly decreasing function on $(1,\infty)$?
Answer: Consider $f(t) = \log_t(t+1)={\log(t+1)\over\log(t)}$, so
$$f'(t) = \frac{\frac{1}{t+1} \log (t) - \frac{1}{t} \log(1+t)}{\log(t)^2}$$
But $x \log x$ is an increasing function on $x>1$, so $t \log t < (t+1) \log(t+1)$, hence $f'(t) <0$ on $(1, \infty)$ so $f$ is decreasing, as required.
Answer: it can be shown to be true also without calculus by AM-GM inequality
$$\frac{a+b}{2} \geq \sqrt{ab}$$
firstly observe that
$$\log_n (n+1) = \frac {\log (n+1)}{\log(n)}$$
$$\log_{n+1} (n+2) = \frac {\log (n+2)}{\log(n+1)}$$
then
$$\log_n (n+1) > \log_{n+1} (n+2) \iff \log^2 (n+1) > \log(n)\log(n+2)$$
now for AM-GM inequality
$$\log(n)\log(n+2)\leq \left( \frac{log(n)+log(n+2)}{2} \right)^2 = \left( \frac{log(n^2+2n)}{2} \right)^2 < \left( \frac{log(n^2+2n+1)}{2} \right)^2 = \log^2 (n+1)$$

logarithm different bases comparison


Como importar dados do DataGridView para textboxs

Question: Como eu faço para,quando eu clicar no botão modificar trazer os dados de  uma linha do DataGridView para os campos de cima e também se eu salvar como inativa deixar a linha como Enabled = false.
Comment: Web forms? Ou Windows
Comment: O tipo de formulário,né? se for é Windows Form.
Answer: txtCodigo.text = dataGridView.Rows[indiceDaLinha].Cells["nomeDaCelula"].Value.toString();

Para o nome a mesma coisa.
Para os Radios vc faz:
if(dataGridView.Rows[inicideLinha].Cells["nomeDaCelula"].Value.toString().Equals("Inativa"))
{
    radioButtonInativa.checked = true;
}

Talvez você precise fazer algumas correções com relação ao nome dos seus componentes. Mas é assim que eu faria.


Half note triplets

Question: I would like know how to count and play this half note triplets on 4/4 time.
Comment: It's not entirely clear to me what your question is. Do you know how to count quarter note or eighth note triplets but not sure how it works for larger values? Or are you asking how to count triplets in general?
Comment: Is your question "I don't know what this means," or "I know what it means, but it's hard for me to do?" The answer to the first is "It's an entire 4/4 bar evenly divided into three parts," just as the more common eighth-note triplets divide a single beat into three.
Comment: What's the piece?  Is it really 4/4 and not 2/2?  The manuscript looks like it ought to be in cut time.
Comment: @Tim I beg to differ. I love the sound of the 4 against a min7 chord. Very common in modal jazz. In fact, based on the excerpt, I would probably play (on piano) A-D-G for the Am7.
Comment: I'd be more concerned about the D note against an Am7 chord shown earlier. Doesn't sound good! And the G nat. written instead of Fx...
Answer: To do this in general, one must develop a feel for 4:3 (four against three). However, since this appears to be for a solo instrument, there can be a simpler option. Here are a few common techniques for handling this situation:
1. "Changing" the time signature (i.e., Count the beats differently)
On an instrument like piano, it is sometimes necessary to play a "true" 4:3. Say, the right hand is playing triplets against sixteenth notes in the left hand.
However, on a solo instrument, there is sometimes a "cheat" which would seem to apply here. Rather than counting the measure in four beats, count it in two (if you're comfortable with 3:2) or even one (especially if the tempo is fast). In the latter case, the half-note triplet just becomes a "normal" triplet.
To do this most effectively, start counting in 2 or 1 a few measures beforehand, so that you have the feel well established, particularly of the downbeat, before getting to the triplet.
2. Counting it mechanically (least common multiple)
Divide the measure into twelve equal pulses

Divide each of the measure's four beats into three sub-pulses (i.e., triplets) — so, twelve total "pulses".
Divide each of the triplet notes into four parts — again, a total of twelve pulses.

This will show how each triplet note aligns with the beat.

Starting at a very slow speed, count to twelve, tapping one hand on the main beat and one hand on the triplet beat.
As the rhythm becomes intuitive at a slow tempo, begin to speed up.

See graphic below.
3. Solfege
Very similar to the above.

Count triplets for each main beat, emphasizing the part of the beat where the half-note triplet occurs.
Count the half-note triplet in four-pulse units (i.e., like counting sixteenth notes), emphasizing the main metrical beats.

MAIN EMPHASIS / _secondary emphasis_ (or vice versa)

ONE two three _one_ TWO three _one_ two THREE _one_ two three

ONE two three FOUR _one_ two THREE four _one_ TWO three four

4. Mnemonic
The most commonly encountered mnemonic for 4:3 is

Pass the gosh darn spinach

or variations thereof.
Comment: The fourth way is to "feel" this section as one beat per bar; then it is easy to divide by 3.
Comment: @AndyBonner I see. Good thought. I've rearranged the answer to put more focus on the "changing how you count" option. Any additional feedback is appreciated.
Comment: @BrianChandler Absolutely. May I add that to my post?
Comment: Yes, of course...
Comment: @BrianChandler's suggestion typically works well for faster tempos and poorly for slower tempos.  People trying to use this approach for slower tempos frequently end up with two dotted quarter notes and a quarter note instead of three even beats.  There's also an intermediate possibility: if the piece is in cut time (and the manuscript style suggests that this is likely, though the question indicates it is 4/4) then the problem is one of 3 against 2 rather than 3 against 4, and many more people can handle 3 against 2.
Comment: @phoog I believe that's consistent with what I wrote in my post.
Comment: Re @phoog's/Brian's suggestion: I didn't contribute an answer since I'm not sure whether the OP is looking for "what is this" or for "how do I do it," but if it were the latter (and the tempo makes it possible), I'd advise to prepare for the triplet by mentally focusing on the downbeat in the preceding measures, to "feel the 'one,'" just as one might make a transition from eighth notes to triplets by focusing more on the beat than the subdivision.
Comment: @AndyBonner Given you're the third to make the comment, I'm starting to wonder ... does my post not clearly (enough) communicate the point about counting in 1?
Comment: Heh, I realize now I abandoned a lengthier comment yesterday as too nitpicky. I was going to say that internalizing the 3:4 (or :2) relationship might only be necessary if you're in a position to care about it (e.g. you're the conductor, or playing multiple voices, etc.) It can of course be helpful in any case. But if you only have to play the "3," practical advice might be to "tune out" the "4." Your recent edit mentions this, but sorta "below the fold."
Comment: @phoog I've reworked the post to put more emphasis on the "changing how you count" aspect. Any thoughts to better the post would be appreciated.
Answer: Perhaps it would be easier to interpret this rhythm notated the following way, especially if you're familiar with jazz swing eight note triplet patterns?

Half note triplets


Does higher variance in predictions result in higher variance error estimation?

Question: Motivation
Everyone knows that fitting high variance models requires more data. A "yes" answer to the question would suggest that more data is also needed to evaluate these models.
Assumptions

$\forall i \neq j, Y_i \perp Y_j$. This is just a usual independence assumption.
$\forall i, j, Y_i \perp \hat{Y}_j$. This should come from the fact that outcomes (in our evaluation or "test" dataset) are independent of data used to fit a model which supplies predictions.

Feel free to assume a bit more if necessary, e.g., we're fitting a Gaussian linear model. Keep in mind that because this problem is about prediction, it's unreasonable to assume $\text{E}(Y_i) = \text{E}(Y_j)$ or $\text{E}(\hat{Y}_i) = \text{E}(\hat{Y}_j)$ for $i \neq j$.
Hypothesis
Variance in predictions contributes to variance in error, as error is a function of predictions.
Progress
Math
Is there a way to decompose, for example, $\text{Var}\Big( \frac{1}{n} \sum_{i=1}^{n} (Y_i - \hat{Y}_i)^2 \Big)$, into a sum including $\text{Var}(\hat{Y}_i)$? So far I've broken down this down into a bunch of granular terms, and am trying to consolidate these into something clearer. For notation sake, let $\epsilon_i = Y_i - \hat{Y_i}$.
$$
\begin{equation}
\text{Var}\Big( \frac{1}{n} \sum_{i=1}^{n} \epsilon_i^2 \Big) = \frac{1}{n^2} \Big( \sum_{i=1}^{n} \text{Var}(\epsilon_i^2) + 2\sum_{i < j} \text{Cov}(\epsilon_i^2, \epsilon_j^2) \Big).
\end{equation}
$$
Breaking down $\text{Var}(\epsilon_i^2)$:
$$
\begin{align}
\text{Var}(\epsilon_i^2) &= \text{Var}(Y_i^2 - 2Y_i\hat{Y}_i + \hat{Y}_i^2) \\
&= \text{Var}(Y_i^2) + 4\text{Var}(Y_i\hat{Y}_i) + \text{Var}(\hat{Y}_i^2) - 2\text{Cov}(Y_i^2, Y_i\hat{Y}_i) - 2\text{Cov}(\hat{Y}_i^2, Y_i\hat{Y}_i) \\
&= \text{Var}(Y_i^2) + \text{E}(\hat{Y}_i)^2 \text{Var}(Y_i) + \text{Var}(\hat{Y}_i)\text{E}(Y_i)^2 + \text{Var}(\hat{Y}_i^2) \\
   &\qquad - 2\text{E}(\hat{Y}_i)\text{Cov}(Y_i^2, Y_i) - 2\text{E}(Y_i)\text{Cov}(\hat{Y}_i^2, \hat{Y}_i).
\end{align}
$$
Note that I used this fact to break down $\text{Var}(Y_i\hat{Y}_i)$ and this fact to break down the two covariance terms.
Breaking down the covariances $\text{Cov}(\epsilon_i^2, \epsilon_j^2)$ where $i < j$, thanks to @gunes' answer (which I think used the above referenced fact about covariance):
$$
\begin{align}
\text{Cov}(\epsilon_i^2, \epsilon_j^2) &= \text{Cov}(Y_i^2 - 2Y_i\hat{Y}_i + \hat{Y}_i^2, Y_j^2 - 2Y_j\hat{Y}_j + \hat{Y}_j^2) \\
&= \text{Cov}(Y_i^2, Y_j^2) - 2\text{Cov}(Y_i^2, Y_j\hat{Y}_j) + \text{Cov}(Y_i^2, \hat{Y}_j^2) \\
   &\qquad - 2\text{Cov}(Y_i\hat{Y}_i, Y_j^2) + 4\text{Cov}(Y_i\hat{Y}_i, Y_j\hat{Y}_j) - 2\text{Cov}(Y_i\hat{Y}_i, \hat{Y}_j^2)\\
   &\qquad + \text{Cov}(\hat{Y}_i^2, Y_j^2) - 2\text{Cov}(\hat{Y}_i^2, Y_j\hat{Y}_j) + \text{Cov}(\hat{Y}_i^2, \hat{Y}_j^2) \\
&= 0 -2(0) + 0 \\
   &\qquad - 2(0) + 4\text{Cov}(Y_i\hat{Y}_i, Y_j\hat{Y}_j) - 2\text{Cov}(Y_i\hat{Y}_i, \hat{Y}_j^2) \\
   &\qquad + 0 - 2\text{Cov}(\hat{Y}_i^2, Y_j\hat{Y}_j) + \text{Cov}(\hat{Y}_i^2, \hat{Y}_j^2) \\
&= 4\text{E}(Y_i)\text{E}(Y_j)\text{Cov}(\hat{Y}_i, \hat{Y}_j) - 2\text{E}(Y_i)\text{Cov}(\hat{Y}_i, \hat{Y}_j) \\
   &\qquad - 2\text{E}(Y_j)\text{Cov}(\hat{Y}_i, \hat{Y}_j) + \text{Cov}(\hat{Y}_i^2, \hat{Y}_j^2) \\
&= \text{Cov}(\hat{Y}_i, \hat{Y}_j)\big( 4\text{E}(Y_i)\text{E}(Y_j) - 2(\text{E}(Y_i) + \text{E}(Y_j)) \big) + \text{Cov}(\hat{Y}_i^2, \hat{Y}_j^2).
\end{align}
$$
At this point, employing the assumption that we're fitting a Gaussian linear model may help.
Simulation - LASSO
See this Python notebook.

Problems with this simulation:

I thought the variance in predictions decreases as the $l_1$ penalization coefficient (alpha in scikit-learn) increases. This simulation does not have this property.
The plot is quite sensitive to dataset parameters. I can explore this more. But overall, the simulation doesn't provide consistent evidence for or against the hypothesis that higher variance in predictions causes higher variance in error.

Simulation - polynomial regression
Maybe there's a weird problem with the way I estimated the total variance in predictions. To avoid estimating it I decided to directly compute it, which is straightforward for unregularized linear regression.
Note that a problem with this simulation is that it currently isn't capable of fitting significantly variant models. Fitting a polynomial regression past degree=6 screws up compute_var_preds, as it needs to compute an inverse. Perhaps there are better numerical approximations to this quantity.
Speaking of compute_var_preds, below is the math for it (the $*$ subscript denotes test data):
$$
\begin{align}
\text{Var}(\hat{\mathbf{y}}_*) &= \text{Var}(X_* \hat{\mathbf{\beta}}) \\
&= \text{Var}(X_* (X^T X)^{-1} X^T \mathbf{y}) \\
&= X_* (X^T X)^{-1} X^T \text{Var}(\mathbf{y}) X (X^T X)^{-1} X_*^T \\
&= X_* (X^T X)^{-1} X^T (\sigma^2 I) X (X^T X)^{-1} X_*^T \\
&= \sigma^2 X_* (X^T X)^{-1} X_*^T.
\end{align}
$$
The total variance of predictions is defined as the sum of elements in $\text{Var}(\hat{\mathbf{y}}_*)$, which is equivalent to (only b/c it simplifies the code):
$$
\begin{align}
\sum_{i=1}^{n_*} \text{Var}(\hat{\mathbf{y}}_*)_i &= \sum_{i=1}^{n_*} \sigma^2 \mathbf{x}_{*i}^T (X^T X)^{-1} \mathbf{x}_{*i} \\
&= \text{trace}( \sigma^2 X_* (X^T X)^{-1} X_*^T ).
\end{align}
$$
# input arguments to simulate simple linear regression
NUM_SAMPLES = 100
NUM_EXAMPLES = 500
TRAIN_FRAC = 0.5
ERROR_SD = 2
X_LB = 0; X_UB = 10
INTERCEPT = 0; SLOPE = 1
POLYNOMIAL_DEGREES = 1:6
# my compute_var_preds code can't handle degrees > 6
# b/c it directly computes an inverse
SEED = 42

# fixed data and parameters
ERROR_VAR = ERROR_SD^2
x = runif(NUM_EXAMPLES, min=X_LB, max=X_UB)
beta = c(INTERCEPT, SLOPE)
y_true_mean = cbind(1, x) %*% beta

compute_var_preds = function(X_tr, X_te, error_var=ERROR_VAR) {
  # Returns sum of true variances of test set predictions using
  # an analytical formula
  X_tr = as.matrix(X_tr)
  X_te = as.matrix(X_te)
  XtX_inv = solve(t(X_tr) %*% X_tr)
  true_var_preds = ERROR_VAR*diag((X_te %*% XtX_inv %*% t(X_te)))
  return(sum(true_var_preds))
}

# loop takes 5-10 seconds
mses = matrix(rep(NA, NUM_SAMPLES*length(POLYNOMIAL_DEGREES)),
              nrow=NUM_SAMPLES,
              ncol=length(POLYNOMIAL_DEGREES))
var_preds = matrix(rep(NA, NUM_SAMPLES*length(POLYNOMIAL_DEGREES)),
                   nrow=NUM_SAMPLES,
                   ncol=length(POLYNOMIAL_DEGREES))
for (i in 1:NUM_SAMPLES) {
  j = 1 # indexes degree
  for (degree in POLYNOMIAL_DEGREES) {
    # x - polynomial transform, handle degree=0
    if (degree == 0) {
      X_poly = data.frame(x=rep(1, length(x)))
    } else {
      X_poly = data.frame(poly(x, degree=degree, raw=TRUE))
    }
    # y - randomly generate
    set.seed(SEED*i*j)
    error = rnorm(NUM_EXAMPLES, mean=0, sd=ERROR_SD)
    y = y_true_mean + error
    # split into train and test
    tr_inds = sample(NUM_EXAMPLES, floor(NUM_EXAMPLES*TRAIN_FRAC))
    y_tr = y[tr_inds]
    y_te = y[-tr_inds]
    X_tr = X_poly[tr_inds,,drop=FALSE] # keep the df's row names
    X_te = X_poly[-tr_inds,,drop=FALSE]
    # fit, predict
    model = lm(y_tr ~ ., data=X_tr)
    # suppress warnings about predictions from rank-deficient fit.
    # should only occur for degree 0, but whatever...
    y_tr_pred = suppressWarnings(predict(model, X_tr, type="response"))
    y_te_pred = suppressWarnings(predict(model, X_te, type="response"))
    # store results
    var_preds[i,j] = compute_var_preds(X_tr, X_te)
    mses[i,j] = mean((y_te - y_te_pred)^2)
    j = j+1
  }
}

var_preds_mean = apply(var_preds, MARGIN=2,
                       FUN=mean) # randomness in splits. aggregate w/ mean
var_mses_estimates = apply(mses, MARGIN=2, FUN=var)

# final plot of interest
poly_degree_col = 'red'
plot(var_preds_mean, var_mses_estimates,
     xlab='true total Var(predictions)',
     ylab='estimated Var(test MSE)',
     pch=16, cex=1)
text(var_preds_mean, var_mses_estimates,
     POLYNOMIAL_DEGREES, pos=2, col=poly_degree_col)
legend('bottomleft',
       legend='polynomial degree', text.col=poly_degree_col)

Note that y's range is around 0-10.
# misc plots
boxplot(mses,
        names=POLYNOMIAL_DEGREES,
        xlab='polynomial degree',
        ylab='test MSE')

# visual check for degree of overfitting for the last degree
# x_tr = x[tr_inds]
# plot(x_tr, y_tr,
#      xlab='x (train)',
#      ylab='y (train)',
#      main=paste('polynomial degree:', degree))
# x_tr_order = order(x_tr) # to plot preds as a line, need x to be sorted
# lines(x_tr[x_tr_order],
#       y_tr_pred[x_tr_order])

IMO, test MSE isn't different enough to draw any solid conclusions about the question. I'm just sharing my progress so far.
Comment: If predictions and outcomes of different $i,j$ are assumed to be independent as well, this can be reduced to a simple distribution of variance over the summands.
Comment: Oh yeah sorry I'll add that assumption in. Basically, assume the model was fit on data that's independent of the evaluation data. Later today I'll add my progress on the calculation so that you can see where I'm stuck. I'd also like to run some simulations to get a quick and dirty answer
Comment: Perhaps I don't understand but are you claiming that $z_i=10x_i + \epsilon_i$ has more test error than $y_i=x_i + \epsilon_i$
Answer: For the cross term (which should be $i\neq j$, and multiplied by $2$)
Assume $i\neq j$:
$$\begin{align}\operatorname{Cov}(\epsilon_i^2, \epsilon_j^2)&=\operatorname{Cov}(Y_i^2-2Y_i\hat Y_i+\hat Y_i^2, Y_j^2-2Y_j\hat Y_j+\hat Y_j^2)\\&=\operatorname{Cov}(-2Y_i\hat Y_i,-2Y_j\hat Y_j)+\operatorname{Cov}(-2Y_i\hat Y_i, \hat Y_j^2)\\&+ \operatorname{Cov}(\hat Y_i^2, -2Y_j\hat Y_j^2)+\operatorname{Cov}(\hat Y_i^2, \hat Y_j^2)\end{align}$$
First term:
$$t_1=4\operatorname{Cov}(Y_i\hat Y_i, Y_j\hat Y_j)=\operatorname{E}[Y_i]\operatorname{E}[Y_j]\operatorname{Cov}(\hat Y_i, \hat Y_j)=\operatorname{E}[Y]^2\operatorname{Cov}(\hat Y_i, \hat Y_j)$$
Second term (and third term, as they are symmetric):
$$t_{2,3}=-2\operatorname{E}[Y]\operatorname{Cov}(\hat Y_i, \hat Y_j^2)$$
And, the fourth term is self-explanatory. I've assumed $Y_i$ come from the same distribution.
I wonder if something like this is what you're seeking for.
Comment: I updated my question w/ your math. Two small corrections for $t_1$: (1) the $4$ coefficient was accidentally dropped (2) we can't assume $\text{E}(Y_i) = \text{E}(Y_j)$ in this prediction problem


Proof Verification: $\text{Hom}(\mathbb Z[x],S)=S$ (as rings)

Question: How to prove $\text{Hom}(\mathbb Z[x],S)=S$ (as rings), where S is any ring?
My attempt: took an element $b$ in $S$, defined a map , $b: \mathbb Z[x]\to S$ which maps $f(x)$ to $f(b)$. Clearly $b$ is a ring homomorphism, hence we proved one side inclusion.
for other side, take $b$ in $\text{Hom}(\mathbb Z[x],S)$ with the same mapping, since $b$ is a ring homomorphism from $\mathbb Z[x]$ to $S$, clearly $f(b)$ is in $S$. Since $f(x)$ is in $\mathbb Z[x]$, surely $b$ is in $S$.
Is my proof valid?
Also in case to show the generalised result for $n$ variables, can I use induction?
Comment: Then this claim cannot be true by the existence of non-commutative rings.
Comment: What is $S$?$\hspace{1pt}$
Comment: S is any ring. Forgot to mention. Sorry!
Comment: @ElliotG The term *as rings* in the statement should refer to the $\operatorname{Hom}$, i.e. we consider ring homomorphisms from $\Bbb Z[x]$ to $S$. The equality sign should then refer to a bijection, i.e. isomorphism in the category $\bf{Set}$.
Comment: @ElliotG Could you please elaborate on it a bit? I am new to study of rings.
Comment: There are, unfortunately, several conventions as to what "ring" means. Could you please give _your_ definition?
Comment: @Arthur Ring( R,+,.)  1. (R,+)is an Abelian Group 2. (R,.) is semigroup  3. + and . distribute
Comment: How are you considering $\text{Hom}(\mathbb Z[x],S)$ to be a ring?
Answer: As I understand, you give a map let's call it $\phi : S \rightarrow Hom(\mathbb{Z}[x], S)$ which takes $b \in S$ to the map $\phi_b: \mathbb{Z}[x] \rightarrow S$ which sends a polynomial $p \in \mathbb{Z}[x]$ to $\phi_b(p) = p(b)$. Note $S$ has an identity element, call it $e$, and $\phi_b(1) = e$. To check this map is well defined, for each $b \in S$ we should check $\phi_b$ is indeed a ring homomorphism. So we would show things like $\phi_b(p\cdot q) = (p\cdot q)(b) = p(b)\cdot q(b) = \phi_b(p) \cdot \phi_b(q)$.
To show that this map is indeed a bijection we can construct an inverse. Let's make a map $\psi : Hom(\mathbb Z[x], S) \rightarrow S$. We want $\psi(\phi_b) = b$ so a natural choice for $\psi$ is a map sending any homomorphism $f : \mathbb Z[x] \rightarrow S$ to $\psi(f) = f(x)$ where $x \in \mathbb Z[x]$ is a polynomial.
We have $\psi(\phi_b) = \phi_b(x) = b$ as desired. We just need to show that $\phi_{\psi(f)} = f$ for any $f \in Hom(\mathbb Z[x], S)$. So take any $p \in \mathbb Z[x]$, we have $\phi_{\psi(f)}(p) = p(\psi(f)) = p(f(x)) = f(p)$. The last equality $p(f(x)) = f(p)$ should be justified carefully! And this is the part which (I think) will break when you try to generalise this to $n$ variables.
Addendum for non-unital rings. If $S$ doesn't have an identity then $Hom(\mathbb Z, S)$ may be trivial when $S$ is non-trivial. E.g. Take $S = 2\mathbb Z$ a non-unital ring contained in $\mathbb Z$, then take $ f \in Hom(\mathbb Z[x], S)$. We have $f(1) = 2n$ for some $n \in \mathbb Z$. Since $f$ is a homomorphism of rings, $2n = f(1) = f(1^2) = f(1)^2 = 4n^2$ hence $n = 0$. So $Hom(\mathbb Z[x], S)$ is the zero ring but $S$ is not the zero ring. To avoid these pathologies we have assumed $S$ has an identity element.
Comment: I don't think the ring is required to have a multiplicative identity. So you'll get into trouble with the constant term of the polynomials.
Comment: Which part of the argument relies on $1 \in S$?
Comment: When you say $p(b)$, which element of $S$ is that if $p(x)=x+1$ and $1\notin S$?
Comment: Yes you're right. I have added an addendum to my answer showing that the result needed hold for non-unital rings.


Attachment added through hook_mail_alter() is making no effect

Question: I'm trying to add one pdf attachment to the mail sent on webform submission. I'm using hook_mail_alter() to add attachments to the mail.
This is the code I am using.
function webfpdf_mail_alter(&$message) {
$attach = genpdf($message['params']['message']);
$message['params']['attachments'][] = $attach;
dpm($message);
return $message;
}
The output message shown by dpm() shows attachment, but the email doesn't contain attachments.
This is the output of dpm().
attachments (Array, 1 element)
0 (String, 568525 characters ) Content-Type: application/pdf; name="Webform-...
Content-Type: application/pdf;
 name="Webform-submission.pdf"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="Webform-submission.pdf"

cSA4NS4wNCAwIDAgMzMuNzMgMjguM......
..../****base64data*****/.....
..... ;
body (Array, 1 element)
headers (Array, 7 elements)
MIME-Version (String, 3 characters ) 1.0
Content-Type (String, 24 characters ) text/html; charset=UTF-8
Content-Transfer-Encoding (String, 4 characters ) 8Bit
X-Mailer (String, 27 characters ) Drupal Webform (PHP/5.3.29)
Return-Path (String, 35 characters ) email@example.com
Sender (String, 35 characters ) email@example.com
From (String, 28 characters ) "LESS" <firstname.lastname@example.com>

Can anyone help me to fix this?
Answer: If you're not already using it, I think you need MIME Mail or something similar to handle attachments, and when you do, your attachment needs to be added as an array of arrays rather than an array.
This is what the README file says.

You can use the following optional parameters to build the e-mail:

'plain': Boolean, whether to send messages in plaintext-only (optional, default is FALSE).
'plaintext': Plaintext portion of a multipart e-mail (optional).
'attachments': Array of arrays with the path or content, name and MIME type of the file (optional).
'headers': A keyed array with headers (optional).


List all users with ALL PRIVILEGES in MySQL

Question: I need to find a way to list all database users that have ALL PRIVILILEGES on *.* (all databases and tables). Additionally, it would be great if it could also list only those with GRANT OPTION.
My initial thought was to select User from mysql.user where each privilege field is ="Y". That would result in a very long query. Is there another, more convenient way?
Answer: On MariaDB there are a total of 28 grants:
SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, SHUTDOWN, PROCESS, FILE, REFERENCES, INDEX, ALTER, SHOW DATABASES, SUPER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER, CREATE TABLESPACE.

To show all users with all grants:
select count(*) as grants, grantee from information_schema.user_privileges group_by grantee having grants >= 28;

To check if it's true, you can check with result from
select * from information_schema.user_privileges;

and compare :)
Comment: Why the underscore in `group_by`?


Primeiro elemento array de um array multidimensional como chave para os demais

Question: Estou lendo uma planilha de Excel que me retorna um resultado como o abaixo, sendo o primeiro array o nome das colunas
Array
(
    [0] => Array
        (
            [0] => nome
            [1] => sobre_nome
        )

    [1] => Array
        (
            [0] => nome1
            [1] => snome1
        )

    [2] => Array
        (
            [0] => nome2
            [1] => snome2
        )

    [3] => Array
        (
            [0] => nome3
            [1] => snome3
        )

)

Existe alguma forma prática de usar o primeiro elemento array, que é sempre o mesmo, como chave para os demais arrays? ficando dessa forma:
Array
(
    [0] => Array
        (
            [nome] => nome1
            [sobre_nome] => snome1
        )

    [1] => Array
        (
            [nome] => nome2
            [sobre_nome] => snome2
        )

    [2] => Array
        (
            [nome] => nome3
            [sobre_nome] => snome3
        )

)

Comment: Também seria bom colocar como você esta criando este array, pois seria mais pratico já retornar o array como deseja ao invés de percorrer todo e criar outro array.
Answer: Sim, basta você recuperar o primeiro elemento:
$chaves = $retorno[0];

depois percorrer o array começando do primeiro índice e adicionar ao novo array:
$novo = [];
for($i=1; $i < count($retorno); $i++){
  $novo[] = [
    $chaves[0] => $retorno[$i][0],
    $chaves[1] => $retorno[$i][1]
  ];
}

você terá o retorno desejado:
Array
(
    [0] => Array
        (
            [nome] => nome1
            [sobre_nome] => snome1
        )

    [1] => Array
        (
            [nome] => nome2
            [sobre_nome] => snome2
        )

    [2] => Array
        (
            [nome] => nome3
            [sobre_nome] => snome3
        )

)

Veja funcionando em ideone.com


Predicting time of user action in action sequence

Question: In my system, my users will perform a set of actions in order to accomplish a task. As each action applies a load to my server, I would like to forecast potential load spikes.
The actions will always happen in order, but the time between each action is up to the user. The same action may occur more than once in an action-set.
I would like to be able to use previous behaviour by a user to predict when they will next perform an action (and whether it will be the same action or the next in sequence) and aggregate this data to forecast total load. The first action in the set doesn't have to be predicted, but It'd be helpful if it could be.
Not being a statistician, I believe I have all the data here to make useful predictions, but I have no idea what sort of algorithm/model to use. What model would be appropriate for the information given?
Answer: This may be a good use case for Discrete Event Simulation (https://en.wikipedia.org/wiki/Discrete_event_simulation). I have utilized with R simmer package. You can set up a simulation where the tasks arrive according to a distribution you determine; and where the users "seize" and "release" the task according to probabilities and task times that you determine.
You could then use that as a foundation and extract data on when the actions are taking place and transform that into your expected server load.
So this would not be "predictive" per se (not taking into account some characteristics for the particular task in flight to predict when it will get worked), but it would be simulated according to the historical data you have on many of these tasks and actions.


Is there a "blown save" if the closer gives up only UNearned runs?

Question: If a closer pitches the ninth inning, coming in with a 3-2 lead, and gives up an earned run, that would characterized as a "blown save."
Suppose the run was UNearned because of a fielding error. Is it still a "blown save?"
Answer: Yes.
Saves and holds, like wins, are wholly dependent on the score and do not account for whether or not a run is earned or unearned.
The save rule's first clause is the one clearly in play here
10.18.a

He is the finishing pitcher in a game won by his team;

The remainder of the save rule defines the context of a save situation. If you enter the game in a save situation and fail to hold the lead then you are awarded a blown save. It does not matter whether or not the runs you allow are earned or unearned.


Is there a way to do string removing in html tags?

Question: {% for term in terms %}

        {{ term }}:    {{ terms[term] }}

{% endfor %}

'terms' is a dictionary with the value as a list in Python:
terms = {'a':['1','2','3'], 'b':['4','5','6'], 'c': ['x', 'y', 'z']}

The current html code will display the 'terms' as follows in the for loop:
a: ['1','2','3']
b: ['4','5','6']
c: ['x', 'y', 'z']

I want the quotes to be removed as follows:
a: [1, 2, 3]
b: [4, 5, 6]
c: [x, y, z]

Is there a way to run a string removing function in the html blocks? If not, is there other possible ways to display as I expected? This is in a Flask project.
Answer: Change your html to
{% for key, value in terms.items() %}

        {{ key }}:    [{{ value|join(', ') }}]

{% endfor %}

it will render to:

    a:    [1, 2, 3]

    b:    [4, 5, 6]

    c:    [x, y, z]

Answer: If you want to show without the '', you will need to show this as an element of the list.
  a = ['1','2']
  So you will need to do like

  {{a[0]}}

Or you can do something like below. It is not the best way but this does work to remove the ''. From the list, the code is:
https://github.com/Programmer-RD-AI/Stackoverflow-Questions/tree/main/Is%20there%20a%20way%20to%20do%20string%20removing%20in%20html%20tags%3F
Check the above link. You can get the code from there.


statsmodels seasonal_decompose(): What is the right "period of the series" in the context of a list column (constant vs. varying number of items)

Question: Assume having a list column so that your time series is nested, see Convert pandas df with data in a "list column" into a time series in long format. Use three columns: [list of data] + [timestamp] + [duration] for details. The question here is not about how to unnest a list column though. Assume that you already have a long format structure with all available list elements unnested into a normal column, here for example taking 2 lists with 4 list elements, making 8 rows in the end.
Part1: Assume that every list of the original list column has the same number of list elements (here, the first list with 4 items around 08:53, and the second with 4 items around 08:55).
<code>                     value
datetimeindex
2016-05-04 08:53:20  1
2016-05-04 08:53:21  2
2016-05-04 08:53:22  1
2016-05-04 08:53:23  9
2016-05-04 08:55:00  2
2016-05-04 08:55:01  2
2016-05-04 08:55:02  3
2016-05-04 08:55:03  0
</code>
Now approaching the actual question. From statsmodels.tsa.seasonal.seasonal_decompose¶ we read:
Definition of period

"period, int, optional"
Period of the series. Must be used if x is not a pandas object or if
the index of x does not have a frequency. Overrides default
periodicity of x if x is a pandas object with a timeseries index.

What is meant here with "Period of the series"? Is it:

the number of lists, here 2.
the standard size of a list. This would be 4 in the example.
something else than 1./2.

Please also explain what would be different if you had a "Part2 setting", if there is any difference:
Part2: Assume that every list of the original list column has a varying number of list elements (here, the first list with 4 items around 08:53, and the second with just 3 items around 08:55).
<code>                     value
datetimeindex
2016-05-04 08:53:20  1
2016-05-04 08:53:21  2
2016-05-04 08:53:22  1
2016-05-04 08:53:23  9
2016-05-04 08:55:00  2
2016-05-04 08:55:01  2
2016-05-04 08:55:02  3
</code>
The examples shall make the question clear, no programming (especially not with the examples) needed for an accepted answer.
Context:
This question arose from decompose() for time series: ValueError: You must specify a period or x must be a pandas object with a DatetimeIndex with a freq not set to None.
Comment: (1) The time-series data's having previously been stored in a nested list isn't obviously relevant. Some explanation is required of what that has to do with the choice of period for decomposition. (2) *Period* is the no. observations in a seasonal cycle - e.g. if you've daily observations & weekly seasonality, the period is 7. But your observations are at irregular intervals, & how to perform seasonal decomposition on irregular time series is perhaps the crux of your question (see e.g. https://stats.stackexchange.com/q/244042/17230 & https://stackoverflow.com/q/12623027/1864816).
Comment: Bounties become more noticeable towards the end of the period, as they move towards the top of the [list](https://stats.stackexchange.com/?tab=bounties). That said, it is notable that this Q has only 21 views, a number of which are me checking on it. You may want to ask a question on [meta.stats.SE] to see if there's a way to raise the profile of this Q &/or make it more answerable. (Again, this doesn't seem to be a programming Q, & there are so many Qs on [SO] that they usually get less attention than here.)
Comment: @Scortchi would you mind making an answer from your comment? It says that the period parameter does not play a role if the data does not have any cycles. Since there is no answer up to now, you are likely right with all. And it should not be just a comment then. Not sure how to reach you since this Q was migrated.
Comment: @Scortchi-ReinstateMonica Please answer.
Answer: Without cycles
The "period" parameter does not play a role if the data does not have any cycles. This short example cannot have a seasonality in it. Thus, the answer to the question is that there is no answer.
With cycles, and with observations at regular intervals
The "period" parameter is the number of observations in a seasonal cycle. For example, if you have daily observations and weekly seasonality, the period is 7.
With cycles, and with observations at irregular intervals
Not asked in the example of this question, but if the data had a seasonality and by the same time irregular intervals, there are questions dealing with this:

Trend in irregular time series data
How to analyse irregular time-series in R

Thanks go to the remarks of user Scortchi.


clr stored procedures with two in parameters

Question: I have started to learn clr stored procedures. I have managed to one work with one singel stored procedure. But when I attempt to create a stored procedure with two in parameters I run into strange results. When I supply the two parameters I am getting the total of all rows available, and not the rows supposed to be matched with parameter pair.
The two parameters in my actual table, tblRoute are named DepCity and ArvCity both with datatype nvarchar50.
I do understand that this current complexity should be rather solved in sql server itself, but this is a training exercise for clr stored procedures on my part.
Thanks
Here is my code:
using System;
using System.Data;
using System.Data.SqlClient;
using System.Data.SqlTypes;
using Microsoft.SqlServer.Server;

public partial class StoredProcedures
{

    [Microsoft.SqlServer.Server.SqlProcedure]
public static void RouteSqlStoredProcedure(SqlString strDepCity, SqlString strArvCity)
{
    SqlConnection conn = new SqlConnection();
    conn.ConnectionString = "Context Connection=true";

    SqlCommand cmd = new SqlCommand();
    cmd.Connection = conn;
    conn.Open();
    cmd.CommandText = "SELECT tblAirline.AirlineName, tblAircraft.Manufacturer, tblAircraft.AircraftModel, tblAircraft.AircraftUnits, tblAircraft.SeatCapacity, tblAircraft.TotalCapacity, tblRoute.FlightNr, tblRoute.DepCity, tblRoute.ArvCity FROM tblAircraft INNER JOIN tblAircraftRoute ON tblAircraft.AircraftID = tblAircraftRoute.AircraftID INNER JOIN tblAirline ON tblAircraft.AirlineID = tblAirline.AirlineID INNER JOIN tblRoute ON tblAircraftRoute.RouteID = tblRoute.RouteID";

    SqlParameter paramDep = new SqlParameter("@strDepCity", SqlDbType.NVarChar, 50);
    paramDep.Direction = ParameterDirection.Input; // optional as it is the default
    paramDep.Value = strDepCity.Value;
    cmd.Parameters.Add(paramDep);

    SqlParameter paramArr = new SqlParameter("@strArvCity", SqlDbType.NVarChar, 50);
    paramArr.Direction = ParameterDirection.Input; // optional as it is the default
    paramArr.Value = strArvCity.Value;
    cmd.Parameters.Add(paramArr);
    SqlDataReader sqldr = cmd.ExecuteReader();
    SqlContext.Pipe.Send(sqldr);

    sqldr.Close();
    conn.Close();
}
}

Answer: You are not using either of the parameters in the query itself. SQL Server cannot determine how supplied parameters should be used in a query. You need to add the proper WHERE condition to the query and reference those parameters.


Synapse LDAPSend library requires username = Domain\User

Question: I'm using Synapse LDAPSend.pas library to connect and authenticate in an Active Directory server. The code is pretty simple and direct, but the problem is that the username needs "DomainName\Username" or "Username@DomainName" format to bind command works correctly. Are there anyhow to make it work only with Username specified?
var
  ldap: TLDAPsend;
begin
  ldap:= TLDAPsend.Create;
  try
    ldap.TargetHost := '192.168.0.12';
    ldap.UserName:= 'AD\Owner'; //here I need to specify only Owner
    ldap.Password:= 'Password!';
    if ldap.Login then
      if ldap.Bind then
      begin
        //do stuff
        ldap.Logout;
      end;
  finally
    ldap.Free;
  end;
end;

I was thinking maybe something like "*\Owner", or any other command from Active Directory to make it works? Or maybe a Synapse setting to make it.
Comment: You are connecting against an Active Directory, which always has a name. If you know the user name, acquiring the domain name should not be a problem.
Comment: I'm not very familiar with Active Directory. Is possible to get the domain name without being logged? How can I do that?
Comment: Is your program running on the client machine or on a server somewhere on the network? If the program is running on the server: is the server part of the AD domain?
Comment: The program is running on a client machine. And the big problem is: sometimes it can run outside the LAN... I'm going nuts with this! Thinking to pay some freelancer to do this with IndyTCPClient, without the need of domain in user, I know it's possible, but I have no time to study the LDAP protocol, which looks more like a big mess.
Comment: I'm looking the Synapse source code, and looks like it just do the normal TCP operations to connect/authenticate into the server. I can't understand why the DOMAIN is needed with the username on this library. Functions like Win32 AdsOpenObject doesn't need it.
Answer: If the current user is logged on to the domain there is a variety of options to retrieve the domain name. The easiest way is to read the environment variable USERDOMAIN using the function SysUtils.GetEnvironmentVariable. There are also the Windows API functions NetWkstaUserGetInfo available on Windows NT and better as well as GetUserNameEx introduced with Windows 2000 among a lot of others, which are usually harder to use.
If your version of Delphi does not contain declarations for these API functions please use the Jedi Windows API headers.
Comment: Thank you for the answer. Are there anyway to get the domain if the current user is not logged on the domain, and outside the local network? Maybe doing the login/bind anonymously on AD server and retrieve the domain name anonymously?
Comment: Try calling [`DsGetDcName`](https://msdn.microsoft.com/en-us/library/ms675983(v=vs.85).aspx) to retrieve a domain name. A domain controller must be able to service the request.


Understanding security relevant fields of CycloneDX files

Question: Basically, title. I have a program that checks for security relevant changes between two cylconeDX SBOM's, formatted in json.
I've written an application to diff the two sbom's, though, I am noticing that there are fields that change from run-to-run that don't seem to be relevant to actual component change, e.g. sha256 hash changes of a particular component property; e.g.:
<code>"components":[
  {
    "bom-ref":"foo",
    "name": "bar",
    "cpe": "foobarCPE",
    "properties": [
      {
        "name":"syft:someLocation:idx:layerID",
        "value":"sha256:some_sha_here"
      }
    ]
  },
  {
   "bom-ref":"{foobar}",
   ...
  }
]
</code>
If I ignore the <code>[components][properties]</code> field, is there the potential for relevant changes in a component's nature to be missed? And as a tag-along, should I be considering less information here? Based on some light reading, it seems like the purpose of CPE's exist to guarantee a common way to indicate significant changes in baselines, but I am somewhat unclear.
Comment: I'm suggesting for the moment where you don't know if whether or not, the only thing you can sanely do - and IMHO that should be done with the same or higher priority than generating SBOMs - is to have a reproducible build so that content hashes like in your example can be tracked to revisions of the source-code rather than to a situation in the past that is long gone from verification. YMMV.
Comment: @hakre I am having a hard time understanding your comment. Are you suggesting they are important, or not? And what're you intimating with three or four letter acronyms, here?
Comment: And what is a _"tag-along"_? If meaning of https://dictionary.cambridge.org/dictionary/english/tag-along could you elaborate a bit more about the meaning in your scenario outlined in the question?
Answer: Following up on this answer, some months later, a few months wiser.
The crux of the issue, here, was that the sha256 hash for some docker layers were changing. This was indicating that the construction of the layer itself had changed. This should be static, otherwise.
You might think to yourself, "that seems problematic". From a security standpoint, indeed it should. The change in hash indicated that the layers themselves had been altered. Due to how the dependencies of this particular project were constructed, the docker containers of several different repoes were using different version numbers of various dependencies, depending on some corner cases.
So, if you have landed here wondering, "Can I disregard the changes in hash?" It depends. Are you trying to make security assurances? If so, this is probably a show-stopper, until you can figure out how to properly attribute your dependencies. Are you trying to build an SBOM? Then it's probably fine. In our case, the solution was to compile a set of keys, comprised of the dependency's name, version, and source.
If things had changed from run-to-run, there was a report that would state what was added, what was removed, and what had changed. This may be insufficient for your use-case, but it solved our problem. YMMV.


Two-tailed one-sample permutation test

Question: I want to test the significance of the classification accuracy of a cross-validated binary classifier.
I performed a permutation test to determine whether the true classification accuracy is higher than the chance level classification accuracy:

How to perform a two-tailed version of such a test?
Comment: If you're generally interested in how to perform a two-tailed permutation test, that's okay, but for your particular purpose, what's wrong with the regular proportion test? The R function is `prop.test`. Also, what did you permute, the group labels and then run the same classification model (retrained, I guess) to check for accuracy?
Comment: It boils down to how, exactly, you quantify "at least as extreme". With location tests, often deviation from a point-null (absolute difference in means) makes sense but in other cases it may make more sense to argue that you should do other things (doubling the smaller p-value is pretty common; sometimes it makes sense to do just that).
Answer: In many cases the P-value of a 2-sided test is twice the P-value of
a 1-sided test.
For demonstration purposes, here are descriptions of a sample of size $n = 100$ from a Laplace distribution
(centered at 2 and with much heavier tails than normal).
summary(x)
    Min.  1st Qu.   Median     Mean  3rd Qu.     Max.
-13.0858  -0.2901   1.0873   0.8850   1.8196  11.6360
sd(x);  length(x)
[1] 3.122601
[1] 100

boxplot(x, col="skyblue2", pch = 20, horizontal=T)

Here is a permutation test, in R,  of $H_0: \mu = 0$ vs $H_a: \mu \ne 0.$
At each iteration, the sign of each observation is randomly changed between $\pm 1.$ At each iteration, the mean of the sign-permuted data is found.
[That is, the 'metric' of the permutation test is the sample mean.]
The observed mean of the sample is 1.885.
set.seed(1234)
a.obs = mean(x)
a.prm = replicate(10^5, mean(sample(c(-1,1),100,rep=T)*x) )
mean(abs(a.prm) >= abs(a.obs))
[1] 0.00496                     # P-value of two-sided test

hist(a.prm, prob=T, col="skyblue2", xlim=c(-7,7))
abline(v=c(a.obs,-a.obs), col="red")

The P-value of the two-sided test is the area in both tails beyond
the vertical red lines.
For a one-sided test (against $H_a: \mu > 0,$ the P-value would be
the value in the right tail beyond 0.885. R-code 'mean(a.prm >= a.obs)`,
which returns 0.00223.
Notes: (1) In the R code for the P-values, the vector with >= is
a logical vector with $10^5$ TRUEs and FALSEs, and the mean is
the proportion of its TRUEs. (2) Eudey et al. has an elementary presentation of permutation tests; Sect. 2 on paired tests is similar to your one-sample test.
Comment: Thanks! In my case, shuffles correspond to permutations of the two classes' labels I classify. So you would just double the p_value I found with the one-sided test?
Comment: If you do a one-sided test and get P-value 0.01, then the P-value for a corresponding 2-sided test is 0.02. // But you can't apply the 'doubling rule' blindly. If your one-sided test is in the "wrong direction" and the P-value is 0.83, then you obviously wouldn't double _that_ P-value.
Comment: Thanks! In which cases doesn't this (doubling the smallest one-sided p-value to get the two-sided one) hold true?
Comment: Getting a probability > 1 is always a clue something's wrong.
Answer: If you have collected $N$ permutation scores, $s_1,\ldots,s_N$, the two-sided p-value for a treshold, $t$, can be estimated as $p_2(t) = {\rm min}(2\frac{|\{ s_i \geq t\}|}{N}, 2(1-\frac{|\{s_i \geq t\}|}{N}))$. I.e. just divide 2 times the number of permutations with scores greater or equal to your treshold with the total number of permutations. If the resulting value is less than one, then use the value directly, othervise subtract the value from 2.

Two-tailed one-sample permutation test


Slove a complicated equation (probably with LambertW function)

Question: For the following equation:
$$\exp\left(\frac{2 d g}{2 d g x-x+1}\right)= \left(\frac{2 d g x-x+1}{1-x}\right)$$
the solution for $x$ is discussed in: Solving an equation with LambertW function
As an extension, I encountered a problem with following equation in $y$:
$$\exp\left(\frac{a b (g+1) y-a b^2}{g y (a b+y (a+1-ag))}\right)=\left(\frac{a b+y (a+1-ag)}{y}\right)$$
As this is different from the previous one, I am not able to follow same technique. I could only solve this without $gy$ term in the denominator of $\exp(\cdot)$.
Does any one have constructive idea to convert this equation in order to get the solution using the LambertW function form? or else any other alternative method?
Answer: I'm pretty sure that this can't be done using LambertW.  Of course the usual numerical methods are available (given numerical values of the parameters), or if some of the parameters are small you might find series expansions useful.


Show that $P(X_k \le x) = P\{N(x) \ge k\}$

Question: Suppose $x_1 ... x_n$ are the order statistics of an iid sample from a continuous distribution $F(x)$. Show that $P(X_k \le x) = P\{N(x) \ge k\}$ where $N(x)$, the number of sample values less than x, is binomial with parameters $n$ and probability $p = F(x)$.
Given the statement looks (i believe it to be) like this:
$$P(X_k \le x) = \sum_{k=0}^n\binom{n}{k}F(x)^k(1-F(x))^{n-k}$$
However, I do not see how $\sum_{k=0}^n\binom{n}{k}F(x)^k(1-F(x))^{n-k} = P\{N(x) \ge k\}$
Additional data:
Use the Q above to show that the density of $X_k$ is
$$p(x) = \binom{n}{k}F(x)^{k-1}(1-F(x))^{n-k}f(x)$$ where $f(x)$ is the density from $F(x)$. Verify the density using the multinomial argument:
$$n!\epsilon^n\prod_ip_{\theta}(x_i)$$
which is a general heuristic to deal with order statistics from an iid sample from a continuous density $p_{\theta}(x)$.
Comment: The sum should read $\sum_{i=k}^n {n \choose i} F(x)^i (1-F(x))^{n-i}$ which is clearly equal to $P(N(x)\ge k)$.
Comment: In any case, in the continuous case, the alternative derivation in https://stats.stackexchange.com/a/78559/77222 is much more illuminating (differentiating the above sum of binomial point masses is a rather tedious exercise).
Comment: If $X_{(k)}\le x$ then at least $k$ of the variates in the $n$ sample are less than $x$ since $$X_{(1)}\le\cdots\le X_{(k)}$$
Comment: $N(x)$ is the total number of observations $\le x$.  Each observation $X_i \le x$ with probabiity $P(X_i\le x)=F(x)$ and hence (because of indepence), $N(x)\sim \operatorname{bin}(n,F(x))$.  And the event that $X_{(k)}\le x$ is the same event that $N(x)\ge k$.
Comment: @JarleTufto I cannot understand why this is true? For example, $X_k$ takes a completely different distribution.
Comment: There's scarcely anything to show, because the usual definition of $X_{(k)}$ says the event "$X_{(k)}\le x$" *means* $N(x) \ge k.$ Because these describe the same event, they must have the same probability!  Exercise questions like this are indeed interesting and useful for connecting different definitions of something, so it would help us to know what *your* definition of the order statistics is: then we would be able to see what needs to be shown.
Comment: @whuber I have updated the post with additional data - I hesitated when looking at $p(x)$ because of the following: - "f(x) is the density from F(x)", I cannot see how my sum is equal to the one provided; moreso on how to derive it. As for the order statistic; I defined it simply as ordering/sorting through iid variables by their minimum to maximum value.
Answer:
Show that $$P(X_{(k)} \le x) = P(N(x) \ge k)$$

You can proof this by showing that the events on the left side and the right side $$X_{(k)} \le x \qquad \text{and} \qquad N(x) \ge k$$ are the same event.
There is however a tricky detail which is that they are not exactly the same event.
You have on the left side an inequality that is not strict (less than or equal) whereas on the right side you have a strict inequality with the definition of $N(x)$ being 'the number of sample values less than $x$'.
Discrete distribution
For discrete distributions this difference in the events will result in the probabilities not being equal. Take for instance a sample of size one drawn from a Bernoulli distribution. Then $$P(X_{(1)} \leq 1) = 1 \qquad  \text{and} \quad P(N(1) \geq 1) = 1-p$$
Continuous distribution
To still make the proof for continuous probabilities, we could use as starting point one of the following alternative equations instead. These are made by replacing either the $\leq$ sign on the left by a $<$ sign, or the $<$ sign on the right by a $\leq$ sign such that the events are the same.
$$\begin{array}{}P(X_{(k)} < x) &=& P(N(x) \ge k) \\
P(X_{(k)} \le x) &=& P(N^\prime(x) \ge k) \\
\end{array}\\$$
with $N^\prime(x)$ meaning the number of sample values less than or equal to $x$.
For these two expressions, you can easily see that the events are the same because they imply each other (and also the events will be the same for the discrete case). Take for example the second statement:

If there the number of values smaller than or equal to $x$ is larger than $k$ then at least the first $k$ order statistics must be smaller than or equal to $x$ and therefore the $k$-th order statistic must be smaller than or equal to $x$.
If the $k$-th order statistic is smaller than or equal to $x$, then so must be at least the $k-1$ order statistics with a lower order, and therefore there are at least $k$ values smaller than or equal to $x$.

The trick to complete the proof for the continuous distribution is that the probability of the different events (with signs $\leq$ or $<$) are the same. We have
$$P(X_{(k)} \leq x) = P(X_{(k)} < x) \\
P(N(x) \ge k) = P(N^\prime(x) \ge k)$$
The reason is because the probability of the complement of the events is zero. For instance $$\begin{array}{}
P(X_{(k)} \leq x) - P(X_{(k)} < x) &=& P(\lbrace X_{(k)} \leq x \rbrace \setminus \lbrace X_{(k)} < x \rbrace)\\ &=& P(X_{(k)} = x) \\ &=& 0
\end{array}$$
Summarizing
$$\begin{array}{ccc}
\rlap{\overbrace{\phantom{P(X_{(k)} \leq x ) = P(X_{(k)} < x)}}^{\substack{\text{different events} \\ \text{but same probability} \\ \text{for continuous distributions}}}} P(X_{(k)} \leq x ) &=& \underbrace{P(X_{(k)} < x) = P(N(x) \ge k)}_{\text{same events}}\\
&& \overbrace{P(X_{(k)} \le x) = P(N^\prime(x) \ge k)}_{} &=& P(N(x) \ge k \llap{\underbrace{\phantom{P(N^\prime(x) \ge k) = P(N(x) \ge k}}_{\substack{\text{different events} \\ \text{but same probability} \\ \text{for continuous distributions}}}})
\end{array}$$
So we have $P(X_{(k)} \le x) = P(N(x) \ge k)$ not entirely because the events are the same, but because they are events with the same probability, which is because the difference between the events has probability zero for continuous distributions.
Comment: Thanks for the breakdown this makes things clearer; for there $\exists X_k(X_k \le x)$ therefore $N(x)$ represents those values of $x$ greater than them values $k \in X_k$?
Comment: @SextusEmpiricus While not done by the OP, it makes more sense to define $N(x)$ as the number of observations less _or equal_ to $x$ (as is usually done to put it on an equal footing with the definition of the cdf of $X_{(k)}$), in which case the statement of the OP would hold also for discrete random variables.  So I think your answer just adds to the confusion here.
Comment: @JarleTufto I have updated it, but i wonder whether it makes it less confusing. When you define N(x) differently then you do not answer the question directly and you still need to show why the equality for the differently defined N(x) also implies the equality for the N(x) from the original question.


if its already open, show the existing application instead of opening another one

Question: I have an application that minimizes to icon state when user press close. I want to make application show again when user tries to open the app, instead of just opening the app again.
I thought of using settings: A boolean value, true if the app is opened and false if it isn't but that would mean you should set it to false when user closes app and true in the load function. But since close event doesn't run when program crashes, that would make program un-openable (:D?), and i have no idea how to call events between two programs.
TL;DR: Show the app to user instead of opening it twice when user opens program again, just like Discord.
Thanks!
Comment: Typically done either using a `Mutex` and or the `Process` way. Does this answer your question [What is the correct way to create a single-instance WPF application?](https://stackoverflow.com/questions/19147/what-is-the-correct-way-to-create-a-single-instance-wpf-application)? Although tagged with `WPF` it's the same. [Here's another solution](https://stackoverflow.com/questions/34617940/single-instance-of-an-app-in-c-sharp) that may help as well.
Answer: In your Program class add the ShowWindow method from the User32 dll:
const int SW_SHOW = 5;

[System.Runtime.InteropServices.DllImport("user32.dll")]
private static extern bool ShowWindow(IntPtr hwnd, int nCmdShow);

Then write that in your main method before calling the Application.Run method:
var currentProcess = System.Diagnostics.Process.GetCurrentProcess();

// search for another process with the same name
var anotherProcess = System.Diagnostics.Process.GetProcesses().FirstOrDefault(p => p.ProcessName == currentProcess.ProcessName && p.Id != currentProcess.Id);

if (anotherProcess != null)
{
    ShowWindow(anotherProcess.MainWindowHandle, SW_SHOW);
    return; // don't continue to run your application.
}

UPDATE:
The con of that way is that you're not actually identify your program, you're just searching for a process with the same name. so when renaming the exe file you will be able to run an another process of the same program with a different name.
or if there is another program in your PC with the same name you will find it as was your program so you will show it instead of yours.
Comment: where do i add the first code (const int etc.)? In the main method or outside
Comment: inside the Program class, outside the main method.
Comment: This doesn't show the opened program, did i do this right? https://gist.github.com/lim10dev/c144f2cbcfaadc093e9c58665119c06e
Comment: yes. that works for me. is your app minimized and shown in the taskbar?
Comment: no, its in icon state - background work
Comment: I think it should work anyway since it's the main window of your application but I didn't test it. try it with a form that will be shown in the taskbar to check if that makes the difference.
Comment: It works fine when the app is minimized
Comment: How to do it for certmgr.msc? It's running as mmc.exe
Comment: @user3625699 Two ideas that I can think of (but didn't try):
1. Inject code to mmc.exe (totally not recommended and can be updated by the Windows Update).
2. Write a program that gets an executable path and checks whether it's already running, if running, it will show its window, otherwise - it will run the executable.
You can then create a shortcut to that program with mmc.exe file path as argument and run it instead of running certmgr.msc directly.


Interval estimation of $\sigma^2$ with the reliability of $95\%$

Question: I got these values from the measurement by a telescope:
20.1, 20.2, 19.9, 20, 20.5, 20.5, 20, 19.8, 19.9, 20.
I know that the actual distance is 20 km and the error of the measurement is not affected by the systematic error. What is the accuracy of the telescope? [Answer is: ${σ}^2$=0.061]
Now I am supposed to calculate ${σ}^2$ with a reliability of 95% using the interval estimation. So, ${σ}^2$ should be:
 $ \frac{n* s^{2} }{  \chi^{2}_{0.975}   \big(n-1\big)   } ;
 \frac{n* s^{2} }{  \chi^{2}_{0.025}   \big(n-1\big)   } $
After using $n = 10$, $k = n-1 = 9$ and $s^{2} = 0.061$ I get this:
$ \frac{10*0.061}{  \chi^{2}_{0.975}  \big(9\big)  };
 \frac{10*0.061}{  \chi^{2}_{0.025}  \big(9\big)  } $
Is it right?  In the numerator I should use n = 10 or n-1 = 9?
In the denumerator I should use n-1 = 9 or n = 10 ?
When I'm looking for the values in the table of $\chi^2$ quantiles, should I look for $0.95$ and $0.05$ or $0.975$ and $0.025$?  When they say they need the reliability of $95\%$, I assume I will use $1- \frac{ \alpha }{2}$ and $\frac{ \alpha }{2}$. Since my  $\alpha$ is $0.05$, I think $0.975$ and $0.025$ are the right values.
[ Answer is: $σ$ = <0.173 , 0.433> ]
Answer: Short answer:
Use $n$ when you know the actual distance. Use $n-1$ when you don't know the actual distance.
Long answer:
Assume that $X_1,\ldots,X_n$ are independent normal random variables with mean $\mu=20$ and unknown variance $\sigma^2$.
By the scale and location properties of the normal distribution, this means that $Y_i=\frac{X_i-20}{\sigma}$ are standard normal, i.e. $N(0,1)$, random variables.
By the definition of the $\chi^2$ distribution, the sum of $n$ independent squared $N(0,1)$ variables is $\chi^2(n)$. This means that $\sum_{i=1}^n Y_i^2$ is $\chi^2(n)$ distributed.
Your estimator of $\sigma^2$ is $$s^2=\frac{1}{n}\sum_{i=1}^n(X_i-20)^2=\frac{\sigma^2}{n}\sum_{i=1}^n Y_i^2.$$
Thus $\frac{ns^2}{\sigma^2}=\sum_{i=1}^n Y_i$ is $\chi^2(n)$ distributed. Hence, using the definition of quantiles, $$P\Big(\chi^2_{\alpha/2}(n)\leq \frac{ns^2}{\sigma^2} \leq \chi^2_{1-\alpha/2}(n)\Big)=1-\alpha.$$
After some algebra, we find that this also means that $P\Big(\frac{ns^2}{\chi^2_{1-\alpha/2}(n)}\leq\sigma^2 \leq \frac{ns^2}{\chi^2_{\alpha/2}(n)}\Big)=1-\alpha$. Consequently, your $1-\alpha$ confidence interval for $\sigma^2$ is
$$\Big(\frac{ns^2}{\chi^2_{1-\alpha/2}(n)},  \frac{ns^2}{\chi^2_{\alpha/2}(n)}\Big).$$
As pointed out by Procrastinator, you obtain the corresponding interval for $\sigma$ by taking the square roots of the interval limits. This works because $$P\Big(\sqrt{\frac{ns^2}{\chi^2_{1-\alpha/2}(n)}}\leq\sigma \leq \sqrt{\frac{ns^2}{\chi^2_{\alpha/2}(n)}}\Big)=P\Big(\frac{ns^2}{\chi^2_{1-\alpha/2}(n)}\leq\sigma^2 \leq \frac{ns^2}{\chi^2_{\alpha/2}(n)}\Big)=1-\alpha.$$
Plugging in your values, the interval for $\sigma^2$ is $(0.02978054, 0.18786730
)$. Thus the interval for $\sigma$ becomes $(0.1725704, 0.4334366)\approx (0.173,0.433)$.
If $\mu$ had been unknown, you would have used the estimator $\hat{\sigma}^2=\frac{1}{n-1}\sum_{i=1}^n(X_i-\bar{X})^2$ instead. It can be shown that $\hat{\sigma}^2$ is $\chi^2(n-1)$ distributed. Using the same derivation as above, we then find that the confidence interval becomes $\Big(\frac{(n-1)\hat{\sigma}^2}{\chi^2_{1-\alpha/2}(n-1)},  \frac{(n-1)\hat{\sigma}^2}{\chi^2_{\alpha/2}(n-1)}\Big)$ instead.
Comment: I think the important point in MansT's answer is that the chi square distribution used to get the confidence interval depends on the observations having a normal distribution.
Comment: (+1) This is a nice clear answer to the problem, as long as the normality assumption is reasonable.
Comment: +1 Nice answer. Did you get the numerical result mentioned by the OP?
Comment: @Procrastinator: I did, yes. I've added the numerical results to the answer now. Thanks for pointing that out :)
Answer: If your data are not normally distributed you can try bootstrap confidence intervals but my research has shown that you need larges samples to get the proper coverage (these intervals have advertised coverage only asymptotically).  Also although  you say that you know the measurements are unbiased and the actual mean parameter is 20 you can compute the sample estimate of bias and test to see if it is different from 0.
Comment: +1 because everything in OP's reasoning is only valid when the data is normally distributed