Upgrading weight parameters to random variable in Gaussian mixtures

Question: In a Gaussian mixture model we model a density like:
$p(\mathbf{x}|\pi,\mu,\sigma)=\sum \pi_i N(\mathbf{x}|\mu_i,\sigma_i)$ [1]
where $\pi,\mu$ and $\sigma$ are parameters.
I would like to know if the following model is of any use / has a name.
Let's suppose that $\mathbf{\pi} \sim Dirichlet(\mathbf{\alpha})$, so that the weights of the Gaussian become a random variable. Than instead of [1] we have:
$p(\mathbf{x}|\alpha,\mu,\sigma)=\int d\mathbf{\pi}  p(\mathbf{\pi} |\alpha) \sum \pi_i N(\mathbf{x}|\mu_i,\sigma_i)$ [2]
Does it make any sense to use [2] instead of [1] in some setting ? Does this operation make any sense in some setting ?
Comment: Have a look at latent Dirichlet allocation, it's similar.
Comment: Actually I was indeed studying LDA. I read this sentence on the article "It is important to distinguish LDA from a simple Dirichlet-multinomial clustering model. A classical clustering model would involve a two-level model in which a Dirichlet is sampled once for a corpus, a multinomial clustering variable is selected once for each document in the corpus, and a set of words are selected for the document conditional on the cluster variable." and I was trying to understand what they meant in this sentence...
Comment: When setting a distribution like the Dirichlet distribution on the $\pi_i$'s, the goal is to facilitate the inference on the $\pi_i$'s, not to make them disappear. (This is the principle beyond Bayesian inference.)
Comment: Maybe I should rewrite my question explaining what I am trying to understand...
Comment: @Xi'an thanks I will start first to understand your comment and than relate to the LDA sentence I wanted to understand. Could you please check my answer below ? If it is totally wrong I will remove it, otherwise please feel free to edit and correct. I would like to check my understanding...
Answer: Trying to answer Xi'an comment, to check if I am understanding his comment. As far as I am understanding he means that upgrading $\pi$ to random variable would have the meaning of modelling our uncertainty of $\pi$ but not the sampling process of the data points. I try to write this better for exercise.
In the standard setting, when we fit a Gaussian mixtures the $\pi$ are parameters. Let's us suppose $\mu$ and $\sigma$ fixed here and suppose we have $M$ samples. In the usual setting the EM algorithm finds the maximum likelyhood estimate:
$\pi_{MLE}=argmax_{\pi} \ log L(\mathbf{x}|\pi)$
Now instead of treating $\pi$ as a parameter we can upgrade it to a random variable. In order to do this we need to define its probability distribution and relation to $x$. In mathematical terms we need:

$P(\mathbf{x}|\pi)$ , that we have ;

A prior  $P(\pi)$ ;

In order to do this we introduce a parameter $\alpha$ that defines the prior as a Dirchlet distribution and arrive at the graphical model:

which fully defines the joint $P(\mathbf{x},\pi|\alpha)$.
So what did we lose/gain with respect to MLE ?

Still all values of $\pi$ are possible/taken into account. If we condition on the value of  $\pi$ we have $P(\mathbf{x}|\pi,\alpha)=P(\mathbf{x}|\pi)$. This is exactly the parametric likelyhood we had before ;

Still we can do inference of the value of $\pi$ by estimating the posterior $P(\pi|\mathbf{x},\alpha)$ ;

As a drawback, we introduced a parameter $\alpha$ which was not present before. Maybe  it would make any sense to estimate $\alpha$ by maximizing the marginal likelyhood:

$$S(\alpha)=\int d\pi P(\pi|\alpha)P(\mathbf{x}|\pi)$$
even if this approach seems to suppose that the uncertainty in $\pi$ is related to the sampling process of $\mathbf{x}$. I have the impression that this way we would mix "uncertainties" and "sampling probabilites", but maybe these concepts are so interrelated that can be mixed together.
Actually this approach ( maximum marginal likelyhood ) in some sense is making the $\pi$ variables disappear the way described in the original message (weird...)
Comment: This reflection upon my comment is essentially sound. It is correct that the inference will depend on the choice of the prior (hyper)parameter $\alpha$, that the MAP will differ from the MLE, and that $\alpha$ could be estimated from the marginal likelihood: the later technique is called empirical Bayes estimation and was proposed by Robbins in the 1950's. A description can be found in some Bayesian textbooks like Carlin and Louis (1992) or mine.
Comment: Great thanks for your feedback and suggestions


PHPWord - table style defaulting always to Arial 10, new style not being applied

Question: I tried another post here re Title styles not being applied, but that code did nothing, either. Here, my PHPWord builds sections that display hard-coded vars using "Open Sans" successfully; but when I change to displaying info in a Table it always defaults to "Arial" 10. Please see if you can help tell why this is happening, thank you! Note - to get output to show lines evenly from cell to cell i am trying other styles within the text. Even when I apply $BodyFontStyle to each addCell, addText, and addRow, still - Arial only. I need the table to display in "Open Sans" 10. Help, please -
<code>$firstRowStyle = array('name' => 'Open Sans', 'size' => 10, 'valign' => 'bottom');
$BodyFontStyle = array('name'=>'Open Sans', 'bold'=>false, 'size'=>14);
$table_cell_style = array('valign' => 'top');
$lineStyle = array('name' => 'Open Sans', 'size' => 10, 'valign' => 'bottom');

//from above, this flag is not set, or set:
if ($signatory_found == 1) { //from mySQL query looking for Signatory.

    $PHPWord->addTableStyle('BodyFontStyle', array('name'=>'Open Sans', 'bold'=>false, 'size'=>14));
    $table = $section->addTable('BodyFontStyle');

    //tried this, to no avail:
    //$PHPWord->addTableStyle($BodyFontStyle);
    //$table = $section->addTable($BodyFontStyle);

    for ($i=0; $i <= $counter; $i++) {
        $table->addRow();
        $table->addCell(1600, $firstRowStyle)->addText("BY:_____________________________________________________");
        $table->addCell(200, $firstRowStyle)->addText('');
        $table->addCell(1600, $firstRowStyle)->addText("     ____________________________");
        $table->addRow();

        $full_name_title = "    " . $signatory_results[$i]['first'] . " " . $signatory_results[$i]['last'] . ", " . $signatory_results[$i]['title'];

        $table->addCell(1600, $table_cell_style)->addText("{$full_name_title}");
        $table->addCell(200, $firstRowStyle)->addText(''); //space between name, and date.
        $table->addCell(1600, $table_cell_style)->addText("                          Date");
    }
} else { //else signatory flag was not set to 1, so no records found.

    $table = $section->addTable('BodyFontStyle');
    $PHPWord->addTableStyle('BodyFontStyle', array('name'=>'Open Sans', 'bold'=>false, 'size'=>14));

    $table->addRow();
    $table->addCell(1600, $firstRowStyle)->addText("BY:_________________________________");
    $table->addCell(200, $firstRowStyle)->addText('');
    $table->addCell(1600, $firstRowStyle)->addText("_______________________");

    $table->addRow();
    $table->addCell(1600, $firstRowStyle)->addText("          Signature");
    $table->addCell(200, $firstRowStyle)->addText('');
    $table->addCell(1600, $firstRowStyle)->addText("                Date");

    $table->addRow();
    $table->addCell(1600, $lineStyle)->addText("     ____________________________________", array(), array('cellMarginLeft' => 6));
    $table->addCell(200, $firstRowStyle)->addText('');
    $table->addCell(1600, $firstRowStyle)->addText("_______________________");

    $table->addRow();
    $table->addCell(1600, $table_cell_style)->addText("          Print Name");
    $table->addCell(200, $firstRowStyle)->addText('');
    $table->addCell(1600, $table_cell_style)->addText("                Print Title");
}

// Hack - Martin/Pitt has a / in the LP Short name which messes up the filename/path
$LPShort = str_replace("/", "-", $LPShort);
$LPShort = str_replace(" ", "-", $LPShort);

$objWriter = \PhpOffice\PhpWord\IOFactory::createWriter( $PHPWord, 'Word2007' );
$filelocation = '/var/www/company/contracts/tmp';
$filename = 'Sig_'.$LPShort."_".date('YmdHis',time()).'.docx';
$FullName = $filelocation.'/'.$filename;
//echo $FullName;
$objWriter->save($FullName);

echo $filename;
exit;
</code>
Answer: THE FIX - i won't post ALL the code again, but this command will show the solution. The key is to put the defined style inline with the addCell command, at the end after the text. Note that, above, i simply had the addCell with the text, but no style after it. i will post a before and after command to explain:
<code>/*BEFORE* $table->addCell(1600, $table_cell_style)->addText("{$full_name_title}");

/*AFTER*/ $table->addCell(1600, $table_cell_style)->addText("{$full_name_title}", $BodyFontStyle);
</code>


circle envelope tangent in another circle

Question: As the picture shows,

One big circle ,$(0,0)$ ,radius=R, there is a small circle in it, $(m,0)$ ,radius=r .
G is on the big circle. From G ,we can do two tangent lines about the small circle. Get the points of intersection E and F. line EF has a envelope about G , which seem like a circle.
How to prove it? Since calculating it requires much effort.
Some additional infomation:
By picking special points,I get
the radius of envelope circle is
$$\frac{R \left(m^4-2 m^2 \left(r^2+R^2\right)-2 r^2 R^2+R^4\right)}{\left(m^2-R^2\right)^2}$$
and the circle center
$$\left(\frac{1}{2} \left(\frac{R \left(-m^2+2 m R+2 r^2-R^2\right)}{(R-m)^2}-\frac{R \left(-m^2-2 m R+2 r^2-R^2\right)}{(m+R)^2}\right),0\right)$$
Comment: Your formulas can be simplified : for the radius into $R \left(1-2r^2 \dfrac{m^2+R^2}{(R^2-m^2)^2}\right)$ and even more for the abscissa of the center which can be written : $4m\dfrac{R^2r^2}{(R^2-m^2)^2}$
Comment: @JeanMarie the origin of this problem is that: prove the envelope is a circle as well as use ruler-and-compass construction to make the tangent point of EF and circle.
Comment: From the references in the link from @brainjam: This question is very similar to the one Poncelet [started with](https://gallica.bnf.fr/ark:/12148/bpt6k96672602/f338.item.r=336)
Comment: I have been working a lot on your issue without real success. Could we share our advances ? Moreover, could you say something about the origin of this problem ?
Answer: Here is an analytic geometry solution :
We can assume without loss of generality that $R=1$, i.e., we work inside the unit circle. Therefore, we have the following condition : $0 < m <1$.
Let us introduce some notations.
Let $B$ be the second intersection of line $GC$ with the unit circle.
Let $a$ be the polar angle of point $G$ (i.e., oriented angle between positive $x$-axis and $AG$).
Let $b$ be the polar angle of $B$ (i.e., oriented angle between positive $x$-axis and $AB$).
Let $\theta$ be the angle of line $GE$ with line $GB$ (equal to the angle between $GB$ and $GF$).
We will use a certain number of trigonometric formulas. An extensive and well structured list of those can be found here.

Fig. 1: $G(\cos a,\sin a)$, $B(\cos b,\sin b)$, and $C(m,0)$, center of the small circle with radius $r$. The circle to which lines $EF$ are all assumed to be tangent is in red.
Let us look for relationships between angles $a,b$ and $\theta$ and lengths $m$ and $r$.

a) The fact that $GE$ and $GF$ are tangent to the small circle is expressed by the following relationship :

$$\sin \theta = \dfrac{r}{GC}$$
which is equivalent to :
$$(\sin \theta)^2 = \dfrac{r^2}{GC^2}= \dfrac{r^2}{(\cos a - m)^2+(\sin a - 0)^2}=\dfrac{r^2}{1 - 2m \cos a +m^2}\tag{1}$$

b) The angle between $AE$ and $AB$ is $2 \theta$ by central angle theorem. The same for the angle between $AB$ and $AF$. Let $I$ be the intersection point of line $AB$ and line $EF$ ; triangle $EAF$ being isosceles, line segment $AI$ is orthogonal to $EF$, implying that its algebraic measure ("signed distance") is $AI=\cos(2 \theta)$ (possibly negative).

Therefore, straight line $EF$ whose normal vector is $\vec{AB} = \binom{\cos b}{\sin b}$ has equation :
$$x \cos b + y \sin b = \cos 2 \theta \tag{2}$$
Using relationship $\cos 2 \theta=1-2 \sin^2 \theta$ with formula (1) :
$$x \cos b + y \sin b =  1-\dfrac{2 r^2}{1 - 2m \cos a +m^2}\tag{3}$$

c) Let us, finally, express that $G, C$ and $B$ are aligned. This relationship, written under the form (see http://mathworld.wolfram.com/Collinear.html) :

$$\begin{vmatrix}m &\cos a &\cos b\\0 & \sin a&\sin b\\1&1&1\end{vmatrix}=0\tag{4}$$
i.e.,
$$m=\dfrac{\sin(a-b)}{\sin a - \sin b}$$
$$\iff \ \ m=\dfrac{\color{red}{2 \sin(\tfrac12(b-a))}\cos(\tfrac12(b-a))}{\color{red}{2 \sin(\tfrac12(b-a))}\cos(\tfrac12(b+a))}=\dfrac{1+\tan(a/2) \ \tan(b/2)}{1-\tan(a/2) \ \tan(b/2)}$$
yielding the rather unexpected following condition :
$$\tan(a/2) \ \tan(b/2)=k \ \ \text{where} \ \ k:=\dfrac{m-1}{m+1}\tag{5}$$
from which we deduce (using a "tangent half-angle formula" and setting :
$$t:=\tan(b/2)$$
that :
$$\cos a=\dfrac{1-\tan(a/2)^2}{1+\tan(a/2)^2}=\dfrac{1-\left(\tfrac{k}{t}\right)^2}{1+\left(\tfrac{k}{t}\right)^2}=\dfrac{t^2-k^2}{t^2+k^2}\tag{6}$$
Plugging (6) into formula (3), we get for the RHS of (3) after some algebraic transformations :
$$\cos 2 \theta=1-\dfrac{2r^2(t^2(m+1)^2+(m-1)^2)}{(m^2-1)^2(1+t^2)}$$
Using once again tangent half-angle formulas, we can now express the equation of straight line $EF$ under the following parametric form in variable $t$ :
$$x \dfrac{1-t^2}{1+t^2} + y \dfrac{2t}{1+t^2} - 1+\dfrac{2r^2(t^2(m+1)^2+(m-1)^2)}{(m^2-1)^2(1+t^2)}=0.\tag{7}$$
It remains to establish that the distance $d$ of the point with coordinates
$$C'(x_0,y_0)=\left(4m\dfrac{r^2}{(1-m^2)^2},0\right)\tag{8}$$
(would-be center of envelope circle) to straight line $EF$ is constant (i.e., is independent from $t$).
This distance, obtained (see http://mathworld.wolfram.com/Point-LineDistance2-Dimensional.html) by replacing $(x,y)$ in the Left Hand Side of equation (7) by the coordinates $(x_0,y_0)$ of $C'$ given by (8)
$$d=(2r^2(1+m^2) - (m^2 - 1)^2)/(m^2 - 1)^2\tag{9}$$
is indeed independent from parameter $t$ ; moreover it is (up to its sign) the awaited expression (I have used a Computer Algebra  System to obtain this expression of $d$).
Remarks :
1) It is possible to characterize the tangent point of $EF$ on the (red) envelope circle, following the method outlined by @Jan-Magnus Økland in comments following this answer.
2) About a possible more (synthetic) geometry solution. I just discovered in this paper dealing with "exponential pencils of conics" that there is so-called "conjugate conic" of the first circle vs. the second circle (see Theorem 2.5 page 4). But I am not sure it will permit a new solution.
Comment: In maxima CAS with the $r$ you dropped in (7) back in `l:x*((1-t^2)/(1+t^2))+y*(2*t/(1+t^2))+(((2*m^2+4*m+2)*r^2-m^4+2*m^2-1)*t^2+(2*m^2-4*m+2)*r^2-m^4+2*m^2-1)/((m^4-2*m^2+1)*t^2+m^4-2*m^2+1);`

Then the envelope is given by `solve([l,diff(l,t)],[x,y]);`

`[[x=(((2*m^2+4*m+2)*r^2-m^4+2*m^2-1)*t^2+((-2*m^2)+4*m-2)*r^2+m^4-2*m^2+1)/((m^4-2*m^2+1)*t^2+m^4-2*m^2+1),y=-(((4*m^2+4)*r^2-2*m^4+4*m^2-2)*t)/((m^4-2*m^2+1)*t^2+m^4-2*m^2+1)]]`
Comment: Then the implicitization in [M2](http://habanero.math.cornell.edu:3690/):
`R=QQ[m,r]`
`S=R[s,t,x,y,z]`
`I=ideal(x-(((2*m^2+4*m+2)*r^2-m^4+2*m^2-1)*t^2+(((-2*m^2)+4*m-2)*r^2+m^4-2*m^2+1)*s^2),y+(((4*m^2+4)*r^2-2*m^4+4*m^2-2)*s*t),z-((m^4-2*m^2+1)*t^2+(m^4-2*m^2+1)*s^2))`
`gens gb I -- (4*r^4+((-4*m^2)-4)*r^2+m^4-2*m^2+1)*z^2
         +8*m*r^2*x*z+((-m^4)+2*m^2-1)*y^2+((-m^4)+2*m^2-1)*x^2`
Comment: $z$ is just there to eliminate the fractions. Setting $z=1$ gives the correct answer.
Comment: @JeanMarie: I don't (yet?) have a better solution, synthetic or otherwise, but I have an possibly-useful observation: let $e$ and $f$ be the distances from $E$ and $F$ to the points where $\overline{GE}$ and $\overline{GF}$ touch $\bigcirc C$, and let $e'$ and $f'$ be the distances from $E$ and $F$ to the point where $\overline{EF}$ touches the target circle. Then $$\frac{e}{f}=\frac{e'}{f'}$$
Comment: @JeanMarie I don't think that the "conjugate conic" approach will be fruitful.  That's because the red circle is coaxal with the big and small circles, so it is a linear combination of the two (i.e. its equation is a weighted sum of the equations of the other 2). See more context in https://math.stackexchange.com/a/3545910/1257. If the problem is generalized projectively the red conic will be in the pencil generated by the first two conics.
Comment: At last, I have found an analytical-geometry proof, after a lot of attempts... before spotting the good parameters
Comment: @Blue Have you seen this problem ? I have finally found a proof, but a long one. Knowing your large "know how" in geometry, would you have tracks for short-cutting some parts of this proof, or have other ideas, maybe on the side of "pure" geometry ?
Comment: @Jan-Magnus Økland : Thanks for your interest in this issue. Comment on you **first comment:** I fully agree : I had attempted this way with a failure (maybe because the CAS I use is linked to Matlab, therefore less powerful than Maxima) : forming and "solving" the system with the expression and its derivative with respect to the parameter for getting the envelope's equation; The expressions you obtain (x=A/C and y=B/C) with A,B,C polynomials with degree at most 2  already prove that we have a conic. In your **second comment,** I don't understand what $z$ is for.
Comment: @Jan-Magnus Økland I see. In this way, we get indeed the equation of a circle centered on the $x$ axis. Thanks !
Comment: @Blue Very interesting.
Comment: @JeanMarie: Also (perhaps unsurprisingly), the target circle is [coaxal](http://mathworld.wolfram.com/CoaxalCircles.html) with the first two.
Comment: @Blue This is interesting because sharing a same radical line means that certain metric relationships hold.
Comment: @Blue I realized that this property can be extended to 3D : spheres instead of circles , tangent cone instead of tangent lines, intersecting the sphere along a circle which defines a plane and a "red" sphere generated as envelope of these planes...
Comment: @JeanMarie: Does an off-center cone meet a sphere in a circle?
Comment: @Blue Not so sure, in fact... Interesting issue...
Comment: @Blue There is another circle that hopefuly has a role to play : the locus of mid-points of line segment $GC$ is a circle (which can equivalently be defined as the locus of midpoints of line segments  $BC$). Property and proof found [here](https://manasataramgini.wordpress.com/2018/07/06/the-hearts-and-the-intrinsic-cassinian-curve-of-an-ellipse/)
Answer: The expository paper Poncelet's theorem by András Hraskó very nicely treats this problem and its relation to Poncelet's Closure Theorem.
In the OP diagram call the big and little circles $e$ and $a$ respectively, and the red circle $c$. (this corresponds to the labels in the paper, see Figures 1,4,5).  Poncelet's Theorem is concerned with scenarios such as Figures 1 and 4, where a polygon is inscribed in $e$ such that the edges touch $a$.  But the paper speculates that Poncelet studied that situation of an inscribed triangle where one of the sides does not touch $a$, and found that the non-touching side generates the envelope of a circle $c$ (Figure 5).  As @Blue has noted in the comments, $c$ is coaxal with the pair $a$ and $e$.
I leave the details to the paper, but can't resist quoting this:

Poncelet's General Theorem: Let $e$ be a circle of a non-intersecting pencil and let $a_1,a_2,\ldots,a_n$ be (not necessarily
  different) oriented circles in the interior of $e$ that belong to the
  same pencil. Starting at an arbitrary point $A_0$ of the circle $e$, the
  points $A_1,A_2,\ldots,A_n$ are constructed on the same circle, such that the
  lines $A_0A_1, A_1A_2, \ldots, A_{n-1}A_n$ touch the circles $a_1,a_2,\ldots, a_n$,
  respectively, in the appropriate direction. It may happen that at the
  end of the construction, we get back to the starting point, that is,
  $A_n=A_0$. The theorem states that in that case, we will always get back
  to the starting point in the $n$-th step, whichever point of $e$ we start
  from. We do not even need to take care to draw the tangents to the
  circles in a fixed order.

The case of $a_1=a_2=\cdots=a_n=a$ gives the classic Poncelet's Closure Theorem.
The case $a_1=a_3=a, a_2=c$ relates to the OP.
Comment: For more on Poncelet's General Theorem, see _Poncelet's_ _porism_: _a_ _long_ _story_ _of_ _renewed_ _discoveries_, [I](https://link.springer.com/article/10.1007%2Fs00407-015-0163-y) and [II](https://link.springer.com/article/10.1007%2Fs00407-015-0164-x) by Andrea Del Centina
Comment: @Jan-MagnusØkland, thanks for this reference.  It also suggests an answer to a projectively dual problem.  See https://math.stackexchange.com/a/3560237/1257


Interpreting of difference between R lm() with intercept and without it?

Question: I have applied the lm() to a data set. The independent variables are categorical.
First, I use lm() with intercept and I got the next results:
> model <- lm(y ~ factor(x))
> summary(model)

Call:
lm(formula = y ~ factor(x))

Residuals:
    Min      1Q  Median      3Q     Max
-5.3085 -1.8132 -0.4136  1.4323 11.2480

Coefficients:
              Estimate Std. Error t value Pr(>|t|)
(Intercept)     9.3085     0.4064  22.907   <2e-16 ***
factor(x)0.75   0.1435     0.6896   0.208    0.836
factor(x)1.5    0.9062     0.6272   1.445    0.151
factor(x)3      0.9040     0.6989   1.293    0.198
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 2.786 on 126 degrees of freedom
Multiple R-squared: 0.0238, Adjusted R-squared: 0.0005601
F-statistic: 1.024 on 3 and 126 DF,  p-value: 0.3844

In the second model I don't use intercept:
> model.1 <- lm(y ~ factor(x) - 1)
> summary(model.1)

Call:
lm(formula = y ~ factor(x) - 1)

Residuals:
    Min      1Q  Median      3Q     Max
-5.3085 -1.8132 -0.4136  1.4323 11.2480

Coefficients:
              Estimate Std. Error t value Pr(>|t|)
factor(x)0.25   9.3085     0.4064   22.91   <2e-16 ***
factor(x)0.75   9.4520     0.5572   16.96   <2e-16 ***
factor(x)1.5   10.2147     0.4778   21.38   <2e-16 ***
factor(x)3     10.2125     0.5687   17.96   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 2.786 on 126 degrees of freedom
Multiple R-squared: 0.9267, Adjusted R-squared: 0.9243
F-statistic:   398 on 4 and 126 DF,  p-value: < 2.2e-16

If I don't understand the difference between the R-squared value of them? Could I accept the second one as a fit model?
Would somebody help me to understand this problem?
Comment: see [R FAQ 7.41](http://cran.r-project.org/doc/FAQ/R-FAQ.html#Why-does-summary_0028_0029-report-strange-results-for-the-R_005e2-estimate-when-I-fit-a-linear-model-with-no-intercept_003f), also FAQ from [UCLA stats page](http://www.ats.ucla.edu/stat/mult_pkg/faq/general/noconstant.htm)
Comment: Read http://stat.ethz.ch/R-manual/R-patched/library/stats/html/summary.lm.html
Answer: It's a well-known problem. However, you could use
R_squared <- (corr(y, fitted(model.1)))^2

to get a comparable $R^2$. In fact, $\text{corr}^2(y,\hat{y})$ is equivalent to the R squared printed by summary when there is an intercept (see M. Verbeek, §2.4).


Transform target/label variable into classes but classes are data-dependent: How to approach this correctly?

Question: I was redirected from StackOverflow because my question is more about theory.
I have a usual set-up with a pandas dataframe with some features and a numeric target variable (financial returns for example). Now I want to make a classification problem out of it: Rather than predicting the numerical value of the return, I want to predict classes. My question deals now about how I do correctly create the classes of the target variable if the creation of the classes are dependent on the data. For example I want to make 4 classes (1,...4) for the target variable based on the quartiles of the target variable. But my believe is, that when I have the full data set, I cannot calculate the quantile on the whole target variable and then make a train/test split afterwards and do a CV on the train set. Because then the calculated quantile values to create the classes are based on the test data as well. So my question is, how can I approach such a task in a sklearn framework? I saw that there exists the class  TransformedTargetRegressor which goes into this direction: One could possibly use this together with KBinsDiscretizer for transforming the target variable. But a problem I see there is that it always backtransforms then the classes into numerical values when using .predict etc. but I want to do a classification not predicting numerical values.
Or: Would it be allowed to estimate the quartiles on the whole dataset and then classify the all target observations based on this? - But I would have data leakage problem there right?
Happy for any help.
Comment: [Discretizing a continuous variable throws away a lot of information](https://stats.stackexchange.com/q/68834/1352), and I have never seen a case where it was beneficial. Why do you want to do this?
Comment: I think you should do it in the same way you would use any other preprocessing transformer (e.g. StandardScaler). 1. Calculate the quartiles using the training set, and 2. Transform your target variable from continuous to categorical on both the training and test sets using the quartiles found in the previous step.
Comment: @alphaH If there aren't any off-the-shelf transformers in scikit-learn that do exactly what you want, you can always use a [custom transformer](https://www.section.io/engineering-education/custom-transformer/).
Comment: Please add a link to your previous answer in StackOverflow.
Comment: This was the link: https://stackoverflow.com/questions/69142504/transform-target-label-variable-into-classes-but-classes-are-data-dependent-how
Comment: @ Adrià: Thanks. I think that is correct, but say I want to do a cross-validation which is implemented directly in sklearn. When I do it for preprocessing features, I create a pipline and then feed in this pipeline into cross_validate() or GridSearchCV(). But is there not also a method such that I can use those routine, otherwise it gets very difficult?
Comment: @ Stephan: Seems to be common, see e.g. article "Forecasting multinomial stock returns using machine learningmethods"
Comment: @ Adrià: Yes that's true, but the thing is KBinsDiscretizer does this. But pipeline always transforms only the features and not the labels; but using pipeline is important for cross-validation etc. So I need to know on how I can implement it but for the target not on the features.
Comment: @AdriàLuz Your comment "I think you should do it in the same way you would use any other preprocessing transformer..." seems like a perfectly good start for an answer. Remember, stats.SE isn't a site where asking for code is on-topic, so you only need to answer the statistical portion of this question ("How to approach this correctly?")
Answer: As per @Sycorax's suggestion, I'm expanding my first comment as an answer...

I think you should do it in the same way you would use any other preprocessing transformer. That is:

Calculate the quartiles using the training set
Transform your target variable from continuous to categorical on both the training and test sets using the quartiles found in the previous step

I'll illustrate this with an example. For simplicity, let's imagine you only had 10 observations. This is how your target variable might look like:

Next, you randomly split your dataset into train (70%) and test (30%).
Train:

Test:

(I know this all looks a bit ridiculous with such a small number of observations but the main idea is the important bit).
Now, you calculate the quartiles from the training set. These are:
$$q_1=-0.040 \\ q_2=0.100 \\ q_3=0.145$$
Using this information, now you proceed to transform your target variable on both the training and test sets using the following logic:
$$Y^* = \begin{cases}
        1 & \text{if } Y\leq q_1\\
        2 & \text{if } q_1 < Y\leq q_2\\
        3 & \text{if } q_2 < Y\leq q_3\\
        4 & \text{if } Y > q_3
\end{cases}$$
Train:

Test:

Now you can train a model using $Y^*$ as your target variable.
Comment: I'm not sure why the tables are not showing properly (they looked fine in the preview as I was writing the answer!)
Comment: Thank you for taking the time. Makes sense. I guess for the answer about implemented sklearn methods I need to change forum.
Comment: @AdriàLuz (+1) Thanks for writing this up! I was also puzzled by the problem with tables -- they're fine in the edit view, but broken when you display the page. It must be some bug.
Comment: Oh I see! I will replace them with pictures then.


Como saber se a posição do scroll está em cima de uma section?

Question: Não sei se ficou claro, mas vou exemplificar, Eu tenho varias sections no meu site, e quando der scroll eu queria realizar alguns efeitos com Javascript, como eu faço para para saber quando exatamente o usuário estará visualizando a div, como no Scroll Spy do Boostrap? Eu tentei o pageYoffset, mas as medidas variam de acordo com a resolução.
Answer: você pode usar o getBoundingClientRect par apegar a posição atual de um elemento na tela.

//todas as paginas ocupam toda a tela.
var pages = [].slice.call(document.querySelectorAll('.page'), 0);
var checkCurrentPage = function () {
  // obtendo posição das paginas.
  var tops = pages.map(function (page) {
    var position = page.getBoundingClientRect();
    return position.top;
  });

  var indexUnique = tops.indexOf(0);
  if (indexUnique != -1) {
    // unica pagina sendo exibida;
    console.log([pages[indexUnique]]);
    return;
  } else {
    // obtendo as paginas que começam acima da posição atual.
    var prevs = tops.filter(function (top) {
      return top < 0;
    });

    var prev = pages[prevs.length - 1];
    var next = pages[prevs.length];
    console.log([ prev, next ]);
  }
}

window.addEventListener("scroll", checkCurrentPage);
window.addEventListener("resize", checkCurrentPage);
html, body {
  position: relative;
  width: 100%;
  height: 100%;
  padding: 0px;
  margin: 0px;
}

.page {
  position: relative;
  width: 100%;
  height: 100%;
}

.page1 {
  background-color: #009688
}

.page2 {
  background-color: #FF9800
}

.page3 {
  background-color: #673AB7
}

se precisar saber se um determinado elemento está dentro do viewport atual (mesmo que apenas parcialmente), então você deve comparar o retorno do getBoundingClientRect com o window.innerWidth e window.innerHeight.
var position = page.getBoundingClientRect();
var vpWidth = window.innerWidth;
var vpHeight = window.innerHeight;

var visibleY = (position.top <= 0 && position.bottom > 0) || (position.top > 0 && position.top < vpHeight);
var visibleX = (position.left <= 0 && position.right > 0) || (position.left > 0 && position.left < vpWidth);

return visibleY && visibleX;

Se o elemento precisar está completamente visível dentro da tela.:
var position = page.getBoundingClientRect();
var vpWidth = window.innerWidth;
var vpHeight = window.innerHeight;

var visibleY = position.top >= 0 && position.bottom <= vpHeight;
var visibleX = position.left >= 0 && position.right <= vpWidth;

return visibleY && visibleX;

Answer: Basta verificar se o scrollTop() é maior ou igual que o position().top da seção que tu quer verificar, o scrollTop() retorna a posição vertical atual da barra de rolagem, o position().top a posição do elemento em relação ao seu pai.
*Nota que no exemplo eu verifico se a posição do scroll esta entre o início da seção de interesse e o início da próxima seção, ou seja, garantindo que está sobre a seção correta.
*Note também que nesse contexto o offset().top também funcionaria, pois o elemento pai das section's é o próprio document.
Segue um snippet de exemplo:

$(document).on('scroll', function() {
  console.clear();
  if ($(this).scrollTop() >= $('#section1').position().top && $(this).scrollTop() <= $('#section2').position().top) {
    console.log("secao1");
  }
  if ($(this).scrollTop() >= $('#section2').position().top && $(this).scrollTop() <= $('#section3').position().top) {
    console.log("secao2");
  }
  if ($(this).scrollTop() >= $('#section3').position().top) {
    console.log("secao3");
  }
})
section {
  height: 500px;
  border: 1px solid red;
}


cors failing for subdomains

Question: CORS not working properly for domain and subdomain.
I've a single NodeJS server hosted at https://api.example.com
I've two ReactJS Clients
Client 1. https://example.com
Client 2. https://subdomain.example.com
I've configured Client 1 to force www so that a single origin is used for Client 1.
When I open Clien1 that works fine.
When I open Client2 I get the error that

Access to XMLHttpRequest at 'https://api.example.com/someAPI' from origin 'https://subdomain.example.com' has been blocked by CORS policy: The 'Access-Control-Allow-Origin' header has a value 'https://www.example.com' that is not equal to the supplied origin

When I press Ctrl+F5 then it works fine but after that If I refresh Client1 then the error comes at Client1

Access to XMLHttpRequest at 'https://api.example.com/someAPI' from origin 'https://www.example.com' has been blocked by CORS policy: The 'Access-Control-Allow-Origin' header has a value 'https://subdomain.example.com' that is not equal to the supplied origin.

Now When I press Ctrl+F5 at Client1 then it works fine but the same error goes to the Client2.
My nodeJS server is configured as follows
var whitelist = ['https://www.example.com', 'https://subdomain.example.com'];
    getCorsCoptions(req, callback) {
    var getOriginValue = () => {
      if (whitelist.indexOf(req.header('origin')) !== -1) {
        return true;
      } else {
        log.error(__filename, 'Not allowed by CORS', origin);
        return false;
      }
    }
    var corsOptions = {
      origin: getOriginValue(),
      methods: ['GET', 'POST'],
      credentials: true,
      preflightContinue: false,,
      allowedHeaders:["Content-Type","X-Requested-With","X-HTTP-Method-Override","Accept"]
    }
    callback(null, corsOptions)
  }

app.use('*', cors(this.getCorsCoptions));
app.options('*', cors(this.getCorsCoptions));

I'm using axios at React Client sides as follows
get(url: string) {
    return Axios.get(url, {
      withCredentials: true
    }).then((res: any) => {
      if (res) {
        return res.data;
      }
      return {};
    });
}

post(url: string, data: any) {
    let requestHeaders = { 'Content-Type': 'application/json' };
    return Axios.post(url, data, {
      headers: requestHeaders
      , withCredentials: true
    }).then((res: any) => {
      if (res) {
        return res.data;
      }
      return {};
    });
}

Comment: Are you using `express` on the server?
Comment: yes, using express on server
Comment: I've posted an answer. Try setting your corsOptions.origin to the array of URL strings that you want to work
Comment: Not sure but it sounds like maybe you want to try adding a `Vary: Origin` response header to the response. See the answers at https://stackoverflow.com/a/46735000/441757 and https://stackoverflow.com/a/46067554/441757 and see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Vary
Answer: I found a solution to this problem. The browser was caching origin.
Solved this problem by adding the following header in response
Cache-Control: no-store,no-cache,must-revalidate

I've also added Vary: Origin but i think it didn't solve the problem
I've managed cors as follows
var whitelist = ['https://www.example.com', 'https://subdomain.example.com'];
router.all("*", (req, res, next) => {
    var origin = req.headers.origin;
    if (whitelist.indexOf(origin) != -1) {
      res.header("Access-Control-Allow-Origin", origin);
    }
    res.header("Access-Control-Allow-Headers", ["Content-Type","X-Requested-With","X-HTTP-Method-Override","Accept"]);
    res.header("Access-Control-Allow-Credentials", true);
    res.header("Access-Control-Allow-Methods", "GET,POST");
    res.header("Cache-Control", "no-store,no-cache,must-revalidate");
    res.header("Vary", "Origin");
    if (req.method === "OPTIONS") {
    res.status(200).send("");
      return;
    }
    next();
});

where router is express.Router()
I hope that someone find this helpful.
Answer: It seems like you're using express. Based on their docs, you can set origin to an array of strings or RegExp which should allow the origins that you want.

https://expressjs.com/en/resources/middleware/cors.html

origin: Configures the Access-Control-Allow-Origin CORS header.
  Possible values:

Array - set origin to an array of valid origins.
  Each origin can be a String or a RegExp. For example
  ["http://example1.com", /.example2.com$/] will accept any request
  from "http://example1.com" or from a subdomain of "example2.com".

Based on that, updating getOriginValue() to the following might work for you:
var getOriginValue = () => {
  if (whitelist.indexOf(req.header('origin')) !== -1) {
    return whitelist; // Return array of strings
  } else {
    log.error(__filename, 'Not allowed by CORS', origin);
    return false;
  }
}

Answer: It's because you're calling your function, you need to just pass it to the middleware
var corsOptions = {
      origin: getOriginValue, // Not origin: getOriginValue()<-no parens,
      methods: ['GET', 'POST'],
      credentials: true,
      preflightContinue: false,,
      allowedHeaders:["Content-Type","X-Requested-With","X-HTTP-Method-Override","Accept"]
    }

Comment: You've noted a valid point, but that doesn't solve my problem

cors failing for subdomains


converting dictionary within a list to JSON format using python

Question: I have a dictionary within a list as below
b = [
    {
        "ServiceLevel": {
            "S": "AA"
        },
        "ReadyTime": {
            "S": "2020-01-31T12:00:00"
        },
        "DeliveryDate": {
            "S": "2020-02-15T12:00:00"
        }
    }
]

I am trying to access the elements within that inner dictionary and convert them to JSON format:
My try:
a = b[0]
for key, value in sorted(a.items()):
    a1 = print(key, value)

for i in a1:
    b1 = a1[i]
    print(b1)

I am trying to get this output:
b = {
        "ServiceLevel": "AA",
        "ReadyTime": "2020-01-31T12:00:00",
        "ReadyDate": "2020-01-31T12:00:00",
}

ERROR : TypeError: 'NoneType' object is not callable
Comment: Hmm, I just tried 3.7.8 and got "... not iterable" as expected.
Comment: What Python version are you using? That error message looks wrong.
Comment: python 3.7 @ superb rain
Answer: You are assigning print() method to a variable, here is the NoneType error
a1 = print(key, value) # variable a1 is NoneType
for i in a1: # error because variable a1 is not iterable

Use :
b1 = {}
for key, value in b[0].items():
    b1[key] = value['S']
print(b1)

Comment: I am a beginner in python programming and I was not aware of using values['S']. I stored it in print and thought I could iterate over the elements in that print statement. I tried your answer and it is working as expected. Thanks


How do I add numbers to each tableview row beginning from 1

Question: I have created a TableView with a Prototype Cell with a lot of content in it, multiple labels, 2 pictures and a view which is colored if the row is selected.
What I want to do now, is I want to add a number for each of the rows created so the user can see how many rows there are. I can't use the ID I get from the Database since it is a random mix of characters.
I added an Int variable 'participantID' which starts from one and increments by one with every cell created and writes the value in a label within the prototype cell.
This does exactly what I want, with just one issue: Every time i scroll in the table the ID is incremented. I know that's because of the reuse of the cells however I couldn't find a way to fix this issue.
This is how it looks (particID):
func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell {
        let cell = tableView.dequeueReusableCell(withIdentifier: "tbcCourseParticipant", for: indexPath) as! participantCell
        let partic = participants[indexPath.row]
        cell.lblId?.text = String(particID)
        cell.lblName?.text = partic.name
        cell.lblFirstName?.text = partic.firstName
        cell.lblBirthDate?.text = partic.birthDate
        cell.lblPhoneNr?.text = partic.phoneNr
        cell.lblPrice?.text = String(partic.payed) + "€"
        particID += 1
        return(cell)
    }

Answer: The total number of rows is participants.count. The number of this row is indexPath.row+1 (I add 1 because the user will not expect to start counting at zero the way a programmer would do).
So I would say:
cell.lblId?.text = String(indexPath.row+1)

Or maybe:
cell.lblId?.text = String(indexPath.row+1) + " of " + String(participants.count)


T-SQL to add a column bound to a existing default constraint

Question: I have a DEFAULT constraint created in the database, which I want to share among several tables, and I want to script an unique alter table statement for adding a field in that tables as follows:
ALTER TABLE MyTable ADD MyField NOT NULL DEFAULT(DF_EXISTING_DEFAULT)

The following error ocurrs:
Msg 128, Level 15, State 1, Line 1
The name "DF_EXISTING_DEFAULT" is not permitted in this context. Valid expressions are constants, constant expressions, and (in some contexts) variables. Column names are not permitted.

Notice that the default value is already defined in DF_EXISTING_DEFAULT, I don't want to write it again. How can I do it? Any reason I shouldn't do it? Is it a t-sql limitation?
Answer: You actually can do it but the mechanism to do so is deprecated so shouldn't be used.
You can CREATE DEFAULT object and then use sp_bindefault.
There is no non deprecated replacement. If the logic for the default is complex you could put it in a scalar UDF to avoid repeating yourself.
Comment: Yes that's a (deprecated) default **object**. Not a (non deprecated) default **constraint**.
Comment: Well, the default is not complex (e.g. DEFAULT('A simple text'), but it can change in the future. In this case, is UDF the only way to avoid that "code duplication"? The main objective at this moment is to keep a *single* default and to avoid those sql automatic constraint naming.
Comment: @natenho - Yes. A default *constraint* can't be shared between columns  and there are no user defined global variables in SQL Server so to avoid repeating `'A simple text'` in the constraint definitions themselves this is the only way. You could also store it in a table but that would also require a UDF to access. The advantage of storing in a table might be easier maintenance (altering the UDF definition would require dropping all defaults referencing it) but hardcoded in the UDF would be more efficient and in constraint definition itself more so. So trading performance vs maintainability.
Comment: Regarding "can't be shared"...when you use `sp_binddefault` for the same DF in multiple table/columns, aren't you sharing the constraint? At least I can't see new constraints being created.
Comment: @natenho - That is sharing a default *object* not default *constraint*. Trying to bind a constraint gives `Cannot bind default 'X'. The default must be created using the CREATE DEFAULT statement.` (at least on my version of SQL Server)
Comment: If I understood what you said, `EXECUTE sp_bindefault N'dbo.DF_EXISTING_DEFAULT', N'dbo.MyTable.MyField'` works in SQL 2008 for every table I want to bind.
Comment: @natenho - `CREATE TABLE T1(X INT CONSTRAINT DF DEFAULT 0);CREATE TABLE T2(X INT);EXEC sp_bindefault 'DF' , 'T2.X'` definitely doesn't work for me on 2008.
Comment: I meant `CREATE DEFAULT [dbo].[DF] AS 0
CREATE TABLE T1 (X INT);
CREATE TABLE T2 (X INT);
EXEC sp_bindefault 'DF', 'T1.X';
EXEC sp_bindefault 'DF', 'T2.X';`
Comment: Now I see, from [msdn](http://technet.microsoft.com/pt-br/library/ms173565.aspx)`CREATE DEFAULT`: This feature will be removed in a future version of Microsoft SQL Server. Avoid using this feature in new development work, and plan to modify applications that currently use this feature. Instead, use default definitions created using the DEFAULT keyword of ALTER TABLE or CREATE TABLE.


Using Bayesian inference to fit functions

Question: I'm trying to understand Bayes Inference better and was wondering if it is possible to use it to fit a function $f(t)$ where I only know the value $f(t)$ for a few values of $t$. For each value of $t$ I have an ensemble of possible values for $f(t)$. As a example I have an experiment which I can repeat as often as I want but my quantaty of interest can only be measured at a few fixed points in time.
As an example lets say $f(t) = sin(\alpha x)$. With $\alpha=1.4$ and the ensemble of each measurment is $Norm(\mu=f(t), \sigma=.01)$. I measure the $f(t)$ at 8 equily distributed points between 0 and $\pi$.
I tried to model this in pymc with:
alpha = pm.Normal('alpha', 3, 1)
tau = pm.Normal('tau', 5, 10)
obs = [pm.CommonDeterministics.Lambda('obs_{}'.format(i),
                lambda alpha=alpha : np.sin(alpha*xx))
       for i, xx in enumerate(x)]
noises = [pm.Normal('noise_{}'.format(i), o, tau,
                    value=m, observed=True) for i, (m, o) in
          enumerate(zip(measurements, obs))]

This assumes that the priors for $\alpha$ and $\sigma$ (note: pymc uses $\tau = \frac{1}{\sigma^2}$ instead that is the reason I write $\tau$ in the code.)
When I run this with the MarkovChainMonteCarlo algorithm from pymc I get the following result.

Which is totally off. It does this because it starts to estimate an unreasonably large error $\sigma$ in the observations.
So my questions are.

Is this because I choose bad priors?
Is so can what would be more appropriate priors?
Is something like this in general possible using bayes?

Update
The be clear. I would like to infer the value of $\alpha$ from the observed data.
The whole example can be found here
Comment: I think you are estimating independent error terms for each of the measurements. I guess the model you want for your data $y$ is $y \sim N( sin(\alpha x), \tau)$ with your priors $\alpha \sim N(3,1)$, $\tau \sim N(5,10)$ (I don't think these priors are good). So you $\tau$ should be independently drawn from the same distribution for each measurement.
Answer: Yes, there are plenty of ways of modelling functions. I'd point you in the direction of Gaussian processes for quite a general framework.
Here is an example of modelling a sine wave (green) with one (red) and credible intervals (orange):

Here (IIRC) is the code used to generate it:
from pylab import *
n = 150

ns = 145

X = linspace(-8*pi, 8*pi, n)
S = 2*sin(X)
Y = S + randn(n)
F = linspace(-7*pi,7*pi,ns)

sig1 = 1.0
sig2 = 0.5
b = 0.5

def K(p,q):
  noise = 0
  if p == q:
    noise = sig1
  return sig2*exp(-b*(p-q)**2) + noise

def makeKmn(Xm,Xn):
  m = len(Xm)
  n = len(Xn)
  Kmn = matrix(zeros((m,n)))
  for i in range(n):
    for j in range(n):
      Kmn[i,j] = K(Xm[i],Xn[j])
  return Kmn

Koo = makeKmn(X,X)
Kos = makeKmn(X,F)
Kss = makeKmn(F,F)

YF = Kos.T * Koo.I * matrix(Y).T
YF = array(YF.T)[0]

VF = Kss - Kos.T * Koo.I * Kos
Vs = diag(VF)**(.5)

YFu = 1.96 * Vs + YF
YFl = -1.96 * Vs + YF

fill_between(F,YFu,y2=YFl,facecolor="orange",alpha=0.2)
scatter(X,Y)
plot(F,YF,color="r")
plot(X,S,color='g')
show()

Have a look at gaussianprocess.org for details.
EDIT:
If you suspect function is close to a sine wave one way to proceed is by modelling the mean function of the GP as something like $\mu(x)=b \sin(ax + c)$. If $b$ comes out near zero I expect that would suggest a sine wave is not a good model for your data.
The reason the GP formalism is useful here is that it makes sense to then talk about the value of the function at time points you have never taken a measurement at, i.e. to interpolate. The covariance can be kept simple if desired, e.g. $K(t,t')=\mathbb I_{t=t'}\sigma^2$ (although my hunch would be keeping it so simple isn't going to tell you so much about whether your sine model is good or not.)
Comment: I've read a little bit about gaussian processes now. As far as I understand there are a good tool to make predictions about future observed values based on the available data. While useful this is not what I want to know. I usually know the function my data should follow but not the exact parameters that fit best (the phase $\alpha$ of the sine-curve eg). I don't see how it is possible to infer the value of $\alpha$ with gaussian processes modeling. Are gaussian processes the right tool for that? If so can you point me to some literature. Thanks
Comment: Added some comments. Also, the page linked above has info on software packages for fitting.


Why is training error a better performance metric than cross-validation error?

Question: What I have learned is that I should use cross-validation performance for selecting the best model. Currently, selecting the models based on cross-validation performance gives lower test performance than if I select the models by training performance. I have 300-400 features and about 10000 samples, with 12 outputs (multi-class classification).
I've split the dataset in 80:20 % shares (training:test). I'm using H2O's deep learning with 5-fold cross-validation.
Is it wrong to use the training log-loss instead of the cross-validation log-loss?
As I've understood it, the cross-validation validation sets are used for scoring the models (5). The XVAL log-loss is then the average (?) of the 5 models trained.
I have 2 loops, one loop for sampling parameters such as epochs (between 1 and 5), number of layers (2-5), and number of nodes in the layers (10-50).
For each network parameter configuration in the outer loop, I train 100 models, and of these I'm checking among others the average log-loss for both training and cross-validation, variance of the log-loss, average true-rate for training and testing, and average profit on test set.
If I selected the 10 best model configurations based on the average training log-loss or average training true-rate, it would give higher true-rate and profit on the test set, compared to using cross-validation metrics.
I think the cross-validation log-loss is good for training (true-rate is probably not good for this), but do I have to also use it for selecting model configurations?
I previously did feature selection and later parameter tuning by following the cross-validation metrics, but now that I would get better results by following the training log-loss instead I'm wondering if I should do the feature selection and parameter tuning again?
Answer: There isn't a single "best" answer to your question. There are many threads throughout CV that discuss these issues. Here's one ...
Understanding how good a prediction is, in logistic regression
Key points to note are that training or calibration data provides useful information regarding optimal fit (e.g., Harrell recommends using a nonparametric loess smoother for this) but is well known to be positively biased:

The calibration curve is both a measure of goodness-of-fit and a great
  way to check the accuracy of probabilities estimated by the model.

Regardless, the emphasis in machine learning is on using validation (out-of-sample) data to evaluate fit. Validation results provide useful information regarding the bias-variance tradeoff that can't be inferred from calibration information.

If your validation error and training error are both high, you have underfitting (bias).
If your validation error is high but training error is low, you have overfitting (variance).

Here's one discussion of this:
Question about bias-variance tradeoff
Comment: Consistently, I got better predictions for the accuracy by using the training error and not the validation error. This happened during many runs, also where parameters were changed.
What I planned to test next, is to split the test set in 2 and select 2 models: one based on validation error, and one on train error and then compare the accuracy.
Answer: I am no expert in this matter but I think in your case it has to do with the model you chose yourself. If 10k is already a small size for a neural network, aren't the fold sub-samples way to little for a dataset with 300-400 features and a min of 20 and max of 250 neurons? Probably your neural network is not able to learn anything from the fold datasets, while the full training dataset maybe just about enough to notice a difference?
I can't find a concrete paper but there's a lot of literature on-line discussing how the complexity of the problem defines the optimal training dataset size.
Here's an oline blog that cites some good papers.


Is it possible to build a fiber bundle whose fibers are different? (Or we should not call it a fiber bundle?)

Question: Suppose there is a fiber bundle $E$. The base space is $M$ so that $\pi:E\rightarrow M$ is the projection. By the definition, the bundle has a typical fiber $F$ such that the local trivialization over one open subset $U_i$ of $M$ is defined in this way $\phi_i:U_i\times F\rightarrow \pi^{-1}(U_i)$. So we can find out that no matter which open set $U_i$ we choose, we always choose the same typical fiber $F$. I guess this is because we want to have a smooth structure. If at different points of $M$, we choose different fibers, for example, at point $p\in M$ we attach a 3 dimensional vector space, while at point $q\in M$ we attach a 2 dimensional vector space and so on, then we are not going to have a smooth structure.
So my question is: can we attach different fibers to different points of a base manifold? If we insist to do so, is this structure still smooth? Is it still called a fiber bundle?
Comment: Sure. The base could be disconnected. (If the base is connected, then no: it's a nice exercise to show that a fiber bundle over a connected base has homeomorphic fibers, and similarly a smooth fiber bundle over a connected base has diffeomorphic fibers.)
Answer: See the notion of fibration:

A fibration is like a fiber bundle, except that the fibers need not be the same space, rather they are just homotopy equivalent.
Answer: The projection $\mathbb R^3\setminus \{0\}\to \mathbb R: (x,y,z)\mapsto x$ is a smooth submersion between connected smooth manifolds and all its fibers are connected.
However the fibers are not diffeomorphic:  the fiber of $0\in \mathbb R$ is difffeomorphic to $\mathbb R^2\setminus \{0\}$ whereas all other fibers are diffeomorphic to  $\mathbb R^2$.
Comment: Right, but this isn't a fiber bundle.
Comment: @Qiaochu: of course it is not a locally fiber bundle, but I was answering the question "can we attach different fibers to different points of the manifold" and I wanted to give an example of a more general situation (which is also a counterexample to Ehresmann's theorem for non proper maps).


What is the guitar effect being used in Fine Young Cannibals' She Drives Me Crazy?

Question: There is a power chord refrain used throughout this song.  Here is the video, queued up to just before where you hear the power chords the first time.

I've often thought it might be a compressor that is used to get this "compressed" sound.  But I've read up on what a compressor does and I'm not sure that's quite it.
Anyone know how to get this effect?
Comment: Side note: *reading* what a compressor does is no substitute for *hearing* what a compressor does
Comment: Sound engineer David Z talks in 2001 about recording guitar (among other things) for that song: https://web.archive.org/web/20050210123643/http://mixonline.com/mag/audio_fine_young_cannibals/
Comment: FWIW, I don't believe these are power chords-- I can hear a third quite distinctly in them, although it is _possible_ it is an overtone that is exacerbated by the YT compression.
Comment: Very little of that song was not massively manipulated "in post," and I think the distortion has volume control set up as an envelope follower. ( https://www.mixonline.com/recording/classic-tracks-fine-young-cannibals-she-drives-me-crazy-375247 ). I cannot find a "live" performance of it that does not look like they are pretending to play. Their awards show performance, the guitarist has no effects set up, his guitar sound rapidly switches from heavy effects to clean chords, and the audio itself sounds almost exactly like the recording. If it is live a lot of it may be midi triggered
Comment: Some guitars do have midi capability even back then.
Answer: Yeah, it's a heavy comp, but it's combined with the actual playing style, which is a really slow drag across the chord each time.
I think it's either combined with a slight delay panned half right, or it has a close & ambient mic, panned centre & half right, so the sound seems to spread slightly towards the right each time.
There could be chorus on it too, or it's a side effect of the delay [however generated]. I'm not hearing a distinct double-track.
From comments
I think they've copied the "Bowie mic" thing from Heroes [but without the gate, perhaps] & had one long ambient mic panned off-centre. That would automatically be duller than the close mic. Distortion doesn't have to make it dull, especially if you use a really saggy amp*. That amp doesn't sound like a modern hi-gain, smooth noise, it's an older amp with perhaps slightly past their best valves or capacitors.
*I loved it when I discovered if you 'turn round' the amp faces in Native Instruments' Guitar Rig you could actually dial in an amount of sag. It made my day because I actually love that noise, if nicely controlled.
Comment: I just looked through the manual - seems most if not all of the amps still have that control - you turn them round to get to deeper edit parameters. Long time since I've actually used it, tbh.
Comment: It could be the same guitar being played through two amps at once, but I 'm pretty sure I hear two tones.  There's some midrange twang coming through that I wouldn't expect to survive that level of distortion.  I heard the chorus too, but decided it might not be there after I settled on "probably two guitar tracks"
Comment: I think they've copied the "Bowie mic" thing from Heroes [but without the gate, perhaps] & had one long ambient mic panned off-centre. That would automatically be duller than the close mic. Distortion doesn't have to make it dull, especially if you use a really saggy amp. That amp doesn't sound like a modern hi-gain, smooth noise, it's an older amp with perhaps slightly past their best valves or capacitors.
Comment: This comment about 'turning round' the amp face.  What kind of amp was that, and what does all that involve?  Very interested.  Could it be done with anything available on the market today?
Answer: It's distortion.  Distortion does have a compressing effect.  You can emphasize the compressing effect of distortion by putting a normal compressor before the distortion effect, which is pretty common.
Aside from that, it sounds like the guitar part is doubled by a cleaner guitar.  This second guitar likely has some milder distortion which also has a compressing effect, it's very common.
Answer: It sounds like a Rockman distortion
Comment: Hi Christopher. Welcome to the site. Just know, the site goal is to leave reasonably substantial answers, with citations when possible. Could you fill out your answer a bit? Perhaps explain what you hear that leads you to your conclusion.
Comment: The Rockman always sounded like an over-treated Marshall on a power-soak. It didn't have the sag I'm hearing in this. The chorus was …let's call it, a sound unto itself.


Changing image tag in OpenShift Deployment with CLI

Question: is there a set of commands to change the docker image name/tag in an existing deployment in a project in an OpenShift cluster?
Answer: You can use the oc set image command to change the image for a container in an existing Deployment / DeploymentConfig:
oc set image dc/myapp mycontainer=nginx:1.9.1

Try oc set image --help for some examples.


Diagonalizing matrix that has variables

Question: So I have the following matrix that I have to diagonalize
\begin{bmatrix}
    0    & 1 & 0 & 0 \\
    -x^2 & 0 & 0 & 0 \\
    0    & 0 & 0 & 1 \\
    0 & 0 & -y^2 & 0
\end{bmatrix}
So the eigenvalues of the matrix are: ${\pm ix}$, ${\pm iy}$.
But now I'm having trouble finding the eigenvectors. Specifically, I'm stuck on what to do when I normally try to reduce the matrix to row-reduced form. When plugging in +ix, I can cancel out the second row, but then stuck on what to do with the third and fourth row. Any help would be great!
Answer: By observation, you can see that multiplying the first column with $\pm\frac{i}{x}$ will cancel the second column out. The same yields for the third and the fourth column with the factor $\pm\frac{i}{y}$. Therefore your eigenvectors will be:
$$\begin{bmatrix} -\frac{i}{x}\\1\\0\\0\end{bmatrix},\begin{bmatrix} \frac{i}{x}\\1\\0\\0\end{bmatrix},\begin{bmatrix} 0\\0\\-\frac{i}{y}\\1\end{bmatrix},\begin{bmatrix} 0\\0\\\frac{i}{y}\\1\end{bmatrix}$$
Corresponding to the eigenvalues: $(ix,-ix,iy,-iy)$
Comment: Thanks! That is what I suspected. Also for the eigenvector corresponding with -iy, I'm assuming that's supposed to be a +i/y in the column vector, right? (it looks like a typo to me)
Comment: Yes that's a typo, sorry! Could you accept as answer please?

Diagonalizing matrix that has variables


Comparing emmeans values multinom model

Question: I am performing quite some binomial/multinomial models for my thesis. After doing the emmeans statement, I used the contrast statement to compare the emmeans of the different groups.         But, I start to think about a few things:

My data is sampled as such that the groups don't have an equal size
Therefore the SE is not equal for each estimate. If my reasoning is correct, the variance between the two groups is not the same. I didn't found how the contrast test works, but if it actually just performs a t-test, then the assumptions for a t-test are violated. How can I adjust for this?

Or am I making things more difficult than they actually are?
Example:
I want to compare the distribution of 4 age classes between two methods. HD and DD, HD contain 322 individuals and DD 1711 individuals. The null hypothesis is that there is no difference in age distribution between the two methods. The Anova test on the model was significant. Therefore I want to know which age classes differ the most between the two methods.
the piece of code (Age class consists of a 0,1,2,3+ age class, method consists of the HD and DD group):
a <- multinom(`Age class` ~  `Method`  , data = goodyears)
emmeans = emmeans(a,~ `Age class`  |`Method`, mode = "prob")
z =contrast(emmeans, "pairwise", simple = "each", combine = TRUE, adjust = "mvt")

For example, the emmeans for the 1y age class in the HD group is 0.32 (95%CL [0.29,0.34]) and the emmeans for the 1y age class in the DD group is 0.24 (95%CL [0.18,0.30]). The contrast test is insignificant (logical as the intervals overlap). Is there a way to deal with the rather large difference in group size? Statistically the smaller group size caused a larger confidence interval which made the effect less significant. Or do I better accept the result as it is, and state in a discussion that the insignificance of the effect might be due to the limited sample size of one group?
Comment: Please edit the question to provide more information about the model `a` that you are using for the basis of the `emmeans` post-modeling analysis. Without that information it's hard to know if the call to `emmeans` actually accomplishes what you want.
Comment: Hi I provided the code, and some additional information on the variables, I hope it helps to understand the question.
Comment: Are you interested in the contrast between 1y age class in the HD group and the 2y age class in the DD group? Can you explain what you mean by "I want to compare the distribution of 4 age classes between the two methods."
Comment: Excuse me, it is a typo. I am interested in the contrast between 1y age class in HD group and the 1y age class in the DD group.
Answer: The emmeans package does not require equal sample sizes, so to that extent you are worrying too much. It uses the variance-covariance matrix of coefficient estimates to compare different scenarios after the initial model has been fit.
Based on that matrix, the formula for a weighted sum of correlated variables provides the variance estimate for any sum/difference of coefficient estimates involved in a comparison of interest. The assumption for significance testing is generally that the coefficient estimates follow a multivariate normal or multivariate t distribution, an assumption that underlies much statistical analysis.
The sample size enters through its effects on those variance/covariance estimates. A coefficient variance estimate typically decreases with the number of observations, so in your case there is probably more precision in estimates for the DD than in the HD class.
Where you might be making things harder for yourself is in the number of comparisons you perform. Results in emmeans are corrected for multiple comparisons. The more comparisons you perform, the harder it is to show that any single comparison is "significant." I'm not sure how many comparisons are provided by your call to the contrast() function, but I don't think that you need to perform more than an HD versus DD comparison within each of the 4 age classes to accomplish what you want, or 4 comparisons total.
Finally, don't jump to an assumption that overlap of confidence intervals necessarily means no significant difference between groups. See this answer for a more nuanced explanation.
Answer:
Is there a way to deal with the rather large difference in group size? Statistically the smaller group size caused a larger confidence interval which made the effect less significant.

You have less information (= fewer samples) about the HD group and about the distribution of age classes within this group. So you should expect to estimate the HD parameters with less accuracy (= wider confidence intervals). To get more precise estimates, you would have to collect more data.
That said, you may be over-adjusting by computing more comparisons than you are actually interested in.
For illustration purposes, I've generated some fake data according to your specifications. There are two methods, DD and HD, and four age classes 0, 1, 2 and 3+.
Here is the output of contrast as you use it:
emm <- emmeans(fit, ~ `Age class` | `Method`, mode = "prob")
contrast(emm, "pairwise", simple = "each", combine = TRUE, adjust = "mvt")
#>  Method Age class contrast estimate     SE df t.ratio p.value
#>  DD     .         0 - 1    -0.03507 0.0170  6  -2.065  0.4246
#>  DD     .         0 - 2    -0.02747 0.0169  6  -1.630  0.6360
#>  DD     .         0 - (3+) -0.01870 0.0167  6  -1.119  0.8811
#>  DD     .         1 - 2     0.00760 0.0175  6   0.435  0.9986
#>  DD     .         1 - (3+)  0.01636 0.0173  6   0.945  0.9377
#>  DD     .         2 - (3+)  0.00877 0.0172  6   0.510  0.9968
#>  HD     .         0 - 1     0.05280 0.0477  6   1.106  0.8861
#>  HD     .         0 - 2     0.22360 0.0400  6   5.590  0.0106
#>  HD     .         0 - (3+)  0.30125 0.0351  6   8.581  0.0011
#>  HD     .         1 - 2     0.17080 0.0387  6   4.409  0.0332
#>  HD     .         1 - (3+)  0.24845 0.0340  6   7.299  0.0028
#>  HD     .         2 - (3+)  0.07765 0.0283  6   2.743  0.2039
#>  .      0         DD - HD  -0.16472 0.0291  6  -5.666  0.0101
#>  .      1         DD - HD  -0.07686 0.0285  6  -2.697  0.2156
#>  .      2         DD - HD   0.08635 0.0235  6   3.677  0.0720
#>  .      3+        DD - HD   0.15523 0.0193  6   8.054  0.0017
#>
#> P value adjustment: mvt method for 16 tests

Why adjust for all possible 16 tests? If you are not interested in the contrast between 1y age class in the HD group and the 2y age class in the DD group and the rest of the comparisons across both ages and methods, you are over-adjusting.
Here is how to compare the two methods at each age class.
# NB: Switch the order in the `condition on` statement.
emm <- emmeans(fit, ~ `Method` | `Age class`, mode = "prob")
contrast(emm, "pairwise", adjust = "mvt")
#> Age class = 0:
#>  contrast estimate     SE df t.ratio p.value
#>  DD - HD   -0.1647 0.0291  6  -5.666  0.0013
#>
#> Age class = 1:
#>  contrast estimate     SE df t.ratio p.value
#>  DD - HD   -0.0769 0.0285  6  -2.697  0.0357
#>
#> Age class = 2:
#>  contrast estimate     SE df t.ratio p.value
#>  DD - HD    0.0863 0.0235  6   3.677  0.0104
#>
#> Age class = 3+:
#>  contrast estimate     SE df t.ratio p.value
#>  DD - HD    0.1552 0.0193  6   8.054  0.0002

Note that now we get no statement about number of adjustments. Since for each age class emmeans calculates a single pairwise comparison, it applies no adjustment to the p-values.
The question if and how to adjust for multiple comparisons of interest is trickier than the fact we shouldn't calculate and adjust for comparisons of no interest. Since you are studying the difference in age distribution between the two methods (rather than, say, focusing on babies and toddlers separately), I think you should do multiple comparisons adjustment.
So finally here is how to compare the two methods at each age class and adjust for making four comparisons across the four age classes.
emm %>%
  contrast("pairwise", by = "Age class") %>%
  summary(by = NULL, adjust = "mvt")
#>  contrast Age class estimate     SE df t.ratio p.value
#>  DD - HD  0          -0.1647 0.0291  6  -5.666  0.0038
#>  DD - HD  1          -0.0769 0.0285  6  -2.697  0.1054
#>  DD - HD  2           0.0863 0.0235  6   3.677  0.0324
#>  DD - HD  3+          0.1552 0.0193  6   8.054  0.0006
#>
#> P value adjustment: mvt method for 4 tests

Comment: Nice demonstration of the problem of more comparisons than you need! (+1)

Comparing emmeans values multinom model


How does this approximation of $\sqrt[R]{n}$ work?

Question: Decades ago, when the slide rule was the norm, and calculators with the ability to calculate square roots were the cutting edge of technology, I was given an interesting formula. It went like this:

To estimate the $R$th root of $n$ using a simple calculator (where both $R$ and $n$ are positive numbers; fractions are allowed), do the following.

Enter $n$ into the calculator.
Press the $\sqrt{}$ button 12 times (i.e. calculate the 4096th root of $n$).
Subtract 1.
Divide by $R$.
Add 1.
Square the result (usually press the $\times =$ buttons) 12 times (i.e. calculate the 4096th power of the result).

Fascinatingly, this works pretty well.
For example, using this method on my calculator to find $\sqrt[8.5]{586.426}$ gives the answer with an error of just 0.05%.
This means that:
$$\sqrt[R]{n} \approx \biggl(\frac{\sqrt[4096]{n} - 1}{R} + 1\biggr)^{4096}$$
Now, to me, the number $2^{12}$ seems arbitrary. So, my question is:
Why does this work?
I've tried muddling through equations to no avail.
EDIT: I've just spotted this question, which seems relevant.
Comment: Love that you included the slide rule link, hahaha!
Comment: Yes, @ChaseRyanTaylor — I have fond memories of them! Not so much the old [log tables](https://en.wikipedia.org/wiki/Mathematical_table#Tables_of_logarithms).
Comment: Oh my goodness. That doesn't sound too bad if you know the change-of-base formula. Those tables remind me of the trigonometric ones we sometimes used in geometry.
Comment: Ah, yes, @ChaseRyanTaylor, I had forgotten about those trigonometric tables. So glad that we don't have to use them any more. These are foreign concepts to my children.
Answer: You're correct that $2^{12}$ is arbitrary: it's good because it's large and a power of $2$ (so you can mash the square root button to get the $2^{12}$ root). Let the root we have available be the $k$th, so that in your case $k=4096$ (this just makes the calculation clearer), and the formula is then
$$ f_k(x) = \left( 1+\frac{x^{1/k}-1}{R} \right)^k $$
We want to show that for large $k$, $f_k(x) \approx x^{1/R}$. Since $k$ is large and $x$ is positive, we can use the exponential's power series to write
$$x^{1/k} = e^{(1/k)\log{x}} = 1 + \frac{1}{k}\log{x} + \frac{1}{2k^2}(\log{x})^2 + O(k^{-3}), $$
where $O(g(k))$ means that the term decreases at the same rate as $g(k)$ as $k \to \infty$, so
$$ 1 + \frac{x^{1/k}-1}{R} \approx 1 + \frac{1}{Rk}\log{x} + \frac{1}{2Rk^2}(\log{x})^2 + O(k^{-3}) $$
At this point it's probably most convincing to take logs of everything, so
$$ \log{f_k(x)} = k\log{\left( 1+\frac{x^{1/k}-1}{R} \right)} = k\log{\left( 1 + \frac{1}{Rk}\log{x} + \frac{1}{2Rk^2}(\log{x})^2 + O(k^{-3}) \right)}, $$
and since $k$ is large, the argument of the logarithm is close to $1$, and we can apply the power series for the logarithm to give
$$ \log{f_k(x)} = \frac{1}{R}\log{x} + \frac{1}{2Rk}(\log{x})^2 - \frac{1}{2R^2k}(\log{x})^2 + O(k^{-2}) $$
Hence
$$ \log{\left( \frac{f_k(x)}{x^{1/R}} \right)} =  \frac{R-1}{2R^2k}(\log{x})^2 + O(k^{-2}). $$
Exponentiating again gives the error in the approximation as approximately
$$ \frac{f_k(x)}{x^{1/R}} -1 \approx \frac{R-1}{2R^2k}(\log{x})^2, $$
which will obviously be very small for large $k$ and "ordinary-sized" $R$ and $x$ (or even large $R$, since the error depends on $1/R$).
Comment: Man, that's clever! Thank you
Answer: As you point out, $2^{12}$ just seems to be arbitrary and is used to denote a number that is sufficiently large. So we can reach the required result by just taking limit to infinity instead of the factor  $2^{12}$. i.e. We need to prove that,$$\sqrt[n]{x} = \lim_{k \to \infty}\left(1+ \frac{x^{1/k} - 1}{n}\right)^k$$
which is not a hard nut to crack.

Here's the computation of the limit. Consider,$$\begin{align}y &=\lim_{k \to \infty}\left(1+ \frac{x^{1/k} - 1}{n}\right)^k\\& = \color{blue}{\lim_{k\to \infty} \left(1 + \frac{x^{1/k} - 1}{n}\right)}^{{\color{blue}{\left(\frac{n}{x^{1/k} -1}\right)}}\cdot \left(\frac{x^{1/k} - 1}{n}\right)\cdot k}\tag{1}\\& = \lim_{k\to \infty}\exp{\left(\frac{x^{1/k} - 1}{n}\right)\cdot k}\tag{2}\\& = \exp{\left(\color{blue}{ \lim_{k\to \infty}\frac{x^{1/k} - 1}{1/k}}\right)\cdot \frac1n} \\&= \exp\left(\log(x)\right) \cdot \frac1n\tag{3}\\& = \exp \left(\log(x^{1/n}) \right) \\& =\boxed{\sqrt[n]{x}}\end{align}$$

Explanation:

$(1)$ Here I made some adjustments in the exponent term to make it in the form of a standard limit.
$(2)$ Notice that, it is of the form $\lim_{x\to\infty} \left(1 + \frac1x\right)^{x} = e.$
$(3)$ Here we can simply use $\lim_{x\to 0} \frac{x^a - 1}{a} = \log (a)$.
Comment: Excellent, thank you!
Comment: You first claim has a $k$ in it, which I guess should be an $n$.
Comment: Yes. A typo. Fixed it. Thanks! :)
Comment: @Sahaj BTW our rep points just differ by a 1 lol


Dinghy Racing: how do I use the kicker/vang properly?

Question: I'm sailing an Enterprise Dinghy, and recently upgraded to a cascading kicker (kicking strap, boom vang, ...). The previous kicker I had I could never operate during a race properly, but now I can, and my question is: what is the best way to learn how to use a kicker properly?
I know that in very light wind it should be tight to flatten the sail, making it easier for the weak wind stream to attach to the sail, off in medium wind which has more power to pull the rounder sail forward, and tight again in strong wind to depower the sail. But how do I go beyond that? How can I best learn to match a sail shape to the wind condition, and then achieve that shape with the kicker?
Comment: @PhilipKendall On the contrary, it is very competitive. Using the kicker properly is essential for dinghy racing, in order to achieve the best boat speed in given conditions. I have sailed recreationally for many years without using the kicker, as it is not essential for cruising.
Comment: I'm voting to close this question as off-topic because it doesn't involve a *competitive* sport. Please edit to focus the question on the competitive aspects, or consider deleting this and reposting on [The Great Outdoors](http://outdoors.stackexchange.com/).
Comment: Good running technique isn't essential for recreational jogging, that doesn't mean a question on good technique is about competitive running. Same deal with your kicker technique and recreational versus competitive sailing.
Answer: The best way to learn how to use a boom vang is really by using it, but here are some guidelines that I have taught at Sailing Schools, and used to great effect when racing:

in very light wind, you actually don't want tension on it at all

But if you have a large swell or rough seas in a light wind, leaving the vang too loose will let your boom bounce up and down, so pull it in enough to stop that happening.

in high winds you want it pulled in tight

This gives more power, as the sail flattens, and the mast bends, but be ready to slacken it off as the sail overpowers.
That basic rule, having your vang tension proportional to the wind strength is an excellent approximation. This allows the sail more power - which is what you want control of in racing.
It would be ideal if the helmsman could also work the vang, because you can play the helm, main sheets and vang off against each other, but on a 2 person boat you'd usually have your crew play the vang.
Edit: Just found a brief page from Dave Dellenbaugh (an America's Cup veteran) that gives a little more detail on these.


Advantages of Particle Swarm Optimization over Bayesian Optimization for hyperparameter tuning?

Question: There's substantial contemporary research on Bayesian Optimization (1) for tuning ML hyperparameters. The driving motivation here is that a minimal number of data points are required to make informed choices about what points are worthwhile to try (objective function calls are expensive, so making fewer is better) because training a model is time-intensive -- some modestly-large SVM problems that I've worked on can take between minutes and hours to complete.
On the other hand, Optunity is a particle swarm implementation to address  for the same task. I'm not overwhelmingly familiar with PSO, but it seems like it must be less efficient in the sense of requiring a larger number of trial points, and therefore objective function evaluations, to assess the hyperparameter surface.
Am I missing a key detail that makes PSO preferred to BO in the machine learning context? Or is the choice between the two always inherently contextual for the hyperparameter tuning task?

(1) Shahriari et al, "Taking the Human out of the Loop: A Review of Bayesian Optimizaiton."
Comment: Apologies for my late arrival to the party, not sure how I managed to overlook a question about Optunity for so long! :-)
Comment: Not well enough educated on the topic to make this a definitive answer, but I would think Bayesian Optimization should suffer the same fate as most efficient optimizers with highly multi-modal problems (see: 95% of machine learning problems): it zeros in on the closest local minimum without "surveying" the global space. I think Particle Swarm would have better luck finding non-local minimums.
Comment: @MarcClaesen I must admit, I was hoping that you would find the time to reply at some point. Late or not, I think we're all glad that you have arrived.
Comment: @CliffAB I haven't observed that in my trials. Beyond my personal uses, BO is also used for precisely these ML hyperparameter tuning problems. Some versions of BO use overly-greedy acquisition functions so the nominally global optimizer tends to revert to a local optimizer. But better acquisition functions will inherently trade off between exploitation and exploration in a way that can provably have zero cumulative regret!
Comment: Right. I think it would be very problem-dependent: if the hyper-parameter space is uni-modal, has a hyper-plane of equivalence, or a very large domain of attraction to the global minimum, then I can't think of a reason for particle swarm outside of ease of implementation (which definitively could be the real answer for its popularity). However, for problems that are highly multi-modal, I could see utility in the particle swarm.
Comment: doesn't need gradient.  works with discontinuity.  moderately efficient.  handles several dimensions.  handles noise well.  Has built-in robustness of estimator.
Comment: @EngrStudent You can say all of those things about BO, except BO appears to be **more efficient** because it requires a smaller number of function evaluation, at least in my reckoning. I'm not asking about PSO in general, I'm asking about its merits relative to BO.
Comment: They might be related.  I wonder if PSO is a heuristic approximation of BO.    I don't know BO at all.  With other Bayes methods you need a non-informative prior but without ephemeris that usually comes from 10k+ samples and Metropolis-Hastings.  In that sort of comparison PSO might be competitive.
Comment: PSO is consistent with "Algorithm 1" in your reference with the caveat that it is an ensemble, and that selection of the maximum per particle is weighted with previous position of the particle and its pseudo-momentum.
Answer: As the lead developer of Optunity I'll add my two cents.
We have done extensive benchmarks comparing Optunity with the most popular Bayesian solvers (e.g., hyperopt, SMAC, bayesopt) on real-world problems, and the results indicate that PSO is in fact not less efficient in many practical cases. In our benchmark, which consists of tuning SVM classifiers on various datasets, Optunity is actually more efficient than hyperopt and SMAC, but slightly less efficient than BayesOpt. I would love to share the results here, but I'm going to wait until Optunity is finally published in JMLR (under review for over a year now, so don't hold your breath ...).
As you indicate, increased efficiency is a commonly used selling point for Bayesian optimization, but in practice it only holds water if the assumptions of the underlying surrogate models hold, which is far from trivial. In our experiments, Optunity's very simple PSO solver is often competitive with complex Bayesian approaches in terms of number of function evaluations. Bayesian solvers work very well when provided with good priors, but with an uninformative prior there is virtually no structural benefit over metaheuristic methods like PSO in terms of efficiency.
A big selling point for PSO is the fact it's embarassingly parallel. Bayesian optimization is often hard to parallelize, due to its inherently sequential nature (hyperopt's implementation being the only real exception). Given opportunities to distribute, which is becoming the norm, Optunity quickly takes the lead in wall-clock time to obtain good solutions.
Another key difference between Optunity and most other dedicated hyperparameter optimization libraries is the target audience: Optunity has the simplest interface and is targetted towards non-machine learning experts, whereas most other libraries require some understanding of Bayesian optimization to use effectively (i.e., they are targetted towards specialists).
The reason we made the library is that despite the fact dedicated hyperparameter optimization methods exist, they lack adoption in practice. Most people are still either not tuning at all, doing it manually, or via naive approaches like grid or random search. In our opinion, a key reason for this is the fact that existing libraries prior to developing Optunity were too difficult to use in terms of installation, documentation, API and often limited to a single environment.
Comment: As informed an answer as we could get! I'm curious: you say PSO solver is competitive with Bayesian Optimization approaches. Is that to say that PSO *run in parallel* is found to be faster than Bayseian Optimization *run sequentially*? Not trying to be mean, but it's an important distinction for me to understand.
Comment: +1 for an excellent answer. I've built my own BO software, which I must admit is mostly a vanity project at this point, so I understand how the BO procedure works in some detail; I'm glad that I can start to scratch the surface of what else is going on the hyperparameter tuning world. Your remark about naive approaches really hits home with me, as one of my older naive tuning programs has been tuning a model for a week now with no end in sight... Thanks for your contribution, and I'm sure I'll have more questions once I digest this.
Comment: I think there are several reasons. For one, hyperparameter surfaces have a lot of local optima (e.g., due to finite sample effects, cross-validation folds, inherent randomness in some learning approaches). Secondly, Bayesian optimization relies on building accurate surrogate objective functions, which is not an easy task until the objective function has been sampled plenty of times. Bayesian optimization takes a while before convergence speeds up (an often omitted detail). By that time metaheuristic methods like PSO have reached their local search phase too. PSO is *very* good at local search.
Comment: Nope, both running sequentially. In our experiments (tuning SVMs), the efficiency of PSO and Bayesian optimization is competitive in terms of number of function evaluations. We didn't compare efficiency in terms of wall-clock time in distributed settings as that would be a bit of a cheap shot since many Bayesian optimization methods simply can't do that.
Comment: That's interesting. Any thoughts as to why? Unstable hyper-parameter surface?
Comment: Interesting. I don't know enough about Bayesian Optimization to know how well it handles lots of noise in the response space: since many ML techniques, even conditional on the fixed set of tuning parameters, don't really find a global min, I wonder if this "optimization noise" is more detrimental to BO than to PSO; i.e. if you can only calculate $f(x) + \epsilon$ rather than $f(x)$, it's *much* harder to get a reasonable estimate of the derivative from finite differences, for example. Here $\epsilon$ is the error associated with failing to find the global minimum.
Comment: (+1) This is quite surprising. How many function evaluations did you use in your benchmark? (ballpark)
Do any of the BO algorithms that you compared implement: (a) full marginalization of hyper-parameters; (b) input and output warping; (c) a portfolio of acquisition functions and/or (c2) predictive entropy search (PES)? A key aspect for BO to work is to take into account uncertainty properly, especially in early stages, via full marginalization. Also, warping can help with nonstationarity (which is a big problem for traditional BO), and "simple" acquisition functions can be too greedy.
Comment: I agree that naive (and also many not-so-naive) implementations of BO can have all sorts of problems and require a lot of fiddling, but I still wonder how PSO would perform against the state of the art of BO. (Of course, with the caveat that BO would be *much* more computationally expensive, and with all the problems that you mentioned.)
Comment: @lacerbi We compared with state-of-the-art hyperparameter optimization libraries that use Bayesian techniques uner the hood (hyperopt, bayesopt and SMAC).
Comment: @Marc: Thanks. However, I asked specifically for the presence of some features that substantially affect the efficiency of BO. I find quite hard to believe that state-of-the-art BO is beaten by PSO in terms of serial efficiency. I may agree that PSO beats BO in terms of usability, parallelization or other properties you mentioned, and that would be good enough for your goal -- I am just wondering about raw efficiency here (which I understand it is not what the OP asked, it's for my personal understanding).
Comment: @Marc: I don't know these libraries in detail, but a quick look at the user reference suggests that none of these libraries implement points (a), (b), or (c2) that I mentioned above, or at least not in their basic configuration (but maybe you used non-standard configurations, or I just missed them in the manual, in which case please correct me). So I wouldn't say that they implement the state-of-the-art of BO. Spearmint perhaps is the package that implements most of these advanced features (although it might be lacking in other aspects, and they might be scattered in different branches).
Comment: @lacerbi the aforementioned libraries are state-of-the-art for hyperparameter optimization. Spearmint may have better optimization algorithms (though I don't know of any clear-cut benchmarks demonstrating this in papers), but it has slightly fallen out of use for hyperparameter optimization. We initially wanted to compare against Spearmint in our own benchmark, but it was agonizingly slow, so we stopped using it.
Comment: @Marc: Well, full-BO is admittedly for very costly target functions (e.g., hours or more), so I am not sure of how or what you're comparing (e.g., before I asked how many function evaluations you were considering). I suppose I'll have to wait for your paper to come out to know the details. I am very interested in knowing the full extent of efficacy of BO; I agree with you that an extensive benchmark is missing (most NIPS papers only show a few examples which you can only hope they were not cherry-picked); for this it's also important to know which flavour of BO is being considered.
Answer: The answer is problem-dependent and cannot be given without additional context.
Typically, the answer would go as follows. Bayesian Optimization is more suitable for low-dimensional problems with the computational budget up to say 10x-100x the number of variables. PSO can be quite efficient for much larger budgets but is not state-of-the-art in its niche.
Comment: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/83006/discussion-on-answer-by-indiesolver-advantages-of-particle-swarm-optimization-ov).