Can one formulate a optimization problem to choose a good step size for gradient descent?

Question: My main goal is to understand how to choose a good step size for gradient descent.
I was trying to understand how to choose a good step size for gradient descent. For this I came across the following (which I will change the notation from the original):

If you want a completely automatic and optimal method, you can just
  take a derivative with respect to the learning rate, and solve for
  when it becomes 0.   Namely, you want to have some update
  $w_{t+1}=w_t−\eta ∇L(w_t)$ for your cost/loss function $L$.  So what you
  can do is take $R( \eta)=L(w_t−\eta ∇L(w_t))$ and find the optimal $\eta$ by
  setting $\frac{ \partial R }{ \partial \eta} = 0$ and solving for $\eta$.
  If solving is too hard, a line search could do as well.

However, I didn't understand conceptually what the optimization problem $R(\eta)$ is trying to achieve. I understand that one can define such an objective function and optimize it, however, it didn't make sense to me why that optimization problem should yield a good step size to choose for gradient descent.
Can someone explain to me intuitively what optimizing for $R$ is trying to do?
In a summary I'd like to address:

Why does $R( \eta)=L(w_t−\eta ∇L(w_t))$ make sense?
How does one choose the loss function for the second optimization problem. Why does it have to be the same loss function as the one used in the original problem?

To see if I understood this, I tried this on a simple example. The example I will try is Kernel function with the squared loss function. Thus:
$$L(y_n,f(x_n)) = (y_n - f(x_n))^2$$
and
$$ f(x_n) = \sum^{N}_{i=1} c_i K(x_n, x_i) = \langle K_{x_n} , c \rangle = K_{x_n}^Tc$$
However, with these definitions of Loss and function, it didn't quite make sense to simply substitute $w_t−\eta ∇L(w_t)$ into $L$. I guess one could re-write $L$ to be only a function of $c$ as follows:
$$L(c) = (y_n - \langle K_{x_n} , c \rangle)^2$$
then:
$$ L(c) = L(w_t−\eta ∇L(w_t)) = (y_n - \langle K_{x_n} , w_t−\eta ∇L(w_t) \rangle)^2 $$
However, this formulation seemed a bit strange and wasn't sure if it was correct.
Comment: Thank you for the clarifying comments.  Why not pick a step that takes you as high as possible, and no further?  That's what this quotation describes.
Comment: The quoted text--about using a line search--appears to have no relationship to choosing a step size.  This makes it difficult to understand what you're asking. Are you trying to understand what a line search is?
Comment: @whuber I am trying to understand how the formulation for $R( \eta )$ makes sense when choosing a step size for gradient descent. My goal is to choose a "good" step size for gradient descent. In fact, its to understand why the quoted text makes sense for choosing a step size for gradient descent. (Btw, I am not trying to understand what a line search is, I am trying to understand what $R( \eta )$ means).
Comment: I'm not quite with you, perhaps because the connection is not (yet) explicit.  Is the claim you are referring to that (any) (approximate) solution $\eta$ to $dR/d\eta=0$ would be a good step size to choose?
Comment: If I understand you correctly, yes. i.e. What I mean is, why is the formulation of $R$ a good one to choose a step size and why does that solution to it make sense to use for GD. Does it make sense now?
Comment: I guess I follow--but your questions are strange. Consider this (excellent) analogy: walking to the highest point in a terrain. The quotation suggests heading straight in the direction at which your current location is ascending most steeply and stopping when you can go no higher. You ask (1) why this procedure makes sense--but doesn't it obviously increase your elevation as much as possible, once you decide to set out in a given direction? Then you ask (2) "why do you still use height (the loss) to decide when to stop?" It would be bizarre to use anything else!
Comment: @whuber I am not asking why to go in the direction of the gradient (nor why the update rule for gradient descent makes sense, $w_t = w_t - \eta \bigtriangledown_w L(w_t)$ makes sense as an update rule), I understand that and I understand directional derivatives too. What I don't understand is the second optimization problem for choosing the step size, it seems to come out of nowhere (at least for me). Following your analogy, what I am interested in understanding is how to decide how big the step sizes one should take when going up in elevation (for each step that one takes).
Comment: @whuber thanks! That makes sense finally. Though, I have a small follow up. Do you know if the above method was computable exactly, if we could reach a local optimum in 1 single step using steepest descent?
Comment: I'm unsure about your question, Charlie, but if you are asking whether it's possible to move the right distance in a given direction in one step, without a numerical search (as described in the quotation), the answer would have to be that it depends on the objective function.  If it's given in the form of a formula, then what we have is a standard one-variable optimization problem, exactly the kind everyone learns to solve in a Calculus course.  If the formula is amenable to that method of solution, then you could find the step size that way.
Comment: @whuber yes what you said makes sense. Though, I guess what I meant was more of a purely theoretical thought experiment. Given a black box that computes the optimal step size (for any function), can we use the optimal step-size to reach the local minimum in 1 single step? Or do we have to always do at least more than one step, even if we knew the perfect optimal step size described in my question.
Answer: The book that I learned gradient descent from (Numerical Analysis by Burden and Faires) describes this as the method for finding the learning rate, so it seems odd to me when people describe other ways of finding a learning rate, such as keeping it constant.
Anyways, the idea is that you are taking a step in the direction of the derivative, i.e.
$w_{t+1} = w_{t} + \alpha ∇L(w_t)$
The question is how to chose the scalar value $\alpha$. Well, the point is that we can just see the choice of $\alpha$ as an optimization problem and chose the value of $\alpha$ that maximizes (or at least increase) $L(w_t + \alpha ∇L(w_t)$).
Note that if $w_{t+1} = w_{t} + \alpha ∇L(w_t)$, and $\alpha$ is the value that maximizes $L(w_t + \alpha ∇L(w_t))$, then clearly $L(w_{t+1}) \geq L(w_t)$.
I think what may seem circular is that it seems really backwards to say "the way we figure out how to do this step of the optimization problem is to see it as another optimization problem". But the key is that we see it as a much simpler optimization problem. It is generally much easier to optimize a scalar problem (i.e. find optimal choice of $\alpha$) than to optimize the multivariate problem (i.e. find optimal choice of $w_{t+1}$, where $w$ is high dimensional). As they state, worst come to worst, you can use a simple line search to find an optimal value of $\alpha$, which is clearly less computationally intense than a high dimensional grid search to find an optimal value of $w$.
Comment: I think their discussion of "directional derivatives" may be helpful for you.
Comment: I understand $L(w_t + \alpha ∇L(w_t)$ as a statement. Whether its sound to do is the question I am sort of more interested in. I do not understand why solving such an equation makes sense. What is the justification for doing that? Maybe a reference or a specific chapter of the book you quoted might explain it? i.e. why is it **the** method for finding the learning rate?
Comment: The point is that you are now minimizing over a scalar parameter ($\alpha$) rather than a (potentially high dimensional) vector ($w$). Univariate optimization will be an easier problem than multivariate.
Comment: In terms of the book, gradient descent is discussed in section 10.4 (8th edition).
Comment: As for being "**the**" method of finding $\alpha$, you've chosen the $\alpha$ that optimizes your objective function. Given that your goal is to optimize your objective function, I can't see any reason for questioning that choice of $\alpha$, other than choosing a constant and *hoping* it works, to slightly reduce computation. But I think my wording demonstrates why I don't think that makes much sense.
Comment: I do understand that a way to choose $\alpha$ is via a different optimization problem.What I do not understand is why if our original goal was to optimize our original objective function (for example, in the empirical risk minimization the original objective function would be $\sum^N_{n=1}L(y_n, f(x_n) )$),then to do so, we'd need to choose the step size via $R(\alpha) = L( w_t - \bigtriangledown_w L(w_t) )$.The connection between the *original* optimization goal (i.e. the one we are doing GD on) and the new formulation to choose $\eta$ is what I don't see. (btw, whats the name of the chapter)
Comment: Chapter 10: Numerical Solutions of Nonlinear Systems of Equations Section 4: Steepest Descent Methods
Comment: The update rule $w_t - \eta \bigtriangledown L(w_t) $, makes sense to me and I understand that the gradient points in the steepest ascent/descent. Thats not what is confusing me. What is confusing me, is where the second optimization problem came from and its relation in minimizing the original problem. I am confused on how to choose the step size when following gradient descent. I am not confused about why moving in the steepest direction minimizes the original objective. That part makes sense to me. Its the second for deriving the step size that seems to come out of nowhere (at least to me).
Comment: are you familiar with using this update rule but for stochastic gradient descent? i.e. if we know the function we want to optimize is a summation of other loss functions, then maybe taking the steepest step with respect to the change of one loss with respect to one data point could be good. Are you familiar with this?
Comment: @CharlieParker: I'm familiar with stochastic gradient descent. If you only calculate a single contribution to the objective function, you will have lots of problems.
Comment: What do you mean you will have lots of problems? Did you mean that you will have lots of problems if you use the method suggested in this question as the step size adjustment, or just using SGD in general? Or both?
Comment: I was reading a new result that claims that "Greedy methods that use precise line search may still get stuck at stationary points." Thought that you might find it interesting: http://arxiv.org/pdf/1602.04915v2.pdf
Comment: Do you know if the above method was computable exactly, if we could reach a local optimum in 1 single step using steepest descent?
Answer: Notice that the goal is to reach to some minim of $g(x)$. i.e. we want:
$$ \min_{x \in R^n} g(x)$$
notice that $x$ might be a multivariable vector.
Since we have no better way to reach a minimum, lets follow the direction of greatest change towards some minimum. Thus the update rule:
$$ x^{(1)} := x^{(0)} - \alpha \nabla g(x^{(0)})$$
However, now we are left with choosing a good step size $\alpha$. Unfortunately, at any point of this algorithm there are only two things that we know, our current guess $g(x^{(0)})$ and the direction of greatest change $\nabla g(x^{(0)})$. If we could, we'd want to choose $x^{(1)}$ such that it corresponds to a minimum i.e. it would be nice to have:
$$ \min_{x \in R^n} g(x) = \min_{x^{(1)} \in X^{(1)} } g(x^{(1)}) $$
where $X^{(1)}$ denotes the set of candidate $x^{(1)}$ that we could potentially choose (based on the two things that we know). It would be awesome if we could reach the minimum like this because it means that we choose a new $x^{(1)}$ such that its at the (local) minimum that we are looking for. Unfortunately, we are restricted to the previous estimate ( i.e. $x^{(0)}$) that we had. So in reality, we have:
$$ \min_{x^{(1)} \in X^{(1)} } g(x^{(1)}) \iff \min_{\alpha \in R } g(x^{(1)}), \text{ subject to } x^{(0)} - \alpha \nabla g(x^{(0)}) $$
Thus these two optimization problems are equivalent!
At this point the key aspect to note is that our original problem is $ \min_{x \in R^n} g(x)$ and we replaced it by choosing a multivariable vector in a restricted set. How is it restricted? We are only allowed to choose $x^{(1)}$ based on the gradient and our previous estimate. So it makes sense to choose $\alpha$ (and hence $x^{(1)}$ such that we minimize $g(x)$ as much as possible. Since, that was our goal the whole time anyway! Our goal is to choose the smallest $g(x)$, so at each step, we are making sure that we choose an $\alpha$ such that we minimize our original goal $g(x)$ as much as possible.

Let's address the following points:

why is that the optimal value of $\alpha$? ( i.e. the solution to the minimization of $h( \alpha )$?)
How is "optimality" value of alpha even mean in this case and why does minimizing $h( \alpha)$ capture this definition of optimality? Maybe that is the choice of alpha that gives the biggest step size possible given the computed gradient?
What is the proof or the rigorous justification that this equation is the right one to optimize to choose the step size? Why is that the correct equation and not something else? Is there a rigorous why to explain why that is the best alpha?
What type of guarantee's does that choice of $\alpha$ give us? If we choose that $alpha$, are we guaranteed that if we keep doing iterations of steepest descent, that we will never overshoot? Will it be in an infinite number of iterations or finite iterations?
Does this mean that we choose a new step size for every step of Steepest descent?

My Response:

The reason that $min_{\alpha} h(\alpha)$ is the optimal value of $\alpha$ is because, we are choosing the step size that makes sure we minimize our objective $g(x)$ as much as possible, given that we only know our previous estimate of the minimum and the direction greatest change. So we choose an alpha such that $min g(x^{(1)})$ is satisfied.
Optimality in this case means that we choose a step size such that we decrease our function as much as possible (given the information that we have, i.e. we only know its previous estimate and the direction of steepest change).
The justification is that given the restrictions to the problem, we cannot do better than choose the next $x^{(1)}$ as close to our real objective $min g(x)$ as possible.
If you notice $\min_{\alpha \in R } g(x^{(1)}), \text{ subject to } x^{(0)} - \alpha \nabla g(x^{(0)}) $ is of the same form as our real objective $min g(x)$. Thus, a solution to $\min_{\alpha \in R } g(x^{(1)}), \text{ subject to } x^{(0)} - \alpha \nabla g(x^{(0)}) $ will only aim to minimize the original objective function $g$ subject to the constraints. The really nice property of this is that if the optimal solution $x^*$ is ever in the restricted set $X^{(1)}$, the solution will be chosen. Also notice that we are choosing a new step size based on our previous estimate. So we will only choose a step size that minimizes our current guess....why? Because thats how we set up the problem, look:

$$\min_{\alpha \in R } g(x^{(1)}), \text{ subject to } x^{(0)} - \alpha \nabla g(x^{(0)}) $$
Choose a step size only if it improves our current solution, is a way of re-phrasing. So its key that in this second optimization problem that we choose the same objective function to optimize over $x^{(1)}$. If we choose some other $g'$ we would have:
$$\min_{\alpha \in R } g'(x^{(1)}), \text{ subject to } x^{(0)} - \alpha \nabla g(x^{(0)}) $$
which doesn't necessarily improve our real objective of minimizing $g$.

Yes, you need to choose a new step size and solve the optimization problem at each step of the algorithm.

I never quite explicitly said this but:
$$ X^{(1)} = \{ x^{(1)} \in R^n \mid x^{(0)} - \alpha \nabla g(x^{(0)}), \alpha \in R  \}$$
so we are only considering points in the hyperplane described by the vector $\nabla g(x^{(0)})$ and the point to "reach" the plane $x^{(0)}$.
Answer: A no-nonsense answer that works is:
Try setting it to something big, like 1. If the optimization works, celebrate, else try for instance 0.5. Again, if it works, great, else try 0.25 and so on. In practice you get faster convergence with bigger values of $\eta$, when they work. If they are too big, the optimization diverges, so you need to try a smaller one. Monitor the target function value and see if it drops steadily (minimization). All the nice theory comes down to something simple like this in the end.
Comment: This is an extremely slow way to do things. As I noted in my answer, I can't imagine any good reason for doing it this way.
Comment: unfortunately, this method doesn't really work when you want to do cross validation or something like that because you have to run lots of GD to train multiple models. Its quintessential for me at least to choose a step size automatically that makes sense. The less human intervention the better. Thanks for the suggestion though.
Comment: True! However, you can run a few tests without CV. Just for some parameters, and when it works try that.


Fork não restaura as variáveis

Question: A idéia do código é:
Dar fork 3 vezes,ou seja, a partir do pai criar três filhos(com count=0,count=1 e count=2).
Os filhos então,ao voltar do loop vão ver que seu i==0 e vão sair do loop.
Ao entrar na outra condição,eles(os filhos,pois i==0),ao printar o count deveria aparecer 0,1,2,mas está aparecendo 0,0,0,depois dar um exec passando:
1-o caminho com o nome do executável.
2-Qual o número do filho(0,1,2).
Tenho o programa pai:...
 i = 1;
for( count = 0; count < 3; count++) {
        if( i != 0 ) {
            i = fork();
        } else {
            break;
        }
    }

    if( i == 0 ) {
        printf("%d",count);
        char arg[10];
        sprintf(arg, "%d", count);
        execl("/home/Downloads/filho",arg,NULL);

....
O meu ouuput do printf esperado era:

0,1,2

Mas está saindo:

0,0,0

EDIT-----
Talvez no filho eu estou pegando o valor de maneira errada:
Eu estou passando assim:
char arg[10];
            sprintf(arg, "%d", count);
            execl("/home/Downloads/filho",arg,NULL);

E no executavel filho:
int n = atoi(argv[1]);

Porém dessa forma ele gera um problema que não aparece nenhum resultado no executavel filho.
e caso coloco assim:
int n = atoi(argv[2]);

Ele printa os resultados do executavel filho,mas de maneira errada.
filho.c:
int main( int argc, char *argv[] )
{
        int n = atoi(argv[2]);

        printf("Filho #%d,n);
    exit(0);
}

Output esperado:

Filho #1 Filho #2 Filho #3

Output quando sai:

Filho #0 Filho #0 Filho #0
Comment: http://pt.stackoverflow.com/help/mcve
Comment: Pelo trecho postado, não consigo entender o que está acontecendo. Cada filho vai tentar o `fork`, fazer `i` virar `0` e então sair do loop - com `count` igual a `1`. A saída então deveria ser `1,1,1`, e não `0,0,0`.
Comment: OK,era isso mesmo,1,2,3 erro meu também,enfim,não funciona :/
Comment: [Experimentei no ideone](http://ideone.com/UwuHNp) e o resultado foi como o esperado (fora de ordem, é claro). O problema deve estar em algum outro lugar...
Comment: Uma coisa de cada vez... O seu resultado no processo filho está consistente com o que está acontecendo com o `count`. Se esse problema for resolvido, creio que o filho vai funcionar corretamente.
Comment: Como você definiu `i`?
Comment: Só com este trecho fica difícil entender o que está acontecendo. E como ele parece ser fazer muito esquisito, seria bom dizer qual é o propósito dele.
Comment: i está inicializado com 1;A idéia é dar um execl no processo filho,passando como parâmetro o valor do loop em que foi dado o fork.Por exemplo:no primeiro loop count=0 então queria passar arg=0 e assim por diante
Comment: @bigown Editado
Comment: @mgibsonbr Expliquei o que eu acho que o código deveria fazer.
Comment: @felipealves Desculpe, ignorância minha, o `fork` [não funciona como eu achava que funcionava...](http://stackoverflow.com/q/11288226/520779) Ainda assim, a saída esperada deveria ser `1,2,3` - e não `0,1,2` - pois o `count` de cada processo filho é incrementado pelo menos uma vez.
Comment: Esta faltando uma chave no ultimo if?
Comment: @mgibsonbr Editei com outro possível problema
Answer: Tu não estás a passar o argv[1] para o filho
execl("/home/Downloads/filho", arg, NULL); // NULL vai para argv[1]
//    path                     argv[0]

devia ser
execl("/home/Downloads/filho", "nome do filho", arg, NULL); // NULL vai para argv[2]
//    path                     argv[0]          argv[1]

Fork não restaura as variáveis


xampp Apache not running

Question: few weeks before my xamppp was working properly.
but today when i started it ....it failed to open its phpmyadmin page..
so i installed it again.
now its apache is not running.
another webserver is running??? whats that??
enter image description here
the above image shows the error.
although apache was not running ...i tried start localhost.
and this cameup.
enter image description here
Answer: got my error.
port 80 was occupied by apache2.
removed apache2 from port 80 and it started woorking again.

xampp Apache not running


"changed by" on nodes

Question: mysql> select nid, uid, created, changed from node;
+-----+-----+------------+------------+
| nid | uid | created    | changed    |
+-----+-----+------------+------------+
|   1 |   8 | 1336040166 | 1336046390 |
+-----+-----+------------+------------+

I'd like to have an "changed by" column on the node table, exactly as we have a "created by" (the uid field). This would keep track of who made the last change on that node. I know that this could be derived from the node_revision table but this depends on the revisions being enabled for the content types I'm interested.
So, what is the best way to do this? And why doesn't Drupal core offer this by default? I thought that "changed by" is a pretty standard piece of information that a CMS should attach to content.
Comment: Is there a reason you can't enable revisioning?  Seems like the easiest way to get what you need.  It's probably what I'd do.  If people are going to be editing nodes frequently, it also means you have a backup of previous versions.
Comment: Yes I can. I'd like to know if it's possible to have it on the main `node` table though. It looks more straightforward.
Answer: I thought this would be quite tough to do, but as it turns out it's fairly easy.
You just need to create a custom module that adds a column to the node table on install, implement hook_schema_alter() so Drupal knows about the new column, and add some logic to provide a value before the node is saved.
Here's a small module that will do the trick:
File: node_table_alter.info
name = Node Table Alter
core = 7.x

File: node_table_alter.install
function node_table_alter_install() {
  // Add the new field to the node table
  $field = array(
    'description' => 'Stores the user id of the last user to alter the node',
    'type' => 'int',
    'unsigned' => TRUE
  );

  db_add_field('node', 'changed_by', $field);
}

File: node_table_alter.module
function node_table_alter_schema_alter(&$schema) {
  // Add the new field to the schema cache
  $schema['node']['fields']['changed_by'] = array(
    'description' => 'Stores the user id of the last user to alter the node',
    'type' => 'int',
    'unsigned' => TRUE
  );
}

function node_table_alter_node_presave($node) {
  // Populate the changed_by column with current user's id
  $node->changed_by = $GLOBALS['user']->uid;
}

You may want to add logic to remove the field again on uninstall, and add an index to the table for the changed_by column (see db_add_index()), but this should give you a good place to start.
The beauty of this method is that you have effectively added a new property to the node. You'll be able to use node_load(), EntityFieldQuerys, etc. with it as if it were any of the other standard properties for a node.
God bless Drupal for being so extendable!
Comment: For complete entity integration, and if your are using the Entity API module, you will also need to implement hook_entity_property_info() to pvoide informatiom about this new property.
Comment: This is exactly what the UUID module does. Check it out for a more complete implementation of something similar. http://drupal.org/project/uuid
Comment: BTW you can use exactly the same logic to answer [your other question](http://drupal.stackexchange.com/questions/30086/custom-node-status).
Comment: @PierreBuyle Good point, didn't think of that
Comment: Thanks a lot for the detailed explanation and clean solution!
Answer: I guess you could add a entity reference field (let's call it field_changed_by_user) to the content type you need to track. Then you could use hook_node_presave to save the user id to the node like this:
function hook_node_presave($node) {
  if ($node->nid && $node->type == 'content_type_to_track_changes_for') {
    global $user;
    $node->field_changed_by_user['und'][0]['target_id'] = $user->uid;
  }
}

I think it's also possible to update the field with the user's id just by creating a rule. You can read more here.

"changed by" on nodes


PyMC for nonparametric clustering: Dirichlet process to estimate Gaussian mixture's parameters fails to cluster

Question: Problem setup
One of the first toy problems I wanted to apply PyMC to is nonparametric clustering: given some data, model it as a Gaussian mixture, and learn the number of clusters and each cluster's mean and covariance. Most of what I know about this method comes from video lectures by Michael Jordan and Yee Whye Teh, circa 2007 (before sparsity became the rage), and the last couple of days' reading Dr Fonnesbeck's and E. Chen's tutorials [fn1], [fn2]. But the problem is well-studied and has some reliable implementations [fn3].
In this toy problem, I generate ten draws from a one-dimensional Gaussian $\mathcal{N}(\mu=0, \sigma=1)$ and forty draws from $\mathcal{N}(\mu=4, \sigma=2)$. As you can see below, I didn't shuffle the draws, to make it easy to tell which samples came from which mixture component.

I model each data sample $y_i \sim \mathcal{N}(\mu_{z_i}, \sigma_{z_i})$, for $i=1,...,50$ and where $z_i$ indicates the cluster for this $i$th data point: $z_i \in [1,...,N_{DP}]$. $N_{DP}$ here is the length of the truncated Dirichlet process used: for me, $N_{DP}=50$.
Expanding the Dirichlet process infrastructure, each $z_i$ cluster ID is a draw from a categorical random variable, whose probability mass function is given by the stick-breaking construct: $z_i \sim Categorical(p)$ with $p \sim Stick(\alpha)$ for a concentration parameter $\alpha$. Stick-breaking constructs the $N_{DP}$-long vector $p$, which must sum to 1, by first obtaining $N_{DP}$ i.i.d. Beta-distributed draws that depend on $\alpha$, see [fn1]. And since I'd like to the data to inform my ignorance of $\alpha$, I follow [fn1] and assume $\alpha \sim Uniform(0.3, 100)$.
This specifies how each data sample's cluster ID is generated. Each of the $N_{DP}$ clusters has an associated mean and standard deviation, $\mu_{z_i}$ and $\sigma_{z_i}$. Then, $\mu_{z_i} \sim \mathcal{N}(\mu=0, \sigma=50)$ and $\sigma_{z_i} \sim Uniform(0, 100)$.
(I was previously following [fn1] unthinkingly and placing a hyperprior on $\mu_{z_i}$, that is, $\mu_{z_i} \sim \mathcal{N}(\mu_0, \sigma_0)$ with $\mu_0$ itself a draw from a fixed-parameter normal distribution, and $\sigma_0$ from a uniform. But per https://stats.stackexchange.com/a/71932/31187, my data doesn't support this kind of hierarchical hyperprior.)
In summary, my model is:
$y_i \sim \mathcal{N}(\mu_{z_i}, \sigma_{z_i})$ where $i$ runs from 1 to 50 (the number of data samples).
$z_i \sim Categorical(p)$ and can take on values between 0 and $N_{DP}-1=49$;  $p \sim Stick(\alpha)$, a $N_{DP}$-long vector; and $\alpha \sim Uniform(0.3, 100)$, a scalar. (I now slightly regret making the number of data samples equal to the truncated length of the Dirichlet prior, but I hope it's clear.)
$\mu_{z_i} \sim \mathcal{N}(\mu=0, \sigma=50)$ and $\sigma_{z_i} \sim Uniform(0, 100)$. There's $N_{DP}$ of these means and standard deviations (one for each of the $N_{DP}$ possible clusters.)
Here's the graphical model: the names are variable names, see the code section below.

Problem statement
Despite several tweaks and failed fixes, the parameters learned are not at all similar to the true values that generated the data.
Currently, I'm initializing most of the random variables to fixed values. The mean and standard deviation variables are initialized to their expected values (i.e., 0 for the normal ones, the middle of their support for the uniform ones). I initialize all $z_i$ cluster IDs to 0. And I initialize the concentration parameter $\alpha=5$.
With such initializations, 100'000 MCMC iterations simply can't find a second cluster. The first element of $p$ is close to 1, and nearly all draws of $\mu_{z_i}$ for all data samples $i$ are the same, around 3.5. I show every 100th draw here for the first twenty data samples, i.e., $\mu_{z_i}$ for $i=1,...,20$:

Recalling that the first ten data samples were from one mode and the rest were from the other, the above result clearly fails to capture that.
If I allow random initialization of the cluster IDs, then I obtain more than one cluster but the cluster means all wander around the same 3.5 level:

This suggests to me that it's the usual problem with MCMC, that it can't reach another mode of the posterior from the one it's at: recall that these different results happen after just changing the initialization of the cluster IDs $z_i$, not their priors or anything else.
Am I making any modeling mistakes? Similar question: https://stackoverflow.com/q/19114790/500207 wants to use a Dirichlet distribution and fit a 3-element Gaussian mixture and is running into somewhat similar problems. Should I consider setting up a fully conjugate model and using Gibbs sampling for this kind of clustering? (I implemented a Gibbs sampler for the parametric Dirichlet distribution case, except using a fixed concentration $\alpha$, back in the day and it worked well, so expect PyMC to be able to solve at least that problem handily.)
Appendix: code
import pymc
import numpy as np

### Data generation

# Means and standard deviations of the Gaussian mixture model. The inference
# engine doesn't know these.
means = [0, 4.0]
stdevs = [1, 2.0]

# Rather than randomizing between the mixands, just specify how many
# to draw from each. This makes it really easy to know which draws
# came from which mixands (the first N1 from the first, the rest from
# the secon). The inference engine doesn't know about N1 and N2, only Ndata
N1 = 10
N2 = 40
Ndata = N1+N2

# Seed both the data generator RNG  as well as the global seed (for PyMC)
RNGseed = 123
np.random.seed(RNGseed)

def generate_data(draws_per_mixand):
    """Draw samples from a two-element Gaussian mixture reproducibly.

    Input sequence indicates the number of draws from each mixand. Resulting
    draws are concantenated together.

    """
    RNG = np.random.RandomState(RNGseed)
    values = np.hstack([RNG.normal(means[i], stdevs[i], ndraws)
                        for (i,ndraws) in enumerate(draws_per_mixand)])
    return values

observed_data = generate_data([N1, N2])

### PyMC model setup, step 1: the Dirichlet process and stick-breaking

# Truncation level of the Dirichlet process
Ndp = 50

# "alpha", or the concentration of the stick-breaking construction. There exists
# some interplay between choice of Ndp and concentration: a high concentration
# value implies many clusters, in turn implying low values for the leading
# elements of the probability mass function built by stick-breaking. Since we
# enforce the resulting PMF to sum to one, the probability of the last cluster
# might be then be set artificially high. This may interfere with the Dirichlet
# process' clustering ability.
#
# An example: if Ndp===4, and concentration high enough, stick-breaking might
# yield p===[.1, .1, .1, .7], which isn't desireable. You want to initialize
# concentration so that the last element of the PMF is less than or not much
# more than the a few of the previous ones. So you'd want to initialize at a
# smaller concentration to get something more like, say, p===[.35, .3, .25, .1].
#
# A thought: maybe we can avoid this interdependency by, rather than setting the
# final value of the PMF vector, scale the entire PMF vector to sum to 1? FIXME,
# TODO.
concinit = 5.0
conclo = 0.3
conchi = 100.0
concentration = pymc.Uniform('concentration', lower=conclo, upper=conchi,
                             value=concinit)

# The stick-breaking construction: requires Ndp beta draws dependent on the
# concentration, before the probability mass function is actually constructed.
betas = pymc.Beta('betas', alpha=1, beta=concentration, size=Ndp)

@pymc.deterministic
def pmf(betas=betas):
    "Construct a probability mass function for the truncated Dirichlet process"
    # prod = lambda x: np.exp(np.sum(np.log(x))) # Slow but more accurate(?)
    prod = np.prod
    value = map(lambda (i,u): u * prod(1.0 - betas[:i]), enumerate(betas))
    value[-1] = 1.0 - sum(value[:-1]) # force value to sum to 1
    return value

# The cluster assignments: each data point's estimated cluster ID.
# Remove idinit to allow clusterid to be randomly initialized:
idinit = np.zeros(Ndata, dtype=np.int64)
clusterid = pymc.Categorical('clusterid', p=pmf, size=Ndata, value=idinit)

### PyMC model setup, step 2: clusters' means and stdevs

# An individual data sample is drawn from a Gaussian, whose mean and stdev is
# what we're seeking.

# Hyperprior on clusters' means
mu0_mean = 0.0
mu0_std = 50.0
mu0_prec = 1.0/mu0_std**2
mu0_init = np.zeros(Ndp)
clustermean = pymc.Normal('clustermean', mu=mu0_mean, tau=mu0_prec,
                          size=Ndp, value=mu0_init)

# The cluster's stdev
clustersig_lo = 0.0
clustersig_hi = 100.0
clustersig_init = 50*np.ones(Ndp) # Again, don't really care?
clustersig = pymc.Uniform('clustersig', lower=clustersig_lo,
                          upper=clustersig_hi, size=Ndp, value=clustersig_init)
clusterprec = clustersig ** -2

### PyMC model setup, step 3: data

# So now we have means and stdevs for each of the Ndp clusters. We also have a
# probability mass function over all clusters, and a cluster ID indicating which
# cluster a particular data sample belongs to.

@pymc.deterministic
def data_cluster_mean(clusterid=clusterid, clustermean=clustermean):
    "Converts Ndata cluster IDs and Ndp cluster means to Ndata means."
    return clustermean[clusterid]

@pymc.deterministic
def data_cluster_prec(clusterid=clusterid, clusterprec=clusterprec):
    "Converts Ndata cluster IDs and Ndp cluster precs to Ndata precs."
    return clusterprec[clusterid]

data = pymc.Normal('data', mu=data_cluster_mean, tau=data_cluster_prec,
                   observed=True, value=observed_data)

References

fn1:  http://nbviewer.ipython.org/urls/raw.github.com/fonnesbeck/Bios366/master/notebooks/Section5_2-Dirichlet-Processes.ipynb
fn2: http://blog.echen.me/2012/03/20/infinite-mixture-models-with-nonparametric-bayes-and-the-dirichlet-process/
fn3: http://scikit-learn.org/stable/auto_examples/mixture/plot_gmm.html#example-mixture-plot-gmm-py
Comment: I wouldn't dismiss the hierarchical prior. It's well-known that you put yourself in a bad position if you put flat priors on the components of a mixture - *especially* when you are simultaneously attempting to learn the number of clusters. This is what is causing the weird spikes in your trace plots I think. The big names in NP-Bayes all seem to set $\mu_{Z_i} \sim N(\mu_0, \sigma_0)$ and either use plug-in estimates of $\mu_0, \sigma_0$ or by placing an *informative* prior on these components designed to put them on the appropriate scales.
Comment: Your prior over the component variances is Uniform(0,100) which could be causing you major problems. Only 2% of the mass of this prior covers the true variances of 1 and 2. The expected variance of your components under this prior is 50, which is such a broad Gaussian that it can easily account for your data with a single component.
Comment: Have you read through this chapter of the [Probabilistic Programming and Bayesian Statistics for Hackers](http://nbviewer.ipython.org/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Chapter3_MCMC/IntroMCMC.ipynb) book? It has an example that may help you!
Comment: This seems a little brief for an answer. It appears to be more of a comment. Could you at least outline what information the OP will obtain by reading it?
Comment: @TimRich yes I've read it, and taken the class in graduate school, and worked in industry doing applied statistics ;) this is a PyMC-specific question.
Answer: I'm not sure if anyone is looking at this question any more but I put your question in to rjags to test Tom's Gibbs sampling suggestion while incorporating insight from Guy about the flat prior for standard deviation.
This toy problem might be difficult because 10 and even 40 data points are not enough to estimate variance without an informative prior. The current prior σzi∼Uniform(0,100) is not informative. This might explain why nearly all draws of μzi are the expected mean of the two distributions. If it does not alter your question too much I will use 100 and 400 data points respectively.
I also did not use the stick breaking process directly in my code. The wikipedia page for the dirichlet process made me think p ~ Dir(a/k) would be ok.
Finally it is only a semi-parametric implementation since it still takes a number of clusters k. I don't know how to make an infinite mixture model in rjags.

library("rjags")

set1 <- rnorm(100, 0, 1)
set2 <- rnorm(400, 4, 1)
data <- c(set1, set2)

plot(data, type='l', col='blue', lwd=3,
     main='gaussian mixture model data',
     xlab='data sample #', ylab='data value')
points(data, col='blue')

cpd.model.str <- 'model {
  a ~ dunif(0.3, 100)
  for (i in 1:k){
    alpha[i] <- a/k
    mu[i] ~ dnorm(0.0, 0.001)
    sigma[i] ~ dunif(0, 100)
  }
  p[1:k] ~ ddirich(alpha[1:k])
  for (i in 1:n){
    z[i] ~ dcat(p)
    y[i] ~ dnorm(mu[z[i]], pow(sigma[z[i]], -2))
  }
}'

cpd.model <- jags.model(textConnection(cpd.model.str),
                        data=list(y=data,
                                  n=length(data),
                                  k=5))
update(cpd.model, 1000)
chain <- coda.samples(model = cpd.model, n.iter = 1000,
                      variable.names = c('p', 'mu', 'sigma'))
rchain <- as.matrix(chain)
apply(rchain, 2, mean)

Comment: With respect to using a finite number of clusters $K$, you can just take $K$ large (and set $\alpha_i = a / K$, as you have) and get a model which is effectively identical to the Dirichlet process. The convergence of this prior to the Dirichlet process prior is very rapid; for $500$ observations, something like $K = 25$ should be more than enough. It's also smart, when using `JAGS`, to **not** use the stick-breaking construction, because `JAGS` won't detect the block update on $p$; what you have done here is better.
Answer: The poor mixing that you are seeing is most likely because of the way that PyMC draws samples.  As explained in section 5.8.1 of the PyMC documentation, all elements of an array variable are updated together.  In your case, that means it will try to update the entire clustermean array in one step, and similarly for clusterid.  PyMC doesn't do Gibbs sampling; it does Metropolis where the proposal is chosen by some simple heuristics.  This makes it unlikely to propose a good value for an entire array.
Comment: STAN doesn't handle discrete variables, but JAGS is worth a try.
Comment: As soon as you said, "it will try to update the entire array in one step", I understood the drawbacks of Metropolis (in this instance) versus Gibbs. Is there anything special about STAN or JAGS that might let them do better on this? In either case, I am going to spend some time implementing Gibbs in PyMC. Thank you! (I've been a fan of your work since lightspeed, so double-thank-you!)


Hausdorff distance between convex sets in $\mathbb{R}^d$.

Question: I have the following property I suspect about the Hausdorff distance between convex sets in $\mathbb{R}^d$, and I thought that perhaps someone can find a fault with this idea if it is wrong.

Given nonempty convex compact sets $A,B\in \mathbb{R^d}$, their Hausdorff distance is given by $d_H(A,B)=\max \Big\{ \underset{a\in \partial A}{\sup} d(a,B), \underset{b\in \partial B}{\sup} d(b,A) \Big\}$. When $d=1$ and $A=[\alpha,\beta]$ and $B=[\gamma,\delta]$, then $d_H(A,B)=\max \big\{  \vert \alpha-\gamma\vert , \vert \alpha -\delta\vert , \vert\beta - \gamma \vert, \vert \beta- \delta \vert \big\}$.

While the Hausdorff distance is given by $d_H(A,B)= \max \Big\{ \underset{a\in A}{\sup}d(a,B), \underset{b\in B}{\sup} d(b,A)  \Big\}$. In the $1$-dimensional case this follows just by looking at all the cases for $a,b,c,d$. I think that in $\mathbb{R}^d$ this should follow by the hyperplane separation theorem and since $A=\partial A \sqcup int(A)$. For example if $a\notin B$ and $a\in int(A)$, then there is a hyperplane separating $\{ a \}$ and $B$. We then take a vector of length $\epsilon$ perpindicular to the separating hyperplane and the openess implies that there exists some $a'\in A$ such that $d(a',B)>d(a,B)$. Therefore, $\underset{a\in A}{\sup}d(a,B)=\underset{a\in \partial A}{\sup}d(a,B)$.
Does this argument seem correct? If not, does the statement seem correct?
Answer: Since I stumbled across the answer, I will say that the answer to this question is yes and one can strengthen it  to  a  general metric space by demanding closedness and boundedness instead of compactness. This is proven very clearly in this paper.
I hope that if someone else stumbles upon this, the reference I gave will help them.
Answer: If A and B are two bounded, closed, non-empty, and convex subsets of a normed space X, then the
Hausdorff distance between A and B is the same as the Hausdorff distance between the boundary of A
and the boundary of B.
You can go through this article. I think the problem is solved here.
https://www.heldermann-verlag.de/jca/jca14/jca0511_b.pdf
Comment: Thanks for the reply, but this  is actually the same paper I linked to in my short answer below.


New sort method added in List after using Collections.sort

Question: Why is a new sort method added in java.util.List in java 8 when we have a provision to sort lists using Collections.sort
Comment: @AbhishekEkaanth what do you want more exactly ? OP can't give you an explanation on a theorical question.
Comment: @Shuchita check this question answer https://stackoverflow.com/questions/34910841/difference-between-collection-sortlist-and-list-sort
Comment: your question is same as your explanation, please improve your explanation.
Comment: @Kailas Thanks for providing this link
Answer:
because it makes the API more intuitive and OO
because it allows implementations of List to use a faster sorting algorithm, best suited to their internal structure. For example, ArrayList can sort its internal array without first doing a copy as the default implementation does.
Comment: Overriding methods may not only be faster, they may enforce additional guarantees. E.g. before Java 8, applying `Collections.sort` to a `synchronizeList` or `Vector` was not safe without additional manual synchronization. Today, it delegates to `List.sort` and these collections override it with a `synchronized` method… Further, unmodifiable lists can throw an `UnsupportedOperationException` unconditionally, even if the `sort` attempt would not result in an actual change.
Comment: @Eugene it doesn't make any sense to check a `List` against `SortedSet`; it's impossible to implement both contracts correctly in one class. Further, I doubt that anyone is doing an explicit check whether sorting changed anything, that just happens implicitly, e.g. `Collections.sort(Collections.emptyList())` does not fail, likewise  `Collections.emptyList().addAll(Collections.emptyList())` does nothing. Perhaps, that has been kept for compatibility. Newer constructs, e.g. `Collections.sort(List.of())` or `List.of().addAll(List.of());` do the idiomatic unconditional throwing.
Comment: @Holger Yes, the old behavior of zero elements to an empty (and unmodifiable) list is kept for compatibility. This allows latent bugs in applications, because this "works" until somebody tries to add a non-zero number of elements, at which time it blows up. The newer implementations provide a more "fail-fast" behavior. The new `sort` default method also can provide fail-fast behavior, as you observed. The old `Collections.sort` algorithm, if called on an unmodifiable list, would copy out, do the sort, and **then** blow up when copying back to the original list.
Comment: @StuartMarks that's a trade-off I can live with. After all, a broken comparator can corrupt the list anyway, e.g. if no exception has been thrown, which was more likely before the introduction of TimSort, so we don't loose much compared to the old behavior (depending on how deep we dig). Being "failure idempotent" is a nice-to-have but shouldn't sacrifice the performance of correct code in a significant amount.
Comment: @Holger I knew about the second one, interesting enough I find it correct (I know I'm asking for trouble here), but my gut is saying that instead of potentially sort and check if sort has changed anything or not, simply throwing the exception is more idiomatic; unless a check against `SortedSet` would be performed for example...
Comment: @Holger This also has a perhaps unfortunate side effect, which is that `Collections.sort` is now no longer failure idempotent. That is, if the sort fails -- for example, if there is a bug in the comparison method resulting in an exception -- sorting in-place can leave the original list corrupted. That didn't occur with the old copy-sort-copy-back technique.
Comment: @OrangeDog Sure, but an exception occurring during copy-back is almost unheard of. In most cases it doesn't even allocate memory, so even OOME is unlikely. It's still possible of course. However, exceptions thrown from the comparison method are quite likely, as you can tell from SO questions on this topic (search for "comparison method violates its general contract").
Comment: @Holger Thanks for your thoughts on this. There is actually a bug on this, but so far we haven't felt the impetus to fix it.
Comment: @StuartMarks it did if an exception occurred in the middle of copying them back.
Answer: JB Nizet's answer already gives you reasons why it was a good idea to add this method. The second aspect of this is:
If it is so obviously a good idea to add this method, why hasn't it been added in some earlier version?
Both the List interface and the static utility Collections were added in the same version 1.2, so it would have been possible to include it from the start.
After that opportunity had been missed, there was no way to add it any more. Adding a method to an interface was a change that would have broken backward-compatibility prior to the introduction of default-methods in Java 1.8.
Comment: Nice one, as nice as figuring that bug 199 on that other question ;-)


A meeting has 12 employees. Given that 8 of the employees are female, what is the probability that all employees are female?

Question: If I use Bayes' theorem here , event A denoting that 12 employees are female and event B denoting that 8 employees are female, (assuming that an employee has equal chances of being a male or female) I get,
$P(A \mid B) = \frac{P(B \mid A) \times P(A)}{P(B)} = \frac{1 \times (0.5)^{12}}{\binom{12}{8}(0.5)^{12}}=1/\binom{12}{8}$
Is this the correct way of doing this? I am especially confused because the gender of the employees are independent of one another , and yet I am using the information that 8 of them are female to determine the probability that all of them are female.
I am sorry if I sound confused and do not make sense.
Comment: if I know they are females, can't I just calculate the probability of the remaining 4 employees to be females? 0.5^4
Comment: $12 \choose 8$ is the number of ways for *exactly* 8 employees to be female, not for *at least* 8 employees to be female. You presumably want the latter, since the probability that all 12 are female given that exactly 8 are female is just 0.
Comment: I'm saying it's impossible that the number of female employees is simultaneously equal to exactly 8 and exactly 12.
Comment: can you use `|` isntead of `/` where appropriate? and also write explicitly what do you think p(A) is, and p(B) is
Comment: Exactly where my confusion is. This makes sense , and yet I cannot figure out where in my calculation using Bayes'  did I do a mistake. Thats the only use of the information that us provided.
Comment: You need to define the event B as "8 or more" employees are female. Then the calculations using the Bayes's theorem should coincide with "just calculating the probability of the ramining 4 employees to be female".
Comment: Someone kindly suggested edits. Thank you! The events A and B are also defined ,rep_ho .  Why is the probability exactly 0 though , @fblundun
Comment: This is just a special case of the "Boy or Girl" (so-called) paradox: https://en.wikipedia.org/wiki/Boy_or_Girl_paradox
Comment: In and of itself, eight of 12 people being female is indicative of nothing, not even if the meeting was exclusively of Amazons or Spartans.

Against what would you like to compare this tiny sample?
Comment: Not to mention that IRL,  even assuming the statement implies "we know at least 8 are female,"  the odds of any given person being female is NOT 50% , even if we define M/F as "birth assignment."  If we allow trans and ambiguous genders, the odds change again.
Comment: It's interesting that a very similar question was recently asked [here](https://math.stackexchange.com/questions/4171471/find-the-mistake-in-my-conditional-probability-calculation#comment8641506_4171471).
Comment: @Carl, the question isn't about demographics or real-world statistics. the issue at hand is contradictions that are created in probability theory when events or sets are improperly defined or calculated. the question already said to assume equal odds. if you want, you can read the question in terms of coin flips instead of a meeting of 12 people.
Answer: The confusion comes from the fact that there are multiple ways to interpret "Given that 8 employees are female":

If it's 8 specific employees - say, the employees in positions 1 thru 8 - then the remaining four have $2^4$ possible gender configurations, only $1$ of which is all-female, giving $\frac{1}{2^4}$

If it's any 8 of the 12 employees, then what's being asked is to look at all configurations of 12 employees, throw out the ones with 5 or more men, and count the proportion that are all female.
Notice that under this interpretation, each employee in the valid configurations does not have a 50% chance of being male/female, since we are assuming that there are at least 8 females in each valid configuration.  What does have an equal chance is each valid configuration.

The reason this is confusing is that our intuition assumes the first interpretation, but the way the question is worded implies the second.

There is a famous statistical "paradox" that stems from this same line of reasoning:

In a family with two children, one of whom is a girl, what's the probability both are girls?

Most people assume the answer is $\frac{1}{2}$, but it's actually $\frac{1}{3}$, for the same reason as the original question.  If you're still confused, see this answer which gives a more thorough explanation of the paradox and its resolution.
Comment: To make it even more complicated, depending on the wording, the probability can depend on *how you know* the information.
Comment: "I'll give you \$100 if you tell me truthfully there is at least one girl, and I'll give you \$10,000 if you tell me truthfully there are two girls". Answer: "At least one girl". Probability of two girls is zero.
Comment: The main part of this answer correctly explains that the wording of the question was ambiguous and didn't contain enough information to correctly calculate the desired probability. But then you cite *'a famous statistical "paradox"'* which contains the same kind of ambiguity, and you conclude *"Most people assume the answer is 1/2, but it's actually 1/3."*? No it's not. The only possible answer is "can you please rephrase the question, it does not contain enough information, I can't give you a probability with these words".
Comment: I strongly disagree with the statement *"The reason this is confusing is that our intuition assumes the first interpretation, but the way the question is worded implies the second."*. Someone tells you they know that 8 of the 12 employees are female. How did they get that information? Did they know personally 8 of the 12 employees (first interpretation), or did they use a computer program to build a list of potential 12-employee groups, then throw away the groups that did not have at least 8 female employees (second interpretation)? Occam's razor would suggest the first interpretation.
Comment: I think your paradox example is wrong. I see it as four scenarios, boy girl (mentioned), girl (mentioned) boy, girl (mentioned) girl, girl girl (mentioned). Two out of four scenarios have two girls.
Comment: @Tvde1 Those scenarios are not equiprobable. The first two have probabilities 1/3 each; the last two have total probability 1/3 (1/6 each if we assume that if both children are girls the respondent picks each with prob. 1/2 to mention, but that is neither known nor relevant).
Comment: @Tvde1 Yes, that's the origin of the "paradox". You are choosing each **girl** equiprobably, while the way the question is worded strongly implies that each **family** should have equal probability. See [the link I mentioned](https://math.stackexchange.com/questions/15055/in-a-family-with-two-children-what-are-the-chances-if-one-of-the-children-is-a/15085#15085) for more details.
Comment: So, the difference is what it means that "one of whom is a girl" / "8 of which are female"? If we know that the older child is a girl, then the probability is 1/2, right? If the know the younger child is a girl, the probability is also 1/2. But if we know either one is a girl, the probability becomes 1/3, because then there are three scenarios?
Answer: Perhaps it would be helpful to give this some clearer structure, via explicit assumptions.  Suppose we are willing to assume a priori that each person is equally likely to be male or female, and we assume that the sexes are mutually independent.  Then the "female-indicator" variables for the people in the group are:
$$X_1,...,X_{12} \sim \text{IID Bern}(\tfrac{1}{2}).$$
Consequently, the number of females in the group has a binomial distribution:
$$\dot{X} \equiv \sum_{i} X_i \sim \text{Bin}(12, \tfrac{1}{2}),$$
and the conditional probability of interest is:
$$\mathbb{P}(\dot{X} = 12 | \dot{X} \geqslant 8)
= \frac{\mathbb{P}(\dot{X} = 12)}{\mathbb{P}(\dot{X} \geqslant 8)} = \cdots$$
Can you take it from here?
Comment: Hello Ben, another question from a beginner: Is this approach fully Bayesian? I tried to answer the question by assuming a Beta(2,2) prior for the Bernoulli parameter $\mu$ (the person being a female), then maybe generate Bernoulli samples (with size 12) given the posterior $\mu|\text{Data}$ and calculate the probability of all persons being females.
Comment: You can certainly do that if you wish --- if you are unwilling to assume equal probability of males and females then you would give that probability parameter a prior and then proceed via Bayes theorem.
Answer: You need to be very, very, very precise with the statements you make, otherwise any results will be utter nonsense - because they might be the correct answer to a totally different questions.
My reading of your question as asked leads to the answer "the probability is zero". Eight out of twelve employees are female, so four are male, so not all employees are female.
Let's interpret it as "Someone picked 12 employees at random, and counted how many were female. The answer was a number from eight to twelve". Or "Someone picked 12 employees at random, then picked eight of those and checked their gender, and all eight were female". Much different situation, and much different answer.
In the first case, if it was nine females, why did I say "the answer was a number from eight to twelve" and not "the answer was a number from nine to twelve"? If it was eight females, why didn't I say "the answer was a number from zero to eight"? I might have an agenda to make the impression that either lots of employees or that few employees are female, so if you don't know about the agenda, you might get different answers.
Let's say I ask you "how many children do you have" and you answer "two". Then I say "I very much prefer boys to girls. So if you tell me that you have at least one boy, I'll give you 100 dollars. If you tell me that you have two boys, I'll give you 10,000 dollars. If you lie, I'll shoot you". If you tell me "I have at least one boy" then the probability that you have two boys is zero.
But still, you can't answer the question at all. We don't know how many employees there are - because your question wasn't clear. I know there were 12 employees in a meeting, but I don't know how many were outside the meeting. Obviously the more outside the meeting, the less likely it is that they are all female. And we don't know the probability that a random employee is female. You guessed that the probability was 0.5. I would assume that the probability is an unknown number, that the chances that eight employees are female depends on that number, and vice versa you can draw conclusions from the number of females in a group what the probability is that some employee is female.
So let's restate the question. You picked 12 employees at random. I tell you "I will ask you about some attribute that each employee might or might not have, and I want you to tell me if the number of employees in the group with that attribute is from eight to twelve or not".
Comment: If we are going to interpret "8 of the employees are female" as "at least 8 of them", shouldn't we interpret "a meeting has 12 employees" as "a meeting has at least 12 employees"?
Comment: We know even less. Most meetings don't have _random_ employees attending (and I cannot find that word in the original statement). If you have meeting at a hospital it could for instance be a meeting for doctors or nurses - with different probabilities. Finally the assumption that all attendees are iid is questionable in practice.
Answer: What you're noticing here is that the entire field of statistics is plagued by serious interpretational issues.
Chief among these (and at fault here) is the reference class problem.  In a frequentist framework, this corresponds to assigning your statement of "probability" to a well-populated space of outcomes (recall that probability may be axiomatized in terms of event spaces - in a Bayesian framework, the formalization is slightly different, but the practical consequences are the same).  For a statement such as the one in this problem, there are multiple ways we could conceivably do this - what's worse, there's no obvious "correct" statistical interpretation purely from the wording of the problem.  We need more information.
This is one of the main problems with statistics as it is currently taught.  Mathematical pedagogy tends to favor short, snappy exercises - they're easy to read and easy to grade.  But statistics abhors this; any sufficiently short statistical statement is almost bound to be uninterpretable.
Answer: Is not there also a potential skewing of the probabilities due to "cultural" (for want of a better word) factors. If 8 of the employees are female, perhaps this is a Women's gym that does not hire men, or perhaps it is a small company run by an entrepreneur who (probably unlawfully, but that is another subject) only or primarliy hires women? Perhaps this meeting is of members of an occupation that skews female like nursing or elementary school teaching.
Comment: this is a question about a contradiction that arises in probability theory when events or sets aren't correctly defined. it's not a question about societal demographics or culture.


What are Tone Pots / Potentiometers?

Question: Can someone please explain to me what tone pots are, and how they affect the sound of a guitar? Thanks very much :)
Answer: A potentiometer is basically a variable resistor.  When it's turned all the way one way, it has no resistance, and when it's turned all the way in the other direction, it has its maximum resistance.
A pot has three soldering points: its input, its output, and ground.  So when the output of your pickups is wired to the input of the pot, the pot splits the signal in two directions, one of which leads to the output jack of your guitar and the other of which leads to ground.  The signal from the pickups wants to go to ground, but if the pot is turned to its maximum resistance, it acts as a block to the signal, which then has to go to the output of the guitar instead.  As you turn the pot, its resistance weakens, so more and more of the signal escapes to ground instead of going out the guitar's output, and so you perceive this as lower volume.  So the pot acts as the volume knob.
Same thing with the tone knob.  If you combine a pot and a capacitor with the capacitor going to ground, you get a low-pass filter, meaning a circuit that allows low frequencies to pass through but blocks high frequencies.  The combination of the values of the capacitor and the resistor determines the frequencies which the filter sees as "high" and those it sees as "low".  As you turn the pot, the filter sees more and more frequencies as high, and so it allows fewer of them to pass through.  Hence, the guitar's output signal contains fewer and fewer high frequencies, and boom: tone knob.  Here's a simple circuit diagram, and it's exactly what's in my guitar---except for not showing the volume and pickup switch, it's not oversimplified at all:

The (very) basic jist of a capacitor is that it can "store" current, but it takes some time to charge up and down.  When it's charged down, a capacitor acts more-or-less like a wire, and current passes through it with no problem.  Once it's charged up, though, it won't let any current through (it's at "capacity", if you will); in that state, it's like a broken wire.  High frequencies alternate so rapidly that they don't give the capacitor a chance to charge up, so they pass right through it; low frequencies, however, charge up the capacitor before they get through it, and so they're blocked.
So as you can see from the diagram, if the pot's resistance is low enough to allow the signal to head towards ground, the high frequencies will pass through the capacitor and escape, while the low frequencies still can't get by the cap and have to go to the guitar's output.  And so you hear the low frequencies but don't hear the high ones.  Of course, if the pot's resistance is all the way up, then none of the signal goes towards ground and all of it goes to the output.
You can easily tweak your tone controls by replacing the capacitors with different ones with different values.  It's a simple soldering job, capacitors are dirt cheap (like $0.10 cheap) and it's much easier than replacing the pots.
Comment: Similar to the tone control, a capacitor can be put across the volume pot so it acts like a [high pass filter](http://www.ratcliffe.co.za/articles/volumepot2.shtml), allowing more highs to pass through as the volume is rolled down. This mod. on a Les Paul can clean it up when you roll the volume down, making chords, especially complex ones, really sparkle when you drop back for rhythm work. I used to do it to all my Gibsons. The linked article specifically says single-coils will gain more, and, yes they would. I always used little pre-amps in my strats which took care of the problem entirely.
Comment: +1 to tin man. This is a good mod often referred to as the "50's wiring mod". I.e. that is how Gibson guitars were wired back then. I really recommend wiring a Les Paul like this if you get a chance, particularly if you're into blues or classic rock. For extra tone, buy some "vintage" pots and caps from a place like RS Guitarworks (http://store.rsguitarworks.net/)
Comment: That actually is NOT vintage wiring. PLacing a capacitor across the volume pot is called a treble bleed capacitor, and it's usually paired with a specific resistor depending on the type of potentiometer you are installing them on along with the output of the pickups. The so-called "vintage" wiring on Les Pauls is very hard to nail down because each guitar could have been wired completely different. A diagram of true '57 wiring (the only consistent one to my knowledge) is located here: http://www.lollarguitars.com/pickup-wiring-diagrams/57LesPaulWiringDiagram.pdf
Comment: Great answer Alex! It would be more illustrative with a picture though. Another good article on this subject is located here: http://www.stewmac.com/freeinfo/Electronics/Pots/w101-tonecontrol.html
Comment: If you'll notice on that diagram, the older way of wiring a tone pot included placing one post at the sweeper lug on the pot and the other post on the input to the tone control. This compensates for a volume drop that occurs as the logarithmic taper of a potentiometer kicks in. When you roll the volume back some of the frequencies that were bled off are allowed back into the output signal, keeping it consistant. The drawback to this schema is that the tone value changes as you move the volume--so every time you change the volume you have to tweak the tone pot to keep the tone uniform.
Comment: Apologies for the time it took for me to accept the answer! Outstanding answer Alex, well done!
Answer: The tone pots are literally the knobs on the front of your guitar that you adjust the tone and volume of your pick ups with.
Edit:
Cant explain in any great technical detail; aside from to say that they allow you to control the low power analogue signal of your pickups.
In the case of the Tone pots these allow you to control output from specific pick-ups with the volume pot controlling overall volume.
The toggle switch allows you to choose individual/combinations of pickups; with the tone pots acting upon those selected.
Depending on the type/configuration of the guitar you have, you may have a tone pot for each pickup (this gives you most tone control); or you may have one tone pot which controls the overall tone of whatever configuration is selected via he toggle switch.
Comment: Could you elaborate? :)
Comment: Not brilliantly see above
Comment: I don't believe toggle switches are, in fact, pots.  When a pot is described as "linear", that refers to the rate at which the resistance changes as the pot is turned.
Comment: Your probably right; i don't know a great deal about electronics. regardless i have removed that from my answer.

What are Tone Pots / Potentiometers?


cannot log in by using password that has been hash

Question: I'm trying to login into the account by using the md5 password. but the error keeps showing that invalid parameter number. Is there anything wrong with my code?
<?php

 include_once '../database.php';
 include_once 'reg_Customer.php';
 session_start();
 if(isset($_SESSION["CustomerName"]))
      {
        header("location:index.php");
      }

 try
 {
       $conn = new PDO("mysql:host=$servername;dbname=$dbname", $username, $password);
       $conn->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
      if(isset($_POST["login"]))
      {
           if(empty($_POST["CustomerName"]) || empty($_POST["CustomerPass"]))
           {
                $message = 'All fields are required';
           }
           else
           {

                $query = "SELECT * FROM customer WHERE CustomerName = :CustomerName AND CustomerPass = ".md5(CustomerPass)."";
                $stmt = $conn->prepare($query);
                $stmt->execute(
                     array(
                          'CustomerName'     =>     $_POST["CustomerName"],
                         md5('CustomerPass')     =>     $_POST["CustomerPass"]
                     )
                );
                $count = $stmt->rowCount();
                if($count > 0)
                {

                    $_SESSION["CustomerName"] = $_POST["CustomerName"];

                     header("location:index.php");
                }
                else
                {
                     $message = 'Wrong Password';
                }
           }
      }
 }
 catch(PDOException $error)
 {
      $message = $error->getMessage();
 }
 ?>

the error:

SQLSTATE[HY093]: Invalid parameter number: number of bound variables does not match the number of tokens
Comment: Regardless of the actual error, you should throw this code away and start again: MD5 is a **really really insecure** function to use for password hashes; you might as well be storing them in plain text. Instead, generate a strong, salted hash with [password_hash](https://php.net/password_hash) and check the user's input against it using [password_verify](https://php.net/password_verify).
Comment: a correct approach imho should be to: store the password on the db with password_hash when the user registers. When he tries to login download the hashed password from the db and match it against the input password with password_verify.
Answer: You are actually setting the value already when defining the query:
[...] AND CustomerPass = ".md5(CustomerPass)."";

It has to be [...] AND CustomerPass = :CustomerPass.
Also, in your array, you use the MD5 hash function on the key, not the value. It has to  be:
'CustomerPass'     =>     md5($_POST["CustomerPass"])

Other than that, don't use md5 as IMSoP already stated. password_hash is your friend, or other hash functions such as SHA2-512.
Comment: Rolling your own hash with SHA2-512 is still a bad idea - data verification hashes are designed to be **fast**, but password hashes are designed to be **slow**, to combat brute force. password_hash and password_verify will use an appropriate algorithm, and manage the salt for you; I can't think of any reason *not* to use them.
Comment: Neither can I, it was just added to name a _more_ secure hash function, not _the most secure_ one. I still think SHA2-512 is doing a decent job - especially when forcing long(er) passwords. But yeah, in the wide array of hash functions available, functions like _bcrypt_ are a better fit for the task, no doubt.


Redirect loop - apache

Question: I'm trying to wrap my head around why my apache configs give me an infinite redirect loop. I've tried everything but the loop is still there.
What's interesting is that when you replace the very last redirect (app.php) with app_dev.php, it runs fine (Symfony Dev environment), but replacing it with the app.php causes the loop.
I'm looking at the code and have no clue what that could be. Any clues?
<code><VirtualHost 126.96.36.199:8080>

ServerName messpace.com
ServerAlias www.messpace.com
ServerAdmin email@example.com
DocumentRoot /home/admin/web/messpace.com/public_html
ScriptAlias /cgi-bin/ /home/admin/web/messpace.com/cgi-bin/
Alias /vstats/ /home/admin/web/messpace.com/stats/
Alias /error/ /home/admin/web/messpace.com/document_errors/
#SuexecUserGroup admin admin
CustomLog /var/log/apache2/domains/messpace.com.bytes bytes
CustomLog /var/log/apache2/domains/messpace.com.log combined
ErrorLog /var/log/apache2/domains/messpace.com.error.log
<Directory /home/admin/web/messpace.com/public_html>
    AllowOverride All
    Options +Includes -Indexes +ExecCGI
</Directory>
<Directory /home/admin/web/messpace.com/stats>
    AllowOverride All
</Directory>

<IfModule mod_ruid2.c>
    RMode config
    RUidGid admin admin
    RGroups www-data
</IfModule>
<IfModule itk.c>
    AssignUserID admin admin
</IfModule>

#IncludeOptional /home/admin/conf/web/apache2.messpace.com.conf*

Alias /sf /$data_dir/symfony/web/sf
RewriteEngine On
# Rewrite assets path
RewriteRule ^/?images/(.*)$ /web/images/$1  [L]
RewriteRule ^/?css/(.*)$ /web/css/$1  [L]
RewriteRule ^/?js/(.*)$ /web/js/$1  [L]
RewriteRule ^/?font-awesome/(.*)$ /web/font-awesome/$1  [L]
RewriteRule ^/?fonts/(.*)$ /web/fonts/$1  [L]
RewriteRule ^/?bower_components/(.*)$ /web/bower_components/$1  [L]
RewriteRule ^/web/app.php - [L]
ErrorLog /var/log/apache2/domains/messpace.com.error.log
#SuexecUserGroup admin admin
CustomLog /var/log/apache2/domains/messpace.com.bytes bytes
CustomLog /var/log/apache2/domains/messpace.com.log combined
LogLevel warn rewrite:trace8
<Directory /home/admin/web/messpace.com/public_html>
    AllowOverride All
    Options +Includes -Indexes +ExecCGI -MultiViews     +SymLinksIfOwnerMatch
    Order allow,deny
    Allow from all
</Directory>
<Directory /home/admin/web/messpace.com/stats>
    AllowOverride All
</Directory>
<IfModule mod_ruid2.c>
    RMode config
    RUidGid admin admin
    RGroups www-data
</IfModule>
# IncludeOptional /home/admin/conf/web/apache2.messpace.com.conf*
<IfModule mod_rewrite.c>
Options +FollowSymlinks
RewriteEngine On

# Explicitly disable rewriting for front controllers
RewriteRule ^/web/app_dev.php - [L]

RewriteRule ^/web/app.php - [L]
# Fix the bundles folder
RewriteRule ^bundles/(.*)$ /web/bundles/$1  [QSA,L]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_URI} !^/web/app.php

# Change below before deploying to production
#RewriteRule ^(.*)$ /web/app_dev.php [QSA,L]
RewriteRule  ^(.*)$ /web/app.php [QSA,L]
</code>

Any ideas? Thanks!
Answer: I've tried your rules on my apache and they are working as intended.
Maybe there are more rules from conf.d or from .htaccess in your setup?
Comment: Right in the button.
There was an htaccess file that messed it up (a Symfony stock one).
I was so sure that it was the conf file that I didn't even bother looking at htaccess.
Thanks a ton for checking it.

Redirect loop - apache


Best statistical methods for assessing change in density

Question: I am measuring the change in density of algae over time. I have seven different species and I am testing their different ability to survive after treatment compared to control. I made 7 control samples for each - and seven treated samples (no replicates). The treatment is the only variable - all physical and chemical conditions were held constant.

Because each was grown in a different tube the density of the algae at the starting point was different on day 1 before adding treatment. I tried indexing my data to a common starting point by using percent change.
I was wondering if there any other methods I can use that are more accurate?
Comment: What do you mean by "density"? What are these numbers? Are they counts of colonies, eg?
Comment: it's a fluorescence measurement that can reflect how much algae is in the tube.
Answer: you have your linear term which is Date which is change over time, but the Date squared term will give you the rate of change over time.


HTML sidebar is cover by the content

Question: My sidebar is covered by the main tabbed menu. I have put it in the jsfiddel.
https://jsfiddle.net/x0j97dtk/
It works with firefox but not chrome. It didn't work in jsfiddle either.
Any idea?
Here is the HTML
<!DOCTYPE html>

    <meta charset="UTF-8">
    <title>Nav</title>
    <link href="nav.css" rel="stylesheet"/>

 My App

            Version

                    V2

{{v2}}

                    V3

{{v3}}

                 <button id = "{{table}}" width = "70">{{table}}</button>

ERD

This content appears on tab ERD.

Columns

This content appears on tab Columns.

Home

This content appears on tab Home. lfkdgl;k  lkfd;lkg ';lkfg ;lkg 'df;lk ;lk ';lk ';fdlkg ';fdlkg';dflk;ldfkg';lkdlfkdfkglkdf lkjhlsdjhfljdfhlkjdh jh jhkjdhfkjsdhf skjdhf lk h dsfjlkjsdlkfj;dslkfj  dskfj;kj sdfkj fkdj;lfkjsd;lkfj   sdkfj ;slkj sdfj;lskjf   skdj flksjdf ; sdfkj ;sdlkfj dskfj sdkjfhueuu suehu heu he u heu heh ueh ufhu fhe uueh ush uhsudh ue huhuhufheuheu u heiu h euh eh ue

    "Copyright© 2016

and the CSS
#logo{
    position: absolute;
    right:10px;
    bottom: 10px;
}
body{
    margin: 0;
    padding: 0;
    font-size: 12px;
    font-family: "Lucida Grande", "Helvetica Nueue", Arial, sans-serif;
}
nav {
    background-color: #333;
    border: 1px solid #333;
    color: #fff;
    display: block;
    margin: 0;
    padding: 0;
    z-index: 9999;
}
nav ul{
    margin: 0;
    padding: 0;
    list-style: none;
    z-index: 9999;
    text-align: center;
}
na ul span {
    width: 300px
    margin-top: 250%;;
}
nav ul li {
    margin: 0;
    display: inline-block;
    list-style-type: none;
    transition: all 0.2s;
}

nav > ul > li > a {
    color: #aaa;
    display: block;
    line-height: 1em;
    /*padding: 0.5em 2em;*/
    padding: 0.2em 3em;
    text-decoration: none;

}

nav li > ul{
    display : none;
    margin-top:1px;
    background-color: #bbb;

}

nav li > ul li{
    display: block;
}

nav  li > ul li a {
    color: #111;
    display: block;
    line-height: 1em;
    padding: 0.2em 2em;
    text-decoration: none;
}

nav li:hover {
    background-color: #666;
}
nav li:hover > ul{
    position:absolute;
    display : block;
}
nav li > ul > li ul  {
    display: none;
    background-color: #888;
}
nav li > ul > li:hover > ul  {
    position:absolute;
    display : block;
    margin-left:100%;
    margin-top:-3em;
}

nav ul > li.sub{
    background: url(ic_keyboard_arrow_down_white_18dp.png) right center no-repeat;
    z-index: 5;
    overflow: visible;
}

nav ul > li.sub li.sub{
    background: url(ic_keyboard_arrow_right_white_18dp.png) right center no-repeat;
    z-index: 5;
    overflow: visible;
}

/*****************************************************************************/
aside {
    display: block;
    position: absolute;
  width: 80px;
  height: 550px;
  padding-right: 0;
  margin: 0;
  align: left;
  overflow: auto;
  background-color: lightblue
}
/*
article.side aside
{
    width: 90px;
    height: 500px;
    margin: 2em auto;
    background-color: red;
  z-index:100;
}*/
article {
/*  position: relative; */
}

article.tabs
{
    position: relative;
    display: block;
    width: 1100px;
    height: 500px;
    margin: 2em auto;
        background-color: green;

}
article.tabs section
{
    position: absolute;
    display: block;
    left: 0;
    width: 1100px;
    height: 500px;
    padding: 10px 20px;
    background-color: #ddd;
    border-radius: 5px;
    box-shadow: 0 3px 3px rgba(0,0,0,0.1);
    z-index: 0;
}
article.tabs section:first-child
{
    z-index: 1;
}
article.tabs section h2
{
    position: absolute;
    font-size: 1em;
    font-weight: normal;
    width: 120px;
    height: 1.5em;
    top: -1.5em;
    left: 10px;
    padding: 0;
    margin: 0;
    color: #999;
    background-color: #ddd;
    border-radius: 5px 5px 0 0;
}
article.tabs section:nth-child(2) h2
{
    left: 132px;
}
/*
article.tabs section:nth-child(3) h2
{
    left: 254px;
}*/

article.tabs section h2 a
{
    display: block;
    width: 100%;
    line-height: 1.5em;
    text-align: center;
    text-decoration: none;
    color: inherit;
    outline: 0 none;
}
article.tabs section:target,
article.tabs section:target h2
{
    color: #333;
    background-color: #fff;
    z-index: 2;
}
article.tabs section,
article.tabs section h2
{
    -webkit-transition: all 500ms ease;
    -moz-transition: all 500ms ease;
    -ms-transition: all 500ms ease;
    -o-transition: all 500ms ease;
    transition: all 500ms ease;
}
footer
{
    color: #999;
  text-align:center
  background-color: #333;
  width: 100%;
  bottom: 0;
  position: fixed;
}

Comment: `align` attribute is depreciated. It's also not a valid CSS property.
Comment: Hungerstar, thanks for mentioning that.
Answer: Adding margin-left: 82px solves the issue.
article.tabs {
  position: relative;
  display: block;
  width: 1100px;
  height: 500px;
  margin: 2em auto;
  background-color: green;
  margin-left: 82px;
}

Answer: article.tabs{margin-left:80px;
}

Comment: Argiris, thanks for the answer. That fixed the issue. Can I ask another favor? when the sidebar button is clicked the focus will change from the second tab to the first one. how can I make the second tab remain on display?
Comment: Hmm,that seems like a JavaScript issue. Unfortunately I am not that skilled at it. The issue I believe is that you have the function onclick that directs you to the app,and the first tab is shown then.
Comment: Thanks, I will open a another one.
Answer: You have a left sidebar with a width of 80px. So for the content to not cover is to push it 80 px to the right.
Do you know what padding and margin does?
You have a (in this case) an article with class tabs.
- Margin is the space around the article.
- You can use padding to give space from the side of the article to the inner elements.
Right now you have margin: 2em auto
if you specify margin with two values the first value one is for the left and right sides and the second value for the top and bottom sides.
Since you probaly only want margin on the left you can change it to two different values:
margin-left: 80px;
or
margin: 0px 0px 0px 80px
This second way of describing margin is a way to set all the margins in one line
margin: top right bottom left;
Hopes this explains your problem properly.
Comment: user3398922, You have given excellent explanation. Thank you. Really appreciate this great community support. I got another issue: how can I keep the 2nd tab remain on display when the sidebar button is clicked?
Comment: The best thing you can do is make another post and add angular or angularjs  as tag. This way other people can benefit from your question. I'm not quite familiar with angular so I don't now how the tabs are handled.

HTML sidebar is cover by the content


Django annotate with conditional expression, how to apply distinct()

Question: I have a little bit complex query and I use annotate with a conditional expression, the code is below:
from django.db.models import Q, F, When, CharField, Value, Case
from django.contrib.auth.models import User

ATT_PREFIX = 'f'

def get_it(mymodel, fields):

    q = Q(modelone__modeltwo__mymodel=mymodel)

    query_set = {
        'aid': F('modelone__id'),
        'name': F('modelfour__name'),
    }

    for f in fields:
        query_set[ATT_PREFIX + str(f.id)] = Case(
            When(Q(modelfive__modelsix=f) &
                 Q(modelfive__user__email=F('email')),
                 then='modelfive__entry'),
            default=Value(''),
            output_field=CharField(),
        )

    return User.objects.filter(q).annotate(**query_set)

Attributes aid and name collected with F() have expected values. Second part which use conditional expression have a problem: If feilds list contain 2 objects, then I have 2 User objects created by Case() when condition is meet. So instead of one object with atributes (aid, name, f17, f18)  I got two objects:
obj1 attributes (aid, name, f17)
obj2 attributes (aid, name, f18)

Django 1.10, Python3.5
Answer: try this
q = Q(modelone__modeltwo__mymodel=mymodel).distinct()


A better approximation than mine for $\int_0^1\log\left(1+\operatorname{gd}(x)\right)\,dx$, where $\operatorname{gd}(x)$ is the Gudermannian function

Question: Let $\operatorname{gd}(x)$ the Gudermannian function, defined as in this MathWorld's article, and implemented in Wolfram Language as Gudermannian[x].
This afternoon I've spent an hour playing with Wolfram Alpha online calculator about integrals, for instance, like this $$I=\int_0^1\log\left(1+\operatorname{gd}(x)\right)\,dx.$$
Notice that the plot in previous article of MathWorld tell us that for $0<x<1$ one has $0<\operatorname{gd}(x)<1$.
My calculations to get an approximation were $$I\approx\int_0^1 \operatorname{gd}(x)\,dx=-\frac{\pi}{2}+i\left(\operatorname{Li_2}(-ie)-\operatorname{Li_2}(ie)\right)-2C,$$ where I've combined with the indefinite integral of the Gudermannian functions that tell us MathWorld. Here $C$ is Catalan's constant. But this quantity is about $\approx 0.464065$.
But WA calculated $I$ as $0.365619$, play this code:
integrate log(1+(gd(x)))dx, from x=0 to x=1

Question. Can you provide us a better approximation, than mine, for our integral $$\int_0^1\log\left(1+\operatorname{gd}(x)\right)\,dx?$$
  Please provide also the more important justifications in your calculations. Also if your approach uses the numerical analysis and/or calculations with your computer provide us some detail of your calculation/method. Thanks in advance.
Comment: To make it more standard, you could substitute $$u = \text{gd}(x),$$ which leads to $$\int_0^\alpha \frac{\log(1+u)}{\cos u} \, du,$$ with $\alpha = \text{gd}(1).$
Comment: You should be able to use the Taylor Series for $\log(1+x)$, take a few terms, and integrate termwise to get a good approximation
Comment: Many thanks, @πr8 and thanks for your help and attention. I am an aficionado but your formulas seem very nice. Feel free to edit the post, any case.
Comment: Seems a very good strategy, many thanks @BrevanEllefsen
Answer: For sure, you could use Taylor expansion
$$\log (1+\text{gd}(x))=x-\frac{x^2}{2}+\frac{x^3}{6}-\frac{x^4}{12}+\frac{3 x^5}{40}-\frac{x^6}{18}+\frac{169
   x^7}{5040}-\frac{19 x^8}{840}+\frac{2171 x^9}{120960}+O\left(x^{10}\right)$$ and integrate which would lead to $$\int_0^1 \log (1+\text{gd}(x))\,dx=\frac{443161}{1209600}\approx 0.366370$$ Continuing the expansion to $O\left(x^{20}\right)$ would give $$\int_0^1 \log (1+\text{gd}(x))\,dx=\frac{35582823156912797}{97316080327065600}\approx 0.365642$$ May be, better would be to use Padé approximants; a very simple would be
$$\log (1+\text{gd}(x))\approx \frac{x-\frac{x^2}{2}}{1-\frac{x^2}{6}}$$ and the integration would lead to  $$\int_0^1 \log (1+\text{gd}(x))\,dx\approx 3+3 \log \left(\frac{6}{5}\right)-3 \sqrt{6} \tanh ^{-1}\left(\frac{1}{\sqrt{6}}\right)\approx 0.361349$$ Using the next one
$$\log (1+\text{gd}(x))\approx \frac{-\frac{17 x^3}{60}+\frac{6 x^2}{5}+x}{\frac{2 x^2}{5}+\frac{17 x}{10}+1}$$ we can still integrate and get a bunch of logarithms for a value $\approx 0.364644$.
Just for the fun, I put in the next table the decimal value of the integral of the $[n,2]$ Padé approximant
$$\left(
\begin{array}{cc}
 n & \text{result} \\
 2 & 0.3613491513 \\
 3 & 0.3646438085 \\
 4 & 0.3661395229 \\
 5 & 0.3657879040 \\
 6 & 0.3654692466 \\
 7 & 0.3655746142 \\
 8 & 0.3656449849 \\
 9 & 0.3656298359 \\
 10 & 0.3656072339
\end{array}
\right)$$
Comment: @user243301. You will know many and more with age (that is the problem). Padé approximants are just beautiful and so superior to Taylor series. Have a look on this site to see many examples. Cheers and thanks for the challenging problem ! Good luck in your studies.
Comment: Padé approximants! You know a lot of tricks. Many thanks I am going to read and study your nice answer.
Answer: The Gudermannian function is lower-bounded by the line going to $(0,0)$ and $(1,\operatorname{gd}(1))$. However, it doesn't make a very large difference, and it hence gives a reasonable approximation.
See, for example, a plot here on Wolfram Alpha.
Now note that
\begin{align*} \int_0^1\log\left(1+\operatorname{gd}(x)\right)\,dx &> \int_0^1\log\left(1+\operatorname{gd}(1)x\right)\,dx \\ &= \left[ x \log(\operatorname{gd}(1) x + 1) + \frac{\log(\operatorname{gd}(1) x + 1)}{\operatorname{gd}(1)} - x\right]^1_0 \\ & =  \log(\operatorname{gd}(1)+ 1)  +\frac{\log(\operatorname{gd}(1)+ 1)}{\operatorname{gd}(1) }  - 1 \\ &\approx 0.344043 \end{align*}

Even more simply, one can approximate the integral by $$\frac{\log(1+\operatorname{gd}(1))+\log(1+\operatorname{gd}(0))}{2} = \frac{\log(1+\operatorname{gd}(1))}{2} \approx 0.31184$$
Comment: @user243301 Thank you for the accept, but I wouldn't accept an answer too fast on this question - there are many users on this site with far more experience with integrals than I, and they might be able to do much better, or even find an exact value for this integral.
Comment: Many thanks, your reasonings an approximations are very good.
Comment: Then I am sorry, you are right, but your answer was very good and quick. Any case many thanks. Now I've unaccepted your answer.
Answer: You considered $\log(1+x)\approx x$ forgetting that gd(x) goes from 0 to 0.6 circa on the given interval, that is too much and gives a huge error
Compare $\log(1+0.4)\approx 0.33;\;\log(1+0.5)\approx 0.4$ and $\log(1+0.6)\approx 0.47$
This causes your approximation being too high. Taking  integration limit lower than 1, like 0.2 or 0.3 gives good approximation.
Comment: Many thanks for this detail, you are right.


How to test X11 Forwarding?

Question: I am trying to enable X11 forwarding so that I can run Firefox from the remote host. The remote host is a Vagrant VM running Ubuntu 12.04. The local host is Mac OS X 10.10. I think I have everything configured correctly, but I've never done this before, and it doesn't seem to be working.
What is an easy way to test that X11 forwarding is configured correctly?
Comment: Then everything isn't configured correctly. Perhaps you should ask your question about that, instead.
Comment: Try running Firefox?
Comment: Sorry, I was simplifying the example...I'm actually trying to run selenium which is supposed to start firefox behind the scenes, but I'm not sure where things are going wrong. When I run `firefox` on the remote host, I get `Error: no display specified`
Comment: So you're saying, all I need to do is try to run `firefox` and it should work? You should make that comment an answer because that's all I need to know. I don't know if I need to start an X window session or specify any additional command line options, so that's why I am asking this question. If I can get a simple example working, then I can move onto getting the selenium configuration working.
Answer: Simply running a graphical program when logged into the remote host is a sufficient test.
Comment: e.g. `ssh host firefox` (assuming firefox is installed on the host)

How to test X11 Forwarding?


Uso de Swing no java EE

Question: Boa Tarde, quando o programador trabalha com java WEB, ele gera interface grafica  usando a classe javax.swing como em Desktop?
Comment: Até onde sei, swing é somente desktop, para web há o [JAVA FX](http://blog.caelum.com.br/falando-em-java-introducao-ao-javafx/).
Comment: Obrigado Diego..
Answer: TL;DR
Não, quando se fala em Java Web, ainda mais em Java EE, as interfaces são construídas usando HTML, CSS e JavaScript.
Mas esta não é a história completa...
Java Applets
Na verdade é possível em algum grau usar Swing numa página web usando Java Applets, que são programas Java que executam na máquina do usuário, dentro do navegador, e podem usar componentes Swing.
É como ter uma janela desktop dentro do navegador.
Porém, Java Applets é uma tecnologia defasada, que ninguém mais recomenda usar, inclusive o Chrome deixou de suportar há alguns meses. Os principais problemas são relacionados à segurança e também ao fato de que isso foge completamente ao conceito de um sistema web.
Java Network Launch Protocol (JNPL)
Outra tecnologia que tenta facilitar o uso de aplicações Java Desktop é JNPL, que permite que uma aplicação desktop, possivelmente baseada em Swing, seja baixada e executada pelo usuário automaticamente, sem necessidade de instalação ou configuração.
Não é incomum sistemas web servirem de ponte para uma aplicação JNDI, isto é, você faz o login no sistema web e então é redirecionado para um descritor JNPL que então carrega sistema desktop na sua JVM local.
Quase Swing #sqn
Para aproveitar a familiaridade que muitos desenvolvedores tem do Swing, algumas soluções para Java Web tentam aproveitar o mesmo conceito de componentização, como é o caso do Vaadin, Wicket, GWT ou até JSF, embora este último esteja hoje um pouco mais próximo do HTML que os demais.
Basicamente eles usam algum template ou mesmo API que permite você construir e manipular componentes via código, entretanto esses componentes são implementados com HTML, CSS e JavaScript.
Alega-se que tal uso dos componentes aumenta a produtividade, já que o desenvolvedor não precisa se preocupar com o design. Entretanto, tais frameworks tem a fama de apresentarem diversas dificuldades para estender os componentes ou de compatibilidade com navegadores, fazendo com que muitas vezes o desenvolvedor tenha que cavar em scripts e tags em algum momento.
Comment: Obrigado pelo Comentario , Esclareceu Minha Duvida..

Uso de Swing no java EE


Showing the the unit sphere is closed using sequences

Question:
Let $(X,\|\cdot\|)$ be a normed space. Prove that every sequence in $S_X=\{x\in X\mid \|x\|=1\}$  converges in $S_X$.

My attempt.
Let $(x_n)\in S_X$. Then, $\|x_n\|=1$ for all $n$. Now assume that $x_n\to x$. Given $\varepsilon>0$, there is $N\in\mathbb N$ such that $\|x_n-x\|<\varepsilon$ whenever $n\geqslant N$. Thus, $$\|x\|=\|x-x_n+x_n\|=\|x_n-x\|+\|x_n\|<\varepsilon +\|x_n\|=\varepsilon+1.$$
As $\varepsilon$ is arbitrary, it follows that $\|x\|=1$.
If the reasoning is correct, this shows that $S_X$ is closed. There have been many questions about this, but they involve showing that its complement is not open and so on. So, can someone please verify that my reasoning above is correct? Should I perhaps argue by contradiction assuming that $x\notin X$ or is this fine?
Thanks in advance.
Comment: I think that at this point you can conclude that $||x||\leq 1$ but not $||x||=1.$
Comment: @Ryuky If for all $\epsilon>0$ we have $1-\epsilon<||x||<1+\epsilon$, then we must have that $||x|| = 1$.
Comment: Closed means $convergent$ sequences in $X$ converge in $S_X$.  There are nonconvergent sequences in $S_X$.  You want to use the "reverse triangle inequality" here too.
Comment: @mathematician Did you mean convergent sequences in $S_X$ converge in $S_X$? This is the definition I was given. I took a convergent sequence in $S_X$ and I want to show that its limit has norm 1. Apparently, what I did above is wrong. So, how can I show this?
Comment: @Baloown So, what do you suggest? With the reverse triangle inequality I get $\|x\|>1-\varepsilon$ but I don't see how this helps.
Comment: @Ryuky : I suggest to listen Marc !
Answer: A more general way to prove this is to derive the result that any norm is continuous, see for example this question. If you take this result as given, then
$$
||x|| = ||\lim_{n\rightarrow\infty}x_n|| = \lim_{n\rightarrow\infty}||x_n|| = \lim_{n\rightarrow\infty}1 = 1.
$$
Comment: Good idea. Thanks!


QT: qmake not recognising 3dcore, 3drender

Question: I am using qmake to compile my QT project. In my .pro file I have a line:
QT += 3dcore 3drender 3dinput 3dextras

When executing qmake myprofile.pro I get the error:
Project ERROR: Unknown module(s) in QT: 3dcore 3drender 3dinput 3dextras

I am using Ubuntu17 and have QT5 installed. How can I get it to work with qmake? I am including libraries like <Qt3DCore> in my code so I require these modules as suggested here.
Comment: Check if your Kit is pointing to a wrong version of Qt. Maybe this [here](https://forum.qt.io/topic/86166/3d-modules-not-found/13) can help you fix it.
Answer: 3dcore introduced in Qt 5.7, ubuntu 17 had only 5.6 by default. If you have such problems, you need to update library.