How do you calculate the expected value of $e^{-X}$?

Question: If $X$ is a random variable, I would like to be able to calculate something like $$E(e^{-X})$$
How can I do this? Thank you so much.
Comment: this is the moment generating function $m_{X}(t)=E[\exp(tX)]$ evaluated at $t=-1$.
Comment: A specific example:  https://stats.stackexchange.com/questions/175196/expectation-of-e-x-when-x-is-log-normal/176162#176162
Answer: As pointed out in the comments, your specific question can be solved by evaluating the moment generating function and $t=-1$, but it appears you may be asking the more general question of how to calculate the expected value of a function of a random variable.
In general, if $X$ has density function $p$, then
$$ E \left( f(X) \right) = \int_{D} f(x) p(x) dx $$
where $D$ denotes the support of the random variable. For discrete random variables, the corresponding expectation is
$$ E \left( f(X) \right) = \sum_{x \in D} f(x) P(X=x) $$
These identities follow from the definition of expected value. In your example $f(X) = \exp(-X)$, so you would just plug that into the definition above.
Continuous example: Suppose $X \sim N(0,1)$, then
\begin{align*}
E \left (\exp(-X) \right) &= \int_{-\infty}^{\infty} e^{-x} \frac{1}{\sqrt{2 \pi}} e^{-x^2/2} dx \\
&=  \int_{-\infty}^{\infty} \frac{1}{\sqrt{2 \pi}} e^{-(x^2 + 2x)/2} dx \\
&=  \int_{-\infty}^{\infty} \frac{1}{\sqrt{2 \pi}} e^{-(x^2 + 2x + 1)/2} e^{1/2} dx \\
&=  e^{1/2} \int_{-\infty}^{\infty} \underbrace{\frac{1}{\sqrt{2 \pi}} e^{-(x+1)^2/2}}_{{\rm density \ of \ a \ N(-1,1)}} dx \\
&= e^{1/2}
\end{align*}
Discrete example: $X \sim {\rm Bernoulli}(p)$. Then
\begin{align*}
E \left( \exp(-X) \right) &= \sum_{i=0}^{1} e^{-i} P(X=i) \\
&= (1-p)e^0 + pe^{-1} \\
&= (1-p) + p/e
\end{align*}
Comment: $Y = f(X)$ is a random variable which we assume for simplicity is a continuous random variable with density function $g_Y(y)$. Thus the _definition_ of $E[Y]$ is $$E[Y] = \int_{-\infty}^{\infty}yg_Y(y)\,\mathrm dy,$$ and so the question is _why_ is the expectation of $Y$ **also** given by $$E[Y] = E[f(X)] = \int_{-\infty}^{\infty} f(x) g_X(x)\,\mathrm dx?$$ That the two competing formulas for $E[Y]$ both give the same value is worth mentioning somewhere, possibly linking to a proof of this remarkable result which the Wikipedia link calls the Law of the Unconscious Statistician.
Comment: @DilipSarwate, doesn't this just follow from the change of variable theorem?
Comment: Measure theoreticaly, you prove it when $f$ is an indicator, then when $f$ is a nonnegative simple function, then when $f$ is a nonnegative measurable function (using monotone convergence), and finally when $f$ is an arbitrary measurable function.


Salvar dataset C#

Question: Estou buscando várias tabelas do banco de dados para um Dataset através de um Stored Procedure no MySQL usando o MySQL Workbrenck, mas na hora de salvar somente a primeira tabela é salva, meu código:
Stored Procedure:
CREATE DEFINER=`root`@`localhost` PROCEDURE `Tabelas_Materia_Prima`()
BEGIN

    SELECT * FROM materia_prima_ar order by 'DATA';
    SELECT * FROM materia_prima_amido order by 'DATA';
    SELECT * FROM materia_prima_art order by 'DATA';
    SELECT * FROM materia_prima_umidade order by 'DATA';
    SELECT * FROM materia_prima_pol order by 'DATA';
    SELECT * FROM materia_prima_infeccao order by 'DATA';
    SELECT * FROM materia_prima_imp_veg order by 'DATA';
    SELECT * FROM materia_prima_imp_min order by 'DATA';
    SELECT * FROM materia_prima_fosfato order by 'DATA';
    SELECT * FROM materia_prima_fibra order by 'DATA';
    SELECT * FROM materia_prima_dextrana order by 'DATA';

END

Método para salvar o Dataset novamente:
public void Salvar_Procedure(DataSet Dados, string Nome_Procedure)
{
     MySqlConnection Conexao = new MySqlConnection(StringConexao);

     //OdbcConnection Conexao = new OdbcConnection("DSN=YasFashion_Sacoleiro_DB");

     MySqlCommand Comando = new MySqlCommand();
     Comando.Connection = Conexao;
     Comando.CommandType = CommandType.StoredProcedure;
     Comando.CommandText = Nome_Procedure;
     MySqlDataAdapter Meu_Adaptador = new MySqlDataAdapter(Comando);

     //OdbcDataAdapter Meu_Adaptador = new OdbcDataAdapter("SELECT * FROM " + Nome_Tabela, Conexao);

     try
     {
          Conexao.Open();

          MySqlCommandBuilder Comando01 = new MySqlCommandBuilder(Meu_Adaptador);

         //OdbcCommandBuilder Comando = new OdbcCommandBuilder(Meu_Adaptador);

         Meu_Adaptador.Update(Dados);

         Conexao.Close();
      }
      catch (Exception ex)
      {
           MessageBox.Show("Erro de salvamento: \n" + ex);
      }
}

Comment: Você que usar a PROCEDURE Tabelas_Materia_Prima() como um select de retorno de todos os select que tem nela?
Answer: A sua Proc executa quando é chamada direto do SGBD? Caso não, da uma olhada se ele não está se perdendo com o nome das colunas dos diferentes tipos de tabelas.
Você pode compor o select por campos explícitos com o mesmo nome para cada consulta. Pelo que parece, o objeto que você salva está conforme o primeiro select.
Você pode usar todos eles em uma View utilizando o UNION do sql para colocar todos os dados como se fosse uma única tabela.

Salvar dataset C#


angularjs: scope issue in directive when using $compile

Question: I'm having an issue with scope lost when using $compile to create a dynamic template for my directive. See the code below (trimmed for clarity) :
(function () {
'use strict';

angular.module('cdt.dm.directives').directive('serviceSources', ['$http', '$templateCache', '$compile', '$parse',
function ($http, $templateCache, $compile, $parse) {
    return {
        restrict: 'E',
        replace: true,
        scope: {
            type: '=',
            sources: '='
        },
        link: function (scope, element, attr) {
            var template = 'Template_' + scope.type + '.html';

            $http.get(template, { cache: $templateCache }).success(function (tplContent) {
                element.replaceWith($compile(tplContent)(scope));
            });

            $compile(element.contents())(scope);
        }
    }
}
])
})();

that works and the html template is loaded.
the html template looks like:

        File

        {{src.fileName}}

sources is an array with two elements. In the scope of the directive, it is definitely correct, but in the template, the ng-repeat is not working (I guess because sources is undefined at this stage).
Does someone know what I'm doing wrong ?
Comment: I wonder if you could just use `templateUrl` instead?
Comment: Just to check, it's not the typo in ng-repeat is it?
Comment: No, the template needs to be dynamic, based on the value passed to the 'type' variable on the scope. So I cannot use templateUrl.
Comment: I've edited the original code as per your recommendation, thanks.
Comment: sorry, forgot to tell you to wrap the code inside `scope.$apply`. Check my updated answer.
Answer: I think there is a typo: ng-repeat instead of data-ng-reapeat, and ng-repeat should be placed on

        File

        {{src.fileName}}

Note that $http.get is asyn and remember to wrap the code inside scope.$apply. You should modify your code like this:
link: function (scope, element, attr) {
            var template = 'Template_' + scope.type + '.html';

            $http.get(template, { cache: $templateCache }).success(function (tplContent) {
              scope.$apply(function(){
                element.replaceWith(tplContent);
                $compile(element)(scope);
                //Or
                //element.html(tplContent);
                //$compile(element.contents())(scope);
               });
            });
        }

Comment: ng-repeat and data-ng-repeat are interchangeable. Same goes for all directives, they can all have data prefixed to them, even custom made directives.
Comment: @francisco.preller: if you look closely, you will see `reapeat` instead of `repeat`
Comment: Oh yeah, you're absolutely right. Thought you meant the latter. My bad :)
Comment: @francisco.preller: I meant both actually. In the question, I see:   `the ng-repeat is not working (I guess because sources is undefined at this stage)` . I thought the OP was referring to `ng-repeat` and I also did not know about `data-ng-repeat`, though. (I usually use `ng-repeat` this way). Thanks for pointing out `data-ng-repeat`
Comment: Sometimes instead of working late on a Friday evening and posting dumb questions on SO I should just enjoy the weekend.... :) Thanks a lot people for spotting the typo, that was just it... And also thanks for the great insights regarding $http.get, I wasn't finished with that and I'm going to improve it now based on your comment @Khanh TO.


DRF - limit objects returned to owner

Question: I have a model
class Widget(models.Model):
    title = models.CharField(max_length=100)
    description = models.CharField(max_length=1024)
    username = models.CharField(max_length=50)
    code = models.CharField(max_length=1024)
    owner = models.ForeignKey('MyappUser', related_name='Myapp_owner')
    list = models.ForeignKey('WidgetList')

What I want to be able to do is when calling a specific view -
class WidgetList(generics.ListCreateAPIView):
    queryset = Widget.objects.all()
    serializer_class = WidgetSerializer
    filter_class = WidgetFilter

I only want the widgets owned by the user logged in.
My current filter is -
class WidgetFilter(django_filters.FilterSet):
    owner = django_filters.NumberFilter(name="owner", lookup_type="exact")
    list = django_filters.NumberFilter(name="list", lookup_type="exact")

    class Meta:
        model = Widget
        fields = ['owner', 'list']

Right now I have to do /api/widgets/?owner=3&list=5
But that doesn't prevent another user from viewing the widget.
I had previously been doing it using the following but it breaks my ability to do a filter class -
class WidgetList(generics.ListCreateAPIView):
    queryset = Widget.objects.all()
    serializer_class = WidgetSerializer

    def get(self, request):
        if request.user:
            queryset = self.get_queryset().filter(owner=request.user)
            for object in queryset:
                object.code = decrypt_code(masterkey,
                                                   personalkey,
                                                   object.code)
        else:
            queryset = self.get_queryset()

        page = self.paginate_queryset(queryset)
        if page is not None:
            serializer = self.get_serializer(page, many=True)
            return self.get_paginated_response(serializer.data)

        serializer = WidgetSerializer(queryset, many=True)
        return Response(serializer.data)

What I want to be able to do is if I have a list like this -
Jane
Widget 1
Widget List ABC

Jane
Widget 2
Widget List DEF (which is 3)

Mike (which is 3)
Widget 3
Widget List ABC (which is 5)

Alfred
Widget 4
Widget List ABC

if I do /api/widgets/?list=5 if I'm logged in as Mike I only want to see Widget 3
Comment: Override `get_queryset` and keep the filter_class.
Comment: explain how?  I've been trying 15 different things with no luck.
Comment: Writing an elaborate answer.
Answer: From the docs:

Note that you can use both an overridden .get_queryset() and generic
  filtering together, and everything will work as expected.

So I would do something like:
class WidgetList(generics.ListCreateAPIView):
    queryset = Widget.objects.all()
    serializer_class = WidgetSerializer
    filter_class = WidgetFilter

    def get_queryset(self):
        owner = self.request.user
        return super(WidgetList, self).get_queryset().filter(owner=request.user)

I didn't manage to test the code however my expectation is that get_queryset would only return the entries filtered by owner. So it will not contain entries from other users.
Comment: But this is what I'm already doing with my working example?  The problem is then trying to do a filter isn't doing anything.

`    def get(self, request):
        if request.user:
            queryset = self.get_queryset().filter(owner=request.user)`?
Comment: Because you skipped the `filter_class` on `WidgetList`?
Comment: I think I know the problem.  I need to have the other code in there to get the return as it's decrypting data in a field.  your code works, the problem is where do I put the `for object in queryset: ....` which needs to be there to decrypt the field object.code?
Comment: What happens when you add the `filter_class` on the 2nd example? I see no `filter_class` here - http://imgur.com/MKIyMCc
Comment: BTW, IMO, the decryption should go to the model and then you should use it in the serializer.
Comment: ah thnx!  Yeah I was never sure where to put the decryption however the decryption is based on the user logged in would that still work in the model?

DRF - limit objects returned to owner


How many real square roots can a real invertible matrix have?

Question: Let A be a real square matrix nxn. The real matrix B with the size nxn is considered to be a square root of a matrix A if A=B*B. So I wonder, how many different real square roots can an invertible real matrix have.
I know there can be 0 square roots. If a matrix has a negative determinant, it cannot have square roots. I also know that a matrix can have infinitely many square roots. The example is an identity matrix of a size 2x2. Also a matrix can have exactly 2^n square roots. The example is any positive definite matrix. Generally, if a matrix has a finite amount of square roots, it has to be even, since for every square root matrix B, the matrix -B is also a square root.
So the question is, are there any other options?
I need the answer to prove another theorem, the link is here Prove that there is no isomorphism between groups of matrices of a size n with determinant 1 and matrices of a size n with determinant +1/-1
If it turns out these are all the options, the theorem is proven. Otherwise, I'll need to find at least a single example of a matrix with the smallest non-zero amount of square roots.
The link to wikipedia article on square roots is here, it might also help: https://en.wikipedia.org/wiki/Square_root_of_a_matrix
Comment: Then correct your post, as commented over there.
Comment: "Also a matrix can have exactly 2^n square roots. The example is any positive definite matrix" No. Look at your previous sentence.
Comment: It is not false, your example does not satisfy the theorem. We look at ALL the matrices wit det=1, you've taken only a subgroup of these matrices
Answer: The matrix $A:=\begin{bmatrix} 1 &1 \\ 0 & 1\end{bmatrix}$ has precisely 2 square roots.
For, if $B = \begin{bmatrix} a&b\\c&d\end{bmatrix}$ is a square root of $A$, then we find that $$\begin{bmatrix} a^2 + bc & ab+bd\\ ca+dc & cb+b^2\end{bmatrix} = \begin{bmatrix} 1 & 1\\ 0 & 1\end{bmatrix}.$$
The bottom left equation tells us that $c(a+d) = 0$ so either $c =0$ or $a=-d.$  But note that if $a=-d$, then the top right entry is $0$, instead of $1$.
Thus, we must have $c = 0$.  Then the top left and bottom right equations implies that $a,d\in \{\pm 1\}$.  Since we already know that $a\neq -d$, we must have $a=d = \pm 1$.  The top right equation now tells us that $b = \frac{1}{2a}$.
In summary, the only square roots of $A$ are $$B = \begin{bmatrix} 1 & \frac{1}{2}\\ 0 & 1\end{bmatrix}\text{ and } \begin{bmatrix} -1 & -\frac{1}{2} \\ 0 & -1\end{bmatrix}.$$
More generally, I claim for each $n$, we can find an $n\times n$ matrix with precisely two square roots.  In fact, we can use an $n\times n$ generalization of the above matrix $A$, which is the identity matrix with 1s right above the main diagonal, but $0$s everywhere else has precisely two square roots.  So, for example, the $4\times 4$ matrix I am talking about is $A = \begin{bmatrix} 1 &1 & 0 & 0\\ 0 & 1 & 1 & 0\\ 0 & 0 & 1 & 1\\ 0 & 0 & 0 & 1\end{bmatrix}$.
The matrix $A$ has at most $2$ square roots.
Assume $B^2 = A$.  Note that $A$ is in Jordan normal form with precisely one block.  It follows that the Jordan normal form of $B$ must also have one block.  Since the eigenvalues of $A$ are all one, the eigenvalues of $B$ must therefore be either all $1$s or all $-1$s.  By replacing $B$ with $-B$ if necessary, we may assume the Jordan normal form of $B$ is a single block with all eigenvalues $1$.  That is, the Jordan normal form of $B$ is $A$.  We will show that this property characterizes $B$, so that there are at most $2$ square roots of $A$:   $B$ and $-B$.
So, assume there are two matrices $B$ and $\widetilde{B}$ which are both square roots of $A$ and with both having $A$ as a Jordan form.
Thus, there is a matrix $C$ with $CBC^{-1} = A$.  Likewise, there is a matrix $\widetilde{C}$ with $\widetilde{C}\widetilde{B}\widetilde{C}^{-1} = A$.  Squaring both of these equations, and using the fact that both $B$ and $\widetilde{B}$ are square roots of $A$, we find $$CAC^{-1} = A^2\text{ and } \widetilde{C} A \widetilde{C}^{-1} = A^2.$$  Thus, $CAC^{-1} = \widetilde{C}A\widetilde{C}^{-1}$.  Rearranging, we find that $\widetilde{C}^{-1} C A  = A \widetilde{C}^{-1} C$.  That is, the matrix $\widetilde{C}^{-1} C$ commutes with $A$.
From this MSE question and answer, we deduce that $\widetilde{C}^{-1} C = p(A)$ where $p(A)$ denotes some polynomial in $A$.  Thus, $C = p(A) \widetilde{C}$.  Since $C$ is invertible, so is $p(A)$.  And, since $A$ obviously commutes with $p(A)$, it now follows that $A$ also commutes with $p(A)^{-1}$.
Thus, we find $$B = C^{-1} A C = \widetilde{C}^{-1} p(A)^{-1} A p(A) \widetilde{C} = \widetilde{C}^{-1} A p(A)^{-1}p(A) \widetilde{C} = \widetilde{C}^{-1} A \widetilde{C} = \widetilde{B}.$$
Thus, there are at most $2$ square roots of $A$:  $B$ and $-B$.
The matrix $A$ has at least $2$ square roots
From this MSE question, we know that $A^2$ has $A$ as a Jordan normal form.  So, there is matrix $C$ with $CA^2 C^{-1} = A$.  Then $\pm CAC^{-1}$ are both square roots of $A$.  Indeed, $$(\pm CAC^{-1})^2 = CA^2 C^{-1} = A.$$
Comment: My guess is that by playing around with various Jordan blocks, you can find examples of $n\times n$ matrices having precisely $2k$ square roots for any given $k\leq n$.  I didn't think too deeply about this, though.
Comment: @Andrew:  The matrix $-B$ of my answer works for $odd\times odd$.
Comment: Jason DeVito
Thanks, you really helped me out


stable match algorithm for different sizes groups and limited places

Question: I want to write a program that oriented student to their specialty in university depending on their choices and their range in the specialty.
each specialty can take multiple student (in this case cnm =1 , dsi=2 , rss=2),and the number of student can be more than the number of places (in this case one student not gonna have a place because there is only 5 places ).
The program run with out a problem but the output is not correct .(and I couldn't know why )
.
The output for this example should be ((a,cnm),(c,rss),(b,dsi),(e,dsi),(f,rss))
i found a video on youtube (https://youtu.be/FhRf0j068ZA ,the source code https://github.com/Schachte/stable-matching-algorithm) where the person explain and write a code for stable matching problem , i liked that code so i toke it and i tried to edited so it can be solve my problem
If anyone could help me out I would really appreciate it.
*****this is my code
students = {
'A': ['CNM', 'DSI', 'RSS'],
'B': ['CNM', 'DSI', 'RSS'],
'C': ['DSI', 'CNM', 'RSS'],
'D': ['DSI', 'RSS', 'CNM'],
'E': ['RSS', 'DSI', 'CNM'],
'F': ['RSS', 'CNM', 'DSI']
    }

choices = {
   'DSI': ['C', 'E','D', 'B','F','A'],
   'RSS': ['B','F','A', 'E', 'D','C'],
   'CNM': ['A', 'B','C', 'D','E','F']
   }

rss=2
cnm = 1
dsi=2
results = []
students_with_out_choice =[]

for student in students:
     students_with_out_choice.append(student)

while (len(students_with_out_choice) > 0 ):
      for i in range (3):
          for student in students_with_out_choice:
              for choice in students[student]:
                    if ((choice == "DSI" and dsi != 0) or (choice == "RSS" and rss != 0) or (choice == "CNM" and cnm != 0)):
                       results.append([student, choice])
                       students_with_out_choice.remove(student)
                       if (choice == "DSI"):
                          dsi -= 1
                       if (choice == "RSS"):
                          rss -= 1
                       if (choice == "CNM"):
                          cnm -= 1
                       break

                    else:
                       for key, value in results:
                          if choice == value:
                             a= key

                       etudiant_actuel = choices[choice].index(a)
                       etudiant_proposer = choices[choice].index(student)

                       if (etudiant_actuel >etudiant_proposer):
                           students_with_out_choice.remove(student)

                           students_with_out_choice.append(a)

                           a = student
                           break
                       elif(i==2):
                            students_with_out_choice.remove(student)
                            break

print(results)

Comment: you should use Deferred Acceptance algorithm. Because in each step, only a tentative matching is stored, when a high priority student comes along later, a school can always match with that student, which assures stability.
Comment: Why is your expected output `((a,cnm),(c,rss),(b,dsi),(e,dsi),(f,rss))`, shouldn't it be `((a,cnm),(c,dsi),(b,rss),(e,dsi),(f,rss))`? Did you perhaps mix up b and c?
Comment: @davidlucius I have been trying to add another condition to your code which is some school may have a minimum grade in a specific subject , So even if there is a place for him in the school put he has a per example 9 in math and school only accept people who have more than 10 in math, so even if he could get to the school but because of his math grade he can't go now.
so we can add to your code two addition dictionary :
students_math_grade{
    'A': 12,
    'B': 15,
    'C': 07,
    'D': 18,
    'E': 05,
    'F': 11

}
min_grades{
    'DSI': 12,
    'RSS': 14,
    'CNM': 0
}
Answer: One problem in your code could be that you are changing a list (students_with_out_choice) while iterating over it. This often leads to unexpected results.
As @enke has pointed out, the deferred acceptance algorithm is the method of choice here. An implementation of this algorithm could look something like this with your example:
s = {
    'A': ['CNM', 'DSI', 'RSS'],
    'B': ['CNM', 'DSI', 'RSS'],
    'C': ['DSI', 'CNM', 'RSS'],
    'D': ['DSI', 'RSS', 'CNM'],
    'E': ['RSS', 'DSI', 'CNM'],
    'F': ['RSS', 'CNM', 'DSI']
}

r = {
    'DSI': ['C', 'E', 'D', 'B', 'F', 'A'],
    'RSS': ['B', 'F', 'A', 'E', 'D', 'C'],
    'CNM': ['A', 'B', 'C', 'D', 'E', 'F']
}

p = {
    'DSI': 2,
    'RSS': 2,
    'CNM': 1
}

def get_matched_students(students, rankings, places):

    waiting_students = [student for student in students]
    matching_results = {choice: [] for choice in places}

    def get_waiting_list_without_student(student):
        return [x for x in waiting_students if x != student]

    def get_sorted_results_with_student(student, choice):
        matching_results[choice].append(student)
        return [x for x in rankings[choice] if x in matching_results[choice]]

    while waiting_students:
        for student in waiting_students.copy():
            if not students[student]:
                waiting_students = get_waiting_list_without_student(student)
                continue
            choice = students[student].pop(0)
            if len(matching_results[choice]) < places[choice]:
                matching_results[choice] = get_sorted_results_with_student(student, choice)
                waiting_students = get_waiting_list_without_student(student)
            else:
                if rankings[choice].index(student) < rankings[choice].index(matching_results[choice][-1]):
                    matching_results[choice] = get_sorted_results_with_student(student, choice)
                    waiting_students = get_waiting_list_without_student(student)
                    waiting_students.append(matching_results[choice].pop())

    return matching_results

print(get_matched_students(s, r, p))

This will give you the lists of students that have been matched to each subject: {'DSI': ['C', 'E'], 'RSS': ['B', 'F'], 'CNM': ['A']}
If you want to see the results from the student's perspective, you can do something like this:
results = get_matched_students(s, r, p)

results_by_students = [[student, course] for student in s for course, result in results.items() if student in result]

print(results_by_students)

# Output: [['A', 'CNM'], ['B', 'RSS'], ['C', 'DSI'], ['E', 'DSI'], ['F', 'RSS']]

Comment: the output should be ((a,cnm),(c,rss),(b,dsi),(e,dsi),(f,rss))
Comment: @Abbad Med: Yes, because you changed the order of the input lists compared to your original question. ;) With the new input lists, my function also outputs the desired result: `{'DSI': ['B', 'E'], 'RSS': ['C', 'F'], 'CNM': ['A']}`.
Comment: @Abbad Med: I also just added a way to display the results from the student's perspective (like in your own solutions) in my post.
Answer: I eventually resolved my problem using two different solutions.
the first soltion is that found the bugs in my first program and i corrected it.
students = {
    'A': ['DSI', 'RSS', 'CNM'],
    'B': ['RSS', 'DSI', 'CNM'],
    'C': ['CNM', 'DSI', 'RSS'],
    'D': ['DSI', 'RSS', 'CNM'],
    'E': ['RSS', 'DSI', 'CNM'],
    'F': ['CNM', 'DSI', 'RSS']
}

choices = {
    'DSI': ['B', 'E', 'C', 'D', 'A', 'F'],
    'RSS': ['C', 'F', 'A', 'B', 'E', 'D'],
    'CNM': ['A', 'D', 'B', 'C', 'E', 'F']
}
rss = 2
cnm = 1
dsi = 2
nbr_of_speciality = 3
results = []
students_with_out_choice = []

for student in students:
    students_with_out_choice.append(student)

while len(students_with_out_choice) > 0:
    for student in students_with_out_choice:
        for i in range(nbr_of_speciality):
            choice = students[student][i]
            if (choice == "DSI" and dsi != 0) or (choice == "RSS" and rss != 0) or (choice == "CNM" and cnm != 0):
                results.append([student, choice])
                students_with_out_choice.remove(student)
                if choice == "DSI":
                    dsi -= 1
                if choice == "RSS":
                    rss -= 1
                if choice == "CNM":
                    cnm -= 1
                break

            else:
                free_choice = [k for (k,v) in results if choice == v]
                student_proposer = free_choice[0]
                for k in free_choice :
                    if choices[choice].index(k) > choices[choice].index(student_proposer):
                        student_proposer = k
                rating_of_propose_student = choices[choice].index(student_proposer)
                rating_of_actuel_student = choices[choice].index(student)

                if rating_of_propose_student > rating_of_actuel_student:
                    students_with_out_choice.remove(student)
                    results.remove([student_proposer, choice])
                    results.append([student, choice])
                    students_with_out_choice.append(student_proposer)
                    break
                elif i==nbr_of_speciality-1 :
                    print(f"student  non-oriented : {student}")
                    students_with_out_choice.remove(student)
                    break

print(results)

the seconde solution :

students = {
    'A': ['dsi', 'rss', 'cnm'],
    'B': ['rss', 'dsi', 'cnm'],
    'C': ['cnm', 'dsi', 'rss'],
    'D': ['dsi', 'rss', 'cnm'],
    'E': ['rss', 'dsi', 'cnm'],
    'F': ['cnm', 'dsi', 'rss'],
}
choices = {
    'dsi': ['B', 'E', 'C', 'D', 'A', 'F'],
    'cnm': ['A', 'D', 'B', 'C', 'E', 'F'],
    'rss': ['C', 'F', 'A', 'B', 'E', 'D'],
}
places = {
    'cnm': 1,
    'rss': 2,
    'dsi': 2,
}

left_places = places
results = {}

students_with_out_choices = ['A', 'B', 'C', 'D', 'E', 'F']
choices_left = students

def list_of_student_with_out_a_choice_left_to_prpose(students_with_out_choices, choices_left):
    results = []
    for e in students_with_out_choices:
        if len(choices_left[e]) > 0:
            results.append(e)

    return results

def last_choice_in_ranking(s, choices_left, results, choices):
    students_with_out_choices = []
    ranging_list = choices[choices_left[e][0]]
    ranging_student_list = []
    for key in results:
        if results[key] == choices_left[e][0]:
            students_with_out_choices.append(key)
    for i in range(len(ranging_list)):
        if ranging_list[i] in students_with_out_choices:
            ranging_student_list.append(ranging_list[i])
    if ranging_list.index(e) < ranging_list.index(ranging_student_list[-1]):
        return ranging_student_list[-1]
    return s

list_of_oriented_student = students_with_out_choices
while len(list_of_oriented_student) > 0:
    e = list_of_oriented_student[0]
    first_choice = choices_left[e][0]
    if places[first_choice] > 0:
        results[e] = first_choice
        left_places[first_choice] -= 1
        students_with_out_choices.remove(e)
    elif len(choices_left[e]) > 0:
        student_less_rank_than_e = last_choice_in_ranking(e, choices_left, results, choices)
        if student_less_rank_than_e != e:
            del results[student_less_rank_than_e]
            results[e] = first_choice
            students_with_out_choices.remove(e)
            students_with_out_choices.append(student_less_rank_than_e)
    choices_left[e].remove(choices_left[e][0])
    list_of_oriented_student = list_of_student_with_out_a_choice_left_to_prpose(students_with_out_choices, choices_left)
print("results: ", results)

liste_of_student_non_oriented=[]
for key in students:
    if key not in results:
        liste_of_student_non_oriented.append(key)

print(f"students non-oriented : {liste_of_student_non_oriented}")


Using Firebug, under  why is the first line

Question: I wanted to view how other people code their websites, but got stuck trying to understand what this does:
<html class="js flexbox flexboxlegacy canvas canvastext webgl no-touch geolocation postmessage no-websqldatabase indexeddb hashchange
             history draganddrop websockets rgba hsla multiplebgs backgroundsize borderimage borderradius boxshadow textshadow opacity
             cssanimations csscolumns cssgradients no-cssreflections csstransforms csstransforms3d csstransitions fontface generatedcontent
             video audio localstorage sessionstorage webworkers applicationcache svg inlinesvg smil svgclippaths"
      lang="en" style="">

What is the purpose of this and can someone please explain it to me?
Comment: They are [using Modernizr](http://stackoverflow.com/q/16138424/11683).
Answer: Because they are using modernizr
Modernizr detects features which are supported and which are not supported by the browser. In response to it appends different classes HTML element. Which are then can be written as to fix cross browsers compatibility issues.
As  is document entry point or main container therefor it adds classes to HTML tag.
Similar Question
Answer: The classes in html tag is because of the Modernizr js included in the project.
Modernizr is a javascript plugin which on the fly identifies browser capabilities and respected classes on the html tag. For instance if the browser which supports border-radius property of CSS Modernizr adds a class named borderradius and if the browser does not supports border-radius it adds a class named no-borderradius, like wise for all other properties.
So when we working on project we can add a fall back design by targeting no-borderradius class for the browsers' not supported by border-radius feature of CSS3.
Comment: Targeting `no-borderradius` will treat browsers with javascript disabled as capable. (Just sayin'.)
Answer: They're using modernizr for feature detection.
As a contrived example, if a browser didn't support CSS3 transitions, you could choose to add graceful alternatives. For example, you might check if the HTML element had the csstransitions class before adding any programmatically w/ js.
Modernizr also has a great set of useful javascript methods. For instance, theres a method to test which vendor prefix you will need to use if any.
very useful, so you should
CHECK IT OUT


Hidden Markov Model: Expectation maximization

Question: I am trying to understand the EM algorithm for Hidden Markov Model (HMM). However, I don't see how the complete log-likelihood is calculated.
Let $x_{1:T}$ be a sequence of observed random variables $X_1,...,X_T$ and $z_{1:T}$ be the sequence of hidden discrete valued random variables $Z_1, Z_2,...,Z_T$.
The joint distribution has the following form:
$$p(z_{1:T}, x_{1:T}) = p(x_{1:T}|z_{1:T})p(z_{1:T}) = \prod_{t=1}^{T}p(x_t|z_i)p(z_1)\prod_{t=2}^{T}p(z_{t}|z_{t-1})$$
I don't see how one can compute either one of this quantities. There is a lot of stuff discussed in the HMM context like $p(z_t = i|x_{1:t})$ - forward algorithm, $p(z_{t}=i|x_{1:T})$-forward/backward algorithm. But I don't know how this can be used to calculate the EM.
Comment: In an HMM you have expressions for $p(z_t | z_{t-1})$ and $p(x_t | z_t)$ so the joint likelihood is just a product.
Answer: The best-known version of EM algorithm applied to a Hidden Markov Model is the Baum-Welch algorithm.  The Wikipedia article to which I have just given a link seems to me to contain a reasonably accessible exposition of it, although I think some of the notation isn't particularly well-chosen (I'd prefer the simpler $\ b_{ji}\ $ instead of the unneccessarily verbose $\ b_j\big(y_i\big)\ $, for instance).
In your notation, starting from your current estimates for the initial state probabilities, $\ \pi_i=p\big(Z_1=i\big)\ $ for all $\ i\ $, transition matrix entries, $\ a_{ij}=p\big(Z_{t+1}=j\,\big|\,Z_t=i\big)\ $ for all $\ i\ $ and $\ j\ $, and observation output probabilities, $\ b_{ik}=p\big(X_t=k\,\big|\,Z_t=i\big)\ $ for all $\ i\ $ and $\ k\ $, you need to calculate:

$\ p\big(Z_t=i\,\big|\,X_{1:T}=x_{1:T}\big)=\frac{p\big(Z_t=i,X_{1:T}=x_{1:T}\big)}{p\big(X_{1:T}=x_{1:T}\big)}\ $ for all $\ i\ $;
$\ p\big(Z_t=i, Z_{t+1}=j\,\big|\,X_{1:T}=x_{1:T}\big)=\frac{p\big(Z_t=i, Z_{t+1}=j,X_{1:T}=x_{1:T}\big)}{p\big(X_{1:T}=x_{1:T}\big)}\ $ for all $\ i, j\ $ and $\ t\ $;
$\ p\big(Z_t=i, X_t=k\,\big|\,X_{1:T}=x_{1:T}\big)=\frac{p\big(Z_t=i, X_t=k,X_{1:T}=x_{1:T}\big)}{p\big(X_{1:T}=x_{1:T}\big)}\ $ for all $\ i, k\ $ and $\ t\ $.

I think the nicest expositions of the Baum-Welch algorithm are those which express the calculations in terms of matrix multiplication.  If $\ n\ $ is the number of hidden states, define the following matrices and column vectors:

$\ C(k)\ $, for each possible observed output $\ k\ $, is the $\ n\times n\ $ matrix whose entry in row $\ i\ $ and column $\ j\ $ is $\ a_{ij}b_{ik}\ $;
$\ \pi\ $ the $\ n\times1\ $ column vector whose $\ i^\text{th}\ $ entry is $\ \pi_i\ $;
$\ \epsilon(i)\ $, for each hidden state $\ i\ $, is the $\ n\times1\ $ column vector whose $\ i^\text{th}\ $ entry is $1$ and whose other entries are all $0$.
$\ \mathbf{1}\ $ is the $\ n\times1\ $column vector whose entries are all $1$.

In terms of these matrices and column vectors, the quantities you need to calculate are given by:
\begin{align}
&p\big(X_{1:T}=x_{1:T}\big)=\pi^TC\big(x_1\big)C\big(x_2\big)\dots C\big(x_T\big)\mathbf{1}\ ,\\
&p\big(Z_t=i,X_{1:T}=x_{1:T}\big)=\\
&\hspace{7em}\cases{\pi_i\epsilon(i)^TC\big(x_1\big)C\big(x_2\big)\dots C\big(x_T\big)\mathbf{1}&if $\ t=1$\\
\pi^T\prod_\limits{s=1}^{t-1}C\big(x_s\big)\epsilon(i)\epsilon(i)^T\prod_\limits{s=t}^TC\big(x_s\big)\mathbf{1}&if $\ 1<t\le T$
}\\
&p\big(Z_t=i,Z_{t+1}=j,X_{1:T}=x_{1:T}\big)=\\
&\hspace{3em}\cases{\pi_ia_{ij}b_{ix_1}\epsilon(j)^TC\big(x_2\big)C\big(x_3\big)\dots C\big(x_T\big)\mathbf{1}&if $\ t=1$\\
\pi^T\prod_\limits{s=1}^{t-1}C\big(x_s\big)\epsilon(i)a_{ij}b_{ix_t}\epsilon(j)^T\prod_\limits{s=t+1}^TC\big(x_s\big)\mathbf{1}&if $\ 1<t\le T-1\ $.
}\\
&p\big(Z_t=i, X_t=k,X_{1:T}=x_{1:T}\big)=\\
&\hspace{11em}\cases{p\big(Z_t=i,X_{1:T}=x_{1:T}\big)&if $\ k= x_t$\\
0&otherwise}
\end{align}
From these quantities, you can now obtain
\begin{align}
 &p\big(Z_t=i\,\big|\,X_{1:T}=x_{1:T}\big)\ ,\\ &p\big(Z_t=i, Z_{t+1}=j\,\big|\,X_{1:T}=x_{1:T}\big)\ \text{, and}\\ &p\big(Z_t=i, X_t=k\,\big|\,X_{1:T}=x_{1:T}\big)\ ,
\end{align}
and calculate the conditional expectations, given the observations, of:

The number of times each hidden state $\ i\ $ was visited during the instants $\ t=1,2,\dots, T-1\ $,
$$\sum_{t=1}^{T-1}p\big(Z_t=i\,\big|\,X_{1:T}=x_{1:T}\big) $$
and the number of times it was visited during the instants $\ t=1,2,\dots, T\ $,
$$\sum_{t=1}^Tp\big(Z_t=i\,\big|\,X_{1:T}=x_{1:T}\big) $$
The number of times the hidden Markov chain made a transition from state $\ i\ $ at time $\ t\ $ to state $\ j\ $ at time $\ t+1\ $,
$$ \sum_{t=1}^{T-1}p\big(Z_t=i, Z_{t+1}=j\,\big|\,X_{1:T}=x_{1:T}\big)$$
The number of times during the instants $\ t=1,2,\dots, T\ $ that $\ X_t=k\ $ and the hidden state was $\ i\ $,
$$ \sum_{t=1}^Tp\big(Z_t=i, X_t=k\,\big|\,X_{1:T}=x_{1:T}\big)\ .$$
This completes the E (Expectation) stage of the EM algorithm.  You now perform the M (Modification) stage by replacing the quantities $\ \pi_i\ $, $\ a_{ij}\ $ and $\ b_{ik}\ $ with their new estimates,
\begin{align}
\hat{\pi}_i&=p\big(Z_1=i\,\big|\,X_{1:T}=x_{1:T}\big)\ ,\\
\hat{a}_{ij}&=\frac{\sum_\limits{t=1}^{T-1}p\big(Z_t=i, Z_{t+1}=j\,\big|\,X_{1:T}=x_{1:T}\big)}{\sum_\limits{t=1}^{T-1}p\big(Z_t=i\,\big|\,X_{1:T}=x_{1:T}\big)}\ \text{, and}\\
\hat{b}_{ik}&=\frac{\sum_\limits{t=1}^Tp\big(Z_t=i, X_t=k\,\big|\,X_{1:T}=x_{1:T}\big)}{\sum_\limits{t=1}^Tp\big(Z_t=i\,\big|\,X_{1:T}=x_{1:T}\big)}\ .
\end{align}


multiple comparison for discrete nonparametric distributions

Question: I have $N=24$ controls and $M=10$ patients. ($M$ can be $20$ later.)
Each subject has one $3D$ image, each element of the image is a discrete value (integer) ranging from $0$ to $4$ $[0,1,2,3,4]$. It's like a score.
I want to find a set of voxels whose scores are significantly different between two groups, i.e. in which regions the scores were significantly increased in patient group compared to controls.
As it is difficult to assume a parametric distribution of the scores across the whole image, I applied Wilcoxon rank sum test for each voxel, using $N$ integers distribution from controls and $M$ integers distribution from patients.
1. My first question is that if Wilcoxon rank sum test is a good choice for this kind of discrete data distributions.
2. My second question is about multiple comparison controlling false discovery rate for this test is appropriate.
If you have better idea please teach me.
Comment: What do you intend by "nonparametric distribution" in your title? (e.g. are you referring to some form of nonparametric density estimation?)
Comment: It was not appropriate to assume that the data histrogram follows student t-distribution or other types of parametric distribution. I didn't mean other form of nonparametric density estimation.
Answer:
First question: your data being ordinal, a Wilcoxon rank sum test seems an appropriate choice.
Second question: yes, you need to correct for multiple comparison (see http://www.researchgate.net/publication/255651552_Neural_correlates_of_interspecies_perspective_taking_in_the_post-mortem_Atlantic_Salmon_an_argument_for_multiple_comparisons_correction for a convincing example with MRI data). Moreover, as your data results from a spatial grid so that they can be considered as not independent each other, using a correction method based on independence could be too conservative and I suggest you to look for correction methods involving a spatial dependence on the data (e.g. http://onlinelibrary.wiley.com/doi/10.1002/hbm.1058/full but many other approaches exist)


Trouble getting my html tags to show up inside my div border

Question: I am new to this, so the page is for practice. I can't seem to get the "p" or "h1" tags to show up in, what should be the middle of my page. Here is my html code:

My CSS looks like this:
#top,
#bottom,
#left,
#right {
background: #666666;
position: fixed;
}

#left,
#right {
top: 0;
bottom: 0;
width: 100px;
}

#left {
left: 0;
}

#right {
right: 0;
}

#top,
#bottom {
left: 0;
right: 0;
height: 100px;
}

#top {
top: 0;
}

#bottom {
bottom: 0;
}

If anyone could let me know what I am doing wrong it would be very appreciated. I tried to mess with the positioning a bit with no luck, and I also attempted to give the p tags a z-index but I don't know if that was even something that could have worked properly to begin with.
Answer: A trick to center elements in the middle is:
#left{
    margin-left: auto;
    margin-right: auto;
}

but if you're just looking to center the
 or
 then this would work:
p {
    text-align: center;
}
h1 {
    text-align: center;
}

Answer: This is a perfectly acceptable and simpler way to achieve what you want.

body {
  background-color: #666666;
  width: 100%;
  margin: 0;
}
#container {
  background-color: blue;
  width: 80%;
  height: 100px;
  margin: auto;
}
h1 {
  text-align: center;
}
p {
  text-align: center;
}

HEADER 1

PARAGRAPH 1

Answer: Go with float and clear in the css segments

top

left
background

right

bottom

Comment: This is awesome! Thank you! It looks like my whole setup was completely wrong :(
Comment: Maybe a little bit, but the attempt was really good for the beginning. Look out for w3schools.com you can find a lot of help on that site, too.


How do I unit test/Mock with fake data with activejdbc?

Question: I'd like to mock some DB inserts and reads, IE, not do them to my actual production DB, but either create a mock, or do a temporary in memory type of thing, to be able to unit test some methods getting back fake data.
How do I do this in activejdbc?
Answer: When you need to test the models' functions in your project,  you would use the class DBSpec as a super class of your JUnit test like this:
public class MovieSpec extends DBSpec {

    @Test
    public void shouldValidateMovieAttributes() {
        Movie movie = new Movie();
        the(movie).shouldNotBe("valid");
        movie.set("title", "blah", "year", "1122");
        the(movie).shouldBe("valid");
    }

    @Test
    public void shouldStoreDirectorAndMovies() {

        Person director = new Person("Stephen Spielberg");
        director.saveIt();

        director.add(new Movie("Saving private Ryan", 1998));
        director.add(new Movie("Jaws", 1982));

        a(Movie.count()).shouldBeEqual(2);

        List<Movie> movies = director.getAll(Movie.class).orderBy("year");

        the(movies.get(0).getTitle()).shouldBeEqual("Jaws");
        the(movies.get(1).getTitle()).shouldBeEqual("Saving private Ryan");
    }
}

A lot of things are happening here. The super class DBSpec will do the following automatically:
Before test executes:

opens a connection
starts a new transaction on the DB connection

After test executes:

rolls back DB transaction (deletes, inserts updates)
closes DB connection

This way, you only write code  that is related to your business logic, and not worry about opening/closing connections.
Also, because DBSpec rollbacks your transactions, your test database is always in a clean state. For a full example of a project that shows how this works,   please refer to: https://github.com/javalite/simple-example


Homogenous different equation solved correctly?

Question: Do I solve this homogeneous differential equation correctly? Thank you.
$$(2x^3 +y^3)dx -3xy^2dy =0$$
Page $\boldsymbol 1$:
\begin{align}
-3xy^2dy &= -(2x^3 +y^3)dx \\
 3xy^2dy &=  (2x^3 +y^3)dx \\
\frac{dy}{dx} &= \frac{2x^3 +y^3}{3xy^2} \\
\frac{dy}{dx} &= \frac{2 +\dfrac{y^3}{x^3}}{3\dfrac{y^2}{x^2}} \quad(\text{divided by }x^3)\\
\frac{dy}{dx} &= \frac{2 +\left(\dfrac{y}{x}\right)^3}{3\left(\dfrac{y}{x}\right)^2}
\end{align}
Let $p=y/x$ so that $y=px$ and $y'=p'x+p$. Then,
\begin{align}
p'x +p &= \frac{2+p^3}{3p^2} \\
p'x &= \frac{2+p^3}{3p^2} -p \\
p'x &= \frac{2+p^3-3p^3}{3p^2} = \frac{2-2p^3}{3p^2} \\
\frac{dp}{dx} \cdot x &= \frac{2-2p^3}{3p^2} \\
\int \frac{3p^2}{2-2p^3} dp &= \int \frac{dx}{x}
\end{align}
Page $\boldsymbol 2$:
\begin{align}
-\frac12 \ln |2-2p^3| &= \ln|x| +c \\
\ln |2-2p^3|^{-1/2} &= \ln|x| +c \\
(2-2p^3)^{-1/2} &= x +c \\
\left(2-2\left(\frac{y}{x}\right)^3\right)^{-1/2} &= x \\
2^{-1/2} -\left(\frac{2y^3}{x^3}\right)^{-1/2} &= x +c \\
\frac{1}{\sqrt{2} -\frac{2y^{-3/2}}{x^{-3/2}}} &= x +c \\
-\frac{2y^{-3/2}}{x^{-3/2}} &= x -\frac{1}{\sqrt{2}} +c \\
-2y^{-3/2} &= x^{-3/2} \left(x -\frac{1}{\sqrt{2}} +c\right) \\
y^{-3/2} &= \frac{x^{-3/2} \left(x -\dfrac{1}{\sqrt{2}} +c\right)}{-2} \\
(y^{-3/2})^{(-2)} &= \left(\frac{x^{-3/2} \left(x -\dfrac{1}{\sqrt{2}} +c\right)}{-2}\right)^{-2} \\
y^3 &= \left(\frac{x^{-3/2} \left(x -\dfrac{1}{\sqrt{2}} +c\right)}{-2}\right)^{-2}
\end{align}
Comment: Now you have to write the rest.
Answer: Your first page is correct. However, your second page has a mistake, since $(a+b)^{-\frac{1}{2}} \neq a^{-\frac{1}{2}}+b^{-\frac{1}{2}}$. Also, I suggest in the future that you simplify arbitrary constants when possible by defining new constants as I have done on my solution below.
Hence, I suggest you do the following:
We start from the point where I believe there are no errors.
$$-\frac{1}{2} \ln{|2-2p^3|}=\ln{|x|}+C$$
$$\ln{|2-2p^3|}^{-\frac{1}{2}}=\ln{|x|}+C$$
Exponentiating both sides:
$$e^{\ln{|2-2p^3|}^{-\frac{1}{2}}}=e^{\ln{|x|}+C}$$
$$(2-2p^3)^{-\frac{1}{2}}=e^{\ln{|x|}}\cdot e^C$$
We let $e^C=k$. This is ok to do, since $C$ is an arbitrary constant, hence $k$ will also be arbitrary.
$$(2-2p^3)^{-\frac{1}{2}}=kx$$
$$2-2p^3=(kx)^{-2}$$
Factorising:
$$2(1-p^3)=\frac{1}{(kx)^2}$$
$$1-p^3=\frac{1}{2(kx)^2}$$
$$p^3=1-\frac{1}{2(kx)^2}$$
$$p^3=\frac{2(kx)^2-1}{2(kx)^2}$$
$$p^3=\frac{2k^2 x^2-1}{2k^2 x^2}$$
Let $k_1=2k^2$
$$p^3=\frac{k_1 x^2-1}{k_1 x^2}$$
Dividing numerator and denominator by $k_1$:
$$p^3=\frac{x^2-\frac{1}{k_1}}{x^2}$$
Let $k_2=-\frac{1}{k_1}$
Substitute $p=\frac{y}{x}$
$$\frac{y^3}{x^3}=\frac{x^2+k_2}{x^2}$$
$$y^3=x^3+k_2 x$$
$$y=\sqrt[3]{x^3+k_2 x}$$
Hence, the general solution to our differential equation is:
$$y(x)=\sqrt[3]{x}\sqrt[3]{x^2+k_2}$$
Where $k_2$ is an arbitrary constant.
Please do not hesitate to ask if you have any doubts or questions.


JQuery mouseenter() and mouseleave()

Question: So I have got this demo navigation, which has a small button on the side and when you hover the button, it slides the menu into the window. though I have got the hover working, but now when the mouse leaves, it's still open. how to fix this? I'm pretty new to jQuery by the way
here's the html:

    <button class="open"></button>

Home Pagina

Product Pagina

Bestel Pagina

And my jQuery:
$("#demoNav").mouseenter(function(){
       $("#demoNav").animate({marginLeft:'0px'}, 500)
});

If you need more info, just tell me, I'll provide more codes.
Comment: this is not the hover on button, this is mouse enter inside the div layout, do a $("#demoNav .open").hover(function(){ //XXXXX code to toggle });
Answer: You haven't actually told it to hide again.
That said, I'd like to suggest this CSS alternative:
#demoNav {
    transition:margin-left 0.5s ease;
    -webkit-transition:margin-left 0.5s ease;
}
#demoNav:hover {
    margin-left:0;
}

Answer: Try this:
$("#demoNav").hover(

function () {
    $(this).animate({
        marginLeft: '0px'
    }, 500)
},

function () {
    $(this).animate({
        marginLeft: '50px'
    }, 500)
});

Answer: Add a mouseleave event -
$('#demoNav').mouseleave(function(){
   $("#demoNav").animate({marginLeft:'50px'}, 500)
});

Answer: Use jQuery:
$("#demoNav").hover(function(){
    // do something when mouse hover
},function(){
    //do some thing when mouseout
});

Answer: you coded for mouseenter but forgot to code for mouseleave event
add these lines in jQuery
$("#demoNav").mouseleave(function(){
       $("#demoNav").animate({'marginLeft':'50px'}, 500)
});

working example here:
http://jsfiddle.net/EP6wR/
but i will suggest use hover method which is cleaner
syntax: $(selector).hover(handlerIn, handlerOut)

Answer: might be this solution works
 $('#demoNav .open').on("mouseenter", function(){  $(this).parent().animate({marginLeft:'0px'}, 500); });
 $('#demoNav').on("mouseleave", function(){  $(this).animate({marginLeft:'50px'}, 500); });

this will activate the pane on hover of button, but when you leave the div, it'll act again making a left move

JQuery mouseenter() and mouseleave()


RHEL 6.3 Upgrading OpenSSH & Apache

Question: We just ran an external security scan from 403Labs against one of our servers (RHEL 6.3 x86_64) for PCI compliance and the results appeared to mainly dictate that we had a hand full of applications that needed to be upgraded to pass the scan.
That having been said, the problem I am encountering is that the package manager (yum) and the use of the remi repo do not have the versions I need for Apache and OpenSSH. I have already performed the following:
<code>yum update
yum --enablerepo=remi,remi-test install httpd mysql mysql-server php php-common
</code>
This resolved our critical and high risk results, but the medium level results are still stating that we need to further upgrade the following packages.
The upgrades we need are:
<code>   Current            Required
Apache 2.2.15 to >= Apache 2.2.23
OpenSSH 5.3   to >= 5.7
</code>
So, since the package manager is not capable of letting me upgrade to those versions, how must I go about doing this? I'm currently under the premise that I will need to install from source. If there's a better alternative, please indicate that.
Also, if I have no choice but to install from source, can someone please help me identify what the proper source packages would be so that I know I am installing the correct versions for my OS?
Thank you very much for any help.
Answer: Push back...
Red Hat Enterprise Linux doesn't work that way. Installing from source to meet audit requirements opens you to additional security issues and more management overhead.
The approach Red Hat takes for its enterprise operating systems is to create a consistent target throughout the support lifecycle of the OS. Larger corporations and enterprise applications need to guarantee binary compatibility throughout the 7+ years that these operating systems are supported for. Red Hat will not change minor version numbers of a package, but will instead backport changes and security patches from newer versions into the older package.
E.g., you'll never see Apache 2.2.23 in RHEL 5, but you'll see the relevant security patches (and some functionality) ported from newer versions of Apache into the 2.2.3.
When I deal with auditors or security-focused people, I insist that they read through the package changelog to see if their specific concern is covered by existing security fixes or bugfixes...
Here's the EL Apache changelog.
Cross-reference the CVE-* numbers against the CERT/National Vulnerability Database. For instance, CVE-2011-3368 lists a vulnerability that affects Apache versions prior to 2.2.21. Obviously, RHEL only has 2.2.3 as a major version, so the security fix was developed, tested and ported to the old version.
Answer: Don't do that !
Before you step outside the OS vendor's support structure you should verify that this is the right thing to do.
Some PCI compliance tests will report that an application has vulnerabilities because it's reported version number is too low. This does not take into account backporting of security and bug fixes that many vendors employ.
For example (from an old Nessus scan) it declares that Apache supplied by CentOS is vulnerable if the version is <2.2.14. If you dig into the detail about what the vulnerabilities are then you discover CVE-2009-3095, CVE-2009-3094 etc.
Looking them up you discover that they have been fixed in current versions of of Apache supplies buy RH and thus CentOS.
Comment: @Skittles: Yuu need a detailed list of the vulnerabilities which you can then cross reference against the fixes.
Comment: Thank you, lain. So, if 403Labs is telling us that our site is not PCI compliant based on that exact scenario and I have applied all updates via yum update, then what coarse of action should I take to let my management know that we actually are fine? I'm not very versed in this sort of thing, so I want to make sure that his concerns have been addressed accordingly?
Comment: Tell them to cross-reference the exploit/bug against the [vulnerability database](http://web.nvd.nist.gov/view/vuln/search) to see if RHEL backported the changes.
Comment: @lain - I do have a detailed list as a scan report from 403Labs.
Comment: @ewwhite - I will try to see if we can do that. Thank you.
Comment: @Skittles: You should work through the list and discount what you can.
Comment: @lain - Here is one of the bugs as noted on nist.gov. https://bugzilla.redhat.com/show_bug.cgi?id=659297 Is this safe to assume that it's one I can discount?
Comment: @Skittles: It is closed not a bug, the comments 2 and 3 say why. I would use it as evidence to discount that vulnerability.
Comment: Thank you, lain! That's what I needed to work this out. I am ever so grateful for your assistance.
Answer: I ran into this with my PCI compliance implementation as well, though I knew about the back-porting of patches mentioned above.  Our solution with apache was to set ServerTokens to 'Prod' and ServerSignature to 'Off' in /etc/httpd/conf/httpd.conf.  This wise setting tells apache not to report its version numbers and spit out all the modules it is using. We then passed the scan.
As for SSH, the ideal setup is to firewall off access so that only your office can connect to the port. This way the scan won't pick it up.
If that's not possible, PCI auditors can be convinced you are secure if you prove you're patched up (usually via some screen sharing utility you show that a 'yum update' shows no updates available) and have a procedure in place for regular tests. I just showed that I was subscribed to RedHat's security mailing list and that we immediately schedule an update if a component we're using is vulnerable.
Barring all that, you should be able to appeal and get a proper penetration test.
Comment: This is an extremely good answer and I have just implemented the changes that you noted and will fire off another scan to see if this takes care of the remain medium level risks that we had. 15 in all and I am in the process of requesting exemptions for many of them.
Comment: Awesome.  Let me know how it goes.

RHEL 6.3 Upgrading OpenSSH & Apache


Help needed to Interpret ln(y) = a +b (Standardized X)

Question: I am analysing server data and I have a scenario where I need to get the % by which Y is changed because of a unit change in X:
EDIT: I am doing a Linear Regression in Python
(and its other forms like Lasso - ultimate aim is to find feature importances)
My Y is a continuous variable.
My Xs are all standardized (meaning : x-xmean/xstd.dev)
Case 1:
ln(y) = a +  b (Standardized X)
When X is Increased by 1 standard deviation, then Y increases by b *100 % or [ exp(b) -1 ] *100 %
So when X increases by 1 unit , does Y increases by b*100/std.deviation of X   % or  [ exp(b) -1 ] *100 / std.dev(X) % ?
or should I un-standardize the coeff and take it as:
% change in Y for 1 standard deviation change in X is [ exp{ b1 / std.dev(X) } -1 ] *100  ?
Case 2:
ln(y) = a +  b (Standardized X)
Here X is a % , Eg:  % of memory used at the moment, or % of cpu time spent on a job , etc.
How should I interpret % change in Y in this case?
Data in my target (Y) is as shown in the pic below:
Comment: Would you please post the raw data before taking logs or standardizing?
Comment: @JamesPhillips, I have added a pic of the raw data for Y , andI have many X cols , like 1300 cols or so ...
Comment: Dears, I have edited my question with more clarity on my understanding ...
Answer: For a one standard deviation increase in $X$, $\ln y$ is expected to increase by $b$ units. That's the only interpretation you can get from this model.
To use the % change interpretation, you need to model $\ln(E[y]) = a + b Z$ (where $Z = X/\sigma$). You've modeled $E[\ln y] = a + b Z$. The first model is a generalized linear model with a log link. The second model is a linear model with a log-transformed outcome.
In the first model, if you take $\exp$ of both sides, you get $$E[y] = \exp(a + bZ)=\exp(a)\exp(bZ)=\alpha \ \exp(bZ)$$
To see how $E[y]$ changes when we increase $Z$ by 1 (i.e., increase $X$ by one stndard deviation), we can simply plug, going from $Z = 0$ to $Z=1$.
$$E[y|Z=0]=\alpha \ \exp(b \times 0) = \alpha$$
$$E[y|Z=1]=\alpha \ \exp(b \times1) = \alpha \ \exp(b)$$
So, for a one standard deviation increase in $X$, $E[y]$ increase by a factor of $\exp(b)$.
In the second model, if you take $\exp$ of both sides, you get
$$\exp(E[\ln y]) = \exp(a + bZ)$$
The left side is not reducible, so we can't go further down this path. The only way to interpret this model is by interpreting the linear change in $E[\ln y]$, as I did in the beginning of this post.
This distinction has been discussed here, here, and here on CV and here.
Another note is that you shouldn't standardize a predictor that is already in interpretable units like percentage points. It only muddies the interpretation.
Comment: Dear @Noah , I am using Python - Linear Regression and was wanting to use a Log-Linear model .... I will check your links and get back to you...
Comment: Dear @Noah , Also , my Y is a continuous value ...
Comment: The term standardized would lead me to guess $Z = (X  - \bar X)\  /\ \text{SD}(X)$
Comment: Dear @NickCox, you are correct...
Comment: Dears , I have updated my post for more clarity...
Comment: Dear @Noah, I am working in python where most models are under Generalised Linear model category ... so do you mean to say that I run a Linear regression , get the predicted values of y and then take a ln() and use this to interpret the coefficients? I am a bit confused and worried how to achieve this ...
Comment: No, you need to run a generalized linear model with a log link. Do not transform anything. I don't know how to use Python, sorry, but you can ask how to do this on StackOverflow. The interpretation of $b$ doesn't change regardless of whether you center $X$ or not, so I left that part out for simplicity.


Adding random numbers into a linked-list using loop

Question: I want to add a completely random number into a linked-list, but instead of having all of my code in main, I want to use the object-oriented approach. I have my standard Node class and it's header, and then in main I want a loop that runs through 20 times and then stops adding more. I was given my insert function and how it would be called in main, but I can't seem to get the random numbers to work. I understand that you can't assign an int to a class, but I don't really know how else to incorporate my random number into my function in order to insert it into my list.
Here is my code. Observe the error on line 20 of main. Any insight would be great. Thanks!
Main.cpp
#include <iostream>
#include <iomanip>
#include <time.h>
#include "Node.h"

using namespace std;

int main()
{
    srand(time(NULL));

    Node link_head;
    Node instance;

    for (int i = 0; i < 20; i++)
    {
        int random = rand() % 100;

        instance.insert(&link_head, &random);
    }
}

Node.h
#include <iostream>
#include <iomanip>

#ifndef NODE_H
#define NODE_H

typedef int ElementType;

class Node
{
public:

    Node();
    ElementType data;
    Node *next;
    int insert(Node *, Node *);

};

#endif NODE_H

Node.cpp
#include <iomanip>
#include <iostream>
#include "Node.h"

using namespace std;

Node::Node()
{
    this -> data = 0;
    this -> next = NULL;
}

int Node::insert(Node *link_head, Node *newNode)
{
        Node *current = link_head;

        while (true)
        {
            if(current->next == NULL)
            {
                current->next = newNode;
                break;
            }
            current = current->next;
        }

    return 0;
}

Answer: You are sending the address of an int to a function requiring a pointer on a Node. Allocate a new node first then send it to the function.
for (int i = 0; i < 20; i++)
{
    int random = rand() % 100;

    Node* newNode = new Node;
    newNode->data = random;
    instance.insert(&linkHead, newNode);
}

As stated, insert method should really be static of even a free function since it access only public members of the struct.
Comment: It's not about the point, it's about the fact that my answer is not wrong but if you really want you 10 points. I am good with that.
Comment: Thank you. I was too busy worrying about the error to focus on the logic of my code. It was a super easy fix. Darn coding.
Comment: @AustinAllenJulio Have a look at my answer, may be this enlightens you a bit more ...
Comment: Hum why the downvote, at least comment so I can improve the answer.
Comment: _'Hum why the downvote'_ `instance.insert(&linkHead, newNode);` will fix the compiler error, but not the flawed concept.
Comment: While I agree, the question is not about that topic and for what it's worth, the method is not static(and he stated that the insert method was given to him) so I wouldn't downvote an answer for that. Cheers.
Comment: @EricFortin Yes, OP's tend to ask the wrong questions about their code ;) ... My experience is that this kind of answer generates follow up questions or comments (even if accepted) like: _But now I have the following problem, blah blah ..._
Comment: Downvote their question then ;) But like I said, insert function was given to him and it is not static so what can I do ...
Comment: @EricFortin Hmmm, no. The question is valid, shows the relevant code, and mentions the (primary) problem, a.s.o. If you really care about your 2pts rep, I'll release my downvote (if you at least hint the OP about their other flaws, or upvote my answer alternatively ;D ).
Comment: @EricFortin Yes sometimes I just want my ten points, and how it is now it should really help the OP, or other researchers landing here ;) ...
Answer: Your code is flawed in several ways.

instance.insert(&link_head, &random); &random doesn't point to a Node, thus the compiler error
int insert(Node *, Node *); should be static int insert(Node **, Node *); and used as follows

Node* head = NULL;

for (int i = 0; i < 20; i++)
{
    Node* newNode = new Node;
    newNode->data = rand() % 100;
    Node::insert(&head, newNode);
}

Where the implementation looks like:
int Node::insert(Node** link_head, Node *newNode)
{
    if(!link_head) {
        return -1;
    }
    if(!(*link_head)) {
        *link_head = newNode;
    }
    else {
        newNode->next = (*link_head)->next;
        (*link_head)->next = new_node;
    }
    return 0;
}

The difference is you use a head reference as an anchor for the linked list and you'll not have a useless instance that always needs to be sorted out from the actual values stored in the list.


How to upload files under taxonomy?

Question: Suppose in a School's website, I have some departments. They want their class lectures, class routines, course curriculum to be uploaded under their departments. Typically we did some very simple things - all the departments are now simple hierarchical pages. How to go with it.
We thought to go with a new content type, where the taxonomy will be the department name, where under the department names will be the type of document (whether class routine, or class lecture etc.) and under that taxonomy if we write a content, then we can go through the taxonomies to the content. In a scene they will be:

Home
  - Departments
     - Zoology
       - Class Lecture
         - Content
       - Class Routine
         - Content
       - Class Curriculum
         - Content
     - Biology
       - Class Lecture
         - Content
       - Class Routine
         - Content
       - Class Curriculum
         - Content

BUT NOT the file directly. We need to attach file under taxonomy, not the content. So there is a problem.
How can we proceed to such extent?
Comment: Are you using Drupal 7?
Comment: @JayendraKainthola: Yap. 7.15.
Answer: You can attach file with taxonomy as below:

Go to Administration -> Structure -> Taxonomy
Edit Vocabulary, which you want to use, from the list of Vocabulary.

Click on "Manage Fields" tab in the top of the page.

Add new field of field type as file and assign the required file extension which you want to upload and save the field.

When you create new term of the vocabulary, it will allow you to upload file with term.

How to upload files under taxonomy?


Still reachable in valgrind

Question: While searching about still reachable in valgrind, some people say its not a problem. we don't nedd to fix it. Some people say it needs to be fixed.I would be better if somebody could exaplain me explicitly what is the logic behind this still reachable. Is it mandatory to fix this?
[EDIT]
I have following valgrind output for my C program.Do i need to fix it?
      LEAK SUMMARY:
      ==27333==    definitely lost: 0 bytes in 0 blocks.
      ==27333==      possibly lost: 0 bytes in 0 blocks.
      ==27333==    still reachable: 96 bytes in 12 blocks.
      ==27333==         suppressed: 0 bytes in 0 blocks.

Comment: You haven't even said if this is a C or C++ program.
Comment: The above result is for my C program.
Answer: It depends. "Still reachable" means you haven't deallocated a block of memory before exiting, but had a pointer to it.
In a C++ program this means that some object could have not been deleted and therefore its destructor might not have been run and thus say some data might have not been saved onto disk for example and some other action might not have been taken and thus your program might produce unexpected behavior.
However there're no destructors in C programs, so your program just can't depend on that. Also deallocating memory takes some time, so by not freeing memory on exit you can save some time - your program will exit faster (this can be significant for programs with lots of data).
So IMO if your C program has "still reachable" blocks it's not a problem but this indicates that some code in the program doesn't free memory and so you can expect bugs when reusing that code.

Still reachable in valgrind


Do I need to remove the 'include module' that already installed by Puppet?

Question: I am very new to Puppet. I just write a code to install software via puppet
node 'myweb.com' {
   include ntp
   include apache
    apache::vhost { 'myweb.com':
      port =>8080,
      docroot => '/var/www/myweb.com',
      ssl => false,
      priority => 10,
      serveraliases => 'myweb.com',
       }

When  the above code applied, should I remove it in order not to double-install in the next time ?
Any suggestion would be appreciated
Answer: Puppet uses a declarative language, not an instructive one. You are telling Puppet how you want the system to look, and then trusting that Puppet will do the right thing to put the system's state into compliance with your manifest(s).
in your "apache" class, if you're using the standard method to install apache:
package { "apache":
    ensure => "installed"
}

...then you don't need to worry about it installing twice.


countUp.js doesnt load

Question: I have tried to set up countUp.js but it fails to load, I'm pretty new to Javascript or Jquery so maybe I did something wrong?

// Could there be a problem with the script?
var options = {
  useEasing: true,
    useGrouping: true,
    separator: ',',
    decimal: '.'
  prefix: ''
  suffix: ''
}
var test = new countUp("count", 1350, 10000, 0, 5000, options);
demo.start();

JSfiddle link
Comment: As a good advice, use your console to debug
Comment: And to add to what @A.Wolff said: you can use `console.log()` and `console.debug()` to print strings and variables.
Answer: First some properties of your object are missing their trailing ,, so you have syntax errors. Also, you need to call start() on the test variable, not demo. Try this:
var options = {
    useEasing: true,
    useGrouping: true,
    separator: ',',
    decimal: '.',
    prefix: '',
    suffix: ''
}
var test = new countUp("count", 1350, 10000, 0, 5000, options);
test.start();

Updated fiddle
Comment: Without seeing your code that's going to be tricky to diagnose.
Comment: You're including jQuery twice there, remove one of those.
Comment: I'd suggest starting a new question for that - including your full HTML and JS code if you can.
Comment: You're missing a closing bracket: http://jsfiddle.net/pd8z71o5/2/ First thing to check if something isn't working: the console.
Comment: Thanks, this worked on the jsfiddle file but not on my website, I'm still getting errors on chrome console http://i.imgur.com/5wra4t2.png
Comment: http://i.imgur.com/Saed6o3.png the problem seems to be on the countup.js file
Comment: and heres my code http://i.imgur.com/AEjY1Sh.png
Comment: Ok so I removed the .min one but it still doesn't load
Comment: Here's the code http://jsfiddle.net/pd8z71o5/

It doesnt work on the jsfiddle either
Comment: Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/68822/discussion-between-common-and-rory-mccrossan).

countUp.js doesnt load


When using protobuf-net, how do I know what fields will be updated (or have been updated) when using merge on an existing object

Question: Using Protobuf-net, I want to know what properties of an object have been updated at the end of a merge operation so that I can notify interested code to update other components that may relate to those updated properties.
I noticed that there are a few different types of properties/methods I can add which will help me serialize selectively (Specified and ShouldSerialize). I noticed in MemberSpecifiedDecorator that the 'read' method will set the specified property to true when it reads. However, even if I add specified properties for each field, I'd have to check each one (and update code when new properties were added)
My current plan is to create a custom SerializationContext.context object, and then detect that during the desearalization process – and update a list of members. However… there are quite a few places in the code I need to touch to do that, and I'd rather do it using an existing system if possible.
It is much more desirable to get a list of updated member information. I realize that due to walking down an object graph that may result in many members, but in my use case I'm not merging complex objects, just simple POCO's with value type properties.
Answer: Getting a delta log isn't an inbuilt feature, partly because of the complexity when it comes to complex models, as you note. The Specified trick would work, although this isn't the purpose it was designed for - but to avoid adding complexity to your own code,that would be something best handled via reflection, perhaps using the Expression API for performance. Another approach might be to use a ProtoReader to know in advance which fields will be touched, but that demands an understanding of the field-number/member map (which can be queried via RuntimeTypeModel).
Are you using habd-crafted models? Or are you using protogen? Yet another option would be to have code in the setters that logs changes somewhere. I don't think protogen currently emits partial method hooks, but it possibly could.
But let me turn this around: it isn't a feature that is built in right now, and it is somewhat limited due to complexity anyway, but: what would a "good" API for this look like to you?
As a side note: this isn't really a common features in serializers - you'd have very similar challenges in any mainstream serializer that I can think of.
Comment: Ultimately, a list of properties that were updated at the end of the merge. The after deseralization attribute seems like a good place to put this type of function. I noticed the callback is already doing some work to figure out what type of parameters that method has, and then populates those parameters during the invoking of the method (awesome). Adding one more property there for some sort of an "accounting" type object would work great. However, even better would be that same function, callback at end of deseralization, that could be applied to the RuntimeTypeModel.
Comment: Adding a callback to the RuntimeTypeModel seems like the better approach because then I don't have to add complexity to my POCO, which I do currently create by hand.


find a kernel for the parameter $\theta = \mathbb{E}[(X_1 - \mathbb{E}X_1)^3]$

Question: let $(X_1,\dots,X_n)$ be an i.i.d sample,
find a kernel for the parameter $\theta = \mathbb{E}[(X_1 - \mathbb{E}X_1)^3]$, basically, find some numerical function $h : \mathbb{R^d} \to  \mathbb{R} $ such that $d \leq n$ and $\theta = \mathbb{E} h(X_1,\dots,X_d)$
I tried $h(x_1, \dots ,x_n) = (x_1 - \frac{x_1+\dots+x_n}{n})^3 $
but it doesn't check out, $\mathbb{E}[(X_1 - \mathbb{E}X_1)^3]$ simplifies down to $EX_1^3-3EX_1EX_1^2+2E^3X_1$ while $\mathbb{E}[(X_1 - \overline{X}_n)^3]$ is a messy expression which can't be simplified.
any help will be greatly appreciated, thanks !
Answer: The kernel in this case is given by
$$
h(x_1,x_2,x_3)=\frac{1}{3}\sum_{1\le i\le 3} x_i^3-\frac{1}{2}\sum_{\substack{1\le i,j\le 3, \\ i\ne j}}x_i^{2}x_j+2 x_1x_2x_3.
$$
A more general result (for the $r$-th central moment) can be found in this paper (Theorem 1 on page 862).


What do statisticians do that can't be automated?

Question: Will software eventually make statisticians obsolete?  What is done that can't be programmed into a computer?
Comment: They *think* (i.e., bring knowledge to bear).
Comment: drink beer! ;o)
Comment: Interpreting the results ;-)
Comment: the same question can be asked for programmers in general then ;)
Comment: We design studies, and in particular we have to deal with real-life problems where the statistically "best" design can't be implemented. We clean up dirty data, bringing real-world knowledge to bear. We also interpret results in plain [insert language of  choice].
Comment: Umm, statistics?
Comment: A physician came to me for help with an advanced longitudinal data analysis declaring "I know what I'm doing, I just want confirmation that this is correct."  His use of a random effects model was flawless except for one simple fact: He had extreme values in the the response variable that were badly tilting the results.  This was caused by his taking logs to deal with "high outliers."  With many tiny values in the original responses, the log transformation blew up, creating "low outliers" instead.  Statisticians deal with many things, including reexamining implicit assumptions.
Comment: Computers can not determine what is right or wrong (more accurately, better or worse), they just do it faster. The consequences of doing the wrong thing very quickly and *en masse* are not good...
Comment: Reading just 1 or 2 thoughtful articles by the likes of Jacob Cohen, John Tukey, Howard Wainer, Gerd Gigerenzer, Gary King, or Donald Rubin will dispel any need to ask this question.
Comment: I recently learned that in general, optimization is area that often requires solutions that are problem-specific. This often requires human input and cannot always be automated.
Answer: Computers will only make statisticians obsolete when strong AI makes humans as a whole obsolete.
The question reminds me of the question about, "If there are all of these robust statistical methods, why do people still use other methods?" Some of the answer is habit and training, but much of it is that the question is naive: "robust" doesn't mean "you don't have to think about and understand what you're doing", as the question implies.
I mean, you could download the R statistics package today, and be doing any basic statistical technique by nightfall. You could then download a couple of packages and start using methods so esoteric that most of us haven't even heard of them. The question is: would you get reasonable answers? The answer is: probably not.
The algorithms are automated, but you still have to make many judgement calls all along the investigative path: from the plan of attack to the final judgement of whether the results actually make sense. To get to that point, you're really talking about Star-Trek-like computers where you can say, "Computer, tell me...", by which point pretty much every human vocation is obsolete.
Comment: +1 for "Computers will only make statisticians obsolete when strong AI makes humans as a whole obsolete."
Answer: @Adam, if you think of statistical researchers analogously to those in other fields - people who build upon the existing methodology and knowledge - then it might make it more clear that the answer to your first question is 'No'.
Statisticians that make a living from simply applying canned software packages could quite possibly be replaced by computers for every step except writing the discussion section of a paper where the results must be interpreted. So, in that sense, yes - it could be automated (although it would have to be a complicated piece of software that has one hell of a natural language processor).
However, as most researchers eventually figure out, the "canned" routines that people often use are pretty limited and must be modified (or new methods entirely must be developed) to answer specialized research questions - this is where the human aspect of statistics is indispensable. Or, a researcher must simply settle for a somewhat different, but related, research question that can be answered using classical methods.
Most statisticians I know work in research jobs (e.g. professors, research scientists) where their primary role is to develop new methodology. If this process could be automated, meaning that a computer can formulate and crank out useful new methodology, then I'm afraid researchers in every field would be obsolete.
Comment: I think your second paragraph misses a point: it's not just the end of the process (result interpretation) that's hard, it's also the beginning - understanding what methods to apply to the data in what ways, which in the general case requires understanding the nature of the data and the system it came from.
Comment: @Jefromi, like I commented to someone below, I think that understanding comes from an expert in the field of application, not a statistician.
Comment: If understanding just "came from" experts in the field of application my job'd be much easier (& much less fun). There's a frame problem: something the expert doesn't think to say can be important for the statistical analysis. In practice the most fruitful collaborations result in the expert learning a fair amount of statistics & the statistician learning a fair amount about the field of application.
Comment: @Macro Why do you think it would be easier to replace the statistician than to replace the domain expert with AI?
Answer: What can a statistician do that a computer can't? Write the original program they get replaced by.
Beyond that somewhat silly answer, the root of the question is ignoring the actual science of statistics in favor of its mechanics, and entirely discounting the role of the creative process in statistical analysis. This is, to use Peter Flom's car example, like saying cars are built using rivets and welds, so there's no reason the new Mustang couldn't be designed by riveting and welding robots.
A tremendous amount of the doing of statistics involves subject-matter expertise, judgement calls, and creativity. "Canned" analysis running from an algorithm often won't get you the best answer, and there are myriad documented examples where using automated methods actually gives you the wrong answer - or at least not the answer you think you're getting. The use of stepwise p-value based variable selection procedures and analysis based on purely numerically defined quantiles are two I'm most familiar with, but I'm sure you can find a wealth of others out there.
Even if all that was still somehow automated, there is the matter of interpreting results. The statistician (or statistically-inclined scientist)'s job isn't done when you obtain a regression coefficient or p-value. What does that finding mean. What are the caveats? What does this represent in the context of what's come before?
Finally, you have the development of new methods. Statistics isn't something that was simply laid out long ago by people whose names we recognize - Fisher, Cox, etc. It's an evolving field, and you can't program a new method into a computer until a person develops the method itself.
Comment: (+1) because "Canned analysis running from an algorithm often won't get you the best answer" is very true. This doesn't mean that human practitioners of statistics don't do this all of the time. (Note: most practitioners of statistics are NOT statisticians... more like people who are using statistics despite not really knowing what they're doing, often resulting in bad science)
Answer: Another way to interpret this question might be: "has the rapid increase in automated statistical techniques in recent years corresponded with a decreased demand in jobs for dedicated statisticians and data analysts?"
We can address this question by looking at the data

Data courtesy of indeed.com & revolutions blog
Comment: I'm not convinced "demand in jobs for dedicated statisticians and data analysts" has strong correlation with the usage of the keywords "data scientist" or "big data" in job ads.  <-- being suspicious of assumptions is what the human brings to the table ;-)
Comment: +1 Even Indeed.com has not made @cboettig obselete.
Comment: @DarrenCook well said!
Answer: I don't entirely agree with the premise of the question, i.e. I think there is no way in which computers could ever hope to replace statisticians, but to put a concrete example to why I think that:
The work which statisticians do with scientists, particularly, in the design and interpretation of experiments, requires not only a human mind but even a philosophical bent which it is inconceivable that computers could ever show.
Unless we end up in some sort of Skynet type situation, of course, in which case I reckon all bets are probably off as far as the future of all humanity, never mind about just the statisticians, is concerned :-)
Comment: Except I have feline overlords to obey. :)
Answer: The question suggests a naive view of a statistician-—that it's all about checking to see if a p < 0.05 and reporting some numbers and standard graphs.  If that's what you mean by statistician then you are correct in your implication that much of it could be entirely automated.  But that's not what statistician means.
Define your term statistician though, and you might get better answers.
Answer: Loading a statistics package onto your computer doesn't make you a statistician any more than buying a car makes you able to drive.
Even if the statistician just applies "canned" routines there are lots of questions.

Which routine? What routine will answer the client's questions?
With what variables? and should they be transformed? Should some levels be combined? Which should be forced into a model?
With what data? Should outliers be deleted? Trimmed? Maybe a robust method?

and so on.
But the job starts way before the computer is turned on, and ends long after the statistical package is turned off.
Before:
What does the client want to do?  Often this is a lot of work!
What data does the client have? Oy vey! The variables are labeled V1 to V828171 Which are which?
What is the state of the literature?
What will the client expect? How technical should it be?
After:
What do results mean? (and not just "this means that the regression is significant")
How should the results be explained to the client?
What other questions do the results raise?
It will, I think, be a long time before computers can do this.
Comment: For me, the analogy is a little closer to "buying a car does not make you a mechanic or a car designer."
Comment: I cannot resist pointing out that Google has been making great advances in the direction where buying a car *will* make you able to drive--it will do so automatically!
Comment: In order for you to answer the questions listed in (1), (2) and (3), you go through some logical process. Theoretically, this logical process could be coded into a computer program. If the computer had a perfect natural language processor and the software contained all "canned" software, and had the logic mentioned above programmed in, it would be able to do these things. Or, are you saying, it's not exactly a logical process?
Comment: @Macro Because its a logical process doesn't necessarily mean if can be programmed into a computer. "Should some levels be combined" isn't always a numeric measurement - it requires considering if those combined levels make sense in the context of the variable itself, for example.
Comment: Deciding whether it makes sense in the context of the application isn't a question for a statistician either - it's a question for an expert in whatever the application is. A statistician can tell you if it's justifiable to combine levels based on whether or not they appear homogeneous, which could certainly be taught to a computer.
Comment: @macro 1, 2, and 3 are not an entirely logical process. They require intuition. Also, I don't think it is even possible to have a perfect natural language processor, either in humans or computers. It would be very hard to program a computer to figure out when it had misunderstood someone.
Comment: @whuber: I would say Sebastian Thrun and his team have. Their work started quite long ago at CMU and they and others made great strides from the US government funded DARPA challenges. Google is now bankrolling some of their work. :)
Comment: @whuber: as long as it is not raining heavily. Google cars might work in California, I would like seeing one working in northern norway in snowstorm (we have that each year).
Comment: @whuber A little late to the party, and perhaps a bit pedantic, but wouldn't that be closer to *riding* a car?
Answer: Academic studies which look at the probability of automation of different occupations or task do not think that statisticians will be soon substituted by computers. See for example the controversial Frey & Osborne (2013) study which ranks occupations according to their probability of computerization, statisticians are ranked low 213 out of 702 with a probability of 22% (see table in the appendix). If you are further interested, see also the Slate article here.
Arntz et al. (2016) (here an The Economist article) look at tasks rather than occupations for the European Union and come to a similar conclusion: Doing "Complex Math or Statistics" is statistically significantly negatively related to job automatibilty (see Table 3).
But some caution is advisable, academics and/or economists have not always been very good in predicting the future (the Nobel laureate Robert Lucas for example concluded in 2003, a few years before the financial crises, that the "central problem of depression prevention as been solved, for all practical purposes, and has in fact been solved for many decades."). Both studies appear to be working paper, which are widely discussed but have not been published in standard peer-reviewed journals.
Regarding the academic debate, here you can find an overview article about the state of research about automation.
Answer: I think that AI will only make statisticians smarter and more competitive. Why? Because this is the intent of artificial intelligence since their conception many decades ago...


Best Fit for Exponential Data

Question: I'm trying to better understand some of the theory behind fitting models that have a nonlinear link between the response and the predictors.
set.seed(1)

#Create a random exponential sample
n<-1000
y<-rexp(n=n,rate=.01)
y<-y[order(y)]
x<-seq(n)
df1<-data.frame(cbind(x,y))
plot(x,y)

Now I will attempt 4 different models fits and explain what I expect as a result versus the actual result.
m1<-lm(I(log(y))~x,data=df1)
summary(m1)
out1<-exp(predict(m1,type='response'))
lines(out1)

I expect the m1 model to be the worst fit.  This is OLS with the log of the response taken before fitting the model.  By taking the log of the response I have a model that, when exponentiated, cannot be negative.  Therefore, the assumptions of normally distributed residuals with a constant variance cannot hold.  The graph appears to under weight the tails of the distribution significantly.
Next, I will fit the model using a GLM.
m2<-glm(y~x,data=df1,family=gaussian(link='log'))
summary(m2)
out2<-predict(m2,type='response')
lines(out2,col='red')

I expect the m2 model to be of a slightly different fit than m1.  This is due to the fact we are now modeling log(y+ϵ)=Xβ rather than modeling log(y)=Xβ+ϵ.  I cannot justify if this fit should be better or worse from a theoretical standpoint.  R gives an R-squared figure for lm() functions calls but not glm() (rather it gives and AIC score).  Looking at the plot it appears that m2 is more poorly matched in the tails of the distribution than m1.  The issues with normally distributed residuals should still be the same.
For m3, I relax the residual distribution assumptions by changing to a Gamma family residual distribution.  Since an exponential distribution is a Gamma family distribution, I expect this model to very closely match the sampled data.
m3<-glm(y~x,data=df1,family=Gamma(link='log'))
summary(m3)
out3<-predict(m3,type='response')
lines(out3,col='blue')

AIC suggests that m3 is a better fit than m2, but it still does not appear to be a very good fit.  In fact, it appears to be a worse estimate in the tails of the distribution than m1.
The m4 model will use nls just for a different approach.
m4<-nls(y~exp(a+b*x),data=df1, start = list(a = 0, b = 0))
summary(m4)
out4<-predict(m4,type='response')
lines(out4,col='yellow')

#for direct comparison
t1<-cbind(out1,out2,out3,out4,y)

This appears to very closely match m2.  I also expected this to very closely fit the data.
My questions are - how come does m3 not more closely match my data?  Is there a way using glm() to more closely fit this model?  I realize that if I am randomly sampling then the tails of my sample data will not fit my model exactly, but these appear to be nowhere close.  Perhaps letting n approach infinity would all one (or more) of the models to converge, but rerunning the code with n=100,000 actually appears to be worse fitting models (perhaps that's because with 100,000 random samples more outliers were selected and due to how the graph is present an undue amount of focus is given to these outliers?).
I also realize that an exponential distribution is not the same as exponentiating a normal distribution.  That is to say, I realize that just because I took the log the of response doesn't mean that I should get a "perfect model" as a result; however, in m3 when I am fitting the models to a Gamma family distribution, I expected to be able to get a very close fit.
Thanks in advance.
Answer: In your simulation you are trying to model the $i$-the order statistic, $Y_{(i)}$ of an exponential distribution of an exponentially distributed sample of size $n$ as a function of $i$. This is not an exponential function, so your fitted exponential curves failed. In fact, you are essentially estimating the quantile function of the exponential distribution.
Try
curve(qexp(x/n, rate=0.01), add=TRUE)

to match your scatterplot. As a homework, derive this function and you will see that it is actually logarithmic.
If instead you want to experiment with exponentially distributed data with one predictor, use something like
y2 <- rexp(n=n, rate=exp(-1 - 0.001*x))

In this case curves 2 - 4 will pretty much overlap and reproduce the correct parameters, while curve 1 will underestimate (as expected).
Answer: To me it looks like your models are all off. For some reason, you decided to use the vector x = seq(n) = (1, 2, ..., n) as a covariate which doesn't make sense. Use a vector of 1s instead (an intercept-only model) and see how it goes. In fact, you don't need to create it because you can use something like:
model = glm(outcome~1, ....);
to fit an intercept-only model.
Comment: Wouldn't an intercept only model just give me a horizontal line?  I'm using the sequential vector as a covariate because I have ordered the data and am trying to fit a model to that order.  I agree that vector X is arbitrary - whether I use seq(n) for x or seq(n)/2, it shouldn't matter.  Perhaps I'm not articulating what I'm trying to accomplish well.  I'll have to think about it some more.
Comment: You are saying that the mean of your response is dependent on some x, and yet the response was generated as: y<-rexp(n=n,rate=.01), that is, all observations have exactly the same mean. For your model statement to be meaningful, "rate" should be a function of x.
Comment: Note, that he sorted `y`. It does not make the regression model appropriate, though.
Comment: Okay, so an exponential model is a 1 parameter model, but I chose an exponential because it's simple and it's an exponential family distribution, but let's pretend we didn't know it was an exponential model but we still assume exponential family and that we are trying to estimate 2 parameters.  Now, I could do MLE fits of the two parameters for each of the exponential family distributions.  I could set up an MCMC to estimate the two parameters for the distribution.  My question is there a way to estimate these with a glm?  I hope this more articulate.  Thank you for this dialogue.
Comment: You could benefit from reading this: http://en.wikipedia.org/wiki/Generalized_linear_model and then the manual for glm() package in R.
Answer: This is due to the fact we are now modeling log(y+ϵ)=Xβ rather than modeling log(y)=Xβ+ϵ.
My understanding is that with log link we are no longer transforming y, we are modeling yhat = e^eta, where eta is the structural model, such as a regression line.
In my experience with a gnarly data set (n = 10 means, unequal sample sizes, sterr increases with mean, except for one known outlier), the StErrSq weighted gamma error model with log link gave the "best fit" by AIC, the Nweighted model gave the strongest evidence (largest LRatio). I would point out, however, that Gamma error together with StErrSq is overkill, it amounts to 2 runs at weighting by the variance, reducing the total deviance to dust.

Best Fit for Exponential Data


Método sendo chamado várias vezes no evento change de um p:ajax

Question: Tenho uma tabela com um p:inputText com um p:ajax dentro, o mesmo possui o evento change, quando preencho esse inputText com um valor uma consulta e algumas validações são realizadas. O problema é que quando eu colo um valor dentro desse inputText (ctrl+v) o método é chamado duas vezes. Se eu digito o valor no inputText ao invés de cola-lo o método funciona da maneira correta, ou seja, é chamado apenas uma vez.
xhtml:

O que pode estar "estragando" essa requisição ajax? Tenho outros eventos ajax na tela, porém servem pra outras coisas.
Obs: Na tela também tenho alguns decorates na tela também, não sei isso pode influenciar.
Comment: Provavelmente o evento atributo event="change" está sendo disparado no keyUp do teclado, como ctrl + v são duas teclas, logo serão 2 eventos (não tenho nenhum projeto pequeno agora para testar). Você já tentou usar o atributo delay do p:ajax ? tente algo como delay="1000" e veja se funciona.
Comment: @wryel Funcionou, pode responder a pergunta :D
Comment: Vou tentar, eu acho que resolvi alterando pra blur e tratando ele, mas se der certo no change é melhor
Answer: Provavelmente o evento do atributo event="change" está sendo disparado no keyUp do teclado, como ctrl + v são duas teclas, logo serão 2 eventos. Teste usar o atributo delay do p:ajax


How much spot checking do I need?

Question: I'm working on software in which Component X is used 130 times. X is going to be replaced with a new and improved version, Component Y.
We need to regression test every instance where X is replaced with Y, which is going to take a lot of effort. If we get through the first 50 tests and there are no failures, intuitively, we're not going to expect any of the other 80 to fail. The risk may be small enough that it's not worth continuing.
Is there a way to quantify this risk? If we start with assumption that there's a 10% chance that at least of the regression tests will fail, how many successful tests do we need to reduce that risk to 1%?
To put it another way:

A bag contains 130 balls that are either red or black.
There is a 10% chance that the bag contains at least one red ball.
How many black balls must be drawn to reduce that chance to 1%?

I've gotten as far as relating the problem to this familiar thought experiment, but don't know where to go from here.
Answer: Lets say that probability of not drawing a red ball given there are any is $p$, and we draw total of $n$ balls.
Total probability of not drawing a red ball is thus $0.1 \cdot p + 0.9$, and probability of having a red ball given we didn' draw one is $\frac{0.1 \cdot p}{0.1 \cdot p + 0.9} = \frac{p}{p + 9}$.
Now we need $\frac{p}{p + 9} \leqslant 0.01$ or $p < \frac{1}{11}$.
Given that $p \leqslant \frac{130 - n}{130}$ (probability of not drawing any red ball if there is only one), $n = 119$ will be sufficient (and $n = 118$ will not, as if there is just one red ball, we will have probability of $\frac{12}{130} > \frac{1}{11}$ of not discovering it).
Comment: Thanks. I'm more concerned with the average case than the worst case, but this is the start that I needed. (Don't worry, I'm not building a self-driving car! :))

How much spot checking do I need?