Is it okay to use different data preparation procedures for different models?

Question: I'm currently learning some basics of building models, and I was wondering whether it's okay to use different data preparation procedures for different models. Specifically, I was considering an example of using linear regression vs ridge regression and whether any features need to be removed. I read that ridge regression can deal with some multicollinearity since it reduces model complexity, so we don't necessarily need to perform feature selection before ridge regression. However, feature selection in linear regression is much more important. Would it be okay to do feature selection just in the linear regression?
My intuition is that this would be fine as this data preparation procedure can be thought of as part of the model - but I was wondering if there could be any issues with comparing models.
Answer: Selecting (or constructing) variables is very much dependent on the model, e.g. if trying to model a quadratic relationship with a linear model you need to include an additional variable to capture this, while a random forest (example) model would not need such a variable since it can learn this relationship on its own (although it doesn't necessarily hurt to include it). Another case is the one you mention, ridge can deal with multicollinearity on its own (although you might not like how it chose to handle it), while linear regression cannot, so you should probably do something about it.
Answer: I wonder if, for your specific example, this counts as "data preparation" at all. While it's true that collinearity is a feature only of the independent variables, the usual procedure for diagnosing it involves doing collinearity diagnostics on a model, so you have to run a model to tell if you need to fix collinearity.
I think of "data preparation" as stuff that you do before running any models. There could be cases where you have to do some preparation for some models but not others. For instance, you may know, a priori that you want to change count data into yes/no data for some variable for one model but not others. This could happen if you have hypotheses about both presence/absence and count.  For example, if you have data on how many cars a family owns, you might have a hypothesis about whether they own a car at all and another about how many cars they own.


Como trazer mais campos no groupby do Pandas, sem necessariamente precisar usá-los no agrupamento?

Question: Boa tarde, pessoal. Estou mexendo com a biblioteca Pandas no Python e gostaria de tirar uma dúvida, se possível. Eu tenho uma lista de valores e preciso reduzir esta lista para apenas um registro por data. O critério é que o terceiro campo (contador) seja o maior de sua respectiva data. Para fazer isso, usei o seguinte código:
import pandas as pd

lista = [("2020-11-16", "803", 4),
    ("2020-11-16", "801", 18),
    ("2020-11-16", "802", 20),
    ("2020-11-17", "801", 12),
    ("2020-11-17", "802", 6)]

df = pd.DataFrame(lista)
df.columns=['data', 'codigo', 'contador']
df2 = df.groupby('data')['contador'].max()
for i in df2.items():
    data = i[0]
    contador = i[1]
    print(data, contador)

#O resultado obtido foi o seguinte e está correto:
['2020-11-16', 20]
['2020-11-17', 12]

A minha dúvida seria se é possível trazer junto neste resultado a coluna 'codigo', sem que esta passe por agrupamento. Se eu a colocar no groupby terei vários registros por data e estará errado, pois só posso ter um único registro por data. Teria alguma maneira de fazer isso, seja pelo pandas ou de outra maneira? Obrigado desde já.
Answer: Uma solução seria criar um 'filtro':
filtro = df.groupby('data')['contador'].max()

E depois utilizar o isin do pandas:
df[df['contador'].isin(filtro)].reset_index(drop = True)

Saída:
    data        codigo  contador
0   2020-11-16    802        20
1   2020-11-17    801        12

Comment: @imonferrari, bela solução! Não tenho o hábito de usar o método `isin()`, pois normalmente trabalho com a base toda. Com certeza me fará pensar diferente de agora em diante.
Comment: Perfeito. Não conhecia essa possibilidade de filtro com o isin(). Muito obrigado!
Comment: Opa! Agradeço, Paulo! `isin` é um método bem rápido, depois que começa a usar se torna um vício kkkk. Brincadeiras a parte... Grande Abraço!
Answer: Acredito que o que você esteja procurando é o aggregate
Para obter o resultado, basta:

Criar uma estrutura de funções desejadas para cada campo

f = {'contador': 'max', 'codigo': 'first'}

Faça o groupby com agg passando f como parâmetro

df2 = series = df.groupby('data', as_index=False)[['contador', 'codigo']].agg(f)

O resultado será:
>>> df2
         data  contador codigo
0  2020-11-16        20    803
1  2020-11-17        12    801

Perceba que o código escolhido pelo aggregate foi o first, o que no caso é o 803. Porém, caso você queira o mesmo que está na linha do contador=20, ou seja, codigo=802 faça exatamente o que vc estava fazendo e depois conclua com um merge()
df3 = pd.merge(df2, df, on=['data', 'contador'])

O resultado será:
>>> df3
         data  contador codigo
0  2020-11-16        20    802
1  2020-11-17        12    801

Espero ter ajudado
Comment: @imonferrari, obrigado pelo comentário. Os campos que existem nos dois dataframes recebem automaticamente os sufixos`_x` e `_y`. Particularmente eu prefiro passar os sufixos como parâmetro no merge. Algo como `suffixes=('_df1', '_df2')` ou `suffixes=('_left', '_right')`. Outros parâmetros bem úteis são `left_on` e `right_on` que permitem juntar dataframes quando os campos possuem nomes diferentes. Veja mais detalhes do merge [aqui](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.merge.html).
Comment: Testei aqui também com o aggregate e deu certo também, obrigado Paulo. É bom saber que temos caminhos diferentes para fazer isso. Agradeço muito pelas duas respostas, até mais pessoal.
Comment: Fala Paulo, boa noite! Tudo bom?  Em um data frame maior parece que cria colunas codigo_x e codigo_y. Creio que tenha que excluir a codigo_x pro resultado ficar igual ao que ele pede no problema. Grande Abraço!
Comment: Sim, foi só uma observação mesmo para a resposta "sair" do jeito que ele espera, no mais ta tudo nos conformes kkkk. Abraço!!


How to recover from bad firewall config in Azure

Question: All our virtual machines recently disappeared from Azure because our trial quotas were reached.
I decided to follow this guide for recovering the VMS: http://blogs.msdn.com/b/narahari/archive/2012/10/18/windows-azure-virtual-machine-disappeared-or-gone-how-do-i-recover.aspx
However, the virtual machines have firewall configurations created with ip-tables (via firewall builder). The new virtual machines get a new set of IPs. This causes problems, since ip-tables will try to reassign the old ip address to the interface. The end result is that we cannot SSH to the machines - they are inaccessible.
The question is how to recover from this. My current idea is to download the virtual machine images and try to mount them with some live-cd and remove the firewall configuration, then upload them again and create virtual machines based on that. However, I do not know how to mount the VHDs.
Is there any faster/better way?
Comment: Don't use a destination IP on your INPUT rules in cloud-instance VMs. Oh, and also check Azure's own firewall; they randomize the port they open for ssh for you.
Answer: I solved this by creating a new virtual machine on Azure, and attaching the old disk as a data disk. I then mounted the old disk and could remove the firewall script. We now have access to our VM's again, although we have had to update som DNS settings.


How can *$R$ be a field in Synthetic Differential Geometry or Smooth Infinitesimal analysis?

Question: In many sources it was written that *$R$ is ring and *$R$ can't be a field because it's not possible that $d^2=0$ in field.
But in some sources it was written that *$R$ is a field.
How can *$R$ be a field in Synthetic Differential Geometry or Smooth Infinitesimal Analysis?
Thanks.
Comment: @Mike_bb, Giordano is *not* doing Synthetic Differential Geometry.  Furthermore, Giordano did not use ${}^\ast\!R$ but rather ${}^\bullet\! R$.
Comment: Which sources are you referring to?
Comment: Why do you use $^\ast \!R$ do denote the line of SDG?  Both sources you linked use simply $R$.
Comment: @MarkSaving https://www.math.ru.nl/~landsman/scriptieTim.pdf (page 16) - ring. https://pages.physics.ua.edu/staff/fabi/InvitationSDG.pdf (page 3) - "Nevertheless most of the literature refers to R simply as a ring while a few authors still refer to R as a field"
Comment: @MikhailKatz It's my mistake. *$R$ was in Giordano's source which I read.
Comment: @MikhailKatz Ok. I find now in Giordano's source that IDG differ from SDG. It's true. Thx.
Answer: Since SDG uses intuitionistic logic, it is important to specify how exactly you define a field.  If you want the property that if $x\not=0$ then $x$ is invertible, then this still holds.  Nilsquare infinitesimals do not provide a counterexample to this, since they are not provably nonzero.


Populate DataGridView from a List

Question: I have a list of string, int and object. I need to populate a datagridview with those values, but when I tried to populate the object value doesn't work.
This is my declaration for the class:
public int ClientAddressId { get; set; }
    public int ClientId { get; set; }
    public CityModel City { get; set; }
    public CountyModel County { get; set; }
    public string StreetName { get; set; }
    public string StreetNo { get; set; }

This is my declaration for the method that return a list:
public BindingList<ClientAddressModel> GetClientAddressBi(int clientId)
    {
        DAClientAddress dac = new DAClientAddress();
        DataTable dt = new DataTable();
        dt = dac.GetClientAddress(clientId);

        BindingList<ClientAddressModel> list = new BindingList<ClientAddressModel>();
        for (int i = 0; i < dt.Rows.Count; i++)
        {
            ClientAddressModel address = new ClientAddressModel();
            address.ClientAddressId = Convert.ToInt32(dt.Rows[i]["ClientAddressId"].ToString());
            address.ClientId = Convert.ToInt32(dt.Rows[i]["ClientId"].ToString());
            address.City = new CityModel
            {
                CityId = Convert.ToInt32(dt.Rows[i]["CityId"].ToString()),
                CityName = dt.Rows[i]["CityName"].ToString()
            };
            address.County = new CountyModel{
                CountyId = Convert.ToInt32(dt.Rows[i]["CountyId"].ToString()),
                CountyName = dt.Rows[i]["CountyName"].ToString()
            };
            address.StreetName = dt.Rows[i]["StreetName"].ToString();
            address.StreetNo = dt.Rows[i]["StreetNo"].ToString();

            list.Add(address);

        }

This is where I suppose to populate datagrid view
BLClientAddress blca = new BLClientAddress();
        BindingList<ClientAddressModel> list = blca.GetClientAddressBi(PartnerId );
        dataGridClientAddress.DataSource = list;

That is what my app return
Comment: It looks like it populated correctly but when you write CityName = dt.Rows[i]["CityName"].ToString() it returns the name of the Object not the actual value so that part is probably confusing you... remove ToString() and see if you get the actual value
Comment: Why don't you create a _ClientAddressViewModel_ class with the properties that you want to display in the grid?
Comment: Or you want to add an override for the ToString() method in the class County and City and return what you want to show in the columns
Answer: For a simple scenario like this one I suggest to simply add an override for the ToString method inside the classes CityModel and CountyModel
public class CityModel
{
   public int CityId { get; set; }
   public string CityName { get; set; }
   public override string ToString()
   {
       return CityName;
   }
}
public class CountyModel
{
   public int CountyId { get; set; }
   public string CountyName { get; set; }
   public override string ToString()
   {
       return CountyName;
   }
}

The DataGridViewTextBoxColumn calls the ToString() on the County and City properties to get the value to display. If you don't override that method you get the object's name. With the override you get what you decide to return from that method.

Populate DataGridView from a List


Multiple interpretations of MSE

Question: I recently discovered that MSE can mean multiple things:

MSE for  predictor (mean of squared errors)
MSE for estimator (variance+ bias^2)
MSE in regression analysis (residual sum of squares divided by degrees of freedom)

My questions are,
1. Why these multiple interpretations?
2. Are there anymore interpretations of MSE?
Answer: MSE or mean squared error literally measures the average squared error for an estimator when estimating a parameter. The different interpretations rise from the estimation of different parameters.

This is when the parameter being estimated is a future value (unobserved response) from current model.
This is when the parameter being estimated is an attribute of a distribution, like the mean etc from statistic
This is when the parameter being estimated is the mean regression function from the OLS line.

In general MSE = $\dfrac{1}{n}\sum$ (parameter - estimate)^2. So the underlying formula is always the same, just the parameters and estimates change from problem to problem.
In regression, sometime MSE refers to Residual sum of squares where then
$$MSE = \dfrac{1}{n-p} \sum(y - y_i)^2 $$
The denominator changes in order for MSE to be an unbiased estimated of the error variance. This is one example of a slight deviation from the general understanding of mean squared error.
Comment: Helpful, but your formula needs a divisor. As it stands, it is SSE. (I'd prefer _estimate_ to estimator personally.)
Comment: Congratulations with 1000 reputation :)
Comment: could you explain more about why the difference in the formulae in each case? Please.
Comment: @Bach The basic formula is *always* the same. But the formula takes a different form, from problem to problem.
Comment: oh, and why does that happen? because in case of regression, we are estimating from same data, so the denominator reduces by the number of parameters estimated and similarly for estimate and predictor?

Multiple interpretations of MSE


A question on the curvature of a regular parametrized curve

Question: The next question is from Do-Carmo's baby book, page 30 question 3 in section 1-6.
The question goes as follows:
Show that the curvature $k(t)\neq 0$ of a regular parametrized curve $\alpha : I\rightarrow \mathbb{R}^3$ is the curvature at $t$ of of the plane curve $\pi \circ \alpha$, where $\pi$ is the normal projection of $\alpha$ over the osculating plane at $t$.
Now I guess I don't understand what is $\pi$ here, I mean I have $$\alpha(s)= (x(s),y(s),z(s))$$ assume it's in arclength parametrization, so basically I want to show that:
$$(\pi \circ \alpha (s))'' = k \cdot n_{\pi \circ \alpha (s)}\;,$$
but I am not sure here what $\pi$ equals to, I mean if its a projection on the osculating plane which is the plane of $xy$ then shouldn't it be the binormal to $\alpha$ (in which case it's $(0,0,z(s))$)?
Any hints?
Thanks.
Comment: You can get the symbol for concatenation using `\circ`. To put punctuation on a line after a displayed equation (instead of at the beginning of the next line), you need to include it before the closing `$$` (preferably preceded by a `\;` to set it off from the equation).
Answer: Don't use coordinates to built your intuition! Use them when you need to make calculations.
Given a point $\alpha(t)$ on our curve, consider two points $X$ and $Y$ which move on the curve. The plane determined by these three points approaches a fixed plane as $X$ and $Y$ approach the point $\alpha(t)$. This plane is called the osculating plane.
They also determine a unique circle passing through all three points, and these circles approach a fixed circle as $X$ and $Y$ get closer to $\alpha(t)$. This is the circle of curvature.
It is an easy exercise to check that these are compatible with the usual definitions. With this in mind the problem reduces to showing that projecting the curve to the osculating plane at $\alpha(t)$ does not change the osculating circle. This should be obvious once phrased as follows:
Given a sequence of triangles whose limit is a single point, the limit of their circumcircles is the same as the limit of their circumellipses provided the eccentricity of these ellipses goes to zero.
Comment: I am not sure I understand, so basically this projection is onto the plane tn of alpha?
Comment: Ok, I think I have got an answer.
 let's assume that the osculating circle of $\alpha$, is given by the equation :
$$x^2(s)+y^2(s)=\frac{1}{k^2}$$

Now under the projection of $\pi$ onto the osculatin plane we get $$\pi\circ \alpha(s)=(x(s),y(s),0)$$, so we get:
$$ |\pi\circ\alpha(s)|^2 =\frac{1}{k^2}$$
So upto a sign the curvatures of both $\alpha$ and $\pi\circ\alpha$, now they have the same sign cause the orientation of both of these curves are the same.

Does it need anymore tweaking?
Answer: Although the geometric intuition given in the other response is amazing. I encountered the same problem and solved it in two (analytic) ways:

We know that the Osculating Plane in $t\in I$ is the set of points $x$ in $\mathbb{R}^3$ such that $x=\alpha(t)+r_1\hat{t}(t)+r_2\hat{n}(t)$ where $\hat{t}(t)=\frac{\alpha'(t)}{\|\alpha'(t)\|}$,$\hat{n}(t)=\frac{\alpha''(t)}{\|\alpha''(t)\|}$ and $r_1,r_2\in\mathbb{R}$.
Now for every $y\in\mathbb{R}^3$ define the projection of $x$ over $y$ to be $P_{y}(x):=\frac{x\cdot y}{\|y\|^2}y$.
To answer your question: the projection of a vector $x$ over a plane $\pi_{_\mathcal{P}}(x)$ is obtained by computing the linear combination of the projection of $x$ to each of the spanning vectors of the plane, in this case: $\pi_{_\mathcal{P}}(x)=P_{\hat{n}(t)}(x)+P_{\hat{t}(t)}(x)=[x\cdot\hat{n}(t)]\hat{n}(t)+[x\cdot\hat{t}(t)]\hat{t}(t)$.
Finally, if we compose $\pi_{_\mathcal{P}}\circ\alpha$, derivate that function two times and evaluate in $t$ (to get its curvature) we get the desired result.
Another way to get the proyection of a point $u\in\mathbb{R}^3$ over the plane $\mathcal{P}$ is to remember that a plane is the geometric space whose points $x=(x,y,z)$ satisfy that $ax+by+cz+d=0$ where the vector $n=(a,b,c)$ is normal to the plane and $n\cdot x_0=-d$ for some $x_0\in\mathcal{P}$. If you consider the line "with foot" in $u$ and spanned by $n$, i.e., $\mathcal{L}=\{u+tn|t\in\mathbb{R}\}$, that line should intersect the plane $\mathcal{P}$ in the point $\pi_{_\mathcal{P}}(u)$. Doing some computations you should get that:
$$\pi_{_\mathcal{P}}(u)=u+t_0n=u-P_{n}(u)-P_n(x_0)$$
Proceding in the same way as in 1. using $\hat{b}(t)=\hat{n}(t)\times\hat{t}(t)$ as the vector normal to $\mathcal{P}$ should give the same result.


Mobile keyboard interferes with full screen modal in react native app

Question: I have a spinning loading wheel (implemented with <ActivityIndicator>) inside a modal with lowered opacity.  I want the modal to cover the full screen.  The modal pops up after the user unlocks their account in another modal.  The problem is that the 'loading wheel' modal only covers the part of the screen that wasn't covered by the keyboard in the 'unlock' modal.

It would seem to be a timing issue as occasionally the 'loading wheel' modal does cover the whole screen.  Here's my react native modals:
<Modal visible={modalUnlockVisible} animationType={"none"}>
     <Text>Enter Pin Code</Text>
     <TextInput
          maxLength={15}
          secureTextEntry={true}
          onChangeText={text => setPincode(text)}
      />
      <Button
          title="Unlock"
          onPress={() => { Keyboard.dismiss(); unlock(pincode) }}
       />
</Modal>
<Modal visible={loader} transparent={true} animationType={"none"} style={styles.loader}>
    <View style={styles.loaderBackground}>
        <View style={styles.loaderFrame}>
             <ActivityIndicator size="large" color="#006EFF"/>
        </View>
    </View>
</Modal>
......
//styles
loader: {
    zIndex: 100
},

loaderBackground: {
    top: 0,
    left: 0,
    bottom: 0,
    right: 0,
    flex: 1,
    alignItems: 'center',
    flexDirection: 'column',
    backgroundColor: 'rgba(0,0,0,0.7)'
},
.....

So the top modal of the above two pops up, and the user unlocks by pressing the Unlock button calling unlock(pincode):
function unlock(pincode) {
    setModalUnlockVisibility(false);  // Hide 'unlock' modal
    setLoader(true);  // Show 'loading wheel' modal
    .....
}

Any way to make the 'loading' modal cover the whole screen?  I added in the Keyboard.dismiss() before unlock(..) in the hope that it might help, but the problem persists most of the time.
Answer: I solved this by using the Dimension component from react-native.  As per this answer at SO.
import { Dimensions } from 'react-native'
^ at the top of your file where your component is.  And in my case change my loaderBackground style to:
loaderBackground: {
    width: Dimensions.get("window").width,
    height: Dimensions.get("window").height,
    alignItems: 'center',
    flexDirection: 'column',
    backgroundColor: 'rgba(0,0,0,0.7)',
    ....
},


How to check if a term has a published content associated with it?

Question: I have taxonomy term list called clients. I have managed to list all the terms on the taxonomy term list page by adding contents for each term, some contents are published some are not.
what I want to do is check and remove the term page link for those terms whose contents are not published and display them without the link.
Answer: There is an existing Question that handles this for views 2 and views 3: https://stackoverflow.com/questions/421972/how-to-display-only-taxonomy-terms-with-nodes-associated-to-in-views-2
Comment: I have checked the link you gave me.... it differs from what i'm trying to achieve. I have listed all the terms by adding contents for each term some published so not. what I want to do is remove the term page link for those term whose contents are unpublished but the term listed without the term page link.


Generating scaffolding template recursively for MVC

Question: Consider following compostion relation between Child & Root classes-
public class Child
{
    public string ChildProperty { get; set; }
}
public class Root
{
    public Child child { get; set; }
    public string RootProperty { get; set; }
}

Creating strongly typed view for Root class, does not include child property.
<fieldset>
    <legend>Root</legend>

        @Html.LabelFor(model => model.RootProperty)

        @Html.EditorFor(model => model.RootProperty)
        @Html.ValidationMessageFor(model => model.RootProperty)

        <input type="submit" value="Save" />

</fieldset>

How do I generate scaffolding template to include child properties as well?
Thank you for your help.
Answer: Generally you would need to create your own T4 templates that do just that:
https://stackoverflow.com/a/16072432/176877
But, I'm not certain you could actually loop through those child models; I'm fairly certain you would need to take things a step farther and actually generate a Custom Scaffold with your own PowerShell script that loops through those models and calls a new set of templates you create.
http://blog.stevensanderson.com/2011/04/07/mvcscaffolding-creating-custom-scaffolders/
In other words, you have a lot of work ahead of you if you want to do this.


Is testing model assumptions considered p-hacking/fishing?

Question: "P-hacking", "fishing", and "garden of forking paths" as explained here and here describes an exploratory data analysis-like style of doing research that produces biased estimates.
Does testing model assumptions (e.g. normality, homoskedasticity in regression) using statistical tests on the same data set that is used to fit the model considered a "p-hacking" or "garden of forking paths" problem?
The results of those tests certainly affect what model the researcher ultimately choices to fit.
Comment: *testing* assumptions doesn't do a thing. But people rarely just *test*, the outcome of the test influences what they do next... and it's the "what they do next" that's the issue. It's when the outcome of the test changes what you would do that there's a "fork". It's important to make sure you're asking the right thing there.
Comment: @Glen_b excellent point - for example, if you test data for normality, and then choose what test to use depending on whether the null was accepted or not, you're essentially using a composite test of unknown properties (Type I & II errors).
Comment: See [Does testing for assumptions affect type I error?](https://stats.stackexchange.com/q/100934/17230).
Comment: @Glen_b, that is exactly what I'm wondering about. I'm not sure if choosing a model based on the results of tests of assumptions would bias an estimator (I'm trying to think of an example of that), but it would definitely affect the standard errors.
Comment: @scortchi, that is a great example of what I'm concerned about. Yet, testing assumptions or at least evaluating them based on the data used to fit the model seems to be standard practice. That's how I have been taught in all my classes.
Answer: I do not believe that checking the assumptions of any model qualifies as p-hacking /fishing. In the first article, the author is talking about analysts who are repeatedly performing analyses on a data set and only reporting the best result. In other words, they are purposely portraying a biased picture of what is happening in the data.
Testing the assumptions of regression or any model is mandatory. What is not mandatory is to repeatedly re-sample from the data in order to ascertain the best possible outcome. Assuming researchers have a large enough sample to pull from, they will sometimes re-sample over and over again...perform hypothesis tests over and over again....until they achieve the result they want. Hence p-hacking. They're hacking the p-value via looking for the desired result and won't quit until they find it (fishing). So even if out of 100 hypothesis tests they only achieve 1 with a significant result, they'll report the p-value belonging to that particular test and omit all the others.
Does this make sense? When checking model assumptions, you're making sure that the model is appropriate for the data that you have. With p-hacking/fishing, you are endlessly searching the data/manipulating the study in order to achieve your desired outcome.
As for the purpose of multiple comparison, if you keep running a model through the mud endlessly trying to find a way to invalidate it (or validate it) then eventually you will find a way. This is fishing. If you want to validate a model, then you'll find a way. If you want to invalidate it, then you'll find a way. The key is to have an open mind and find out the truth - not just see what you want to see.
Answer: It's not quite the same thing in the sense that the practice of testing whether assumptions were violated was originally intended to make sure an appropriate analysis was done, but as it turns out, it does have some of the same consequences (see e.g. this question). But it is in a milder form than the more extreme variants of p-hacking that are specifically targeted at somehow getting the p-value for the effect of interest below 0.05. That is unless you start combining multiple problematic practices (e.g. checking for normality, checking for homoscedasticity, checking covariates that "should" be in the model, checking for linearity of covariates, checking interactions etc.). I am not sure whether anyone has looked into how much that invalidates the final analysis.
Of course the other issue is that testing for normality is not normally meaningful (see e.g. this discussion). For small sample sizes you do not reliably pick up massive deviations that truly violate your assumptions, while for large sample sizes e.g. the t-test becomes quite robust to deviations but the normality test will start to detect tiny deviations that do not matter. It is much better to (whenever possible) specify an appropriate model based on previous data or subject matter knowledge. When that is not possible, it may be best to use methods that are more robust to violations of distributional assumptions or have none/fewer.


Number theoretic problem - Representation of even number as difference of two numbers that are coprime to it.

Question: Someone gave me this problem:

Show that every even number $n$ is the difference of two natural numbers $a,b$ both of which are coprime to a given number $c$.

I have trouble to use the somehow make the information that $a$ and $b$ are coprime to $c$ useful. Any hints for this are appreciated. Thanks in advance I usually come up with something but this time I have no clue. I know we can write
$$
\frac{a+b}{2}=m
$$
where $n=2m$ but this does not help either. I also constructed a few examples but I found no real pattern.
Comment: You seek an integer $\,x = \,a\,$ such that $\,x\,$ and $\,x+n\,$ are coprime to $\,c,\,$  which [is equivalent](https://math.stackexchange.com/a/3555162/242) to their product $\,f(x)= x(x+n)\,$ being coprime to $\,c.\,$ CRT allows us to easily tackle such problems for any polynomial $f(x)$ with integer coef's - see my comment on Calvin's answer.
Answer: Assume $c>1$ (otherwise the answer is obvious). The problem is equivalent to: find two elements of $\Bbb Z_c^\times$ (the set of invertible elements of $\Bbb Z_c$) which differ by $n\bmod c.$
By the Chinese remainder theorem, we may restrict to the case where $c$ is a power of some prime $p.$
If $p\ne2$ then $\varphi(c)>c/2$ so the injective map $\Bbb Z_c\to\Bbb Z_c,\;t\mapsto t+(n\bmod c)$ cannot send every invertible to a non-invertible, and we are done.
If $p=2,$ the set of differences of two elements of $\Bbb Z_c^\times$ is $2\Bbb Z_c.$ It contains $n\bmod c$ since $n$ is even, so again we are done.
Answer: Let $P$ be the set of primes such that $1 + n \not \equiv 0 \pmod{p}$.
Let $Q$ be the set of primes such that $ -1 + n \not \equiv 0 \pmod{p}$.
Claim: Prove that $P \cup Q$ is the set of all primes.
(I'm not requiring that it's a disjoint union.)
Consider the prime decomposition of $c = \prod p_i^{r_i}$.
Solve the modular arithmetic system
$a \equiv  \begin{cases} 1 \pmod{p_i}  & p_i \in P \\ -1 \pmod{p_i} & \text{otherwise} \end{cases}$
Claim: Prove that $a, b=a+n$ satisfy the conditions of the problem.
Followup: Where did we use that $n$ is even?
Can we modify the proof so that this works for all numbers $n$?
Comment: More generally, to show $\,f(x) = x(x+n)\,$ has a value coprime to $\,c = \prod p_i^{e_i}$ it suffices to show for every $\,i\,$ that $f\,$ has a value coprime to  $p_i,\,$ i.e. $\,f(x_i)\not\equiv 0\pmod{\!p_i},\,$ since then we get a solution by using CRT to solve the system $\,x\equiv x_i\pmod{\!p_i}.\,$ It's clear if $\,p_i > 2 = \deg f\,$ since $\Bbb Z_p$ is a field so $f$ has as most $2 < p_i$ roots, so there is a nonroot $\,x_i.\,$ If some $\,p_i = 2\,$ then $\bmod 2\!:\ n\equiv 0\Rightarrow f = x(x+n)\equiv x^2\,$ so $\,f(1)\not\equiv 0$.
Comment: The same idea works for higher degree polynomials $\,f\in\Bbb Z[x]\,$ (that don't vanish mod any $p_i,$ i.e. $p_i$ does not divide some coef). As above, we reduce to finding a nonroot for all primes $p_i \le \deg f)$.


HAProxy to terminate SSL also send SSL to backend server

Question: I would like terminate SSL at HAProxy, do some manipulation on the header, rewrite URL and re-encrypt traffic and send to backend servers as SSL?
I can't seem to find a way to do this. I can get regular SSL termination done, and send plain HTTP requests to backend. But I need to send SSL to backend.
I would like to have the following features:

Extract x-forwarded-for headers, to get the real client IP behind proxy.
Implement session stickiness using cookie.
Do some URL rewriting.
Send SSL traffic to backend using cookie based session stickieness.

Unless I terminate SSL at haproxy end, I cannot get URL rewriting done.
Any help from the good people here would be highly appreciated.
Comment: Can you post your current config, and tell us what's not working?
Answer: There's nothing special to do in haproxy.cfg. You simply configure whatever URL rewrites and header manipulations you want within your HAProxy frontend and then redirect traffic to your SSL backend. Here's an quick example:
<code>frontend app1_ssl
    bind *:443 ssl crt /etc/haproxy/certs.d/example.com.crt crt /etc/haproxy/certs.d/ no-sslv3

    option http-server-close
    option forwardfor
    http-request add-header X-Forwarded-Proto https
    http-request add-header X-Forwarded-Port 443

    # set HTTP Strict Transport Security (HTST) header
    http-response add-header Strict-Transport-Security max-age=15768000

    # some ACLs and URL rewrites...

    default_backend             backend_app1_ssl

backend backend_app1_ssl
    server mybackendserver 127.0.0.1:4433 ssl verify none
</code>
Comment: Awesome! Let me try that. I wasn't able to find that "server .... ssl" thing anywhere.
Comment: With `ssl verify none` traffic between HAProxy and backend server is still encrypted, but validity of backend's SSL certificate isn't checked.
Comment: @eli you are right. Deprecation warning was added after my initial answer. I've updated my answer. Cheers!
Comment: Many thanks for your help. I was able to get it working. However, I could not get SSL sticky to work. Could you help please? The question is here: http://serverfault.com/questions/738397/haproxy-ssl-roundrobin-not-working-when-ssl-terminated-and-forwarded
Comment: Great, I'm glad I was able to help. Don't forget to accept my answer so others can see right away that it was helpful ;-)
Comment: I just found this answer and discovered I had already upvoted it... Wish I could upvote again! Thanks!
Comment: Does the encryption really works by by giving ssl verify none? From the haproxy documentation, " If set
to 'none', server certificate is not verified. In the other case, The
certificate provided by the server is verified using CAs from 'ca-file'". Can someone please clarify this?
Comment: according to (\*) `reqadd` is an old directive and `https-request set-header X-Forwarded-Proto https` can be used. (\*: https://stackoverflow.com/a/51940105/4940240 from https://stackoverflow.com/questions/51928504/x-forwarded-proto-https-in-frontend-or-backend-haproxy/51940105#51940105). (just noting)
Comment: http://cbonte.github.io/haproxy-dconv/2.5/configuration.html#5.2-ssl
the `server` keywords support a lot of interesting settings, documented here.


Is reading bass clef by reference to the corresponding position in treble clef a bad learning method?

Question: I've taught myself to play the piano when I was young, but I never really learned to read sheet music. Usually I would just look at a synthesia tutorial online or try to do it by ear (which I'm not that good at).
I've decided to take it a bit more serious and buy a new digital piano with weighted keys etc. and also want to learn how to read sheet music. I started with the treble clef notes yesterday and I can identify all the notes fairly quickly already (40 notes in 1 minute without mistakes). Now if I try to learn the bass clef notes, I will automatically pretend it's a treble clef but drop down 1 line. This is also fairly fast but I'm afraid this way of learning it, is the wrong way. I don't want to always keep using this extra link to identify the right note.
Is it bad to continue practicing like this? Or do you think I will eventually automatically do it without having to think about what note it would be on the treble clef?
Comment: Eventually your brain will instantly associate lines and spaces with notes, either due to your method being great or it being so annoying it'll naturally try to find a way around it.
Answer: I suppose the 'correct' way to think about the clefs is to understand what they are actually supposed to indicate. Each clef indicates which line is the reference note. (I'm not sure if there is a specific terms for the reference note.)
So, the basic clef types are 'G' (treble), 'F' (bass), and 'C' (alto and tenor):

the little spiral of the G clef shows the line where G is
the double dots of the F clef shows where F is
the center of the two curves of the C clef shows where C is

Instead of 'dropping down a line,' you can try to read from the F line as indicated by the clef.
Answer: I really don't think it's a bad way. Especially as it works for you!
There are several methods of reading notes, and in the initial stages, where you are, anything that's successful proves its point.
Eventually you'll cut out the middle stage. At the moment, you're calculating what the note is called on the staff - either of them - and then finding that note of the same name on the keys.
Eventually, you won't be thinking of individual note names - no-one thinks that fast !What will happen is you'll be playing a note, and see that the next one is on the next line, space, up two lines, whatever, and automatically hit it. Its name is academic, as long as you have a point from which to go, you'll find the next one, sort of by interval, but you probably won't even be thinking 'that's a M3, I'm here, so I need to go there'.
Others have pointed out why the grand stave works as it does. That's fine, but if your method works, and it seems to, stick with it, until it atrophies and you don't need it any more!
Answer: This is very bad, don't do this - you will always be confused!
A good practice is to draw the grand staff with a space between the lines that you can write the names of the notes directly on and between the lines (where the dots are meant to be:
starting with scales than copying tunes and finally your own melodies and ideas.
Counting the notes per minute is a very good idea, but transposing is bad.

Comment: You're right about not accepting an answer too early. Shouldn't have done that before I hear all sides. My view on things can change when I hear different perspectives. And if I see a post get dislikes, it's obvious most people don't agree with it.
Comment: Can't understand why you state 'transposing is bad'. I see nothing wrong in it - after all, that's what the C clef does! Guitarists do it all the time, albeit a whole octave. In fact, it was after a couple of years playing that it dawned on me that's what we do!
Comment: @user45266 - perhaps it's a dyslexic way of writing 'sneaked'..?
Comment: This kind of transposing is not the same what guitarists do when they transpose the patterns from bend to bend. It took a long time to me to develop reading the bass clef without transposing and to understand the system of the grand staff just because I used to know only the treble clef as I started piano playing only with 13 years. What I mean is rather in the direction of Michael Curtis post. I will show you tomorrow what I mean. You will say everyone has to find his own way, there are no rules for everyone: I fully agree.
Comment: Well, @Tim, it was interesting, until you snaked its accept. Now, it's just a downvoted answer :)
Comment: @user45266 - not my choice! Many times I've suggested to OPs that to accept an answer of mine too early is not a good thing to do. What's 'snaked'?
Comment: @Tim True, true. I just noticed it and thought it was funny :) "To snake" (slang/informal), ie. to sneakily and quickly take another's possession (in the manner of a striking snake). It's not standard or anything, I just use it (although I do believe surfers often describe other unscrupulous surfers as having "snaked" a wave). For example, *"Bobathan **snakes** my Cheetos when I'm not looking, and I can tell because his fingers are all orange"*
Comment: @Tipsi - to be fair, it's hardly 'most people' - there were two dvs only, meaning only two people disagreed. Not particularly mind-changing, and since we don't actually know why they dv-ed (rarely do !) it's all not too convincing.
Comment: @Tim, that's true. But I'm not only basing it off downvotes. The difference between the 2 answers is 6 votes. Also after hearing a different perspective, it did make more sense to me.
Comment: there has been a very similar question two weeks ago: one was asking whether there is a double c' by the as he didn't understand  the e' on the ledger line in the bass clef ...My first reaction was how can one ask such a elementary basic question? if he is able to post an question in SE he should be able to google for bass and treble clef. Then I looked up myself, and what did I found in the explanations? Nothing! I also looked up in books. Nothing! That's why I post here these 2 pictures.


Basic problem in Mechanics

Question:
A body starts from rest and builds up to a velocity of 7.2km/hr in half a minute.
A) What is the acceleration?
  B) What distance is travelled in the half minute?

Apparently the answer is $1/15 m/s^2$ for part A, I've tried used the equation $v = u + at$ but for some reason I keep receiving the answer $240m/s^2$. What on earth am I doing wrong?
Comment: Try the equation a=(vf-vi)/t.  Converting the velocity to m/s and time to s.
Comment: $7.2 km/h=2m/s.$
Comment: I'm a high school student and have not been provided with the equation of converting velocity to m/s. I assumed that the 7.2 would be 7200 in meters and that time would be 30 seconds. Could you please elaborate on your equation?
Comment: $$7.2\text{km} \times \frac{1000\text{m}}{1\text{km}}=?\text{m}$$ That's the conversion. Also, use $$1\text{hour}\times\frac{3600\text{s}}{1\text{hour}}=?\text{s}$$
Answer: You need to do the conversion first. For distance:
$$7.2 \text{ km}\times\frac{1000\text{ meters}}{1\text{ km}}=7200\text{ meters}$$
For time:
$$1\text{ hour}\times\frac{3600\text{ seconds}}{1\text{ hour}}=3600\text{ seconds}$$
This is then
$$\frac{7200\text{ meters}}{3600\text{ seconds}}=2\text{ m}/\text{s}$$
Now use the kinematic equation
$$v_f=v_0+at$$
Re-arrange to get
$$a=\frac{v_f-v_0}{t}$$
and solve for the acceleration (You know that $t=30\text{ seconds}$). Distance traveled can be found by
$$x=v_0t+\frac{1}{2}at^2$$
Having the body start from rest ($v_0=0$) simplifies these a lot.
Answer: Sure, I can elaborate.  you have $\frac{7.2km}{1hour}$  Convert the top part to meters and the bottom part to seconds.  $\frac{7.2 \times 1000}{1 \times 60 \times 60}$  That is how mfl got $2m/s$ above.  So now you have an initial velocity of 0.  A final velocity of 2.  And it had 30 seconds to make the increase.
Answer: It looks as though you're not properly converting your units. I think that the easiest way to do this problem is to convert your velocity to meters per second, and your time to seconds. In the comments below your question, mlf provided a formula to convert the units of velocity from $\text{km/h}$ to $\text{m/s}$. I'll explain how mlf came to that formula. There are 60 seconds in a minute, and 60 minutes in an hour. Therefore, $1\,\text{h} = 60*60 \,\text{s} = 3600 \,\text{s}$. Similarly, there are 1000 meters in a kilometer, so $1 \,\text{km} = 1000 \,\text{m}$. Combining these conversion factors, we get:
$1 \, \text{km/h} = (1000 \, \text{m})\left(\frac{1}{ 3600 \, \text{s} }\right) = \frac{5}{18} \,\text{m/s}$.
If we multiply both sides of this equation by 7.2, we can convert 7.2 $\text{km/h}$ to meters per second:
$7.2 \, \text{km/h} = (\frac{5}{18}\times 7.2) \,\text{m/s} = 2.0 \,\text{m/s}$.
Now that we've converted our velocity from kilometers per hour into meters per second, we can convert our change in time to seconds. Half a minute corresponds to $30 \, \text{s}$, so the time-value is $30 \, \text{s}$. From here, all that's left to do is plug our velocity and change in time values into the equation $a = \frac{\Delta v}{\Delta t} $, where $\Delta v$ represents the change in velocity, and $\Delta t$ represents the change in time Do so and you'll get the desired answer.
Hope this helps!

Basic problem in Mechanics


Como deixar o meu Button inabilitado mas aparecendo a imagem?

Question: Alguém sabe se tem alguma opção de eu deixar meu Button inabilitado mas aparecendo a imagem do  Button?
Como fica se eu desabilito o Button.

Código XAML do Button.
<Button x:Name="BtnCalc" Content="" HorizontalAlignment="Left" Margin="773,206,-172,0"
        VerticalAlignment="Top" Width="200" Click="BtnCalc_Click" Height="85"
        BorderBrush="{x:Null}">
    <Button.Background>
        <ImageBrush ImageSource="images/calculador.png"/>
    </Button.Background>
</Button>

Answer: Assim:
Coloque um <Button.Content> dentro um <StackPanel> e por fim um <Image>, vai resolver esse problema
<Button Name="BtnCalc"
        HorizontalAlignment="Left"
        Margin="53,62,0,0"
        VerticalAlignment="Top" Width="266" Height="136" Click="BtnCalc_Click_1">
    <Button.Content>
        <StackPanel>
            <Image Source="images/1.jpg"></Image>
        </StackPanel>
    </Button.Content>
</Button>

Esses é um exemplo adeque ao seu com as medidas da imagem.


Moq WCF service doesn't return value

Question: My problem is mocked object doesn't return the value it return null instead
My MSpec UnitTest as follows
<code>public class With_Fake_Data_Service
{
    protected static Mock<IMProposalWCFService> _fakeDataService;
    protected static FaultContract fault;

    private Establish context = () =>
        {
            _fakeDataService = new Mock<IMProposalWCFService>();

            _fakeDataService.Setup(
                service =>
                service.ReplyToProposal(new ReplyToProposalRequest(Moq.It.IsAny<Proposal>(), Moq.It.IsAny<bool>())))
                            .Returns(new ReplyToProposalResponse( Moq.It.IsAny<bool>(), fault));
            _fakeDataService.Setup(
                service => service.ReplyToProposalEmail(new ReplyToProposalEmailRequest(Moq.It.IsAny<string>(), Moq.It.IsAny<bool>())))
                             .Returns(new ReplyToProposalEmailResponse(Moq.It.IsAny<string>(), fault));

            _fakeDataService.Setup(service => service.GetAllProposals(Moq.It.IsAny<GetAllProposalsRequest>()))
                            .Returns(() => new GetAllProposalsResponse(new List<Proposal>(){new Proposal()}, fault));

            _fakeDataService.Setup(service => service.GetAllProposals(Moq.It.IsAny<GetAllProposalsRequest>())).Verifiable();

        };
}

 public class When_Testing_HomeController_Index : With_Fake_Data_Service
{
    protected static HomeController _homeController;
    protected static IList<Proposal> _proposals;

    private Establish context = () =>
        {
            _homeController = new HomeController(_fakeDataService.Object);
        };

    private Because of = () =>
        {
            var result = _homeController.Index() as ViewResult;
            _proposals = result.Model as IList<Proposal>;
        };

    private It Should_Have_Called_GetAllProposals_At_Least_Once =
        () => _fakeDataService.Verify(service => service.GetAllProposals(Moq.It.IsAny<GetAllProposalsRequest>()), Times.Exactly(1));
}
</code>
Inside the Actual Code HomeController Index Method I get response as null when running above unit test
<code> public HomeController(IMProposalWCFService  service)
    {
        _service = service;
    }

    public ActionResult Index()
    {
        var response = _service.GetAllProposals(new GetAllProposalsRequest());

        if (response.fault == null)    /*Right Here reponse is null when running unit test*/
        {
            var proposals = response.GetAllProposalsResult;
        }

        ViewBag.Message = "Modify this template to jump-start your ASP.NET MVC application.";

        return View();
    }
</code>
Why am i Not getting not null response  in other words why i'm not getting
new GetAllProposalsResponse(new List(){new Proposal()}, fault)
please help
Answer: Found the problem its
<code>    public class With_Fake_Data_Service
{
    protected static Mock<IMProposalWCFService> _fakeDataService;
    protected static FaultContract fault;

    private Establish context = () =>
        {
            _fakeDataService = new Mock<IMProposalWCFService>();

            _fakeDataService.Setup(
                service =>
                service.ReplyToProposal(new ReplyToProposalRequest(Moq.It.IsAny<Proposal>(), Moq.It.IsAny<bool>())))
                            .Returns(new ReplyToProposalResponse( Moq.It.IsAny<bool>(), fault));
            _fakeDataService.Setup(
                service => service.ReplyToProposalEmail(new ReplyToProposalEmailRequest(Moq.It.IsAny<string>(), Moq.It.IsAny<bool>())))
                             .Returns(new ReplyToProposalEmailResponse(Moq.It.IsAny<string>(), fault));

            _fakeDataService.Setup(service => service.GetAllProposals(Moq.It.IsAny<GetAllProposalsRequest>()))
                            .Returns(() => new GetAllProposalsResponse(new List<Proposal>(){new Proposal()}, fault)).Verifiable();

        };
}
</code>
Comment: I recommend [Machine.Fakes](https://github.com/machine/machine.fakes). It makes your code more readable and succinct and gives you better ways to reduce duplication.

Moq WCF service doesn't return value


Preventing data modification from outer source in SQLite

Question: I recently created a password manager using Java for my college project in OOP. To handle database I picked SQLite since using MySQL or SQL server was getting hectic for a small project. Though I am already done with the submission, I was thinking if I could do any further improvement in the project.
Biggest drawback that I have observed yet is that if anyone manages to find the location of database in the system (which is way too easy) it would be very simple to open the database.
Now here two problem arises -

User's password list will be visible
Anyone would be able to modify the data using SQLite manager.

In order to solve the first problem, I already used AES encryption and it is working just fine. However, the second problem still remains.

So in a nut shell, How can I prevent my SQLite DB to get modified except from the Password Manager itself?

Point to note that my application is just an offline Password Manager used on a household PC. So, you can consider the level of threat accordingly. Moreover, the Password Manager itself would have to modify the database content, so assigning the permission should be such that it should not prevent the application to do so.
Note: I was wondering if we can use the limitation of SQLite that only one connection to write the data can be established at a time. Using this the intruder won't be able to modify it. But, I am not sure how it can be implemented.
Comment: Just `chmod` the database as readable only to you. The other users have their own accounts, right? (Also you should be hashing passwords with a password hashing function)
Comment: If someone gets hold of the physical database files nothing prevents them from looking into the data or changing the data (this is also true for any server based DBMS - however it's usually harder to access the files on the database server). Not even a single connection "limit" would prevent that - shutting down your application would render that useless.
Comment: You can't practically defend against an intruder with admin rights. They can set breakpoint and dump the RAM of your program to get pretty much any information they want out of it.
Comment: _" I was wondering if we can use the limitation of SQLite that only one connection can be established at a time."_ This is incorrect. Only one connection can have the database locked for writing at a time, but multiple connections can exist.
Comment: @ColonelThirtyTwo Thanks for correction.
Comment: @ColonelThirtyTwo as I said you'll have to consider intruder have admin rights too. I don't it will still prevent the modification. Moreover, Password Manager itself would have to rewrite the data as user would login and modify it through the application legitimately.
Comment: @ColonelThirtyTwo Considering the complexity I have made few changes in the scenario.
Comment: @Gamma: I tried to answer your question. Does it help? Or are you trying to solve a different problem?
Comment: @ChristianStrempfer Thanks for the answer. Despite providing a near solution to the problem you have made several assumption (like using Cloud). As I said it was a small offline application, I had my expectations that answers lie within that area.

Although I liked your alternative idea and I can give you +1 for this but accepting this answer might not be the best. As for the bounty, if none other suitable answers are submitted I will award that too.

Thanks
Comment: @Gamma: I tried to say that there is no other alternative, because file access permissions are part of the operating system not of your app. I added a part about encrypting the whole database, but I don't think it makes much difference as you're already encrypting the passwords.
Answer: Restrict user access
Only the operating system can secure files against access by unauthorized persons. Put the database into a folder, which is only accessible by the current user, and have a separate database for each user.
Encryption
You're already encrypting the passwords, that's good. If you want to encrypt the whole database, you could have a look at the SQLite Encryption Extension.

The SQLite Encryption Extension (SEE) is an add-on to the public domain version of SQLite that allows an application to read and write encrypted database files.

Also have a look at the question SQLite with encryption/password protection.
Attack
What would actually happen if someone has access to the database file?
If the database is secured properly, the attacker is not able to get the plain passwords (at least not in reasonable time). In the worst case a password is replaced by another one, but that would achieve nothing, besides you using the wrong password and maybe resetting it. Therefore the worst case would be that you'll lose your saved passwords.
You can do nothing to prevent a data loss on a single machine. For example hard disks sometimes just stop working, someone could steal the whole PC, format the hard disk, etc.
Backups
If you really want to make sure that the data is not modified, you need to keep backups on different machines to minimize the possiblity that someone has access to all of them. For example you could upload the database file to a cloud service. Then you sign the file, so that you can see if a file was compromise and if so fallback to another version.
Conclusion
Your password manager is good enough for an offline tool. If you want to improve the data integrity you have to transfer the data to other machines.


c# A sorting by columnheader problem in datagridview

Question: I have a datagridview that is created by using Datatable. When I try to give a name to a column if this name includes a ","  like this : DataColumn dc8 = new DataColumn("22,5"); and if  i try to sort this column by
columnheader it gives error but i change the "," to "." like : DataColumn dc8 = new DataColumn("22.5"); it works.
i need to make the code works in first situation
Comment: not working for me it says not a valid sequence input
Answer: The second answer at  DataTable sorting with Datacolumn Name with comma does work.
Try this in your sorting method:
<code>protected void gridView_Sorting(object sender, GridViewSortEventArgs e)
        {
            DataTable dataTable = GridView1.DataSource as DataTable;

            if (dataTable != null)
            {
                dataTable.Columns["22,5"].ColumnName = "Temporary";
                dataTable.DefaultView.Sort = "Temporary DESC";
                dataTable = dataTable.DefaultView.ToTable();
                dataTable.Columns["Temporary"].ColumnName = "22,5";

                GridView1.DataSource = dataTable;
                GridView1.DataBind();
            }
        }
</code>


Find windows process that is changing time back 1 hour

Question: I have a windows machine which will periodically change the system time, for reasons unknown.  It appears to happen every hour.
This windows machine is a virtual machine (Parallels Desktop 9, Win7 guest, OSX host).  It has an NTP service (NetTime) running that promptly corrects the error, but in those brief few seconds between change and correction, it causes problems.
I have checked:

VM time synchronisation is disabled
Windows "Internet Time" is disabled
Windows Time service is disabled
I only have a single NTP client running, updating every 15 minutes

There is a complication.  We run an overnight astronomy service.  In order to avoid issues arising from automatic DST changes, we disable automatic DST changes, and manually set the machine timezone later in the day to a zone with the correct offset.  Eg in Spain it is DST right now.  Standard time is UTC+1, DST is UTC+2.  The morning after DST changes we set the machine timezone to Greece UTC+2.  Host machine is configured normally (correct timezone, automatic DST changes).  The complication is that the clock changes back to the current time at UTC+1 (pre-DST time).
SOME process is changing this.  Possibly it has it's own timezone setting.  But I have been unable to track it down.  The changes are logged in the System Log.  There are two key entries: where the time is set incorrectly, and when it is corrected:

(Full disclosure, the event log is filtered by Event ID = 1, but the other events appear meaningless).
It's interesting how regular these occur (every hour, to the second).  What's more interesting is that these are hours of uptime.  I can watch the system Up Time in Task Manager, and when it ticks over the hour, the clock changes.
Also interesting is looking at the event details:
- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
        <Provider Name="Microsoft-Windows-Kernel-General" Guid="{GUID}" />
        <EventID>1</EventID>
        <Version>0</Version>
        <Level>4</Level>
        <Task>0</Task>
        <Opcode>0</Opcode>
        <Keywords>0x8000000000000010</Keywords>
        <TimeCreated SystemTime="2018-04-18T00:31:28.500000000Z" />
        <EventRecordID>500706</EventRecordID>
        <Correlation />
        <Execution ProcessID="4" ThreadID="56" />
        <Channel>System</Channel>
        <Computer>T07-VM-GUEST</Computer>
        <Security UserID="SID" />
    </System>
    - <EventData>
        <Data Name="NewTime">2018-04-18T00:31:28.500000000Z</Data>
        <Data Name="OldTime">2018-04-18T01:31:28.861800000Z</Data>
    </EventData>
</Event>

We can see this event changes from 01:31 to 00:31 (UTC times, 03:31 to 02:31 local as seen in event log).  What is particularly interesting is this line:
<Execution ProcessID="4" ThreadID="56" />

PID 4 is the System process:

Using ProcessExplorer I can inspect the System process (PID 4) and I can see some details on ThreadId 56 (assuming they don't get recycled and I'm looking at the correct one):

But it's all gibberish to me.  The only meaningful thing I can see here is the Start Time, and how it relates to the clock change event times (as I said above, every hour in sync with Uptime).
This answer talks about finding time changes in the Security Log, and all of the changes initiated by the NetTime service are in there.  But the problematic changes are suspiciously missing:

Am I correct in my analysis, and if so, why is the System Process changing my system clock every hour?
Comment: @Appleoddity yep, it does seem that way (setting aside the fact that host time sync is disabled).  I've just double-checked, and the other system we've seen it happening on is also a Parallels VM.  Except, the way Parallels changes the guest OS time is using Parallels Tools.  I would have expected the Parallels Tools process to be the one changing the time, not the System process.  I was hoping I could intelligently determine the offender, rather than blindly disabling things and waiting for an hour each time.
Comment: Well, you've done a good job, but sometimes it just takes trial and error. You can find out pretty quickly if it's a running process by disabling most everything and waiting an hour. At least you know if you're wasting your time. I think it's your vm host. But the idea about the tools is good. Maybe it's a driver doing this, maybe the parallels drivers, and that isn't going to show up in the things you've looked at, I don't think. You could try uninstalling the drivers for now.
Comment: @Appleoddity Because I had already disabled VM time sync, and because the changes weren't initiated by the Parallels Tools process (it actually runs 3), I had largely discounted that.  Now you've made me look at it more closely, and I think you're right.  I've found a total of 4 systems exhibiting this problem now, the only 4 we have running Parallels (most are running VMWare).  I'm killing and disabling the Parallels services, just waiting for the next "hour" tick in 15 minutes to confirm.  Then I can take it up with Parallels.
Comment: It sounds like your attempts to "avoid" DST issues are probably the _cause_. As long as the Windows timezone data is accurate, you should either let it handle DST, or run it in UTC.
Comment: @MichaelHampton DST is unacceptable.  UTC would be a big overhaul.  I should mention that there are roughly 30 other similar systems not exhibiting this problem.
Comment: This feels like a vm host issue. Are you sure there is nothing on the host affecting time? You'll need to start troubleshooting this by disabling all non-microsoft services and all startup items. This is easily done with msconfig in Win 7. See if the problem persists. If it does, kill any Microsoft services you can or try running in safe mode. This is a process of elimination.
Comment: Well, not the definitive proof I was looking for.  Even after killing 3 services and an additional process, the time still changed.  I still suspect Parallels (it's happened on every Parallels system, and no where else I can identify).
Answer: Ok, as all the worst types of problems do, this one had two parts.

Windows updates the system time every hour to the hardware clock. (Note it also does this on boot, which sped up testing)
Parallels was virtualising the hardware clock incorrectly.  I have to say I'm using an old version of Parallels (PD9), I would hope they have fixed this by now.

I haven't found any definitive statement about this, but I've seen plenty of people in my shoes confirming the same thing: every hour Windows reads the Real Time Clock (RTC, or the hardware clock in the BIOS) and re-syncing with it.  See references: #1, #2, #3, #4.
Obviously most of these are to do with faulty RTC, flat BIOS batteries, or dual-booting with a *nix OS (which stores UTC in the RTC, not local time).  But the fact remains that Windows will do this every hour.  I have not found a way to disable this yet.
In addition, Parallels doesn't keep a hardware clock per-se, instead it keeps an offset (from System time) in the VM config file.  The problem is that this offset doesn't correctly take into account DST.  So for example I have a host Mac in Madrid, Spain which is normally UTC+1, but currently in Summer Time it is UTC+2.  When I set the time in my guest machine, Parallels computes the difference between my guest time and my host timezone WITHOUT DST.
Let's do an example:
Current time is UTC 00:00.
Madrid standard time would be UTC+1, so 01:00.  Except it's currently DST, UTC+2, 02:00.
I set my guest machine to 02:00, Windows tries to write this to RTC, Parallels computes the difference between my guest time and Madrid Standard (01:00), and saves <TimeShift>-3600</TimeShift> into the config file (file is only updated on reboot, I imagine this variable is tracked in memory during runtime).  So every time Windows reads the RTC (Parallels reads the host system time), it thinks the RTC is set to HostTime-3600s (-1 hr) and updates the time.
I know my guest machine has a complicated setup (manually set to Cairo to find a timezone without DST), I thought I'd give Parallels the benefit of the doubt and see if it works correctly with both guest and host set to the correct timezone (Madrid with DST).  Nope, it still screws up.
Solution:
I can't find a way to disable Windows from reading the RTC every hour, so for the time being I've forced the host machine to use a timezone that doesn't use DST (eg Cairo, UTC+2).  This works.  When I save my guest time to 02:00, and Cairo time (UTC+2) is 02:00, Parallels saves <TimeShift>0</TimeShift>.
Ugly.
Answer: There are two times available to software, the System Time and Local Time.  System time should not be updated regardless of the DST setting.  local Time is what is adjusted.  If this is not the case then there is a programming error in your OS or more likely the application.  it may be that the application is using e.g. GetLocalTime() when it should be using GetSystemTime() and the UI is erroneously stating that it is system time.
Answer: I've been struggling with this exact system event for days. Numerous attempts of disabling or configuring the time synchronization services failed. I ended up finding a workaround to prevent the system from adjusting the system time every 3600 Seconds.
Win32 API offers a SetSystemTime function to set the system time to any desired time. PowerShell offers a more convenient function Set-Date that wraps the API call. By calling Set-Date -Adjust 0 the system time stays unchanged, but the system seams happy. The ugly part, the call needs to be repeated within every 3600 Seconds, to prevent the system event continuously. Hence, I run a PowerShell script to register a scheduled task that is triggered every 59 Minutes:
$taskName = "Mock System Time Synchronization"
$taskDescription = "This task prevents the system to synchronize the system time with the hardware clock every 3600 seconds."

$startTime = (Get-Date) + (New-TimeSpan -Minutes 1)
$action = New-ScheduledTaskAction -Execute 'Powershell.exe' -Argument '-WindowStyle Hidden -command "& Set-Date -Adjust 0"'
$trigger =  New-ScheduledTaskTrigger -Once -At $startTime -RepetitionInterval (New-TimeSpan -Minutes 59)

Register-ScheduledTask -Action $action -Trigger $trigger -TaskName $taskName -Description $taskDescription -User "System"


understanding natural policy gradient

Question: I'm reading this paper on Natural Policy Gradient https://papers.nips.cc/paper/2073-a-natural-policy-gradient.pdf and have some questions regarding how it works.
I'm coming at this from an ML background and do not have a very strong math background so please bear with me. The argument that is being made is that, the gradient that is calculated in standard gradient descent is noncovariant. The definition that I've found of noncovariant is that (from what I can tell) if you reparameterize the policy but the output probability distribution is the same, you can get different directions of steepest descent?
To deal with this, the author looks to find a direction which is covariant using a particular metric tensor.
When looking to choose a $d\theta$ such that the distance between $\eta(\theta)$ and $\eta(\theta + d\theta)$ is maximized, solving gives us $$d\theta = \frac{1}{2 \lambda} G^{-1}  \nabla \eta({\theta}) $$ where $G$ is a metric tensor which, in euclidean space, corresponds to the identity matrix.
My understanding of what a metric tensor is is very sketchy but as far as I can tell it's like a mapping from some parameter space to a distance-value according to some metric (?). To improve the optimization procedure he uses the Fisher Information Matrix $F$ instead of $G$.
The author says that the average reward is technically a function on the set of distributions $\pi_\theta$ rather than on the parameters $\theta$ themselves and suggests transforming the gradient $\nabla \eta(\theta)$ by $F(\theta)^{-1}$ instead since, instead, what we should be looking for is to improve the probability distribution directly.
I'm not entirely sure what exactly this transformation does? If I think about the gradient as just a general greatest rate of change and then transforming it by the metric tensor then I guess intuitively $d\theta$ will be the update that maximizes the informational difference (which seems to try and push the policy to be almost deterministic?). While I can buy this to some degree I'm having quite a bit of trouble understanding how these are going to be different in terms of how $\eta(\theta)$ changes. Is it possible that optimizing using policy gradient and natural gradient will simply arrive at the same solution if given enough time?
Hopefully someone can spot some glaring flaws in my understanding that might help me understand!
Thanks
Answer: This question applies more broadly for any gradient (stochastic or not) descent method as applied to a cost function defined on a space of distributions.
The reason for at least some of the confusion is that the concept of a gradient is often not taught well. The gradient of a function $f$ is supposed to be a vector that points in the direction of steepest ascent of $f$. This direction is found by computing $(\partial f/\partial x_1,...,\partial f/\partial x_n)^T$ only if $x_i$ are Cartesian coordinates in a Euclidean space. I will explain below how this generalizes to other cases, but let me first address why the generalization may be needed.
The point of natural metrics on spaces of distributions is that the Euclidean structure is unnatural. There are several ways to convince oneself of this unnaturalness: one is that one may freely reparametrize a distribution without changing it, and if we have Euclidean structure in one parametrization, we lose it if we go to another one (see below); parameters may have different physical units, or other constraints that preclude a Euclidean structure; the Euclidean structure leads to inefficient optimization paths. The Fisher information metric is a more natural metric on spaces of distributions because it is compatible with transformations of the state space (not just the parameter space). But in the context of the discussion below, it is just a special case; once we understand how to define a gradient for an arbitrary non-Euclidean metric, we can apply this to the Fisher information metric.
Next, let's try to understand why the "vector" of partial derivative does not correspond to a direction in the space. Let's look at a simple example in $\mathbb{R}^2$. Suppose we want to minimize the function $f(x,y)=x^2+2y^2$. Let's do the usual thing and use the following system of ODE's to get to the minimum
$$\dot x=-\frac{\partial f}{\partial x}=-2x,$$
$$\dot y=-\frac{\partial f}{\partial y}=-4y.$$
Then we switch to polar coordinates: $x=r\cos\varphi$ and $y=r\sin\varphi$. In those coordinates, the function reads $f(r,\varphi)=r^2(1+\sin^2\varphi)$. If we do the same as above, we would write
$$\dot r=-\frac{\partial f}{\partial r}=-2r(1+\sin^2\varphi),$$
$$\dot \varphi=-\frac{\partial f}{\partial \varphi}=-2r^2\cos\varphi\sin\varphi.$$
Now, one may check that the two trajectories, although they both converge to $x=y=0$, look different. What happened?
The important point to realize is that the "vector" of partial derivatives $(\partial f/\partial x,\partial f/\partial y)^T$ or $(\partial f/\partial r,\partial f/\partial \varphi)^T$ does not give directions, but rather acts on directions (tangent vectors) to see how $f$ locally changes in that direction. This can be made precise by considering a curve given by $x(t)$ and $y(t)$ (not necessarily a solution to the ODEs above) such that $x(0)=x_0$ and $y(t)=y_0$. The vector tangent $v$ to the curve at the point $p$ given by $x=x_0$ and $y=y_0$ (corresponding to $t=0$) is given by $v^x=\dot x(0)$ and $v^y=\dot y(0)$. Now if we compute the time derivative of $f$ evaluated along the curve at $t=0$, we get
$$\frac{d}{dt}f(x(t),y(t))\Big\vert_{t=0}=\frac{\partial f}{\partial x}(x_0,y_0)\dot x(0)+\frac{\partial f}{\partial y}(x_0,y_0)\dot y(0)=(\partial f/\partial x,\partial f/\partial y)\begin{pmatrix}v_x\\v_y\end{pmatrix},$$
so we see that the partial derivatives form a dual vector that acts on the tangent vector to produce the directional derivative in the direction of the tangent vector. The result of the calculation also does not depend on higher-order behavior of the curve near $t=0$.
We can do the same calculation as above in polar coordinates. Let's pick a curve $r(t)$ and $\varphi(t)$ such that $r(0)=r_0$ and $\varphi(0)=\varphi_0$, as well as $\dot r(0)=v^r$ and $\dot\varphi(0)=v^{\varphi}$. In order for it to be the same curve (at least to first order at $t=0$), we have to have the relations
$$x_0=r_0\cos\varphi_0, \quad y_0=r_0\sin\varphi_0,$$
$$v^x=\frac{\partial x}{\partial r}v^r+\frac{\partial x}{\partial \varphi}v^{\varphi}=\cos\varphi_0v^r-r_0\sin\varphi_0v^{\varphi},$$
$$ v^y=\frac{\partial y}{\partial r}v^r+\frac{\partial y}{\partial \varphi}v^{\varphi}=\sin\varphi_0v^r+r_0\cos\varphi_0v^{\varphi}.$$
Then we can write
$$\frac{d}{dt}f(r(t),\varphi(t))\Big\vert_{t=0}=\frac{\partial f}{\partial r}(r_0,\varphi_0)\dot r(0)+\frac{\partial f}{\partial \varphi}(r_0,\varphi_0)\dot \varphi(0)=(\partial f/\partial r,\partial f/\partial \varphi)\begin{pmatrix}v_r\\v_{\varphi}\end{pmatrix},$$
and one may check that it gives the same result as in cartesian coordiantes because the components of $(\partial f/\partial r,\partial f/\partial \varphi)$ actually transform as
$$ \frac{\partial f}{\partial r} = \frac{\partial x}{\partial r}\frac{\partial f}{\partial x}+\frac{\partial y}{\partial r}\frac{\partial f}{\partial y},\quad \frac{\partial f}{\partial \varphi} = \frac{\partial x}{\partial \varphi}\frac{\partial f}{\partial x}+\frac{\partial y}{\partial \varphi}\frac{\partial f}{\partial y}.$$
To go one step further, we can actually regard the vector $v$ as an abstract object attached to the point $p$. Likewise, the dual vector $df_p$ (differential of $f$ at $p$) is an object in its own right. We may express $v$ and $df_p$ in different coordinates as
$$ v=v^x\partial_x|_p+v^y\partial_y|_p=v^r\partial_r|_p+v^{\varphi}\partial_{\varphi}|_p,$$
$$ df_p=\frac{\partial f}{\partial x}dx+\frac{\partial f}{\partial y}dy=\frac{\partial f}{\partial r}dr+\frac{\partial f}{\partial \varphi}d\varphi.$$
The computation above may be written in a coordinate-invariant way as
$$\frac{d}{dt}f(p(t))\Big\vert_{t=0}=vf=df_p(v),$$
reflecting once again the duality of $v$ and $df_p$.
Ok, now that we understand the distinction between directions (tangent vectors) and their duals, of which $(\partial f/\partial x,\partial f/\partial y)$ or $(\partial f/\partial r,\partial f/\partial \varphi)$ are particular examples (a general dual vector does not need to come from a function, at least not globally), we may consider the problem of defining a gradient operator.
The gradient applied to a function $f$ at $p$ should produce a tangent vector that in some sense maximizes the local change in $f$ when walking in the direction of the tangent vector. In order for this to make sense, we have to constrain the length of the vector (otherwise the target of this maximization is unbounded). Thus, in order to define a gradient, we have to have a way of measuring the length of vectors. This is called a Riemannian metric, which is a way of specifying at each point a scalar product such that we can compute $\langle v,w\rangle_p$ for any two vectors $v,w$ at $p$, in a way that depends smoothly on the point. We may then define a steepest direction of $f$ at $p$ as the unit vector that maximizes the directional derivative of $f$ at $p$
$$sf_p = \arg \max_{\langle v,v\rangle_p=1} vf= \arg \max_{\langle v,v\rangle_p=1} df_p(v), \tag1$$
and then we define the gradient as a scaled version that also takes into account how fast $f$ actually changes in that direction. We therefore multiply $sf_p$ by the directional derivative of $f$ in the direction of $sf_p$:
$$\text{grad} f_p=df_p(sf_p)\;sf_p.$$
It may then be shown that $\text{grad} f_p$ satisfies
$$\langle \text{grad} f_p, v \rangle_p = df_p(v) \tag2$$
for every tangent vector $v$. By the non-degeneracy of the scalar product, this then uniquely defines $\text{grad} f_p$ in terms of the inverse metric tensor written in the OP. In order to show (2), we may write down a Lagrangian for the constrained optimization problem (1):
$$\mathcal{L}(v)=df_p(v)+\lambda(\langle v,v\rangle_p-1),$$
where $\lambda$ is some Lagrange multiplier. The first variation of this reads
$$\mathcal{L}(v+\delta v)-\mathcal{L}(v)=df_p(\delta v)+2\lambda \langle v,\delta v\rangle_p = 0, \quad \forall\delta v,$$
from which we obtain (by setting $v=\delta v=sf_p$) that $-2\lambda=df_p(sf_p)$ and then by plugging this back into the first variation for $v=sf_p$,
$$\langle \text{grad} f_p, \delta v \rangle_p=df_p(sf_p)\langle sf_p, \delta v \rangle_p = df_p(\delta v), \quad \forall\delta v.$$
We may now come back to the example using cartesian and polar coordinates. The first observation is that if we use a Euclidean scalar product to measure the length of a vector in cartesian coordinates, it becomes non-Euclidean in polar coordinates in view of the transformation of the vector components from cartesian to polar:
$$\langle v,v \rangle_p=(v^x)^2+(v^y)^2=(v^r)^2+r^2 (v^{\varphi})^2.$$
This means that while in cartesian coordinates $(\partial f/\partial x,\partial f/\partial y)^T$ is the gradient, this property does not hold in polar coordinates! Instead, we have from (2) that $\text{grad} f_p=(\text{grad} f_p)^r\partial_r|_p+(\text{grad} f_p)^{\varphi}\partial_{\varphi}|_p$ such that
$$(\text{grad} f_p)^r v^r+ r^2 (\text{grad} f_p)^{\varphi} v^{\varphi}=\frac{\partial f}{\partial r}(p)v^r+\frac{\partial f}{\partial \varphi}(p)v^{\varphi}$$
holds for all $v^r,v^{\varphi}$.
We immediately obtain
$$(\text{grad} f_p)^r =\frac{\partial f}{\partial r}(p)=2r(1+\sin^2\varphi),$$ $$(\text{grad} f_p)^{\varphi} = \frac{1}{r^2}\frac{\partial f}{\partial \varphi}(p)=2\cos\varphi\sin\varphi.$$
If we now write a dynamical system as $\dot r=-(\text{grad} f)^r$ and $\dot\varphi=-(\text{grad} f)^{\varphi}$, we obtain the same solution curve as in cartesian coordinates.
Comment: Thank you so much for sharing all of this. It's quite overwhelming but I will struggle through it. For further background reading do you have any recommendations that might help?
Comment: Just to clarify, earlier you used x˙(0)= -∂f/∂x, however later on you've written that d/dt f(x(t),y(t))∣∣∣t=0 = ∂f/∂x(x0,y0) x˙(0), which looks confusing to me? is x˙(0) not supposed to be equal to ∂x/∂t in this equation?
Comment: The first was a dynamical system that I defined explicitely (gradient descent on the example $f$). Later on, I'm doing completely general calculations (after the remark that I'm considering a curve that is not necessarily a solution to the dynamical system above). The dot notation is just a shorthand for the time derivative.

understanding natural policy gradient


List is not null, but can't call .isEmpty without nullpointer exception

Question:     Laerer laerer = (Laerer) session.getAttribute("laererInnlogget");
    List<Spoerreskjema> ss = laerer.getSkjemaliste();

    if(ss == null)
        System.out.println("1");
    if(ss != null)
        System.out.println("2");
    if(ss.isEmpty())
        System.out.println("3");
    if(!ss.isEmpty())
        System.out.println("4");

The if-loops are there to show what the problem is.
The console will show 2, and then throw a NullPointerException upon reaching ss.isEmpty()
The laerer object is not null, but contains a List which should be null (or atleast empty)
All of this is stored, and fetched in/from a postgresql database.
Any idea what the problem could be?
Btw, calling ss.size() will also cause NullPointerException
I essentially want to be able to check if the list is empty, or not.
SEVERE: Servlet.service() for servlet [no.hib.prosjekt01.controllers.laerer.LaererHjemServlet] in context with path [/prosjekt01] threw exception
java.lang.NullPointerException
    at org.apache.openjpa.enhance.no$hib$prosjekt01$models$Laerer$pcsubclass.pcReplaceField(Unknown Source)
    at org.apache.openjpa.kernel.StateManagerImpl.replaceField(StateManagerImpl.java:3254)
    at org.apache.openjpa.kernel.StateManagerImpl.storeObjectField(StateManagerImpl.java:2681)
    at org.apache.openjpa.kernel.StateManagerImpl.storeObject(StateManagerImpl.java:2671)
    at org.apache.openjpa.jdbc.meta.strats.StoreCollectionFieldStrategy.load(StoreCollectionFieldStrategy.java:596)
    at org.apache.openjpa.jdbc.meta.FieldMapping.load(FieldMapping.java:934)
    at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.load(JDBCStoreManager.java:681)
    at org.apache.openjpa.kernel.DelegatingStoreManager.load(DelegatingStoreManager.java:117)
    at org.apache.openjpa.kernel.ROPStoreManager.load(ROPStoreManager.java:78)
    at org.apache.openjpa.kernel.StateManagerImpl.loadFields(StateManagerImpl.java:3146)
    at org.apache.openjpa.kernel.StateManagerImpl.loadField(StateManagerImpl.java:3226)
    at org.apache.openjpa.kernel.StateManagerImpl.fetchObjectField(StateManagerImpl.java:2468)
    at org.apache.openjpa.kernel.StateManagerImpl.fetchField(StateManagerImpl.java:890)
    at org.apache.openjpa.kernel.StateManagerImpl.fetch(StateManagerImpl.java:852)
    at org.apache.openjpa.enhance.RedefinitionHelper$1.invoke(RedefinitionHelper.java:230)
    at com.sun.proxy.$Proxy68.isEmpty(Unknown Source)
    at no.hib.prosjekt01.controllers.laerer.LaererHjemServlet.doGet(LaererHjemServlet.java:35)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.tomee.catalina.OpenEJBValve.invoke(OpenEJBValve.java:44)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Unknown Source)

Edit1:
Creating subclass for "[class no.hib.prosjekt01.models.Spoerreskjema, class no.hib.prosjekt01.models.Laerer, class no.hib.prosjekt01.models.Svar, class no.hib.prosjekt01.models.Spoersmaal, class no.hib.prosjekt01.models.Kryptering]". This means that your application will be less efficient and will consume more memory than it would if you ran the OpenJPA enhancer. Additionally, lazy loading will not be available for one-to-one and many-to-one persistent attributes in types using field access; they will be loaded eagerly instead.

Edit2:
@Entity
@Table(name = "laerer")
public class Laerer implements Serializable {
private static final long serialVersionUID = 1L;

@Id
private String id;
private String fornavn;
private String etternavn;
@Lob
private byte[] passord;

@OneToMany(fetch=FetchType.LAZY, cascade = CascadeType.ALL, mappedBy = "laerer")
private List<Spoerreskjema> skjemaliste;

public Laerer(String epost, String fornavn, String etternavn, byte[] passord) {
    this.id = epost;
    this.fornavn = fornavn;
    this.etternavn = etternavn;
    this.passord = passord;
    this.skjemaliste = new ArrayList<Spoerreskjema>();
}

public Laerer() {
    id = null;
    fornavn = null;
    etternavn = null;
    passord = null;
    skjemaliste = null;
}

Comment: Please use a debugger and check out of which implementation your list is. When you are using an ORM Framework for example, most lists are based on proxi objects which tries to load your data lazy. So when you access your list outside of your database transaction, strange behaviors are normal ;) So inspect your list impl. look up the API for this to handle it correctly.
Comment: whats the return type of laerer.getSkjemaliste()
Comment: Try printing `laerer.getSkjemaliste().isEmpty()' directly, what is printed? You should try adding a break point and check the ss type.
Comment: Well maybe the object you get is of a class with such a strange implementation of isEmpty method.
Comment: @Pintouch Still gives me a nullpointer exception
Comment: @Nailgun the isEmpty method is from the  api
Comment: @NiravPrajapati it is List
Comment: @Patidati `List` is just an interface. What is the class of your object?
Comment: @MarounMaroun I don't use it here to illustrate a point :)
If the console wrote 1, i would expect a nullpointer exception, but since it writes 2 i shouldnt get one
Comment: Try breaking on the first if statement, and check ss object
Comment: @rmertins I actually am loading this data lazy, could that be the problem? If so, could you point me in the correct direction by any chance :)?
Comment: What ORM are you using, where are your transaction borders? How does the instance of your list look in the debugger after: laerer.getSkjemaliste(); ?
Comment: @rmertins Using postgresql, not sure what you mean with transaction borders :)
Added something in the description, if that helps?
Comment: my guts feeling is telling me something else is throwing exception somewhere else in you program. isEmpty() method should not throw NullPointException. unless ur list is empty to begin with.
Comment: I see it is a servelet so you are in a webaplication using openJPA as ORM. Also it says that your ORM is loading one2one and one2many relations eager. This means the list (if it is one2many) should already be loaded. The problem now can only exist in the list implementation of openJPA.
Comment: Are you sure that this where exception is occurring. It could occur somewhere else in another thread, coincidentally display when you reach `ss.isEmpty()`.
Comment: Transaction borders, means the definition from thet start of an database transaction till the end. In an application using jpa for example you can spread a transaction over several database calls to handle for example lazy loading.
Comment: I have narrowed this down to this :) The problem originally was in a jsp. It is the usage of the list that is causing the error.
Comment: @rmertins yes that's correct, this is in a servlet:)
You say the problem can only exist in the list implementation of openJPA... what now? Any idea what I could do to try to fix this?
Comment: Can you add your Laerer class with JPA annotations in your question?
Comment: @Patidati just out of curiosity mate. could you put a print out before your if statements (e.g. here1).and put a print out right at the end(e.g. here2) and see if it print "here1" and "here2" at all?
Comment: So after some reachare and this line: com.sun.proxy.$Proxy68.isEmpty(Unknown Source) after a doGet() shows me that you access the list after his transaction. Try to get the size of the list in your DAO Object after the query execute. Before you return the Laerer Object. The I just saw that you getting the Laerer  object from the session. There it is always out of the database transaction.
Comment: I think that you're facing this problem : http://stackoverflow.com/questions/6864574/openjpa-lazy-fetching-does-not-work?rq=1 . You have to re-attach the context of the query to fetch the list
Comment: @rmertins I'm using  getSingleResult, and handling it if it should return null. This is handled a good while before this error has the potential to occur
Answer: OK now I have it.
You are loading the Laerer  Object and then placing it in the session.
Somewhere later you try to access a lazy loaded collection on it.
But the problem is, that the proxy inside of your list, which should call your database to get the data, needs to be in the same transaction as your call to the database when you loaded Laerer.
There a several solutions.

When you know that there are not so many objects in that list change to "EAGER" and you are happy.
In your DAO after you got the Laerer Object from the entitymanager call the list to load it. Same as loading it directly "EAGER". (JPA defines that an entity can only have one EAGER Loaded list)
Don't load directly the list. Instead load the list of IDs when loading Laerer Object. When you then access the list of ids over the session you can call the database again to load the required objects by IDs. For example you can place the list of IDs as a comma seperated string in a transient variable of your entity. This string can directly be use in an SELECT a FROM a.class WHERE a.id IN (:IDS) Query.
Comment: This makes a lot of sense :) I will implement it and come back to tell if it works or not, but i think it should.
Comment: AHHHHH LOVE YOU <3
It works :D !!
Used method 3, as it was the most simply of them :)


Changing the key/value pair of a field list in Drupal 7 once content has been added to the site

Question: I have a field list on my content type that I need to edit. Specifically I just need to change one of the key/value pairs. Unfortunately there's already content in the site so I can't just change it in the admin interface (it gives me an error that there are already records that use this).
I'm looking through my database (mysql) and I can't even find where the key/value pairings are stored. I can see the keys being attached to my nodes and can manually change them there, but that doesn't solve my underlying problem.
So my question is two-fold:
1) where is the table to edit field lists for a content type?
and
2) what's the preferred way to change a field list key/value pairing once content has been added to the site (too much to delete and start from scratch)?
Thanks!
Answer: In future use taxonomy for the list. That way you can add/edit to the list without any problems. Lists should be used for things that are constant, like the seasons of the year etc.
Use MySQL Workbench (assuming you are using MySQL). Open the table 'field_config' and do a select on it.
Look for your field in question. It's data is a blob. Right click on the blob and select 'Open Value in editor'. Select text.
You will see something like this:

a:7:{s:12:"translatable";s:1:"0";s:12:"entity_types";a:0:{}s:8:"settings";a:2:{s:14:"allowed_values";a:2:{i:1;s:3:"one";i:2;s:3:"two";}s:23:"allowed_values_function";s:0:"";}s:7:"storage";a:5:{s:4:"type";s:17:"field_sql_storage";s:8:"settings";a:0:{}s:6:"module";s:17:"field_sql_storage";s:6:"active";s:1:"1";s:7:"details";a:1:{s:3:"sql";a:2:{s:18:"FIELD_LOAD_CURRENT";a:1:{s:21:"field_data_field_list";a:1:{s:5:"value";s:16:"field_list_value";}}s:19:"FIELD_LOAD_REVISION";a:1:{s:25:"field_revision_field_list";a:1:{s:5:"value";s:16:"field_list_value";}}}}}s:12:"foreign
  keys";a:0:{}s:7:"indexes";a:1:{s:5:"value";a:1:{i:0;s:5:"value";}}s:2:"id";s:1:"6";}

You can see under allowed values I have two list entries:
1 => one
2 => two
To add 3 => three I change this bit:
"allowed_values";a:2:{i:1;s:3:"one";i:2;s:3:"two";}
To this:
"allowed_values";a:3:{i:1;s:3:"one";i:2;s:3:"two";i:3;s:5:"three";}
a:2 changed to a:3 as it is the count of values in the list
i:3 indicates the new key
s:5 is the string length of the word three
Then hit apply in the edit box, and hit apply in the SQL window and you are done.

So you don't want to add a new value, but edit an existing one. Use same process as above, but just edit the allowed_values, then run an SQL update on your two tables for the field : 'field_data_XXX', 'field_revision_XX' to update old key to new key.
Comment: @J Reynolds Brilliant. Thank you, it works like a charm. Also very interested to be learning a technique in Workbench finally. I'd downloaded it and played around with it, but not used it until now. I'd also wondered about that table field_config but figured I couldn't change the Blob. Anyway, thanks again!
Comment: For Drupal 8 it would appear this data is in the config table. I modified a key, value in the data blob, adjusted the string lengths but Drupal reported Attempt to create a field field_my_field that does not exist on entity type XX. I'm unsure of your statement to run SQL update to update the keys. When I look in the table field_my_field it doesn't seem to need this key? What did I not understand? thx.


Ember.js: How to filter a model by route

Question: I'm trying to adopt the completed/archived-route-behavior from this TodoMVC-Tutorial.
I'm using Ember 1.7.0, Ember Data 1.0.0-beta.9 and Handlebars 1.3.0.
I want to get all active users on my index route (example.com/organization/users) and all archived users on my archived route (example.com/organization/users/archived).
But the filter doesn't work: archived users are shown on my index route and vice versa, but my console.log output is correct.
Update: Please check this JSBin: http://emberjs.jsbin.com/puxeludeviga/3/

That's how I tried it:
Docket.OrganizationUsersRoute = Docket.AuthenticatedRoute.extend({

  model: function() {
    return this.store.find('user');
  },

  renderTemplate: function() {

    // render all posts
    this.render('organization/users', {
      into: 'application'
    });

    // render toolbar
    this.render('organization/toolbar', {
      into: 'application',
      outlet: 'toolbar'
    });
  }

});

Docket.OrganizationUsersIndexRoute = Docket.OrganizationUsersRoute.extend({
  model: function() {
    console.log('index');
    return this.store.filter('user', function(user) {
      return !user.get('archived');
    });
  }
});

Docket.OrganizationUsersArchivedRoute = Docket.OrganizationUsersRoute.extend({
  model: function() {
    console.log('archived');
    return this.store.filter('user', function(user) {
      return user.get('archived');
    });
  }
});

And that's my template:

    {{#each}}

                <button {{action "remove" this}} class="ei-icon-close"></button>

                {{initial}}{{name}}

    {{else}}

No users

    {{/each}}

My (shortened) router.js file:
Docket.Router.map(function() {

  // organization route
  this.resource('organization', function() {

    // user routes
    this.resource('organization.users', { path: '/users' }, function() {
      this.route('archived');
    });

  });

});

Comment: Are you sure the archived values are returning what you think they are? ie. If you log the value of archived for each user, is it true/false when it should be?
Comment: Yes, I added `return this.store.filter('user', function(user) { console.log(user.get('name'), user.get('archived')); });` to both routes and it returns the correct results.
Comment: could you please show your `router.js` file
Comment: @slevin At this point, I'd suggest making a JSBin (in general, a JSBin does wonders for revealing your problem to others). I'm having to make too many assumptions about your application. How you are going back in forth between the routes. Link-to helpers? Or do you have render actions. Or are you using the URL. Does it happen in one case and not the others? Etc...you get my point
Comment: OK, I added a link to a JSBin in my question.
Answer: I think I found your issue. The thing here is that template you're rendering the users into, has this name:
organization/users

and turns out that it matches to you organization.users route, so, it is always rendering the data from that route. You need to change the name of you template, let's say to:
organization/users_list

and then, use that in your routes, it will not have any problems because the name is different to the users route.
something like:

and in your routes:
renderTemplate: function() {

  // render all posts
  this.render('organization/users_list', {
    into: 'application'
  });

  // render toolbar
  this.render('organization/toolbar', {
    into: 'application',
    outlet: 'toolbar'
  });
}


How do you use numPerms to get the total number of unique permutations for a PERMANOVA?

Question: I am trying to calculate the number of unique permutations for a PERMANOVA using the R adonis2 function but I am getting stuck on defining my permutation scheme.
As suggested here I am using the numPerms function from the permute package to get the answer. However, I am getting stuck with how it works, even in simpler designs. Here is an example with an attempt:
# Create sample dataframe with counts for 3 species from 10 different locations
# Counts are randomly generated from 1-100
set.seed(2)
df = data.frame(
  sp1 = sample(1:100, 10),
  sp2 = sample(1:100, 10),
  sp3 = sample(1:100, 10)
)

# Create a vector of data with 2 groups of 5.
myGroups = c(rep("A",5), rep("B", 5))

# Define the permutation scheme.
hh = how(blocks = as.factor(myGroups))

# My PERMANOVA would therefore be as follows with the permutation scheme specified (I think?)
adonis2(df~myGroups, perm = hh)

# Calculate number of permutations
numPerms(df, hh)

However, $2$ groups of $5$ should have $126$ unique permutations (e.g., see here), but this returns $14400$. What did I do wrong?
Answer: You did nothing wrong; {permute} doesn't know how to do permutations of the sort you want.
The permutation test you described via how() is a randomization test restricted by blocks, so there are 5! * 5! such permutations, which is what numPerms() returns.


square root of the matrix

Question: I would like to calculate the square root of a matrix that may be after no positive defined by using MATLAB(so I cannot use chol instruction) . Somewhere I saw the function nearestspd  but I can not find it in MATLAB . Thank you for the help
Answer: Per https://www.mathworks.com/help/matlab/ref/sqrtm.html , in MATLAB, X = sqrtm(A) returns the principal square root of the matrix A, that is, X*X = A.
This is defined for any matrix A, positive definite or not, but may have complex entries (for instance, think of the 1 by 1 matrix "-1").

square root of the matrix