Comparing frequencies or proportions between two groups to find differences

Question: Suppose I have two groups of users, A and B. In case it is relevant, B is much smaller than A.
I have a feature of products purchased across two groups. I want to find items whose popularity is different between the groups and ultimately classify group membership based off this for unlabelled data.
Example: Suppose the feature is "product". My data might look like:
Product, Percentage Group A, Percentage Group B,
Apple, 90%, 80%
Banana 50%, 49%
Chicken 0%, 90%
Lamb   1%, 60%
Kale 50%, 20%
… more stuff
So 90% of people in group A purchased an apple. Nobody in group A purchased chicken. (I also have the raw counts if that is useful).
Data notes:

The lists are very long, Group A may buy products that do not appear in Group B and vice versa, like chicken above.
In the real use case the product names may not carry a semantic meaning.
There are many products which only one user has purchased. So far I am just dropping items which only a "few" users have purchased.

Goal:
I am trying to find differences in the purchases between the two groups and any related insights, ultimately to determine likely group membership based on products purchased. For example above we could observe that chicken is very popular in group B but does not appear in group A. What I am particularly interested in is any stand out differences, like chicken in the example above, where if someone purchased chicken I would be much more likely to think they belong to group B instead of group A.
What have I tried:
One thing I have tried is to calculate the difference in ranking between the two groups and look for items where the difference is large.
Domain experts tell me there are practical differences between the groups. Above it might turn out there is a statistically significant difference between the two groups in purchasing bananas, but the difference between 50 and 49 percent is so small, I doubt my stakeholders would be that interested, compared to the insight about the chicken.
Comment: To answer this interesting question, a few more details are needed
( sorry here is not an answer yet, but I lack reputation to just comment your question): is it right that you have the complete purchase data for each individual in group A and B? and are the data just Apple:yes/no or more detailed? how many individuals roughly, and how many items (apples, bananas)? running tests on each category (apple, banana…) separately would create a multiple testing situation. Standard multiple testing corrections (Bonferroni…) assume independence, which is the choice when you do not have complete informa
Comment: [tag:chi-square]? [tag:logistic] regression?
Comment: Hi  @Ute, I  have the complete purchase data for all users, and I also have the quantity of each item they have purchased (and when). I thought it might make the analysis easier to just say have they bought this product, but I do have the complete purchase history. I have thousands of items and say hundred thousand users in group A and several thousand in B.
Comment: Hi @Sycorax. Thanks for your suggestions. Would I use chi-square separately for each product type or across everything at once? If I have hundreds of products with a stat significant difference, is there a "good" way to find the "big" items (something like an effect size)? Would the LR be at the individual user level, predicting group based on purchases? I am thinking out loud here but something like an odds ratio might be useful here for explainability.
Answer: I can think of different ways to do this. I think the most straightforward however is logistic regression, given your requirements (you need which items are most important and at the same time make class assignments for future datapoints).
You could fit a logistic regression model to predict whether an individual is a member of group A or of group B, using the number of purchases of each item as features. (I'm assuming you have a wealth of individual-level data here). You can then calculate an odds ratio for each feature (i.e. how much each additional purchase of item X increases the odds of being Group B) and a 95% CI on that odds ratio. This functionality is standard in most software packages you can use to do logistic regression (e.g. in R or in statsmodels or scikit-learn in Python, see this StackOverflow for example). This therefore simultaneously answers the question posed by your end users (which features are most important) and constructs a predictive model you can use on new data.
Since your dataset is imbalanced (significantly more class A than class B datapoints), you'll need to use class weighting. You might also need to use L1 (promotes sparsity) or L2 (promotes small weight values) regularization. L1 regularization can be particularly useful if you want a sparse solution (i.e. want most of the weights to be zero so that only a small subset of the features are used, which I think is what you are after). A regularization technique that may be especially useful for your case is elastic net, which is a combination of L1 and L2 regularization. I would do some experiments (e.g. 5x cross validation) on a small subset of your training data to see if regularization is beneficial and if so what hyperparameter setting you need.
Alternatively, you might want to know which items offer the best differentiation between categories and then use only those as inputs to your logistic regression model, in other words, do feature selection before fitting. In this case, you could use a chi-square test at the level of each item (i.e. how many in group A purchased this item or did not and how many in group B purchased this item or did not) or mutual information to determine which items provide the best distinction between groups. Both of these are implemented as feature selection methods in the scikit-learn package in Python, and statsmodels of course has the chi-square test. (I guarantee all this can be done in R as well, although I use R less frequently and am therefore less familiar with all of the useful packages).
I would be aware of course that if using the chi-square test, be careful about how you interpret the p-value since you are of course performing many statistical tests (since you have thousands of items) and therefore if you decide to use a p-value cutoff as some measure of significance, if you do not perform a multiple-test correction you have an increased chance of false positives. Also, if you are running cross-validation experiments when fitting your model, or if you assess your model on some validation or test set, you should perform the feature selection on the training data only to avoid over-estimating the performance of your model.
There are other ways to model this data, but I think these are probably the simplest given what I think you're trying to do.
EDIT: Another possible problem you may encounter involves highly correlated predictors. It is possible that some of the items you're tracking are highly correlated, in other words, people who buy tofu nearly always buy Impossible Burgers and almost never buy chicken, etc. There are a variety of ways you can deal with this.
One approach is to do principal component analysis (PCA) if the data is number of items purchased (if it is instead one-hot encoded yes or no purchase, you can use MCA instead, although it sounds like this is number of items purchased); there are a lot of tutorials on PCA online, it's implemented in scikit-learn and any other common statistical package. In a nutshell, PCA uses the eigenvectors of the covariance matrix as a new basis set for the data. The share of the variance explained by a given principal component (eigenvector) is its associated eigenvalue divided by the sum of all the eigenvalues; so, the principal components can be sorted by eigenvalue and we can use the new features associated with the n largest eigenvalues as input to our model while discarding the rest.
This will eliminate the multicollinearity problem (if present), but it's not ideal for your application, because now the odds ratios from logistic regression will be odds ratios for the principal components, each of which is a linear combination of the original items. This will somewhat complicate your analysis when you're trying to explain your insights to stakeholders. (Although maybe you can find a way to use this to give them an explanation that's sufficient for their needs).
So, I think what I would suggest instead as a place to start is doing feature selection, e.g. using mutual information, which should give you a greatly reduced number of features. At that point, you can check for correlations among those features (e.g. using the Spearman's-r correlation coefficient if the data is the number of an item purchased by a customer). In any case where multiple items are very strongly correlated, you can keep one and eliminate the others as inputs to your logistic regression model. (This doesn't completely eliminate the possibility of multicollinearity but should significantly reduce the likelihood that it is a problem.) This way, when providing an interpretation to stakeholders, you can say "this is the odds ratio for tofu, we find that purchases of impossible meat are strongly correlated with tofu, i.e. customers who purchase the one nearly always purchase the other".
Also, elastic net regularization (combined L1 / L2) is likely to be especially helpful if you have multiple correlated features. L1 regularization alone will help with this but it will tend to keep one of the correlated features and drop the others, whereas elastic net can avoid this problem. Again some experiments on a subset of your data might be helpful here.
Comment: I would definitely cut down on the number of variables if you want to use a logistic model. when your variables are so high dimensional, it is almost certain that some of the variables basically tell the same story. Someone who buys more apples will likely also buy more bananas than the average customer. When variables tell the same story, it becomes statistically very difficult to tell their influence apart. The estimates for the variables effects become extremely shaky and non significant.
I
Comment: How can you do a chisquare test comparing A and B for one item only ?
Comment: It is true that there may be multicollinearity; I will add some advice on how to deal with this if there are multiple highly correlated predictors.

It is straightforward to do a chi-squared test for feature selection, see scikit-learn's implementation for example: https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.chi2.html The details depend on the data, but let's say for example you have category A and category B and # who purchased this item, yes or no; now you have a contingency table
Comment: PCA can be useful to re-express the data in an orthogonal basis, and you are correct to point out that doing so complicates interpretation. However, OP has stated interpretation is the explicit goal of the analysis, so PCA might not be the best choice. An alternative solution to rank-deficiency, prefect separation and multicollinearity in logistic regression is to use a penalized regression (e.g. [tag:ridge-regression], [tag:elastic-net]).
Comment: elastic-net is a good point, thanks, it is likely to work better than L1 regularization alone in that case, will add that as an edit.
Comment: linear discriminant analysis with shrinkage is another option (shrinkage probably necessary, if there is multicollinearity) or nearest shrunken centroids. Still, I tend to think logistic regression is the easiest to interpret, which is the main requirement here
Comment: Yes, I also suggested LDA before. The answer got deleted though ;-)
Thank you for clarifying what was meant with chisquare, and giving a link. According to the link, it is not a chi-square test for one item (that would be new to me).


Guardar uma imagem identificando-a a partir do FrameworkElement

Question: Tenho este layout:
<StackPanel>
    <Button x:Name="button" Content="Button" Click="button_Click" Height="50" />
    <Image x:Name="image" Source="vermelho.jpg" />
</StackPanel>

O código que testei para gravar a imagem para um ficheiro png é o seguinte:
private void button_Click(object sender, RoutedEventArgs e)
{
    Microsoft.Win32.SaveFileDialog dlg = new Microsoft.Win32.SaveFileDialog();
    dlg.FileName = "imagem";
    dlg.DefaultExt = ".png";
    dlg.Filter = "Imagem PNG (.png)|*.png";
    Nullable<bool> result = dlg.ShowDialog();
    if (result == true)
    {
        var encoder = new PngBitmapEncoder();
        SaveUsingEncoder(image, dlg.FileName, encoder);
    }
}

void SaveUsingEncoder(FrameworkElement visual, string fileName, BitmapEncoder encoder)
{
    RenderTargetBitmap bitmap = new RenderTargetBitmap(
        (int)visual.ActualWidth,
        (int)visual.ActualHeight,
        96,
        96,
        PixelFormats.Pbgra32);
    bitmap.Render(visual);
    BitmapFrame frame = BitmapFrame.Create(bitmap);
    encoder.Frames.Add(frame);
    using (var stream = File.Create(fileName))
    {
        encoder.Save(stream);
    }
}

Porque razão é que quando abro a imagem gravada no disco, também é gravada, em transparente, a área correspondente ao botão?
Answer: Para te ajudar reproduzi aqui seu problema e fiz um resumo do que entendi.
RESUMO: O que parece acontecer, é que a classe RenderTargetBitmap, mesmo com o método "Render" contendo o Visual especificado sua medidas definidas de acordo com o alvo. ele não se posiciona corretamente para transformar o alvo em um Bitmap. É como se ele formasse um quadrado posicionado no inicio da Janela ao invés de ficar sobre a imagem. Não sei o motivo disso acontecer, estou pesquisando.
Ademais, consegui solucionar o problema colocando a imagem dentro de um DrawingVisual.
 public void SaveUsingEncoder(FrameworkElement visual, string fileName)
 {
        // SOLUÇÃO DO PROBLEMA: Colocar a imagem dentro de um DrawingVisual que é uma classe de desenho.
        DrawingVisual drawing = new DrawingVisual();
        using (DrawingContext ctx = drawing.RenderOpen())
        {
            VisualBrush vb = new VisualBrush(visual);
            ctx.DrawRectangle(vb, null, new Rect(new Point(0, 0), new Point(visual.Width, visual.Height)));
        }

        // Cria uma nova instância do RenderTargetBitmap onde será convertido um alvo visual para bitmap.
        RenderTargetBitmap bitmap = new RenderTargetBitmap((int)visual.Width, (int)visual.Height, 96, 96, PixelFormats.Pbgra32);
        // Converte o DrawingVisual com sua imagem em Bitmap
        bitmap.Render(drawing);

        // Cria instância do encode
        PngBitmapEncoder encoder = new PngBitmapEncoder();
        // Transforma nosso BitmapSource em uma imagem
        BitmapFrame frame = BitmapFrame.Create(bitmap);
        // Adiciona a imagem ao encoder. Transformando ela em PNG...
        encoder.Frames.Add(frame);

        using (var file = File.Create(fileName))
        {
            encoder.Save(file);
        }
    }

Não sei ao certo se essa é a solução ideal e até mesmo se o que eu disse sobre o RenderTargetBitmap está certo. Caso alguém tenha uma explicação melhor, nos ajude a entender essa situação.
Answer: Eu entretanto abandonei esta abordagem e resolvi assim, de forma mais simples:
//Criar um ficheiro no disco correspondente à imagem exibida na área de preview
private void guardarImagemDiscoDoPreviewButton_Click(object sender, RoutedEventArgs e)
{
    try
    {
        //Converter source da imagem num byte array
        byte[] ImageData = getJPGFromImageControl(previewImage.Source as BitmapImage);

        //Save As Dialog ...
        Microsoft.Win32.SaveFileDialog dlg = new Microsoft.Win32.SaveFileDialog();
        dlg.FileName = "imagem";
        dlg.DefaultExt = ".jpg";
        dlg.Filter = "Imagem JPEG|*.jpg";
        Nullable<bool> result = dlg.ShowDialog();

        //Gravar a imagem no disco
        using (Image image = Image.FromStream(new MemoryStream(ImageData)))
        {
            image.Save(dlg.FileName, ImageFormat.Jpeg);  // Ou Png ...
        }
    }
    catch
    {
        MessageBox.Show("Nenhuma imagem foi ainda carregada.", "ERRO!", MessageBoxButton.OK, MessageBoxImage.Error);

    }
}

Claro que isto resolve o meu problema mas não responde à questão inicial.


Regarding the definition of a Closed Linear Operator

Question: I found this definition:
Let $X, Y$ be normed spaces, $D$ a
subspace of $X$, and $T : D → Y$ a linear operator. The operator $T$ is closed
if its graph is closed in $X\times Y$.
What if $X$ were a subspace of a normed space, say, $V$? So $D$ is also a subspace of $V$. Then for the above definition to be valid, the graph of $T$ must also be closed in $V\times Y$. I understand that if it were closed in $V\times Y$ then it will be closed in $X\times Y$, being the intersection of a closed set in the bigger space with $X\times Y$. But why is it true the other way around as well?
Comment: Oh wait, is there a dependency on the space $X$ itself?
Answer: When we talk about closed operators we think of $X,Y,D$ as all given in advance. The converse implication here is not true unless $X \times Y$ is closed in $V \times Y$. The question of closedness of an operator not being well defined does not arise because we fix the spaces in advance.
Comment: To add more clarity to this an unbounded operator is a quadruple $(X,D,Y,T)$ where $D$ is a subspace of $X$ and $T:D \to Y$ is a linear map. If you change any of the four elemenst you are talking about a different unbounded operator. Unfortunately books rarely clarify this point.


Trees on $\{1,2,3,4,5,6,7,8\}$ with conditions

Question:
1) How many trees on these vertex have at least 2 leaves ?
2) How many trees on these vertex have more than 2 leaves ?

Attempt -
I will use inclusion-exclusion for both cases -
$1)$ $$\binom{8}{2}\cdot6^6-\binom{8}{3}\cdot5^6 + \binom{8}{4}\cdot4^6 - \binom{8}{5}\cdot3^6 + \binom{8}{6}\cdot2^6 - \binom{8}{7}$$
I exclude from the "universe" where 2 of my vertices are leaves.
2) My problem now is when I have at least 3 leaves, and using the same method $$\binom{8}{3}\cdot5^6 - \binom{8}{4}\cdot4^6 +\binom{8}{5}\cdot3^6 -\binom{8}{6}\cdot2^6 + \binom{8}{7}$$
Is everything true ? thank you !
Comment: I am not sure I understand how you get your formula.
Answer: By definition, a tree is connected, and it is a classical result that a tree with at least two vertices has at least two leaves.
Hence 1. is simply asking for the total number of trees on $8$ vertices, given by Cayley's formula:
$$8^6=262144$$
Now a tree has exactly two leaves if and only if it is linear, and such trees almost correspond to permutations on $8$ elements, the only caveat is that each tree corresponds to two permutations (since the trees are not supposed to be oriented). Hence there are $$\dfrac{8!}{2}$$ such trees, and so $$8^6-\dfrac{8!}{2}=241984$$ trees with more than $2$ leaves.


Cannot debug Cortana code with ngrok in botframework

Question: We have enabled our bot on Cortana channel but I am not able to debug my code using ngrok?
When I set the URL of NGROK in azure and run my code,it does not hit the breakpoint . Is there any other way to debug if i am using cortana channel?
Or is it possible to do so with ngrok and how?
Comment: Is any other channel working with ngrok for you? I just tested Cortana with ngrak with no issues.  Are you using the https ngrok endpoint? and. Are you sure everything is configured correctly?  Is cortana able to invoke your skill?
Comment: I agree with JasonSowers, you @user9180315 should check if everything is configured correctly. I shared the steps that I followed to debug cortna channel locally using ngrok, you could refer to it.
Answer:
When I set the URL of NGROK in azure and run my code,it does not hit the breakpoint .

I do a test on my side, I can debug cortna channel locally using ngrok, you could following these steps(or compare with your did):
1) specify your BotId, MicrosoftAppId and your MicrosoftAppPassword in bot application Web.config, and then run your bot application
<appSettings>

  <add key="BotId" value="fehanbasicbot" />
  <add key="MicrosoftAppId" value="878927ac-xxxx-xxxx-xxxx-a87f8eaf0dce" />
  <add key="MicrosoftAppPassword" value="xxxxxxxxxxxxxxxx" />
</appSettings>

2) run the following command and get the ngrok forwarding link (replace 3980 with your bot app port)
ngrok http 3980 -host-header="localhost:3980"

3) On your bot service Settings blade, configure the ngrok forwarding link as Messaging endpoint. Ensure that you maintain "/api/messages" at the end of the URL.

4) send message from Cortana channel, and it could hit the breakpoint

Note:
As JasonSowers mentined in comment, please check if everything is configured correctly. And you could also debug other channel(s) with ngrok.


Exchange's New-MailboxExportRequest behaves strangely with DateTime parameters to ContentFilter

Question: I am attempting to extract a certain date-range of emails from a mailbox into a PST, but using a DateTime parameter for ContentFilter doesn't seem to work as expected.
This should export everything for the last 5 days:
$endDate = Get-Date "00:00:00"
$startDate = $endDate.AddDays(-5)

Write-Host "Exporting items between $startDate and $endDate..."

New-MailboxExportRequest -ContentFilter {(Received -ge $startDate) -and (Received -lt $endDate)} -Mailbox "EmailLog" -FilePath "\\ReadyNAS\backup\Mailboxes\EmailLog\EmailLog.pst"

But the PST ends up with the full 15+Gb of the mailbox in it.
If I specify the dates manually, it works fine:
New-MailboxExportRequest -ContentFilter {(Received -ge "01-06-2013 00:00:00") -and (Received -lt "06-06-2013 00:00:00")} -Mailbox "EmailLog" -FilePath "\\ReadyNAS\backup\Mailboxes\EmailLog\EmailLog-man.pst"

I'm not sure if this is due to some sort of regional-settings conflict, but I would have thought that passing a typed parameter would avoid that sort of sillyness. The reason I suspect this is because if I print out the dates (the Write-Host in the first example) the dates come out in US format:
[PS] C:\Windows\system32>Write-Host "Exporting items between $startDate and $endDate..."
Exporting items between 06/01/2013 00:00:00 and 06/06/2013 00:00:00...

Comment: Did you ever come up with a resolution for this? I am experiencing the same issue.
Answer: I confirm I have solved this by specifying the content filter exactly as "(Received -ge '$startDate') -and (Received -lt '$endDate')" to prevent it from becoming $null with help from this question.
My client's Exchange Server 2013 version/build no. is 15.0.847.32.
It could be a rare condition, still I added my answer here just in case it helps someone.
Edit: Based on the concept, I have written a PowerShell Script to archive Journal Mailbox monthly with email reporting.
Answer: Yes its a known bug
The workaround is to change the mailboxes date format to US format export, do the backup then change back to your own regional format, and run the script from an account using US regional settings.
ie
Get-mailbox -Server $Server -resultsize unlimited | Set-MailboxRegionalConfiguration -Language 1033 -DateFormat "M/d/yyyy"

Messy.
Comment: Cool, that sounds like it might fix the problem. Is it possible to set the regional settings for the current account to US, just for the current session? Or do I have to have a new user that has these settings?
Comment: @Justin, do you have a link with published information about that bug?


SOLVED - Question involving the normal distribution

Question: I have some variable that's normally distributed with mean A and standard deviation B (variance B^2).
I make two (independent) observations. What's the probability that these observations will deviate from each other by at least a standard deviation B?
Comment: Do you know how $X_1-X_2$ is distributed? $X_1, X_2\sim \mathcal N(A, B^2)$
Comment: @callculus No, sorry.
Answer: Let $X_i$ be IID $N(\mu, \sigma^2)$. Then $X_1-X_2$ is $N(0, 2\sigma^2)$.
Proof:
Note that, using independence,
$$\mathbb{E}(e^{t(X_1-X_2)})=\mathbb{E}(e^{tX_1})\mathbb{E}(e^{-tX_2})$$
$$=e^{\mu t+0.5\sigma^2 t^2}\cdot e^{-\mu t +0.5\sigma^2 t^2}=e^{\sigma^2 t^2},$$
which is the MGF of a $N(0,2\sigma^2)$ RV.
Thus, $P(X_1-X_2>\sigma)=P(Z>\tfrac{1}{\sqrt{2}})\approx 0.23975$. By the way, linear combinations of independent normal RVs are very useful. Highly suggest googling a bit deeper or reviewing your textbook, etc.
Answer: The difference is normally distributed as well, with $\mu_{x_1-x_2}=\mu_{x_1}-\mu_{x_2}=\mu_y=0$ and $\sigma^2_{x_1-x_2}=\sigma_{x_1}^2+\sigma_{x_2}^2=\sigma_y^2=2B^2$.
Therefore $X_1-X_2=Y\sim \mathcal N(0, 2B^2)$. Now we calculate the probability that the deviation of the absolute value of $Y$ is more than $B$. For this purpose we can standardize Y:
$Z=\frac{Y-\mu_y}{\sigma_y}=\frac{Y-0}{\sqrt 2\cdot B}$
Since $Z$ is symmetric around $0$ we have $P(|Z|>B)=2-2\Phi(B)$.
Thus $P(|Z|>B)=2-2\Phi\left(\frac{B}{\sqrt 2\cdot B} \right)= 2-2\Phi\left(\frac{1}{\sqrt 2} \right)=2-2\Phi\left(0.707107\right)$
Using a calculator for thes (standard) normal distribution we get $\Phi\left(0.707107\right)=0.76$
$P(|Z|>B)=2-2\cdot 0.76=2-1.52=0.48$


Postfix omits realm (@domain) when checking for smtpd pam+mysql authentication

Question: I've trying to configure Postfix so users can send email through SMTP having a username and password. And I'm trying to store credential within a MySQL database. I've taken the path of saslauthd and pam and I'm stuck in the midway.
Enabling the pam debug feature I can see that when I try to send an email using Thunderbird the query executed to retrieve the password is:
pam_mysql - SELECT password FROM mailbox WHERE username = 'mehran'

It's in the case that username holds the value of firstname.lastname@example.com. But when I try:
# testsaslauthd -u email@example.com -p 123 -s smtp
0: OK "Success."

And the log reads:
pam_mysql - SELECT password FROM mailbox WHERE username = 'firstname.lastname@example.com'

Showing that it's Postfix who's omitting the @domain off the username before handing it to saslauthd (or at least I believe so). I've searched up and down Postfix's configuration but I don't see what affects this!?
Answer: I found the problem, so here it is in case it would help anyone:
There's a configuration file for saslauthd, mine is located in /etc/sysconfig/saslauthd:
# Directory in which to place saslauthd's listening socket, pid file, and so
# on.  This directory must already exist.
SOCKETDIR=/var/run/saslauthd

# Mechanism to use when checking passwords.  Run "saslauthd -v" to get a list
# of which mechanism your installation was compiled with the ablity to use.
MECH=pam

# Options sent to the saslauthd. If the MECH is other than "pam" uncomment the next line.
# DAEMONOPTS=--user saslauth
# OPTIONS="-c -r -m /var/spool/postfix/var/run/saslauthd"
OPTIONS="-c -m /var/run/saslauthd"

# Additional flags to pass to saslauthd on the command line.  See saslauthd(8)
# for the list of accepted flags.
FLAGS=

The OPTIONS variable is supposed to contain the options passed to saslauthd. It's missing -r which does what exactly I was looking for. According to its documentation:

-r
  Combine the realm with the login (with an '@' sign in between).
  e.g.  login: "foo" realm: "bar" will get passed as login:
  "foo@bar".  Note that the realm will still be passed, which may lead to unexpected behavior.

But my problem was not solved just by this change! It turned out that due to some bug in the /etc/init.d/saslauthd script, even though the configuration file mentioned above is loaded, but it is not applied! The $OPTIONS was never used!!!
Here's the original start section of /etc/init.d/saslauthd script:
start() {
    [ -x $path ] || exit 5
        echo -n $"Starting $prog: "
        daemon $DAEMONOPTS $path -m $SOCKETDIR -a $MECH $FLAGS
        RETVAL=$?
        echo
    [ $RETVAL -eq 0 ] && touch $lockfile
        return $RETVAL
}

And this is how I amended it:
start() {
    [ -x $path ] || exit 5
        echo -n $"Starting $prog: "
        daemon $DAEMONOPTS $path $OPTIONS -a $MECH $FLAGS
        RETVAL=$?
        echo
    [ $RETVAL -eq 0 ] && touch $lockfile
        return $RETVAL
}

All done! Now the query contains the complete email address to check for the password.
Answer: Your query contains the wrong/incomplete parameter for the query.
This query will append the domain to the local part of the login:
SELECT password FROM mailbox WHERE username = '%u@%r'

Correct the query in your SASL configuration for postfix, e.g. /etc/postfix/sasl/smtpd.conf
Comment: Could you please include a complete list of what should I have in that file? Right now all I've got is: `pwcheck_method: saslauthd`, `mech_list: login plain`, `log_level: 7`
Comment: If you're unable to find the location of that wrong error yourself, you should most probably not run a mailserver facing the public internet! You need to find the file which contains the wrong query definition. As my guess is wrong, it must be elsewhere. You can use `grep` to do that.


Left hand position for high fretting (electric guitar)

Question: I'm self-taught and I want to check that I'm not doing myself a disservice with my left hand positioning when I'm playing high up the neck (say, above fret 15). I play with my thumb behind the neck, fingers angled 'backwards' down the neck (from my perspective looking down, they point to my left - just a little). This has served me very well thus far I think, and means my pinky doesn't flail around and all fingers stay fairly close to the fretboard.
However, I've been working on a particular fiddly passage I wrote and it's proving very difficult to nail at the intended tempo. I was researching to check my technique and have actually been unable to find a single example article, video, photo, etc of anyone who didn't put their thumb over the top of the neck and angle fingers 'upwards' (to the right if looking down).
While I can see how in theory reach might be increased by this, it feels far less natural to me, breaks the wisdom I've read before that thumb should be behind the neck, and naturally positions the 2nd, 3rd and 4th fingers away from the fretboard some distance.
Basically, do I have anything to worry about? :)
EDIT: Now with video for your comment/critique/enjoyment/bemusement (first half - how I play currently; second half - initial attempts at what I see others do):

Also, this particular passage has been my nemesis for some time.  Any tips for getting it cleaner, smoother, etc - do let me know.  Or an analysis of why it's so difficult? :)

(At the same time, I have some specific questions around my technique and how I practise. Really looking for some like-minded (and more experienced than me!) folks how I can bounce some ideas off, get confirmation on some suspicions, that kind of thing! Is this site that place, or is there a forum, different site, youtube channel or something else that anyone knows of?  I can find all of those things for beginners, maybe some lessons hinting on intermediate/advanced, but oddly the resources seem to cut off at about that point.)
Comment: I'm also learning guitar and struggling a bit with hand position. Adjusting the angle of the guitar helps a bit – a higher neck makes higher frets more accessible – but I haven't really worked it all out for myself.
Comment: @all, Playing around I just found something really interesting!  While using my current technique feels better usually, trying to trill with my pinky, even completely relaxed, gets difficult very quick.  With the thumb-over-top method, the trill is easier!  As such I think I will re-train.. I'm fairly sure this trill effort is what's causing my pain point at the moment!
Answer: It is, of course, possible to play the guitar with a variety of techniques and hand positions. Although some are generally accepted as being more orthodox than others, you can always find examples of good players who use less orthodox techniques.
Often, keeping the thumb behind the neck while playing is considered to be a more 'technically-correct' way of playing, certainly for classical guitar, but there are real benefits for quite a lot of electric technique too. It's interesting that you say you adopt this technique; from experience, it seems that self-taught guitarists often don't. (I certainly played with my thumb over the top of the neck until I took guitar lessons…) Having said this, once you get to the very high frets this can actually become more of a hindrance than a benefit. Let me suggest a few reasons for this:

Having the fingers point 'to-the-right' a little while playing at the top of the neck allows you to reach further up the neck than the position your hand and thumb are actually in.
If your guitar has quite an obstructive shape to the body, where it joins the neck (I play a Telecaster usually so it really gets in the way!), you need to be able to reach your fingers further than your hand and thumb can reach (as they have to stop at the point where the neck joins the body).
If you are performing techniques such as pull-offs, hammer-ons and bends on the top few strings on these high frets, having your thumb above the neck can help you to get more of a grip on the guitar, making these techniques easier to execute.

It is also worth noting that you are likely to be playing the top strings far more than the bottom strings when up this high on the neck (otherwise you would usually play on a higher string further down the neck!) For this reason, you don't need to be able to reach the bottom strings as easily. And, you probably won't be playing chords as much up there, so having flatter fingers is fine, too. (Ordinarily, these are problems when playing with the thumb above the neck.)
You certainly don't need to worry about your technique though. I would say that the technique you have developed is probably fairly sound. However, you might just want to adapt it slightly when playing very high up the neck. Incidentally, when you play with your fingers pointing slightly to the right, your 2nd, 3rd and 4th fingers don't need to be further away from the neck. Instead, they are just bent differently, with the 1st more bent than the others, and the 4th quite straight - I think of this as being like a "flute-grip". It is also similar to how one positions the fingers when playing the violin.
EDIT: @Kieren Johnstone I just watched your video. I think your technique is pretty good and doesn't need to be changed drastically. To be honest, my advice above is what most guitarists would do when playing that high up the neck because it can be difficult to play with the more orthodox "thumb-behind-the-neck" technique. You manage to do that fine so stick with it! There are a couple of reasons that your first technique seems to work for you: you play with your guitar neck angled up quite a lot (see Bradd's comment…); the kind of guitar you are playing allows you good access to the high frets. Again, to be perfectly honest, you don't seem to be struggling with that passage that much - if you feel you are, you probably just need to keep practising it, especially below tempo and gradually increasing tempo. I can't see that your playing technique is stopping you playing that passage.
However, having had another couple of views of your vid, if I was to be super critical, you could try to keep your left-hand little finger a bit closer to the neck, and this would help you play more smoothly. Lots of scale practice can help with this; especially chromatic scales...
Comment: Excellent feedback, thank you.  I will experiment some more tomorrow.  Going to leave the question unanswered for a wee bit to see what others can come up with!
Comment: No probs! There are quite a few Music.SE users with good guitar knowledge, so you should get plenty of good advice.
Comment: If it makes a difference or inspires comment, I made a quick video just now - https://www.youtube.com/watch?v=4lQwR8POWlo ?  The more typical method (second half) feels less natural, do I have it right and need to do slow, deliberate practise on that would you say?  Thanks again
Comment: Thanks again! That's where I feel I need the most work too.  To get really in the groove for playing the section well, I focus on super light touch etc, and the little finger behaves better. I came straight home and recorded that with no warm up, and that's usually what happens when I do that (any extra tension flails out of that little finger). Overall, I am reassured (thank you), although one niggle: I *do* need to angle the 'normal' way to reach, see 3sec in ... for one stretchy bit of the passage.  Don't know if it's worse to be inconsistent like that.  Note: I do obsess over this I know..
Answer: While the 'correct' method historically is to have the thumb right in the back of the neck, it may not matter for the style of music you play. I think Bob's answer says almost everything I would write here.
For years I played with it there, and still do when playing classical or flamenco, but for most of my current gigging, I find myself moving more to using the thumb as an extra fretting point (I use a 7 string with my current band, and being able to fret the low A string is quite useful). I never have my fingers leaning to the left, or the nut end of the guitar though - always either straight up or towards the bridge.
At high frets though, I almost never wrap my thumb round, as despite having really deep cutaways on most of my guitars it would still get blocked somewhere between the 15th and 19th frets, so my thumb moves away from the neck the higher I get. Occasionally I will actually have my thumb on the block where the neck joins the body - with my fingers either straight up or slightly pointing bridge-wards.
If I am playing at the 23rd fret or higher my thumb is actually alongside my first finger and my entire hand is in front of the guitar with my fingers heavily angled towards the bridge, almost lying along the strings rather than across them. This allows me to play rapidly on most strings, but especially on my top 3, which are the important ones at the high frets.
Comment: Thank you, that makes sense - my hand does come around a bit and very high up, my thumb naturally comes around in the way you describe too.  If it makes any difference, my fingers are pointing pretty much straight up vertically, it's just with the guitar angled 'up', it makes they are pointing headstockwards rather than bodywards, if you see what I mean!  I'll experiment some more and see what I can work out
Answer: It looks like the tilt of your guitar is one of the factors here. That is : the head sits higher than the body.
It appears that you're able to play (well!) with your fingers pointing to the left and this looks like it's because the neck of your guitar is tilted upwards quite a lot.
My fingers (like many guitarist I guess?) point to the right when I'm playing higher up but generally my guitar is less tilted, almost horizontal. I'm not sure there's a "right" way but playing with your guitar more horizontally would probably mean your fingers would naturally end up pointing more to the right on the neck ?
There is an advantage to this stance (as others have said) in that it's probably easier to reach the higher frests, because you're kind of pointing at the fretted note rather than reaching across it, and of course your hand won't foul on the body of the guitar so much.
But then "Easier" is subjective and means "you might find it easy", which is a bit self defeating given you're clearly doing damn fine as you are :-)
Maybe it's worth experimenting with the tilt of your guitar?  Might work, or maybe it's not worth the change.
Sometimes when playing a solo I pick the guitar up so the strap goes loose and hold it quite close to my head. Seems to help somehow, probably a similar kind of thing.
Comment: Since everybody has mention all other things, you might try keeping your guitar strap a little shorter so that your left doesn't have that extreme curled up position. I'd keep it up to just below shoulder level, then it would benefit your left hand's awkward angle..
Comment: Yeah I agree - might feel really weird at first I guess but could help once you get used to it.


Which of the followings are correct:

Question: Which of the followings are correct:

I know that the set in D is closed and hence D is false.
Comment: @SritiMallick : by seeing two users giving same hint at a time, i think you should see how concrete this example is :)
Comment: if you know that D is false, i hope you can clearly see what C would be...
Comment: @PraphullaKoushik: Right you are. It's a straight line and so can be proved similarly to be closed.
Comment: For A, I suggest looking at how $\Bbb Q$ and $\Bbb R$ are related to one another.
Comment: good. keep going.. try for A,

$\mathbb{Q}$ is dense in $\mathbb{R}$. What about its compliment.
Comment: For (B), write down the definition of nowhere dense, and reword it in terms of $X\setminus \overline{A}$. Now note that $X\setminus A$ contains $X\setminus\overline{A}$
Comment: $\Bbb Q\subset \Bbb R$ is the standard textbook example of a dense set with dense complement, because by the time you get to introductory topology, you are most likely very familiar with it. It helps with the intuition of what "dense" means. Then a year or so later comes the Zarisky topology, and that intuition more or less has to be built up from scratch again.
Answer: Counterexample to (A): $X=\mathbb R$, $A=\mathbb Q$
(B): Let $U$ be any nonempty open set. $U\not\subseteq A$ because $A$ is nowhere dense. Therefore $U\cap (X\setminus A)\ne \emptyset$, i.e. $X\setminus A$ is dense.
(C): $\mathbb R\times \{0\}$ is a closed proper subset, hence not dense.
(D): The diagonal is a closed (because $\mathbb R$ is Hausdorff) proper subset, hence not dense.

Which of the followings are correct:


Save PHP form to a separate XML per submission

Question: I have been searching here for a solution and found the script below to save a form submission as XML. It appends the form data to an existing XML file.
I need to be able save each form submission as a separate XML file. The name of the file can just be a timestamp. I changed the script a little because instead of $_REQUEST it was using $_POST.
<?php

// Script by Fred Fletcher, Canada.

$name = $_REQUEST['name'];
$email = $_REQUEST['email'];
$message = $_REQUEST['message'];

$xml = new DOMDocument('1.0', 'utf-8');
$xml->formatOutput = true;
$xml->preserveWhiteSpace = false;
$xml->load('file.xml');

$element = $xml->getElementsByTagName('Lead')->item(0);

$timestamp = $element->getElementsByTagName('timestamp')->item(0);
$name = $element->getElementsByTagName('Contact')->item(0);
$email = $element->getElementsByTagName('email')->item(0);
$message = $element->getElementsByTagName('message')->item(0);

$newItem = $xml->createElement('Lead');

$newItem->appendChild($xml->createElement('timestamp', date("F j, Y, g:i a",time())));;

$newItem->appendChild($xml->createElement('Contact', $_REQUEST['name']));
$newItem->appendChild($xml->createElement('email', $_REQUEST['email']));
$newItem->appendChild($xml->createElement('message', $_REQUEST['message']));

$xml->getElementsByTagName('entries')->item(0)->appendChild($newItem);

$xml->save('file.xml');

echo "Data has been written.";

?>

Answer: Try this solution, it save the data to the file which name include timestamp and microtime (to prevent file rewrite if to requests come in same second:
$xml = new DOMDocument('1.0', 'utf-8');

$lead = $xml->createElement('lead');
$lead->appendChild($xml->createElement('timestamp', date('d.m.Y H:i:s')));
$lead->appendChild($xml->createElement('name', $_REQUEST['name']));
$lead->appendChild($xml->createElement('email', $_REQUEST['email']));
$lead->appendChild($xml->createElement('message', $_REQUEST['message']));

$xml->appendChild($lead);
$xml->formatOutput = true;
$xml->save(microtime(true).'.xml');


Problema ao retornar os models do Entity: "The entity or complex type ... cannot be constructed in a LINQ to Entities query"

Question: Estou tentando retornar uma lista dos objetos gerados pelo DataBase First do Entity Framework mas me aparece esse erro:

The entity or complex type 'leaosites04Model.TB_LEMBRETES' cannot be constructed in a LINQ to Entities query.

Estou tentando de alguma forma não criar uma classe para cada tipo diferente de consulta que faço com colunas diferentes no EF...
O método está assim:
public List<TB_LEMBRETES> getLembretes(int? situacao)
{
    try
    {
        using (dbEmpEntities db = new dbEmpEntities())
        {
            List<TB_LEMBRETES> result =
                (from l in db.TB_LEMBRETES
                 join c in db.TB_CLIENTE on l.TB_CLIENTE.id_cliente equals c.id_cliente
                 where l.situacao == 0
                 select new TB_LEMBRETES
                 {
                    dt_lembrete = l.dt_lembrete,
                    obs = l.obs,
                    TB_CLIENTE = new TB_CLIENTE() { id_cliente = c.id_cliente, nome = c.nome }
                 }).ToList();

            return result;
        }
    }
    catch (Exception ex)
    {
        throw new Exception(ex.Message);
    }
}

TB_LEMBRETE e o TB_CLIENTE são models, dentro do Model1.edmx (dentro do Model1.tt), que foram gerados pelo Entity no momento do Database First.
Já vi na net em alguns locais que isso não é possível, mas não entendi porque!
Segue abaixo os models:
[Table("leaosites04.TB_LEMBRETES")]
public partial class TB_LEMBRETES
{
    [Key]
    public int id_lembrete { get; set; }

    public int? id_cliente { get; set; }

    public DateTime? dt_lembrete { get; set; }

    [Column(TypeName = "text")]
    [StringLength(65535)]
    public string obs { get; set; }

    public virtual TB_CLIENTE TB_CLIENTE { get; set; }
}

[Table("leaosites04.TB_CLIENTE")]
public partial class TB_CLIENTE
{
 public TB_CLIENTE()
 {
     TB_LEMBRETES = new HashSet<TB_LEMBRETES>();
     TB_TELEFONES = new HashSet<TB_TELEFONES>();
 }

 [Key]
 public int id_cliente { get; set; }

 [Required]
 [StringLength(100)]
 public string nome { get; set; }

 public virtual ICollection<TB_LEMBRETES> TB_LEMBRETES { get; set; }

}

Answer: A solução sugerida pelo Harry Potter, é realmente a mais prática e mais sugerida, porém, complementando a forma que começou a fazer, deveria ficar algo assim:
public List<TB_LEMBRETES> getLembretes(int? situacao)
{
    try
    {
        using (dbEmpEntities db = new dbEmpEntities())
        {
            var resultTipoAnonimo =
                (from l in db.TB_LEMBRETES
                 join c in db.TB_CLIENTE on l.TB_CLIENTE.id_cliente equals c.id_cliente
                 where l.situacao == 0
                 select new
                 {
                    dt_lembrete = l.dt_lembrete,
                    obs = l.obs,
                    id_cliente = c.id_cliente,
                    nome = c.nome
                 }).ToList();

            List<TB_LEMBRETES> result =
                resultTipoAnonimo.Select(r => new TB_LEMBRETES()
                {
                    dt_lembrete = r.dt_lembrete,
                    obs = r.obs,
                    TB_CLIENTE = new TB_CLIENTE() { id_cliente = r.id_cliente, nome = r.nome }
                }).ToList();

            return result;
        }
    }
    catch (Exception ex)
    {
        throw new Exception(ex.Message);
    }
}

Dessa forma, você recupera uma lista de tipo anônimo e converte depois para o seu tipo desejado.
Acredito que somente em casos onde se queira uma quantidade mínima de colunas dentre muitas existentes, ou então em caso de ter coluna com grande volume de dados na mesma tabela, como um blob, por exemplo, é que seria recomendado usar dessa última forma, assim pouparia de buscar por um volume tão maior de dados que nem se quer faria uso.
Comment: Muito obrigado Luiz, isso me ajudou muito! Resolveu meu problema =D
Eu não tinha pensado em dar um novo Select pq isso gera um novo processo pro sistema...
Ou esse select possui algum tipo de performance que não demora mto?
Me preocupo quando eu tiver muuuitos dados, são praticamente um select demorado no banco e mais outro passando pelos mesmos dados
Comment: Reynaldo, claro que esse novo select terá algum custo, mas muito longe de ser algo crítico, pois os dados estarão todos em memória já, apenas criará uma lista de objetos com dados já disponíveis, não fará uma nova pesquisa em banco.

Que isso @HarryPotter, estamos aqui pra somar!
Comment: @Luiz perfeito, obrigado pelos créditos, são poucos aqui que reconhecem.


Convert UNIX time (INT) to timestamp in BigQuery

Question: I have a column "date_time" in a BigQuery table which contains unix timestamp values like "1569888224". The problem is that these values are integer data types, not timestamp data types, so I do not seem to have an easy way to convert them to human readable date/times. Does anyone have a good way of converting these integers into datetime values in BigQuery?
Thanks!
Answer: This can be solved by using BigQuery's TIMESTAMP_SECONDS function - https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#format_timestamp
select TIMESTAMP_SECONDS(date_time) from table;

Comment: Correct link for the function is https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#timestamp_seconds
Comment: It depends on the source data.  Epoch time from Cisco required TIMESTAMP_MILLIS not TIMESTAMP_SECONDS
Answer: A shortcut to anyone who is trying to convert a Firebase event_timestamp in BigQuery:
SELECT
TIMESTAMP_SECONDS(CAST(CAST(event_timestamp as INT64)/1000000 as INT64)) as converted,
event_timestamp,
event_date
FROM [table] LIMIT 10

Comment: an even better one is

```
SELECT
TIMESTAMP_MICROS(event_timestamp) as converted,
event_timestamp,
event_date
FROM [table] LIMIT 10```
Answer: Depend on your column values, but in general int64 values consist of 16 digits such as;
- event_timestamp
- user_first_touch_timestamp
- etc...

you should use below functions;
TIMESTAMP_MICROS(event_timestamp) event_datetime
TIMESTAMP_MICROS(user_first_touch_timestamp) user_first_touch

if it doesn't consist of 16 digits, you can try the other functions which one is suitable for you;
TIMESTAMP_SECONDS
TIMESTAMP_MILLIS

Answer: Use the timestamp of your preference; here I have used IST(Indian Standard Time):
SELECT
  TIMESTAMP_SECONDS(Column_Name) AS unix_timestamp,
  CONCAT(FORMAT_TIMESTAMP('%Y-%m-%d %H:%M:%S', TIMESTAMP_SECONDS(Column_Name), 'Asia/Kolkata'), ' IST') AS IST_timestamp
FROM
  `bigquery-public-data.google_analytics_Table_Name*`


How to implement a cron job for every minute and every 5 minutes?

Question: In D8 I programmed hook_cron. I tested it several times but I cannot see any log in the watchlog.
The name of the module is nncbt. The code is placed in nncbt.module.
Any suggestions what I should do?
<code>function nncbt_cron() {

    $last_run_per_minut = \Drupal::state()->get('nncbt.last_run', 0);
    // If 1 minutes passed since last time.
    if ((REQUEST_TIME - $last_run_per_minut) > 60) {

        \Drupal::logger('nncbt')->info('Runnning Cron every 1 minut log');

        // Update last run.
        \Drupal::state()->set('nncbt.last_run', REQUEST_TIME);

    }

    // Run the following every 5 minutes
    $last_run_per_5_minuts = \Drupal::state()->get('nncbt.last_run_5_minuts', 0);

    // If 5 minutes passed since last time.
    if ((REQUEST_TIME - $last_run_per_5_minuts) > 300) {

        \Drupal::logger('nncbt')->info("Runnning Cron every 5 minut log");

        // Update last run.
        \Drupal::state()->set('nncbt.last_run_5_minuts', REQUEST_TIME);

    }

}
</code>
Comment: How are you running cron?
Comment: Inside, by clicking admin menu> run cron. Also tried the button "Run Cron" on url admin/config/system/cron.
Comment: Did you rebuild the caches after adding your hook? Is cron completing? Check the status page, cron last run X ago.
Comment: If I run the function nncbt_cron() throught developers, execute php code everything is working fine.
Comment: Then it sounds like drupal doesn't know about your hook. Rebuild the caches.
Comment: Rebuild caches already 10 times but no luck in that :-(
Comment: You most likely mean `$_SERVER['REQUEST_TIME']`
Answer: hook_cron()

The engine will then call the hook whenever a cron run happens, as
  defined by the administrator.

In other words, hook_cron gets executed after a cron run happens. So if you have cron set to run once a day, then your hook_cron script only gets executed once per day. So you need to set cron to run every 5 minutes or less.
A better alternative, I suggest you use a module like Ultimate   Cron which allows you to schedule different cron run times for different cron tasks. This way, cron doesn't run every single task every 5 minutes and instead it runs your specific task only.
Comment: Totally, and to confirm that Ultimate cron is the solution to granular control over Drupal's cron (even though its 8 version is in alpha), its interface and options are a bit daunting but it really gives you control.
Comment: Even if Drupal's cron runs every 5 minutes there is little additional overhead as tasks are (or should be using state API) scheduled independently of cron running. Having cron run more regularly is desirable; if it only runs once a day then ALL cron tasks are going to run at once, adding to overhead. (assuming you don't have badly written cron tasks that is)
Comment: @Duncanmoo which is why I mentioned about using [Ultimate Cron](https://www.drupal.org/project/ultimate_cron) in my answer, so you can run specific cron tasks and different times.


Django Form Does Not Find Right ID int Increment on Production vs LH

Question: For some reason on Localhost (LH) everything works fine but on my Production server, my form does not add a new user submission properly. The error I am getting is:
duplicate key value violates unique constraint "..."
DETAIL:  Key (id)=(8) already exists.

Is there some sort of production "sudo systemctl restart gunicorn" I need to run (I have already tried the example above)? Maybe it's only working on LH because there I have tested there more and the increment naturally fell on the same level as the total amount of users? I really am out of ideas here.
models.py
class Lead(models.Model):
    username = models.CharField(max_length=15, blank=True, null=True)
    email = models.CharField(unique=True, max_length=150, validators=[validate_email])
    created = models.DateTimeField(auto_now_add=True)
    ...

forms.py
class LeadCaptureForm1(forms.ModelForm):
    birth_date = forms.DateField(widget=SelectDateWidget(years=range(1999, 1910, -1)))

    class Meta:
        model = Lead
        widgets = {
            'email': forms.TextInput(attrs={'class': 'form-control'}),
        }
        fields = ('email', 'birth_date',)

views.py
def iframe1(request):
    ip = get_real_ip(request)
    created = timezone.now()

    if request.method == 'POST':

        form = LeadCaptureForm1(request.POST)
        if form.is_valid():

            # Save lead
            lead = form.save()
            # attempt at fixing it
            #lead.id = Lead.objects.get(all).count()
            #print(lead.id)
            lead.created = created
            lead.birth_date = form.cleaned_data.get('birth_date')
            lead.ipaddress = get_real_ip(request)
            lead.joinmethod = "Iframe1"
            lead.save()

            print(lead)

Answer: I'm not sure why you are setting the ID manually, and especially why you are setting it to the count of items. You should always let the database manage the primary key itself - it is an autoincrement field, and is completely opaque to your data.
The reason why you are getting this conflict is that items can be deleted, so that there can be 8 entries in the database but ID 8 already exists. But as I say, don't do this at all.
Also, don't set created manually, as that will be done automatically as well because you have auto_now_add=True in the model field.. And birth_date is set by the form save already. Finally, you should call save with commit=False if you want to set some other fields manually.
So just do:
lead = form.save(commit=False)
lead.ipaddress = get_real_ip(request)
lead.joinmethod = "Iframe1"
lead.save()

Comment: Hey thanks for taking a look. Noticed the code above and just commented out that section because it was an attempt at fixing. So I added the "(commit=False)" and still having the same issue on production. Weird that it works on LH
Answer: This is due to the fact that we uploaded leads manually and Django uses PostgreSQL's SERIAL data type to store auto-incrementing primary keys.
"A SERIAL column is populated with values from a sequence that keeps track of the next available value. Manually assigning a value to an auto-incrementing field doesn't update the field's sequence, which might later cause a conflict."
https://docs.djangoproject.com/en/2.0/ref/databases/#manually-specifying-values-of-auto-incrementing-primary-keys
To solve this we can either force a new serial number or build an exception to fix the serial. The latter option would be the ideal since we may upload users manually in the future. However, for now, we'll try and force serial number.
Run code: python manage.py sqlsequencereset [app_name]

For some reason, this did NOT work, so I was about to try and figure out how to build some sort of "python if exemption" but instead found this post first (IntegrityError duplicate key value violates unique constraint - django/postgres) that helped me directly update the "setval":
SELECT setval('tablename_id_seq', (SELECT MAX(id) FROM tablename)+1)


For vector spaces $V,W$ over $\mathbf{k}$, is every additive $\phi: V \to W$ also $\mathbf{k}$-linear?

Question: Since every map of vector spaces is a map of abelian groups, I was wondering if the converse also holds:
Given an additive map $\phi: V \to W$ between two vector spaces, does it follow that $\phi$ is also $\mathbf{k}-$linear? I'm interested in the case of $\mathbf{k}$ having characteristic zero, specially if $\mathbf{k}$ is a famous field like rational or real or complex numbers.
I'm guessing it's false, but I tried to come up with counter-examples for $\mathbf{k} = \mathbf{Q}, \mathbf{R}$ and couldn't find any. Finding a counter-example in characteristic $p>0$ may not be that hard, for instance since taking $p-$th power is additive. However that's not the case I care most about. Appreciate any help!
Answer: This is true only for $k\cong\mathbb{Q}$.  In the case $k=\mathbb{Q}$, suppose $\phi:V\to W$ is a homomorphism of abelian groups between two vector spaces, $v\in V$ and $\frac{a}{b}\in\mathbb{Q}$ (with $a,b\in\mathbb{Z}$).  Note that then $$a\phi(v)=\phi(av)=\phi\left(b\cdot\frac{a}{b}v\right)=b\phi\left(\frac{a}{b}v\right),$$ so multiplying by $\frac{1}{b}$ we find that $\phi(\frac{a}{b}v)=\frac{a}{b}\phi(v)$ so $\phi$ is linear.
On the other hand, if $k$ is not isomorphic to $\mathbb{Q}$, it is a nontrivial field extension of $\mathbb{Q}$.  In particular, $k$ can be considered as a $\mathbb{Q}$-vector space of dimension greater than $1$.  Picking a basis, there is then a $\mathbb{Q}$-linear map $\phi:k\to k$ which is the identity on all but one basis vector but maps one basis vector to $0$.  This $\phi$ cannot be $k$-linear, since any $k$-linear map $k\to k$ is either $0$ or injective.
(If you consider fields of arbitrary characteristic, similar arguments show that every abelian group homomorphism of $k$-vector spaces is $k$-linear iff $k$ is a prime field.)
Comment: thank you very much for that solid answer!
Answer: Consider the case of conjugation $f=(-)^*: \mathbb{C} \to \mathbb{C}$, where $\mathbb{C}$ is a 1-dimensional vector space over itself (and the scaling is ordinary complex multiplication, of course).  This is an additive homomorphism since $(z+w)^* =z^* + w^*$.  However, $f(cz) = (cz)^* = c^*z^*$ is not necessarily equal to $cf(z)=cz^*$, so it is not a $\mathbb{C}-$vector space map. (Of course they will be the same for all $z$ only when $c$ is real.)
Comment: thank you, an important, simple counterexample!


different p-values from resampling and traditional methods

Question: I'm not sure if this is hard to answer without the original data, but let's give it a go.
I'm calculating the difference in proportions for two diagnostic methods:
      outcome
test     0  1
method1 22 30
method2 18 39

Using R and prop.test(c(30, 39), c(52, 57)) gives me a p = 0.3361.
Using R and the infer package "Two categorical (2 level) variables" gives me p = 0.250.
Is such a big difference to be expected when using permutation, or am I doing something wrong?
Comment: If these are the same cases assessed by the different methods, don't you need a method for paired samples?
Comment: What exactly are you doing and why?
Comment: I'm not sure I understand what you're asking. You mean how and why I'm using that R package? Or the purpose of the comparison?
Comment: How did you get that table? What are those values and methods? What is the purpose of prop.test and what are you doing with the infer package?
Comment: Alright :) The table is from R `table(mydata)`. The methods are two different biopsy methods for diagnosing cancer, so the outcomes are 0 = no cancer, 1 = cancer. The purpose of prop.test is to test the null hypothesis that there is no difference in detection of cancer with the two methods. I'm testing the same hypothesis using infer with the method described under "Two categorical (2 level) variables" - comparing the observed difference in proportions to the null distribution of 1000 permuted samples.
Comment: If I understand your table correctly you should be comparing the methods, that is `prop.test(c(22,30),c(18,39))`.
Comment: Thats what I think I'm doing with `prop.test(x = c(30, 39), n = (52, 57))`. `prop.test(c(22,30),c(18,39))`throws an error `elements of 'x' must not be greater than those of 'n'`
Comment: Never mind, it's correct. So the question is why the two tests give different p-values? The difference doesn't seem that big, considering that one is simulated. Either case you retain your null hypothesis.
Comment: You need to give us more details about the study design as it looks on the face of it that @Björn is correct here and you are doing the wrong thing.
Comment: They are not the same cases, 52 got method1 and 57 got method2. However, the group of 52 ALSO got method2, and I'm planning to use McNemars test in that group comparison to account for paired/repeated measures.
Answer: The difference was due to Yates continuity correction. It's set to default = TRUE in prop.test().
prop.test(c(30, 39), c(52, 57), correct = FALSE) yields a more similar p-value.


Accommodating entrenched views of p-values

Question: Sometimes in reports I include a disclaimer about the p-values and other inferential statistics I've provided.  I say that since the sample wasn't random, then such statistics would not strictly apply.  My specific wording is usually given in a footnote:

"While, strictly speaking, inferential
  statistics are only applicable in the
  context of random sampling, we follow
  convention in reporting significance
  levels and/or confidence intervals as
  convenient yardsticks even for
  nonrandom samples. See Michael Oakes's
  Statistical inference: A commentary for the social and behavioural
  sciences  (NY: Wiley, 1986).

On a couple of occasions--once for a peer-reviewed paper, once or twice in a non-academic setting--the editor or reviewer objected to this disclaimer, calling it confusing, and felt that the inferential findings should simply stand as written (and be given the mantle of authority).  Has anyone else encountered this problem and found a good solution?  On the one hand, people's understanding of p-values is generally dismal, even in the context of random sampling, so perhaps it doesn't matter much what we say.  On the other, to contribute further to misunderstandings seems to make one part of the problem. I should add that I frequently deal with survey studies, where random assignment does not apply and where Monte Carlo simulations would often fail to address the issue of representativeness.
Comment: Correct me if I'm wrong, but the randomness of sampling simply affects the degree to which you can generalize findings. In contrast, random assignment is the more critical feature for causal inference.
Comment: the comment on a reviewer is extremely sad, one would hope that a person in that position would at least not openly display their ignorance, and by doing so, further support the misinterpretation of the statistical method.
Comment: Mike, I agree with you.  Do you make this point to extend the discussion or to indicate disagreement with something I've said?
Comment: @richiemorrisroe: one would be foolish to expect that of all reviewers, but I suppose one can hope for a future in which we can expect that, and we should certainly pressure publishers to do more to demand and enforce that than they do presently...Rolando, I think Mike's is merely a point of clarification to disambiguate this discussion from causal-inference-related issues.  Evidently some people have found that helpful, though I thought it was clear enough already, personally.  If I'm right, this inadvertently measures others' confusion about _p_ values, which motivates the original post!
Answer: The use of inferential statistics can be justified not only based on a population model, but also based on a randomization model. The latter does not make any assumptions about the way the sample has been obtained. In fact, Fisher was the one that suggested that the randomization model should be the basis for statistical inference (as opposed to Neyman and Pearson). See, for example:
Ernst, M. D. (2004). Permutation methods: A basis for exact inference. Statistical Science, 19, 676-685. [link (open access)]
Ludbrook, J. & Dudley, H. (1998). Why permutation tests are superior to t and F tests in biomedical research. American Statistician, 52, 127-132. [link (if you have JSTOR access)]
I somehow doubt though that the editors or reviewers in question were using this as the reason for calling your disclaimer "confusing".
Comment: If the primary goal is to make some kind of inference to the population and the sampling mechanism is of such a nature that the representativeness of the sample is questionable, then indeed, any inference will also be rather questionable. Essentially, you can only make an inference to that part of the population that the sampling mechanism provides a representation of. In principle, the inferences you make will be appropriate for that part of the population. Whether that part of the population is of any interest to you (or the readers) is another issue.
Comment: Wolfgang - interesting and helpful points.  I should have made clear, though, that much of my work is on surveys.
Answer: There is indeed an argument to be had not to include the disclaimer. Frankly, I'd find a brief treatise on the nature of p-values in a journal article to be a little off-putting, and for a moment would have to pause and try to figure out if you'd done something particularly...esoteric...to warrant devoting that space to a definitional point.
Basically, as a reviewer, I'd call it unnecessary because the reader should already know what a p-value is and does. I might even object to it because making such a note does not actually prevent any of the many crimes of analysis and interpretation that accompany p-values, it merely puts on a cloak of "trust me, I know what I'm doing". It's also a little odd - "I'm going to make a bold stand against p-values, but not so bold I don't report them".
When I consider "entrenched views on p-values", I'm much less concerned about something like what you posted above, and much more concerned about reviewers' insistence on statistical significance in order to be published or the focus of the paper (put a star by a finding and suddenly its a Big Deal) or blending statistical significance with the significance of a finding.
Comment: I don't think this answers the OP. I'm assuming @rolando2 reports other stats that are more central to his discussions (e.g., effect sizes), and mostly reports _p_ values as one way of accommodating conventional expectations, even though they don't apply strictly. As such, we should be off-put to whatever extent we're reading too much into _p_ values; we should consider his motivation for the disclaimer. Readers don't know what they should; the OP mentions this. The disclaimer promotes doubt, not trust. It's not that odd to object to a standard while conforming to it; it's not a bold stand.
Comment: The OP's question: "Has anyone else encountered this problem and found a good solution?" Your answer ignores the literal question to respond to the idea, and mostly offers your opinions on why the idea should be shot down. You're starting to hint at a constructive critique of the OP's idea though: you don't seem to think the Oakes citation tells you why it matters. I'll expand on this a little in an answer of my own.
Comment: @NickStauner I don't see how it doesn't "answer" the OP. Perhaps it doesn't support what they want to do, but in my mind it's both a really weird break from the actual content of the paper, and also useless - "This is wrong, but I'm going to soldier on as if its right because it's what you all expect" doesn't tell me if the wrongness *matters*.
Answer: I haven't had to do battle with any bad reviewers yet, so I wouldn't claim any knowledge of how to get out of a battle that's already begun. However, if their objections are a mere matter of obstructive ignorance, a little preemptive diversion might do the trick. If $p$ values are in fact necessary to report despite their non-negligible invalidity in a problematic study (a class into which all too many published articles fall), one might downplay them implicitly. Consider focusing your narrative instead—maybe even exclusively—on effect sizes. If your study is sufficiently representative to be usefully informative (this shouldn't necessitate perfectly random sampling, only caution in the generality of interpretations), your effect sizes ought to have broader implications than merely indicating the existence and directions of relationships or differences anyway. Focusing one's discussion on effect sizes can facilitate a deeper understanding of how much the relationships or differences matter in a practical sense, though this still needs to be considered in the context of the subject of study (e.g., one cannot conclude by size alone that an $r = .03$ is necessarily unimportant if it might pertain to a matter of life and death; Rosenthal, Rubin, & Rosnow, 2000). You can do this by discussing results in terms of "weak," "moderate," or "strong" relationships or "small" or "large" differences instead of referring to them as "significant" and "insignificant"; the latter two words shouldn't be necessary whatsoever to make most of the points researchers want to make. If the $p$ values are necessary, let them speak for themselves. Do meta-analysts a favor and just sandwich them in more comprehensive reports of valuable statistics: effect sizes, confidence intervals, and test statistics. Maybe hope for a day when readers and reviewers will ignore $p$ values and demand confidence intervals, so that the $p$ values can be ditched entirely.  (Or maybe not!  See post-postscript!)
Another, potentially complementary option would be to expand on your footnote. Both your descriptions of the problem as reviewers have experienced it, and the presently accepted answer on this page, suggest that not enough information is conveyed to explain your motivation for including the footnote, nor enough to motivate the reader to follow your citation to the reference that you use to explain it so tersely. A single, additional sentence, even a brief quote from your reference, could go a long way toward explaining the value of your footnote and motivating readers to read deeper.  Evidently, your footnote as is sooner motivates a simple, negative, dismissive reaction toward your understated attempt to disrupt their complacency about their improper assumptions. Readers might be a little less intellectually lazy if you spoonfeed them one or two of the main points about problems that they probably overlook routinely. Also, for many particular problems with $p$ values, consider citing not just that book, but also a fairly concise journal article that's freely available online presently (e.g., Goodman, 2008, Wagenmakers, 2007). That might help reduce any resistance due to the difficulty of obtaining a book and finding the relevant info within.
P.S. Thanks to @rpierce for Wagenmakers (2007) and much of the logic of my answer, and to @FranciscoArceo for Goodman (2008)! See also Francisco's loosely related answer, as well as some other popular posts here on Cross Validated about interpreting $p$ values properly:

What is the meaning of p values and t values in statistical tests?
Understanding p-value

P.P.S. @MichaelLew's counterpoint is also worth considering before tossing the $p$ values out entirely! See Senn (2001) and Lew (2013) for some rare and valuable (but only partial) defenses of $p$. [Edit]: Also, I brought up this question in a new question, "Why are 0.05 < p < 0.95 results called false positives?" In discussing my answer, the OP brought up Hurlbert and Lombardi (2009), which I brought up with my colleagues, one of whom then brought up Nuzzo (2014), a brand new Nature News article that led to even more references (Goodman, 2001, 1992; Gorroochurn, Hodge, Heiman, Durner, & Greenberg, 2007)...I am obviously not keeping up at this point, but Michael is just as clearly not alone in defending the possibility of extracting useful information from exact $p$ values (when they do "strictly apply", at least).
References

- Goodman, S. N. (1992). A comment on replication, P‐values and evidence. Statistics in Medicine, 11(7), 875–879.
- Goodman, S. N. (2001). Of P-values and Bayes: A modest proposal. Epidemiology, 12(3), 295–297. Retrieved from http://swfsc.noaa.gov/uploadedFiles/Divisions/PRD/Programs/ETP_Cetacean_Assessment/Of_P_Values_and_Bayes__A_Modest_Proposal.6.pdf.
- Goodman, S. (2008). A dirty dozen: Twelve P-value misconceptions. Seminars in Hematology, 45(3), 135–140. Retrieved from http://xa.yimg.com/kq/groups/18751725/636586767/name/twelve+P+value+misconceptions.pdf.
- Gorroochurn, P., Hodge, S. E., Heiman, G. A., Durner, M., & Greenberg, D. A. (2007). Non-replication of association studies: "pseudo-failures" to replicate? Genetics in Medicine, 9(6), 325–331. Retrieved from http://www.nature.com/gim/journal/v9/n6/full/gim200755a.html.
- Hurlbert, S. H., & Lombardi, C. M. (2009). Final collapse of the Neyman–Pearson decision theoretic framework and rise of the neoFisherian. Annales Zoologici Fennici, 46(5), 311–349. Retrieved from http://xa.yimg.com/kq/groups/1542294/508917937/name/HurlbertLombardi2009AZF.pdf.
- Lew, M. J. (2013). To P or not to P: On the evidential nature of P-values and their place in scientific inference. arXiv:1311.0081 [stat.ME]. Retrieved from http://arxiv.org/abs/1311.0081.
- Nuzzo, R. (2014, February 12). Scientific method: Statistical errors. Nature News, 506(7487). Retrieved from http://www.nature.com/news/scientific-method-statistical-errors-1.14700.
- Rosenthal, R., Rosnow, R. L., & Rubin, D. B. (2000). Contrasts and effect sizes in behavioral research: A correlational approach. Cambridge University Press.
- Senn, S. (2001). Two cheers for P-values? Journal of Epidemiology and Biostatistics, 6(2), 193–204. Retrieved from http://www.phil.vt.edu/dmayo/conference_2010/Senn%20Two%20Cheers%20Paper.pdf.
- Wagenmakers, E. J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14(5), 779–804. Retrieved from http://www.brainlife.org/reprint/2007/Wagenmakers_EJ071000.pdf.
Comment: Not all of the criticisms of P-values are correct or warranted, despite their vehemence. You should see these two papers for a few counterpoints to the commentaries that you cite: Two Cheers for P Values (by Stephen Senn) http://www.phil.vt.edu/dmayo/conference_2010/Senn%20Two%20Cheers%20Paper.pdf; To P or Not To P (by me) http://arxiv.org/abs/1311.0081
Comment: Do you have any experimental evidence in favor of your claim that Edwards' assumption extends to p-values?  I find my self extremely skeptical.  My way of analogy, I've seen a couple papers that demonstrate that even experienced data scientists have trouble estimating a correlation coefficient from a scatterplot.  It seems like you are asking a lot more from scientists in gaining a sense of what a p value means in terms of likelihood.  Your argument in favor of likelihood functions is interesting... they do tend to look a bit like posterior distributions, no?
Comment: @rpierce I don't have experimental evidence for the understanding of users of statistical methods. I would, however, contend that at least some of the studies that have been done to see if scientists 'understand' p-values are fatally flawed by not including among the options a true evidentially meaningful description of the p-value. Your analogy is not close because the fact that correlation coefficients are not easily estimated is not the same problem as estimating the strength of evidence from a p-value.
Comment: Excellent point! Thank you! I've edited slightly to include your contributions, and I may edit a bit more once I understand them well enough to incorporate their implications into the rest of what I've said. This is why I love Cross Validated...
Comment: @rpierce The posterior probability density function from a uniform prior will be proportional to the likelihood function.
Comment: @rpierce I only ask that scientists understand that the evidence from a smaller significance test p-value is stronger than the evidence from a larger p-value, and that the evidence from a p-value from a larger sample size points to a smaller region than that from the same p-value with a larger sample. I would contend that scientists already understand that. (Note that I do not contend that a p-value is an adequate description of the results without a clear display of the data!)
Comment: It is useful to hear that the posterior probability density function from a uniform prior will be proportional to the likelihood function.  Although it is telling that what is required for that to be true is essentially an uninformative prior... and one of the big stumbling blocks for many in accepting Bayesian analyses is an allergy to priors.
Comment: Nevertheless, the extent to which scientists are all to often not aware of the effect sizes in their experiments and treat failures to reject as evidence for the null under conditions of say p > .1 p < .15 is prima facie evidence that their level of understanding is vastly insufficient; i.e., they are not able to psychologically represent anything like a likelihood function given the p value (or for many of them, even if given N and the effect size).
Comment: You may hold what you may - but I'd encourage you to consider the pragmatic consequences of your argument.  What do you think the the take-away from your article for scientists who are at greatest risk in terms of their inability to interpret their statistical results (e.g., those who pay little head to effect sizes) will be?

I fear that the readership most in danger will boil it down to "...the conclusion is drawn that P-values are useful indices of experimental evidence. The widespread use of P-values in scientific research is well justified by the actual properties of P-values...".
Comment: Ha!  Just got an unexplained downvote...Entrenched indeed!
Comment: @MichaelLew It very nice to see a detailed description of the relationship between the p-value and the underlying likelihood function.  Nevertheless, I think your argument is problematic on pragmatic grounds.
Comment: The analogy is apt to the extent to which it seems counter-intuitive that scientists would have a problem with such a simple issue.
Comment: I would grant that some, certainly far from all, scientists using statistics have at least a directional sense of the consequences to the state of a posteriori knowledge subsequent to a hypothesis test.  This sense almost certainly drives a lot of the failures to reproduce a technically correct description of the p-value.


SEM: switching order of indicators in latent variable definition changes results

Question: I want to do a path analysis with lavaan but encounter a few problems and would appreciate any help.
The structural model looks like this:

The relation between one observed independent (s) and one observed dependent variable (v) is mediated through a latent variable (m) that is defined by two observed indicator variables (x1, x2). This is basically a simplified version of the SEM example in the tutorial on the lavaan project website.
When I enter my code (given further below) into R, I encounter two problems:
(1) The results change when I change the order of the indicator variables.
This model:
# measurement model
    m =~ x1 + x2

returns a different result than this model:
# measurement model
    m =~ x2 + x1

How can that be? Isn't the order of the indicators arbitrary? And if not, how do I know which is the correct order, if my model does not presuppose a specific order?
(2) There are a few warnings that I don't understand: for the first model, no standard errors could be computed; and the second model did not "converge" (whatever that means). The warnings are given in context in the full code posted below.
What do I have to do to obtain reliable estimates?

Here is the full R output to provide context to my questions.
# data

s <- c(2, 5, 4, 4, 4, 8, 2, 9, 1, 1, 3, 3, 2, 3, 2, 5, 5, 7, 4, 7, 8, 4, 10, 10, 2, 4, 0, 2, 4, NA, 1, 5, 2, 6, 3, 5, 0, 5, 3, 6, 4, 9, 4, 9, 4, 5, 6, 1, 8, 0, 6, 9, 1, 5, 1, 6, 2, 5, 0, 5, 6, 2, 4, 10, 3, 4)
v <- c(8, 10, 1, 4, 0, 2, 3, 2, 1, 1, 2, 5, 1, 5, 0, 5, 4, 5, 2, 10, 0, 6, 5, 5, 6, 1, 1, 0, 0, NA, 1, 0, 1, 8, 1, 3, 0, 5, 6, 3, 2, 10, 0, 5, 5, 10, 4, 1, 1, 0, 0, 0, 2, 10, 1, 8, 2, 3, 2, 2, 4, 4, 2, 5, 6, 2)
x1 <- c(2.500000, 3.789474, 1.514563, 5.846868, 4.588235, 5.600000, 5.066667, 11.647059, 2.000000, NA, 4.461538, 18.000000, 1.058824, 9.217391, 27.840000, 15.375000, NA, 6.000000, 9.714286, 12.484848, 16.503497, 20.666667, 3.500000, 4.658824, 4.750000, 4.000000, 2.800000, 14.228571, 11.000000, NA, 2.666667, 3.764706, 4.705882, 13.272727, 2.000000, 18.444444, 17.555556, 14.222222, 2.000000, 4.000000, 8.461538, 19.200000, 13.902439, 13.000000, 3.000000, NA, 7.360000, 1.611374, 1.500000, 3.365854, 22.375000, 10.838710, 2.923077, 3.488372, 5.176471, 37.666667, 1.176471, 7.454545, 36.235294, 6.823529, 2.222222, 6.133333, 11.428571, 42.705882, 28.105263, 18.333333)
x2 <- c(8.125000, 14.273684, 7.339806, 23.387471, 113.058824, 22.200000, 17.466667, 43.647059, 9.230769, NA, 13.538462, 83.555556, 5.058824, 37.391304, 100.000000, 59.250000, NA, 22.470588, 38.428571, 50.787879, 76.223776, 92.888889, 15.375000, 16.235294, 18.875000, 13.647059, 10.133333, 55.885714, 36.428571, NA, 6.933333, 13.294118, 14.117647, 81.818182, 6.117647, 67.777778, 76.333333, 51.888889, 6.428571, 14.200000, 34.000000, 59.680000, 68.634146, 40.500000, 12.250000, NA, 29.760000, 8.909953, 5.400000, NA, 71.125000, 39.741935, 9.846154, 13.116279, 18.823529, 204.000000, 4.588235, 49.090909, 188.470588, 19.647059, 10.222222, 22.933333, 38.285714, 140.235294, 137.526316, 79.000000)
dat <- data.frame(cbind(s, v, x1, x2))

# first model

model <- '
    # measurement model
        m =~ x2 + x1
    # regressions
        m ~ s
        v ~ s + m
    # residual correlations
        x1 ~~ x2
'
fit <- sem(model, data = dat, missing = "fiml")

# Warning messages:
# 1: In lav_data_full(data = data, group = group, group.label = group.label,  :
#   lavaan WARNING: some cases are empty and will be removed:
#   30
# 2: In lav_model_vcov(lavmodel = lavmodel, lavsamplestats = lavsamplestats,  :
#   lavaan WARNING: could not compute standard errors!
#   lavaan NOTE: this may be a symptom that the model is not identified.

summary(fit, fit.measures=TRUE, standardized=TRUE, rsquare=TRUE)

# lavaan (0.5-18) converged normally after 147 iterations
#
#                                                   Used       Total
#   Number of observations                            65          66
#
#   Number of missing patterns                         3
#
#   Estimator                                         ML
#   Minimum Function Test Statistic                0.565
#   Degrees of freedom                                 0
#   Minimum Function Value               0.0043451960201
#
# Model test baseline model:
#
#   Minimum Function Test Statistic              126.904
#   Degrees of freedom                                 6
#   P-value                                        0.000
#
# User model versus baseline model:
#
#   Comparative Fit Index (CFI)                    0.995
#   Tucker-Lewis Index (TLI)                       1.000
#
# Loglikelihood and Information Criteria:
#
#   Loglikelihood user model (H0)               -797.558
#   Loglikelihood unrestricted model (H1)       -797.275
#
#   Number of free parameters                         12
#   Akaike (AIC)                                1619.115
#   Bayesian (BIC)                              1645.208
#   Sample-size adjusted Bayesian (BIC)         1607.435
#
# Root Mean Square Error of Approximation:
#
#   RMSEA                                          0.000
#   90 Percent Confidence Interval          0.000  0.000
#   P-value RMSEA <= 0.05                          1.000
#
# Standardized Root Mean Square Residual:
#
#   SRMR                                           0.027
#
# Parameter estimates:
#
#   Information                                 Observed
#   Standard Errors                             Standard
#
#                    Estimate  Std.err  Z-value  P(>|z|)   Std.lv  Std.all
# Latent variables:
#   m =~
#     x2                1.000                              14.272    0.330
#     x1                0.384                               5.482    0.588
#
# Regressions:
#   m ~
#     s                 1.732                               0.121    0.323
#   v ~
#     s                 0.335                               0.335    0.306
#     m                 0.012                               0.171    0.059
#
# Covariances:
#   x2 ~~
#     x1              292.112                             292.112    0.951
#
# Intercepts:
#     x2               35.558                              35.558    0.823
#     x1                7.220                               7.220    0.775
#     v                 1.761                               1.761    0.604
#     m                 0.000                               0.000    0.000
#
# Variances:
#     x2             1663.119                            1663.119    0.891
#     x1               56.783                              56.783    0.654
#     v                 7.591                               7.591    0.892
#     m               182.367                               0.895    0.895
#
# R-Square:
#
#     x2                0.109
#     x1                0.346
#     v                 0.108
#     m                 0.105

model <- '
    # measurement model
        m =~ x1 + x2
    # regressions
        m ~ s
        v ~ s + m
    # residual correlations
        x1 ~~ x2
'
fit <- sem(model, data = dat, missing = "fiml")

# Warning messages:
# 1: In lav_data_full(data = data, group = group, group.label = group.label,  :
#   lavaan WARNING: some cases are empty and will be removed:
#   30
# 2: In lavaan::lavaan(model = model, data = dat, missing = "fiml", model.type = "sem",  :
#   lavaan WARNING: model has NOT converged!

summary(fit, fit.measures=TRUE, standardized=TRUE, rsquare=TRUE)

# ** WARNING ** lavaan (0.5-18) did NOT converge after 9438 iterations
# ** WARNING ** Estimates below are most likely unreliable
#
#                                                   Used       Total
#   Number of observations                            65          66
#
#   Number of missing patterns                         3
#
#   Estimator                                         ML
#   Minimum Function Test Statistic                   NA
#   Degrees of freedom                                NA
#   P-value                                           NA
#
# Parameter estimates:
#
#   Information                                 Observed
#   Standard Errors                             Standard
#
#                    Estimate  Std.err  Z-value  P(>|z|)   Std.lv  Std.all
# Latent variables:
#   m =~
#     x1                1.000                               0.526    0.056
#     x2             1606.326                             845.326   19.343
#
# Regressions:
#   m ~
#     s                -0.001                              -0.001   -0.004
#   v ~
#     s                 0.355                               0.355    0.325
#     m                 0.004                               0.002    0.001
#
# Covariances:
#   x1 ~~
#     x2              -69.375                             -69.375   -0.009
#
# Intercepts:
#     x1               10.099                              10.099    1.083
#     x2               48.281                              48.281    1.105
#     v                 1.761                               1.761    0.604
#     m                 0.000                               0.000    0.000
#
# Variances:
#     x1               86.614                              86.614    0.997
#     x2            -712666.446                            -712666.446 -373.157
#     v                 7.617                               7.617    0.895
#     m                 0.277                               1.000    1.000
#
# R-Square:
#
#     x1                0.003
#     x2                   NA
#     v                 0.105
#     m                 0.000
# Warning message:
# In .local(object, ...) :
#   lavaan WARNING: fit measures not available if model did not converge

Note. I have posted the same question to the lavaan Google Group, but this is part of my bachelor's thesis, which I have to turn in on Monday, so I'm a bit pressed for time and hope you forgive me for crossposting.
Answer: Your model is not identified. That means that Lavaan cannot find a unique solution, and cannot compute standard errors. Because it can't find a unique solution, it's finishing in different places.
[Very simple example of identification:
$x + y = 5$
There are many (an infinite number) of solutions to this equation for x and y, and they're all equally good, so it's not identified.]
This has happened for a couple of reasons.
First:
        x1 ~~ x2

Is not going to work. You need the latent variable to explain the covariance of these two items. You can't have a covariance between the two variables, because there's nothing left for the latent variable to explain. So let's get rid of that. Oh, still problematic.
Second, your sample size is small. Nothing we can do about that.
Third, the variances of your variables are quite different. This causes convergence problems for most SEM programs. A rule of thumb is to make sure your SDs are less than an order of magnitude different. Let's check:
> apply(dat, 2, na.rm=TRUE, sd)
    s         v        x1        x2
 2.686326  2.940680  9.397064 43.761456

OK, let's fix that.
dat$x1 <- dat$x1 / 10
dat$x2 <- dat$x2 / 10

Hmmmm... still doesn't work.
Using FIML probably doesn't make life any easier for Lavaan. Let's turn that off.
Nope, still not working.
Now we're getting a bit desperate. We need some additional restrictions in the model. You have two indicators of a latent variable - that's not many. Let's fix the loadings to be equal, that makes it easier to converge.
That worked, we have convergence. But we also have a negative error variance on x1 (a Heywood case). Is it just a little bit negative? If so, we'll ignore it. Nope, it's not. It's big.  OK, we need to fix that.
Again, because we're desperate, we're going to fix the error variance of x1 and x2 to be equal.
x1 ~~ a*x1
x2 ~~ a*x2

That gives us convergence, but horrible fit.  What about fixing it to zero?
x2 ~~ 0*x2

Nope, still horrible fit. Let's go back a step and see if we can work out what's going on (which is worth starting with). Let's look at the correlations.
> cor(dat, use="pairwise.complete.obs")
           s         v        x1        x2
s  1.0000000 0.3246869 0.1811305 0.0808834
v  0.3246869 1.0000000 0.1479880 0.1192254
x1 0.1811305 0.1479880 1.0000000 0.9206230
x2 0.0808834 0.1192254 0.9206230 1.0000000

OK, x1 and x2 are very highly correlated, so they are going to have similar loadings on M, and if they have similar loadings on M, they need to have similar relationships with the other variables (because M is explaining their relationship with those other variables).
The correlation of S with x1 is about twice as high as the correlation of s with s2. This is the crux of the problem. Somehow we need to account for that. The two paths are:
s -> m -> x1
s -> m -> x2
The first path needs to get a value about twice as high as the second. It can't do it from the s->m path, because there's only one of that. So to account for the correlation difference it wants to have different loadings. But it can't have different loadings, because the correlation is so high between x1 and x2.
You have (as I see it) three possible solutions.

Drop one of x1 or x2. Then you no longer have a latent variable. (But you hardly have a latent variable anyway, with only two indicators.
Add a correlation between either x1 or x2 with S.  That probably doesn't make theoretical sense.
Lose the latent, and have x1 and x2 as mediators of the relationship between s and v directly, so your model is:
v ~ s + x1 + x2
x1 ~ s
x2 ~ s
x1 ~~ x2

Which is a regular multiple mediator model.
Comment: Holy cow, thank you!!! I owe you a crate of beer for that, @JeremyMiles (or whatever you like to bathe in). Two questions: (1) Shouldn't it be `x1 ~~ x2`? Because `x1 ~ x2` imposes a unidirectionality that is not there (theoretically). Think of x1 and x2 being the time to lift the left and right hand respectively, when being told to lift both hands. They sure correlate, and one does not cause the other. And (2), does the order of `x1 + x2` matter in any way?
Comment: Oops, yes, x1 ~~ x2.
Comment: Order shouldn't matter, it will change estimates, etc, but not the standardized estimates, and not the p-values or interpretation. Because the model wasn't identified, it did matter.  (And yes, that went on for longer than I expected. ;)
Comment: I didn't expect x1 and x2 to correlate as much as they do (the hand lifting was just an example, not what I really measured, the study is about tooth grinding), and I feel I cannot just drop one variable post hoc.
Comment: Why do you think the hands correlate differently with s?


Newly Created toolbar icon is not Showing in the Toolbar

Question: I want to add a new icon to the Notepad++ toolbar.
I've tried following code
internal static void SetToolBarIcon()
{
    toolbarIcons tbIcons = new toolbarIcons();
    tbIcons.hToolbarBmp = tbBmp.GetHbitmap();
    IntPtr pTbIcons = Marshal.AllocHGlobal(Marshal.SizeOf(tbIcons));
    Marshal.StructureToPtr(tbIcons, pTbIcons, false);
    Win32.SendMessage(PluginBase.nppData._nppHandle, NppMsg.NPPM_ADDTOOLBARICON, PluginBase._funcItems.Items[idMyDlg]._cmdID, pTbIcons);
    Marshal.FreeHGlobal(pTbIcons);
}

Now my problem is that instead of an icon I only see a black square in the toolbar
Comment: any solution found for this issue ? i am facing the same issue now, its running fine on win10 but showing a square black box on win 7 enterprise
Comment: I have the same problem with Notepad++ 6.9.1 on Windows Server 2008 R2 running in a VMWare.
Strange is that the icon shows fine on Windows 7 Enterprise 64bit that is not running in a VMWare with the same Npp version!
I have no idea if this is related to VMWare or not
Answer: Remember to save your icon in the Resources.resx