Dúvida funcionalidade Regex - Java

Question: Alguém poderia me explicar o que esse Regex permite?
private static final String MAIL_PATTERN = "^[_A-Za-z0-9-\\+]+(\\.[_A-Za-z0-9-]+)*@[A-Za-z0-9-]+(\\.[A-Za-z0-9]+)*(\\.[A-Za-z]{2,})$";

Comment: Alguns sites como [Regexr](https://regexr.com/) e [Regex101](https://regex101.com/) explicam todos os componentes de uma expressão regular e ainda permitem que você as teste online
Comment: Talvez sua pergunta esteja muito ampla. Sugiro dar uma olhada em tutoriais como [este](https://www.regular-expressions.info/tutorial.html) e [este](http://rexegg.com/). Em seguida você pode **[edit]** sua pergunta para que ela fique mais específica sobre algum ponto em particular no qual você ficou em dúvida.
Comment: Repare que eu escrevi "*Talvez*". Ainda estou em dúvida, pois inicialmente pensei que precisaria escrever um tutorial de regex para responder adequadamente (e por isso ela é considerada ampla, não pq está perguntando várias coisas, mas pq a resposta ficaria longa demais). Mas talvez dê para fazer uma resposta não tão longa, ainda vou ver isso mais tarde (se ninguém responder antes, claro). De qualquer forma, se vc puder ler os links que eu e o Leonardo colocamos, talvez vc consiga entender o básico e focar suas dúvidas em pontos mais específicos. Isso ajuda a deixá-la "menos ampla"
Comment: Com certeza, fico muito agradecido pela força @hkotsubo! Abraços grande.
Comment: @hkotsubo minha pergunta está ampla? Eu quero saber o que esse Regex acima faz.
Comment: No fim, achei que valia a pena postar uma resposta. Mesmo tendo ficado bem grande, está bem focada na sua regex e no seu uso para validar emails (não ficou um tutorial de regex, até pq vc está usando poucos recursos da sintaxe), e no fundo o que importa [não é o tamanho da resposta, e sim o foco](https://pt.meta.stackoverflow.com/q/4801/112052), então acho que me equivoquei ao consider a pergunta "ampla demais". Divirta-se e bem vindo ao estranho/maravilhoso mundo das expressões regulares :-)
Comment: Apenas complementando, [nesta outra resposta](https://pt.stackoverflow.com/a/348859/112052) coloquei umas expressões (bem complicadas) para validar emails - não testei com Java, somente com JavaScript, mas não deve ser "difícil" adaptá-las, eu acho :-)
Answer: Basicamente, esta regex verifica se uma String corresponde a um endereço de email. Mas ela também pode acabar aceitando coisas "estranhas" (mais sobre isso no final).
Analisando a regex em detalhes:
^[_A-Za-z0-9-\\+]+(\\.[_A-Za-z0-9-]+)*@[A-Za-z0-9-]+(\\.[A-Za-z0-9]+)*(\\.[A-Za-z]{2,})$

^ e $ são marcadores que significam, respectivamente, o início e fim da string. Com isso você garante que a string só terá o que está na regex (sem esses marcadores, é possível verificar se somente parte da string corresponde à regex).
Os colchetes ([]) definem uma classe de caracteres: eles servem para indicar que você quer qualquer caractere que esteja dentro deles. Por exemplo, [abc] significa "a letra a ou a letra b ou a letra c" (apenas uma delas, qualquer uma serve).
Só que dentro dos colchetes você pode usar alguns atalhos, como A-Z, que significa "letras de A a Z" (maiúsculas). De modo similar, a-z significa "letras minúsculas de a a z" e 0-9 significa "dígitos de 0 a 9".
Ou seja, [_A-Za-z0-9-\\+] significa "o caractere _ ou letras (maiúsculas ou minúsculas) ou dígitos ou hífen (-) ou o sinal de mais (+)". As duas barras invertidas seriam para escapar o +, mas creio que dentro dos colchetes não seja necessário. Nos meus testes não fez diferença, mas de qualquer modo, deixei assim mesmo (a sintaxe de regex diz que apenas um \ é usado para escape, mas como estamos dentro de uma String, precisamos colocar \\).
O detalhe é que "qualquer coisa dentro de colchetes" é uma expressão que corresponde a apenas um caractere. Se você quiser mais ocorrências, deve usar os quantificadores, e foi isso que fizemos colocando um + depois dos colchetes:
[_A-Za-z0-9-\\+]+
                ^

Fora dos colchetes, o + significa "uma ou mais ocorrências" do que está imediatamente antes dele. No caso, é uma ou mais ocorrências de "caractere _ ou letras (maiúsculas ou minúsculas) ou dígitos ou hífen (-) ou o sinal de mais (+)". Ou seja, strings como abc, A124_fadfd-a12 e até mesmo +_a são consideradas válidas por esta expressão.

Em seguida temos (\\.[_A-Za-z0-9-]+)*. Vamos começar com o que está dentro dos parênteses:

\\.: corresponde ao caractere ponto (.). Está escapado com barra invertida porque o ponto tem significado especial em regex (significa "qualquer caractere"). Usando a barra invertida, ele "perde seus poderes" e vira um caractere comum
[_A-Za-z0-9-]+: similar ao caso anterior, é uma ou mais ocorrências de _, ou letras, ou números, ou hífen

Juntas, essas 2 partes acima são "um ponto, seguido de uma ou mais letras/números/hífens/underlines".
Só que tudo isso está entre parênteses, e logo depois tem um *. Isso quer dizer "zero ou mais ocorrências do que está dentro dos parênteses". Ou seja, essa sequência "um ponto, seguido de uma ou mais letras/números/hífens/underlines" pode ocorrer várias vezes (ou nenhuma). Podemos ter strings como .abc.cde123.fgh ou simplesmente não ter nada.
Juntando isso com a expressão anterior, temos a primeira parte do email (antes do @), que pode ser desde um username comum (como joaosilva) até coisas mais complicadas como 32teste.abd2-cdef12_4232.xyz.afd.
Em seguida, temos um @, que corresponde ao próprio caractere "arroba".

Agora temos a segunda parte do email, depois do @.
Primeiro temos [A-Za-z0-9-]+ (uma ou mais letras/números/hífens).
Depois temos (\\.[A-Za-z0-9]+)*, cuja lógica já vimos. Neste caso, significa "zero ou mais repetições de (ponto seguido de letras/números)". Isso serve para domínios como algum.nome.comprido.com.br: os trechos .nome, .comprido e .com correspondem a esta expressão.
Por fim, temos (\\.[A-Za-z]{2,}). O quantificador {2,} significa "duas ou mais ocorrências". Portanto, esta expressão significa "ponto, seguido de 2 ou mais letras". Com isso, o domínio do email não pode terminar com .a ou .b, por exemplo, é preciso ter pelo menos duas letras.
Juntando as 3 expressões acima, temos o domínio do endereço de email, que pode ser desde gmail.com até endereços com vários sub-domínios, como a333-bcd.abc.co.uk.

Sobre a validação de emails
Infelizmente, validar emails não é tão simples quanto parece. Há regras demais (cada parte do email tem um limite de tamanho, o domínio também pode ser um endereço IP, etc), e quanto mais precisa é a regex, mais complicada ela fica - veja estes exemplos, só para ter uma ideia. E quanto mais simples (ou menos complicada), maior a chance de falsos positivos.
A regex em questão, por exemplo, aceita emails como -----email@example.com e abc@-----abc.com (veja aqui ela funcionando). Isso acontece porque usamos o hífen dentro dos colchetes e com quantificadores (uma ou mais ocorrências), por isso uma String com vários hífens é considerada válida.
Da mesma forma, no primeiro par de colchetes há um caractere +, e estes colchetes tem um quantificador + (uma ou mais ocorrências), então a regex considera que +++++@abc.com é um email válido (veja aqui). Tecnicamente, segundo a RFC 5322, acredito que até seja, mas cabe a você decidir se seu sistema vai aceitar tais endereços.
Outro problema é que ela não verifica os limites de tamanho de cada parte, nem aceita endereços IP no lugar do domínio (só para citar alguns casos). O fato é que, para cada regra a mais que você tenta colocar, mais difícil vai ficando: veja neste link como a regex vai ficando cada vez mais complicada a cada vez que adicionamos uma regra.
No fim, você deve achar um meio-termo adequado para os seus casos de uso. Uma regex muito complicada pode virar um problema de manutenção futura (pegue uma das expressões dos links indicados; se já é difícil entender, imagine se tiver que modificar para adicionar um caso novo ou corrigir algum bug).
Considere os prós e contras: simplificar a regex significa abrir mão da precisão em troca de facilidade de manutenção (mas você passa a ter mais falsos positivos). E deixá-la mais precisa pode levar àqueles códigos complicados que todos têm medo de mexer.
O quão ruim será aceitar alguns emails inválidos (ou "estranhos", como +++@---.com ou +_+_+@---.com, ou ainda +.-._-_@-.com, todos considerados válidos pela regex em questão)? O quão ruim será não aceitar alguns casos válidos? E o que são casos válidos? user@localhost pode ser um caso válido, dependendo da situação, mas esta regex considera que é inválido. Algumas soluções pela internet consideram que a primeira parte (antes do @) também pode ter caracteres como %, $ e ! (que a sua regex não considera).
Para não aceitar os casos "estranhos" (ou ao menos minimizá-los), você pode alterar a regex para aceitar emails que comecem somente com letras, por exemplo, colocando esta condição logo após o ^:
^[A-Za-z][_A-Za-z0-9-\\+]+.... (o resto é igual)
 ^^^^^^^^

Com isso, o primeiro caractere deve ser uma letra, e o restante pode ser "uma ou mais letras/números/hífens/underlines/sinal de mais" etc. A regex ainda vai aceitar firstname.lastname@example.com, mas você pode adicionar uma regra que não aceite dois + seguidos, etc (percebe o padrão "quanto mais regras, mais complicado fica"?).
Cabe a você testar, analisar seus casos de uso e decidir por um caminho. Regex é uma ferramenta extremamente poderosa (e na minha opinião, muito legal), mas nem sempre é a melhor solução para tudo.
Talvez uma solução mais simples seja separar em várias expressões (faz um split no @ e verifica cada parte com uma regex diferente), o que pode ser mais fácil (de fazer, entender e manter) do que uma super-regex-monstruosa-que-faz-tudo. Outra solução é usar uma lib externa.

Só lembrando que a regex não verifica se o email existe (se há uma conta para aquele usuário naquele servidor, se a conta está ativa, se alguém lê de fato os emails enviados para este endereço, etc).
Talvez no fim das contas seja mais fácil usar uma regex mais simples (que verifique algo que **pareça** um email, sem se preocupar com as regras mais complicadas) e em seguida você envia um email com um link de confirmação/ativação, como muitos sites costumam fazer.

Alguns tutoriais úteis sobre regex:

https://www.regular-expressions.info/tutorial.html
http://rexegg.com/
Comment: Muito obrigado hkotsubo! Ajudou demais.
Comment: @AndréFilipe De nada! Se vc achar que a resposta de fato responde/soluciona a pergunta, vc tem a opção de aceitá-la: veja [**aqui**](https://pt.meta.stackoverflow.com/q/1078/112052) como e porque fazê-lo. Claro que vc não é obrigado a aceitar nenhuma resposta (apenas se vc achar que realmente respondeu à pergunta), mas isso serve para indicar a futuros visitantes que a resposta soluciona/responde ao que foi perguntado. E bons estudos com regex, pois é um assunto sem fim!
Comment: @AndréFilipe Sem problemas!
Comment: Tenho um monte de sistemas que tenho que modificar depois de ler essa resposta.
Comment: Perdão, eu até que votei na sua resposta na sexta-feira mesmo, porém, fechei o navegador rapidamente. Acredito que tenha sido por isso que não foi contabilizado!

Dúvida funcionalidade Regex - Java


rsync webapps sub-directories

Question: Can I run tomcat as ec2-user? The issue is: I need to rsync some sub-folders of webapps folder between 2 ec2 instances. The rsync works for all folders owned by "ec2-user" but for the webapps sub-folders (whose owner is 'tomcat'), rsync fails with permission denied.
Any suggestions please?
Answer: You could run rsync with sudo. ec2-user has sudo-without-a-password permissions.
i.e. sudo rsync yourdir rsync://host/dst

rsync webapps sub-directories


Compare list elements in python

Question: I have a list and want to compare if the last value is greater than the past 10 values, however, I know there is a much easier way to approach this then the code below:
list = [1,2,3,4,5,6,7,8,9,10]

if list[-1] > list[-2] and list[-1] > list[-3] and list[-1] > list[-4]:  (etc)
     print "It's bigger"

Any suggestions would be greatly appreciated!
Answer: One way is to take the maximum of the past values and compare it with the last:
>>> l = [1,2,3,4,5,6,7,8,9,10]
>>> if l[-1] > max(l[:-1]):
...     print "It's bigger"
...
It's bigger

Comment: If your list has more than 10 values and you only want to compare with the 10 that precede the last value, you can do `max(l[-11:-1])`.
Answer: We iterate through the list except the last element and we check if all these elements are smaller than the last element.
print all(num < my_list[-1] for num in my_list[:-1])

In Python, the last element can be accessed with the negative indexing. So my_list[-1] refers to the last element and my_list[-2] refers to the last but one, etc.
Also we use slicing notation to exclude the last element from the list, with my_list[:-1].
Note: The advantage of this method is that, it short circuits immediately after finding a failure. So, we don't have to process the entire list always. In the best case, it returns after processing the first element. This is very useful if the list being processed is very big.
As Ashwini Chaudhary points out in the comments, this is the optimal way to do this
from itertools import islice
print all(num < my_list[-1] for num in islice(my_list, len(my_list) - 1))

Comment: Note that if OP has a big list then you should use `islice` instead of normal slicing.

Compare list elements in python


Symfony 3 Changing a Submit label from a Form class inside of controller

Question: So I have a form that is created in it's own class:
<code><?php
// src/AppBundle/Form/Type/CoffeeShopForm.php
namespace AppBundle\Form\Type;

use Symfony\Component\Form\AbstractType;
...

class CoffeeShopType extends AbstractType
{
   public function buildForm(FormBuilderInterface $builder, array $options)
{
    $builder
        ->add('name', TextType::class)
        ...
        ->add('save', SubmitType::class, array('label' => 'Create a coffee shop'))
        ->getForm();
    ;
}

}
</code>
And then I am using it in one controller
<code>/**
 * @Route("/admin/edit/{coffee_shop_id}")
 */
public function editCoffeeShopAction(Request $request, $coffee_shop_id)
{

    $repository = $this->getDoctrine()->getRepository('AppBundle:CoffeeShop');

    $coffeeshop = $repository->find($coffee_shop_id);

    $form = $this->createForm(CoffeeShopType::class, $coffeeshop);

    $form->get('name')->setData('New name value');

    $form->handleRequest($request);

    return $this->render('AppBundle:CoffeeController:edit_coffee_shop.html.twig', array(
        'form' => $form->createView(),
    ));
}
</code>
So in the controller as you may see I am able to change the value of the name field with $form->get('name')->setData('New name value');
My question is how may I change the label of the SubmitType field - I searched for the documentation of this thing but I can't find it, and it is really helpful if I could reuse this form since I am using it for the add form and then for the edit form and basically it is possible in Zend Framework, so I think it should be possible also in Symfony
Comment: http://symfony.com/doc/current/cookbook/form/dynamic_form_modification.html
Comment: thanks a lot for your answer! I actually found this before. My question is more: is it not possible to be made just with something similar as $form->get('name')->setData() . Something like set label, or if there is a special structure of the argument in setData() for submit's label
Comment: The approaches you can use are in the answer to this question: http://stackoverflow.com/questions/22893735/symfony2-form-type-how-to-set-form-css-class-as-attribute/22896777
Answer: I have stumbled across the same question, and after extensive debugging, I can say with some confidence that what you want to do is not possible, at least in Symfony 2.8 .
However, what you can do is setting another label when rendering the form:
<code>{{ form_row(form.name, { 'label': 'Test 123'}) }}
</code>
See http://symfony.com/doc/current/reference/forms/types/text.html#label for details.
Comment: I think it's easier, faster and more maintainable to do it in Twig like this. Thanks for sharing.


for Python's plotnine, how to get % of count by total count in each facet

Question: I am trying to get a density plot to show percentage of count by the total number in each facet.
For instance, I have this density plot:
Produced by this code:
ggplot(data, aes(x = 'ratio'))

+ geom_histogram(aes(y = '..density..'),
                  binwidth = 0.5,
                  na_rm = True,
                  alpha = 0.8)

 + geom_vline(xintercept = 1, colour = 'red', linetype = 'dashed', size = 0.5)  # add a red vertical line

 + facet_grid('industry ~ state')

 + labs(x = 'ratio', y = 'density')

 + scale_x_continuous(breaks = [0,1,2], labels = ['0','1','2'])

 + theme(strip_text_y = element_text(angle = 0,         # angle text in y-fact (industry names)
                                     ha = 'left'),      # left alignment
         strip_background_y = element_text(width = 2.5),# change width of the grey box (on y)
         strip_background_x = element_text(width = 1),
         figure_size=(5, 5))

As you can see, the height of the columns do not sum to 1.
How do make sure the height of each column corresponds to the count percentage of each facet.
For instance using the facet (NSW, Construction) as an example. This is now a Count plot:

The sum in the NSW/Construction facet is 3760.
The count in the 1st, 2nd... 5th column are 350, 950, 1630,630 and 200 respectively
I want the columns to show for the:

1st column = 350/3760  = 9%
2nd column = 950/3760  = 25%%
3rd column = 1630/3760 = 43%
4th column = 630/3760  = 17%
5th column = 200/3760  = 5%

I tried using aes(y='..count../sum(..count..)'), but that gave me count over the whole population, not the population within each facet.
Please help.
Answer: use aes(y=stat(width*density)).
Got the idea from a post on R's ggplot2, which is what plotnine is based upon.
https://github.com/tidyverse/ggplot2/issues/2499


React Native Deep Linking Application Not Opening

Question: I followed all the steps from both the articles mentioned below
https://hackernoon.com/react-native-deep-linking-for-ios-and-android-d33abfba7ef3
https://medium.com/react-native-training/deep-linking-your-react-native-app-d87c39a1ad5e
After the app gets installed on my phone, I tried opening the app from the browser by giving the URL as peopleapp://people/1 format. Instead of opening the app, the browser opens Google search to search for the above.
I Used this application to open my Application by using my App Link(https://play.google.com/store/apps/details?id=com.manoj.dlt&hl=en_US) it's working.
But how to open the application from browser or from another application Using my App Link ?
Anyone has idea, how to solve this issue ?
Here is my Total AndroidManifest Code`
<code><uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.RECORD_AUDIO"/>
<uses-feature android:name="android.hardware.camera" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.VIBRATE" />
<uses-permission android:name="android.permission.ACCESS_FINE_LOCATION" />
<uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION"/>
<uses-permission android:name="android.permission.FOREGROUND_SERVICE"/>
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<application
    android:name=".MainApplication"
    android:allowBackup="false"
    android:icon="@mipmap/ic_launcher"
    android:label="@string/app_name"
    android:theme="@style/AppTheme"
    android:usesCleartextTraffic="true">
    <activity
        android:name=".MainActivity"
        android:configChanges="keyboard|keyboardHidden|orientation|screenSize"
        android:screenOrientation="portrait"
        android:label="@string/app_name"
        android:windowSoftInputMode="adjustResize"
        android:launchMode="singleTask">
        <intent-filter>
            <action android:name="android.intent.action.MAIN" />
            <category android:name="android.intent.category.LAUNCHER" />
        </intent-filter>
        <intent-filter android:label="filter_react_native">
            <action android:name="android.intent.action.VIEW" />
            <category android:name="android.intent.category.DEFAULT" />
            <category android:name="android.intent.category.BROWSABLE" />
            <data android:scheme="whizzard" android:host="article" />
        </intent-filter>
    </activity>

</application>
</code>
`
My Link is whizzard://article
Comment: Please post the code you used, not the link.
Comment: which browser have you tried the deeplink?
Comment: @RishiRaj from Chrome or from another apps Like slack,gmail.
Comment: @hongdevelop added now Manifest File too,can you please check now
Comment: @PraveenReddyVangala [read this](https://stackoverflow.com/questions/29189172/google-chrome-does-not-open-application-on-deep-link-on-android)
Answer: As per google docs:
The functionality has changed slightly in Chrome for Android, versions 25 and later. It is no longer possible to launch an Android app by setting an iframe's src attribute. For example, navigating an iframe to a URI with a custom scheme such as paulsawesomeapp:// will not work even if the user has the appropriate app installed. Instead, you should implement a user gesture to launch the app via a custom scheme, or use the "intent:".
You can read more about how intents work here. However, the tutorial you followed, for deeplinking will work on other browsers which are not chrome or chrome based.
Comment: can this be done using firebase dynamic links or not ?


Show $\int_{-1}^{1} T_{n}(x)\,dx = \frac{(-1)^{n}+1}{1-n^{2}}$

Question: $T_{n}$ being
$$T_{n}(\cos (\theta)) = \cos(n\theta)$$
Could anyone explain me why
$$\int_{-1}^{1} T_{n}(x)\,dx = \frac{(-1)^{n}+1}{1-n^{2}}$$
if $n \neq 1$ and $0$ if $n=1$?
With the integral I came as far as the integral evaluates to
$$ \frac{\cos(n+1)}{n+1}-\frac{\cos(n-1)}{n-1}$$
but from here I am not sure how to get the results with given boundaries.
Comment: I suspect that you did $x=\cos\theta$ but forgot the change of limits.
Comment: Can you [provide more context](https://math.meta.stackexchange.com/questions/9959/how-to-ask-a-good-question/9960#9960)?  In particular, remind us of the definition of $T_{n}(x)$, and show how you arrived at the result you did.
Comment: Not only more context: what in the world is $\;T_n\;$ ?
Comment: Oh sorry. That's the chebyshev polynomial of the first kind
Answer: Pretty straightforward proof.
$$\int_{-1}^1 T_n(x)\mathrm{d}x=\int_0^\pi \cos(n\theta)\sin(\theta)\mathrm{d}\theta$$
Now we use the identity
$$\cos(a)\sin(b)=\frac{1}{2}(\sin(a+b)-\sin(a-b))$$
So
$$\int_0^\pi \cos(n\theta)\sin(\theta)\mathrm{d}\theta=\frac{1}{2}\int_{0}^{\pi} \sin\big((n+1)\theta\big)-\sin\big((n-1)\theta\big)~\mathrm{d}\theta$$
$$=\frac{1}{2}\bigg[\frac{-1}{n+1}\cos\big((n+1)\theta\big)\bigg]^\pi_0-\frac{1}{2}\bigg[\frac{-1}{n-1}\cos\big((n-1)\theta\big)\bigg]^\pi_0$$
$$\frac{-1}{2(n+1)}\big((-1)^{n+1}-1\big)+\frac{1}{2(n-1)}\big((-1)^{n-1}-1\big)$$
$$=\frac{\big((-1)^{n-1}-1\big)(n+1)-\big((-1)^{n+1}-1\big)(n-1)}{2(n^2-1)}$$
$$=\frac{(-1)^{n-1}n+(-1)^{n-1}-n-1-(-1)^{n+1}n+(-1)^{n+1}+n-1}{2(n^2-1)}$$
$$=\frac{(-1)^{n-1}n-(-1)^2(-1)^{n-1}n+(-1)^{n-1}+(-1)^2(-1)^{n-1}-2}{2(n^2-1)}$$
$$=\frac{2(-1)^{n-1}-2}{2(n^2-1)}=\frac{(-1)^n+1}{1-n^2}.$$
Done.


finding KNN for larger Dataset

Question: I am trying to find the nearest neighbors  for a data set A consisting of 25000 rows, for that, I am trying to fit a dataset B to the KNN model that consists of 13 million rows, the goal is to find s 25000 rows of dataset B  which are similar to dataset A
model_knn= NearestNeighbors(n_neighbors=10, algorithm = 'kd_tree')
model_knn.fit(B)

knn_distances,knn_indices=model_knn.kneighbors(A.values, n_neighbors=10)

here when I am fitting B up to 600000 rows there is no issues
model_knn.fit(knn_test_pd[:600000])

beyond 600000 the model is not fitting, there is no error but for fitting 600000 it takes 2 sec  beyond 600000 its taking hours  and the data I'm fitting is scaled data
I tried splitting the data frame and fitting it is it a correct approach? then also the model is taking hours to fit
splited_B=np.array_split(B, 113)
model_knn= NearestNeighbors(n_neighbors=10, algorithm = 'kd_tree')
for df in splited_B:
    model_knn.fit(df)

What shall I do to fit these big data to knn? Or is there another model similar to knn which can accept large datasets?
Comment: What do you mean "*not fitting*"? Do you get any error? Please edit & update your post to clarify.
Comment: Please do not quote-format your text (edited).
Comment: there is no error but for fitting 600000 it takes 2 sec  beyond 600000 its taking hours and the data is scaled data
Answer: You can split dataset B into 600,000-row chunks, which gives you 22 datasets(respectively, 22 KNN models).
In prediction, for each row in A, find the nearest data point on each of those 22 models; this gives you 22 data points. And finally, search for the nearest point among those 22 points.
Answer: Scikit-learn will work with smaller datasets and is limited to your main memory for loading and processing data. If you want to perform machine learning on larger datasets, I recommend using Apache Spark which is designed to run on multiple nodes, beating the limits you are currently facing. spark-sklearn will help you get the job done. Another option is to use random sampling. Get a smaller sample of data to represent your full dataset and perform your operations on it. You can even divide the samples into smaller ones and run your algorithm on them and compare the results from each. See this solution for random sampling.

finding KNN for larger Dataset


Verificar Login e Senha AngularJS

Question: Tenho a seguinte função para login:
function login(username, password, callback) {
        var myResponse;
        var systemUser;

        Users.getByLogin(username)
        .then(function (response) {
            systemUser = response.data;
        });

        // Linha testada
        console.log("Senha: " + systemUser.password);
        console.log("Senha informada: " + password);

        if (typeof systemUser == "undefined") {
            myResponse = { success: false, message: "Usuário inexistente!" };
        } else if (password == systemUser.password) {
            myResponse = { success: true, user: systemUser };
        } else {
            myResponse = { success: false, message: "Usuário ou senha incorretos!" };
        }

        callback(myResponse);
    }

Normalmente ocorre o erro TypeError: Cannot read property 'password' of undefined.
Se eu declarar a variável systemUser de fora da função , observando a linha do comentário Linha testada, o valor dá undefined na primeira vez que chamo o método, mas na segunda vez passa.
Answer: Isso acontece porque o método Users.getByLogin() é assíncrono.
Seu erro ocorre porque no momento em que a linha console.log("Senha: " + systemUser.password); é executada o retorno de getByLogin ainda não ocorreu; systemUser é ainda uma variável nula.
Uma vez que o resultado seja retornado, a função especificada em .then() será executada. Modifique o seu código como sugerido abaixo:
function login(username, password, callback) {
    var myResponse;
    var systemUser;

    Users.getByLogin(username)
    .then(function (response) {
        systemUser = response.data;

        // Linha testada
        console.log("Senha: " + systemUser.password);
        console.log("Senha informada: " + password);

        if (typeof systemUser == "undefined") {
            myResponse = { success: false, message: "Usuário inexistente!" };
        } else if (password == systemUser.password) {
            myResponse = { success: true, user: systemUser };
        } else {
            myResponse = { success: false, message: "Usuário ou senha incorretos!" };
        }

        callback(myResponse);
    });
}

Comment: Esqueci mesmo desta questão da sincronização, realmente as consultas AJAX são assíncronas! Obrigado!

Verificar Login e Senha AngularJS


Excel verificação de valor repetido

Question: Como checar no Excel se o valor de uma célula está repetido, valor das linha na mesma coluna,não fiz nenhuma tentativa de comando.Gostaria de saber se existe uma formula especifica?
Comment: Sim, o excel tem um botão de remover duplicatas na barra de ferramentas de dados.
Comment: Por favor, leia o [Manual de como NÃO fazer perguntas](https://pt.meta.stackoverflow.com/q/5483/75104), [Como devemos formatar perguntas e respostas?](https://pt.meta.stackoverflow.com/questions/1084/como-devemos-formatar-perguntas-e-respostas) e [Seja mais específico na pergunta](https://pt.meta.stackoverflow.com/a/1481/75104). Portanto, evite usar o CapsLock e coloque [valores de exemplo de uma tabela, sendo mais específico](https://ozh.github.io/ascii-tables/).
Comment: Ok. Não queria remover e sim sinalizar que está repetido. Tem como? Formatação condicional?
Comment: A última vez que precisei de algo assim, acabei duplicando colunas e criando umas fórmulas. Que eu saiba, não há padrão e formatação condicional não resolve. Vamos aguardar pra ver se alguém aparece com uma resposta detalhada (mas já aviso que Excel não está entre os assuntos mais tratados aqui, pode demorar um pouco).
Answer: Você pode usar formatos condicionais:

Excel verificação de valor repetido


SUMIF with several conditions with google apps script

Question: I am trying to write a google apps script similar to SUMIF in google sheets. My goal is to sum all values in column hours for all rows which contain a particular ID (id) during a defined period (column period), which are marked as productive (productive  = 1) and the value of column unit for the particular row is different from the string stated bellow. For some reason, it does not work. I am new to javascript and can't figure out the reason why it is not working. I tried defining a sum function and then call it for each of the values in the id column (about 7000 rows).
//Productive hours
//=if($A2<> "", SUMIFS(hours,hrmid,$AC2, productive,1,period,$A2)-SUMIFS(hours,hrmid,$AC2, productive,1,period,$A2, units,"Revision Stand-by"), "")
  function sumPrdHrs (item) {
    var hoursToSum = 0;
    var summedHrs = [];
    var hours = ytdSheet.getRange(2,18, ytdSheet.getLastRow()).getValues();
    var productive = ytdSheet.getRange(2,33, ytdSheet.getLastRow()).getValues();
    var units = ytdSheet.getRange(2, 9, ytdSheet.getLastRow()).getValues();
    var id= ytdSheet.getRange(2,29, ytdSheet.getLastRow()).getValues();
    var period = ytdSheet.getRange(2,1, ytdSheet.getLastRow()).getValues();
    //var date = ytdSheet.getRange(2,1, ytdSheet.getLastRow()).getValues();

    for (var i=0; i < hours.length; i++) {
      if ((id[i][0] == item) && (productive[i][0] == 1) && (period[i][0] == period[i+1][0]) && (units[i][0] != "Revision Stand-by")) {
      hoursToSum += hours[i][0];
      summedHrs.push(hoursToSum);
      }
    }
    return sumPrdHrs;
    console.log(summedHrs);
}

function productiveHrs () {

  var id= ytdSheet.getRange(2,29, ytdSheet.getLastRow()).getValues();
  var sumHours = [];

  id.forEach(function(item,i) {

     var summedHours = sumPrdHrs (item);

    //sumHours.push(summedHours);

    //Logger.log(summedHours);

   // ytdSheet.getRange(2,31,hrmId.length,1).setValues(summedHours);

                });

}
Comment: can you give a screenshot on what you are trying to work on?
Comment: Updated my answer below. Also provided some changes with your conditions that I think would be better suited with what you are doing. So far, it does what your functions want to achieve. If you have additional conditions you want to be added, you should be able to add them easily. feel free to comment below if you need some assistance.
Answer: I see some questionable lines on your code, I'll discuss them below in comparison with the code I have created and tested based on what you want to achieve and still makes sense.
function sumPrdHrs (item) {
    var hoursToSum = 0;

    // Removed summedHrs array. Based on your code structure, you want to return the sum of productive hours.
    // I don't see any use of creating an array when you can already sum it up directly.

    var hours = ytdSheet.getRange(2,18, ytdSheet.getLastRow()).getValues();
    var productive = ytdSheet.getRange(2,33, ytdSheet.getLastRow()).getValues();
    var units = ytdSheet.getRange(2, 9, ytdSheet.getLastRow()).getValues();
    var id= ytdSheet.getRange(2,29, ytdSheet.getLastRow()).getValues();
    var period = ytdSheet.getRange(2,1, ytdSheet.getLastRow()).getValues();

    for (var i=0; i < hours.length; i++) {
      // In the conditions below, I don't get why you want to compare current period to the period below.
      // It always makes the last item with the same period invalid since there is no last date.
      // But will include it since it might be important to you
      // You also have another condition mentioned but wasn't added initially so I skipped it for now. But feel free to add it on your code.
      if ((id[i][0] == item) && (productive[i][0] == 1) && (period[i][0] == period[i+1][0]) && (units[i][0] != "Revision Stand-by")) {
        hoursToSum += hours[i][0];
        // Total hours is calculated per item, that's all you need to return below.
      }
    }
    return hoursToSum;
}

// Based on what you want, this will sum all hours per item that follows the conditions above.
function productiveHrs () {
  // This will transform the 2d array into 1d array
  var id = ytdSheet.getRange(2,29, ytdSheet.getLastRow()).getValues().flat();
  // We want to loop only the unique items, removing duplicates
  id = id.filter((item, i, ar) => ar.indexOf(item) === i);

  var sumHours = [];

  id.forEach(function(item,i) {
    // Skip blank/invalid items
    if(item) {
      var summedHours = sumPrdHrs(item);
      sumHours.push(summedHours);
    }
  });

  Logger.log(sumHours);
}

This is my sample sheet

This is the output

See that the total of item b which should be 14 is 10 right? That's because of the period condition that checks the period below. That's what I discussed above.
What you can do instead with the "period vs period below" condition is get the unique dates per item, and then sum them per date, per item.
This should be a better way checking period.
// Added time for per item summation of hours
function sumPrdHrs (item, time) {
    var hoursToSum = 0;
    var hours = ytdSheet.getRange(2,18, ytdSheet.getLastRow()).getValues();
    var productive = ytdSheet.getRange(2,33, ytdSheet.getLastRow()).getValues();
    var units = ytdSheet.getRange(2, 9, ytdSheet.getLastRow()).getValues();
    var id = ytdSheet.getRange(2,29, ytdSheet.getLastRow()).getValues();
    var period = ytdSheet.getRange(2,1, ytdSheet.getLastRow()).getValues();

    for (var i=0; i < hours.length; i++) {
      // Compare period[i][0] to unique period fetched from productiveHrs() instead.
      if ((id[i][0] == item) && (productive[i][0] == 1) && (period[i][0] == time) && (units[i][0] != "Revision Stand-by")) {
        hoursToSum += hours[i][0];
      }
    }
    return hoursToSum;
}

function productiveHrs () {
  // Get unique ids and periods
  var id = ytdSheet.getRange(2,29, ytdSheet.getLastRow()).getValues().flat();
  var period = ytdSheet.getRange(2,1, ytdSheet.getLastRow()).getValues().flat();
  id = id.filter((item, i, ar) => ar.indexOf(item) === i);
  period = period.filter((item, i, ar) => ar.indexOf(item) === i);

  var sumHours = [];

  // Traverse all combination of id and period
  id.forEach(function(item,i) {
    if(item) {
      period.forEach(function(time,j) {
        if(time) {
          // Pass period to sum hours under item per period
          var summedHours = sumPrdHrs(item, time);
          // Only push data when where is summedHours
          if (summedHours) {
            sumHours.push([item, time, summedHours]);
          }
        }
      });
    }
  });

  Logger.log(sumHours);
}

This is the sample data:

This is the output:

As you can see, a prettier and more readable output and conditions that would make sense.
If you have some clarifications and additional questions, feel free to comment below.
Comment: Thanks @Marios, I am still awaiting OPs comment/verification as he did mention some conditions that I did not include in my code. Although, what I think he tries to achieve is already included here. and if he does want some conditions added, he can easily add them on.
Comment: Thank you very much @Nazi A! Your solution perfectly solved my problem! I see what was wrong with my code.


fontawesome not displaying in local

Question: I am using fontawesome in my laravel project, the problem is that it will not display if I use it in local without using CDN.
<link href="{{ URL::asset('css/font-awesome.min.css') }}" rel='stylesheet' type='text/css'>

NetworkError: 404 Not Found - http://localhost/mysite/fonts/fontawesome-webfont.woff2?v=4.6.3"
"NetworkError: 404 Not Found - http://localhost/mysite/fonts/fontawesome-webfont.woff?v=4.6.3"

NetworkError: 404 Not Found - http://localhost/mysite/fonts/fontawesome-webfont.ttf?v=4.6.3
here is my fullpath of font-awesome.min.css

C:\wamp\www\mysite\public\css\

Thank you in advance.
Answer: This is because you are using the URL helpers incorrectly.
If font-awesome.min.css is inside public/css then you want to be referencing it just using the asset() helper - you do not need to add the URL facade too.
So just tweak your <link> tag a little like so:
<link href="{{ asset('css/font-awesome.min.css') }}" rel='stylesheet' type='text/css'>

Comment: Hi @James,I tried you solution but still not working
Comment: @jemz can you confirm the full file path for `font-awesome.min.css` please?
Comment: I put the full path in my question
Comment: @jemz have you followed the correct steps to use font awesome locally? It looks like you haven't got the font files in your `public/fonts` directory... http://fontawesome.io/get-started/
Comment: what do you mean the steps ?the fonts are place inside the fonts folder separate to css folder
Comment: @jemz yes but are they located at `public/fonts`? Your 404 errors suggest that those files cannot be found
Comment: it is there I donwloaded again and put the fonts folder inside the public folder and it doesn't work
Comment: @jemz when you follow the 404 error link can you see the contents of them?
Comment: yes I can see the content, I viewed source then click the link of fontawesome then it display the content of it
Comment: yeah me too, I am pulling my hair ...I don't know how to fix this
Comment: @jemz why can't you just use a CDN?
Comment: because the server has no internet connection. and they want to run the application in local
Comment: I put public asset('public/css/fontawesome.min.css')

fontawesome not displaying in local


Como configurar um campo existente para auto-incremento?

Question: Gostaria de definir o campo id já criado como auto-incremento, porque eu criei a tabela e não coloquei essa opção para ele, como poderia fazer isto?
Comment: Pode colocar na sua pergunta a sentença de criação da tabela?
Answer: Para alteração basta usar os seguintes comando:
Para SQL Server:
ALTER TABLE <TABELA>
ALTER COLUMN <Nome_Coluna> Identity (1,1) // caso for sql server

Para MySQL:
ALTER TABLE <TABELA>
MODIFY COLUMN <Nome_Coluna> AUTO_INCREMENT // caso for My SQL

Comment: Isso, porém vc precisa substituir o TABLE pelo nome da sua tabela... Disponha e se este campo for chave primária adicionar PRIMARY KEY ao final.
Comment: Haha obrigada. Mas pena que você demorou para responder porque eu apaguei a tabela e fiz outra com o identity. Mas vou testar sim.
Comment: É sql server mas o nome da coluna é id...Como ficaria isso?Oo ALTER TABLE ALTER COLUMN ID Identity (1,1) // caso for sql server?


What is meant by the statement- "Histograms plot binned quantitative data"?

Question: The problem lies especially in the word-"binned". I've tried to consult the dictionary but it doesn't gives a satisfactory result.
I came across this while I was reading the difference between histogram and bar graph.
Comment: Why not [consult our site](http://stats.stackexchange.com/search?q=bin)?
Answer: In a histogram you typically plot the values of a variable on the x-axis while plotting the frequency/count of the variable's values on the y-axis.
Especially with continous variables it is not very likely that you have the exact same value several times, rather you have similar values or values within a certain range (e.g. 1 to 1.5) that occur more than once.
Binning, thus, simply means that instead of plotting the count of each and every single observed value of a variable, you take ranges or bins of that variable and plot the frequency/count pertaining to values in that bin.
See: https://en.wikipedia.org/wiki/Data_binning
Answer: A histogram is a graphical representation of the estimated distribution of a variable (usually continuous). It consists of numbers for bins represented as bars.
Each bin is defined by an interval. The bar height for each bin represents the  number of values that lies in the corresponding interval.
So, for example if we have a sample X = (-4.5, -2.7, 3.4, 4.7, 5.5, 5.8, 6.0, 8.9, 9.2, 12.0), we can plot a histogram with, let's say, 4 equal-sized bins.
Intervals for bins would be [-5, 0), [0, 5), [5, 10), [10, 15]. So your 3rd bin will have value of 5, because 5 values of X lie within it (between 5 and 10).
Comment: You describe a *bar chart* of frequencies.  By definition, a histogram represents frequencies or probabilities with *areas*, not heights. The distinction becomes clear in the histograms that have unequal bases to their bars.
Comment: I've edited slightly for style. Strictly your open intervals leave ambiguous which bin receives values that are exact multiples of the bin width, e.g does 5 go up or down? One convention is ( ], ( ], ..., (); another is [  ), [ ), [) can make sense too.
Comment: Thanks. It was sort of intentional, because I hesitated to additional information abou right or left-closed bins.
Answer: As a matter of general graphics, a histogram is $-$ or I would say should be considered as $-$ just a special case of a bar chart. The argument is simple: a histogram uses bars to show information, just like any other kind of bar chart. The information here can be frequencies, probabilities, proportions or fractions, probability densities or frequency densities.
In statistics, a histogram is often considered to be not a bar chart. (The term bar chart seems more common than, say, bar graph or bar plot, but I won't argue for one rather than another: feel free to follow personal or collective taste.) It's hard for me to work out exactly why this is said. It could be just a convention handed down: we have a long-established term (histogram was one of Karl Pearson's coinages) and specific rules that should be used. For example, if bins (classes, intervals) are of unequal width, then arguably only showing some kind of density that scales for those varying widths makes sense. But that still doesn't explain the denial to me. I detect a whiff of academic snobbery in denying that histograms are bar charts. Bar charts are, after all, often taught at an early age and their use by academics, researchers, or students in papers or texts is often considered indicative of limited, low-level analyses.
But I'd argue, and I doubt that I am alone, that any bar chart showing frequencies for a discrete variable is also a histogram. Conventionally, gaps are left to underline that the scale is discrete. If the bars shrink to spikes, many might demur in calling such plots either histograms or bar charts, but again that seems a matter of conventional usage rather than logic. Naturally, or arguably, there is no harm in using a simple evocative term: I've used the term "spike plots" and it is echoed in (for example) Stata's spikeplot command:  see documentation here.
As already implied, I would be happy also to regard a histogram showing a continuous distribution by touching bars as a kind of bar chart. But watch out: if you follow that usage, you might be regarded as ignorant, or a little less bluntly, as just not having learned a standard term.
Either way, bin is a common word for each class or interval used on a histogram, or more generally in any grouping of a distribution into disjoint intervals. General-purpose dictionaries seem slow in catching that usage, which seems to have become increasingly common in recent decades. I think I first encountered it in J.W. Tukey. 1977. Exploratory data analysis Reading, MA: Addison-Wesley. The earliest use I can recall was by the statistically-minded geologist W.C. Krumbein in the 1930s.
(I'd add a precise reference later if I can find it again. Conversely, earlier references would be welcome.)
Comment: I think you are off base with that "whiff of academic snobbery."  I believe there are several underlying, truly compelling reasons to insist that histograms represent frequencies with *areas* rather than *heights*.  The best is that the area interpretation is the one that generalizes to probability density functions: the height interpretation does not.  Another is that the distinction brings with it the realization that geometric elements of a graphic can function in differing ways to represent data effectively: we are not required to use lengths (or heights) alone.
Comment: @whuber Naturally I agree on, and am fully aware of, the area principle (although my answer could make that more obvious). But in turn you don't explain whether a histogram is, or is properly described as, a bar chart. I wasn't aware that anyone specifies that the bars in a bar chart have to be of equal width along one axis, although many people won't see many examples to the contrary.
Comment: In a "bar chart," the *lengths* of the bars directly represent the quantity of interest.  Even in variants where the widths represent something else, the lengths retain that meaning.  Additionally, some people (see [Wikipedia](https://en.wikipedia.org/wiki/Bar_chart#Usage)) allow that "bars may be arranged in any order": it's a *categorical* plot. Moreover, I am reluctant to allow gaps between the bars of a histogram: that would be deceptive. If you would like, then, to describe a histogram as a "contiguous ordered bar chart of probability (or frequency) density," that would be accurate.
Comment: Indeed; as my answer makes explicit, bar height can encode density, and only does that validly when density calculation adjusts for unequal bin widths. The first two paragraphs of my answer contrast what I see as the graphics view and the statistics view of histograms, which don't coincide. I am more than happy to allow gaps between bars of a histogram when the bars represent discrete values.
Comment: Unsurprisingly to any who know my preferences, I like the Stata view of histograms which by default gives touching bars but lets the user insist on gaps (and I hate the Excel view which is entirely the opposite). To Stata and Excel, add any and all programs jumping either way.
Comment: In short, I won't insist on "contiguous" as part of the definition, and remain curious about the precise logical grounds why that is not accurate. If it is described as customary usage, that's fine by me; if it can be documented that that was precisely Karl Pearson's definition, that is of historical interest.
Comment: @whuber Your answer with your logic for definitions and distinctions would be much appreciated, including where and why you differ from any others.


Where shall I place "PHP SAML" library so that SAML Authentication module could find it?

Question: I have a multisite Drupal 7 installation (not using Composer), and I can't figure out where shall I place "PHP SAML" library so that the SAML Authentication module would find it.
The instructions within the module simply say to install OneLogin's SAML PHP module (and the only provided instructions set uses Composer). The Libraries API module is installed, but wherever sites/domainname/libraries I try to place the content of the cloned https://github.com/onelogin/php-saml.git, the SAML Authentication module still says the "PHP SAML" isn't installed.
How and where do I place the right files in absence of Composer?
Comment: I placed mine in the root directory and added the library path in my settings.php: $settings['simplesamlphp_dir'] = '/path/to/library';
Comment: You know, it's possible to use Composer with Drupal 7. But even if you don't want to do that, there's nothing stopping you from using Composer to install simpleSAMLphp. It should show up in the `vendor` directory if you do this. You can do this in a separate repo and copy that into your project if you don't want to be able to update the code easily.
Answer: I dabbled with this before. In order to get phpsimplesaml to work you need a module that autoloads composer libraries. I ultimately gave up trying it on d7 because it was a nightmare to get everything working, but I at least had the library recognized.
Comment: Thanks for the idea, I'll try "X Autoload" module first.


ldap client failed to bind to ldap server

Question: I have a windows ldap client setup and I have been able to connect to it using the gui ldap admin client. When I try to connect to it using my ubuntu 12.04 client I get the following error in my auth.log:

Aug 21 16:38:20 uby getent: nss_ldap: failed to bind to LDAP server
  ldap://pdc.ad.company.com:389: Invalid credentials
Comment: Please read error messages instead of just posting them here and expecting us to do it for you. By posting that line your question answers itself.
Answer: Aug 21 16:38:20 uby getent: nss_ldap: failed to bind to LDAP server ldap://pdc.ad.company.com:389: Invalid credentials
The "invalid credentials" part means that the credentials your Ubuntu client is using are not valid credentials.  Supply it with correct credentials that have access to read your LDAP directory (or none, if you allow anonymous access), and that should do the trick.
Comment: Thanks for the reply, do you know what the file it is that I set the user name and password? It use to be some sort of db file in the past.
Comment: @rahrahruby that would really depend on what program you're using to auth against LDAP... which is...?


Limit number of webservice call in Swift

Question: I need to do a lot of web service call - downloading images -, but want to limit it to 5 establish connection at the same time.  Do I need a concurrent queue or should I use dispatch group?
I think I need a queue, but an item from queue should remove only when web service response returned.
Answer: The answer depends on what you want achieve next.
If you want to wait all answer before going further, you should use dispatch_group, but if you want perform action ( like updating UI ) after each WS response, you need concurrent queue


#limit_validation_errors submit problem

Question: I have a problem using #limit_validation_errors for a submit button on which I dont want all the validations to run. Here's my custom submit button :
$form['actions']['draft'] = array(
     '#type' => 'submit',
     '#value' => 'Save as draft',
     '#submit' => array(''),
     '#limit_validation_errors' => array(
       array('title'), // Validate $form_state['values']['title'].
     ),
);

It works perfectly in the sense that it checks the validation on the title field only, but the weird thing is, it does not submit any other field's values when submitted. I wrote a custom submit handler to check that. However, the values are present in $form_state['input'] but not in $form_state['values'].
What I want simply is, this form will submit with just the values given and overlook the validations when this custom submit button is clicked.
FYI : I can get $form_state['values'] perfectly in my custom validate function.
Any ideas? What am I doing wrong here?
Answer: It works as expected - in $form_state['values'] you have validated values, and in $form_state['input'] you have raw values. If you will limit validation to title field only - only this field will be transferred into "values" array.
Answer: I had to switch from a type="submit" to type="button" in order to disable errors + use ajax.
  public function buildForm(array $form, FormStateInterface $form_state) {

    // ...

    // Create Draft Button
    $form['#prefix'] = '
';
    $form['#suffix'] = '
';
    $form['actions']['save_draft'] = [
      '#type' => 'button',
      '#value' => $this->t('Save Draft'),
      '#name' => 'submit__save_draft', // TODO: Make constant
      '#limit_validation_errors' => [],
      '#ajax' => [
        'callback' => [$this, 'ajaxSubmitForm'],
        'wrapper' => 'send-emails-manual-send-by-role-wrapper',
        'method' => 'replace',
        'effect' => 'fade',
      ],
    ];

    return $form;
  }

  /**
   * Submit the form using ajax
   */
  public function ajaxSubmitForm(array &$form, FormStateInterface $form_state) {
    $this->submitForm($form, $form_state);
    return $form;
  }

#limit_validation_errors submit problem


One disk of a software RAID10 was repartitioned. What to do?

Question: I have a software array in one machine
root@c ~# mdadm --query --detail /dev/md0
/dev/md0:
        Version : 0.90
  Creation Time : Mon Mar 17 10:22:38 2014
     Raid Level : raid10
     Array Size : 976772096 (931.52 GiB 1000.21 GB)
  Used Dev Size : 488386048 (465.76 GiB 500.11 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Mon Jan 19 10:30:49 2015
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : far=2
     Chunk Size : 256K

           UUID : e21d2907:a7b46e36:220b319d:884b2099
         Events : 0.296

    Number   Major   Minor   RaidDevice State
       0       8       32        0      active sync   /dev/sdc
       1       8       48        1      active sync   /dev/sdd
       2       8       64        2      active sync   /dev/sde
       3       8       80        3      active sync   /dev/sdf

md0 is a RAID-10 array. A coworker made a fdisk on /dev/sdf and created three linux partitions there. Currently md0 is working fine, but I'm worried what will happen when the server will be restarted.
Is my information safe? Does the fdisk destroyed one of the pair of RAID1? How to fix sdf without damaging the array?
Thanks in advance.
Answer: Your information is not safe; you have destroyed one disk in the array and it will not come up on reboot.  Your kernel will likely still be using the old partition table, because it was in use when the fdisk was done, but you need to take these steps:

Back up your data (it should go without saying, but I want to make it clear that I take no responsibility if this procedure doesn't work ;-)
Fail the disk: mdadm --fail /dev/md0 /dev/sdf

The array should now show as degraded in /proc/mdstat

Remove the disk from the array: mdadm --remove /dev/md0 /dev/sdf

The disk should now have disappeared from /proc/mdstat

Clear the partition table: dd if=/dev/zero of=/dev/sdf bs=1k count=1
Re-add the disk to the array: mdadm --add /dev/md0 /dev/sdf
The array should now be rebuilding.  Monitor /proc/mdstat until complete.
Back up your data again to a different location. Compare it with the first backup and ensure that only new/updated files have changed.

You should be able to do all of the above while the array is online, because RAID 10 means that you have at least 2 copies of every block.
What your coworker did to the array is a good reason for not using raw disks in RAID arrays.  For future arrays, you should create a partition on each disk of type FD so that the system knows they are RAID partitions and anyone who comes along will see partitions on it.  You only lose a tiny amount of space and you gain a really big red flag that tells people that the disk is in use.
Good luck!
Comment: Thank you very much, Paul! I will follow dutifully your advice. From now on, only RAID in partitions.  Thanks again!


Find the subrepresentation of a cyclic group

Question: This is a spin-off of this question:
https://math.stackexchange.com/questions/1636682/show-that-representation-rho-can-be-divided
I came across the problem of dividing representation $\rho$ of a cyclic group given as below:
$$
g \longmapsto \begin{pmatrix}
     1 & -1 \\
     1 & 0  \\
     \end{pmatrix}
$$
$$
g^2 \longmapsto \begin{pmatrix}
     0 & -1 \\
     1 & -1  \\
     \end{pmatrix}
$$
$$
g^3 \longmapsto \begin{pmatrix}
     -1 & 0 \\
     0 & -1  \\
     \end{pmatrix}
$$
$$
g^4 \longmapsto \begin{pmatrix}
     -1 & 1 \\
     -1 & 0  \\
     \end{pmatrix}
$$
$$
g^5 \longmapsto \begin{pmatrix}
     0 & 1 \\
     1 & -1  \\
     \end{pmatrix}
$$
$$
1 \longmapsto \begin{pmatrix}
     1 & 0 \\
     0 & 1  \\
     \end{pmatrix}
$$
into two irreducible representations.
Formally the task is to:
Reduce this representation to a direct sum of irreducible representations and find the matrix representation of $\rho$ in a new basis that's a sum of the basis of the found irreducible representations
How I think this one of the subrepresentation should look like (in case of the second subrepresentation it should be to the negative power):
$$\rho^1: \ 1 \longmapsto 1 \\ \ \ \ \ \ \ \  g \longmapsto \lambda \\ \ \ \ \ \ \ \ \ \ \ g^3 \longmapsto \lambda^3 \\ \ \ \ \ \ \ \ \ \ \ g^4 \longmapsto \lambda^4 \\  \ \ \ \ \ \ \ \ \ \ g^5 \longmapsto \lambda^5$$
Then, by Matschke's theorem the initial subrepresentation would look like this:
$$
g \longmapsto \begin{pmatrix}
     \lambda & 0 \\
     0 & \lambda^{-1}  \\
     \end{pmatrix}
$$
$$
g^2 \longmapsto \begin{pmatrix}
     \lambda^2 & 0 \\
     0 & \lambda^{-2}  \\
     \end{pmatrix}
$$
$$
g^3 \longmapsto \begin{pmatrix}
     \lambda^3 & 0 \\
     0 & \lambda^{-3}  \\
     \end{pmatrix}
$$
$$
g^4 \longmapsto \begin{pmatrix}
     \lambda^4 & 0 \\
     0 & \lambda^{-4}  \\
     \end{pmatrix}
$$
$$
g^5 \longmapsto \begin{pmatrix}
     \lambda^5 & 0 \\
     0 & \lambda^{-5}  \\
     \end{pmatrix}
$$
$$
1 \longmapsto \begin{pmatrix}
     1 & 0 \\
     0 & 1  \\
     \end{pmatrix}
$$
Comment: This doesn't make sense as a representation.  $1$ should map to the identity matrix.  Otherwise, if it made sense, you would take the traces of the matrices and find an integer linear combination of traces of the irreducible representations to determine the decomposition.  (You "split" or "reduce" a representation, rather than "divide.")
Comment: Those summands are necessarily one-dimensional. In a 1-dimensional space a group can only act by scalars. Those scalars are eigenvalues of the matrices. Why don't you find an eigenbasis of $g$, and see what comes out?
Comment: I fixed the question and added some calculations - after finding the eigenvalues i tried to find the spanning vector but without any positive result
Answer: The first thing to note is that you have the wrong eigenvalues. The characteristic polynomial for $g$ is $\lambda^2-\lambda+1$ and has eigenvalues
$$\lambda=\frac{1+i\sqrt{3}}{2}=e^{2\pi i/6}$$
and
$$\overline{\lambda}=\frac{1-i\sqrt{3}}{2}=e^{-2\pi i/6}=\lambda^{-1}.$$
Now, let $\{x,y\}$ be a basis for $\mathbb{C}^2$ for which $g$ acts by the matrix
$$
g=\begin{pmatrix}1&-1\\1&0\end{pmatrix}
$$
You want a basis of eigenvectors, so you want a vector $v=ax+by$ such that
$$\lambda v=g.v=g.(ax+by)=(a-b)x+ay.$$
This gives the equations
$$\begin{cases} \lambda a=a-b\\\lambda b=a\end{cases}.$$
A solution to this system is $a=\lambda$ and $b=1$, and we get and eigenvector
$$v=\lambda x+y.$$
Similarly,
$$w=\lambda^{-1}x+y$$
is an eigenvector with eigenvalue $\lambda^{-1}$. Reverting to column vector notation, the basis
$$\left\{\begin{pmatrix}\lambda\\1\end{pmatrix},\begin{pmatrix}\lambda^{-1}\\1\end{pmatrix}\right\}$$
decomposes your representation into two 1-dimensional invariant subspaces.
Comment: Sorry. I wrote the wrong $P$. It should be $$P=\begin{pmatrix}\lambda&\lambda^{-1}\\1&1\end{pmatrix}.$$ You should calculate $PGP^{-1}$.
Comment: Could you please elaborate on why did you find the vector v the way you did? From what I've been told i should check it with this condition $(g-\lambda I)v=0$ ?
Comment: The equation $(g-\lambda).v=0$ is the same as $g.v=\lambda v$.
Comment: If you need further help, please tell me where you are getting confused. Which line? That will help me answer your question.
Comment: I am confused with why is my reasoning wrong: $\begin{pmatrix}1 -\lambda&-1\\1&- \lambda \end{pmatrix} \begin{pmatrix}x\\y\end{pmatrix}=\begin{pmatrix}0\\0\end{pmatrix}$, so i get $(1-\lambda)x-y=0$ and $x-\lambda y=0$, from which i get $(1-\lambda)\lambda y-y=0$ and $y(-\lambda^2+\lambda-1)=0$
Comment: There is nothing wrong with it, you just aren't making useful conclusions. You have the equation $x-\lambda y=0$. What does that tell you? It is also worth noting that 1-\lambda=\lambda^{-1}$, so the system has a consistent solution.
Comment: Thanks for your help, but so far I only have two 1-dimensional invariant subspaces and what I need is  to find the representations in those subspaces - with that find the matrix representation of my basis representation should be rather easy.
Comment: I think you need to think about what it means to find "the representations in those subspaces". How does $g$ act on $v$ and $w$? What is the matrix for $g$ in that basis?
Comment: I updated the question with how I understand it should look like.
Comment: Can you write the matrix for $g$ in the basis $\{v,w\}$ above? This can be obtained by conjugating by the matrix $$P=\begin{pmatrix}\lambda&1\\1&\lambda^{-1}\end{pmatrix}$$ but hopefully you can identify the final answer without making this computation.
Comment: Normally i would find $P^{-1}$ and multiply $P^{-1}GP$, and get the matrix in the new basis, but i cannot do it since $detP=0$. Also is the subrepresentation $\rho^1$ correct?
Comment: Ok, i finally calulated it - it doesn't look nice, but w/e. Once more - are the one dimensional subrepresentations I wrote correct? In what other way could I get the answer without calculating the inverse matrix?
Comment: could you please help me with the last question? I really want to know if I got the subrepresentations right. Also you said that I can with the matrix for g in the new basis without the said computation - what would be an easier way for it?
Comment: Since $gv=\lambda v$ and $gw=\lambda w$ the matrix for $g$ in the basis $\{v,w\}$ is $\begin{pmatrix}\lambda&0\\0&\lambda\end{pmatrix}$. It is possible this should be $P^{-1}GP$. Is this what you got?
Comment: No the result i get from $\begin{pmatrix}\lambda& 1 \\ -\lambda^{-1} &1\end{pmatrix} \begin{pmatrix} -1 & \lambda\\1&0\end{pmatrix} \begin{pmatrix}\lambda&\lambda^{-1}\\1&1\end{pmatrix}$ is different from the result you wrote. Also i think that $gw=\lambda^{-1}w$, so the matrix should be $\begin{pmatrix}\lambda&0\\0&\lambda^{-1} \end{pmatrix}$.
Comment: You are right. The correct computation is actually $P^{-1}GP$. To see this, note that $Px=v$ ($x=(1,0)^T$), and $Gv=\lambda v$, so $$P^{-1}GPx=P^{-1}Gv=P^{-1}(\lambda v)=\lambda P^{-1}v=\lambda x$$


Decidability of model of set theory.

Question: I am wondering if there exists a model of set theory (KPU or ZFC) that has decidable theory?
Thanks.
Comment: Such a model will be able to solve the halting problem.
Comment: what do you mean by a model having a decidable theorem?
Comment: I mean , is there a model of set theory for which we can design an algorithm to decide all the first order formula sastified in it .... Thanks , obviously I am not talking here about all theorem of the structure ...
Comment: Do you have the references of the work you mentionned of David Harris ? Please...
Comment: @Dehn: I believe that Peter Smith is talking about David Harris's answer below.
Comment: @Peter Smith: Please see my comment below the answer by David Harris.
Comment: [My comment deleted -- at least until I've had time to rethink in the light of Carl Mummert's comments below.]
Comment: Ok thank you very much guys, I have now understood what I was doing wrong. Thanks!
Answer: Any model of set theory includes the full true theory of arithmetic. Even very weak theories of arithmetic, let alone the full True Arithmetic, are undecidable.
Comment: An arbitrary model could have nonstandard natural numbers and so the set of arithmetical formulas that is true in that model could be very different than the true theory of arithmetic. Naively, that set could even be computable. So it is not enough to note that the set of true arithmetic sentences is not computable. ...
Comment: However, the answer can be remedied via Tennenbaum's theorem that no nonstandard model of PA has a computable elementary diagram (neither addition nor multiplication can be computable). Because the theory of each model of ZFC computes the theory of a model of PA, if the model of ZFC is not an $\omega$-model this shows its theory cannot be computable. Or you could use the fact that the theory of the model of ZFC computes a completion of PA, and no such extension is computable (even if it is nonstandard). But it takes more than the uncomputablity of true arithmetic, in any case.
Comment: That, and we don't even know for sure if there exists *any* model of ZFC. :)
Comment: Any reasonable model of set theory, anyway... (Even models of finite set theory can be undecidable – ZFC minus Infinity and Peano arithmetic are biinterpretable, for example.)
Answer: The viewpoint that is usually taken in the literature is to look at extensions of the original theory rather than models of it. A theory is said to be essentially incomplete if it has no decidable complete consistent extension.  Essential incompleteness implies, in particular, that the theory cannot have a model whose elementary diagram is decidable.
A corollary of the incompleteness theorems is that many effective theories are essentially incomplete. For example, Peano arithmetic and ZFC are both essentially incomplete, so they cannot have models whose elementary diagrams are decidable.
On the other hand, the theory of groups is incomplete, because it cannot prove or disprove $(\exists x)(\exists y)(x \not = y)$, but it can be made complete by adding the axiom $(\forall x)(\forall y)(x = y)$, and the resulting theory (which is categorical) has a model whose elementary diagram is decidable.

Decidability of model of set theory.


Ring theory question.

Question: Why is a field with 27 elements has characteristic 3?
I was solving a question and I came to know this fact which I didn't know before.
Is there anyone who can explain this to me?
Thanks in advance.
Answer: The characteristic has to be a prime. Since the additive group has $27$ elements, it must divide $27$.
Comment: Is there any theorem for this?
Comment: Yes. ${}{}{}{}$
Comment: @user114873 Note that a field has no (non-zero) zero-divisors so that if $ab=0$ it follows that $a=0$ or $b=0$. If we call $1+1+\dots +1, r$ where there are $n$ $1$s added together, and if the order of $1$ is $n=pq$ as integers then we have $pq=0$ in the field. So $p=0$ or $q=0$ (which means a number of copies of $1$ add to give zero).
Answer: The characteristic of $F$ is the order of $1$ in the additive group of $F$, which is a divisor of $|F|$ per group theory and is a prime (or zero) for any field.
Answer: Everything is well explained in Finite Field on wikipedia
All finite fields of some fixed size are unique (up to isomorphism), and any field of size $p^k$ has characteristic $p$.
Comment: must that p be prime?
Comment: @user114873 yes of course, if you want a field. Otherwise, you only get rings, with divisors of zero.

Ring theory question.


Why does the log likelihood need to go to minus infinity when the parameter approaches the boundary of the parameter space?

Question: In a recent lecture I was told that, in order for the maximum likelihood estimate to be valid, the log likelihood needs to go to minus infinity as the parameter goes to the boundary of the parameter space. But I don't understand why this is essential. Suppose the log likelihood goes to some kind of asymptote. Then the parameter that maximizes the likelihood is still the maximum likelihood estimate, right?
Comment: (+1).  Gosh: so if I perform ML fitting of a Normal distribution to my data and limit the possible values of the SD to the range from $10^{-1000}$ to $10^{1000}$ and the mean to the range $\pm 10^{1000},$ I guess my estimates will no longer be valid ... . :-).  Since those endpoints are beyond the range of IEEE floating point precision, that must mean nobody can trust any of the statistical software that runs on standard computing devices.  It must be time for all of us to pull out that old abacus (it's on the shelf with the slide rule) and go back to doing the calculations by hand.
Comment: The usual argument for the asymptotic normality of the ML estimator  uses an assumption that  true value of the parameter is in the interior of the parameter space.  Presumably, the assumption you are talking about is used to prove this interiority.  The condition you mention is definitely not essential, in the sense of being necessary.
Comment: What is the parameter space, what is the parameter in question, and which distribution? What you are told lacks a lot of critical information, in order for one to be able to assess its validity.
Answer:
in order for the maximum likelihood estimate to be valid, the log likelihood needs to go to minus infinity as the parameter goes to the boundary

This is equal to saying, the Likelihood of a parameter needs to become 0 at the boundary of the parameter space in order for the result to be valid.
Well first of all, you can restrict the parameter space to values that all have a positive likelihood and still obtain a valid estimate.
Secondly, even if you use, say $(-\infty,\infty)$, you don't come close to the boundary since any off the shelf optimisation package performs some sort of random initialisation and then approaches the minimum using some method such as gradient descent, conjugate gradient or another. In either case, you almost never end up approaching the boundary of the parameter space, so I don't quite understand why the boundaries matter in the first place.
And even if you do that on purpose, at one point you will hit the floating point precision of your operating system. I can guarantee you that at that point, you haven't really approached the boundary $-\infty$ by much. :)
Personally I find the underflow problem arising when calculating sums and products of very small likelihoods and the log sum exp trick much more interesting and more noteworthy issue that actually matters a lot in practice, unlike reaching the boundaries of the parameter space.


bash getopts behavior different in while loop vs commandline

Question: Commandline
$ getopts ":mnopq:rs" Option -q
$ echo $Option
?
$ echo $OPTARG

Given
$ cat getopt.sh
#!/bin/bash

echo '$@' is $@
while getopts ":mnopq:rs" Option
do
  echo Option is  $Option
  echo OPTARG is $OPTARG
  case $Option in
    m     ) echo "Scenario #1: option -m-   [OPTIND=${OPTIND}]";;
    n | o ) echo "Scenario #2: option -$Option-   [OPTIND=${OPTIND}]";;
    p     ) echo "Scenario #3: option -p-   [OPTIND=${OPTIND}]";;
    q     ) echo "Scenario #4: option -q-\
                  with argument \"$OPTARG\"   [OPTIND=${OPTIND}]";;
    #  Note that option 'q' must have an associated argument,
    #+ otherwise it falls through to the default.
    r | s ) echo "Scenario #5: option -$Option-";;
    *     ) echo "Unimplemented option chosen.";;   # Default.
  esac
done

I get
Script
$ ./getopt.sh -q
$@ is -q
Option is :
OPTARG is q
Unimplemented option chosen.

Why is there a difference in output between Commandline and Script ?
Comment: I can't reproduce it in bash 3.2.57 or 4.2.37.
Answer: Have you run the "Commandline" test multiple times in the same shell? getopts uses the variable OPTIND to keep track of where it is in the argument list, so it doesn't just process the same option over and over. As a result, if you run the test multiple times it'll skip over what it processed last time. In your case, I suspect this is making it think it's at the end of the argument list (in which case it'll have an exit status of 1). Here's an excerpt from the bash man page:

[...] Each time it is invoked, getopts places [...] the index
  of the next argument to be processed into the variable OPTIND. OPTIND
  is initialized to 1 each time the shell or a shell script is invoked.
  [...] The shell does not reset OPTIND automatically; it
  must be manually reset between multiple calls to getopts within the
  same shell invocation if a new set of parameters is to be used.

Here's an example:
$ getopts ":mnopq:rs" Option -q
$ echo "status=$?, Option='$Option', OPTARG='$OPTARG', OPTIND=$OPTIND"
status=0, Option=':', OPTARG='q', OPTIND=2
$
$ getopts ":mnopq:rs" Option -q
$ echo "status=$?, Option='$Option', OPTARG='$OPTARG', OPTIND=$OPTIND"
status=1, Option='?', OPTARG='', OPTIND=2
$
$ unset OPTIND
$ getopts ":mnopq:rs" Option -q
$ echo "status=$?, Option='$Option', OPTARG='$OPTARG', OPTIND=$OPTIND"
status=0, Option=':', OPTARG='q', OPTIND=2

The first time it does what you expect. The second time it indicates it's out of options to process, which matches what you're seeing. The third time OPTIND has been unset, so it defaults back to the beginning of the argument list.


ERR_SSL_PROTOCOL_ERROR after http2 enabled

Question: I'm setting HTTP2 for my application and I get randomly ERR_SSL_PROTOCOL_ERROR on y browser.
this is happening only on Google Chrome browser. Everything is OK on Firefox.
I'm using Let's Encrypth to encrypt with SSL and everything was working well. The problem occured after I have enabled HTTP2 today. I'm using Apache2. I updated it to 2.4.38 to support HTTP2 module
I tried the following :

disabling http2 module / restarting apache2
clearing all the browser cache
resetting the HSTS header using .htacess :

Header set Strict-Transport-Security "max-age=0; includeSubDomains;" env=HTTPS
No results, this is still have randomly the error.
here is my vhost config :
<IfModule mod_ssl.c>
    <VirtualHost *:443>
            DocumentRoot /var/www/domain/prod/web
            ServerName sub.domain.tld
            ServerAlias domain
            ServerAdmin firstname.lastname@example.com

    <Directory "/var/www/domain/prod/web/">
            Options FollowSymLinks MultiViews
            AllowOverride All
            Order allow,deny
            Allow from All
    </Directory>
    ErrorLog /var/www/domain/prod/logs/error.log
    LogLevel warn
    CustomLog /var/www/domain/prod/logs/access.log combined
    ServerSignature On

            SSLEngine On
            Include /etc/letsencrypt/options-ssl-apache.conf

            Protocols h2 http/1.1

            SSLCertificateFile
            /etc/letsencrypt/live/domain/fullchain.pem
            SSLCertificateKeyFile
           /etc/letsencrypt/live/domain/privkey.pem
    </VirtualHost>
 </IfModule>

last update
I've market Martin's answer as best answer because installing PHP-FPM seems to have solved my problem (don't see the SSL_ERROR)
what i did: https://www.vultr.com/docs/use-php5-fpm-with-apache-2-on-ubuntu-14-04
but for one of my websites, I get a 500 error for a specific page. le apache error logs says :
[Mon Apr 01 14:58:31.844703 2019] [:error] [pid 30434] [client 220.127.116.11:30902] FastCGI: server "/usr/lib/cgi-bin/php5-fcgi" stderr: PHP message: PHP  12. Project->projectFormat() /var/www/jachete/models/Project.php:137

I don't really understand what's the error is about, any idea ?
update
interesting things happens here after disabling "pagespeed" module, here the errors I get from error.log
[Sun Mar 31 16:13:07.090348 2019] [http2:warn] [pid 16845] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
[Sun Mar 31 16:13:07.119247 2019] [mpm_prefork:notice] [pid 16845] AH00163: Apache/2.4.38 (Ubuntu) OpenSSL/1.1.1b mpm-itk/2.4.6-01 PHP/5.5.9-1ubuntu4.27 configured -- resuming normal operations

according to https://http2.pro/doc/Apache this is due to mpm-itk not supproting http2. as it's just notices/warnings I think there is no reason to cause SSL_ERROR ?
here is the test results of Qualys Lab (A)Qualys : https://i.stack.imgur.com/deMnu.jpg
here are some apache2 error logs :
    [Sun Mar 31 15:31:07.393421 2019] [pagespeed:error] [pid 13785] [mod_pagespeed 220.127.116.11-0 @13785] Failed to make directory /var/cache/mod_pagespeed/v3/domain.tld/https,3A/,2Fapp.domain.tld: Permission denied
[Sun Mar 31 15:31:07.393454 2019] [pagespeed:error] [pid 13785] [mod_pagespeed 220.127.116.11-0 @13785] Could not create directories for file /var/cache/mod_pagespeed/v3/domain.tld/https,3A/,2Fapp.domain.tld/views/assets/img/loading.gif,.temp
[Sun Mar 31 15:31:07.393488 2019] [pagespeed:error] [pid 13785] [mod_pagespeed 22.214.171.124-0 @13785] /var/cache/mod_pagespeed/v3/domain.tld/https,3A/,2Fapp.domain.tld/views/assets/img/loading.gif,.tempqeySBV:0: opening temp file: No such file or directory
[Sun Mar 31 15:31:08.115111 2019] [pagespeed:error] [pid 13785] [mod_pagespeed 22.214.171.124-0 @13785] Failed to make directory /var/cache/mod_pagespeed/v3/domain.tld/https,3A/,2Fapp.domain.tld: Permission denied
[Sun Mar 31 15:31:08.115142 2019] [pagespeed:error] [pid 13785] [mod_pagespeed 18.104.22.168-0 @13785] Could not create directories for file /var/cache/mod_pagespeed/v3/domain.tld/https,3A/,2Fapp.domain.tld/views/assets/img/logo/favicon.png,.temp
[Sun Mar 31 15:31:08.115162 2019] [pagespeed:error] [pid 13785] [mod_pagespeed 184.108.40.206-0 @13785] /var/cache/mod_pagespeed/v3/domain.tld/https,3A/,2Fapp.domain.tld/views/assets/img/logo/favicon.png,.tempT2JwQU:0: opening temp file: No such file or directory
[Sun Mar 31 15:31:08.118059 2019] [pagespeed:error] [pid 13785] [mod_pagespeed 188.8.131.52-0 @13785] Failed to make directory /var/cache/mod_pagespeed/v3/domain.tld/https,3A/,2Fapp.domain.tld: Permission denied
[Sun Mar 31 15:31:08.118103 2019] [pagespeed:error] [pid 13785] [mod_pagespeed 18.104.22.168-0 @13785] Could not create directories for file /var/cache/mod_pagespeed/v3/domain.tld/https,3A/,2Fapp.domain.tld/views/assets/img/logo/xfavicon.png.pagespeed.ic.coNvSghNBK.webp,.temp
[Sun Mar 31 15:31:08.118126 2019] [pagespeed:error] [pid 13785] [mod_pagespeed 18.104.22.168-0 @13785] /var/cache/mod_pagespeed/v3/domain.tld/https,3A/,2Fapp.domain.tld/views/assets/img/logo/xfavicon.png.pagespeed.ic.coNvSghNBK.webp,.tempg9bR5T:0: opening temp file: No such file or directory

there is a permission denied error, but when I look to permissions, I get this
drwxr-xr-x  6 www-data www-data 4096 Oct 30 06:34 http,3A/
drwxr-xr-x  3 www-data www-data 4096 Jun 30  2018 https,3A/

should I add write permission for all users ? (chmod a+w dir/ -R) ?
By the way, the SSL error still occure, even when I disable the PageSpeed module, so i guess the errors in log above aren't related to my SSL problem ?
Comment: @anx copy/paste mistake, sorry. I updated the post. any idea ?
Comment: @MichaelHampton I just updated the post with tail error.log all is in relation with pagespeed
Comment: Are you unfamiliar with Linux? The error message is obvious, but perhaps not to someone who doesn't know Linux.
Comment: @anx I updated the post with Qualys results (note: A), Where can I find the TLS+HTTP/2 logs ? by disabling TLS1.3 you mean remove Lets Encrypt certificate ?
Comment: You have permission issues. And directory paths look suspicious `/var/cache/mod_pagespeed/v3/domain.tld/https,3A/,2Fapp.domain.tld` has commas in it and two characters that resembles URI encoding characters.
Comment: @MichaelHampton I see the Permission Denied and Directory not found error, but is this in relation with SSL issues ?
Comment: It might be related, and it might not be. You won't be able to rule it out until you fix it.
Comment: @MichaelHampton & ychaouche ok I updated the post because I actually have rwx perpission on the user www-data, do you recommand doing something like chmod a+w ? by the way, the SSL error still even when I disable PageSpeed module
Comment: @MichaelHampton I diasbled pagespeed module and get another warning related to http2 / ssl. any chance for you to have another look to my updates and give your feedback ?
Comment: I'm pretty sure you need to be using mpm_event (or possibly mpm_worker) for http2. prefork should have died in a fire years ago, and itk is a niche thing that you might not be able to get much help for here.
Comment: @MichaelHampton acutually, I don't need http2 anymore, I just want to get my app works on Google Chrome again. I disabled http2 for now.
Answer: Disable mpm_prefork, don't use mod_php use PHP FPM as a fast CGI proxy and all will be well.
See: https://http2.pro/doc/Apache
I hope that helps.
Comment: @MarinMcEvoy thank you for your answer. I've marked it as the best response cause it seems to solve my problem. but on an other site, I have a 500 error with the following verbose: `[Mon Apr 01 14:58:31.844703 2019] [:error] [pid 30434] [client 184.108.40.206:30902] FastCGI: server "/usr/lib/cgi-bin/php5-fcgi" stderr: PHP message: PHP  12. Project->projectFormat() /var/www/domain/models/Project.php:137` any idea ?


batch-training LSTM with pretrained & out-of-vocabulary word embeddings in keras

Question: My goal is to batch-train an RNN LSTM mode using Stochastic Gradient Descent to predict named entities from labeled text in keras. The input to my model are word-sized units. I chose to represent words using pretrained GloVe embeddings.
Since my training data comes from a specialized domain, it contains words that do not have matches in GloVe (out-of-vocabulary words). For these I generated pseudo-random embeddings by sampling from a truncated normal distribution based on summary statistics from GloVe embeddings; I also made sure that tokens of the same type have the same embedding.
Given this heterogenous dataset, the usual approach of creating a lookup table of embeddings equal to the length of the vocabulary and generating embedding batches on the fly based on the lookup table was impractical. So instead, I padded the sequences to MAX_SEQ_LEN and pre-generated my batches.
So my batch data is a NUM_BATCHES-long list of (X,y) tuples where:

X are inputs consisting of embedding matrices of shape MAX_SEQ_LEN, D], where D is the dimensionality of the embedding;
y are labels consisting of vectors of length MAX_SEQ_LEN consiting of integers of ranging from 0 (padded cell response) to NUM_CLASSES.

My question:
I am using an external loop for whole data epochs and then an internal queue within the keras model.fit() fuction to feed my batches one by one. This does not seem like the most appropriate way to do this. I wonder because my learning curve (see loss function plot) is quite jagged, even after decreasing the learning rate; I'd expected batch-training to yield smoother learning curves. What is a more appropriate way to implement batch training given my mixed embedding and fixed batch constraints?
<code>from keras.preprocessing.sequence import pad_sequences
from keras.models import Sequential
from keras.layers import LSTM, Dense, TimeDistributed, Bidirectional
from keras import optimizers

# define LSTM
model = Sequential()
model.add(Bidirectional(LSTM(HIDDEN_SIZE, return_sequences=True), input_shape=(MAX_SEQ_LEN, D)))
model.add(TimeDistributed(Dense(N_CLASSES, activation='sigmoid')))
model.compile(loss='sparse_categorical_crossentropy', optimizer=optimizers.SGD(lr=.1), metrics=['mse'])

# train LSTM
for epoch in range(1):

   for X,y in train:

        # reshape input and output data to be suitable for LSTMs
        X = X.reshape(1, MAX_SEQ_LEN, D)
        y = y.reshape(1, MAX_SEQ_LEN, 1)

        # fit model for one epoch on this sequence
        model.fit(X, y, epochs=1, batch_size=1, verbose=2)
</code>
Answer: If the batch size is small and the size of the dataset is not massive, you are likely to get jagged curves, as the gradient of the loss could be noisy. Note that noisy gradients are what makes mini-batching effective: noise helps escape local minima. That said, the plots you are showing refer to a single epoch, is it correct? If so, they are showing you how the loss behaves in a single epoch, for each data batch. During a single epoch, you process potentially different batches, so the loss might oscillate. Instead, you should see the learning curve where you plot different epochs on the x-axis and the average loss over all batches in the y-axis.


asp core 1.0.1 web api deploy

Question: I m trying to deploy a WebApi in asp.net core 1.0.1
Can any one point me the way to do it on Windows 2012 server ?
If i create a new project on vs 2015 (with the webapi template with the default valuescontroller) and publish it as is it gives a 404 error page not found.
I instaled DotNetCore.1.0.1-WindowsHosting, created amnd app pool with no managed code and added a new website to it with the content from the publish folder (created by visual studio).
What am i doing wrong?
PS: if i run the app on visual studio it works (returns a json file)
thanks in advance
Answer: I cannot add comment so I add my 2 cents here :
Suggestion to find the reason:
I try the following step (1) and it works.
Maybe you can try (2) & (3) and then you may find out reason for your issue.

Use default web site (not a new web site) with DefaultAppPool (not a new pool)
If it works, use new web site with DefaultAppPool.
If it works, the issue may be the publishing-user with wrong credential on your new folder.

hope this help

asp core 1.0.1 web api deploy


Regular expression to find all function calls with parameters in VS2010

Question: Say I have a function called myFunc(). It can take parameters. I want to find all instances of myFunc() which HAS parameters passed in. What regular expression should I use? I'm using Ctrl+Shift+F in VS2010 and have Use Regular Expressions option selected.
So for example, I want to see
myFunc( varA )
myFunc( varB, varC)
and not
myFunc()
Thanks!
Comment: Wow, I got down voted hard for this. LOL.
Answer: This is very simple but should do what you are looking for:
myFunc\([^)]+\)

basically looking for:
 - myFunc\( = myFunc(
 - [^)]+ = one or more of something besides )
 - \) = followed by a )
visually:
Comment: Thank you for the breakdown of the solution!
Answer: Here you go
myFunc\(\s*\S.*\)

The \s* checks for zero or more spaces before. \S makes sure at least one non-space is found, and .* tales everything that might also be there.
note that the parantheses is also escaped


How to prompt message box in remote computer where application is running as service?

Question: I have an C++ MFC application running in master machine that installs update from a server in remote computers through an agent service. This service takes care of authorization in remote machines before update. Now, I need to prompt the currently active user in remote machines for confirmation. AfxMessagebox or MessageBox in the service context doesn't show up any window in remote machines. Is there a way to achieve this?
Comment: Services run in session 0, and any UI shown there is not visible. You need to show the UI from a process running in a user session. Typically that means creating a helper process in the user session, and having it show the UI.
Answer: Look at WTSSendMessage():

Displays a message box on the client desktop of a specified Remote Desktop Services session.
Comment: Thank you for the reply!
Can this messagebox be customized? Or I have only the option of UINT type i.e MB_YESNO, MB_YESNOCANCEL etc. Instead of 'yes'/'no', can there be customized texts?
Comment: @user2541490 sorry, the only customizations are title, message text, and standard buttons, just like a normal MessageBox


Rational or irrational

Question: We know  that  $$0.12345\cdots = \frac{1}{10}+\frac{2}{10^2}+\frac{3}{10^3}+\frac{4}{10^4}+\frac{5}{10^5}\cdots = \sum_{k=1}^{\infty}\frac{k}{10^k}$$
prove that $0.12345\cdots$ is irrational .
I though if  let $0.1345\cdots$ be rational like proof of $\mathbb e$.
also I compute that series
NOTE :if $|x| < 1$ then   $$\sum_{k=1}^{\infty}x^k=\frac{1}{1-x}  \implies \sum_{k=1}^{\infty}kx^k= \frac{x}{(1-x)^2}$$
$$\sum_{k=1}^{\infty}\frac{k}{10^k}=\sum_{k=1}^{\infty}k\left(\frac{1}{10}\right)^k= \frac{\frac{1}{10}}{\left(1-\frac{1}{10}\right)^2}=\frac{10}{81}=0.1234567912346679\cdots$$ which is rational.
please give me hints. I don't want answer buddies.
thanks a lot.
Comment: The sum is rational, you have basically proved it. A sort of related number $0.1234567891011121314\cdots$ is irrational, indeed transcendental. The proof of irrationality is easy.
Comment: if i understand correct, you say this number is like $0.1234567891011121314\cdots$
Answer: I don't know how to give you a hint

You're talking about two different things

$0.12345678910111213\cdots$ known as $C_{10}$ (Champernowne constant) is a real transcendental
 constant
On the other hand You're talking about
$$\begin{align}
\frac{1}{10}+\frac{2}{10^2}+\frac{3}{10^3}+\frac{4}{10^4}+\frac{5}{10^5}\cdots
&=\sum_{k=1}^{\infty}\frac{k}{10^k}=\frac{10}{81}\end{align}=0.\overline{123456790}$$ which is rational and

Clearly $\displaystyle \quad \quad \quad \quad \quad \quad 0.12345678910111213\cdots\neq0.\overline{123456790}$

Since you're interested in proof of irrationality A bit of googling yields Answer by André Nicolas

Rational or irrational


Não consigo usar os pacotes do PIP

Question: Sou novo no python e com gerenciamento de pacotes, então várias dúvidas acabam surgindo, gostaria que alguém pudesse me ajudar a esclarecer.
Quando eu instalo um pacote via pip, ex: pip install opencv-python, eu simplesmente abro o CMD em admin e executo o comando.
Quando eu crio o projeto e importo a biblioteca, nesse caso import cv2, os compiladores não reconhecem; somente se eu der o pip dentro da pasta do projeto.
Gostaria de saber se realmente é para ser assim, ou a instalação deve ser de forma "global"?
O que eu estou fazendo errado?
Answer: Você pode estar passando por esses 2 problemas.
1 - A "chamada" do pacote pode estar errada, você pode ter que especificar onde está o cv2 com um "from opencv-python < (Isso daqui pode variar) import cv2".
2 - Se você estiver usando PyCharm para desenvolver o compilador que o seu PyCharm está usando é o virtual, ai ele pode não pegar o pacote que você baixou na sua maquina. Em configurações do Pycharm e até mesmo nas configurações de execução do programa você pode editar o compilador que está utilizando, se o PyCharm estiver usando o compilador virtual só aponta ele para o compilador da maquina passando o caminho de instalação do Python.
Comment: Como está escrito atualmente, sua resposta não é clara. Por favor, pode [edit] para adicionar mais detalhes, que ajudarão outras pessoas a entender como isso resolve a pergunta feita. Você pode encontrar mais informações sobre como escrever boas respostas [na Central de Ajuda](/help/how-to-answer).
Answer: Anaconda
Se você possui o IDE da anaconda e o sistema operacional Windows, você simplesmente pode executar esse seguinte código:
conda install -c https://conda.binstar.org/menpo opencv

Linux
Se você possui o sistema operacional Linux, você pode simplesmente fazer dessa maneira
pip install opencv-python

ou
conda install opencv

ou
sudo pip3 install opencv-python

Windows sem a IDE anaconda
Se você possui apenas o Windows e o Python, você pode utilizar esses seguintes comandos no terminal do CMD (obs: não pode ser no terminal do Python):
pip3 install opencv-python

ou
pip install opencv-python

Teste
Para certificar se você conseguiu instalar a biblioteca com sucesso, basta você abrir o terminal do Python e importar o módulo cv2. Caso o terminal não informe nenhum erro, deu certo

PyCharm
Se utiliza o PyCharm como IDE e importou o módulo cv2 (já instalado) e não funcionou, você tem que fazer algumas configurações na IDE para funcionar. Da uma olhada nesse site para saber como que configura a IDE para reconhecer o cv2 PyCharm não reconhece cv2
Comment: Também deve falar sobre ambientes virtuais. https://docs.python.org/pt-br/3/tutorial/venv.html e https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

Não consigo usar os pacotes do PIP


Horizontal Menu that will add drop down option if too long

Question: I'm build a site that needs to have horizontal menus and they all need to fit on one row, however if they add too many menu items we would like for the last item to convert to a drop-down with any menu items that didn't fit.
I don't want it to be a solution where "after the 5 menu" start a dropdown, I want the script  to know that the menu is to long and automatically create it when needed.
So if they put have a menu like
Short Link  -  Short Link  -  Short Link  -  Short Link  -  Short Link
it will display normal with no drop down, however if they go hog wild and add twenty items it will go like
Short Link  -  Short Link  -  Short Link  -  Short Link  -  Short Link  -  [Drop Down with 15 other links]
Would love for it to be a jquery solution but doesn't have to be.
Answer: With jQuery you can assign an event to window resize. When it hits a specific point, make the change.
<code>$(window).resize(function() {
    if ($(window).width() < whatever) {
        //Change the menu...
        //eg. $('#menu').addClass('condensed');
    }
});
</code>
You could even try it with purely CSS.
<code>@media only (max-width: whatever) {
    #menu {
        /* Override with new styling */
    }
}
</code>