Not allowed to reindex Elasticsearch?

Question: I've set my AWS Elasticsearch instance so that anyone can do anything (create, delete, search, etc.) to it.
These are my permissions (replace $myARN with my Elasticsearch ARN):
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "$myARN"
    }
  ]
}

When I PUT a new index:
 PUT http://my-elasticsearch-domain.us-west-2.es.amazonaws.com/index-name

Or I DELETE an index:
 DELETE http://my-elasticsearch-domain.us-west-2.es.amazonaws.com/index-name

I get this:
{
  "acknowledged": true
}

Which means I can create and delete indexes but when I try to POST a reindex I get:
{
  "Message": "Your request: '/_reindex' is not allowed."
}

Do I have to sign this request? Why should I have to sign this request but not creating or deleting indexes?
Answer: The reason is simply because the Amazon Elasticsearch Service is a kind of restricted environment where you don't have access to the full range of services and endpoints provided by a barebone install of Elasticsearch.
You can check the list of endpoints that you're allowed to use on the Amazon Elasticsearch Service and _reindex is not part of that list.
UPDATE
There's another way to achieve what you want, though. By leveraging Logstash, you can source the data from ES, apply any transformation you wish and sink it back to ES.
input {
  elasticsearch {
   hosts => ["my-elasticsearch-domain.us-west-2.es.amazonaws.com:80"]
   index => "index-name"
   docinfo => true
  }
}
filter {
 mutate {
  remove_field => [ "@version", "@timestamp" ]
 }
 # add other transformations here
}
output {
 elasticsearch {
   hosts => ["my-elasticsearch-domain.us-west-2.es.amazonaws.com:80"]
   manage_template => false
   index => "%{[@metadata][_index]}"
   document_type => "%{[@metadata][_type]}"
   document_id => "%{[@metadata][_id]}"
 }
}

Comment: There is another way ;) I've updated my answer to showcase another solution.
Comment: I have put this part of my project on hold for the moment. I will update when I work on it again.
Comment: Gotcha, let us know.
Answer: Reindex feature will not be available in previous versions 1.5 and 2.3. So currently if you use the versions 1.5 or 2.3, it would be good for you to move on to the latest ES version so that you will get better indexing performance and other features which are not supported in previous versions.
Also have a look into the below link to know more the APIs which are supported in different versions of AWS Elasticsearch. If you look into the 5.1 section you can the "_reindex" is listed there.
http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-supported-es-operations.html#es_version_5_1
Answer: I was able to do this using the following tool
taskrabbit/elasticsearch-dump
After installing it, you can run this on the command line:
elasticdump \
  --input=http://es.com:9200/api/search \
  --input-index=my_index \
  --output=http://es.com:9200/api/search \
  --output-index=my_index \
  --type=mapping

NOTE: I did have to use the --awsChain option to find my credentials.

Not allowed to reindex Elasticsearch?


How to compute the closure of a set of binary strings in term of the "AND" and "OR" operators?

Question: Given a set of $n$ binary strings of length $m$, how many binary strings at least should we add to the set to make sure that the binary "AND" and "OR" operators are closed on the set? The output of the problem is the number of binary strings needed to add.
For example, given the set {001, 010, 110}, we should at least add {000, 011, 111} to the set to make both "AND" and "OR" operators closed. So the output is 3.
Is there any good algorithm or idea related to the problem? Thanks a lot.
Comment: You could repeatedly add "AND" and "OR" of strings you already have. This will certainly terminate, because your strings have a certain length. Now try to optimize this approach.
Comment: But how to optimize it?
Answer: I believe you are trying to compute the "sublattice of the boolean lattice of length m" generated by your n binary strings. I don't know a fast way to compute sublattices.  In general, it is exponential in n.
Here is an example:  For any positive integer n, and any positive integer m ≥ n, start with the binary strings of length m which are zero in position i for all i > n, and which are nonzero in all but one other place.  There are exactly n such strings.  How many strings do you have to add?  2n−n, since every string with exactly one nonzero entry (in the first n positions) is an AND of n−1 of the initial strings.  This output is exponential length in n.

If you included "not" or otherwise included "xor", you would suddenly upgrade to finding "ideals in the boolean algebra" and this becomes linear algebra, and things like gaussian elimination should make the problem quadratic in n.
Of course the output itself could still be exponential in n, so you have to settle for a different, but very useful, kind of output: a basis.

As pointed out by Ross Millikan, the critical problem for a practical solution is a better data-type for the output.  Just listing all the words is going to be exponential in general, but sometimes one could produce ridiculously fast membership test functions (and product them ridiculously quickly) — in my example, the membership function simply checks that the positions from n+1 to m are all 0.
However, no data-type can work magic.  Depending on how many possible answers there are, the data-type will have to be at least a certain size. Queyranne–Tardella (2008) attempt to answer the question of how many sublattices are there and in fact study exactly the original problem.  They provide a data-type for the output and prove it is memory optimal up to a constant factor:

Queyranne, Maurice; Tardella, Fabio.
  "Sublattices of product spaces: hulls, representations and counting."
  Discrete Math. 308 (2008), no. 9, 1508–1523.
  MR2392592
  DOI:10.1016/j.disc.2007.03.080
Comment: Thanks for your information. There is something I didn't mention clearly. In the problem, the output is the least number of binary strings we need add. So we needn't output all the binary strings. In this case, is there any polynomial algorithm?
Comment: I think the paper is a little too theoretical for a good implementation though.  However, I suspect its ideas of cylinders and epigraphs can be made concrete in a simple way.  I *think* these are just generalizations of Ross's observation (+1, sorry was distracted) about the bit-implication.
Comment: @Timothy: oh yikes, that is very different.
Comment: @Timothy: just in case, my latest answer is saying that there is a way to produce a very fast membership test.  That is, there is a polynomial time algorithm which produces a polynomial time algorithm which decides if a binary string of length m can be written as a bunch of ANDs and ORs of your original n strings. — I don't know if it can be used to count the number of strings where it says yes in polynomial time.
Comment: @Timothy: the way the problem works is that, when you're concerned with closure of the set, there is no 'least'. Yes, you could add _all_ strings to satisfy that you have all possible ANDs and ORs (plus maybe a lot more. But to ask for closure, you're asking for exactly that already, _the_ subset of the universe that satisfies the criteria (adding more would be adding strings that don't fit the criteria of being 'reachable' by ANDs and ORs).
Answer: You can look for cases where one bit implies another.  In your example bit 2 (the MSB) implies bit 1-there are no examples in your given set with bit 2 on and bit 1 off.  So you can never get rid of this and 100 and 101 will not be required.  This is $n^2m$ in complexity
Added in response to comment:  You are looking for cases where bit $i$ is set in all your input words whenever bit $j$ is set.  So you could make an array $m \times m$ and initialize it to all $1$.  Go through all $n$ input words.  For each word, check if bit i=0 and bit j=1.  If so, set the $(i,j)$ element to $0$.  At the end, if any entries in the array are still $1$, having bit $j$ on means for sure bit $i$ is on in your input words.  As AND and OR cannot destroy this, none of your output words will have $i$ off and $j$ on.  As you check $m(m-1)/2 \approx m^2$ bits in $n$ input words, this is of order $m^2n$
Comment: The checking is only nnm, right?  The full algorithm cannot be better than exponential in general, since the output can be exponential.
Comment: @Jack:  you are right about how many checks it takes-I had already said that.  But the exponential may just come from the output.  As you point out, If I can find the 0 word and all words with a single 1 bit on, I can just assert that all words are in the closure.  I don't know if these checks will always find the basis.
Comment: I think you are exactly right that we need some clear idea of a basis.  I found a paper that does just that!
Comment: How did you get $n^2 m$? Can you say a little more about the idea of your algorithm? I can't fully understand it. Thanks.


Is $c \frac{\partial}{\partial x}=\frac{\partial c}{\partial x}=0$?

Question: Is $c \frac{\partial}{\partial x}=\frac{\partial c}{\partial x}=0$?
Or is one supposed to treat $c \frac{\partial}{\partial x}$ as "partially applied" operator (which expects a function).
Comment: Please think for example on $\hat{p}=-i\hbar \frac{\partial}{\partial x}$ - if this would equal zero than would be useless
Answer: I don't see how "linearity" of the derivative, that $\frac{\partial}{\partial x}(af(x)+ bg(x))= a\frac{\partial f}{\partial x}+ b\frac{\partial g}{\partial x}$, has anything to do with this. It is just a matter of notation- understanding what $c\frac{\partial}{\partial x}$ means as differential operator.
Answer: Let's have your 2 operators:

$\hat{D}=\frac{\partial}{\partial x}$
$\hat{C}=c$ -yes, in the theory can be as well

Now test on a function $f(x)$- $\hat{C}(\hat{D}f(x))$ and $\hat{D}(\hat{C}f(x))$:
1) $\hat{C}(\hat{D}f(x))= \hat{C}(\frac{\partial}{\partial x}f(x))=\hat{C}f'(x)=c\cdot f'(x)$
2) $$\hat{D}(\hat{C}f(x))=\hat{D}(c\cdot f(x))=\frac{\partial}{\partial x}(c\cdot f(x))=[\frac{\partial}{\partial x}c]\cdot f(x) +c\cdot [\frac{\partial}{\partial x}f(x)]=0\cdot f(x)+c\cdot f'(x)=c\cdot f'(x)$$
True is here $\hat{C}\cdot\hat{D}=\hat{D}\cdot\hat{C}$ but not $c\cdot\hat{D}=\hat{D}\cdot c$
Answer: If c is a number , yeah thats right
We know that Differentiate of every number relative to x is equal to 0 so the second side is equal to 0 .
(differentiate of c and 1 relative to x is 0) so in the left side we have : [c(0)=0]
So yeah the equation is true.
Attention: my original language is not english and im not proffesional on it . so excuse me if i have mistake in my writing or in my words)
Comment: Shirin Sari has, I think, misunderstood the question.  It is true that the derivative of a constant is 0 but "$c\frac{\partial}{\partial x}$" is NOT "$\frac{\partial c}{\partial x}$.  $c\frac{\partial}{\partial x}$ is an **operator** to be applied to some function.
Comment: @user247327 Okay so the linearity of differentation applies only when one applies that to a function?


How to count web-site previews in Whatsapp etc?

Question: The problem is:
1.  The message, which I want to send to a list of recipients in WhatsApp, contains the link to a web-page.
2.  The number of recipients is going to be more than a hundred (typically, a group of students).
3.  I need to keep records how many recipients received this message by counting the number of page previews or in some other way. I don't need to count IPs, or collect their telephone numbers or names. I need to get just a number of people received it.
As I understand, WhatsApp client application only once generates the web-page preview, and then puts it into a cache in order to show it when necessary. That means there is no way to count the web-pages previews on a server side. Am I correct?
Is there another way to count the receipts of the message?
Answer: I see two different approaches to check who has received your link.
Use WhatsApp Read Receipts
You can easily see who has read your message in WhatsApp group chats by the blue check marks or the message info as described in their faq
Checking the access log of your web server to see how many clients have accessed your site to generate a rich link preview
However your title says you want to count how many times the website has been previewed on different devices.
I assume you have access to the access log of the web server hosting the website.
Then you can check how often WhatsApp clients created the rich link preview of your site by counting the number of client whose user-agent contained WhatsApp.
If I send a link to my website in a WhatsApp chat I can see entries like this in my web servers access log:
domain.tld 0 22.214.171.124 - - [01/Sep/2018:09:35:10 +0200] "GET / HTTP/1.1" 200 5928 "-" "WhatsApp/2.18.163 A"

You said you want to know on how many devices the preview has been displayed. You can try to distinguish between the clients by their ip addresses, but that solution isn't perfect since the ip of a device can change frequently.
Count how many times the preview of the website has been generated in php without access to the web server logs
You can also determine whether a user visits your site a web browser or if it is just WhatsApp generating the preview in php.
It is done by checking whether the user agent contains "WhatApp" or not.
if (strpos($_SERVER['HTTP_USER_AGENT'], 'WhatsApp') !== false) {
    increasePreviewCount();
}

The problem of distinguishing between different clients remains with this approach.
Comment: Thanks for your reply. I haven't been clear from the beginning, sorry. What I want to do is to form a link with request string, like: http://example.com?coupon=197234&id=972345 and I need to be able to record the number of previews of that code. I do not have access to log files, as the site is on hosting servers.
Comment: Of course, there is a way of doing smth like this: https://hackerone.com/reports/309367
but there might be another way to do so
Comment: Okay, I have updated my answer and described how to count the times the preview has been generated in php. The problem of telling apart different devices remains.
Comment: Thanks again. The thing is that it works once and does not go further, as if php server keeps generated preview in cache. I was trying to disable php cache, but it did not work in preview either.


Assinar xml Nfse paulistana

Question: Eu fiz um programa em C# que assina o XML em lote da nota legal de Porto Alegre. Agora a empresa em que trabalho abriu uma filial em São Paulo e preciso assinar o XML em lote da nota fiscal paulistana para declarar serviços prestados. Tentei usar o mesmo programa que fiz e obtive um erro:

"Malformed reference element."

Reparei algumas diferenças entre os xmls. O xml de São Paulo não tem o atributo id nem nas tags rps nem no lote. O atributo id é obrigatório para o elemento reference que computa a assinatura. Tentei setar manualmente o id, mas deu o mesmo erro.
Outra diferença que notei é que na nota legal de Porto Alegre há elementos "InfRps", e cada um precisa de um elemento signature, com todos os dados de como foi feita a assinatura, como o algoritmo, etc. O lote também precisa do elemento signature. Na nota fiscal paulistana, o lote no fim do arquivo precisa o elemento signature, mas os elementos RPS que equivalem aos "InfRps" da nota legal de Porto Alegre contém apenas uma tag assinatura, com uma codificação em base64 da assinatura do elemento RPS. Não sei ainda como gerar essa assinatura.
Desde já agradeço qualquer ajuda.
Abaixo há um exemplo de como deve ficar o lote de xml da nota fiscal paulistana já assinado:
<?xml version="1.0" encoding="UTF-8"?>
<PedidoEnvioLoteRPS xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"                         xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://www.prefeitura.sp.gov.br/nfe">
<Cabecalho Versao="1" xmlns="">
<CPFCNPJRemetente>
  <CNPJ>04642554000143</CNPJ>
</CPFCNPJRemetente>
<transacao>false</transacao>
<dtInicio>2007-01-20</dtInicio>
<dtFim>2007-01-20</dtFim>
<QtdRPS>2</QtdRPS>
<ValorTotalServicos>2000</ValorTotalServicos>
<ValorTotalDeducoes>200</ValorTotalDeducoes>
</Cabecalho>
<RPS xmlns="">
       <Assinatura>ro6Og7L5BMPpYZKXfSSITNe8U9C4a95P9dIspX8R0Okg1CRsA87N08Llaq+q6IVOGLhjGMkAELBIkQ8T0BGR3czgtJAuOcgMlOmGZlgziWAg3Kww3pFD+rYxE+DRgL2M5QXTaZEYwVNj0lZyJpt1nS9LHUqO7PB+ivYRk+ewUVA=</Assinatura>
<ChaveRPS>
  <InscricaoPrestador>31000000</InscricaoPrestador>
  <SerieRPS>LLLLL</SerieRPS>
  <NumeroRPS>1</NumeroRPS>
</ChaveRPS>
<TipoRPS>RPS-M</TipoRPS>
<DataEmissao>2007-01-20</DataEmissao>
<StatusRPS>N</StatusRPS>
<TributacaoRPS>T</TributacaoRPS>
<ValorServicos>1000</ValorServicos>
<ValorDeducoes>100</ValorDeducoes>
<CodigoServico>2658</CodigoServico>
<AliquotaServicos>0.05</AliquotaServicos>
<ISSRetido>false</ISSRetido>
<CPFCNPJTomador>
  <CPF>18564071878</CPF>
</CPFCNPJTomador>
<RazaoSocialTomador>PATRICIA TUPYNAMBA</RazaoSocialTomador>
<EnderecoTomador>
  <TipoLogradouro>R</TipoLogradouro>
  <Logradouro>Guararapes</Logradouro>
  <NumeroEndereco>495 A</NumeroEndereco>
  <ComplementoEndereco>ap 102</ComplementoEndereco>
  <Bairro>BROOKLIN PAULISTA</Bairro>
  <Cidade>3550308</Cidade>
  <UF>SP</UF>
  <CEP>4561000</CEP>
</EnderecoTomador>
<EmailTomador>firstname.lastname@example.com</EmailTomador>
<Discriminacao>Desenvolvimento de Web Site Pessoal</Discriminacao>
</RPS>
<RPS xmlns="">
            <Assinatura>jkTucEqcj8Qkes8RalHZW+p6Zc6weeqnPabOK7IHSkyVTO9reNqpMclEh8n6nHXSPGcn   svhQx4JWx/sK26IxVTOMKp3i41/1w3OyC9SvL0VPtzrARKQMEqTf2kqwu9skHMzw5d4T/jfjYLrp5n/Y    AbMqKQ5zaa4qAcY0A066OAc=</Assinatura>
<ChaveRPS>
  <InscricaoPrestador>31000000</InscricaoPrestador>
  <SerieRPS>LLLLL</SerieRPS>
  <NumeroRPS>2</NumeroRPS>
</ChaveRPS>
<TipoRPS>RPS-M</TipoRPS>
<DataEmissao>2007-01-20</DataEmissao>
<StatusRPS>N</StatusRPS>
<TributacaoRPS>T</TributacaoRPS>
<ValorServicos>1000</ValorServicos>
<ValorDeducoes>100</ValorDeducoes>
<ValorPIS>10</ValorPIS>
<ValorCOFINS>20</ValorCOFINS>
<ValorINSS>30</ValorINSS>
<ValorIR>40</ValorIR>
<ValorCSLL>50</ValorCSLL>
<CodigoServico>2658</CodigoServico>
<AliquotaServicos>0.05</AliquotaServicos>
<ISSRetido>false</ISSRetido>
<CPFCNPJTomador>
  <CNPJ>06215299000188</CNPJ>
</CPFCNPJTomador>
        <RazaoSocialTomador>Grixco Soluções</RazaoSocialTomador>
<EnderecoTomador>
  <TipoLogradouro>R</TipoLogradouro>
  <Logradouro>Augusta</Logradouro>
  <NumeroEndereco>01642</NumeroEndereco>
  <ComplementoEndereco>3 Andar</ComplementoEndereco>
  <Bairro>Cerqueira Cesar</Bairro>
  <Cidade>3550308</Cidade>
  <UF>SP</UF>
  <CEP>1304001</CEP>
</EnderecoTomador>
<Discriminacao>Manutenção de Sistema</Discriminacao>
</RPS>
<Signature xmlns="http://www.w3.org/2000/09/xmldsig#">
<SignedInfo>
  <CanonicalizationMethod Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315" />
  <SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1" />
  <Reference URI="">
    <Transforms>
      <Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature" />
      <Transform Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315" />
    </Transforms>
    <DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />
    <DigestValue>AkHyCjCwkANg3aRAnltAXR1YQ4c=</DigestValue>
  </Reference>
</SignedInfo>
<SignatureValue>IkLB0qfZLDuTNXNB83tXXsZ2TFNK9X0l7gq8jRCOcwhit059iF5gNHfmuM4NoUhyhZ+rC6UGn9lSMv1A35lofsplIuWUJO13yPtHsxaY6/rP9DTB4Ve3ihzwrEkpenANoEU1C5wLenX0lRtYc1k3fWeDmZUvv+b/M81pwoPBL8k=</SignatureValue>
<KeyInfo>
  <X509Data>
    <X509Certificate>MIIFUzCCBDugAwIBAgIQSUJS8pELZyjasDkgGzKm0TANBgkqhkiG9w0BAQUFADBuMQswCQYDVQQGEwJCUjETMBEGA1UEChMKSUNQLUJyYXNpbDEsMCoGA1UECxMjU2VjcmV0YXJpYSBkYSBSZWNlaXRhIEZlZGVyYWwgLSBTUkYxHDAaBgNVBAMTE0FDIENlcnRpU2lnbiBTUkYgVjMwHhcNMDYwNzE5MDAwMDAwWhcNMDkwNzE4MjM1OTU5WjCB1DELMAkGA1UEBhMCQlIxEzARBgNVBAoUCklDUC1CcmFzaWwxKjAoBgNVBAsTIVNlY3JldGFyaWEgZGEgUmVjZWl0YSBGZWRlcmFsLVNSRjETMBEGA1UECxQKU1JGIGUtQ05QSjELMAkGA1UECBMCUkoxFzAVBgNVBAcUDlJJTyBERSBKQU5FSVJPMUkwRwYDVQQDE0BUSVBMQU4gQ09OU1VMVE9SSUEgRSBTRVJWSUNPUyBFTSBJTkZPUk1BVElDQSBMVERBOjA0NjQyNTU0MDAwMTQzMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCx86LAoJRVmtQMzmtdWpyNgKy200+bwjtz/TuywNcTjvfw7qHFGIgTjipmuZ3zhX28CgYLYXp3tj1Dfh2B7EhjHdLJPfvoF4MgbN/dQGXmGpMpF5cNxYusOGCZiyASvI7Gqt/xE4xLSIalNr6kF6CaPLkpFgTNNe+WQkG0fMqsQQIDAQABo4ICCDCCAgQwgbEGA1UdEQSBqTCBpqA/BgVgTAEDBKA2DDQyNDA3MTk3NjA3MTM4NTM3Nzg2MDAwMDAwMDAwMDAwMDAwMDAwOTI5OTA2MjFDTkggIFJKoB8GBWBMAQMCoBYMFEZFUk5BTkRPIFNJTFZBIEJSQUdBoBkGBWBMAQMDoBAMDjA0NjQyNTU0MDAwMTQzoBEGBWBMAQMHoAgMBjIzOTU0OIEUZmJyYWdhQHRpcGxhbi5jb20uYnIwCQYDVR0TBAIwADBiBgNVHR8EWzBZMFegVaBThlFodHRwOi8vaWNwLWJyYXNpbC5jZXJ0aXNpZ24uY29tLmJyL3JlcG9zaXRvcmlvL2xjci9BQ0NlcnRpU2lnblNSRlYzL0xhdGVzdENSTC5jcmwwHwYDVR0jBBgwFoAU9p1ZXf6/xXLN3c7ELmYbLu4Iz3YwDgYDVR0PAQH/BAQDAgXgMFUGA1UdIAROMEwwSgYGYEwBAgMGMEAwPgYIKwYBBQUHAgEWMmh0dHA6Ly9pY3AtYnJhc2lsLmNlcnRpc2lnbi5jb20uYnIvcmVwb3NpdG9yaW8vZHBjMB0GA1UdJQQWMBQGCCsGAQUFBwMEBggrBgEFBQcDAjA4BggrBgEFBQcBAQQsMCowKAYIKwYBBQUHMAGGHGh0dHA6Ly9vY3NwLmNlcnRpc2lnbi5jb20uYnIwDQYJKoZIhvcNAQEFBQADggEBAC5w/CBXAykvPSbBGf+u0UPcWVJATL2ix0hCfNUVtHaCjMz8hRjgYqmhpefzDm2LCTvoCPzG6XQBYxAmnDhX1f/gyjHz+E1xJg451qtqcyCJ9861o9R2bHd4zR0DuyxCNGOTiYJ4Gc/Xa4xqECorAx5ktkk1T/HOc1K/ntRGpdL+llsO/jqSRmTOnRgdeNHcKkyXsOgL5BwxxgGNuIyqirgGXW0by4Io1GnSXtixxfvEOnqOicxBY6AcVS9HHuhmOBYiK9skAUp0Sm2v41hpsC8uIkfUeRxsJIp2CNZ4DjoyfmKwNLMCRZQAKpwMXyyHZlX1a4o/9iGTszNeeShw61g=</X509Certificate>
  </X509Data>
</KeyInfo>
</Signature>
</PedidoEnvioLoteRPS>

Segue abaixo o programa que fiz para assinar a nota legal de Porto Alegre:
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;
using System.Security.Cryptography.Xml;
using System.Text;
using System.Threading.Tasks;
using System.Xml;
using System.Xml.Linq;

namespace escolhercertificadosimples
{
    public static class DocumentExtensions
    {
        public static XmlDocument ToXmlDocument(this XDocument xDocument)
        {
            var xmlDocument = new XmlDocument();
            using (var xmlReader = xDocument.CreateReader())
            {
                xmlDocument.Load(xmlReader);
            }
            return xmlDocument;
        }

        public static XDocument ToXDocument(this XmlDocument xmlDocument)
        {
            using (var nodeReader = new XmlNodeReader(xmlDocument))
            {
                nodeReader.MoveToContent();
                return XDocument.Load(nodeReader);
            }
        }
    }

    class Program
    {
        static void Main(string[] args)
        {
            //Abre o deposito de certificados da maquina local
            var store = new X509Store(StoreName.My, StoreLocation.CurrentUser);
            //var store = new X509Store(StoreName.My, StoreLocation.LocalMachine);
            store.Open(OpenFlags.ReadOnly);
            //Abre a tela de escolher certificado
            var selectedCertificate = X509Certificate2UI.SelectFromCollection(
                store.Certificates,
                "Title",
                "MSG",
                X509SelectionFlag.SingleSelection);

            //Pega o objeto x509 do certificado selecionado
            foreach (X509Certificate2 x509 in selectedCertificate)
            {
                try
                {
                    //Imprime informações do certificado, se existirem
                    byte[] rawdata = x509.RawData;
                    //Console.WriteLine("Content Type: {0}{1}", X509Certificate2.GetCertContentType(rawdata), Environment.NewLine);
                    //Console.WriteLine("Friendly Name: {0}{1}", x509.FriendlyName, Environment.NewLine);
                    //Console.WriteLine("Certificate Verified?: {0}{1}", x509.Verify(), Environment.NewLine);
                    //Console.WriteLine("Simple Name: {0}{1}", x509.GetNameInfo(X509NameType.SimpleName, true), Environment.NewLine);
                    //Console.WriteLine("Signature Algorithm: {0}{1}", x509.SignatureAlgorithm.FriendlyName, Environment.NewLine);
                    //Console.WriteLine("Private Key: {0}{1}", x509.PrivateKey.ToXmlString(false), Environment.NewLine);
                    //Console.WriteLine("Public Key: {0}{1}", x509.PublicKey.Key.ToXmlString(false), Environment.NewLine);
                    //Console.WriteLine("Certificate Archived?: {0}{1}", x509.Archived, Environment.NewLine);
                    //Console.WriteLine("Length of Raw Data: {0}{1}", x509.RawData.Length, Environment.NewLine);
                    //X509Certificate2UI.DisplayCertificate(x509);
                    //x509.Reset();

                    //==============================
                    // Comeca a ler os arquivos xml
                    //==============================
                    var txtFiles = Directory.EnumerateFiles("./", "*.xml");
                    //Console.WriteLine(txtFiles.Count());break;
                    foreach (string currentFile in txtFiles)
                    {

                        Console.WriteLine("Lendo o arquivo" + currentFile + ":");

                        var originalDoc = XDocument.Load(currentFile);

                        //string id = xe4.FirstAttribute.Value.ToString();
                        //Console.WriteLine(id);
                        //Console.WriteLine(xe4.ToString());

                        XmlDocument doc = DocumentExtensions.ToXmlDocument(originalDoc);

                        //====================================
                        // Comeca a Criacao da pasta Assinados
                        //====================================
                        // Specify the directory you want to manipulate.
                        string path_originais = "Originais";

                        try
                        {
                            // Determine whether the directory exists.
                            if (Directory.Exists(path_originais))
                            {
                                //Console.WriteLine("That path exists already.");
                                //Console.WriteLine("Pasta Assinados já existe.");
                                //return;
                            }
                            else
                            {

                                // Try to create the directory.
                                DirectoryInfo di = Directory.CreateDirectory(path_originais);
                                Console.WriteLine("The directory was created successfully at {0}.", Directory.GetCreationTime(path_originais));
                            }
                            // Delete the directory.
                            //di.Delete();
                            //Console.WriteLine("The directory was deleted successfully.");
                        }
                        catch (Exception e)
                        {
                            Console.WriteLine("The process failed: {0}", e.ToString());
                        }
                        finally { }

                        //doc.Save(path + "/teste.xml"); //break;

                        //Salva o novo arquivo assinado na pasta Assinados
                        doc.Save(path_originais + "/" + currentFile.Replace("./", "")); //break;

                        //==============================
                        // Comeca a Leitura das Notas
                        //==============================
                        XmlNodeList ListInfRps = doc.GetElementsByTagName("RPS");
                        //Console.WriteLine(ListInfRps.Count); break;

                        //Console.WriteLine("Aqui");

                        //Console.WriteLine(ListInfNFe);

                        //SignedXml signedXml = new SignedXml(doc);

                        //Console.WriteLine(ListInfRps.Count);

                        int NodeCounter = 1;

                        foreach (XmlElement InfRps in ListInfRps)
                        {
                            Console.Write("Assinando Nota " + NodeCounter + "...") ;
                            //Console.WriteLine(InfRps.InnerXml);
                            //continue;

                            //Console.WriteLine(InfRps.Value);

                            //string id = InfRps.Attributes.GetNamedItem("Id").Value;
                            //Console.WriteLine(id);
                            string id = "lote";

                            SignedXml signedXml = new SignedXml(InfRps);

                            signedXml.SigningKey = x509.PrivateKey;
                            //RSACryptoServiceProvider privateKey = x509.PrivateKey as RSACryptoServiceProvider;

                            // Transformações p/ DigestValue da Nota
                            //Reference reference = new Reference("#" + id);
                            Reference reference = new Reference("#" + id.Replace(":",""));
                            //Console.WriteLine("#" + id.Replace(":", ""));
                            //Reference reference = new Reference("#" + "lote");

                            reference.AddTransform(new XmlDsigEnvelopedSignatureTransform());

                            reference.AddTransform(new XmlDsigC14NTransform());

                            signedXml.AddReference(reference);

                            KeyInfo keyInfo = new KeyInfo();
                            keyInfo.AddClause(new KeyInfoX509Data(x509));
                            signedXml.KeyInfo = keyInfo;

                            signedXml.ComputeSignature();

                            XmlElement xmlSignature = doc.CreateElement("Signature", "http://www.w3.org/2000/09/xmldsig#");

                            XmlAttribute attr = doc.CreateAttribute("Id");
                            attr.Value = "Ass_" + id;

                            //Add the attribute to the node
                            xmlSignature.Attributes.SetNamedItem(attr);

                            XmlElement xmlSignedInfo = signedXml.SignedInfo.GetXml();
                            XmlElement xmlKeyInfo = signedXml.KeyInfo.GetXml();

                            XmlElement xmlSignatureValue = doc.CreateElement("SignatureValue", xmlSignature.NamespaceURI);
                            string signBase64 = Convert.ToBase64String(signedXml.Signature.SignatureValue);
                            XmlText text = doc.CreateTextNode(signBase64);
                            xmlSignatureValue.AppendChild(text);
                            //xmlSignature.AppendChild(xmlSignatureValue);

                            xmlSignature.AppendChild(doc.ImportNode(xmlSignedInfo, true));

                            xmlSignature.AppendChild(xmlSignatureValue);

                            xmlSignature.AppendChild(doc.ImportNode(xmlKeyInfo, true));

                            //InfRps.AppendChild(xmlSignature);

                            XmlNodeList ListRps = doc.GetElementsByTagName("Rps");
                            //Console.WriteLine(ListInfRps.Count); break;

                            //Console.WriteLine("Aqui");

                            //Console.WriteLine(ListInfNFe);

                            //SignedXml signedXml = new SignedXml(doc);

                            int RpsCounter = 1;

                            foreach (XmlElement Rps in ListRps)
                            {

                                //Console.WriteLine(Rps.InnerXml); return;
                                //Console.WriteLine(RpsCounter);
                                //Console.WriteLine(NodeCounter);

                                if (RpsCounter == NodeCounter)
                                {
                                    Rps.AppendChild(xmlSignature);
                                }
                                RpsCounter++;
                            }

                            Console.WriteLine("Ok");
                            NodeCounter++;
                        }

                        //==============================
                        // Comeca a Leitura do Lote
                        //==============================
                        XmlNodeList ListLoteRps = doc.GetElementsByTagName("PedidoEnvioLoteRPS");
                        //Console.WriteLine(ListInfRps.Count); break;

                        //Console.WriteLine("Aqui");

                        //Console.WriteLine(ListInfNFe);

                        //SignedXml signedXml = new SignedXml(doc);

                        foreach (XmlElement LoteRps in ListLoteRps)
                        {
                            Console.Write("Assinando Lote...");

                            //Console.WriteLine("Aquiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii");
                            //Console.WriteLine(InfRps.InnerXml);
                            //continue;

                            //Console.WriteLine(InfRps.Value);

                            string id = LoteRps.Attributes.GetNamedItem("Id").Value;
                            //Console.WriteLine(id);

                            SignedXml signedXml = new SignedXml(LoteRps);
                            //Console.WriteLine("1");
                            signedXml.SigningKey = x509.PrivateKey;
                            //RSACryptoServiceProvider privateKey = x509.PrivateKey as RSACryptoServiceProvider;
                            //Console.WriteLine("2");
                            // Transformações p/ DigestValue da Nota
                            Reference reference = new Reference("#" + id);
                            //Reference reference = new Reference("#" + "lote");
                            //Console.WriteLine("3");
                            reference.AddTransform(new XmlDsigEnvelopedSignatureTransform());
                            //Console.WriteLine("4");
                            reference.AddTransform(new XmlDsigC14NTransform());
                            //Console.WriteLine("5");
                            signedXml.AddReference(reference);

                            //Console.WriteLine("6");

                            KeyInfo keyInfo = new KeyInfo();
                            keyInfo.AddClause(new KeyInfoX509Data(x509));
                            signedXml.KeyInfo = keyInfo;

                            //Console.WriteLine("7");

                            signedXml.ComputeSignature();

                            //Console.WriteLine("8");

                            XmlElement xmlSignature = doc.CreateElement("Signature", "http://www.w3.org/2000/09/xmldsig#");

                            XmlAttribute attr = doc.CreateAttribute("Id");
                            attr.Value = "Ass_"+id;

                            //Add the attribute to the node
                            xmlSignature.Attributes.SetNamedItem(attr);

                            XmlElement xmlSignedInfo = signedXml.SignedInfo.GetXml();
                            XmlElement xmlKeyInfo = signedXml.KeyInfo.GetXml();

                            //Console.WriteLine("9");

                            XmlElement xmlSignatureValue = doc.CreateElement("SignatureValue", xmlSignature.NamespaceURI);

                            string signBase64 = Convert.ToBase64String(signedXml.Signature.SignatureValue);
                            XmlText text = doc.CreateTextNode(signBase64);
                            xmlSignatureValue.AppendChild(text);
                            //xmlSignature.AppendChild(xmlSignatureValue);

                            //Console.WriteLine("10");

                            xmlSignature.AppendChild(doc.ImportNode(xmlSignedInfo, true));

                            xmlSignature.AppendChild(xmlSignatureValue);

                            xmlSignature.AppendChild(doc.ImportNode(xmlKeyInfo, true));

                            //LoteRps.AppendChild(xmlSignature);

                            XmlNodeList ListEnviarLoteRpsEnvio = doc.GetElementsByTagName("EnviarLoteRpsEnvio");
                            //Console.WriteLine(ListInfRps.Count); break;

                            //Console.WriteLine("Aqui");

                            //Console.WriteLine(ListInfNFe);

                            //SignedXml signedXml = new SignedXml(doc);

                            foreach (XmlElement EnviarLoteRpsEnvio in ListEnviarLoteRpsEnvio)
                            {
                                EnviarLoteRpsEnvio.AppendChild(xmlSignature);
                            }

                            Console.WriteLine("Ok");

                            //Console.WriteLine(InfRps.InnerXml);

                            //XmlNode root = doc.DocumentElement;
                            //XmlNode myNode = root.SelectSingleNode("descendant::LoteRps");
                            //myNode.AppendChild(xmlSignature);

                            //====================================
                            // Comeca a Criacao da pasta Assinados
                            //====================================
                            // Specify the directory you want to manipulate.
                            string path = "Assinados";

                            try
                            {
                                // Determine whether the directory exists.
                                if (Directory.Exists(path))
                                {
                                    //Console.WriteLine("That path exists already.");
                                    //Console.WriteLine("Pasta Assinados já existe.");
                                    //return;
                                }
                                else
                                {

                                    // Try to create the directory.
                                    DirectoryInfo di = Directory.CreateDirectory(path);
                                    Console.WriteLine("The directory was created successfully at {0}.", Directory.GetCreationTime(path));
                                }
                                // Delete the directory.
                                //di.Delete();
                                //Console.WriteLine("The directory was deleted successfully.");
                            }
                            catch (Exception e)
                            {
                                Console.WriteLine("The process failed: {0}", e.ToString());
                            }
                            finally { }

                            //doc.Save(path + "/teste.xml"); //break;

                            //Salva o novo arquivo assinado na pasta Assinados
                            doc.Save(path + "/" + currentFile.Replace("./","")); //break;

                            //Console.WriteLine("11");

                            //Encoding enc = Encoding.UTF8;
                            //MemoryStream ms = new MemoryStream();
                            //XmlTextWriter writer = new XmlTextWriter(ms, enc);

                            //Console.WriteLine("12");

                            //writer.Formatting = Formatting.Indented;
                            //writer.Indentation = 2;
                            //writer.Namespaces = true;

                            //Console.WriteLine("13");

                            //doc.WriteTo(writer);
                            //writer.Flush();
                            //writer.Close();

                            //Console.WriteLine("Aquiiiiiiiiiiiiiiiiiiiii");
                            //Console.WriteLine(writer.Namespaces);

                            //XmlWriterSettings oSettings = new XmlWriterSettings();
                            //oSettings.Indent = true;
                            //oSettings.OmitXmlDeclaration = false;
                            //oSettings.Encoding = Encoding.ASCII;

                            //XmlWriter writer1 = XmlWriter.Create("xmlfile.xml", oSettings);
                            //System.Xml.XmlElement xml = InfRps;
                            //xml.WriteContentTo(writer1);

                            //signedXml = new SignedXml(infNFe);
                            //signedXml.SigningKey = ObtemCertificado().PrivateKey;
                        }

                        //signedXml.SigningKey = x509.PrivateKey;
                        //signedXml.SignatureMethod;

                        //Console.WriteLine(signedXml.SignatureMethod);
                        //Console.WriteLine(signedXml.Signature);
                        //break;
                    }
                }
                catch (CryptographicException ce)
                {
                    Console.WriteLine(ce.ToString());
                    Console.WriteLine("Information could not be written out for this certificate.");
                }
            }
        }
    }
}

Answer: Tive esse mesmo problema.
Para resolver basta passar o ID em branco.
Por padrão o URI no reference vem NULL basta passar string.Empty e tudo fica certo.

Assinar xml Nfse paulistana


Linear Algebra Practice Problems

Question: I am currently watching a lecture series on Youtube by Dr. Gilbert Strang, and it's been great so far. The only issue that I'm running into is that, without practicing the problems, I have no way to really gauge if I'm really understanding the material, and the concepts aren't sticking as well.
Specifically, we are currently covering the four major subspaces of linear algebra: the column space, the null space, the row space, and the left null space. One of the ideas in this chapter is to be able to find the basis for each of these spaces given a certain mxn matrix, and determining whether or not columns or rows are linearly independent.
I mostly understand the concepts themselves, but I really need to practice to be sure, and to get it to stick. I've tried searching google for practice problems, as well as past questions on this site, and so far I'm not really finding what I need. Would anyone happen to know a good source of practice problems for subspace bases and linear independence of vectors for matrices at the first year (undergraduate) linear algebra level? Thanks, I would appreciate it.
Comment: Strang has multiple textbooks on linear algebra; I'm sure they will have plenty of problems and probably will align fairly well with his lecture series.
Comment: Thanks! Now that you mention it, looking back at lecture 1 of the online course, he lists the book on his blackboard at the beginning of the class.
Comment: Notably, there is his _Linear Algebra and Its Applications_.  I used to have a copy and liked it.  Used it for reference.   There was an emphasis on applications,  as the title suggests.
Answer: Here is a practical way to find the four subspaces you mention. By finding I mean: find a basis for them. (Because if we know a basis for a subspace, then we know that every other vector in the subspace is a linear combination of the basis vectors).
I am assuming you know that every linear function $f:{\mathbb R}^n \longrightarrow {\mathbb R}^m$ is given by an $m\times n$ matrix $A$ in the sense that $f(x)=Ax$ for every $x$ in ${\mathbb R}^n$. So assume we have such a matrix $A$.
For the column space: compute $A^T$ (the transpose), then do row operations until $A^T$ is in Row Echelon Form. Then the non-zero rows of the Row Echelon Form will be a basis for the Column Space (that is the same thing as the range of the linear function).
For the Null Space: put $A$ in Row Echelon Form. Remember that the leading entry of a row is the first non-zero entry of that row (if there is one). In the Row Echelon Form of $A$, each column that has no pivot(=no entry that is a leading entry of some row) corresponds to a free variable. If you find no free variables, the null space is the just the zero space (only has the zero vector in it, which means the function if one-to-one). If you find $k$ free variables, that means the Null Space has dimension $k$ (meaning: there will be $k$ vectors in a basis for it). To find the $k$ vectors for a specific basis, proceed as follows: set one free variable to $1$, and all others to zero, and solve the homogeneous system corresponding to the Row Echelon Form. The answer will be a vector in a basis for the Null Space. Now set another free variable to $1$ and all the others to $0$, and solve the linear system again, to get a second vector in a basis. So on, until you finish all the free variables, and get $k$ vectors for a basis for the null space.
As for the Row Space, it's just like the column space, but you work on $A$ instead of $A^T$.
The left Null Space is the same thing as the Null Space of $A^T$, so again switch $A$ with $A^T$.
To practice, try the three matrices below. Find a basis for the four subspaces in each case.
$A=\left(\begin{array}{ccc} -1 & 1 & 1 \\ 3 & -1 & 0 \\ 2 & -4 & -5 \end{array} \right)$.
$B=\left(\begin{array}{ccc} 1 & 2 & 3 \\ 2 & 5 & 3 \\ 1 & 0 & 8 \end{array} \right)$.
$C=\left(\begin{array}{ccc} -1 & 1 & 2 \\ -2 & 2 & 4 \\ 3 & -3 & -6 \end{array} \right)$.

Linear Algebra Practice Problems


Azure Function calls itself after 3 minutes

Question: i have the following code in my azure function with 5 minutes manual timeout.

when i run the above function in azure, i see the function creates a new instance after 3 minutes.(check the below image)

both the instances completes successfully ,but returns  Status: 504 Gateway Timeout which in turn fails my function execution.
i have hosted the function in App Service Plan, and also increased the timeout in host.json file to 10 minutes
<code>"functionTimeout": "00:10:00"
</code>
Comment: @ABaig as mag_zbc suggests, please do not post bitmaps of code and function output. This makes it much harder to work with. e.g. no easy way to grab the invocation IDs without manually copying them.
Comment: How is the trigger defined?
Comment: i have used http trigger for the function.
Comment: Also, clarify how you are invoking the function, and what makes you believe that the function is calling itself.
Comment: Do not post code as images, ever. Please edit your question, remove the images and add the code as text. [Why may I not upload images of code on SO when asking a question?](https://meta.stackoverflow.com/q/285551/1288408)
Comment: @David Ebbo i used the image instead of code as it was giving me syntax error for my code making it unable to save. thanks for the suggestion will keep it in my checklist henceforth.
Answer: Several questions in here:

Timeouts - The function timeout in host.json applies to the underlying function runtime; not the http pipeline. You should not have an http function running longer than a minute. The http calls will timeout independently of the runtime (as you see with the 504). However, you could use that timeout for a long-running (ie, 60 minute on appservice plan) queue trigger. If you need a long-running function, the http call could queue a message for a queue trigger, or you could use the Durable Function support.
Why is it invoking again? The simplest interpretation here is that your function is just receiving a second http request message. Do you have evidence that's not the case? You could bind to the HttpRequestMessage and log additional http request properties to track this down.
Comment: should i use a different trigger for my function?can you suggest any(actually need to implement azure functions for my long running web-jobs. above function was a sample i was trying.) and about invoking again, i am testing the function in azure portal, hence it is invoked only once manually.


Showing if there is statistical significance between two data sets (data analysis)

Question: Hey guys so I have been given a homework problem where there is a table of video views with one column being views(in millions) before an event and then the other column is views after the event. The question asks to make meaningful data analysis insights. I made some graphs and such comparing the two and also did a t test showing if it was statistically different or not. Can anyone think of any other tests or anything I could do for insights? Thanks
Comment: What graphs did you do?  When you say "graphs and such" it's hard to offer further suggestions in that direction, because we don't know what you've done already.
Answer: You could show and maybe compare distributions of the data. You can show it with histograms, density curves, kernel density etc. And you can assess the normality of distributions, skewness, homoscedasticity and potential outliers.
Answer: Comment: It is not clear exactly what additional input you are looking for. Here
are a few thoughts.
I put the first six Before/After pairs into Minitab. (Typing
them all would have been too tedious.) Dif is Aft - Bef so that all of these differences are positive.
Data Display

Row   Bef   Aft   Dif
  1  1.19  4.36  3.17
  2  1.60  4.01  2.41
  3  1.28  5.08  3.80
  4  1.59  4.46  2.87
  5  1.77  4.55  2.78
  6  1.51  4.18  2.67

There is no reason to suspect that these differences are other than normal.
A t test of the differences rejects the null hypothesis that
there the differences center around $0$ with a P-value less than $0.0005,$ shown in the printout as 0.000.
One-Sample T: Dif

Test of μ = 0 vs ≠ 0

Variable  N   Mean  StDev  SE Mean      95% CI          T      P
Dif       6  2.950  0.485    0.198  (2.441, 3.459)  14.90  0.000

Just scanning down the entire dataset, it seems that the 'After' values
are mainly larger then corresponding 'Before' values---except for a few
pairs at the very end.
Roughly speaking, this is confirmed by the paired t test on the first six pairs.
It might be interesting to investigate why the last
few pairs are different from the rest in this regard.
(If you plot After (y) vs Before (x), do you see a trend?
If so, a regression might show that 'After' can be roughly
predicted by 'Before'.)
I would not
expect any further insights from a paired t-test on the whole dataset; the P-value may be a little larger (but I guess still significant at the 5% level).
If you have further questions please post the entire dataset
in computer-readable format (actual text, not a picture). And ask specific questions by editing your original post.
For example:
Bef = (1.19, 1.60, 1.28, 1.59, 1.77, 1.51, ..., 5.06)
Aft = (4.36, ...                                3.38)


How to handle JS serialized item in PHP?

Question: Hey guys!
I am having a bit of trouble handling the data from a submitted form that is serialized with JS.
In the form there a textarea fields which users can input any text they want but when they enter & like "apples & oranges", I have trouble handling that.  As we know that JS serializes & to %26 and spaces with a "+" sign.
So now back in PHP, I am trying to put this data into the database and call it back out elsewhere but for the life of me don't know how to handle the special characters properly.
Note that I am also php exploding the data at the "&" sign to make it into an array, then is put into the database as an array.  I tried several methods like urldecode, htmlentities but those don't help because remember I need to explode at "&".
I am serializing in JS because its an AJAX submitting form.
Comment: i think you might want to convert it to Json before sending over http post/get
Comment: Can you not urldecode() and then explode()? or explode() with %26 as the delimiter?
Comment: Why are you serialising with JS?
Comment: Can you show us, how you serialize your data? I think, you could completely bypass this problem if you would build a proper POST request.
Comment: Sorry guys, I should have added I am AJAX submitting the form therefore I am serializing it.
Comment: I think nobody here will be able to help you, unless you post some of your code. For normally what you are describing shouldn't be a problem at all, if handled correctly.
Answer: What about url*decode*?
Comment: No that would not work because it changes it into "&" and I need the "&" to explode from...
Answer: parse_str()
Answer: If something is serialized in JS and then sent to a PHP script - it can only be via GET or POST method, which in return means that the serialized data will be stored in one of the following arrays: $_GET if method was GET, $_POST if method was POST and $_REQUEST regardless of the method (so you can capture both POST and GET variables with a single array).
That means that you shouldn't have your custom function that creates from and to php array since it's already done for you.
If you want to store the array in string format, you should serialize that array before storing it into the db and then unserialize it when you want to use it in array format after pulling it out from db.
Answer: I would use a double replace to prevent using a loop.
First I would replace all & to '####-####' or any other code that is highly unlikely to appear (or use chr(0)).
Then use urldecode() to replace all %26 to & and all + to ' '
Last explode on '####-####' instead of the &
Comment: Thanks Hugo your comment gave me an idea..I use what you said and then I used urldecode() after and it did the trick...
Answer: You can use urldecode to get the text representation of those encoded characters. I doubt that JS is doing this by itself though.
Comment: No that would not work because it changes it into "&" and I need the "&" to explode from...


The AIC for my Poisson Regression is INF. Does it mean i shouldn't use Poisson regression for my model?

Question: I am trying to create a regression model for this variable (Y) based on 2 categorical variables. So, I created dummy variables to replace them. These dummy variables (i.e. int_collab, Q1,Q2,Q3) have values of 1 and 0.
Y is a double, with values ranging from 0 to 348.19, and about 10% of it has values of 0. It follows Poisson distribution.
But, when I model it:
glm(Y ~ int_collab + Q1 + Q2 + Q3, data = capdata,
                 family=poisson(link="log"))

It returned INF for the AIC value. I am guessing something is wrong? Is it because i am not supposed to use Poisson to model this variable?
I have been reading online, but it confused me even further. It seems i should consider using either Negative Binomial or Quasi Poisson (as the variance greater than mean), etc ... Any pointer on which distribution would be more suitable would be much appreciated!
Comment: Y is an indicator called FWCI. It is calculated by taking the ratio between a publication citations with the average number of citations received by all other similar publications. I am trying to see if any of the predictors (journal quality ~ quartile 1 to quartile 4 and international collaboration) has any affect on this FWCI.
Answer: Here's your problem:

Y is a double, with values ranging from 0 to 348.19 ...  It follows [a] Poisson distribution

You should never use a Poisson distribution for non-integer response values.
The AIC is based on the negative log-likelihood, which in turn is based on the log probability of the observed values given the model.  The probability of a non-integer value is zero, so the log-likelihood is -Inf, so the negative log-likelihood is Inf.  If you have even a single non-integer value in your data set, this will happen. As an example, here's how R would calculate the negative log-likelihood of $Y=2.3$ for a Poisson distribution with $\lambda$ (mean) equal to 2.3:
-dpois(2.3,lambda=2.3,log=TRUE)

this is Inf, and also produces a warning:

In dpois(2.3, lambda = 2.3, log = TRUE) : non-integer x = 2.300000

You should have gotten a bunch of warnings of this type: they may have been hidden in a catch-all phrase like There were 20 warnings (use warnings() to see them).
If you want to analyze $x/Y$ where $x$ is an integer count and $Y$ is a known value, you should look up the use of offsets, i.e. glm(x~ ... + offset(log(Y)) ...
Comment: And if you want variance proportional to mean, but not a Poisson distribution, use family="quasipoisson".  There will not be any AIC (which is correct, since there is no likelihood).
Answer: I think you have a dummy variable problem or singularities. As your variables are dummies you can not use all in the regression, otherwise there is a problem of identification. See the summary of the results, one of the Qi coefficients would be NA. Also the warnings said that the function does not produce integer values.
One solution is to retire one of the Qi dummies, the one you want as a base of comparison (the glm function actually omits one) and use another link function. I tried quasipoisson (see here a discussion on it) and get the results. But of course, it depends on your data.
Here I generated same data and I was able to replicate the same AIC = Inf problem.
FWCI <- c(c(18,17,15,20,10,20,25,13,12)/1.2,c(18,17,15,20,10,20,25,13,12)/0.8)
#int_collab, Q1,Q2,Q3
no=length(FWCI)
int_collab=rep(c(0,1),length.out=no)
Qs <- gl(3,1,no,labels=c("Q1","Q2","Q3")) #factor

glmp <- glm(FWCI ~ int_collab + I(Qs=="Q1")+I(Qs=="Q2")+I(Qs=="Q3"), family = poisson(link="log"))
glmp

#Call:  glm(formula = FWCI ~ int_collab + I(Qs == "Q1") + I(Qs == "Q2") +
#    I(Qs == "Q3"), family = poisson(link = "log"))
#
#Coefficients:
#      (Intercept)         int_collab  I(Qs == "Q1")TRUE  I(Qs == "Q2")TRUE  I(Qs == "Q3")TRUE
#          2.77893            0.02667            0.29299           -0.16127                 NA
#
#Degrees of Freedom: 17 Total (i.e. Null);  14 Residual
#Null Deviance:     34.63
#Residual Deviance: 23.21   AIC: Inf

#summary(glmp)

glmr <- glm(FWCI ~ int_collab + Qs, family = quasipoisson())

glmr

#Call:  glm(formula = FWCI ~ int_collab + Qs, family = quasipoisson())
#
#Coefficients:
#(Intercept)   int_collab         QsQ2         QsQ3
#    3.07192      0.02667     -0.45426     -0.29299
#
#Degrees of Freedom: 17 Total (i.e. Null);  14 Residual
#Null Deviance:     34.63
#Residual Deviance: 23.21   AIC: NA

#anova(glmr, test = "F")
#summary(glmr)


Are algebraic groups defined over $\mathbb{R}$ Lie groups?

Question: In the notes I am reading, which is about algebraic groups, in the section about over $\mathbb{R}$ all the sudden they started using the word Lie groups. I understand Lie groups and algebraic groups are related but I am finding this section of the notes confusing.
If I have $G$ a linear algebraic group defined over $\mathbb{Q}$ say, and if I consider $G(\mathbb{R})$ the real points, does it automatically become a Lie group (with the Zariski topology)?
Comment: The Zariski topology is not Hausdorff, so we do not have a Lie group.
Comment: More generally, the Zariski topology is way too coarse for analytic endeavours
Comment: A connected linear algebraic group is a Lie group in the Euclidean topology. I think only a finite group would be a Lie group in the Zariski topology.
Answer: If $G$ is a linear algebraic group over $\mathbb{R}$, (it is automatically smooth and) $G(\mathbb{R})$ can be viewed as a polynomial subset of some affine space $\mathbb{R}^n$. For the usual topology on $\mathbb{R}^n$ (and not the Zariski topology), $G(\mathbb{R})$ is a smooth sub-manifold, and since the multiplication and inverse on $G(\mathbb{R})$ are given by polynomials, they are smooth, so $G(\mathbb{R})$ is a real Lie group in the sens of differential geometry. But I think there are real Lie groups that don't come this way.
It is the same thing for complex algebraic linear groups.
Comment: (+1) Two things. First, there are Lie groups that don't come this way (e.g. the universal cover of $\mathrm{SL}_2(\mathbb{R})$. Secondly, note that $G(\mathbb{R})$ being even relatively well-behaved is hard. For example, the fact that $G(\mathbb{R})$ has finitely many connected components is not even obvious!
Comment: Writing $G$ as $\mathrm{Spec}(\mathbb{R}[X_1,\cdots,X_n]/(P_1,\cdots,P_m))$ ($G$ is a closed subvariety of an affine space), the set of $\mathbb{R}$-points of $G$ is $\mathrm{Hom}_{\mathbb{R}-alg}(\mathbb{R}[X_1,\cdots,X_n]/(P_1,\cdots,P_m), \mathbb{R})$ which can be identified with $\lbrace x=(x_1,\cdots,x_n) \in \mathbb{R}^n \, | \, P_1(x)=\cdots=P_n(x)=0 \rbrace \subset \mathbb{R}^n$.
Comment: Thank you for your answer! One of the things I'm still not getting is how one puts Euclidean topology on $G(\mathbb{R})$... The points of linear algebraic groups correspond to prime ideals of some ring (at least in this notes I'm reading). The maximal ideals correspond to the "actual points" in $\mathbb{R}^n$, I think, so what happens to the other points? How do you put Euclidean topology when $G(\mathbb{R})$ contains "other points"? (or do we just ignore the points that do not correspond to maximal ideals for this purpose of viewing it as a Lie group?)
Comment: $\def\RR{\mathbb{R}}$If $A$ is a commutative $\RR$-algebra and $X = \text{Spec}(A)$, then $X(\RR)$ is the set of $\RR$-algebra homomorphisms from $A$ to $\RR$. Each $f \in A$ can be thought of as a function $X(\RR) \to \RR$, and we topologize $X(\RR)$ by taking as a basis of open sets $\{ x \in X(\RR) : |f(x)| < \epsilon \}$ for $f \in A$ and $\epsilon > 0$. If $A = \RR[x_1, x_2, \ldots, x_n]/(p_1, p_2, \ldots, p_m)$, then  $X(\RR)$ is the solutions to $p_1=p_2=\cdots=p_m=0$ in $\RR^n$, with the topology induced from the Euclidean topology on $\RR^n$.
Comment: For example, if $A = \mathbb{R}[x,y]/(x^2+y^2-1)$, then $X(\mathbb{R})$ is the circle $x^2+y^2=1$ in $\mathbb{R}^2$, with its usual topology. The fact that $A$ has maximal ideals that don't correspond to points of $X(\mathbb{R})$ (for example $\langle x-2, y^2+3 \rangle$) is irrelevant.


The website is way too slow Error while sending QUERY packet. PID=11597 in /home/******/public_html/sites/all/modules/apdqc/apdqc.mysql.inc

Question: all the pages on the website are taking more than 1 minutes to be displayed, and sometimes they are showing Connection Reset,
I had a previous issue which caused the server to be down once I clear the cache / enable a module and to solve it I have added the Asynchronous Prefetch Database Query Cache Module, I have even switched to a new Server.
Error that's being logged is the following:
[27-Sep-2016 14:55:10 UTC] PHP Warning:  Error while sending QUERY packet. PID=11602 in /home/****/public_html/includes/database/database.inc on line 2204
[27-Sep-2016 14:55:10 UTC] PHP Warning:  Error while sending QUERY packet. PID=11592 in /home/****/public_html/sites/all/modules/apdqc/apdqc.mysql.inc on line 871
[27-Sep-2016 14:55:10 UTC] PHP Warning:  Error while sending QUERY packet. PID=11592 in /home/****/public_html/includes/database/database.inc on line 2204
[27-Sep-2016 14:55:10 UTC] PHP Warning:  Error while sending QUERY packet. PID=11593 in /home/****/public_html/includes/database/database.inc on line 2204
[27-Sep-2016 14:55:10 UTC] PHP Warning:  Error while sending QUERY packet. PID=11597 in /home/****/public_html/sites/all/modules/apdqc/apdqc.mysql.inc on line 871
[27-Sep-2016 14:55:10 UTC] PHP Warning:  Error while sending QUERY packet. PID=11597 in /home/****/public_html/includes/database/database.inc on line 2204

Setting / specification of server and website are right below:

Drupal 7
PHP Version 5.6.x
realpath_cache_size  150k
realpath_cache_ttl   4000
memory_limit    1200M
Apache 2.4
MYSQL DB Server Version 5.6.X

$conf['cache_backends'][] = 'sites/all/modules/apdqc/apdqc.cache.inc';
$conf['cache_default_class'] = 'APDQCache';
$databases['default']['default']['unix_socket'] =
  '/var/lib/mysql/mysql.sock';
$databases['default']['default']['host'] = NULL;
$conf['lock_inc'] = 'sites/all/modules/apdqc/apdqc.lock.inc';
$databases['default']['default']['init_commands']['innodb_lock_wait_timeout']
  = "SET SESSION innodb_lock_wait_timeout = 100";
$databases['default']['default']['init_commands']['wait_timeout'] =
  "SET SESSION wait_timeout = 1000";
$databases['default']['default']['init_commands']['read_rnd_buffer_size']
  = "SET SESSION read_rnd_buffer_size = 8388608";
$conf['session_inc'] = 'sites/all/modules/apdqc/apdqc.session.inc';
$databases['default']['default']['init_commands']['isolation'] = "SET
  SESSION tx_isolation='READ-COMMITTED'";
$databases['default']['default']['init_commands']['join_buffer_size']
  = "SET SESSION join_buffer_size = 8388608";

in an Average day, we have almost 300 concurrent users.
the Process Manager is showing a huge CPU % usage from the following COMMAND
/usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/lib/mysql/****.*******.***.err --open-files-limit=10000 --pid-file=/var/lib/mysql/****.*******.***.pid

What I want is to know how to track what is the issue here? and why there's this huge of CPU Usage, and how to stop the Connection Reset of the pages / elements inside the website?
PS: I have disabled the ERROR logging, access log, no devel module is added, the cron is set to be run every day at midnight, my status report has zero errors/notices related to apqdc and there's an Expire caching module, in addition to XMLSITEMAP, and Metatags
[edit]

Model: HP - DL180 G9
Processor: 2x Intel Hexa-Core Xeon E5-2620v3 [ 6 Core(s) ]
RAM: 64 GB
HDD: 2x 480GB
Linux CENTOS 7.2
WHM version 58.0

my.cnf information:
core file size          (blocks, -c) 1000000
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 256494
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 14335
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

default-storage-engine=MyISAM
innodb_file_per_table=1
performance-schema=0
max_allowed_packet=268435456
open_files_limit=107000
innodb_buffer_pool_size=5G
innodb_checksum_algorithm=crc32
innodb_flush_method=O_DIRECT
innodb_read_io_threads=50
innodb_spin_wait_delay=30
innodb_sync_spin_loops=200
innodb_write_io_threads=20
metadata_locks_hash_instances=256
binlog_format=ROW
innodb_flush_log_at_trx_commit=2
innodb_autoinc_lock_mode=2
table_open_cache_instances=250
table_open_cache=10000
thread_cache_size=100
max_user_connections=800
max_connections=7000
innodb_lock_wait_timeout=25
wait_timeout=600
read_rnd_buffer_size=8388608
join_buffer_size=8388608
query_cache_size=2000000

php.ini information:
 max_execution_time = 200
max_input_time = 120
memory_limit = 250M

Comment: This problem was solved, by hosting provider, where they have re installed the whole web service Apache, which fixed all the performance issues on the website. thank you all for your support.
Answer: Reduce the php memory limit to no more than 512M. Depending on the version of PHP you have, anything higher (even 512) is ineffective and can cause issues.
Secondly, completely uninstall apdqc module, and see if that alleviates the issue.
Third, profile your application with xhprof and/or New Relic and/or Blackfire.io.
Without knowing more about your setup, you might want to review the issue queue for APDQC module to narrow down what the issue could be:
https://www.drupal.org/project/issues/apdqc?categories=All
Comment: Thanks for your reply, I will try to follow the above steps, but have one concern, that's before `apdqc` module, the whole server was going down because of metadata lock issue, and this module fixed this issue for me , maybe I have missed some configuration, while adding the module , the module owner is  @mikeytown2
Comment: what kind of more setup information do you need?
Comment: This question looks very similar. http://drupal.stackexchange.com/a/108673/57
Comment: how is it similar? the module is already installed and the deadlock was removed, and the solution is already integrated in my website, my problem is that I have a lot of an ERR_CONNECTION_RESET, and then if I refreshed the page the page will take up 60 seconds to work.
Comment: You'll need to profile that yourself though @RafahNehmeh, it can't be done from here. All we can give you are generic, broad suggestions that may or may not be relevant; it's why this type of question isn't really suited to Q+A
Answer: The php errors you listed happen in core and in apdqc. This php bug report points to the max_allowed_packet being too small. The php memory limit shouldn't cause the issue you are describing. Another cause could be a low wait_timeout value, but you are setting that to 1000.
If you can move the 'init_commands' from the settings.php into your my.conf file that has a small chance of improving your issue. You can remove them and then go to the status report page and apdqc should tell you what to add to my.conf.
In terms of getting to the bottom of this issue; knowing the stacktrace of what caused this is key. This core patch is a must: Add backtrace to all errors and add $conf['error_level'] = 2; to your settings.php file. If you have syslog use that, if not re-enable the dblog in order to log this issue.
Comment: Thanks :) , I will follow the steps and update you, but I think it was related to a module called advagg for compressing and bundling JS and CSS, because once it was disabled the website is back to normal, I am still keeping an eye on the mysql CPU Usage,and I will definitely follow your mentioned steps. Thanks again @miketown2
Comment: Hey, I having in the log error a lot of the following sentence `[core:error] [pid 25305] [client 188.8.131.52:39378] AH00126: Invalid URI in request Connection: Close`
Comment: @RafahNehmeh all that is saying is that the settings used for connecting to the db is failing.
Comment: I guess it's something to go with the actual configuration that I am using, it's not optimized, I will update my questions to list all the specification of the server that I am using in addition to my.cnf and php.ini max values, Can you confirm if it's actually optimized or should be updated?
Comment: The APDQC status report covers most of the basic things that need to be adjusted in my.conf. `Invalid URI` my guess is this is an issue with the db connection configuration, which is usually inside of your setting.php file (don't post the contents of it for security reasons). How did tracking down your original issue go `PHP Warning:  Error while sending QUERY packet`? Do you have a backtrace of the error now?
Comment: it was fixed directly by increasing the size of the Packet, the backtrack didn't provide me with the exact error.
Comment: I am following up with hosting provider, the Load on the Server has decreased ( from 6.X to 3.X) same for the CPU Usage ( from 134.X to 64.X) since I increased the size of the QUERY CACHE, that's why I think it's a server configuration issue, and not a Drupal or plugin issue, Thanks for your help.


overall Information of content

Question: I have created content on 30/8/15 and the same created content is edited by other user on 1/9/15, when I go check in the dashboard or if i create view and see the edited only i could see, but in my case i want to see overall like when the content is created and by whom its created and also by whom edited?on what date for that particular content I want to see it in a view. how to achieve this please can any one suggest me?
Answer: First if you haven't enabled revisions for your content type, please enable like below:
admin/structure/types/manage/[your-content-type]

After that you need to create view to show revisions like below:

This will help you to achieve all information of event on node.

Thanks!
Comment: if i follow this i could see all the created content node right? i want only for specific content type. i want to track only one content type not all, thanks in advance
Comment: You can enable revisions for your required content type. And In view pass filter criteria "content type" for which you want to track changes.
Comment: :do i need to select the filter condition as content type? well i want to track the user also ,that who all are editing the content.
Comment: I have added one more image please have look. You can try adding required fields and filters accordingly. Further you can explore relationships of author and user fields etc basic controls available in views. Thanks!
Comment: Have look at https://www.drupal.org/project/diff  module helps to track the difference between revisions as well.
Comment: Thank u so much, and also sorry that i keep on asking you, but i want to capture the user also that  who are editing are adding the content along with the date. please i am not getting it please share that along with this how to track the user....sorry again for asking
Comment: not a problem, I have updated image with help of that you can set view and track the user user who have made changes to your node :)
Comment: Thanks i am close to my requirement. i will try again with other methods and update you on monday.):):):):

overall Information of content


Modulation Techniques in 'Sword Art Online - Swordland (Main Theme)'

Question: In Sword Art Online - Swordland (Main Theme) (https://musescore.com/user/13887436/scores/4364901), it goes through a series of modulations and was wondering how the modulations work and how they prepare for it.
It begins in E minor and from what I can tell, beings to prepare at measure 9 to a modulation to C#m that happens in measure 11. Measure 9 contains the chords F#m7, D#dim, back to F#m7 and measure 10 is G# which is the V chord of C#m.

My question is why does the F#m work as a preparation? Was there something before that set this up? Or are we just simply saying "Oh, we're in E major temporarily now and doing a ii -V chord progression?"
Answer: Let me start with the first question: all those chords in bars 9 and 10 iv -> ii -> V7 -> i cadence in c# minor.  This cadence adds the sharps that would change e minor to E Major, so it feels like a lift to E major, landing on its relative minor, c#.
Bar 19
There are some similarities here.  In bar 9-11 we moved from e-minor to c#-minor, down a minor 3rd.  In bar 18-19, that's happening again: c# minor moving down a minor third again to b♭.  But the difference is that in 9-11, the cadence strongly led to the NEW key, while in 18-19, it very strongly leads to the old key, c#.  This isn't really a modulation at all-- it's quite jarring, and deliberately so.
The same thing happens again in bars 25-27: a big set-up for a landing on e♭ minor, but instead jumping down to c minor.  So in each of those cases, they are landing a minor 3rd down from where you'd expect.
Imagine a movie trailer with a lot of sudden cut scenes.  This is kind of the musical equivalent of that-- a series of musical cut scenes.
So what holds it together?  The constant repetition of a couple of rhythmic / melodic devices in the right hand: first a 3-3-2 division of 8, and later, then a kind of buhhhh-buh-buh-buhhhh in bar 11, which has a 6/8 feel and can be counted 1, (2)-and-a, 1, (2)-and-a. . .  In both rhythmic devices, the scales mostly move in a very similar way: holllld-step-step-holllllld.
Comment: A couple of clarifications regarding your mm. 9-10 analysis: 1) Do you mean iv/ii ("four of two"), or do you mean there is a iv chord and a ii chord? 2) I don't understand your reference to E minor and E major; it's just a V chord in C# minor, isn't it?
Comment: (1)  Because they're 7ths and the doubled notes can just be thought of as passing tones, then I'm not too picky about whether to label that as iv->ii or a kind of conglomerate.  The important thing is that it's part of a cadence in c#.  (2)  e minor would normally like to modulate to C rather than C#.  So in this case, there's an implied shift to E major.  Since the composer repeats this pattern throughout the piece WITHOUT modulating, I think it's worth pointing that out.
Comment: (1) I think we might be misunderstanding each other "iv/ii" (four of two; i.e., an applied/secondary chord) would be G#min (equivalent to the minor v chord in C# minor), but I don't see G#min anywhere. If you mean "there is a mixture of iv chords and ii chords",  consider changing the notation so as not to confuse other readers.
Answer: My question is why does the F#m work as a preparation?
Although the right-hand has an apparent F#m chord, when the left-hand is included, the chord is revealed as a D# half-diminished chord. D#m7b5 (half diminished) is the ii chord in the key of C# minor. It's followed in the next measure (m. 10) by G# major, which, of course, is the V chord. So, measures 9-11 form a ii-V-i progression in C# minor.
Note that the E in m. 9 is a suspension from the previous measure, and is not part of the overall harmony of m. 9.
Answer: A simple answer is - yes, the ii V I sequence is so strong that jumping straight into a ii V I in a new, unrelated key is a common device.  (It leads us into C♯ minor, not E major, here.)


Set MaximumDate in DatePicker to CurrentCulture Property

Question: I have a TextBlock in which a date is getting shown which is set to CurrentCulture property. Now below that I have a DatePicker in which I have to disable all future dates so I have to set its maximum date to ProcessingDate. Please tell me how to do it? Thanks in advance.
 <TextBlock x:Name="ProcessingDate" FontSize="10" Text="{Binding ProcessingDate, StringFormat={}{0:d}, ConverterCulture={x:Static gl:CultureInfo.CurrentCulture}}"
                               ToolTip="Processing Date"/>

   <my:DatePicker   Name="dpStartDte" SelectedValue="{Binding Path=DataContext.CONTRACT.CONTRACTSTARTDTE, UpdateSourceTrigger=LostFocus}" MaximumDate = 'WHAT TO WRITE' />

Answer: Instead of maximum Data, you could use  DisplayDateEnd Which makes  any date that's outside of the specifided DateRange doesn't even appear.
XAML:
   <DatePicker DisplayDateStart="12/7/1941" DisplayDateEnd="8/14/1945" />

Add this style to your Resources and set the Culture,

Comment: But how to generalize it? I will have to set DisplayDateEnd to the date set through CultureInfo.CurrentCulture
Comment: CultureInfo ci = new CultureInfo(Thread.CurrentThread.CurrentCulture.Name);
 dp.DateTimeFormat.ShortDatePattern = "dd/MM/yyyy";
 dp.DateTimeFormat.DateSeparator = "/";
 Thread.CurrentThread.CurrentCulture = dp;
Comment: I am using your earlier approach. But it is throwing an exception. The vale in CONTRACTSTARTDTE is 9/19/2044 12:00:00 AM. This is line of code: dpStartDte.DisplayDateEnd = Controller.DataContext.CONTRACT.CONTRACTSTARTDTE;
Comment: what is the exception
Comment: The calling thread cannot access this object because a different thread owns it.
Comment: I have used an alternate way to do this but thanks a lot for your help.


Pass variable to Linux 'awk'

Question: How can I pass a previously defined variable to awk in bash? For example,
hostname="server1"
cat /etc/passwd | awk -F":" '{print $1 " of $hostname"}
And the expected output is
root of server1
sys of server1
......
......  // for all user accounts

But what I get is
root of $hostname
sys of $hostname

Answer: awk has a built in variable option with -v.
cat /etc/passwd | awk -F":" -v hostname="server1" '{print $1 " of " hostname}

Answer: You have a quoting problem:

Enclosing characters in single quotes (''') preserves the literal value of each character within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash.

(from Bash Reference Manual).
All you need to do is to place $hostname outside the single quotes:
cat /etc/passwd | awk -F":" '{print $1 " of '$hostname'"}'

or use double quotes and escape some characters with \:
cat /etc/passwd | awk -F":" "{print \$1 \" of $hostname\"}"

Pass variable to Linux 'awk'


Need an algorithm to resize an organization chart to fit window

Question: Context:
I need to be able to render any organization hierarchy tree onto a web page of any given dimension and then, using a headless browser, convert the page to a PDF, while keeping the text legible so that large companies can print it on large print outs or zoom in to see it.
End goal:

Company can create a hierarchy of their organization through an app
Visually preview it through the web interface
Save to a PDF (and optionally print a physical copy)

This allows them to have a single source of truth and to visually represent their organization's tree.  Some organizations would like to define a large pdf size (eg, 24" x 36") and be able to take their PDF on a flash drive to Kinkos and print it and display it in their office.
Note: There is no upper limit on the depth or width of an organization.  Currently have companies that are 300+ across and 5 levels deep.  So that is why I need an approach that will scale.
Current Approach:
Currently, I'm taking a brute force approach by scaling the font-size of the parent container.

Set font-size on chart element to 0
Create a while loop and increment the font-size by some increment (ex. 0.1px)
Let the browser redraw the chart
Check if it's overflowing
if not overflowing, increment & repeat, else, decrement by increment amount and exit loop

Large charts need finer increments, otherwise they don't render at all.
Example, say the font-size increment is 0.5.  If a chart is so large 0.5px is overflowing, the current script will increment - see it's overflowing - then revert to 0, meaning it's not displayed.
If I lower the increment, then most smaller charts render time will increase drastically.
Data structure:
<code>Seat: {
   name: 'Seat Name',
   children: Array<Seat>,
}
</code>
Prototype:
https://github.com/danfoust/org-chart-prototype
Desired Solution
I could feed a recursive data structure & window dimensions to a function and it could efficiently determine the best font-size to fit the generated DOM in the window.
If this is possible with CSS, that would also be great.  I'm using Puppeteer & by extension Chromium, so if there's some latest and greatest CSS magic, that would be awesome.
Comment: What is the point in wanting it to fit it in the viewport? If the company is big you won't be able to fit in an iPhone 5S viewport AND keep it readable. Working on a good pdf)html format with well thought referencing/links will likely be a better solution.
Comment: I think you didn't get my point. When you have 10000 people in a company, you can't print it on a single page. Or maybe your case is only applicable on small/medium size company? If a file is enough then as I mentioned before think about a structure of links/references that will enable your html/pdf file to redirect users on appropriate sub categories or upper categories
Comment: You can improve the running time of your current solution with an [exponential search](https://en.wikipedia.org/wiki/Exponential_search). Start with a font size of 1. If the chart fits, keep doubling the font size until it doesn't fit. Then you have a lower and upper bound on the font size, and you can do a binary search to find the largest font size that fits. (If font size 1 doesn't fit, then the lower bound is 0 and the upper bound is 1, and you do a binary search within that range.)
Comment: The use case is not mobile devices.  These are enterprise clients.  They want to be able to save the output to a file and print it off on a large print out or share the file with employees and view on desktop.
Answer: I followed @user3386109's recommendation and found the upper bound then did a binary search.  For 300 seats, this cut the time to find in half.
<code>'use strict';

const $ = (selector) => document.querySelector(selector);
const $$ = (selector) => document.querySelectorAll(selector);

// Prevent loop from blocking render for 3 ms
async function isOutOfBounds(chart, w, h) {
  return new Promise((res) => {
    setTimeout(() => res(chart.scrollWidth > w || chart.scrollHeight > h), 3);
  });
}

///////////////////////////////////////////////////////////////////////////////////

const scrollbarPX = 16; // Prevent empty last page

const screenWidth = window.innerWidth - 1;
const screenHeight = window.innerHeight - scrollbarPX;

const headHeight = $('.header').scrollHeight;
const footHeight = $('.footer').scrollHeight;

const maxChartHeight = screenHeight - headHeight - footHeight;

(async () => {
  // Set each page to window dimensions
  const pages = $$('.page');
  for (const page of pages) {
    page.style.width = `${screenWidth}px`;
    page.style.height = `${screenHeight}px`;
  }

  //////////////////////////////////////////////////////////////////////

  const cache = {};

  const charts = $$('.chart');
  for (const chart of charts) {
      let cacheKey = `${chart.scrollWidth}x${chart.scrollHeight}`;
      if (cache[cacheKey]) {
          chart.style.fontSize = `${cache[cacheKey]}px`;

          // Handle rare case where same sized charts render differently at same font-size
          if (!await isOutOfBounds(chart, screenWidth, maxChartHeight)) {
             continue;
          }
      }

      //////////////////////////////////////////////////////////////////

      let max = 1;

      while (1) { // Get upper bound
          chart.style.fontSize = `${max}px`;

          if (await isOutOfBounds(chart, screenWidth, maxChartHeight)) break;

          max *= 2;
      }

      let mid;

      while (1) { // Use binary search
          mid = max / 2;

          chart.style.fontSize = `${mid}px`;

          if (!(await isOutOfBounds(chart, screenWidth, maxChartHeight))) break;

          max = mid;
      }

      // The binary search takes the first number that works
      // which may not be the best number.  Let's try scaling
      // up by a fraction of the working number.
      let incr = mid / 10;
      let size = mid + incr;
      while (1) {
         chart.style.fontSize = `${size}px`;

          if (await isOutOfBounds(chart, screenWidth, maxChartHeight)) {
              chart.style.fontSize = `${size - incr}px`;
              cache[cacheKey] = size - incr;
              break;
          }

          size += incr;
      }
    }

    // Using waitForSelector with Puppeteer
    $('body').classList.add('complete');
})();
</code>


Não consigo solucionar um objetivo usando ":hover". O que fazer?

Question: Estou fazendo um site institucional e é um tipo de agência de web-design, só que é mais pra aprender.
Então na HOME eu fiz vários bloquinhos com links para as páginas dos serviços. E em cada bloco minha intenção é fazer que ao passar o mouse por cima, apareça uma breve descrição antes do usuário clicar e ir até a página.
Eu consegui (em partes) fazer isso de duas maneiras (nas duas algo deu errado):
1º maneira - sobrepondo duas carreiras de blocos (a de cima com os ícones e títulos e a de baixo com as descrições) e quando passa o mouse em cima de algum bloco de cima ele muda o height para 0% fazendo aparecer o bloco com a descrição em baixo. Problema: qualquer movimento no mouse faz a classe sair do :hover (logicamente porque o elemento some e o mouse sai de cima).
2º maneira - só uma carreira de blocos. desse jeito a minha intenção era fazer assim: o bloco está lá com o ícone e o titulo em forma de background-image e a descrição lá (em texto dentro da div) em display:none, ai no :hover o background-image passa a ser none e descrição vira block. Problema Dessa maneira não deu certo; eu passo o mouse por cima da div e não muda o display, tentei com visibility e também não funcionou. (deixei a parte em questão visível só pra galera ver mas tá comentado no código).
Da segunda maneira eu poderia fazer a descrição em forma de imagem, daria certo: apenas mudar o background-image. Mas prejudicaria no SEO.
Também tentei com JavaScript e não consegui.
Comment: Veja http://meta.pt.stackoverflow.com/questions/297/quando-se-deve-colocar-o-nome-da-linguagem-no-t%C3%ADtulo e http://meta.pt.stackoverflow.com/questions/846/sauda%C3%A7%C3%B5es-e-agradecimentos. Tente ir direto ao assunto, faça sua pergunta de forma objetiva, aí sobra mais tempo e tecladas para caprichar na ortografia e gramática.
Comment: Obrigado, sempre escorrego nesses mesmos erros.
Answer: Solução genérica:
Procurei fazer só o essencial do CSS, e "encavalei" o quadro com informações justamente para você testar o hover. Basta alterar as dimensões e posições para o layout desejado, pois essa solução é bem genérica e adaptável.
Se quiser que as duas coisas apareçam exatamente no mesmo espaço, basta colocar a largura e altura do span igual a do bloco pai.
O "segredo" da solução é por o conteúdo alternativo dentro do elemento que vai "sofrer" o hover, assim, quando ele aparece, o hover continua ativo.
(Lembre-se de ver a 2ª demonstração, lá no fim do post, que é mais completa)

a.bloquinho {
  display: block;
  position: relative;
  width: 200px;
  height: 20px;
}
.bloquinho span {
  display: none;
  position: absolute;
  top: 0;   /* ajustar como desejado */
  left: 20px; /* ajustar como desejado */
  background: #eee;
  z-index: 10;
}
.bloquinho:hover span {
  display: block;
}
Link 1
   descrição do link bla bla bla bla bla bla bla

Link 2
   descrição do link bla bla bla bla bla bla bla

Link 3
   descrição do link bla bla bla bla bla bla bla

Link 4
   descrição do link bla bla bla bla bla bla bla

Adaptando ao exemplo linkado:
Se preferir o efeito de "cortina", basta usar o span cercando o link em vez de cercar o conteúdo:

a.bloquinho {
  float:left;
  margin-right: 20px;
  display: block;
  position: relative;
  width:100px;
  height:150px;
  background: #eee;
}

.bloquinho span {
  display: block;
  position: absolute;
  top: 0;
  left: 0px;
  width:100px;
  height:150px;
  -webkit-transition: height .5s;
  transition: height .5s;
  z-index: 10;
  overflow:hidden;
}

.bloquinho:hover span {
  height: 0;
}

  descrição do link bla bla bla bla bla bla bla

  descrição do link bla bla bla bla bla bla bla

  descrição do link bla bla bla bla bla bla bla

  descrição do link bla bla bla bla bla bla bla

Comment: Como vc mesmo você mencionou, se eu coloco o hover no _span_, ele deixa de estar em hover quando some. Desta forma, mesmo o _span_ sumindo, o "bloquinho" vai continuar em _hover_.
Comment: Tudo isso porque eu não conhecia essa forma de usar o :hover ".bloquinho:hover span ". É similar a acessar uma variável... Mas por que o :hover não funcionou sendo colocado diretamente no span?
Comment: Sim, eu já entendi 100%.