When combining p-values, why not just averaging?

Question: I recently learned about Fisher's method to combine p-values. This is based on the fact that p-value under the null follows a uniform distribution, and that $$-2\sum_{i=1}^n{\log X_i} \sim \chi^2(2n), \text{ given } X \sim \text{Unif}(0,1)$$
which I think is genius. But my question is why going this convoluted way? and why not (what is wrong with) just using mean of p-values and use central limit theorem? or median? I am trying to understand the genius of RA Fisher behind this grand scheme.
Comment: It comes down to a basic axiom of probability: p-values are probabilities and probabilities for the outcomes of independent experiments don't add, they *multiply.*  Where multiplication is concerned, logarithms simplify a product to a sum: that's where $\sum\log(X_i)$ comes from. (That it has a chi-squared distribution is then an ineluctable mathematical consequence.) Far from begin "convoluted," this is perhaps the simplest and most natural (legitimate) procedure conceivable.
Comment: Say I have 2 independent samples from the same population (let's say we have a one sample t-test). Imagine the sample mean and standard deviations are just about the same. So the p-value for the first sample is 0.0666 and for the second sample is 0.0668. What should the overall p-value be? Well, should it be 0.0667? Actually, it's quite obvious it must be smaller. In this case the "right" thing to do is combine the samples, if we have them. We'd have about the same mean and standard deviation, but *twice the sample size*. The std. error of the mean is smaller, and the p-value must be smaller.
Comment: There are other ways to combine p-values, of course, though the product is the most natural way to do it. One could add the p-values for example; under the joint null the sum of them should have a triangular distribution. Or one could convert the p-values to z-values and add those (and if you were combining results from similar size not-too-small samples from a normal population, this would make a lot of sense). But the product is the obvious way to proceed; it makes logical sense every time.
Comment: Note that Fisher's method is based on the product, which is what I'm describing as natural -- because you multiply independent probabilities to find their joint probability. Considering GM is not really different from product other than there's then an additional step in figuring out what the corresponding combined p-value is because having worked out the GM ($g$, say) by taking the product, you'd then need to look at $-2n \log g=-2 \log (g^n)$ get the combined p-value. Which is to say you'd converting the GM back to the product before taking logs to find the combined p-value.
Comment: I would ask that every one read Duncan Murdoch's piece "P-values are Random Variables" in "The American Statistician". I find a copy online at: https://hypergeometric.files.wordpress.com/2013/09/pvaluesarerandomvariablespresentation.pdf
Comment: @Glen_b if product is the obvious way to proceed, what about geometric mean?
Answer: You can perfectly use the mean $p$-value.
Fisher's method set sets a threshold $s_\alpha$ on $-2 \sum_{i=1}^n \log p_i$, such that if the null hypothesis $H_0$ : all $p$-values are $\sim U(0,1)$ holds, then $-2 \sum_i \log p_i$ exceeds $s_\alpha$ with probability $\alpha$. $H_0$ is rejected when this happens.
Usually one takes $\alpha = 0.05$ and $s_\alpha$ is given by a quantile of $\chi^2(2n)$. Equivalently, one can work on the product $\prod_i p_i$ which is lower than $e^{-s_\alpha/2}$ with probability $\alpha$.
Here is, for $n=2$, a graph showing the rejection zone (in red) (here we use $s_\alpha = 9.49$. The rejection zone has area = 0.05.

Now you can chose to work on ${1\over n} \sum_{i=1}^n p_i$ instead, or equivalently on $\sum_i p_i$. You just need to find a threshold $t_\alpha$ such that $\sum p_i$ is below $t_\alpha$ with probability $\alpha$; exact computation $t_\alpha$ is tedious – for $n$ big enough you can rely on central limit theorem; for $n = 2$, $t_\alpha = (2\alpha)^{1\over 2}$. The following graph shows the rejection zone (area = 0.05 again).

As you can imagine, many other shapes for the rejection zone are possibles, and have been proposed. It is not a priori clear which is better – i.e. which has greater power.
Let's assume that $p_1$, $p_2$ come from a bilateral $z$-test with non-centrality parameter 1 :
<code>> p1 <- pchisq( rnorm(1e4, 1, 1)**2, df=1, lower.tail=FALSE )
> p2 <- pchisq( rnorm(1e4, 1, 1)**2, df=1, lower.tail=FALSE )
</code>
Let's have a look on the scatterplot with in red the points for which the null hypothesis is rejected.

The power of Fisher's product method is approximately
<code>> sum(p1*p2<exp(-9.49/2))/1e4
[1] 0.2245
</code>
The power of the method based on the sum of $p$-values is approximately
<code>> sum(p1+p2<sqrt(0.1))/1e4
[1] 0.1963
</code>
So Fisher's method wins – at least in this case.
Comment: Thanks, great edits (+1). For $n=2$ the null distribution is a triangular distribution, so indeed $t_\alpha=\sqrt{2\alpha}$. For $n=3$ the null distribution is already quite involved (its a three part density) but luckily for $n > 3$ it can already be approximated very well by a normal distribution with mean $0.5*n$ and variance $n/12$.
Comment: +1. Note that adding $p$-values is called **Edgington's method**, see my answer below for some bibliography.
Answer: What is wrong with summing up all individual $p$-values?
As @whuber and @Glen_b argue in the comments, Fisher's method is essentially multiplying all individual $p$-values, and multiplying probabilities is a more natural thing to do than adding them.
Still one can add them up. In fact, precisely this was suggested by Edgington (1972) An additive method for combining probability values from independent experiments (under pay-wall), and is sometimes referred to as Edgington's method. The 1972 paper concludes claiming that

The additive method  is shown to be more powerful than the multiplicative  method, having a greater probability than  the multiplicative method
  of yielding  significant results when  there actually are treatment effects.

but given that the method remains relatively unknown, I suspect that this was at least an oversimplification. E.g. a recent overview Cousins (2008) Annotated Bibliography of Some Papers on Combining Significances or
p-values does not mention Edgington's method at all and it seems that this term has never been mentioned on CrossValidated either.
It is easy to come up with various ways of combining $p$-values (I have once come up with one myself and asked why it is never used: Stouffer's Z-score method: what if we sum $z^2$ instead of $z$?), and what is a better method is largely an empirical question. Please see @whuber's answer there for an empirical comparison of statistical power of two different methods in a specific situation; there is a clear winner.
So the answer to the general question about why using any "convoluted" method at all, is that one can gain power.
Zaykin et al (2002) Truncated Product Method for Combining p-values runs some simulations and includes Edgington's method in the comparison, but I am not sure about the conclusions.
One way to visualize all such methods is to draw rejection regions for $n=2$, as @Elvis did in his nice answer (+1). Here is another figure that explicitly includes Edgington's method from what appears to be a poster Winkler et al (2013) Non-Parametric Combination for Analyses of Multi-Modal Imaging:

Having said all that, I think there still remains a question of why Edgington's method would (often?) be suboptimal, as follows from it being obscure.
Perhaps one reason for obscurity is that it does not conform to our intuition very well: for $n=2$, if $p_1 = 0.4$ (or higher) then no matter what the value of $p_2$ is, the combined null will not be rejected at $\alpha=0.05$, that is even if e.g. $p_2 = 0.00000001$.
More generally, summing $p$-values hardly distinguishes very small numbers like e.g. $p=0.001$ from $p=0.00000001$, but the difference in these probabilities is actually huge.

Update. Here is what Hedges and Olkin write about Edgintgon's method (after reviewing other methods for combining $p$-values) in their Statistical Methods for Meta-Analysis (1985), emphasis mine:

A quite different combined test procedure was proposed by Edgington
  (1972a,b). Edgington proposed combining $p$-values by taking the sum $$S = p_1 + \cdots + p_k,$$ and gave a tedious but straightforward method for obtaining significance
  levels for $S$. A large sample approximation to the significance levels of $S$ is
  given in Edgington (1972b). Although it is a monotone combination procedure and therefore is admissible, Edgington's method is generally thought
  to be a poor procedure since one large $p$-value can overwhelm many small
  values that compose the statistic. However, there have been almost no
  numerical investigations of this procedure.
Comment: Thanks, @Glen_b! I am glad this thread got some additional and well-deserved visibility. By the way, I was not aware that this procedure is called "Edgington's method" until I started researching this answer.
Answer: So if you did three studies of similar sizes and got a p-value of 0.05 on all three occasions, your intuition is that the "true value" should be 0.05? My intuition is different. Multiple similar results would seem to make the significance higher (and therefore the p-values which are probabilities should be lower). P-values are not really probabilities. They are statements about the sample distribution of observed values under a particular hypothesis. I believe that it may have given support to the notion that one can misuse them as such. I regret making that assertion.
At any rate, under the null hypothesis of no difference, the chances of getting multiple extreme p-values would seem to be much more unlikely. Every time I see the statement that the p-value is uniformly distributed from 0-1 under the null hypothesis I feel compelled to test it with simulation, and so far the statement seems to hold. I'm apparently do not think consciously on a logarithmic scale, although at least part of my cerebral neural net must.
If you want to quantify this intuition, the formula you offered (with slight revisions) appears in the Wikipedia page: http://en.wikipedia.org/wiki/Fisher%27s_method , and the associated graphic lets you quantify visually and semi-quantitatively the impact of getting two small p-values on the overall significance. For example reading from the color coded graphic, 2 simultaneous p-values of 0.05 would give a synthetic p-value around .02. You could also investigate the impact on the t-statistics of doubling your sample size. The sample size enters into the sample t-statistic as 1/sqrt(n-1) so you could look at the impact of that factor as a result of going from 50 to 100. (in R:)
<code> plot(1:100, 1/sqrt(1:100) ,ylim=c(0,1) )
 abline(h=1/sqrt(c(50,100)))
</code>
Those two approaches yield different quantitative results, since the ratio the 1/sqrt(n) values for 50 and 100 are not the same as the ratio of 0.05 to 0.02. Both approaches support my intuition, but to different degrees. Maybe someone else can resolve this discrepancy. Yet a third approach would be to consider the probability of getting two random draws of "True" when the binomial probability of each draw was .05. (an extremely unfair dice) That joint event should have a probability of .05*.05=.002, which result could be considered on the "other side" of the Fisher estimate. I just ran a simulation of 50,000 simultaneous t.tests. If you plot the results it looks very much like the maps of the cosmic background radiation field... ie. mostly random.
<code> t1 <- replicate(50000, t.test(rnorm(50))$p.value )
     t2 <- replicate(50000, t.test(rnorm(50))$p.value )
 table(t1 < 0.05, t2 < 0.05)
 plot(t1, t2, cex=0.1)
#        FALSE  TRUE
#  FALSE 45099  2411
#  TRUE   2380   110
 110/(50000-110)
#[1] 0.002204851
</code>
Comment: You don't seem to notice that with the "averaging method" and two experiments with $p_1 = 0.05$ and $p_2 = 0.05$, the null hypothesis is rejected (see second drawing in my answer above).
Comment: Thank you for your answer. The intuition you mentioned actually makes sense. I would consider those cases you mentioned as more significant. But is there a way to express this idea more mathematically rigorously?
Comment: This answer (first sentence) assumes that when averaging the $p$-values, the significance cutoff $alpha$ would stay the same, but is not true. Averaging can work just fine. See the answer by @Elvis.
Comment: I saw it. Wasn't convinced.
Comment: The averaging method "emphasizes" or weights the compound hypothesis that both individual hypotheses with be rejected together. That seems to be an unspoken constraint.
Comment: Yes, you're right (though is hardly "unspoken") : the rejection region contains only tuples of "small' p-values. See Zaykin's paper mentioned my amoeba for some stuff about the case where the null is true in many of the tests.


An equilateral triangle has sides of length 8cm. A circle passes through each vertex of the triangle. Find an expression for the circumference.

Question: The equilateral triangle PQR has sides of length 8cm. A circle, centre O, passes through each of the vertices of the triangle. Find an expression for the circumference of circle.
I thought about breaking it up into sectors but I don't know the radius.
I thought about perpendicular bisectors meeting in centre but that wouldn't help me know the radius.
Grateful for help!
Comment: Do you know that the perpendicular bisector and the angle bisector are the same thing in an equilateral triangle?
Comment: Hint: Think of the inscribed angle theorem and what that means for the angle being formed at the center.
Comment: If you want the measure of the radius of the circle use this formula: $R=\frac{abc}{4S}$ , where R is radius and S is the area of triangle, in your case a=b=c= 5 cm.First find S.
Answer: First, draw $\triangle\mathit{PQR}$. The angles between each side are all $60$ degrees. The center of the circle, $O$, is exactly at the center of the triangle. Then, connect each vertex of the triangle to $O$. These segments are all $r$, the radius of the circle. Next, choose a side and draw the perpendicular bisector of the side between said side and $O$.
To make that easier to understand, I'll provide an example. Let's use segment $\mathit{PQ}$. Define the midpoint of $\mathit{PQ}$ as $M$. Segment $\mathit{MO}$ is the perpendicular bisector between $\mathit{PQ}$ and $O$.
Segments $\mathit{PM}$, $\mathit{MO}$, and $\mathit{MP}$ form a 30/60/90 triangle which relates the side length of the triangle to the radius of the circle.
Let's call the side length of the circle $s$. The length of segment $\mathit{PM}$ equals $\frac{s}{2}$ and $\angle \mathit{OPM}$ is $30$ degrees. Therefore, $\cos \angle \mathit{OPM} = \cos \frac{\pi}{6} = \frac{\mathit{PM}}{\mathit{OP}} = \frac{s}{2r}$. Rewriting, we get that:
$$
r = \frac{1}{2} s \sec \frac{\pi}{6}
$$
The circumference of a circle is:
$$
C = 2\pi r
$$
Substituting $r$, we get:
$$
\boxed{C = \pi s \sec \frac{\pi}{6}}
$$
This equation directly relates the side length of the triangle to the circumference of the circle.
In order to better understand the derivation, I drew this diagram.
In the context of your problem, you were meant to find an expression for the circumference of the circle with the side length of the triangle being $8\text{cm}$. That expression would be:
$$
\boxed{8\text{cm} \cdot \pi \sec \frac{\pi}{6}}
$$
Answer: A circle that passes through the triangle's vertices is known as its circumcircle, and its radius is known as its circumradius $r_c$.  The circumference is obviously $2\pi r_c$.  The formula for the circumradius of a triangle with side lengths $a$, $b$, and $c$ is shown below:

Since it's an equilateral triangle, $a = b = c$.
Comment: I can include the derivation of this formula if you want.
Comment: For an equilateral triangle with side length $s$ the formula simplifies to: $\frac{s^3}{\sqrt{3s(s)(s)(s)}} = \frac{s^3}{\sqrt{3s^4}}$.
Comment: So the length of the circumference is $2\pi(\frac{512}{110.85}) \approx 2\pi(4.6188) \approx 29cm$.


Erro no programa de criptografia c#

Question: Olá, estou tentando executar meu programa feito em c# com Aplicação Windows, porém está dando um erro de compilação que não estou conseguindo solucionar. Agradeço a quem puder me ajudar ! Grato ! ( O erro está acontecendo no método void Decodificar )
<code>using System;
using System.Linq;
using System.ComponentModel;
using System.Data;
using System.Text;
using System.Collections.Generic;
using System.Drawing;
using System.Windows.Forms;
using System.Security.Cryptography;

namespace CriptografiaTrabalho
{
    public partial class MainForm : Form
    {
        public MainForm()
        {
            InitializeComponent();

        }
        void Button1Click(object sender, EventArgs e)
        {
            textBox2.Text = Codificar(textBox1.Text);
        }
        void Button2Click(object sender, EventArgs e)
        {
            textBox3.Text = Decodificar(textBox2.Text);
        }

        public static string Codificar(string entrada)
        {
            TripleDESCryptoServiceProvider cript = new TripleDESCryptoServiceProvider();
            MD5CryptoServiceProvider md5 = new MD5CryptoServiceProvider();

            try
            {
                if (entrada.Trim() !="")
                {
                    string chave = "cx3f2e4ct4d2";
                    cript.Key = md5.ComputeHash(Encoding.Default.GetBytes(chave));
                    cript.Mode = CipherMode.ECB;
                    ICryptoTransform dencrypt = cript.CreateEncryptor();
                    byte[] buff = Encoding.Default.GetBytes(entrada);

                    return Convert.ToBase64String(dencrypt.TransformFinalBlock(buff, 0, buff.Length));
                }
                else
                {
                    return "";
                }
            }
            catch (Exception exception)
            {
                throw exception;
            }
            finally
            {
                cript = null;
                md5 = null;
            }
        }

        public static string Decodificar(string entrada)
        {
            TripleDESCryptoServiceProvider decript = new TripleDESCryptoServiceProvider();
            MD5CryptoServiceProvider md5 = new MD5CryptoServiceProvider();

            try
            {
                if (entrada.Trim() != "")
                {
                    string chave = "cx3f2e4ct4d2";
                    decript.Key = md5.ComputeHash(Encoding.Default.GetBytes(chave));
                    decript.Mode = CipherMode.ECB;
                    ICryptoTransform dencrypt = decript.CreateEncryptor();
                    byte[] buff = Convert.FromBase64String(entrada);

                    return Encoding.Default.GetString(dencrypt.TransformFinalBlock(buff, 0, buff.Length));
                }
            }
            catch (Exception exception)
            {
                throw exception;
            }
            finally
            {
                decript = null;
                md5 = null;
            }
        }
    }
}
</code>
Comment: Qual o erro de compilação? Qual a mensagem?
Comment: Pessoal desculpe o erro na em *public static void Decodificar (string entrada)* a mensagem é (CriptografiaTrabalho.MainForm.Decodificar(string): nem todos os caminhos de código retornam um valor (CS0161)
Answer: Esta faltando um return no método, tem um return dentro do if mas somente ele não adianta, pois se a condição do if não for atendida, não haverá pra "onde ir".
Tenta assim:
<code> public static string Decodificar(string entrada)
    {
        TripleDESCryptoServiceProvider decript = new TripleDESCryptoServiceProvider();
        MD5CryptoServiceProvider md5 = new MD5CryptoServiceProvider();

        try
        {
            if (entrada.Trim() != "")
            {
                string chave = "cx3f2e4ct4d2";
                decript.Key = md5.ComputeHash(Encoding.Default.GetBytes(chave));
                decript.Mode = CipherMode.ECB;
                ICryptoTransform dencrypt = decript.CreateEncryptor();
                byte[] buff = Convert.FromBase64String(entrada);

                return Encoding.Default.GetString(dencrypt.TransformFinalBlock(buff, 0, buff.Length));
            }
    return "";
        }
        catch (Exception exception)
        {
            throw exception;
        }
        finally
        {
            decript = null;
            md5 = null;
        }
    }
</code>
Repare que depois do if eu coloquei um retorno vazio..
Comment: Muito Obrigado Igor realmente nem reparei q não tinha o else dando um retorno. Vlw cara ! Agradeço !

Erro no programa de criptografia c#


How to make an Apache 301 redirect to retain the referrer?

Question: I have an Apache 301 redirect in my httpd.conf file which redirects all traffic to a single domain for canonicalization purposes.  I recently noticed this redirect does not retain the referrer in the headers.  RFC2616 seems to indicate preserving the referrer is optional on a redirect and actually advises against it when crossing between HTTP and HTTPS.
Though, I have found other questions in which a simple Apache 301 redirect preserves the referrer and this generally seems to be the case by default.  What are some other factors which may influence this behavior that I can play around with to preserve the referrer?  Are the only factors based on the Apache server version and/or the client's browser settings (IE. Outside my control)?
My redirect:
ServerAlias www.example.com sub1.example.com

## Redirect all sub-domains to www
RewriteCond %{HTTP_HOST} !^www\.example\.com$ [NC]
RewriteRule ^(.*)$ http://www.example.com$1 [R=301,L]

With respect to the above rewrite rule, I'm visiting http://sub1.example.com. I'm currently just working with HTTP for simplicity until I can figure this out.
The way I'm determining the referrer is not being carried over is by visiting the redirected site manually via a browser with developer tools open.  I have also determined the referrer is not included in PHP's $_SERVER global on the receiving page.  I can confirm there is only 1 redirect occurring.  The server side language is PHP.
Here is the request header for the initial page (sub1.example.com):
GET / HTTP/1.1
Host: sub1.example.com
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
Upgrade-Insecure-Requests: 1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8

Here is the response header for the initial page (sub1.example.com):
HTTP/1.1 301 Moved Permanently
Date: Wed, 04 Oct 2017 18:14:34 GMT
Server: Apache
Upgrade-Insecure-Requests: 0
Location: http://www.example.com/
Cache-Control: max-age=1
Expires: Wed, 04 Oct 2017 18:14:35 GMT
Content-Length: 243
Keep-Alive: timeout=5, max=99
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8

Here is the request header for the receiving page (www.example.com):
GET / HTTP/1.1
Host: www.example.com
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
Upgrade-Insecure-Requests: 1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
Cookie: <redacted>

Response header for the receiving page (www.example.com):
HTTP/1.1 200 OK
Date: Wed, 04 Oct 2017 18:14:34 GMT
Server: Apache
Upgrade-Insecure-Requests: 0
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Vary: Accept-Encoding
Content-Encoding: gzip
Keep-Alive: timeout=5, max=99
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8

Comment: A redirect, 301 or otherwise, sends a reponse back to the client (user's browser) which simply says "send your request again, but this time use `foo.com`." _If_ the client chooses to send again, then it does, and how it formulates the request is _entirely_ up to the client. Because there is _no_ action on the server side, you cannot influence inclusion of the referer.
Comment: Thank you @Colt , post as an answer and I'll accept.   I'll have to post another question to figure out how to determine if my canonicalized domains are actually in use then.
Comment: "by visiting the redirected site manually via a browser" - How are you making the initial request? (By "manually" typing it in the browser?!) What are you expecting the `Referer` to be set to in your example? "the referrer is not being carried over..." - carried over _from where_?
Comment: See https://serverfault.com/a/877174/438557 for explains and solution
Comment: @MrWhite Yes, I was manually typing `http://sub1.example.com` into my browser which was redirecting to `http://www.examle.com`.  I was expecting the referrer to be `http://sub1.example.com`.  However @Webdesigner has pointed out that this will not be the case since a referrer and a redirect are different.
Answer: A redirect, 301 or otherwise, sends a reponse back to the client (user's browser) which simply says "send your request again, but this time use foo.com." If the client chooses to send again, then it does, and how it formulates the request is entirely up to the client. Because there is no action on the server side, you cannot influence inclusion of the referer.


$ \lim_{x \rightarrow \pi}\left \lfloor \frac{3\sin(x^2)-1}{2\cos(x)+1} \right \rfloor$=?

Question: How to find this limit:
$$ \lim_{x \rightarrow \pi}\left \lfloor \frac{3\sin(x^2)-1}{2\cos(x)+1} \right \rfloor $$
where $\left \lfloor x \right \rfloor$ is the floor-function?
Generally, how to find limit of floor-function?
tnx
Comment: Could you find the limit if the floor wasn't there?
Answer: Hint. The floor function $t\to\lfloor t\rfloor$ is continuous for $t\in \mathbb{R}\setminus \mathbb{Z}$. On the other hand for $n\in\mathbb{Z}$,
$$\lim_{t \rightarrow n^+}\lfloor t\rfloor=n\quad\mbox{and}\quad
\lim_{t \rightarrow n^-}\lfloor t\rfloor=n-1.$$
Evaluate
$\displaystyle \lim_{x \rightarrow \pi}\frac{3\sin(x^2)-1}{2\cos(x)+1}.$ Is it an integer number?


c# - Deserialize a Json in two steps

Question: I would like to deserialize my Json in two step because I have to read the first part to know  what kind of Object it is.
I have to read this kind of Json :
{"header":3,"data":{"result":"myResult"}}

it's more readable like that
{
    "header":3,
    "data":{
        "result":"myResult"
    }
}

I deserialize this Json in a class named ProtocolHeader :
public class ProtocolHeader
{
    [JsonProperty("header")]
    public int Header { get; set; }

    [JsonProperty("data")]
    public string Data { get; set; }
}

To do this I use this code :
JsonConvert.DeserializeObject<ProtocolHeader>(Json)

Depending on the value of the Header, I will choose different class to deserialize the end of the file.
For example, I could have another class
public class ProtocolResult
{
    [JsonProperty("result")]
    public string Result{ get; set; }
}

or like that
public class ProtocolError
{
    [JsonProperty("errorNumber")]
    public int ErrorNumber{ get; set; }

    [JsonProperty("additionalInformation")]
    public string AdditionalInformation{ get; set; }
}

Do you have an idea to seperate the Deserialize Object in two steps ?
Thanks
Comment: 2 similar questions in stackoverflow http://stackoverflow.com/questions/4272972/json-deserialization-to-inherited-types and http://stackoverflow.com/questions/12381636/why-is-json-net-is-not-working-with-inheritance-while-deserializing
Answer: You could make 3 classes.
One common class (not base) which has all of the fields, then a ProtocolResult & ProtocolError
Then implement an implicit cast to each.
You could also put a IsError getter on to your common class to decide how to use it.
Answer: You can use a reader to only read as long as you need, then skip out of the reader and do your real deserialization.
Probably not a whole lot better than deserializing into a simple object first then a real object later, but it's an alternative.
You can probably tweak this a bit.
string json = @"{""header"":3,""data"":{""result"":""myResult""}}";

using (var stringReader = new StringReader(json))
{
    using (var jsonReader = new JsonTextReader(stringReader))
    {
        while (jsonReader.Read())
        {
            if (jsonReader.TokenType == JsonToken.PropertyName
                && jsonReader.Value != null
                && jsonReader.Value.ToString() == "header")
            {
                jsonReader.Read();
                int header = Convert.ToInt32(jsonReader.Value);
                switch (header)
                {
                    case 1:
                        // Deserialize as type 1
                        break;
                    case 2:
                        // Deserialize as type 2
                        break;
                    case 3:
                        // Deserialize as type 3
                        break;
                }
                break;
            }
        }
    }
}

Answer: Option 1: Without using an abstract base class for your data classes.
The easiest I've found to do this is as follows:

Declare your class using JToken as the field type for the unknown object.
[JsonObject(MemberSerialization.OptIn)]
public class ProtocolHeader
{
    [JsonProperty("header")]
    private int _header;

    [JsonProperty("data")]
    private JToken _data;
}

Expose the specialized data in properties.
public ProtocolResult Result
{
    get
    {
        if (_data == null || _header != ResultHeaderValue)
            return null;

        return _data.ToObject<ProtocolResult>();
    }
}

public ProtocolError Error
{
    get
    {
        if (_data == null || _header != ErrorHeaderValue)
            return null;

        return _data.ToObject<ProtocolError>();
    }
}

Option 2: With using an abstract base class for your data classes.
Another option is to create an abstract base class for the various data types, and create a static method in the abstract base class to perform the type selection and proper deserialization. This is particularly useful when the type information is contained in the object itself (e.g. if header was a property inside the data object).

The LoadBalancerConfiguration<T>._healthMonitor field has the type JObject, but the HealthMonitor property in the same class returns a HealthMonitor object.
The HealthMonitor.FromJObject method performs the actual deserialization.
Comment: You could be more explicit on the way to do it. With more details about the topic.

c# - Deserialize a Json in two steps


Jolt Spec - Map a field within an array conditionally based on an outer field

Question: I want to conditionally map fields within an array based on the value of a field that's outside the array.

If message.conditionCode is "AA", then map data[].field1 to data[].conditionalField
If message.conditionCode is "BB", then map data[].field2 to data[].conditionalField

Also the conditionCode itself needs to be mapped to message.conditionCd.
I tried traversing back from within data[] to message node and check conditionCode but not sure how to travel back inside the data[].
Current Spec does not have this conditional mapping:
[
  {
    "operation": "shift",
    "spec": {
      "event": {
        "id": "event.id"
      },
      "message": {
        "id": "message.idNbr",
        "conditionCode": "message.conditionCd",
        "data": {
          "*": {
            "id": "message.data[&1].idNbr",
            "field3": "message.data[&1].directField"
          }
        }
      }
    }
  }
]

Input JSON 1:
{
  "event": {
    "id": "12345678"
  },
  "message": {
    "id": "1001",
    "conditionCode": "AA",
    "location": "US",
    "data": [
      {
        "id": "1",
        "field1": "value1",
        "field2": "value2",
        "field3": "value3"
      },
      {
        "id": "2",
        "field1": "value4",
        "field2": "value5",
        "field3": "value6"
      }
    ]
  }
}

Expected Output JSON 1:
{
  "event": {
    "id": "12345678"
  },
  "message": {
    "idNbr": "1001",
    "conditionCd": "AA",
    "data": [
      {
        "idNbr": "1",
        "directField": "value3",
        "conditionalField": "value1"
      },
      {
        "idNbr": "2",
        "directField": "value6",
        "conditionalField": "value4"
      }
    ]
  }
}

Input JSON 2:
{
  "event": {
    "id": "12345678"
  },
  "message": {
    "id": "1001",
    "conditionCode": "BB",
    "location": "US",
    "data": [
      {
        "id": "1",
        "field1": "value1",
        "field2": "value2",
        "field3": "value3"
      },
      {
        "id": "2",
        "field1": "value4",
        "field2": "value5",
        "field3": "value6"
      }
    ]
  }
}

Expected Output JSON 2:
{
  "event": {
    "id": "12345678"
  },
  "message": {
    "idNbr": "1001",
    "conditionCd": "BB",
    "data": [
      {
        "idNbr": "1",
        "directField": "value3",
        "conditionalField": "value2"
      },
      {
        "idNbr": "2",
        "directField": "value6",
        "conditionalField": "value5"
      }
    ]
  }
}

Answer: You can use the following shift transformation spec
[
  {
    "operation": "shift",
    "spec": {
      "event": {
        "id": "event.&"
      },
      "message": {
        "id": "&1.&Nbr",
        "conditionCode": "&1.conditionCd",
        "data": {
          "*": {
            "id": "&3.&2[&1].&Nbr",
            "field3": "&3.&2[&1].directField",
            "@2,conditionCode": {// bring the value of conditionCode after going two levels up the tree
              "AA": { "@2,field1": "&5.&4[&3].conditionalField" },
              "BB": { "@2,field2": "&5.&4[&3].conditionalField" }
            }
          }
        }
      }
    }
  }
]

where path lookup wildcard & is used where needed instead of repeating the literals id, data, message


Random number generation

Question: Generating n random variables whose summation will be 1. [I got the answer.]
EDIT
On genetic algorithm, we have to maintain population. Say, I have two individuals a and b. Every individual consists of $n$ pairs of ($x_i, \theta_i$), where $ 0 \leq i < n$. A fitness function evaluates fitness, $f$ of every individual. Constraint is for every individual is $\Sigma\theta_i \approx 1$ ($0.95 \leq \Sigma\theta_i < 1.05$ would suffice). $\theta_i$ associated with individual a will be adapted by some function (which I haven't figured out yet) of $d(a, b)$ & $\Delta f$. $\theta_i$ will be adaptive (by I guess something like covariance matrix). So if I increase value of $\theta_i$, values of some $\theta_j$ have to be decreased to maintain summation $\Sigma\theta_i \approx 1$. So I am seeking suggestion how can be $\theta_i$ adapted based on $d(a, b)$ & $\Delta f$?
Comment: One answer: generate $n$ random values.  Divide each by their total.  This works whenever their total is nonzero.  The generality of this solution highlights the real question: *how do you want those random values to be distributed*?
Comment: **Questions**: What do you mean by "parameter"? (Do you mean you want to generate $n$ *random variates* according to your condition?) What do you mean by "error margin"? What other (distributional, etc.) constraints would you like to be satisfied? The answer to your question is undoubtedly **yes**, but how to go about it will depend other details of your problem. Please edit your question accordingly.
Comment: FWIW, question is edited. TIA
Comment: @whuber: great solution. You have solved another problem of mine. Wish I could vote up you more.
Comment: @whuber: this is may be silly question. The range of generated value 1<*x*<100 or 1<*x*<10, will it make any difference between these range?
Comment: @crucified If you are generating only integers, then the resolution of the final values will be finer when the ranges are greater.  Otherwise there is no difference.
Comment: @whuber: So, in a word, your method is range invariant? What if, i use different ranges for generating n variables for different individuals, will it make any difference?
Comment: @crucified I think it would be good for you to experiment so you can understand what this simple formula does and see what it does *not* do.  In particular, my reply has not specified *any* method to generate the random values.  *That* is the important consideration.  You already have two specific replies (one of which is a special case of the other), but they don't begin to describe the scope of the possibilities.  Again: *how do you need the numbers to be distributed?*
Comment: @whuber FWIW, question is edited again. I don't know how much I clarified it. Thanks for your patience.
Answer: You problem is that of generating pseudo random numbers on the edges of a simplex. The best method is to:

Generate $x_1,...,x_{n^2}\sim \exp(1)$,
Stacke them in a $n\times n$ matrix,
Divide them by their row-wise sum.

I can't find a seminal paper now, but you should look for the algorithms described here.
Comment: Here is a reference for your answer: "Andrew Gelman, Bayesian Data
Analysis, pp.482" as given here: http://www.mathworks.com/matlabcentral/newsreader/view_thread/22703
Comment: This is just a special case of @Harsh's answer. Also, what do you mean by "the best method"?
Answer:
How can I generate n random numbers whose summation will be around 1?

You can try the Dirichlet distribution (see also,  https://statipedia.org/wiki/index.php?title=Dirichlet).
You need to specify more details about how they're distributed; using only the sum-constraint condition allows too many possible distributions over parameters. The Dirichlet is a commonly used distribution which satisfies this property.

Random number generation


Position of best-fit circle center with defined Radius

Question: I am trying to calculate the center point of a best fit circle, based on points in XY where the radius is defined. I have read the post of Get the best fit circle if radius is specified (constrained) but fail to understand the calculation steps needed.
What I have is points in XY (measured) and I know what the radius should be. So my function is not looking for minimizing the residuals but rather fitting a circle with defined radius through these given points. I'm only interested in the results for X and Y.
Answer: Making the problem totally formal, the shortest distance between a point  $(x_i,y_i)$ and the circle of radius $R$ centered at $(X,Y)$ is given by
$$d_i=\left|\sqrt{(x_i-X)^2+(y_i-Y)^2}-R\right|$$ So, we want to minimize
$$\color{red}{\Phi(X,Y)=\sum_{i=1}^n d_i^2=\sum_{i=1}^n \Big[\sqrt{(x_i-X)^2+(y_i-Y)^2}-R\Big]^2}$$ Computing the derivatives and setting them equal to $0$, we then need to solve the equations
$$\color{blue}{\sum_{i=1}^n \frac{(x_i-X) \left(\sqrt{(x_i-X)^2+(y_i-Y)^2}-R\right)}{\sqrt{(x_i-X)^2+(y_i-Y)^2}}=0}$$
$$\color{blue}{\sum_{i=1}^n \frac{(y_i-Y) \left(\sqrt{(x_i-X)^2+(y_i-Y)^2}-R\right)}{\sqrt{(x_i-X)^2+(y_i-Y)^2}}=0}$$ which are highly nonlinear which means that, at least in principle, a reasonable guess of $(X,Y)$ is required to start iterations. If you have it, no problem using Newton-Raphson method. What I gave in the first part of my answer to this question is just a simple way to obtain this estimate (notice that $R$ is not present in this part) while the second part is an approximation of the formal problem as I set it here.
Using the same data points, taken frol page $2$ in this paper, the first part gave as estimates $X_0=3.06030$ and $Y_0=0.743607$. These will be used to the full process for a few values of $R$
$$\left(
\begin{array}{ccc}
R & X_{\text{previous}} & Y_{\text{previous}}& X_{\text{actual}}& Y_{\text{actual}} \\
 3.7 & 3.11165 & 0.800459 &  3.10985 & 0.810044 \\
 3.8 & 3.10106 & 0.788820 &  3.09472 & 0.795389 \\
 3.9 & 3.08933 & 0.775878 &  3.07880 & 0.779987 \\
 4.0 & 3.07628 & 0.761420 &  3.06201 & 0.763781 \\
 4.1 & 3.06172 & 0.745189 &  3.04431 & 0.746706 \\
 4.2 & 3.04537 & 0.726870 &  3.02560 & 0.728695 \\
 4.3 & 3.02693 & 0.706070 &  3.00583 & 0.709670
\end{array}
\right)$$
As you can see, the results are not identical but sufficiently close for practical purposes.
If you have an optimizer, then attack directly the problem of the minimization of $\Phi(X,Y)$ without any guess. If the errors are not too large, it should converge like a charm (you can do it using Excel solver).


How to "ignore" username and password prompt in net use

Question: I have at the moment a logon.cmd script, that I'm using to map network drives to the users profile. It looks like this:
::Onboarding
net use m: /delete
net use m: \\BOB\onboarding
::Bookings
net use n: /delete
net use n: \\BOB\bookings

::Accounts
net use j: /delete
net use j: \\BOB\accounts

It works fine up until it gets up to a folder that the current user cannot access, it then asks for a username and password instead of erroring and continuing.
Notes: This very script used to work on another Samba PDC network, but I've moved it over to another server (Still Samba PDC) and now its breaking.
Is there anyway for it to ignore the username/password prompt and just continue?
Comment: And this is a prime example of why it's usually not a good idea to try and replace Active Directory with something that isn't Active Directory.
Comment: This is very true MDMarra, but $0 software budget causes certain restrictions :P
Comment: Gotta ask - Why are you running a script to map drives that the user may not have a access to?
Comment: Logon.cmd is the only script that can be ran during boot, and it has no way of knowing who should have access to what, so, it has to try to map everything otherwise it would get skipped
Comment: I don't use Samba, so I'm not fully familiar with it, but are you saying that all users must have the same script? Can't you set a logon script on a per-user basis?
Comment: Unfortunately no :/, Samba is a lot more basic than your normal AD PDC, you have one script for everyone on logon.
Answer: I was never able to find a finite solution to the problem. But a usable (but very hacky and unclean) work around is using this command:
::Helpdesk
copy /Y NUL "\\BOB\helpdesk\.writable"
IF %ERRORLEVEL%==0 (
    del \\BOB\helpdesk\.writable
    GOTO:ALLOWEDHELPDESK
 )

GOTO:SECT2
:ALLOWEDHELPDESK
net use k: /delete
net use k: \\BOB\helpdesk
:SECT2

::Onboarding
copy /Y NUL "\\BOB\onboarding\.writable"
IF %ERRORLEVEL%==0 (
    del \\BOB\onboarding\.writable
    GOTO:ALLOWEDONBOARDING
 )

GOTO:SECT3
:ALLOWEDONBOARDING
net use m: /delete
net use m: \\BOB\onboarding
:SECT3

::Etc,etc,etc

Basic explanation:
It checks to see if a folder is writeable first by coping a blank file .writable, if it succeeds it executes the command, if it fails it skips the command and continues on.
This is purely a workaround....
Comment: Hey @Mattisdada, I'm adding an answer that extends yours. Maybe a community wiki would make sense here; I'm not sure. Thanks for posting question/answer. Your solution was the best fix I could find for my problem and I just hope my answer can add a little value.
Answer: Extending Mattisdada's answer, this script will make it a bit easier to modify any drive mappings by simply adding, deleting or changing the mapDrive function calls. Additionally, this function checks for read access by calling DIR (instead of writing the temp file) and maps accordingly:
@ECHO off

setlocal
set SHARE=BOB
CALL:mapDrive K: helpdesk
CALL:mapDrive M: onboarding
CALL:mapDrive Z: watercooler
endlocal

net use
ECHO Mapping Complete. Thank You.
PAUSE&GOTO:EOF

:mapDrive
DIR \\%SHARE%\%~2 > nul 2>&1
IF %ERRORLEVEL%==0 (
    net use %~1 /DELETE > nul 2>&1
    net use %~1 \\%SHARE%\%~2
)
GOTO:eof

Answer: I seem to have figured this out on my end, I just enter a password as an argument of net use, after the folder to be mapped, as below:
..
net use t: \BOB\helpdesk password /PERSISTENT:NO
not exactly sure why it works, as its not the right password for the user but it seems to map it anyway if the user has access and just throw an access denied if no access to drive and moves on :)
Answer: The solution:
net use t: \BOB\helpdesk password /PERSISTENT:NO
seems to have worked but fails after recent updates.
if EXIST \BOB\helpdesk net use t: \BOB\helpdesk /PERSISTENT:NO
is a solution that keeps working for me. It skips if the user does not have sufficient access rights(works for network shares).
Comment: i would use if NOT exist to run it only in case its not existing else your exmaple runs only if its exists


Wind instruments: why is the left hand on top?

Question: Every wind instrument which require the two hands to press keys (saxophone, clarinet, oboe…) have the left hand of the player being the top one. This consistency makes it easier to switch from one instrument to another but is there a particular reason for that?
One could think that both hands are equally used on these instruments but I remember, as a left-handed person, trying to grab the saxophone the other way around when I started (it did not last long). There are lefties guitar, while, even if both hands do not have the same role, both of them are pretty active. If there is a physiological reason for having the left hand on top, then why don't we have left-handed wind instruments?
Comment: @phoog - ambidextrous winds go back further than that.  Renaissance recorders almost always have duplicated holes (or keys with basses) for the bottom note, the unused hole being plugged with wax.
Comment: @phoog - I guess anyone wanting to learn l.h. piano might struggle to find a teacher. A rich guy did in fact have a l.h. grand made for him. But I doubt l.h. 'boards would be a viable option.
Comment: Also see https://music.stackexchange.com/questions/60492/how-important-is-it-for-the-left-hand-to-be-nearest-the-mouth-when-playing-tin-w
Comment: A factor that I don't see mentioned is that on many instruments, particularly the saxophones, the right hand bears most of the weight.
Comment: No significant benefit but significant price difference due to very small number of instruments is sufficient to discard the idea from my point of view. Note, that all fingering tables would also be affected.
Comment: Would a left-handed transverse flute be able to march with right-handed transverse flutes?  But there is a counterexample: baroque oboes frequently have an ambidextrous key.  See for example http://www.baroqueoboes.com/OBOES/oboe415.html
Comment: Since about 85% are right-handed, often the right hand (fingers!) has more to do. It's probably down to market forces, making left hand saxes would be a small market - and most players adapt. As they often do on guitar.
Comment: @phoog - on the left hand side of a row, yes!
Comment: @Tim indeed.  What about left-handed pianos or other keyboards?  For an electronic keyboard it ought to be fairly trivial to achieve.  Would it help anyone?  I tend to doubt it.  Do left-handed jazz pianists play more interesting bass parts?
Comment: @phoog Joe Zawinul experimented with 'inverted' keyboards. It's easy enough to program.
Comment: @badjohn That's a good point indeed!
Answer: Western woodwind instruments were played with either hand on top up to the eighteenth century when keys started to be added. The first key on the modern flute was the D-sharp/E-flat key which was on an extra joint and could be rotated for either hand. As more keys were added people finally settled on having the left hand on top and all modern western instruments are built this way.
The concept of right- or left-handedness is irrelevant for woodwind instruments because neither hand dominates, although I once saw a "left-handed" keyed Irish flute, and I've heard of "left-handed" recorders.
Comment: Thanks for this answer! Interesting to see how it shifted from ambidextrous to a single design. As I remember trying in the beginning in the wrong way, I would say that the current choice is probably the more natural for right-handed people…
Comment: @phoog With only one key it was easy to play either way. Once they added a second key they had to settle on one system. The first key was probably left ambidextrous for those who didn't want to use the second key.
Comment: I believe I've also heard of left-handed recorders.  What do you make of the baroque oboe, though?  There are two keys, and only one of them is ambidextrous as far as I can see.
Comment: Same with the baroque bassoon. Instruments with only a few keys had a "forked" key on front that allowed you to play either hand on top. When more and more keys where added, somehow the choice was made to have left hand on top. The modern bassoon start at around 22 keys, but may have up to 28 or even a few more. Most busy finger is left thumb where I have 10 keys on my bassoon.
Comment: @Tom_C Consider it from the perspective of *closer in* and *farther out*, and it's not surprising that the majority dominant hand is the one doing the reaching.
Comment: @chrylis-cautiouslyoptimistic- Which is odd, because on the piano (in some styles, at least), it's the left hand which reaches farther (both in terms of playing octaves, and in moving up and down the keyboard).  I think similar would apply to guitars and other (plucked) stringed instruments too.
Comment: @phoog - by the time of baroque oboes, the ambidextrous key was just an esthetic vestige, since the assymetrical double holes made it only possible to play with the left hand above.


what families of guitar strings (lower and upper) will give me differents sound timbres?

Question: They are plenty kinds of guitar strings for my classical guitar : silver or titanium (or platinum ?) coated, crystal ones...
Some are for commercial attractiveness, some might increase longevity of the strings, but which of them really changes the timbre of their sound ?
If I want to experiment many different timbres possible when I change my set of strings, to find the final set I enjoy the most, what families of :

Lower strings
Upper Strings

will give me a different timbre ?
Comment: I'd think that, somewhere like Just Strings or the D'addario site, there'd be an explanation of the different variations. My classical guitar has a busted but and hasn't seen new strings in a decade, so I'm not the one to answer, but it seems like a low-fruit answer outside of MSE.
Comment: I'm not a guitarist, but in the world of woodwind instruments material doesn't affect sound at all. I imagine, with strings it affects responsiveness which, in turn, might affect timbre. Back in the day, some bassists would let their strings rust, to give them more of a "dirty" sound. I believe, Todd Rittmann of U.S. Maple substituted the low 3 strings on his electric guitar with bass strings, which provided for some of the band's unique sound, so that could be something to experiment with as well.
Answer:
but which of them really changes the timbre of their sound ?

The shortest answer would be - "all of them." It's one of the main reasons why there are so many different types of strings to begin with. The gauge, material, and winding  affect the overall quality of the sound. Even coating on guitar strings changes the timbre and tone - even though strings are coated mainly for the purpose of preservation, the coating definitely changes the quality of sound... love it or hate it.
Some guitarists spend years experimenting with strings to find the right fit... generally, once a musician discovers their preferred string, they tend to play with it exclusively.
As far as the material of a string, for classical guitar in particular, this article is helpful in answering your question.

Treble Strings
Clear Nylon: This is the most common material for treble nylon strings. Clear nylon offers a good balance of brightness and warmth in
tone and are known for their projection and sustain qualities and easy
vibrato.They aren't as loud or as powerful as other string materials
like titanium or composite.
It is the smoothest material used for nylon strings, which can feel
the best when playing finger style.
Black Nylon: This produces a warmer, more mellow tone with more treble overtones than clear nylon. It is a popular choice for folk.
Titanium: Titanium produces a brighter crisper sound. Strings made with titanium are often used to brighten up a darker sounding guitar.
They also have good sustain and easy vibrato.
Carbon Fibre: Strings made using carbon produce a loud sound and tend to be longer lasting. They are characterized by a short sustain
and can go beyond bright to a more 'thin' or 'harsh' sound. These can
work well depending on the guitar.
If your guitar already has a bright sound, then carbon strings are
likely to sound too bright but they can be good for a darker sounding
guitar.
Composite: This material produces a very bright sound with strong projection. Composite is often used for G strings to make for a
smoother transition in sound between the treble and bass strings.
Bass Strings
Bronze: Bronze windings are typically composed of 80% copper and 20% zinc. This material is sometimes known as 80/20 bronze, brass or
gold. These strings produce a bright sound with good sustain.
Silver: Silver windings are typically made of copper with a silver coating.  Silver coated windings help to produce a warmer tone.


Wireshark display filter for Protocol != 802.11

Question: I've set Wireshark's capture filter set to capture only packets from the MAC address of interest, but the result is dominated by zillions of packets whose Protocol is "802.11".  I want to view all of the packets that are NOT 802.11, e.g. ARP, DCHP, DNS, TCP, etc.
For example, in the following, I'd like to hide all of the 802.11 packets and show the DHCP packets (and any others that are NOT 802.11):

I tried "wlan.fc.type != 0", but clearly that's not correct.  What's the magic keyword for such a filter?
update
@ChristopherMaynard's comment is close, and I now understand that a simple filter of:
ip

is almost what I want, but it fails to display packets of type ARP, EAPOL, LLC, MDP and perhaps others.
Comment: I think your question is off-topic on SO, and should be on https://networkengineering.stackexchange.com/questions/tagged/wireshark
Comment: Have you tried `dhcp` as a display filter for only displaying DHCP packets?  If you want DHCP and ARP packets, try `dhcp or arp`.  DHCP, ARP and TCP - try `dhcp or arp or tcp`.  DNS too?  How about `dhcp or arp or tcp or dns`?

Curious about all possible display filters?  Have a look at https://www.wireshark.org/docs/dfref/.  Need even more information on Wireshark display filters?  Have a look at https://www.wireshark.org/docs/man-pages/wireshark-filter.html.
Comment: Downvotes do convey some information, so I'm not against them.  But it sure would be nice to get some actionable info along with the downvote!  (FYI, I've searched online docs, cheat sheets, tutorials.  I simply don't know the name of the filter.)
Comment: Ah!  Okay.  But I note that there are LOTS of other Wireshark questions here on SO, which is what emboldened me to post it here.  Thanks for the hint.
Comment: @ChristopherMaynard - My challenge is that I want *any* packet that's not 802.11 -- I don't know what they'll be in advance.  A simple filter of `ip` comes close, but doesn't show MDP, ARP, LLC, EAPOL packets (and likely others).
Comment: I don't understand what you want.  You can use `!wlan` to exclude all 802.11 packets, but that hardly seems like it'll be useful.  If you want to exclude 802.11 management frames, then maybe `wlan and !(wlan.fc.type == 0)` is what you're after?  Have a look at this cheatsheet?: https://www.semfionetworks.com/uploads/2/9/8/3/29831147/wireshark_802.11_filters_-_reference_sheet.pdf
Comment: @ChristopherMaynard > I don't understand what you want.
I apologize for not being clearer: In the column labeled "Protocol", I want to hide any entry labeled "802.11" and display everything else.   Thanks.
Comment: @ChristopherMaynard: I did look at that cheatsheet before posting the OP.  Evidently I'm handicapped by not knowing the distinction between management, control and data frames.
Comment: If you just want data frames but those without un-dissected data (which would still show 802.11 in the Protocol column), then you could try `wlan.fc.type == 2 and !data`.  Or you could try `!(frame.protocols == "radiotap:wlan_radio:wlan:data" or frame.protocols == "radiotap:wlan_radio:wlan")`?
Comment: @ChristopherMaynard yes - !(frame.protocols == "radiotap:wlan_radio:wlan:data" or frame.protocols == "radiotap:wlan_radio:wlan")? was the winning ticket.  Want to make an answer out of it so I can accept it?
Answer: [Converted comment to answer and added an alternate solution while I'm at it.]
If you just want data frames but those without un-dissected data (which would still show 802.11 in the Protocol column), then you could try wlan.fc.type == 2 and !data. Or you could try !(frame.protocols == "radiotap:wlan_radio:wlan:data" or frame.protocols == "radiotap:wlan_radio:wlan").
Another alternative is to download the filtcols.lua script written by Chuck Craft, save it to your plugins directory (Wireshark: Help -> About Wireshark -> Folders -> Personal Lua Plugins), the [re]start Wireshark.  Now you can apply a display filter such as wlan and !(filtcols.protocol == "802.11").
Comment: That is so stupid. *sigh*


Left exactness of $-(U)$

Question: A few questions about the proof given here: $\Gamma(U,\cdot)$ is a left-exact functor $\mathfrak{Ab}(X)\to\mathfrak{Ab}$

Let $0\to\mathcal{F}'\xrightarrow{\phi}\mathcal{F}\xrightarrow{\psi}\mathcal{F}''$ be a left-exact sequence. Then for any open set $U\subseteq X$, the sequence $0\to\Gamma(U,\mathcal{F}')\to\Gamma(U,\mathcal{F})\to\Gamma(U,\mathcal{F}'')$ is left-exact.

$$\begin{array}{cccccc}
0 & \rightarrow & \Gamma(U,\mathcal{F}') & \xrightarrow{\phi_U} & \Gamma(U,\mathcal{F}) & \xrightarrow{\psi_U} & \Gamma(U,\mathcal{F}'')\\
& & \downarrow & & \downarrow & & \downarrow\\
0 & \rightarrow & \mathcal{F}'_p & \xrightarrow{\phi_p} & \mathcal{F}_p & \xrightarrow{\psi_p} & \mathcal{F}^{''}_p
\end{array}$$

Now if $s\in\Gamma (U,\mathcal{F}')$, we have $\psi_U(\phi_U(s))_p=\psi_p(\phi_p(s_p))=0$ for every $p\in U$, hence $\psi_U(\phi_U(s))=0$. The inclusion left to show is $\ker(\psi_U)\subseteq\operatorname{im}(\phi_U)$.

First, I suppose vertical maps are denoted by $(-)_p$. Also, I suppose the LHS of $\psi_U(\phi_U(s))_p=\psi_p(\phi_p(s_p))$ should be $(\psi_U(\phi_U(s)))_p$ (the image of $\psi_U(\phi_U(s))$ under the vertical arrow). I don't see how $(\psi_U(\phi_U(s)))_p=0$ in $\mathcal F''_p$ implies $\psi_U(\phi_U(s))=0$ in $\mathcal F''(U)$. The former means that there is an equality of equivalence classes $[(\psi_U(\phi_U(s)), U)]=[(0,X)]$, which only means that $res_{U,W}(\psi_U(\phi_U(s)))=res_{X,W}(0)$ for some $W\subset U$. How does this imply that $\psi_U(\phi_U(s))=0$?
Next, the answer there says:

In my opinion, you need to reason why the $s'_P$ do glue to some $s'\in\mathcal{F}'(U)$. However, this is not that tough. We pick neighborhoods $V_P\subseteq U$ for each point $P$ such that $s'_P=(f_P,V_P)$ with $f_P\in\mathcal{F}'(V_P)$. For $W:=V_P\cap V_Q\ne 0$, we have $\phi_W(f_P|_W)=t|_W=\phi_W(f_Q|_W)$ and we know that $\phi_W$ is injective.

Here, I suppose, the notation $f_P|_W $ means $res_{V_P,W}(f_P)$ and similarly for $f_Q|_W$ and $t|_W$. How does $\phi_W(f_P|_W)=t|_W=\phi_W(f_Q|_W)$ follow? I tried drawing the diagram that witnesses the functoriality of $\phi$, but I didn't see how it helps. And probably even a more basic question: why do we need to check this? if we're given some $s_p\in\mathscr F_p'$, it has the form $[(f,U)]$ for some open $U$ that contains $p$ and some $f\in \mathcal F'(U)$. Its preimage under the vertical map is just $f$.
Answer: $(\psi_U(\phi_U(s)))_p=0$ holds for every $p$. You need to show $a_p = 0 \> \forall p \implies a = 0.$ That's a good exercise but here is the solution if you need:

 $a_p = 0$ means there exists $V_p \ni p$ such that $a|_{V_p} = 0$. Such $V_p$ cover $U$ and $a|_{V_p} = 0|_{V_p},$ so $a = 0.$

In the proof that $\ker(\psi_U)\subseteq\operatorname{im}(\phi_U)$, you end up with a bunch of $s'_p\in\mathcal{F}'_p, p \in U.$ You want them to come from a single element of $\Gamma(U,\mathcal{F}').$ The idea is that those $s'_p$ are represented by some $f_p \in \Gamma(V_p,\mathcal{F}'),$ and one can use that $\mathcal{F'}$ is a sheaf to glue $f_p$ together to get the desired element. For that, one has to check the compability of $f_p.$ This means showing that $f_p|_W=f_q|_W$ for $W \subset V_p \cap V_q.$ It is enough to show $\phi_W(f_p|_W)=\phi_W(f_q|_W)$ since $\phi_W$ is injective. But here it follows from how $f_p$ are defined. They are defined so that elements $s'_p$ they represent are mapped to $t_p$ by $\phi_p.$
This means that both  $\phi_W(f_p|_W)$ and $\phi_W(f_q|_W)$ locally look like $t$ (so have the same germs) and by the exercise above they must be equal.
Comment: So the fact that $t_p$ (which is equal to the equivalence class $[(t,U)]$) has a preimage means that there is some $[(f_p,V_p)]\in\mathcal F_p$ such that $[(t,U)]=[(\phi_{V_p}(f_p),V_p)]$? (Somehow I used to think that it means that the $V_p$ above must be the same as the $U$ in the top row of the commutative diagram.)
Comment: I suppose the fact $f_p|_W=f_q|_W$ needs to be checked because it is a hypothesis in the gluability axiom. Then, applying the conclusion of the gluability axiom, we obtain that there is $f\in\mathcal F'(U)$ such that $res_{U,V_p}(f)=f_p$. Do we need to use the part after "such that" somehow?
Comment: @user634426 Germs need not come from a "global" element, yes.
Comment: @user634426 Once you've obtained $f$, you use our exercise once again to show that it's indeed the preimage you've been looking for, just as InvisiblePanda describes in the question.
Comment: That's what I did, but I didn't use $res_{U,V_p}(f)=f_p$. (I just inferred from $t_p=\phi_U(f)_p $ that $t=\phi_U(f)$ more or less immediately; I don't see why that condition is needed.)
Comment: @user634426 Yes, it doesn't seem to be used.

Left exactness of $-(U)$


How to solve $\int_0^\infty J_0(x)\ \text{sinc}(\pi\,x)\ e^{-x}\,\mathrm dx$?

Question: I need some help with solving this integral involving Bessel function:
$\hspace{2in}\displaystyle\int_0^\infty$$J_0(x)\ $$\text{sinc}(\pi\,x)\ $$e^{-x}\,\mathrm dx.$
Answer: Prudnikov-Brychkov-Marychev (Vol. II, formula 184.108.40.206) tells us that
$$\int_0^{\infty}\frac{1-e^{-px}}{x}J_0(x)dx=\ln\left(p+\sqrt{p^2+1}\right)$$
Taking the difference of this formula with $p=1-i\pi$ and $p=1+i\pi$, one obtains
\begin{align}\int_0^{\infty}J_0(x)\mathrm{sinc}(\pi x)e^{-x}dx&=\frac{1}{2\pi i}\ln\frac{1+i\pi+\sqrt{(1+i\pi)^2+1}}{1-i\pi+\sqrt{(1-i\pi)^2+1}}=\\
&=\frac{1}{\pi}\arctan\frac{\pi\sqrt{2}+\sqrt{\pi^2-2+\sqrt{\pi^4+4}}}{
\sqrt{2}+\sqrt{2-\pi^2+\sqrt{\pi^4+4}}}.
\end{align}
Comment: Amazing answer!
Comment: Great answer! It is a shame that Mathematica cannot solve it, especially when they claim "It can do almost any integral that can be done in terms of standard mathematical functions." http://reference.wolfram.com/mathematica/howto/DoAnIntegral.html
Comment: @nbubis: I wouldn't say so. One should just know the right book.
Comment: @Zakharia, you did notice the word "almost", no? ;)
Answer: Here's an evaluation of the integral used by O.L..
$$\begin{align} \int_{0}^{\infty} \frac{1-e^{-px}}{x} J_{0}(x) \ dx &= \int_{0}^{\infty} \int_{0}^{\infty} (1-e^{-px})J_{0}(x) e^{-xt} \ dt \ dx  \\ &= \int_{0}^{\infty} \left( \int_{0}^{\infty} J_{0}(x) e^{-tx} \ dx \ - \int_{0}^{\infty} J_{0}(x) e^{-(p+t)x} \ dx \right) \ dt  \\ &= \int_{0}^{\infty} \Bigg( \frac{1}{\sqrt{1+t^{2}}} - \frac{1}{\sqrt{1+(p+t)^{2}}} \Bigg)\ dt \tag{1} \\ &= \text{arcsinh}(t) -\text{arcsinh}(p+t) \Bigg|^{\infty}_{0} \\ &= \text{arcsinh}(p) \\ &= \ln \left(p+\sqrt{p^{2}+1} \right) \end{align}$$
$(1)$ http://mathworld.wolfram.com/LaplaceTransform.html
Answer: O.L.'s answer is absolutely correct. I just want to generalize the formula a little:
$$\forall a>0,\hspace{5mm}\int_0^\infty J_0(x)\ \text{sinc}(a\,x)\ e^{-x}\,\mathrm dx=\frac1a\arctan\sqrt{\frac{2}{\sqrt{a^4+4}-a^2}-1}.$$


Missing CSS styles for Carbon Charts. Please read the Carbon Charts getting started guide

Question: I'm using the react library of carbon design charts and I'm getting intermittent error 'Missing CSS styles for Carbon Charts. Please read the Carbon Charts getting started guide.'
I am importing the css in app.scss which is imported in index.js.
@import '~@carbon/charts/styles.css';
It works when I execute the code on localhost, but when the code is built with npm build it gives this error sometimes and the complete page goes blank.
also tried importing in the css in component file,
import '@carbon/charts/styles.css';
Comment: Did you get any solution for this?
Comment: Yes, I did a temporary workaround by adding the minified css manually in the code repo so that the page doesn't go blank...
Answer: Spent a lot of time experimenting to see what was missing. Finally was able to get it working by adding these to my package.json (in addition to @carbon/charts and @carbon/charts-angular):
"carbon-components": "^10.52.0",
"@carbon/themes": "^10.50.0",

In my global styles.scss (although could be in a component's scss)...
@import "https://fonts.googleapis.com/css?family=IBM+Plex+Sans+Condensed|IBM+Plex+Sans:400,600&display=swap";
@import '@carbon/themes/scss/themes';
@import 'carbon-components/scss/globals/scss/styles.scss';
@import "@carbon/charts/styles/styles.scss";

Answer: @import, is for importing into css file, so put your import inside of index.css on top. Or you can try doing import '~@carbon/charts/styles.css without @ inside of index.js
Comment: Sorry for the incorrect description. I am importing the css in app.scss which is imported in index.js.
@import '~@carbon/charts/styles.css';
Answer: Pls verify that the styles are actually being applied.
I've seen this issue many times in angular, and usually you have to move the import around to see where it'll actually take effect. I usually use app.component.ts or a higher-level file like that
Also, we do have an angular getting started guide
https://charts.carbondesignsystem.com/?path=/story/docs-getting-started--angular


Ubuntu 6.06: Mounting /root/sda1 /root failed: No such device

Question: An old machine in our office, running Ubuntu 6.06 all of a sudden will not boot up. I get the following info during boot:
Uncompressing Linux... Ok Booting the kernel
mount: Mounting /root/sda1 /root failed: No such device
mount: Mounting /root/dev on /dev/.static/dev failed: No such file or directory
mount: Mounting /sys /root/sys failed: No such file or directory
mount: Mounting /pro /root/pro failed: No such file or directory
Target filesystem doesn't have /sbin/init

Busybox v1.01 (debian 1:1.01-4ubuntu3) Built-in shell (ash)

Enter 'help' for a list of built-in commands
/bin/sh: can't access tty; job control turned off
#

I haven't changed anything on the system as far as I'm aware, and I ran some HD diagnostics and everything seems fine.
After Googling around, I found this thread: http://ubuntuforums.org/showthread.php?t=246895 Which seems very similar to my problem, however when I try to mount the drive with the following command:
sudo mount -t ext3 -o rw /dev/hda1 /mnt

I get the following error:

mount:  wrong fs type, bad option, bad
  superblock on /dev/sda1, missing code
  page or helper program, or other error
  In some cases useful info is found in
  syslog - try dmesg | tail or so

I ran fdisk -l and it says the partition type is Linux.
The output after running dmesg | tail :
[12207.483801] init_special_inode: bogus i_mode (101)
[12207.483809] EXT2-fs: corrupt root inode, run e2fsck
[12260.427078] init_special_inode: bogus i_mode (101)
[12260.427086] EXT2-fs: corrupt root inode, run e2fsck
[13716.998799] e100: eth0 NIC Link is Up 100 Mbps Full Duplex
[13716.998923] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[13727.061967] eth0: no IPv6 routers present
[13896.700449] kjournald starting.  Commit interval 5 seconds
[13896.700458] init_special_inode: bogus i_mode (101)
[13896.700465] EXT3-fs: corrupt root inode, run e2fsck

After running e2fsck -p /dev/sda1, I get the following info:
/dev/sda1: clean, 142449 / 9584640 files, 5402711 / 19161520 blocks

Now I'm really not sure where to go with this :x
Thank you for looking
Answer: Most of the time, e2fsck will not run unless it thinks it should.  You can force it in a situation like this.  Try adding the -f flag, like e2fsck -fp /dev/sda1.
Comment: But never, ever do it on a mounted filesystem.  Ever.
Comment: Thanks a lot, doing this instructed me to manually run fsck. I did that, and cleared the errors and now the drive will mount. :) I tried to restart the machine and I get Error 15 from GRUB. I'll play around with this and see if I can restore GRUB.


MSTest fails: Failed to find configuration section 'log4net' in the application's .config file

Question: This is my setup:
/
solution.sln
Core/
    Core.csproj
web/
    web.csproj
    Web.config
Test/
    Test.csproj
    App.config
build/
    _PublishedWebsites/web/etc...
    test.dll
    test.dll.config
    Core.dll
    log4net.dll

I wrote Unit Tests in the Test project that test the functionality of the Core project. All of the projects use log4net, all of them are configured using the respective App.config or Web.config files using
[assembly: log4net.Config.XmlConfigurator(Watch = true)]

in their AssemblyInfo.cs.
The projects are built using NAnt into a build directory.
When I run the Unit Tests from Visual Studio (2010 Professional), everything works fine.
But when I try to execute MSTest directly
MSTest.exe /resultsfile:TestResults.trx /noisolation /testcontainer:build\test.dll

All test cases fail with the following error:
log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file

This confuses me, because the test.dll.config file most definitely contains the log4net configuration section.
The folder created by MSTest also contains the config file:
/flopes_MyMachine 2015-10-08 08_48_59/Out/
    Core.dll
    test.dll
    test.dll.config
    log4net.dll

It appears to me that maybe MSTest is using a wrong configuration file or doesnt find the configuration file of the test.dll? Does that even make sense? Is there a way to tell MSTest which config file to use?
I did a small test output to see what configuration file is loaded, and it added even more mystery, because apparently the configuration file that was loaded is:
D:\Programme\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe.Config

How is this possible? Why does MSTest not just load the configuration file that is right beside the test.dll? How does this make sense?
Comment: Weird, `noisolation` basically controls the creation of separate sandboxes for the test-instances, but maybe the creation of separate processes is needed to use a "dynamic" and non-hardcoded config for ms-test.
Comment: Does it change if you run mstest from within the build-folder? (ie so that working directory is different)
Comment: @flindeberg Nop, same config file is loaded, no matter what working dir.
Comment: Is it possible you can create a minimal working example which I can have a look at? I cannot recreate your scenario =/ (given that the .config is part of the deployment as a deployment item)
Comment: @flindeberg Well... I can't either apparently. I tried creating a MWE for two hours now, but somehow it works when I start from scratch. Still this is information; it means that something else besides what I listed here must be causing the problem...
Comment: @flindeberg Okay, as soon as I remove the `noisolation` parameter from the MSTest call, it 1. uses the correct configuration file and 2. runs the tests as normal... Can't figure out how the isolation should actually work though
Comment: @flindeberg Well it seems this is also the answer... It all boils down to `/noisolation`. Do you want to update your answer so I can accept it?
Answer: I would have a look at this question and answer: MSTest.exe not finding app.config
Which states that you should create a test run configuration which includes the .config as a deployment item.
Also check if the config being used is the correct one; ways of finding active config-file:
// Get the user-config:
ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None).FilePath

// Simple approach
AppDomain.CurrentDomain.SetupInformation.ConfigurationFile? Is it the correct .config

Checking a couple of things with @florianpeschka led us to believe that the problem is the noisolation-flag. And that is a bit weird, since noisolation basically controls the creation of separate sandboxes for the test-instances, but maybe the creation of separate processes is needed to use a "dynamic" and non-hardcoded config for ms-test.
So in short, do not use the noisolation-flag and everything works.
Comment: This doesn't seem to solve the underlying problem. The `config` file is properly deployed into the `TestResults/Out/` folder, see my question. The file is there, it's just somehow not picked up by MSTest...
Comment: What is `AppDomain.CurrentDomain.SetupInformation.ConfigurationFile`? Is it the correct `.config`-file?
Comment: How should I find that out? Can I debug into the MSTestExecution?
Comment: Maybe remove log4net (so it runs) and write the variable to file?
Comment: Well that's interesting... I added the output to my question.


Setting a default checkbox value in FAPI

Question: Trying to set a default value on an entity reference field.
$unitid = node_load($unit);
$form['field_choose_a_service']['und']['#default_value'] = $unitid;

Warning: Illegal offset type in form_type_checkboxes_value() (line
  2390 of
  /homepages/43/d503288075/htdocs/example.co.uk/includes/form.inc).
Notice: Object of class stdClass could not be converted to int in
  form_process_checkboxes() (line 3316 of
  /homepages/43/d503288075/htdocs/example.co.uk/includes/form.inc).

And targeting the nid from that loaded node...
$unitid = node_load($unit);
$form['field_choose_a_service']['und']['#default_value'] = $unitid->nid;

Warning: Invalid argument supplied for foreach() in
  form_type_checkboxes_value() (line 2389 of
  /homepages/43/d503288075/htdocs/example.co.uk/includes/form.inc).

The first one actually works, but kicks up the errors. What am I doing wrong?
Essentially I want to check if the nid im passing in the URL as a _GET query matches an option in the entity reference checkbox options, and make that option selected if it does.
Answer: Try this one:
$unitid = node_load($unit);
$form['field_choose_a_service']['und']['#default_value'] = array($unitid->nid);

And please note that node_load returns an object but not a ID. I see you use $unitid as the variable name. It can make people confused.
Comment: I supposed that `$unitid` should be a node as it is the value from `node_load()`....
Comment: Yep, that worked. I'm guessing that worked because the default value is expecting an array, even if there is only one? Thanks!
Comment: @Collins I think field uses `checkboxes` rather than `checkbox` in all cases. And `checkboxes` only accept array as value. And the error happen on `form_type_checkboxes_value()` can prove it.


How can I see anonymous visitors in real time Google Analytics?

Question: I can't see anonymous visitors in real-time in Google Analytics, I can only see
registered users. But I need see all visitors in Real-time>Overview , including anonymous visitors.
In the Google Analytics module all roles are set.
Answer: First of all this is not a Drupal Question, but anyways.
You can see Google Analytics for Anonymous users in your Google webmasters dashboard. Drupal has nothing to do with this.
Drupal Module for Google Analytics has only one function - Tell Google that it's your website and track its activities. Google does the real work of tracking.


Why is use of specifc enum invalid in generic method with enum as the type

Question: Why does this line of code (context below)
if (spellEnum == Spell.Fireball) return "Fire ball";

produce the error:

'Spell' is a type parameter, which is not valid in the given context

public abstract class DataSet : ScriptableObject
{
    public abstract string DisplayName<T>(T arg) where T : Enum;
}

public enum Spell { Fireball, Icestorm, MagicMissile };

public class SpellDataSet : DataSet
{
    public override string DisplayName<Spell>(Spell spellEnum)
    {
        if (spellEnum == Spell.Fireball) return "Fire ball";

        return "Other spell";
    }
}

public class DataLibrary
{
     void main()
    {
        SpellDataSet spellDataSet = new SpellDataSet();
        strint test = spellDataSet.DisplayName(Spell.Fireball);
    }
}

I want a generic way to access data about various types (spells, items, characters), with data about specific kinds of that type (Spell1, Spell2, etc) accessed via that type's respective enum (Spell), and a universal method (defined in abstract class) across all types will produce the user-friendly name (DisplayName).
Answer: I still don't understand why the original code produces an error, but I have achieved what I wanted via making the abstract class generic, rather than only the DisplayName method.  Code that works:
public abstract class DataSet<T> : ScriptableObject where T: Enum
{
    public abstract string DisplayName(T arg);
}

public enum Spell { Fireball, Icestorm, MagicMissile };

public class SpellDataSet : DataSet<Spell>
{
    public override string DisplayName(Spell spellEnum)
    {
        if (spellEnum == Spell.Fireball) return "Fire ball";

        return "Other spell";
    }
}

public class DataLibrary
{
    void main()
    {
        SpellDataSet spellDataSet = new SpellDataSet();
        string test = spellDataSet.DisplayName(Spell.Fireball);
    }
}


Django Haystack __contains field lookup not working

Question: Recently I use Haystack and whoosh for keyword search in my django project.But I use SearchQuerySet to filter with "__contains" and return error result.There is model and index.
class Team(models.Model):
   name = models.CharField(max_length=NAME_MAX_LENGTH, default='')
   leader = models.CharField(max_length=NAME_MAX_LENGTH, default='')
   slogan = models.CharField(max_length=SHORT_TEXT_LENGTH, default='')
   about = models.CharField(max_length=LONGTEXT_MAX_LENGTH, default='')
   b_type = models.IntegerField(default=0)
   ...

class TeamIndex(indexes.SearchIndex, indexes.Indexable):
   text = indexes.CharField(document=True, use_template=True)

   team_name = indexes.CharField(model_attr='name')
   team_logo = indexes.CharField(model_attr='logo_path')
   team_about = indexes.CharField(model_attr='about')
   team_type = indexes.CharField(model_attr='b_type')

   def get_model(self):
       return Team

   def index_queryset(self, using=None):
       return self.get_model().objects.all()

As fallow, I want search some result which contains key words.such as use "student" to match "The student is good.".
condition = reduce(operator.and_, (Q(content__contains=x) for x in keys))
res = SearchQuerySet().filter(condition).models(model)

But It also return null.So I look up the indexes which whoosh return.It can return a good result.
enter image description here
But when I use haystack to filter the result, It return error result.
(1)"__contains" looks like "__exact"
>>> SearchQuerySet().filter(text='rw\n').count()
3
>>> SearchQuerySet().filter(content='rw\n').count()
3
>>> SearchQuerySet().all().filter(content__contains='w').count()
0
>>> SearchQuerySet().all().filter(text__contains='w').count()
0

(2)"__exact" return error result
>>> SearchQuerySet().filter(text__contains='y\n1231').count()
3

But I only have one index which match "y\n1231".
Exceptally, I try some ways but fails.

use "NgramField" or "EdgeNgramField" instead of "CharField"
use
"SearchQuerySet().exclude(content="XXX").filter(content__contains='w').count()"

Ps:
Python: 3.5.2
Django: 1.10.5
django-haystack: 2.6.0
whoosh: 2.7.4
jieba: 0.38

Answer: In my opinion, I finally solve the problem.I want to share my error if some other one meet it again.Before, I only use one charter for keyword, so whoosh never create index on one character. such as "a", "b","c".I use whoosh API to debug the code.
>>> from whoosh.index import open_dir
>>> ix = open_dir('whoosh_index')
>>> searcher = ix.searcher()
>>> list(searcher.lexicon("text"))
[b'1231', b'about', b'jack', b'rw', b'tom']

I think it needs more characters if whoosh want to work.such as "jack", "about".


How to map internal OIDC group to external K8s cluster roles

Question: I have successfully connected my K8s cluster + dashboard to my Keycloak server, now i have asked myself the question:
I have followed these instructions here.
Furthermore I also made the appropriate adjustments to the kube-apiserver.yml. In my opinion these two yaml's should do the corresponding mapping. Does anyone happen to have a corresponding configuration and can validate my assumption.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: devops-cluster-admin
  namespace: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: devopstales

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
subjects:
- kind: User
  name: "devopstales"
  namespace: "kube-system"
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name:  cluster-admin

How can I achieve a role mapping between the OIDC roles/ groups and the K8s roles?
Answer: You will want to specify --oidc-groups-claim= with the JWT claim containing the list of strings that identify the groups to which the authenticated user belongs, then you can reference those names (with any optional oidc-groups-prefix you specified) in RBAC situations to grant cluster permissions to those groups
Comment: Ok, thanks for your answer. So if I understand this correctly it is not possible to map a rights structure. E.g. that several OIDC groups are mapped to different RBAC rights.
Comment: I don't know keycloak well enough to know how much it would help push RBAC Roles into the cluster for you. If your situation is extremely dynamic, using a webhook authorization scheme may interest you, and this would be worth reading, also: https://github.com/keycloak/keycloak-operator
Comment: Thanks for the hint, currently I use the [louketo-proxy](https://github.com/louketo/louketo-proxy). Maybe the Keycloak is the better alternative to map multiple groups and roles. I will check that.


Arimax Prediction : Using Forecast Package

Question: The arimax function in the TSA package is to my knowledge the only R package that will fit a transfer function for intervention models. It lacks a predict function though which is sometimes needed.
Is the following a work-around for this issue, leveraging the excellent forecast package? Will the predictive intervals be correct? In my example, the std errors are "close" for the components.

Use the forecast package arima function to determine the pre-intervention noise series and add any outlier adjustment.
Fit the same model in arimax but add the transfer function
Take the fitted values for the transfer function (coefficients from arimax) and add them as xreg in arima.
Forecast with arima

library(TSA)
library(forecast)
data(airmiles)
air.m1<-arimax(log(airmiles),order=c(0,0,1),
              xtransf=data.frame(I911=1*(seq(airmiles)==69)),
              transfer=list(c(1,0))
              )

air.m1

Output:
Coefficients:
  ma1  intercept  I911-AR1  I911-MA0
0.5197    17.5172    0.5521   -0.4937
s.e.  0.0798     0.0165    0.2273    0.1103

sigma^2 estimated as 0.01223:  log likelihood=88.33
AIC=-168.65   AICc=-168.09   BIC=-155.02

This is the filter, extended out 5 more periods that the data
tf<-filter(1*(seq(1:(length(airmiles)+5))==69),filter=0.5521330,method='recursive',side=1)*(-0.4936508)
forecast.arima<-Arima(log(airmiles),order=c(0,0,1),xreg=tf[1:(length(tf)-5)])
forecast.arima

Output:
Coefficients:
         ma1  intercept  tf[1:(length(tf) - 5)]
      0.5197    17.5173                  1.0000
s.e.  0.0792     0.0159                  0.2183

sigma^2 estimated as 0.01223:  log likelihood=88.33
AIC=-168.65   AICc=-168.28   BIC=-157.74

Then to Predict
predict(forecast.arima,n.ahead = 5, newxreg=tf[114:length(tf)])

Comment: +1, very clever I see nothing wrong with the approach, s.e looks good, I'm not sure which s.e. You are referring?
Comment: Ah forecaster! I missed you! :) I meant the SE of the MA and intercept between the two fits. The  residual variance is identical. I think that the residual variance is all that matters for the prediction error of the forecast from an ARIMA model (although I am rusty).
Comment: Net stumblers, for the filtering, I found quite an insight here: https://onlinecourses.science.psu.edu/stat510/node/75
Comment: Now I get it, I would be surprised if the s.e. is not close, being close is good, sigma^2 is used for prediction intervals, there is no difference between two models so prediction intervals will be same for arimax or Arima I believe.
Comment: The way you define `tf <- filter(...)`... I am lost. Do you have any tips to understand it? What would happen if I had: `I911-AR1: 0.55`, `I911-AR2: 0.66`, `I911-MA0: 0.49`, `I911-MA1: 0.39` ?
Answer: Prediction intervals are based on residual variance as estimated by the maximum likelihood optimization.
I like forecast function from the forecast package:
fc <- forecast(forecast.arima,h = 5, xreg=tf[114:length(tf)])

which gives me the following predictions:
  Point Forecast    Lo 80    Hi 80    Lo 95    Hi 95
Jun 2005       17.61393 17.47222 17.75565 17.39720 17.83067
Jul 2005       17.51725 17.35753 17.67697 17.27299 17.76152
Aug 2005       17.51725 17.35753 17.67697 17.27299 17.76152
Sep 2005       17.51725 17.35753 17.67697 17.27299 17.76152
Oct 2005       17.51725 17.35753 17.67697 17.27299 17.76152

the way prediction interval works is by following formula:
$y_t \pm z\sigma$ where $z$ is a multiplier which takes values such as 1.96 for 95% prediction interval and 1.28 for 80% prediction interval $\sigma$ is the standard deviation of the residual also forecast distribution which is also the square root of sigma^2 in the maximum likelihood estimate. As you show sigma^2 ( 0.01223) is identical in both your models, prediction intervals will also be same and will match.
If you want to check it,
Upper Limits:
> fc$mean[1]+sqrt(forecast.arima$sigma2)*1.96
[1] 17.83068

Lower Limits:
> fc$mean[1]-sqrt(forecast.arima$sigma2)*1.96
[1] 17.39719

which matches the prediction interval provided by the forecast function. To answer your question, yes leveraging forecast package will work in this case and the prediction intervals will be correct.


Hibernate One To Many Eager Not Pulling in all Data

Question: I am doing an eager load on a one to many relationship in hibernate.  The parent items are pulled back correctly but they only recieve one item each in their child list.  One of the parent's should have two.  I run the query Eclipse spits out, and it pulls the correct results.  Question is why would only one item go into each list when one should have two?
<code> @OneToMany(mappedBy="badge", fetch=FetchType.EAGER)
     public List<BadgeLevel> getBadgeLevels() {
        return this.badgelevels;
     }
</code>
SQL
<code>select
        *
    from
        ( select
            this_.ID as ID10_2_,
            this_.ACTIVE as ACTIVE10_2_,
            this_.DATECREATED as DATECREA3_10_2_,
            this_.DATEMODIFIED as DATEMODI4_10_2_,
            this_.DESCRIPTION as DESCRIPT5_10_2_,
            this_.ENDDATE as ENDDATE10_2_,
            this_.GLOBAL as GLOBAL10_2_,
            this_.NAME as NAME10_2_,
            this_.PUBLISHDETAILS as PUBLISHD9_10_2_,
            this_.STARTDATE as STARTDATE10_2_,
            (SELECT
                COUNT(*)
            FROM
                BADGELEVELS bl
            WHERE
                bl.BADGEID = this_.ID) as formula0_2_,
            badgelevel2_.BADGEID as BADGEID4_,
            badgelevel2_.ID as ID4_,
            badgelevel2_.ID as ID9_0_,
            badgelevel2_.ACTIVE as ACTIVE9_0_,
            badgelevel2_.ASSETID as ASSETID9_0_,
            badgelevel2_.BADGEID as BADGEID9_0_,
            badgelevel2_.DATECREATED as DATECREA3_9_0_,
            badgelevel2_.DATEMODIFIED as DATEMODI4_9_0_,
            badgelevel2_.DESCRIPTION as DESCRIPT5_9_0_,
            badgelevel2_.FILTERS as FILTERS9_0_,
            badgelevel2_."ORDER" as ORDER9_0_,
            asset3_.ID as ID2_1_,
            asset3_.ACTIVE as ACTIVE2_1_,
            asset3_.DATECREATED as DATECREA3_2_1_,
            asset3_.DATEMODIFIED as DATEMODI4_2_1_,
            asset3_.DESCRIPTION as DESCRIPT5_2_1_,
            asset3_.FILENAME as FILENAME2_1_,
            asset3_.FILEPATH as FILEPATH2_1_,
            asset3_.TITLE as TITLE2_1_,
            asset3_.TYPE as TYPE2_1_
        from
            TEST.BADGES this_
        inner join
            TEST.BADGELEVELS badgelevel2_
                on this_.ID=badgelevel2_.BADGEID
        inner join
            TEST.ASSETS asset3_
                on badgelevel2_.ASSETID=asset3_.ID
        where
            this_.ACTIVE=1
        )
</code>
Comment: Can you post the mapping of both classes? At least ids, the relationships and any annotations at the class level.
Comment: It was rownum which was three.  But my one to many is pulling five results instead of three.  The five come from the outer join happening.  I thought hibernate was smart enough go combine these into three entities with their children attached.  Am I not correct?
Answer: To get what I needed, the page size was off.  So increasing that showed all the outer joins coming through, but they weren't being constructed correctly.  So I added @Fetch annotation to fix that issue.
<code>     @OneToMany(mappedBy="badge", fetch=FetchType.EAGER)
     @Fetch(value=FetchMode.SELECT)
     public List<BadgeLevel> getBadgeLevels() {
        return this.badgelevels;
     }
</code>


Visual Studio Online SDK

Question: I am working on a project where I have a requirement to create workitem on Visual Studio Online instance. I am using personal access token. This will set CreatedBy as my name (Expected behavior). I am considering to use Oauth2; However, I am not sure if there's the way to do this Server-to-Server (Non-Interactive)? Any suggestions thoughts?

            var personalAccessToken = "PAT Value fro Config";
            HttpClient client = new HttpClient();
            client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
            client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic",
                Convert.ToBase64String(Encoding.ASCII.GetBytes(string.Format("{0}:{1}", "", personalAccessToken))));

            return client;

PAT's are created in Security context of the user. I need to find a way to use Oauth without having to involved UI. So I'm looking for Server-to-Server Auth.

object[] patchDocument = new object[5];
            patchDocument[0] = new { op = "add", path = "/fields/System.Title", value = bugTitle };
            patchDocument[1] = new { op = "add", path = "/fields/Microsoft.VSTS.TCM.ReproSteps", value = bugReproSteps };
            patchDocument[2] = new { op = "add", path = "/fields/Microsoft.VSTS.Common.Priority", value = "1" };
            patchDocument[3] = new { op = "add", path = "/fields/Microsoft.VSTS.Common.Severity", value = "2 - High" };
            patchDocument[4] = new { op = "add", path = "/fields/System.IterationPath", value = deserializeIteration };
            //System.IterationPath
            string postUrl = $"{_vsoInstanceUrl}/DefaultCollection/ProjectName/_apis/wit/workitems/$Bug?api-version=1.0";
await ExecutePatch(patchDocument.ToArray(), postUrl, "application/json-patch+json");
Comment: Your best bet is to research this (asking a Q here is not really research because of how it works) and try some things. Come back with some code and your results if you run into problems. https://stackoverflow.com/help/how-to-ask
Answer: No there is no Server-to-Server OAuth support. If you use the .NET Client Object Model you can leverage Impersonation support.
If your account has "Act on behalf of others" permissions you can also achieve a "User X via YourAccount".

Visual Studio Online SDK


How to save and append comment form without refresh the page in drupal 7

Question: I have a form with comments but this page have counter to cout views of this page.
but every time when comment added on this page that page loaded and my counts increasing
  after comment added.
but i want to count views when page load first time not by comments module  in drupal 7.
Answer: One option would be to use the AJAX Comments module:

Provides ajax comments to Drupal sites. Features:

Makes AJAX submits
Allows replies to comments on the actual thread page
AJAX comments deleting
Works with CAPTCHA
Cool jQuery sliding
views integration

It will allow users to submit comments without a page refresh, bypassing the extra view count.


Facebook API's fb:registration returning "Invalid 'client_id'" when not connected to Facebook

Question: I'm trying to use Facebook's registration tool for my website. I would like to allow Facebook users to login seamlessly to my website, but also non-Facebook users to create an account through that unified UI.
I'm using the <fb:registration> tag for that purpose. If I'm already connected to Facebook when arriving to the page, everything works as expected (the form is prefilled with information from my Facebook account). But if I'm not connected to Facebook, the following error is displayed: Invalid 'client_id'..
What am I doing wrong?
Thanks very much for your help.
You'll find below the HTML code rendered from the server:
<!doctype html>

<title>MyWebSite - Signup</title>
<link rel='stylesheet' type='text/css' href='/client/Core.css'/>

<fb:registration
    fields='[{"name":"name"},{"name":"first_name"},{"name":"last_name"},{"name":"email"},{"name":"location"},{"name":"gender"},{"name":"password","view":"not_prefilled"},{"name":"captcha"}]'
    redirect-uri='http://172.16.100.31/DoSignup'
    width='600'
    fb_only='false'
    allowTransparency='true'>
</fb:registration>

Answer: You might have to configure your application to let users other than the developers to access the app. I had a hard time finding it, but if I remember well, it was a matter of sandboxing the app or not.
Comment: Thank you very much, that was it! That makes sense, but the ambiguous error message doesn't really help...
Comment: In your app on the developer page, go to About->Advanced->Authentication->Sandbox Mode and click Disabled.

I enabled this thinking it'd be fine since I only want developer access, but I guess that's incompatible with registation! Silly me.
Comment: OMG this was killing us - thank you so much! This was a real doh! moment.
Comment: and sandbox was the solution.
Comment: Phew, thanks for that. Facebook and their lack of documentations!
Answer: I ran into the same problem, except I had the sandbox disabled already.
I had included the script tag as

<fb:registration
  fields="name,birthday,gender,location,email"
  redirect-uri="https://developers.facebook.com/tools/echo/"
  width="530">
</fb:registration>

as it was on the registration example page, however, the JavaScript SDK page had

Removing the #appId={YOUR_APP_ID}&xfbml=1 from the external script tag did the trick.
Answer: If you registered your application with Facebook they will give you an appID which you will need to use instead of the '123456789' AppID in the code above.
Comment: Thanks for your answer, but I already own a Facebook app ID, which I'm using instead of the "123456789" above. I was using this fake number just to avoid giving out my app ID.
Other than that, do you see anything wrong in the code above? If you own an app ID, could you copy/paste this code and try it out with your app ID?
Answer: Just guessing here - because I had a similar but non-identical problem:
If you use the Facebook XFBML code on http://developers.facebook.com/docs/plugins/registration/ and replace the word {YOUR_APP_ID} by {1234123434534} (or whatever), it fails because it isn't expecting the curly brackets. (Different from various other areas that use JSON and are expecting them).
I don't know if this is what you've done wrong, but it's an easy mistake to make, as I just discovered.
Answer: For testing set your YOUR_APP_ID to 113869198637480
Answer: In my case, I was displaying the fb registration link without logging in the user first. After scratching my head for a long time, I asked the user to log in through facebook before presenting the registration form. The error was gone much to my delight.


How can I use an owl dataset anyOntology.owl for generating grails domain classes

Question: I'm an owl/protege newbe with some grails experience.
I would like to transform my protege ontology file anyOntology.owl to grails domain classes. In the next step I would use grails generate-all command to obtain the respective controllers and views to obtain the executable model with CRUD functionality, which would be my executable ontology.
Is there already something like a protege-export or a grails-import as a plugin, or is someone already planning/working on such a feature?
Peter
Answer: in the meantime I developed a 0.1 Version for the above Problem and promoted it to
https://github.com/pstrotmann/ontology/tree/HEAD/owlToOO
It is a grails Application, that transforms an xxxx.owl file to grails Domain classes that can be transformed to a grails application by using the generate_all feature of grails thus obtaining a CRUD functionality.
In addition a graphViz input is generated from the xxxx.owl file that can be transformed by graphViz to an UML-Diagram.
Answer: in addition there is runnable version of owlToOO that can be accessed via
http://partnerontology.strotmann.org
with a generated Partner application