Add custom fields to JSON file based on user input from YAML file - Python3

Question: I am new to Python. I am having the one YAML file and I am accessing it with a Python file. In YAML file, there is fields option. In YAML file, User can set variables with values. The Python file reads the variables with values and then adds it to the JSON file. Note that the variables and values can be changed according to user.
How can I make this possible?
Here is the sample code:
import yaml
from datetime import datetime
import os
import json

#name for json file
name = "stack.json"

#load data from yml file
data = yaml.safe_load(open('stack.yml'))
data2 = data.get('heartbeat.monitors')

#Current time stamp
timestamp = datetime.now().strftime("%B %d %Y, %H:%M:%S")

#ip
ip ='192.168.1.1'

#getting data from the field and assign it to variable
for item in data2:
    if item["type"] == "icmp":
        fields_under_root = (item["fields_under_root"])

        # if fields_under_root is true,code goes here
        if fields_under_root == True:
            fields = (item["fields"])
            print(fields)
            locals().update(fields)
        #code to be entered

        #if fields_under_root is false, code goes here
        elif fields_under_root == False:
            fields = (item["fields"])
            print(fields)
        #code to be entered
#For writing in JSON File
#Creates a JSON file if not exists
if not os.path.exists(name):
    with open(name, 'w') as f:
        f.write('{}')

#list for storing the values
result = [(timestamp, {'monitor.ip': ip,"fields": fields })]

#For writing in JSON File
with open(name, 'rb+') as f:
    f.seek(-1, os.SEEK_END)
    f.truncate()
    for entry in result:
        _entry = '"{}":{},\n'.format(entry[0], json.dumps(entry[1]))
        _entry = _entry.encode()
        f.write(_entry)
    f.write('}'.encode('ascii'))

In YAML file:
heartbeat.monitors:
- type: icmp
  fields:
    a: steven
    b: kumar

  fields_under_root: True

My output in JSON file:
{"February 18 2019, 17:04:30":{"monitor.ip": "192.168.1.1", "fields": {"b": "kumar", "a": "steven"}},
}

Required output if fields_under_root is True:
{"February 18 2019, 17:04:30":{"monitor.ip": "192.168.1.1", "b": "kumar", "a": "steven"},
}

Required output if fields_under_root is False:
{"February 18 2019, 17:04:30":{"monitor.ip": "192.168.1.1", "fields.b": "kumar", "fields.a": "steven"},
}

Comment: why do the conditional checks in your code do the same thing? also why are you updating `locals`
Comment: I didn't add the code on conditional fields. Tell me some suggestion to make the output.
Comment: Next time, just edit your question instead of [deleting](https://stackoverflow.com/questions/54746721/add-custom-fields-based-on-user-input-to-json/54746977#54746977) it, and copy-and-pasting the contents into a new question.
Answer: Given an input file:
...

heartbeat.monitors:
- type: icmp
  fields:
    a: steven
    b: kumar
  # If this option is set to true, the custom fields are stored as top-level
  # fields in the output document instead of being grouped under a fields
  # sub-dictionary. Default is false.
  fields_under_root: True

your program should have a function that does all the work, then it is easy to test
both versions of the input, by updating the input file.:
import ruamel.yaml
from datetime import datetime
import os
import json

# name for json file
json_name = 'stack.json'
yaml_name = 'stack.yaml'

# Current time stamp
timestamp = datetime.now().strftime('%B %d %Y, %H:%M:%S')

def gen_output(data, json_filename):
    ip = '192.168.1.1'
    val = {'monitor.ip': ip}
    result = dict(timestamp=val)

    # getting data from the field and assign it to variable

    for item in data:
        if item['type'] == 'icmp':
            fields = item.pop('fields')
            if item['fields_under_root']:
                val.update(fields)
            else:
                nf = {}
                for k in fields:
                    nf['fields.' + k] = fields[k]
                val.update(nf)

    with open(json_filename, 'w') as fp:
        json.dump(result, fp, ensure_ascii=False)

# load data from YAML file
yaml = ruamel.yaml.YAML(typ='safe')
with open('stack.yaml') as fp:
    data = yaml.load(fp)
data2 = data.get('heartbeat.monitors')

gen_output(data2, json_name)

# show the output file
with open('stack.json') as fp:
    print(fp.read())

print('---------')
# update the YAML input file
with open('stack.yaml') as fp:
    yamlrt = ruamel.yaml.YAML() # default is round-trip, preserving comments
    data = yaml.load(fp)
data['heartbeat.monitors'][0]['fields_under_root'] = False
with open(yaml_name, 'wb') as fp:
    yamlrt.dump(data, fp)

with open('stack.yaml') as fp:
    data = yaml.load(fp)
data2 = data.get('heartbeat.monitors')

gen_output(data2, json_name)
with open('stack.json') as fp:
    print(fp.read())

which gives:
{"timestamp": {"monitor.ip": "192.168.1.1", "a": "steven", "b": "kumar"}}
---------
{"timestamp": {"monitor.ip": "192.168.1.1", "fields.a": "steven", "fields.b": "kumar"}}

Generic comments:

don't test for == False or == True, if at all test with is
False or is True, but better just test a variable x that
contains a boolean with if x: and don't do both, a boolean, if not
True is False, hence use a plain else:
don't hassle with outputting JSON. No need to truncate, seek
etc. Just make sure your data is correctly formed and then dump in one go. That
makes sure your output is valid JSON.
don't add comments stating the obvious  #ip
add a space after the comment token #, many lint-ers complain about not having that
I use ruamel.yaml in the above (disclaimer: I am the author of that package), because with
PyYAML programmaticly updating your YAML file would lose you the comment information.
And because YAML(typ='safe').load() is faster than PyYAML's yaml.load()
(Apart from that PyYAML only supports the outdated YAML 1.1)
Comment: Thanks for the Code. It worked. Thanks for the Comments. I will keep in mind and rectify my mistakes in future.


When are even symmetric matrices invertible?

Question: Let $A$ be a symmetric $n\times n$ real matrix. By even, I mean that
$$i+j \text{ is odd } \Rightarrow A_{ij}=0$$
What can be said about the invertibility of such symmetric even matrices? I would be looking for a characterization of which such matrices are singular and which are non-singular that exploits both the even and symmetric properties of the matrix.
Comment: Surely if you re-order the basis you are just looking at pair of symmetric matrices?
Answer: By rearranging the rows and columns, we can produce a block-diagonal matrix. For example,
$$
\pmatrix{a_{11}&0&a_{13}&0\\
0 & a_{22} &0 & a_{24}\\
a_{31}&0&a_{33}&0\\
0&a_{42}&0&a_{44}} \to
%
\pmatrix{a_{11}&a_{13}&0&0\\
0 & 0& a_{22} & a_{24}\\
a_{31}&a_{33}&0&0\\
0&0&a_{42}&a_{44}} \to
\pmatrix{a_{11}&a_{13}&0&0\\
a_{31}&a_{33}&0&0\\
0 & 0& a_{22} & a_{24}\\
0&0&a_{42}&a_{44}}.
$$
In this case, the first matrix will be invertible if and only if the two matrices
$$
\pmatrix{a_{11} & a_{13}\\ a_{31}&a_{33}}, \quad \pmatrix{a_{22} & a_{24}\\ a_{42} & a_{44}}
$$
are invertible.


Prove that if both $N$ and $G/N'$ are nilpotent, then $G$ is nilpotent.

Question: Let $G$ be a group, and let $N$ be a normal subgroup of $G$. Let $N'$ be the derived group of $N$. Prove that if both $N$ and $G/N'$ are nilpotent, then $G$ is nilpotent. Furthermore, assume that $G$ is arbitrary, not necessarily finite. Prove that if $N$ is nilpotent and $G/N'$ is abelian, thn $G$ is nilpotent and $C_{i}(G)=C_{i}(N)$ for $2 \leq i $
Recall that $C_{i}(G)$ is the higher commutator subgroup of a group $G$
I think the lemma maybe usefull: If the factor group $G/ \Phi G$ is nilpotent for a finite group $G$, then $G$ itself is nilpotent.
So I think we should prove that if $G/N'$ is nilpotent then $G/ \Phi G$ is nilpotent.
How do I prove it?
P/s: There is a details proof on the book The Theory of Infinite Soluble Groups, JOHN C. LENNOX and DEREK J. S. ROBINSON, $(1.2.17)$ page $12$, but it looks complicated because they use stronger tools
Comment: This post maybe helpfull http://math.stackexchange.com/questions/113894/prove-if-n-lhd-g-and-n-and-fracgn-are-nilpotent-then-g-is-nilpot
Comment: Can anyone help me?
Answer: For finite groups there is an easier proof, using the following facts. The proof of these facts can be found in the spoiler. All groups here are finite. Recall that $\Phi(G)$ is the set of non-generators of $G$ and the intersection of all maximal subgroups of $G$. And that a group is nilpotent if and only if all its maximal subgroups are normal (here is where we need finiteness).
Lemma 1 A group $G$ is nilpotent if and only if $G' \subseteq \Phi(G)$.
Lemma 2 If $N \unlhd G$, then $\Phi(N) \unlhd \Phi(G)$.
Now assume that $G/N'$ and $N$ are nilpotent. By Lemma 1, we have $N' \subseteq \Phi(N)$, and by Lemma 2, it follows that $N' \unlhd \Phi(G)$. If $M$ is a maximal subgroup of $G$. Then $\Phi(G) \subseteq M$, whence $N' \subseteq M$, implying $M/N'$ is maximal in $G/N'$. Since $G/N'$ is nilpotent it follows that $M/N' \unlhd G/N'$ and by the correspondence theorem this is equivalent to $M \unlhd G$. So all maximal subgroups are normal, and $G$ is nilpotent.

!Proof of Lemma 1 If a group is nilpotent and let $M$ be a maximal subgroup. Then $M$ is normal and hence $G/M$ has no non-trival subgroups. This can only be if $M$ has prime index in $G$, whence $G/M$ is abelian and so $G' \subseteq M$. Since $\Phi(G)$ is the intersection of all maximal subgroup, we get $G' \subseteq \Phi(G)$. Conversely, if $G' \subseteq \Phi(G)$, then for any maximal subgroup $M$ we have $G' \subseteq \Phi(G) \subseteq M$, so $G' \subseteq M$ and $M$ is normal.
Proof of Lemma 2 Observe that $\Phi(N)$ is characteristic in $N$ and $N$ is normal in $G$, so $\Phi(N)$ is normal in $G$. Hence for a maximal subgroup $M$ of $G$, $M\Phi(N)$ is a subgroup containing $M$. Assume $G=M\Phi(N)$. By Dedekind Lemma, we get $N=N \cap M\Phi(N)=(N \cap M)\Phi(N)=N \cap M \subseteq M$. Hence $\Phi(N) \subseteq N \subseteq M$ and it follows that $G=M\Phi(N)=M$ a contradiction. We conclude that $\Phi(N) \subseteq M$ for any maximal $M$, hence $\Phi(N) \subseteq \Phi(G)$ as required.
Comment: Can you explain more precise the second statement?
Comment: @chuyenvien94 Observe by the way, that all I wote down above, does not depend on $G$ being finite. OK, let me give you one step further: every abelian group is nilpotent, so you can apply the first statement: $G$ must be nilpotent. Further $G'=N'$.
Comment: @chuyenvien94 I have put a spoiler in my answer for the last part of your question. But I encourage you to give it a try yourself.
Comment: @chuyenvien94 you are welcome. And if you deem my answer as the right one please tick as such, thanks!
Comment: Perhaps I have misunderstood, but your first paragraph does not seem to help. You seem to have shown that $\gamma_n(G) \subseteq N' \Rightarrow \gamma_{n+1}(G) \subseteq N'$ (which doesn't need an elaborate proof since $\gamma_{n+1}(G) \subseteq \gamma_n(G)$).
Comment: @Nex 23 - you are perfectly right, my proof is bogus and I thank you for noticing it. I replaced it with a proof that only works for *finite* groups as far as I know.
Comment: @NickyHekster Assuming $N$ is nilpotent of degree n and $G/N$ is nilpotent of degree $m$, what does this say about the degree G is nilpotent of?
Comment: @NickyHekster Really?  Can you please provide a counterexample?
Comment: @user558377 Nothing, since the nilpotence of $N$ and $G/N$ does not imply that $G$ is nilpotent. Take $G=S_3$ and $N=A_3$.
Comment: @user558377 What would be a really good question is as follows: (and you should post this as a new question), let $G$ be a group, $N \lhd G$, such that $G/N'$ and $N$ are both nilpotent. What can be said about the nilpotency class of $G$ in terms of those of $G/N'$ and $N$? I know that a theorem of P. Hall (1958) gives a bound but better bounds have been found. I do not know if for finite groups the proof of Hall can be simplified, since he uses the machinery of commutative algebra.


How do I pull content from a view into an array?

Question: I hav a strange task where I need to pull the content from a (well some) drupal view into an array so I can use that data later. I dont want to display contents right away, but I'll need to access it later.
I have already built the view but it was suggested that I can export the view and then create a module to grab the view data but I have no idea how to do that.
Basically what Im doing, is creating a view to pull say, 9 records of node type "blog" and on the front page, I want to pull that view, throw the results into a json array, and then do stuff with it later.
Answer: Not sure if it's what you're looking for, but if you're using a template to display that view, you can access an array of the view data by using the php code
<code>$view->style_plugin->rendered_fields;
</code>
This will give you an array of everything that the view has returned.


Surface area of platonic solids inscribed in unit sphere.

Question: If one inscribed each of the platonic solids in the unit sphere, what can be said about the surface areas? Is it a monotonic sequence? Does any "meaningful" (see paragraph below) continuous (or discrete) function $f(n)$, n the number of faces, outputting the inscribed area exist?
By "meaningful", I mean not just curve-fitting, but something that is significant for values of n other than 4, 6, 8, 12, or 20, even though there are no platonic solids for these other values of n. The most obvious thing in this category is that $f(n) \rightarrow 4\pi$ as $n \rightarrow \infty$.
I'm not sure exactly $\textbf{how}$ this significance would occur. I guess I'm wondering if you could discover other patterns that exist. Starting places might be properties involving n even or odd, prime, a Fermat number, or perhaps having the inscribable solid be as close as possible to regular (whatever that means), non-convex, etc.?
Related questions:
Strong equidistribution of points on the n-sphere
What property of certain regular polygons allows them to be faces of the Platonic Solids?
Comment: In particular, note that the surface area is an increasing function of the number of vertices, not the number of faces.
Comment: The surface areas of the solids in question are $8/\sqrt{3}\approx 4.6188$ for the tetrahedron, $8$ for the cube, $4\sqrt{3} \approx 6.9282$ for the octahedron, $2\sqrt{50-10\sqrt{5}}\approx 10.5146$ for the dodecahedron, and $2\sqrt{3}(5-\sqrt{5})\approx 9.5745$ for the icosahedron.  The sphere itself has a surface area of $4\pi \approx 12.5664$.
Answer: The Platonic solids are best viewed as consisting of two families – those with triangular faces (the tetrahedron, the octahedron and the icosahedron), which for given edge count have maximal number of faces and minimal number of vertices, and their duals (the tetrahedron, the cube and the dodecahedron), in which three faces meet at each vertex and which for given edge count have maximal number of vertices and minimal number of faces. (The tetrahedron appears in both families because it's self-dual.)
For each of these families, we can derive a formula of the sort you're looking for. In both cases, we'll need the area and the solid angle of a regular polygon inscribed in a sphere. Consider a regular $N$-gon as consisting of $N$ triangles formed by its centre and the $N$ pairs of neighbouring vertices. Let $r$ be the distance from the centre to the vertices. Then each such triangle has area
$$\frac{r^2}2\sin\frac{2\pi}N\;,$$
and the $N$-gon has $N$ times that area. The solid angle $\Omega$ subtended by the triangle can be calculated from the dihedral angles of the tetrahedron it forms with the origin (see Wikipedia), namely as the excess over $\pi$ of the sum of the three dihedral angles at the origin. The calculation is a bit of a mess but leads to a nice result: If we want $n$ of these $N$-gons to subtend the solid angle $4\pi$ of the sphere, we need
$$
\sqrt{1-r^2}=\frac{\tan\left(\frac\pi N-\frac{2\pi}{Nn}\right)}{\tan\frac\pi N}\;.
$$
Solving this for $r$, substituting into the first equation and multiplying by $Nn$ to get the total area of the polyhedron yields
$$
\begin{eqnarray}
A(n,N)&=&\frac{Nn}2\left(1-\left(\frac{\tan\left(\frac\pi N-\frac{2\pi}{Nn}\right)}{\tan\frac\pi N}\right)^2\right)\sin\frac{2\pi}N\\
&=&\frac{Nn\cos^2\frac\pi N}{\tan\frac\pi N}\left(\tan^2\frac\pi N-\tan^2\left(\frac\pi N-\frac{2\pi}{Nn}\right)\right)\;.
\end{eqnarray}
$$
Now we can apply this to the two families of Platonic solids. For the first one, the one with triangles, we have $N=3$, and
$$
A_1(n)=A(n,3)=\frac{\sqrt3}4n\left(3-\tan^2\left(\frac\pi 3-\frac{2\pi}{3n}\right)\right)\;.
$$
For the three actual Platonic solids in this family, this yields
$$
\begin{array}{c|c|c|c}
n&\text{name}&A_1(n)&\simeq\\
\hline
4&\text{tetrahedron}&8\,/\sqrt3&4.6188\\
8&\text{octahedron}&4\sqrt3&6.9282\\
20&\text{icosahedron}&2\sqrt3\left(5-\sqrt5\right)&9.5745\\
\end{array}
$$
For the second family, we need to first determine $N$ as a function of $n$. In this family, three faces meet at each vertex, so the numbers $v$, $e$ and $f$ of vertices, edges and faces are given by $3v=Nn$, $2e=Nn$ and $f=n$, respectively. Then by Euler's polyhedron formula, $Nn/3-Nn/2+n=2$, so $N=6-12/n$. Thus, for this family we have
$$
\begin{eqnarray}
A_2(n)&=&A\left(n,6-\frac{12}n\right)\\
&=&\frac{6(n-2)\cos^2\left(\frac\pi6\frac n{n-2}\right)}{\tan\left(\frac\pi6\frac n{n-2}\right)}\left(\tan^2\left(\frac\pi6\frac n{n-2}\right)-\tan^2\left(\frac\pi6\frac n{n-2}-\frac\pi{3(n-2)}\right)\right)\;.
\end{eqnarray}
$$
For the three actual Platonic solids in this family, this yields
$$
\begin{array}{c|c|c|r}
n&\text{name}&A_2(n)&\simeq\\
\hline
4&\text{tetrahedron}&8\,/\sqrt3&4.6188\\
6&\text{cube}&8&8.0000\\
12&\text{dodecahedron}&2\sqrt{10\left(5-\sqrt5\right)}&10.5146\\
\end{array}
$$
Here's a plot of $A_1(n)$ (in blue), $A_2(n)$ (in red) and the area $4\pi$ of the sphere (in green):

The second family, with maximal number of vertices and minimal number of faces for given edge count, converges much faster than the first, which is related to what Jim pointed out in a comment, that for the five Platonic solids taken together, the area increases with the number of vertices, not with the number of faces.


If $L$ is a diagonalizable linear operator, why is $f(L)$ well-defined for $f: \mathbb{C} \rightarrow \mathbb{C}$?

Question: In a book on quantum mechanics, I encountered a statement equivalent to the following. Suppose $V$ is a finite dimensional inner product space over $\mathbb{C}$. Let $L : V \rightarrow V$ be a normal operator (or any operator that is diagonalizable). Suppose $\beta = \{ e_i \}$ is a corresponding orthonormal basis of eigenvectors, with eigenvalues $\{\lambda_i\}$.  Let $f: \mathbb{C} \rightarrow \mathbb{C}$ be a function. Then $f(L)_\beta : V \rightarrow V$ defined by $f(L)_\beta(e_i) = f(\lambda_i)e_i$ is independent of the choice of the orthonormal basis of eigenvectors $\{e_i \}$, that is, if $\gamma$ is any other orthonormal basis of eigenvectors of $L$, then $f(L)_\beta = f(L)_\gamma$(So we can meaningfully speak of $f(L)$ if $L$ is diagonalizable).
The book does not provide a proof of this statement, and I seem to be only able to prove it if $L$ has no degenerate eigenvalues. I would be glad if someone could give me an approach to proving it in general.
Comment: What is the problem if you have degenerate eigenvalues?
Answer: For the distinct eigenvalues $\lambda_1,\dots,\lambda_k$, let $V_i$ be the space of eigenvectors for eigenvalue $\lambda_i$. Then $V=V_1+V_2+\cdots V_k$ and $f(L)(v_1+v_2+\cdots+v_k)=f(\lambda_i)v_1+\cdots +f(\lambda_k)v_k$ for $v_i\in V_i$, no matter which eigen-basis you chose. So it is less about a basis, and more about the eigenspaces $V_i$. And the $V_i$ are entirely determined by $L$.
Comment: thank you! so you mean $f(L)$ just multiplies things in the $\lambda_i$-eigenspace by $f(\lambda_i)$?
Comment: Yes. That's it exactly. @M.Van


CSV from list of dictionaries with differing length and keys

Question: I have a list of dictionaries that I want to write to a csv file.
The first dictionary is of a different length and has different keys than the following dictionaries.
dict_list = [{"A": 1, "B": 2}, {"C": 3, "D": 4, "E": 5}, {"C": 6, "D": 7, "E": 8}, ...]

How do I write this to a csv-file so that the file looks like this:
A B C D E
1 2 3 4 5
    6 7 8
    . . .

Comment: Do you really want 1,2,3,4,5 on one single line while they come from 2 different dictionaries?
Answer: The problem is that you will need the full column set to write the header at the beginning of the file. But apart from that, csv.DictWriter is what you need:
# optional: compute the fieldnames:
fieldnames = set()
for d in dict_list:
    fieldnames.update(d.keys())
fieldnames = sorted(fieldnames)    # sort the fieldnames...

# produce the csv file
with open("file.csv", "w", newline='') as fd:
    wr = csv.DictWriter(fd, fieldnames)
    wr.writeheader()
    wr.writerows(dict_list)

And the produced csv will look like this:
A,B,C,D,E
1,2,,,
,,3,4,5
,,6,7,8

If you really want to combine rows with disjoint set of keys, you could do:
# produce the csv file
with open("file.csv", "w", newline='') as fd:
    wr = csv.DictWriter(fd, sorted(fieldnames))
    old = { k: k for k in wr.fieldnames }     # use old for the header line
    for row in dict_list:
        if len(set(old.keys()).intersection(row.keys())) != 0:
            wr.writerow(old)                  # common fields: write old and start a new row
            old = row
        old.update(row)                       # disjoint fields: just combine
    wr.writerow(old)                          # do not forget last row

You would get:
A,B,C,D,E
1,2,3,4,5
,,6,7,8

Comment: This is basically what I was looking for and I appreciate the simpleness. However, the order of the columns gets messed up. Columns "A" and "B" appear between "C", "D", and "E". What exactly is happening during fieldnames.update()?
Comment: @MaxJ.: You can sort the field names (see the edit in the first part of my answer). And I also show how you could combine rows with disjoint set of keys.
Comment: I set the fieldnames by adding the keys of the second dict to the keys of the first dict. Your first solution may even be more suitable for my problem than what I was asking for (the second part). As this works with minimal requirements, I made this the accepted solution.
Answer: You can also use only the built-in functionalities that come with the python language. My example below is similar to that proposed by @Serge Ballesta. The code is as follows:
import csv

# sample data
data = [{'A': 1, 'B': 2}, {'A': 3, 'D': 4, 'E': 5}, {'C': 6, 'D': 7, 'E': 8}]
# Collect from elements in **data** (they are dict object) the field names and store
# them in a **set** to preserve their uniqueness
fields = set()
for item in data:
    names = set(item.keys())
    fields = fields | names   # we used the **or** i.e | operator for **set**

fields = list(fields)   # cast the fields into a list
# and sort the content so that during the display everything is in order :)
fields.sort()

# Now let write a function that return a cleaned data from the original, that is all
# data items have the same field names.

def clean_data(origdata, fieldnames):
    """Turn the original data into a new data with similar field in data items.

    Parameters
    ----------
    origdata: list of dict
         original data which will be cleaned or harmonized according to the field names
    fieldnames: list of strings
         fields names in the new data items

    Returns
    -------
    Returns a new data consisting of list of dict where all dict items have the same
    keys (i.e fieldnames)
    """
    newdata = []
    for dataitem in data:
        keys = dataitem.keys()
        for key in fieldnames:
             if key not in keys:
                  # In this instance we update the datitem with **key** and value= ' '
                  dataitem[key] = ' '
        newdata.append(dataitem)

    return newdata

def main():
    """Test the above function and display the result"""
    newdata = clean_data(data, fields)

    # write the data to a csv file
    with open("data.csv", "w", newline='') as csvfile:
        writer = csv.DictWriter(csvfile, fieldnames=fields)
        writer.writeheader()
        for row in newdata:
            writer.writerow(row)

    # Now let load our newly written csv file and print the content
    # -- some fancy display formatting here: not needed but I like it. :)
    nfields = len(fields)
    fmt = " %s " * nfields
    headInfo = fmt % tuple(fields)
    line = '-'* (len(headInfo)+1)
    print(line)
    print("|" + headInfo)
    print(line)
    with open("data.csv", "r", newline='') as csvfile:
        reader = csv.DictReader(csvfile)
        for item im reader:
            row = [item[field] for field in fields]
            printf("|" + fmt % tuple(row))

    print(line)

main()

The script above will produce the following output:
---------------------
| A | B | C | D | E |
---------------------
| 1 | 2 |   |   |   |
|   |   | 3 | 4 | 5 |
|   |   | 6 | 7 | 8 |
---------------------

Answer: Pandas is able to generate a dataframe from a list of dictionaries if you call pd.DataFrame() on the list. In the resulting dataframe, every dictionary will be one row and every key will correspond to a column. The value corresponding to the 3rd key (I'll call it key3) in the 7th dict therefore will be located in the 7th row of the key3-column.
What this means for your problem: you'll first have to modify your dict_list to include the merged dictionary like so:
dict_list.insert(2, dict(**dict_list[0], **dict_list[1]))
print(dict_list)

[{'A': 1, 'B': 2},
 {'C': 3, 'D': 4, 'E': 5},
 {'A': 1, 'B': 2, 'C': 3, 'D': 4, 'E': 5},
 {'C': 6, 'D': 7, 'E': 8}]

This inserts the combination of the first two dictionaries at index 2 into your list. Why index 2? That allows you to conveniently slice your list when converting it to a dataframe, giving you the desired output
df = pd.DataFrame(dict_list[2:])
print(df)

     A    B  C  D  E
0  1.0  2.0  3  4  5
1  NaN  NaN  6  7  8

For comparison, calling pd.DataFrame on the unmodified list directly gives you
df_unmodified = pd.DataFrame(dict_list)
print(df_unmodified)

     A    B    C    D    E
0  1.0  2.0  NaN  NaN  NaN
1  NaN  NaN  3.0  4.0  5.0
2  NaN  NaN  6.0  7.0  8.0

Afterwards, you can use df.to_csv() to save the dataframe to a csv-file
Comment: This leads to the desired result. Thank you!


How are shorter power-plays counted in power play percentage?

Question: One of the team statics in ice-hockey is power play percentage. From example, you can see it here for the currently played World Chapmionship or here for this year's NHL season.
I understand that this should be something like ratio of power play goals and total number power plays. I wonder whether in some cases the fact that the power play is shorter than 2 minutes is factored in when calculating PP%.
There are several reasons why power play might be shorter, here are some I can think of:

The power play ended because a team scored a goal.
The power play was shorter because a team playing power power received a penalty.
The power play was shorter because the penalty of the opposing team was called while playing shorthanded, which means that there was 4 on 4 play before the power play stared.
The power play was shorter because it started less than two minutes until the end of the game.

In the first case, this was a successful power play - so there is no doubt how this should be counted. In the second case, it is the mistake of the team who had the advantage that they infringed the rules. So it should be counted an unsuccessful power play and the fact that it was shorter does not seem to be relevant. Probably the similar argument can be in the third case.
But at least in the fourth case we can say that it is not the team's fault that the power play they played was shorter. So the team has less time than 2 minutes to score a goal in the power play. So it seems a bit illogical that if they do not score this should be called in the same way as if they  do not score during the whole 2 minutes.
In any case, what I wrote above are just my speculations. I would be glad to find some official source showing how this statistic is counted in these special cases.
Answer: The NCAA statistics rules define "Teams are on a power play when they have at least a one-player advantage on the ice for any amount of time." (section 8).
Using this definition, all of your examples count as one power play opportunity, even if the man advantage situation lasted for only a second. As you point out, this probably isn't actually the best way of measuring the effectiveness of special teams as there's a big difference between a one second power play and a two minute one, so there are various "advanced stats" like power play goals per two minutes of power play and things like that which attempt/claim to be a better measure of special teams effectiveness.
I believe the NHL uses essentially the same definition for power play opportunities, but I haven't been able to find an official definition.


Is it possible to render video from HDMI on iPadOS?

Question: It's very general question. I just want to understand if it's technically possible and why? Also to get some idea of what's the good starting point to investigate this topic.
So question is: Is it technically possible

Connect another device with HDMI-out to iPad with HDMI-in to USB-C adapter
Capture HDMI signal from the device
Render video from the device in iPad app

I guess that it was very limited in previous iOS version.
But in iOS 16 there is a DriverKit now.
Can it be used to solve this problem?
Answer: Short answer: yes.
Long answer:
Previously, this was possible for members of the MFi programme who designed their own MFi-compliant devices. It continues to be the only way to do it for iPhones and iPads which are not based on the M1 SoC.
On iPadOS 16+, on iPads with M1 (or in future, presumably, better) SoC, you can indeed drive near-arbitrary USB devices using a DriverKit extension. Note that there is currently no video capture API you can implement on iPadOS, so you cannot make the device available to any app through a standard interface - each app that uses the DriverKit driver will need to implement that driver's specific IOService user client API. (And the app providing the driver needs to be installed.)
If your capture device also captures audio, it should already be picked up automatically if it's a USB Audio Class compliant device; if it's not class-compliant, you can make it available as a system wide audio device using AudioDriverKit.
Answer: This is available in iPadOS 17+ which adds support for UVC (USB Video Class) input devices -- so they show up in AVFoundation. You can use HDMI-to-USB dongles to plug an HDMI source into the USB-C port of your iPad. I've confirmed cheap dongles that expose UVC work (e.g. ones based on Macrosilicon MS2109 chipset)


How do I apply a discount for the second and successive items only if the quantity for those items is higher than 1?

Question: Using Commerce Discount and Commerce Discount Extra modules there is no problem to set discount rule conditions for product type and quantity. This way I can set up 20% off discount that is applied for a cart line item when this line item has specific product type and quantity more than one. 20% off discount is then calculated for this line item and multiplied by quantity.
However, my need is to apply such discount not to each item from line item quantity. I simply want to exclude applying discount for one (first one) on the basis: 0,2*(n-1)*X where n is qty and X is product base price.
The problem is that line item discount type in Commerce Discount module is calculating before quantity calculation (only for qty of 1) and proper price component is added to line item price.
The solution I've tried was to divide calculated discount (0.2*X/n) but the result calculated for quatity 3 or 6 or 9 is inaccurate (wrong, decimal number).
Is there a hook or other solution that can allow me to "return" the part of discount that should not be applied, or am I simply missing something and there is a more natural way to achieve what I described (for example,  the order discount rule)?
Answer: You should be able to do this out of the box if you have the Commerce Discount Extra module installed. The offer type to use is Per-quantity product discount (% off). This offer type (and the other per-quantity offer types) was created to support BOGO style offers (buy 1, get 1 free), but we developed them to support any type of discount for any number of matching products.
In your case, you'd set the "Buy #" to 1 and the "Get up to #" to a very high number... whatever you consider reasonable for the quantity you'd expect to discount. This will result in the creation of a new discount line item.
Note that the pattern for matching products will be SKU based or product category based.
Comment: Sounds like it would require a customization to the matching products, then. The quickest thing to do if you aren't concerned about creating a merchant-facing user interface for this would be to hook into hook_commerce_discount_per_quantity_manifest_alter(). For more information refer to the docs in commerce_discount_extra.api.php.
Comment: Hi Ryan, thank you very much for help.

Commerce Extra was the first module I tried, but as you said "pattern for matching products will be SKU based or product category based" and my situation is that all products comes from the CRM system. Imports are few times a day. Categories are shared between few types of products and I want to apply discount to only one product type. In my opinion SKU and category patterns are not suitable in this situation.

That is why I was looking for "Product type" based discount solution. Do you see any solution, including code modification?


How to forward packets not destined for linux host to container

Question: I am trying to do some traffic inspection on a lxc container in my host machine.  My host machine is connected to a mirror port and being sent duplicate copies of the traffic of 4-5 other machines on a local network.
I can see all the traffic from these other machines through wireshark, so I know that they are reaching my interface, but I am having trouble forwarding these the last hop to my lxc container.
I have tried doing NAT through iptables, but I suspect after reading a few other posts that they are being filtered prior to reaching iptables.  I saw one or two suggestions for ebtables and have tried to do some static routing unsuccessfully.
Any help or pointers to existing literature would be greatly appreciated.
Comment: Yes, just read the packets passively no response to original host necessary. It is not a dedicated mirror port. I am duplicating all the packets with an open flow rule. I am getting the whole ethernet frame including original MACs
Comment: No I don't, as I'm working with a small simplistic network and edge device, it's only one hop from the switch. So I can differentiate it based on the fact that it's destination address is the ip of the other machines.
Answer: I strongly suggest you encapsulate your traffic somehow. Easiest is probably VLAN (because I know how to handle this easily with tc), but of course ERSPAN is a defacto standard. Without encapsulation, it's not easy to guess if the received traffic is for your system or from the mirror (what about broadcasts? multicast? ARP?)
So I give here a simplistic way to duplicate again all your received traffic to your containers, with the exception of IPv4 packets having your own IP (I won't care about ARP packets or other such less frequent traffic for your host). If the received traffic was encapsulated, more options would be available (starting with redirecting rather than mirroring everything), and this answer would have to be reworked.
You can't expect normal routing, or even bridging to apply when receiving mirrored traffic. You can use a low level tool available to do this: Traffic Control (tc). It's a quite complex tool able to handle various different network features primarily intended for traffic shaping and alike. Suffice to say that there's an action to duplicate packets described there: tc mirred, that an action is used by a filter and that a filter applies on classful queue discipline (qdisc). As it's incoming, the special ingress qdisc is used.
I won't describe how to integrate this automatically with LXC (it's probably possible by supplying a fixed name in lxc.net.[i].veth.pair for the side attached to the bridge intended for LXC rather than letting LXC choose a random name and then using some hook for scripts, as long as the interface are already created when the hook is invoked). Because the interface names must be existing when creating the tc filters, when doing this manually the containers must be started before,
So let's suppose this (edit to suit your configuration):

your unique real interface is called eth0,
your "external" (assigned to eth0) system IP is 192.0.2.2,
you started two containers, and their veth pair's side on the host are called veth123456 and veth89ABCD. The fact that they are enslaved as bridge port (eg: to lxcbr0) doesn't matter.

Let's do the initial plumbing:
tc qdisc add dev eth0 ingress

Let's add the first copy, with a filter priority (to override it later) and provisions to add more:
tc filter add dev eth0 ingress prio 10 matchall \
   action mirred egress mirror dev veth123456 continue

Because the continue control was used, this filter is not final. More can be added (and continue could again be added for even more filters):
tc filter add dev eth0 ingress prio 10 matchall \
   action mirred egress mirror dev veth89ABCD

Since the criteria are the same (match all), the two previous filters could also have been rewritten, using the pipe action control, with this unique filter instead:
tc filter add dev eth0 ingress prio 10 matchall \
   action mirred egress mirror dev veth123456 pipe \
   action mirred egress mirror dev veth89ABCD

Now it's quite easy to insert a filter with an earlier priority to prevent the copy of packets with the destination IP (but not ARP or broadcasts or multicasts or IPv6 ...) intended for the host (pass makes the action final):
tc filter add dev eth0 ingress prio 5 protocol ip \
   u32 match ip dst 192.0.2.2 action pass

Note that the bridge (and almost all the network stack) was entirely bypassed when doing this: the packet was moved from an interface to an other, short-circuiting everything in between. You could also create an additional veth interface with the host side not connected to a bridge (if LXC can allow this? else just use a dummy bridge) and copy the packets there, thus having a dedicated interface in the container.
To display filters with usage statistics:
tc -statistics filter show dev eth0 ingress

To cancel everything, deleting the qdisc will also delete the filters:
tc qdisc delete dev eth0 ingress

UPDATE: the interface is receiving traffic not intended for it, that means the ethernet frame's destination addresses are most likely not the interface's MAC address. By default the interface's hardware will filter out those frames and they'll never be seen by the system, including by tc, leaving only mirrored broadcasts (eg: ARP requests) visible. So better turn the interface in promiscuous mode:
ip link set dev eth0 promisc on

Comment: Also note that for the *ingress* case only, like here, nftables can do something equivalent when using the [*netdev*](https://wiki.nftables.org/wiki-nftables/index.php/Nftables_families#netdev) address family (which hooks to pre-existing interfaces, just like *tc*).


Sum of squared difference and Gaussian noise model

Question: I have been reading that when the underlying error is distributed normally, then minimising the sum of squared difference between the observed data and the model is the appropriate cost function to do the model fitting. However, I am having trouble understanding why that is the case.
I was wondering if there is an intuitive way to understand why this is the case? What would happen if I use other cost functions say point wise mutual information or some local measure like cross correlation.
Comment: Normality has nothing to do with it. Normality comes later, to endow the estimator with certain desirable properties. You may want to check the following post. http://stats.stackexchange.com/questions/70539/is-there-a-regression-method-for-fitting-general-n-order-polynomials-of-two-or-m/70647#70647
Answer: The sum of squares cost function arises naturally from the assumption,
$y = t(x,w) + \epsilon$, where $\epsilon \sim N(0,\beta^{-1})$. Regarding your concrete example, you take a series of measurements, which it amounts to a series image points from images $x$ and $y$. For that set, the likelihood function is,
$$\prod_{n=1}^{M}N(y_{n}|t(x_{n},w);\beta^{-1})$$
Which is equivalent to minimizing the log-likelihood function, that is,
$$\sum_{n} \log N(y_{n}|t(x_{n},w);\beta^{-1}) = -\frac{\beta}{2}\sum_{n}\left[ y_{n} - t(x_{n},w)\right]^{2} + \frac{M}{2}\log \beta - \frac{M}{2} \log(2\pi)$$
The non-linear mapping is embedded in the deterministic function $t(x,w)$, which you previously choose, depending upon your camera calibration settings.
Answer:
I have been reading that when the underlying error is distributed normally, then minimising the sum of squared difference between the observed data and the model is the appropriate cost function to do the model fitting. However, I am having trouble understanding why that is the case.

"Appropriate" might mean any number of things, but my guess is that they're referring to the fact that when the errors are normal, the least squares estimates are maximum likelihood.
Using maximum likelihood (ML), the estimates are the ones that give the best chance of observing the data you got (conditional on the remaining assumptions).
ML estimators have a number of nice properties and are very widely used - but there's plenty of good reasons to use least squares even when the data aren't normal (on the other hand, when the data are very far from normal, many of those nice properties aren't likely of much use - it's no use being BLUE when all linear estimators are terrible).

What would happen if I use other cost functions say point wise mutual information or some local measure like cross correlation.

Well, presumably they wouldn't be ML, but that's not a problem if you're looking to optimize something else.
Excuse my ignorance; can you give an example (or point to one) showing how these would work in this context?


Find the pdf of piecewise random variable involving identity functions

Question: $X$ is a standard normal random variable.
$Y = g(X)$, where $g(x)=-I_{(-\infty,0)}(x) + \frac12I_{\{0\}}(x)+I_{(0,\infty)}(x)$.
I'm trying to find the pdf for $Y$. I know to start with $F_Y(y)=P(Y\leq y)$, but I don't know where to go from here with the piecewise function, especially when it's just the identity function.
Comment: $Y$ has no PDF since it is a discrete random variable (as was shown by V.V). It has a PMF.
Answer: As $X$ is continuous r.v., then the probability of getting $x=0$ is $0$, thus $Y$ can only receive two values; either $-1$ or $1$, hence
$$
P_Y(y) = P(Y=y)=\frac{1}{2}, \quad y\in\{-1,1\}.
$$
While $1/2$ stems from the fact that $P(X\le 0) = \Phi(0)=1/2$.
Comment: Thanks! What does the phi stand for here?
Comment: It is the cdf of standard normal random variable


Draw pie Chart from a data frame

Question: I have data frame with two variables; attendance 4 5 6 7 2 5 7 8 and another with treatment  A B B A A B B A. How do you make a pie chart comparing sum percentages of A and B in rstudio
Answer: Using dplyr and the pie function, we first group by treatment and do the total sum per group.
a = data.frame(attendance=c(4,5,6,7,2,5,7,8),
                             treatment=c("A","B","B","A","A","B","B","A"),
                             stringsAsFactors = FALSE)

A = a%>%group_by(treatment)%>%summarise(tot=sum(attendance))

pie(A$tot/sum(A$tot),labels=paste(A$treatment,round(A$tot/sum(A$tot),2)),main="Pie")

Draw pie Chart from a data frame


Prove $\text{rank}(A^2) — \text{rank}(A^3) \leq \text{rank}(A) — \text{rank}(A^2)$

Question: Let $A$ be an $n \times n$ matrix over a field $K$. Prove that
$\text{rank}(A^2) — \text{rank}(A^3) \leq \text{rank}(A) — \text{rank}(A^2)$
In particular - I am not aware of any relationship that must hold in general between $\text{rank}(A)$ and $\text{rank}(A^2)$.
Comment: $\text{rank}(A^2)\le\text{rank}(A)$. This can be seen if you think about what happens when applying a linear transformation twice
Comment: @Shuri2060 that makes sense to me.... are you telling me its just that easy then???
Comment: No - I was pointing out a relationship that holds between the two ranks. I think you might need something else to solve the question. (But perhaps using a similar concept)
Comment: I'm not completely sure how to write it out formally, but I think you can see where the inequality comes from if you look at $A_1:\Bbb R^n\rightarrow\Bbb R^n$ (applying transformation $A$ on $\Bbb R^n$), $A_2:\text{Im}(A_1)\rightarrow\Bbb R^n$ (applying transformation $A$ on $\text{Im}(A_1)$) and $A_3:\text{Im}(A_2)\rightarrow\Bbb R^n$ (applying transformation $A$ on $\text{Im}(A_2)$). Then the LHS of the inequality is equal to $\ker(A_3)$ and the RHS is equal to $\ker(A_2)$.
Answer: Hint: Consider epimorphism $\operatorname{im}A\to \operatorname{im}A^2/\operatorname{im}A^3$ given by $x\mapsto Ax + \operatorname{im}A^3$ which factors through $\operatorname{im}A/\operatorname{im}A^2$.


Select a column in dataframe and mask the duplicate

Question: I have a dataframe like this:-
import pandas as pd

dict_data = {
    'Date':pd.Timestamp('20200720'),
    'Number': 123,
    'course':pd.Series(['Python', 'Quant', 'CFA', 'Finance', 'Python', 'Python', 'Finance', 'Finance']),
    'Company':['AA', 'BB', 'CC', 'DD', 'BB', 'BB', 'DD', 'CC']
}

pd.DataFrame(dict_data)

I can select a column. For example, dict_data['course'] and it will output all data of this column. May I know is there any method it can mask the duplicate value? Look like this?
0     Python
1      Quant
2        CFA
3    Finance

Comment: Can, thank you so much.
Answer: You can use df.drop_duplicates():
df = pd.DataFrame(dict_data)

In [1327]: df.course.drop_duplicates()
Out[1327]:
0     Python
1      Quant
2        CFA
3    Finance
Name: course, dtype: object


Multiple regression in R results different to simple linear regression?

Question: Hi I am having trouble acquiring the final results for presentation.
The results from a multiple regression are different to my results in a simple linear regression.
For example, the multiple regression model
mul <- lm(Response ~ Temp + Wave + Overcast, data = env) gives me a p-value for 'Temp' of $0.01$, while the others are $>0.05$.
However, a simple linear regression sim <- lm(Response ~ Temp, data =env)
returns a p-value for 'Temp' of $0.03$.
Which result should I be presenting, and also how to I interpret the relationship between the predictors in the multiple regression?
Comment: Hint: look at the degrees of freedom for the t statistics in the simple and multiple linear regression. Then compare them.
Comment: There is a superb answer to this general question [here](http://stats.stackexchange.com/a/78830/28500). In the terminology of that answer, think of each linear regression coefficient as the result of _ignoring_ all of the other variables, while each multiple regression coefficient is _controlling for_ all the other variables.
Answer: You are comparing two entirely different models. Which one you choose has a lot to do with experience in the field of study, aided by things such as adjusted R squares and ANOVA testing of both models side-by-side.
In multiple regression, the idea is that when we add a regressor ($x_2$) to a pre-existing model with just one regressor ($x_1$), we are regressing the "dependent variable" $y$ not over $x_2$, but over the residuals of the regression of $x_2$ over $x_1$. This changes everything.
If we set up the model as:
$y = \color{blue}{5}\, x_2 + \color{red}{15}\, x_1$ with both $x_1$ and $x_2$ generated as random $\sim N(0,1)$, the correlation $y$ with $x_1$ will be much higher than with $x_2$:
The regression of $y$ over $x_2$ in isolation will not capture the slope we set p of $5$, because $x_2$ will be compensating for the absence of the main explanatory variable in the model, i.e. $x_1$. The slope of $x_2$ will be, in fact, very close to the coefficient we assigned to $x_1$: $\color{red}{15}$. Leaving the intercept out, summary(lm(y ~ x2 - 1))$coefficients will return a slope of $\color{red}{15.42}$.
However, if we now include $x_1$ in the regression in a sneaky way, and instead of calling lm(y ~ x1 + x2 - 1), we first regress $x_2$ over $x_1$ and keep the residuals before "tossing" $x_1$ as errors <- residuals(lm(x2 ~ x1 - 1)) and then call summary(lm(y ~ errors - 1))$coefficients the slope will be $\color{blue}{4.681616}$, very close to the coefficient we set up for $x_2$, and... here comes the punch of the story... identical to coefficient for $x_2$ in summary(lm(y ~ x2 + x1 - 1))$coefficients: $x_2 \,\,\color{blue}{4.681616}$. The coefficient for $x_1$ will be $x_1 \,\,\color{red}{15.091305}$.
So ignoring the hidden confounder $x_1$ in the model $y \sim x_2$ forced $x_2$ to explain all by itself as much of the variation in $y$ as possible, resulting in a completely different slope as compared to the more accurate $y \sim x_1 + x_2$. You can check the concept of omitted variable bias.
It makes sense that the $p$-values are going to be significant in both instances, albeit different.
Comment: Thank you so much for your detailed reply :)
However I've had cases where the relationship between a predictor and a dependent data was significant in a multiple regression but not significant in simple linear regression.. How would I interpret this? Would I try to find which other predictor is affecting the relationship and control for it?


Hook path in main menu not hold active trail

Question: In custom module names supply_area I've created custom path.
function supply_area_menu() {
    $items['supply-area'] = array(
        'title' => 'Supply area',
        'page callback' => 'supply_area_page',
        'type' => MENU_NORMAL_ITEM,
        'access callback' => true,
        'menu_name' => 'main-menu',
    );
    return $items;
}

Everything works perfectly except that even if I assign this path into main menu structure it does not hold menu structure so breadcrumbs looks like:
Home > Supply area and not Home > About > Supply area.
Is there something wrong with my hook_menu?
Comment: Given just the above code, why would you presume it should know about "About" ?
Comment: It's normal menu item so I think I could add this path to main menu. If it's in main menu in Home | About | Supply area so it should hold active trail?
Answer: I am not sure what you are doing in a lot more areas of your system than described above, but doing the following on a d7 testbed here:
function MYMODULE_menu() {

  array $items;

  $items['about'] = array(
    'title' => 'About',
    'page callback' => 'MYMODULE_about_page',
    'type' => MENU_NORMAL_ITEM,
    'access callback' => TRUE,
    'menu_name' => 'main-menu',
  );

  $items['supply-area'] = array(
    'title' => 'Supply area',
    'page callback' => 'MYMODULE_supply_area_page',
    'type' => MENU_NORMAL_ITEM,
    'access callback' => TRUE,
    'menu_name' => 'main-menu',
  );

  return $items;

}

function MYMODULE_about_page() {
  return 'What\'s it all about, Alfie?';
}

function MYMODULE_supply_area_page() {
  return 'Making Supplies';
}

...followed by clearing caches and then setting it up via the Menu UI:

...provides the following:

so I would check how you might be doing it slightly differently on your system.


Confused about an integral from MIT integration bee 2012

Question: One of the integrals is:
$$\int \frac{\mathrm{d}x}{2+2\sin x + \cos x}\, \mathrm{d}x $$
How can there be two $\mathrm{d}x$?
MIT's Integration Bee
Comment: It's a "typo"... (one I sometimes make)
Comment: @DavidMitra Well, its from MIT and I thought maybe it was some strange high level math, or it was a typo.  It's is a fun list of integrals to practice my skill one.
Comment: It is a 'typo', definitely.
Answer: The extra $dx$ is definitely a typo. To solve this integral we have to substitute $u=\tan\frac{x}{2}$. Then, $\sin x=\frac{2u}{1+u^2}$, $\cos x=\frac{1-u^2}{1+u^2}$ and $dx=\frac{du}{1+u^2}$. Thus,
$$\int\frac{dx}{2+2\sin x+\cos x}=\int\frac{du}{2+2u^2+4u+1-u^2}=\int\frac{du}{u^2+4u+3}=\int\frac{du}{(u-1)(u-3)}=\frac{1}{2}\int\frac{du}{1-u}+\frac{1}{2}\int\frac{du}{u-3}=\frac{1}{2}\ln\left|1-u\right|+\frac{1}{2}\ln\left|u-3\right|+c=\\
\frac{1}{2}\ln\left|1-\tan\frac{ x}{2}\right|+\frac{1}{2}\ln\left|\tan\frac{ x}{2}-3\right|+c$$


How to add a column to the list of group members or create a Views equivalent of it?

Question: I use the "Group" module and I want to add a column "Email Address" on the page that lists the members of the groups (I want to put this "Email" column between the column "member name" and the column "statut").
This page is not a view, it is the page that comes with the module, and which is located at /group/gid/member (with gid the id of the group). Here is a screenprint of it:

As an alternative (if adding such Email column on that page is too difficult or impossible), I want to create a Views page (to replace the module's member page). This view page should be identical (or as close as possible) to the screenshot above.
Here is a screenprint of the view I created so far:

And here is an export of this view:
$view = new view();
$view->name = 'group_membres';
$view->description = '';
$view->tag = 'default';
$view->base_table = 'group_membership';
$view->human_name = 'Membres';
$view->core = 7;
$view->api_version = '3.0';
$view->disabled = FALSE; /* Edit this to true to make a default view disabled initially */

/* Display: Master */
$handler = $view->new_display('default', 'Master', 'default');
$handler->display->display_options['title'] = 'Membres';
$handler->display->display_options['use_more_always'] = FALSE;
$handler->display->display_options['use_more_text'] = 'plus';
$handler->display->display_options['access']['type'] = 'none';
$handler->display->display_options['cache']['type'] = 'none';
$handler->display->display_options['query']['type'] = 'views_query';
$handler->display->display_options['exposed_form']['type'] = 'basic';
$handler->display->display_options['exposed_form']['options']['submit_button'] = 'Appliquer';
$handler->display->display_options['exposed_form']['options']['reset_button_label'] = 'Réinitialiser';
$handler->display->display_options['exposed_form']['options']['exposed_sorts_label'] = 'Trier par';
$handler->display->display_options['pager']['type'] = 'none';
$handler->display->display_options['style_plugin'] = 'views_bootstrap_table_plugin_style';
/* Relation: Adhésion au groupe : Groupe gid */
$handler->display->display_options['relationships']['group']['id'] = 'group';
$handler->display->display_options['relationships']['group']['table'] = 'group_membership';
$handler->display->display_options['relationships']['group']['field'] = 'group';
/* Relation: Adhésion au groupe : Utilisateur uid */
$handler->display->display_options['relationships']['user']['id'] = 'user';
$handler->display->display_options['relationships']['user']['table'] = 'group_membership';
$handler->display->display_options['relationships']['user']['field'] = 'user';
/* Champ: Groupe : Titre */
$handler->display->display_options['fields']['title']['id'] = 'title';
$handler->display->display_options['fields']['title']['table'] = 'groups';
$handler->display->display_options['fields']['title']['field'] = 'title';
$handler->display->display_options['fields']['title']['relationship'] = 'group';
/* Champ: Utilisateur : Nom */
$handler->display->display_options['fields']['name']['id'] = 'name';
$handler->display->display_options['fields']['name']['table'] = 'users';
$handler->display->display_options['fields']['name']['field'] = 'name';
$handler->display->display_options['fields']['name']['relationship'] = 'user';
$handler->display->display_options['fields']['name']['label'] = 'Nom d\'utilisateur';
/* Champ: Utilisateur : Courriel */
$handler->display->display_options['fields']['mail']['id'] = 'mail';
$handler->display->display_options['fields']['mail']['table'] = 'users';
$handler->display->display_options['fields']['mail']['field'] = 'mail';
$handler->display->display_options['fields']['mail']['relationship'] = 'user';
$handler->display->display_options['fields']['mail']['link_to_user'] = '0';
/* Champ: Adhésion au groupe : Statut de membre */
$handler->display->display_options['fields']['status']['id'] = 'status';
$handler->display->display_options['fields']['status']['table'] = 'group_membership';
$handler->display->display_options['fields']['status']['field'] = 'status';
/* Champ: Adhésion au groupe : Group role */
$handler->display->display_options['fields']['role']['id'] = 'role';
$handler->display->display_options['fields']['role']['table'] = 'group_membership';
$handler->display->display_options['fields']['role']['field'] = 'role';
$handler->display->display_options['fields']['role']['label'] = 'Rôles';

/* Display: Page */
$handler = $view->new_display('page', 'Page', 'page');
$handler->display->display_options['path'] = 'membres';
$translatables['group_membres'] = array(
  t('Master'),
  t('Membres'),
  t('plus'),
  t('Appliquer'),
  t('Réinitialiser'),
  t('Trier par'),
  t('Asc'),
  t('Desc'),
  t('Groupe'),
  t('Utilisateur'),
  t('Titre'),
  t('Nom d\'utilisateur'),
  t('Courriel'),
  t('Statut de membre'),
  t('Rôles'),
  t('Page'),
);

My questions about this view:

How to add the "Actions" column with links ?
How to add filters ?
Answer: Have a look at this updated version of your view (in Views export format, which you should be to import in your own site):
$view = new view();
$view->name = 'group_members';
$view->description = 'List of all members in a specific Group';
$view->tag = 'default';
$view->base_table = 'group_membership';
$view->human_name = 'Group Members';
$view->core = 7;
$view->api_version = '3.0';
$view->disabled = FALSE; /* Edit this to true to make a default view disabled initially */

/* Display: Master */
$handler = $view->new_display('default', 'Master', 'default');
$handler->display->display_options['title'] = 'Membres';
$handler->display->display_options['use_more_always'] = FALSE;
$handler->display->display_options['use_more_text'] = 'plus';
$handler->display->display_options['access']['type'] = 'role';
$handler->display->display_options['access']['role'] = array(
  3 => '3',
);
$handler->display->display_options['cache']['type'] = 'none';
$handler->display->display_options['query']['type'] = 'views_query';
$handler->display->display_options['exposed_form']['type'] = 'basic';
$handler->display->display_options['exposed_form']['options']['submit_button'] = 'Appliquer';
$handler->display->display_options['exposed_form']['options']['reset_button_label'] = 'Réinitialiser';
$handler->display->display_options['exposed_form']['options']['exposed_sorts_label'] = 'Trier par';
$handler->display->display_options['pager']['type'] = 'none';
$handler->display->display_options['style_plugin'] = 'table';
/* Relationship: Group membership: Group gid */
$handler->display->display_options['relationships']['group']['id'] = 'group';
$handler->display->display_options['relationships']['group']['table'] = 'group_membership';
$handler->display->display_options['relationships']['group']['field'] = 'group';
/* Relationship: Group membership: User uid */
$handler->display->display_options['relationships']['user']['id'] = 'user';
$handler->display->display_options['relationships']['user']['table'] = 'group_membership';
$handler->display->display_options['relationships']['user']['field'] = 'user';
/* Field: Group: Group ID */
$handler->display->display_options['fields']['gid']['id'] = 'gid';
$handler->display->display_options['fields']['gid']['table'] = 'groups';
$handler->display->display_options['fields']['gid']['field'] = 'gid';
$handler->display->display_options['fields']['gid']['relationship'] = 'group';
$handler->display->display_options['fields']['gid']['exclude'] = TRUE;
/* Field: User: Uid */
$handler->display->display_options['fields']['uid']['id'] = 'uid';
$handler->display->display_options['fields']['uid']['table'] = 'users';
$handler->display->display_options['fields']['uid']['field'] = 'uid';
$handler->display->display_options['fields']['uid']['relationship'] = 'user';
$handler->display->display_options['fields']['uid']['exclude'] = TRUE;
/* Field: Group: Title */
$handler->display->display_options['fields']['title']['id'] = 'title';
$handler->display->display_options['fields']['title']['table'] = 'groups';
$handler->display->display_options['fields']['title']['field'] = 'title';
$handler->display->display_options['fields']['title']['relationship'] = 'group';
/* Field: User: Name */
$handler->display->display_options['fields']['name']['id'] = 'name';
$handler->display->display_options['fields']['name']['table'] = 'users';
$handler->display->display_options['fields']['name']['field'] = 'name';
$handler->display->display_options['fields']['name']['relationship'] = 'user';
$handler->display->display_options['fields']['name']['label'] = 'Nom d\'utilisateur';
/* Field: User: E-mail */
$handler->display->display_options['fields']['mail']['id'] = 'mail';
$handler->display->display_options['fields']['mail']['table'] = 'users';
$handler->display->display_options['fields']['mail']['field'] = 'mail';
$handler->display->display_options['fields']['mail']['relationship'] = 'user';
$handler->display->display_options['fields']['mail']['link_to_user'] = '0';
/* Field: Group membership: Membership status */
$handler->display->display_options['fields']['status']['id'] = 'status';
$handler->display->display_options['fields']['status']['table'] = 'group_membership';
$handler->display->display_options['fields']['status']['field'] = 'status';
/* Field: Group membership: Group role */
$handler->display->display_options['fields']['role']['id'] = 'role';
$handler->display->display_options['fields']['role']['table'] = 'group_membership';
$handler->display->display_options['fields']['role']['field'] = 'role';
$handler->display->display_options['fields']['role']['label'] = 'Rôles';
/* Field: Global: Custom text */
$handler->display->display_options['fields']['nothing_3']['id'] = 'nothing_3';
$handler->display->display_options['fields']['nothing_3']['table'] = 'views';
$handler->display->display_options['fields']['nothing_3']['field'] = 'nothing';
$handler->display->display_options['fields']['nothing_3']['label'] = 'View';
$handler->display->display_options['fields']['nothing_3']['alter']['text'] = 'view';
$handler->display->display_options['fields']['nothing_3']['alter']['make_link'] = TRUE;
$handler->display->display_options['fields']['nothing_3']['alter']['path'] = 'group/[gid]/member/[uid]';
/* Field: Global: Custom text */
$handler->display->display_options['fields']['nothing']['id'] = 'nothing';
$handler->display->display_options['fields']['nothing']['table'] = 'views';
$handler->display->display_options['fields']['nothing']['field'] = 'nothing';
$handler->display->display_options['fields']['nothing']['label'] = 'Modifier';
$handler->display->display_options['fields']['nothing']['alter']['text'] = 'modifier';
$handler->display->display_options['fields']['nothing']['alter']['make_link'] = TRUE;
$handler->display->display_options['fields']['nothing']['alter']['path'] = 'group/[gid]/member/[uid]/edit?destination=group/[gid]/member';
/* Field: Global: Custom text */
$handler->display->display_options['fields']['nothing_1']['id'] = 'nothing_1';
$handler->display->display_options['fields']['nothing_1']['table'] = 'views';
$handler->display->display_options['fields']['nothing_1']['field'] = 'nothing';
$handler->display->display_options['fields']['nothing_1']['label'] = 'Annuler';
$handler->display->display_options['fields']['nothing_1']['alter']['text'] = 'annuler';
$handler->display->display_options['fields']['nothing_1']['alter']['make_link'] = TRUE;
$handler->display->display_options['fields']['nothing_1']['alter']['path'] = 'group/[gid]/member/[uid]/cancel?destination=group/[gid]/member';
/* Field: Global: Custom text */
$handler->display->display_options['fields']['nothing_2']['id'] = 'nothing_2';
$handler->display->display_options['fields']['nothing_2']['table'] = 'views';
$handler->display->display_options['fields']['nothing_2']['field'] = 'nothing';
$handler->display->display_options['fields']['nothing_2']['label'] = 'Metadata';
$handler->display->display_options['fields']['nothing_2']['alter']['text'] = 'metadata';
$handler->display->display_options['fields']['nothing_2']['alter']['make_link'] = TRUE;
$handler->display->display_options['fields']['nothing_2']['alter']['path'] = 'group/[gid]/member/[uid]/metadata';
/* Contextual filter: Group: Group ID */
$handler->display->display_options['arguments']['gid']['id'] = 'gid';
$handler->display->display_options['arguments']['gid']['table'] = 'groups';
$handler->display->display_options['arguments']['gid']['field'] = 'gid';
$handler->display->display_options['arguments']['gid']['relationship'] = 'group';
$handler->display->display_options['arguments']['gid']['title_enable'] = TRUE;
$handler->display->display_options['arguments']['gid']['title'] = 'Members of the group with Group Id = %1';
$handler->display->display_options['arguments']['gid']['default_argument_type'] = 'fixed';
$handler->display->display_options['arguments']['gid']['summary']['number_of_records'] = '0';
$handler->display->display_options['arguments']['gid']['summary']['format'] = 'default_summary';
$handler->display->display_options['arguments']['gid']['summary_options']['items_per_page'] = '25';
$handler->display->display_options['arguments']['gid']['specify_validation'] = TRUE;
$handler->display->display_options['arguments']['gid']['validate']['type'] = 'numeric';
/* Filter criterion: Group membership: Membership status */
$handler->display->display_options['filters']['status']['id'] = 'status';
$handler->display->display_options['filters']['status']['table'] = 'group_membership';
$handler->display->display_options['filters']['status']['field'] = 'status';
$handler->display->display_options['filters']['status']['exposed'] = TRUE;
$handler->display->display_options['filters']['status']['expose']['operator_id'] = 'status_op';
$handler->display->display_options['filters']['status']['expose']['label'] = 'Membership status';
$handler->display->display_options['filters']['status']['expose']['operator'] = 'status_op';
$handler->display->display_options['filters']['status']['expose']['identifier'] = 'status';
$handler->display->display_options['filters']['status']['expose']['remember_roles'] = array(
  2 => '2',
  1 => 0,
  3 => 0,
  4 => 0,
  5 => 0,
  6 => 0,
);

/* Display: Page */
$handler = $view->new_display('page', 'Page', 'page');
$handler->display->display_options['path'] = 'group/%/memberlist';

Some more details about the changes in this view:

I changed the machine name of it (to group_members), to avoid overwriting a same named view if you try to import this view.
I changed the "style_plugin" to a standard "table" (to not have to depend on that bootstrap-variation, feel free to change that again).
I added 2 fields that are "excluded from display", ie: the Group id (gid) and User id (uid). That way I can use their tokens in URLs in other fields I added afterwards.
I added the fields (with appropriate links) for "Modifier", "Annuler" and "Metadata". For each of those fields I used the "rewrite results" technique.
Note that I also added a similar "View"-link, since in your original view the links for users point to user/uid (which is not what matches with the orginal screenprint you started from ...).
And I added an (exposed) filter for "Membership status".

Apart from the changes detailed above, I also included these enhancements (which I recommend you to also include):

Add a contextual filter to specify the group ID for which the members should be shown.
Adapt the path of this view to /group/%/memberlist (I couldn't use /group/%/member because that path is already used by the Group module). The % in this path is to be replaced by the group Id you want to show its members for. And if you want to see a list of members for all groups together, then just use /group/all/memberlist.
Add a title like "Members of the group with Group Id = x" (with x = the selected group id).
Limit access to the results of this view. The view in your question does not have any access restrictions. So I restricted (within "Page settings") the access to something with "Role", and selected as role "Administrator". Without that, even an anonymous user who knows the path to get access to this view can see a list of all group members, including there eMail IDs ... However, as delivered by the Group module, a path like /group/gid/member (with gid equal to some group id) cannot be accessed by an anonymous user.

With that, I think ... I'm done.
PS (1 more recommendation to further improve it): use the DataTables module to change the table-display to a dataTables display (try it, you'll love it ... especially if you have a high number of views results).
Comment: :-) Thank you it works but lacks the mass operations as in the original page.
Comment: @Mathieu about the mass Operations: if you make it a VBO view, and then combine that with Rules components, pretty sure you could come up with the equivalent of that also. You think I should create a new question about that, and add my own answer/solution also? Or you know how to do that yourself?


Finding the Value of an Object Element within an Array

Question: I have a table that im inserting into , The table holds Plate Values
How would i go about iterating over the table to return the following as true
function table.contains(table, element)
  for _, value in pairs(table) do
    if value == element then
      return true
    end
  end
  return false
end

My current table insert
for k, v in pairs(returnVehicleData) do
    platesAvailable = v.plate
    newTable =  {['plate'] = platesAvailable}
    table.insert(vehiclePlateTable, newTable)
end

The data thats being returned
[{"plate":"47QVS009"},{"plate":"86KIE632"}]

I want to check the subsequent value after the : on both objects.
Comment: If your returned values are strings, you can try a string pattern such as `:"(.-)"` with `string.gmatch` to iterate over the results.
Answer: I ended up doing this and getting it to work
newValueTable = {}
    for Key = 1, #tabledPlate, 1 do
            table.insert(newValueTable, tabledPlate[Key].plate)
        end

Because the elements are part of an even bigger table of data , they dont actually return as a string. But thank you for the suggestion Coordinate Newton


Methods to check if my data fits a distribution function?

Question: My shortened data is:
y <- c (2,2,1,5,6,7,1,2,1,6,6,7,3,2,4,4,4,4,3,3,9,1,1,9)

I firstly normalize my data:
y_scale <- scale(y)

Then, I generate a model dataset with normal distribution based on y_scale's mean and stdev:
y_norm <- rnorm(n=24, m=mean(y_scale), sd=sd(y_scale))

To check if my data fits the normal distribution, I do
ks.test(y_scale,y_norm)

I found the result is as follows:
Two-sample Kolmogorov-Smirnov test

data:  y_scale and y_norm
D = 0.2083, p-value = 0.6749
alternative hypothesis: two-sided

Warning message:
In ks.test(y_scale, y_norm) : cannot compute correct p-values with ties

Here, my question is:
(1) My real data set has ~ 700,000 numbers, I found I cannot use shapiro.test.
shapiro.test(y_scale)
Error in shapiro.test(y_scale) : sample size must be between 3 and 5000

(2) Is the p-value calculated above by ks.test wrong? How to solve this problem of p-values?
Warning message:
In ks.test(y_scale, y_norm) : cannot compute correct p-values with ties

(3) The reasons why I tried to use ks.test instead of other methods, is because I want to compare with other model datasets that have other distribution functions.  It seems to me that I can simply replace y_norm with other model dataset, and compare their p-values or D-values (the smaller the better), to choose which distribution function fits my data most.
(4) Is it a must to normalize my data first?
Comment: Unless your data are artificially generated by a particular, known distribution, I can guarantee that any reasonably effective distribution test will reject them!  Nevertheless, it can be meaningful to use a goodness of fit statistic, like the KS statistic, to measure the deviation between the distribution of your data and any ideal distribution (there will be no problem with ties or discreteness, either), but the p-values are worthless.
Comment: Probably QQ-plots might guide you also, though I'm not sure what is the reason for scaling at such sample size if you do know that the data has two finite moments then scaling will bring you to the central limit theorem conclusions.
Comment: Interesting read: http://stats.stackexchange.com/questions/2492/normality-testing-essentially-useless
Answer: Why are you testing for normality?
As stated by @user603 the discreteness of your data suggests that it is not normal.  But are you interested in normal enough?
With 700,000 data points you will have power to detect differences from normality that are very minor and probably not important (I just compared a sample from a t distribution with 100 df and got a p-value of 0.011).  If you are doing the normality test in preparation to using normal theory inference (t-tests, confidence intervals, etc.) then for most population distributions 700,000 should be enough for the central limit theorem to let you use the normal theory inference, but knowledge about the population is probably more important here than any test of normality.
Also, if your data really is discrete, then normalizing ruins the discreteness and could be misleading.  For meaningful results you should not normalize.
Computing the mean and standard deviation after normalizing is meaningless since you have set the mean to 0 and standard deviation to 1.
The KS test is designed to test against a fully specified distribution, normalizing the data first is equivalent to comparing to a distribution with estimated parameters which will also invalidate the p-value.
Comment: I have some questions. Isn't the data always discrete? We can depict numbers only up to a number of digits. Can't the marks in a school be normal distributed?
Comment: There are 2 concepts here, the actual data entered into the computer which will always be discrete, and the theoretical mechanism for generating the data which we can theorize about being continous (even though the data we enter will be a discrete approximation to this).  Your data is all integers which means that either the mechanism for generating the data is discrete, like counts from a poisson or binomial where it is impossible to get fractional counts, or you have done a lot of rounding.  This is one of the reasons that understanding beats testing.
Answer:
If your data is discrete, it cannot possible be drawn from a continuous distribution.
Try ks.test(y,"pnorm",mean(y),sd(y)).
Comment: I tried to do this as follows, but I don't know if I have misunderstood your reply.  I found the D value are different as show below.  However, the p-value and the warning message is the same as below. Could you mind to give me some more guidance?
Comment: One-sample Kolmogorov-Smirnov test

data:  y
D = 0.0703, p-value < 2.2e-16
alternative hypothesis: two-sided

Warning message:
In ks.test(y, "pnorm", mean(y), sd(y)) :
  cannot compute correct p-values with ties
Comment: > ks.test(y, y_norm)

        Two-sample Kolmogorov-Smirnov test

data:  y and y_norm
D = 0.9386, p-value < 2.2e-16
alternative hypothesis: two-sided

Warning message:
In ks.test(y, y_norm) : cannot compute correct p-values with ties
>
Comment: i can't help but point out again that if your data is discrete (as it seems to be) it cannot possible be a draw from a Gaussian (i.e. the correct p-value in this case is 0).
Answer: As suggested by user603,  you can test directly by comparing with the normal distribution.
As you can see at help page the $p$-values are not exact if there are ties because "the continuous distributions do not generate them." So the theory (and hence the algorithm) does not allow ties.
I did some testings using
 x <- rnorm(N)
 ks.test(x,"pnorm")
and increased the size of $N$. I noticed the $p$-value is unreliable, jumping up and down randomly. And there were no ties (give credit to R for that). However, the $D$-value is consistent and decreases with the increase of the size. So I suggest you to use the $D$-value.
And about the normalizing thing, there is no such requirement. So I see no point in doing it.


NGINX: Lots of connection timed out Errors

Question: I am getting this error a lot:
2014/11/26 21:01:30 [error] 3475#0: *4028 upstream timed out (110: Connection timed out) while reading response header from upstream
Here is my configuration:
user www-data;
pid /run/nginx.pid;
worker_processes 2;
worker_rlimit_nofile 16384;

events {
  worker_connections    4096;
  use                   epoll;
  multi_accept          on;
}

http {

  sendfile              on;
  tcp_nopush            on;
  tcp_nodelay           on;
  keepalive_timeout     65;

  # free up connection after client stops responding...
  reset_timedout_connection on;

  # If the client stops reading data, free up the stale client connection after this much time. Default 60.
  # send_timeout          2;

  types_hash_max_size   2048;

  server_names_hash_bucket_size 256;
  # server_name_in_redirect off;

  include               /etc/nginx/mime.types;
  default_type          application/octet-stream;

  ssl_protocols         TLSv1 TLSv1.1 TLSv1.2;  # don't use SSLv3 ref: POODLE
  ssl_ciphers           "AES256+EECDH:AES256+EDH";

  access_log            /var/log/nginx/access.log;
  error_log             /var/log/nginx/error.log;

  ##
  # Gzip Settings
  ##

  gzip                  on;
  gzip_vary             on;
  gzip_min_length       10240;
  gzip_proxied          expired no-cache no-store private auth;
  gzip_types            text/plain text/css text/xml text/javascript application/x-javascript application/xml;
  gzip_disable          "MSIE [1-6]\.";
  ##
  # general proxy settings
  ##
  proxy_cache_path      /var/cache/nginx levels=1:2 keys_zone=one:8m max_size=3000m inactive=600m;
  proxy_temp_path       /var/tmp;
  proxy_intercept_errors on;

  include               /etc/nginx/conf.d/*.conf;
  include               /etc/nginx/sites-enabled/*;
}

Comment: please also post the content from the files in /etc/nginx/sites-enabled/
Comment: Increase `proxy_read_timeout` or check why your proxied backend doens't answer in time.
Comment: @XavierLucas increasing proxy_read_timeout got rid of the errors completely, but looking into the underlying cause seems important as well. It turns out most of these connections are WebSocket connections that are not being treated as such by nginx. I have just found in the documentation that you need to add an "Upgrade" header to force nginx to treat proxied connections as WebSockets. Do you know about this? I'm curious if adding this header will affect my non-websocket connetions negatively...
Comment: @AllTheTime Check my full answer
Answer: Solution for the former issue : increase proxy_read_timeout or check why your backend doens't answer in time.
Your second question : There are two things, the Upgrade header and the Connection header.
Both of these need to be passed to the backend when using websockets and you can use a map to change the value depending on original request headers (i.e. if it's a websocket connection handhaske or standard HTTP traffic). The backend must reply to the Upgrade header with HTTP 101. Then nginx will enter in a specific case where a tunnel is set up between the origin of the request and the backend.
Such a case is explained in the official documentation :

http {

    map $http_upgrade $connection_upgrade {
        default     upgrade;
        ''          close;
    }

}

server {

   ...

   location /foo/ {
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }

}


Is the graph of (x^2/4) + y^4 = 1 an ellipse?

Question: My first thought was that it was not an ellipse because it was not because y was not to the 2nd power. But I decided to graph it to find out. I graphed the following four equations: y = +/-[(+/-) (1-(x^2/4))^1/2]^1/2. I got 2 imaginary values for y and 2 real values for y. The graph of the function was an ellipse with a=2 and b=1. Is this the correct solution? Thank you.
T. Grode
Comment: What is the definition of an ellipse?
Comment: Your first thought was right. The graph does not have all the properties of ellipse.
Comment: Maybe of interest https://en.wikipedia.org/wiki/Superellipse and http://mathworld.wolfram.com/Superellipse.html
Answer: It is not an ellipse.
If it were a circle (a special case of an ellipse) then you'd be able to find a point in the plane $P$ (the centre) such that the distance from $P$ to every point $X$ on the curve, written $|XP|$, is constant.
If it were a "proper" ellipse then you would be able to find two, and only two, points in the plane $P_1$ and $P_2$ (the foci) such that $|XP_1|+|XP_2|$ is constant for all $X$.
http://www.mathwords.com/e/ellipse.htm
Answer: I used WolframAlpha to graph
$$\frac{x^2}{4}+y^4=1$$
and the resulting plot is shown in the image below:

To cut the long story short, this is certainly not an ellipse.


When is it acceptable to collapse across groups when performing a factor analysis?

Question: Imagine that

responses were collected on a 20 item scale which was designed to measure 4 factors with 5 items on each scale.
participants were drawn from two groups (Group 1) and (Group 2) with sample size $n_1 = 150$ and $n_2 = 150$.
a researcher wanted to assess the factor structure of the scale

Common scenarios that I see in my consulting:

Group 1 are first year psychology students and Group 2 is sampled from the general community
Group 1 are sampled at one period of time and Group 2 is sampled several years later
Group 1 is a normal population and Group 2 is a clinical population

Question

Under what circumstances would it be appropriate to collapse across groups?
How would these circumstances be assessed?

My initial Thoughts:
My own initial thoughts were as follows:

Theoretical assessment: assess the degree to which the two groups were sampled or measured in ways that would alter the means, sds, or correlation between the items
Empirical assessment: Examine differences between means, sds, and intercorrelations on the scales and optionally on other relevant variables (e.g., demographics); perform a two-group confirmatory factor analysis to assess the consistency of the factor structure across groups.

Essentially, if the empirical evidence suggests that the groups are similar and the theoretical assessment suggests that they are similar, then it should be reasonable to combine.
Conclusion

Does the approach above seem reasonable?
Do you have alternative strategies?
Are there any references that provide recommendations or examples regarding best practice in this situation?
Answer: There seems to be two cases to consider, depending on whether your scale was already validated using standard psychometric methods (from classical test or item response theory). In what follows, I will consider the first case where I assume preliminary studies have demonstrated construct validity and scores reliability for your scale.
In this case, there is no formal need to apply exploratory factor analysis, unless you want to examine the pattern matrix within each group (but I generally do it, just to ensure that there are no items that unexpectedly highlight low factor loading or cross-load onto different factors); in order to be able to pool all your data, you need to use a multi-group factor analysis (hence, a confirmatory approach as you suggest), which basically amount to add extra parameters for testing a group effect on factor loading (1st order model) or factor correlation (2nd order model, if this makes sense) which would impact measurement invariance across subgroups of respondents. This can be done using Mplus (see the discussion about CFA there) or Mx (e.g. Conor et al., 2009), not sure about Amos as it seems to be restricted to simple factor structure. The Mx software has been redesigned to work within the R environment, OpenMx. The wiki is well responding so you can ask questions if you encounter difficulties with it. There is also a more recent package, lavaan, which appears to be a promising package for SEMs.
Alternatives models coming from IRT may also be considered, including a Latent Regression Rasch Model (for each scale separately, see De Boeck and Wilson, 2004), or a Multivariate Mixture Rasch Model (von Davier and Carstensen, 2007). You can take a look at Volume 20 of the Journal of Statistical Software, entirely devoted to psychometrics in R, for further information about IRT modeling with R.
You may be able to reach similar tests using Structural Equation Modeling, though.
If factor structure proves to be equivalent across the two groups, then you can aggregate the scores (on your four summated scales) and report your statistics as usual.
However, it is always a challenging task to use CFA since not rejecting H0 does by no mean allow you to check that your postulated theoretical model is correct in the true world, but just that there is no reason to reject it on statistical grounds; on the other hand, rejecting the null would lead to accept the alternative, which is generally left unspecified, unless you apply sequential testing of nested models. Anyway, this is the way we go in cross-cultural settings, especially when we want to assess whether a given questionnaire (e.g., on Patients Reported Outcomes) measures what it purports to do whatever the population it is administered to.
Now, regarding the apparent differences between the two groups -- one is drawn from a population of students, the other is a clinical sample, assessed at a later date -- it depends very much on your own considerations: Does mixing of these two samples makes sense from the literature surrounding the questionnaire used (esp., it should have shown temporal stability and applicability in a wide population), do you plan to generalize your findings over a larger population (obviously, you gain power by increasing sample size). At first sight, I would say that you need to ensure that both groups are comparable with respect to the characteristics thought to influence one's score on this questionnaire (e.g., gender, age, SES, biomedical history, etc.), and this can be done using classical statistics for two-groups comparison (on raw scores). It is worth noting that in clinical studies, we face the reverse situation: We usually want to show that scores differ between different clinical subgroups (or between treated and naive patients), which is often refered to as know-group validity.
Reference:

De Boeck, P. and Wilson, M. (2004). Explanatory Item Response Models. A Generalized Linear and Nonlinear Approach. Springer.
von Davier, M. and Carstensen, C.H. (2007). Multivariate and Mixture Distribution Rasch Models. Springer.
Comment: Thanks for the comprehensive response. OpenMX does look like a good open source R package option for multi-group CFA. The IRT options also look interesting. Good point about the problems of hypothesis testing in CFA. Thanks for the references.
Comment: I'd recommend lavaan as well, stupid I didn't think of it earlier.
Answer: The approach you mention seems reasonable, but you'd have to take into account that you cannot see the total dataset as a single population. So theoretically, you should use any kind of method that can take differences between those groups into account, similar to using "group" as a random term in an ANOVA or GLM approach.
An alternative for empirical evaluation would be to check formally whether an effect of group can be found on the answers. To do that, you could create a binary dataset with following columns :
yes/no  -  item  -  participant  -  group
With this you can use item as a random term, participant nested in group and test the fixed effect of group using e.g. a glm with a logit link. You can just ignore participant too if you lose too many df.
This is an approximation of the truth, but if the effect of group is significant, I wouldn't collapse the dataset.
Comment: Regarding your last point about GLM, this is basically how IRT models work: they can be viewed as generalized mixed-effects model with a logit function link and individuals treated as random effets (not items!). This yields the so-called marginal likelihood approach, where subjects' parameters are assumed to follow a standard normal distribution, for model identification purpose. The problem is that it has to be done on each subscale which are hypothesized to measure a single construct.
Comment: Good point. And in fact, It would be better to use GEE models or mixed effect models instead of GLM, a standard GLM doesn't allow for random effects.
Comment: FYI, this article from Tuerlinckx et al. 2006 provides a nice review of GLMMs: http://bit.ly/9k8vJs. Also, about the use of GEE as an alternative marginal model, it was proposed by Feddag, Grama, and Mesbah (2003) to estimate parameters of the Rasch Model (one-parameter IRT model), or loglinear multidimensional IRT models as defined by Kelderman and Rijkes (1994). It is currently available in the Stata `raschtest` package.
Answer: It might be a little fly by night, but your theory may suggest whether the two groups have the same factor structure or not.  If your theory suggests they do, and there is no reason to doubt the theory, I'd suggest you could go right ahead and trust that they have the same factor structure.
Your empirical assessment would probably be a good route to go just to spot check the theoretical assessment as whether they are likely to share the same structure.  However, I don't intuitively see why mean differences between items would imply they have a different underlying factor structure.  It seems to me that might just suggest that one group has higher or lower scores on a given factor.
Comment: Interesting point about the means. I suppose differences between means would suggest that the groups are different in some respect. And empirically I'd expect that where there's large differences between means, there's more likely to be differences in factor structure. Differences in means might also have implications for how the findings regarding the factor structure could be generalised. It might also increase correlations between items in general due to an effect the opposite of range restriction.
Comment: Good point I hadn't thought of it that way.  Thank you.  FYI I'm not sure how common it is to use this term, but Rosenthal (of Rosenthal and Rosnow, 2007) refers to the effect opposite of restriction of range as hypertrophy of range.


How can I remove the "This translation is published" checkbox?

Question: In my custom entity module I have already defined a status checkbox to determine the entities status. What I noticed is when I enable show_revisions_ui in the annotation and have the entity setup to support multiple languages there are two check boxes at the bottom of the page.
One of them is to publish this entities language copy and as a result the published checkbox I created is being ignored. How can I disable the multilingual published checkbox?
Comment: Have you tried hook_form_alter already?
Comment: I checked but only the revisions field had a group labeled advanced. Not sure where that field is coming from.
Answer: So it turns out I needed to define a translation handler for my entity.
inside of MyEntityTranslationHandler that extends the ContentTranslationHandler I override the entityFormAlter function and add this:
<code>$form['content_translation']['status']['#access'] = FALSE;</code>
Comment: This code really "hides" mentioned checkbox, but it does not stop the previous checkbox beeing ignored. I have tested it on Media entity type. I have disabled $form['content_translation']['status'] access and "status"  base field stays ignored. Maybe I am doing somwthing wrong.


Android: Timer doesn't have leading zeroes

Question: I have a timer that works, but I don't have leading zeros in the form of 00:00.000. How can I make that happen? Here's my code:
mEndTime = System.currentTimeMillis();
mTotalTime = mEndTime - mStartTime;
int seconds = (int) (mTotalTime / 1000);
int minutes = seconds / 60;
seconds = seconds % 60;
int millis = (int) mTotalTime % 1000;
String sectime = Integer.toString(seconds);
String milsectime = Integer.toString(millis);
String time = minutes + ":" + sectime + "." + milsectime;

Comment: See also [How to format a duration in java? (e.g format H:MM:SS)](http://stackoverflow.com/questions/266825/how-to-format-a-duration-in-java-e-g-format-hmmss)
Comment: check this http://stackoverflow.com/a/37065062/2032561
Answer: String time = String.format("%02d:%02d.%03d", minutes, seconds, millis);


Não estou conseguindo exbir um listView que usa um ArrayAdapter

Question: Eu tenho uma classe principal que herda da Activity onde tenho minhas views, mas precisei criar uma outra classe que herda ArrayAdapter. Porém não estou conseguindo apresentar minha listView, o estranho é que quando pega um item da lista e imprimo funciona. Alguém pode dar um help? Agradeço desde já!
Este método está na minha classe que herda Activity. Este é o método para preencher a activity que não está mostrando nada.
//recebo um vetor de String com dados do banco
public void preencheListView(String[] paises) {

        lv = new ListView(this);
        //instancio aqui
        adapter = new ArrayAdapter(this, android.R.layout.simple_list_item_1, paises);
        //aqui eu consigo imprimir um pais para teste, o que significa que o adapter está ok
        System.out.println("teste: "+adapter.getItem(1));
        lv.setAdapter(adapter);
        lv.setOnItemClickListener(this);
        linearlayout.addView(lv);

    }
}

Esta é a minha classe que herda ArrayAdapter
public class ArrayAdapter extends ArrayAdapter<String> {

    public Context ctx;
    public int rec;
    public String[] paises;

    public ArrayAdapter(Context context, int resource, String[] p) {
        super(context, android.R.layout.simple_list_item_1, p);
        this.ctx = context;
        this.rec = resource;
        this.paises = p;
    }

    @Override
    public int getCount() {
        // TODO Auto-generated method stub
        return 0;
    }

    @Override
    public long getItemId(int position) {
        // TODO Auto-generated method stub
        return 0;
    }

    @Override
    public View getView(int position, View convertView, ViewGroup parent) {
        // TODO Auto-generated method stub
        return null;
    }

    public int getRec() {
        return rec;
    }

    public void setRec(int rec) {
        this.rec = rec;
    }

    public Context getCtx() {
        return ctx;
    }

    public void setCtx(Context ctx) {
        this.ctx = ctx;
    }

    public String[] getPaises() {
        return paises;
    }

    public void setPaises(String[] paises) {
        this.paises = paises;
    }

}

Obs: Antes de eu criar a classe ArrayAdapter funcionava certinho, porém tive que criar a ArrayAdapter porque vou precisar usar o getView().
Obs2: Coloquei um adapter.getCount() agora e descobri que o adapter está vazio mesmo, porém imprimo o item, com o adapter.getItem(x);
Obs3: Será que estou chamando a classe ArrayAdapter da maneira correta?
Answer: O problema está no método
   @Override
   public View getView(int position, View convertView, ViewGroup parent) {
         // TODO Auto-generated method stub
         return null;
    }

Ele não deve retornar null, mas deve retornar uma View que será o item do seu ListView, então dentro do seu método getView faça o seguinte código:
@Override
public View getView(int position, View convertView, ViewGroup parent) {
    // TODO Auto-generated method stub
    return super.getView(position, convertView, parent);
}

Você deve implementar os métodos add e getItem também.
Desta forma:
@Override
public void add(String value) {
    super.add(object);
}

@Override
public String getItem(int position) {
    return super.getItem(position);
}

Não implementar os métodos também resolverá seu problema.
  Caso você não precise implementar o método, ou não faça @Override ou chame sempre o código da super classe, desta forma: super.METODO
Comment: obrigado, funcionou perfeitamente!
Comment: Valeu, fico feliz em poder lhe ajudar.
Answer: Em seu adapter vi que você está retornando 0 no metodo getCount neste trecho do seu codigo:
@Override
public int getCount() {
    // TODO Auto-generated method stub
    return 0;
}

Tente ao invés disso retornar o tamanho da sua Array de Strings assim:
@Override
public int getCount() {
    // TODO Auto-generated method stub
    return paises.length;
}

E também no método getItemId tem que retornar a posição que é mandada como parametro:
@Override
public long getItemId(int position) {
    // TODO Auto-generated method stub
    return position;
}

Assim você consegue retornar o getCount do seu objeto corretamente e também retornar o Id de cada item que nada mais é do que a própria posição do item.