Pattern "inside" prime numbers

Question: Update $(2020)$
I've observed a possible characterization and a possible parametrization of the pattern, and I've additionally rewritten the entire post with more details and better definitions.
It remains to prove the observed possible characterization, and then complete it by finding the sequences for locations of the "parabolic shapes" that live in the characterized regions.

Table of contents:

Visualization of factorizations

Inside prime visualizations

The pattern and the question

Pattern parametrization

Visualization of factorizations
Here I will explain how we can visualize positive integers in an interesting way which also has the property that the factorization of $n$ is encoded into it in the form of a fractal-like pattern.
$2$-digit palindromes represent factorizations
We say that a positive integer is palindromic in some integer number base (number-system) $b\ge 2$ if when written in that number base, its digits are read the same forward and backwards.
The $2$-digit palindrome is an integer $n=(a,a)_b=ab^1+ab^0=a(b+1),a\lt b,a\ge 1$, where in the $(n_1,n_2,\dots)_b$ notation, $n_1,n_2,\dots$ stand for digits of $n$ in the integer base $b\ge 2$.
Factorizations of positive integers $n$ are related to $2$-digit palindromes. That is, if $n$ can be factorized as $n=p\cdot q$ where $p\lt q-1$, this means that $n$ is a $2$-digit palindrome in the number base $q-1$, which we write as:
$$
n=p q=p(q-1)+p=(p, p)_{q-1}, p<q-1,
$$
where $(p,p)$ are digits of $n=pq$ in the number base $q-1$.
 Representing $n\in\mathbb N$ as a sum of $n\times n$ images (matrices)
First we define a "matrix of point", then we define the "sum" of these matrices.
Let $x_0,y_0$ be nonnegative integers. Let $P_n(x_0,y_0)$ be a $n$ by $n$ matrix "of point $(x_0,y_0)$" whose top left entry is denoted with $P_n(x_0,y_0)(0,0)$ and whose bottom right entry is denoted with $P_n(x_0,y_0)(n-1,n-1)$. It is defined for $x,y\in[0,n)$ as:
$$
P_n(x_0,y_0)(x,y)=
\begin{cases}
1, & \text{if }x+nx_0\text{ is a two digit palindrome in base }y+ny_0\\
0, & \text{else}
\end{cases}
$$
We can visualize these "point" matrices as images by coloring different entries with different colors.  For example, "$P_{100}(x_0,y_0)$"'s for $(x_0,x_0)=(0,0),(1,0),(2,0),(3,0)$ are:

Where the green pixels represent the matrix entries that are $1$.
Notice that $P_n(x_0,y_0)$ in point $(x_0,y_0)=(0,0)$ contains "dotted lines" $L_t$ of points from
$$\{(t^2-1+(t-1)r,t+r),t\ge 2, r\ge 0\}$$
and that if we look at $P_n(x_0,y_0)$ in some other point, we get continuations of these lines.
 Visualization of the "image of $n$" matrix
We define $N_n(y_0)=\sum\limits_{k=0}^{\infty} P_n(k,y_0)$ as "entry-wise sum of $P_n$'s along the $x$-axis at level $y_0$". Notice that this sum always sums finitely many matrices because for large enough $k=x_0$, all entries of the matrix are $0$.
To visualize $N_n(y_0)$ we will color all $0$ entries white, and all other entries blue. This visualization remains the same for all $y_0\ge 1$ except in the top left entry which is zero only for $y_0=1$. If we choose $y_0=0$, then the visualization is "degenerated".
This motivates us to write $N_n=f(N_n(1))$ and call $N_n$ the "image of $n$", where $f$ returns the same matrix but all nonzero entries are set to "blue" and all zero entries are set to "white".
For example, below is the image showing the $N_n$'s for $n=7,\dots,31$. Notice that the prime numbers are solid blue squares with no additional details, and that other numbers have visualizations that correspond to their factorizations.

 Patterns in $N_n$ based on factorizations of $n$
For example, notice that even semi-primes $n=2p$ where $p$ is prime, all have the following look:

Or for example, notice that squares of primes $n=p^2$ have the least details:

On the other hand, numbers with a lot of factors such as factorials like $4!=2\cdot3\cdot4 = 24$ or primorials like $p_3\#=2\cdot 3\cdot 5 = 30$ have much more details (factors), and numbers such as powers of primes like $27=3^3$ and $32=2^5$ have regular fractal-like patterns.

One might ask if we can for $n$ in general compute the matrix $N_n$ more efficiently than calculating and summing up all those $P_n$'s. That is, is there an efficient test to get a $N_n(x,y)$ entry directly? $-$ This question does not need to be answered, it's just my thought on the current chapter.

Inside prime visualizations
The $N_p$ where $p$ is a prime number resembles a solid blue square and looks boring. But, we can "look inside" this solid blue square by replacing our $f$ function.
Recall that we are WLOG observing $y_0=1$ because all $y_0\ge 1$ yield equivalent images.
 "Heat-image" matrix of prime number $p$
We will write $\overline{N}_p=h(N_p(1))$ and call $\overline{N}_p$ the "heat-image of prime $p$", where $h$ assigns distinct colors to distinct values of entries of $N_p(1)$ and returns such matrix. Let the zero value be "black", the smallest nonzero value be "red" and the second-smallest nonzero value be "yellow". Other values can be "white" or represented in shades of "grey".
For example, here are $\overline{N}_p$'s of primes $p=101,103,109$:

At first glance these appear to be different sizes of the same image, but this is not the case.
Within different $M_p$'s we find different patterns in certain regions of the image. Thanks to Hyperplane's comment, we can "enhance" these regions by removing the needless details.
That is, if we ignore the last row then these $\overline{N}_p$'s appear to be symmetric along the central horizontal line. In other words, if an entry $(x,y)$ is red then the mirrored entry $(x,|p-y-2|)$ is yellow, and vice versa. However, this is not true for all entries.
This motivates us to observe the "non-mirrored" entries, and set the other ones to $0$.
 The pattern inside prime number $p$
We will observe the irregularities in the "heat-image". That is, we define a new matrix $M_p$:
$$
M_p(x,y)=
\begin{cases}
1, & \text{if }N_p(1)(x,y) = N_p(1)(x,|p-y-2|)\\
0, & \text{else}
\end{cases}
$$
We will represent these entries by coloring $1$'s and $0$'s with white and black, respectively.
For example, here are $M_p$'s of primes $p=101,103,109$:

Observe the white pixels near the central horizontal line. Notice that there we can distinguish between a "thick dotted parabola" and a "thin dotted parabola".
For example, zooming into the central horizontal line of $109$ shows

that the "thick dotted parabola" one is on the left side, and that the "thin dotted parabola" one is on the right side. It is the same for the prime $101$, but not for the prime $103$ which has it the other way around.
It turns out that primes of the form $p=4k+1$ are "left-handed" (have the "thick" parabola on the left side) and that the primes of the form $p=4k-1$ are "right-handed" (have the "thick" parabola on the right side).
It also turns out that these are not the only "parabolic" groups of pixels that exist. I've tried tracing the prime $p=101$ from different groups of pixels and found a set of three distinct parabolas. Observe the last (fourth) image below.

These have one of the following "shapes" of pixels at their "head":

The first ("green parabola") has $I_{2}$ - "I shape of height of 2 pixels"
The second ("blue parabola") has $J_{2,1}$ - "J shape of height of $2+1$ pixels"
The third ("red parabola") has $I_{3}$ - "I shape of height of $3$ pixels"

Note that these "shapes" are mirrored below/above the central horizontal line.
All $6$ permutations of these "shapes" can appear in some prime number $p$ (in some $M_p$). Depending on the permutation that appears in the image (the matrix), we can determine if the prime number is of form $p=18k+m$ where $m\in\{-1,1,5,7,11,13\}$.
For example, all primes of the form $p=18k+11$ have the same $(I_{2},J_{2,1},I_{3})$ permutation of these parabolic patterns as the prime $101=18\cdot 5+11$, in that region of $M_p$.
The larger the prime $p$, the more of these regions we can find.
For example, in primes as large as $p=997$ I've highlighted four of these regions.  Some regions repeat multiple times. For example, notice that there are four instances of the yellow region highlighted.

If the picture above isn't clear, we can alternatively color (highlight) the regions by connecting the pixels (entries) that represent ("belong to") corresponding "parabolic" shapes:

In every region $R$, we can observe a set of "parabolic" shapes. Different permutations of those shapes will appear in different primes $p$ (in different $M_p$'s), depending on the remainder $m$ from division of $p$ by some constant $c_R$.
The question is now, can we characterize every such region and its shapes?

The pattern and the question
I think I'm having a hard time characterizing all these regions, because I'm actually not sure how to properly define them.
Essentially, it appears $M_p$ contains "regions $R_s$" and within those regions we can find one of the possible permutations $P=(p^{(s)}_1,p^{(s)}_2,\dots,p^{(s)}_s)$ of $s$ "parabolic shapes".
It seems that region $R_s$ contains some  permutation $P_m$ if and only if $p\equiv m\pmod{c_s}$ where $c_s$ is some constant related to the $s$th region.

Question. $1.$ Can we characterize all regions $R_s$ and their $P_m$'s and $c_s$ ?

The "pattern inside a prime $p$" would be the collection of all regions $R_s$ of the matrix $M_p$.
Note that this is related to Moiré pattern's, as this user pointed out.
Below are the regions I've observed so far.
 Characterization of regions $R_s$ for small $s$
I will be using $P=(1,2,\dots,s)$ as a shortened notation for $P=(p^{(s)}_1,p^{(s)}_2,\dots,p^{(s)}_s)$. Also, the interpretation of this notation is that $p^{(s)}_1$ is in the leftmost part of the region, and that $p^{(s)}_s$ is in the rightmost part of the region.

$$ (s=1) $$
In every $M_p$, if you look either at the very top or at the very bottom, you will find the same "half-parabola" shape $p^{(1)}_1$. This gives the trivial region:
$$
R_1 : P=(1) \iff p \equiv 1\pmod{2}
$$
That is, there is only one possible permutation within $R_1$, and every prime $p$ has it.

$$ (s=2) $$
Given a $M_p$ and looking around the central horizontal line we can find $2$ distinct parabolic shapes. These are the "thick parabola" $p^{(2)}_1$ and the "thin parabola" $p^{(2)}_2$. I've observed that:
$$\begin{align}
R_2 : P=(1,2) &\iff p \equiv 1\pmod{4} \\
R_2 : P=(2,1) &\iff p \equiv 3\pmod{4} \\
\end{align}$$
This region splits the primes into two sets.

$$ (s=3) $$
In the previous chapter we've observed $R_3$ to contain parabolic shapes characterized by a "short I shape" = $p^{(3)}_1$, a "long I shape" = $p^{(3)}_2$ and a "J shape" = $p^{(3)}_3$. It appears that:
$$\begin{align}
R_3 : P=(1,2,3) &\iff p \equiv 1\pmod{18} \\
R_3 : P=(2,1,3) &\iff p \equiv 5\pmod{18} \\
R_3 : P=(2,3,1) &\iff p \equiv 7\pmod{18} \\
R_3 : P=(1,3,2) &\iff p \equiv 11\pmod{18} \\
R_3 : P=(3,1,2) &\iff p \equiv 13\pmod{18} \\
R_3 : P=(3,2,1) &\iff p \equiv 17\pmod{18} \\
\end{align}$$
This region splits the primes into six sets. Unlike $R_2$ that appears only once, this $R_3$ region appears twice in the image: above and (mirrored) below the central horizontal line.

$$ (s=4) $$
The region $R_4$ is similar to $R_3$ but of course has more permutations. This region contains parabolic shapes characterized by following shapes (groups of pixels): "3 long pixels" = $p^{(4)}_1$, "2 short pixels" = $p^{(4)}_2$, "2 long pixels" = $p^{(4)}_3$ and "3 short pixels" = $p^{(4)}_4$. The corresponding congurences are:
$$\begin{align}
R_4 : P=(1,2,3,4) &\iff p \equiv 1\pmod{16} \\
R_4 : P=(1,4,3,2) &\iff p \equiv 3\pmod{16} \\
R_4 : P=(4,1,2,3) &\iff p \equiv 5\pmod{16} \\
R_4 : P=(2,1,4,3) &\iff p \equiv 7\pmod{16} \\
R_4 : P=(3,4,1,2) &\iff p \equiv 9\pmod{16} \\
R_4 : P=(3,2,1,4) &\iff p \equiv 11\pmod{16} \\
R_4 : P=(2,3,4,1) &\iff p \equiv 13\pmod{16} \\
R_4 : P=(4,3,2,1) &\iff p \equiv 15\pmod{16} \\
\end{align}$$
Interestingly, not all permutations of four shapes are possible to appear in this region. That is, we can see that only $8$ out of theoretical $24=4!$ appear. Also notice that we alternate even and odd numbers ("short" and "long" shapes) in all of these permutations.

$$ (s\ge 5) $$
Unexpectedly, unlike $R_3$ and $R_4$ that appear only once (on each mirrored side), we can say that the $R_5$ appears twice (on each mirrored side). But, in each of those two mirrored pairs we have different parabolic shapes. Therefore, we can also say that there are two different $R_5$ regions.
So far, I've been manually tracing pixels in many consecutive primes to observe these regions and their permutations. Can we characterize these regions more efficiently?
I've examined the $R_5$ region(s) and found that $c_5=50$, which gives $20$ possible permutations. I've also examined $R_6$ and found $c_6=36$, which gives $12$ possible permutations.
Notice that a pattern emerges here. The $c_s $appears to follow the sequence:
$$
c_s=2, 4, 18, 16, 50, 36,\dots
$$
Which corresponds to only one OEIS sequence, namely the A137933(s) = $c_s = \DeclareMathOperator{\lcm}{lcm}\lcm(s^2,2)$. If this is true, then the number of possible permutations is $s\phi(s)$, which is A002618.
In other words, it appears that only the permutations (of "parabolic" shapes $\{1, 2, 3, \dots, s\}$) that are "in arithmetic progressions modulo $s$" can exist within the $R_s$ region!
Can we prove that this holds for all $s$?

Beside finding how the individual parabolic shapes permute within corresponding regions, we also need to find their exact locations within those regions, to complete the characterization.
 Partition of pixels (entries) into regions $R_s$
An interesting question that we can ask is, can we color every pixel (entry) $(x,y)$ such that it belongs to exactly one (or two neighbouring) regions $R_s$?
That is, $(x,y)$ should belong to a region if it is a part of a shape that is being permuted in that region, or if it "extends" one of those shapes up to the border of a neighbouring region.
I've decided to color $R_1,R_2,R_3,R_4$ with $\color{purple}{\text{purple}},\color{red}{\text{red}},\color{green}{\text{green}},\color{blue}{\text{blue}}$, respectively. I'll use white to color pixels that "overlap" (could be in either region) or those that "are forming next region".
Prime numbers $p=2,3,5$ seem too small so we can start with $p=7,11,13,17,19,23$.

The first three regions are already visible in these small primes.
Observing the next few primes $p=29,31,37,41$, we see that the fourth region emerges too, but still intersects the third region at some parts.

It is at primes $p=43,47,53$ when the fourth region should be clearly visible for the first time.

The fifth region and beyond will appear at a bit larger primes.
If we want to be able to color larger examples, we need to first characterize the regions (answer my first question) and then we need to be able to additionally determine their exact locations within the corresponding region.

Question. $2.$ Can we determine locations for the parabolic shapes for given $R_s$ region?

I've looked at $R_2$ (first nontrivial region), and it is not hard to see where exactly the two corresponding parabolic shapes will appear. This allowed me to come up with the parametrization of the entire pattern (matrix $M_p$) and is presented in the following chapter.

Pattern parametrization
The central region $R_2$ can be used to generate the entire matrix $M_p$. This is much more efficient than computing it and might shed some light on the pattern.
That is, all regions $R_s$ emerge from the following parametrization.
 Parametrization of $M_p$
To start, we locate the start (the "head") of the "thin" and the "thick" parabola. The starting $y$ will be given by $y=y_P$ for both, where the starting $x=x_L$ of the leftmost one and the starting $x=x_R$ of the rightmost one are:
$$\begin{align}
y_P&=\frac12(p-3)\\
x_L&=\frac14\left(p-(p\bmod{4})_{\in\{-1,1\}}\right)\\
x_R&=p-x_L
\end{align}$$
where the "mod" takes values in $\{-1,1\}$ as noted. To generate $M_p$, we start with a matrix of $0$'s and fill in the $1$'s with the following sequences, where the "mod" now takes nonnegative values:
$$\begin{align}
\left(x^{(i)}_n,y^{(i)}_n\right)&=
\left(
\left(\left(x^{(i)}_{n-1}+(2n-1)\right)\bmod{p}\right),
y^{(i)}_{n-1}+1
\right), i\in\{1,2\},\\
\left(x^{(i)}_n,y^{(i)}_n\right)&=
\left(
\left(\left(x^{(i)}_{n-1}+(2n-1)\right)\bmod{p}\right),
y^{(i)}_{n-1}-1
\right), i\in\{3,4\},\\
\left(x^{(5)}_n,y^{(5)}_n\right)&=
\left(
\left(\left(x^{(5)}_{n-1}+2n\right)\bmod{p}\right),
y^{(5)}_{n-1}+1
\right),\\
\left(x^{(6)}_n,y^{(6)}_n\right)&=
\left(
\left(\left(x^{(6)}_{n-1}+2n\right)\bmod{p}\right),
y^{(6)}_{n-1}-1
\right),\\
\end{align}$$
where the starting conditions are:
$$\begin{align}
\left(x^{(i)}_0,y^{(i)}_0\right)&=\left(x_a,y_P+(i\bmod{2})\right),i\in\{1,2,3,4\},\\
\left(x^{(i)}_0,y^{(i)}_0\right)&=\left(x_b,y_P+(i\bmod{2})\right),i\in\{5,6\},\\
\end{align}$$
where $x_a,x_b\in\{x_L,x_R\}$ are the $x$'s of the "head"'s of "thick","thin" parabolas, respectively.
A sequence terminates after its $y$ coordinate leaves the $[0,p)$ interval.
The only correction that needs to be done is that $M_p(0,0)=M_p(0,p-2)=M_p(0,p-1)=0$ and that $M_p(0,y)=1$ for $y=1,2,\dots,p-3$. The rest of entries $M_p(x^{(i)}_n,y^{(i)}_n)=1$ are correctly generated by sequences $i=1,2,3,4,5,6$ and $n$'s up until the $y$ escapes $[0,p)$.
These sequences represent extensions of the "thick" and "thin" parabolas from $R_2$, that loop around the right and left edges of the image (matrix) up until the top (bottom).
As these sequences intertwine, they will produce all other regions $R_s,s\ne 2$. That is, all regions emerge from these sequences of pixels (entries).
Can we use this to help us predict the characterizations of regions $R_s$?
Comment: Mathematicians don't like base-dependant maths because it often doesn't contain an underlying truth, merely a coincidence in base $b$. But this seems to find a pattern connecting primes and ALL bases $b$, something that certainly has potential for underlying truths!
Comment: When is the book coming out?
Comment: The plots for the prime numbers seem to have a sort of symmetry around the central horizontal axis: If the pixel $(x,y)$ is red, then $(x,N-y)$ is very often yellow and vice versa. Maybe it would be worth to do a difference plot, i.e. only plot pixels that don't satisfy this property. From the images I would suggest that maybe their relative occurrence (#mismatches/$N^2$) goes to $0$ as $N$ gets big.
Comment: @GerryMyerson I assume you are referring to the length of this post. Anyhow, I'm just exploring things like this in my free time as a hobby.
Comment: See, this is the beauty of math. What may come from research like this? Who knows, perhaps an unknown secret of primes or perhaps nothing at all. Nonetheless, patterns like this are very fun to dive into. I encourage you to keep going!
Comment: Why would you call the upper left pixel the "center pixel?" That seems a name designed for confusion.
Comment: @Hyperplane I just recently updated the question with difference plots and wanted to add on in the comments; indeed, the ratio of asymmetrical to symmetrical pixels tends to $0$. Interestingly, despite their number being virtually nullified compared to the rest of pixels, the asymmetrical pixels seem to form similar patterns to the far ends of $2$-digit palindrome regions in the palindromic plot presented in the introduction.
Comment: The longest post I have ever seen on the math.SE
Comment: @user477343 To be honest, I wasn't active on this topic since [this answer](https://math.stackexchange.com/a/2463426/318073) was posted. But your comment reminded me to rewrite this as it was messy in its initial form after the initial edits.
Comment: Well, I'm glad I helped in some way, hahah. Everything happens for a reason, I guess :)
Answer: I would put this in the comments, but I don't have enough points.
The patterns you are seeing are Moiré patterns. From wikipedia:

In mathematics, physics, and art, a moiré pattern (/mwɑːrˈeɪ/; French: [mwaʁe]) or moiré fringes are large scale interference patterns that can be produced when an opaque ruled pattern with transparent gaps is overlaid on another similar pattern. For the moiré interference pattern to appear, the two patterns must not be completely identical in that they must be displaced, rotated, etc., or have different but similar pitch.

This is a broad topic with lots of different approaches and applications, so I encourage you to research it.
In terms of finding equations to the patterns you are seeing, Mathematica is great for this. If you don't have Mathematica, you can use Wolfram Alpha. Here is an example with a list of numbers using findsequencefunction. You can also provide a list of points (generally, wolfram alpha needs five or more points to find a sequence):
<code>findsequencefunction[{{x1, y1}, {x2, y2}, {x3, y3}, {x4, y4}}, n]
</code>
Hope this helps!

Pattern "inside" prime numbers


Como apagar arquivos em um diretório com script batch?

Question: Preciso que ao executar um script.cmd ele apague todos os arquivos com as seguintes extensões: .txt e .mp4 se existirem. Esses arquivos estão localizados na seguinte pasta C:\comando\batch\diretorio\teste e o script.cmd está localizado em C:\comando\batch.
Como eu poderia fazer isso, sabendo que depois destes comandos, outros irão ser executados, mas não para a pasta teste, e sim na pasta batch.
Ou seja, é possível fazer isso sem sair da pasta batch?
Os comandos precisam ser compatíveis com o Windows 2003/XP.
Answer: É possível sim.
Para deletar os arquivos, use o del.
Se preferir deletar os arquivos usando o caminho absoluto, use a variável %~dp0 para obter a pasta do script e indique a subpasta alvo.
Você pode iterar sobre um array com as extensões e usar o coringa para corresponder à arquivos com certas extensões.
@echo off

set "subpasta=%~dp0%diretorio\teste\" REM a pasta do batch e subpasta
set "extensoes=txt mp4"               REM separe as extensoes com espaco

for %%i in (%extensoes%) do (
    echo Deletando arquivos do tipo %%i da pasta %subpasta%
    del %subpasta%*.%%i
)

if %errorlevel% EQU 0 (
    echo Operacao realizada com sucesso
) else (
    echo Nao foi possivel deletar um ou mais arquivos
)

Obs: Para não mostrar nenhuma saída, você redirecionar o stdout e stderr para null assim: del %subpasta%*.%%i 2> nul
Comment: @Florida Até onde sei não, mas se você quiser pode usar o `/p` para perguntar antes. Sobre o `%~dp0`, nesse link tem outros também: https://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/percent.mspx?mfr=true
Comment: Só uma duvida, da forma como está aí, não é necessário nenhuma confirmação ao executar o `del` né? Ou tem de colocar o `/q` como mencionado nos comentários? Se sim, como seria? ps: gostei do comando `%~dp0` muito interessante :)
Answer: Como é uma sub pasta do seu diretorio raiz, não a necessidade de sair, mas eu em particular prefiro sempre usar o comando com um path absoluto para maior segurança.
del C:\comando\batch\diretorio\teste\*.txt /s
del C:\comando\batch\diretorio\teste\*.mp4 /s
O '/s' apaga recursivamente.
ex.: dentro da pasta tiver outra pasta(teste/videos)com arquivos mp4, eles tambem seram apagados.
Dai você pode continuar com suas rotinas!
Comment: Eu acho que `/s` é para deletar os arquivos de como recursivo, [veja aqui](http://ss64.com/nt/del.html). Para ignorar as perguntas, deve ser `/q`.
Comment: É isso mesmo @stderr , obrigado pela observação.


Does the existence of a chord tone (and its enharmonic spellings) imply the absence of another?

Question: My question is about the notation of chords and what that notation conveys (mostly applied to jazz and the notes that are implied by the chord symbol). A chord symbol notated with a '7' alone indicates a particular quality for the 3rd, 5th and 7th. So if someone writes CM7, then we know that the 3rd is major (and not minor, augmented or diminished), the 5th is perfect (and not augmented or diminished), and the 7th is major (and not minor, augmented, diminished, double diminished). The existence of a perfect 5th eliminates the existence of an augmented 5th, and vice versa. It doesn't make sense to convey a major augmented ♯5 mood by writing a CM7 chord, because the CM7 chord symbol indicates that the 5th is not augmented. However, since the CM7 symbol doesn't specify the quality of the 2nd, 4th and 6th, it could convey a lydian, jonian, harmonic major mood, and others. Is this reasoning correct?
If this reasoning is correct, CM7(♭5) cannot express a lydian intention because the chord notation would imply the absence of a perfect 5th and a lydian mode contains a perfect 5th. So if the intention is a lydian feel, then the appropriate chord would be: CM7(♯11) (being aware of the 9th, which usually is not a problem).
Likewise, G7(♯5) couldn't express a phrygian dominant mood because the phrygian dominant mode contains the ♮5. If the intention is for the soloist to play phrygian dominant, then the chord should be written as G7(♭13), thus allowing the existence of a perfect 5th alongside the ♭13. (I know the 5th is usually omitted when voicing this chord, but that's not something the chord notation is formally telling us to do; it should be a matter of taste, not obligatory. Even if the pianist/guitarist omits the 5th when voicing the G7(♭13) chord, the chord symbol nevertheless implies what notes are available to play.)
In practice it seems to me that this information, embedded in the chord symbols, is overlooked and G7(♯5) is treated as the same chord--with the same implications--as G7(♭13). The idea here is that different enharmonic spellings imply different available notes (and therefore scales), particularly in regards to alterations like ♯5 vs. ♭13 and ♯11 vs. ♭5. This idea could only be implemented when dealing with the known diatonic scales and their modes, because one can point easily to scales with two types of 5ths or 7ths, etc. Is this something to pay attention when reading charts or is there no hope in this level of formalism?
Comment: Welcome to Music.SE; terrific question!!
Comment: You're right in what you've said. In chord symbols, a ♯5 usually implies a ♯4. By contrast, having a ♭13 in the chord symbol usually implies a ♮5.
Comment: @jdjazz Noob question: is there some way to contact you to explain some edits I'd change? There's no direct contact in stack sites?
Comment: We can chat here: https://chat.stackexchange.com/rooms/70683/are-chord-tones-implied
Answer: Why This Question is Important
I agree completely with everything you've said, including the motivation for the question and the specific examples you've given. Musicians rely on written music for communication. Granted, when learning an unfamiliar song, the most ideal scenario is to listen through original recordings of the tune, talk to other musicians, and hear how they play over the song. But sometimes those options simply aren't available. Imagine we're on a gig and the bandleader calls a tune we don't know. We open our real book, and it only gives the chords. We see G7(♯5) and need to know what chords to play and what scales to use. All we can rely on is what we see on the page. The conventions for writing and interpreting chord symbols are important and have many real-world implications.
Chord Symbol Conventions
As a general principle, the particular enharmonic spelling we choose for the chord alterations matters a lot. A ♯5 is different from a ♭13, and the choice has different implications. At the most fundamental level, a ♯5 implies the omission of a ♮5. Similarly, a ♭13 implies the omission of a ♮13. Given this fundamental, basic distinction, we can build larger implications too.
In general, I was taught the following four conventions when reading chord symbols for dominant seventh chords. These conventions come directly from chord-scale theory. They describe chord symbols, but more importantly, they describe the underlying scales from which the chords are constructed.

A ♯5 implies a ♯4.

(G altered: 1 ♭2 ♯2 3 ♯4 ♯5 ♭7)
(G whole tone: 1 2 3 ♯4 ♯5 ♭7)

A ♭13 implies a ♮5.

(G phrygian dominant: 1 ♭2 3 4 5 ♭6 ♭7)
(G mixolydian ♭6 scale: 1 2 3 4 5 ♭6 ♭7)

A ♯11 implies a ♮5.

(G lydian dominant: 1 2 3 ♯4 5 6 ♭7)

A ♭9 typically implies a ♯9 and vice versa.

(G half-whole diminished: 1 ♭2 ♯2 3 ♯4 5 6 ♭7)
(G altered: 1 ♭2 ♯2 3 ♯4 ♯5 ♭7)

There are two cases where a dominant 7th scale breaks one of these conventions:

G mixolydian ♭2 scale: 1 ♭2 3 4 5 6 ♭7
G phrygian dominant scale: 1 ♭2 3 4 5 ♭6 ♭7

Both scales contain a ♭2 but no ♯2--this breaks the fourth convention from above. The good news is that adding a ♯2 to either scale would generally sound good and wouldn't clash. So following the fourth convention is still safe thing to do, as a general guideline. Doing so won't really get us into much trouble.
Because these conventions describe scales, they apply to the chord symbols too. Chords are built from scales, and when deciding which extra notes to add to the chord, one safe bet is look to the underlying scale. (Granted, I've only listed scales built from the major scale, melodic minor scale, harmonic minor scale, and harmonic major scale. And there will be forms of music that don't want any link whatsoever between chord and scale, but those forms of music are much less likely to use chord symbols.)
Matching Chord Symbols with Scales
Using the conventions above, we can look at some common dominant 7th chords and figure out what scales would sound good over each. The first chord symbol shows what we might see on a lead sheet and the second chord symbol shows the chord again with the implied tones written in.

G7(♯5♮9) → G7(♯4♯5♮9) → use G whole tone scale
G7(♮9♭13) → G7(♮5,♮9,♭13) → use G mixolydian ♭6
G7(♯5♯9) or G7(♯5♭9) → G7(♯4♯5♯9) or G7(♯4♯5♭9) → use G altered scale
G7(♭9♭13) or G7(♯9♭13) → G7(♮5♭9♭13) or G7(♮5♯9♭13) → use G phrygian dominant scale
G7(♯11) → G7(♮5♯11♮13) → use G lydian dominant scale
G7(♯9♮13) → G7(♮5♯9♮13) → use G half-whole diminished
G7(♭9♮13) → G7(♮5♭9♮13) → use G half-whole diminished or G mixolydian ♭2

Caveat
As with anything, these are guidelines are loose, and that's especially true in this case. These conventions are best viewed as providing a solid first choice in scenarios where several options might exist. Out of everyone in the band, the soloist generally has the most liberty. For example, it's extremely common for a soloist to take a G7(♯5♯9) chord and treat it as G7(♯9♭13) with a ♮5. But the rhythm instruments must be a little more careful when making these changes. Imagine a pianist is playing chords behind a horn solo. The pianist sees G7(♯5♯9), replaces it with G7(♯9♭13), and plays the ♮5. This is a little riskier because it would likely clash if the soloist is playing a G altered scale or if the bass is playing a G altered scale. If I were the pianist in this scenario, I wouldn't replace G7(♯5♯9) with G7(♯9♭13) until I heard the soloist play the ♮5.
Comment: Nice! That's the kind of answer I was hoping for. Could you indicate sources (books, articles) where these guidelines are suggested?
Comment: @AllanFelipe, I tend to think that the logic you presented stands on its own, but you are correct to seek references and I'll see if I can dig some up. My teacher taught me these conventions, but I'll see if there's a text somewhere that indicates this too.
Comment: @dfhwze, I'm not being arbitrary--there is a rationale behind the spelling I've used. I'm describing the scale tones from a functional perspective. The Bb in G altered does not function as a b3 as your spelling would suggest. It functions as a #9. The B natural functions as the 3rd, not a b4. As evidence, we can look to the chords. If we are asked to play a G altered triad, we do not play Bb as the 3rd--we play B natural.
Comment: @dfhwze, we must be thinking about different musical traditions. Every jazz lead sheet is written with chord symbols that "name the functional relations of the notes in a chord." By contrast, you've said this is incredibly uncommon, which seems to suggest that we're thinking in different frameworks. Your statement suggests that G7 and G7alt have a different 3rd, which means that we change the 3rd simply by adding altered extensions to an existing chord. To me (thinking in a jazz framework), that sounds like a failure of the theory to adequately describe the facts.
Comment: @dfhwze, if you told a jazz musician "play the 3rd in G7, and then play the 3rd in G7alt," they would play the same note. The notion of an un-flatted 3rd and a flatted 7th is the very foundation of a dominant 7th chord in jazz.
Comment: @dfhwze, regarding the altered scale, see **[here](https://en.wikipedia.org/wiki/Altered_scale)**, for example: "In jazz, the altered scale or altered dominant scale is a seven-note scale that is a dominant scale where all non-essential tones have been altered. This means that it comprises the three irreducibly essential tones that define a dominant seventh chord, which are root, major third, and minor seventh and ... the ninth is altered to a b9 and a #9"
Comment: @dfhwze, under the view you cite, it would be wrong to call it a dom7 chord and we should instead be written and analyzed as a minor chord or a sus(b4). Is that a useful way to answer think about that chord and this question?
Comment: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/93251/discussion-between-jdjazz-and-dfhwze).
Answer: Is a so-called C7M(♭5) occurring in a C lydian, really a C7M(♭5) or more properly a C7M(♯11)?
Is a so-called G7(♯5) occurring in a G phrygian, really a G7(♯5) or more properly a G7(♭13)?
How are some chord notations more suitable than others?
What does it mean by "foo is more properly called bar"?
What does a suitable name conveys in the realm of music analysis?
Why do we call a chord, a certain chord rather than others? What is in a name?
It depends. Without context, it is futile to continue such discussion.
A note might be a ♭5 if it is passing from 5 --> ♭5 --> 4.
A note might be a ♯11 if it is superimposed by many thirds, being in the original position, thus the 11th interval is able to be perceived.
In the high classical age, a chord symbol inherently suggest a major or minor key.
Despite modern freedom, one usually has in heart the canonical example.
More important is why you think a non-scale note can exist by itself, without breaking the sense of a mode?
The 4 ancient modes, dorian, phrygian, lydian, mixolydian, were mainly used in the strict-counterpoint age.
If you want to convey the sense of Church music, you might not add too much non-Chord tones.
It's true that notes with accidentals --- false notes --- did exist, but they were mainly a temporary device to avoid tritone (augmented 4th).
I can think a good example of situation close to what you described.
The very beginning of L'Isle joyeuse by Debussy reads:

What is it? We are not sure. The key signature has A major or f minor, but the music does not suggest so, and neither does a pure listener know the key signature.
But A, B, ♯C, ♯D, F, G are particularly stressed: a totally symmetric, uniform, so called whole-tone scale, mysterious, without any direction.
Among them, ♯C is very evident, while G is also picked out because it is both the lowest point and highest point of the phrase.

But later, a perfect fifth is outlined, namely A -- E. Because ♯C is always hovering in the right hand, this seems to be a A major chord.
But ♯C -- D are perversely repeated, and a sense of lydian mode persists.
You can continue the process to for the rest of the piece.
Following your logic, you might call this a A7(♭5).
Is this meaningful? Maybe. But nothing can replace the effort of really attending to the music in real time, recognizing something to be another, or turning out mistaken.
Listening to music is an active process. You don't passively identify, "Oh yeah, this is a C7M(♭5). Tell me it is if it is, or tell me not if it's not."
No, hear it yourself, or hear it another way.
It is a holistic process to perceive music.
This is why a computer cannot analyze a music piece, so much as it cannot "read a book" in the human sense.
If it could --- listen to or read in the human sense --- then this is a real AI, completely indistinguishable with the humankind, and we had better be prepared.
I am simplifying a lot of things, and each topic can easily be extended to a short essay. Tell me whether this address your question.
If you have a particular passage in mind, I would suggest that you open another question to draw more attention, and I will be willing to analyze (and to be upvoted).
Comment: Nice answer. About the second example, since there's no G (in the specific bar), I guess the option would be Aadd(♯4) if this ♯4 is considered an integral part of the harmony. Calling it A(♭5) (I've never seen that) would mean that the perfect 5th is not an important interval, but it gets a lot of attention. (Just some typos: "phrygian dominant" in the first paragraph, and "♯C -- ♯D" later)
Answer: Let me first clarify that we can talk about two DIFFERENT topics here (both of which I think are interesting): 1. theoretical musings of music theory for the sake of discussion, and 2. Theoretical music for the purpose of PRACTICAL playing (e.g. what is HELPFUL).
You make some interesting points (and I think we would get a long just fine with our mutual interest music-theory-musings) but allow me to make a point or two as I'm primarily concerned with what will help me play better... what SOUNDS good.
The real problem with asking the above question, is that you are trying to create rules for yourself that DON'T EXIST. The thing is, when somebody has written CM7#11 on a page of music, they are not at all telling YOU what to play. Instead, they are giving you notes to RELATE to (in the case of C7M#11: C, E, G, B, F#). Moreover, I think even the rules are made to be broken. I don't have time to dig, but you wouldn't have to look far in a Charlie Parker or Kenny Garret tune to see them 'breaking the rules', i.e. playing notes that 'don't belong.' We are all familiar idea of 'playing outside' and of course, a player of great taste can make bad notes sounds delicious!
Then again, as Aminopterin pointed out, context is everything, and there are certainly situations where a soloist is aware of an 'avoid' note as it might clash with another instrument, certainly more often with dominant chords. Now in regards to CM7b5... I suppose theoretically a composer could desire the absence of the perfect 5th. But he might want to mention it to the musicians before the session -- and they might still play that perfect 5th anyway! Of course, the perfect 5th will be a fine sounding note.
Comment: I agree, of course the sound is always what matters. You have an interesting point, that people write a chord thinking about the notes of it and nothing more. Given these sounds, that build the foundation, the rest is up to you. If this is the semantics of a chord, then C7M(♭5) doesn't contain the meaning: "play the ♭5 and not the 5", but "F#/G♭ is an important foundational note, make sure you know that"


No log handling enabled - turning on stderr logging Cannot find

Question: I'm trying to setup a Cron Job that runs a .php file that will automate client email correspondance. I've created the .php file and the Cron Job through cPanel. Everything works fine in that I'm getting my test email as I require it when I require it but I'm also getting an additional server email on every send with the following content;

No log handling enabled - turning on stderr logging Cannot find module (MTA-MIB): At line 1 in (none) Cannot find module (NETWORK-SERVICES-MIB): At line 1 in (none)

The Cron Job is as follows;

0.3 * * * * php -q /home/USERNAME/public_html/cron/test.php

I've done research on this site adn could find a solution. I've also chatted with my hosting company and believe it or not his best advice was to refer me to this site.
Thank you in advance.
Comment: possible duplicate of [No log handling enabled - turning on stderr logging Cannot find](http://serverfault.com/questions/299970/no-log-handling-enabled-turning-on-stderr-logging-cannot-find)
Comment: Can you run your script from the command line and does it give the same errors ?
Comment: It seems related to SNMP. Which distro are you running?
Answer: There's a good chance you're a customer of yet another clueless hosting provider. That error or should I say warning is a spit out from php that they have the php-snmp library compiled in but their MIBs are not mapping correctly. They need to fix their snmp install and they will go away.
It is not your problem.. it is theirs.
Comment: Hi Mike, thanks for your response. I originally thought it was their config issue so I'll head back to them with a little more emphasis this time.


JS - EventListener click not working properly

Question: When the page loads it's supposed to show several small squares of different colors (and it does) and when I click the desired square, it changes the overall page background color to it. But the click doesn't work. For some reason the click goes through once when the page loads, so if I comment the last line in setbgcolor() out, the page becomes blue, otherwise it stays white like in presant code. Clicking after the page has loaded does nothing.
var colorArr = ["Red", "Orange", "Yellow", "Green", "Teal", "Blue", "Purple", "Grey"];

for(var i=0; i<colorArr.length; i++){

    var div = document.createElement('div');
    div.style.backgroundColor = colorArr[i];
    div.setAttribute('class', 'palette');
    div.addEventListener("click", setBgColor(div));

    document.getElementById('palette-div').appendChild(div);

}

function setBgColor(clickedDiv) {
        var bgColor = clickedDiv.style.backgroundColor;

    document.body.style.backgroundColor = "blue";
    document.body.style.backgroundColor = bgColor;
}

Answer: You are invoking setBgColor instead of passing it.
div.addEventListener("click", setBgColor(div));

To pass a method you need to leave the parenthesis out:
div.addEventListener("click", setBgColor);

You can then figure out which div was clicked, by investigating the this context in setBgColor.
Comment: Assignment states that I need to call setBgColor and that I should pass 'this' as the parameter so that the function knows which div was called. However I understand what you mean, it's like in Java, I should've though about it. So basiclly the assignment was wrong then?
Answer: You shouldn't pass callback function this way. There are two ways to do this properly:
Way 1:
div.addEventListener("click", setBgColor);

Note that using this way you cannot define custom arguments for the called function. However, in this case, you don't need it.
setBgColor will be called with 1 argument - MouseEvent object which has some properties like target.
You can make your setBgColor function look like:
function setBgColor(ev)
{
    ev.target.style.backgroundColor = '#FF0000';
}

This way is preferable because you add event listeners to multiple objects in a loop. As @Rodik mentioned in comments, the usage of anonymous functions within loops is definitely a bad practice.
Way 2:
If you only needed to assign a single function call, you could implement an anonymous function which would made the desirable function call:
div.addEventListener("click", function()
{
    setBgColor(div);
});

Comment: While this will work, defining anonymous functions inside loops is a bad pattern.
Comment: @Rodik Can you explain why?
Comment: @Rodik I agree. I didn't notice a loop. Of course, way 1 is better in this case. I have updated the answer.
Comment: @Zim84 defining functions in loops can lead to scoping errors in some cases. it is best to avoid the practice even when such an error does not occur in the current example. for more info - https://jslinterrors.com/dont-make-functions-within-a-loop
Answer: ele.addEventListener("event", function(){//handler});//this syntax of addevent
 // but your code

 div.addEventListener("click", setBgColor());
 // setBgColor ==>this function reference
 // setBgColor()  ==> this function calling (in case of you its returning undefined )

div.addEventListener("click", setBgColor);//modify like this

check this

var colorArr = ["Red", "Orange", "Yellow", "Green", "Teal", "Blue", "Purple", "Grey"];

for(var i=0; i<colorArr.length; i++){

    var div = document.createElement('div');
    div.style.backgroundColor = colorArr[i];
    div.setAttribute('class', 'palette');
    div.addEventListener("click", setBgColor);

    document.getElementById('palette-div').appendChild(div);

}

function setBgColor(event) {
        var bgColor = event.target.style.backgroundColor;

    document.body.style.backgroundColor = "blue";
    document.body.style.backgroundColor = bgColor;
}
.palette{
width:50px;
  height:50px;
  display:inline-block;
}


How to configure p3p policy on Azure

Question: Our app is hosted in Facebook. As you know, Facebook hosts third party app in an IFrame. You may also know that if a web site in an Iframe, and parent website is on a different domain, then 3rd party (cross domain) cookies that do not have a compact policy will be blocked in IE for Medium privacy settings. So one needs to configure a compact p3p policy for a web site. My general question is: What is the recommended way to do this on Azure. I would appreciate any documentations on this. I looked and looked and can only find bits and pieces that do not work for me.
Note, I have already created the p3p policy files (html, xml, compact and reference files). So basically I think I have all the pieces that I need. I am just missing the directions on how to enable them.
Further subquestions:
Do I have to serve the compact p3p policy as part of the response header every time a page is requested?
or
Can I configure the IIS on Azure to set the headers for the entire site? I found this link
http://richardprodger.wordpress.com/2011/04/06/p3p-and-iis-in-azure/
that talks about how to create and run an appcommand on Azure. However, when I do it and try to deploy, my role instances never start. They hang infinitely.
Kind regards,
Archil
Answer: On another site (social.msdn.microsoft.com) someone made the following post:

Instead of configuring IIS in startup
  task, we can write the setting in the
  web.config file in our web role. The
  setting I used to resolve p3p issue is:

<code><configuration>
    <system.webServer>
        <httpProtocol>
            <customHeaders>
                <add name="p3p" value="CP="IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT""/>
            </customHeaders>
        </httpProtocol>
    </system.webServer>
</configuration>
</code>

The p3p header value is quoted from
  http://adamyoung.net/IE-Blocking-iFrame-Cookies.
  Please try if it works for you or not.

Hope this helps.
Comment: I have tried this but unfortunately my role instance never starts. Visual Studion just sits there displaying "Running the Deployment" in the lower left corner but it never transitions to actually running it. If I remove the above block then I can run the deployment no problem.
Comment: Correction to the above: I was incorrectly placing the httpProtocol node under the System.web. Once I put it under System.webserver then I can deploy the solution with no problem. However, I am still having a problem of a "Privacy Settings eye" showing up at the status bar of the IE. I wonder if one of my p3p compact three letter durectives is incorrect.
Comment: I realized I was modifying a wrong web.config file. The aobve suggestions should go into the web.config file of an app that is hosted inside an IFrame (not the host one). After I made those changes then the problem was solved.

How to configure p3p policy on Azure


How to make an invisible border around a JLabel? (JAVA)

Question: I have been searching how to make a border around a JLabel.  But I don't want it to have no color.  Thanks in advance.
public TitlePanel()
{
    title = new JLabel("This is some text!", JLabel.CENTER);
    add(title);

    //This will make a black border around the "title" label
    title.setBorder(new LineBorder(new Color(0,0,0)));
}

Comment: What is a border if it is invisible?
Comment: In that case probably one with zero thickness.
Comment: @gangqinlaohu Padding, empty space, dangerous ;)
Answer: Simply use an EmptyBorder
title.setBorder(new EmptyBorder(10, 10, 10, 10));
Check out How to use borders for more examples
Updated
As suggested by Eng.Fouad, you really should be using the BorderFactory to generate borders.
title.setBorder(BorderFactory.createEmptyBorder(10, 10, 10, 10));
Generally speaking, this can reduce the number of objects created by the application
Comment: +1 I always use the static-factory method: `BorderFactory.createEmptyBorder(10, 10, 10, 10)` ;)
Comment: @Eng.Fouad It did say "where possible" :P
Comment: @Eng.Fouad You're absolutely right, we should be using the factory methods as it reduces the over header and re-uses the borders. I'm just lazy that way (bad me) ;)
Comment: `Border` cannot be re-used. Actually, both are the same. Check the source code :)
Comment: @Eng.Fouad Really, I thought that was the point of the `BorderFactory` (mind you, it's been nearly 10 years since I last used it)
Comment: @Eng.Fouad From the [JavaDocs](http://docs.oracle.com/javase/7/docs/api/javax/swing/BorderFactory.html) *Wherever possible, this factory will hand out references to shared Border instances*
Comment: I might be wrong about the re-usability of the `Border` object (I must be confused with the re-usability of swing components or layout manager objects), but most of the static-factory methods in `BorderFactory` use `return new XXXBorder();`.


Problem in importing chessboard.js in vue application

Question: I am trying to import chessboard.js on my jetstream-vue application. After creating the project I ran
npm install @chrisoakman/chessboardjs
and it's in my node_modules folder. But when i tried import Chessboard from '@chrisoakman/chessboardjs' on app.js it shows error as
ERROR in ./resources/js/app.js
Module not found: Error: Can't resolve '@chrisoakman/chessboardjs' in '/home/user/project-chess/resources/js'
 @ ./resources/js/app.js 9:0-51
 @ multi ./resources/js/app.js ./resources/css/app.css

Is there any other way I should try to import it or from to a different file than app.js.
Answer: @chrisoakman/chessboardjs can't be imported, since it doesn't export anything.
Look at the code, at the end it says:
  // TODO: do module exports here

Comment: oh, just checked that. So, i have to write export code now. Or is there any other way in which i could use the module functions without importing.
Comment: Perhaps you can learn something here: https://www.npmjs.com/package/ng2-chessboard


What is the alternative for IN sql operator in LINQ for string field?

Question: Please find the example below which I am trying to achieve.
List<string> list = new List<string>() { "Test", "Test1", "Test2","Test","Test1" };

SQL = select * from table_name
     where column_name IN ("Test","Test1");

output = { "Test","Test1","Test","Test1"}

LINQ= from x in list where list.Contains("Test","Test1") select x

output = { "Test", "Test1", "Test2","Test","Test1" }

Contains Does not do an exact match.
How can we get the same output as SQL IN operator with string list in LINQ?
Thanks
Comment: you could use `Contains` or `Any` it should give you similar results.
Comment: The LINQ statement is not what you intend it to be. `where list.Contains("Test","Test1")` is always true.
Answer: You're using Contains incorrectly (and indeed, your example code doesn't compile). Instead, declare a set that you wish to match against and use Contains on that set, like this:
List<string> list = new List<string>() { "Test", "Test1", "Test2","Test","Test1" };

var match = new HashSet<string>() { "Test", "Test1" };

var result = from x in list where match.Contains(x) select x;

Result:
Test
Test1
Test
Test1

Comment: Note you could create `match` inline (thought not very concisely as a `HashSet`) without performance penalty (in this small example): `new[] { "Test", "Test1" }.Contains(x)`
Comment: Yep, indeed I usually do that. I thought the separate declaration would make it easier to follow.
Comment: I agree it is easier to follow, but perhaps a bit less like the SQL.
Answer: A similar approach to Eric's answer but using join:
var list = new List<string>() { "Test", "Test1", "Test2","Test","Test1" };
var match = new HashSet<string>() { "Test", "Test1" };

var results = from x in list
              join y in match on x equals y
              select x;


Alternative proof of Monotone Convergence Theorem

Question: Let $(X, \mathfrak{A}, \mu)$ be a measure space. Show that the Monotone Convergence Theorem holds if $f$, $f_{1}$, $f_{2}$, ... are real-valued measurable functions and $f_{1}$ is integrable. Further,
(1) $f_{1} \leq f_{2}...$
and
(2) f = $\lim_{n} f_{n}$ hold almost everywhere.
My try:
Assume (1) and (2) hold everywhere.
From the monotonicity of the integral we have
$\int f_{1} d\mu \leq \int f_{2} d\mu \leq$ .. $\leq \int f d\mu$.
We need to show the reverse inequality.
Let $g^{+}_{n,k}$  and $g^{-}_{n,k}$ be two sequences of positive valued simple functions such that
$\lim_{k} g^{+}_{n,k} - g^{-}_{n,k} = f_{n}$ and $g^{+}_{n,k} - g^{-}_{n,k} \leq f_{n}$ for all $k$.
Define $h_{n}^{+}$ and $h_{n}^{-}$ as $h^{+}_{n} = \max_{k}(g_{k,n}^{+}$) and $h^{-}_{n} = \max_{k}(-g^{-}_{k,n})$, then both $h_{n}^{+}$ and $h^{-}_{n}$ are non-decreasing measurable simple functions and $\lim_{n} h^{+}_{n} + h^{-}_{n } = f^{+} - f^{-} = f$. Also $h^{+}_{n} \leq f^{+}_{n}$ and $h^{-}_{n} \leq -f^{-}_{n}.$ Since $h^{+}_{n}$ and $h^{-}_{n}$ are simple functions, it follows
$\lim_n \int h_{n}^{+} d\mu = \int f^{+} d\mu$ and $ \lim_{n} \int h^{-}_{n} d\mu = -\int f^{-} d\mu$
and
$\lim_{n} \int h^{+}_{n} + h^{-}_{n} d\mu = \int f d\mu$.
Since $\int f^{+} d\mu = \lim_{n} h^{+}_{n} d\mu \leq \lim_{n} \int f^{+}_{n}d\mu$ and similary $-\int f^{-}d\mu = \lim_{n} \int h^{-}_{n} d\mu \leq \lim_{n} -\int f^{-}_{n}d\mu$
$\int f d\mu = \lim_{n} \int h^{+}_{n} + h^{-}_{n} d\mu \leq \lim_{n} \int f^{+}_{n} - f^{-}_{n} d\mu = \int f_{n}d\mu$.
Assume now (1) and (2) hold almost everywhere, let $N$ be the set consisting all $x\in X$ for which at least one of (1) and (2) fails.
Then the functions $f\chi_{n^{C}}$ and $f_{n}\chi_{N^{C}}$ fulfills (1) and (2) and thus
$\int f\chi_{N^{C}} d\mu = \lim_{n} \int f_{n}\chi_{N^{C}}d\mu$ and from this we can conclude that
$\int f d\mu = \lim_{n} \int f_{n}d\mu$.
This is very similar to a proof for MCT in the book Measure Theory by Donald Cohn, however, the book is for positive valued functions only.
My question is whether this is correct or if I have missed something.
Secondly, why is the constraint that $f_{1}$ is integrable necessary?
Comment: One thing as I'm reading it through: "We need to show the reverse inequality." There is no reason why the reverse inequality should hold. Why do you want to show this?
Comment: I don't really understand what you mean. We have $\lim_{n}\int f_{n} d\mu \leq \int f d\mu$ and we want to show that $\int f d\mu \leq \lim_{n} \int f_{n} d\mu$ so that $\lim_{n} \int f_{n} d\mu = \int f d\mu$ ?
Answer: Some comments while I'm reading your proof and the original from Donald Cohn's (that you cited). These might sound critical, but I'm just trying to raise your awareness in details.

The verse "We need to show the reverse inequality" is a bit puzzling. If, e.g. $(X,\mathcal{M},\mu)=([0,1],\text{Leb}([0,1]),m_{1})$, where $m_{1}$ is the Lebesgue measure, and we choose $f_{n}\equiv 1-\frac{1}{n}$ for all $n\in\mathbb{N}$, then $\int f_{n}\,dm_{1}<\int f_{n+1}\,dm_{1}$ for all $n\in\mathbb{N}$. So there is really no reason why the reverse inequality should hold, and I doubt this is what you meant. You want to show that $\lim_{n} \int f_{n}\,d\mu\geq \int f\,d\mu$.
The way you defined the functions $h_{n}^{+}$ and $h_{n}^{-}$ does not work, i.e. defining them as maximums over an infinite set. Are you sure these maximums exist, or should you replace it with a supremum? And if you take supremum, what would be the result? Would these functions, as a supremum of simple functions, be simple functions? In any case, what you want to do is to define $h_{n}^{+}=\max\{g^{+}_{1,n},...,g^{+}_{n,n}\}$ instead, and $h_{n}^{-}$ similarly (but instead minimums).

I am going to assume from now on that the functions $h_{n}^{+}$ and $h_{n}^{-}$ are defined as above. Also it is worth to note that you have not defined $f^{+}$ or $f^{-}$ before you start using these notations.

Note that the sequence $(h_{n}^{-})$ does not consist of non-negative simple functions, so you can't use Proposition 2.3.3 as it stands in the next step. Something analogous could be argumented though.
And here comes the situation where some level of integrability from $f_{n}$ would be required:
\begin{equation*}
\lim_{n}\int h_{n}^{+}+h_{n}^{-}\,d\mu=\int f\,d\mu.
\end{equation*}
What if $\int |f_{n}|\,d\mu=\infty$ for all $n$? Would it be possible that one of the sides would yield $\infty-\infty$ in the steps where you proved this equality, which would not be defined?

Here's a suggestion for an alternative approach to this result, using the MCT for non-negative functions as Donald Cohn has proven it. I don't think you have to make the proof from scratch again.
Take $f_{1}\leq f_{2}\leq ...\leq f$ as before and assume that $f_{1}$ is integrable. Then $0\leq f_{2}-f_{1}\leq f_{3}-f_{1}\leq ... \leq f-f_{1}$ is a non-negative sequence, and thus
\begin{equation*}
\lim_{n}\int f_{n}\,d\mu-\int f_{1}\,d\mu=\lim_{n}\int f_{n}-f_{1}\,d\mu\overset{MCT}{=}\int f-f_{1}\,d\mu=\int f\,d\mu-\int f_{1}\,d\mu.
\end{equation*}
Since $-\infty<\int f_{1}\,d\mu<\infty$, then this implies $\lim_{n}\int f_{n}\,d\mu=\int f\,d\mu$.
Edit: Note that the integrability of $f_{1}$ is necessary. Let $(X,\mathcal{M},\mu)=(\mathbb{R},\text{Leb}(\mathbb{R}),m_{1})$ and $f_{n}=-\chi_{[n,\infty)}$ for all $n\in\mathbb{N}$. Now $f_{1}\leq f_{2}\leq ...\leq f=0$, but $\lim_{n}\int f_{n}\,dm_{1}=-\infty\neq 0=\int f\,dm_{1}$.
Comment: Thanks a lot, I agree I was a bit sloppy with some details such as the $\max$-definitions etc, and yes 2.3.3 must be modified, I forgot that.
Comment: @Erik. Sure. And you would probably also want to note that $g_{n,k}^{+}$ are chosen to approximate $f_{n}^{+}$ and $g_{n,k}^{-}$ to approximate $f_{n}^{-}$. Even though it is implicitly clear from the steps that follow.
Comment: So with the correct definition with $h^{+}$ and $h^{-}$ and a modified version of 2.3.3, is my proof valid?
Comment: @Erik. There is one more detail which does not seem correct. Note that $(g_{n,k}^{-})$ increases to $f_{n}^{-}$, whence $(-g_{n,k}^{-})$ decreases to $-f_{n}^{-}$. So you want to take minimums when defining $h_{n}^{-}$ instead of maximums.
Comment: Yes, you are right, I was too quick with that.
Comment: @Erik. This also affects some future steps, since now $h_{n}^{-}\geq -f_{n}^{-}$ instead of $\leq$.
Comment: So $h^{+}_{n} \leq f^{+}$ and $h^{-}_{n} \geq -f^{-}_{n}$ and then $h^{+}_{n} + h^{-}_{n} \leq f^{+}_{n} - f^{-}_{n}$ so with some corrections of the proof we can still get what we want?
Comment: @Erik. Except that these two conditions do not imply $h_{n}^{+}+h_{n}^{-}\leq f_{n}^{+}-f_{n}^{-}$. Consider for instance $h_{n}^{+}=1$, $f_{n}^{+}=2$, $h_{n}^{-}=-1$, $-f_{n}^{-}=-3$. Then $h_{n}^{+}+h_{n}^{-}=1-1=0>-1=2-3=f_{n}^{+}-f_{n}^{-}$.
Comment: @Erik. I also added an example in my answer which shows that integrability of $f_{1}$ is necessary (or $f_{n}$ for any $n$), if you want to take a look at it.


Code that creates a txt file but can't write on it

Question: So, i'm new to coding in general and i'm learning C, i'm trying to make a program that executes an action according to the number the person types in (For example type 1 to open the browser),
and i'm having some trouble with the notepad part.
The notepad consists  in: you will be able to write anything, and when you press enter it's gonna save it as a txt file.
Now the problem: when i execute the code, i can't write when i was suppose to, but yeah, it creates the txt file but without nothing because i couldnt type.
Btw i copy pasted the notepad code into an empty C file and it worked fine
<code>
#include <stdio.h>
#include <stdlib.h>
#include <windows.h>

#define DATA_SIZE 1000

int main()
{
    char data[DATA_SIZE];
    int selection = 0;
    printf("\t\t\t\t\HELLO, WELCOME TO THE ACTION CENTER\n");
    printf("Select what you want by typing the number according to the list below\n");
    printf("\t1. Type something on notepad\n");
    printf("\t2. Open browser\n");
    printf("\t3. Open Zoom\n");
    scanf("%i", &selection);
    //char data[DATA_SIZE];

    if (selection == 1)
    {
        //char data[DATA_SIZE];
        printf("Opening Notepad...\n");
        FILE * fPtr;
        fPtr = fopen("txtfiles/text1.txt", "w"); // it needs to be in the same folder as your code
        if (fPtr == NULL)
        {
            printf("Unable to create your txt file\n");
            exit(EXIT_FAILURE);
        }

        printf("NOTEPAD - TYPE WHAT YOU WANT:\n");
        fgets(data, DATA_SIZE,stdin);
        fputs (data, fPtr);
        fclose(fPtr);
        printf("File created successfully.\n");
    }

    if (selection == 2)
    {
        ShellExecute(NULL, "open", "C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe" , NULL, NULL, SW_SHOWDEFAULT);
    }

    return 0;
}
</code>
Comment: "but without nothing" - I don't know what you mean by this.
Comment: Does this answer your question? [fgets doesn't work after scanf](https://stackoverflow.com/questions/5918079/fgets-doesnt-work-after-scanf)
Comment: Don't use notepad, at least download [notepad++](https://notepad-plus-plus.org/downloads/) or [geany](https://www.geany.org/download/releases/) or [VScode](https://code.visualstudio.com/download) -- but note, hacking the `.json` config files to make VScode work can be frustrating. A proper editor will make life much, much easier.
Comment: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Answer: When you got the menu, you probably typed "1" and then "enter". The "1" selected the notepad function. Then the "enter" supplied an empty line.
If you want to put in "test" for the notepad, you need to enter "1test" and then hit enter. That will supply "1" for the menu and "test" followed by enter for the line.


Why does an (Augmented) Dickey Fuller test apply in an ARMA situation when it (seems to) assume an $AR(p)$ situation?

Question: The set-up of this question may come out a little confused. But that's because I'm just that -- confused! If you can resolve my confusion, then that would be a full and satisfying answer!
To be as precise as possible amidst my confusion, let's break my question down to parts:

What is the null hypothesis of the (non-augmented) Dickey Fuller test? Is it that you begin with a time series that satisfies some $AR(1)$ equation such that its characteristic polynomial has no root at $1$? Do you further assume that the time series is stationary? (There are non-stationary time series that satisfy an $AR(1)$ equation with no root at $1$, right?)
Same as Question 1, but for the augmented Dickey Fuller. Presumably we assume that it satisfies some $AR(p)$ equation. Do we assume stationarity? (Is it not true that there are time series that satisfy some $AR(p)$ equation but are not stationary? Sure it is.)
It seems to me that the (augmented) Dickey Fuller test is often used for situations that are ultimately modeled using an $ARMA$ model. In that case -- why would ADF (augmented Dikcey Fuller) apply? Doesn't it assume that we're in an $AR(p)$ situation?
Is there a theorem that reduces an $ARMA$ situation to an $AR$ situation for checking whether the $AR$-characteristic polynomial has a root at $1$? What is it exactly?
It is often colloquially said that ADF is a "test for stationarity". Can you word that more precisely?

Any answer to any of these questions will be very welcome! Don't feel obligated to answer all of them in one fell swoop.
Edit: If I'm reading http://faculty.chicagobooth.edu/ruey.tsay/teaching/uts/lec11-08.pdf correctly, doesn't Dickey Fuller assume something much milder than $AR(1)$? Does it actually just assume that $z_t=\pi z_{t-1}+y_t$, where $y_t$ is some stationary process that satisfies very mild conditions? (This seems to subsume $ARMA(1,q)$, no?)
Comment: Related post [here](https://stats.stackexchange.com/questions/305038/is-the-ar-characteristic-polynomial-independent-of-the-arma-equations).
Answer:
The null hypothesis for DF test is a type of non-stationarity. $H_0: \phi_1 = 1$ where $X_t - \mu = \phi_1(X_{t-1}-\mu) + Z_t$.
The null hypothesis for ADF test is a type of non-stationarity as well, but with an AR(p) model. If your model is $X_t - \mu = \phi_1(X_{t-1}-\mu)+\cdots + \phi_p(X_{t-p} -\mu) + Z_t$, then $H_0: \sum_{i=1}^p \phi_i = 1$.
If an ARMA model is invertible, we can approximate it with an AR process. That's because we can divide both sides of $\phi(B)(X_t - \mu) = \theta(B)Z_t$ by the polynomial $\theta(z)$ to get $\frac{\phi(B)}{\theta(B)} (X_t - \mu) = Z_t$.
See above.
See above.
Comment: @AndrewNC, out of 6 questions you asked (including this one but not any newer ones) you got answers to 4 of them (even 2 answers for one of those). Thus more often than not, you *do* get answers. You also got an answer to your 7th question, which is a good sign, too. However, you have not accepted any answers that you got, thus giving no reward to the people who spend their effort helping you, which may discourage future answers.( You do not need to accept answers that are lacking, but you should then indicate what is missing and ask for clarification.)
Comment: Ah, so the crux is your answer to #3!

So this only applies to invertible ARMA's? Does every time series that satisfies *some* ARMA equation also satisfy an invertible ARMA equation? (In that case, is the invertible ARMA equation unique? Is it unique if you fix p and q?)
Comment: @AndrewNC good question; it sounds like a meaty-enough question to warrant it's own thread/question.
Comment: Alas, I can only ask a new question every 40 minutes. And more often than not, nobody answers my questions.


Multi-stage sampling together with hierarchical/ mixed effects models: which R packages?

Question: Analyzing educational datasets we have samples of children from samples of class in samples of schools - we have sampling weights, so I use the survey package e.g. to do a linear model. But this kind of design also requires looking at the mixed effects. But this isn't possible using the survey package. I can do this in nlme – but then I don't know how to account for the weighting. I guess I could use the sample weights as predictors in nlme regressions but I don't think that is correct.
It seems that this kind of design (in fact any stratified survey sample which includes nested levels) needs analysing from both perspectives – (survey weights and mixed effects) at once – but the packages of choice for each of these perspectives, survey and nlme, each don't seem to have slots for the other perspective.
Can someone put me on the right track, or suggest another package which does both at the same time?
Answer: I have looked at this recently, and concluded that nothing works in R the way I wanted to. So I analyzed my multilevel complex survey data in Stata using gllamm package, which can account for weights at multiple levels and clustering (but still can't do stratification). I would be happy to hear of otherwise available packages, but generally the multilevel people and the survey people do not overlap that much.
Comment: I know you posted this a few years ago. Since then have you had any luck using sampling weights with mixed effects models in R? I'm particularly interested in using them with the Ordinal package and `clmm()` function.
Comment: I have not looked back in a long while; I think last time I looked was in 2014.


What is the application for using the `Boltzmann Machines`?

Question: I've came across this post from wikipedia about Boltzmann machines, which concludes that it is not a model that is usually used in practice, and that its' constrained version (i.e. where the network resembles a bipartite graph) is more usable.
My question is - what the Boltzmann machines are used for, and why is the constrained version is more usable than the unconstrained one?
Answer: The idea behind the Boltzmann Machine is that it represents a closed system where an energy flows from one part to another, i.e. heat dissipation, and models the decrease in the entropy of a closed model - while the model starts with relatively low entropy (i.e. when there is a separation between 'hot' and 'cold' parts), it tends to the state of equilibrium, or high entropy (i.e. all the items of the same energy, or 'heat'). Those networks are a type of Hopfield networks, which are used for associative memory modelling (here is the link from wikipedia).

To the reason why it is not so useful - it is due to the infeasibility of its' solution in the general case, when there are an edge between each node to the other, even a relatively small network will have n(n-1) connections.

The answer for the first question is also in part an answer for the second, but in addition to it, Constrained Boltzmann Networks may model a more realistic scenarios, where nodes are divided into two separate groups, and the edges between them may represent weight of data flow, etc.


Drop down forms that when submitted load url

Question: I am trying to make a basic drop down forum which when options are selected they will load a URL on a submit button click.
I would like something like this
Drop down one
Make of bike:
KTM
BMW
Ect
Model of bike:
(if ktm was picked above would show)
990
EXC
Ect
(if BMW picked above would show)
GS
Adventure
Ect
So if make of bike was ktm and model was 990 on click of submit button would load
Www.website.com/ktm/990
this is the code i have i have got the picking part sorted so if you pick KTM it shows you models of KTM and the same with BMW the only problem is i cant get the url to work at all and have no real idea how to do it.

and the form code
  <form name="myform">

<select name="bike" size="1"
onchange="setOptions(document.myform.bike.options[document.myform.bike.selectedIndex].value);">
<option value=" " selected="selected"> </option>
<option value="BMW">BMW</option>
<option value="KTM">KTM</option>
<option value="Yamaha">Yamaha</option>
</select>

<select name="model" size="1">
<option value=" " selected="selected">Please select one of the options above first</option>
</select>
<input type="button" name="go" value="Search"
onclick="alert(document.myform.model.options[document.myform.model.selectedIndex].value);">

</form>

Answer: You can create the URL using simple concatenation of the selected options, and redirect to the chosen page by setting location = "<new URL>" in your button click handler:
onclick="location='http://www.website.com/' + myform.bike.options[myform.bike.selectedIndex].value + '/' + myform.model.options[myform.model.selectedIndex].value;"

Let me know if you need a worked solution.
You should also do a check to ensure that the user has selected something in the bike drop-down before allowing the button to be clicked. Otherwise the URL will not be valid.
Edit: here's a JSFiddle showing the complete solution: http://jsfiddle.net/V8jPn/1/
Answer: Add this function to your script
Javascript:
function loadURL(){

    var bike = document.myform.bike.options[document.myform.bike.selectedIndex].value;
    var model = document.myform.model.options[document.myform.model.selectedIndex].value;

    if(bike != ' ' && model != ' '){

        var url = 'http://YOURWEBSITE.com/' + bike + '/' + model;
        window.location = url;

    }//if

}

And make this your submit button
HTML
<input type="button" name="go" value="Search" onclick="loadURL();">


vSphere power on recommendations?

Question: I have 5 hosts in a cluster. In vCenter when I power on a VM, I get a Power on recommendations popup. But i can only choose 1 host. How do I force it to turn on on a certain host?
Answer: Sounds like an issue with how DRS is managing that VM: https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.resmgmt.doc/GUID-25B61435-40FA-46B6-998B-8AD059655C6F.html
You can use DRS affinity rules to control where a VM can run.

vSphere power on recommendations?


How to create a membership using rules, using membership entity module

Question: I'm using the membership entity module, and the membership entity commerce module.
I am trying to create a membership using  rules, without success yet.
There is one membership action available..to activate the membership but thats all.
EDIT:
This is the rule I used for checking out a membership:
<code>    { "rules_at_listing_import_create_user_and_membership" : {
    "LABEL" : "At listing import create user and membership",
    "PLUGIN" : "reaction rule",
    "OWNER" : "rules",
    "REQUIRES" : [
      "rules",
      "commerce_cart",
      "commerce_checkout",
      "membership_entity_commerce",
      "feeds"
    ],
    "ON" : { "feeds_import_listing_import" : [] },
    "DO" : [
      { "entity_create" : {
          "USING" : {
            "type" : "user",
            "param_name" : "[node:field-first-name][node:field-last-name]",
            "param_mail" : [ "node:field-user-email" ]
          },
          "PROVIDE" : { "entity_created" : { "user_created" : "Created user" } }
        }
      },
      { "user_unblock" : { "account" : [ "user-created" ] } },
      { "entity_save" : { "data" : [ "user-created" ], "immediate" : "1" } },
      { "commerce_cart_product_add_by_sku" : {
          "USING" : {
            "user" : [ "user-created" ],
            "sku" : "Listingmember-1y",
            "quantity" : "1",
            "combine" : "1"
          },
          "PROVIDE" : { "product_add_line_item" : { "product_add_line_item" : "Added product line item" } }
        }
      },
      { "commerce_checkout_complete" : { "commerce_order" : [ "product-add-line-item:order" ] } },
      { "membership_entity_commerce_activate_term" : { "commerce_order" : [ "product-add-line-item:order" ] } }
    ]
  }
}
</code>
Using this rule, I assume that the membership is created automatically because of the membership product being checked out.
Alternatively, I tried the method without using the membership product, by straight away creating an entity of type membership using rules. That does not work, the membership is not created.
Comment: your rule has to do various save operations, and all this in a single rule. Unexpected things may happen (or not happen). At this point I'd enable Rules debugging (within "settings"), to see if any errors show up there. If you do, please add relevant rules logging info (shown near the top of the page) as an extra update to your question.
Comment: @Pierre.Vriens No errors appear in the rules log or dblog. The ruls log shows: https://i.imgur.com/eAGCEfm.png   (I did not expand the default commerce reactions).. Basically it doesn't show much info about a membership
Answer: I did a good comparison of the Commerce membership entity module and the Commerce Licenses module, and the commerce licences can do te same, and has a lot more extra features, is more flexible. So switching to that module was my solution.


Is it possible to assign an Access listbox value to an output format on EmailDatabaseObject

Question: As the title suggests, what I need to know is a way to get a listbox value which has an email address to appear in the CC section of the output format of an Emaildatabaseobject.
I've tried multiple ways now but the best I've had is the =[Forms]![Training_admin]![List54] show in the outlooks CC, rather than the actual result.
There is only one result within the query which is appearing in the listbox, but I just cannot get the value to the email.
Answer: If the multiselect property is 'None', this should do:
<code>Forms.Training_admin.List54.Value
</code>
For multiselect, have a look here.


can Jackson serialize to a Java Set<> in RESTEasy?

Question: I'm using JAX-RS with RESTEasy and Jackson as a provider. I know from past experience (if I remember correctly) that I can have Jackson serialize my FooBar POJOs to a Java List<FooBar>, like this, generating either text or JSON:
@GET
@Produces({ MediaType.TEXT_PLAIN, MediaType.APPLICATION_JSON })
public List<FooBar> getFooBars() throws IOException, WebApplicationException;

But can Jackson handle Set<FooBar> as well?
@GET
@Produces({ MediaType.TEXT_PLAIN, MediaType.APPLICATION_JSON })
public List<Set> getFooBars() throws IOException, WebApplicationException;

Answer: Jackson will look at actual type of the instance during serialization, changing the return type of the method won't make a difference - you can change it to Object if you want. Deserialization is another matter however.
Jackson will handle all Iterables the same way, it'll serialize as an array


Manually computing fisher test with Big Numbers in Cells

Question: Hi all I have a 2x2 contingency table like this:
            Product B   Not in B    Row Total
Product A       25000       175000      200000
Not in A        4           24899996    24900000
Column Total    25004       25074996    25100000

since I have a cell with less than 5, I have to use the fisher test formula but I have to solve it manually thus I can't used the fisher.test function in R. I have to used this formula http://en.wikipedia.org/wiki/Fisher's_exact_test in order to do the test but factorial function in R and excel can't handle large numbers. Is their a way to compute the fisher exact test without using the factorial or simplifying the equation? Thanks.
Also I'm curious about the equation in fisher.test function in R for it can get the p-value above easily and I can't do it manually due to large factorial. I think that function uses some sort of shortcut. I am still studying it.
Comment: It does not matter that one cell has a small count.  What matters is its *expected* value, which is huge.  You can trust the chi-squared distribution.
Comment: I meant something different: the 80% is irrelevant here, because 100% of your cells have huge expected values.  In fact, the departure from independence is so blatantly obvious that any formal test would be superfluous.
Comment: Thanks @whuber. Forgot that the assumptions only state 80% of the cells must be 5 or more. But I am still looking on how R computes fisher.test. I got a close answer using the algorithms on the code.

**B=2000
almost.1 <- 1 + 64 * .Machine$double.eps
STATISTIC <- sum(lfactorial(x))
(STATISTIC/almost.1)/(B+1)**
Comment: @whuber thank you again. The chi square test and fisher are exactly the same with this kind of data.
Answer: The usual rules of thumb about using the chi-square don't refer to the observed data counts, but to the expected counts.
None of your expected counts are small. There's simply no issue here.
As whuber says, you can trust a chi square here -- or you could do a G test -- both will behave similarly in this case. (The Fisher test should also give a similar outcome.)
If you must do a Fisher test, I'd be inclined to use a normal approximation to the hypergeometric; I believe it will give essentially the same result you seek. Alternatively you could simulate to get the p-value.
Note that many packages - R included - have functions to compute log-gamma, log-factorial and log-binomial-coefficient functions. (In R they're lgamma,lfactorial, and lchoose.)

Some comparisons in R:
# set up data
ctb <- matrix(c(175000, 200000, 24899996, 24900000),nr=2,byrow=TRUE)

#chi square
> chisq.test(ctb)

        Pearson's Chi-squared test with Yates' continuity correction

data:  ctb
X-squared = 1654.073, df = 1, p-value < 2.2e-16

 # G test
 expec <- outer(rowSums(ctb),colSums(ctb))/sum(ctb)
 (G <- 2*sum(ctb*log(ctb/expec)))
[1] 1655.443
 pchisq(G,1,lower.tail=FALSE)
[1] 0

# Fisher test. If we proceed naively --
r <- rowSums(ctb)
dhyper(ctb[1,1],r[1],r[2],colSums(ctb)[1])
phyper(ctb[1,1],r[1],r[2],colSums(ctb)[1])

# then these numbers all zeros. This is enough that we can just say p is too small
# that trying to give an exact value makes no sense. But let's try to compute it
# anyway by working on the log scale:

dhyper(ctb[1,1],r[1],r[2],colSums(ctb)[1],log=TRUE)
[1] -834.3587

Well, that's a ridiculously low number, but we need it to find what tables are in the "at least as extreme" category. The corresponding smallest value at least as extreme (in terms of likelihood) in the other direction is 199814:
dhyper(199814,r[1],r[2],colSums(ctb)[1],log=TRUE)
[1] -834.4395

So the logs of the required tail probs are:
phyper(ctb[1,1],r[1],r[2],colSums(ctb)[1],log=TRUE)
phyper(199814-1,r[1],r[2],colSums(ctb)[1],log=TRUE,lower.tail=FALSE)

which give -832.2798 and -832.3606 respectively
We'd need to add those two probabilities given on the log-scale. One way to do that is take out a common factor (e.g. shift up by 832 and exponentiate), add the scaled probabilities, take logs and shift back down.
The result gives a combined log-probability of -831.6263, or a p-value of about $10^{-361.17}$. Which we already knew was largely meaningless (for a start, the assumptions won't hold closely enough for that to be even remotely close to a meaningful number).
By contrast the chi-square test gave a log p-value of -830.97 and the unadjusted G test gave a log p-value of -831.65, if I made no errors.
Normal approx to hypergeometric
Mean is nK/N
Variance is nK/N x (N-K)/N x (N-n)/(N-1)
N=sum(ctb)
K=colSums(ctb)[1]
n=rowSums(ctb)[1]
m=n*K/N
v=n*K/N*(N-K)/N* (N-n)/(N-1)
s=sqrt(v)
(z=(ctb[1,1]-m)/s)
pnorm(z,log=TRUE)+log(2) # 2nd term for other tail
-831.0353

With continuity correction:
(z=(ctb[1,1]-m+.5)/s)
pnorm(z,log=TRUE)+log(2) # 2nd term for other tail
-830.97

All these correspond to p-values on the order of $10^{-361}$. It surprises me quite how close together they all are.
[For all of them we might as well just have said "p-value < 2.2e-16" just as the inbuilt chi-square test does.]
Comment: +1 I'm not in the least surprised all the results are consistently tiny (the p-values actually vary by a factor of four to five), but I am quite surprised they are as *large* as this!


How to change the order of attributes in the Ubertcart cart page

Question: In the screenshot, you can see that the attributes for this product are listed in seemingly random order.  Actually, it seems to be ordered by the attribute ID.  Is there a way to change this order?  Each attribute has a "List position" property but this only seems to be for the product page, it doesn't affect the cart.
Comment: Are you using D6 or D7?
Comment: change order of field display under content type manage display settings.
Comment: Hi, I'm using Drupal 7.23 with UberCart 7.x-3.5.
Comment: Hi @ARUN, thanks for the answer but that doesn't seem to be the solution.  That only lets you change the order of the fields in the catalog or teaser view, but what I need is to change the order of the product attributes in the shopping cart.
Answer: By looking at ubdercart's theme_uc_attribute_add_to_cart function and how it is themeing the output including attributes, it appears that you can achieve what you want by implementing hook_form_alter. I would start by finding the form_id, perhaps by adding a debug statement in the function. Then I would implement hook_form_alter and would alter the way the attributes are ordered to my desired order.
If this does do what you want, try and follow the logic to see where the attributes are being themed and/or built. Then hook into that mechanism (form_alter or a preprocess function) and  I am confident you will achieve what you want.
Hope this helps.


Find app list which were added since last export for each device

Question: My question deals with selecting items from a series that were not present in a previous observation. I am trying to adapt a technique from SQL: selecting rows where column value changed from previous row.
I have an application that includes a table that stores a list of apps installed on devices. Devices submit this data daily, and the full list is saved each day. I have been exporting new records each day for import into a reporting system but the data is growing very large because my reporting dataset has to retain a much larger time range that the actual application. And the way the app stores this data is not optimal for reporting. I would like to export only records that indicate that a new app has been installed on a device since the last time I performed a data export. The report_ids of last export and for the most recent report from the device are known. The data structure is not ideal. Essentially I am trying to convert a time series of data into a change log.
I don't own the source application so I can't change the database's data structures, triggers, etc.
*** Please see fiddle for data structure and sample data: http://sqlfiddle.com/#!9/ec6040/17
SQL:
-- Sample data:
CREATE TABLE last_exported_reports
  (report_id int, mobile_device_id int);
INSERT INTO last_exported_reports
  (mobile_device_id, report_id)
VALUES
  (1, 1), (2, 6), (3, 7);

CREATE TABLE mobile_devices_denormalized
  (mobile_device_id int, last_report_id int);
INSERT INTO mobile_devices_denormalized
  (mobile_device_id, last_report_id)
VALUES
  (1, 4), (2, 6), (3, 8);

CREATE TABLE reports
  (`report_id` int, `mobile_device_id` int, `date_entered_epoch` bigint(32));
INSERT INTO reports
  (`report_id`, `mobile_device_id`, `date_entered_epoch`)
VALUES
  (1, 1, 1529981397691),
  (2, 2, 1529981397692),
  (3, 3, 1529981397693),
  (4, 1, 1529981397694),
  (5, 2, 1529981397695),
  (6, 2, 1529981397696),
  (7, 3, 1529981397697),
  (8, 3, 1529981397698);

CREATE TABLE mobile_device_installed_applications
    (`report_id` int, `identifier` varchar(8), `application_short_version` varchar(5));
INSERT INTO mobile_device_installed_applications
    (`report_id`, `identifier`, `application_short_version`)
VALUES
    (1, 'Chrome', 'c1.1'),   -- device 1
    (1, 'Word',   'w2.1'),   -- device 1
    (2, 'Skype',  's1.0'),   -- device 2
    (3, 'Excel',  'e3.0'),   -- device 3
    (4, 'Chrome', 'c2.1u'),  -- device 1
    (4, 'Word',   'w2.1n'),  -- device 1
    (4, 'Excel', 'w2.0'),    -- device 1
    (5, 'Skype',  's1.0'),   -- device 2
    (6, 'Skype',  's1.9'),   -- device 2
    (7, 'Excel',  'e3.0'),   -- device 3
    (8, 'Excel',  'e3.0');   -- device 3

SELECT
  mdd.mobile_device_id AS md_id, mdia.report_id, mdia.identifier, mdia.application_short_version AS ver
FROM
  -- List of all devices
  mobile_devices_denormalized mdd
  INNER JOIN
    -- Add in apps
    mobile_device_installed_applications mdia
  ON
    -- Only if they are from the last report from a device
    mdia.report_id = mdd.last_report_id
  AND
    -- And only if the latest report has not already been exported
    mdia.report_id NOT IN (select report_id FROM last_exported_reports)
  AND
    -- And only if the app in the new report was not in the last exported report
    NOT EXISTS (
      SELECT *
      FROM mobile_device_installed_applications exported
      WHERE exported.identifier = mdia.identifier
      AND  exported.report_id = mdd.last_report_id
    )
;

If I run the above without the NOT EXISTS clause, it works fine in that I get all app records for new reports as expected. When I add in the final step, the subquery to get rid of repetative apps, I get no results.
RESULT IF RUN WITHOUT THE SUBQUERY CLAUSE:
md_id    report_id    identifier    ver
1        4            Chrome        c2.1u
1        4            Word          w2.1n
1        4            Excel         w2.0
3        8            Excel         e3.0

This is correct for device 1 because report 4 is most
recent and is not the previously exported report (4 != 1)
This is correct for device 2 because report 6 is most
recent but it was already exported so no records are exported.
This is correct for device 3 because report 8 is most
recent and is not the previously exported report (8 != 7)

RESULT WHEN I ADD IN THE FINAL SUBQUERY CLAUSE:
No records.
EXPECTED RESULT:
md_id    report_id    identifier  ver
1        4            Excel       w2.0

The above record should have been printed because it is in
the most recent report (report_id=4) for device #1 and the app was not on the device the last time it was exported (report_id=1).
Comment: Let me know if you are still looking for an answer. The answer will exclude the first comment case scenario and returns output like the second comment.
Comment: What if an app is uninstalled? What should be expected output?
Comment: I think the output should be `1     4        Excel    w2.0`. As Word & Chrome was already present in report 1 (except different version for Word) but Excel was the new application installed in report 4. I have a query for that let me know if I am right.
Comment: @JitendraYadav, good question! Once I understand how to evaluate the current status based on the previous, I think I will be able to adapt the answer to deal with deletes as well as version upgrades. But I didn't want to over-complicate the question. It's already complicated enough.
Comment: You are right @jitendra. I corrected the question. Thank you.
Answer: SELECT a.*
FROM (
    SELECT
        mdd.mobile_device_id,
        mdia.report_id,
        mdia.identifier,
        mdia.application_short_version AS ver
    FROM mobile_devices_denormalized mdd
    INNER JOIN mobile_device_installed_applications mdia ON mdia.report_id = mdd.last_report_id
    ) AS a
LEFT JOIN (
    SELECT
        ler.mobile_device_id,
        mdia.report_id,
        mdia.identifier,
        mdia.application_short_version AS ver
    FROM last_exported_reports ler
    INNER JOIN mobile_device_installed_applications mdia ON mdia.report_id = ler.report_id
    ) AS b ON a.mobile_device_id = b.mobile_device_id  AND a.identifier = b.identifier
WHERE b.report_id IS NULL;

Logic: Match new report app list with app list from last exported report for the same device and select which are not matching from new report.
Comment: If you want to check which apps were un-installed, use full outer join. Here we have used left outer join.
Comment: Thanks for educating me. Greatly appreciated.


Ao fazer uma requisição ajax, há problemas se usuário recarregar a página logo em seguida?

Question: Ola, eu estou desenvolvendo um sistema em PHP onde ele trabalha com listas de IPTV armazenadas em serviços como o pastebin. O usuário submete uma url dessas, por exemplo (https://gitlab.com/bieldry02/FilmeseSeries.m3u8/raw/master/FilmeseSeries.m3u8), em um formulário e meu script vai analisar linha por linha do conteúdo da lista. O problema é que o servidor demora muito de responder quando a lista é muito grande. Então a minha ideia é fazer a requisição usando ajax. Só que como o servidor pode demorar a responder, o usuário pode continuar navegando no site, a minha grande dúvida é que se quando o servidor enviar a resposta da requisição haverá algum problema.
Adicionando mais informações:
É um site que tem um painel de usuário, eu quero da a liberdade de ele poder submeter uma lista dessas e enquanto o servidor não envia a resposta, ele continuar navegando normalmente no site.
Comment: Depende de como a navegação da página for feita. Podes explicar mais sobre a página que tens? ela é toda carregada de uma vez ou é dinâmica?
Comment: A página é carregada de uma vez.
Answer: Olá @Marcos Silva,
Utiliza o beforeSend() do ajax, quando a lista estiver carregando, mantém o usuário informado de que isso está ocorrendo através de um buffer/preloader, e seria relevante deixar o usuário focado nessa requisição feita por ele mesmo não o deixando clicar em outros botões durante esse processo,
então no success function(), vc obtém o resultado assim que a lista for definitivamente carregada, e o libera para continuar a navegação deixando assim os demais botões disponíveis.
<code>$.ajax({
    type: "GET",
    url: "pasta/",
    beforeSend: function(){

      //buffer preloader
      //bloqueia demais botoes

    },
   success: function( data ){
     setTimeout(function(){

       //carrega a lista completa(com paginação se possivel ou necessario)
       //desbloqueia demais botoes permitindo que o usuario continue sua navegação

     }, 2000);
   }
});

return false;
</code>
Comment: Mas quero da a liberdade de o usuário poder submeter uma lista dessas e enquanto o servidor não envia a resposta, ele continuar navegando normalmente no site.
Comment: Se o usuário recarregar a página inteira, via http, a lista vai iniciar o carregamento novamente do zero. Então vc precisa organizar essa estrutura de um modo dinâmico. *Vamos imaginar que vc tem um menu com varios links/botões, então cada link deverá ser carregado com ajax, e cada conteudo deverá ser carregado em divs diferentes, dessa forma vc consegue o resultado desejado. $(".lista").load("pasta/lista.php"); $(".conteudo2").load("pasta/conteudo2.php");


Get the country name from Latitude and Longitude-No Google

Question: Can I get the country information just from Latitude and Longitude of the user? I know it should be easy to get it from Geocoder APIs from Google. But is it possible to derive that information without using any services like Google's?
Could you please let me know?
Comment: Duplicate of **[Country code from a latitude longitude position (without using geocoding services)](http://stackoverflow.com/questions/5154593/country-code-from-a-latitude-longitude-position-without-using-geocoding-service)**
Answer: The only way of doing this is by using a third party API or creating your own database, which is a lot more work that it's worth. Have you looked to see if there is a GeoIP database that you can download for your programming language of choice (Or a generic one?)
Edit: This looks like it might have the data you need
Comment: We have to do that using JavaScript and Application cache.Size limit is a concern in our case.


display data (taxonomy terms) after uploading it directly to the database

Question: I have created a taxonomy vocabulary and then uploaded via mysql all the needed tables for the different fields.
I have created a view where I can see all the terms of the vocabulary, but if I go to <code>/admin/structure/taxonomy/myvocabulary</code> I can only see the terms that I have edited and saved, where, which table I have to modify for the terms to be displayed at the vocabulary page?
Answer:
From what I can tell those are all the taxonomy tables, but none of them seem to contain the term (url) path; thus, they must be in the url_alias table. Also, don't forget to flush your cache to see the changes.
I think it's better/safer if you use feeds to import your taxonomy terms.
Feeds

Import or aggregate data as nodes, users, taxonomy terms or simple
  database records.

One-off imports and periodic aggregation of content
Import or aggregate RSS/Atom feeds
Import or aggregate CSV files
Import or aggregate OPML files
PubSubHubbub support
Create nodes, users, taxonomy terms or simple database records from import
Extensible to import any other kind of content
Granular mapping of input elements to Drupal content elements
Exportable configurations
Batched import for large files

Video Tutorial
Comment: Thanks @No Sssweat, I had to do some work around, export the fields first, and some other things, but finally reimporting with feeds made the trick.
Comment: Just to clarify, I did export the vocabulary using Views Data Export module (https://www.drupal.org/project/views_data_export)


NuGet\Install-Package in only downloading package not creating package.config for projects inside the solution

Question: I used this NuGet\Install-Package Microsoft.CodeAnalysis.NetAnalyzers -Version 8.0.0 command to install the nuget package for the solution it is only creating packages folder i am not able to see the package.config in the projects folder is there any way that i can do this from package manger cli.
Answer: In NuGet 4.0+ the package.config is not common for packages anymore instead the packages have reference in your project file please check your project.csproj than in your .csproj file there should be <ItemGroup> where you should see your <PackageReference> it should look something like this:
<ItemGroup>
    <PackageReference Include="jQuery" Version="3.7.0" />
    <PackageReference Include="Microsoft.AspNetCore.Authentication.Negotiate" Version="6.0.15" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.Localization" Version="2.2.0" />
    <PackageReference Include="Microsoft.EntityFrameworkCore" Version="7.0.7" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="7.0.7" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="7.0.7">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.Extensions.Logging.Log4Net.AspNetCore" Version="6.1.0" />
    <PackageReference Include="Microsoft.VisualStudio.SlowCheetah" Version="4.0.50">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="MySql.EntityFrameworkCore" Version="7.0.5" />
    <PackageReference Include="MySqlConnector" Version="2.2.6" />
    <PackageReference Include="Newtonsoft.Json" Version="13.0.3" />
    <PackageReference Include="Select2.js" Version="4.0.13" />
    <PackageReference Include="System.Drawing.Common" Version="7.0.0" />
    <PackageReference Include="toastr" Version="2.1.1" />
    <PackageReference Include="ZXing.Net" Version="0.16.9" />
  </ItemGroup>

there is also a posibility that your project includes a whole folder for some reference like for my select-2 and toastr:
<ItemGroup>
    <Folder Include="wwwroot\lib\select-2\" />
    <Folder Include="wwwroot\lib\Resources\Fonts\" />
    <Folder Include="wwwroot\lib\Resources\Images\" />
    <Folder Include="wwwroot\lib\toastr\" />
    <Folder Include="wwwroot\Resources\imgs\" />
  </ItemGroup>

also you should see your packages in your solution explorer under dependencies =>
I believe if your package is installed correctly there should be a line like this in your .csproj
<PackageReference Include="Microsoft.CodeAnalysis.NetAnalyzers" Version="8.0.0" />

You can also add your PackageReferences yourself if for some reason is your nuget manager not creating them