Can I use K-Means to group customers based on a single variable?

Question: I have a test dataset of 11m records. The dataset contains a global customer id and spend figure.
I need to group customers into the following categories:

0  Low
1 Low/Med
2  Med
3 Med/High
4 High

I tried K-Means to group. See results below.
As you can see 10m records or so are in the low group as 80% of the db has low to negative spend.
If I want to further segment that low group, should I just increase the number of clusters? Or is there a better algorithm given the distribution of the data?
Thanks

count
mean
std
min
25%
50%
75%
Max

Cluster

0
10498822.0
21.147982
30.447597
-22885.364
6.78600
11.4520
26.30600
160.854

1
714573.0
300.654938
115.836596
160.855
207.94600
269.0280
366.02400
651.081

2
57318.0
1002.263623
400.515911
651.084
723.53375
841.8320
1118.37575
2370.803

3
14415.0
3739.988910
924.921881
2371.056
2993.61250
3599.1800
4319.69000
6162.907

4
3010.0
8584.038995
2476.616904
6163.451
6905.48800
7861.5815
9318.03100
22884.357
Comment: You could well also pre-specify intervals of values for your classes in a non-data driven way, or choose the classes so that they all have the same size. Why do you think k-means is better than this? In fact the terms "low", "low/med" etc. seem to refer to the interpretation of the observed values, and k-means is not interested in this interpretation, so why do you think k-means will come up with clusters that behave well according to your pre-specified description?
Comment: Thanks for the reply. The issue is that the values could change over a period of time and there are 30+ columns I need to do this for so pre-defining will be an issue. I was hoping there was a programmatic way to group in low/low-med etc.
Comment: This depends on what you really need. k-means is defined by an objective function that may or may not deliver what you want. I suspect that out of the purely data driven clustering methods it may well be the best here, but you may want to involve side conditions that make sure it gives you something that works even better for you. But then you need to specify what's wrong with the k-means solution and what you need instead. As said already, if you want the same number of observations in each category, you can enforce this directly rather than doing k-means.
Answer: Given your description, I would just assign cutoff points at percentiles of the distribution of total spend. With five categories, equal intervals (same number of observations in each category) would be quintiles, i.e. [0%,20%), [20%,40%),[40-60%),[60%,80%),[80%-100%].
Answer: If you want a data driven clustering, k-means looks promising in the sense that it will produce clusters with similar within-cluster variance, which may make sense in your application. The problem of imbalanced cluster sizes may come from a very skew distribution of values, and k-means may produce something more to your liking if you transform the data first, say, by taking logs or square roots. As written in the comments, it all depends on what you really need in this situation. There is no uniquely "true" clustering that could be found reliably from data. Different clusterings are possible on the same data and background knowledge regarding the meaning of the data and aim of clustering is required to decide between them.


Single row in ag-grid using autoHeight has extra space after row

Question: I'm using ag-grid in an angular project and I'd like to use autoHeight. When there is only one row in the grid, there is space below the row inside the grid.
grid with single row and extra space
When a second row is added, autoHeight works fine and does not display the extra space.
grid with two rows and no extra space
I can remove the space by manually setting the height when there is a single row
this.systemOptions.length === 1 ? this.gridApi.setDomLayout() : this.gridApi.setDomLayout('autoHeight');

but I would prefer to use autoHeight for any number of rows. Is there a way to display a single row using autoHeight without the extra space below the row?
Comment: did you try messing with the CSS?
Answer: Here is what the documentation says, but I have altered the code from unset to 40 as this worked for me and it really looks weird without min-height.

When using Auto Height, there is a minimum of 150px set to the grid
rows section. This is to avoid an empty grid which would look weird.
To remove this minimum height, add the following CSS:

.ag-center-cols-clipper {
    min-height: 40 !important;
}

Source:
https://www.ag-grid.com/javascript-data-grid/grid-size/


Store drawable or bitmap in internal storage

Question: I have a object with two string values and one drawable (or it may be bitmap - doesn't matters). I get pictures from the WEB. In my project I need to save couple of that objects in ArrayList for further use.
That ArrayList used to fill the ListView with custom adapter.
Now I using those methods for write and read ArrayList from file. It use serialization.
public final class InternalStorage{

private InternalStorage() {}

public static void writeObject(Context context, String key, Object object) throws IOException {
    FileOutputStream fos = context.openFileOutput(key, Context.MODE_PRIVATE);
    ObjectOutputStream oos = new ObjectOutputStream(fos);
    oos.writeObject(object);
    oos.close();
    fos.close();
}

public static Object readObject(Context context, String key) throws IOException,
        ClassNotFoundException {
    FileInputStream fis = context.openFileInput(key);
    ObjectInputStream ois = new ObjectInputStream(fis);
    Object object = ois.readObject();
    return object;
} }

Problem is Drawable or Bitmap objects are not serializable.
So what I suppose to do to store drawables along with every serialized object of ArrayList?
Maybe I need somehow associate every stored image file with every ArrayList item? And how to read It after all?
Thanks for answers and sorry for bad English.
Answer: you do not want to serialize the bitmap anyway as pure pixel-buffer would be to big. use the compress method at Bitmap
ByteArrayOutputStream baos = new ByteArrayOutputStream();
immage.compress(Bitmap.CompressFormat.JPEG, 100, baos);

Comment: And When I compress it - it just save the file to Internal srorage, right? So how I can associate these file with my Object and, finally, read it for using in ListView adapter?
Answer: save the ByteArray of those images then you can restore it.
Use these functions..
 public static byte[] convert(Bitmap bitmap) throws IOException {
            ByteArrayOutputStream stream = new ByteArrayOutputStream();
    bitmap.compress(Bitmap.CompressFormat.PNG,100,stream);
    byte[] array = stream.toByteArray();
    stream.close();
    return array;

}
public static Bitmap convert(byte[] array) {
    return BitmapFactory.decodeByteArray(array,0,array.length);
}


Calculations in a $sl_\alpha$-module

Question: I want to understand the proof of the following theorem:
For $\lambda\in H^*$ we have: $\dim M_\lambda<\infty \Leftrightarrow \lambda \in \Lambda^+$
($H$ is a Cartan subalgebra of a Lie algebra $L$ and $M_\lambda:=V_\lambda/S_{max}$ where $V_\lambda$ is the Verma-module of $L$ and $S_{max}$ is the unique maximal proper submodule of $V_\lambda$)
At the beginning of the proof we consider $M_\lambda$ as a finite-dimensional $sl_\alpha$-module ($sl_\alpha$ is a Lie subalgebra of $L$, which is generated by $\{e_\alpha,f_\alpha, h_\alpha\}$ and these three elements correspondend to the generators $\{e,f,h\}$ of $sl(2,\mathbb{C})$)
Now they say, if we have an irreducible, finite dimensional $sl_\alpha$-module $V$ and $e.v=0$ for some $v\in V$, then we get $h.v=mv$ with $m\in\mathbb{Z}_{+}$. Can you give me an explanation for that?
Later in the proof they say: If $V$ is a finite-dimensional $sl_\alpha$-module and $x\in V\backslash\{0\}$ satisfies $h.x=mx$, then $f^m.x\neq0$ if $m\geq 0$ and $e^{-m}.x\neq 0$ if $m<0$. How can I prove that?
I know that any irreducible finite dimensional $sl(2,\mathbb{C})$-module $N$ is isomorphic to $M_\lambda$ with $\lambda=\dim(N)-1$. ($M_\lambda=V_\lambda/S_\lambda$ and $V_\lambda$ is the Verma-module of $sl(2,\mathbb{C})$ and $S_\lambda$ is the unique maximal proper submodule of $V_\lambda$ for $\lambda\in \mathbb{Z}_{+}$).
Comment: What is $\Lambda^+$?
Comment: $\Lambda^+$ contains all dominant integral weights of $L$ resp. $H$.
Comment: But it is not used again after the second line, right?!
Comment: Yes, I only have problems with a few steps of the proof, but they don't need $\Lambda^+$
Answer: You posted two questions.
1) That's because of the classification of the irreducible $\mathfrak{sl}(2,\mathbb{C})$-modules. For each $m\in\mathbb{N}$ there is (up to isomorphism) one and only one such module and, for that module, the eigenspace with eigenvalue $0$ of the action of $e$ on that module is one-dimensional and if $v$ belongs to that space, $h.v=(m-1)v$. Since $m\in\mathbb N$, $m-1\in\mathbb{Z}_+$.
2) Again, this follows from the classification of the irreducible $\mathfrak{sl}(2,\mathbb{C})$-modules. I am assuming here that there is a typo in your question and that, when you wrote $e^m.x$ what you really meant was $e^{-m}.x$.

More details for 1) Take $m\in\mathbb N$. There is one and, up to isomorphism, only one irreducible $m$-dimensional representation $V_m$ of $\mathfrak{sl}(2,\mathbb{C})$. For that representation, the eigenvalues of the action of $h$ are $m-1,m-3,m-5,\dots,-(m-1)$. Furthermore, if $v$ is an eigenvector of the action of $h$ with eigenvalue $m-1$, then $e.v=0$. On the other hand, the kernel of the action of $e$ is precisely $\mathbb{C}v$. Indeed, there is a basis $(v_{m-1},v_{m-3},\ldots,v_{-(m-1)})$ of $V_m$ such that, if $k\in\{m-1,m-3,\ldots,-(m-1)\}$, then

$h.v_k=kv_k$;
$e.v_k=\begin{cases}v_{k+2}&\text{ if }k<m-1\\0&\text{ otherwise.}\end{cases}$

More details for 2) It turns out that the action of $f$ on the previous base is given by$$f.v_k=\begin{cases}0&\text{ if }k=-(m-1)\\\eta_kv_{k-2}&\text{ otherwise,}\end{cases}$$where the $\eta_k$'s are non-zero integers. Suppose now that $h.v=kv$ for some integer $k$; suppose furthermore that $k\geqslant0$. Could we have $f^k.v=0$? No, because the kernel of the action of $f$ on this module is $\mathbb{C}v_{-(m-1)}$, $h.v_{-(m-1)}=-(m-1)v_{-(m-1)}$, and $-(m-1)<0$. For a similar reason, if $k\leqslant0$, then $e^{-k}.v\neq0$.
Comment: Can you explain part 1) a little bit more? Why is that true? We didn't learn much about the classification of irreducible $sl(2,\mathbb{C})$ modules. I only know that each irreducible $sl(2,\mathbb{C})$-module $N$ admits a basis $\{v_m\}_{m=0,..,\dim(N)-1}$, such that $e,f,g$ acts as following:
$e.v_m=(\lambda-m+1)v_{m-1}$
$f.v_m=(m+1)v_{m+1}$
$h.v_m=(\lambda-2m)v_m$
(Again we have $\lambda=\dim(N)-1$)
Is this helpful for answering my questions?

In part 2) you are right, it must be $-m$, I changed that in my question.
Comment: @user95 I added a few more details to my previous answer.
Comment: I get the first part, thanks. Can you be a little more precise at the second part?
Comment: @user95 I've added more information for the second part.

Calculations in a $sl_\alpha$-module


Find a limit in an efficent way

Question: I'm trying to calculate the following limit:
$$\mathop {\lim }\limits_{x \to {0^ + }} {\left( {\frac{{\sin x}}{x}} \right)^{\frac{1}{x}}}$$
What I did is writing it as:
$${e^{\frac{1}{x}\ln \left( {\frac{{\sin x}}{x}} \right)}}$$
Therefore, we need to calculate:
$$\mathop {\lim }\limits_{x \to {0^ + }} \frac{{\ln \left( {\frac{{\sin x}}{x}} \right)}}{x}$$
Now, we can apply L'Hopital rule, Which I did:
$$\Rightarrow cot(x) - {1 \over x}$$
But in order to reach the final limit two more application of LHR are needed. Is there a better way?
Comment: Related : http://math.stackexchange.com/questions/632527/finding-the-limit-displaystyle-lim-x-to-0-left-frac-sin-xx-right1-x
Answer: Note that as $x\to 0^+$ you have
$$\frac{\ln\frac{\sin x}x}x\sim \frac{\frac{\sin x}x-1}x\sim \frac{\sin x-x}{x^2}\sim \frac{-x^3}{6x^2}\to 0$$.
Comment: I used an "asymptotic" version of Taylor-Mac Laurin formula for $\sin$ and $\ln$.
Comment: How can I explain those steps? I understand what you did, but how can I write it down formally? certainly, those tildas won't do the work :)
Comment: The tilda means asmiphtotic (see http://en.wikipedia.org/wiki/Asymptotic_analysis), so my writing is formally complete and correct. In particular, I applied $\ln(1+x)\sim x$ and $\sin(x)-x\sim -\frac{x^3}6$ as $x\to 0$.
Comment: hmm..OK, didn't know that. Thanks!
Comment: Would you add verbal explanation to this in a proof?
Answer: Elementary proof using well known limits and inequalities:
$$\frac{{\ln \left( {\frac{{\sin x}}{x}} \right)}}{x} =  \frac{{\ln \left( \left( \frac{{\sin x}}{x}-1 \right) + 1 \right)}}{\frac{{\sin x}}{x}-1 } \cdot \frac{\frac{{\sin x}}{x}-1 }{x} =\frac{{\ln \left( \left( \frac{{\sin x}}{x}-1 \right) + 1 \right)}}{\frac{{\sin x}}{x}-1 } \cdot \frac{\sin x-x }{x^2}$$
but for $x >0$ we have: $x-\frac{x^3}{6} \le \sin x \le x \Rightarrow -\frac{x^3}{6} = x-\frac{x^3}{6}-x\le\sin x -x \le 0 \Rightarrow -\frac{x^2}{6} \le\frac{\sin x-x }{x^2} \le 0$ the rest is squeeze theorem.
Comment: nice one, I don't think I could come up with that.
Answer: According to Bernoulli's Inequality, for $0\le x\le1$
$$
\left(\frac{\sin(x)}{x}\right)^{1/x}=\left(1-\frac{x-\sin(x)}{x}\right)^{1/x}\ge1-\frac1x\frac{x-\sin(x)}{x}\tag{1}
$$
This answer provides an elementary proof of Bernoulli's Inequality for rational exponents.
In this answer, it is shown geometrically that for $0\lt x\lt\frac\pi2$, we have $\sin(x)\le x\le\tan(x)$. Therefore,
$$
\begin{align}
0\le\frac{x-\sin(x)}{x^2}&\le\frac{\tan(x)-\sin(x)}{\sin^2(x)}\\[4pt]
&=\frac{\tan(x)-\sin(x)}{\sin(x)\tan(x)}\sec(x)\\[4pt]
&=\frac{1-\cos(x)}{\sin(x)}\sec(x)\\[4pt]
&=\frac{\sin(x)}{1+\cos(x)}\sec(x)\\[4pt]
&\to\frac02\cdot1\\[9pt]
&=0\tag{2}
\end{align}
$$
Thus, by the Squeeze Theorem,
$$
\lim_{x\to0}\frac{x-\sin(x)}{x^2}=0\tag{3}
$$
and therefore,
$$
1\ge\lim_{x\to0}\left(\frac{\sin(x)}{x}\right)^{1/x}\ge1-0\tag{4}
$$
Comment: To make the limit more elementary. I thought it would be nice to use only pre-calculus concepts.
Comment: Thank you sir, nice approach!
Comment: @SuperStamp: I've just removed L'Hospital :-)
Comment: Why did you remove LHR?
Comment: Cool. very appreciated! :)
Comment: Thanks robjohn for the excellent and most elementary handling of $\lim\limits_{x \to 0}\dfrac{x - \sin x}{x^{2}}$. I was thinking that L'Hospital/Taylor was the only way out here.
Answer: Here, as often, Taylor series are much more efficient than L'Hopital's rule.
$$\begin{eqnarray}\dfrac{\sin(x)}x \ &=&\ \color{#c00}1\color{#0a0}{\,-\,\dfrac{x^2}6+O(x^4)}\!\!\!\!\!\!\!\\&=&\ \color{#c00}1\, +\,\color{#0a0}z\\
\Rightarrow\ \ x^{-1}\log\left(\dfrac{\sin(x)}x\right) &=&\ x^{-1}\log(\color{#c00}1+\color{#0a0}z)\ &=&\  x^{-1}\left(\ \ \ \color{#0a0}z\ \ \ +\ \ O(z^2)\right)\\
&& &=&\ x^{-1}\left(-\color{#0a0}{\dfrac{x^2}6+O(x^4)} \right)\\
&& &=&\ -\dfrac{x}6\, +\ O(x^3)\\
&& &&\!\!\!\!\!\!\!{\rm which}\overset{\phantom{I}} \to \ 0 \ \ \, {\rm as}\, \ \ x\to0^+\\
\end{eqnarray}\qquad\quad\qquad\qquad$$

Find a limit in an efficent way


How to programmatically get the field cardinality

Question: Is there any way to get the number of paragraphs' repeating limit for the entity's field which those paragraphs depends on?
I'm writing a code which creates paragraphs according to the data taken from a CSV data and and insert their target ID (and target revision ID) into user fields programmatically.
The repeating limits of paragraphs for each user field differs.
Some user fields allow paragraphs to repeat only a few times and others expect them to repeat 10 times.
So if my guess was correct it would be useful if I can get the number when collecting data from CSV and that's why I'm asking.
Comment: Probably you find your solution here: https://drupal.stackexchange.com/questions/191402/get-a-field-instances-settings , except if it's dynamically set with validation criteria on form submit. So you get the field settings, and you search for the right settings, it should be there.
Answer: Thanks for the comments, I figured out a way to get the number(cardinality). Hope it helps someone.
<code>(array) $user_fields = \Drupal::service('entity_field.manager')->getFieldDefinitions('user', 'user');

foreach ($user_fields as $user_field_name => $user_field_definition) {
    (array) $cardinality = $user_field_definition->get('fieldStorage')->get('cardinality');
}
</code>
Comment: Instead of the generic `get()` methods, you really should be using the dedicated getters. This is because `get('fieldStorage')` may return NULL if the fieldStorage property has not been initialized yet. This always works:

`$cardinality = $user_field_definition->getFieldStorageDefinition()->getCardinality();`


Separar tag

Question: Tenho uma variável que pode armazenar texto simples e as tags ; '; // Com problema
console.log(test.split(';'));

Answer: O problema não está nos ; mas sim no fato de que os navegadores fazem o parse da string .
Como tens no conteúdo da variável test o texto ), veja a resposta do Zuul para uma possível solução (outra seria mover seu script para um arquivo externo, que no meu entendimento não estaria sujeito a esse problema).
Quanto à questão dos pontos-e-vírgulas dentro dos scripts, na ausência de uma solução mais completa (i.e. um parse apropriado do conteúdo) o que você pode fazer é - depois de separar em pedaços - juntar de novo aqueles pedaços que deveriam estar juntos. Exemplo:

var test = 'texto1;texto2;";
for ( var i = 0 ; i < resultado.length ; i++ )
    document.querySelector("body").innerHTML += "
" + resultado[i].replace(/</g, "<") + "
";
document.querySelector("body").innerHTML += "
";
Comment: [Como se forma o plural da palavra ponto-e-vírgula?](http://www.ciberduvidas.com/pergunta.php?id=27946) :)

Separar tag


Error Code: 1064 - MySQL

Question: Olá, pessoal!
Ao construir uma procedure simples de inclusão deu essa seguinte mensagem!

Error Code: 1064. You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'natural,  naciona,  profiss,  nac_res,  cpf_res,  cin_res,  est_civ,  data_mat, ' at line 89

O Avisa dá justamente na variável natural
Segue o meu código da SP:
CREATE PROCEDURE sp_IncluirDadosAluno(
    m_j_codcli char(5) ,
    m_j_nome char(45),
    m_cind char(10),
    m_oexp char(15),
    m_cpff char(14),
    m_sexo char(1) ,
    m_natural char(25),
    m_naciona char(25),
    m_profiss char(25),
    m_nac_res char(25),
    m_cpf_res char(14),
    m_cin_res char(10),
    m_est_civ char(25),
    m_data_mat char(10),
    m_data_nas char(10),
    m_pai char(45),
    m_mae char(45),
    m_data_nap char(10),
    m_data_nam char(10),
    m_endereco char(100),
    m_cidade char(35),
    m_bairro char(35),
    m_estado char(2) ,
    m_cep char(10),
    m_telefone char(14),
    m_telefon1 char(14),
    m_email char(45),
    m_j_curso char(3) ,
    m_j_serie char(3) ,
    m_j_areas char(3) ,
    m_ncurso char(35),
    m_nserie char(35),
    m_nareas char(35),
    m_j_turma char(10),
    m_j_turno char(1) ,
    m_j_sala char(5) ,
    m_j_situ char(1) ,
    m_j_bols char(1) ,
    m_descto double(10,2),
    m_foto char(30),
    m_J_CONV char(3) ,
    m_NCONVE char(30),
    m_MATRCONV char(12),
    m_MATRREAL char(22),
    m_numero char(5) ,
    m_ecivil char(12),
    m_certnasc char(25),
    m_ota char(1) ,
    m_lestrang char(20),
    m_nac_mae char(25),
    m_civil_mae char(12),
    m_profis_mae char(30),
    m_cracha_mae char(25),
    m_cpf_mae char(14),
    m_ci_mae char(10),
    m_cracha_pai char(25),
    m_naci_mae char(25),
    m_estac_mae char(12),
    m_rf char(1) ,
    m_nom_f char(45),
    m_pro_f char(30),
    m_cpf_f char(14),
    m_ci_f char(10),
    m_eci_f char(12),
    m_nac_f char(25),
    m_dna_f char(10),
    m_cra_f char(25),
    m_rped char(1) ,
    m_nom_ped char(45),
    m_pro_ped char(30),
    m_cpf_ped char(14),
    m_ci_ped char(10),
    m_eci_ped char(12),
    m_nac_ped char(25),
    m_dna_ped char(10),
    m_cra_ped char(25)
)

BEGIN

INSERT INTO l01j14
(
    j_codcli,
    j_nome,
    cind,
    oexp,
    cpff,
    sexo,
    natural,
    naciona,
    profiss,
    nac_res,
    cpf_res,
    cin_res,
    est_civ,
    data_mat,
    data_nas,
    pai,
    mae,
    data_nap,
    data_nam,
    endereco,
    cidade,
    bairro,
    estado,
    cep,
    telefone,
    telefon1,
    email,
    j_curso,
    j_serie,
    j_areas,
    ncurso,
    nserie,
    nareas,
    j_turma,
    j_turno,
    j_sala,
    j_situ,
    j_bols,
    descto,
    foto,
    J_CONV,
    NCONVE,
    MATRCONV,
    MATRREAL,
    numero,
    ecivil,
    certnasc,
    ota,
    lestrang,
    nac_mae,
    civil_mae,
    profis_mae,
    cracha_mae,
    cpf_mae,
    ci_mae,
    cracha_pai,
    naci_mae,
    estac_mae,
    rf,
    nom_f,
    pro_f,
    cpf_f,
    ci_f,
    eci_f,
    nac_f,
    dna_f,
    cra_f,
    rped,
    nom_ped,
    pro_ped,
    cpf_ped,
    ci_ped,
    eci_ped,
    nac_ped,
    dna_ped,
    cra_ped
)
VALUES
(
    m_j_codcli,
    m_j_nome,
    m_cind,
    m_oexp,
    m_cpff,
    m_sexo,
    m_natural,
    m_naciona,
    m_profiss,
    m_nac_res,
    m_cpf_res,
    m_cin_res,
    m_est_civ,
    m_data_mat,
    m_data_nas,
    m_pai,
    m_mae,
    m_data_nap,
    m_data_nam,
    m_endereco,
    m_cidade,
    m_bairro,
    m_estado,
    m_cep,
    m_telefone,
    m_telefon1,
    m_email,
    m_j_curso,
    m_j_serie,
    m_j_areas,
    m_ncurso,
    m_nserie,
    m_nareas,
    m_j_turma,
    m_j_turno,
    m_j_sala,
    m_j_situ,
    m_j_bols,
    m_descto,
    m_foto,
    m_J_CONV,
    m_NCONVE,
    m_MATRCONV,
    m_MATRREAL,
    m_numero,
    m_ecivil,
    m_certnasc,
    m_ota,
    m_lestrang,
    m_nac_mae,
    m_civil_mae,
    m_profis_mae,
    m_cracha_mae,
    m_cpf_mae,
    m_ci_mae,
    m_cracha_pai,
    m_naci_mae,
    m_estac_mae,
    m_rf,
    m_nom_f,
    m_pro_f,
    m_cpf_f,
    m_ci_f,
    m_eci_f,
    m_nac_f,
    m_dna_f,
    m_cra_f,
    m_rped,
    m_nom_ped,
    m_pro_ped,
    m_cpf_ped,
    m_ci_ped,
    m_eci_ped,
    m_nac_ped,
    m_dna_ped,
    m_cra_ped
);

END;
Answer: NATURAL é uma palavra reservada do MySQL (ver http://dev.mysql.com/doc/refman/5.0/en/reserved-words.html)
Assim sendo, no script, deve ser substituido
<code>(...)
INSERT INTO l01j14
(
    j_codcli,
    j_nome,
    cind,
    oexp,
    cpff,
    sexo,
    natural,
    naciona,
    profiss,
   (...)
</code>
por
<code>(...)
INSERT INTO l01j14
(
    j_codcli,
    j_nome,
    cind,
    oexp,
    cpff,
    sexo,
    `natural`,
    naciona,
    profiss,
   (...)
</code>
Ou seja, colocar o natural entre backticks (`)
Comment: Agora entendi muito bem, estou trabalhando na conversão de um sistema de escola, que a empresa desenvolveu aqui e **Fivewin + xHarbour**, atualmente pode-se dizer que ele tem de tudo, exceto a webservice. Agora vou poder ver quais são as palavras reservadas :) [Palavras reservadas do MySQL](http://dev.mysql.com/doc/refman/5.0/en/reserved-words.html) Muito Obrigado!

Error Code: 1064 - MySQL


Can every curve that is not a plane curve be embedded in the quadric hypersurface?

Question: Fix an algebraically closed field $k$. It is well known that the quadric hypersurface $Q=\mathbb{P}_k^1\times_k\mathbb{P}_k^1$ can be embedded in $\mathbb{P}^3$, and that any nonsingular projective curve $X$ can be embedded in $\mathbb{P}^3$. Can we sharpen this statement to say that any curve $X$ can be embedded in $Q$ or $\mathbb{P}^n$?
My reason for thinking this is that while plane curves only have a very narrow range of degrees, any nonsingular closed curve of degree $(a,b)$ in $Q$ will have genus $(a-1)(b-1)$, so in practice, if we want to find a curve that cannot be embedded in $\mathbb{P}^2$, we tend to look in $Q$. So do these give us all the examples?
My intuition is that this must be false, and any proof by counterexample would probably involve constructing some field $K$ such that $\operatorname{tr.deg}_k K=1$, and showing that such a curve could not exist in $Q$. However, this would seem to require more advanced theory than what is taught in the basic theory of algebraic curves, so I don't right away see how to do this.
Comment: As Mohan points out, not every curve embeds in $\mathbb{P}^1\times\mathbb{P}^1$. In fact, there is no surface which every curve embeds into, see [this question](https://mathoverflow.net/q/221957/21564).
Answer: Yes, it is false. As you said, any curve in $\mathbb{P}^1\times\mathbb{P}^1$ of type $(a,b)$ has genus $(a-1)(b-1)$, So, if a curve has genus $p$, a prime, of this type, it follows that $a$ or $b$ must be 2. But then, by a suitable projection, the curve must be a double cover of $\mathbb{P}^1$, that is hyperelliptic. If $p\geq 3$, there exists curves which are not hyperelliptic.


Pegar todos níveis de categorias dos produtos

Question: estou com uma dúvida. Tenho uma tabela de categorias:
CREATE TABLE `categorias` (
  `id` int(10) UNSIGNED NOT NULL,
  `parent_id` int(10) DEFAULT NULL,
  `lft` int(10) DEFAULT NULL,
  `rght` int(10) DEFAULT NULL,
  `nome` varchar(255) DEFAULT NULL,
  `publicado` int(11) NOT NULL DEFAULT '0',
  `icon` varchar(255) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

E uma tabela de produtos:
CREATE TABLE `produtos` (
  `id` int(11) NOT NULL,
  `titulo` varchar(255) DEFAULT NULL,
  `data` date DEFAULT NULL,
  `categoria_id` int(11) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

Onde produtos.categoria_id é uma chave estrangeira da tabela de categorias.
A tabela de categorias tem categorias e subcategorias, sendo que categorias.parent_id faz referência a um categoria.id para indicar que é filha da mesma.
Preciso exportar a tabela de produtos com um JOIN pegando 3 primeiros níveis das categorias, e não sei qual função utilizar para melhor adequação.
Resultado deve estar:
NOME PRODUTO - ID PRODUTO - NOME CATEGORIA NÍVEL 1 - NOME CATEGORIA NÍVEL 2 - NOME CATEGORIA NÍVEL 3 - DATA
Podem me ajudar?
Answer: Pelo que entendi, o produto sempre será relacionado com o terceiro nível ? Veja se te ajuda o seguinte código:
SELECT
p.titulo,
p.id,
n1.nome AS cat1,
n2.nome AS cat2,
n3.nome AS cat3
FROM produtos p
INNER JOIN categorias n3 ON n3.id = p.categoria_id
LEFT OUTER JOIN categorias n2 ON n2.id = n3.parent_id
LEFT OUTER JOIN categorias n1 ON n1.id = n2.parent_id

Comment: Sensacional, Rovann. Isso mesmo, muito obrigado!


Is there an event that is fired when a view exits?

Question: In asp.net mvc appliction, is there an event that is fired when a view exits? I want to save some screen settings when the view exits.
This can happen when a user clicks on any other (page/view) link in the nav bar or logs off.
Comment: you mean a event when the page refreshes
Comment: no only when some other view opens or application is logged off
Comment: what are those view that you are constantly talking about?aren't those pages?html pages?
Comment: yes for example about.cshtml or index.cshtml..so for example i want to leave Maps page and go to some other page at that point i want to save settings on maps page before leaving
Comment: i see so your views are pages, you need a unload event and some localstorage/sessionstorage/cookie
Comment: yes whenever the page unloads i need to do something
Comment: read https://developer.mozilla.org/ro/docs/Web/Events/unload    and https://developer.mozilla.org/en-US/docs/Web/API/Window/sessionStorage
Comment: or you simply use some get variables in your url
Answer: So i am doing it on window.onbeforeunload event. Thanks to madalin and this answer.
<code><script type="text/javascript">
 window.onbeforeunload = function (e) {

    var dataItems = [...];
    var model = JSON.stringify({ 'model': dataItems });

    $.ajax({
        url: "@Url.Action("SaveDefaultSettings", "Maps")",
        type: 'get',
        data: { defaultSettings: model },
    success: function (data) {

    }
});
};
</script>
</code>


How to evaluate "external" quality of clustering?

Question: Let's say you want to cluster some objects, say documents, or sentences, or images.
On the technical side, you first represent these object somehow so that you could calculate distance between them, and then you feed those representations to some clustering algorithm.
Externally, however, you just want to group similar (in some sense -- and that's where things become pretty vague for me) objects together. For example, in case of sentences we want for clusters to contain sentences about similar topic/concept; we feel that sentences "oh look at this pic of a cute lolcat" and "facebook revealed new shiny feature tonight" should be in different clusters.
What are the usual approaches for measuring this "external" quality of clustering? I.e. we want to measure how well our clustering procedure groups initial objects (sentences, images); we're not interested in internal measures (like averaged cluster radius, clusters sparseness), since those measures deal with objects' representations, not with real objects. Meaning, the chosen representation may be awful, and even if internal measures is great, externally we'll end up with clusters that are complete junk from our vague, subjective, "some sense"-ish point of view.
P.S. Having limited knowledge in clustering domain, I suspect I may be asking about really obvious thing, or my terminology may sound strange to clustering experts. If so, please advice what should I read on the subject.
P.P.S. Just in case, I asked the very same question on Quora: http://www.quora.com/How-to-evaluate-external-quality-of-clustering
Comment: Please note that there is an important difference between methods [clustering](http://en.wikipedia.org/wiki/Cluster_analysis) and [classification](http://en.wikipedia.org/wiki/Classification_%28machine_learning%29): for the first you don't know what your clusters are, for the second they are defined and you want to predict membership. What you mean by ""some sense"-ish point of view" may be the difference between the two.
Comment: I imply the clustering task, that is the classes/groups/clusters/topics are unknown. The intent of adding the "classification" tag was to maximise visibility of the question ;)
Comment: I have retagged accordingly :-) It looks like you want to put a priori constraints on your cluster solution, right?
Answer: I do not perfectly understand what you mean by internal and external quality. I assume that internal refers to a measure computed on the obtained partition while external is the result that you would like to obtain.
Usually, internal measure aims at comparing the within cluster distance compared to the distance between the cluster. Intuitively, if clusters are dense and well separated, then you have a good clustering. As this is the objective of clustering, you cannot really do better, unless you ask people to look at your partitions and say whether or not they are good.
If the resulting clustering does not seems good to you, it is probably that either your points are not correctly placed or your distance is not adapted to your problem. For example, suppose that your expected clusters form long parallel rectangle in your representation. If you use an euclidean distance, you won't be able to find the expected partition.
To solve this problem, if in the resulting partition, you find that their is points in the same cluster that should not belong together, then ask yourself why the chosen distance considered them as close. Then, just build (or read about) a new distance function that avoid this problem.
To sum up, if you find that the computed partition does not make sense, it is not necessarily because your quality measure is wrong, but more likely because the clustering performed the wrong task. Finding a good distance and space representation is probably the main task when doing clustering.
Answer: Based on what I understand from your question...
It sounds like you are clustering based on one set of characteristics, but then wanting the clusters to ideally reflect other characteristics that your clustering wasn't based on. The way to fix this is to give everything some thought before estimating any models, and making sure your "internal" charateristics match up with the "external" ones. In other words, if you want sentences that are clustered based on topic, then the crux is to find the distance metric (or whatever) that reflects that. EDIT: This is also basically what Mougel says.
That said, maybe you don't have any ideas on how to do this at the start and want to experiment a bit. The trouble with clustering is that the class labels are unknown...but from your example it seems like a person could look at the individual results and decide if they were happy with the clustering. So you could cluster, then take a manageable random sample of the output and see how well it did for yourself. Extending from this, you could assign class labels at the start and turn this into a classification problem.
Bottom line: I'm not aware of a "statistical" or automated approach to this - this seems like a problem that will be solved with more user input and thoughfulness.
Answer: Solving a real problem takes a combination of technique and domain knowledge. You're asking about technique, and you're getting good answers on that front. But you can't really succeed without some defined domain knowledge. Your main task is to elicit, focus, distill, etc, that domain knowledge. Then you'll be able to properly apply a technique.
So you need to help them (whoever it is that's waving their arms and crying "No good, no good!" about your clusters) focus. If you're clustering photos, make or solicit 20 specific examples of good clusters (of say 5 photos each), and 20 specific examples of bad clusters. You can then take the advice of the other answers in this thread and try to align your clustering distance with the answers.*
This is the part of statistics that books and classes really don't cover. It's the consultant side of the coin. But I can't see any way you'll succeed without doing it. (Even if the them is you.)
_* There are also semi-supervised techniques that use some labeled data and mostly unlabeled data, but as a first step you can explore your data with what labels you can elicit and then figure out what metric works best.


Is a C .lib file platform specific?

Question: I'm trying to use an API for a proprietary interface device on an embedded system (Freescale HCS08), and the provided files include headers (.h) and libraries (.lib).  The header compiles fine with the rest of my code (standard C), but when trying to link against the library I get memory errors saying the file might be corrupted.
My understanding of libraries in C is somewhat limited as I work almost exclusively on embedded systems where magic things like stdio, files, and dlls don't exist; but would the (or any) library be platform specific?  Does it contain fully (if there is any sort of level there) compiled code?  Some of the other files provided are VS project files, so if it is the case that the .lib is platform-specific, it wouldn't be unexpected that linking a file meant for x86-Windows to an 8-bit compiler would fail; it could be just me.
Answer: Not only is a .lib file CPU specific (there would be no way to link HCS08 code to x86 code), it is toolchain specific (CodeWarrior won't talk to SDCC, GCC/binutils won't talk to Visual Studio).
Comment: ...and also OS-specific: on UNIX you have `.a` files instead of `.lib` ones.
Answer: Yes the .lib contains compiled code so it is platform-specific.  If you have the source you should be able to re-compile it to your platform.
Comment: Even with the source code, it's not always so easy. Often, source code either has dependencies or use certain platform-specific features (in C, #ifdef _WIN32_, and so on), or some compiler-specific features. I'm sure you can find some code written for gcc 2.97 that won't work with gcc 4, even on the same Linux environment for example.
Comment: It's remotely possible I could get the source code; there shouldn't be much secret stuff in it as it's mostly a ton of functions that generate/parse serial messages, and the communications spec is well defined.  Like any library, would just save me a ton of time if I don't have to reimplement everything :P

Is a C .lib file platform specific?


How do I display all available field types?

Question: I want generate an new form in a custom module via the generate:form command.
In this step, I must enter a field type.

How can I get all available field types?
Answer: With the term "field type" you probably don't find what you are looking for. Better search for "form and render elements". You can find a list of all available elements here:
https://api.drupal.org/api/drupal/elements/8.2.x
You can use all elements of the type "FormElement".
Comment: Thank you :) I'm surprised , I don't see any taxonomy field type
Comment: That's a real field type (which are used to store data), in D8 that is not a separate field type, but an entity reference field.
Comment: If you want to store data, generate a custom entity with field types and then use an entity form.


Telerik Javascript error on unload page

Question: I am using Telerik controls in my Asp.net web application. I am facing javascript error on unload page:

Unable to set value of the property '_events': Object is null or undefined

I am facing this issue when i have merged/combined my asp.net website with MVC4.
I have searched on Google but unable to find any solution.
Edit:
I found problem:
When RadComboBox is empty then this error is occurred. RadComboBox Load data at runtime (when user clicks on it).
Any solution for this?
Answer: Which telerik controls are you using? What is their version?
If you are using the AjaxControlToolkit in the same project, try removing it (delete the assembly and the reference, rebuild) and see if this helps.
Comment: I have resolved it. It was version issue. I have added updated dll. Thanks for your response
Comment: Hi I haved added issue...can you please look into it?
Comment: Again, which Telerik controls are you using? Are they the MVC extensions? If they are the AJAX controls you may have issues with MVC environments.
What is the versio, does the latest fix the issue?
Are you using the AjaxControlToolkit? It messes up event handlers and MS AJAX.

Telerik Javascript error on unload page


How to convert low level code from TF 1 to TF 2

Question: I want to initialize weights and biases from a normal distribution in TF 2 to be similar to TF 1.
I cant seem to find the tf.truncated_normal in the API for TF 2.
w = tf.Variable(tf.truncated_normal([5, 5, channels_in, channels_out], stddev = 0.1))
b = tf.Variable(tf.constant(0.1, shape = [channels_out])

Answer: Random number operations are now in tf.random. So you just need to use tf.random.truncated_normal.


write back cache on network scsi disks

Question: I have a machine in the cloud which has a persistent disk. For the host, it appears as a SCSI disk. I see the following seting:
<code>$ cat /sys/block/sda/device/scsi_disk/0\:0\:1\:0/cache_type
write back
</code>
What does this mean? Does it mean that on this machine, there is a write back cache being used by the scsi layer and so a write I/O op might be just making it to this cache before   being deemed to be complete? (and later somehow get flushed to the actual disk across the network asyncronously)
Thanks.
Comment: That is what it means. I would also interpret it to mean "ok for temporary usage, don't put a database on it".
Comment: @GregAskew databases often are smart and issue write barriers (to ensure the data reached the target, not just the cache), so it should be safe to use for databases too.
Answer: Yes.
It may be safe to use for a database if the disk is on a controller with battery backed caching or it might not be a single disk at all but rather a SAN. In both cases you shouldn't be using the default IO scheduler. OTOH it might be a virtual disk on a SATA drive plugged directly into the host - in which case there is a good chance that things could go very pear shaped in the event of a sudden power outage.
But most cloud providers won't tell you that much about the hardware.
Comment: The `write back` setting is on the SCSI initiator machine.  So, what is the significance of whether the controller (i.e on the remote SCSI target machine) has a  battery or not? The said write back cache is on the initiator machine iiuc , (is that right?) and so data is going  to be cached on the initiator side itself and so suspect to loss.

write back cache on network scsi disks


How is FIFA world cup schedule created?

Question: For example, on 13/06 it was Spain and Netherlands. How are the matching teams decided and based on what? What are these groups? Sorry, I'm new to football and this site.
Answer: There are 32 teams in the 2014 FIFA World Cup in Brazil.  In order to get into the tournament, teams had to participate in regional preliminary qualifying tournaments.  Only Brazil, as the host country, received an automatic bid into the final tournament. (Source)
These 32 teams are organized into 8 groups of 4, labelled A through H.  To decide which teams are in each group, first the top eight teams are seeded into the first position of each group.  (Brazil, as the host country, is automatically seeded as team A1.)  After these 8 teams are each put into separate groups, the rest of the teams are placed into the groups based on geography and random draw. (Source)
The first stage of the tournament is called the Group Stage.  In this stage, each team plays the other three teams in their group.  That means that each of the 32 teams plays 3 matches, for a total of 6 matches in each group, and 48 matches in the entire group stage.  The Group Stage is scheduled for June 12-26, 2014.
After the Group Stage is done, the teams in each group are ranked by their performance in their three matches, and the top two teams in each group go to the next stage of the tournament.
These remaining 16 teams go into the Second Stage (or Knockout Stage), a single-elimination tournament.  The Second Stage begins on June 28, and the final match is scheduled for July 13, 19:00 UTC, in Rio De Janeiro.

How is FIFA world cup schedule created?


De Moivre's theorem application

Question: Suppose I am given a complex number
$$z=\frac{(1+i\cos(x)+\sin(x))^n} {(1-i\cos(x)+\sin(x))^n}$$ and I need it to convert to simpler form
First of all I know that a complex number is of form $$z'=r(\cos(x) + i\sin(x))$$ so I tried to convert $z$ into it (denominator and numerator separately  )
So it becomes
$$z=(\frac{i-1\cos(x)+i\sin(x)} {i+\cos(x)+i\sin(x)})^n$$
Now how to change it to polar form? So that I can apply de movires theorem?
Comment: Could you edit your title?
Comment: I think you mean De Moivre!
Comment: Edited the title.
Answer: A way to compute this is to write $1=\cos(0)+i\sin(0)$ and then use the prosthaphaeresis formulas
$$\begin{align}\sin(x)+\sin(y)&=2\sin(\frac{x+y}{2})\cos(\frac{x-y}{2})\\\cos(x)+\cos(y)&=2\cos(\frac{x+y}{2})\cos(\frac{x-y}{2})\end{align}$$
This turns the stuff inside the powers in numerator and denominator into polar form and then you can use De Moivre's formula to write numerator and denominator in polar form. Finally the quotient of complex numbers in polar form is easy to compute.
Answer: Multiply by $1+\sin(x) + i\cos(x)$ (the conjugate of the denominator) in the numerator and denominator to get
$$\frac{1+i\cos(x) +\sin(x)}{1-i\cos(x) +\sin(x)} = i\cos(x) + \sin(x) = e^{-ix+\pi i/2}$$
so
$$\left(\frac{1+i\cos(x) +\sin(x)}{1-i\cos(x) +\sin(x)}\right)^n = e^{-ixn + \pi i n/2} = \cos(nx - \pi n/2) - i\sin(nx - \pi n/ 2)$$

De Moivre's theorem application


Exclude keys from array with continue

Question: I have an array with keys from 0 to n, and i need to exclude the following pattern of keys with continue when the array is iterated:
3,4,8,9,13,14,18,19...
The problem is that i need some kind of formula so that the pattern is not limited only to the numbers above.
Answer: <code>$exclude = array(3,4,8,9,13,14,18,19);
foreach ( $inputArray as $key => $value ) {
  if ( in_array($key, $exclude) ) {
    continue;
  }

  // do something
}
</code>
Answer: <code>$array = range( 0, $n);
foreach( $array as $i)
{
    if( in_array( $i, array( 3,4,8,9,13,14,18,19)))
        continue;
}
</code>
Comment: So he can pull out what I set `$array` to and use his own - Seems like a pretty obvious thing to do.
Answer: <code>$exclude_keys = array(3,4,9);

for($array as $index => $value) {
    if(false !== array_search($index, $exclude_keys)) {
        //do what you want
    }
}
</code>
Answer: <code>$exclude_mods = array(3,4,8,9);
foreach ( $inputArray as $key => $value ) {
  if ( in_array($key % 10, $exclude) ) {
    continue;
  }

  // do something
}
</code>

Exclude keys from array with continue


Date picker in views for Drupal 6

Question: Is there any way one can display date pickers as filter for a drupal 6 view?
Answer: Yes, just use Date-Filter (node) and expose as Popup. I assume you have the Date module (6.x-2.9 in my case) and Date Popup module installed and set your Datepicker on admin/settings/date_popup...
In Views it probably won't work in the preview, but on the view itself it will.
Comment: what is the url for date_popup module?
Comment: It's bundled with the date module (http://drupal.org/project/date) and also requires the jQuery UI module (http://drupal.org/project/jquery_ui) for the Date Popups datepicker to work!
Comment: What about date fields that don't come from Nodes, Users or CCK?

Date picker in views for Drupal 6


Should I use redirect_to :back or request.refferer?

Question: When I want to redirect a user back to the previous page they were on (the refferer), should I use redirect_to :back or use the following block of code:
session[:return_to] = request.referer
# Do whatever you want
redirect_to session.delete(:return_to)

Are there any advantages to the longer method, or should I switch to the quick way :)
Answer: In Rails 4, redirect_to :back is the same as redirect_to(request.env["HTTP_REFERER"]).  And request is an instance of ActionDispatch::Request which inherits from Rack::Request which defines referer as:
def referer
  @env['HTTP_REFERER']
end
alias referrer referer

So, it doesn't look like it matters all that much which you use.  I would be inclined to use :back myself.
Comment: Thanks, using :back would also result in a cleaner code, so I think I'll switch!
Answer: redirect_to :back

is a shorthand for
redirect_to(request.env["HTTP_REFERER"])

so, your longer method is different, so the choice is up to you (i would use :back, you are storing something on session and then deleting that something, that's extra work)
Answer: Use:
redirect_to request.env["HTTP_REFERER"]

http://apidock.com/rails/ActionController/Base/redirect_to (pre Rails 3) or http://apidock.com/rails/ActionController/Redirecting/redirect_to (Rails 3)


Unstaged changes immediately after cloning in Eclipse

Question: I'm using Atlassian Stash, and the Git plugin for Eclipse Kepler, and I'm trying to clone a repository. As soon as I complete it, I immediately end up with N unstaged changes, where N vary everytime I try it. When I try and compare the two files that are apparently different, there isn't a single change. I'm guessing there may be some differences between the metadata of the files, but since I can't find what exactly has changed I don't want to commit/push the files either. Like I said, the number of files that have changed are completely random, and the kind of files that have been changed are also pretty random. (.project, .java, .groovy, +more).
I have had a look at this question, this question, and this question but none of them solved my question.
Any ideas as to why this happens?
Comment: I don't know for sure, but it sounds like a line-ending problem... Does the project use a different platform to the one you use?
Comment: I did look into that, but the entire project is using the same OS and developer setup, so it seems unlikely
Comment: Have a look at this question http://stackoverflow.com/questions/3195140/git-status-reports-modified-files-in-a-freshly-cloned-repo, it should solve your problem
Answer: Like already mentioned in the comments,this is a line-ending problem.
I also had this issue one day.
The git plugin(egit) you are using does have a issue with <code>.gitattributes</code> where you can configure how line endings should handled.
 - If you are using the plugin and still want to use it, you need to have to fix the line endings first and than configure your line endings in your IDE to use Unix-Line-Endings, as well as on the other(usually Windows) machine. This will solve the line ending problem for new files.

I am totally against the fact that git should handle your line-endings, you as a developer are responsible for managing this. So let your IDE handle this issue and configure it as already mentioned.


adding data to an existing content type programmatically

Question: I am sure this question has come up before, but right now my google skills are failing me.  I am trying to create a migration module since I haven't had any luck with either the migrate module or migrate_d2d.  Right now I am trying to populate a content type with the data from my field remap, but I can't find out how to do it.
would a simple db_insert work or is there something special I need to do to generate the field data?
Comment: http://fooninja.net/2011/04/13/guide-to-programmatic-node-creation-in-drupal-7/
Answer: Matt,
To create a node programmatically,

You firstly need to create a content type for same. This will make sure that Drupal architecture is followed to create a node.
Now you need to create a node from UI, and in hook_node_presave, make sure you print the node object - dprint_r($node); die; and see what is the Object architecture like.
Now you need too create the same object before saving it.
To, create the object, create $node = new stdClass();, define the node type $node->type = 'YOUR CONTENT TYPE NAME' and do a node_object_prepare($node), this will make sure that all the basic settings are added to the node object.
Post this, make sure you create the object, same as it was in node presave stage.
Just before calling node_save, call node_submit, which will prepare the node for submission.

Hope this helps !!!
Comment: Hi Vaibhav, thanks for the help.  though for some reason it still isn't working.  I am creating the node exactly as the original node is structured, i'm even running an array_udiff_assoc comparing both the keys and data types of the 2 arrays.  then double checking with an array_diff_assoc.  on node_save however, only the node is getting populated, the content_type tables are not.
Comment: Also, the node type is getting set as "node", this is probably the problem, however once I run node_object_prepare changing the $object->type causes errors all over the place.


Unicode representation to formatted Unicode?

Question: I am having some difficulty understanding the translation of unicode expressions into their respective characters. I have been looking at the unicode specification, and I have come across various strings that are formatted as follows U+1F600. As far as I have seen, there doesn't appear to be a built in function that knows how to translate these strings into the correct formatting for Python, such as \U0001F600.
In my program I have made a small regular expression that will find these U\+.{5} patterns and substitute the U+ with \U000. However, what I have found is that this syntax isn't the same for all unicode characters, such as the zero width join that actually is supposed to be translated from U+200D to \u200D.
Because I don't know every variation of the correct unicode escape sequence, what is the best method to handle this case? Is it that there are only a finite amount of these special characters that I can just check for or am I going about this the wrong way entirely?
Python version is 2.7.
Comment: Sorry, I'll add that in. It's Python 2.7. :)
Comment: `u'\u200D' == u'\U0000200D'`
Comment: Thanks for that! I wonder if the example I added will accept an extra zero. If that was the case maybe that could solve the non-uniform error. - Turns out that it does, as `\U00001F600` is a different character. Thanks anyway!
Answer: I think your most reliable method will be to parse the number to an integer and then use unichr to lookup that codepoint:
unichr(0x1f600)  # or: unichr(int('1f600', 16))

Note: on Python 3, it's just chr.
Comment: No, that's fine, it's just a crappy build of python.  Could you try `struct.pack('i', 0x1f600).decode('utf-32')` and see the result?
Comment: Using `struct.pack` is brilliant and disgusting at the same time. You should edit it into your answer. @lindsay there's no reason `unichr` couldn't do the same, it's just not good enough.
Comment: Thanks for these answers. It might be me, but I just tried the Python 2.7 example and it throws this error:

`unichr() arg not in range(0x10000)  (narrow Python build).` Any ideas?
Comment: What do you get for `import unicodedata; unicodedata.unidata_version` ?
Comment: Terminal says that I'm using `5.2.0`. Perhaps that's dated?
Comment: @lindsay that's a limitation of Python 2.7 in its narrow build form (which I believe is the default) - it's incapable of representing those higher number characters as a single Unicode codepoint, and needs to resort to a surrogate pair. In some version of Python 3 they fixed that problem, can't remember exactly which at the moment.
Comment: I just tried the `struct.pack` method. It seemed to work on that string. I'm baffled. What is going on behind the scenes there which isn't the case for `unichr`? This was my output. `u'\0001f600'`.
Comment: I'll leave the "disgusting" hack in the comments.  IOW, get a better build of Python :)
Comment: Thanks for these solutions. Really helps a lot. :)
Answer: U+NNNN is just common notation used to talk about Unicode.  Python's syntax for a single Unicode character is one of:

u'\xNN' for Unicode characters through U+00FF
u'\uNNNN' for Unicode characters through U+FFFF
u'\U00NNNNNN' for Unicode characters through U+10FFFF (max)

Note: N is a hexadecimal digit.
Use the correct notation when entering a character.  You can use the longer notations even for low characters:
u'A' == u'\x41' == u'\u0041' == u'\U00000041'

Programmatically, you can also generate the correct character with unichr(n) (Python 2) or chr(n) (Python 3).
Note that before Python 3.3, there were narrow and wide Unicode builds of Python.  unichr/chr can only support sys.maxunicode, which is 65535 (0xFFFF) in narrow builds and 1114111 (0x10FFFF) on wide builds.  Python 3.3 unified the builds and solved many issues with Unicode.
If you are dealing with text string in the format U+NNNN, here's a regular expression (Python 3).  It looks for U+ and 4-6 hexadecimal digits, and replaces them with the chr() version.  Note that ASCII characters (Python 2) or printable characters (Python 3) will display the actual character and not the escaped version.
>>> re.sub(r'U\+([0-9A-Fa-f]{4,6})',lambda m: chr(int(m.group(1),16)),'testing U+1F600')
'testing \U0001f600'
>>> re.sub(r'U\+([0-9A-Fa-f]{4,6})',lambda m: chr(int(m.group(1),16)),'testing U+5000')
'testing \u5000'
>>> re.sub(r'U\+([0-9A-Fa-f]{4,6})',lambda m: chr(int(m.group(1),16)),'testing U+0041')
'testing A'
>>> re.sub(r'U\+([0-9A-Fa-f]{4,6})',lambda m: chr(int(m.group(1),16)),'testing U+0081')
'testing \x81'

Comment: @lindsay Before Python 3.3 Unicode characters were stored internally using [UTF-16LE](https://en.wikipedia.org/wiki/UTF-16) encoding in a narrow build, and UTF-32LE in a wide build.  UTF-16 encoding uses one 16-bit word for Unicode codepoints below U+10000, and two 16-bit words for everything else.  So there were oddities like `len(u'\U0001F600') == 2` but `len(u'\U0000FFFF') == 1`.  `unichr()` only supported returning codepoints below U+10000 in a narrow build.  @wim's `struct.pack` trick works around that.
Comment: @lindsay Added a regular expression to parse that syntax, but note that Python 2 has issues with U+10000 - U+10FFFF if you are using a narrow build.
Comment: O.P. is dealing with *text* like `"U+1F600"`.  Since it's not the Python syntax, there is a need for some parsing or conversion, no?
Comment: Thanks for those answers. Python 3.3 is sound really good right about now. I probably should be using it in all honesty. I tried using the `unichr` method beforehand but it didn't work due to the narrow build issue. I assume the Python 3 `chr` method can handle this issue.
Comment: Yeah, as @wim said i'm editing the text expression of the unicode character. Thanks for the detailed answer non-the-less! :)
Comment: @MarkTolonen thanks for adding in that answer. I'd never heard of this narrow issue until now.
Comment: @MarkTolonen thank you for that in depth response! This unicode stuff is tricky, but that helps me get a better grasp of it all. :) Explains why I had gotten myself stuck trying to encode and decode these strings.
Answer: You can look into json module implementation. It seem that it is not that simple:
# Unicode escape sequence
uni = _decode_uXXXX(s, end)
end += 5
# Check for surrogate pair on UCS-4 systems
if sys.maxunicode > 65535 and \
0xd800 <= uni <= 0xdbff and s[end:end + 2] == '\\u':
    uni2 = _decode_uXXXX(s, end + 1)
    if 0xdc00 <= uni2 <= 0xdfff:
        uni = 0x10000 + (((uni - 0xd800) << 10) | (uni2 - 0xdc00))
        end += 6
char = unichr(uni)

(from cpython-2.7.9/Lib/json/decoder.py lines 129-138)
I think that it would be easier to use json.loads directly:
>>> print json.loads('"\\u0123"')
ģ

Comment: If it's json, sure.  But who said anything about json?
Comment: @wim, you can create strings from unicode sequences and feed them to json parser


When someone plays vocal warmup scales on a piano, what do they play?

Question: A very typical vocal warmup is to sing scales up and down from a note. Then again from the next note up. And so on, so you're going up and down but slowly moving higher and higher (or sometimes lower and lower). Often they'll play a chord at the end of each scale up/down.
But what is actually being played here, typically? Would it be as simple as playing each major scale, moving the starting note up chromatically each run? Or staying in the same key but moving the starting note one degree each time?
I can find an example video if needed but I'm assuming any vocalist or pianist who works with singers knows exactly what I'm talking about.
Answer: You are right. Typically these are just major or minor scales, moving up by a semitone each time.
But you can use anything that is relevant for the singer - sometimes you may want wider intervals to help them warm up over fifths or octaves for example.
Comment: When the pianist begins/ends each 'run' with a chord, would that typically be the major chord of the 1st degree of the scale then? e.g. play C major chord, then run up/down C major scale, play C#major chord, run up down C# major scale?
Comment: That's what I often see, yes. The chord matches the scale.
Answer: Adding to Dr Mayhem's answer, if the scale stayed in the same key, but just moved up a note each time, you'd be singing modes rather than scales. It's another good way of doing the warm up, and will get you used to how music works probably better, using both strategies. Also don't forget to use different sounds. With choirs, I've done the usual 'lah', but 'mee', 'ooh', 'doh', and 'day' work, along with 'bah' at the risk of sounding like sheep... Playing a chord at each beginning and end is just to adjust the ear. If, at the end of a Cmaj. run, you play G#7, it'll take you straight into C#.


image picker controller takes insane memory even if released

Question: my image picker is initialized like:

self.imgPicker = [[UIImagePickerController alloc] init];
self.imgPicker.allowsEditing = YES;
self.imgPicker.delegate = self;
self.imgPicker.sourceType = UIImagePickerControllerSourceTypePhotoLibrary;

presentation:

[self initImagePickerFromLibrary]; ----(this calls the lines I have mentioned before)
      [self presentModalViewController:self.imgPicker animated:YES];

      [imgPicker release];

apple instrument says:
before calling the picker memory used is 3 Mb.While the picker is in function 12Mb.While editing image is 28Mb and my app crashes.Anyone did find a solution for this?
Answer: The one thing that comes to mind is that you're doing self. which may mean that your image picker is retained twice. Once, because you're allocating it and once because you're using a setter.
What does the @property line look like for imgPicker? Does it have retain? If it does, you should either change the first line in your first quote above to:
imgPicker = [[UIImagePickerController alloc] init];

I.e. removing self..
Also in your second line, you're releasing imgPicker directly. If your @property does have retain, you should instead just do:
self.imgPicker = nil;

The reason is that the @property synthesizing retains and releases objects when you set them. If you release the imgPicker without setting it to nil, and then, later on, try to set it to a new value, the system will attempt to release the already released object, and probably crash. The above is the same as doing:
[imgPicker release];
imgPicker = nil;


BPM matching pre-recorded vocals in GarageBand

Question: I have some recorded track vocals that I would like to work with in GarageBand. For example, add a beat soundtrack to it. The problem is I fail at exactly matching the bpm of the vocals track.
I try playing around with the master bpm, sometimes seems like I've got it, but after a while you can hear the gap in synchronization increasing more and more.
Is there any way to systematically succeed at matching a track's beat in GarageBand?
Comment: Right, there's no "systematic way" that doesn't depend on the acapella having a system—an underlying tempo. If you want the best answer you'll need to provide more info on the acapella and whether it was recorded to a defined tempo. A vocal recorded to a click should sync up from any given point. An isolated vocal recorded to a backing band but no click should mostly sync up but may drift out gradually because backing bands aren't drum machines. A vocal recorded without a steady tempo at all might have to be sync'd phrase by phrase and/or use time stretching.
Comment: Is it a professional created acapella? The reason I ask is that if it was just someone singing the song, then it probably wasn't recorded to a click.
Comment: As far as I'm aware, there is no way to do beat matching in Garageband, also called Beat Mapping.  You can automate the tempo to have it change and try to follow the prerecorded track though, which is basically the same thing but involves more work/guess and check.  This is one of the downfalls of using something like Garageband.  It's free (with purchase of expensive computer), which is great, and does a really good job for most needs of a normal musician, however, it falls short on any more advanced needs.  You can often accomplish what you need but it's way more work and time to do so.
Answer: Can you slice your drum track into individual hits and then place them manually, maybe phrase by phrase? Do you have a midi keyboard or pads that would allow you to create a simple beat along with your vocals? You could then try matching the one you want to use to the simple one you created. The other option would be to try editing the vocal to match the tempo.
Answer: Despite the commented objection to the answer above from @robert winsly, he may have the right idea.   It seems that your vocal track wasn't recorded at a constant tempo. In fact, unless it was specifically recorded against a click, it's almost certain its tempo varies.  It IS possible to align a sequencer tempo map to such a recording, either manually, by entering individual tempo changes, or with some degree of automation - though I think you'll need a more advanced program than Garage Band for that.   But think what you'll get.  A continually-varying tempo map.  When you program a beat over that, you'll just get what sounds like a drummer who can't keep time!  Not a very good groove.
So you try the other way.  Fool around with the vocal track to tweak it into a steady tempo.  This will involve slicing it into chunks, moving and/or time-stretching each one to get it 'on the beat'.  But however skilfully done, you'll hear the artefacts.
Much better to re-record the vocals to a click, or (better) to a dummy drum track.   Though what you're asking for CAN be done, it's turd-polishing (you can polish a turd all you like, but you'll never get a good shine).  Far too much of today's 'music production' involves turd-polishing, when a better (and easier) approach would be to go back and record it right in the first place.
Answer: You can follow the method i use but for this you should know how to write a vocal in music notation by hearing the vocal or by playing the lead using a instrument.
1) Write the vocal in music notation (Lead sheet) form in any of the notation writing software with defined tempo.i use muse score for this (free music writing software)
2) Export the vocal in wav format and import in garage band- select the scale, meter and tempo of the song.
3) Add a software instrument dum beat and allow to run.
4) Sing and record the vocal along with the tempo of the Vocal wav file  imported in garage band.Follow the tempo of wav fil without any lapse.
Now save with the vocal recorded and the drum part - There will not be any mismatch of tempo.
Your song is ready with excellent drum beats.
Comment: This doesn't seem to answer the OP's question. OP already has a vocal track, and wants to add rhythm tracks. It seems like you are suggesting that OP should record rhythm tracks first, then record the vocal track to the rhythm tracks.
Answer: If you want to measure the thickness of a piece of paper, that's hard to do with one sheet, but easy if we have 500 sheets - we can measure the total thickness and divide by 500. We can use a similar trick if we have a long instrumental or vocal solo part that was recorded to a click (or steady beat):

load up the solo into an audio editor of your choice (or your DAW might be good enough)
place a marker exactly on a '1' beat (first beat of the bar) near the start of the solo recording
place another marker on a '1' beat near the end of the solo recording
count (in your head) how many bars there are between those two points
calculate the time difference in minutes between your two markers
the BPM is then (no. of bars x beats per bar) / time difference in minutes

e.g. if the difference between the markers is 1 minute, and you counted 30 bars of 4/4, then the BPM  = (30 * 4) / 1 = 120 BPM.
The main point of this technique is that if you use the longest section possible of your vocal track to work out the BPM, it minimises error - you're measuring lots of 'beats' all together and then dividing, just like with the paper.
Of course there may be challenges in counting if there are long silences in the recording - in this case it might be best to split up the recording and sync each piece separately. And if the original vocal wasn't recorded to a click, that's when you start to need fancy time-stretching and beat matching tools.
Comment: I basically use this method to determine the BPM of video game themes that I transcribe.


How can I add state to a lexer in Racket?

Question: I am trying to write a lexer for the sed language in Racket (ex. "s/find/replace/"). One problem I have been encountering is dealing with the fact that a lot of the tokens have no definitive form and can change. For example, I can write the above example as "ssfindsreplaces", where the letter 's' is used instead of '/'.
I've started writing a lexer such as,
(define sed-lexer
  (lexer-srcloc
   ["\n" (token 'NEWLINE lexeme)]
   ["/" (token 'DIVIDER lexeme]
   [(:or "s" "y" "d" "p") (token 'CMD lexeme)]
   [(:* (complement "/") (token 'LITERAL lexeme)]))

but this fails on multiple levels:

The command can only come at the beginning (in this simplified example). After a command has been read I want to ignore the command case until a newline.
The DIVIDER token can't be set always as a slash.

I can imagine a solution to this problem could be adding states to this lexer. So for example, the lexer starts in a 'start' state where it looks for a command, then it goes to the 'divider1' state, looking for what will be the divider character. Such a feature seems to exist here http://pygments.org/docs/lexerdevelopment/.  What would be the best way to go about solving this problem given the tools in the Racket ecosystem?
Comment: Which lexer are you using? https://github.com/racket/racket/wiki/language-development-toolchain#lexers-and-parsers
Comment: I was trying to get it work with the lexer at "br-parser-tools/lex" but I'd be open for a solution with something else.
Answer: A lexer is simply a function that consumes an input port and returns a token. If (br-)parser-tools/lex is not enough for you, you can write it yourself (it should not be difficult).
In theory, finite state machine and regular expression are equally expressive, so I think you can in fact use parser-tools/lex to accomplish what you want. It will just look really tedious due to how you need to split cases over all possible "divider"s (because pure regular expression doesn't have the backreference ability). I think that pygments that you mentioned will have a similar problem.
Another possibility is to use something more powerful than regular expression. As sed grammar is pretty simple, you can even parse it right away without lexing first. Here's a crappy version I wrote quickly using megaparsack, a parser combinator library
#lang racket

(require megaparsack megaparsack/text
         data/monad data/applicative)

(struct substitution (search replace flags) #:transparent)

(define substitution/p (do (char/p #\s)
                           [divider <- any-char/p]
                           [search <- (many/p (char-not/p divider))]
                           (char/p divider)
                           [replace <- (many/p (char-not/p divider))]
                           (char/p divider)
                           [flags <- (many/p (char-in/p "gIp"))]
                           (pure (substitution search replace flags))))

(define dummy-command/p (string/p "dummy-command"))

(define line/p (or/p substitution/p
                     dummy-command/p))

(define program/p (do [result <- (many/p line/p #:sep (char/p #\newline))]
                      eof/p
                      (pure result)))

(pretty-print
 (parse-result!
  (parse-string program/p
                "s/hello/world/\ndummy-command\ns|search|replace|gp")))

#|

Result:

(list
 (substitution '(#\h #\e #\l #\l #\o) '(#\w #\o #\r #\l #\d) '())
 "dummy-command"
 (substitution
  '(#\s #\e #\a #\r #\c #\h)
  '(#\r #\e #\p #\l #\a #\c #\e)
  '(#\g #\p)))

|#

Comment: Thanks! This is really helpful. It's great to see a solutions to this problem that I didn't know about.


does neo4j really require big RAM size for large set of data?

Question: Current data what we have is node store has almost 65MB of data, and 260MB of relationship store ?
And while going to production, expect 200 users using the application concurrently, which behind the scenes trigger neo4j queries. And I went through the neo4j calculator, it is recommended to have 64G RAM.
Currently the staging where I test has 3 RHEL servers with 8G memory each, and 4G has been assigned for java heap. And it has HDD not SSD. And each server 2 cpu cores.
And as a single user, the simple queries brings the fast, and some complex ones which has to run bigger data nodes run in 20 seconds. It goes well till 2 to 3 users. But after sometime, the results are not returned, and the query runs indeterminately.
I am not sure what is the expected system configuration for the kind of data, and the management wants to justify the requirement for higher system procurement.
Is there any chance, this configuration can be tweaked to support production load.
And I already did the performance tweaks as suggested in the neo4j site, from linux performance to memory cache performance parameters.
Or is it necessary to have bigger RAM to load all relationship into memory, to make the queries and concurrent loads have faster response

Query which is in Question
match (c:company)
where c.company IN [ "GENERAL ELECTRIC-TELEPRESENCE","PNOC_REG02","testzenos2","testzenos3","PT10","CMSP_SLT_SYNC","CMSP_SLT_SYNC_2","Smoke Test Company","PTrans-Regr","SPWIFI-POC","PNOC_REG01","IBM","MERAKISP-MERAKICUST","0101rms","Meraki_SP01-new3" ]

match (c)-[r1]->(s:physical_location)-[r5]->(im:im_tkt)
where (
   ( im.assigned_group_id IN [ "SGP000000000259" ] and im.company = "GENERAL ELECTRIC-TELEPRESENCE")
OR ( im.assigned_group_id IN [ "SGP000000000259" ] and im.company = "PNOC_REG02")
OR ( im.assigned_group_id IN [ "SGP000000000259","SGP000000000175" ] and im.company = "testzenos2")
OR ( im.assigned_group_id IN [ "SGP000000000259","SGP000000000175" ] and im.company = "testzenos3")
OR ( im.assigned_group_id IN [ "SGP000000000259","SGP000000000088" ] and im.company = "PT10")
OR ( im.assigned_group_id IN [ "SGP000000000175","SGP000000000088" ] and im.company = "CMSP_SLT_SYNC")
OR ( im.assigned_group_id IN [ "SGP000000000175","SGP000000000088" ] and im.company = "CMSP_SLT_SYNC_2")
OR ( im.assigned_group_id IN [ "SGP000000000088" ] and im.company = "Smoke Test Company")
OR ( im.assigned_group_id IN [ "SGP000000000088" ] and im.company = "PTrans-Regr")
OR ( im.assigned_group_id IN [ "SGP000000000088" ] and im.company = "SPWIFI-POC")
OR ( im.assigned_group_id IN [ "SGP000000000088" ] and im.company = "PNOC_REG01")
OR ( im.assigned_group_id IN [ "SGP000000000088" ] and im.company = "IBM")
OR ( im.assigned_group_id IN [ "SGP000000000088" ] and im.company = "MERAKISP-MERAKICUST")
OR ( im.assigned_group_id IN [ "SGP000000000088" ] and im.company = "0101rms")
OR ( im.assigned_group_id IN [ "SGP000000000088" ] and im.company = "Meraki_SP01-new3") )

RETURN collect(distinct(im.incident_number)) as im_tkt__incident_number ,
       collect(distinct(im.company)) as im_tkt__company ,
       collect(distinct(im.assigned_support_organization)) as im_tkt__assigned_support_organization ,
       collect(distinct(im.assignee)) as im_tkt__assignee ,
       collect(distinct(im.description)) as im_tkt__description ,
       collect(distinct(im.priority)) as im_tkt__priority ,
       collect(distinct(im.status)) as im_tkt__status ,
       collect(distinct(im.impact)) as im_tkt__impact ,
       collect(distinct(im.entryid_TS1)) as im_tkt__entryid_TS1;

This is the query which runs for a longer time
Comment: Perhaps you can share your data model + queries. SSD would be a great plus. Your store is tiny, so it should be no problem.
Comment: Neo4j version is 2.1.6 enterprise, and added the query in the question. Basically company is related to a site, site related to a device, device is related to alarm or im_tkt.    If an user logins to the application, he need to see all these data in nodes company,site,device,alarm,im based on the company and the im he is entitled to
Comment: did you profile the query? what does your :schema indexes and constraints look like? If you send an email to neo4j.com/contact-us we can help you also sort it out 1:1
Answer: Did you create indexes or constraints ?
create index on :company(company);
create index on :im_tkt(assigned_group_id);

Use Parameters in your real program !!
Pass in a list of maps for param like this:
[{company:"GENERAL ELECTRIC-TELEPRESENCE",
  assigned_group_ids:[ "SGP000000000259" ]},
...]

Then query:
UNWIND {param} as pair
MATCH (c:company {company:pair.company})-[r1]->(s:physical_location)-[r5]->(im:im_tkt)
WHERE im.assigned_group_id IN pair.assigned_group_ids
RETURN collect(distinct(im.incident_number)) as im_tkt__incident_number ,
       collect(distinct(im.company)) as im_tkt__company ,
       collect(distinct(im.assigned_support_organization)) as im_tkt__assigned_support_organization ,
       collect(distinct(im.assignee)) as im_tkt__assignee ,
       collect(distinct(im.description)) as im_tkt__description ,
       collect(distinct(im.priority)) as im_tkt__priority ,
       collect(distinct(im.status)) as im_tkt__status ,
       collect(distinct(im.impact)) as im_tkt__impact ,
       collect(distinct(im.entryid_TS1)) as im_tkt__entryid_TS1;

for testing you can use
UNWIND [{company:"GENERAL ELECTRIC-TELEPRESENCE", assigned_group_ids:[ "SGP000000000259" ]}] as pair
MATCH (c:company {company:pair.company})-[r1]->(s:physical_location)-[r5]->(im:im_tkt)
WHERE im.assigned_group_id IN pair.assigned_group_ids
RETURN ...


Searching for duplicate values in an array

Question: I am very new to C and I am hoping for some pointers. I am attempting to take an input of 7 integers of an array and search through them to see if any number appears only once. Here is what I have so far:
#define size 7
int main(void)
{
int array[size], target, i, prev, count;
//Initialize the array
printf("Please enter %d integers", size);
scanf("%d", &target);
prev = array[0];
count = 1;

for(i = 0; i<size; i++)
{
scanf("%d", &array[i]);
...

I realize it is quite terrible but C is completely strange to me.  I figured out how to input the 7 integers from the user but I haven't the first clue as to where to start attempting to index them. I also realized that there are more advanced ways to figure it out; however, I am attempting find the solution using basic concepts that an amateur could understand.
Comment: Don't forget to check that `scanf()` was successful.  Once you have the array entered, you can do a brute force, quadratic search for duplicates by looking at the each value in the array in turn (one loop), and search through the rest of the array (a nested loop) to see whether any other value is equal to the current value from the outer loop.
Comment: hey for your reference i found one link with good explanation about this problem check it out http://stackoverflow.com/questions/7117388/finding-out-the-duplicate-element-in-an-array
Comment: Thanks that makes sense.  Any idea how to check the array entered?  Or how to index it?
Comment: "I am very new to C and I am hoping for some pointers." Ha! You got 'em :)
Answer: The simplest way to search for duplicates (though not the most efficient one) is to sort the array. You can use the built-in qsort function as such:
int compare (const void * a, const void * b)
{
  return ( *(int*)a - *(int*)b );
}

/* ... */

qsort (array, size, sizeof(int), compare);

int seen = 0;
for (int i = 1; i < size; ++i) {
  if (array[i] == array[i - 1]) {
    if (!seen) {
      printf("%d\n", array[i]);
      seen = 1;
    }
  } else {
    seen = 0;
  }
}

Comment: This takes `n log n + n` time. It beats n ^ 2 for large n !!
Comment: O(nlogn + n) = O(nlogn). This is doable in O(n) using a hashtable, but I figured that's too complicated for the poster's level of understanding.
Comment: Right, espcially that his numbers are not ranged (so that an array cannot be used for hashing) and C doesn't provide `set` data structure
Comment: Right, if it were C++ or Java, I'd definitely go with a set-based implementation.
Comment: +1 In general, this is the best strategy for finding duplicates.
Answer: I know 3 methods to find duplicates, 2 are already answered, so here is the third (simplified)-
Complexity O(N) time, O(M) memory.
If the numbers are within some range like 0 - M and M is comparable to N the number of elements, you can use a array of size M+1 to check if the number has appeared before.
Code -
int exists[M+1]; //set M to appropriate value
memset(exists, 0, sizeof(exists)); //set all 0
for (i = 0; i < N; i++)
{
  if (exists[array[i]])
  {
    printf("Duplicate found\n");
    break; //or something else
  }
  exists[array[i]] = 1;
}

Note - Don't forget that input elements should be positive integers, not greater than M
Comment: you can also use some kind of hashmap instead of bitmap, and when you have hash collision on some element you just compare current value with ones in the bucket, will increase amount of code significantly, but also should reduce memory consumption
Comment: @Vorber: Of course, I mentioned the "simplified" approach for the OP.
Answer: This one is a little pain in the neck to read, so I explain what I did a little bit here:
First, include the standard i/o library, #define array size to whatever you want, declare you array (I called mine: int entries[SIZE];).
The first for loop after "Enter 10 numbers" is the main one, that allows you to push the 10 numbers into the array.
The following if statements are tests that are applied to value entered right after is typed:
1) The first if statement makes sure we enter the value within the proper range.
2) The following 'else if' suggests that if entries[i] = entries[0] (meaning if this is the first object in the array) let's not do anything because there's nothing to compare it to.
3) The last 'else' contains a nested loop. The outer loop gets initialized to 1 so we make sure that in the comparison that takes place in the inner loop we're always comparing the current value against a previous one.
I hope this helps... cheers :)
*/
#include <stdio.h>

#define SIZE 10

//declarations
int entries[SIZE];

int main(void)
{
    printf("Enter 10 numbers:\n");

    for(int i = 0; i <= SIZE-1; i++)
    {
        printf("[%d]:\n", i);
        scanf("%d", &entries[i]);

        if(entries[i] < 10 || entries[i] > 100) {
            printf("Please enter valid number (between 10 and 100)\n");
            scanf("%d", &entries[i]);
        }
        else if(i == 0) {
            ;
        } else
        {
            for(int j = 1; j <= i; j++)
            {
                *//internal loop goes through all the previous entries (entries[i-1], entries[i-2], etc)*
                for(int k = 0; k < j; k++) {
                    if(entries[j] == entries[k])
                        printf("%d is a duplicate value\n", entries[i]);
                }
            }
        }
    }

}

Answer: This can be done in O(n^2) algorithm:
int yes = 1, i, j;
for (i = 0; i < n; ++i)
{
   for (j = i + 1; j < n; ++j) if (arr[i] == arr[j])
   {
       printf("Found a duplicate of %d\n", arr[i]);
       yes = 0;
       break;
   }
   if (!yes) break;
}
if (yes) printf("No duplicates");

Comment: I think the 'doubles' referred to were 'duplicates' — hence my edit of the title.


How can I find the limit of the following sequence $\frac{1}{2} . \frac {3}{4} ....... \frac {2n - 1}{2n} $?

Question: How can I find the limit of the following sequence:
$$\frac{1}{2} . \frac {3}{4} ....... \frac {2n - 1}{2n} $$
The problem for me is that they are multiplied and not added, also I can see that the numerator is the set of odd numbers and the denominator is the set of even numbers but then what?
Comment: Suppose it has a limit, call it $L$.  Now, consider $\ln L$.  Now it is a summation and you can use your usual techniques.
Comment: @mathguy Moreover, the question I link when I voted to close actually evaluates the sum, so yes it is a duplicate whichever way you want to slice it. If there three newer questions each asking how to compute $\int_1^2 x^2dx$, and yet there was an older post showing how to evaluate $\int_1^t x^2dx$ for all $t$, wouldn't you agree that the older answer is sufficient?
Comment: @Batominovski - perhaps, but the linked threads (the answers) use more work to do more than simply answer the question about the limit. I didn't see the very elementary answer to the present question in those threads.
Comment: @mathguy  The second answer by Ahmad Bazzi does not use much knowledge.  Plus, there are two more links that are linked to that thread.   About that you "didn't see the very elementary answer," why should you expect "very elementary answer"?  Just because a question looks simple does not mean it will have a "very elementary" solution.  In fact, this question happens many times, but I am too lazy to find all old links.
Comment: @Batominovski - I didn't assume something, I asserted it. The problem does have a very elementary solution, I just posted one.
Comment: @Batominovski thanks for the diss ;)
Comment: @mathguy  As I said, this question popped up many times.  This is another thread: https://math.stackexchange.com/questions/2402508/convergence-point-of-frac-prod-k-1n-2k-1-prod-k-1n-2k/2402623#2402623.  It has an answer with exactly the same solution that you gave.  (I just realized that I also gave exactly the same solution in two threads.)
Comment: lol - OK, I see it. Then, if you want to suggest a question is a duplicate of an older one, you should perhaps spend just a little longer to find the right "original".
Comment: @mathguy  What?  That is an absurd statement.  Any link with an answer to this question is a "right original."  The method used in an answer is irrelevant.  And I didn't need to give the oldest link to prove that this question is a duplicate.  This question is a duplicate no matter which old link I happen to use to justify my claim.
Comment: @JavaMan - how can a question asking to compute a limit be a duplicate of a question asking to compute a closed form of the general term? As I showed, finding the limit is obviously a different (much easier) question, since it has a solution that doesn't help in any way with finding a closed form. "This question can be answered as a trivial corollary of another question" doesn't mean "this" question is a *duplicate* of the other one.
Comment: Duplicates include: https://math.stackexchange.com/questions/2886753/limit-of-the-product-prod-k-1n-left-frac2k-12k-right, https://math.stackexchange.com/questions/1586773/evaluate-prod-frac-2k-1-2k, https://math.stackexchange.com/questions/93001/closed-form-expression-for-the-product-prod-limits-k-1n-left1-frac1, https://math.stackexchange.com/questions/2402508/convergence-point-of-frac-prod-k-1n-2k-1-prod-k-1n-2k?noredirect=1&lq=1 OP was too lazy to look for duplicates, so question should be closed. Your answer is nice, but answer a different version of the question if you want the points.
Comment: @JavaMan - In your example, I wouldn't agree if the definite integral had a much simpler way to be computed. Change your example to $\int_{-1}^{1} x^3 dx$ vs $\int_{-1}^{t}\dots$. The former question has a trivial answer which has nothing to do with the more general question. Neither is a duplicate of the other.
Answer: Write $a_n$ for $\frac 1 2 \cdot \frac 3 4 \cdots \frac{2n-1}{2n}$ and $b_n$ for $\frac 2 3 \cdot \frac 4 5 \cdots \frac{2n}{2n+1}$. Then clearly $a_n < b_n$; but $a_n \cdot b_n = \frac 1 {2n+1}$ so $a_n \cdot b_n \to 0$ as $n \to \infty$. And $a_n^2 < a_n \cdot b_n$, so it must also be that $a_n \to 0$. That's your limit.
Comment: Cute. You don't need to use anything about the harmonic series.......+1
Answer: Apply the log we get
\begin{equation}
 \log \prod_{n=1}^N \frac{2n-1}{2n}=\sum_{n=1}^N \log \left(1-\frac{1}{2n}\right)\le -\frac12 \sum_{n=1}^N\frac 1n,
\end{equation}
because $\log (1 + x) \leq x $ for all $x > -1$. So
\begin{equation}
  \prod_{n=1}^N \frac{2n-1}{2n}
 \leq
 e^{-\frac12 \sum_{n=1}^N\frac 1n}
\end{equation}
 But $\sum_{n=1}^N\frac 1n \rightarrow \infty$ by the $p-$test. So,
 \begin{equation}
 \lim_{N \rightarrow \infty}
  \prod_{n=1}^N \frac{2n-1}{2n}
 \leq
  \lim_{N \rightarrow \infty}
 e^{-\frac12 \sum_{n=1}^N\frac 1n}
 =
 0
\end{equation}
By the sandwich theorem
\begin{equation}
 \lim_{N \rightarrow \infty}
  \prod_{n=1}^N \frac{2n-1}{2n}
  =
  0
\end{equation}
Comment: @JavaMan thanks for letting me know..
Comment: Just FYI: You should use `\prod` for the product operator to get a nicer looking product.
Answer: Let $S_n=\prod_{j=1}^n\frac {2j-1}{2j}.$ We have $$\frac {1}{S_n}=\prod_{j=1}^n(1+\frac {1}{2j-1})\geq \sum_{j=1}^n\frac {1}{2j-1}.$$ E.g. if $a,b,c$ are non-negative then $(1+a)(1+b)=1+a+b+ab\geq 1+a+b,$ and $(1+a)(1+b)(1+c)\geq$ $ (1+a+b)(1+c)=1+a+b+c+(a+b)c\geq 1+a+b+c.$
So $\frac {1}{S_n}\to \infty.$ So $S_n\to 0.$
Remarks: Two useful elementary tools:
(1). If each $a_n\geq 0$ then $\sum_{n\in \Bbb N}a_n$ converges iff $\prod_{n\in \Bbb N}(1+a_n)$ converges.
(2). If $0\leq a_n<1$ for each $n$ then $\sum_{n\in \Bbb N}a_n$ converges iff $\prod_{n\in \Bbb N}(1-a_n)\ne 0.$
In this Q, we may apply (2) with $a_n=1/2n$ to get $S_n\to 0$ directly, or, as I did above, apply (1) to the lower bound $\sum_{j=1}^n(1+1/(2n-1))$ for $1/S_n$ to get $1/S_n\to \infty.$


Heavy tailed rather than normal distribution for a stock simulator

Question: Have a program that simulates picking stocks 10000 times according to different parameters. I then look at the ending portfolio amounts, and the distribution is pretty heavy tailed. This doesn't violate 'normal distribution rules', correct? I remember in undergrad that the normal distribution would appear if I took a mean of all 10000 ending portfolio amounts, and then did that like, many times. Is my understanding correct? Is there a ELI5 version of this phenomenon?
Secondly, I aim to calculate the confidence intervals for the ending portfolio amounts. I generate 10000 samples for ending portfolio amount and then sort these results. For a 70 percent CI, I pick the 3000th and 7000th sample from the sorted results. Is this correct? I think that's not correct; I remember in undergrad when calculating CI, we had to collect a random sample from the 10000, and then resample with replacement, and then take percentiles of the resampled results? Vaguely remember something like that.
Answer:

You're alluding to the central limit theorem (CLT), which says:
$$
\sqrt{n}\dfrac{
\bar X-\mu
}{
\sigma
}
\overset{d}{\rightarrow}
N(0,1)$$
(The $d$ above the arrow means a particular type of convergence in probability theory called convergence in distribution.)
Loosely speaking, the sample mean converges to a normal distribution as the sample size increases.
However, there are conditions on the observations used to calculate that $\bar X$. There are multiple variants of the CLT, but a commonality is a requirement that the observations used to calculate $\bar X$ come from distributions with finite variance.
Financial processes often are thought to be too heavy-tailed to have finite variance, so the central limit theorem would not apply.
Even if the variance is finite, the tails might be so heavy that a finite sample size does not get you particularly close to the limit. Vanderbilt's Frank Harrell has tweeted about this, for instance.
2)
You're alluding to bootstrap confidence intervals, particularly using a percentile bootstrap, which is easy to calculate but not considered a great method. Software like the boot package in R makes it easy to calculate preferred bootstrap confidence interval, such as bias-corrected and accelerated (BCa).
Comment: When calculating confidence intervals, is resampling vs non resampling a conversation to have? Or do you always have to resample?
Comment: Meaning like, let's say we are estimating US voting preferences and we have 10000 samples out of the 350million population. Resampling would makes sense in that case. However, what if we have (hypothetically), 340million samples. In that case, resampling would not make sense, correct?
Comment: For my finance example, specifically users enter into the calculator what edge they think they have on each trade, how much of their portfolio to commit to each trade (this can either be fixed or variant), and also how many trades they plan to make in the upcoming year.
Comment: There are lots of ways to calculate confidence intervals. The first way people tend to learn has to do with a z-statistic or t- statistic. The Pennsylvania State University has some [online lessons](https://online.stat.psu.edu/stat200/lesson/4/4.2) about this topic.


Displaying the csv file content in a view

Question: We imported the content of the excel sheet into csv and generated a view using feeds importer(Feeds module). But we want show the csv file content onto a slidershow. How to display them on a revolution slider.
Answer: Change the said view's display format to "Slideshow". You also will have a whole bunch of settings to customize your slideshow.
PS: You only get this format when you install the Views Slideshow module


The set of zero divisors is the union of radicals of annihilators

Question: I am trying to figure out why the statement
$$\text{the set of zero divisors }=\bigcup_{0\ne x\in R} \sqrt{\text{Ann}(x)}$$
is true. Here $R$ is a commutative ring, $\text{Ann}(x)=\{r\in R\mid rx=0\}$ the annihilator of $x\in R$ and $\sqrt I=\{r\in R\mid \exists n:\ r^n\in I\}$ the radical of some ideal $I\lhd R$.
I can certainly see that the set of zero divisors is $\bigcup_{0\ne x\in R}\text{Ann}(x)$, but is it still true if we take the radical of the annihilator?
Answer: If $r^n \in \mathrm{Ann}(x)$ and $r$ is not a zero divisor, then show by descending induction that $r^k x =0$ for $k \leq n$, in particular $x=0$, contradiction.


Opcache not working properly

Question: I got problem with Opcache. It's not working as I wish/desire..
This is my opcache configuration
cat /etc/php/7.1/apache2/php.ini | grep opcache
[opcache]
opcache.enable = 1
opcache.enable_cli = 1
opcache.memory_consumption = 256
opcache.interned_strings_buffer = 16
opcache.max_accelerated_files = 8882
;opcache.max_wasted_percentage=5
;opcache.use_cwd=1
opcache.validate_timestamps = 0
opcache.revalidate_freq = 0
;opcache.revalidate_path=0
;opcache.save_comments=1
opcache.fast_shutdown = 1
;opcache.enable_file_override=0
;opcache.optimization_level=0xffffffff
;opcache.inherited_hack=1
;opcache.dups_fix=0
;opcache.blacklist_filename=
;opcache.max_file_size=0
;opcache.consistency_checks=0
;opcache.force_restart_timeout=180
;opcache.error_log=
;opcache.log_verbosity_level=1
;opcache.preferred_memory_model=
;opcache.protect_memory=0
;opcache.restrict_api=
;opcache.mmap_base=
;opcache.file_cache=
;opcache.file_cache_only=0
;opcache.file_cache_consistency_checks=1
; Implies opcache.file_cache_only=1 for a certain process that failed to
;opcache.file_cache_fallback=1
;opcache.huge_code_pages=1
;opcache.validate_permission=0
;opcache.validate_root=0

Since my setting opcache.validate_timestamps = 0 is set to false, as far as i know, modifying any php file shouldn't take any affect until i release it with php function opcache_reset();
So I made a little test and I have changed my index.php file (via terminal). I have inserted a method print_r(opcache_get_status());
This is the result:
array (
    [opcache_enabled] => 1
    [cache_full] =>
    [restart_pending] => 1
    [restart_in_progress] =>
    [memory_usage] => Array
        (
            [used_memory] => 37994008
            [free_memory] => 230441448
            [wasted_memory] => 0
            [current_wasted_percentage] => 0
        )

    [interned_strings_usage] => Array
        (
            [buffer_size] => 16777216
            [used_memory] => 504816
            [free_memory] => 16272400
            [number_of_strings] => 11672
        )

    [opcache_statistics] => Array
        (
            [num_cached_scripts] => 64
            [num_cached_keys] => 111
            [max_cached_keys] => 16229
            [hits] => 0
            [start_time] => 1513234045
            [last_restart_time] => 1513263246
            [oom_restarts] => 0
            [hash_restarts] => 0
            [manual_restarts] => 124
            [misses] => 358
            [blacklist_misses] => 0
            [blacklist_miss_ratio] => 0
            [opcache_hit_rate] => 0
        )

)

My problem is that results of print_r(opcache_get_status()); should not be visible, since my opcache.validate_timestamps is set to false and changes should take affect only when I reload opcache files, via opcache_reset(); method.
Am I right or am I missing something? If you need any additional informations, please let me know and I will provide. Thank you!
UPDATE
I found out that not all files are caching...
Running following command find . -type f -print | grep php | wc -l reports that there are 6167 php files, so why there are only 64 num_cached_scripts
Comment: I don't understand what you're asking. What exactly is wrong with the statistics?
Comment: there should be around 6000 cached files no only 64... not all php files are caching
Answer: I found out what the problem was. After going trough code i found out, that some one implemented opcache_reset(); after each request. After removing that line of code, cache is working.

Opcache not working properly