How to fit polynomial to a time series data in python

Question: I have a dataframe as:
The index of the dataframe is time object and one column named as 'Idle duration' as some numeric value.
14:09:00           1644
14:22:35            321
14:25:17            498
14:26:10            195
14:28:22            216
14:31:58            101
14:32:07            268
14:36:26            154
14:40:26            160
14:42:46           3085
14:45:21            172
14:50:29             71
14:51:14            594
14:52:03             79
14:55:32            130
14:55:39             69
14:57:29             80
14:58:15             68
14:59:57             78
15:02:15            112
15:03:09            191
15:04:10            537
15:04:26             85
15:04:45             65
15:05:20            223
15:07:28             95
15:13:26            117
15:13:39            176
15:15:22             73
15:15:30             70
...                 ...
16:08:29            181
16:09:47            137
16:10:28            345
16:12:10            138
16:14:34             65
16:15:00            104
16:15:41             65
16:16:40             91
16:16:43            415
16:17:36            302
16:18:12           2478
16:19:17            644
16:24:19            654
16:24:52            163
16:25:32            276
16:29:08             65
16:29:23             72
16:30:19             65
16:32:10             79
16:32:56             85
16:34:32             90
16:34:41           1261
16:34:52             65
16:38:13            277
16:40:06            155
16:43:11            110
16:50:57           1190
16:52:59            142
16:56:30            756
17:00:02            116

[102 rows x 1 columns]

I want to fit a best polynomial to this data to predict Idle time duration for next 30 minutes. I have used polyfit function to fit the polynomial. But I am getting this error:

TypeError: unsupported operand type(s) for +: 'datetime.time' and 'float'

Can anyone suggest what is wrong here. My code is:
Idletimes=pd.DataFrame(SampleOne, index=Times, columns=['Idle_duration'])
Idletimes.sort_index(inplace=True)
z = np.polyfit(Idletimes.index, Idletimes['Idle_duration'], 2)
print(z)

Comment: Convert your time stamps to something more sensible, such as seconds (from whatever your zero point is).
Answer: You cant apply np.polyfit on date object. Both x and y should be arrays of numbers.
For instance, see the following example from the documentation:
x = np.array([0.0, 1.0, 2.0, 3.0,  4.0,  5.0])
y = np.array([0.0, 0.8, 0.9, 0.1, -0.8, -1.0])
z = np.polyfit(x, y, 3)
z
array([ 0.08703704, -0.81349206,  1.69312169, -0.03968254])

So you need to translate your date column to some numeric values. As @Evert suggested, you could try to transform it to the number of seconds from your initial starting point.


Junção de tabelas

Question: Tenho três tabelas uma chamada  CRM_PROCESSO, outra chamada CRM_PROCESSO_VARIAVEL e a ultima chamada CRM_PROCESSO_ATIVIDADE, ambas possuem a variável idprocesso em comum.
O que preciso é que o comando traga o campo IDPROCESSO, USUARIO, DATAINICIO, DATATERMINO da tabela CRM_PROCESSO,o campo DESCRICAO da tabela CRM_PROCESSO_ATIVIDADEe o campo valoratualda tabela CRM_PROCESSO_VARIAVEL
Consegui fazer o comando, porém com os dados separados.
Rodei o comando abaixo, e trouxe o campo IDPROCESSO, USUARIO, DATAINICIO, DATATERMINO da tabela CRM_PROCESSO,o campo DESCRICAO da tabela CRM_PROCESSO_ATIVIDADE, quando ambas tem o IDPROCESSO igual.
    select crm_processo.idprocesso,
       crm_processo.usuario,
       crm_processo.datainicio,
       crm_processo.datatermino,
       crm_processo_atividade.descricao
   from crm_processo join crm_processo_atividade on (crm_processo_atividade.idprocesso = crm_processo.idprocesso and crm_processo_atividade.idatividade = crm_processo.idatividadeatual) where status = 1 and idprocedimento = 34 and idatividadeatual <>2

E consegui fazer o comando abaixo, trazendo o campo valoratualda tabela CRM_PROCESSO_VARIAVEL (Utilizei um exemplo com IDPROCESSO = 39)
SELECT  cpv.descricao,
cpv.valoratual,
(CASE CPV.DESCRICAO
     WHEN '/*MOTIVOCANCELAMENTO*/' THEN 'Motivo Cancelamento'
     END)  from crm_processo_variavel cpv  where idprocesso = 3089 AND CPV.DESCRICAO IN ('/*MOTIVOCANCELAMENTO*/')

Mas sinceramente, não sei como juntar os dois comando.
A relação processo x atividade é 1 para 1.
E processo x variável também 1 para 1.
Alguém consegue me auxiliar ?
Comment: A relação processo x atividade é `1 para 1`? E processo x variável também `1 para 1`? Ou é `1 para 0`, ou `1 para muitos`? É importante saber a multiplicidade das relações para você saber se você vai fazer um JOIN ou um OUTER JOIN para recuperar os registros em uma consulta única. Edite a pergunta e informe as multiplicidades.
Comment: Olá @Pagotti a relação processo x atividade é 1 para 1. E processo x variável também 1 para 1.
Comment: Tem algo estranho porque na sua query original tem `crm_processo_atividade.idatividade = crm_processo.idatividadeatual`. Se é `1 para 1` porque você tem esse `idatividadeatual`? Cada processo pode ter mais de 1 atividade?
Comment: O `crm_processo_variavel` também tem um filtro da descrição. Isso significa que cada `crm_processo_variavel` também é `1 para muitos`? E você está aplicando os filtros para que resulte apenas 1 registro de processo. É isso?
Comment: Uma ferramenta legal que você pode usar quando tiver questões desse tipo é o [DBFiddle](https://www.db-fiddle.com). Lá você pode colocar a estrutura e dados de exemplo das tabelas e depois compartilhar na pergunta para facilitar quem for te ajudar. Coloque nas tags também qual é o banco que você está usando.
Answer: Se a relação é 1 para 1 entre as três tabelas, então bastaria um JOIN normal entre elas, mas pelos comentários parece que a relação é 1 para N e os filtros nas tabelas forçam uma relação 1 para 1. Nesse caso você pode fazer o filtro no JOIN ou criando SUBQUERY com filtros internos.
A versão com filtro no JOIN é essa:
SELECT P.idprocesso,
       P.usuario,
       P.datainicio,
       P.datatermino,
       PA.descricao AS pa_descricao,
       PV.valoratual,
       CASE PV.descricao
         WHEN '/*MOTIVOCANCELAMENTO*/' THEN 'Motivo Cancelamento'
         END AS pv_descricao
FROM crm_processo AS P
LEFT JOIN crm_processo_atividade AS PA
     ON PA.idprocesso = P.idprocesso
     AND idatividade = P.idatividadeatual
     AND idatividade <> 2
LEFT JOIN crm_processo_variavel AS PV
     ON PV.idprocesso = P.idprocesso
     AND PV.descricao IN ('/*MOTIVOCANCELAMENTO*/')
WHERE
     P.status = 1
     AND P.idprocedimento = 34

A versão com SUBQUERY é essa:
SELECT P.idprocesso,
       P.usuario,
       P.datainicio,
       P.datatermino,
       PA.descricao AS pa_descricao,
       PV.descricao AS pv_descricao,
       PV.valoratual
FROM crm_processo P
LEFT JOIN (SELECT descricao FROM crm_processo_atividade
           WHERE idprocesso = P.idprocesso
           AND idatividade = P.idatividadeatual
           AND idatividade <> 2 ) AS PA
     ON PA.idprocesso = P.idprocesso
LEFT JOIN (SELECT valoratual,
             CASE descricao
             WHEN '/*MOTIVOCANCELAMENTO*/' THEN 'Motivo Cancelamento'
             END AS descricao
           FROM crm_processo_variavel
           WHERE idprocesso = P.idprocesso
           AND descricao IN ('/*MOTIVOCANCELAMENTO*/') ) AS PV
     ON PV.idprocesso = P.idprocesso
WHERE
     P.status = 1
     AND P.idprocedimento = 34

Comment: Deu certo @Pagotti. Obrigado. Vlw pelas dicas.

Junção de tabelas


Is My Understanding of Covariant Derivative Right?

Question: Let $E\overset{\pi}{\twoheadrightarrow}M$ be a fiber bundle, and $\sigma$ be a smooth local section, i.e. $\pi\circ\sigma=\mathrm{id}$. For $\forall X\in T_{x}M$, if $E$ is trivial, then one can talk about the directional derivative $d\sigma(x)(X)$, where $d\sigma(x)$ is a tangent map $T_{x}M\rightarrow T_{\sigma(x)}E$. This is because there's an obvious way to parallelly transport each element in $E$.
For example, let $\gamma(t)$ is a curve in $M$, such that $\gamma(0)=x$, and $\gamma^{\prime}(0)=X$, then, since $E\overset{\pi}{\twoheadrightarrow}M$ is trivial, one has $$d\sigma(x)(X)=\frac{d}{dt}\Bigg|_{t=0}\sigma(\gamma(t))=\lim_{t\rightarrow 0}\frac{\sigma(\gamma(t))-\sigma(\gamma(0))}{t},$$
where $\sigma(\gamma(t))\in E_{\gamma(t)}$. It makes sense because $E_{\gamma(t)}$ is isomorphic to $E_{\gamma(0)}$ in a canonical way.
In general, if $E$ is none trivial, then one has to use the covariant differential $\nabla\sigma(x)$ to define the tangent map. The covariant derivative is usually defined by introducing the parallel transport:

For each curve $\gamma(t)$ in manifold $M$, the collections of diffeomorphisms $$\Gamma(\gamma)_{s}^{t}: E_{\gamma(s)}\rightarrow E_{\gamma(t)}$$
such that
\begin{align}
&1.\,\,\,\Gamma(\gamma)_{s}^{s}=\mathrm{Id}_{E_{\gamma(s)}} \\
&2.\,\,\,\Gamma(\gamma)_{\epsilon}^{t}\circ\Gamma(\gamma)_{s}^{\epsilon}=\Gamma(\gamma)_{s}^{t} \\
&3.\,\,\,\Gamma(\gamma)_{s}^{t}\,\,\,\mathrm{depends\,\,on\,\,\gamma,\,\,s,\,\,\mathrm{and}\,\,t\,\,\mathrm{smoothly}.}
\end{align}

Then, for a given curve $\gamma(t)$ in M, such that $\gamma(t)=x$, and $\gamma^{\prime}(t)=X$, one defines the covariant derivative $$\nabla\sigma(x)(X)=\nabla\sigma(x)(\gamma^{\prime}(t))\equiv\nabla_{X}\sigma(x)\equiv\frac{d}{d\epsilon}\Bigg|_{\epsilon=0}\Gamma(\gamma)_{t+\epsilon}^{t}\circ\sigma(\gamma(t+\epsilon)).$$
What confused me a lot recently is that under the diffeomorphism $\Gamma(\gamma)_{t+\epsilon}^{t}: E_{\gamma(t+\epsilon)}\rightarrow E_{\gamma(t)}$, the section $\sigma(\gamma(t+\epsilon))\in E_{\gamma(t+\epsilon)}$ is mapped to another section $\varsigma_{\epsilon}(\gamma(t))\in E_{\gamma(t)}$. For convenience, I denote $$\frac{d}{d\epsilon}\Bigg|_{\epsilon=0}\varsigma_{\epsilon}(\gamma(t))=\xi(t)\in T_{\varsigma_{\epsilon}(\gamma(t))}E.$$
Then, from the canonical projection $E\overset{\pi}{\twoheadrightarrow}M$, which locally gives $$\pi\circ\sigma=\pi\circ\varsigma_{\epsilon}=\mathrm{id},$$
one has $$d\pi(\xi(t))=\frac{d}{d\epsilon}\Bigg|_{\epsilon=0}\pi(\varsigma_{\epsilon}(\gamma(t)))=\frac{d}{d\epsilon}\Bigg|_{\epsilon=0}\gamma(t)=0.$$
In other words, it seems to me that the push forward $d\pi(\nabla\sigma(x)(X))=0$.
Or, if one views the covariant differential $\nabla\sigma(x)$ as a tangent map $T_{x}M\rightarrow T_{\sigma(x)}E$, then the above calculation really showed that the covariant derivative actually maps the tangent vector $X\in T_{x}M$ to the vertical subspace $V_{\sigma(x)}E$. i.e. $$\nabla\sigma(x): T_{x}M\rightarrow V_{\sigma(x)}E.$$
Is that correct?
Comment: If you differentiate a function of the form $F(x) = (x,f(x))$, $f\colon \Bbb R\to \Bbb R^m$, is the derivative an element of $\Bbb R^m$?
Comment: @TedShifrin I guess the answer is yes. Am I right?
Comment: No. Draw some pictures (even from beginning calculus).
Comment: @TedShifrin $dF/dx=(1,df/dx)$. If $f:\mathbb{R}\rightarrow\mathbb{R}^{m}$, then this seems a $m$-dimensional vector.
Comment: What about the $(1,$?
Answer: I found the same claim from several sources. For example, in this lecture notes, Chris Wendl also claimed that the covariant derivative is the vertical part of the tangent map. So I believe that my understanding of the covariant derivative is correct.
In the following, I will prove the result for a vector bundle from a different perspective. From this Wikipedia page, given a connection $\nabla$ on a vector bundle $E$ over $M$, and $\gamma(t)$ a smooth curve in $M$, a section $\sigma$ of $E$ is called parallel if $$\nabla_{\dot{\gamma}(t)}\sigma=0. \tag{0}$$
This is supposed to be equivalent to the definition of $\Gamma(\gamma)_{s}^{t}$ of parallel transport in a vector bundle. I will use this equivalence to prove that for a generic section, its covariant derivative is indeed vertical.
Starting from equation (0), locally, in an neighborhood $U\subset M$ with local coordinates $\left\{q^{\mu}\right\}$, the parallel section $\sigma$ can be expressed as $\sigma|_{U}=\sigma^{i}e_{i}$, where $\left\{e_{i}\right\}_{i=1,\cdots,m}$ is a local frame. For convenience, I will denote $\sigma^{i}(\gamma(t))\equiv\sigma(t)^{i}$, and $q^{\mu}(\gamma(t))=q^{\mu}(t)$. Denoting the connection $1$-form of the vector bundle $E$ by $A_{i}^{j}=A_{i\mu}^{j}dq^{\mu}$. Then, one has $$\nabla e_{i}=A_{i}^{j}e_{j}.$$
Using the Leibniz rule, one has $$\nabla_{\dot{\gamma}(t)}\sigma=\nabla_{\dot{\gamma}(t)}(\sigma^{i}e_{i})=\left(\frac{d\sigma^{i}}{dt}+\sigma^{j}\frac{dq^{\mu}}{dt}A_{j\mu}^{i}\right)e_{i}. \tag{1}$$
So the condition that the section $\sigma$ is parallel implies that the differential equation $$\frac{d\sigma^{i}}{dt}+\sigma^{j}\frac{dq^{\mu}}{dt}A_{j\mu}^{i}=0 \tag{2}$$
has a solution, which is certainly true according to the theory of ordinary differential equations.

Lemma: Let $\nabla$ be a connection on the vector bundle $E\overset{\pi}{\twoheadrightarrow}M$, with the vector space $\mathbb{V}$ its typical fiber. Let $x\in M$, and $\sigma\in\pi^{-1}(x)$. Then, the horizontal subspace $H_{\sigma}E$ is isomorphic to $T_{x}M$ under the push forward $d\pi_{\sigma}$.
proof: Let $\left\{q^{\mu}\right\}$ be the local coordinates of $x\in U\subset M$. The vector bundle $E|_{U}$ has local trivialization $\varphi: U\times\mathbb{V}\rightarrow\pi^{-1}(U)$. Denote $\left\{f_{i}\right\}_{i=1,\cdots,m}$ as the basis of the vector space $\mathbb{V}$, and $e_{i}\equiv\varphi(x,f_{i})$. Accordingly, $\left\{\frac{\partial}{\partial q^{\mu}}\right\}$ is a local frame on $TM|_{U}$, and $\left\{e_{i}\right\}$ is a local frame of the vector bundle $E|_{U}$. Then, any local section $\sigma|_{U}$ can be written as $\sigma=\sigma^{i}e_{i}$. For any point $\sigma\in\pi^{-1}(x)$, there's a local coordinate $\left\{\tilde{q}^{\mu},p^{i}\right\}$ of $\sigma\in\pi^{-1}(U)$ such that \begin{equation}
\begin{cases}
\tilde{q}^{\mu}(\sigma):=q^{\mu}(x) \\
p^{i}(\sigma):=\sigma^{i}(x).
\end{cases}
\end{equation}
Denote the frame of $TE|_{U}$ by $\left\{X_{\mu},Y_{i}\right\}$, i.e. $$X_{\mu}=\frac{\partial}{\partial\tilde{q}^{\mu}},\quad Y_{i}=\frac{\partial}{\partial p^{i}}.$$
Let $\gamma(t)$ be an arbitrary smooth curve in $M$, such that $\gamma(0)=x$. Then, one can consider its horizontal lift $\tilde{\gamma}(t)$ in $E$ (i.e. it is parallel along $\gamma(t)$), such that $\tilde{\gamma}(0)=\sigma$. Using the above coordinate, one can express the parallel curve $\tilde{\gamma}(t)$ as $$\left\{\tilde{q}^{\mu}(t),p^{i}(t)\right\},$$
where $\tilde{q}^{\mu}(t)\equiv q^{\mu}(\gamma(t))=q^{\mu}(t)$, and $p^{i}(t)\equiv\sigma^{i}(\gamma(t))=\sigma^{i}(t)$. Then, by definition, it has to satisfy the equation $$\frac{dp^{i}}{dt}+p^{j}\frac{d\tilde{q}^{\mu}}{dt}A_{j\mu}^{i}=0.$$
Then, the tangent vector $\tilde{\gamma}^{\prime}(0)\in T_{\sigma}E$ is \begin{align}
\tilde{\gamma}^{\prime}(0)&=\frac{d\tilde{q}^{\mu}}{dt}\frac{\partial}{\partial\tilde{q}^{\mu}}\Bigg|_{\sigma}+\frac{dp^{i}}{dt}\frac{\partial}{\partial p^{i}}\Bigg|_{\sigma} \\
&=\frac{d\tilde{q}^{\mu}}{dt}\left(X_{\mu}(\sigma)-p^{j}(\sigma)A^{i}_{j\mu}(x)Y_{i}(\sigma)\right). \tag{3}
\end{align}
Comparing the above vector with $$\gamma^{\prime}(0)=\frac{dq^{\mu}}{dt}\frac{\partial}{\partial q^{\mu}}\Bigg|_{x}\in T_{x}M,$$
one finds that the horizontal lift of the tangent vector $\gamma^{\prime}(0)$ is given by (3). Under the map $$\frac{\partial}{\partial q^{\mu}}\Bigg|_{x}\longleftrightarrow Z_{\mu}(\sigma)\equiv X_{\mu}(\sigma)-p^{j}(\sigma)A^{i}_{j\mu}(x)Y_{i}(\sigma),$$
there is an isomorphism between $T_{x}M$ and $H_{\sigma}E$. Under the projection $\pi_{\ast}|_{\sigma}$, $Z_{\mu}(\sigma)$ is mapped to $\frac{\partial}{\partial q^{\mu}}\Bigg|_{x}$.

The above statement implies that the tangent space $T_{\sigma}E$ has a decomposition

$T_{\sigma}E=H_{\sigma}E\oplus V_{\sigma}E,$
$d\pi_{\sigma}:H_{\sigma}\rightarrow T_{\pi(\sigma)}M$ is an isomorphism.

Then, finally, one has the following theorem:

Theorem: Let $E\overset{\pi}{\twoheadrightarrow}M$ be a vector bundle, with connection $\nabla$ and the connection $1$-form $A^{i}_{j}$. Let $\gamma:(-\epsilon,\epsilon)\rightarrow M$ be a smooth curve, and $\sigma\in\Gamma(E)$ be any section of $E$. Then, $$\nabla_{\gamma^{\prime}(0)}\sigma=\Pi_{\mathrm{v}}(\sigma^{\prime}(0)),$$
where $\Pi_{\mathrm{v}}$ is the projection on the vertical subspace under the decomposition $T_{\sigma}E=H_{\sigma}E\oplus V_{\sigma}E$.
proof: The section $\sigma\in\Gamma(E)$ along the curve $\gamma(t)\subset M$ is also a curve in the total space $E$. Denote this curve $\sigma|_{\gamma(t)}$ by $\tilde{\gamma}(t)$, then using local coordinates, one has $$\tilde{\gamma}^{\prime}(t)=\frac{d\tilde{q}^{\mu}}{dt}\frac{\partial}{\partial\tilde{q}^{\mu}}\Bigg|_{\sigma(t)}+\frac{dp^{i}}{dt}\frac{\partial}{\partial p^{i}}\Bigg|_{\sigma(t)},$$
where $\sigma(t)\equiv\sigma(\gamma(t))$. Then, using the decomposition $T_{\sigma}E=H_{\sigma}E\oplus V_{\sigma}E$, one has $$\tilde{\gamma}^{\prime}(t)=\frac{d\tilde{q}^{\mu}}{dt}\left(\frac{\partial}{\partial\tilde{q}^{\mu}}\Bigg|_{\sigma(t)}-p^{j}(t)A^{i}_{j\mu}(\gamma(t))\frac{\partial}{\partial p^{i}}\Bigg|_{\sigma(t)}\right)+\left(\frac{dp^{i}}{dt}+p^{j}(t)\frac{d\tilde{q}^{\mu}}{dt}A^{i}_{j\mu}(\gamma(t))\right)\frac{\partial}{\partial p^{i}}\Bigg|_{\sigma(t)}.$$
Then, one has $$\Pi_{\mathrm{v}}(\tilde{\gamma}^{\prime}(t))=\left(\frac{dp^{i}}{dt}+p^{j}(t)\frac{d\tilde{q}^{\mu}}{dt}A^{i}_{j\mu}(\gamma(t))\right)\frac{\partial}{\partial p^{i}}\Bigg|_{\sigma(t)}.$$
Then, using equation (1), one has $$\Pi_{\mathrm{v}}(\tilde{\gamma}^{\prime}(t))=\nabla_{\gamma^{\prime}(t)}\sigma(\gamma(t)), \tag{4}$$
under the natural identification $$\frac{\partial}{\partial p^{i}}\longleftrightarrow e_{i},\quad\mathrm{or}\quad T_{\sigma(t)}(\pi^{-1}(\gamma(t)))\simeq V_{\sigma(t)}E.$$

The above theorem implies that the covariant differential as a tangent map is indeed vertical on a vector bundle. It acts on a section by taking ordinary differential and then eating up the horizontal component. Unsurprisingly, if the section is parallel (i.e. horizontal), its covariant derivative vanishes.


the order of a group of matrices over $\mathbb{Z}/2\mathbb{Z}$

Question: Let $G$ be the group of 3 by 3 matrices over $\mathbb{Z}/2\mathbb{Z}$ of determinant 1. Determine the order of $G$.
So does this mean all the elements of matrix $A$ in $G$ are 0 or 1(and the determinant is 1)? and we need to find the order of this group?
So the operation should be multiplication, right?
This is not a large set. But is there a way to find the order instead of check it one by one?
Thank you very much!
Comment: Hint: Apply the first isomorphism theorem.
Answer: It means that not only are the elements of the matrix 0 or 1, but that when multiplying matrices (yes multiplication is the group operation), you treat the entries of the matrix as lying in $\mathbb{Z}/2\mathbb{Z}$. So $1+1 = 0$...etc.
Here's a way you can find the order.
Split the matrix into its three columns. You know that a matrix has nonzero determinant if and only if its columns are linearly independent. Thus, the first column can be anything except the zero vector. Thus, you have $2^3 - 1 = 7$ choices for the first column.
The second column can be anything except a multiple of the first column, so you have $2^3 - 2$ choices (there are only two possible multiples of the first column!) The third column can be anything except a linear combination of the first two columns. I'll let you figure out how many choices there are for the third column.
Lastly note that if a matrix over $\mathbb{Z}/2\mathbb{Z}$ has nonzero determinant, it must have determinant 1.
Answer: Did you know that $\mathbb Z/2\mathbb Z$ is field which is normally denoted $\mathbb F_2$?  And did you know that everything you learned about matrices in linear algebra is valid so long as the entries of those matrices come from a field?
In particular a matrix has nonzero determinant if and only if it's columns are linearly independent.  Well the only nonzero element in $\mathbb F_2$ is $1$ so you're just trying to count how many sets of $3$ linearly independent vectors there are.
Well you have $2^3 - 1$ choices for the first vector cause you can choose anything that's nonzero.  For the second vector you can choose anything that's not a scalar multiple of the first vector.  Well the only scalar multiples of the first vector are $0$ and the vector so that's $2^3 - 2$ choices.  I'll let you figure out how many choices you have for the third vector (don't get fooled by the obvious pattern, $2^3 - 3$ isn't the right answer!).
Answer: Define
$$H:=\left\{\;A\in M_3(\Bbb F_2)\;:\;\;\det A\neq 0\;\right\}$$
or in other words: $\;H\;$ is the group of all invertible $\;3\times3\;$ matrices over the field $\;\Bbb F_2\;$ with two elements.
(1) Show that $\;|H|=(2^3-1)(2^3-2)(2^3-2^2)=168\;$
(2) Define
$$\phi: H\to\Bbb F_2^*\;,\;\;\phi(A):=\det A$$
Show that $\;\phi\;$ is a surjective group homomorphism
(3) Prove that $\;G=\ker\phi\;$
(4) Use the first isomorphism theorem now to deduce $\;|G|\;$
If you need a little more help read the next spoiler after you try to work out the above.

spoiler Can you see that $\;G=H\;$?


Is standard gradient descent possible when we have constrained parameters?

Question: Is it true that standard gradient descent algorithm (be it batch or mini batch or stochastic) cannot be used when we have certain constraint on parameters? If yes, why is it so? Is it because gradient descent just determines gradient of the cost function with respect to the parameter, changes parameter value based on it, and hence has no mechanism to handle the constraints?
Could anyone please help me understand this concept?
Answer: Yes, you are right, this is indeed the problem.
What can be done? You could either go for an optimization method other than gradient descent, or you try to find an equivalent problem where your search space is again an open set so that you can still use gradient descent.
E.g., if your objective function is $g$ and the parameters are $x$ and your constraint is of the form:
$$
f(x) = 0,
$$
you could try to fix this by parameterizing the submanifold $M$ given by $f(x)=0$, consider a chart $\xi:U\to M$ from an open subset $U\subset \mathbb{R}^d$ to $M$ and then apply gradient descent to $g\circ \xi$.
Of course, there are many different possible constraints and many other approaches to dealing with them.
Comment: Thank you for the clarification!


Let V be an n-dimensional complex vector space and let $T\in \mbox{End}(V)$ satisfy $E^2$ = E

Question: Let V be an n-dimensional complex vector space and let $E\in \mbox{End}(V)$  satisfy $E^2 = E$. Show that there is an $r \geq 0$ and a basis$(v_1, v_2, ..., v_n)$ for $V$ such that
$E(v_i) = \begin{cases}
v_i & \text{if } 1 \leq i \leq r \\
0 & \text{if } r + 1 \leq i \leq n
\end{cases}$
Comment: Be careful with your writing: is $\;T=E\;$ ?
Comment: @Joanpemo Thanks, I didn't notice.
Answer: Hint:
Show $V=\ker T\oplus\operatorname{Im}T$.
Note this is true for a vector space over any field, not merely $\mathbf C$or $\mathbf R$ .
Comment: I have proven your hint. Is this all I need to do or is there more?
Comment: Only some words to conclude, if you didn't do it. $E$ is the projection onto the second factor.


Spark Clusters: worker info doesn't show on web UI

Question: I have installed spark standalone on a set of clusters. And I tried to launch clusters through the cluster launch script. I have added cluster's IP address into conf/slaves file. The master connects to all slaves through password-less ssh.
After running ./bin/start-slaves.sh script, I get the following message:

starting org.apache.spark.deploy.worker.Worker, logging to /root/spark-0.8.0-incubating/bin/../logs/spark-root-org.apache.spark.deploy.worker.Worker-1-jbosstest2.out

But the webUI of the master (localhost:8080) is not showing any information about the worker. But when I add localhost entry onto my conf/slaves file the worker info of localhost is shown.
There are no error messages, the message on terminal says the worker is started, but the WebUI is not showing any workers.
Comment: Look at the worker's logs, it will tell you why it cannot connect to the master.
Answer: I had the same problem.  I noticed when I could not telnet master:port from the slaves.  In my etc/hosts file (on master) I had a 127.0.0.1 master entry (before my 192.168.0.x master).  When I removed the 127.0.0.1 entry from my etc/hosts file I could telnet and when I start-slaves.sh (from the master) my slaves connected
Comment: Your answer was a clue to my problem. The port in which the slaves connect to the master (7077) was blocked by a firewall.
Answer: When you run the cluster, check command $jps in worker nodes, whether its up correctly and check it in the logs with the worker's PID.
or
set the following: run the cluster and check if the ports are up or not with your configured ports
export SPARK_MASTER_WEBUI_PORT=5050
export SPARK_WORKER_WEBUI_PORT=4040

Answer: check your /etc/hosts and see the bindings for master
If your master is binding to localhost as well as ip address (eg 192.168.x.x), remove localhost. if you have local host intact master will be mapped to localhost which wont allow slaves to connect to master Ip address
Answer: You can use: ./start-master.sh --host 192.168.x.x instead of changing the file: /etc/hosts
Answer: I met the same issue and finally solved by adding the following line in $SPARK_HOME/conf/spark-env.sh:
SPARK_MASTER_HOST=your_master_ip_address


Adding Conv Layer in front of pretrained model gives ValueError

Question: I want to combine a pretrained VGG16 model with a special input block, which is an input layer and a convolutional layer. The goal is to use a pre-trained RGB VGG16 imagenet model on grayscale images:
from keras.applications.vgg16 import VGG16
from keras.layers.convolutional import Conv2D
from keras.layers import Input
from keras.models import Model

img_height = 299
img_width = 299

def input_block(img_height = 299, img_width = 299):
    input_shape = (img_height, img_width, 1)
    img_input = Input(shape=input_shape, name = 'grayscale_input_layer')
    x = Conv2D(3, (3,3),  padding= 'same', name = 'grayscale_RGB_layer')(img_input)
    return x

pretrained_model = VGG16(weights = 'imagenet', include_top=False, input_tensor = input_block(img_height, img_width))

When I set the weight initalization of VGG16() to 'None', the model builds correctly, with the following desired structure:
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
grayscale_input_layer (Input (None, 299, 299, 1)       0
_________________________________________________________________
grayscale_RGB_layer (Conv2D) (None, 299, 299, 3)       30
_________________________________________________________________
block1_conv1 (Conv2D)        (None, 299, 299, 64)      1792
_________________________________________________________________
block1_conv2 (Conv2D)        (None, 299, 299, 64)      36928
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, 149, 149, 64)      0
_________________________________________________________________
block2_conv1 (Conv2D)        (None, 149, 149, 128)     73856
_________________________________________________________________
block2_conv2 (Conv2D)        (None, 149, 149, 128)     147584
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, 74, 74, 128)       0
_________________________________________________________________
block3_conv1 (Conv2D)        (None, 74, 74, 256)       295168
_________________________________________________________________
block3_conv2 (Conv2D)        (None, 74, 74, 256)       590080
_________________________________________________________________
block3_conv3 (Conv2D)        (None, 74, 74, 256)       590080
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, 37, 37, 256)       0
_________________________________________________________________
block4_conv1 (Conv2D)        (None, 37, 37, 512)       1180160
_________________________________________________________________
block4_conv2 (Conv2D)        (None, 37, 37, 512)       2359808
_________________________________________________________________
block4_conv3 (Conv2D)        (None, 37, 37, 512)       2359808
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, 18, 18, 512)       0
_________________________________________________________________
block5_conv1 (Conv2D)        (None, 18, 18, 512)       2359808
_________________________________________________________________
block5_conv2 (Conv2D)        (None, 18, 18, 512)       2359808
_________________________________________________________________
block5_conv3 (Conv2D)        (None, 18, 18, 512)       2359808
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, 9, 9, 512)         0
=================================================================
Total params: 14,714,718
Trainable params: 14,714,718
Non-trainable params: 0
_________________________________________________________________
None

However, when I set the weight initialization to 'imagenet',
I get the following error:

ValueError: You are trying to load a weight file containing 13 layers into a model with 14 layers.

This error makes sense, since I have added two layers in front of the VGG16 model instead of a single layer.
As a workaround, I have tried the following:
def input_block_model(img_height = 299, img_width = 299):
    input_shape = (img_height, img_width, 1)
    img_input = Input(shape=input_shape, name = 'grayscale_input_layer')
    x = Conv2D(3, (3,3),  padding= 'same', name = 'grayscale_RGB_layer')(img_input)
    model = Model(img_input, x, name='input_block_model')
    return model

input_model = input_block_model(299,299)
pretrained_model = VGG16(weights = "imagenet", include_top=False)
combined_model = Model(input_model.input,
pretrained_model(input_model.output))
print(combined_model.summary())

Then, the model structure is:
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
grayscale_input_layer (Input (None, 299, 299, 1)       0
_________________________________________________________________
grayscale_RGB_layer (Conv2D) (None, 299, 299, 3)       30
_________________________________________________________________
vgg16 (Model)                multiple                  14714688
=================================================================
Total params: 14,714,718
Trainable params: 14,714,718
Non-trainable params: 0
_________________________________________________________________
None

The disadvantage of this structure, is that I cannot set properties of layers within the VGG16 model. I want to freeze certain layers for example in this model, which I cannot access via combined_model.layers. Does anyone have a working solution, such that I get the model structure as with the 'None' initialization, but with pretrained ImageNet weights?
Comment: Of course you can access the layers of VGG16 model using `combined_model.layers[2].layers`.
Answer: You can freeze or train layers using combined_model.layers[2].layers as mentioned in the comment above. You can may be simplify the model as follows:
```
<code>img_input = Input(shape=(img_height, img_width, 1), name = 'grayscale_input_layer')
x = Conv2D(3, (3,3),  padding= 'same', name = 'grayscale_RGB_layer')(img_input)
x = VGG16(weights = None, include_top=False)(x)
model = Model(img_input, x)
model.summary()

for layer in model.layers[2].layers:
    layer.trainable = False
</code>
```
Comment: can you say please why do we use (img_input) in front of our Conv2D? How is python syntax?


Capturing event of Windows LAN proxy settings change?

Question: Is there a way to capture event for a LAN Proxy settings change in C# program/service? I would like to capture event when the user enables or disables the LAN proxy server or changes the address or the port of the proxy server.
I already tested with NetworkAddressChangedEventHandler but it doesn't seem to capture this event.
Comment: Related: https://stackoverflow.com/questions/844467/c-sharp-auto-detect-proxy-settings
Answer: It is possible to listen events from Windows Registry (including change of Registry value) but apparently the Registry value of Software\Microsoft\Windows\CurrentVersion\Internet Settings is not being changed when the user changes the LAN proxy settings.


Como armazenar imagens dentro da aplicação?

Question: Estou começando agora em Java e preciso terminar uma aplicação que ao clicar em um botão mostre uma imagem, eu até consegui fazer isso usando o método
Desktop.getDesktop().open(new File()
 mas eu precisava que esse programa funcionasse em qualquer computador, teria como colocar a imagem dentro da própria aplicação ou algo assim?
Obrigado
Comment: Se o usuário da sua aplicação tem a opção de cadastrar imagens, salve o caminho onde ela está, e depois quando for visualizar, busque pelo caminho o qual salvou.
Comment: Poderia mostrar o que vc já fez para capturar e/ou pegar o caminho da imagem? Assim fica mais fácil de podermos ajudar! Saberemos qual caminho tomar! Obrigago
Comment: Estou fazendo assim http://imgur.com/Lu2RFPP
Answer: Pode colocar a imagem numa pasta e depois ao gerar o jar gera com a pasta tambem
Comment: mas ai eu coloco qual caminho no código? o caminho onde a pasta está no meu computador?


Markings and symbols in Paganini violin compositions

Question: In a lot of Paganini's violin compositions, there are bolded but sometimes not bolded capital letters like A, B, and C. I look at the note that it is above and it doesn't click for me. In Nel cor piu non mi sento (1st variation), there is a marking, "A1" below the staff and under the note g-sharp, which I really don't understand. Somewhere along the ways of many compositions there is a Roman numeral with a lowercase letter, sometimes with a dot underneath that letter, below the staff. I know that the numeral means the string but what is that letter next to it for? These are really advanced markings I'd assume and I couldn't find what they are in music dictionaries and such, so I'd like to know what all these symbols mean.

Excerpt from score published by B. Schott & Söhne, 1829

Excerpt from score published by Schott, 1909

Excerpt from score published by Editio Musica Budapest, 1968

Excerpt from score of 24 Caprices, op.1, publisher unknown
Comment: Can we get pictures of the letter markings, please? Based on your description, I doubt they're rehearsal markings, unlike one answer below.
Comment: Hi, welcome, etc. -- in cases like this, it's a good idea to check out alternate editions of the piece to see what's notated there.  Of course, you should post both an image and the name of the edition you are referring to.
Comment: I added a few images I found online of what i think you're describing; feel feel to remove or replace them if they don't show the correct markings.
Answer: Comparing editions, we can see that one of them marks so-called "artificial harmonics" with the open-diamond mark.  The other edition has "A3" there which presumably indicates the same pitch to be played (A three octaves up), and indirectly this requires artificial harmonic compared with the written note there.
I don't recognize "Vi-"  (and there's no fifth string :-) ), but perhaps it's some shorthand for the same bowing technique indicated with "saltato," i.e.

on the violin, a springing, jumping or bouncing bow, sauté (French),
  gesprungen (German): however, see also saltellato

With thanks as always to Dolmetsch
All the  "IIa," IVa"  are string-choice indicators.
Answer: They're probably not rehearsal letters because those are helpful only when there's more than one performer, while this is a solo.
In the half dozen editions and arrangements of Nel cor piu non mi sento that I found online, some modern, some ancient, none included such symbols.  So these symbols are unlikely to be musically important.  They may be peculiar to that publisher.
Edit: They mark which string to play.  Here's an example.
Comment: I agree that all the   IIa, IVa, markings are string choice, but I find it a bit unlikely that one edition would switch between "Ia"  and "A3" notation.
Answer: Sounds like they could be rehearsal letters.
Thought I'd explain it myself, but the cited article is already pretty clear.

A rehearsal letter is a boldface letter of the alphabet in an orchestral score, and its corresponding parts, that provides the conductor, who typically leads rehearsals, with a convenient spot to begin at places other than the start of movements or pieces. Rehearsal letters are most often used in scores of the Romantic era and onwards, beginning with Louis Spohr.

Hope this helps!
Comment: This answer was posted before the edit. Seeing the images of the score, it's clear this isn't correct in this case.


Get the form to add a term to a vocabulary programmatically

Question: I am working on Drupal 8.9.6.
I am trying to retrieve the form to add a term to a vocabulary programmatically. I know how to do it for a content type:
<code>    $values = array('type' => $content_type);

    $node = \Drupal::entityTypeManager()
      ->getStorage('node')
      ->create($values);

    $form = \Drupal::entityTypeManager()
      ->getFormObject('node', 'default')
      ->setEntity($node);
    $modal_form =  \Drupal::formBuilder()->getForm($form);
</code>
Nevertheless, I am not able to do the same for the form of adding a term to a vocabulary. Any hint would be more than welcome.
Thanks!
Answer: <code>$vocab_machine_name = 'tags';
$values = array('vid' => $vocab_machine_name);

$term = \Drupal::entityTypeManager()
  ->getStorage('taxonomy_term')
  ->create($values);

$form = \Drupal::entityTypeManager()
  ->getFormObject('taxonomy_term', 'default')
  ->setEntity($term);

$term_form =  \Drupal::formBuilder()->getForm($form);
</code>

Shorter way
<code>$vocab_machine_name = 'tags';

$term = \Drupal\taxonomy\Entity\Term::create([
  'vid' => $vocab_machine_name,
]);

$term_form = \Drupal::service('entity.form_builder')->getForm($term, 'default');
</code>
Comment: It works perfectly. Thanks!


$\prod\limits_{k = 0}^\infty {(1 - {p_k})} = 0$ if and only if $\sum\limits_{k = 0}^\infty {{p_k}} $ diverges

Question: Let $p_0,p_1,p_2,...$ be real numbers in $(0,1)$. I am trying to prove that
$\prod\limits_{k = 0}^\infty  {(1 - {p_k})}  = 0$ if and only if the series
$\sum\limits_{k = 0}^\infty  {{p_k}} $ diverges.
By taking logarithms in the product we have
$\log \left( {\prod\limits_{k = 0}^\infty  {(1 - {p_k})} } \right) = \sum\limits_{k = 0}^\infty  {\log \left( {1 - {p_k}} \right)} $
and the latter series should diverge to $-\infty$ if the product were to be $0$.
Equivalently, $\sum\limits_{k = 0}^\infty  {\left( { - \log \left( {1 - {p_k}} \right)} \right)} $ should diverge to $+\infty$ and the conclusion may follow by approximating $-\log(1-p_k)$ by $p_k$ near $0$.
Any help on how to write this last step formally? How can we prove that $\sum\limits_{k = 0}^\infty  {\left( { - \log \left( {1 - {p_k}} \right)} \right)} = +\infty$ if and only if
$\sum\limits_{k = 0}^\infty  {{p_k}} = +\infty$ ?
Answer: You can try this: if $p_k \not \to 0$ then clearly both $\sum u_k = \infty$ and $\prod_k(1-p_k)=0$. Assume now $p_k \to 0$. Then we have
$$\log\frac{1}{1-p_k} \simeq p_k$$
so the series
$\sum \log\frac{1}{1-p_k}$ and $\sum p_k$ are simultaneously convergent.
Obs: You can also throw in the product $\prod_k (1+p_k)$
Comment: You have $\frac{1}{1-p_k}>1+p_k$ so $\prod_k \frac{1}{1-p_k} > 1 + \sum p_k$ by Bernoulli. It's the other way that needs some limits.
Answer: As $1-x_i>0$            it is obvious that $$\lim_{n \to \infty}\prod_{n \ge i \ge 1} (1-x_i)=0$$ is equivalent to $$\lim_{n \to \infty}\prod_{n \ge i \ge 1} \frac{1}{(1-x_i)} = \infty.$$ At first let us transform the latter product into something that has the form of the first (namely $\prod (1+a_i)$) by writing $\frac{1}{1-x_i} = 1+ \frac{x_i}{1-x_i}$. Let $y_i=\frac{x_i}{1-x_i}$. Now all we need to do is show that $\prod(1+y_i) $ and $\sum x_i$ are of the same nature. The result follows directly from the two following facts :
Lemma 1.
For any positive sequence $(a_i)_{i>0}$, $$S_n = \sum_{n \ge i \ge 1} a_i$$ and $$P_n= \prod_{n \ge i \ge 1} (1+a_i)$$are both of the same nature (they either both converge or both diverge).
Lemma 2.
For $0<x_i<1$ $$A_n= \sum_{n \ge i \ge 1} x_i $$ and $$B_n=\sum_{n \ge i \ge 1} \frac{x_i}{1-x_i}$$ are of the same nature.

Now just set $a_n = x_n$ and note that $B_n \sim A_n=S_n \sim P_n$ where $ \sim$ means "is of the same nature as" which is an equivalence relation.

Proof of Lemma 1. By the inequality $\exp(x) \ge x+1$ and induction (or alternatively by Newton's inequalities) we can see that $$\exp(\sum_{n \ge i \ge 1} a_i) \ge \prod_{n \ge i \ge 1} (1+a_i) $$
On the other hand, by Bernoulli's inequality we have $$\prod_{n \ge i \ge 1} (1+a_i) \ge 1+ \sum_{n \ge i \ge 1}a_i$$ hence $S_n$ and $P_n$ are really of the same nature.
Proof of Lemma 2.
Note that we are dealing with positive numbers here and $0<1-x_i \to 1$. Therefore there exists an integer $n_0$ such that for all $i>n_0$ we have $1>(1-x_i)> 0,5$  and therefore $x_i<\frac{x_i}{1-x_i}< 2 x_i$. From here we get
$$\sum_{n \ge i >n_0} x_i \le \sum_{n \ge i >n_0}\frac{x_i}{1-x_i} \le 2\sum_{n \ge i >n_0} x_i$$
which implies the result.
Comment: Sooo complicated... :-) For a shorter approach, see http://math.stackexchange.com/q/950653/.


Example of a function whose directional derivatives are always positive

Question: I need an example of a function  $f: \Bbb R^n \rightarrow  \Bbb R$ such that it`s directional derivative at the direction of the vector $y$ is such that $\mathbf{D_y}(a)>0$ for a fixed vector $y$ and every $a$ in $\Bbb R^n$
I'm having my doubts with this one since I just proved that there no $f: \Bbb R^n \rightarrow  \Bbb R$ such that $\mathbf{D_y}(a)>0$ for every  $y$ at a fixed point $a$ in $\Bbb R^n$ by using the property that states that $\mathbf{D_y}(a)=y \nabla f(a)$.
Answer: Since $y=(y_1,\ldots,y_n)$ is fixed, if you have $\nabla f(a)=(y_1,\ldots,y_n)$ you will have $y\cdot \nabla f(a)=y_1^2+\cdots+y_n^2>0$ (of course, if $y\ne 0$).
This can be easily achieved by taking $f(x_1,\ldots,x_n)=y_1x_1+\cdots+y_nx_n$. In shorter notation, you can define $f(x)=y\cdot x$.
Comment: Why did you change your answer to this one? It seems to me that your previous solution (which coincidentally I have come up with, too) was correct and more generic.
Comment: Perhaps you noticed that Weierstraß Ramirez edited his question long ago by changing "every fixed vector y" to "a fixed vector y"? I think the value of this question to others is higher if we keep the generic version for arbitrary y which he even flagged as the right answer.
Comment: Yes you are right. I'll roll back.
Answer: My understanding of your question is as following:
For each $y \in \mathbb{R}^n$ find a function $f : \mathbb{R}^n \rightarrow \mathbb{R}$ such that the directional derivative $\partial_yf = \mathbf{D}_yf$ is positive.
Please note that for $\|y\|=0$ the directional derivative is always $0$, so I'll exclude that case.

I suggest the following approach:
For an arbitrary $y \in \mathbb{R}^n \setminus \lbrace 0 \rbrace$ let $f: \mathbb{R}^n \rightarrow \mathbb{R}$ be given by
$$
    f(x) = \sum_{i=1}^n y_i x_i
$$
Then we have for $i \in \lbrace 1, \ldots, n \rbrace$ and $a \in \mathbb{R}^n$
$$
    \frac{\partial f}{\partial x_i}(a) = y_i
$$
which means that it is $\nabla f(a) = y$.
This immediately shows that the directional derivative of $f$ in the direction of $y$ is positive:
$$
    \mathbf{D_y}f(a) = y \cdot \nabla f(a) = y \cdot y = \|y\|^2 > 0
$$
Comment: After answering, I noticed that the question is quite old already... It must have been bumped up because Martin just changed his answer's content radically. Actually it seems his previous, old solution was the same as the one I have just presented.


Problem using Java's Math functions to come up with a page number for view pagination

Question: If I have 2 database records and 25 records per page, then the following code:
System.out.println("page count: " + (double)2/25);

results in this output:
page count: 0.08

But because I am using this figure for pagination, I need the next highest integer, in this case: 1.
Both Math.ceil and Math.abs produce the result 0 or 0.0.
How do I end up with a page number integer?
Comment: It sounds like the error is in code you're not showing us.
Comment: Why do you say that? Is there something wrong with what I have written above? Could you explain what you mean?
Answer: The correct way to do that is:
int pages = (totalRecords+recordsPerPage-1)/recordPerPage;

In your case: pages = (2 + 25 - 1)/25 = 26/25 = 1
Comment: +1: I can tell I'm used to dealing with PHP and the like... I thought about int / int equaling an int as to why `Math.ceil` wasn't working properly, but not for producing the correct result without using `ceil`/`floor`.
Answer: Math.ceil should never give you 0 for 0.08.  Clearly there's a bug in the code you didn't post.
System.out.println(Math.ceil((double)2/25));

outputs 1.0 in Java 6u21 just like you would expect.
At a guess, your other code is missing the cast to double on one of the arguments, and int / int always returns int in Java.
System.out.println(Math.ceil(2/25));

prints 0.0.
Comment: @Linc: You need to cast `2` or `25` to a double first, or you get int `0` before doing `Math.ceil`.  I've edited my answer to note that.
Comment: I was doing: Math.ceil((2/25))
Comment: @Linc Math.ceil((double)2/25) . I am sure this will work for you.
Answer: You can add 1 to your result and then do a cast to int.
(int) (1 + (double)2/25)

This will go to the next higher number and then truncate.
But, the correct equation would be:
(int) (0.5 + (double) 2/25)

So, if you have 25 then it would be:
0.5 + 25/25

or
0.5 + 1 so int of that is 1

You can round up by adding .5 and taking the int.
Comment: This is wrong, imagine if there is 25 records, with pageSize=25. The formula gives 2 pages, even though there is only 1 page.
Answer: Once you get 0.08, add 0.5 then do Math.floor.
Comment: @Linc: you ALWAYS add 0.5 before using floor.  Or, you could use Math.round which works like this: (int)Math.floor(a + 0.5f)


Git create full file patches to download/distribute?

Question: Either i'm going nuts or nobody likes/liked this feature, but a long time ago i used to use subversion with sourceforge system. I had the ability to create full file patches for commits done.
I cannot figure out anyway to create these in git, all i want is the files that were changed from XX commit and only those files in their entirety so that i could hand them off to someone/upload just those files to a location.
As it currently stands i'm stuck reuploading the entire project because i have nothing that tells me what was changed. Since we're also on a shared web host there is no way to get git on the server without upgrading to more expensive package.
I've tried
git archive --output=/home/username/temp.zip <commit id>

which put everything into a zip nicely but it did just that, it put everything. Also tried a variation of format-patch but that didn't seem to work.
Answer: It seems that I misunderstood what you wanted in my other answer.  git archive can take a list of paths to include in the archive, so you could do:
git archive -o /tmp/foo.zip HEAD $(git diff --name-only oldcommit HEAD)

If your filenames contain surprising characters, though, it would be safer to do:
git diff -z --name-only oldcommit HEAD | xargs -0 git archive HEAD -o /tmp/blah.zip --

Comment: Thanks, this worked. In my case I needed it again a branch instead of commit. So I did: `git diff -z --name-only origin/branch_name | xargs -0 git archive HEAD -o /tmp/blah.zip`, and then applied those changes by extracting file in my repo: `tar -xf /tmp/blah.zip`
Answer: The first thing to try would be to save the output of git diff, which might be sufficient in your situation:
git diff oldcommit > test.patch

If you have changed binary files since then, you could try:
git diff --binary oldcommit > test-with-binaries.patch

In that case you'd then need to apply the patch with git apply.  Otherwise, I would create a new branch based on oldcommit, do a squashed merge from master, commit the result and and use git format-patch to produce the patch.  That method (and several other solutions to your problem) are described in more detail in the answers to this similar stackoverflow question: How do you squash commits into one patch with git format-patch?
Comment: That still only outputs the section that was changed, i need the whole files in their entirety. If i changed 2 lines on config.inc i want the entire config.inc not just the changed lines.
Comment: @iarp: Ah, OK - sorry.  I've added another answer with a different suggestion that uses `git archive` instead.


What are ways to "decorate" a chord progression played on guitar?

Question: I picked up the guitar half a year ago. Much of my playing is me playing a song's chord progression and a friend singing over it. Some ways that I've been experimenting with to "decorate" my playing on the spur of the moment are:

different strumming patterns
damping the strings
switching in sus2/sus4 chords
hammering on or pulling off one of the strings in the chord
playing the chord at different places
even sneaking in some melody notes if I've planned ahead

All of these ideas or patterns I've just been lucky to come across. How can I find more ways to decorate "simple" chord progressions? I've been trying to google such lists, but maybe I'm not using the right words, or maybe as a newbie I'm not thinking from the right perspective.
Comment: Another good trick: if you have, say a bar of one chord (e.g. **C7**), split the bar into two different chord shapes to introduce some variety without changing the harmony (e.g., play **C7** in root position, then in 1st inversion).
Comment: Chord substitutions can sound great when you are playing by yourself. With other players you need to be careful; sometimes they change the quality of the harmony enough that it can throw other players off a bit. For the most part you probably want to avoid making substitutions that conflict with the melody, so it shouldn't cause problems for singers, except that unexpected chords might surprise them ;)  This is the kind of thing that you just have to experiment with by yourself and also when playing with others.
Comment: @foreyez - check in any thesaurus, and the two words are in fact synonymous. Decorate = embellish; embellish = decorate, with or without quotation marks.
Comment: You can make some chord substitutions; you might take a look [at this question and answers](https://music.stackexchange.com/questions/1024/what-are-some-chord-substitutions-for-a-i-iv-v-blues-progression).
Comment: @DavidBowling Thank you, great resource! Do you think these kinds of substitutions sound good even if I'm the only one playing, or could e.g. substituting V for ii make it sound like something else and throw the singer off?
Comment: the keyword you're looking for is "embellish" not decorate. look up on google/youtube how to embellish chords on guitar.
Comment: @Tim I know, but I meant there's alot more youtube results for that particular keyword "embellishment" in music esp in guitar
Comment: I'm a little late to the party but I'd suggest learning _lots_ of songs. You'll pick up loads of different techniques just by studying other people.
Comment: You could arpeggiate the chords. You could also add a bass line in with them. For example, if a progression goes from G to Em, the bass line may well include an F♯ In the transition. So play G G/F♯ Em instead.
Answer: Sounds like you've already picked up quite a few effective ideas!
6ths sound good in some songs. 7ths also - maj7, m7 and if going to a chord a 4th above what you're on, dominant 7ths. 'Add 9ths' also can fit.
Instead of written dom.7ths, 9ths sound good, as do 13ths in the right place.
Tritone substitutes work well, on your own. Playing with others, you may surprise them, and two different chords together rarely sound harmonious.
For turnarounds, several options are available. Instead of a bar of V, play ii, V before the next verse. Or given a couple of bars, (in C here), play C, Eb/ D, Db. Or the ubiquitous I, vi / IV, V.
An occasional harmonic breaks things up too, especially in G - where there's 12th fret 2,3,4 strings (G), 7th fret 2,3,4 strings (D) and 12th fret 1,3 strings (C).
A few little ideas for you to add to the armoury.
Comment: With "9ths sound good" on dominant 7th chords, I imagine that you include altered 9ths since sometimes a 7♭9 or a 7♯9 chord works well where a natural 9th chord fails. In general, dominant 7th chords can often be replaced by altered versions (using ♯5, ♭9, ♯9, ♯11, etc.) to add some variety without causing too many problems for other players. Good ideas here, +1
Comment: @DavidBowling - thanks. Don't forget the humble b5 too! Or would that be encompassed in the #11...
Answer: Learn all of the basic CAGED system chords. That will give you 5 different voicings for just about any basic chord. Another trick, depending on style, is the b3 to 3 hammer on for major and 7th chords,great for folk, blues and rock.
Answer: Apart from the great answers that beat me to this thread, I'd add:

Varying volume
Pinch harmonics or artificial harmonics
Adding the 9ths
Sliding into the chord from a half-step below
Arpeggiate!
Add percussive elements
Any kind of chord substitutions (but make sure everyone's on the same page)
Find different voicings on the guitar and move up and down the fretboard stylistically
Pay attention to the bass notes, maybe add some stuff
Go wild on the dominant (now's the time for that crazy scale you learned)
Try using fewer notes to imply the harmony. Double stops are particularly effective, and you often can emulate the melody much more easily.
Drone notes (let a few notes ring through the entire progression)
Syncopation in general

And consider taking a more chord-melody/fingerstyle approach to playing. It's very helpful to be able to blend the melody and harmony, because then one finds that it's easy to embellish the harmony when accompanying. Of course, this is harder and takes planning, but it can be extremely rewarding. Plus, one becomes a better guitarist.
Answer: for beginners very simple but most expressive is the bass change of the root to the 5th (4th down), that's do-so do-so , re-so re-so, or the 3rd in the bass. this will fit with strumming and picking).
Country songs and blues use the change of the 5th and the 6th: then you'll hear in the chord accompaniment: so-la so-la.
In the C and E this is most easy.
Comment: Thank you! Do you mean this as alternating the bass for fingerstyle, or as playing the inversions while strumming chords?
Answer: You're on the right track. 'Playing the chord at different places' particularly caught my eye - the term for this is inversions if you want to dig in. You should also look into voicings and that will open up plenty of new options, like throwing in drone notes or creating wide intervallic spacing.
You can get a lot of mileage out of a voicing by simply modifying what notes you sound with your picking hand. 'Man of Constant Sorrow' is a great example - track down the tab and study the opening riff, with the same chord shape they're able to create movement by working different sections; be selective in what notes you sound - often less is more and gives you more options.
There's also alternate tunings. You should get your fundamentals down first but know that they are out there and the options are limitless. I'm mentioning this in case you are trying to learn something by ear and can't figure out how in the world it's played.
Answer: Primarily, you listen to other performers, both live and recorded.  When you hear an effect that you like, but it isn't obvious how to do it, then you ask.
Sit down with another guitarist for an hour.  He'll have some tricks to show you, and you'll have some to show him.
Sorry if this is all a bit obvious :-)
Comment: this is kind of the way I've been doing it, asking friends I hear making call sounds but mostly looking at youtube videos. I was hoping to structure my development a little bit more :)


Rails: Convert string to variable (to store a value)

Question: I have a parameter hash that contains different variable and name pairs such as:
param_hash = {"system_used"=>"metric", "person_height_feet"=>"5"}

I also have an object CalculationValidator that is not an ActiveRecord but a ActiveModel::Validations. The Object validates different types of input from forms. Thus it does not have a specific set of variables.
I want to create an Object to validate it like this:
validator = CalculationValidator.new()
validator.system_used = "metric"
validator.person_height_feet = 5

validator.valid?

my problem right now is that I really would not prefer to code each CalculationValidator manually but rather use the information in the Hash. The information is all there so what I would like to do is something like this, where MAKE_INTO_VARIABLE() is the functionality I am looking for.
validator = CalculationValidator.new()
param_hash.each do |param_pair|
  ["validator.", param_pair[0]].join.MAKE_INTO_VARIABLE() = param_pair[1]
  # thus creating
  # "validator.system_used".MAKE_INTO_VARIABLE() = "metric"
  # while wanting: validator.system_used = "metric"
  # ...and in the next loop
  # "validator.person_height_feet".MAKE_INTO_VARIABLE() = 5
  # while wanting: validator.person_height_feet = 5
end

validator.valid?

Problem:
Basically my problem is, how do I make the string "validator.person_height" into the variable validator.person_height that I can use to store the number 5?
Additionally, it is very important that the values of param_pair[1] are stored as their real formats (integer, string etc) since they will be validated.
I have tried .send() and  instance_variable_set but I am not sure if they will do the trick.
Answer: Something like this might work for you:
param_hash.each do |param, val|
  validator.instance_eval("def #{param}; @#{param} end")
  validator.instance_variable_set("@#{param}", val)
end

However, you might notice there's no casting or anything here.  You'd need to communicate what type of value each is somehow, as it can't be assumed that "5" is supposed to be an integer, for example.
And of course I probably don't have to mention, eval'ing input that comes in from a form isn't exactly the safest thing in the world, so you'd have to think about how you want to handle this.
Answer: Have you looked at eval. As long as you can trust the inputs it should be ok to use.


How to add 2 Cognito Pools in AWS Amplify

Question: I am creating an app using vue3 and amplify. My app needs to have 2 different user types. I am able to add 1 user type using amplify auth API and that user type is able to register and log in. However, when I do amplify add auth to add a second Cognito pool, I get the result: "Auth has already been added to this project. To update run amplify update auth."
Is there a way to support this either through amplify studio or amplify cli?
Answer: No, it is not possible to add two user pools. Instead, you can use single user pool and create different groups for different user types inside that user pool.
Could you elaborate more on your user case? If you really need two user pools, there might be also advanced ways to solve your case. For example, using Cognito Federated Identities (identity pools) and adding two user pools as authentication providers.


Getting Started Recording and Arranging Drums, Guitar, Bass, Vocals, Etc

Question: I'm new to the recording scene, and I'd like to jump right into making professional sounding, high-quality productions. Now that I can play the instruments, I need to know a few things about recording:
How do I record acoustic drums without having to buy 5 or 6 mics?
I have tried Garage Band and thought that the overall sound is good for 15 bucks, but I'm willing to spend more than that in order to truly get clean sounding recordings. So, what makes ProTools worth its price? If not ProTools, then what's another good program for editing and processing?
Finally, what is a kind of snare drum that really makes a crisp sound like the one Travis Barker uses in Feeling This?
Thanks!
Comment: For the future you might want to think about asking multiple questions. This will yield _specialized_, _in_ _detail_ answers rather than a good answer to one question and rough guesses on the rest. (Also: Easier headlines for you, better search results for all others, differentiated voting on question and/or answers and much more :D )
Comment: Just a quick tip, getting "professional sounding, high-quality productions" will take time and a lot of practice. Producing is a skill just like playing an instrument, the more you do it the better you get. As far as the drums go, it will be a **lot** easier if you start with drum loops/constructing beats with samples, instead of trying to record a whole drum kit, especially if you only have one mic. As far as software, that's a more complicated decision. I like Ableton but it's more for jamming with yourself or writing/arranging tracks and trying ideas, less popular for straight up recording.
Comment: I don't want to be rude, but if you want to make a "professional sounding", you will have to split all your components and recording them all together, but on a different mic, and then remaster it. It is a little investment, but you can find mics and mixing console for less than 500€ on the internet, and it will be enough for your needs.
Comment: One on-topic question, one blatantly-off-topic question and the answer to the third question: Piccolo snare drum.
Comment: I would suggest splitting this question into the three separate questions that you have stated although be warned the second question WILL be closed as off topic. and the third will not be well received as it is purely down to personal preference. but for the first question you need only buy 4 cheep mics 2 for overhead 1 for snare and 1 for bass drum.
Answer: First of all, "making professional sounding, high-quality productions" takes years of experience and investment in decent quality gear. So, don't be disappointed if you can't "jump right into" it. Unfortunately, as with anything about music, patience is the key here.
I would recommend you to start with reading some books on recording, mixing and mastering; probably in that order. All three are equally important and a good recording engineer is not necessarily a good mixing engineer or a good mastering engineer (and vice versa). They are different domains of expertise.
A good producer needs to know about all these fields, at least enough to know who to hire. Producer's job is to conceptualize the final product from the beginning and direct the song writers, performers and recording, mixing and mastering engineers towards that concept (whether he does or does not perform one/some/all of these functions).
Now, about your specific questions:
Recording acoustic drums
It depends on the genre. For jazz, you can get away with a pair of good overhead mics and a good drummer with good control, possibly complemented with a good bass drum mic. But if you're aiming at processed rock drums for example, you're going to need more than that so you can process the bass drum and the snare drum separately at the very least. Drum mic setups with 10+ mics are not uncommon.
There are cheaper high-quality mics on the market now, but it's still quite an investment. Also, the quality of the recording room is at least as important as the quality of the drum set and the mic set, if not more so. You will also need good mic preamps and a good sound interface.
The only cheaper option I can think of is using a MIDI drum set (or writing MIDI data by hand) and a good software drum sample library. Depending on the genre, they can be totally unacceptable or better than real drums.
Recording software
Believe it or not, all major recording software (or DAWs -digital audio workstations- as they call them) have more or less the same functionality and the same sound quality. ProTools can come with some external cards that takes the CPU load off of the main computer but with a powerful computer it's not an absolute necessity. The choice is more about workflow habits (and budget!) than sound quality.
You can buy a cheap DAW (like Reaper) and spend the rest of your money on effects and synth plugins and/or external gear. In fact, effects and synth plugins are more important than your DAW.
Crisp snare drum
I don't feel qualified to answer this question but you can definitely achieve good results with most crisp sounding snare drums with proper room, mics, preamps and effects (eq, compressor etc.).
Answer: You can record a drumset with just two overhead microphones or a stereo pair, but it'll sound jazzy with not much punch on bass-drum and snare-drum. Also with just two microphones, the room in which you record gets more important, as in such a setup the mics pick up substantial amouts of "room" compared to the direct sound from the drums. Placing mics near the drums mitigates this problem, but you'll need more mics or the drums close to the mics will sound too loud.
If you want EQed/compressed fat bass and snare you need dedicated microphones for these. You can buy a mic for 50 bucks or less. It should tolerate high sound levels, but other than that quality requirements are moderate for bass and snare (and somewhat higher for the overheads).
As for the choice of snare drum, I am not an expert, but I would assume that the overall sound is determined primarily by the way it is played, mic'ed and EQed, the damping of the room and not so much by the snare drum itself.


Adicionar valores a TextView tratamento

Question: Estou criando um app de controle financeiro, é um projeto da faculdade, no local onde mostra o valor no formulário fiz uma TextView que a ser clicada abre um Dialog onde criei um teclado personalizado para colocar o valor, quando o usuário adiciona o valor e clica em ok o valor passa pra TextView exatamente como o usuário passou o valor,oq eu quero fazer é tratar esse valor, mas não conheço a forma de tratar esse valor já que sempre trabalhei com C# e comecei agora android,
ex: se o usuário adicionar um valor sem virgula quero que seja adicionado a string ,00 . se o usuário adicionar um valor ex: 15,5 quero adicionar depois do 5 + um 0.
Comment: acredito que não tenha problema o fado de ser TextView, continua sendo texto, a logica é a mesma
Answer: Utilize a classe DecimalFormat para formatar valores numéricos, com ela você pode definir um padrão e então formatá-lo, por exemplo:
DecimalFormat df = new DecimalFormat("#,##0.00");
String formatted = df.format(12345.67);

textView.setText(formatted);

Uma outra opção é utilizar a classe NumberFormat, com ela você poderá formar porcentagens, moedas, valores inteiros etc, por exemplo:
NumberFormat nf = NumberFormat.getNumberInstance(Locale.GERMAN);
String formatted = nf.format(12345.67);
textView.setText(formatted);

Comment: vou tentar aqui ver se tenho o resultado esperado
Comment: casa quando coloquei o metodo o valor que a minha TextView retornou sempre foi 123456,67, ela sempre retorta o que ta aqui: String formatted = nf.format(12345.67);
Comment: @RomãoGomes Qual deles você utilizou?
Comment: usei todos os dois, todos me retornam o valor (12345.67)

Adicionar valores a TextView tratamento


Execute a limited number of Tasks concurrent

Question: Given I have an IEnumerable<Func<Task>> to generate some (crawler)-Tasks which I want to execute concurrent, how can I set an upper bound for the concurrency?
E.g., I do not want that more than 5 of these Tasks are running at the same time. On the other side, there always should be 5 Tasks running if possible.
My current approach is this code:
    public static async Task ExecuteConcurrent(IEnumerable<Func<Task>> taskGenerators, int maxDegreeOfConcurrency)
    {
        var executingTasks = new HashSet<Task>();
        foreach (var taskGenerator in taskGenerators) {
            while (executingTasks.Count >= maxDegreeOfConcurrency) {
                executingTasks.Remove(await Task.WhenAny(executingTasks));
            }

            executingTasks.Add(taskGenerator());
        }

        await Task.WhenAll(executingTasks);
    }

I am wondering if there is a better method to do this? Maybe, there is already a method available?
Thank you
Answer: Sounds like a job for TPL Dataflow
The advantages are :

It work well with async and await and both CPU bound and IO bound workloads
You can limit concurrency with MaxDegreeOfParallelism and many other options
You can Chain it into more complicated PipeLines
Works well with Reactive Extensions
its all built for you by Mr Parallel Stephen Toub

Nuget System.Threading.Tasks.Dataflow

Very basic example
public static async Task DoWorkLoads(List<IPAddress> addresses)
{
   var options = new ExecutionDataflowBlockOptions
                     {
                        MaxDegreeOfParallelism = 50 // limit here
                     };

   var block = new ActionBlock<SomeObject>(MyMethodAsync, options);

   foreach (var ip in addresses)
      block.Post(ip);

   block.Complete();
   await block.Completion;

}

...

public async Task MyMethodAsync(SomeObject obj)
{
    // await something here
}

Comment: @KevinMeier: You'd want to set the `BoundedCapacity` option for that.
Comment: Cool solution:)! Thanks:)! What if my `IEnumerable` is huge (=bigger than my memory). In this solution the complete `IEnumerable` (if we replace `List` by `IEnumerable`) would be consumed as soon as possible. Can this be avoided or reduced? Because it would not really be necessary.
Comment: Yeah there is a lot of options, and this is designed for complicated and demanding workloads. Possibly look up Stephen toubs pod cast on tpl dataflow, although they look a bit bgrade he will cover a lot of things I can't on my phone
Answer: Starting from .NET 6, you can use the Parallel.ForEachAsync method:
public static Task ExecuteConcurrent(IEnumerable<Func<Task>> taskGenerators,
    int maxDegreeOfConcurrency)
{
    ParallelOptions options = new() { MaxDegreeOfParallelism = maxDegreeOfConcurrency };
    return Parallel.ForEachAsync(taskGenerators, options, (taskFactory, ct) =>
    {
        await taskFactory();
    });
}

The ct argument is a CancellationToken provided by the Parallel infrastructure, that will be signaled immediately after a taskFactory fails. You could pass this argument to your factory and sprinkle your code with ct.ThrowIfCancellationRequested() in strategic points, to speed-up the termination of the parallel execution in case of an error.


Why does it make sense to pair existence and uniqueness?

Question: Edit. September 8th, 2013. I've added another another section to the question, which should hopefully constrain its scope. Please see below the horizontal line!!
Lets write the statement 'there is at most one $x$ such that $Px$' symbolically as $!xPx,$ and assert the following definition.
$$!xPx \Leftrightarrow \forall xy(Px \wedge Py \rightarrow x=y).$$
Furthermore, lets define the statement 'there exists unique $x$ such that $Px$' symbolically as follows.
$$\exists! x Px \Leftrightarrow \exists xPx \;\wedge\:\, !xPx$$
I don't have a good understanding of why the conjunction $\exists xPx \;\wedge\:\, !xPx$ is useful. I know, for example, that:

It shows up a lot.
The $\exists$ quantifier can be viewed as asserting the existence of $\geq 1$ of a thing, and the $!$ quantifier can be viewed as asserting the existence of $\leq 1$ of a thing. So together, they say that there is precisely one of a thing.
The statement $\forall x\exists! y Q(x,y)$ holds iff there exists a unique function $f$ such that $\forall xQ(x,f(x))$.

But still, I don't really understand how uniqueness and existence 'work' together. Clearly there's some sort of synergy going on here. The whole is greater than the sum of the parts, kind of thing.
Why does it make sense to pair existence and uniqueness?

Edit. I noticed that, in the presence of the assumption that $Px$ holds for a unique $x$, we have that
$$\forall x(Px \rightarrow Qx) \iff\exists x(Px \wedge Qx)$$
So it makes sense to create the notation $\exists! x(Px \mid Qx)$ to mean: "There exists unique $x$ such that $Px$, and for the unique such $x$ we have $Qx$." In light of the above observation, this can be defined by either of the following formulae.

$\exists! x Px \wedge \forall x(Px \rightarrow Qx)$
$\exists ! x Px \wedge \exists x(Px \wedge Qx)$

Lets also create the notation $\exists x(Px \mid Qx)$ as shorthand for "There exists (not necessarily unique) $x$ such that $Px$, and for all such $x$ we have $Qx$." Symbolically, this can be defined as in a similar way to 1, except without the uniqueness requirement. Namely:

$\exists x Px \wedge \forall x(Px \rightarrow Qx).$

Now its clear that the following implications hold.
$$\exists ! x (Px \mid Qx) \implies \exists x(Px \mid Qx) \implies \exists x(Px \wedge Qx)$$
My question is this:
Does $\exists ! x (Px \mid Qx)$ have better properties than $\exists x (Px \mid Qx),$ that make it more mathematically useful? If so, what are they?
Comment: Why do you think it doesn't make sense?
Comment: @AsafKaragila, I *do* think it makes sense! I'm just not sure *why*.
Comment: I think you answered your own question with your #1.  It shows up a lot, so it's useful and convenient.
Comment: It seems to me this is not a very common way to define uniqueness. More common and more intuitive is the equivalent definition given by amWhy.
Answer: Note that you need not create an "at most one" quantifier, since as you note, $$\forall x \forall y\Big((P(x) \land P(y)) \rightarrow x = y\Big)$$ states the same: there is at most one "thing" in the universe, let's just call it x, for which P(x) holds. It claims nothing about whether or not P(x) it holds for any "x" in the universe, so it tells us nothing about the existence of anything unique. It tells us that if there is anything such that P is true of that thing, there is at most one such thing.
To make the stronger assertion of existence and uniqueness, then, we simply add the assertion of existence, rearrange quantifiers, giving us: $$\exists x\Big(P(x) \land \forall y(P(y) \rightarrow x = y)\Big)$$
Comment: You didn't really answer the question, but this answer was still very useful to me, so consider it accepted.
Answer: More philosophically,
when you want to find a solution to something,
the first step is to show
that a solution exists
(i.e.,
$\exists x\ P(x)$)
or show that no solution exists
($\not\exists x\ P(x)$).
In many cases, just showing that a solution exists
is enough.
Once you know that a solution exists,
the next step is
to find the number of solutions
($\#x | P(x)
$).
If there is only one
(some have asserted that
"there can be only one",
but that is clearly not true),
then you have uniqueness
($\exists x(P(x) \land \forall y(P(y) \rightarrow x = y))$
or
$(\exists xP(x) \land \exists yP(y) \implies x = y)$
).
The ultimate goal is
to exhibit all the solutions
($\{x | P(x)\}$).
I'm in a meandering mood this morning.


What are the advantages of the Linux server over the Windows server?

Question: I know this is a pretty short and straight forward question, but what are the advantages of the Linux server over the Windows server?
Is linux better than windows? And if it is/isn't, how?
Answer: Is Linux better than Windows?  Mu!
This may be a short question, but it's definitely not straightforward to answer. Without knowing what function you are going to put the server to, it's even more difficult to answer (it's like the perennial Mac vs Windows or Emacs vs. vi question).
At the most basic level, all an OS is, is a platform for running applications (i.e., boring);  it's the applications that make computing interesting.  If the application that you need to run is best supported on Windows, then use Windows.  If all of your experience is with Linux, and you have an application that does what you need in Linux, then use Linux.
In general, the initial price of the Operating System is only a small part of the TCO of a server.  You must also take into account server Maintenance, Application maintenance, the management cost, the cost of any software you install, etc.  And these days, you can't even say that Linux is more stable than Windows — we aren't living in the '90s any more...
As much as some of the more rabid zealots would like to say otherwise, both platforms are equally valid, depending.  It's up to the Sysadmin to look objectively at a given problem, and decide what OS/Application combination is best going to solve it.
Comment: +1 especially for the last sentence. Platform choice depends largely (arguably almost entirely) on the purpose of the server, the duties it will be performing, and the clients it will be serving.
Comment: Good points all "we aren't living in the '90s any more". +1 Just for mu reference great answer.
Comment: I think your answer salvages this question.
Answer: Probably the single biggest reason I like UNIX-like servers over Windows servers is that I've found any problems are so much easier to diagnose and fix. I spent much more time beating my head against a wall or on hold with tech support with Windows servers. With OpenBSD, FreeBSD, or Linux the problem is usually in the logs, and if the solution isn't obvious then it usually yields to Googling a bit.
When everything is running smoothly then Windows can be a fine server platform, as can Linux and *BSD. It's when things aren't working that the difference comes in.
As for the cost of admins, you have to take competence into account. Sure Windows admins are plentiful, but to hire a good one will cost you just like hiring a good UNIX admin. If you just need someone to click "Next" on install dialogs it'll be cheaper to go Windows, but you don't want that.
Comment: +1 for the cost of admins part, liked that... -1 for the easier to troubleshoot, but that's very subjective so I won't down vote (seing this is not yet a CW). I personally think it's about the same especially with later Microsoft server/application versions but hey, experiences differ a lot and no one is biased ^^
Comment: Yes, experiences differ but that's been mine. And I "grew up" on DOS, then Windows. The later Windows stuff may be much better than what I was working with. I hope so!
Answer: Well the big one and the most obvious one is the price. Being able to literly be able to buy two servers instead of one if you put linux on them is a major pull (if it can do the job).
Answer: Well, this question cries out for a "subjective" tag.
Anyway, the answer here is `to each their own'.
I mean, it all depends on what you need to run on this server. A directory service? Should I use openldap? No, thanks, I'll probably use Active Directory (on a Windows-server-based domain controller). An application needs SQL Server (microsoft one :P)? Looks like a job for windows server to ne. On the other hand, I don't see much point in running DNS or (you must be kidding me) DHCP on a windows server. Or a web proxy (hello, squid). Or a firewall/router.
To each their own.
Comment: Certainly there will be subjective info here, but there are also factual points to be made on both sides of the issue. I know good admins that run both!
Answer: TCO is the most important factor here.  Yes Linux is free and it can match Windows Server for functionality.  But have you considered...

Cost of hiring administrators - Windows admins are far more plentiful and to an extent (very much open to debate) the MCP certificates provide a neat way of qualifying applicants.
Ongoing costs - *nix admins generally tend to command higher salaries (very much depends on location)

In reality it is rare to actually get the choice of what platform to go with, the decision is made by 'PHBs' not on the technical merits, or the platform is chosen by the fact that a line of business application only runs on one or the other.
What you get for your money with Microsoft is a platform that is easier to setup/manage than Linux, by virtue of only having one Mail server, one DNS server, one directory service and so on it is simpler because you have no options.
Whether this ease of use / simplicity equates to better is really a matter of opinion.  Ultimately, as with anything in business, just use the one best suited to your specific needs, neither platform is better but one will be better suited to you.
Answer: Is linux better than windows?
The answer is BLUE!
Is windows better than Linux?
The answer is CHEESE!
The point is that this is a nonsensical question.
"For my workload, budget, expenses, skillset, future plans, and applications that I run, which platform is likely to be the best choice for me moving forwards?"
There are other variables, of course- but without at least the info above, you're just wanking off fanboys.
Comment: Mmmm blue cheese...
Comment: He meant blue screen of death
Answer: I think it depends on the scenario. Many guys may reply with "Linux is better" but I suggest to select the server with your available resources \ your software requirements etc.
As an example if you are using Linux servers the users will need bit more expertise to work with that (If there are no sys admins to manage them). And even there is a system administrator he will need some more knowledge to work with linux and get the things configured because the effort required to do some configuration is higher in Linux (Not in all cases but in most cases)
In the other hand you may need to install some softwares that doesn't have a "Linux" option.  Generally we can find a windows version for almost any Unix like OS bases softwares but not wise versa. (Proprietary softwares in most cases). So I recommend to consider those factors as well.
I just wanted to show some "Unseen" side of a practical problem ;)
Answer: One area where Linux (or *BSD) really shines is repurposing old hardware for a specific task. You can take any old box that's gathering dust in a corner or attic and have a serviceable little server in minutes.
Answer: In my experience I have benchmarked an identical database query on an identical database with WIN Server 2005 - 64 bit loaded vs OpenSuse 10.3 64 bit on an Intel 64 bit machine. The results I achieved was that the Linux response was 10x faster on a resultset that returned 530,000 rows. Linux = 3 seconds, Windows = 30 seconds. I have not tweaked anything - everyting was installed out of the box with no tweaking. The database used was PostgreSQL, identical versions - 8.3.3.
I did this exercise to satisfy a customer demand for faster response, I am not sorry that I used OpenSuse (10.3) - I have had no response related complaints since the switch.
I hope you will find this info useful - it is supplied for that purpose and not to knock Windows.
Comment: Postgresql is not supposed to be used in production environment on Windows... So yes it's really slow on Windows, try SQL Server instead


C# MVC - Using LIKE in LINQ query

Question: I want to display names from a SQL database that ends with certain char but it won't let me use WildCard in the Controller ActionResult.
This is my current code that I want to work.
public ActionResult GetClients()
    {
        MyDBEntities myDBEntites = new MyDBEntities();
        var clients = from c in myDBEntites.Client
                      where c.name LIKE '%r'
                      select c;
        return View(clients);

    }

What am I missing here for it to work? If its not working with wild card then how do I get the names that ends with a specific char?
Comment: Can't do `LIKE` in LINQ AFAIK. Do `WHERE c.name.EndsWith("r")`
Comment: @apokryfos thank you! i wonder why WildCard didnt work, i mean its like a complete SQL code except from wildcard. weird.
Comment: LINQ is similar but not the same as SQL . It's simply different query languages. The compiler will convert LINQ to SQL as best it can when performing queries so I wouldn't be surprised if `.EndsWith` actually gets converted to `LIKE` in SQL.
Comment: @apokryfos Ahh ok i get it now. Thank you so much man
Answer: You can use EndsWith
public ActionResult GetClients()
    {
        MyDBEntities myDBEntites = new MyDBEntities();
        var clients = from c in myDBEntites.Client
                      where c.name.EndsWith("r")
                      select c;
        return View(clients);

    }

Answer: LIKE is a SQL keyword, therefore you could use EndsWith
c.name.EndsWith("r");

As a best practice, you may want to consider wrapping your DBEntities within a using block so it is correctly disposed. You could also move it into a new method/service layer, rather than retrieving it within the Controller which would be a cleaner approach.
Service Layer
public IEnumerable<Client> RetrieveClients(string endString)
{
   using (MyDBEntities myDbEntities = new MyDBEntities())
   {
        var clients = from c in myDBEntites.Client
                      where c.name.EndsWith(endString)'
                      select c;

        return clients;
   }
}

Controller
public ActionResult GetClients()
{
    var clients = RetrieveClients("R");
    return View(clients);
}

Answer: What I would suggest you to do here is to try to solve this with String.EndsWith (LINK TO DOCUMENTATION) method.
Let me write an example based on your question, using linq ofcourse:
public ActionResult GetClients()
{
  MyDBEntities myDBEntites = new MyDBEntities();
  var clients = from c in myDBEntites.Client
                where c.name.EndsWith("r")
            select c;

 return View(clients);
}

Comment: @Dolev if you've found this answer useful please let me know :)

C# MVC - Using LIKE in LINQ query


Bayesian vs MLE, overfitting problem

Question: In Bishop's PRML book, he says that, overfitting is a problem with Maximum Likelihood Estimation (MLE), and Bayesian can avoid it.
But I think, overfitting is a problem more about model selection, not about the method used to do parameter estimation. That is, suppose I have a data set $D$, which is generated via $$f(x)=sin(x),\;x\in[0,1]$$, now I might choose different models $H_i$ to fit the data and find out which one is the best. And the models under consideration are polynomial ones with different orders, $H_1$ is order 1, $H_2$ is order 2, $H_3$ is order 9.
Now I try to fit the data $D$ with each of the 3 models, each model has its paramters, denoted as $w_i$ for $H_i$.
Using ML, I will have a point estimate of the model parameters $w$, and $H_1$ is too simple and will always underfit the data, whereas $H_3$ is too complex and will overfit the data, only $H_2$ will fit the data well.
My questions are,
1) Model $H_3$ will overfit the data, but I don't think it's the problem of ML, but the problem of the model per se. Because, using ML for $H_1,H_2$ doesn't result into overfitting. Am I right?
2) Compared to Bayesian, ML does have some disadvantages, since it just gives the point estimate of the model parameters $w$, and it's overconfident. Whereas Bayesian doesn't rely on just the most probable value of the parameter, but all the possible values of the parameters given the observed data $D$, right?
3) Why can Bayesian avoid or decrease overfitting? As I understand it, we can use Bayesian for model comparison, that is, given data $D$, we could find out the marginal likelihood (or model evidence) for each model under consideration, and then pick the one with the highest marginal likelihood, right? If so, why is that?
Comment: @nbro, sorry it's been a while, I don't remember clearly, but maybe it meant Bayesian could somehow mitigate the overfitting issue, not avoid completely.
Answer: Optimisation is the root of all evil in statistics.  Any time you make choices about your model$^1$ by optimising some suitable criterion evaluated on a finite sample of data you run the risk of over-fitting the criterion, i.e. reducing the statistic beyond the point where improvements in generalisation performance are obtained and the reduction is instead gained by exploiting the peculiarities of the sample of data, e.g. noise).  The reason the Bayesian method works better is that you don't optimise anything, but instead marginalise (integrate) over all possible choices.  The problem then lies in the choice of prior beliefs regarding the model, so one problem has gone away, but another one appears in its place.

$^1$ This includes maximising the evidence (marginal likelihood) in a Bayesian setting.  For an example of this, see the results for Gaussian Process classifiers in my paper, where optimising the marginal likelihood makes the model worse if you have too many hyper-parameters (note selection according to marginal likelihood will tend to favour models with lots of hyper-parameters as a result of this form of over-fitting).
G. C. Cawley and N. L. C. Talbot, Over-fitting in model selection and subsequent selection bias in performance evaluation, Journal of Machine Learning Research, 2010. Research, vol. 11, pp. 2079-2107, July 2010. (pdf)
Comment: +1, thank you so much, I'll read your paper and see if I have any further questions, ;-)
Comment: Just to note here that optimization can usually be thought of as approximately integrating - the Laplace method is an example of this.  Optimising usually fails when it isn't a good approximation to integrating - hence why REML is usually better than ML.
Comment: @dikran marsupial -  Perhaps a better way to explain it is that integration is often well approximated by estimating a parameter by ML, and constraining that parameter to be equal its MLE.  The Laplace approximation provides a "correction factor" to this intuition - in the same way that REML does.
Comment: @probabilityislogic, I am not sure I understand, ML is a bit like MAP, there is no integration performed.  Using the Laplace approximation (in the way I have seen it used) is optimising in the sense that you optimise an approximation to the function your want to integrate and integrate that instead, but there is still integration going on.
Comment: @probabilityislogic thanks for the reply, I'll give it some thought!
Answer: As a general response, if you're using "least squares" type regression models there really isn't much difference between bayes and ML, unless you use an informative prior for the regression parameters.  In response to specifics:
1)$ H_9 $ won't necessarily overfit the data - only when you have close to 9 observations.  If you had 100 observations, most of the supposedly "overfitted" coefficients will be close to zero.  Also $ H_1 $ would almost always result in "underfitting" - as there would be clear curvature missed
2) This is, not true for "linear" like polynomial expansions ("linear" meaning linear with respect to parameters, not $ x $).  ML estimates for least squares are identical to posterior means under non informative priors or large sample sizes.  In fact you can show that ML estimates can be thought of as "asymptotic" posterior means under a variety of models.
3) The Bayesian approach can avoid overfitting only for proper priors.  This operates in a similar manner to penalty terms you see in some fitting algorithms. For example,  L2 penalty = normal prior, L1 penalty = laplace prior.
Comment: upvoted, and you're right that with more observations at hand, $H_9$ won't overfit. But am I right with the claim that *overfitting is a problem of choosing the wrong model, not the ML per se*? And we can use Bayesian in model selection, but we can't do that with ML, can we?
Comment: Surely all choices of H here will be the wrong model, other than $H_\infty$.  The problem is the error in estimating the parameters of the model, which has both bias and variance components.  If you choose the model using a Bayesian criterion, you can still over-fit that as well (I'll add a reference to support that in my answer).
Comment: @loganecolss - I think $ H_9 $ here would be closer to the truth than any others.  Overfitting is more closely linked with the sample size and the type of model structure that it can provide support for (sometimes called the "sure thing" model).
Answer: Basically, what you're doing by increasing the degrees of your polynomials is increasing the number of parameters or degrees of freedom of your model space, ie. its dimension.
The more parameters you add, the more the model can fit the training data easily. But this also depends heavily on the number of observations. Your models $H_1$ and $H_2$ might just as well overfit the training data if the number of observations is low, just as $H_3$ may not overfit at all if the number of training instances is large enough.
For example, let's grossly exaggerate and suppose you are given only $2$ training examples, than even $H_1$ will always overfit your data.
The advantage of imposing priors for instance through regularisation is that the parameters are either shrunk to zero or some other predefined value (you can even add parameters to "tie" the coefficients together if you like), and thus you are implicitly constraining the parameters and reducing the "freedom" of your model to overfit. For example, using the lasso (ie. $l^1$ regularisation or equivalently a Laplace prior) and tuning the corresponding parameter (using 10x cross validation for example) will automatically get rid of the surplus parameters.
The Bayesian interpretation is similar : by imposing priors, you are constraining your parameters to some more probable value, inferred from the overall data.
Comment: A simple hypothesis (e.g. h1, h2) with insufficient training samples would be an example of under fitting (for cv) and not over fitting due to model bias on the few training examples given.

Bayesian vs MLE, overfitting problem


Why does YY represent a 4 digit year in a date format for the jQueryUI datepicker?

Question: When specifying a date format for the jQuery UI datepicker YY is a 4 digit year and Y is a 2 digit year:
YY = 2011
Y  = 11
See http://docs.jquery.com/UI/Datepicker/formatDate
Was curious why this is.
Comment: You'd probably want to ask them.
Comment: While I've offered an answer to this question I'm *also* voting to close as 'not constructive' since it solicits opinion, and without someone from the jQuery datepicker team answering the question, speculation and opinion is all we can offer.
Answer: All of the single character variations (d, m, y) are the short version. Double is long version. Since the short version of a year is two digits, using yy specifies the long version, which is four digits. The format is not directly related to the number of digits that the date outputs.
Just as M (single) means: "Jan, Feb, Mar, Apr", and MM (double) means: "January, February, March, April". The same goes for all of the format chars.
This is unlike the standard format for PHP, which may confuse some.
The chosen format was to meet the web standards (RFC)s for date formatting.
See: RFC-822
Comment: Which web-standards RFCs? Often I have seen "YYYY" as the long form and "YY" as the short form, although I could see an argument -- for symmetry -- against this, as per the post. Dates like [IOS8601](http://en.wikipedia.org/wiki/ISO_8601) are actually *defined* in terms YYYY for the 4-digit year (or +-YYYYY extended).
Comment: Source: http://www.ietf.org/rfc/rfc822.txt
Comment: In RFC822, the only thing I can find is a "Military Date" (or [Little-endian form date](http://en.wikipedia.org/wiki/Calendar_date)): `DD MON YY`, where `YY` is only two digits. While still common for [web server] log files, I wouldn't consider it a modern [1982] web-standard. ISO8601, and the small common subset of it, is much less ambiguous and variants thereof are used most systems... in any case, the question is about choosing `YY/Y` instead of `YYYY/YY` to designate the format, and I can see merit with the argument for *why* it was chosen (so +1). However, I do not agree with the rest :)
Comment: Okay, unfortunately, more things use RFC822 than that ... I being slightly facetious about it not being a "current standard". In any case, RFC822 only specifies a 2-digit year and is otherwise not related to the the `YY/Y` question. In RFC822, `date = 1*2DIGIT month 2DIGIT ; e.g. 20 Jun 82`.
Answer: Presumably because they wanted to keep things such as option/variable names brief; in this usage it seems clear that Y indicates the short-form, whereas YY clearly indicates the long-form, of the date format.


Find Two Closest Solutions

Question: Find the two closest solutions to $z=0$ of
$$e^{e^z}=1$$
I started with setting $t=e^z$
$$e^{t}=1\iff e^t=e^{(0+2\pi k)i}\iff t=2\pi ki$$
so
Because we are on the unit circle every two solution $|f(z_1)|=|f(z_2)|=1$ so we need to take the square root.
$$e^z=\frac{2\pi k i}{2} \text{    Where }   k=0,1$$
$$z=ln|\frac{2\pi k}{2}|=ln|\pi k|$$ for $k=1$ we get $z=ln|\pi|$ when $k=0$ we get $ln|0|$ so we need to take the 3rd root
$z=ln|\frac{2\pi k}{3}|$ for $k=0,1,2$
for $k=1$ we get $ln|\frac{\pi}{3}|$ for $k=1$ we get $ln|\frac{2\pi}{3}| $
Is it correct?
Answer: Changing $e^z = t$ we have $e^t = 1$ so $t=2\pi i k$ with $k$ integer, i.e. $e^z = 2\pi ik$ with $k$ integer. If $k=0$, $e^z=0$ has no solution, so if $k\neq 0$
$$
z = \ln|2\pi ik| + \arg(2\pi i k)i = \ln(2\pi |k|) + \arg(2\pi i k)i
$$
Simplifying
$$
z = \begin{cases}
\ln(2\pi k) + \pi i/2 & k >0\\
\ln(-2\pi k) - \pi i/2 & k<0
\end{cases}
$$
Comment: @newhere If $k>0$ then $\arg(2\pi i k) = \pi/2$ (is a pure imaginary number with positive imaginary part). If $k<0$ then $\arg(2\pi i k) = -\pi /2$.
Comment: How did we get $\pm \frac{\pi i}{2}$?

Find Two Closest Solutions