Taking the expectation of Taylor series (especially the remainder)

Question: My question concerns trying to justify a widely-used method, namely taking the expected value of Taylor Series. Assume we have a random variable $X$ with positive mean $\mu$ and variance $\sigma^2$. Additionally, we have a function, say, $\log(x)$.
Doing Taylor Expansion of $\log X$ around the mean, we get
$$
\log X = \log\mu + \frac{X - \mu}{\mu} - \frac12 \frac{(X-\mu)^2}{\mu^2} + \frac13 \frac{(X - \mu)^3}{\xi_X^3},
$$
where, as usual, $\xi_X$ is s.t. $|\xi_X - \mu| < |X - \mu|$.
If we take an expectation, we will get an approximate equation which people usually refer to as something self-apparent (see the $\approx$ sign in the first equation here):
$$
\mathbb{E}\log X \approx \log \mu - \frac12 \frac{\sigma^2}{\mu^2}
$$
QUESTION: I'm interested in how to prove that the expected value of the remainder term is actually negligible, i.e.
$$
\mathbb{E}\left[\frac{(X - \mu)^3}{\xi_X^3}\right] = o(\sigma^2)
$$
(or, in other words, $\mathbb{E}\bigl[o(X-\mu)^2\bigr] = o\bigl(\mathbb{E}\bigl[(X-\mu)^2\bigr]\bigr)$).
What I tried to do: assuming that $\sigma^2 \to 0$ (which, in turn, means $X \to \mu$ in $\mathbb{P}$), I tried to split the integral into two, surrounding $\mu$ with some $\varepsilon$-vicinity $N_\varepsilon$:
$$
\int_\mathbb{R} p(x)\frac{(x-\mu)^3}{\xi_x^3} \,dx = \int_{x \in N_\varepsilon} \ldots dx + \int_{x \notin N_\varepsilon} \ldots dx
$$
The first one can be bounded due to the fact that $0 \notin N_\varepsilon$ and thus $1/\xi^3$ doesn't bother. But with the second one we have two concurring facts: on the one hand
$$
\mathbb{P}(|X - \mu| > \varepsilon) \to 0
$$
(as $\sigma^2 \to 0$). But on the other hand, we don't know what to do with $1/\xi^3$.
Another possibility could be to try using the Fatou's lemma, but I can't figure out how.
Will appreciate any help or hint. I realize that this is sort of a very technical question, but I need to go through it in order to trust this "Taylor-expectation" method. Thanks!
P.S. I checked out here, but seems it's a bit of another stuff.
Comment: (+1) This issue recently came up in discussions of two questions related to finding the moments of $X^{-1}$. It pays to take additional care with such matters. :-)
Comment: The first order approximation may actually be better in some cases, because of the mean value theorem.  Not sure if the mean value theorem would help in the general case.
Comment: I would have thought that the dominated convergence theorem might be useful here, as the equation $E(o(..))=o(E(..))$ is an interchange of limits and integration.
Comment: Why is there a minus sign in front of the third term of the Taylor expansion? Also why in the fourth term  is there $3$ and not $3!$? What am I missing?
Comment: @Alecos: Just look at the $n$th derivative of $\log x$. That will answer both of your questions.
Comment: @cardinal Naturally. I guess I am going blind. Thanks!
Comment: @cardinal so, as far as I understood, there's still no clear way to do that? Could you to point these topics out to me, so as I can see that discussion?
Comment: I believe you also need $X > 0$ (or $X \geq 0$ in the continuous case) for that result to hold...
Comment: Have you tried to use Taylors formula with an integral remainder?
Answer: You are right to be skeptical of this approach. The Taylor series method does not work in general, although the heuristic contains a kernel of truth. To summarize the technical discussion below,

Strong concentration implies that the Taylor series method works for nice functions
Things can and will go dramatically wrong for heavy-tailed distributions or not-so-nice functions

As Alecos's answer indicates, this suggests that the Taylor-series method should be scrapped if your data might have heavy tails. (Finance professionals, I'm looking at you.)
As Elvis noted, key problem is that the variance does not control higher moments.  To see why, let's simplify your question as much as possible to get to the main idea.

Suppose we have a sequence of random variables $X_n$ with $\sigma(X_n)\to 0$ as $n\to \infty$.
Q: Can we guarantee that $\mathbb{E}[|X_n-\mu|^3] = o(\sigma^2(X_n))$ as $n\to \infty?$

Since there are random variables with finite second moments and infinite third moments, the answer is emphatically no.  Therefore, in general, the Taylor series method fails even for 3rd degree polynomials. Iterating this argument shows you cannot expect the Taylor series method to provide accurate results, even for polynomials, unless all moments of your random variable are well controlled.
What, then, are we to do? Certainly the method works for bounded random variables whose support converges to a point, but this class is far too small to be interesting.  Suppose instead that the sequence $X_n$ comes from some highly concentrated family that satisfies (say)
$$\mathbb{P}\left\{ |X_n-\mu|> t\right\} \le \mathrm{e}^{- C n t^2} \tag{1}$$
for every $t>0$ and some $C>0$. Such random variables are surprisingly common. For example when $X_n$ is the empirical mean
$$ X_n := \frac{1}{n} \sum_{i=1}^n Y_i$$
of nice random variables $Y_i$ (e.g., iid and bounded), various concentration inequalities imply that $X_n$ satisfies (1). A standard argument (see p. 10 here) bounds the $p$th moments for such random variables:
$$ \mathbb{E}[|X_n-\mu|^p] \le \left(\frac{p}{2 C n}\right)^{p/2}.$$
Therefore, for any "sufficiently nice" analytic function $f$ (see below), we can bound the error $\mathcal{E}_m$ on the $m$-term Taylor series approximation using the triangle inequality
$$ \mathcal{E}_m:=\left|\mathbb{E}[f(X_n)] - \sum_{p=0}^m \frac{f^{(p)}(\mu)}{p!} \mathbb{E}(X_n-\mu)^p\right|\le \tfrac{1}{(2 C n)^{(m+1)/2}} \sum_{p=m+1}^\infty |f^{(p)}(\mu)| \frac{p^{p/2}}{p!}$$
when $n>C/2$.  Since Stirling's approximation gives $p! \approx p^{p-1/2}$, the error of the truncated Taylor series satisfies
$$ \mathcal{E}_m = O(n^{-(m+1)/2}) \text{ as } n\to \infty\quad \text{whenever} \quad \sum_{p=0}^\infty p^{(1-p)/2 }|f^{(p)}(\mu)| < \infty \tag{2}.$$
Hence, when $X_n$ is strongly concentrated and $f$ is sufficiently nice, the Taylor series approximation is indeed accurate. The inequality appearing in (2) implies that $f^{(p)}(\mu)/p! = O(p^{-p/2})$, so that in particular our condition requires that $f$ is entire.  This makes sense because (1) does not impose any boundedness assumptions on $X_n$.
Let's see what can go wrong when $f$ is has a singularity (following whuber's comment). Suppose that we choose $f(x)=1/x$.  If we take $X_n$ from the $\mathrm{Normal}(1,1/n)$ distribution truncated between zero and two, then $X_n$ is sufficiently concentrated but $\mathbb{E}[f(X_n)] = \infty$ for every $n$.  In other words, we have a highly concentrated, bounded random variable, and still the Taylor series method fails when the function has just one singularity.
A few words on rigor.  I find it nicer to present the condition appearing in (2) as derived rather than a deus ex machina that's required in a rigorous theorem/proof format.  In order to make the argument completely rigorous, first note that the right-hand side in (2)  implies that
$$\mathbb{E}[|f(X_n)|] \le \sum_{i=0}^\infty \frac{|f^{(p)}(\mu)|}{p!} \mathbb{E}[|X_n-\mu|^p]< \infty$$
by the growth rate of subgaussian moments from above.  Thus, Fubini's theorem provides
$$ \mathbb{E}[f(X_n)] = \sum_{i=0}^\infty \frac{f^{(p)}(\mu)}{p!} \mathbb{E}[(X_n-\mu)^p]$$
The rest of the proof proceeds as above.
Comment: @whuber You are correct; you'll need the support of $X$ to be in the ROC of the Taylor series, so in particular, $00$. You can also verify that (2) does not hold because any function that satisfies (2) also satisfies $\log (p! f^{(p)}(\mu) )/ p \to -\infty $, and hence $f$ has no singularities (its *entire*, per the link).
Comment: @gron You need two things: (1) ensure that your RV has support strictly within the ROC of the power series of log (i.e., $[0+\varepsilon, 2 \mu-\varepsilon]$ for $\varepsilon > 0$), and (2) make sure that the moments of the RV decrease fast enough that an error estimate for $\mathcal{E}_m$ above is finite. As for how to control the moments, you should ask a new question because it will take way too many characters (and I'm curious about new ways myself).
Comment: The "exact remainder" formula has one major issue: the point $\zeta$ in the remainder depends on $x$. This means that the remainder term is not, in fact, a 3rd degree polynomial. I've added some more discussion to your other question.
Comment: Many thanks for this nice answer! That's exactly the kind of thing I was looking for when I was writing that we need hypotheses on the decay of the density of $X_n$... (+1, of course) (and welcome!)
Comment: @whuber The inequality in particular implies that $f^{(p)}/p! = O( p^{-p/2})$, which means that $f$ is an [entire function](https://en.wikipedia.org/wiki/Entire_function#Properties). This makes sense because we didn't assume that $X_n$ is bounded.
Comment: FYI, I removed the postscript about the logarithm. From @whuber's comments, I realized that I misapplied the alternating series test. Apologies.
Comment: I made many, many typos in this.  If you see any more, please let me know so that I can get it fixed up.
Comment: Hi @MikeMcCoy, sorry for a long silence. I'm now returning to this problem and in process of reading through it I got trapped with the example of $1/x$. So I understand why $\mathbb{E}[X_n] = \infty$, but when I look back at $(2)$, I see that the series
$$
\sum_3^\infty \frac{p^{p/2}}{p! \cdot p!} = \sum_3^\infty \frac{p^{p/2}}{p^{2p-1}}
$$
converges (via the ratio test). Where I got to see things wrong? Thanks a lot!
Comment: @MikeMcCoy sure, silly mistake. Yes, so now I get why logarithm is also a "bad function".

Do you by any chance know any literature about super exponentially concentrated distributions (I need now to find the conditions to impose on the distributions to "overgo" the obstacles caused by the $log$ function)?
Comment: The second order Taylor expansion is given by $E[f(X)] = f(\mu) + \frac{1}{2} f''(\mu) E[(X-\mu)^2] + R_2(X)$ where the remainder $R_2(X) = \frac{f^{(3)}(\zeta)}{3!}E[(X-\mu)^3]$. Moments higher than the third moment do not appear in this formula at all, so it appears that we only need require the remainder $R_2(X)$ to be sufficiently smaller than the second moment in order for the Taylor approximation to be accurate. This conflicts with the statement that we need to control **all** moments.
Comment: So am I missing something, why would we need control of the higher moments when they are not used at all in the formula for the Taylor approximation of $E[f(X)]$ in my previous comment?
Comment: I have posted a question about this issue to try and get some clarification [here](https://stats.stackexchange.com/questions/511763/do-all-moments-of-a-random-variable-need-to-be-well-controlled-for-a-valid-2nd-o)
Answer: Although my answer will nowhere approach the level of mathematical sophistication of the other answers, I decided to post it because I believe it has something to contribute -although the result will be "negative", as they say.
In a light tone, I would say that the OP is "risk-averse", (as most people are, as well as science itself), because the OP requires a sufficient condition for the 2nd-order Taylor series expansion approximation to "be acceptable". But it is not a necessary condition.
Firstly, a necessary but not sufficient pre-requisite for the expected value of the Remainder to be of lower order than the variance of the r.v., as the OP requires, is that the series converges in the first place. Should we just assume convergence? No.
The general expression we examine is
$$ E\Big[g(Y)\Big] = \int_{-\infty}^{\infty}f_Y(y)\Big[\sum_{i=0}^{\infty}g^{(i)}(\mu)\frac{(y-\mu)^i}{i!}\Big]dy  \qquad [1]$$
As Loistl (1976) states, referencing Gemignani's "Calculus and Statistics" book (1978, p. 170), a condition for convergence of the infinite sum is (an application of the ratio test for convergence)
$$y-\mu < |y-\mu|<\lim_{i\rightarrow \infty}\left | \left(\frac {g^{(i)}(\mu)}{g^{(i+1)}(\mu)}(i+1)\right)\right| \qquad [2]$$
...where $\mu$ is the mean of the r.v. Although this too is a sufficient condition (the ratio test is inconclusive if the above relation holds with equality), the series will diverge if the inequality holds in the other direction.
Loistl examined three specific functional forms for $g()$, the exponential, the power, and the logarithm (his paper is in the field of Expected Utility and Portfolio Choice, so he tested the standard functional forms used to represent a concave utility function). For these functional forms, he found that only for the exponential functional form no restrictions on $y-\mu$ were imposed. On the contrary, for the power, and for the logarithmic case (where we already have $0 <y$), we find that the validity of inequality $[2]$ is equivalent to
$$y-\mu < \mu \Rightarrow 0 < y < 2\mu$$
This means that if our variable varies outside this range, the Taylor expansion having as expansion center the variable's mean will diverge.
So: for some functional forms, the value of a function at some point of its domain equals its infinite Taylor expansion, no matter how far this point is from the expansion center. For other functional forms (logarithm included), the point of interest should lie somewhat "close" to the chosen center of expansion. In the case where we have a r.v., this translates to a restriction on the theoretical support of the variable (or an examination of its empirically observed range).
Loitl, using numerical examples, showed also that increasing the order of the expansion before truncation could make matters worse for the accuracy of the approximation. We must note that empirically, time-series of observed variables in the financial sector do exhibit variability larger than the one required by the inequality. So Loitl went on to advocate that the Taylor series approximation methodology should be scrapped entirely, regarding Portfolio Choice Theory.
The rebound came 18 years later from Hlawitschka (1994). The valuable insight and result here was, and I quote

...although a series may ultimately
  converge, little can be said about any
  of its partial series; convergence of a series
  does not imply that the terms immediately
  decrease in size or that any particular term
  is sufficiently small to be ignored. Indeed, it
  is possible, as demonstrated here, that a
  series may appear to diverge before ultimately
  converging in the limit. The quality
  of moment approximations to expected utility
  that are based upon the first few terms
  of a Taylor series, therefore, cannot be determined
  by the convergence properties of
  the infinite series. This is an empirical issue,
  and empirically, two-moment approximations to the utility functions studied here
  perform well for the task of portfolio selection. Hlawitschka (1994)

By example, Hlawitschka showed that the 2nd-order approximation was "successful" whether the Taylor series converged or not, but he also verified Lotl's result, that increasing the order of the approximation may make it worse. But there is a qualifier for this success: In Portfolio Choice, Expected Utility is used to rank securities and other financial products. It is an ordinal measure, not cardinal. So what Hlawitschka found is that the 2nd-order approximation preserved the ranking of different securities, compared to the ranking stemming from the exact value of $E(g(Y)$, and not that it always gave quantitative results that where sufficiently close to this exact value (see his table A1 in p. 718).
So where does that leave us? In limbo, I'd say. It appears that both in theory and in empirics, the acceptability of the 2nd-order Taylor approximation depends critically on many different aspects of the specific phenomenon under study and the scientific methodology employed -it depends on the theoretical assumptions, on the functional forms used, on the observed variability of the series...
But let's end this positively: nowadays, computer power substitutes for a lot of things. So we could simulate and test the validity of the 2nd-order approximation, for a wide range of values of the variable cheaply, whether we work on a theoretical, or an empirical problem.
Comment: One way to see the condition $0 < y < 2\mu)$ is that the logarithm and power functions are only defined on the positive real line so the best you can hope for is a radius $\mu$ of convergence around $\mu$.
Answer: Not an actual answer, but an example to show that things are not so nice, and that extra hypotheses are needed to make this result true.
Define $X_n$ as a mixture between a uniform $U\left( \left[ -{1\over n} ; {1\over n} \right] \right)$ and a normal $\mathcal N({n \over n-1}, {1\over n})$, the uniform component being chosen with probability $1\over n$, and the normal with probability $1 -{1\over n} = {n-1 \over n}$. You have $E(X_n) = 1$ and its variance converges to $0$ when $n$ goes to infinity, as
$$E\left(X_n^2\right) = {1\over 3 n^2} \times {1\over n} + \left(\left({n \over n-1}\right)^2+{1\over n}\right)\times{n-1 \over n},$$
if I am not mistaking.
Now define $f(x) = 1/x$ (and $f(0) = 0$ or whatever). The random variables $f(X_n)$ are well defined but does not have an expected value, as
$$ \int_{-{1\over n}}^{1\over n} {1\over x} \mathrm dx$$
is not defined, no matter how big $n$ is.
My conclusion is that you clearly need hypotheses on either the global behavior of $f$ or – more likely, more elegantly – on the speed at which the density of $X_n$ decays when you're far from the expected value. I am sure that such hypotheses can be found in the classic literature (and even in textbooks), unfortunately my training was not in statistics and I still struggle with the literature myself... anyway I hope this helped.
PS. Isn't this example a counter-example to Nick's answer? Who's wrong then?
Comment: A more general statement of your argument is that $E\left[X^k\right]$ exists and is finite for $k=1,2,3$
Comment: Elvis, yes, we need a global condition. Essentially, the remainder has to behave nicely after it's weighted by the tails of the distribution. For something similar to your example that came up recently, see [here](http://stats.stackexchange.com/questions/70045/mean-and-variance-of-inverse-of-a-normal-rv/70052#comment135568_70052), [here](http://stats.stackexchange.com/questions/70045/mean-and-variance-of-inverse-of-a-normal-rv/70052#comment135639_70045) and [here](http://stats.stackexchange.com/questions/41896/varx-is-known-how-to-calculate-var1-x#comment135573_41905).
Comment: I think my above comment is not correct - what should be there is that the function $f(x)$ admits a Taylor Series expansion at the point $x=\mu$.  The example you provide, you have $f(x)=\frac{1}{x}$ which is not continuous at $x=0$.  I think this means that $f$ cannot be expanded in a Taylor series for your example.
Comment: It can be, at $\mu = 1$. Then there's the radius of convergence... May be you need an infinite radius of convergence?! That's a strong requirement.
Answer: This is not a complete answer, just a different way of arriving at the second order approximation.
I think the best way to go is to use Cauchy's mean value theorem, rather than work with the remainder term of a Taylor series.  If we apply it one time we have
 $$f(X)=f(\mu)+f'(\xi_1)(X-\mu)$$
for some $X\leq\xi_1 \leq \mu$ when $X \leq \mu$ or $X\geq\xi_1 \geq \mu$ when $X \geq \mu$.  We now apply the mean value theorem again to $ f'(\xi_1)$  and we have
$$ f'(\xi_1)= f'(\mu) + f''(\xi_2)( \xi_1-\mu)$$
for some $X\leq\xi_1\leq\xi_2\leq\mu$ when $X\leq\mu$ or $X\geq\xi_1\geq \xi_2 \geq\mu$ when $X\geq\mu$.  putting this into the first fomula gives
$$f(X)=f(\mu)+ f'(\mu) (X-\mu) + f''(\xi_2)( \xi_1-\mu) (X-\mu)$$
Note that this result only requires that $f$ is continuous and twice differentiable between $X$ and $\mu$.  However this applies only for a fixed $X$, and changing $X$ will mean a corresponding change in $\xi_i$.  The second order delta method can be seen as making the global assumption that $\xi_1-\mu=\frac{1}{2}(X-\mu)$ and $\xi_2=\mu$ over the entire range of the support of $X$, or at least over the region of high probability mass.


how to edit big images in android?

Question: I am trying to save small images from the original which I load from SDcard. Android doesn't allow me to create bitmap of bigger image sizes so I had to use options, doing so my images' resolution is altered so now how i am supposed to edit the original image??
And saving images, do I have the bitmap.compress option alone? If so, this reduces the quality of the image when i open and save it in JPG
And how can I display them independent of the screen???
Answer: I found a way to display big image in Android by using BitmapRegionDecoder. http://developer.android.com/reference/android/graphics/BitmapRegionDecoder.html
Comment: Simple link giving is not an answer, you should also write something about it.
Comment: Hello and welcome to Stackoverflow. Please read the guidelines for good answers: http://stackoverflow.com/questions/how-to-answer. One of the rules is to add context to links, instead of just pasting them. You should provide an answer that is valid without the need for the user to navigate to another side, but might wish to do so for more detail on the answer. That becomes a bigger problem when the links for some reason become invalid.

how to edit big images in android?


How do I add a new content status?

Question: What hook would I use to add a new content status under the path admin/content?  See image below.

I am trying to give my content type(s) a new status so that I can do some more filtering on it.  Specifically, I want to create a new "draft" status.  Any pointers on where to start and API's to look at?  Or are there any existing modules that do this that I can take a look at the code?
EDIT: I would like the user filling out the form to set their own status as 'draft'.  The reason is that I am using the save_draft module (http://drupal.org/project/save_draft) that allows the user to save values in a form without submitting it.  The problem is that Drupal considers the node to be unpublished.  My editors should not see the content type if it is in "draft" mode, but since Drupal saves it as "unpublished", my editors see this and they are not sure if the content is ready to be edited or if the end user is just saving their work and finishing it up later.
The save_draft module gives me 2 buttons (Publish and Save as Draft).  This allows the user to select their own status.  I would also like this new "save as draft" status to be searchable/sortable so my editors could find out who hasn't finished submitting their unfinished draft, if need be.
Comment: the purpose od the save draft module is just to make the  unbublish option a bit more friendly. So what it does it to save the content as unpublished and nothing else. so I would suggest to go with either workflow are with workbench as mentioned by Mohammed Ali Akbari. Both would do what you want. There are lots of option to control the workflow of content creation.
Comment: Thanks, Mohammed Shemeem.  I took a quick look at the Youtube video on Workbench and it looks like it could work.  I will give it a try and report back.
Comment: Unfortunately the Workflow and Workbench modules is not able to do what I want since I want the user to self select their status to let my reviewers know if it is done or not.  THe workflow and workbench module auto sets the status for them upon clicking the submit button (e.g. "draft"), which is not what I want.
Answer: You should look at the workflow module. This one is used to create additional work flows and status in your content authoring.
Comment: I added some more details about in my EDIT section.  I don't think the workflow module can allow the user to set their own status.
Answer: In addition to Workflow you can use Workbench with Workbench Moderation
Its very difficult to alter Built-in Content administration page (admin/content), the filters hard-coded in modules/node/node.admin.inc
<code>/**
 * List node administration filters that can be applied.
 */
function node_filters() {
  // Regular filters
  $filters['status'] = array(
    'title' => t('status'),
    'options' => array(
      '[any]' => t('any'),
      'status-1' => t('published'),
      'status-0' => t('not published'),
      'promote-1' => t('promoted'),
      'promote-0' => t('not promoted'),
      'sticky-1' => t('sticky'),
      'sticky-0' => t('not sticky'),
    ),
  );
  ....
  return $filters;
}
</code>
Comment: Will the workbench module allow the user filling out the form to set their own workflow status?
Answer: I was able to solve my own problem by creating a field that sets the status (e.g. "draft", "to be reviewed").
I then used jQuery to hide this field.  Also in jQuery, I made it so that when the user clicks on the "Publish/Save" button, then it sets the value of this hidden field to "to be reviewed".  If the user hits the "Save as draft" button, then this hidden field is by default set to "draft".
In other words, my 2 buttons will set the value of this hidden field and the user can decide if their submission is ready to be reviewed or not, and my editors will know based on this hidden field on the form.
Answer: you can use Custom Publishing Options module for add some status to your content

How do I add a new content status?


SQL IN query behavior

Question: Executing the following SQL statement;
select  '2312' in ('2312,254,2111') as result1, 2312 in ('2312,254,2111') as result2
I am getting the following result
+---------+---------+
| result1 | result2 |
+---------+---------+
|       0 |       1 |
+---------+---------+

I would expect the opposite result. Having result1 to be true and result2 to be false. Could someone explain why?
Answer: Using IN() with a CSV-string as parameter is not supported.
It should be IN ('2312','254','2111') instead of IN ('2312,254,2111')
The reason for the observed behaviour is the implicit type conversion from string to integer that is happening here.
Observe:
SELECT 2312 IN ('2312,254,2111') -- result: 1
SELECT 2312 IN ('254,2312,2111') -- result: 0 -- interesting
SELECT 2312 = '2312,254,2111'    -- result: 1 << see ??
SELECT 2312 = '254,2312,2111'    -- result: 0

Only the first number in the string is relevant. The rest is ignored due to the implicit type conversion.
Also,
SELECT '2312' in ('2312,254,2111') -- result: 0

is false, because no type conversion happens here. '2312' does not equal the only string '2312,254,2111' in the value list and hence the IN() operator return false.
If you use a list of values for IN() instead of a CSV-string, everything works als expected:
SELECT
   2312  IN ('2312','254','2111')  -- result: 1
, '2312' IN ('2312','254','2111')  -- result: 1
,  254   IN ('2312','254','2111')  -- result: 1
, '254'  IN ('2312','254','2111')  -- result: 1
,  2312  IN (2312,254,2111)        -- result: 1
, '2312' IN (2312,254,2111)        -- result: 1
,  254   IN (2312,254,2111)        -- result: 1
, '254'  IN (2312,254,2111)        -- result: 1

From the manual:

Implicit type conversion may produce nonintuitive results:
mysql> SELECT 'a' IN (0), 0 IN ('b');
        -> 1, 1

In both cases, the comparison values are converted to floating-point values, yielding 0.0 in each case, and a comparison result of 1 (true).
Comment: Yes, when comparing in an integer context, a string like `'2312,254,2111'` is cast to its integer value, which is based on any leading digits, and ignores subsequent characters like the `,` and anything that follows. Whereas comparing it to a string `'2312'` is a string-to-string comparison, so it compares the whole string verbatim.

SQL IN query behavior


Método mais simples para alterar parâmetros GET

Question: Não entendo muito de programação e preciso inserir no meu site uma função simples.
O caso é o seguinte:
Um usuário clica em um link como:
http://site.com.br/titulo-da-noticia?id=1

Eu preciso remover a parte ?ida=1 da URL e adicionar, por exemplo, ?parametro=1&a=2, carregando a url:
http://site.com.br/titulo-da-noticia?parametro=1&a=2

Alguém pode gerar este código ou mostrar qual o caminho mais rápido?
Muito obrigado!
Abs.
Comment: Posta o seu código. Isso é bem simples, mas sem o código não adianta.
Comment: É um site em wordpress
Answer: Já que o site é feito no WP, vai em no painel (wp-admin) e Configurações > Links Permanentes.
Você pode escolher entre as configurações comuns ou personalizar de acordo com suas preferências.


how to get php/apache to schedule tasks using the AT daemon

Question: is there a way to enable a php script (apache user) to schedule a task using the AT daemon?
I would like to pipe the task to the atd from my php script like this:
exec( "echo 'date > /some/dir/date.txt' | at now + 1 minute > /dev/null &" );

when I run the php script from the command line (root) everything is fine,
however, when I run the script via a http request the at job is never created.
I am aware that apache doesn't have shell access, is there any work around available?
Answer: exec() does not use a shell so you can't use pipes etc..
Use backticks, shell_exec() or system() instead to make it work.
shell_exec("echo 'date > /some/dir/date.txt' | at now + 1 minute");

Comment: tnx ThiefMaster, using shell_exec my error_log shows "shell_exec() has been disabled for security reasons". I'm not sure what the implications would be if I were to enable it.
Comment: Usually shared hosting companies disable it via php.ini because they don't want php applications to run shell commands. Usually they have really poor security (all customers in the same user group and thus only restricted by things like open_basedir from nosing around other customers' files - and shell commands obviously circumvent that restriction)
Answer: If the account your script runs under (Apache or a cgi user) does not have shell access, at will not work, because the disfunctional shell (like /bin/bad) will be tried to run the command later on.
Check if you can exec anything:
$output = array();
exec('whoami',$output);
print_r($output);

If this does work, you could try something like (pseudocode):
write "date > /some/dir/date.txt" in /tmp/atcmd
exec("/usr/bin/at -f /tmp/atcmd now + 1 minute");

So you need:
- A shell to run the command in, later on
- Permision inside php to run the exec function
- No problems created by /etc/at.{deny,allow}
This basically means (I guess) you need a PHP that runs in php-cgi switched to a "real" account.
The only other solution I can think of right now is to create the script you need on the webserver, and to run the at job on a machine you do have access to, using wget.
echo "wget -q http://myserver.mydomain/my/atscript"  | at now + 1 minute

People actually use such constructs for cron-like behaviour. Hey, I might even setup a service for stuff like that on my hosting service :-)


If $m(Z)=0,W=\{x^2: x\in Z\}$, how to prove that $m(W)=0$?

Question: Here is my question,
If $m(Z)=0, W=\{x^2: x\in Z\}$, how to prove that $m(W)=0$? Here $m$ is the Lebesgue measure.
I have no idea about it, can anyone give me a hint or answer?
Comment: Note $f (x) = x^2$ is locally Lipschitz and that the enumerable union of sets with measure zero has measure zero.
Answer: Denote $g : x \mapsto x^2$. We have to prove that $m(g(Z)) = 0$. We have $W = \cup_n W_n$ where $W_n = W \cap [n,n+1)$ where $n \in \mathbb N$.
It is enough to prove that for all $n \in \mathbb N$, $m(W_n) = 0$. And this is the case as $W_n = g([\sqrt n, \sqrt{n+1})) \cup g([-\sqrt{n+1}, - \sqrt n))$ and $g$ is absolutely continuous on $[\sqrt n, \sqrt{n+1}), [-\sqrt{n+1}, - \sqrt n)$ as it is Lipschitz continuous on those bounded intervals.
Comment: I got it, thanks a lot!


How to rename a table in Athena?

Question: probably a very trivial question but I'm not sure about this and also don't want to lose the table. How do I rename a table in Athena?
Database name - friends
table name - centralPark
desired table name -centralPerk

Answer: you can't!
see the list of unsupported DDL in Athena.
what you can do is to make a new table using select:
CREATE TABLE centralPark
AS SELECT * FROM centralPerk
WITH DATA

and drop the old table:
DROP TABLE IF EXISTS centralPerk

Comment: What is with data?
Comment: @DawsonSmith to bring data in that table to the new table  if you have any. you can use '`with no data` , so only you make the empty table
Comment: Can you help me with a new question I posted?
Comment: Works but if the table is partitoned it doesn't keep the partitions, any idea for this?
Comment: @dduque you have to create table with  option `WITH partitioned_by = ARRAY[ col_name[,…] ]` and then list the column names in the select statement that the partition(s) is based on . See docs  for `using select` in my answer
Comment: I'm not sure why this answer was accepted.
This might be VERY long, VERY expensive, might fail, and eventually you might get table with wrong structure.
This command creates new Parquet table with a single partition with the data copied from original. If originally you had 50TB table with 2000 partitions this will probably fail, or create inefficient table.
Answer: I know this is old, but for new people that seek the answer I'll write anyway.
There's no single command but you can do it easily with 2, either manually or programatically.
You do
show create table `your_table`

Then copy-paste the "create external table" command to the editor, replace table name and run.
If the table includes non-projection partitions, you will also need to run this to detect and load your partitions.
MSCK REPAIR TABLE `your_table`

You can do the same programmatically by doing simple regexp replace of the table name and rerun. It's more complicated of course than running "CREATE TABLE ... AS SELECT ..." but it's immediate, free and will promise the same structure pointing to the same place in S3.
Answer: Using a CTAS query is effective, but I found it to be quite slow. It needs to copy all the files.
But you don't need to copy the files. You can create a new table directly in the Glue catalog and point it at the existing files. This works in seconds or less.
If you're using Python, I highly recommend the awswrangler library for this kind of work.
import awswrangler as wr

def wrangler_copy(db, old_name, new_name):
    wr.catalog.create_parquet_table(
        db,
        new_name,
        path=wr.catalog.get_table_location(db, old_name),
        columns_types=wr.catalog.get_table_types(db, old_name),
        # TODO: partitions, etc
    )

And then drop the old table if you like.
DROP TABLE IF EXISTS <old_name>

How to rename a table in Athena?


How to run Redis on Docker using docker-compose.yml?

Question: Have found an official Spring tutorial about the application developing that uses Redis keystore is described but don't know almost nothing about Docker and don't really want to learn it. The app's source code contains docker-compose.yml file with multiple Redis oriented settings and Spring docs are say:

There is a docker-compose.yml file in the source code in Github which
  you can run really easily on the command line with docker-compose up.

But it seems to be not that easy and Docker docs are too complicated.
Have installed Docker and deployed Redis there:
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
81cbeeb08153        redis               "docker-entrypoint.sh"   22 hours ago        Up 21 minutes       6379/tcp            Server

The docker-compose.yml
redis:
image: redis
ports:
  - "6379:6379"

What's next? How to import this in Docker Redis?
I'm trying to up Redis on the Windows machine to let my simple localhost app finally work.
Answer: Do you have Docker Compose installed? If yes, just run docker-compose up - it will start redis image and make it listen on a correct port.
Alternatively, you will have to start redis manually and correctly expose specified port.
Comment: Yeah it's installed. But where have I place the `docker-compose.yml` before I run this command?
Comment: Usually, it's in project's root directory so you can start something right after you pull it.
In your specific case, `docker-compose.yml` does not rely on other files so it's fine anywhere it is. You `cd` to that directory and then `docker-compose up`.
If you've run redis image without docker-compose on the same port you'd need to stop it to avoid conflict.


Restrict admin to search specific content types

Question: 1.Using search configuration module I have restricted the user role to search particular content type, this worked for me. But when I tried to apply the same approach to administrator this did work. I have unchecked the Admin bypass (primary user with id 1) the option too.
Is there a way to restrict the admin too to search specific content ?
2.And my second issue is about pagination, using the same module pagination is successfully applied, but for admin it is default set to 10 items per page, can I alter this?
Answer: I don't think so; you would have to either:

Create a new role and then limit search configuration for that node.
Use Views to simply create another page for administrators where they can search the title/body, limited to a certain type.
Install Views Bulk Operations and Admin Views which will then allow you to change the default administration pages, and limit the displayed content types if you want.
If you need more complex search, look into Search API with Search API Database Search. These modules integrate with Views and you will be able to cover more complex scenarios.

Solutions 2, 3 and 4 use Views, so you will be able to change the number of items displayed per page.
Hope this helps.


Customer satisfaction over time in R

Question: As a side project I am currently working on determining customer satisfaction over time for quite a large company. We have over 100,000 records in our dataset which need to be analysed. The dataset looks like this:
<code>╔════════╦════════╦═════════╗
║ CustID ║ Rating ║ Loyalty ║
║ 3001   ║ 5      ║ 1       ║
║ 3001   ║ 4      ║ 2       ║
║ 3001   ║ 4      ║ 3       ║
║ 3001   ║ 5      ║ 4       ║
║ 5214   ║ 3      ║ 1       ║
║ 5214   ║ 5      ║ 5       ║
║ 5214   ║ 2      ║ 15      ║
║ 5214   ║ 4      ║ 16      ║
╚════════╩════════╩═════════╝
</code>
A customer can rate a product from 1 to 5. The loyalty is the amount of products the customer purchased from us. Please note: there are a lot of gaps in this data as customers don't always respond. As you can see, customer 5214 only responded 4 times while he purchased more than 16 products.
The research question is:
Do customers become happier with our products as they buy more?
So I looked at both panel data and time-series cross-sectional data analysis, but this seems like the wrong direction to go because it gives detailed information about each specific customer (see here: http://www.princeton.edu/~otorres/Panel101R.pdf)
What we want is an overview of what customers generally speaking think. So I am looking for the best way to achieve this. Preferably in R, but this is of course not necessary. A link to a theory would also be incredibly helpful!
I hope I have provided enough background information on the case. If not, please do let me know!
Comment: The title of your question refers to this as being an analysis "over time" but you presently do not have time as a variable in your data.  May we presume that each record of a rating and loyalty (a strange name for the number of items bought) also comes with a date/time?
Answer: I assume you use have information about products customers purchased, so there are few things one can calculate:

Customer retention rate quarterly or maybe annually, depending on your company needs. This is a general picture that can be used as some sort of simple "scoring" mechanism.
I would also suggest getting total amount customer paid, and looking at correlation between how much customer paid per order vs. rating and loyalty.
To do additional analysis you can use those 2 metrics from above and cluster customers into several groups to suggest the strategy to increase "loyalty", that is conversion of customer into profit.
And, finally, simplest thing is to just do a scatter plot of rating vs loyalty and try to see either linear or some geometric regression. There might be some sort of regression model that could fit the data. Just remember, if there is a big difference between data, try to use logarithmic scale rather than normal values.
Comment: Thank you for your comment. I will definitely look into customer retention. The amount the customer paid isn't really interesting information, as all the customers pay the same (it's a subscription customers can cancel on a monthly basis). I didn't think of doing the regression model on a log scale. Will look into this as well. Thanks for the help!
Answer: I would check the relationship between Rating and Loyality over time.
Either by a simple scatter plot and check visually for a correlation or you could also calculate it.
Next, I would run a linear model by Rating ~ time + Loyalty and check the summary/statistics. Hence, I don't think a linear/random model is wrong, here. A random model makes only sense when you can group/summarize all the customers somehow, but a linear model without random effects will provide general information.
Comment: this question is six years old..   why did it show up in my feed..? ^^

Customer satisfaction over time in R


Prove that $243^{2020} \equiv 1 \pmod 5$

Question: I was supposed to use that  $a^4 \equiv 1 \pmod 5$, but that's not helping me, specifically, I don't know how to prove that $243^{505}$ is not as $5k, k \in \mathbb{Z}$.
Comment: Apply the first corollary in the linked dupe with $\,e = 4.\ \ $
Answer: $$243^{2020}=(243^4)^{505}\equiv (3^4)^{505}\equiv 1^{505}\equiv 1\pmod 5 $$
Comment: Please strive not to add yet more diupe answers to FAQs.


Firefox SSL security warning

Question: We have a weird issue on one of our sites.
The site in question forces https for all traffic but it throws a security error for some users. Different browsers, settings and network environments make it almost impossible to determine the cause of why the user is getting the error.
But, and here comes the question: when I try the same site on my machine, I get the security error in Firefox (v22) and not in Chrome or Safari, or in a VM running Windows 7 and IE9.
www.domain.com uses an invalid security certificate.
The certificate is not trusted because no issuer chain was provided.
(Error code: sec_error_unknown_issuer)
Is this an issue on the back-end and is Firefox the only browser failing because of it, or is it a problem in Firefox, thus making it a local browser problem?
Any feedback would be highly appreciated!
Comment: please check if this error comes with the sites having https(like gmail,odesk,facebook) and also tell me that ,its just case of firefox alone,the same websites working fine in other browsers?
Comment: Happens only for this site and the same site works fine in other browsers.
Answer: Are you sure that you're including the certificate chain provided by the issuer?  Many issuers these days use intermediate certificates to establish the chain of trust back to the root CA.  Different browsers have different sets of CAs built in.
Check the instructions provided by your certificate issuer and make sure you're actually serving any intermediate certificates.

Firefox SSL security warning


Invalidcastexception error while using set or bag cascading

Question: I have a parent class which contains a child object.
I am using set to save the child object when parent is saved.
I m not sure whether set is used for just saving a child object.
And im getting this error below
System.InvalidCastException: Unable to cast object of type 'x' to type 'Iesi.Collections.ISet'
Does anybody knows the solution.

<class name="Customer" table="Customers" lazy="false" dynamic-update="true">

  <id name="Id" column="CustomerID" type="Guid" >
  <generator class="guid.comb" />
  </id>

  <property name="Name" column="CustomerName" type="String" length="50" not-null="true" />

 <bag  name="users" table="Users" cascade="all-delete-orphan">
  <key column="CustomerID" />
  <one-to-many class="User" />
</bag>

<set name="customerPreferences" table="Preferences" cascade="all-delete-orphan">
  <key column="CustomerID" />
  <one-to-many class="CustomerPreferences" />
</set>

</class>

Entity class:
public class Customer
    {
        private Guid _id;
        private string _name;
        private IList _users = new List();
        private CustomerPreferences _customerPreferences;
    public Guid Id
    {
        get { return _id; }
        set { _id = value; }
    }

    public string Name
    {
        get { return _name; }
        set { _name = value; }
    }

    public CustomerPreferences CustomerPreferences
    {
        get { return _customerPreferences; }
        set { _customerPreferences = value; }
    }

    public IList<User> Users
    {
        get { return _users; }
        set { _users = value; }
    }

Comment: Can you include your mapping file and entity?
Comment: Yeah i have added the entity and im getting error in customerpreferences not users
Answer: Your mapping files look fine. So I am guessing that the problem is to do with your implementation of the CustomerPreferences class.
The set attribute in the mapping file defines a property that (when mapped to your entity) must implement the Iesi.Collections.Generic.ISet<T> interface. Apparently you can also use the standard ICollection<T> interface with NHibernate 2.1 but I have never tried this.
Therefore if your CustomerPreferences class does not implement the ISet<T> interface then you will get an InvalidCastException.


Octopack fails to build when I add dependencies

Question: Octopack fails to build with the following xml
<?xml version="1.0"?>
<package >
  <metadata>
    <id>WebCrawlerMVC</id>
    <version>220.127.116.11</version>
    <title>Web Crawler</title>
    <authors>lardern</authors>
    <owners>Nostrum Group</owners>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <description>packaging webcrawler mvc</description>
    <releaseNotes>no release notes</releaseNotes>
    <copyright>Copyright 2013</copyright>
  </metadata>
   <dependencies>
        <dependency id="Ninject" version="22.214.171.124" />
        <dependency id="HtmlAgilityPackHelpers" />
    </dependencies>
    <files src="C:\Users\lardern\Documents\Visual Studio 2010\Projects\WebCrawlerMVC\packages\Ninject.220.127.116.11\lib\net40\Ninject.dll"/>
    <files src="C:\Users\lardern\Documents\Visual Studio 2010\Projects\HtmlAgilityPack\HtmlAgilityPack\HtmlAgilityPackHelpers\bin\Release\HtmlAgilityPackHelpers.dll" />
</package>

the error i receive is

Error 2   The command ""C:\Users\lardern\Documents\Visual Studio
  2010\Projects\WebCrawlerMVC\WebCrawlerMVC..\packages\OctoPack.1.0.111\tools\NuGet.exe"
  pack "C:\Users\lardern\Documents\Visual Studio
  2010\Projects\WebCrawlerMVC\WebCrawlerMVC\WebCrawlerMVC.nuspec"
  -OutputDirectory "bin"  -basePath "C:\Users\lardern\Documents\Visual Studio 2010\Projects\WebCrawlerMVC\WebCrawlerMVC" -Version "22.214.171.124"
  -NoPackageAnalysis" exited with code 1.   WebCrawlerMVC

I'm not quite sure what this means, if I don't have the dependencies added, it builds in release fine otherwise it comes back with this error.
Edit: I edited the code above, thanks for spotting the mistake adam, but the error still occurs.
      Also took out each dependency to see if one was causing the error, failed both times.
Answer: If you view your full build log, you should be able to find more information than just the error line above - NuGet.exe is probably complaining about them.
However, Octopus packages should not include dependencies anyway.

Another NuGet feature that doesn't translate well to Octopus is dependencies. NuGet has a convenient feature for tracking dependencies between packages. This is used when, for example, a package like NHibernate depends on a package like log4net. NuGet will install NHibernate, resolve the dependency, and install log4net. It's important to note that NuGet will install them to two separate folders on disk:
packages/NHibernate.x.y.z
packages/log4net.x.y.z

While this makes sense for libraries referenced via Visual Studio, it doesn't make sense in Octopus, for the same reasons as above. For example, if your Octopus application package had a dependency on log4net, on disk you would have:
packages/YourApp.x.y.z
packages/log4net.x.y.z

At runtime, the executables or ASP.NET site under YourApp would try to invoke a method in log4net, and fail because the DLL can't be found. While this could theoretically be solved using probing paths in your configuration file, in practice it's better to avoid this approach.
For this reason, Octopus NuGet packages cannot have dependencies.
Instead, any libraries you need to use should be bundled inside of your package in the standard binaries folder
Answer: It looks like there is a > missing at the end of the second to last line. 8)
Edit: ok, maybe it wasn't that. But I found this link describing a problem with a space in the path.
Maybe that is relevant?
Answer: https://docs.nuget.org/create/nuspec-reference#specifying-dependencies

The dependencies element is a child element of the metadata element

I suppose it is why you had the error while running nuget pack


Como manter o nome do arquivo codificado (UTF-8) ao descompactar em R?

Question: Estou com problemas para acessar os arquivos de uma pasta compactada. Os nomes dos arquivos contém acentos, mas ao descompactar os nomes são alterados de acordo com a configuração de cada computador que abre e executa o código. Existe alguma maneira de manter o nome do arquivo codificado ao descompactar ou de conseguir acessar os arquivos dessa pasta? Preciso fazer tudo via código do R (versão 4.0.2) no windows 10. Ao executar o seguinte código:
dir.create("./federal_highway")
setwd("./federal_highway/")
temp <- tempfile()
download.file(URLencode("https://servicos.dnit.gov.br/dnitcloud/index.php/s/oTpPRmYs5AAdiNr/download"), destfile = "./original/Repositorio.zip", method="auto", mode="wb")
unzip("./original/Repositorio.zip")
zipfiles <- list.files(path = "./Repositório/SNV Bases Geométricas (2013-Atual) (SHP)", full.names = T, pattern = "*.zip")

Ele roda sem erros, mas eu recebo a seguinte mensagem:
Warning message:
In list.files(path = "./Repositório/SNV Bases GeomÃ©tricas (2013-Atual) (SHP)",  :
  unable to translate './Repositório/SNV Bases GeomÃ©tricas (2013-Atual) (SHP)' to native encoding

E zipfiles retorna "character(0)"
Answer: Você pode usar o comando dir() para ir montando o o caminho do arquivo da seguinte forma:
setwd("./federal_highway/")
repo <- dir("./sf_all_years_original")
repozipfiles <- dir(paste0("./sf_all_years_original/",repo[1]))
SNV <- repozipfiles[3]
dir_repo <- paste0("./sf_all_years_original/",repo[1],"/",SNV,"/")
setwd(dir_repo)

a <- getwd()
zipfiles <- dir(a)

Descompactar as pastas zipadas
for(i in 1:length(zipfiles)){
  unzip(zipfiles[i])
}

Listar somente arquivos shp, por exemplo
raw_shapes <- list.files(".", pattern = ".shp", recursive = T, full.names = T)
raw_shapes <- raw_shapes[raw_shapes %nlike% '.xml']


Como fazer este sql

Question: Eu tenho a seguinte tabela:

Eu utilizo um comando SELECT, tipo:
SELECT estudante, conceito_1, conceito_2 FROM estudante WHERE etapa = '1ª'

Como eu poderia fazer um único select para pegar a 2ªs e 3ªs etapas também?
E gostaria que o SQL gerasse:
Comment: O objetivo da consulta é mostrar tudo menos a etapa?
Answer: Imaginando que sua tabela chama-se Estudante, isto pode ser realizado com alguns subqueries:
select e.estudante,
  a11.conceito_1, a11.conceito_2, a11.etapa,
  a22.conceito_1, a22.conceito_2, a22.etapa,
  a33.conceito_1, a33.conceito_2, a33.etapa
FROM Estudante e
JOIN
  (
        SELECT estudante, conceito_1, conceito_2, etapa
        FROM Estudante where etapa = '1a'
    ) AS a11
ON e.estudante = a11.estudante
JOIN
  (
        SELECT estudante, conceito_1, conceito_2, etapa
        FROM Estudante where etapa = '2a'
    ) AS a22
ON e.estudante = a22.estudante
JOIN
  (
        SELECT estudante, conceito_1, conceito_2, etapa
        FROM Estudante where etapa = '3a'
    ) AS a33
ON e.estudante = a33.estudante
GROUP BY e.estudante

Você pode ver funcionando aqui neste link: http://sqlfiddle.com/#!9/f8e0e/1
Não consegui pensar ainda em uma consulta que funcione independente do número de etapas, sem necessidade de adicionar uma nova subquery manualmente.
Comment: Na hora de digitar eu me esqueci do FROM tabela. O problema que eu quero pegar o conceitos de todos os estudantes e depois manipulá-los. Ex.: Maria, conceito_1 (1ª etapa), conceito_1 (2ª etapa), conceito_2 (1ª etapa), conceito_2 (2ª etapa), assim por diante. Como eu poderia pegar todos os estudantes e conceitos e manipulá-los?
Comment: @LuisSouza, infelizmente não consegui entender... edite a sua pergunta ilustrando melhor o que gostaria de ter como resultado.
Comment: Eu editei a pergunta. Eu penso que agora dá para entender melhor.
Comment: Ops, está funcionando mais ou menos, vou corrigir.
Comment: Agora está correto.
Comment: Perfeito! Obrigado!
Comment: @LuisSouza , legal! :D Poderia marcar a minha resposta como a solução? Veja como fazendo um [tour](http://pt.stackoverflow.com/tour)! Obrigado!

Como fazer este sql


home/username directory chmod 750 best for security but are some errors

Question: in directory /home/username/public_html
I have a website with CMS.
When I change chmod of /home/username directory to 750, my site doesn't work. I believe my CMS have no access to it.
Is there any way to improve /home/username directory security by chmod, so the website in subdirectory still works well?
Comment: Do you have any interesting thing in the logs, the one of the CMs, or `/var/log/syslog` ?
Answer: If you absolutely need your website to be in /home/$user/..., then you could change the permission of your directory like this :
Create a group for this directory
groupadd home_user

add your user, and the user executing the CMS (e.g. wordpress) inside this group.
usermod -A -g $user  home_user
usermod -A -g wordpress  home_user

Then change the permission of your home folder:
chown $USER:home_user /home/$USER
chmod 770 /home/$USER # or 750 if it is enough

Then it should work.

BUT !
Having your website in the home folder, this is bad practice.
It would be better to have it under /var/www/ or /www/ or /opt/www/
Then it is easier to segregate. You give the authorisation to this folder ONLY to the user which is running the website. And he has no other permission.
Comment: Sometimes services save some config files under /home/user folder. When change chmod to 750 of the user dir, does it break something too?
Comment: Not that i know of. Who is gonna write to your home : The softare you may launch (they have your UID),  the system services (they are root). So I don't see any reason for that to not work
Comment: no service writes by default to any home folders. tell us more about this kind of story


Como fazer para o MySQL aceitar um numero do tipo 1.457,40?

Question: Estou desenvolvendo uma aplicação que precisar exportar automaticamente planilhas excel para o banco de dados. Estou usando o MySQL, porém os dados da planilha não estão todos bem formatados do jeito que o banco aceita.
Já setei como utf8_general_ci, todas as tabelas com utf8 e não resolveu. O problema esta quando eu leio um numero do tipo 1.478,45 e ele não consegue armazenar em uma variável do tipo double ou float.
Como proceder?
Comment: E nem deveria ser armazenado neste formato. No DB o dado deve sempre que possível ser armazenado pelo seu valor, não pela sua "estética". Armazenar número com separador de milhar no DB é _quase_ como querer "armazenar um texto em negrito". O DB não tem esses conceitos visuais.
Comment: Porque o número não segue esse formato. Ele só tem divisão do ponto decimal (não tem o separador de milhar). E normalmente o separador decimal é "`.`" e não "`,`" exceto se estiver configurado com o locale adequado
Answer: Na sua linguagem de programação preferida, faça algo como:
var numero = foo; // onde "foo" é o número quebrado;
numero = (numero + '').replace('.', '').replace(',', '.');

Answer: Cara como nosso amigo disse antes, no mysql ele utiliza o padrão americano de moeda que segue este padrão '15,123.50' .
Ao contrario do padrão brasileiro que é '15.123,50' que é ao contrario como vc viu.
porem no BD mysql alem de ser padrão americano ele nao conta as casas de milhares ficando assim '15123.50' .... sendo assim vc deve fazer uma forma de colocar neste padrão na linguagem utilizada por vc !!
Em PHP eu faço da seguinte forma amigo :
valor = valor.replace(".", "") ; //IMPORTANTE - Tirar os pontos (a cada 3 dígitos, exemplo, 1.355.000,00)
valor = valor.replace(",", ".") ; // Depois remover a virgula da casa decimal por ponto, ficando no padrão d mysql


Add Taxonomy Term automatically based on term from user's account

Question: I have a vocab called "state of residence" with states in the form of TN, MS, AL, etc. Each user account has one of these references to show the state of residence. I can also place this taxonomy field on a forum post. But how can I make the post pull the "state of residence" term from the account of the user who has created it? I don't want the user to have to choose. I want the choice to be made automatically based on the user's account.
Or is there a better way than a vocab. Basically, I am trying to allow content to be sorted by the geography of the user who created the post. and I don't want the user to have to make the choice.
Comment: Do you want this sorting of the content at some listing made by views(I mean forum post listing is made by views) ?
Comment: i will use views to sort the content, yes.
Answer: This sort of thing is generally handled through contextual filters in your view. Views, Panels and Page Manager work well together to manage passing contextual filters to a view for filtered output, but there is a bit of a learning curve involved.
I'm not generally one for video tutorials, but I found this one to be a good introduction to this methodology: http://drupalize.me/videos/overview-page-manager-series
If you want a quick-and-dirty way to achieve this, use the module views_field_view https://drupal.org/project/views_field_view and do something like this:

in your forum posting view, set a contextual filter on your state term reference field tid
create a new view based on users, and add a contextual filter on "User: Uid"
on the contextual filter configuration page, under "when the filter value is not available" select "provide default value"
under type, select user id from logged in user
add a field for the state term reference field tid
add a field Global:View ( provided by the views_field_view module ), with these settings:
View: your forum postings view, as modified with contextual filter above
Display: Master
Contextual filter: [tid]

This should now display all forum postings whose state of residence matches the currently logged in user's state of residence.
note that if you are on a page URL that points to a profile of a different user, and you've set it up so the view is visible on this page, it will show all postings for THAT user, as the view will be able to grab a uid from the URL. ... so ... that may be a benefit or a drawback, depending on your intent.
Best'o'luck
disclaimer: written on short sleep. hope i got all the i's dotted and t's crossed
Comment: Thanks so much for the info here. I'm going to take a good look at your suggestions. i had been approaching this from the organic group perspective but I keep thinking that OG has more than what I need for this. Contextual filters is the way to go. Thanks so much for this answer. I love Drupal but sometimes I think that it is hard to see the forest for the trees because there are just so many ways to do things.


Jqgrid Change values on inline edit

Question: I have got a single grid on my page, and when it drops into inline edit mode there are some fields i would like to fill in (price based ones) as soon as the user has filled the first one in, for example they put a price of 50 in, i want to update the nett and gross fields.
Normally i would do this using a jquery blur function, but i'm not sure how to get the names of the input boxes on the grid, they seem to be a number depending on the row, then the name of the field.
Thanks
Luke
Answer: Typically is you need update the cell after the user modified the cell value one use cell editing mode. In case of usage inline editing mode for row editing you have some options:

you initialize inline editing with keys: true parameter and the row will be saved after the user press Enter key. The user can any time discard the changes by pressing Esc key.
you provide some button which the user can click to save the changes. You can use formatter: 'actions', inlineNav or your custom button to do this.
You can implement saving of current editing row inside of onSelectRow or some other callback.

In any way the standard way in inline editing mode is working with rows without additional actions on cell basis. If you do need to bind blur or focusout event to all input fields of editing row you can use dataEvents with type: 'blur' (or type: 'focusout'). To easy define the editoptions for all columns you can use cmTemplate option of jqGrid (see here).

Jqgrid Change values on inline edit


F sharp or F natural from Minuet in G major

Question: I found scores of Minuet in G major on https://imslp.org/wiki/Minuet_in_G_major%2C_BWV_Anh.114_(Pezold%2C_Christian). But in some sheets, there's an F natural in the left hand of measure 29.

This appears to be different from the original. Is this an acceptable arrangement, or is it a typo?
Answer: This is an error.
I found this F-natural in this score, originally uploaded to the Mutopia Project here. But the change log of this Mutopia file clarifies that, in January of 2017, the maintainer addressed the issue:

<code>Date:   Thu Jan 19 12:14:28 2017 +0000
Correct F -> F# in bar 29 bass
Convert to LilyPond 2.19
</code>

Furthermore, you can see a clear F-sharp in an early manuscript source here.


AX Base Data Type Validation

Question: I'm looking for more information on how the basic data types in AX 2009 are validated, in an effort to expand that system.
We know that some kind of data validation occurs in almost all the base data types in AX, most notably the date field, because of some of the shortcuts you can do that take advantage of this: For example, if you type only the number 4 in a date field, the system automagically turns it into a date representing the 4th day of the current month and year. I am looking to find if there is any similar functionality for strings that I might be able to override or control.
Because strings apparently are not currently modified due to contents the way dates are, I want to put one in place myself. My current approach is to override the modifiedField method of a handful of tables to call a custom class, which would clean the data so it does not contain problematic characters, such as \r, \n or \t (some fields would be excluded, of course). Is there a better approach, preferably one that could be made more global?
Comment: That is correct, you can't physically enter it from the keyboard except in memo-style textboxes. However, you can easily get it in other fields by copying/pasting the character from an alternate source (like Notepad or an Email). I have done this myself several times.
Comment: The characters \r, \n or \t are automatically removed by AX. So are trailing spaces. Newlines (\n) are preserved in multi line string controls of cause.
Comment: Not from my experience. I've actually had \r\n at the beginning of an item number before (I think someone copied/pasted the number incorrectly). We only caught it with an Export to Excel from the Item Details screen. There have been several other issues related to those 3 characters, which lead us to implement this [change to the Export to Excel feature](http://axgrind.azurewebsites.net/post/AX-2009-Export-to-Excel-problems) (which leads to other problems, as it turns out)
Comment: I think it is data import problem. There is no way you can enter the \r character in AX.
Answer: There are no low-level string or date overloads that can be accessed.  Being able to manipulate every string on a global scale just doesn't make sense as something we would be able to access.
If that existed, the potential overhead you could introduce would be astronomical.
The way the Date assignment works, is most likely a kernel (or compiler?) operator assignment overload of the = sign to function as you described.  And it only executes at variable assignment, not data validation as you suggested.

AX Base Data Type Validation


Inverse factorial function

Question: I am wondering what is the inverse/opposite factorial function?
e.g $\text{inverse-factorial}*(6)=3$
Furthermore, I am intrigued to know the answer to:
$$a!=\pi $$
Find $a$.
I would really appreciate if anyone could explain this to me as I have found nowhere online with a good explanation of inverse factorial functions.
Thanks
Comment: See https://math.stackexchange.com/questions/931846/does-the-gamma-function-have-an-inverse and http://www.ams.org/journals/proc/2012-140-04/S0002-9939-2011-11023-2/
Comment: Can you elaborate on what you want out of this? The most popular continuous version of factorial is the [Gamma function](https://en.wikipedia.org/wiki/Gamma_function), but I don't know if "ask a calculator to approximate the solution to $\Gamma(x)=\pi$ and add $1$ to the answer" is the sort of thing you're looking for. For detailed strategies for approximating answers, maybe see https://math.stackexchange.com/a/2739498/26369
Comment: No integer's factorial is $pi$ the only thing you have seen is $\frac{1}{2}!=\pi$ which is not true. See definition of Gamma function : https://en.wikipedia.org/wiki/Gamma_function and see what happened when we put $\frac{1}{2}$
Comment: also refer to https://math.stackexchange.com/questions/1624347
Answer: Unfortunately there is not a closed form or nice series for the inverse of the factorial (or Gamma function).
First obstacle is that the factorial has a local minimum at $x:\;\psi(x)=0\; \to \; x=0.4616..$, so , considering only positive values of the argument, that gives you two values for the inverse.
For an analysis of the problem please refer to this and this papers.
 A lighter look is given in this other paper.
Finally an interesting approximated function is given here.
Answer: inverse functions are not well defined when it is not a $1:1$ function, and as there is a minimum where:
$$\Gamma'(z+1)=0$$
that is:
$$\int_0^\infty\partial_nt^ze^{-t}dt=0$$
which can be numerically estimated.

Inverse factorial function


Regression analysis with time series data

Question: I'm completely stuck. I'm making a regression analysis in Python :
my dependent variable - share price of company,
independent variables - price of steel, price of coal and changing in local currency.
According theory I can't use this time series depended and independent variables due to autocorrelation.
Can I solve this problem by taking percent changing day-to-day of share price and percent changing day-to-day independed variable ? I have check this data for stationarity with Dickey Fuller test and get appropriate results, but I think that isn't right way.
or
What are the ways to build such a model ?
P.S I'm recently read that for time series data ARMA model is the good choice, but I don't understand how take in account independent variable, not predict on dependent variables on itself.
Thank you in advance!
Answer: Conventional wisdom suggests that you should use ARCH (GARCH) models rather than OLS or ARIMA models, for the modeling of returns (percent change in prices). The literature of those models is very vast, but basically the starting model should be a GARCH(1,1).


Exportar Tipo Date para openOffice/libreOffice Delphi

Question: Como posso exportar um tipo Date para o openoffice/libreoffice
quando preciso exportar um tipo String faço da seguinte forma.
<code> SheetOpenOffice.GetCellByPosition(vCont,vlinha).SetString(Q_Assoc.Fields[i].AsString);
</code>
Como posso fazer quando o tipo que vem na query for um tipo Date?
Answer: Tente fazer da seguinte maneira:
<code>if ((DateToStr(Q_Assoc.Fields[i].Value) <> Null) and (DateToStr(Q_Assoc.Fields[i].Value) <> '')) then
    SheetOpenOffice.GetCellByPosition(vCont,vlinha).SetString(Q_Assoc.Fields[i].Value);
</code>
Comment: @rodrigo.oliveira, editei a resposta no lugar do SetValue coloque SetString
Comment: Deu esse erro:  InterfaceOleWrapper_Impl::Invoke : [automation bridge]UnoConversionUtilities::variantToAny Cannot convert the value of vartype :"8" to the expected UNO type of type class: 11
Comment: @rodrigo.oliveira, deu certo?


Looking for a measure of variance of variance?

Question: I was hoping you would be able to help me identify the statistic I am looking for, or point me in the right direction.

Is there a statistical measure that will represent variance of variance (or s.d. of s.d.)? In the graphs attached, is there such a measure that will identify graphs 1 and 2 as similar because variance is pretty constant for all values of x, and show graphs 1 and 2 as different from graph 3 as the variance of graph 3 fluctuates greatly as we change x?
I understand that I could split the x axis into bins, calculate the variances of each bin, then calculate the variance of the bin variances but is there a better way that wouldn't involve more decisions to be made (like bin size)?
Thank you for any help you can give me,
Chrissy x
Answer: The term you are looking for is called heteroscedasticity.
As for testing if you have heteroscedasticity, the tests basically proceed as you guessed, except that often you use the fact that you know the underlying model (i.e. some type of regression model), and so you don't exactly "split" up the data. At the extreme, this is precisely what you do for grouped data. See the detection section of the wiki link for some common tests.
Looking at the test statistic of some of these tests should give you a sense of how to measure it. Again, given some model (such as a regression model), this is usually done by taking the residuals of your model and seeing if the variance of the residuals changes as a function of the regressors.
Hopefully this is enough to get you going!
Answer: In quant finance there are several models that incorporate such a concept. For instance, SABR model looks as follows in a simplest form:
$$dF_t=\sigma_t \left(F_t\right)^\beta\, dW_t,$$
$$d\sigma_t=\alpha\sigma^{}_t\, dZ_t,$$
were, $\alpha$ is the variance of the variance of the changes in rate series $F_t$.
In order to estimate the variance of the variance it's best to have some kind of a model in mind. The variance is not directly observed, to start with. This makes measuring variance of variance even more difficult. If you have a model, then you can come up with a fitting approach to extract the parameter from the data.


Complex Analysis, bounding modulus of derivative of a function with given constrains

Question: I've encountered the following problem:

Let $\mathbb{D}$ denote the unit disk. Let $f$ be an entire function,
such that $f(0)=3+4i$ and $\forall z \in \mathbb{D}:|f(z)|\leq 5$.
Find $f'(0)$

I suppose we can apply the maximum modulus principle in some way but I don't see how I can make any statement about the derivative. Any help is appreciated
Comment: You're right, and now the hint: $\;|f(0)|=5\;$, so $\;|f|\;$ attains its maximum value on $\;\Bbb D\;$ at the point $\;z=0\;$ , so...can you take it from here?
Answer: If $f$ was not constant, then $|f|$ would have no maximum. But $|f(0)|=5\geqslant|f(z)|$, for each $z\in\Bbb D$, and therefore the maximum of $|f|$ is attained at $0$. So, $f$ is constant. It follows from this that $f'(0)=0$.


Handling Files With Different Columns in USQL

Question: I have a USQL script and CSV extractor to load my files. However some months the files may contain 4 columns and some months they may contain 5 columns.
If I setup up my extractor with a column list for either 4 or 5 fields I get an error about the expected width of the file. Go check delimiters etc etc. No surprise.
What is the work around to this problem please given USQL is still in a newbie and missing some basic error handling?
I've tried using the silent clause in the extractor to ignore wider columns which is handy for 4 columns. Then getting a row count of the rowset with an IF condition that then has an extractor for 5 columns. However this leads to a world of rowset variables not being used as scalar variables in the IF expression.
Also I tried a C# style count and a sizeof(@AttemptExtractWith4Cols). Neither work.
Code snippet to give you a feel for the approach I'm taking:
DECLARE @SomeFilePath string = @"/MonthlyFile.csv";

@AttemptExtractWith4Cols =
    EXTRACT Col1 string,
            Col2 string,
            Col3 string,
            Col4 string
    FROM @SomeFilePath
    USING Extractors.Csv(silent : true); //can't be good.

//can't assign rowset to scalar variable!
DECLARE @RowSetCount int = (SELECT COUNT(*) FROM @AttemptExtractWith4Cols);

//tells me @AttemptExtractWith4Cols doesn't exist in the current context!
DECLARE @RowSetCount int = @AttemptExtractWith4Cols.Count();

IF (@RowSetCount == 0) THEN
    @AttemptExtractWith5Cols =
        EXTRACT Col1 string,
                Col2 string,
                Col3 string,
                Col4 string,
                Col5 string
        FROM @SomeFilePath
        USING Extractors.Csv(); //not silent
END;

//etc

Of course if there was such a thing as a TRY CATCH block in USQL this would be a lot easier.
Is this even a reasonable approach to take?
Any input would be greatly appreciated.
Thank you for your time.
Answer: U-SQL now supports OUTER UNION so you can handle it like this:
// Scenario 1; file has 4 columns
DECLARE @file1 string = @"/input/file1.csv";

// Scenario 2; file has 5 columns
//DECLARE @file1 string = @"/input/file2.csv";

@file =
    EXTRACT col1 string,
            col2 string,
            col3 string,
            col4 string
    FROM @file1
    USING Extractors.Csv(silent : true)

    OUTER UNION ALL BY NAME ON (col1, col2, col3, col4)

    EXTRACT col1 string,
            col2 string,
            col3 string,
            col4 string,
            col5 string
    FROM @file1
    USING Extractors.Csv(silent : true);

@output =
    SELECT *
    FROM @file;

OUTPUT @output
    TO "/output/output.csv"
USING Outputters.Csv();

In my example, file1 has 4 columns and file2 has 5 columns.  The script runs successfully in either scenario.
My results:

Hope that makes sense.
Answer: The OUTER UNION is a great solution. Alternatively, you can also write your own generic extractor if you expect your rows in a file to be different. See this blog post for an example.
Comment: Thanks Mike, I'll take a look.
Answer: Here's another solution which I found helpful. You can read the file as a single text column (using "\t" as delimiter since there aren't any) and then dynamically split using C# string functions. I have tested this on a similar problem. The advantage of this approach is that you could potentially use the same approach for any number of columns.
SELECT
      (String)(ColList[0])   AS ColA
    , (String)(ColList[1])   AS ColB
    , (String)(ColList[2])   AS ColC
    , (String)(ColList[3])   AS ColD
    , (int?)(NumColumns >= 5 ? (String)(ColList[4]) : (String)null)
                             AS ColE
FROM (
    SELECT ColList
         , ColList.Count AS NumColumns
    FROM (
        SELECT SqlArray.Create(RowText.Split(','))   AS ColList
        FROM (
            EXTRACT RowText string
            FROM @SomeFilePath
            USING Extractors.Text(delimiter: '\t', quoting: false)
        ) AS [T1]
    ) AS [T2]
) AS [T3]

Warning: This solution is not aware of text quoting. Any commas in field values will break this logic.


SQL: Convert duration in microseconds to DD:HH:MM:SS

Question: I am working with data containing [duration] in microseconds, some examples:

5982312624, 181240328664, 2981750461, 993806305107, 3015857413, 283850264906, 5987597545, 240770859583

I converted this to number of hours, minutes and seconds as follows:
(CONVERT(VARCHAR(10), Dateadd(second, (((([duration]/10^7)/3600*60)/1000000.0)*60),0),114)) AS [HH:MM:SS]

The problem I am facing is when this exeeds 24 hours it does not register correctly.
Question: I would either like to add a day in here for example: DD:HH:MM:SS -> 01:23:04:33 or just continue the count of hours example: 45:12:52.
Hope anyone can help. Thanks in advance.
Comment: This might answer your question: https://stackoverflow.com/a/12757846/1926181 , convert to TIME instead of DATE
Answer: Assuming a duration can't be longer than 100 days, which I hope it can't if you're recording microseconds, then given this sample data:
CREATE TABLE dbo.Source(id int identity, duration bigint);

INSERT dbo.Source(duration) VALUES
(5982312624),(181240328664),(2981750461),(993806305107),
(3015857413),(283850264906),(5987597545),(240770859583);

One way is to determine the day of year after adding seconds (not microseconds, which will overflow) to any arbitrary January 1st, and prepend the number of days onto the conversion to time:
SELECT id, duration,
  [DD:HH:MM:SS] = RIGHT('0' + RTRIM(DATEPART(DAYOFYEAR, dt) - 1), 2)
    + ':' + CONVERT(char(8), dt, 108)
  FROM dbo.Source
  CROSS APPLY
  (
    VALUES(DATEADD(SECOND, duration/1000000.0, '20000101'))
  ) AS d(dt);

Results:

id
duration
DD:HH:MM:SS

1
5982312624
00:01:39:42

2
181240328664
02:02:20:40

3
2981750461
00:00:49:41

4
993806305107
11:12:03:26

5
3015857413
00:00:50:15

6
283850264906
03:06:50:50

7
5987597545
00:01:39:47

8
240770859583
02:18:52:50

If it can be more than 100 days (or more than a year), you can make this slight change:
SELECT id, duration,
  [DDDD:HH:MM:SS] = RIGHT('000' + RTRIM(DATEDIFF(DAY, '20000101', dt)), 4)
    + ':' + CONVERT(char(8), dt, 108)
  FROM dbo.Source
  CROSS APPLY
  (
    VALUES(DATEADD(SECOND, duration/1000000.0, '20000101'))
  ) AS d(dt);

Results:

id
duration
DDDD:HH:MM:SS

1
5982312624
0000:01:39:42

2
181240328664
0002:02:20:40

3
2981750461
0000:00:49:41

4
993806305107
0011:12:03:26

5
3015857413
0000:00:50:15

6
283850264906
0003:06:50:50

7
5987597545
0000:01:39:47

8
240770859583
0002:18:52:50

And for completeness, if you want all the hours added together instead of separating days, it's actually slightly more complicated:
SELECT id, duration,
    [HHHH:MM:SS] = RIGHT('000' + CONVERT(varchar(11),
      24 * DATEDIFF(DAY, '20000101', dt) + DATEPART(HOUR, dt)), 4)
      + RIGHT(CONVERT(char(8), dt, 108), 6)
  FROM dbo.Source
  CROSS APPLY
  (
    VALUES(DATEADD(SECOND, duration/1000000.0, '20000101'))
  ) AS d(dt);

Results:

id
duration
HHHH:MM:SS

1
5982312624
0001:39:42

2
181240328664
0050:20:40

3
2981750461
0000:49:41

4
993806305107
0276:03:26

5
3015857413
0000:50:15

6
283850264906
0078:50:50

7
5987597545
0001:39:47

8
240770859583
0066:52:50

If it can be more than 1,000 days, well, perhaps you should consider storing start and end as datetime instead of duration in microseconds.

Examples at db<>fiddle
Answer: I think the answer above causes a little bit of a headache and is not obvious how to tweak for most I guess.
It's far easier to work with FLOOR() and FORMAT().
Example for [DDDD:HH:MM:SS]:
DECLARE @Table TABLE (id int identity, duration bigint)
INSERT @testTable VALUES
(5982312624),(181240328664),(2981750461),(993806305107),
(3015857413),(283850264906),(5987597545),(240770859583)

DECLARE @micro int = 1000000

SELECT id, [HHHH:MM:SS] =
FORMAT(FLOOR(duration/@micro/86400), '0000')+ ':' +    --Convert to total days
FORMAT(FLOOR(duration/@micro/3600) % 24, '00')+ ':' +  --Convert to total hours and modulo the days
FORMAT(FLOOR(duration/@micro/60) % 60, '00')+ ':' +    --Convert to total minutes and modulo the hours
FORMAT(FLOOR(duration/@micro)  % 60, '00')             --Convert to total seconds and modulo the minutes
FROM
@testTable

Example for [HHHH:MM:SS]:
DECLARE @durationTable TABLE (id int identity, duration bigint)
INSERT @durationTable VALUES
(5982312624),(181240328664),(2981750461),(993806305107),
(3015857413),(283850264906),(5987597545),(240770859583)

DECLARE @micro int = 1000000

SELECT id, [HHHH:MM:SS] =
FORMAT(FLOOR(duration/@micro/3600), '0000')+ ':' +     --Convert to total hours
FORMAT(FLOOR(duration/@micro/60) % 60, '00')+ ':' +    --Convert to total minutes and modulo the hours
FORMAT(FLOOR(duration/@micro)  % 60, '00')             --Convert to total seconds and modulo the minutes
FROM
@durationTable


Matrix Equation Solution

Question: Does anybody know how to solve this matrix equation
\begin{equation}
{
I = S - RSQ,
}
\end{equation}
where
$I$ is the identity matrix,
$R$, $Q$ are known matrices, and
$S$ is the unknown matrix.
Answer: Rewrite the equation as $$I=ISI-RSQ$$ and apply vectorization
$$\eqalign{
 {\rm vec}(I) = (I\otimes I-Q^T\otimes R)\,{\rm vec}(S) \cr
 {\rm vec}(S) = (I\otimes I-Q^T\otimes R)^{-1}\,{\rm vec}(I) \cr
 S = {\rm devec}\Big((I\otimes I-Q^T\otimes R)^{-1}\,{\rm vec}(I)\Big) \cr
}$$
If you can transform the equation into a Sylvester equation, then you can use a more efficient solver like Bartels–Stewart instead of this brute force approach.
Answer: Suppose the following series converge, then
$$S=I+RQ+R^2Q^2+\cdots$$
Comment: Yeah, I tried to find the sum of the series and obtained the equation.

Matrix Equation Solution


Using Microsoft CA to create TLS key/cert for Postfix

Question: I'm trying to configure a postfix mailserver to use TLS encryption for SMTP (and IMAP with dovecot, but I haven't gotten to that point yet). It's running on a Fedora 14 machine and I'm using webmin to manage it.
I have a Windows domain with the DC also setup as a Microsoft Certificate Authority and I created a GP that adds the CA as a trusted root ca. I have web services installed so I can go to localhost/certsrv to request new certificates. Anyway, I'd like to use the Microsoft CA to create the Private Key, Public key, and CA Cert for the postfix mail server.
The intent is that all users connecting to the server with SMTP (and later IMAP) will use SSL/TLS and will trust the server because they trust the CA.
Let me know if you need any more info or any suggestions.
Comment: I tried creating a new certificate from the 'website' template. I didn't provide a 'request' or anything from my linux server, just started by going to http://localhost/certsrv and tried to create a new cert. **It never asked for the common name of the server**, so I assumed I was doing something wrong.
Comment: Am I supposed to generate a key pair on the linux server using openssl and then just use my Microsoft CA to sign the public key? If so, the openssl commands would be super useful.
Answer: If your intended goal is to have a verifiable certificate on the postfix server, you can provide the CA chain files to postfix and import the CA root certificate into all clients; this will follow the chain for all certificates issued by the CA.
Quoting from http://www.postfix.org/postconf.5.html#smtpd_tls_cert_file:

To enable a remote SMTP client to verify the Postfix SMTP server
certificate, the issuing CA certificates must be made available to the
client. You should include the required certificates in the server
certificate file, the server certificate first, then the issuing CA(s)
(bottom-up order).
Example: the certificate for "server.example.com" was issued by
"intermediate CA" which itself has a certificate of "root CA". Create
the server.pem file with "cat server_cert.pem intermediate_CA.pem
root_CA.pem > server.pem".


Best way to check if file matching abc* exists in a particular directory

Question: I want to know how to check if a file matching core* exists in /tmp/ path. The actual file can be anything like below.

corexyz.txt
coreabc.tar.gz
core*

If the complete file name was known, I could have used stat or access to know if the file existed. But here, I want to know if there is any file matching core*.
bool checkFile (std::string ABC)
{
    return true; /* return true if file ABC* exists */
}

Comment: https://www.boost.org/doc/libs/1_71_0/libs/filesystem/doc/index.htm
Answer: If you can use c++17 the filesystem library will be interesting for you.
You can use std::filesystem::directory_iterator to iterate over all the files in a directory and check if they start with the given prefix using path.filename().


Etale algebras over dual numbers

Question: How to classify all etale algebras over $k[x]/x^2$ where $k$ is some field? All etale covers? What is the classification for an arbitrary Artin algebra?
My attempt: Let's first assume that $k$ is algebraically closed. Let $k[x]/x^2 \to A$ be an etale map. Then $k \to A \otimes_{k[x]/x^2} k$ is etale. Therefore, $A/x = \Pi_{i \in I} k$ where $|I|$ is the degree of $A$. Does it follow that $A = \Pi_{i \in I} k[x]/x^2$? I'm not sure about that.
Comment: I mean, this makes intuitive sense from the topological perspective. You probably need to mumble something about how lack of ramification implies that the components of $\operatorname{Spec}A/x$ "spread out" to copies of $\operatorname{Spec}k[x]/(x^2)$ without intersecting.
Comment: @TabesBridges Yes, that makes sense but I don't see how to make it rigorous.
Answer: One can just do this explicitly, but I thought I'd mention that there is a nice aspect of the theory that makes the comments from above precise. The key thing that makes everything nice is the topological invariance of the etale site:

Theorem(Topological invariance of the etale site, Tag 04DZ): *Let $f\colon X\to Y$ be a universal homeomorphism. Then, the pullback functor $(V\to Y)\mapsto (V\times_Y X\to X)$ defines an equivalence of categories
$$\left\{\begin{matrix}\text{Schemes etale}\\ \text{over }Y\end{matrix}\right\}\to \left\{\begin{matrix}\text{Schemes etale}\\ \text{over }X\end{matrix}\right\}.$$

In particular, as $\mathrm{Spec}(k)\to \mathrm{Spec}(k[x]/(x^2))$ is a universal homeomorphism (being a nilpotent closed immersion) we see that the functor
$$\left\{\begin{matrix}\text{Schemes etale}\\ \text{over }\mathrm{Spec}(k[x]/(x^2))\end{matrix}\right\}\to \left\{\begin{matrix}\text{Schemes etale}\\ \text{over }\mathrm{Spec}(k)\end{matrix}\right\}.$$
is an equivalence. Now, as you are certainly well-aware, the etale algebras over $\mathrm{Spec}(k)$ are of the form $\bigsqcup_i\mathrm{Spec}(L_i)$ with $L_i/k$ a finite etale extension. As $\bigsqcup_i \mathrm{Spec}(L_i[x]/(x^2))$ pulls back along $\mathrm{Spec}(k)\to \mathrm{Spec}(k[x]/(x^2))$ to $\bigsqcup_i\mathrm{Spec}(L_i)$ we deduce that, up to isomorphism, every etale scheme over $\mathrm{Spec}(k[x]/(x^2))$ is isomorphic to such a $\bigsqcup_i \mathrm{Spec}(L_i[x]/(x^2))$.
Comment: Oh, sure, I'm an idiot! That makes a lot of sense!

Etale algebras over dual numbers


Delete file selected in dropdown list

Question: Trying to make a dropdown list, displaying files from a specific folder, with a Delete button to delete the file selected.
Dropdown list:
<?php
    $dirname = "files";
    $dir = opendir($dirname);
    echo '<form action="delete.php" method="get">';
    echo '<select name="file2">';
    while(false != ($file = readdir($dir)))
    {
        if(($file != ".") and ($file != ".."))
        {
            echo "<option value=".$file.">$file</option>";
        }
    }
    echo '</select>';
    echo '<input type="submit" value="Delete" class="submit" />';
    echo '</form>';
?>

the delete.php file:
<?php
    $dirpath = "files";
    $file_to_delete = $_POST['file2'];
    if ( unlink ($dirpath.'/'.$file_to_delete) ) {
        echo $file_to_delete . " deleted.";
    } else {
        echo "Error.";
    }
?>

When I then try to selected a file and press delete, I get the following error:

Warning: unlink(files/): Is a directory in
  /xxx/xxx/xxx/xxx/xxx/xxx/xxx/xxx/delete.php on
  line 4 Error.

xxx'ed out due to privacy :) all the files trying to be deleted are chmod 777.
its a simply .txt file I'm trying to delete.
Not sure what I'm missing or what I did wrong here... :/
Comment: Did you ever try to look at what `print_r($_POST['file2'])` comes before you try to delete?
Answer: It's $_GET, because your form has method="get":
$file_to_delete = $_GET['file2'];

so:
<?php
    $dirpath = "files";
    $file_to_delete = $_GET['file2'];
    if ( unlink ($dirpath.'/'.$file_to_delete) ) {
        echo $file_to_delete . " deleted.";
    } else {
        echo "Error.";
    }
?>

If you want post change your form to method="post".
Comment: Ah, brilliant. :) learn some new every day.
Thank you very much Mihai.

Delete file selected in dropdown list


How can I use a RandomAccessCollection with a ForEach list?

Question: I have a class that looks something like this:
public class TransactionResults: Identifiable {
    public typealias CustomClassValue = Transaction
    public typealias CustomClassIndex = Array<CustomClassValue>.Index

    public let id = UUID()
    private let transactions = sampleTransactions
}

extension TransactionResults: RandomAccessCollection {
    public var startIndex: CustomClassIndex { return transactions.startIndex }
    public var endIndex: CustomClassIndex { return transactions.endIndex }

    public subscript(index: CustomClassIndex) -> CustomClassValue? {
        get { if index >= 0 && index < transactions.count {
            return transactions[index]
            } else {
            return nil
            }
        }
    }
}

When I try to use it with a ForEach, I get an error: Cannot convert value of type 'TransactionResults' to expected argument type 'Range<Int>'
struct TransactionListView: View {
    @State private var transactions = TransactionResults()

    var body: some View {
        NavigationView {
            List() {
                ForEach(transactions) { transaction in
                    EmptyView()
                }
            }
        }
    }
}

What do I need to change in my TransactionResults class for this to work right?
Answer: If you correctly declare your Transaction class (which by the way can conflict with SwiftUI Transaction type, so I would recommend to rename your type), then all is needed (at least as tested with Xcode 11.4) is to use different ForEach constructor:
ForEach(transactions, id: \.self) { transaction in
    EmptyView()
}

Update: here is a transaction class with which I tested you code snapshot (as I mentioned I renamed it to avoid conflict with built-in type)
class ATransaction: Equatable, Identifiable, Hashable {
    static func == (lhs: ATransaction, rhs: ATransaction) -> Bool {
        lhs.id == rhs.id
    }

    func hash(into hasher: inout Hasher) {
        hasher.combine(id)
    }

    let id = UUID()
}

Comment: This doesn't allow me to use the id of the transaction, which seems strange to me. This does fix the error tho.


AWS autoscaling: downscale too fast

Question: The scaling configured according to cpu usage:

On the chart it's about 8 minutes between upscaling and 2 mins between downscalings.

We cannot upscale faster because an instance requires ~5min to launch, however I'd like to have downscaling with similar delay too. Other variables:
Default Cooldown 600
Health Check Grace Period 400
What's configured wrong here?
Answer: In your decrease logic, increase the value for "consecutive periods...".
That should induce delay in your ASG shrink actions.
Comment: but '4 consecutive periods of 300 seconds' is 20 minutes, why we have only two?
Comment: @Putnik probably because the "remove 2" condition is being hit, so 2 are being removed at relatively the same time.

AWS autoscaling: downscale too fast


How to split a pandas dataframe based on column prefix

Question: I have a dataframe that I imported from a csv.  It goes something like this:
df
A.1  B.1  A.2  B.2
1    1    1    1
2    2    2    2

My question is, what would be an efficient way to turn this into seperate data frames just comprised of A's and B's
df_a
A.1  A.2
1    1
2    2

df_b
B.1  B.2
1    1
2    2

I am not picky as far as the column names, would be fine with having them just be stripped to 1 and 2 etc but haven't been able to find a good way to do this.  I am also open to other/better ways to accomplish what I am trying to do in case this doesn't make sense to someone more knowledgable.  Thanks!
Comment: You should go with @unutbu's answer, IMO.
Answer: You could use df.filter with regex patterns:
df_a, df_B = df.filter(regex=r'^A'), df.filter(regex=r'^B')

or
df_a, df_B = df.filter(like='A'), df.filter(like='B')

Note that if you use like='A' then all columns whose name contains 'A' will be selected. If you use regex=r'^A' then only those columns whose name begins with an A will be selected.

In [7]: df
Out[7]:
   A.1  B.1  A.2  B.2
0    1    1    1    1
1    2    2    2    2

In [8]: df_a, df_B = df.filter(regex=r'^A'), df.filter(regex=r'^B')

In [9]: df_a
Out[9]:
   A.1  A.2
0    1    1
1    2    2

In [10]: df_B
Out[10]:
   B.1  B.2
0    1    1
1    2    2

Answer: Ok, if I understand correctly you just need N new dataframes according to their column name.
dfa = df[[col for col in df.columns if col.startswith("A")]].copy()
# same for dfb, dfc...

Note that copy() is required if you wish to later apply changes to that new dataframe dfa. Otherwise, if I remember correctly, you would be applying changes by pointer, much like when using dictionaries.
Answer: To select the columns:
dfa = df[['A.1', 'A.2']]

To change the name of the columns:
dfa.reindex=["a1","a2"]


Puppet changing source file permissions

Question: I'm setting up jail shells using puppet to maintain a copy of the required libraries within the jails.
Using the following puppet code I can successfully copy the files across:
<code>file { "/home/${username}/lib/x86_64-linux-gnu/libdl.so.2":
        ensure => present,
        source => "/lib/x86_64-linux-gnu/libdl.so.2",
        mode => '0700',
        links => 'manage',
        owner => $username,
        group => $username,
        require => File["/home/${username}/lib/x86_64-linux-gnu/"]
}
</code>
It appears though that the source file permissions are being changed to $username:$username as well as the destination file permissions.
The file /lib/x86_64-linux-gnu/libdl.so.2 end's up with the following permissions:
<code>-rwx------ 1 $username $username ld-2.19.so
</code>
The jailed users are obviously able to login with this setup, but for everyone else that would be using /bin/bash in the real /lib folder it is breaking the login.
Comment: I ran your code on a test machine, and it works fine for me, i.e. it creates a broken symbolic link. Is there any chance that the resource File["/home/${username}/lib/x86_64-linux-gnu/"] might be causing the issue, i.e. your permissions are changed there? Please add the output of the puppet run with `-v` and `--debug`.
Answer: You have to quote:

owner => '$username',
Comment: it's not the fact that it is setting the file permissions incorrectly, i.e the $username is just a placeholder in the permissions example above. The problem is that the permissions are being set on the source file as well as the destination file

Puppet changing source file permissions


Trigonometry in isoceles trisected triangles

Question: Hello,

A $\triangle ABC$ has a point $D$ on $BC$ such that $\angle BAD = 30$ and $\angle CAD = 45$. Both $AB$ and $AC$ are of the length $\sqrt{2}$. Find the length of $AD$.

This is what I have done:-
Taking $AD$ as $x$ and $DC$ as $y$, and then using law of cosines for $y$, we get $y^2 = 2 + x^2 - 2x$.
Taking $BD$ as $k$, and applying law of cosines on $k$, we get $k^2 = x^2 + 2 - x*root6$
Taking law of cosines on $k + y$ we get, $(k + y)^2 = 5 - root3 ( cos 75 = root3 - 1/ 2*root2)$
And now I'm stuck at this step. I have no idea on how to go forward after this step.
ANY HELP IS APPRECIATED(if possible in trigonometry plz).
This is not a part of the question but can someone give me a link for the quick reference of mathjax editing?
Comment: Here's a [MathJax tutorial](https://math.meta.stackexchange.com/q/5020/122782).
Answer:
Since $|AB|=|AC|$,
\begin{align}
\angle ACD&=\angle ABD=52.5^\circ
,\\
\angle DAB&=97.5^\circ
,\\
\angle ADC&=82.5^\circ
.
\end{align}
Using the standard notation for $\triangle ABC$,
the area $[ABC]$ in terms of known side $a$ and
the angles $\alpha$, $\beta$, $\gamma$ is
\begin{align}
[ABC]&=\frac{a^2\sin\beta\sin\gamma}{2\sin\alpha}
.\\
\text{Then}\quad[ADC] \text{ in terms of } |AC|:\quad
[ADC]&=\frac{(\sqrt2)^2\sin52.5^\circ\sin45^\circ}{2\sin82.5^\circ}
=\frac{\sin52.5^\circ}{\sin82.5^\circ}
,\\
\text{and}\quad[ADC] \text{ in terms of } x:\quad
[ADC]&=\frac{x^2\sin82.5^\circ\sin45^\circ}{2\sin52.5^\circ}
=
\frac{x^2\sin82.5^\circ}{2\sin52.5^\circ}
,\\
x&=\frac{\sqrt2\sin52.5^\circ}{\sin82.5^\circ}
.
\end{align}
Edit:
\begin{align}
x&=\frac{\sqrt2\sin52.5^\circ}{\sin82.5^\circ}
\\
&=\sqrt2\cdot\frac{\sin(82.5^\circ-30^\circ)}{\sin82.5^\circ}
\\
&=\sqrt2\left(\cos30^\circ
-\sin30^\circ\cot82.5^\circ
\right)
\\
&=\sqrt2\left(\tfrac{\sqrt3}2
-\tfrac12\tan(\tfrac{30^\circ}4)
\right)
\\
&=\sqrt2\left(\tfrac{\sqrt3}2
-\tfrac12(\sqrt2\sqrt3-\sqrt3+\sqrt2-2)
\right)
\\
&=(\sqrt2-1)(1+\sqrt3)
.
\end{align}
Comment: Very sorry, i gave the wrong angle specification, can you plz check the question again and give me the answer
Comment: Can you please tell me how you got sin 52.5/sin 82.5 in the 3rd step. Also can you please send me a link for the identity if you used it.
Comment: Can you please tell me which software you used to draw that diagram?
Comment: @Adithya Dsilva: [Asymptote](http://asymptote.sourceforge.net/),
see also the comments to [this answer](https://math.stackexchange.com/a/2456737/122782).
Answer: Answer to a changed problem.
The easiest way to solve the problem is with introduction of the coordinate system. Let $A=(0,0)$ and $D=(0,d)$. We have to find $d$. Then $AC$ is a line $y=x$ and $AB$ is a line $y=-{\sqrt{3}\over 3}x$. Since $B,C$ lies on the circle $x^2+y^2=2$ we can easily calculate coordinate of $B$ and $C$:
$$ B = ({\sqrt{6}\over 2},-{\sqrt{2}\over 2})\;\;\;\;{\rm and}\;\;\;\; C = (1,1)$$ Since $B,C,D$ are colinear we get $$d= (\sqrt{3}+1)(\sqrt{2}-1)$$
Comment: Any trigonometric method? And what's the length of AD? because the answers are integer type and the length of AD is in the form root m/2. We have to give the value of m and I thought I needn't bother this community for such simple simplifications.