Should I standardize data using sample statistics?

Question: Suppose I sample 1,000 $\mathcal{N}(0,1)$ observations
>>> import numpy as np
>>> x = np.random.randn(1000)

The sample mean of x is
>>> x.mean()
-0.034215203795331606

The (biased) sample standard deviation of x is
>>> x.std(ddof = 1)
0.9923790554909595

Now suppose that I standardize these observations using these sample statistics
>>> y = (x - x.mean()) / x.std(ddof = 1)

The sample mean and (biased) sample standard deviation of y are
>>> y.mean()
1.5987211554602253e-17
>>> y.std(ddof = 1)
0.9999999999999999

However, I feel that this standardization procedure is a bit misleading, since y is no longer normally distributed. More formally, if
\begin{align}
X &\sim \mathcal{N}(0,1) \\
\hat{\mu} &= \frac{1}{1000} \sum_{i=1}^{1000} X_i \\
\hat{\sigma} &= \sqrt{\frac{1}{999} \sum_{i=1}^{1000} (X_i - \hat{\mu})^2}
\end{align}
then
$$
Y = \frac{X - \hat{\mu}}{\hat{\sigma}}
$$
My questions are:

If $Y$ is not normally distributed, then when is standardizing data using sample statistics appropriate?
How is $Y$ distributed?
Does this standardization procedure using sample statistics have a name?

Update
I was not precise enough in my original question. Suppose that
$$
X_1,X_2,...,X_{1000} \sim \mathcal{N}(0,1)
$$
are independent and identically distributed, and suppose that
\begin{align}
\hat{\mu} &= \frac{1}{1000} \sum_{i=1}^{1000} X_i \\
\hat{\sigma} &= \sqrt{\frac{1}{999} \sum_{i=1}^{1000} (X_i - \hat{\mu})^2}
\end{align}
If
$$
\forall i \in \{1,2,...,1000\}, Y_i = \frac{X_i - \hat{\mu}}{\hat{\sigma}}
$$
then are
$$
Y_1,Y_2,...,Y_{1000} \sim \mathcal{N}(\mu_{Y},\sigma_{Y})
$$
for some mean $\mu_{Y}$ and standard deviation $\sigma_{Y}$? It seems that the answer is no, since each $Y_i$ is a non-linear function of $X_1,X_2,...,X_{1000}$ via $\hat{\sigma}$. If I am wrong, then what would $\mu_{Y}$ and $\sigma_{Y}$ be?
Comment: @krkeane if $Y$ is a non-linear function of $X_1,X_2,...$, which are normally distributed, via $\hat{\sigma}$, then how is $Y$ normally distributed?
Comment: @krkeane I've edited my question to clarify.
Comment: But both $\hat{\mu}$ and $\hat{\sigma}$ are random variables with their own distributions. If I sampled $X_1,X_2,...$ infinitely many times and each time I performed the same transformation, I would not get the same $Y_1,Y_2,...$
Comment: $X$ is normally distributed.  It is characterized by two parameters: a location $\mu$ and a scale $\sigma$.  $Y$ represents a changed location ($X-\hat{\mu}$) and changed scale (multiplication by $\hat{\sigma}^{-1}$) for $X$, but $Y$ is still normally distributed if $X$ is normally distributed.
Comment: Only two things happened to $X_i$ to obtain $Y_i$: you changed location: $(X_i - \hat{\mu})$, then you changed scale: $\frac{1}{\hat{\sigma}}$. These are linear transforms of $X_i$ to $Y_i$.  It does not change the distribution family.
Comment: @krkeane "It does not change the distribution family." - It actually does, see my answer. mhdadk has a point there that deserves to be acknowledged.
Comment: I think this question requires some motivating context to be answerable. "Is __ appropriate?" isn't really answerable without know the goals of the procedure. Facts about the particular distribution of $Y$, and whether or not the distribution of $Y$ depends in a linear or nonlinear way on $X$, might or might not be important, depending on what you're trying to do. A lot of the discussion seems to originate in folks arriving at this thread with different ideas of what problem is being solved, which implicitly shapes which solutions are viable.
Comment: @Sycorax thanks for the feedback. Should I completely re-wrote the question? Or is there a better way of editing it such that the answers don't seem irrelevant? I could also delete this one and post a new one.
Comment: I think it's best to make a new question describing what problem you're trying to solve, how standardization fits into solving it, and articulating exactly what your concerns are.
Comment: @Sycorax should I delete this one before posting a new one? Or should I post a new one with a link to this one?
Comment: The software won't let you delete this question because it has 1 answer with positive score. If you feel this provides helpful context, it's perfectly fine to link to it. But the new question should stand on its own.
Answer: I had to edit this after already having picked up 2 votes because I realised that what I wrote about the conditional distribution isn't correct. In fact this is a surprisingly tricky question.

A major reason to standardise is if you have several different variables and want to make their values comparable. If you only have a single variable, I don't really see the point of it. Generally whether standardisation makes sense or not depends on what you do with the (un-)standardised data.

Note in particular that there is no reason to standardise if you want to estimate $\mu$ and $\sigma$, because standardisation amounts to removing the information about $\mu$ and $\sigma$ from the data.

The fact that $Y$ is not normally distributed is a nice observation. In fact, it's not i.i.d. normally distributed.
"I.i.d." means identically and independently distributed. Independence is violated, because in particular the $Y_i$ have to sum up to zero, meaning that if you know $n-1$ of them, you know exactly what the last one has to be.

The marginals have mean $\mu_Y=0$ and variance $\sigma_Y^2=\frac{n-1}{n}$, however they are not normal, see below. That their mean is zero is easy to check (note that if random variables that are identically distributed sum up to zero, all their expected values must be zero regardless of dependence). The variance can be obtained from a similar argument, realising that $\sum Y_i^2=n-1$.
However, the marginals cannot be normal, because for example, if $n=2$, $Y_1=3$ is impossible because then $Y_2=-3$ because of $\sum Y_i=0$, and $\sum Y_i^2=9\neq 1$, which it need be. This means that the range of values of $Y_i$ is bounded, hence cannot be normal. (In fact, if $n=2$, $Y_i$ can only be $\pm\sqrt{0.5}$; from $n\ge 3$ it should be continuously distributed, but still with bounded range, as too large absolute values would spoil $\sum Y_i^2=n-1$.)
I had earlier claimed that  $Y_i$ is normally distributed conditionally on $\hat \mu$ and $\hat\sigma$, but that's not true either, because knowing $\hat \mu$ and $\hat\sigma$ doesn't change the argument above.
Interestingly, if you plot $Y_1,\ldots,Y_n$, they will have the same distributional shape as $X_1,\ldots,X_n$, which are i.i.d. normal, just with different mean and sd, so normality will be kept in a certain sense, but I don't know how to make that a formal statement.

Many names are in use, see https://en.wikipedia.org/wiki/Standard_score
Comment: What is gained by such standardization?  Why use a method that assumes a symmetric distribution?  Why use non-robust statistical quantities to do the standardization?
Comment: @FrankHarrell: Not sure why you discuss this in the comments to my answer, because my answer addresses the distributional issue raised by mhdadk (which I think is pretty interesting) without making specific recommendations about how to standardise in practice.
Comment: "However in practice what is relevant is that $Y$ is normally distributed *conditionally* on $\hat{\mu}$ and $\hat{\sigma}$." What does this sentence mean exactly?
Comment: I think you could have been clearer by writing "$Y$ is normally distributed given $X_1,X_2,...,X_{1000}$", since both $\hat{\mu}$ and $\hat{\sigma}$ are functions of $X_1,X_2,...,X_{1000}$.
Comment: I was wrong and have changed my answer.
Comment: Thanks for the clarification.
Comment: Be patient, there's still something wrong.
Comment: Interesting. I appreciate all the explanation I can get. Upvoted for the effort.
Comment: Not sure about this paragraph: "I had earlier claimed that $Y_i$ s normally distributed conditionally on $\hat{\mu}$ and $\hat{\sigma}$, but that's not true either, because knowing $\hat{\mu}$ and $\hat{\sigma}$ doesn't change the argument above." From what I understood, since $$Y_i = \frac{X_i - \hat{\mu}}{\hat{\sigma}}$$ then if you are given all $X_1,X_2,...,X_{1000}$, $Y_i$ would no longer be random, since the right-hand side of this equation is a constant.
Comment: "Conditionally on $\hat\mu$ and $\hat\sigma$" means that  $\hat\mu$ and $\hat\sigma$ are given, but *not* all $X_1,\ldots,X_n$.
Comment: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/127601/discussion-between-mhdadk-and-lewian).
Comment: @FrankHarrell I used to perform this standardization before I realized that the sample mean and standard deviation are not the same as the population mean and standard deviation. More precisely, after sampling some data, I would compute the sample mean and standard deviation and I would find that they are not exactly equal to the population mean and standard deviation. To fix this, I would perform this kind of standardization to make sure the sample mean and standard deviation match the population mean and...
Comment: ...standard deviation. Of course, I no longer perform this kind of standardization, since I am aware that the sample mean and standard deviation will never be equal to the population mean and standard deviation. However, I was curious to see how this standardization procedure affects the distribution of $Y_i$, and therefore see how it would affect maximum likelihood estimation of the parameters of the distribution from which the $X_i$ were sampled from.
Comment: @Lewian since $\hat\mu$ and $\hat\sigma$ are functions of $X_1,X_2,...X_{1000}$, how would $\hat\mu$ and $\hat\sigma$ be given without being given $X_1,X_2,...X_{1000}$?
Comment: @mhdadk you are making the common error of believing that the sample mean and standard deviation are adequate parameters to be used for standardization.   They have problems even if sample estimates are close to population truths.
Comment: @FrankHarrell would you like to expand on this in an answer? Specifically, what problems do the sample and standard deviation have and what better statistics can be used for standardization instead.
Comment: First of all I question the use of standardization.  But if one **must** standardize, just question everything.  Mean and SD are not robust, SD assumes symmetric distributions, and require distributions to be somewhat similar to normal distributions.  There are many quantile-based measures that have fewer problems but require larger sample sizes.
Comment: @mhdadk: "How would $\hat\mu$ and $\hat\sigma$ be given without being given $X_1,X_2,,\ldots,X_{1000}$?" I'm addressing theoretical questions here and conditional probabilities given $\hat\mu$ and $\hat\sigma$ but not $X_1,X_2,,\ldots,X_{1000}$ are well defined. Somebody may have told you what $\hat\mu$ and $\hat\sigma$ are but not the original data. What's the problem with that?
Answer: If $X$ is a standard normal, than $Y = (X - \mu)/\sigma$ is normally distributed with mean $\mu$ and standard deviation $\sigma$, as normal distribution is of location-scale family.

However, I feel that this standardization procedure is a bit
misleading, since y is no longer normally distributed.

What is y in here? You took 1000 samples from a normal distribution and calculated the sample mean and sample standard deviation of the samples. In statistics, you treat those 1000 observed values as 1000 i.i.d. normally distributed random variables (not as a random variable). You transformed the samples. The statistics you used are estimates of the true parameters, so you can think of the result of the operation as approximation of what you would observe if you knew the population parameters and transformed the random variables.
The calculated statistics are numbers, you used those numbers to do the transformation. You can use any numbers to scale a variable. Subtracting sample mean and dividing by sample standard deviation is called normalization, standardization, or converting to $z$-scores. You use the sample statistics here, not the population parameters, because you don't know the population parameters.
Moreover, you usually care about transforming the samples, not the underlying random variables. Standardization is done mostly for numerical reasons, to ease up the computations. Having samples with a non-zero mean and standard deviation other than one, even if the samples were taken from a standard distribution, would not help with the computations. Don't confuse samples and results of the computations with random variables.
Comment: @mhdadk you transform $X$ by adding a number and dividing by a number, it's still normal. You are transforming the samples that you observed, not transforming the random variable using the sample statistics.
Comment: @mhdadk no it wouldn't. If you sample from $\mathcal{N}(0,1)$, then the mean of the population is 0 and sd of the population is 1, sample means are different. If you sample from other distribution and standardized, then the mean of the sample is 0, and sd of the sample is 1. You're confusing random variables with their realizations.
Comment: While I appreciate the practical aspect of your answer, I am actually interested more in the last paragraph of your answer than the rest of it. Could you expand on this? I was told that standardization using sample statistics is never a good idea since it modifies the underlying random variable, but I am not sure how to formalize this.
Comment: I've edited my question to explain what I mean.
Comment: @mhdadk but you almost never know the population parameters, so you cannot standardize using them. Btw, I added a numerical example to illustrate the answer.
Comment: thanks a lot for the numerical example. It seems that $Y_i$ is indeed non-normal for small sample sizes and approaches normality as the sample size increases, correct?
Comment: @mhdadk yes, but as said above, you don't standardize random variables but samples. I'm removing the example since it's completely misleading.
Comment: wouldn't knowing that the $Y_i$ are no longer normal be important if you are trying to estimate the parameters of a normal distribution using $Y_i$ instead of $X_i$? It would be like trying to estimate the parameters of one distribution given samples from another distribution.
Comment: @mhdadk indeed non-normal for small sample sizes - (1) with a small sample, you could not reject the $Y_i$a are from a normal distribution; (2) the size of the sample does not change the distribution generating the sample.
Comment: @Tim by the way, I thought your numerical example was informative. Maybe add it back with a disclaimer?
Comment: @mhdadk it wasn't, because it was showing something that never happens. It was showing what would happen if you calculated the statistics on one sample and applied to another sample.
Comment: @Tim understood. Any ideas on my previous comment?
Comment: @Tim got it, thanks. If I sample $\mathcal{N}(\mu,\sigma)$ then standardize these samples, would this be the same as sampling from $\mathcal{N}(0,1)$?
Comment: @NickCox yep, thanks, muscle memory to add `^2`.


Rendering a field in a custom print template

Question: I'm using the Printer, email and PDF versions module to render a custom print page in Drupal 7. I'd like to theme the fields individually and have worked that out to the point of using nasty php in my custom template itself.
So, I have a content type called ecard and following the module's documentation, I'm using print--node--ecard.tpl.php (based on the modules's core print.tpl.php file) for my custom print template. That works fine but the usual ways of rendering individual fields like in a node template don't.
I can do this within print--node--ecard.tpl.php to render an indivdual field called field_ecard_logo:
<?php
$ecard_logo = field_view_value('node', $node, 'field_ecard_logo', $node->field_ecard_logo[$node->language][0]);
print render($ecard_logo);
?>

... works but kind of nasty. I'd rather put all the logic in my theme's template.php and simply print a variable I've created. Looking at what preprocess functions the Print module uses, I see preprocess_print and preprocess_node. So I tried this in template.php:
  function MYTHEME_preprocess_print($vars) {
      $node = $vars['node'];
      $vars['ecard_logo'] = field_view_value('node', $node, 'field_ecard_logo', $node->field_ecard_logo[$node->language][0]);
    }

and then in print--node--ecard.tpl.php, I simply do:
<?php print $ecard_logo; ?>

but that does not work. I tried different incarnations of all this but no joy so at this point I'm stumped.
Comment: You should pass $vars variable by reference..
Answer: After a few days of working on this, it ended up being pretty trivial to accomplish this so I'll answer my own question for those who need help in the future. I had most of the solution working but in my custom print template, the key was to have:
<?php print render($ecard_logo); ?>

NOT:
<?php print $ecard_logo; ?>

So the full function in template.php is:
/**
  * Renders the content of the print page using the theme api.
  */
function usfig_preprocess_print(&$vars, $hook) {

// Global node.
$node = $vars['node'];
// Compartmentalize only for ecard content type.
if($vars['node']->type == 'ecard') {

// Create a var and render using field_view_value, we don't need anything special for this, otherwise use field_get_items for data parts.
$vars['ecard_logo'] = field_view_value('node', $node, 'field_ecard_logo', $node->field_ecard_logo[$node->language][0]);
}

}

... and then in my custom print template I do:
 <?php print render($ecard_logo); ?>

and it works!
Extra credit:
You can create and render an image style with field_view_value as well like so in your theme's template.php file:
 $vars['ecard_image'] = field_view_value('node', $node, 'field_my_image', $node->field_my_image[$node->language][0], array(
      'type' => 'image',
      'settings' => array(
      'image_style' => 'my_custom_image_style',
      'image_link' => 'none',
      ),
      ));

Then just render the variable in your custom template file!  What's the advantage to all this? It allows for ultra minimalist markup and precise themeing where layout and exactness are crucial. It cuts way down on the "divitis" that drupal tends to have.
Answer: From personal experience in these situations it's better to incorporate all the logic into a custom module, so basically you create there your functions for processing data, and these functions are available throughout the whole site by calling the function.
In your current code i think that the variable can be accessed by defining it in your tpl through GLOBAL $ecard_logo; . I haven't tested this so let me know if this helps.
Cheers


How to run commerce_order_save() without triggering any rules?

Question: I need to run commerce_order_save() on several Orders in loop but without triggering any Rules.
Is that possible ?
Answer: It's possible, but the change would need to be sitewide (not just in the context of rules). You'd need to override the entity controller class used for commerce_product (which you could do with hook_entity_info_alter()), and re-implement its save() method so it doesn't invoke the entity hooks.
If that is undesirable (probably is), and you just need to update field values, you can use field_attach_update() instead, which will bypass the entity save process altogether:
$order->some_field[LANGUAGE_NONE][0]['value'] = 'foo';
field_attach_update('commerce_order', $order);

Other than those I don't think there's any way to accomplish it using the API.
Comment: I just need to update the content of a field and it needs to be saved in database. That's it.


Really: why is the Kelvin transform harmonic?

Question: So, it is a famous fact that if $u:\mathbb{R}^n \to \mathbb{R}$ is an harmonic function, then its Kelvin transform
$$
(Ku)(x) := \frac{1}{|x|^{n-2}} u\left(\frac{x}{|x|^2} \right)
$$
is harmonic too. Apparently, all the proofs of this fact that I have seen are some variation of "do the computations and all the terms simplify". Moreover, I already did the computation once and so I am not interested in seeing the computations done in some other equivalent form.
My point is the following: why, really, is this function harmonic? I don't buy the fact that the first person that discovered this just randomly did the computations with the correct power of the norm in front. For $n=2$ this can be proved very easily by properties of holomorphic functions and indeed there's a geometric motivation, in this case $n=2$, on why it sends harmonic functions to harmonic functions. But what for general $n$? Does anyone have at least some geometric/physical intuition on why $Ku$ is harmonic if $u$ is?
I tried many characterizations of harmonic functions like the mean value property or integrating against test functions, and all seem not to bring the result in a clean way. Indeed, at some point, there's always a big computation (being that some tangential Jacobian or Laplacian of composition) that I find basically equivalent to doing the computation from the beginning.
I am searching for a proof or at least some motivation on why would someone think that this transformation is the natural candidate to send harmonic functions to harmonic functions.
Comment: One guess as to why you'd want to look at (a variant) of this transformation: Conformal maps on $\mathbb{C}$ send harmonic functions to harmonic functions, and in higher dimensions, reflections across spheres are the prototype of conformal transformations (see [Liouville's theorem](https://en.wikipedia.org/wiki/Liouville%27s_theorem_(conformal_mappings))). This means looking at $u(x/|x|^2)$ which is not harmonic, but the extra factor $|x|^{2-n}$ is the fundamental solution, so you might guess it based on the computations for $u(x/|x|^2)$ and formulas for the laplacian of a product.
Comment: @Jose27 I am not sure what you said is really convincing. If the reason for this would be that $|x|^{2-n}$ is the fundamental solution in dimension $n\ge 3$ then for $n=2$ it would be $\log(|x|) u(x/|x|^2)$ but in this case it works without the $\log$ in front.
Comment: Compute the expression for $\Delta$ in $y$ coordinates under the change of coordinates $x = y/|y|^2$. The Kelvin transform pops up. See Folland's PDE book.
Comment: I could have phrased it a bit better: My comment is more along the lines of "why would we ever look at something like $u(x/|x|^2)$?" A possible answer is that this inversion is conformal. In two dimensions, you stop here, since composition with inversion does give a harmonic function. In higher dimensions this is no longer true, but you might **guess** the factor that's missing if you run the computations. I make no claim to justify why the factor $|x|^{2-n}$ is there other than it makes things work, hence my second comment.
Comment: I'd be really interested to see a geometric reason for the $|x|^{2-n}$ factor, without (or at least motivated) computations!
Answer: If the pull-back of a harmonic function $H$ is some radial multiple  $f(r)$  of another harmonic function,  the constant function 1 must be pulled back  to $f(r)$, i.e.  $f(r)$ is a radial harmonic function. This motivates trying $f(r)= r^{-(n-2)}$.


How to solve the SyntaxError: Unexpected identifier problem occurring during compilation of ExpressJS (ejs) file?

Question: <% include partials/header.ejs %>

        <% if (players.length > 0) {%>

                        ID
                        Image
                        First Name
                        Last Name
                        Position
                        Number
                        Username
                        Action

                    <% players.forEach((player, index) => { %>

                            <%= player.id %>
                            <img src="/assets/img/<%= player.image %>" class="rounded-circle player-img" alt="">
                            <%= player.first_name %>
                            <%= player.last_name %>
                            <%= player.position %>
                            <%= player.number %>
                            @<%= player.user_name %>

                                <a href="/edit/<%= player.id %>" target="_blank" rel="noopener" class="btn btn-sm btn-success">Edit
                                <a href="/delete/<%= player.id %>" class="btn btn-sm btn-danger">Delete

                    <% }) %>

        <% } else { %>

No players found. Go here to add players.

        <% } %>

This is the error being displayed after running the .ejs file from terminal,

SyntaxError: Unexpected identifier in C:\Users\A\Desktop\Ms\demo-app\node-mysql-crud-app\views\index.ejs while compiling ejs
If the above error is not helpful, you may want to try EJS-Lint:
https://github.com/RyanZim/EJS-Lint
Or, if you meant to create an async function, pass async: true as an option.
    at new Function ()
    at Template.compile (C:\Users\A\Desktop\Ms\demo-app\node-mysql-crud-app\node_modules\ejs\lib\ejs.js:626:12)
    at Object.compile (C:\Users\A\Desktop\Ms\demo-app\node-mysql-crud-app\node_modules\ejs\lib\ejs.js:366:16)
    at handleCache (C:\Users\A\Desktop\Ms\demo-app\node-mysql-crud-app\node_modules\ejs\lib\ejs.js:215:18)
    at tryHandleCache (C:\Users\A\Desktop\Ms\demo-app\node-mysql-crud-app\node_modules\ejs\lib\ejs.js:254:16)
    at View.exports.renderFile [as engine] (C:\Users\A\Desktop\Ms\demo-app\node-mysql-crud-app\node_modules\ejs\lib\ejs.js:459:10)
    at View.render (C:\Users\A\Desktop\Ms\demo-app\node-mysql-crud-app\node_modules\express\lib\view.js:135:8)
    at tryRender (C:\Users\A\Desktop\Ms\demo-app\node-mysql-crud-app\node_modules\express\lib\application.js:640:10)
    at Function.render (C:\Users\A\Desktop\Ms\demo-app\node-mysql-crud-app\node_modules\express\lib\application.js:592:3)
    at ServerResponse.render (C:\Users\A\Desktop\Ms\demo-app\node-mysql-crud-app\node_modules\express\lib\response.js:1012:7)

I am new to expressjs and this a code from a demo app in GitHub.
  It would be a great help if this problem is solved.
Answer: Read this document https://github.com/tj/ejs and i think your foreach loop having some problem check this first and then try once.
 <% for(var i=0; i<players.length;i++) { %>
                                <tr id="idrow_<%= players[i]._id%>">

                                        <%= players[i]._id%>

                                        <%= players[i].first_name%>

                                        <%= players[i].last_name%>

                                        <%= players[i].position%>

                                        <%= players[i].number%>

                                        <%= players[i].user_name%>

                                    <button type="button" class="btn btn-secondary" id="edtbtn" onclick="edituser('<%= players[i]._id %>')" name="edit">edit</button>
                                    <button type="button" class="btn btn-secondary" id="delbtn" onclick="deleteuser('<%= players[i]._id %>')" name="delete">
                                    Delete
                                </button>

                                <% } %>

========================================================================


Problem accessing static variables from another thread

Question: I've got a class
class KeyConstants:
   VK_CTLR = 37
   VK_SHIFT = 50
   VK_ALT = 65

and some code using the class
class EventListener(threading.Thread)
    ... snip ...
    def is_modkey(self, event):
         return event.detail == KeyConstants.VK_CTRL

And then I get AttributeError, KeyConstants does not have attribute VK_CTLR , why can't I access the static variables from another thread?
Comment: The thread shouldn't have anything to do with this.
Comment: I noticed that the attribute is defined as `VK_CTLR` but used as `VK_CTRL` (with R and L swapped)... is it so also in your original code?
Answer: I noticed that the attribute is defined as VK_CTLR but used as VK_CTRL (with R and L swapped)... Maybe it is just a typing error...
Answer: If you can access KeyConstants, you should be able to access it's attributes. The error is likely somewhere else.


(Python - openpyxl) Can't import the exported xlsx file

Question: I'm using openpyxl to import/export xlsx files.
But I can't import load_workbook() the file that i exported with openpyxl.
I must open the exported xlsx file in Excel, change some value of the imported xlsx file and save it (its size changes also (for ex: from 148 Ko to 180 Ko). Then i can import it with openpyxl.
I think that there is a problem in export because after saving manually import works.
Thanks.
Comment: It is unclear where you import and export from. Are you using an external program that does use `import` and `export` or are you just using that terminology instead of `load_workbook()` and `Workbook.save()`. If you provide some code, that would be helpful preferably with something that creates the file that cannot be 'imported'. Did you try to read the faulty file with Excel?
Answer: It seems this doesn't currently work: https://bitbucket.org/ericgazoni/openpyxl/issue/46/problem-reading-one-excel-file-and-writing
import openpyxl
wb = openpyxl.Workbook()
ws = wb.get_active_sheet()
ws.cell('A1').value = 1
wb.save('test.xslx')
wb2 = openpyxl.load_workbook('test.xlsx')
ws2 = wb2.get_active_sheet()
ws2.cell('A2').value = 2
wb2.save('test2.xlsx')


Given $f$ differentiable & nonzero on $[1, \infty)$, $\lim_{x\to\infty} \frac{x f'(x)}{f(x)} < -1$ and non-infinite. Show $\int_1^\infty f$ converges

Question: We know that $1 + \frac{x f'}{f} < 0$.
I'm unsure how to proceed but guessing the proof involves knowing that:
If $\lim_{x \to b} \frac{f(x)}{g(x)}$ exists and is greater than 0, then $\int_a^b g(x)$ converges if and only if $\int_a^b f(x)$ converges.
If $\lim_{x \to b^-} (f(x))^\frac{1}{x}$ exists in the extended reals, then $\int_a^b f(x)$ converges if the limit is less than 1 and diverges otherwise.
Answer: Since $f$ is continuous and non zero, then $f$ is positive or negative.
Assume that $f$ is positive.
By assumption, there exist numbers $\alpha < -1$ and $K \geq 1$ such that
$$
\frac{x f'(x)}{f(x)} \leq \alpha \qquad \forall x\geq K,
$$
i.e.
$$
\frac{d}{dx} \log f(x) = \frac{f'(x)}{f(x)} \leq \frac{\alpha}{x}
\qquad \forall x\geq K.
$$
Hence
$$
\log \frac{f(x)}{f(K)} \leq \alpha \log \frac{x}{K},
\qquad\forall x\geq K,
$$
i.e. $f(x) \leq C x^{\alpha}$ for every $x\geq K$, for some constant $C>0$.
Since $f\in C([1,\infty))$ and $\alpha < -1$, this implies that $\int_1^\infty f < \infty$.
A similar argument works also in the case $f<0$.
Answer: Hint if $\lim_\limits{x\to +\infty}\frac{x f^\prime(x)}{f(x)} = s < -1 $, define $f(x) = g(x) x^s$ and substitute in $\frac{x f^\prime(x)}{f(x)}$.


Manage large amount of environment specific test data in testNG

Question: I have an enterprise android application and a package created for testing the app. The package runs on a framework that uses Appium and testNG. The test package has various tests that check the content, labels, functionality of the app. The package is able to run tests on the app in various stages like alpha, beta.. All the different stages have different test data.
For example, let's take a food delivery application. In this application, the testing is done through a test package that can run in various development stages (alpha, beta, gamma, prod). There can be different types of orders like with long and short distances, discount and without discount, delivery orders and dine-in bookings etc. which will be different for different stages.
I need a way of managing this test data in a generic way in JAVA. Like in JS, we can have TestData.BETA.DISCOUNT_ORDER.EMAIL_ID,
TestData.PROD.DISCOUNT_ORDER.EMAIL_ID
I have tried java relflection but it becomes very complex when dealing with maps, innerclasses and maps inside innerclasses.
Answer: I ended up creating classes like:
CommonTestData

which is inherited by
AlphaTestData
BetaTestData
GammaTestData
ProdTestData

and an interface implemented by all of these that would return the correct object based on the current environment stage.
TestDataController


How to disabled Drupal builtin "poormanscron" cron when using Ultimate cron?

Question: I want to disabled Drupal 7 builtin "poormanscron" to then put it into a crontab task as it should, but when using Ultimate cron module i do not see an option to disabled Drupal 7 builtin "poormanscron"
Do i have to uninstall ultimate cron?
Answer: You don't need to uninstall any module. The setting you are looking for is on admin/config/system/cron/settings/poorman.

Change Launcher to -- Disabled -- from the default Serial, and Poormanscron should not run anymore.
If you disable or uninstall the Ultimate cron module, Drupal shows the equivalent settings in admin/config/system/cron, at the bottom of the page.

In this case, change Run cron every to Never.
Comment: Yes but Ultimate Cron module overwrite that system path
Comment: I think i got it, under admin/config/system/cron/settings/poorman there is an option which says "Select the launcher to use for handling poormans cron." and i can desabled so i guessing it is that. The select Defaults to "Serial"
Comment: Looks like it is [https://www.drupal.org/node/2473575](https://www.drupal.org/node/2473575)
Comment: I misread the module name, and I thought you were referring to Poormanscron which is part of Drupal 7. I will update my answer.


Stop task when there is an exception thrown in ItemProcessor

Question: I am designing a Spring Batch, which reads multiple csv files. I have used partitioning to read each file in chunk and process it to decrypt a certain column in the csv. Before decrypting if i encounter any validation error , i throw custom exception.
Now what i want is if the processing finds any validation error in the first line, the other lines should not be processed, and the job should end.
How can i achieve this? I tried to implement ProcessorListener too but it has no StepExecution object so that i can call SetTerminateOnly() or ExitStatus=Failed
Also note that i have multiple thread accessing the file in different lines.I want to kill all threads in the event of the first encountered error.
Thanks in advance
Answer: So, I identified that running multiple asynchronous concurrent threads (Spring Batch partitioning) was the real issue. Though one of the thread threw an Exception, the other threads were parallely running, and finished executing till the end.
Ath the end, the Job FAILED overall and there was no output processed, but it consumed time to process rest of the data.
Well,the solution to it is as simple as it gets. We just need stop the Job while encountering an error during processing.
The Custom Processor
  public class MultiThreadedFlatFileItemProcessor implements ItemProcessor<BinFileVO, BinFileVO>,JobExecutionListener{
    private JobExecution jobExecution;
    private RSADecrypter decrypter;

    public RSADecrypter getDecrypter() {
        return decrypter;
    }

    public void setDecrypter(RSADecrypter decrypter) {
        this.decrypter = decrypter;
    }
    @Override
    /**
     This method is used process the encrypted data
     @param item
     * */
    public BinFileVO process(BinFileVO item) throws JobException {
        if(null!=item.getEncryptedText() && !item.getEncryptedText().isEmpty()){
            String decrypted = decrypter.getDecryptedText(item.getEncryptedText());
            if(null!=decrypted && !decrypted.isEmpty()){
                 if(decrypted.matches("[0-9]+")){
                     if(decrypted.length() >= 12 && decrypted.length() <= 19){
                         item.setEncryptedText(decrypted);
                     }else{
                         this.jobExecution.stop();
                         throw new JobException(PropertyLoader.getValue(ApplicationConstants.DECRYPTED_CARD_NO_LENGTH_INVALID),item.getLineNumber());
                     }
                 }
         }else{
             this.jobExecution.stop();
             throw new JobException(PropertyLoader.getValue(ApplicationConstants.EMPTY_ENCRYPTED_DATA),item.getLineNumber());
         }
         return item;
     }
    @Override
    public void beforeJob(JobExecution jobExecution) {
            this.jobExecution=jobExecution;
    }
    @Override
    public void afterJob(JobExecution jobExecution) {

    }
  }

The Job xml config
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
   .....>

  <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean" />
  <bean id="jobLauncher"    class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
     <property name="jobRepository" ref="jobRepository" />
  </bean>

    <job id="simpleMultiThreadsReaderJob" xmlns="http://www.springframework.org/schema/batch">
        <step id="step" >
            <partition step="step1" partitioner="partitioner">
                <handler grid-size="5" task-executor="taskExecutor"/>
            </partition>
        </step>
        <listeners>
            <listener ref="decryptingItemProcessor"/>
        </listeners>
    </job>

    <step id="step1" xmlns="http://www.springframework.org/schema/batch">
        <tasklet>
            <chunk reader="itemReader" writer="itemWriter" processor="decryptingItemProcessor" commit-interval="500"/>
            <listeners>
                <listener ref="customItemProcessorListener" />
            </listeners>
        </tasklet>
    </step>

    <bean id="decryptingItemProcessor" class="com.test.batch.io.MultiThreadedFlatFileItemProcessor">
        <property name="decrypter" ref="rsaDecrypter" />
    </bean>

    <bean id="rsaDecrypter" class="test.batch.secure.rsa.client.RSADecrypter"/>

    <bean class="org.springframework.batch.core.scope.StepScope" />
    <bean id="partitioner" class="com.test.batch.partition.FlatFilePartitioner" scope="step">
        <property name="resource" ref="inputFile"/>
    </bean>
     <bean id="taskExecutor"
class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
        <property name="corePoolSize" value="10"/>
    </bean>

    <bean id="transactionManager" class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />
    ........
    .................
</beans>

Here are the logs

2016-09-01 06:32:40 INFO  SimpleJobRepository:273 - Parent JobExecution is stopped, so passing message on to StepExecution
2016-09-01 06:32:43 INFO  ThreadStepInterruptionPolicy:60 - Step interrupted through StepExecution
2016-09-01 06:32:43 INFO  AbstractStep:216 - Encountered interruption executing step: Job interrupted status detected.
; org.springframework.batch.core.JobInterruptedException
2016-09-01 06:32:45 ERROR CustomJobListener:163 - exception :At line No. 1 : The decrypted card number is less than 12 or greater than 19 in length
2016-09-01 06:32:45 ERROR CustomJobListener:163 - exception :Job interrupted status detected.
2016-09-01 06:32:45 INFO  SimpleJobLauncher:135 - Job: [FlowJob: [name=simpleMultiThreadsReaderJob]] completed with the following parameters: [{outputFile=/usr/local/pos/bulktokenization/csv/outputs/cc_output_EDWError_08162016.csv, partitionFile=/usr/local/pos/bulktokenization/csv/partitions/, inputFile=C:\usr\local\pos\bulktokenization\csv\inputs\cc_input_EDWError_08162016.csv, fileName=cc_input_EDWError_08162016}] and the following status: [FAILED]
2016-09-01 06:32:45 INFO  BatchLauncher:122 - Exit Status : FAILED
2016-09-01 06:32:45 INFO  BatchLauncher:123 - Time Taken : 8969
Answer: If we throw Custom Exception in Processor, Spring Batch will terminate and mark the job failed unless you setup 'skipable' exception. You have not mentioned where you perform validate step, are you doing in Processor or Reader? Let me know because it is where Spring Batch decides.
In my project, if I want to stop the job and throw Custom Exception, we put validation logic in a Tasklet or Processor and throw exception as below
private AccountInfoEntity getAccountInfo(Long partnerId) {
        if(partnerId != null){
            .....
            return ....;
        } else {
            throw new ReportsException("XXXXX");
        }
    }

Comment: I am doing in Processor..i use BinFileVO process(BinFileVO item) throws Exception{ If(item.getEncryptedText().length > 10) throw new Exception("Invalid input length"); return item }.    This does not stop the job to process the rest of the file. It only shows the exception at last in afterJob in JobListener as JobExecution.getAllFailureExceptions and shows Job Status = FAILED
Comment: I am implementing ItemProcessor interface and throwing the exception from it. But it will always return the item.. i tried using ItemProcessorListener too and use the method afterProcess and onProcessError. But it has no stepexecution object to stop the batch explicitly
Comment: Are you sure you don't setup 'retry-policy' or 'skippable-exception-classes'?

If you want to have StepExecution object in Processor, you can achieve it as below

1. Implement StepExecutionListener for your Processor
2. Declare a StepExecution variable in your Processor
3. In BeforeStep method, save stepExecution object
4. In Process method, you can use it.
Comment: No i am not declaring any retry-policy or skippable exception. Do i have to inject stepexecution in my processor bean and stepexecutionlistener bean? And will it be shared between the two? I will try it and share the code here
Comment: You can try as below
public class XXX implements ItemProcessor, StepExecutionListener {
 private StepExecution stepExecution;

 public void beforeStep(StepExecution stepExecution) {
        this.stepExecution = stepExecution;
    }

    public X process(X item) throws Exception {
  // you can start using this.stepExecution
 }
}
Comment: Thanks a lot i will try it and update your answer. :)
Comment: Thank you for your time. Please look at my answer i have posted above. I need a different question now as to how to terminate all threads if one fails in spring batch partitioning. I know that Spring partitioning uses asynchronous threads, and that is creating a huge problem for me.
Comment: I think you can research a bit about SimpleJobOperator. That is one option I can think of right now. You can have a bean for that one and inject it to your Processor. And if there is any exception happens in your processor, you can stop the job by using 'stop' method of SimpleJobOperator. The method requires 'jobId', you can have it by implement JobExeuctionListener from your Processor.
Comment: I tried using .stop() method too. But the slave steps(threads) keep executing. The job does not exits at this point. :(
Comment: I have just written an small program to check if jobOperator works as we thought. I do see it is working. Please refer to the link https://bigzidane.wordpress.com/2016/08/30/stop-spring-batch-job-if-a-thread-of-the-processor-caught-an-exception/
Comment: You are using `JobOperator` , but here i am using `JobLauncher` to run my job, and return the `ExecutionContext`. Should i set `Job` in JobOperator and start my job execution by calling the start method? Your above code is still not working for me.
Comment: I'm also using JobLauncher to launch the job. Please see the code I updated from the above link. I just use JobOperator to stop the job in Processor.
Comment: I updated my answer. I was able to achieve it by implementing JobExecution listener in processor and declaring my processor as "job listener" as well as "item processor" in the Job. :)


How can I calculate the partial derivative to a function that uses matrices with frobenius norm?

Question: I have not taken calculus for while so I need some help with the following partial derivative. If I am shown how to do this one I should be able to figure out the rest of them.
So I have
$$f(A,B,C) = ||P - AB^T||^2 + ||Q - AC^T||^2$$
where A,B,C,Q,P are matrices and $|| ||$ is the frobenius norm.
and I want partial derivative:
$$\frac{\partial{f}}{\partial B}$$
Comment: Differentiating with respect to a matrix is quite a bit more advanced than what is covered in typical calculus courses! You may have to look up the identities for [matrix calculus](https://en.wikipedia.org/wiki/Matrix_calculus#Scalar-by-matrix_identities), and use the fact that $\|A\|^2 = {\rm tr}(A^TA)$.
Comment: @Rahul ok thanks. Yeah I have been asked to perform this derivative and it isn't like anything I have seen before. I only have undergraduate calculus courses.
Answer: Let's use a colon to denote the trace/Frobenius product
$$A:B = {\rm tr}(A^TB)$$
and define new variables
$$\eqalign{
 X &= BA^T-P^T \cr
 Y &= CA^T-Q^T \cr
}$$
Write the function in terms of these new symbols. Then find its differential and gradient
$$\eqalign{
 f &= X:X + Y:Y \cr
df &=2X:dX + 2Y:dY \cr
 &= 2X:dB\,A^T + 0 \cr
 &= 2XA:dB \cr
 &= 2(BA^TA - P^TA):dB \cr
\frac{\partial f}{\partial B} &= 2(BA^TA - P^TA) \cr\cr
}$$
There are lots of rules for rearranging the terms in a Frobenius product, which follow from the properties of the underlying trace function.
For example, all of the following are equivalent
$$\eqalign{
 A:BC &= B^TA:C\cr &= AC^T:B\cr &= A^T:(BC)^T\cr &= BC:A \cr
}$$


how to lock linux user account after certain unsuccessful attempts

Question: How to lock user account on Red Hat Enterprise Linux Server release 5.7 after certain no of unsuccessful attempts
Answer: Use the pam_tally module in your PAM authentication setup. Insert as desired into the relevant /etc/pam.d/* configuration files.
Sample:
auth     required       pam_tally.so deny=10 unlock_time=60 per_user

10 failed logins generates a 1 minute lockout.
Comment: which pam config file I need to insert this line?? is it /etc/pam.d/system-auth


coloring a cube with six different colors

Question: a cube of length "a", width "b" and depth "c" where "a" is not equal to "b","b" is not equal to "c" and "c" is not equal to "a" . How many different ways are there to color the faces with six different colors?
Comment: and the answer will be 180
Comment: Hint: There are four symmetries: the identity, and a $180^\circ$ rotation around any of the three axes.
Comment: explain please.
Comment: Those are the things you could do to a cuboid like that that don't change its shape. ("Identity" is "doing nothing," which trivially doesn't change the shape.)
Comment: how could you found the symmetries ?
Comment: Experimenting with anything cuboid-shaped (like a book or a phone) gives it to you pretty clearly
Comment: But,  I found three symmetries . please explain how you get four.
Comment: I also counted the identity ("doing nothing")
Comment: what are the four symmetries you have found?
Comment: There are three pairs of opposite faces. For each pair, rotate 180 degrees around the line through the midpoints of those faces. That gives you three. The identity is the fourth.
Comment: i don't understand the fourth symmetry  ("identity") . what that means?
Comment: The identity transformation means leaving the cube as it is.
Comment: i get it . thanks
Comment: A symmetry is something you do to an object that doesn't change its chape. Those rotations don't change the shape, so they're symmetries. Doing nothing doesn't change the shape either, so it's a symmetry. _Every_ object has the identity as one of its symmetries.
Comment: 180 is correct, I believe.
Comment: @Tushar You should explain your reasoning in the question itself rather than the comments, where it might be missed.
Answer: Choose two of the six colors for the largest faces.  Orient the rectangular prism so that one of these faces is on the bottom.  Notice that it does not matter which of the selected colors is on the bottom since the two selected colors will still be opposite each other.
Choose two of the four remaining colors for the next largest faces.  Orient the rectangular prism so that one of these faces is facing you.  Notice that it does not matter which which of the selected colors is facing you since the two selected colors will still be opposite each other and contiguous with both the top and bottom faces.
For the given orientation of the rectangular prism, choose which of the two remaining colors is on the left.  The remaining color is then placed on the right.  Notice that it does matter which color we place on the left.  If we orient the rectangular prism so that the color on the right is facing left without changing the bottom face, then the side facing you is rotated so that it faces away from you.
Hence, there are
$$\binom{6}{2}\binom{4}{2} \cdot 2!$$
distinguishable ways of painting the rectangular prism.
Comment: @ChristianBlatter I realized that I was thinking about a related problem, namely a cube in which each face is painted a different color.  However, in this problem, the faces of the rectangular prism are different sizes. I replaced my (evidently cryptic) hint with an answer.


Logrotate isn't deleting old files

Question: I have a daily backup running using logrotate. I don't have root access to this machine to edit the normal logrotate.conf, therefore I'm calling it via crontab like so:
0 4 * * * /usr/sbin/logrotate /home/myuser/backups/logrotate.conf --state /home/myuser/backups/logrotate.tmp

My logrotate.conf looks like this:
/home/myuser/backups/mysql/production/db.sql.gz {
    daily
    nocompress
    dateext
    missingok
    rotate 30
    extension *.sql.gz
    sharedscripts
    postrotate
        mysqldump --login-path=production mydatabase --single-transaction | $(which gzip) > /home/myuser/backups/mysql/production/db.sql.gz
    endscript
}

The issue is that old "logs" (SQL backups) aren't being rotated off/deleted. They just stick around. Running ls -lt in my backup directory gives me some odd results:
-rw-r--r-- 1 myuser mygroup 3799057 Apr 17 04:00 db.sql.gz
-rw-r--r-- 1 myuser mygroup 3799221 Apr 16 04:00 db.sql.gz-20190417
-rw-r--r-- 1 myuser mygroup 3799167 Apr 15 04:00 db.sql.gz-20190416
-rw-r--r-- 1 myuser mygroup 3799090 Apr 14 04:00 db.sql.gz-20190415
-rw-r--r-- 1 myuser mygroup 3799136 Apr 13 04:00 db.sql.gz-20190414
-rw-r--r-- 1 myuser mygroup 3799107 Apr 12 04:00 db.sql.gz-20190413
-rw-r--r-- 1 myuser mygroup 3792122 Apr 11 04:00 db.sql.gz-20190412
-rw-r--r-- 1 myuser mygroup 3794498 Apr 10 04:00 db.sql.gz-20190411
-rw-r--r-- 1 myuser mygroup 3798082 Apr  9 04:00 db.sql.gz-20190410
-rw-r--r-- 1 myuser mygroup 3793882 Apr  8 04:00 db-20190409.sql.gz
-rw-r--r-- 1 myuser mygroup 3793695 Apr  7 04:00 db-20190408.sql.gz
-rw-r--r-- 1 myuser mygroup 3794418 Apr  6 04:00 db-20190407.sql.gz
-rw-r--r-- 1 myuser mygroup 3790567 Apr  5 04:00 db-20190406.sql.gz
-rw-r--r-- 1 myuser mygroup 3792162 Apr  4 04:00 db-20190405.sql.gz
-rw-r--r-- 1 myuser mygroup 3783503 Apr  3 04:00 db-20190404.sql.gz
-rw-r--r-- 1 myuser mygroup 3781192 Apr  2 04:00 db-20190403.sql.gz
-rw-r--r-- 1 myuser mygroup 3782857 Apr  1 04:00 db-20190402.sql.gz
-rw-r--r-- 1 myuser mygroup 3785592 Mar 31 04:00 db-20190401.sql.gz
-rw-r--r-- 1 myuser mygroup 3771475 Mar 30 04:00 db-20190331.sql.gz
-rw-r--r-- 1 myuser mygroup 3771263 Mar 29 04:00 db-20190330.sql.gz
-rw-r--r-- 1 myuser mygroup 3751782 Mar 28 04:00 db-20190329.sql.gz
-rw-r--r-- 1 myuser mygroup 3632604 Mar 27 04:00 db-20190328.sql.gz
-rw-r--r-- 1 myuser mygroup 3744550 Mar 26 04:00 db-20190327.sql.gz
-rw-r--r-- 1 myuser mygroup 3608617 Mar 25 05:00 db-20190326.sql.gz
-rw-r--r-- 1 myuser mygroup 3642558 Mar 24 05:00 db-20190325.sql.gz
-rw-r--r-- 1 myuser mygroup 3600705 Mar 23 05:00 db-20190324.sql.gz
-rw-r--r-- 1 myuser mygroup 3735972 Mar 22 05:00 db-20190323.sql.gz
-rw-r--r-- 1 myuser mygroup 3600285 Mar 21 05:00 db-20190322.sql.gz
-rw-r--r-- 1 myuser mygroup 3737448 Mar 20 05:00 db-20190321.sql.gz
-rw-r--r-- 1 myuser mygroup 3741454 Mar 19 05:00 db-20190320.sql.gz
-rw-r--r-- 1 myuser mygroup 3603502 Mar 18 05:00 db-20190319.sql.gz
-rw-r--r-- 1 myuser mygroup 3617021 Mar 17 05:00 db-20190318.sql.gz
-rw-r--r-- 1 myuser mygroup 3612493 Mar 16 05:00 db-20190317.sql.gz
-rw-r--r-- 1 myuser mygroup 3753894 Mar 15 05:00 db-20190316.sql.gz
-rw-r--r-- 1 myuser mygroup 3742415 Mar 14 05:00 db-20190315.sql.gz
-rw-r--r-- 1 myuser mygroup 3733984 Mar 13 05:00 db-20190314.sql.gz
-rw-r--r-- 1 myuser mygroup 3594213 Mar 12 05:00 db-20190313.sql.gz
-rw-r--r-- 1 myuser mygroup 3594515 Mar 11 05:00 db-20190312.sql.gz
-rw-r--r-- 1 myuser mygroup 3622672 Mar 10 05:00 db-20190311.sql.gz

Am I somehow mis-understanding the behavior of logrotate?
Comment: You shouldn't use `*` in the `extension` line.
Comment: Oh, why do you say that?
Comment: Because `*.sql.gz` is not the extension, `.sql.gz` is. `*.sql.gz` is a glob pattern matching those files, but that's not what is required when telling logrotate what the extension is.
Comment: I changed the extension, and the weird naming seems to have gone away...however - old logs are still no longer being removed. I currently have 43 in the backups directory.
Answer: Looks like 30 of your files (the oldest ones) are using a naming scheme (db-DATE.sql.gz) and the newer ones are named differently (db.sql.gz-DATE). Also looks like the newer ones have the dateext option applied. This make me think about two possibilities:

You changed the naming scheme before hitting the threshold of files for rotating/deleting, so logrotate will remove the files matching the new naming scheme (db.sql.gz-DATE), but only when there are more than 30 of them (currently there are 9 files). The files with the old naming must be removed manually by you.
You have 2 different logrotate conf files processing the same set of log files.   (probably inside /etc/logrotate.d). Maybe when you tested, you ran logrotate /etc/logrotate.d/your_conf.conf and thus bypassed some settings of /etc/logrotate.conf (like dateext). Now crontab is executing the logrotate main file and generated the difference in the files naming.
Comment: Just an update to this (over 2 years later!) I left things be for a little while and it does look like it's working after the name change...
Comment: Yeah, I did notice the weird naming. The only logrotate config I've edited is the one posted above (this is shared hosting so I don't have root access to edit `/etc/logrotate.d/*`). I haven't changed the naming scheme since I wrote the logrotate config. The only thing I've changed at all is the time that the cron runs. Is it possible that the hosting provider is somehow running another logrotate that's found my config file and is conflicting with it?
Comment: I just changed `*.sql.gz` to `.sql.gz` for the extension setting and the file naming has gone back to the old format. Old logs are not being removed though

Logrotate isn't deleting old files


How to test whether it is valid causal inference

Question: definition of average causal effect
$$ ACE = E(C_1) - E(C_0) $$
$$ ACE = E(Y|X=1) - E(Y|X=0) $$
given the condition that
$$ X \bot (Y(0), Y(1)) $$
So if I have a regression
<code>m <- lm(y~x)
</code>
whose residual is perfectly normally distributed:

Can I assert that coefficient of $X$ is the average causal effect of $X \rightarrow Y$ ?
If so, how can I test whether that $m\$redisuals$ is normally distributed along $X$?
Thank you a lot!
Comment: did this answer your question?
Answer: If you mean that, given your experimental design, you can justify the statements about independence of assignment of treatment from treatment outcome, you can assert the coefficient of X to be the average causal effect of X on Y.
If you mean that you can't necessarily assert the independence of assignment of treatment from treatment outcome, you cannot identify the coefficient of X as the causal effect.
The question of normality of residuals is covered here
`


Forge viewer: setNavigationLock does not work properly in BimWalk mode

Question: If setting the navigation lock when in BimWalk mode, mouse gestures in the viewer seem to take effect when the navigation lock is deactivated. This does not happen in orbit mode.
Steps to reproduce:

Open a model, e.g. at https://lmv.ninja.autodesk.com/
Enable First Person mode (BimWalk)
In the debugging console, call NOP_VIEWER.setNavigationLock(true)
Do a dragging gesture in the viewer. Nothing moves, as expected.
Call NOP_VIEWER.setNavigationLock(false)
The view moves, seemingly according to the last mouse gesture that was made.

I assume this is not expected behavior? Is there a way to avoid it?
Answer: You're right, I've been able to reproduce the issue. Good catch! Let me report the issue to the engineering team, and please stay tuned for updates on the changelog to see when the issue is fixed.
Comment: I see this has been fixed now, in version 7.58. Thanks!


Is it possible to convert an Android device into a midi input device for Logic Pro

Question: I tried this app and used WireShark to see what's being sent, it seems to be sending midi commands over udp, although none is recognized by the network driver in the Audio MIDI Setup.
Also tried ipMIDI and MolCp3 but could't get to the udp settings, it seems those use a custom protocol or some wrapper on top of the midi commands.
Answer: The USB connector of almost all recent Android devices can be switched into "host mode", making it to follow the same communication standard as if it would be USB port on PC.
In most of devices, Android operating system also has the necessary programmer support. However on some devices this is intentionally disabled. There are USB checkers on Play Store that can verify if your device supports the needed protocols before dealing with music-specific stuff.
Hence, in general, yes, in theory it should be possible. It makes sense to try apps that claim to be capable of this, it is not so that cannot be done in principle. However this is a complex programming task and I expect to find many broken/buggy/incomplete implementations in Play Store. Try multiple apps, and if something "almost works", try to talk with developers of the app.
Answer: I couldn't find something like what I was looking for, so I made a small app to capture the midi commands coming from upd and play the corresponding midi notes in a core midi virtual source.


and MYSQL to display data in php

Question: OK so when I try to display data using a MYSQL query and a <textarea> it crashed and dose not run.
It works well without text area (but not all the data fits in the text box).
bellow is the code.
while($row = mysql_fetch_array($result_reviews))
        {
            echo "" .$row['username']. "	";
            echo "" .$row['title']. "	";
            //echo "" .$row['review']. "	";
            echo "<textarea rows="8" cols="50">" .$row['review']. "</textarea>	";

            echo ("
                <button>Edit Review</button>	");

            echo ("
                <button>Deleat Review</button>
");
        }

I need textarea as there is a review I wish to output that can be quite long.
I have looked at some of the others answers for this problem and non seem to work.
Comment: time to migrate to mysqli
Comment: @Shidil Dinesh I think you may be right
Answer: Maybe a conflict with double quote.
Try
echo "<textarea rows=\"8\" cols=\"50\">" .$row['review']. "</textarea>	";

OR
echo '<textarea rows="8" cols="50">' .$row['review']. '</textarea>	';

Comment: Thank you I will give it a go and let you know
Comment: Fantastic, it works. is there anyway to make it read only as I can also write in the textarea and I would like not to?
Answer: If you want to make it "not editable" you can use the "readonly" property.
<textarea readonly>

Answer: Maybe you could use a div with 'overflow-x: scroll'?

and MYSQL to display data in php


Shouldn't the joint probability of 2 independent events be equal to zero?

Question: If the joint probability is the intersection of 2 events, then shouldn't the joint probability of 2 independent events be zero since they don't intersect at all? I'm confused.
Comment: The probability that I watch on a given day TV is 1/2. The probability that it rains on a given day is 1/2. These are independent events. What is the probability of me watching TV on a rainy day?
Comment: @user1936752 Strictly speaking, your example events are *not* independent for most people (e.g., they might be more willing to spend time outdoors when it does not rain)
Comment: @Gaston: Don't confuse "independent" with "mutually exclusive". _Independent_ events are completely unrelated to each other, whereas _mutually exclusive_ events are **inherently** related. For example, suppose I flip two coins: whether I get heads on Coin 1 is unaffected by the result of Coin 2, but it's inherently connected to whether I get tails on Coin 1! =)
Comment: This video [here](https://www.youtube.com/watch?v=pV3nZAsJxl0&list=PLvxOuBpazmsOGOursPoofaHyz_1NpxbhA&index=5) and [this](https://www.youtube.com/watch?v=UOfsad9WWwk&index=6&list=PLvxOuBpazmsOGOursPoofaHyz_1NpxbhA) other one will be helpful in understanding these concepts.
Comment: @HagenvonEitzen OK, good point. Change *rainy day* to *eat chocolate*.
Answer: There is a difference between

independent events: $\mathbb P(A \cap B) =\mathbb P(A)\,\mathbb P(B)$, i.e. $\mathbb P(A \mid B)= \mathbb P(A)$ so knowing one happened gives no information about whether the other happened
mutually disjoint events: $\mathbb P(A \cap B) = 0$,  i.e. $\mathbb P(A \mid B)= 0$ so knowing one happened means the other did not happen

You asked for a picture.  This might help:
Comment: Is there a reason you wrote "almost" in the second bullet point?  Is that one of those "possible with probability zero" things?  I would think it is by definition impossible (such as the probability of heads and the probability of tails), then why write "almost certainly" rather than "certainly"? I suppose this is the probabilistic interpretation.
Comment: I'm no expert, but even after your last comment I agree with @gerrit: *Heads* and *Tails* are disjoint.  It's possible to get *Not heads and not tails*, but it's impossible to get *heads and tails*. Thus knowing *heads* happened means that *tails* **could not possibly** have happened - no "almost" about it.  I could be wrong on my terminology, but if so please explain patiently as I'm not the only one missing it
Comment: @Barranka I get that, but that doesn't look like what is drawn in the picture on right.  The joint probability of a uniformly drawn random number in [0, 1] being both smaller than 0.4 and larger than 0.6 is not only zero, it is also completely impossible.  Isn't that what the wide band in the right figure illustrates?  Or am I misreading the figure?
Comment: @Barranka I could throw the coin so fast that it escapes earth's gravitational pull.  I would venture P(HEADS)=0.499..., P(TAILS)=0.499..., 00.5, or does it?  When two outcomes are mutually disjoint, does that not mean that it's *by definition* absolutely certain (not almost certain), that they are not both true?
Comment: @Braanka Your coin example is a poor one, as presumably landing on a side has non-zero probability, and if you say that it has zero probability, well, now you're pretty much just begging the question.
Comment: The only explanation for saying that something has p=0 but could happen that I can think of is if one is distinguishing between a set that has measure zero versus a set that is empty. In a normal distribution, the probability of getting a value *exactly* equal the mean is zero (it's a measure zero set), but that's true of every particular value, and obviously when you take a sample, you get *some* value (although, practically, one doesn't measure to infinite accuracy, and given any measured value, the set of actual values that would be measured as having that value has non-zero measure).
Comment: @gerrit because an event *can* occur, even if it's probability is zero. For example, the probability of not winning and not loosing when throwing a fair coin is zero, *but it can land on its side in a crack of the floor*: P{Heads}=P{Tails}=0.5, and P{Not Heads Or Tails}=0, and yet "No Heads or Tails" can happen
Comment: I think the right graph would be much clearer if A label is below the B label. Then it is much more clear that they are disjoint
Comment: is it possible to illustrate this with shapes/drawings? I'd be very grateful if you can point me to any drawings.  Sorry I understand better with drawings/geometry
Comment: @gaston I added a picture
Comment: Maybe it's clearer to say "An event with zero probability *is* posible, although it's occurrence is so unlikely that it's measure of occurrence is zero". In the example of a coin, both results are mutually disjoint (they cannot happen at the same time), but the "Empty" result (no result), although its probability is zero, can happen. The "almost" part in the answer deals with such occurrences, I think.
Comment: @gerrit "[Almost certainly](https://en.wikipedia.org/wiki/Almost_surely)" is a term of art in probability. In certain situations (particularly with infinite outcome sets) you can have P(X)=0 and still see an event. E.g. Pick a random natural number from the full set of natural numbers. What's the probability that it's even? What's the probability that it's prime? What's the probability that it's prime *and* even? -- If you picked an prime number, would you say it's impossible that it's even?
Comment: Another example is flipping a coin an infinite number of times. Any particular sequence of heads and tails has probability zero of happening, and yet *some* sequence will come up (if one could actually flip a coin an infinite number of times, that is).
Answer: What I understood from your question, is that you might have confused independent events with disjoint events.
disjoint events: Two events are called disjoint or mutually exclusive if they cannot both happen. For instance, if we roll a die, the outcomes 1 and 2 are disjoint since they cannot both occur. On the other hand, the outcomes 1 and "rolling an odd number" are not disjoint since both occur if the outcome of the roll is a 1. The intersect of such events is always 0.
independent events: Two events are independent if knowing the outcome of one provides no useful information about the outcome of the other. For instance, when we roll two dice, the outcome of each is an independent event – knowing the outcome of one roll does not help determining the outcome of the other. Let's build on that example:
We roll two dice, a red and a blue. The probability of getting a 1 on the red is given by P(red = 1) = 1/6, and the probability of getting a 1 on the white is given by P(white = 1) = 1/6. It is possible to get their intersect (i.e. both get 1) simply by multiplying them, since they are independent. P(red = 1) x P(white = 1) = 1/6 x 1/6 = 1/36 != 0.
In simple words 1/6 of the time the red die is a 1, and 1/6 of those times the white die is 1. To illustrate:
Answer: The confusion of the OP lies on the notions of disjoint events and independent events.
One simple and intuitive description of independence is:

A and B are independent if knowing that A happened gives you no
  information about whether or not B happened.

Or in other words,

A and B are independent if knowing that A happened does not change the
  probability that B happened.

If A and B are disjoint then knowing that A happened is a game changer! Now you would be certain that B did not happen! And so they are not independent.
The only way independence and "disjointedness" in this example are the same is when B is the empty set (which has probability 0). In this case A happening does not inform anything on B
No pictures but at least some intuition


Wrong Javascript order: Undefined variable: jQuery

Question: Since I moved all scripts form the header to the bottom of the page (in the template file of my theme) I have the problem that some scripts of contributed modules (chosen and device geolocation from smart ip) are added before the JQuery library. Using Jquery this leads to an error:
Undefined variable: jQuery

What can I do to get the right order and what could be the reason of the fact that the order is only wrong when the scripts are added at the bottom of the page and not in the header?
Comment: Realizing after I answered, the first question should be: how did you move all of the scripts? Did you just print `$scripts` in your footer?
Comment: @ChrisRockwell Yes, this is, what I did.
Answer: Modules, via drupal_add_js() can define the 'scope' and 'weight'. The 'scope' defaults to header, so unless you change this via drupal_add_js() in your template.php file, you can't move print $scripts from your header, or you'll have this problem with, likely, most contrib modules.
To change this, you could do something like this in yourtheme_preprocess_page($vars):
$js = drupal_add_js();
$newJs = array()
foreach ($js as $s) {
  $s['scope'] = 'footer';
  $newJs = $s;
}

drupal_add_js($newJs);

Comment: As long as everything is following Drupal best practices for Javascript, I don't recall ever having a problem moving my `` to the end of the body, but before `$page_bottom` in html.tpl.php.


cherrypy/wsgi behind apache ssl

Question: I wrote a cherrypy application and now I need SSL. I am working with apache/wsgi and have a working python file running and returning pages. Now I am trying to get POSTs to work.
This is my working script:
import sys
sys.stdout = sys.stderr

import atexit
import threading
import cherrypy

cherrypy.config.update({'environment': 'embedded'})

if cherrypy.engine.state == 0:
    cherrypy.engine.start(blocking=False)
    atexit.register(cherrypy.engine.stop)

class Root(object):
    def index(self):
        # restrict access by ip address
        clientIP = cherrypy.request.headers["Remote-Addr"]
        if clientIP not in self.allowedIPs:
            return "Access Denied"

        return cherrypy.url()

    index.exposed = True
    allowedIPs = ["127.0.0.1", "192.168.174.1"]

application = cherrypy.Application(Root(), None)

If i make modifications to catch the post:
class Root(object):
    def index(self, files):

I receive the following error:

404 Not Found

Nothing matches the given URI

My apache config,
WSGISocketPrefix run/wsgi
NameVirtualHost *:443
<VirtualHost *:443>
        DocumentRoot /var/www/ssl_html
        ServerName   192.168.174.130:443
        ServerAlias 192.168.174.130

        SSLEngine On
        SSLCertificateFile /etc/ssl/certs/devel.crt
        SSLCertificateKeyFile /etc/ssl/certs/devel.key
        <Location />
                SSLRequireSSL
        </Location>

        WSGIDaemonProcess 192.168.174.130 processes=2 threads=15 display-name=%{GROUP}
        WSGIProcessGroup 192.168.174.130

        WSGIScriptAlias / /var/www/wsgi-scripts/helloWorld.py

</VirtualHost>

any help would be greatly appreciated! :)
Answer: def index(self, files):

should be
def index(self, file):

ughh!! alas, fixed!

cherrypy/wsgi behind apache ssl


E-Mail Hosting for Mid-Size-Business

Question: We are a medium sized company with 5,000 employees. Every day we send and receive about 10,000 e-mails.
We want to set up our own Linux mail server (Postfix+Cyrus). What is recommended hardware for such needs?
Comment: Five thousand employees is hardly small. I have a few less users then that in my Exchange 2007 environment but do 30K messages/day, and we run between 3-4TB of space for mailboxes. We use multiple servers.
Comment: Just an FYI, 5,000 employees is not small on any scale.
Comment: You want 5000 employees using POP3 and IMAP? Have you considered looking at enterprise systems (such as Exchange)?
Comment: Yeah, sounds ignorant to focus on email when both, exchange and lotus do provide a LOT more, including scheduling in an integrated platform.
Answer: I'd recommend looking for VPS's, take a look at Logicworks they have infiniCloud services, with CentOS , Postfix+Cyrus... all you may need for your services.

E-Mail Hosting for Mid-Size-Business


Unstandardize the slope of standardised variables in a linear regression

Question: If I standardize my dependent and independent variable, and run a linear regression between them, the slope estimate which I have will be standardised. The variables were standardised by subtracting the mean and dividing by twice the standard deviation. I know the means and SD of the input variables
Is there any way, I can unstandardise the slope (perhaps the obvious answer is to run the regression between dependent and independent variable without standardizing them, but lets say, for the sake of argument, I cannot unstandardize my dependent and independent variables). What would be the best solution to this.
If the question is not clear, please let me know and I will try my best to restructure the question.
Thanks
Comment: Well the package that I am using in R `arm` does this internally. So it is actually not in my hand.
Comment: To quote Andrew Gelman it's "so that the generic comparison is with inputs equal to the mean ±1 standard deviation. The resulting coefficients are then directly comparable for untransformed binary predictors...  We recommend our rescaling as a default option — an improvement upon the usual approach of including variables in whatever way they are coded in the data file — so that the magnitudes of coefficients can be directly compared as a matter of routine statistical practice" STATISTICS IN MEDICINE Statist. Med. 2008; 27 :2865–2873
Comment: If you want to recover the full regression equation, do you know the means and standard deviations for your variables?
Comment: yes, I know the means and standard deviations of the variable. Also the variables were standardised by subtracting the mean and dividing by 2SD
Comment: Are you sure you wanted to divide by 2 sd? I'm curious what the rationale is to do that rather than by one sd.
Comment: Accessible at: http://www.stat.columbia.edu/~gelman/research/published/standardizing7.pdf
Answer: Assuming you know, or can calculate, the standard deviations of the values of $x$ and $y$, the standardized estimate $b'$ is given by:
$$
b'=b \times \frac{s_x}{s_y}
$$
So to unstandardize:
$$
b=b' \times \frac{s_y}{s_x}
$$
Comment: Hi Sorry I forgot to mention, the variables were standardised by subtracting the mean and dividing by twice the standard deviation
Comment: Also can I use this to calculate the confidence interval of my slope estimate (the actual confidence interval which I have are also standardised)
Comment: First it doesn't matter what multiple of SD you divide by, as long as both $x$ and $y$ are divided by the same sd. For the bottom equation, you're just going to put $\times 0.5$ on the top and bottom, and they will cancel out.
Comment: I think you can. Do you know the sample size? Once you have the slope and the sample size, you can work out the standard error and from that the confidence intervals. (Assuming you only have one predictor. If you have more, you need to know the correlations too.)
Comment: Ya. I know the sample size. I have 33 observations of x and y.


Cant get python to properly work on visual studio code

Question: this is the error I'm getting
Set-Location : A positional parameter cannot be found that accepts argument 's'.
At line:1 char:1
+ cd 'c:\Users\Sean's awesome\Documents\Python Projects\Test Prject'; $ ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (:) [Set-Location], ParameterBindingException
    + FullyQualifiedErrorId : PositionalParameterNotFound,Microsoft.PowerShell.Commands.SetLocationCommand

Comment: The problem should be clearly visible by looking at VS Code's syntax highlighting.
Answer: It looks like the apostrophe in Sean's awesome is causing that error
The apostrophe is ending the location string here 'c:\Users\Sean'
Remove this from your user and try again.


SSH to EC2 Instances suddenly slow

Question: Suddenly, for the last few days, it takes a very long time (up to 30 seconds) to establish a SSH connection to most, but not all, of my Amazon EC2 instances. The issue has been raised with Amazon to see if it's environmental with them, but I wonder if there's anything I can check on the instances themselves.
Most of that time is spent at the step:
Authenticating with public key "imported-openssh-key"
Once on the instance, changing users via
su - newusername
hangs indefinitely.
Other commands (ps, top, find) run fast as ever.
My application running on the instances (a web service) is very responsive. CPU, IO and disk load on the instances is not very high.
EDIT:
Last few lines of output from strace su - myusername as suggested by Dave:
connect(4, {sa_family=AF_INET, sin_port=htons(389), sin_addr=inet_addr("W.X.Y.Z")}, 16) = -1 EINPROGRESS (Operation now in progress)
poll(

This line keeps recurring in 10 second intervals... it hangs at poll( for 10 seconds then repeats the same output.
The IP address referenced is the public IP address of our LDAP server.
The problem was that the instances were attempting to resolve the LDAP server via the public IP address rather than the private one.  Opening the public IP to the other instances resolved the issue.
Answer: Usually when weird hangs happen, I can trace it back to RDNS - either the host you are connecting from doesn't have it set up or there is an issue with the server resolving RDNS.
However su shouldn't be doing anything with RDNS as far as I know.
What happens you trace the process with 'strace'?
EDIT:
So it seems like the connection to the LDAP server is timing out. Have you confirmed the LDAP server is working with other systems? Maybe you can trace the traffic on the LDAP server with:
tcpdump -v tcp port ldap

To get an idea of what happens when it is connecting.
Comment: @Dave: strace gave a nice clue.  strace su - myusername provides output indicating it's doing something with our LDAP instance, though I'm not exactly how to interpret it.  Posted the output as an edit to my question for formatting.
Comment: Can you connect to LDAP from the problematic hosts? Can your LDAP server reverse lookup the problematic hosts?
Comment: Solved, see edited post.
Comment: Good to hear! :)

SSH to EC2 Instances suddenly slow


Tenho um BD local em c# sql. Quando fui instalar no cliente deu o seguinte erro

Question: Alguem poderia me ajudar?
Instalei meu projeto no cliente com banco de dados local, levei o bd junto e tal , mas deu esse erro:
Comment: [Edite](http://pt.stackoverflow.com/posts/64574/edit) a pergunta, poste o código e informe a linha onde ocorre o erro.
Answer: Isto está acontecendo porque a consulta não está encontrando a tabela vendedor no Banco de Dados.
Comment: Adicionalmente, ele deve verificar se executou os scripts no banco de dados para criação dos objetos (tabelas, views, stored procedures).


Dependency Injection and the Url object

Question: I'm making the leap from Drupal 7 to Drupal 8 and am still fuzzy on some core concepts, particularly relating to dependency injection (DI). I've read plenty of tutorials and have a grasp on DI with services, but am unclear on using DI with something like \Drupal::Url().
For instance, I am using the following code in a class:

use Drupal\Core\Url;
...
$url = Url::fromRoute('<current>', [], ['absolute' => 'true'])->toString();

From what I can tell, there is no Url service. So, what I am wondering is: Is it possible to access Url via DI? And if so, how?
Bonus points: I intend for the class I am writing to be used as a service. Does this have any impact on the answer?
Edit: Fixed erroneous code.
Edit 2: I should also point out that the purpose of this call is to retrieve the current absolute URL so that I can determine if it is HTTPS.
Answer: You can inject the UrlGenerator as service and use UrlGenerator::generateFromRoute, but in the linked doc you are discouraged to do so:

@internal Should not be used in user code. Use \Drupal\Core\Url
  instead.

So the code in your question to create an url object is OK for a service, but not the toString() method. Render the url only with this method when the page rendering process has started, for example in a theme.
In a service you usually don't know this and you should return an unrendered url object, so that it can be rendered later in the rendering pipeline.
$url = Url::fromRoute('<current>', [], ['absolute' => 'true']);

toString() in the wrong place generates ugly error messages about early rendering which are hard to debug when this happens somewhere in a service.
Comment: Awesome! This is great information, especially about avoiding toString(). So, am I to understand correctly that it is acceptable to use the static Url::fromRoute procedurally and without DI?
Comment: "The Url class is basically a value object, so in that sense doesn't need to be injected. It's only when you render it to a string that the UrlGenerator dependency is pulled in. But you basically shouldn't be rendering it yourself - use a link render array."  https://www.drupal.org/node/2491981#comment-11556477 - #36
Comment: I really appreciate this information. I've edited my question to add clarification. Does the 2nd edit impact this answer?
Comment: No, because this would be about how to inject the request stack, get the current request and then `$request->getScheme()`
Comment: Do you think that is a good idea to inject the renderer service and then render the link render array inside the service?
Comment: @AdrianCidAlmaguer, this depends on where the service is used. Like the render service itself  other services which depend on it should not run too early.
Comment: Hm... I'm a bit in the dark about this question and it's answer, I know there exists a `url_generator` service. Shouldn't we be using that? And also, "... is basically a value object, ...", it seems to me that you might want to mock routes for `url_generator` to be able to write decent unit tests, right? Or am I completely wrong here?

Dependency Injection and the Url object


GIF as css background image

Question: I have something like this in css
.ajax-loader {
    background-image: url(../images/icon/loader.gif);
}

and used on HTML elements like this

my question is:
If the ajax-loader style is used multiple times in a single page, will the image loader.gif be loaded multiple times or only once?
Comment: What is the question exactly?
Comment: sorry, forgot the question mark. updated XD
Comment: Sorry, I still don't get what you're asking..
Comment: @Patrick, updated again, hope my explanation is understandable this time XD...........
update to your comment: welcome
Answer: This image will be loaded once.This is very simple like object-oriented programing.Define a class once and use everywhere.The same thing is followed in css.
Once the image is loaded you can use it everywhere and also this is a good programming approach

GIF as css background image


Can't access params in ActivatedRoute

Question: I have a sub routing-module with a route like this:
path: 'zoneslist/desktop/:zoneid

I'm trying to access to zoneid from a service. I'm getting always null or undefined.
Given this constructor:
constructor(private route: ActivatedRoute){}

I tried the following solutions:
this.route.snapshot.paramMap.get('zoneid')
this.route.snapshot.params['zoneid']
this.route.snapshot.params.zoneid
this.route.params.subscribe(
    (params: Params) => {
         console.log(params.get('id))
    }
);

Why am I getting null or undefined?
Answer: I think you cant get route params in your service.
You should subscribe to them in ngOnInit in your Component and then pass them as a parameter in a function from your service
Comment: So there isn't a "cleaner way" to do that?
Comment: this is as clean as it gets

Can't access params in ActivatedRoute


How to use rowspan in react-table

Question: Any idea how to use rowSpan in react table. for below example usernameFormatter there are multiple values.
Note: attach picture is copied but that's looks like what I wanted
const columns = [
     {
         accessor: "id",
         show: false,
     },
     {
        Header: "Username",
        id: "username",
        accessor: "username",
        Cell: this.usernameFormatter,
        enableRowSpan: true
     }
]

<ReactTable
    data={this.state.users}
    filterable={true}
    columns={columns}
    defaultPageSize={18}
    getTdProps={(state, rowInfo, column) => {
        const index = rowInfo ? rowInfo.index : -1;
        return {
            onClick: (e, handleOriginal) => {
                if (column.Header === undefined) {
                    handleOriginal();
                } else {
                    if (rowInfo !== undefined) {
                        this.selectedRow = index;
                        this.onRowSelect(rowInfo.original.id);
                    }
                }
            },
        };
    }}
/>

Comment: Does this answer your question? [HTML table rowspan in reactjs](https://stackoverflow.com/questions/70872995/html-table-rowspan-in-reactjs)
Answer: Please check this demo
Idea is array of data set, We are going to render inside table should know, what are the cells going to use row span.
This case, individual person has category and that and "categorySpan"
const defaultData: Person[] = [
  {
    category:"category one",
    categorySpan:4,
    firstName: 'tanner',
    lastName: 'linsley',
    age: 24,
    visits: 100,
    status: 'In Relationship',
    progress: 50,
  },
  {
    category:"category one",
    firstName: 'tandy',
    lastName: 'miller',
    age: 40,
    visits: 40,
    status: 'Single',
    progress: 80,
  },
  {
    category:"category one",
    firstName: 'joe',
    lastName: 'dirte',
    age: 45,
    visits: 20,
    status: 'Complicated',
    progress: 10,
  }
  ,
  {
    category:"category one",
    firstName: 'joe',
    lastName: 'dirte',
    age: 45,
    visits: 20,
    status: 'Complicated',
    progress: 10,
  }
  ,
  {
    category:"category two",
    categorySpan:2,
    firstName: 'joe',
    lastName: 'dirte',
    age: 45,
    visits: 20,
    status: 'Complicated',
    progress: 10,
  }
  ,
  {
    category:"category two",
    firstName: 'joe',
    lastName: 'dirte',
    age: 45,
    visits: 20,
    status: 'Complicated',
    progress: 10,
  }
  ,
  {
    category:"category three",
    categorySpan:1,
    firstName: 'joe',
    lastName: 'dirte',
    age: 45,
    visits: 20,
    status: 'Complicated',
    progress: 10,
  }
  ,
  {
    category:"category four",
    categorySpan:1,
    firstName: 'joe',
    lastName: 'dirte',
    age: 45,
    visits: 20,
    status: 'Complicated',
    progress: 10,
  }

]

So you can add row span property to td base on category and category span

      {row.getVisibleCells().map(cell => {
        if(cell.column.id=="category"){
          if(row.original.categorySpan){
          return (
            {flexRender(cell.column.columnDef.cell, cell.getContext())}
          	);
          }
        }
        else{
          return (

              {flexRender(cell.column.columnDef.cell, cell.getContext())}

          )
        }
      })}

Final code will be like this

import * as React from 'react'
import ReactDOM from 'react-dom/client'

import './index.css'

import {
  createColumnHelper,
  flexRender,
  getCoreRowModel,
  useReactTable,
} from '@tanstack/react-table'

type Person = {
  category:string
  categorySpan?:number
  firstName: string
  lastName: string
  age: number
  visits: number
  status: string
  progress: number
}

const defaultData: Person[] = [
  {
    category:"category one",
    categorySpan:4,
    firstName: 'tanner',
    lastName: 'linsley',
    age: 24,
    visits: 100,
    status: 'In Relationship',
    progress: 50,
  },
  {
    category:"category one",
    firstName: 'tandy',
    lastName: 'miller',
    age: 40,
    visits: 40,
    status: 'Single',
    progress: 80,
  },
  {
    category:"category one",
    firstName: 'joe',
    lastName: 'dirte',
    age: 45,
    visits: 20,
    status: 'Complicated',
    progress: 10,
  }
  ,
  {
    category:"category one",
    firstName: 'joe',
    lastName: 'dirte',
    age: 45,
    visits: 20,
    status: 'Complicated',
    progress: 10,
  }
  ,
  {
    category:"category two",
    categorySpan:2,
    firstName: 'joe',
    lastName: 'dirte',
    age: 45,
    visits: 20,
    status: 'Complicated',
    progress: 10,
  }
  ,
  {
    category:"category two",
    firstName: 'joe',
    lastName: 'dirte',
    age: 45,
    visits: 20,
    status: 'Complicated',
    progress: 10,
  }
  ,
  {
    category:"category three",
    categorySpan:1,
    firstName: 'joe',
    lastName: 'dirte',
    age: 45,
    visits: 20,
    status: 'Complicated',
    progress: 10,
  }
  ,
  {
    category:"category four",
    categorySpan:1,
    firstName: 'joe',
    lastName: 'dirte',
    age: 45,
    visits: 20,
    status: 'Complicated',
    progress: 10,
  }

]

const columnHelper = createColumnHelper<Person>()

const columns = [
  columnHelper.accessor('category', {
    cell: info => info.getValue(),
    footer: info => info.column.id,
  }),
  columnHelper.accessor('firstName', {
    cell: info => info.getValue(),
    footer: info => info.column.id,
  }),
  columnHelper.accessor(row => row.lastName, {
    id: 'lastName',
    cell: info => {info.getValue()},
    header: () => Last Name,
    footer: info => info.column.id,
  }),
  columnHelper.accessor('age', {
    header: () => 'Age',
    cell: info => info.renderValue(),
    footer: info => info.column.id,
  }),
  columnHelper.accessor('visits', {
    header: () => Visits,
    footer: info => info.column.id,
  }),
  columnHelper.accessor('status', {
    header: 'Status',
    footer: info => info.column.id,
  }),
  columnHelper.accessor('progress', {
    header: 'Profile Progress',
    footer: info => info.column.id,
  }),
]

function App() {
  const [data, setData] = React.useState(() => [...defaultData])
  const rerender = React.useReducer(() => ({}), {})[1]

  const table = useReactTable({
    data,
    columns,
    getCoreRowModel: getCoreRowModel(),
  })

  return (

          {table.getHeaderGroups().map(headerGroup => (

              {headerGroup.headers.map(header => (

                  {header.isPlaceholder
                    ? null
                    : flexRender(
                        header.column.columnDef.header,
                        header.getContext()
                      )}

              ))}

          ))}

          {table.getRowModel().rows.map(row => (

              {row.getVisibleCells().map(cell => {
                if(cell.column.id=="category"){
                  if(row.original.categorySpan){
                  return (
                    {flexRender(cell.column.columnDef.cell, cell.getContext())}
                  	);
                  }
                }
                else{
                  return (

                      {flexRender(cell.column.columnDef.cell, cell.getContext())}

                  )
                }
              })}

          ))}

      <button onClick={() => rerender()} className="border p-2">
        Rerender
      </button>

  )
}

const rootElement = document.getElementById('root')
if (!rootElement) throw new Error('Failed to find the root element')

ReactDOM.createRoot(rootElement).render(
  <React.StrictMode>
    <App />
  </React.StrictMode>
)

How to use rowspan in react-table


After module installation, updating translations still at 0%

Question: I installed Drupal 8.5.6, in French. It seemed to work well, apart from no CSS working, but I corrected by unchecking CSS aggregation in the performance settings.
I decided to install some modules, one by one. Every time I install a module, after I click on the Install button, I get this.

After waiting more than 2 minutes (which is abnormal to me), I click on the Extend link to see if the module I am trying to install has been installed. The module seems to be installed, since it is checked, but it is not working.
So I uninstall that module, and try installing another module, but I get the same result.
Notice that the first modules I tried installing were the Coffee and Admin Toolbar modules. They get installed properly offline.
I have also noticed that using Drupal 8.2.5 everything is fine.
Is there any version problem, or am I doing something wrong?
I changed the PHP version using on the host to PHP 7.1.
Answer: After a few time doing something else, I went back to the issue. In fact a lot of JavaScript was not working correctly.
Then I decided to go back to Configuration -> Performance and unchecked JavaScript aggregator and saved that configuration. See the image below :

Now everything works fine!!!
Now I can use admin toolbar's dropdown menu as well as coffee and all the other stuff is working well.
And thank you for reading me.
Comment: This is only a work around. The cause of the error is that your Drupal installation is not able to create the aggregated CSS/JS files in sites/default/files/. The same happens for the translation files or the image styles in your follow up question https://drupal.stackexchange.com/questions/267401/no-image-is-showing


How to retrieve the IP Address Assigned to the machine by ISP

Question: my question is regarding the IP Address assigned to a machine, whenever we go to any site which reveal our IP Address, they display a number which is assigned to the machine through which we are accessing the Internet, i want to know how is it possible to retrieve the IP Address from the local machine using which commands, if any, rather than going to the sites to know the IP Address assigned to the machine by ISP.
the local machine is having an IP Address of 192.168.1.2 and is having a DSL router provided by the ISP which will connect to the Internet with Win XP SP2
Looking for favorable replies.
Thanks
Answer: Login to your DSL router and look at the status page.  If you want to log this information build a script that simply downloads the status page.
You could also write a script with curl/wget that simply gets the page (http://checkip.dyndns.com/) on occasion.
Comment: This obviously depends on the DSL router being configured to allow customers to gain access to them. Many consumer grade ISPs tend to ship routers with no user-visible interface.
Answer: You're just looking for the IP assigned to your NIC? Execute:
IPCONFIG

from a command prompt.
Edit: Re-re-re-reading your question again, it looks like you're asking for the IP address assigned to the Internet-side interface of your router. That's not stored on the local machine. You're going to have to write something to get it from the router itself, or rely on a third-party site to get you that.
Comment: You're downvoting me for at least trying to parse his poor English?  Yeesh! I guess I should ask for clarification in a comment rather than in an answer first, eh?

Yeah-- if he has multiple NICs he's going to see multiple IP addresses. Far be it from me to assume that the poster has some degree of competence and would be able to figure that out.
Comment: This will only work if they are directly connected to the internet, I believe he is looking to get the external IP when connected to a NAT router
Comment: Also, what if user has many NICs? What if a NIC has more than one IP?
Comment: I agree that the question is poorly phrased, and your answer could be a valid response, I don't feel it deserves a downvote.
Answer: If your DSL router has SNMP, you can usually fetch via snmpget the IP address that is assigned to the external interface of your DSL router. Most, (but not all) DSL router manufacturers have SNMP, and they support MIB-II.
example:
[root@myhost ~]# snmpwalk -v1 -c ***** 10.1.10.1 ipAdEntAddr
IP-MIB::ipAdEntAddr.184.108.40.206 = IpAddress: 18.104.22.168

In this case, I am querying the "inside" IP with SNMP and I get the following IP as my external IP. where ****** is my SNMP community string or password. ipAdEntAddr is the SNMP OID string i queried to get the answer.
Answer: You could download WGet then run the following command:
wget -q -O - http://automation.whatismyip.com/n09230945.asp

You will need to run this either from the WGet directory, or add it to your systems Path.
Comment: I think this is the best option to do it programatically.  It is unclear from the question whether the information can come from an online source or must only come from information stored on the local machine.
Comment: The local machine has no idea, and in fact there's no guarantee that there is one and only one such IP address.
Answer: I like http://www.ipchicken.com/ .
It doesn't provide the additional information that Kevin's site does, but I find the name sticks in people's heads and is easy to remember.
Answer: You can download this:  http://curl.haxx.se/latest.cgi?curl=win32-nossl
Extract it, then go to a command prompt and type "curl http://whatismyip.com/automation/n09230945.asp"
(same idea roughly as above by Sam)


Variance of the inner product of independent vectors

Question:
Note: I am not asking about the interpretation of covariance as an inner product on the space of random variables.

I have two $n$-dimensional random variables $\vec X, \vec Y\in\mathbb{R}^n$.
Each has some covariance matrix $\Sigma_{\vec X}, \Sigma_{\vec Y}$.
Each is also "nice", namely

$\vec X\sim\mathcal{N}(0, \sigma^2I_n)$, and
$\vec Y\gets \mathsf{Unif}(K)$ for a compact convex set $K$.

I want to compute $\mathsf{Var}(\langle \vec X, \vec Y\rangle)$.
Is there a simpler way to do this than via explicitly working with the density of $\langle \vec X, \vec Y\rangle$?
For example, is it some function of $\Sigma_{\vec X}, \Sigma_{\vec Y}$?
Comment: Did you mean $X\sim\mathcal{N}(0,\,\Sigma_X)$? Also, can we assume $X$ is independent of $Y$?
Answer: I assume $X$ and $Y$ are independent (your post doesn't say so, but your title does). Then taking $X\sim N(0,\Sigma_X),Y\sim \text{Unif}(K)$ as column random vectors, we have
$$\begin{align}V(X'Y)&=E[X'YY'X]-(E[X'Y])^2\\
&=E[X'YY'X]-(\underbrace{E[X']}_{=0}E[Y])^2 \quad (\text{by indep})\\
&=\text{tr}E[X'YY'X]\\
&=E[\text{tr}(X'YY'X)]\\
&=E[\text{tr}(XX'YY')]\\
&=\text{tr}E[XX'YY']\\
&=\text{tr}E[XX']E[YY'] \quad (\text{by indep})\\
&=\text{tr}(\Sigma_X(\Sigma_Y+\mu_Y\mu_Y'))\\
\end{align}$$
where $\mu_Y,\Sigma_Y$ are respectively mean and covariance matrix of $Y$.
If $\Sigma_X=\sigma^2I_n,$ then this simplifies to
$$\sigma^2\text{tr}(\Sigma_Y+\mu_Y\mu_Y').$$