Bayesian Learning: Finding the variance of noise

Question: Suppose $x_i \sim N(10,4)$ - ie, the distribution is known.
There is a noisy signal $s_i \sim N(x_i, \sigma_e^2)$ and I want to estimate $\sigma_e$.
I see some pairs ($s_i, x_i$) but they are not 'random' (there is some selection on $s_i$ - say, I see these values mostly for large values of $s_i$).
What approach would you recommend for estimating $\sigma_e$?
My first thought was to simply take $\hat\sigma_e = \sqrt{\sum(s_i-x_i)^2/N}$ but I think this would be wrong since my data contains mostly larger $s_i$?
I next tried to model the distribution of $x_i | s_i$. The Bayesian posterior distribution is
$N((\frac{\sigma_{e}^2}{\sigma_{e}^2+4})10 + (\frac{4}{\sigma_{e}^2+4})s_{i},  \frac{\sigma_{e}^2*4}{\sigma_{e}^2+4})$
where the $4$ and $10$ come from the distribution of $x_i$ above.
But I do not know how to proceed. I believe that the difference between the expected value of $x_i|s_i$ and the actual observed value should be informative of $\sigma_e$ somehow, but I am confused since $\sigma_e$ enters both the expected value term and the variance term above. Trying to find the MLE by differentiating the log likelihood above becomes very complicated. Is there something obvious that I am missing? Or any approach that you would recommend to estimate $\sigma_e$?
Edit: To be clear, I know that there is selection on $s_i$, but I am sure that there is no selection of $x_i$ or $x_i|s_i$.
So I feel $x_i|s_i$ should be informative about $\sigma_e$ (For instance, if I only see $x_i$ very close to $s_i$ for each $s_i$, I know that $\sigma_e$ is small; but if I see $x_i|s_i \sim N(10,4)$ (ie, close to the original distribution) I would believe that $\sigma_e$ is large. How can I formalize this intuition to form an estimator?
Edit2: Consider $a_i = \frac{x_i-s_i}{10-s_i}$
$E[a_i|s_i] = E[\frac{x_i-s_i}{10-s_i} | s_i]$
$= \frac{E[x_i | s_i] - s_i}{10-s_i}$
From the equation above, I know that $E[x_i | s_i] = (\frac{\sigma_{e}^2}{\sigma_{e}^2+4})10 + (\frac{4}{\sigma_{e}^2+4})s_{i}$
Therefore, $E[a_i|s_i] = \frac{\sigma_{e}^2}{\sigma_{e}^2+4}$.

Is this correct? If yes, can I take the mean of $[a_i|s_i]$ as an estimator of $\frac{\sigma_{e}^2}{\sigma_{e}^2+4}$? This should overcome the selection problem, since there is no selection conditional on an $s_i$.
Can I get an estimate of $\sigma_{e}^2$ by making it the subject of the formula in 1. above?

Any help, even directional, would be really appreciated.
Thanks!
Comment: I don't think you have much hope unless you have some method for characterizing the missingness distribution as a function of $s_i$?
Comment: If $\frac1N\sum(s_i-x_i)$ is close to $0$ and roughly half the $s_i < x_i$ then you may not have a problem.  Alternatively, if you think you are only seeing $s_i$ above some threshold and are seeing all such $s_i$ then there may be a possibility.  But otherwise I so not see a satisfactory approach
Comment: @Henry Thanks! That expression is significantly greater than 0, and there is no clean threshold for $s_i$ (just that, *mostly*, larger values are selected).
The conditional distribution of $x_i|s_i$ should overcome the selection issue, right? Is there any convenient way of estimating $\sigma_e$ from that?
Comment: @CliffAB You are possibly right, but I am not able to see exactly why. For instance, even if I have 1 $s_i$ but multiple corresponding $x_i$, shouldn't the conditional distribution of $x_i | s_i$ be informative about $\sigma_e$? Or am I just being dense?
Thanks!
Comment: Not if some other $x_i$ corresponding to that $s_i$ are missing but not independently of $x_i$, if you do not know which ones or why
Comment: @Henry That I am sure of - there is no selection on $x_i$ or $x_i|s_i$. Only on $s_i$. Is there any convenient estimator in this case?
Comment: When the sample is random, subtracting $4^2$ from the mean squared residual (the average of the $(x_i-10)^2$) estimates $\sigma_e^2.$  But when there's only some vague "selection" method you can't conclude *anything.*
Comment: @whuber: Thanks. But as I wrote in the earlier comment, I am not able to see exactly why. For instance, even if I have 1 $s_i$  but multiple corresponding $x_i$, shouldn't the conditional distribution of $x_i | s_i$ be informative about $\sigma_e$? I know that, while there is a selection on $s_i$, there is no selection on $x_i$ or $x_i | s_i$
Comment: The problem is that the results are likely to be biased and it's difficult to see how the bias could be corrected without knowing more about the selection process.  You could make progress with very little information.  For instance, if only values of $s_i$ exceeding some threshold are taken and you know what fraction have been dropped, a decent estimate looks possible.  *Some* estimate is possible even without knowing that fraction: the threshold becomes a "nuisance parameter" in the model.
Comment: @whuber Thank you so much for your patience so far! But I am still not sure why taking the conditional distribution of $x_i | s_i$ does not overcome the selection problem. It is very likely I am just being stupid, but could I request you to look at edit2 above and let me know where that logic breaks down? Thanks again!
Comment: BTW, the title is confusing: you are asking about estimating the variance of the *noise,* not the signal.
Comment: Thanks! Updated the title.
Answer: We need to address two questions: (1) demonstrating that how the data are selected does matter for estimation and (2) suggesting how to estimate the parameter.
Everything flows from the joint distribution.  So--choosing a unit of measurement in which the Normally distributed variable $X$ (representing the $x_i$) has zero mean and unit variance, the joint distribution of $(S,X)$ (where $S$ represents $s_i$) is Bivariate Normal with density given by the product of the density of $X$ and the conditional density of $S,$
$$f_{S,X}(s,x;\sigma) = f_{S\mid X}(s;\sigma)f_X(x) =  \frac{1}{\sqrt{2\pi \sigma^2}} \exp\left(-\frac{(s-x)^2}{2\sigma^2}\right)\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{x^2}{2}\right)\,\mathrm ds\mathrm dx.$$
The only parameter, to be estimated, is $\sigma.$  (It is the ratio of the error standard deviation to the standard deviation of $X.$)
The twist is that we don't have a random sample from this distribution.  Instead, some unspecified "selection" was used to choose observations from this distribution with probabilities determined by $S.$  This can be stated mathematically in terms of those selection probabilities: namely, let the chance of keeping an observation $(s_i,x_i)$ be $p = p(s_i),$ a function depending on $s_i.$  This makes the probability density equal to
$$f_{S,X}(s,x;\sigma,p) = C(\sigma,p)\,p(s)f_{S,X}(s,x;\sigma)$$
where $C(\sigma,p)$ is the normalizing constant.
Consider the log likelihood of such a sample $((s_i,x_i),i=1,\ldots, n)$ of $n$ selected observations.  It is the sum
$$\begin{aligned}
&\sum_{i=1}^n \log f_{S,X}(s,x;\sigma,p) \\
&=n\log C(\sigma,p) - \frac{n}{2}\left(\log(2\pi) + 2 \log \sigma\right) - \frac{1}{2}\sum_{i=1}^n \left(\frac{(s_i-x_i)^2}{\sigma^2}+x_i^2\right) + \sum_{i=1}^n \log p(s_i)  .
\end{aligned}$$
Because we haven't specified $p,$ we don't know $C(\sigma,p),$ so there's nothing we can do to analyze the log likelihood.  We cannot proceed without information about $p.$
Consider the case where $p$ is constant.  That implies the normalization constant does not depend on $\sigma,$ giving the usual maximum likelihood estimate
$$\hat\sigma^2 = \frac{1}{n} \sum_{i=1}^n (s_i-x_i)^2.$$
Now suppose $p(s)=1$ when $s\ge T$ and $p(s)=0$ otherwise.  That is, you select only values of $s_i$ exceeding some specified threshold $T.$  The normalization constant is the reciprocal of the chance that $S\ge T,$ given by $\Phi(-T/\sqrt{1+\sigma^2})$ where $\Phi$ is the standard Normal CDF.  Differentiating the log likelihood w.r.t. $\sigma$ and equating it with zero to find critical points now gives
$$0 = n \frac{\sigma T}{(1+\sigma^2)^{3/2}} \frac{\phi(-T/\sqrt{1+\sigma^2})}{\Phi(-T/\sqrt{1+\sigma^2})} - \frac{n}{\sigma} + \sum_{i=1}^n \frac{(s_i-x_i)^2}{\sigma^3}\tag{*}$$
(This requires numerical solution.  I found it works well to multiply the r.h.s. by $\sigma^3/n,$ thereby avoiding some of the divisions that can cause problems near $\sigma=0.$)
As $T\to-\infty,$ the first term in $(*)$ makes a vanishingly small contribution and the (unique) solution closely approximates $\hat \sigma.$  But for larger (especially positive) values of $T,$ this solution is nearly unbiased and far superior to $\hat\sigma.$  Here, for example, is a simulation of $200$ samples of $(S,X),$ each with approximately $n=100$ selected values, where $\sigma=1$ and $T=3.$  In each sample the solution to $(*)$ (the MLE) is plotted horizontally and the root mean square difference $\hat\sigma,$ plotted vertically, is consistently biased about 25% low:

This starkly shows the effect on the native estimator $\hat \sigma$ of selecting the observations based on the values of $s_i.$  Evidently, a more complex procedure described by a general function $p$ will likely exhibit a comparable bias.
Generally, this bias will increase with both $T/\sigma$ and $\sigma.$  For instance, for $\sigma \approx 1/4$ and $T=4\sigma=1,$ the estimator $\hat\sigma$ is biased only a few percent low.  Thus, when the noise is not too great compared to the variation in the signal, you likely have little to worry about and could use $\hat\sigma$ to estimate its standard deviation.
This little example also shows that maximum likelihood estimation is likely to work: it certainly does the job here with relatively small samples and a stringent cutoff threshold (only one in every $59$ random observations will pass this threshold).
We can go further and be more general even if we make broader assumptions about $p.$  For instance, we might suppose there is an unknown selection threshold $T.$  Now this must be estimated along with $\sigma,$ giving a two-parameter problem where $0\lt \sigma$ and $-\infty \lt T \le \min(s_i).$  That latter condition indicates the solution can sometimes be $\hat T = \min(s_i),$ a value on the boundary of the parameter space, so take some care in seeking solutions and applying the standard theorems of maximum likelihood (many of which fail to hold for boundary solutions).
Comment: Thank you! This answer is very helpful, and incredibly detailed.
You say that I need to know the joint distribution to be able to estimate $\sigma$, and conclude that 'We cannot proceed without information about p'. I still do not understand why this is the case - but thanks to your approach, I have been able to proceed with the MLE estimator based on the distribution of $x_i|s_i$. (I have added it as a separate answer). I am able to recover $\sigma$ even without knowing p, which contradicts your conclusion. Could you please check my answer and see if you agree, or point out the error in it?
Comment: I offered two examples with differing $p,$ showing how $\hat\sigma$ thereby changes.  That demonstrates you *cannot* estimate $\sigma$ accurately without some information about $p,$ unless by a matter of luck your estimator happens to be the one appropriate for your $p.$
Comment: But I am able to recover the true $\sigma$ fairly accurately, even for a wide range of $p$ (threshold $T$ varying from 2 to 10) in my answer below - so it should not be the case that it was just luck. It also intuitively makes sense because the conditional distribution of $x_i|s_i$ (which I use for my estimator) is independent of $p$. If you still disagree, can you let me know where exactly the logic in my answer breaks down? Thanks!
Comment: You have explored only a very tiny subset of the parameter space, in a region where the selection doesn't matter.  I discussed that in my answer.  The problems occur when $\sigma$ is comparable to or larger than the variance of $X$ and $T$ is several times greater than $\sqrt{\operatorname{Var}(X)+\sigma^2}.$
Comment: In my answer, variance of $X$ is 1, and $\sigma$ ranges from 3 to 15 - so it satisfies the conditions pointed out in your answer where the problems should occur. Also, for $\sigma = 3$, $T$ ranges upto 10, which is ~3x times that expression.
Comment: Well, you are obtaining results starkly different from those I did, which is why I am suspicious.  Your estimator appears to be exactly what I have called $\hat\sigma.$
Comment: No - my estimator is very different from your $\hat{\sigma}$ - I have used the conditional distribution of $x_i|s_i$ - so $\sigma$ also enters the expectation term (not just the variance term), and the LL has no easy analytic expression.
Answer: I followed the approach outlined in whuber's answer (Thanks!). Considering the case where $x_i \sim N(0,1)$ (equivalent to the original problem, but with simpler algebra), I know that the conditional distribution of $x_i | s_i \sim N(\frac{s_{i}}{\sigma_{e}^2+1},  \frac{\sigma_{e}^2}{\sigma_{e}^2+1})$.
Crucially, this distribution is conditional on $s_i$ and hence does not suffer from the selection problem.
I can write this conditional density using the normal formula, which gives the log likelihood:
$LL = \sum[-0.5log(\frac{\sigma^2}{\sigma^2+1}) -\frac{(x\sigma^2+x-s)^2}{2\sigma^2(\sigma^2+1)}] $
I maximize this log likelihood using numeric methods to get $\sigma$.
I simulated this using different values for $\sigma$ and different filtering thresholds (ie, keeping only those $s_i$ which exceed a threshold $T$) and am able to recover the original $\sigma$ quite well.
$\sigma$" />
So I can recover the true parameter value, even without knowledge of the selection criteria.


When do I apply the distributive property?

Question: I'm a bit lost here....
Equation 1: $(5p − 6) + (1 − p)$.
Shouldn't I apply distributive property here? By distributing the '$+$' sign into $(1 - p)$ to give $(1 + p)$?
If that is the case, then the new formula reworded is:
$$(5p - 6) + 1 + p = 6p - 5,$$
right?
But the book has a different answer and it is, $4p - 5$ instead.... deductively examining where I went wrong, it seems the '$+$' sign isn't distributed and thus the $p$ in $(1 - p)$ didn't change into a positive
If the book has the right answer, then this procs my title question, when do we use distributive property?
Consider the following equation:
$$−10 − 4(n − 5),$$
the $-4$ is distributed into $n$ and $-5$.... If I'm seeing how the formula is worded, whats the difference between this and the case above? Don't they both prompt distributive property cycle? The above case just has an invisible $+1$ right?
I got all the wrong answers in my math test on this part lol but i'm determined to know why.
Comment: Why do you think that $(1-p)$ will become $(1+p)$ ? $(5p − 6) + (1 − p)= 5p-6+1-p$.
Comment: i thought the + has an invisible 1 that it can distribute... is this not true? but then again whenever a positive times a negative it'll still be a negative anyways, so p won't turn positive i believe, just figured this out now lol
Comment: $1(1 - p) = 1 \cdot 1 + 1 \cdot (-p) = 1 - p$.
Answer: If you multiply +1 by -p, you will get -p not +p


Rewriting triple integrals

Question: I'm having trouble rewriting a triple integral.
The question is rewrite the following integral in five different ways:
$\int_0^1\int_y^1\int_0^z f(x,y,z) dx dz dy$
I am having trouble with visually seeing the projections onto the different planes because x runs from 0 to z, but z runs from y to 1, so it's hard for me to picture what the bounded regions look like.
Answer: the region of integration is simply $$
\{(x,y,z) : 0\le x\le z, 0\le y\le  z, 0\le z\le 1
\}
$$
Now you can rewrite the integrals in the following ways:
$$
\int_{x}\int_y\int_z\\
\int_{x}\int_z\int_y\\
\int_{y}\int_x\int_z\\
\int_{y}\int_z\int_x\\
\int_{z}\int_y\int_x\\
\int_{z}\int_x\int_y\\
$$
the step is everytime the same:
for the most outer integral, take the smallest possible lower bound and the greatest possible upper bound. Then for the next integral, take the smallest possible lower bound given the value of the outer variable and go on.
For example:
the integral given is $$
\int_{y=0}^1\int_{z=y}^1\int_{x=0}^z\\
$$
Now if you want to change this to
$$\int_{x}\int_y\int_z
$$then $x$ varies  between $0$ and $\max z = 1$. So your first integral is
$$
\int_{x=0}^1
$$
Then you go on until
$$
\int_{x=0}^1 \int_{y=0}^1 \int_{z=\max(x,y)}^1
$$
Comment: Do you mean z = min(x,y)?
I still don't get how you find the max and min given the outer variables
Comment: the max is just the maximum possible value: here, as in the first integral the max for $x$ is $z$ whose upper bound is 1, when $x$ becomes the outer variable its upper bound becomes 1.
Comment: So if I'm following you correctly, the answer in the order dzdydx should be

$\int_{x=0}^1\int_{y=0}^1\int_{z=0}^1$ ??
Comment: no. because for the inner integrals, the bounds depends on the values of the outer variables.
Comment: see my message: "Then for the next integral, take the smallest possible lower bound given the value of the outer variable"

Rewriting triple integrals


What would the Architecture of Uber, Lyft be like?

Question: Please fill me up with the possible architecture of Uber, Lyft like applications and their servers. What kind of technologies they use? The protocols for mobile-server communication? (esp tracking the user/car location and updating them live)
https://www.youtube.com/watch?v=vujVmugFsKc
This video explains a lot on server architecture and technology uber uses, but its bit old now. The speaker mentions 'mobiles update GPS data every 4 second', what would the protocol be in this case? Poker apps in common connects upto 9 players per table and updates the cards in realtime for each mobile, what protocol they use for mobile-server communication? technologies they use and architecture of their server?
Few useful links http://yalantis.com/blog/uber-underlying-technologies-works/
Comment: I will be interested in knowing if they use any sort of messaging platform for real-time communication.
Answer: Stackshare lists this information:

http://stackshare.io/uber
http://stackshare.io/lyft
Answer: You can get a good understanding of a companies basic stack by looking at their jobs page:

https://www.uber.com/jobs/list?team=Engineering
https://www.lyft.com/jobs


How to replace While loop by CROSS JOIN?

Question: As I have heard WHILE loop can be replaced by CROSS JOIN to optimize a query, but I have no idea how to do this.
Can anyone help me on this?
Thanks in advance!
Comment: We need more examples. And you probably mean CROSS APPLY anyway. See https://www.google.com.mt/?ion=1&espv=2#q=cross%20apply%20replacing%20cursor
Answer: Example : If you want to produce numbers from 1 to 10000, use
<code>select top 10000 row_number() over (order by (select 1)) as number from
sys.objects as t1 cross join sys.objects as t2
</code>


Configuring Solr/Sarnia Fulltext Search in views

Question: I am pulling in content from a Solr server using the Sarnia module, and displaying it using views. If no keyword is entered into the views fulltext filter all results are displayed, but for some reason (which I need help figuring out) the views fulltext search is not producing any results when a keyword is entered into the search box. I've spent days searching the issue queues and google but haven't found a solution. Any help appreciated.
Example query:
<code>Index: sarnia_sarnia_solr_search
Keys: 'scientific explorer'
Parsed keys: array (
    0 => 'scientific',
    1 => 'explorer',
    '#conjunction' => 'OR',
  )
Searched fields: content, spell, tm_title
Filters:
    ss_type = 'product'
Sorting: fs_field_percent_savings DESC
Options: array (
    'parse mode' => 'terms',
    'conjunction' => 'OR',
    'filter class' => 'SearchApiQueryFilter',
    'search id' => 'search_api_views:shop:page',
  )
</code>
Update: I now get some results by going to Views>>Query settings and changing the parse-mode from multiple terms to single term. The problem is that while a search for the full title "scientific explorer" yields results, a search for just "scientific" or "explorer" comes up empty.
Comment: It would be helpful to state the reason for the down vote. Is it that I did not supply enough information, the question was not worded clearly, it's not considered appropriate for this site, duplicate or other reason? I am always willing to improve, but it's hard if I don't know what I'm doing wrong.
Answer: After much digging around (and prayer) I was able to get the search working:

Make sure the content you wish to search by is indexed into Solr as type fulltext
Under /admin/config/search/search_api/server//sarnia/schema/fulltext, make sure the content you wish to search by is of type tm_* (text).

Additionally, under Views>>Query settings you can set the parse-mode to single term to get more targeted search (Exact matches) or multiple terms to get results for each word of the search phrase (or even word stem if so configured in Solr)


The Essential Supremum as a Limit

Question:
Let $(X, \mathcal F, \mu)$ be a finite measure space and let $f\in L^\infty(X, \mu)$.
  Define $\alpha_n=\int_X |f|^n\ d\mu$.
  Then $$\lim_{n\to \infty}\frac{\alpha_{n+1}}{\alpha_n}=\|f\|_\infty$$

I am a bit lost here. Can somebody please help.
Thanks.
Comment: **HINT** I think this may help. Note that
$$
\frac{\alpha_{n+1}}{\alpha_n}=\frac{\|f\|_{n+1}^{n+1}}{\|f\|_{n}^{n}} = \left( \frac{\|f\|_{n+1}}{\|f\|_{n}} \right)^n \|f\|_{n+1}
$$ Now, use the fact that when the space has finite measure, $\lim_{p\to \infty }\| f\|_p=\|f\|_\infty$, which has already been proven here http://math.stackexchange.com/questions/242779/limit-of-lp-norm
Answer: Since $f\in L^\infty(X,\mu)$ and $\mu(X) < \infty$, then $f\in L^n(X,\mu)$ for all $n \ge 1$. Since
$$\alpha_{n+1} \le \|f\|_\infty \int_X  |f|^n\, d\mu = \|f\|_\infty \alpha_n,$$
we have $$\varlimsup_{n\to \infty} \frac{\alpha_{n+1}}{\alpha_n} \le \|f\|_\infty.$$
On the other hand, since $\alpha_n^{1/n} = \|f\|_n \to \|f\|_\infty$ as $n\to \infty$,
$$\varliminf_{n\to \infty} \frac{\alpha_{n+1}}{\alpha_n} \ge \varliminf_{n\to \infty} \alpha_n^{1/n} = \|f\|_\infty.$$
Therefore $$\lim_{n\to \infty} \frac{\alpha_{n+1}}{\alpha_n} = \|f\|_\infty.$$
Note. We can get $\varliminf_{n\to \infty} \frac{\alpha_{n+1}}{\alpha_n} \ge \|f\|_\infty$ alternatively by applying Holder's inequality with conjugate exponents $n+1$ and $(n+1)/n$. Doing so, we obtain $\alpha_n \le \mu(X)^{1/(n+1)}\alpha_{n+1}^{n/(n+1)}$. Thus
$$\varliminf_{n\to \infty} \frac{\alpha_{n+1}}{\alpha_n} \ge \varliminf_{n\to \infty} \mu(X)^{-1/(n+1)} \alpha_{n+1}^{1/(n+1)} = \lim_{n\to \infty} \|f\|_{n+1} = \|f\|_\infty.$$
Comment: That's great. Thanks!

The Essential Supremum as a Limit


C++ Qt unable to open a file

Question: I'm attempting to write a text editor. I'm unable to open a file. Why? Here's my code:
void MainWindow::readfromfile(){
QFile inputFile(":/text.txt");
inputFile.open(QIODevice::ReadOnly);
QTextStream in(&inputFile);
QString line = in.readAll();
inputFile.close();
ui->textEdit->setPlainText(line);
QTextCursor cursor = ui->textEdit->textCursor();
cursor.movePosition(QTextCursor::Start, QTextCursor::MoveAnchor, 1);

}

and
void MainWindow::on_actionOpen_triggered()
{
    connect(ui->actionOpen,SIGNAL(triggered()),ui->textEdit,SLOT(readfromfile()));
}

Comment: Shouldn't the slot connection be to `this` instead of `ui->textEdit` if the slot is defined inside MainWindow?
Comment: "it doesn't work" - means what? Crash? Empty `line`? Wrong data?
Comment: @MichaelBurr this means when i click , nothing happens.
Comment: @ChrisV tried, the same.
Answer: First, it seems that you are using a text file, so it might be worth specifying that when you open the file:
inputFile.open(QIODevice::ReadOnly | QIODevice::Text);

Then also check the output of the open() call:
bool ok = inputFile.open(QIODevice::ReadOnly || QIODevice::Text);
if (!ok) qWarning() << "Cannot open file";

If the file could not be opened, check that the file path is correct.
Comment: Good practice is to include inputFile.errorString() in the error message, as it might gives clues on why the file couldn't be opened.
Comment: There should be only 1 | in both open statements.inputFile.open(QIODevice::ReadOnly | QIODevice::Text);
Comment: when i put this : inputFile.open(QIODevice::ReadOnly || QIODevice::Text); , it gives me errors , http://img535.imageshack.us/img535/2172/screenshot20120218001.png

C++ Qt unable to open a file


Quantize/dither image using ImageMagick commands on image style actions

Question: Is there a way to use custom imagemagick commands on a image style action.
I'm trying to create a dither effect like the ones shown on the examples on the imagemagik site: http://www.imagemagick.org/Usage/quantize/#dither.
Here's another example: http://caca.zoy.org/study/out/lena5-2-1.png.
The command on the imagemagick website looks like this:
<code>  convert colorwheel.png -dither Riemersma      -colors 16 dither_riemersma.gif
</code>
plus
<code>  convert  logo.png  -remap pattern:gray50  mono_remap.gif
  convert -size 15x640 gradient: -rotate 90 \
                   -remap pattern:gray50     mono_remap_gradient.gif
</code>
but I don't know how or if it is possible to add this to an action.
Can someone shed some light on this? Is there an alternative method to accomplish this?
Answer: Imagecache actions has a 'custom' action which allows you to run your own actions. It may help you, though I used it years ago on a drupal 6 site with GD so can not say for certain if it will work for you.
Answer: Solved it by using the Raw ImageMagick Effect module and adding a custom action with the following option:
<code>-remap pattern:gray50
</code>
Final action should look like this after saving it:
<code>Raw ImageMagick options -remap pattern:gray50
</code>


Cannot persist relations in Core Identity's ApplicationUser

Question: I cannot find a way to properly extend ASP.NET Core's IdentityUser (after extending: ApplicationUser). I can override it and link it from other models, but cannot link from the user to other models. Simple data like CustomTag works.
Source code of the issue: https://github.com/chanibal/CoreIdentityIssue
Or, to be specific, it's second commit that contains all the changes over the default template
What I did:

Created new project:
ASP.NET Core Web Application (Model-View-Controller)
Authentication: Individual User Accounts, store user accounts in-app
Updated the database
Changed the model overriding the IdentityUser (as in official docs):
public class ApplicationDbContext : IdentityDbContext<ApplicationUser>
{
    public ApplicationDbContext(DbContextOptions<ApplicationDbContext> options)
        : base(options)
        {}

    public DbSet<Bar> Bars { get; set; }
    public DbSet<Foo> Foos { get; set; }
    public DbSet<ApplicationUser> ApplicationUsers { get; set; } //< doesn't help
}

public class ApplicationUser : IdentityUser
{
    public string CustomTag { get; set; }   //< this works
    public Bar Bar { get; set; }            //< THIS DOES NOT WORK
}

public class Bar
{
    public int Id { get; set; }
    public int Value { get; set; }
}

public class Foo
{
    public int Id { get; set; }
    public ApplicationUser User { get; set; }   //< this works
}

And migrated the changes
I'm updating the Bar values this way:

    /// This should ensure a Bar is connected to a ApplicationUser
    /// and increment it's value
    [Authorize]
    public async Task<IActionResult> IncrementBar()
    {
        // This DOES NOT work
        var user = await _userManager.GetUserAsync(HttpContext.User);
        if (user.Bar == null)   //< this is always null
        {
            user.Bar = new Bar() { Value = 0 };
            // _context.Add(user.Bar); //< doesn't help
        }
        user.Bar.Value++;
        // await _userManager.UpdateAsync(user); //<> doesn't help
        // await _signInManager.RefreshSignInAsync(user);  //< doesn't help, starting to get desperate
        await _context.SaveChangesAsync();

        return RedirectToAction(nameof(Index));
    }

The information is there in the database, accessible by SQL.
But somehow it doesn't hydrate the ApplicationUser model:

Using Visual Studio 16.3.9
Comment: How do the foreign keys look like in the generated database?
Comment: Added a minimal demo of the issue
Comment: The foreign keys configuration is available from the [model snapshot](https://github.com/chanibal/CoreIdentityIssue/blob/master/Data/Migrations/ApplicationDbContextModelSnapshot.cs). But please note, that the question has been already answered. Thank you for the help.
Answer: EF does not ever automatically load related entities. You must either eagerly or explicitly load the relationships. Eager loading is the preferred way, as it does joins to get all the data in a single query. However, UserManager<TUser> provides no way to eagerly load relationships. As such, you have two choices:

Explicitly load the relationship. This will require an extra query, though.
var user = await _userManager.GetUserAsync(HttpContext.User);
await _context.Entry(user).Reference(x => x.Bar).LoadAsync();
// note: for collection props, you'd use `Collection(x => x.CollectionProp).LoadAsync()` instead.

Query the user from the context by the user id, instead of using UserManager<TUser>:
var userId = User.FindFirstValue(ClaimTypes.NameIdentifier);
var user = await _context.Users.Include(x => x.Bar).SingleOrDefaultAsync(x => x.Id == userId);

Comment: You were not mislead. You misunderstood. EF does have an object cache, and it does automatically fix-up relationships when it has the data for those relationships in the object cache. That part is key, though. It has to already be there. If you query something later EF won't go back and alter entities you've already pulled out. Additionally, the context is scoped to the request, so you might have.queried something in a previous request, but that no longer exists
Comment: Thanks, both your solutions work. I was mislead, that EF does indeed load eagerly - because if it has data available, it links it despite not joining at SQL level, for example, having a user loaded via `UserManager` and then selecting `_context.Foos.ToArray()` (`Foo` has a `ApplicationUser` property), each Foo does have a `User` instance - because it was loaded before and is in some kind of cache.
Comment: True. Thank you again for the help.


jQuery only works if I hard code it into the html.tpl.php

Question: Running into this issue where my custom jQuery code only works in my drupal theme if I end up adding  in the  tag of the html.tpl.php file.
I tried all the other ways via template, info and still none of them work other then doing this.
Suggestions?
Thanks,
.-C
Answer: Drupal core ships with jQuery. You can check the version in your <drupal_install_directory>/misc/jquery.js file. If you want to use the later version of the library, you need to install the jQuery Update module.

Upgrades the version of jQuery in Drupal core to a newer version of jQuery.
  Updates...

Drupal 5 to jQuery 1.2.6
Drupal 6 to jQuery 1.3.2
Drupal 7 to jQuery 1.5.2/1.7.1/1.8.2, and jQuery UI 1.8.11
Comment: Yeah have that module installed an using 1.8.
Comment: I had used "Save Form State Module" in a recent project, and it worked fine for me using the jQuery update module.
Answer: Figured it out. Requires one to enclose all jQuery scripts with:
(function($){
\\  Enter script here ///
})(jQuery)


What is an exact characterization for the functions $f$ such that $xf'(x) \leq 2f(x)$?

Question: What is an exact characterization for the functions $f$ such that $xf'(x) \leq 2f(x)$?
I know, for instance, that the inequality holds for all functions $f(x) = c_0 + c_1x + c_2x^2$, with $c_0, c_1, c_2 \geq 0$. But it does not hold for $f(x) = x^2 - x$ or $f(x) = e^x$. I'm interested in the broadest possible class of functions that it holds for.
Note 1: For my application, we can also assume that $f(x), f'(x), f''(x) \geq 0$ for all $x \geq 0$ (i.e., it is positive, non-decreasing, and convex over $[0, \infty)$).
Note 2: I am interested in a slightly more general solution, $xf'(x) \leq bf(x)$, where $b \geq 1$ is a constant. But I should be able to generalize any solution for the $b = 2$ case.
Comment: Just a small note: You have equality exactly for functions of the form $f(x)=Kx^2$.
Comment: All positive differentiable  functions satisfying $f(x)\le Cx^2$ for some $C$?
Answer: One very general class of functions: It will hold for any "generalized" polynomial $\sum_n c_n x^{\alpha_n}$ for any choices of positive $c_n$ and any choice of positive real powers $\alpha_n < 2$ (i.e. you can have as many powers that you want, and they can be real instead of just integer). This should extend to infinite series that have the same types of powers, although they might not converge for all non-negative $x$ unless you restrict the sequence of $c_n$. But for example you could put $c_n \leq K/n!$ and get a series that converges for all non-negative $x$, thus defining another class of functions.
Answer: I'm not sure what you mean by "characterization".  For $x>0$, this should hold whenever $x^{-2} f(x)$ is a nonincreasing function, i.e. $f(x) = x^2 g(x)$ for any nonincreasing differentiable function $g$.
To derive this, write the given inequality as
$$
x\,f'(x) - 2\,f(x) \;\leq\; 0
$$
Assuming $x> 0$, we can multiply through by the integrating factor $x^{-3}$ to get
$$
x^{-2}f'(x) \,-\, 2x^{-3}f(x) \;\leq\; 0,
$$
which can be written
$$
\frac{d}{dx}\biggl[x^{-2} f(x) \biggr] \;\leq\; 0.
$$
Answer: For $x > 0$, the condition is equivalent to $\frac{d}{dx} \frac{f(x)}{x^2} \le 0$ while for $x = 0$, it simply says $f(0) \ge 0$. Therefore these are the differentiable functions on $[0,\infty)$ that satisfy
$$f(0) \ge 0, \quad x \mapsto x^{-2}f(x) \quad \text{ is non-increasing.}
$$
The generalization to the case $xf'(x) \le b f(x)$ is now obvious.
Answer: The inequality yields (Let's say you have $b$ instead of $2$)
For $x>0$,  $xf'(x)-bf(x)\leq 0\Rightarrow x^bf'(x)-bx^{b-1}f(x)\leq 0$
$\Rightarrow \frac{x^bf'(x)-bx^{b-1}f(x)}{x^{2b}}\leq 0 \Rightarrow (\frac{f(x)}{x^b})'\leq 0 $
This means that the function $g(x)=\frac{f(x)}{x^b}$ is strictly decreasing.
I think this is what you want.


if $A \subset X$ is compact then $d(x,A) = d(x,a)$ for some $a$

Question: Let $X$ be equipped with the metric $d$. If  a nonempty subset $A \subset X$ is compact, then $d(x,A ) = d(x,a)$ for some $a \in A$.
My attempt:
$$d(x,A ) : = \inf\{d(x,a)\ | \  a \in A\}$$.
$$d : X \times A \to \mathbb{R}$$.
Now, all I need to show is that if $A$ is compact, it's image under $d$ is closed  (1)
+
infimum of closed sets are contained in closet sets (in $\mathbb{R}$).  (2)
Can anyone help me out here?
Comment: Some variant of this question seems to arise every few days. I guess various syllabi must exhibit convergent evolution?
Comment: Show that $x \mapsto d(x,A)$ is Lipschitz continuous.
Answer: Hint: a real-valued continuous function on a compact set attains its infimum.
Answer: Find some $(a_{n})$ such that $a_{n}\in A$ and $d(x,a_{n})\rightarrow d(x,A)$. Since $A$ is also sequentially compact, there exists some convergent subsequence $(a_{n_{k}})$ with limit point $a\in A$, so $d(x,a)=d(x,A)$.
Answer: Recall that in a metric space:

The image of a compact set under a continuous map is compact.
A compact set is closed.

Instead of the map you consider it is better to consider:
$f_x: X \to \mathbb{R}$ given by $f_x(a)= d(x,a)$
or also just defined on $A$ but the former feels a bit more natural.


what number pyramids have one unique solution?

Question: Question
I came up with this question myself, so I don't know if it has a (nice) solution.
Consider number pyramids like this one, where two neighbouring numbers added are equal to the one on top of them. Only some of them are filled in, though.
If we are given a number pyramid of size n, and at least n spots where numbers have been put in so that it has at least one valid solution, how can we determine if it has one solution or infinitely many?
Example
We are given a pyramid of size 4 where the following spots have been filled in
<code>          --------
          |Filled|
       ---------------
       |      |      |
   -----------------------
   |      |Filled|      |
-----------------------------
|Filled|      |      |Filled|
-----------------------------

</code>
For example they could be filled like this
<code>          ---------
          |   15  |
       ---------------
       |      |      |
   -----------------------
   |      |   3   |      |
-----------------------------
|   1  |      |      |   5  |
-----------------------------
Wich leads to infinitely many solutions:
          ---------
          |   15  |
       ---------------
       | 4+a  | 11-a |
   -----------------------
   |  1+a |   3   |  8-a |
-----------------------------
|   1  |  a   |  3-a |   5  |
-----------------------------

</code>
Infact, all solvable pyramids of this kind have infinitely many solutions.
One idea I have is to interpret this pyramid as a system of linear equations with n(n+1)/2 variables.  I do not now how to use this idea, though.
Comment: Your clarifications are good, but they should go in the body of the question, not the comments.
Comment: Your comments are on point. If you rephrase this problem as a system of linear equations, then you can use the tools of linear algebra to solve the problem/
Comment: @ThomasAndrews height, and by that length. So the example has size 4 and the link has size 7.
Comment: You wouldn't need that many variables.  Once you fill in the blanks in the bottom row, everything percolates up.  In the example you posted, you just need to fill the box to the right of the $9$ with an $a.$  That determines the boxes to the right until you get to the $23$.  The fill the last two boxes on the bottom with $b$ and $c$.  Now just go upwards.  Eventually you will run into boxes that give you linear equations involving $a,b,c$ and you just have to solve the system.
Comment: There will certainly be times where there is only one solution - where ever value can be determined from every other value. Indeed, if the $n$ values in the bottom row are filled, there is only one solution, at most (possibly zero if more values are filled.)
Comment: One idea i would have is to interprete the pyramid as a system of linear equations with n(n+1)/2 variables, one for each brick. I don't know how to take advantage of that, though.
Comment: I came up with this question myself, so idk if it is even solvable
Comment: What is your definition of "size" of a number pyramid? Is it the height?
Comment: It's not clear what "at least $n$ spots where numbers will put in" means. Do you mean where the starting condition would start with $n$ filled, or $n$ spots without numbers?
Comment: @ThomasAndrews we start with at least n spots filled, the others are to be filled in by us to get to a solution. Would "and at least n spots where numbers have been put in" be a better description?
Comment: @saulspatz which box do you mean with "the box to the right of the 9" ?
Comment: The second box from the left on the bottom.
Comment: If the last value (bottom-most row rightmost  value) anything else other than 5, (everything else remain same), how many solutions are you getting?
Comment: @amitava then there are none. Let a,b,c,d be the values of the bottom row, so that d is the one your intrested in. a=1 and 3=b+c. The top value,15, is equal to a+3b+3c+d, you can figure that out by filling the pyramid bottom to top and starting with the values a,b,c and d. So 15=a+3b+3c+d=15. Combining these equasions gives you d=5.
Comment: Yes I also could not see a possible solution, then I am missing something, why did you write "all solvable pyramids of this kind have infinitely many solutions". I mean exactly what kind.
Comment: @amitava the kind means pyramids of size 4 with given numbers in the corners and in the middle. I wrote "all solvable pyramids" to say that the given numbers are arranged in a way that must allow for a solution.
Answer: Here are some observations:

You can fill in all but $n$ values and still have an undetermined solution:
You can get an overdetermined system with as few as three values:

It takes at most $n$ values to completely, uniquely determine the rest.
Here's a useful question: what is the maximum number of spaces you can freely fill in? How many— and which —spaces can you fill in such that (1) the resulting pyramid has a unique solution, but (2) if you remove any of your specified values, the pyramid no longer has a unique solution.
For example, you can fill in the bottom row of $n$ spaces. This determines the entire pyramid, and if you erase any of those spaces, the pyramid no longer has a unique solution.   Can you fill in more than $n$ spaces somewhere and still have this property?
By a dimensionality argument, it turns out that for any pyramid, you can fill in at most $n$ spaces before the rest are all determined. Free pyramids have exactly $n$ independent spaces filled in. It doesn't have to be the bottom row, but it also cannot just be anywhere; here are all of the fully specified pyramids for $n=3$:

For any pyramid (in this case $n=3$), if your filled in values match a free pyramid, the problem is guaranteed to have exactly one solution, no matter what values you've chosen.  If you've filled in a superset of these values, the problem either has one solution (if you chose those values consistently), or no solution at all.  If you've filled in a subset, the problem has infinitely many solutions. If you've filled in some other subset of values, all bets are off: the problem may have no solution, or infinitely many solutions, or one.

I note that if you represent the summation requirement as a homogeneous system of equations $A$, the null space of that matrix has a basis comprised of the "ancestral graphs" of the leaves. (My term: the ancestral graph of a leaf is a vector where the $i$th entry is the number of paths from the leaf to the root passing through node $i$.)  The filled-in solutions that uniquely determine the whole pyramid have some relation to the null space, but I haven't determined exactly what.
Comment: Thanks for your observations. In the case n=3 you missed the one with the corners filled
Comment: @Laurens Thanks! Which case?
Comment: https://primakom.dzlm.de/primafiles/uploads/Unbenannt_7.png this case. The bottom middle value can be solved for by subtracting the left and right value from the top and then dividing by two
Answer: You can test whether the values uniquely determine a pyramid solution by testing whether a certain matrix is invertible.
Theorem. Consider a pyramid with height $h$ and $m\equiv h(h+1)/2$ nodes. Define the $h\times m$ ancestral path matrix $N$ as follows:
$$N_{i,j} = \text{number of distinct upward paths from the }i\text{th leaf node to node }j.$$
In other words, if you express each node's value as a sum involving just the values of the leaf nodes, $N_{i,j}$ is the coefficient of the $j$th leaf in the sum defining the value of the $i$th node. (More intuitively, each row of $N_{i,j}$ records the values of the nodes in the pyramid when one base node has value 1 and the other base nodes have value 0. These are primitive cases; every valid solution is made of multiples of these; they're a basis for the solution space.)
(!)  If you fill in $h$ values of a pyramid, then those values form a unique solution to the pyramid if and only if the corresponding $h$ columns of $N$ are linearly independent.
(Note: So the uniqueness is determined solely by the placement of the $h$ variables; their values may be chosen freely.)

For example, the ancestral path matrix for height $h=3$ looks like:
$$\begin{bmatrix}1&1&&1&&\\2&1&1&&1&\\1&&1&&&1\end{bmatrix}$$
(In this basis, variables are listed in "topological" order, where higher nodes come before lower nodes.) You can pick any $h=3$ columns of this matrix, corresponding to filling in any $h$ variables. Those columns are linearly independent if and only if filling in those variables will uniquely determine a solution.

Proof. A pyramid defines a homogeneous system of equations. This system can be represented as a matrix $A$. You can find a basis for the solution space by forming the augmented matrix $[A^T|I]$ and reducing it using Gaussian elimination, forming a matrix $[M | N^\prime]$. The rows of $N^\prime$  corresponding to all-zero rows of $M$ comprise a basis for the solution space (null space) of $A$. Call that submatrix $N$.
For the system of equations defining a pyramid, it is easy to prove that the null matrix $N$ is the ancestral path matrix of the pyramid. (This is because row reduction of $[A^T | I]$ always happens in a straightforward way; starting from the top of the pyramid, each parent row is added to all of its child rows. After this process is complete, the matrix is in reduced form.)
If $h$ columns of the null matrix are linearly independent, then those variables uniquely determine a solution, as described in my answer here: How to define pivot columns? .

So you can determine whether a certain set of filled in values uniquely determines a solution by selecting the corresponding columns of $N$, and computing the determinant of that submatrix.  That set of values uniquely determines a solution if and only if the determinant is non-zero.
Comment: Maybe more intuitively: Put $h$ arbitrary values in places where you can't solve for any one value given the others. Those $h$ values uniquely determine a solution for the pyramid, and this characterizes all such solutions.
Answer: Let $x_v$ be the value of cell $v$ in the pyramid in a solution, and let $c_v$, for $v \in V$, be the given fixed values. The $x_v$ are goverened by the equations
$$
x_v = x_{\ell(v)} + x_{r(v)},
$$
where $\ell(v),r(v)$ are the two children of $x_v$.
After substituting the values $c_v$ for $x_v$, you get an inhomogeneous linear system with one variable per each unfilled cell. If the rank of the system equals the number of variables, then there will be a unique solution, and otherwise there will be infinitely many solutions.
As an example, let us denote the cells in your pyramid as $1,\ldots,10$, reading from top to bottom and from left to right. We get the following system:
$$
\begin{align}
15 &= x_2 + x_3 \\
3 &= x_2 - x_4 \\
3 &= x_3 - x_6 \\
1 &= x_4 - x_8 \\
3 &= x_8 + x_9 \\
5 &= x_6 - x_9
\end{align}
$$
This is a set of 6 equations in 6 unknowns. However, the constraints are not linearly independent, since
$$
(x_2+x_3)-(x_2-x_4)-(x_3-x_6)-(x_4-x_8)-(x_8+x_9)-(x_6-x_9) = 0.
$$
The rank of the system is 5, and so there is a 1-dimensional space of solutions.
Answer: Pyramid of size n will have exactly $\dfrac{n(n+1)}{2} = \dfrac{n^2}{2} + \dfrac{n}{2}$ variables and exactly $\dfrac{n(n-1)}{2} = \dfrac{n^2}{2} - \dfrac{n}{2}$ equations. Note that variables in the bottom-most row do not create any new equations but rest all do. So we have exactly $(\dfrac{n^2}{2} + \dfrac{n}{2}) - (\dfrac{n^2}{2} - \dfrac{n}{2}) = n $
 extra free variables. Which when we fill up makes number of unknowns same as number of equations.
Unique solution is possible if the row vectors associated with the equations are linearly independent, otherwise not.
For this size 4 pyramid we have $(4*5)/2=10$ variables and $(3*4)/2 = 6$ equations. When we choose the 4 free variables, as is done here (15,3,1,5) in a particular way we get 6 unknowns and 6 equations.
$a_1+a_2+0+0+0+0 = 15--(1)\\
a_1+0-a_3+0+0+0 = 3--(2)\\
0+a_2+0-a_4+0+0 = 3--(3)\\
0+0+a_3+0-a_5+0 = 1--(4)\\
0+0+0+a_4+0-a_6 = 5--(5)\\
0+0+0+0+a_5+a_6 = 3--(6)$
Now the row vectors associates with this are linearly dependent. We get the last row vector as a linear combination of others, that indicates when we solve the first five equations we always get $0+0+0+0+a_5+a_6 = k$ for some k. So unique solution is not possible. For  solutions to exist values at the right side of the equation also should match, here k should be equal to 3, which is true here, so infinitely many solutions exist. If $k\ne 3$ then no solution exist. Here as k = 3, solutions exist and we can choose any value for $a_5$ that will give a value for $a_6$ and from this systematically we can get all other values.


The most general type of a given function

Question: I am studying for a test from operational semantics and type systems and I came across a task I am not sure how to approach.
The task is as follows: Determine the most general type for a function f. f(a,b,c,d) = g(c,d), where g = a(b).

I guess that function f(a,b,c,d) returns the output of a function g(c,d). So lets suppose that c,d are some variables of a basic type and a is a function with one argument b of some basic type. But I do not know what g = a(b) means when used without arguments when there is a call of this function in a form of g(c,d)... or if anything I just said is correct :-/
I am not sure what to do here. Can you give me a hint or redirect me to some article concerning this topic (ideally with an example like this one). The only thing I have found so far are general texts about type systems and semantics. Thanks a lot!
Comment: a is a function of one argument - that returns a function of two arguments (called g)
Comment: @user3125280 you should make it an answer.. it is an answer and a good one.
Answer: You have to ask yourself "what kind of function is a?"
Since a(b) is later used as g(c , d) (which is the same as (a(b))(c, d)), we know that the result of a(b) is a function. so "a" is a function that returns a function. Specifically it is a function that returns a function which takes two arguments c and d.
If the above is confusing (I'm not sure how familiar you are with functional programming, etc) it might help to consider a concrete example. Imagine you have two functions, foo and bar. foo multiplies two numbers (c and d) and bar adds two numbers.
a could be a function which chooses whether we want to use foo or bar.
So f(a , "bar", 1, 2) would be (a("bar"))(1, 2) = 1 + 2 = 3.
and f(a , "foo", 1, 2) would be (a("foo"))(1, 2) = 1 * 2 = 2.
Comment: Thanks.. this actually makes sense and helps a lot!


Convert Excel Range to HTML-Table

Question: Because I didn't find a proper solution, I developed an own code to convert an excel range into an html table. The result is a string and can be used within a mail for example.
Answer: The following code considers: cell dimension, font-family, -style, -size and alignment, border-style and -color (top, right, bottom, left) and interior-color
Function convertRangeToHtml(rng As Range) As String

    Dim r As Range
    Dim c As Range

    Dim strHtml As String

    strHtml = "
 " & vbNewLine & "
"

    For Each r In rng.Rows
        strHtml = strHtml & vbTab & "
" & vbNewLine
        For Each c In r.Columns
            strHtml = strHtml & vbTab & vbTab & "" & Trim(c.Text) & "	" & vbNewLine
        Next c
        strHtml = strHtml & vbTab & "
" & vbNewLine
    Next r

    strHtml = strHtml & "
" & vbNewLine & "
"

    Debug.Print strHtml

    convertRangeToHtml = strHtml

End Function

Function getInteriorColor(r As Range) As String

    getInteriorColor = "background-color: rgb(" & color2rgb(r.DisplayFormat.Interior.Color) & "); "

End Function

Function getBorder(r As Range) As String

    Dim varBorderSytle As Variant
    varBorderSytle = "border-style: "

    Dim varBorderWidth As Variant
    varBorderWidth = "border-width: "

    Dim varBorderColor As Variant
    varBorderColor = "border-color: "

    For Each b In Array(8, 10, 9, 7)
        Select Case r.Borders(b).LineStyle
            Case -4115
                varBorderSytle = varBorderSytle & "dashed "
            Case -4142
                varBorderSytle = varBorderSytle & "none "
            Case 1
                varBorderSytle = varBorderSytle & "solid "
            Case -4118
                varBorderSytle = varBorderSytle & "dotted "
            Case Else
                varBorderSytle = varBorderSytle & "solid "
        End Select

        Select Case r.Borders(b).Weight
           Case 1
                varBorderWidth = varBorderWidth & "1px "
            Case -4138
                varBorderWidth = varBorderWidth & "2px "
            Case 4
                varBorderWidth = varBorderWidth & "3px "
            Case 2
                varBorderWidth = varBorderWidth & "1px "
            Case Else
                varBorderWidth = varBorderWidth & "1px "
        End Select

        varBorderColor = varBorderColor & "rgb(" & color2rgb(r.Borders(b).Color) & ") "

    Next b
    varBorderSytle = varBorderSytle & ";"
    varBorderWidth = varBorderWidth & ";"
    varBorderColor = varBorderColor & ";"

    getBorder = varBorderSytle & " " & varBorderWidth & " " & varBorderColor

End Function

Function getCellDimension(r As Range) As String

    getCellDimension = "width=" & Chr(34) & r.Width * 96 / 72 & "" & Chr(34) & " height=" & Chr(34) & r.Height * 96 / 72 & "" & Chr(34)

End Function

Function getFontStyle(r As Range) As String

    Dim varFontColor As Variant
    Dim varFontName As Variant
    Dim varFontStyle As Variant
    Dim varFontSize As Variant
    Dim varFontBold As Variant
    Dim varFontItalic As Variant
    Dim varTextAlign As Variant

    varFontColor = "color: rgb(" & color2rgb(r.DisplayFormat.Font.Color) & "); "
    varFontName = "font-family: " & r.Font.Name & "; "
    varFontSize = "font-size: " & r.DisplayFormat.Font.Size & "pt; "

    varFontBold = "font-weight: normal; "
    If r.DisplayFormat.Font.Bold Then varFontBold = "font-weight: bold; "

    varFontStyle = "font-style: normal; "
    If r.DisplayFormat.Font.Italic Then varFontStyle = "font-style: italic; "

    Select Case r.HorizontalAlignment
        Case -4131
            varTextAlign = "text-align: left; "
        Case -4152
            varTextAlign = "text-align: right; "
        Case -4108
            varTextAlign = "text-align: center; "
        Case -4130
            varTextAlign = "text-align: justify; "
        Case 1
            Select Case Application.Evaluate("TYPE(" & r.Address(0, 0, external:=True) & ")")
                Case 1
                    varTextAlign = "text-align: right; "
                Case 2
                    varTextAlign = "text-align: left; "
                Case 4, 16
                    varTextAlign = "text-align: center; "
                Case Else
                    varTextAlign = "text-align: start; "
            End Select
    End Select

    Dim strCSS As String

    strCSS = varFontColor & varFontName & varFontSize & varFontBold & varFontStyle & varTextAlign

    getFontStyle = strCSS

End Function

Function color2rgb(varColor As Variant) As String

    color2rgb = Format((varColor Mod 256), "00") & ", " & Format(((varColor \ 256) Mod 256), "00") & ", " & Format((varColor \ 65536), "00")

End Function

Additionally I would suggest to add a css-style wihtin the final html-file:
td {
    padding:1pt 4pt 1pt 4pt;
}

Comment: I thought so too, but outlook can handle it
Comment: That CSS doesn't seem to be right one? Don't you need to suffix the numbers with `px`?

Convert Excel Range to HTML-Table


How did a Mexican soccer (aka "football") team end up with a stadium named "Nou Camp"?

Question: The Mexican team Club León's home stadium is unofficially known as the "Nou Camp," which is also the name of the stadium in Spain where FC Barcelona play their home games. However, "Nou Camp" has no meaning in Spanish and I don't believe there are many Catalan speakers in Mexico so I was surprised to encounter a seemingly-Catalan name.
The relevant entry in Spanish-language Wikipedia makes an attempt at explaining where the name came from but it doesn't quite make sense. Here's my translation:

Contrary to what people believe and because of a mistranslation from Spanish, the stadium was erroneously named "Nou Camp", which is the name (in Catalan) of FC Barcelona's stadium. People in Barcelona (?!) started calling their stadium "Camp Nou" (which means "new field" in Spanish), and that's the name that was eventually chosen.

Even if the word Barcelona should be León, I still can't think of what sort of translation error would lead to a Catalan name for a Mexican stadium.
For posterity, here's the Spanish text I made my translation from:

En contra de lo que la gente cree y por una mala traducción del español, se le dio el erróneo nombre de "Nou Camp", que coincidía con el del FC Barcelona, cuyo nombre sí proviene del catalán. La gente en Barcelona empezó a llamarle a su estadio Camp Nou, que en español es Campo Nuevo, y al final fue el nombre elegido.
Comment: The stadium is officially named "Estadio León," and "Nou Camp" is a nickname.  I wonder if there were a lot of Barcelona fans in León when they built the stadium in the 1960s, and the campo nuevo (new field) was essentially nicknamed after the stadium in Barcelona.
Comment: The second sentence from the Spanish wikipedia is not inaccurate. Although the stadium was always called "Camp Nou" by most people, it was named something else until around 2000, when the name was officially changed to Camp Nou. I'm not sure why the wikipedia includes that fact about the stadium in Mexico, but it's accurate. It still doesn't really say why the Mexican stadium is called that, though.
Comment: @BenMiller Why don't you post that as an answer?
Comment: @JacobJanTuinstra Because it's only an observation and a guess.
Answer: From what I found, Angel Fernandez, the great commentator who is well known for his breathless GOOOOAAAAAAL screams, had a hand with creating the nickname 'Nou Camp'.
Here's the translated version from this link:

Angel Fernandez, an excellent football commentator, was the first person to call it the "Nou Camp" in those distant years, which resembled that of the Camp Nou in Barcelona.

For those who know Spanish, here's the original text

Ángel Fernández, un excelente comentarista de fútbol, fue la primera persona en llamarle "Nou Camp" por el ambiente del estadio que en aquellos años lejanos, que se asemejaba al de el Camp Nou de Barcelona. Eso fue en una narracion de un juego de fútbol, poniéndole asi al estadio y citando el estadio de Barcelona.
Comment: Here's a full translation (with a bit of poetic license for the last sentence, as I'm not sure exactly what the commenter is trying to say): Angel Fernandez, an excellent football commentator, was the first person to call it "Nou Camp" because of the atmosphere of the stadium in those distant years, which was similar to that of Camp Nou of Barcelona. This was in a narration of a football game, putting you in the stadium and summoning the stadium of Barcelona.


Computing probability 1 player win given both follow normal distribution with same standard deviation

Question: I am given that score ("logscore") of 2 players follow a normal distribution. They both have the same standard deviation of 0.205 but the mean varies. Player A has mean logscore that exceeds B by 0.15. I need to compute the probability A wins B. How do I approach this problem?

I think the its also very similar to the below question (e). How do I do such comparisons on 2 normal random variables?
Answer: Consider two variables, $X_1$ and $X_2$  (logscores in this particular case)
Note that $P(X_1>X_2)=P(X_1-X_2>0)$
So if you can work out the distribution of $X_1-X_2$, you can solve the problem easily.
The second problem doesn't have independence, so it's a little more complex, but the same strategy applies.


Migrating from Google Apps to Office365. Prevent losing Google Cloud Services

Question: My company is currently using GoogleApps and we recently set up a Google Cloud Services account to manage our DNS records.
We now are moving out of Google Apps and into Office365 and to prevent losing the management privileges on Google Cloud I tried granting permissions to my personal gmail account with no luck, it throws an error saying that the email account must be from the domain.
I have some questions regarding to this problem:
Will my GoogleApps account be deleted?
If so, how can I prevent losing privileges in Google Cloud?
The only way out that I see is creating a generic "firstname.lastname@example.com" account, then create a Google Cloud Account with that Id, then migrate the DNS records to the new Google Cloud Account, finally change the DNS in the Domain Registrar.
Is there  an easier solution to this problem
Answer: Well I will answer this if someona has the same problem.

I was able to contact google support and they informed me that: There
is no way of adding a non-domain (non-google-apps-account) as
administrator for Google Cloud when Google Cloud account was created
with a google-apps-account.
There is also no way of migrating the Google Cloud Account.

The only solution is to create a new Google Cloud Account and manually migrating the services.


Finding the MVUE of the center of a circle of unknown location

Question: Is there a known analytic solution for finding the minimum variance unbiased estimator of a disk of an unknown location given that a sample of $n$ points was drawn uniformly and randomly from the disk of known radius $r$; and, if so, what is it?
Intuitively, I would expect it to be the midrange of the longest segment of the set of all possible pairs of points, but that might discard information created by the added dimension.
EDIT
Unfortunately, I am stuck rather early in the process and I am aware that an MVUE may not exist.  A unique maximum likelihood estimator does not exist.
First, I define $$\hat{\theta}_x=\hat{\theta}_x(x_1,x_2,\dots,x_n)$$ and $$\hat{\theta}_y=\hat{\theta}_x(y_1,y_2,\dots,y_n).$$
Then, I am trying to solve for $$\min_{\hat{\theta}}\mathbb{E}[(\theta_x-\hat{\theta}_x)^2+(\theta_y-\hat{\theta}_y)^2]$$ subject to $$\mathbb{E}(\theta_x-\hat{\theta}_x)=0$$ and $$\mathbb{E}(\theta_y-\hat{\theta}_y)=0$$ and $$(x_i-\theta_x)^2+(y_i-\theta_y)^2\le{r}^2$$ and $$f((x_i,y_i)|(\theta_x,\theta_y))=\frac{1}{\pi{r^2}}.$$
I had hoped that if I built a Lagrangian, the inequality constraints would be useful in determining $\hat{\theta}$ even if through some envelope condition, but, of course, as $\hat{\theta}$ isn't a term in the inequalities, they drop out entirely.
The other thought I had was to possibly, in some way, use the fact that the variance of $x$ and $y$ were known to constrain the estimator in some way.
Comment: Could you please elaborate? how does your data look like and what are you trying to estimate (the disk center? the disk radius, which differs from $r$?)
Comment: @Xi'an I updated the question.
Answer: Here is a homework problem from Mark Schervish's Theory of Statistics that addresses a similar question:

Let $(X_1, Y_1),\dots,(X_n, Y_n)$ be conditionally IID with uniform distribution on the risk of radius $r$ centered at $(\theta_1, \theta_2)$ in $\mathbb R^2$ given $(\Theta_1, \Theta_2, R)=(\theta_1, \theta_2, r)$.
a. If $(\Theta_1, \Theta_2)$ is known, find a minimal sufficient statistic for $R$.
b. If all parameters are unknown, show that the convex hull of the sample points is a sufficient statistic.

When only the center $\theta$ is unknown, the likelihood is constant over the set
$$\mathfrak O = \bigcap_{i=1}^n \{\theta; d(x_i,\theta)\le r\}$$
which is therefore a (set-valued) "sufficient statistic". There is however no sufficient statistic in the classical sense and any value in $\mathfrak O$ is a MLE.
The problem can be considered from a different perspective, namely as a location model, since
$$Z_1,\ldots,Z_n\sim\mathcal U(\mathcal B(\theta,r))$$
is the translation by $\theta$ of a
$$U_1,\ldots,U_n\sim\mathcal U(\mathcal B(0,r))$$
[ancillary] sample. Therefore one could consider the best equivariant estimator¹ attached with this problem (and squared error loss),
$$\hat\theta^\text P = \dfrac{\int \theta\,\mathbb I_{\max_i \vert\theta-z_i\vert<r}\,\text d\theta}{\int \mathbb I_{\max_i \vert\theta-z_i\vert<r}\,\text d\theta}$$
as established by Pitman (1933). This best equivariant estimator is

unbiased
unique
the center of the "sufficient" region mentioned above
with constant risk (by construction)
minimax (because of 4.)
the MVUE if the later exists
admissible under squared error loss (as Stein phenomenon only occurs in dimension 3 and more for spherically symmetric distributions), which means that it cannot be dominated everywhere by another estimator
but not sufficient.

Both 5. and 7. indicate that this estimator is minimum variance in the weak sense that there is no other estimator with a strictly everywhere smaller maximal MSE.

¹See Theory of Point estimation, by Lehmann and Casella, for a textbook entry to equivariance and best equivariant estimators.
Comment: I guess I will be buying a book.
Comment: I've revised your answer to use math typsetting in place of an image because this is a more accessible format for presenting information.
Comment: @Xi'an I prefer to think of it as the political nature of unbiasedness.
Comment: I do thank you for this answer, but I am looking for the unbiased estimator. This I have. It is very possible for the unbiased estimator to sit outside the intersection.
Comment: Isn't this answer giving an unbiased estimator?


Solution of $\int\frac{\sqrt {x^3-4}}{x}dx$ : Need Hints

Question: $$\int\frac{\sqrt {x^3-4}} x \, dx$$
My attempt: $ \displaystyle \int\frac{3x^2\sqrt {x^3-4}}{3x^3}\,dx$
Then, substituting $u=x^3$; $\displaystyle \int\frac{\sqrt {u-4}}{3u} \, du$
$$\int\frac{u-4}{3u\sqrt{u-4}} \, du$$
$$\int\frac{1}{3\sqrt{u-4}}\,du-4\int\frac{1}{3u\sqrt{u-4}}\,du $$
I am having trouble with the 2nd part. And Wolfram Alpha says
Can you give me some hints on how to get arctanh function here?
Comment: You can simply assume x^3-4 as u ,then simplify it and proceed with multiplication rule .
Answer: The function given by Wolfram Alpha is somewhat problematic in that is contains both $\sqrt{x^3-4}$ and $\sqrt{4-x^3}$, so it unnecessarily brings in complex trig identities.
Hint: One great thing about $\frac{\mathrm{d}x}x$ is that the substitution $x\mapsto x^a$ maps $\frac{\mathrm{d}x}x\mapsto a\,\frac{\mathrm{d}x}x$. Thus, setting $x=u^{2/3}$ gives a standard trig substitution on $u$.
Full answer: if you want to peek, mouse over

$$ \begin{align} \int\frac{\sqrt{x^3-4}}{x}\,\mathrm{d}x &=\frac23\int\frac{\sqrt{u^2-4}}{u}\,\mathrm{d}u\\ &=\frac43\int\frac{\tan(\theta)}{\sec(\theta)}\tan(\theta)\sec(\theta)\,\mathrm{d}\theta\\ &=\frac43\int\tan^2(\theta)\,\mathrm{d}\theta\\ &=\frac43\int\left(\sec^2(\theta)-1\right)\mathrm{d}\theta\\ &=\frac43(\tan(\theta)-\theta)+C\\ &=\frac43\left(\sqrt{\frac{u^2}4-1}-\sec^{-1}\left(\frac u2\right)\right)+C\\&=\frac43\left(\sqrt{\frac{x^3}4-1}-\sec^{-1}\left(\frac{x^{3/2}}2\right)\right)+C \end{align} $$
Answer: $HINT$ : let $u-4 = t^2$, so $du = 2t\,dt$ and $u = t^2+4$. So it's easy to see where $\arctan$ comes.
Answer: \begin{align}
w & = \sqrt{u-4} \\
w^2 & = u-4 \\
2w\,dw & = du \\
w^2+4 & = u\\[15pt]
\int \frac{\sqrt {u-4}}{3u} \, du & = \int\frac{w}{3(w^2+4)} (2w\,dw) = \frac 2 3 \int \left( 1 - \frac 4 {w^2+4} \right) dw
\end{align}
etc.
Comment: I believe it should be $u=w^2\color{#C00}{+}4$.


JavaScript keydown not working on Firefox

Question: <code>$("body").keydown(function(e) {
    if (e.which == 116) {
        window.location.href = "google.com";
    }
});
</code>
I used this code so that every-time someone press f5 it will force them to go to other URL. I tested this with Chrome and it worked fine but with FireFox, nothing happens.
Please help me with this.
Thank you!
Comment: which version of `FireFox` you are using ? try to use `e.keyCode`
Comment: This is a pure guess, but FF protects some system keyboard shortcuts from overriding, like CTRL + D, maybe "The Refresh Button" is also one of these?
Comment: Both `e.which` and `e.keyCode` are deprecated. Use `e.key` instead.
Comment: @connexo Nope, see [e.which](https://api.jquery.com/event.which/#event-which1). Notice, that OP has a jQuery event, not native.
Comment: Well I wouldn't consider jQuery for a second in the first place to achieve that. Well spotted nonetheless.
Answer: this is working fine
<code>$(function() {
  $("body").keydown(function(e) {
    e.preventDefault();
    if (e.which == 116) {
      window.location.href = "http://google.com";
    }
  });
});
</code>
Comment: just a question, how can I detech if Ctrl +f5 is pressed? like this: `e.which == 116 && e.which == ctrlKey`
Comment: something like this if (e.which == 116 && e.ctrlKey)


How to get the difference of different row

Question: I have the following table:
name        date      price
productA    Jan2019   3000
productA    Feb2019   3500
productA    Mar2019   3200
productB    Jan2019   2500
productB    Feb2019   2700
productB    Mar2019   2800

I want to calculate the % change of Mar2019 and Feb2019 of each product. For example, for productA, the percentage is (3200-3500)/3500=-8.6%
The desired result is
name         date        %change
productA     Mar2019     -8.6
productB     Mar2019     3.7

How to write the sql command?
Thanks
Answer: You can use conditional aggregation.  Something like this:
select name,
       (1 -
        max(case when date = 'Mar2019' then price end) /
        max(case when date = 'Feb2019' then price end)
       )
from t
group by name

Comment: @PPJN . . . The question is specifically about March 2019 and Feb 2019.  If the OP has a more general requirement, then that should be asked as another question.
Comment: One downfall would be that this would have to be actively managed. You'd have to constantly update case statements. My solution would avoid this.
Answer: To make the process much more clean, I would recommend storing your date as a date and making it the first of the month. This way it would be an easier join. You would join on the product and the date less 1 month for the prior month. Where there isn't a prior month, I input NA.
create table #test  (
                        name    varchar(16)
                        ,[date] date
                        ,price  decimal(18,2)
                    )

insert into #test
values      ('productA',    '2019-01-01',   3000),
            ('productA',    '2019-02-01',   3500),
            ('productA',    '2019-03-01',   3200),
            ('productB',    '2019-01-01',   2500),
            ('productB',    '2019-02-01',   2700),
            ('productB',    '2019-03-01',   2800)

select      a.name                      name
            ,format(a.date,'yyyyMM')    [date]
            ,case
                when b.date is not null then cast(cast((a.price - b.price)/b.price * 100 as decimal(18,1)) as varchar(25)) + '%'
                else 'NA'
            end                         change
from        #test   a
left join   #test   b   on      b.name = a.name
                            and b.[date] = dateadd(month,-1,a.date)

drop table #test


Standard effect pedal connector/barrel size?

Question: I use the regular 9V DC negative center power adapter, and every so often the connector gets ripped or the internal soldering wears off and in middle of a gig the adapter stops working.
It usually happens at the connector (barrel) side (see following fig):

What I had in mind is buy a heavy duty cable and solder it myself to the adapter so it's more durable, plus having an external power cord to the adapter so if I need extended length it's the power cord rather than the adapter cord.
TL;DR
What's the size of the connector in standard (i.e. Boss) 9V DC adapters? Here are some examples, looks like it's the 5.5*2.5, but maybe it's not even in this list.
Please disregard the other side of the connectors in the picture below. All I'm interested to know is the size of the connector (to the left).
Thanks
Comment: You can post it as your own answer if you want. Definitely I think you should keep the question. It's a pretty good one.
Comment: You might be better soldering straight to the internal socket, and taking a cable out through a small hole you make, so the socket is still available. Open the pedal and hardwire to the socket itelf. that way, basic options are still available
Comment: That's what I'm planning to do, but I need the right connector and I'm not sure what's the size of it.
Comment: No. I mean solder inside, so the connector isn't involved.
Comment: The Voodoo Lab pedal power manuals will have this info. I'll come here and quote it later if you haven't found it before then. It's not only the outer diameter but also the inner diameter that counts.
Comment: @Tim solder inside isn't an option. I'm dealing with multiple FX.
Comment: @ToddWilcox, thank you - the info was [right there](http://www.voodoolab.com/pedalpower_2.htm) (see at the table under _Output_), please post as answer or should I delete the question?
Comment: I suspect you use several on stage, daisychained up. If they're all separate, and get bundled away at the end, there's way too much handling going on. That apart, the pots get knocked and twisted in the gigbag. Far better to obtain a case where they all sit happily protected and no need to touch the plugs/wires at all. Cases are available, or build your own, even put the power supply in there as well. An old briefcase was adapted for some of mine. Foam helped.
Answer: As Todd Wilcox referenced in his comment, the standard barrel used in FX is of:
5.5 x 2.1 mm
Answer: OK, so it's 5.5 x 2.1 mm.
But don't.  When there's a plug-and-socket connection that is vulnerable to getting 'ripped', the answer isn't a heavier plug.  That just ensures that when it IS ripped, more damage gets done, probably to the socket.
Mount your pedals on a board, tether the cables so strain will never be placed on the plug/socket interface.
Comment: It's not the plug that gets damaged, it's the connection to the cable between the barrel and the wire. But you definitely got a point.


CSS to display three column content

Question: I am trying to get three column output using
 for each.
Right now my div on the left is lined properly even the middle one, but when I try to put right it is just clipping down.
CSS
/* left div */
 #left-sidebar, #right-sidebar {
    width:15%;
    height:700px;
    float: left;
    padding-left: 15px;
    padding-top: 20px;
}

/* middle div */
 #mid-content {
    width:75%;
    padding-left: 30px;
    padding-top: 20px;
}

/* right div */
 #right-sidebar {
    float:right;
}

Comment: your right div should be just 10% or less, specify it in your css
Answer: First of all, your elemnents add up to 105% in all.
Second, you have added padding to a floating element which have a width specified in %, not a good idea. Look at http://www.w3schools.com/css/css_boxmodel.asp
Here is a solution http://jsfiddle.net/Ru6CL/
#left-sidebar,
#right-sidebar,
#mid-content {
    float:left;
}

#left-sidebar,
#right-sidebar {
    width: 15%;
    background-color: red;
}
#mid-content {
    width: 70%;
    background-color: blue;
}

remember to use clear: both;
Comment: You should never use tables for design purpose.
Tables are made for data.

Can you try to make a jsfiddle version of your page so we can see it?
Comment: @KyleMit i did the modifications but still the right-sidebar clips down.. also please tell me where and how to use clear:both
Comment: @KyleMit should i use a table to place all the divs in each respective coloumns???? my left-sidebar contains a navigation panel like when we hover on it the sub navigation is floated left and then the mid-content and later again the right-sidebar...
Answer: What happens if you change the values to add up to 100%?. Right now, 75%, 15% and 15% adds up to 105%.
Comment: @AndrewEvt lol, another miscalculation in the same post :) thanks for the hint. I edited the answer.
Comment: manipulated my calculations.. got it under 100%.. but still the problem exist
Answer: That happens because according to CSSs box model,  the size of the element is the sum of its width, border and padding thus - your content+padding exceeds the 100%.
in order to solve this, just add
box-sizing: border-box;

to your columns and it will recalculate the necessary width along with the above taken into consideration.
In addition, you might consider floating the elements to the same direction, i.e to the left since the box-sizing thing might get thrown off otherwise.
I made a working example here: http://jsfiddle.net/v6peK/
Comment: I wouldn't advice to use box-sizing, i know it's supported in all major browsers, but it's bad practice.
Comment: It's being endorsed by Paul Irish for almost two years now: http://www.paulirish.com/2012/box-sizing-border-box-ftw/ and from personal experience - i beg to differ, however i do want to ask what makes you think so?
Comment: @arpymastro - i've update my answer and provided a working example.

CSS to display three column content


Proving $U=U$ is derivable from any set of relators.

Question: This is Exercise 1.1.1 of Magnus et al's book on combinatorial group theory.
The details:
Let $G=\langle X\mid R\rangle$ for sets $X, R$.

Definition 1: The empty word $\varepsilon$ and the words $\{aa^{-1}\mid a\in X\}\cup \{b^{-1}b\mid b\in X\}$ are called trivial relators.
Definition 2: Suppose $\{P, Q, \dots\}\subseteq R$. Then $W$ is derivable from $\{P, Q, \dots\}$ if the following operations, applied a finite number of times, turn $W$ into $\varepsilon$:
(i) Inserting one of $P, P^{-1}, Q, Q^{-1}, \dots$ or one of the trivial relators between two consecutive symbols of $W$, or before $W$, or after $W$.
(ii) Deleting one of $P, P^{-1}, Q, Q^{-1}, \dots$ or one of the trivial relators, if either forms a subword of $W$.
Definition 3: We say the equation $W=V$ is derivable from the relators $P_1=P_2, Q_1=Q_2, \dots$ (of $G$) exactly when $WV^{-1}$ is derivable from the relators $P_1P_2^{-1}, Q_1Q_2^{-1}, \dots (\in R)$.

The Question:

Show (A) by induction on the length of the word $U$ that $UU^{-1}$ is derivable from the trivial relators, and hence (B) that the relation $U=U$ is derivable from any set of relators.
[Hint: If $U=Va^\eta$, where $\eta=\pm 1$, then $UU^{-1}=Va^\eta a^{-\eta}V^{-1}$.]

My Attempt:
That (B) follows from (A) by definition, is obvious. Thus it suffices to prove (A).
Let $U$ be a word in $X$.
We proceed by induction on the length of $U$.
Suppose $\lvert U\rvert=0$. Then $U=\varepsilon$. One applies (i) and (ii) zero times to get the empty word from $UU^{-1}=\varepsilon\varepsilon=\varepsilon$. Thus $UU^{-1}$ is derivable from the trivial relators.
Assume that for $\lvert U\rvert=u$, we have that $UU^{-1}$ is derivable from the trivial relators.
Consider when $\lvert U\rvert=u+1$. Then $U=Va^\eta$ for some $a\in X$, $\eta=\pm 1$, $V$ a word in $X$ s.t. $\lvert V\rvert=u$, so that $UU^{-1}=Va^\eta a^{-\eta}V^{-1}$. The word $a^\eta a^{-\eta}$ is trivial, so apply (ii) once to get $VV^{-1}$, which is itself derivable from the trivial relators by the induction hypothesis.
Hence $UU^{-1}$ is derivable from the trivial relators. $\square$

Is this proof rigourous enough?

Thoughts:
The step . . .

"The word $a^\eta a^{-\eta}$ is trivial, so apply (ii) once to get $VV^{-1}$, which is itself derivable from the trivial relators by the induction hypothesis.
"Hence $UU^{-1}$ is derivable from the trivial relators."

. . . could do with some justification, I think.

Please help :)
Comment: ["Why ask?"](https://chat.stackexchange.com/transcript/message/46192457#46192457)
Answer: I think there are two improvements you can make. Once is minor, the other major.

In the step you ask about, I think it would be good if you wrote "$UU^{-1}=Va^{\eta}a^{-\eta}V^{-1}=VV^{-1}$, by (ii), so the result holds by induction". That is, I think you need to write the actual maths/use notation here to make it clear what is going on.
Your base case is wrong. You actually need to justify that $\epsilon\epsilon=\epsilon$. This holds because $\epsilon$ is a trivial relator so you can removed one of them (the left or right one - you choose!) via (ii). So one application of (ii), not zero.

As a side-note, I still remember my PhD supervisor saying that these exercises were an amazing achievement. I am not sure how many of them I have actually done though...(I think my supervisor might have done them all).
[Also, try to avoid the words "clearly" and "obvious". If it is clear then you don't need to say so. If you feel like something is missing from your explanation then neither of these words will fill that gap! So instead of
"That (B) follows from (A) by definition, is obvious."
write
"By Definition 3, (B) follows from (A)."]
Comment: Yes, I figured that was what you were thinking. But such thoughts are outwith the rules of the game!
Comment: My reasoning for $\varepsilon\varepsilon=\varepsilon$ was down to the fact that the empty word is just that: empty. It acts as the identity in the monoid that is any language under concatenation.


Deedle OptionalValue.Missing can't be dropped by Series.dropmissing

Question: this is the code example from http://bluemountaincapital.github.io/Deedle/reference/deedle-seriesmodule.html
let s1 = series [ 1 => 1.0; 2 => Double.NaN ]
s1 |> Series.dropMissing

the Missing value will be dropped as expected. however if I change this to
let s2 = series [ 1 => OptionalValue(1.0); 2 => OptionalValue.Missing ]
s2 |> Series.dropMissing

the missing value won't be dropped.
I noticed S2 is a
Series<int,OptionalValue<float>>

type while s1 is
Series<int,float>

Is this behaviour by design?
The Reason I asked this question is I have this code from this answer Deedle moving window stats calcuation with a dynamic condition and boundary.atending
let lastKey = ref None
let r =
  ts |> Series.aggregateInto
      (WindowWhile(fun d1 d2 -> d1.AddMonths(1) >= d2)) (fun seg -> seg.Data.LastKey())
      (fun ds ->
         match lastKey.Value, ds.Data.LastKey() with
         | Some lk, clk when lk = clk -> OptionalValue.Missing
         | _, clk -> lastKey := Some clk; OptionalValue(ds.Data))
     |> Series.dropMissing

The Series.aggregateInto somehow can return a non OptionalValue Series while still contain missing value. if I want to use OptionalValue.Missing in a series I create so they can be properly ignored by Stats.mean what is the right way to do it?
Also writing a Series/Frame to csv with missing value Deedle will put blank in the output. however if Series/Frame contains OptionalValue Deedle will put the string  in the output. is this by design?
Answer: The OptionalValue type in Deedle is the internal representation of optional values inside a series. So, if you have Series<Date, float>, it will actually store data as OptionalValue<float>. We do not completely hide this from users - sometimes (like in the aggregation sample), Deedle will take OptionalValue and then use it directly in the internal representation to make things faster.
However, you probably never want to use Series<K, OptionalValue<T>>, because that is an odd kind of series (Deedle handles missing values automatically, so there is no need for this).
If you want to specify missing values when creating a series, you can use:
let s2 = Series.ofOptionalObservations [ 1 => Some(1.0); 2 => None ]
s2 |> Series.dropMissing

The F# API generally prefers standard F# option type, so this is what ofOptionalObservations takes. The dropMissing function works fine on series created in this way.
Comment: Thank you for the explanation. This makes sense now. I was trying to build on logic of aggregateinto to have another level of check on the DS and ignore any DS who has the required number of window size but not all of the items in the window contain values.


Random data generation for hurdle model using R

Question: I would like to generate random data for a distribution which takes the form of hurdle-like model. Let's denote a random variable X with probability mass function
\begin{align}
Pr(X=0)&=\alpha\\
Pr(X=1)&=\beta\\
Pr(X=x)&=(1-\alpha-\beta)f(x;\theta)\quad x\geq2,
\end{align}
where
$\sum_{x=2}^{\infty}{f(x;\theta)}=1, 0<\alpha<1, 0<\beta<1, \theta>0$. How to generate random data, say $N=1000$?
P/s: This is not related to GLM. Just curious to generate data like this using R.
Comment: The cdf is in closed form:$$F(x)=\sum_{i=0}^x Pr(X=i)$$
Comment: Just the usual: generate a uniform $U$ and compare with the cdf.
Comment: In that case the question is too vague since you do not provide any information about $f(x;\theta)$.
Comment: Thanks, the cdf is not in closed form. So, I have to figure out another way to do it. Thank you by the way for your comment.
Comment: I understand that but when $x\geq2$, the $f(x;\theta)$ is not in the closed form.
Answer: As suggested by @Xi'an:
Start by creating a table containing:

$k=0,1,2,...,n_\text{large}$
$p_k=P(X=k)$

Then sample from your PMF with the sample function with parameters:

x = k
prob = p_k
replace = TRUE
size = N (1000 in your example)
Answer: A possible R implementation of FPO's answer, using an example distribution and parameter values:
set.seed(921078628)
f <- function(x, theta) dpois(x - 2L, theta) # example f(x;theta)
alpha <- 0.1; beta <- 0.04; theta <- 3.5 # example values

pdf <- c(alpha, beta)
currsum <- sum(pdf)
const <- 1 - currsum
i <- 2L
while(currsum < 1) {
  if (i == length(pdf)) length(pdf) <- i*2L
  pdf[i <- i + 1L] <- const*f(i, theta)
  currsum <- currsum + pdf[i]
}
pdf <- pdf[1:i]
sample(0:(length(pdf) - 1), 100, TRUE, prob = pdf)
#>   [1]  3  5  9  4  6  9  9  3  6  8  4  4  5  3  8  8  9  6  5  5  6  5  4  1  4
#>  [26]  5  6  6  5  3  4  3  4  5  4  5  5  6  5  3  9  0  7  9  0  6  7  5  3  6
#>  [51]  6 10  5  5  6  7  2  4  1  5  6  6  3  6  5  6  5  4  4  6  3  6  7  6  8
#>  [76]  6  7  6  5  5  8  6  5  0  4  2  9  9 12  4  5  0  7  3  6  4  3  8  3  4

This approach fails if $f(x;\theta)$ is long-tailed (e.g., dnbinom(x - 2L, theta, 0.01)).
Here is another approach that uses memoization to compute only as much of the CDF as needed, based on a uniform sample:
set.seed(921078628)
f <- function(x, theta) dnbinom(x - 2L, theta, 0.01) # example f(x;theta)
alpha <- 0.1; beta <- 0.04; theta <- 3.5 # example values
cdf <- c(alpha, alpha + beta)
dr <- function(n, theta, f) {
  U <- runif(n)
  maxU <- max(U)
  const <- 1 - cdf[2]
  i <- length(cdf)
  while(cdf[i] < maxU) {
    if (i == length(cdf)) length(cdf) <- i*2L
    cdf[i <- i + 1L] <- cdf[i] + const*f(i, theta)
  }
  findInterval(U, cdf <<- cdf[1:i], left.open = TRUE) # sample with memoization
}
dr(10L, theta, f) # generate 10 samples (notice the size of cdf changes)
#>  [1] 451   0 619 240 138 637 623 465 194 534
dr(100L, theta, f) # generate 100 samples (notice the size of cdf changes again)
#>   [1]  271  268    1  412  502  535  668  189    1    0  133    0  236  601  266
#>  [16]    0  114  184    1  471  217  414  244   41  226  101    0  144    0  426
#>  [31]  662  359  322  649  345  189  289    0  420  129  129  848    0    0  200
#>  [46]  314  730  273  554    0  193  161  406  115    0  171    0  226  278  167
#>  [61]  469  152  307  120  474  140  298  115    0  108  509  115    1  360  236
#>  [76]  734  624  647 1063  222    0  402  292  442  162  212  426  525  407  212
#>  [91]  334  266  437  424  321   72  238  123  235    0

Comment: I have run the first program and it seems to work greatly. my $f(x;\theta)$ is not long tailed but thanks for that too.


D-pointer/pimpl pattern base class pointer access derived class member

Question: I have misunderstanding in the concept in this link. http://wiki.qt.io/D-Pointer

In the section Inheriting d-pointers for optimization,
when creating a Label object, say
Label B;

The base constructor will be called first and then the derived constructor will be called. Since the derived constructor is in this form:
Label::Label()
: Widget(*new LabelPrivate) // initialize the d-pointer with our own
Private
{
}

This base constructor is called.
Widget::Widget(WidgetPrivate &d) : d_ptr(&d)
{
}

The base class member
WidgetPrivate *d_ptr

will have the value of
*new LabelPrivate

I know that the object B can access the protected member d_ptr which is in a type of WidgetPrivate *. My problem is, can one use this d_ptr to access the text member in LabelPrivate?
If yes, my concern is that why d_ptr in the type of WidgetPrivate* can have access to the derived class member text?
If no, does this mean Label has no access to the member text in LabelPrivate and then what is the point of using the pattern?

To avoid ambiguity, I post the original material here:
widget.h
class Widget
{
public:
    Widget();
    // ...
protected:
    // only subclasses may access the below
    // allow subclasses to initialize with their own concrete Private
    Widget(WidgetPrivate &d);
    WidgetPrivate *d_ptr;
};

widget_p.h
struct WidgetPrivate
{
    WidgetPrivate(Widget *q) : q_ptr(q) { } // constructor that initializes
the q-ptr
    Widget *q_ptr; // q-ptr that points to the API class
    Rect geometry;
    String stylesheet;
};

widget.cpp
Widget::Widget() : d_ptr(new WidgetPrivate(this))
{
}

Widget::Widget(WidgetPrivate &d) : d_ptr(&d)
{
}

label.h
class Label : public Widget
{
public:
    Label();
    // ...
protected:
    Label(LabelPrivate &d); // allow Label subclasses to pass on their         Private
    // notice how Label does not have a d_ptr! It just uses Widget's d_ptr.
};

label.cpp
#include "widget_p.h"

class LabelPrivate : public WidgetPrivate
{
public:
    String text;
};

Label::Label()
 : Widget(*new LabelPrivate) // initialize the d-pointer with our own Private
{
}

Label::Label(LabelPrivate &d) : Widget(d)
{
}

Comment: Looks like d-pointer is another name for an implementation of the PIMPL idiom. Here `LabelPrivate` does not mean that it has private members, but that it is a "private" detail of `Label`. That is what I gather anyway...
Comment: Understood. LabelPrivate is an implementation of the detail of Label. My question is why a WidgetPrivate* ptr can access to the member of LabelPrivate when LabelPrivate is a derived class of WidgetPrivate.
Comment: The definition of `LabelPrivate` and `WidgetPrivate` are not given here. I suppose `Widget` has (at least) two constructors. One takes no parameters and initializes `d_ptr` with a `new WidgetPrivate()`. The other takes a `WidgetPrivate&` parameter and initializes `d_ptr` as shown here.
Comment: To a Widget, d_ptr is only a WidgetPrivate*.  A Label would have to cast it back to a LabelPrivate* to access its contents.
Comment: So can I say that Label can access to text member of LabelPrivate via the casting of the WidgetPrivate* d_ptr to LabelPrivate*?
Comment: @RonaldKu yes, see my answer
Answer: To a Widget, d_ptr is only a WidgetPrivate*. A Label would have to cast it back to a LabelPrivate* to access its contents.
If you read one section further down on the page you cite, you'll see the full code for actually getting to the contents of LabelPrivate:
void Label::setText(const String &text)
{
    LabelPrivate *d = static_cast<LabelPrivate*>(d_ptr); // cast to our private type
    d->text = text;
}

Then the document goes on to describe some Qt macros that hide this verbosity, but the implementation is still the same.
To summarize the implications of the pattern: Both the top-level class and the PIMPL pointer are hierarchies.  A subclass of Widget (Label) sets the PIMPL pointer to a subclass of WidgetPrivate (LabelPrivate).  Now a Widget implementation can access any part of the WidgetPrivate superclass, because that's the covariant type of the stored pointer (a subclass pointer is always castable to its superclasses).  However, it knows nothing about the subclass LabelPrivate.
Once you're inside a Label implementation, you can recast the PIMPL pointer back to the subclass version (LabelPrivate) that you know is in there, and have full access to the contents.


Is my deduction about the import statements right?

Question: I'm talking about the import statements at the top.
Eg:
import java.util.Scanner;
import javax.swing.JFrame;

So, can I just say that:

Anything that starts with a lower-case letter (java, util, javax and swing here) is always a package and everything that begins with an uppercase is always a Class (which may be followed by an Inner class in some cases).
Comment: You need to further explain your question? It seems you are asking if there is a way for your coding environment to know what imports are packages and what imports are classes. However, it is not clear. If what I assume to be your question is indeed what you are asking, Eclipse has an intelligent system that auto-completes import statements if you want to complete import statements faster. However, as I have already said, your question is vauge and needs to be further explained
Comment: It's not about Eclipse or any other IDE. It's about me looking at some code and figuring out by myself (I know I don't need to, but still)>
Comment: I suggest you select the correct answer to your question.
Answer: That's not necessarily true, as you can create classes starting with lowercase letters and packages starting with uppercase letters, but it is generally good practice to follow the Java coding conventions.
These dictate that what you said should be true.
Answer: If you look up Java naming conventions on the Oracle site, you will see that package names should, by convention, start with a lower-case letter and class names, method names, interface names, and others, should, by convention, start with an upper-case letter. This allows you and anyone who reads your code later to know what statements are imports of packages (although the import keyword makes it obvious) and where you have classes and methods. It also makes your code appear neater aiding readability. You could say that it allows your one's brain to compile what it is reading faster. For example, seeing a constant value in all caps like static final int MIN_WIDTH = 4; makes it much easier to understand what is going on.
I would like to point out that the information is relatively easy to find via google and there are other questions on Stackoverflow that would have the answer to your question. Please do your research in the future.
Comment: Method names... upper case?
Comment: @pbabcdefp with method names, if there are more than one word, subsequent words begin with an uppercase letter e.g. `scanner.equalsIgnoreCase(String str);`
Answer: If the naming conventions have been followed and we are not talking about static imports, then what you say is true.
With static imports you can import constants and methods as well.
import static java.lang.Math.PI;
import static java.lang.Math.cos;


Get number of inserted rows by just plain sql

Question: is there a way to get the number of inserted rows inside of the same transaction?
I see that PL/SQL command:
SQL%ROWCOUNT

does the job, however I don't want to create a procedure just for that!
I tried to simply call
insert into T ...
select SQL%ROWCOUNT;

but it gives me "invalid character".
If I remember well mysql actually had a way to obtain this information, does oracle really not provide any means for that?
Answer:
I don't want to create a procedure just for that

No need to create any procedure, you could simply use an anonymous PL/SQL block.
For example,
SQL> SET serveroutput ON
SQL> DECLARE
  2  var_cnt NUMBER;
  3  BEGIN
  4  var_cnt :=0;
  5  FOR i IN(SELECT empno FROM emp)
  6  LOOP
  7  INSERT INTO emp(empno) VALUES(i.empno);
  8  var_cnt := var_cnt + SQL%ROWCOUNT;
  9  END loop;
 10  DBMS_OUTPUT.PUT_LINE(TO_CHAR(var_cnt)||' rows inserted');
 11  END;
 12  /
14 rows inserted

PL/SQL procedure successfully completed.

SQL>

Update If you cannot use PL/SQL, and just plain SQL, then you cannot use SQL%ROWCOUNT.
The only option that comes to my mind is to have a timestamp column in your table, and query the count based on the timestamp to know the number of rows inserted.
Comment: thumbs up, but my problem is that I will be calling those sql statements inside of an ORM and so I cannot make anything like that :/
Comment: Not the best but I guess it's the only way :(
Answer: Try following,
DBMS_OUTPUT.put_line(TO_CHAR(SQL%ROWCOUNT)||' rows inserted');

Comment: I think @Phate is meaning that `DBMS_OUTPUT.PUT_LINE` is not an SQL command and it cannot run unless it is in a PL/SQL block.
Comment: You have using oracle or sql server?


How to enable IPv6 Autoconfiguration with static address in Linux (RHEL)

Question: I have  a static IPv6 address I would like to assign.  When I add the address to the /etc/sysconfig/network-scripts/ifcfg-eth0 file and restart networking I realize I get several IPv6 addresses.  The first one is my static address (Prefix::host) and the second one is my autoconfiguration address (Prefix::MAC_ADDRESS).
I realized traffic was going out of the autoconfiguration address so my solution was to disable it.  I added "IPV6_AUTOCONF=no" to the /etc/sysconfig/network-scripts/ifcfg-eth0 file.  That solved the issue since the autoconfiguration address was gone.
However since there is no autoconfiguration I was not getting the default gateway.  I believe Solaris has something called "token based IPv6" where you can specify the host portion of the address and autoconfiguration still runs to assign the prefix, dns, and default gateway.  Can I do this in linux?
Comment: Any particular reason you don't want to configure the default gateway statically as well?
Answer: Token IPv6 addresses is a thing in Linux, man ip-token. Essentially,
ip token set ::123/64 dev eth0
There isn't really configuration supporting this in EL scripts, so I just dump it in a /sbin/ifup-local script.
Auto configuration addresses are not necessarily based on your MAC address.
    sysctl -w net.ipv6.conf.default.use_tempaddr=1
should turn on privacy extensions and also prefer your static global addresses, if you want that kind of predictability.  Save to /etc/sysctl.d/  if you are happy with it and want it permanently. See ip-sysctl.txt for other options.
Answer: You need to accept the Router Advertisement (RA) messages from the router.  That will have the prefix (network) and prefix length as well as the default gateway.
Check out the kernel config vars
net.ipv6.conf.default.accept_ra
net.ipv6.conf.all.accept_ra
and make sure they are set to (typically) 1
Also, make sure that something on your network (usually your router) is sending the RAs.
Answer: One possible answer is DHCPv6 (on the network side).  In my own experience where some type of IPv6 advertisement service (instead of DHCPv6) is available for addressing is that if I want my own "static" address, I have to statically define the rest (gateway, etc) as well.