Imputation in production

Question: I have a question regarding imputation I was not able to find an answer to. Any help would be greatly appreciated.
Let's suppose I have a dataset, impute missing values using the median, train a model and test it. The model has good performance. Now I put this model into production.
When I impute missing values for new records that keep getting fed into the model should I use the median of the original dataset or compute the median of the original dataset + the new records?
Comment: Use the median of the original dataset. How else would you apply your model in a sample of size 1?
Comment: I was also leaning towards using the median of the original dataset. Using the median of the original dataset + the one sample would also be a possibility but it does not sound right. Just wanted a second opinion.
Comment: what you will get in that case is a good sequential estimate of the median, and predictions which are difficult to interpret and apply.
Comment: A more general comment: using the median is not good practice for imputation. Better use a Bayesian multiple imputation model, see e.g. https://stats.stackexchange.com/questions/303722/methods-to-work-around-the-problem-of-missing-data-in-machine-learning/303737#303737
Comment: Unfortunately I am strongly limited by what technology I am allowed to use since things need to be productionized really quickly. Python and R are out, I can only use Alteryx at the moment which has none of the advanced imputation methods.
Comment: I do not know Alteryx, but using the median is really not a good idea. A simple (though still suboptimal) option is to use conditional mean imputation. If your outcome is continuous you could use the least square linear regression estimate as imputation, which is a simple matrix projection.
Comment: NB: you need to @ people (e.g. @tomka) as otherwise they are not notified
Answer: As AdamO suggested, use the median from the original dataset.
The intuition is this: the data you used to develop the model should be large enough and representative to production distribution. Otherwise (production data is dramatically different) the whole model building process will not make any sense. If the model building data is representative and large, why bother to add few new production data?
Let's have a concrete example. Suppose we are building a model to predict housing price and the feature is number of bed rooms. The model building data (we can divide it into training and testing to build the model), has the median of 3 bed room. This numbers should be representative to the housing market overall. (say we had 500K data points /houses to build our model)
Now assume in production time, the first day is a special day and we have luxury 10 houses feed into the model, and these houses has ~8 bed rooms. In second day, that there are 5 houses with missing value.
Intuitively what would we do? To me, it is intuitive / natural to use 3 for imputation, because this number is calculated from a large representative historical data.
Comment: Thanks for the example. That is what I believed as well but just wanted to make sure since it is the first time I have to put a model into production.

Imputation in production


What is the usefulness of detection rate in a confusion matrix?

Question: In the R caret documentation for confusionMatrix():
                        TP
sensitivity    = -----------------
                      TP + FN

                        TP
detection rate = -----------------
                 TP + FP + FN + TN

Practically, why would someone want to calculate detection rate? What is its usefulness?
Many people elsewhere think detection rate is the same as sensitivity. Why is such confusion so common?
I'm a bit confused!
Comment: With all due respect to Max, who authored the function and package, "Detection rate" is more than a little bit misleading. I would never use such terminology in reporting the validation of a diagnostic test. Merely, "probability test positive" or some other contextually appropriate description more than suffices.
Answer: One possible implication for this measure in policy and evaluation is for cost-effectiveness. Suppose, for instance, we intend to implement targeted screening for a disease in a population of healthcare subscribers receiving routine care. The current standard of care is to wait for the patient to present with symptoms of disease in clinic; however, treatment at this point tends to have a low success rate, many complications, and high cost. The rationale for screening is to deliver more timely, less invasive, and more successful treatment at a comparable cost.
If a patient is identified as a true positive, not only must they undergo treatment, but their longer survival requires them to be on treatment for a longer duration, and individuals who would have died from other disease before becoming symptomatic undergo treatment unnecessarily for the disease. The cost of the screening program must incorporate the costs of treating those identified with the disease.
Comment: @Underminer indeed. There are four aspects to evaluating a screening program: the sensitivity and cost of the screen, the cost of the confirmation, the cost of treatment in the screen detected persons, the reduced cost of treatment in symptomatic persons (hopefully fewer go on to develop symptoms and thus be identified as diseased as a result of screening if lessons from cancer screening studies are to teach us anything.)
Comment: @AdamO I suppose this makes more sense in the context of a confirmatory secondary test. In effect, though, *this artificially sets the FPR of the original test to 0*. In the breast cancer example, you assume that the biopsy test has 0 FPs so that no healthy people undergo surgery/chemo/etc. Without that perfect secondary test, some healthy people will be treated at exactly the same cost as treating someone with cancer. My point is that you will never know who is TP and who is FP unless you have that second test, and at that point we're no longer discussing metrics of the original test.
Comment: I thought of a similar example at first but realized it doesn't quite fit. The problem is, the cost of this screening program will be determined by the total number of patients selected by the screen for further treatment, which includes both true positives and false positives. In this case, TP+FP/N is a better measure of cost than TP/N (where N is the total number). Without another test, TPs and FPs are totally indistinguishable in practice. If we want to know how cost effective the test is, the positive predictive value appropriately measures how often the test is right for positive cases.
Comment: @AdamO Clever example.  I would add that in order to be identified as a true positive (vs false positive) a secondary test (that is more specific than the first test and less costly than the full treatment, perhaps) would have to be administered.
Comment: @NuclearWang often times the considerations for confirmatory tests is not the cost, but rather that it is invasive. In breast cancer for instance, a screening mammogram can identify abnormal masses, but the biopsy confirms it is cancer. The cost of the biopsy among TPs and FPs may be negligible. The surgical resection, years of chemotherapy, and surveillance that takes places among the TPs incurs the most cost.
Comment: @NuclearWang I think you are still missing my major point. This no longer has anything to do with tests: it's the "so what" after the test is done. If a woman with a positive screen actually receives some form of treatment, she is considered a "TP" in this case. (A sad truth is that many women receive treatment for non-aggressive, non harmful benign growths or ductal carcinoma in-situs). It takes some mulling over to come to grips with what a "true positive" actually reflects in life and in practice.
Comment: @AdamO And my point is that a woman with a positive screen that receives treatment is *not* necessarily a TP. As you point out, many women receive treatment for growths that do not require it - those women are FPs! The total cost of the screening program must account for everyone who was treated, whether they actually needed it or not. In clinical practice, TPs and FPs are indistinguishable - if we knew otherwise we wouldn't get it wrong - but that doesn't change the fact that some people are treated unnecessarily. Detection rate ignores the cost of unnecessary treatment.
Comment: If we are eliminating any False Positives through a confirmatory test, then the "detection rate" of the initial test is equivalent to the "predicted positive condition rate" of the combined test since we have no remaining False Positives.
Comment: @NuclearWang But you will never know that. In the absence of gold-standard you accept the outcomes as the truth.
Comment: @AdamO I agree completely with the practical aspect of TPs and FPs appearing identical in the clinic. If that's the case, though, the formula is better characterized as P/N, where P is the number of predicted positives and N is the total. The formula TP/(TP+FP+TN+FN) is a bit misleading, since you're setting TP=P, FP=0 and FN=0 by assuming the test is always correct. If we assume a 100% accurate test, TP/FP/TN/FN aren't particularly relevant concepts to begin with.
Answer: It is a parameter that will vary according to the dataset. If you get a high overall performance on a moderate detection rate, it would make me wonder how the performance would vary on a smaller dataset with more positives.


Element goes way off screen

Question: I have a fixed div which I made draggable through JQuery UI. The div is centered on startup.
The problem is, when the div gets dragged off screen, (meaning, because it goes off screen, the page starts scrolling,) the div goes way over to the side. The more you drag, the more it goes away from the cursor.
Also, how can I position the div with percentage, this way when the browser gets resized, the div will maintain its position? (Even after it gets dragged.)
JSFiddle

var dragDiv = $('#draggable');

dragDiv.css({
  'top': ($(window).height() / 2) - (dragDiv.outerHeight() / 2),
  'left': ($(window).width() / 2) - (dragDiv.outerWidth() / 2)
});

dragDiv.draggable();
body {
  width: 2000px;
  height: 2000px;
  background-image: url("http://www.freevector.com/site_media/preview_images/FreeVector-Square-Patterns-Set.jpg");
}
#draggable {
  color: lightblue;
  background-color: red;
  width: 200px;
  position: fixed;
}

Drag Me!

Comment: Try `position:absolute;` , and what do you mean by the div will maintain its position ?
Comment: But I need `position: fixed` because I want the `div` to stay in its position when you scroll.
Comment: What I mean by maintain its position, is I want it to stay on screen even if you resize the browser.
Comment: Sorry about that... I didn't fully grok what you were trying to do at first, but I took another stab at it.
Answer: A simple solution would be to use a fixed position overlay/wrapper:
Working Example

var dragDiv = $('#draggable');

dragDiv.css({
  'top': ($(window).height() / 2) - (dragDiv.outerHeight() / 2),
  'left': ($(window).width() / 2) - (dragDiv.outerWidth() / 2)
});

dragDiv.draggable({
  containment: "parent" // <- keep draggable within fixed overlay
});
body {
  width: 2000px;
  height: 2000px;
  background-image: url("http://www.freevector.com/site_media/preview_images/FreeVector-Square-Patterns-Set.jpg");
}
#fixed {
  position: fixed;
  top: 0;
  bottom: 0;
  left: 0;
  right: 0;
  background: rgba(0, 0, 0, 0.25)
}
#draggable {
  color: lightblue;
  background-color: red;
  width: 200px;
  position: absolute;
}

fixed overlay

Drag Me!

As for setting the position in percent, you can check this answer.
Comment: Thanks Really like the answer, and the link! The only problem is, onstartup, before the `div` actually drags, it isn't in percentage values. How can I make `draggable` into a percentage value on startup?
Comment: Also, is it important to have `position: absolute` for `draggable`?
Comment: @Jessica as far as I can tell the only way to force jQuery UI's Draggable to use percent rather than pixels would be to edit the jQuery UI core. I would strongly advise against that, or if you decide to do it, document it thoroughly, so the next developer knows that they're not working with the default functionality.
Comment: @Jessica It looks like relative and absolute positioning will work, it just bugs out on fixed positioning.

Element goes way off screen


how to add new font to Xcode 3.2?

Question: please tell me how to add new font to iphone xcode3.2.
Comment: Updated my answer with UIAppFonts info!
Answer: This functionality isn't built into UILabel/UIFont in iOS < 3.2. But there is a third-party solution you can use: FontLabel
In iOS 3.2+, there's the UIAppFonts plist key!
Just do this before using it: add your font file (such as myfont.ttf) to your project, then edit the Info.plist like so:
Comment: i use this approach but when i load font using below code

UIFont *myF = [UIFont fontWithName:@"my font name" size:12];

myF is return null;
Comment: but i want to use UIAppFonts you have any idea how to use way
Comment: I would think that the name is wrong or the file isn't getting included in the project. There's a good post about this here: http://www.perfectline.co.uk/blog/uiappfonts-custom-fonts-with-the-iphoneipad-3-2-sdk
Comment: Hello! For me it doesn't work it crashes with an EXC_BAD_ACCESS when i want to retrieve the fontfamilynames. I already posted a questions [here](http://stackoverflow.com/questions/8727710/adding-a-uifont-and-looking-for-the-fontfamily-crashes-with-exc-bad-access). Could you tell me, what i'm doing wrong? Thx
Answer: The above methods worked for me in xcode 3.2!  Thanks!:
step 1: drag your font into your project (prob a good idea to not have spaces in the font name)
step 2: add a row to your info.plist file like shown above to enter your font.
step 3: in, viewDidLoad() or wherever you are setting your label, enter code:
 UIFont *myFont = [UIFont fontWithName:@"secondbreakfast" size:40];
 myLabel.font = myFont;

Answer: I did it with the following way:

Add the font type to the Xcode : Under the XIB, go the font setting -> manage font
Add the font to the label:
UIFont *myFont = [UIFont fontWithName:@"myfont" size:12];
label.font = myFont;

Hope it help
Answer: But in xcode 5

Go to xcode preferences -> Fonts and colors.
Then click on T button in the textField of Font.

A list of all colors open then click on setting button in bottom left corner and go to manage fonts
Here select All fonts and then add your own ttf file by clicking on plus button
and your Font will be added.

how to add new font to Xcode 3.2?


Fast way to compute Fréchet/Gateaux Derivatives

Question: I was wondering if there is any fast way to compute Fréchet/Gateaux Derivatives, or at least a reasonable guess in most cases, say from the usual derivatives table.
The Fréchet derivatives $\dfrac{\delta F}{\delta x}$of a functional $F: (X,||\cdot||)\mapsto \mathbb{R}$ is defined, if exists, as the functional $\dfrac{\delta F}{\delta x}: (X,||\cdot||)\mapsto \mathbb{R}$ such that
$$\lim_{||h||\to 0}\dfrac{|F(x+h)-F(x)-\frac{\delta F}{\delta x}(h)|}{||h||}=0.$$
The Gateaux derivatives $\mathrm{d}F(x;h)$ of a functional $F: (X,||\cdot||)\mapsto \mathbb{R}$ in the direction $h\in X$ is defined, if exists,
$$dF(x;h) = \lim_{\varepsilon \to 0} \dfrac{F(x+\varepsilon h)-F(x)}{\varepsilon}.$$
Comment: I am guessing $\mu$ is a measure? The more usual setting is to deal with densities $d\mu=f(x)dx$, and what you wrote are integrals linear and quadratic in densities, so of course it is directly related to elementary derivatives. More generally, $\int L(x,y; f(x),f(y);f'(x),f'(y))\,dxdy$ and the like, where $L$ is some elementary function in $f$ and $f'$, are handled by the calculus of variations. But $\sup/\inf$ functionals are not of this sort.
Comment: This question is way too broad. Do you have specific examples of functionals in mind?
Comment: We should have this $\frac{\delta F}{\delta x}(x)$. And as @daw said i found it very brand to.
Comment: There is if the functionals you have are integral functionals with elementary functions under the integral. That's what the [classical calculus of variations](https://en.wikipedia.org/wiki/Calculus_of_variations#Euler%E2%80%93Lagrange_equation) is about. But speaking generally, why would there be any relation between some abstract functionals and the usual table of derivatives for elementary functions?
Comment: @Conifold for example, the functional $\mathcal{V}: \mu \mapsto \int V\mathrm{d}\mu$ has the first variation as $\dfrac{\delta \mathcal{V}}{\delta \mu}=V$ in the sense of $\dfrac{\delta \mathcal{V}}{\delta \mu}(\nu)= \langle V,\nu\rangle$, provided with a suitable space for $\mu$ to live in. This reminders taking a derivative of a linear function. For the functional $\mathcal{W}:\mu \mapsto \int W(x,y)\mathrm{d}\mu(x)\mathrm{d}\mu(y)$ has a Frechet derivative, which looks like a product rule in the usual derivatives, but I cannot convince myself.
Comment: @daw I replied to Conifold with two known examples, while I am interested into some functional defined in a form of "sup/inf", and look for the relationship between its Frechet/Gateaux derivatives with the usual derivative, so that to compute Frechet/Gateaux derivatives in a formal but correct way.
Answer: There is a way to quickly calculate Gateaux derivatives. It can also be used to calculate Frechet derivatives, if you are sure that a functional is Frechet differentiable, because in this case the Gateaux derivative also exists and coincides with the Frechet derivative.
Note that
$$F(f+th): \mathbb{R}\to\mathbb{R},\ \text{fixed}\ f\in X,\ h\ \text{admissible}, t\in\mathbb{R}$$
$$\frac{dF(f+th)}{dt}\bigg|_{t=0}=\lim_{t\to0}\frac{F(f+th)-F(f)}{t}=dF(f;h)$$
Example: take the functional
$$\int_0^1||f'(x)||^2dx,\ \text{restrict to}\ f(0)=a,\ f(1)=b$$
This is a convex functional over a convex set. For differentiable $h(x)\ \text{s.t.}\ h(0)=0,h(1)=0$:
$$F(f+th)=\int_0^1||f'(x)+th'(x)||^2dx=\int_0^1||f'(x)||^2+2tf'(x)\cdot h'(x)+t^2||h'(x)||^2dx$$
$$dF(f;h)=\frac{dF(f+th)}{dt}\bigg|_{t=0}=\int_0^12f'(x)\cdot h'(x)dx=\int_0^12f''(x)h(x)dx$$
Note that $dF(f;h)=0$ when $f$ is the straight line from $a$ to $b$, this is the minimum of the functional.


opencv warpPerspective parameter count

Question: In my script I have the following code:
src = numpy.array(cornersSheet, numpy.float32)
dst = numpy.array(cornersDesired, numpy.float32)
transform = cv2.getPerspectiveTransform(src,dst)
finished = cv2.warpPerspective(img, transform, img.shape)

Python says:
Traceback (most recent call last):
File "./script.py", line 138, in <module>
    finished = cv2.warpPerspective(img, transform, img.shape)
TypeError: function takes exactly 2 arguments (3 given)

but according to documentation:
    Python: cv2.warpPerspective(src, M, dsize[, dst[, flags[, borderMode[, borderValue]]]]) → dst

three parameters are OK. I have the same issue with cv2.warpAffine.
Answer: Problem solved. img.shape returns tuple with 3 elements, warpPerspective expects tuple with 2.
Answer: Try this
finished = cv2.warpPerspective(img, transform, img.shape[1::-1])

Comment: @tomfriwel In Python list indexing is [begin:end:step], so [1::-1] means from the second element (indexes start at 0) to end, reversed. See https://stackoverflow.com/questions/509211/understanding-pythons-slice-notation
Comment: What is `[1::-1]`mean?

opencv warpPerspective parameter count


Inject returns NULL when activating jqwik in Quarkus test

Question: We want to introduce property-based testing into our Quarkus project, preferably with jqwik. We already got numerous test cases using junit jupiter. We also use CDI in out test cases.
Getting jqwik running in a small Quarkus example project went well, so I wanted to write some properties in out big project. jqwik is running, however, the @Property, @Example and @Provider methods do not have access to the injected bean (as in: the injected bean is Null, TreeRepository in the example below). Same for ArbitrarySupplier subclasses. If I replace the @Example with a @Test the referring test can access the bean and the test passes.
My first guess is that this has something to do with the jqwik lifecycle. I did not find enough information about how (and if?) jqwik integrates with injection.
How do I get this running?
In the example I expect treeRepository to be an instance of TreeRepository (the class is @ApplicationScoped). Instead it is null,  except for in the method with the @Test annotation.
@QuarkusTest
class MyTestClass {

  @Inject
  TreeRepository treeRepository;

  @Test
  void testSimple() {
    final Collection<Tree> trees = this.treeRepository.getTrees() // works
    assertThat(trees).isNotEmpty();
  }

  @Example
  void testSimple() {
    final Collection<Tree> trees = this.treeRepository.getTrees() // does not work
    assertThat(trees).isNotEmpty();
  }

  @Property
  void treesHaveLeaves(@Forall("tree") Tree tree) { // does not work
    assertThat(tree.getLeaves()).isNotEmpty();
  }

  Arbitrary<Tree> tree() {
    final Collection<Tree> trees = this.treeRepository.getTrees(); // does not work
    return Arbitraries.of(trees);
  }
}

Answer: There is bad news and maybe a bit of good news.
The bad news: @QuarkusTest is specially made for use with JUnit Jupiter. Jupiter and jqwik are two different test engines for the JUnit Platform. The lifecycles of different engines do not compose, that's why treeRepository will not be injected automatically when you're executing jqwik properties or examples.
The good news is that it's probably not too difficult to create something like @JqwikQuarkusTest for your integration of jqwik and Quarkus.
If you want to do that yourself you can use jqwik's Spring support as an inspiration or template: https://github.com/jlink/jqwik-spring. Or you read up about jqwik's hooks here: https://jqwik.net/docs/current/user-guide.html#lifecycle-hooks and start from scratch.
If you don't feel up to the task yourself, you might want to think about paying someone to do it for you or your company. I'd be happy to help out.
Comment: Hi johanneslink, thanks for the quick answer. Thats really a pity that this is not supported yet. But I will have a look and come back here to with my conclusions.
Comment: You could also ask the Quarkus folks to support jqwik. I'd be happy to give them some hints.
Comment: Thank you for the support. I have tried writing the extension myself. I failed but not because of jqwik but for a lack of understanding the Quarkus test engine. I hope the Quarkus community can help. I have posted in the Quarkus community and here on Stackoverflow (https://stackoverflow.com/questions/74812675/property-based-testing-support-for-quarkus-tech-doku-about-the-quarkus-test-en).
If I manage to produce a jqwik extension I will get in contact about a potential publication.


Integral of Derivative squared

Question: For a function $f$ I know that: $$\int{f'(r)dr}=f(r)$$ where $f(r)$ is known. knowing the result of this integral how can i calculate $$\int{(f'(r))^2dr}$$ Is there any relation between these integrals?
Comment: If you know the second derivative you can solve it by integrating by parts.
Comment: No, there is no general formula involving only $f$, $f'$ and $r$ for this integral. It might be a nice exercise to try to prove this.
Comment: I don't think there's any meaningful relation. Just think of $\;f(x)=\log x\;,\;\;f'(x)=\frac1x\;$ ...
Answer: While there is no explicit formula of the exact kind you desire, if one is willing to reparametrize, there is an integral-free parametrization of the curves of the form $\bigl(x,f(x),g(x)\bigr)$ for which $g'(x) = (f'(x))^2$: As long as $f''(x)$ is non-vanishing, one can reparametrize such a curve in the form
$$
\bigl(x,f(x),g(x)\bigr)
= \bigl(u''(t),\ t\,u''(t){-}u'(t),\ t^2u''(t){-}2tu'(t){+}2u(t)\,\bigr)
$$
where $u$ is a function of $t = f'(x)$.  Conversely, if $u$ is an arbitrary function of $t$ for which $u'''(t)$ is nonvanishing, the curve on the right hand side of the equation is always of the form $\bigl(x,f(x),g(x)\bigr)$ for some functions $f$ and $g$ satisfying $g'(x) = (f'(x))^2$.
Perhaps this is the sort of thing that one can use when one is trying to get an integral-free description of the relationship between the two functions $f$ and $g$ (and $x$, of course).
Comment: @ccbreen:  Yes, there is a systematic way to search for such 'integral-free' formulae for curves.  This is known in the control theory literature as 'differential flatness'.  The test for such in general is too involved to write out here, but the idea is that you want to find coordinates in which the differential system $$ df - p\,dx = dg - p^2\,dx = 0$$ can be written in "Engel normal form", i.e. $$ du - v\,dt = dv - w\,dt = 0,$$ whose integral curves can be written in the form $(t,u,v,w) = \bigl(t,h(t),h'(t),h''(t)\bigr)$.  Engel's theorem ensures that you can do this.
Comment: Hi, I don't have the math background to understand what to make of the curve you've described. Do you have any suggestions as to good background material? This kind of seems like differential geometry, is that right?
Comment: @JohnMadden:  Yes, it's from differential geometry, but you don't need any differential geometry to see that with $\bigl(x,f(x),g(x)\bigr)$ as given, we have $$\frac{\mathrm{d}g}{\mathrm{d}x}
= t^2 = \left(\frac{\mathrm{d}f}{\mathrm{d}x} \right)^2,$$ just the Chain Rule.
Comment: Thanks for this helpful formula! Where did this come from, and do you know of a systematic way of generating these sorts of formulas? It's not completely clear to me how this was found...
Comment: @RobertBryant, could you possibly comment on this: https://math.stackexchange.com/questions/4771881/weak-version-of-maximum-principle-for-not-quite-subharmonic-functions ?
Answer: Let $f(x)=e^{x^2}$, so $(f'(x))^2=4x^2e^{2x^2}$. But $\int4x^2e^{2x^2}\,dx$ can't be evaluated in terms of elementary functions (exponentials, trig functions, polynomials, $n$th roots, etc.).

Integral of Derivative squared


If $\tan\theta+\sin\theta=x$ and $\tan\theta-\sin\theta=y$, then $(x^2-y^2)^2=16 xy$

Question: I have tried to prove this trig identity but I seem to be heading nowhere…

If $\tan\theta+\sin\theta=x$ and $\tan\theta-\sin\theta=y$, prove that $(x^2-y^2)^2=16 xy$.
Comment: Welcome to MSE. It is in your best interest that you type your questions (using [MathJax](https://math.meta.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference)) instead of posting links to pictures.
Answer: Let $x = \tan(\theta) + \sin(\theta)$ and $y = \tan(\theta) - \sin(\theta)$.
R.T.P: $(x^2 - y^2)^2 = 16xy$
Proof: LHS = $(x^2-y^2)^2 = ((x-y)(x+y))^2 = (x-y)^2(x+y)^2 = (2 \sin(\theta))^2(2 \tan(\theta))^2 = 16 \sin^{2}(\theta) \tan^{2}(\theta)$.
Note that $xy = \tan^{2}(\theta) - \sin^{2}(\theta) = \frac{\sin^{2}(\theta)[1 -\cos^{2}(\theta)]}{\cos^{2}(\theta)} = \tan^{2}(\theta) \sin^{2}(\theta).$
Hence we have that $(x^2 - y^2)^2 = 16xy$.
Answer: Hint:
Let us start without knowing the proposition to be established.
Solve for $\tan\theta,\sin\theta$
Now use $$\tan^2\theta=\dfrac{\sin^2\theta}{1-\sin^2\theta}$$
and simplify.


What type of exercises should a guitar beginner do?

Question: I'm learning classical guitar theory from internet, i get free random courses video on the net.
I've tried to play some tabs that i found them on youtube, most tabs I've seen are very hard to play..
I just learned study 2 and 3 of Matteo Carcassi from a pdf.
I'm a bit frustrated, i didn't found a full guided courses that allow get faster improvement..
In short, assume that i'm  a very guitar beginner, what things should i learn in order:

Learning chords
Chord progression
Arpeggios
Learning Scales
Playing solo music

In addition of the previous list, do i need to master music sheet reading ?
Cheers.
Comment: Music is also about rhythm.
I don't see anything about the latter in your list..I would add some.
Comment: What types of rhythm exercises would you recommend?
Comment: Well...I am looking them myself. Now I am using the plan and content of http://rhythmmasters.org/.  And trying to implement those rhythms with my strumming or hybrid picking techniques.
Comment: Please leave thanks out of questions.  An appropriate comment on the relevant answers would be OK, and better yet make sure to upvote :)
Answer: I think you've got a good list already, but I'd quibble with the in order part. I'd recommend splitting up your practice time to work on each of these fronts every day. So for a 1-hour practice, you might do

Practice yesterday's scale a few times, pick a new scale and practice it. ~ 10 min.
Put on your new favorite pop song and try to transcribe the chord progression (starting with the bass-line). ~ 10 min.
Play around with those chords and write down a few riffs. ~ 10 min.
Guitar Etudes. Carcassi, Sor, Aguado, Tarrega: all these guys have excellent collections of etudes which work your sight-reading, technique, and are very pretty. ~ 30 min.

For me personally, learning is most effective when it touches upon multiple interests. So to learn something about chord progressions, I start by trying to find a chord progression that I like. Then I'm much more motivated to do the hard work involved. That's also why I recommend a generous portion of just play in a practice. It should be fun.
And yes, I think reading the sheet music is essential. It opens up the whole catalog on imslp.org. But music that is edited for guitar has many helpful annotations in addition to the dots and beams. There should be the letters p, i, m, a (and rarely e) to indicate which right-hand finger to use to play the note. And there will be the numbers 1, 2, 3, 4 to indicate which left-hand finger to use to fret the note. There will also be roman numerals III, IV, V, VII, to indicate where on the fret-board your left-hand index finger should be, and usually the letter C to indicate that the index finger needs to barre two or more strings at that fret. And where it's really necessary, there may be circled numbers which indicate exactly which string the note should be played on.
As you become familiar with the chord shapes on the fretboard and chord shapes in the notation, reading well-edited guitar music can be done quite easily. Actually playing the piece may be more or less challenging, though.
Edit: Keep going with the Carcassi, too. I think it's number 4 nope: number 16 that's my favorite. It's a 2-voiced pas-de-deux in F that really works your partial barres. And Carcassi's etude number 1 basically covers everything you need to know about the C major scale and arpeggios.
Edit: The subject of rhythm has come up in the comments. For scales, you want to play in a nice even time. Perhaps practice with a metronome a few times a week to check that you're playing evenly. But learning to use your ear requires some degree of learning to trust your ear. So I would not recommend always using the metronome. It can become a crutch. Classical music should have a clearly defined sense of time. But it should not necessarily be strictly rigid to the point of, well, being unmusical. (It's difficult to precisely quantify that last point. :))
The Etudes mentioned above will exercise your understanding and playing of rhythms as well as notes and technique.
Edit: You also mention wanting something like a "full guided course". What you're looking for, then is a method-book. There are many out there. My favorite, by Celedonio Romero, is out of print. I would like to recommend the Dover edition of Fernando Sor's technique, but I think the antiquated translation would be too formidable for a beginner. On the other hand, it does give you some commentary on his exercises and etudes, which you don't have without it.
For pure technique work, you should get a copy of Steve Vai's "10-hour workout" from Guitar World back in the 90s. It's priceless. He gives you a massive amount of material to practice on. Lots of copies available through google search. If you can find a scan from the magazine (or better still, purchase a back issue), it'll be much more readable than the ascii-tab versions. While the 10-hour workout is mostly focused on electric guitar playing, it can easily be adapted for classical. Instead of down-up alternate picking, alternate between two (or more) fingers: p-i-p-i- *i-m-i-m-* p-a-p-a- *p-i-m-a-* as appropriate. Instead of bends, do glissandos: slide gracefully into the upper (lower) note.
Answer: Well, all of those are useful skills to have at some point, but through the lens of 'Never picked up a guitar before', I would tell you first to learn the basic motor skills needed, being picking/plucking, fretting and how to control your hands in a synchronous manner.
After that, get comfortable with the fretboard. In any playing, comfort is key, but that is taken to an extreme in classical playing. Personally, I would do things in this order:

learn the scales
get extremely comfortable with them
learn chords and progressions (The two usually go hand-in-hand).
learn to play arpeggios

By the time you get comfortable with those skills, you should be naturally coming up with solo material as you explore new ideas and sounds.
If you plan to play classical guitar, I would say the ability to read sheet music is pretty crucial. Tabs lack clarity in areas where sheet music doesn't, and classical can be very intricate, which may be lost in a tab. Its a useful skill to have either way if you intend to play ANY music, so I definitely would learn it if I were in your shoes.
Answer: Invest into good beginner manual or teaching computer program, then let your manual to decide that to do next. With such scenario, most important decision you still need to make yourself is to determine if this book or program is competently prepared for the self learning. Maybe some ratings and, well, the price may help.
Answer: This is my first post on this site, please bear with me if I'm doing something wrong.
Here's another near-absolute beginner speaking, so take what I'm saying with a bucket of salt. I will add to what others say, trying not to repeat them.
These are three things you can do to improve your playing.

Get a teacher.
Get a teacher.
Get a teacher.

I cannot overemphasize it, really. I'm speaking from experience here. I've tried to do it on my own, and failed miserably. (I did manage to learn chords and accompany easy songs, but that's a far cry from playing classical guitar). With a teacher I'm making progress. You can find the same advice everywhere on the internet, site after site, forum after forum.
If you absolutely positively cannot get a teacher, get a method book. But get a modern method book! I hear modern books are good. I cannot vouch for any one personally. Back when I was trying to study on my own, I only had access to the old 19th and early 20th century ones like Pujol and Carcassi. They tend to be frustrating for novices (they were for me!) Avoid them. They can be good if you are a bit more experienced.
Follow the the book, study methodically, and go s-l-o-w-l-y. I mean, when you learn a piece, deliberately pick a tempo which is much much slower than required. Practice at that speed until you are very comfortable with it, then accelerate gradually. Learn to move your fingers precisely, rather than fast. You can always pick up speed later. If you go too fast from the start, you tend to make mistakes and learn them, and they become bad habits. You will have unlearn them at some point or another, which is painful.
Try to talk to more experienced players at every opportunity. Beg, bribe, bully or blackmail them to check your technique and correct your mistakes.
Some teachers offer lessons over video chat, consider taking those at least from time to time.
Watch guitar videos on the intertubes. Pick videos where hands are clearly visible. Ideally you want to watch as good players as possible playing as easy pieces as possible, so that you can analyse and emulate their technique.
Try to exercise for at least 3.5 hours a week, 7 is better. If you are serious enough to invest more time than that, definitely do get a teacher!
Comment: @SmartyTwiti Whatever works for you!
Comment: Thank you for pointing out about getting a teacher, but i believe in self learning. I'm getting remarkable improvement since this topic!
Answer: IMHO The longer you do metronome work the better. It does not have to be scales at first but just learning to play to a beat from the beginning will do wonders. It cannot be emphasised that the longer you stay at the low tempos the greater your progress will be.
Teachers need to emphasise that patience is key. If you do sloppy work while practicing your progress will be slow. You will shoot yourself in the foot.
A good theory teacher will teach you most of the things you mention. Get one he or she will teach you the nuts and bolts of music!
Answer: Turn off the lights. Turn on some soft background music in a rhythm you enjoy. Apply fingertips to neck of guitar... and play. Do this once a day for 15min (or more). Rinse & repeat.
This is to engage your creativity -- not necessarily improve your actual skill, but I appreciate this as being as integral to playing any instrument as your technical ability.


How many integer solutions there are for this equation $\sum_{k=1}^r |x_k| =n$?

Question: Using a proper generating function I need to evaluate the number of integer solutions to this equation :
$\sum_{k=1}^r |x_k| =n$
Can I assume that
$\sum_{k=1}^r x_k =n$
and
$\sum_{k=1}^r -x_k =n$
has the same amount of integer solutions and that the answer is actually $2*p$
for p equals the number of integer solutions to the following equation $\sum_{k=1}^r x_k =n$ ?
Comment: This is the sphere of radius $n$ in the $\ell^1$ norm of $\mathbb R^r$; its uncountable. Or did you mean integer solutions?
Comment: Yeah I meant integer solutions , i'll edit the post
Comment: Hi, my answer is wrong, can you unaccept it?
Comment: Done, although it is wrong it helped me realize a correct solution.
will share it if anyone is intrested. @CalvinKhor
Answer: For $r=1$ the generating function for the number of solutions is
$$1+2x+2x^2+\dots=\frac{1+x}{1-x}$$
as every value of $n\ne0$ has two possible solutions namely $x_1=\pm n$ with the same absolute values and $n=0$ has only one solution namely $x_1=0$. Then, for $r\ge1$, it follows that the generating function for the number of solutions is
$$(1+2x+2x^2+\dots)^r=\left(\frac{1+x}{1-x}\right)^r$$
as each tuple of absolute values $(|x_1|,\dots,|x_r|)$ which produces a valid solution corresponds to a number of tuples of $r$ integers $(x_1,\dots,x_r)$ given by multiplying the coefficients of the monomials in each bracket with the powers $|x_1|,\dots,|x_r|$ respectively.
Answer: If $r=2$, then $(0,n)$ for natural numbers becomes two solutions $(0,\pm n)$, but $(m,n-m)$ becomes four solutions $(\pm m,\pm(n-m))$
Comment: I need to solve for every $r$ not for a specific one
Comment: Just saying a factor of $2^r$, except when some $x_k$  are zero.


expect script + write "if"  in expect script

Question: please advice what's wrong with my exepct syntax ( I run expect in ksh script on Solaris 10 machine )
I want to verify if OS is linux
so if it true the expect check if prompt is "#" or ">"
and then run the "ls" command
but I get the error below
please advice what could be the problem
<code>     OS=solaris

     expect_do_something=`cat << EOF
     set timeout -1
     spawn   telnet 0 $IP_ADDRESS
     expect login:           {send $LOGIN\r}
     expect Password:        {send "$PASS\r"}

 if { "$OS" == "solaris" }
    {
      expect -re {#|>}        {send ls\r}
    }

 expect eof
 EOF`

 expect -c  "$expect_do_something"
</code>
the error from expect:
<code>  wrong # args: no script following " "solaris" == "solaris" " argument
     while executing
  "if { "solaris" == "solaris" }"
</code>
Answer: Expect, being an extension of Tcl, has to respect Tcl's very few syntax rules. In Tcl, a newline signifies the end of the command (see this rule), so you must put the open brace of the if-body on the same line.
<code>if { "$OS" == "solaris" } {
  # ......................^
  expect -re {#|>} {send ls\r}
}
</code>


How sampling distribution of sample proportion is related to binomial distribution

Question: I am reading a book on introductory statistics. In the book, the author introduces the concept of the "sampling distribution of sample proportion" just after explaining the binomial distribution.
I think I've understood the concept of "sampling distribution" and how to take one.
But I have trouble grasping how "sampling distribution of sample proportion" is related to the binomial distribution.
I am also confused with the terminology used in these contexts. For example, when I studied the binomial distribution, $p$ stands for the probability of success on each trail. In the context of "sampling distribution of sample proportion", $p$ represents the population proportion.
Are these two same?
I am feeling completely lost at this point.
Comment: The sample proportion is $\hat{p}=X/n$ where $X$="the number of successes in the n trials" and $n$ is the number of trials. If the outcome of each trial is $B_i$, which takes the value $1$ for a success and $0$ otherwise then $X= B_1+B_2+...+B_n$. If the outcomes -- the $B$'s -- are independent  and the population $p$ is the same for all of them (independent, identically distributed, or iid), then $X$ is [binomial](https://en.wikipedia.org/wiki/Binomial_distribution) and $X/n$ (the sample proportion) is simply a constant multiple of that.
Comment: under iid sampling, n times the sample proportion would be a binomial count.
Comment: @Glen_b-ReinstateMonica, I am sorry but your words don't make sense to me. Can you please elaborate a little?
Comment: [4.2.1](https://online.stat.psu.edu/stat800/book/export/html/665) section "Normal Approximation to the Binomial " of  Book "STAT 800: Introduction to Applied Statistics" can give more insight.
Answer: The terminology may be the key here: The two $p$ you cite are in fact the same (if we regard the population as infinitely large). In this context, one trial is drawing one element from the population and recording whether it has the property of interest or not. So the probability of success $p$ (the probability of said element having said property) is the relative frequency of the property in the population.
Let's write $X$ for the total number of successes, i.e. how many elements we draw overall that do have the property of interest. We assume the population size is infinite, which is practically justified if it is much larger than the sample size $n$. Then there is no difference between drawing with or without replacement, and the probability of success always stays the same $p$, regardless of what we have already drawn so far.
So we have a run of $n$ independent trials, each with a probability of success $p$, and we are interested in the total number of successes $X$. By definition, this means that $X$ has a binomial distribution with parameters $n$ and $p$.
Now the sample proportion is $X/n$, so it differs from $X$ only by the constant (non-random) scaling factor $1/n$, and therefore the shape of its distribution is the same as the distribution of $X$, i.e. a binomial distribution.
Comment: can you shed some more light on how proportion and binomial distribution are related and why binomial distribution is considered as the population?
Comment: I expanded my answer.
Comment: It's sinking in now, thanks for the clerfication.


ignoreInit not working with dynamic content

Question: This is a cross post from github to get more traction.
I recognized that observeEvent is triggered upon start when the UI is created dynamically, even with ignoreInit=TRUE. I found this thread on stackoverflow, but the presented solution does not work for checkboxInput because the value of the eventExpr is either T/F, so it's not possible to decide if it's the first call during initialization or a regular call.
In the minimal working example below I found a workaround by using shinyjs::delay, but I am not sure if this would always work and what minimum delay time is required.
To me, ignoreInit should also work for dynamic UIs, i.e. the observer for dynamicBox should not be triggered upon initialization as is the case for staticBox. Is there any chance to fix that issue or is this expected behavior?
library(shinydashboard)
library(shiny)
library(shinyjs)

ui <- dashboardPage(
  dashboardHeader(title = "observeEvent ignoreInit"),
  dashboardSidebar(),
  dashboardBody(
    useShinyjs(),
    checkboxInput("staticBox", "static"),
    uiOutput("body"))
)

server <- function(input, output, session) {

  output$body <- renderUI( tagList(
    checkboxInput("dynamicBox", "dynamic"),
    checkboxInput("dynamicBoxDelayed", "dynamic delayed")
  ))

  # observe static checkBox
  observeEvent(input$staticBox, {
    cat("observeEvent of static checkBox is executed\n")
  }, ignoreInit=TRUE)

  # observe dynamic checkBox
  observeEvent(input$dynamicBox, {
    # this is executed upon start, although ignoreInit is set to TRUE
    cat("observeEvent of dynamic checkBox is executed\n")
    cat( paste0("dynamicBox value = '", input$dynamicBox, "'\n") )
  }, ignoreInit=TRUE)

  # observe dynamicDelayed checkBox
  shinyjs::delay(100, {
    observeEvent(input$dynamicBoxDelayed, {
      cat("observeEvent of dynamic checkBoxDelayed is executed\n")
    }, ignoreInit=TRUE)
  })

}

shinyApp(ui, server)

Comment: I had the same problem with checkboxes and change from NULL before render to boolean after render, so the fix is just a flag that is check inside observer that is modified after first call.
Answer: Unfortunately, I can't comment due to missing reputation, but here are my thoughts:
I guess the observeEvent does get blocked while initializing. But after that you render the UI Element. The input$dynamicBox changes from NULL to FALSE which triggers the observeEvent function.
You could fake the behavior by introducing a unique global variable within the session, say unique_init_variable <<- FALSE and then change it to true once you've used the checkbox. Here's an example code
library(shinydashboard)
library(shiny)
library(shinyjs)

ui <- dashboardPage(
  dashboardHeader(title = "observeEvent ignoreInit"),
  dashboardSidebar(),
  dashboardBody(
    useShinyjs(),
    checkboxInput("staticBox", "static"),
    uiOutput("body"))
)

server <- function(input, output, session) {

  # unique init variable
  unique_init_variable <<- FALSE

  output$body <- renderUI( tagList(
    checkboxInput("dynamicBox", "dynamic"),
    checkboxInput("dynamicBoxDelayed", "dynamic delayed")
  ))

  # observe static checkBox
  observeEvent(input$staticBox, {
    cat("observeEvent of static checkBox is executed\n")
  }, ignoreInit=TRUE)

  # observe dynamic checkBox
  observeEvent(input$dynamicBox, {
    if (input$dynamicBox == TRUE) {
      unique_init_variable <<- TRUE
    }
    if (unique_init_variable) {
      # this is executed upon start, although ignoreInit is set to TRUE
      cat("observeEvent of dynamic checkBox is executed\n")
      cat( paste0("dynamicBox value = '", input$dynamicBox, "'\n") )
    }
  }, ignoreInit=TRUE)

  # observe dynamicDelayed checkBox
  shinyjs::delay(100, {
    observeEvent(input$dynamicBoxDelayed, {
      cat("observeEvent of dynamic checkBoxDelayed is executed\n")
    }, ignoreInit=TRUE)
  })

}

shinyApp(ui, server)

Comment: Using a variable to ignore the first call is a similar work around than using `shinyjs::delay` in the original post. Since I do not understand why `ignoreInit` is not working, I cannot be sure that the workarounds always work. Interestingly, if `checkboxInput("dynamicBox", "dynamic")` is replaced with `actionButton("dynamicBox", "button")`, the program works as expected.
From my real program I have reason to believe that the GUI elements are not created when the server code is executed, as if there were two threads, but I did not yet come up with a minimal example demonstrating that.


To remove more than predictors in lm() function in r

Question: I've 10 predictors and 1 response variable.
I tried running linear model using
lm(y~., data=mydata)

If I just need to remove one predictor 'age', I can write
lm(y~.-age, data=mydata)

If the summary of the model suggest that more than one variables are not significantly contributing to the model. How can I efficiently write a code for linear model removing these variables.
I tried
lm(y.-c(age, weight), data=mydata)

But I got the error
Error in model.frame.default(formula = y ~ . - c(age, weight),  :
variable lengths differ (found for 'c(age, weight)')

Please help.
Comment: Software-related questions are off-topic on this site so I'm voting to close this thread. As a hint, you can always use `lm(mpg~.-cyl-disp-hp, mtcars)` or `lm(mpg~., mtcars[,-c(5:10)])`.
Comment: No problem :) You can always take a tour https://stats.stackexchange.com/tour to learn more about the site ;)
Answer: You can chain several predictors with a minus sign
lm(y ~ . - age - weight, data=mydata)

Comment: Thanks. I should have tried it before asking. I was trying to use a vector of variables instead.
Answer: Following are some of the methods which can be used :

Subset selection : Identifies subset of predictors that are related to the response. This can be accomplished using best subset selection or stepwise subset selection methods.
Shrinkage methods : Coefficients of predictors weakly related to response are shrunken towards zero. Ridge and Lasso regression can be used for this.
Dimension reduction : Find the predictors which are linearly correlated to other predictors. Can be done using PCA (Principal component analysis).

For more details and worked out examples in R you can refer to chapter 6 from "Introduction to Statistical Learning", which can be downloaded for free from here -> http://www-bcf.usc.edu/~gareth/ISL/
Comment: I am already following the same book. And recently read about these methods. But here I am concerned about how to write a code for removing selected variables from the model.


Computing adjoint of operator

Question: I want to find $T^*$, when the kernel $K(x,y)$ satisfies $K(y,x)=\overline{K(x,y)}$ and for any real $a$, the operator
$$ (Tu)(x)=au(x)+i\int_0^1 K(x,y)u(y)dy$$
on $L^2([0,1])$.
In fact, $T$ is normal, but I have confused how to compute $T^*$.
Answer: Hint: $ \langle T^{*}u, v \rangle =\int_0^{1} u(x)\overline {Tv (x)} dx=\overline {a} \langle u, v \rangle-i\int_0^{1} u(x) \int_0^{1} k(x,y) \overline {v(y)} dy dx$. Interchange the integrals on the right and express RHS in the form $\int_0^{1} \overline {v(y)} g(y)dy$. The conclusion then  is $T^{*}u=g$.

Computing adjoint of operator


Como alterar height de tabela?

Question: Estou desenvolvendo o site da empresa onde trabalho e preciso que a tabela de downloads que aparece no site preencha a tela toda. Imagino que o height teria que estar em 100%, mas quando faço essa alteração, não acontece nada.

Comment: Voce pode usar `height: 100vh;`
Comment: poste seu código inteiro, heigth: 100%, depende dos containeres para funcionar.
Comment: O código é muito extenso pra colocar aqui... é só ir na página e dar Ctrl + U.
Comment: @RafaelAugusto, funcionou parcialmente... Agora faltaria eliminar os espaços em branco em cima e embaixo da tabela.
Comment: @RafaelAugusto, como posso alinhar o texto da célula para baixo?
Comment: Consegui...
Comment: para consultar o suporte de vh nos browsers acesse http://caniuse.com/viewport-units/embed/
Answer: O height é relativo ao container no qual o elemento está contido.
Considerando que o container da sua tabela é o próprio body do documento, é necessário que o documento também possua altura 100%. Para isso, uma forma de definir a altura height=100% do documento (body, html) é aplicar no CSS:

Desta forma, sua tabela com height:100% irá ter a mesma altura do documento (área visível do navegador).
Pode-se usar também o 100vh no elemento (height:100vh), mas alguns navegadores mais antigos não suportam esse valor vh (ex. Safari para Windows, Safari e Chrome em iPhones mais antigos).
Outra solução com jQuery, sem o CSS citado acima:

Como alterar height de tabela?


Construct a regular expression for a given language

Question: I'm currently working on some exercises to get used to create regular expressions from given languages and i'm stuck with a fairly simple exercise. So could you please tell me how to construct it step by step (if possible) from this given language. I would really appreciate your help.
L={w∈{a,b,c}∗:w contains an odd number of one or more of a,b or c}
Comment: I don't think this can be done. In some sense, you're trying to count the total occurrences of a letter in an arbitrarily long string with other letters.
Answer: For starters: an even number of $a$'s: $((b+c)^*a(b+c)^*a(b+c)^*)^*$.


How to reference multiple fields? eg. "$$ROOT.field.$$id" - FieldPath field names may not start with '$'

Question: I am trying to figure out how to reference a field using a variable...
What I am trying to do:
I have an array of categories in one field with an ids field (array) in it, and a field (dictionary) where the keys are an id from these arrays.
I am trying to map the array of ids in each category into the values in the dictionary.
What I am starting with is something like this:
Document {
    "categories": [
      {"name": "One", "ids": [4, 7]},
      {"name": "Two", "ids": [7]}
    ],
    "data": {
      4: {"name": "John"},
      7: {"name": "Nick"},
      2: {"name": "Oliver"}
    }
}

And the result I am looking for is something like this:
{
    "One": [
        {"name": "John"},
        {"name": "Nick"}
    ],
    "Two": [
        {"name": "Nick"}
    ]
}

What I tried doing at first was the straight-forward solution that failed me...
"$replaceWith": {
    "$arrayToObject": {
        "$map": {
            "input": "$$ROOT.categories",
            "in": {
                "k": "$$this.name",
                "v": {
                    "$map": {
                        "input": "$$this.ids",
                        "as": "id",
                        "in": "$$ROOT.data.$$id"
                    }
                }
            }
        }
    }
}

Which didn't work... (Playground link)
I got the "FieldPath field names may not start with '$'. Consider using $getField or $setField." error.
I then also tried using "$getField" like I was suggested but the expression given there must be a constant.
From there I tried finding a solution on Google but couldn't find anything. I decided to resort to using "$function":
"$replaceWith": {
    "$arrayToObject": {
        "$map": {
            "input": "$$ROOT.categories",
            "in": {
                "k": "$$this.name",
                "v": {
                    "$map": {
                        "input": "$$this.ids",
                        "as": "id",
                        "in": {
                            "$function": {
                                "body": "function(id, data) {return data[id]}",
                                "args": [
                                    "$$id",
                                    "$$ROOT.data"
                                ],
                                "lang": "js"
                            }
                        }
                    }
                }
            }
        }
    }
}

Which worked. (Playground link)
But, since I believe that '$function' isn't as efficient and also due to my curiosity, I would like to know if there is a better solution to my problem.
Thanks!
EDIT
I just found out that PyMongo doesn't support '$function' so I definitely would love some help.
Comment: you need to change schema, you can do those in MongoDB without `$function` but it will be complicated and slow. You shouldn't use data on fields, fields are for the schema, MongoDB query language doesn't work with dynamic keys, to make it work you do complicated and slow things.
Comment: I don't see filtering here...?   If you're not filtering and you need a whole doc worth of data to perform the transform you seek, easier to just get the doc client-side into a programming language and use 2 for loops and a hash and you're done.
Answer: Query

the problem is that in MongoDB we cant until now at least use dynamic fields, for example we don't have a get(myDoc, $$akey)
a way to do it in MongoDB aggregation with objectToArray and back to object
query is so big, slow, and even if you try to make it smaller(probably its possible) it will be still be so complicated for the simple thing you wanted to do
you need to change schema, no matter how better you write that it will still be complicated and slow

Test code here
aggregate(
[{"$set":{"data-arr":{"$objectToArray":"$data"}}},
 {"$project":
  {"categories":
   {"$map":
    {"input":"$categories",
     "in":
     {"$let":
      {"vars":
       {"d":
        {"$map":
         {"input":{"$objectToArray":"$$this"},
          "in":["$$this.k", "$$this.v"]}}},
       "in":
       {"$let":
        {"vars":
         {"k":{"$arrayElemAt":[{"$arrayElemAt":["$$d", 0]}, 1]},
          "v":{"$arrayElemAt":[{"$arrayElemAt":["$$d", 1]}, 1]}},
         "in":
         {"k":"$$k",
          "v":
          {"$map":
           {"input":"$$v",
            "in":
            {"$first":
             {"$filter":
              {"input":
               {"$map":
                {"input":"$data-arr",
                 "in":
                 {"$cond":
                  [{"$eq":[{"$toString":"$$this"}, "$$dk.k"]},
                   "$$dk.v", null]},
                 "as":"dk"}},
               "cond":{"$ne":["$$p", null]},
               "as":"p"}}}}}}}}}}}}}},
 {"$set":{"categories":{"$arrayToObject":["$categories"]}}},
 {"$replaceRoot":{"newRoot":"$categories"}}])


How to prove that the sequence $X_{n+1} = 1/2 × (X_n+\sin(X_n)\cos(X_n)) >0 $

Question: In a exercise i need to prove that the sequence :
$$X_{n+1} = \frac12(X_n+\sin(X_n)\cos(X_n))$$
converges to do that i need to prove that the sequence is decreasing by using $\sin(X_n)\cos(X_n)=\sin(2X_n)/2$ and than i have to prove that $X_n > 0$ by induction.
Comment: To make questions easy to read,  it is recommended the use of [MathJax](https://math.meta.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference) to write mathematical equations and/or expressions. Also, adding some context in the way of things you tried but failed, where you get stuck, or ask for hints will improve your question. Some people are too sensitive to this kind of things and are quick to pull the trigger on downvotes without giving any positive feedback to questioners.
Answer: Hint: $\frac{1}{2}(x+\sin x\,\cos x)=\frac{1}{4}(2x+\sin(2x))=\frac{x}{2}\big(1+\frac{\sin(2x)}{2x}\big)$
If $t>0$, what can you say about $\frac{\sin t}{t}$?
Comment: I am sorry but i didn't get it, can you pleas help me i want to prove that Xn > 0
Comment: Try to do some analysis on the function $t\mapsto \frac{\sin t}{t}=\operatorname{sinc}(t)$. It is known that $|\operatorname{sinc}(t)|\leq 1$. Use some differential calculus to see that.
Comment: A plot say between 0 to 20 should give you an idea. Notices that as $t\rightarrow0$, $\operatorname{sinc}(t)\rightarrow1$, which is something you may already know.
Comment: Thanks, i didn't heard about sinc function before


PHP Chilkat imap component memory leaking

Question: Given:

ubuntu 18.04
PHP 7.2.2 (cli) (built: May 12 2019 14:29:04) ( ZTS )
Chilkat component 184.108.40.206

Session log is disabled.
Script running permanently, connection is via ssh and keep opened.
The following function called periodically:
protected function fetchSingle(int $uid)
{
    $e = $this->imap->fetchSingleAsMime($uid, true);
}

It is just to demonstrate, that RES and %MEM on linux top keep increasing:

And very important moment:
after fetching several big emails if there is executed then next code:
$this->imap = null;
unset($this->imap);
sleep(324);

Then RES and %MEM go down to initial values!
But without
$this->imap = null;
unset($this->imap);

RES and %MEM stay on high values!
Logically I may predict that problem inside imap CkImap object...
What can I do ? Unset periodically CkImap object and reconnect is not good solution...
Why is it so ? Maybe chilkat is storing data somewhere in itself ? And I should call some method to clear it ?
Reproducible code:
<?php

require "core/Chilkat.php";

$ckGlobal = new CkGlobal();

$ckGlobal->UnlockBundle('XXXX');

$imap = new CkImap();
$imap->put_Utf8(true);
$imap->put_Ssl(true);
$imap->put_Port(993);
$imap->put_PeekMode(true);

$imap->Connect('imap');

$imap->Login('login', 'password');

$imap->SelectMailbox('Inbox');

// UID of last message
preg_match('/UID (\d+)/', $imap->sendRawCommand("FETCH * (UID)"), $result);

$i = 0;
while(true){
    var_dump('FETCHING');
    $mime = $imap->fetchSingleAsMime($result[1], true);
    var_dump('SLEEP');
    sleep(5);
    if(++$i >= 5) {
        var_dump('UNSET');
        $imap = null;
        unset($imap);
        sleep(666);
    }
}

With the above code we fetch email each 5 seconds and RAM consumption is increasing unstoppable. But it must not, because we overwrite the $mime variable. And once imap is unset - RAM goes down to first value.
get_VerboseLogging - this is false by default. And Peek mode I also tried to set to false. Nothing helped to solve or figure out the problem.
Even after $imap->Disconnect(); RAM will not be decreased..
Comment: If you can provide a [mcve] that demonstrates this, it might be possible to tell whether it's a bug in your code or whether it is a bug somewhere else. Without that info, it's impossible to tell the right way to proceed.
Comment: just fetch several big emails, I provided the code, that's enough. It will increase RAM continueasly. I will add such full code
Comment: Please read the link: Neither the snippits you provide are okay (not enough to be reproducible) nor the full code (which probably contains irrelevant parts)!
Comment: @Ulrich Eckhardt, I have added simple code tested by me
Answer: When it stays at about the same utilization level, just ignore it and add more RAM ...but when it keeps growing, then you'll have to periodically purge the leakage. With closed source you can only complain at the vendor, who might also have established best practices for using their library.
According to Chilkat_9_5_0.Imap, there is a property VerboseLogging
... and also property PeekMode might affect the memory utilization.
Comment: You are absolutely right! RAM is increasing permanently. Without vendors help I can only do something like reopen process...
Comment: @Mikael it may also have to do with [RFC 4549 - Synchronization Operations for Disconnected IMAP4 Clients](https://tools.ietf.org/html/rfc4549), which means: the mailbox size. Even with closed source, one can attach a debugger - despite one cannot fix anything.
Comment: the link you gave is just explanation of IMAP protocol. I don't think there maybe something regarding memory leaking problem
Comment: @Mikael I find the memory usage in the screenshot not even extreme (this is far from OOM), and IMAP works differently than POP3, since it rather expects a connection - and it fetches more than just the index. I'd rather wonder about the SSH tunnel instead of IMAPS, since mail-servers most often have a public IP.
Comment: on screenshot it is not extreme, it's just to show what I mean. On live server with 250GB RAM it increases unstoppable.. Ssh is socket connection, it uses buffers.. I really think problem is in chilkat CkImap object
Comment: @Mikael This might be by design - and not caching to local file-system; there is a difference in between online and offline mode, when it comes to IMAP clients. One can also write them rather lightweight, but a fully implemented IMAP client needs some RAM & drive-space. The SSH tunnel just adds one additional component, which might not be required with IMAPS. Fetching some mails can also be easily done without third party code.
Answer: I apologize! I have checked this code more carefully and seems RAM increased until specific limit! But still interesting why it is increasing..


Minidom: How can I check if I have the expected root and children?

Question: I have this xml structure,
<root>

    <child1>
    </child1>
    <child2>
    </child2>
    <child3 />
    <extendedchild:name>
    </extendedchild:name>

</root>

How can I check with minidom, that root is root, and that children are always the followin elements?
child1
child2
child3
extendedchild

I want also to print the elements that are out of root or not in the above "children list"(outofroot, notachild):
...
    <notachild />
</root>
<outofroot />

Edit:
It seems that outofroot element is treated by the minidom parser, it gives xxxxxxx.xml has an error: junk after document element: line 12, column 0
Comment: It seems that `outofroot` element is treated by the parser, it gives `xxxxxxx.xml has an error: junk after document element: line 12, column 0`
Answer: As far as the root element name check, looks like you can do it like this:
import xml.dom.minidom
dom = xml.dom.minidom.parseString(xmlString)
if dom.documentElement.tagName == "root" ...

You should be able to for..in iterate on the root's .childNodes.
If something is outside of the root, it is not a well-formed XML document (which can only have one root node).
Comment: As far as outofroot, it is treated by the fact that it is not allowed. :) yes, it should be `documentElement` if it behaves like normal DOM.
Comment: It seem that `outofroot` is treated by the parser, it gives `xxxxxxx.xml has an error: junk after document element: line 12, column 0`
Comment: `root=`?, `root=documentElement`?
Answer: You can use minidom to walk though the children and verify that the name of the root node is "root". You can then process children one at a time and verify the other requirements.
if not root.tagName == "root":
   # do something
for node in root.childNodes:
   # do something more

If necessary, you can process the child nodes recursively.
def processChild(node):
    # do some checks on node
    for child in node.childNodes:
        processChild(child)

Comment: `root=`?, `root=documentElement`?
Comment: Yes, `root=doc.documentElement`


Byte array image crop performance issue

Question: Have two pointers (byte*) to 1. B8G8R8A8 pixel data 2. byte buffer to put cropped pixel data, very simple issue. Here is my implementation:
private unsafe void Crop(byte* src, byte* dst, Rectangle rSrc, Rectangle rDst)
{
    int sSrc = 4 * rSrc.Width, sDst = 4 * rDst.Width; // stride

    for (int x = 0; x < rDst.Width; x++) for (int y = 0; y < rDst.Height; y++)
    {
        dst[(x * 4 + y * (sDst)) + 0] = src[((rDst.X + x) * 4 + (rDst.Y + y) * (sSrc)) + 0];
        dst[(x * 4 + y * (sDst)) + 1] = src[((rDst.X + x) * 4 + (rDst.Y + y) * (sSrc)) + 1];
        dst[(x * 4 + y * (sDst)) + 2] = src[((rDst.X + x) * 4 + (rDst.Y + y) * (sSrc)) + 2];
        dst[(x * 4 + y * (sDst)) + 3] = src[((rDst.X + x) * 4 + (rDst.Y + y) * (sSrc)) + 3];
    }
}

And it works: ~ 100ms for 1920x1080, but need ~ 10 - 15 ms. Is it possible to make it faster? For example plain copy (without crop) provides much better performance 8 ms (for the same resolution), here is function:
private unsafe void Copy(void* dst, void* src, long count)
{
    long block; block = count >> 3;

    long* pDst = (long*)dst; long* pSrc = (long*)src;
    {
       for (int i = 0; i < block; i++) { *pDst = *pSrc; pDst++; pSrc++; }
    }
}

Need your thoughts!
Thanks.
Comment: Have you tried the .NET framework cropping methods, creating a new bitmap from an existing one and a rectangle? It may not perform as badly as you think. Otherwise there are unmanaged methods for bit blitting that may be faster by copying an entire row at once instead of pixel-by-pixel-by-byte.
Comment: Yes. it is very possible to make it faster.  Precompute what you can in your loops, and find a way to get .NET to use some of the SIMD instructions.
Comment: store `((rDst.X + x) * 4 + (rDst.Y + y) * (sSrc))` in variable and use it 4 times instead of calculating it 4 times. the same thing for `(x * 4 + y * (sDst))`
Comment: What version of .NET Framework are you targeting?
Comment: @IvanStoev 4.5 is target
Answer: Actually you can make it run in in 3 milliseconds and don't need to be unsafe.
void Main()
{
    int width;
    int height;
    int border;

    byte[] src;
    using (var bm = new Bitmap(@"C:\Data\sample-photo.jpg"))
    {
        width = bm.Width;
        height = bm.Height;
        border = width / 10;

        src = new byte[width * height * 4];

        for (var y = 0; y < bm.Height; y++)
        for (var x = 0; x < bm.Width; x++)
        {
            var pixel = bm.GetPixel(x,y);
            var i = GetIndex(width, x,y);
            src[i + 1] = pixel.R;
            src[i + 2] = pixel.G;
            src[i + 3] = pixel.B;
        }
    }

    var sw = Stopwatch.StartNew();
    byte[] dst = null;

    dst = Crop(src, width, border, width - border, border, height - border);
    sw.Stop();

    Console.WriteLine(sw.Elapsed);

    using (var dstBmp = new Bitmap(width - border * 2, height - border * 2, PixelFormat.Format32bppArgb))
    {
        for (var y = 0; y < (height - border * 2); y++)
            for (var x = 0; x < (width - border * 2); x++)
            {
                var i = GetIndex(dstBmp.Width, x, y);
                var r = dst[i + 1];
                var g = dst[i + 2];
                var b = dst[i + 3];
                dstBmp.SetPixel(x, y, Color.FromArgb(255, r, g, b));
            }
        dstBmp.Save(@"C:\Data\sample-photo-cropped.jpg");
    }
}

byte[] Crop(byte[] src, int srcWidth, int xStart, int xEnd, int yStart, int yEnd)
{
    var width = xEnd - xStart;
    var height = yEnd - yStart;

    var dst = new byte[width * height * 4];

    for (var iY = yStart; iY < yEnd; iY++)
    {
        Array.Copy(src, GetIndex(srcWidth, xStart, iY), dst, GetIndex(width, 0, iY - yStart), width * 4);
    }

    return dst;
}

int GetIndex(int width, int x, int y)
{
    return (x + (width * y)) * 4;
}

Output:
00:00:00.0007131

To work with pointers you can use Marshal class, which has a bunch of overloads for copying memory or the Buffer.MemoryCopy Method
Comment: Thanks for quick reply! To be "unsafe" (pointers) is a limitation of an external system, I'm getting "images" from a device and need to apply some transformations and encode mp4. Sure its possible to get an array via IntPtr and then apply crop but not sure its a good idea.
Comment: Anyway fixed the code for a more complete example and fixed the errors. Now it actually crops a bitmap in around a millisecond time.
Comment: This is a great example.  I would optimize it further by using bit shifts instead of multiplication, and skip the calls to GetIndex in the loop for the destination address since it will always be consecutive, and skip the calls to GetIndex in the loop and preinit it outside the loop and use add to skip the unused bytes.
Comment: Nice example, thanks. But I don't think any Marshal overload or Buffer can be used to replace Array.Copy.
Answer: There you go
private static unsafe void Crop(byte* src, byte* dst, Rectangle rSrc, Rectangle rDst)
{
    int* pDst = (int*)dst, pSrc = (int*)src + (rDst.Y * rSrc.Width + rDst.X);
    int srcSkip = rSrc.Width - rDst.Width;
    int rowCount = rDst.Height, colCount = rDst.Width;
    for (int row = 0; row < rowCount; row++)
    {
        for (int col = 0; col < colCount; col++) *pDst++ = *pSrc++;
        pSrc += srcSkip;
    }
}

Byte array image crop performance issue


Why does DATE_SUB function not work for mysql query with node js?

Question: I have a datetime field in a MySQL database table on Planetscale. I want to return the datetime value after some time has been subtracted from it using the DATE_SUB function. This works in the database console on Planetscale's website but is not working from my node js server request.
Server code:
const mysql = require('mysql2');
mysql.query('SELECT DATE_SUB(log_time,INTERVAL 4 HOUR) AS time FROM time_table;',[], function(err, records){
   if (err) {
      console.log(err);
   }
   else {
      console.log(records);
   }
})

And this is my table:
CREATE TABLE time_table (
    log_time DATETIME
);

But is not subtracting the 4 hours as I am expecting and as it does on the db console on Planetscale's website. Is this an issue with Planetscale, MySQL, MySQL2, node js, or something else? Thanks.
Comment: But the timezone thing could be a real issue if you are talking about planet times, so yeah, we only have "not subtracting 4 hours" , ok, how many did it subtract, or did you get a different error?
Comment: Check your data types?   Is there a date that is actually a string in your node?
Comment: This sounds like a time zone issue. What happens if you do `SELECT CAST(DATE_SUB(...) AS CHAR) AS time`?
Comment: @easleyfixed They have the table definition in the question.
Comment: Yes, but I don't see what the node expects it to be is my point.  Yes it is SUPPOSE to be a date, but what does his node think it is I wonder?
Comment: There was no error, it just subtracted no time at all.
Comment: Are you sure of the current data in the table? Try changing your query to include `log_time` in the results so you can compare... `SELECT log_time, DATE_DUB(...`
Answer: Seems my issue was rooted in the fact that there are different base times for the results of the query.
If I were to just enter SELECT log_time AS time FROM time_table; then the PlanetScale console will return a different time than the results output by node js query result.
I needed to subtract 4 hours to get the Planetscale time into my timezone, but coincidentally the node js results were another 4 hours ahead of planetscale. Thus, when I subtracted 4 hours it appeared to me that no time was being subtracted at all. In reality I just needed to subtract 8 hours in the node query. I guess there is some automatic time zone adjustments going on that make the output that nodes gives different from PlanetScale, but I have solved my issue.


Search Results Web results Collapsible Navbar

Question: How to click on a collapsible nav bar in selenium webdriver?

            <textarea id="terms_text" name="terms_text" class="form-control tinymce" rows="4"></textarea>

I have tried it through XPath but element not clickable at(x,y) arrives.
Comment: Can you show what you have done ? The above is not clear - which line would you except to be clickable ?
Answer: You could try clicking with Javascript to bypass the ElementNotClickable error you are seeing:
menu = driver.find_element_by_xpath("//div[contains(@class, 'panel-collapse')]")

driver.execute_script("arguments[0].click();", menu)

Some elements do not accept regular Selenium click() method, so we use Javascript occasionally to work around these special elements.
Hope this helps a bit.
Comment: What kind of elements have you found where the regular Selenium `click()` method does not work?
Comment: I have seen issues with `click()` not working when trying to click `div` elements. `click()` usually works on `button`, `a`, `input`, and a few others. But clicking `div` gives issues sometimes -- not always however. There is no catch-all case where you know you need to use Javascript. It always depends on the web page you are automating, and what the behavior is like. I try one method, if it does not work, then I try the other.
Comment: The only times I've had issues like the one your describing is when the element I'm trying to click is off the screen, or is covered up by another element.  It sounds like what you're doing could work in those situations though, if making the element be available for Selenium to click on is not feasible.
Comment: @Phil these are also cases, yes. In some React or Javascript-heavy applications, I have seen cases where a perfectly visible element still throws the error.
Comment: I see.  Are there any publicly available sites you've written automation against that display this behavior?  I'd be interested in seeing this happen.
Comment: This was a private QA environment, so unable to share -- in production, custom HTML themes were applied on top so QA and PROD web pages were completely different (testing nightmare). But the application was written in React.js with a mixture of Ext and Angular JS frameworks. I ran into cases where clicking `button` would fail, because a hidden `div` element was obscuring -- visually, the `button` looked clickable, but hidden elements higher up in the tree would cause visibility issues, making Javascript click necessary in some cases.
Comment: Oh, ok.  It sounds like this is similar to the situation I was thinking of, where a div is covering something you want to click, the only difference being that the end user would not be able to tell, because the covering div is invisible.


Gephi preview export pdf cuts off label at left and right margin

Question: I would like to export a SVG/PDF/PNG from the preview tab in Gephi. The exported network includes all the nodes, but cuts off the node labels on the left and right sides. How can I get an exported pdf which does not cut off the labels?
Answer: You have to adjust the size of the output by controlling the Options. For example if you save in PNG format you see the following dialog where you should click at the bottom right corner

Which brings you to the following dialog where you adjust the image width

You have to play a little bit with the settings to see what works well for you


Quantify strength of association of two continuous variables while controlling for random effects

Question: I have a data set from a repeated measures experimental design with different sets of stimuli. I want to know how strong the association between the continuous dependent variable and the continuous predictor is while accounting for the interindividual and interstimulus variation.
My lmer model description in R looks like this
dv ~ pred + (1 | subject) + (1 | stimulus)

Question 1: I understand that it is non-trivial to calculate R squared for random intercept-slope models. Is the same true for random intercept models? Is there an R-implementation of any of the available methods?
Question 2: If I z-transform my dependent variable and predictor, will the parameter estimate for the fixed effect reflect the strength of the association such as it would in an ordinary regression model? Update: I think I phrased this questions too vaguely. I was wondering if scaling variables would yield standardized regression estimates. I found this question has been answered before in another question.
Question 3: Is there an entirely different/more appropriate way to quantify the association strength while controlling for the interindividual and interstimulus variation?
Answer: Q1: Yes, it is not trivial.  Please check the following link http://glmm.wikidot.com/faq under the section How do I compute a coefficient of determination (R2), or an analogue, for (G)LMMs?  as provides a quite good treatment of the issue; small 3-4 liners of code are presented that calculate an $R^2$ like measure.
A rather helpful reference of the subject seems to be Xu's: Measuring explained variation in linear mixed effects models (2003) where the author computes the residual variance of the full model against the residual variance of a (fixed) intercept-only null model (in R syntax:  1-var(residuals(m))/(var(model.response(model.frame(m)))).
Q2: No, it would not reflect that. A crossed random effects structure in your data will still apply to the error structure you would assume in an OLS; you will just change the variances in your sample.
Just to put things into perspectice: your OLS assumes that  $y \sim N(X\beta,\sigma^2 I)$, but you actually have that error diagonal structure only if you account for the "grouping in your samples" through the indicator variables $\gamma$, ie. $y|\gamma \sim N(X\beta+Z\gamma,\sigma^2I)$ (or unconditionally   $y\sim N(X\beta,\sigma^2I+ZDZ^T)$), whitening your $X$ and $y$ does change the amplitude of $D$ and $\sigma$ ($D$ is the diagonal matrix holding the std. deviation of your random effects) but other than that your structure remains unchanged.
Unless you have problems with numerical stability I would not recommend z-transforming / whitening your sample variables.
Q3: Some people might suggest Generalized Estimating Equations (GEEs) but I think your approach is rather sensible and straightforward.
Comment: Thanks a lot. From the glmm-FAQ: "Researchers often want to know if there is a simple (or at least implemented-in-R) way to get an analogue of R2 or another simple goodness-of-fit metric for LMMs or GLMMs." Haha, true. ^^


Why does unity show question mark instead of desired sprite?

Question: In unity3d, I am using WWW class to download a texture from link, then I am creating a Sprite with the obtained texture and I am displaying it on the screen. This is working just fine in Unity Editor, but when I run it in the browser as HTML5 instead of displaying my Sprite, there is a red question mark. Why?
This is my C# code:
GameObject myImage;
Sprite neededSprite;

IEnumerator Start () {

 WWW getMyImage= new WWW("http://previews.123rf.com/images/burakowski/burakowski1202/burakowski120200228/12221967-Grunge-Example-stamp-Stock-Vector-demo.jpg");

 myImage = GameObject.Find("myImage");

 yield return getMyImage;
 Texture2D rawImage = getMyImage.texture;
 neededSprite = Sprite.Create(rawImage, new Rect(0.0f, 0.0f, rawImage.width, rawImage.height), new Vector2(0.5f, 0.5f), 100.0f);
 myImage.GetComponent<Image>().sprite = neededSprite;

}

Comment: Have you got any error in your browser console ? I suspect a cross-origin problem : https://docs.unity3d.com/Manual/webgl-networking.html
Comment: Here is a nice website to learn how to deal with CORS : http://enable-cors.org. If the server is an Apach, the easiest way is to put a htaccess with specific instructions next to the resources you are trying to get
Comment: After placing `Header set Access-Control-Allow-Origin "*"` in `.htaccess` file, everything is working as excepted
Comment: Possible duplicate : http://stackoverflow.com/questions/30090907/unity3d-webgl-cross-origin-request-blocked
Comment: Try saving the downloaded texture as png to see if it is a valid texture. Use [`Texture2D.EncodeToPNG`](https://docs.unity3d.com/ScriptReference/Texture2D.EncodeToPNG.html)
Comment: It is a cross-origin problem, just like @Hellium said. I know how to solve it for a url with a string content. I put in the php file `header('Access-Control-Allow-Origin: *');` How do I apply this for a `.jpg` file? How do I allow access on a server to obtain an image?
Comment: Thanks @Hellium, you helped a lot!
Answer: After placing Header set Access-Control-Allow-Origin "*" in .htaccess file, everything is working as excepted


Uniform boundedness in $L^p$ norm

Question: Given a Banach space $X$ and a sequence $(f_n)_{n\in \mathbb{N}}\subset L^p([0,T];X)$ such that
$$
\sup_{n\in \mathbb{N}} \Vert f_n \Vert_{L^p([0,T];X)} <+\infty
$$
Can we say  that
$$
\sup_{n\in \mathbb{N}} \Vert f_n(t) \Vert_X <+\infty
$$
for some (or every) $t\in [0,T]$? Thank you in advance.
Comment: Is it true when $X=\mathbb R?$
Comment: I don't know, have you a counterexample?
Answer: Hint: Consider $1^{1/p}\chi_{[0,1]}, 2^{1/p}\chi_{[0,1/2]}, 2^{1/p}\chi_{[1/2,1]},3^{1/p}\chi_{[0,1/3]},3^{1/p}\chi_{[1/3,2/3]},3^{1/p}\chi_{[2/3,1]}, \dots$

Uniform boundedness in $L^p$ norm


Find a maximum likelihood $\theta$ for $N(\theta,1)$

Question: $X$ is a random variable following $N(\theta,1)$
$\hat{\theta}$ is a maximum likelihood estimator of $\theta$ and $b(\theta)=E[\hat{\theta}]-\theta$.
I need to prove $b(\theta) \geq 0$ and express a function $f(\theta)$ with $\theta,b(\theta)$, where $$f(\theta)=E[(X-\theta)^2]-E[(\hat{\theta}-\theta)^2]-E[(X-\hat{\theta})^2]$$
We also have
$$\phi (x) = \frac{1}{\sqrt{2\pi}} \exp\{-\frac{x^2}{2}\}, \Phi(x)=\displaystyle \int_{-\infty}^{x} \phi (y) dy$$

I got $\hat{\theta}$ as $\frac{\displaystyle \sum_{i=1}^n x_i}{n}$, but I found out that I am not given $n$ nor $X_i,x_i$.
If I am correct, $b(\theta)$ is going to be 0, and I don't think that is a conclusion they want.
For $f(\theta)$,
I opened up expected value as $$E(X^2)-E(2X\theta)+E(\theta^2)-E(\hat{\theta}^2)+2E(\hat{\theta}\theta)+E({\theta^2})-E({X^2})+E({2X\hat{\theta}})+E({\hat{\theta^2}})$$
but I could not find out how to deal with some terms especially $E(2X\hat{\theta}),E(\hat{\theta}^2)$
Comment: The sample is $X$ which is also the MLE. The question does not make sense as written since the bias is never positive.
Answer: In short, your answer is correct: the MLE for $\theta$ is unbiased.
Remark 1.
The question not explicitly stating $n$ (and, correspondingly, the observations $X_1, \ldots, X_n$) can be interpreted in one of two ways: (i) $n$ is arbitrary or (ii) $n=1$.
Remark 2.
The MLE requires at least one sample from the distribution: it is, by definition, the estimator which maximizes the likelihood.
It doesn't make sense to talk about the likelihood without any samples.
Proof.
Given $X_{1},\ldots,X_{n}\sim N(\theta,1)$, the likelihood is
$$
\mathcal{L}(\theta)=\prod_{i}\varphi(X_{i})\propto\exp\left(-\frac{1}{2}\sum_{i}\left(X_{i}-\theta\right)^{2}\right).
$$
Taking logarithms,
$$
\log\mathcal{L}(\theta)=-\frac{1}{2}\sum_{i}\left(X_{i}-\theta\right)^{2}+\text{const.}=-\frac{1}{2}\left(n\theta^{2}-2\theta\sum_{i}X_{i}\right)+\text{const.}
$$
Taking derivatives,
$$
\frac{d}{d\theta}\left[\log\mathcal{L}(\theta)\right]=-n\theta+\sum_{i}X_{i}.
$$
Setting this to zero and solving for $\theta$ yields the max likelihood estimator
$$
\hat{\theta}=\frac{1}{n}\sum_{i}X_{i}.
$$
This estimator is unbiased because
$$
b(\theta)=\mathbb{E}\hat{\theta}-\theta=0.
$$
Comment: @ohisamadaigaku: I added some remarks that should answer your question.
Comment: Is there perhaps I can get a MLE without $n$ and $i$?


Spatial4j - Create circle using centroid and radius in miles

Question: Using Spatial4j, I want to create a circle using a centroid and radius specified in miles (or kilometers).  The Javadoc says "The units of "distance" should be the same as x & y." which I am confused by.
I understood x and y (lat, lon) to be in degrees and I don't know how to express the radius in degrees.  I want to express it in miles, so I imagine I need to convert miles to degrees, however I also know the conversion is different based on where the centroid is on the surface of the planet.
Is there a way within Spatial4j to easily handle this conversion so I can express the radius in miles and create an accurate circle?
Thanks!
Answer: The DistanceUtils class has the conversion methods you seek, Brooks:

public static double calcBoxByDistFromPt_deltaLonDEG(double lat, double lon, double distDEG)

Should sort you, hope that helps!
Comment: The Javadoc for that method says "The delta longitude of a point-distance. In other words, half the width of the bounding box of a circle."  Parameters for that method are lat, lon and distance in degrees.  I'm not clear on how to get from distance in miles to distance in degrees, otherwise I would just create the circle with distance in degrees to begin with, right?
Comment: Look at the other methods in the DistanceUtils class, perhaps one of those may be helpful?
Answer: From meters:
static double metersToDistanceDeg(int meters) {
    return (1.0 * meters / (1000 * DistanceUtils.DEG_TO_KM));
}

Or if you have miles:
static double milesToDistanceDeg(int miles) {
    return (DistanceUtils.MILES_TO_KM * miles / DistanceUtils.DEG_TO_KM);
}


One-Max fitness function (Java)

Question: Whilst searching on Google about Genetic Algorithms, I came across OneMax Problem, my search showed that this is one of the very first problem that the Genetic Algorithm was applied to. However, I am not exactly sure what is OneMax problem and how can the One-Max problem be represented as a fitness function in java using the following formula which I came across during my search? Can anyone help me out?
S is = to N length of binary string
$
\begin{align*}
OneMax(S) = \sum_{i=1}^ns_i
\end{align*}
$
Any help is appreciated
Comment: As you've said, the problem is simply to maximize the sum of ones in a bitstring. The fitness you'd like to maximize *is* that sum. Is there something else that's unclear?
Answer: With the OneMax problem you are searching for a bitstring where all values are filled with ones.
The fitness can be caluculated by summing all the values of the binary string
<code>int calcFitness(Bit[] bitstring){
    int fitness = 0;
    for( int i = 0; 1 < bitstring.length; ++i)
    {
         fitness += bittring[i];
    }
    return fitness;
}
</code>
You should replace Bit[] by something that actually exist, like int[] (where you would habe to make sure that only 0s and 1es are created or by using an own class...

One-Max fitness function (Java)


How to maintain Session with putty?

Question: I found that when I use putty for our UNIX server connection, it gets time out message after certain time period. Every time I open putty window which is already connected, it it has been idle for more that say 30 minutes, it gets time out and I have to login again to new putty session.
Answer: Found one simple solution for the putty to resolve the "Idle Session Timeout", which overcomes logging again and again to Unix box Putty Sessions. This solution ensures that even if the login remained Idle for a longer time the session does not get killed till exit is explicitly performed.
In Putty Login Screen, Inside Category goTo Connections and change the value for "seconds between keepalives" to a positive number greater than 0. most probably 2 to keep the session alive. This keeps the session alive for even many days.
Comment: 2 seconds is massively overkill. My sessions seem to time out after 5 minutes, so I've set my keep-alive to 60 seconds.
Comment: The fact is, I was not aware the measure of value '2'. Thanks for letting me know that it is in seconds.

How to maintain Session with putty?


Addition of two matrices in quadratic form

Question: Say you have
$x^T\Sigma{x} - y^T\Sigma{y}$
where $\Sigma$ is a symmetric $n\times{n}$ covariance matrix and $x$ and $y$ are $n\times{1}$ vectors. What are the steps to simplify this to
$(x + y)^T\Sigma(x - y)$
Say I didn't know that the simplified version is $(x + y)^T\Sigma(x - y)$. What is the brainwave required to simplify?
Answer: Equality is transitive and then reverse the argument:
$$
\begin{align}
(x + y)^T\Sigma (x - y) &= x^T\Sigma (x - y) + y^T\Sigma (x - y)\\
&= x^T\Sigma x - x^T\Sigma y + y^T\Sigma x - y^T \Sigma y\\
&= x^T\Sigma x - ((x^T\Sigma y)^T)^T + y^T\Sigma x - y^T \Sigma y\\
&= x^T\Sigma x - (y^T\Sigma^T x)^T + y^T\Sigma x - y^T \Sigma y \\
&= x^T\Sigma x - (y^T\Sigma x)^T + y^T\Sigma x - y^T \Sigma y\\
&= x^T\Sigma x - y^T\Sigma x + y^T\Sigma x - y^T \Sigma y\\
&= x^T\Sigma x - y^T \Sigma y.
\end{align}
$$
So why does this work? We are using (in order):

(Left) distributivity of + and matrix multiplication.
(Right) distributivity of + and matrix multiplication.
Transpose of the transpose is the original matrix.
Expand transpose: reverse order and add transposes.
$\Sigma$ is symmetric so does not change by taking transpose.
Actually, we are taking the transpose of a number! So it does nothing.
+/- cancel.

Note that another way is to show that a symmetric $\Sigma$ corresponds to a symmetric bilinear form $B$ and then:
$$
\begin{align}
(x + y)^T \Sigma (x - y) &= B(x + y, x - y)\\
&= B(x, x - y) + B(y, x - y)\\
&= B(x, x) - B(x, y) + B(y, x) - B(y, x)\\
&= B(x, x) - B(y, y)\\
&= x^T \Sigma x - y^T \Sigma y.
\end{align}
$$
This is a more general result: it holds even if $B$ did not come from a symmetric matrix, for example on the vector space of continuous real-valued functions on $[0, 1]$ there is a symmetric bilinear form $B$ given by
$$
    B(f, g) = \int_0^1 f(t)g(t)dt.
$$
There cannot be a matrix in this case, because the space has infinite dimension.
Edit due to change in question (understanding).
How do you get this brainwave? If you don't know where you are going in this case? Pattern matching and heuristics.
This questions evokes feelings of the identity
$$
x^2 - y^2 = x x - y y = (x + y)(x - y)
$$
in me, in fact, this comes from the result that inspired this question on the matrix $(1)$ in vector space $\mathbb{R}^1$ and associated the symmetric bilinear form given by $B(x, y) = xy$.
The following are often useful:

Adding 0, written as $c−c$.
Multiplying by 1, written as $\tfrac{c}{c}$.
Some other operation, like taking the double transpose, that does nothing but can be expanded.

All can lead to simplified expressions, but you need to know if and how to use them. For example, the multiplying by 1 is used for rationalizing denominators. This is something you get good at with experience.
Comment: Thanks, I see how to get from the result back to what I started with, but the real question I'm asking is how to get from what I started with to the result (as if I didn't know it is $x^T\Sigma{x} - y^T\Sigma{y}$. Say I didn't have the brainwave to add and subtract $y^T\Sigma{x}$, and then go from there?
Comment: @NeoCortex64, there isn't really one answer to that. In many cases, adding 0, written as $c - c$, multiplying by 1, written as $c / c$ or some other operation (like taking the double transpose, or the transpose of a scalar) get you to a simplified expression.

This is something you get good at with experience. Often it is simpler to go from the complicated form the simplified form, though here it was not immediately obvious I grant you.
Comment: Thanks. I see this is something I would easily get if I interact with linear algebra all the time. I suppose it's similar to factorising $(a^2-b^2)$, you would add and subtract $ab$ and then go from there.
Comment: @NeoCortex64, exactly! In fact, that result is simply a specialization of the general result. Often mathematics goes both ways: sometimes you've got to make an expression bigger, to simplify it in the end, sometimes you've got to go and specialize a theorem, sometimes you've got to and generalize a theorem.
Answer: We have $(x+y)^T\Sigma(x-y) = (x^T+y^T)\Sigma(x-y)=x^T\Sigma x-x^T\Sigma y+y^T\Sigma x-y^T\Sigma y$
Now as $\Sigma$ is symmetric, it is a self-adjoint operator, so we have $x^T\Sigma y = <x,\Sigma y> = <\Sigma y,x>= <y, \Sigma x> = y^T \Sigma x$, where $<a,b> = a^T b$ is the standard inner product.
So the expression simplifies to $x^T \Sigma x - y^T \Sigma y$.


SQL error log can't create/write to file

Question: I'm quite new to server administration but a keen learner, so if I miss anything or you need any extra info let me know and I'll do my best to provide it ASAP :)
I'm running WordPress on a developer plan VPS over at media temple (meaning I have root access, everything is managed by me etc). My OS is Fedora, Apache is latest version and am using MariaDB for MySQL.
When trying to read/write menus from the WP admin panel Apache writes this to its error log:
[Fri Dec 13 04:28:59.296672 2013] [:error] [pid 11195] [client (my IP, omitted)] WordPress database error Can't create/write to file '/var/tmp/#sql_1ff_0.MAI' (Errcode: 2) for query (**lots of different queries**)

This error repeats for about 20 different queries, and I've checked with perror that error code 2 for SQL means "No such file or directory".
I would contact my host about this problem, but as I said I'm on a developer plan meaning they won't help me with anything. Generally from Googling around I can find an answer or solution but in this case I've come up with nothing.
Answer: You can use the perror command to get an explanation on system errors:
~]$ perror 2
OS error code   2:  No such file or directory

It may be that the directory /var/tmp/ does not exist.
Alternatively the WordPress MySQL user may not have been granted sufficient permissions to create files. You may need to GRANT FILE and/or CREATE TEMPORARY TABLES priviliges to the WordPress user.  Refer to the the GRANT manual page.
Comment: Or the permissions on `/var/tmp` may be wrong (should be 1777).
Comment: More reasons not being able to write could be a full partition (`df`), read-only volume (`mount`) or possibly SELinux related (`getenforce`).
Comment: @HBruijn /var/tmp does exist (from root, that's where I presume you're referring to). Additionally, my WordPress MySQL user has access to `FILE` and `CREATE TEMPORARY TABLES` privs.

@MadHatter The permissions on /var/tmp are 1777 and I've even changed the ownership to mysql[27].
Comment: I would love to hear any other suggestions though, because I'm exhausted. I've spent over 18 hours researching and trying to fix this 1 stupid problem
Comment: All mounts have loads of space, the file system is not read-only because I am able to create folders/files manually, and getenforce=disabled.

This whole problem is getting to be a bit of a joke.. Would it be worth trying setting a tmpdir in my.cnf for SQL (I've tried once, yes I made sure mysql had permissions for the new directory but it still didnt work)? Is there any more detailed method of debugging this error to find out more info?
Comment: The .MAI file exention is apparently used by the MariaDB aria engine for indexes. So it may be an `INDEX` privilege?