How to interpret NearestNeighbor results obtained using cosine similarity for tf-idf vectors

Question: Why is the top result obtained using cosine similarity extremely close to 0 not the expected 1?
That implies complete orthogonality.
Data:
100k documents/rows with 2000 features(TF_IDF values of tokens, mostly zeroes/sparse)
Used NearestNearest neighbors with cosine similarity from this thread:
How to find nearest neighbors using cosine similarity for all items from a large embeddings matrix?
<code># import NearestNeighbors
from sklearn.neighbors import NearestNeighbors
n_neighbor = 10

model = NearestNeighbors(n_neighbors=n_neighbor,
                         metric='cosine',
                         algorithm='brute',
                         n_jobs=-1)
model.fit(df_tfidf)
# let's check first 3 documents
model.kneighbors(df_tfidf.iloc[:3, :])

Out:
(array([[2.22044605e-16, 5.96041399e-01, 6.02695589e-01, 6.05811405e-01,
         6.06062605e-01, 6.06394780e-01, 6.10842747e-01, 6.12047848e-01,
         6.12241848e-01, 6.13224358e-01],
        [0.00000000e+00, 4.26139518e-01, 4.29424721e-01, 4.29424721e-01,
         4.29424721e-01, 4.35292677e-01, 4.41962536e-01, 4.51103908e-01,
         4.52376335e-01, 4.54416818e-01],
        [2.22044605e-16, 6.13395642e-01, 6.20716271e-01, 6.38328419e-01,
     6.43845160e-01, 6.60639675e-01, 6.61006505e-01, 6.66453839e-01,
     6.70172442e-01, 6.71674788e-01]]),
 array([[     0,  40197,   7517,  41572,  40659,  95641,  15538,  36000,
           3115,  36789],
        [     1, 117889,  78484,  37726,  72102,  75169,  91649,   4410,
          18514,  88808],
        [     2,  80311,  63605,  40658,  18017,  33410,  14809,   8880,
           2964,    157]], dtype=int64))
</code>
So we can see that the document is similar to itself - index matches.
The cosine similarity for this top match is 0 not the expected 1.
Thus the vector appears completely transposed
Are the other 9 matches trustworthy - that is are they not similarly transposed?
Notably they are sorted in ascending order.
So the results come out my top match, then 10th match, 9th match, ... 2nd match
It seems like an inconsistency in the model displaying results.
Surely you would want the identical document to have a score of 1 not 0?
EDIT2: I rerun the model looking for 50 closest matches and got the exact same results for the first 10 matches and then 40 more increasing values.
So are the cosine similarity values here inverted - that is 1-real value?
EDIT PER Community Bot request: My question is why kneighbors gives similarity score of 0 for the top match not the expected 1 in the case of document itself.
Comment: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Answer: My hypothesis that model.kneighbors returns 1 - real_cos_similarity score appears correct.
I took a small 10 document x 2000 tf_idf score matrix from the original data and computed cosine similarity
using
<code># import cosine_similarity
from sklearn.metrics.pairwise import cosine_similarity

# compute cosine similarity
cosine_sim = cosine_similarity(df_tfidf.iloc[:10, :])
cos_rounded = cosine_sim.round(2)

# convert kneighbors
one_minus = (1 - kneighbors[0]).round(2)

print(one_minus[0])
# [1.  , 0.2 , 0.15, 0.08, 0.08, 0.06, 0.06, 0.05, 0.03, 0.02]
</code>
The results from one_minus makes sense now they are in sorted descending sequence now of similarity scores as expected.
<code>print(cos_rounded[0])
# [1.  , 0.03, 0.08, 0.05, 0.15, 0.08, 0.02, 0.06, 0.06, 0.2 ]
</code>
Results from cos_rounded are as expected the scores are not sorted,
first row has perfect match with itself.
<code># compare set of values in first row of one_minus with first row of
cos_rounded

set(one_minus[0]) ==  set(cos_rounded[0])
</code>
Out: True
Only question is why does it happen that kneighbors returns 1 - real_cos_similarity?
Comment: As it's currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer).


Mongodb Text Search and ID field

Question: I want to search multiple fields in a collection for a phrase/string. I also need to filter the results to a single ID field.
For example, search the users collection for "john"
db.users.runCommand( "text", { search: "john" } )

When the document looks like:
{
    "_id" : ObjectId("525d5f3fa385ab082e8b4693"),
    "first_name" : "John",
    "last_name" : "Doe",
    "email" : "email@example.com",
    "account_id" : 1,
    "updated_at" : ISODate("2013-10-15T15:29:03.951Z"),
    "created_at" : ISODate("2013-10-15T15:29:03.951Z")
}

What's the best way to also filter the results to a specific ID, in the case above "account_id", so I'd only be searching/returning results where "john" appears and the account_id equals "1" for example?
Answer: I believe this is done using the filter option:
db.users.runCommand( "text", {
                                   search: "john",
                                   filter: { account_id : 1 }
                                 }
                       )

Or from the docs - http://docs.mongodb.org/manual/tutorial/limit-number-of-items-scanned-for-text-search/

Mongodb Text Search and ID field


Calculating portfolio volatility from portfolio returns vs. from covariance matrix

Question: I'm having trouble understanding the difference in calculating portfolio volatility via the portfolio returns vs. via the covariance matrix.
To be more specific: I understand that on the individual security level, volatility is calculated as the sample standard deviation of arbitrarily periodic returns over an arbitrary amount of time. Given that a portfolio has an NAV which can be treated as its total value, those same periodic returns can be calculated and a sample standard deviation can also be calculated over some time frame to produce its volatility as well.
However, I know this is not the method typically done for calculating portfolio volatility. Portfolio volatility is calculated with a covariance matrix and the weights associated with each of the securities/assets comprised within the portfolio.
I understand the math behind the covariance matrix and am not questioning its accuracy. What I don't fully understand is why the volatilities calculated via the portfolio's returns and the volatilities calculated via the covariance matrix differ as they do.
A coworker told me this was due to the small changes in the weights throughout the time period (since the returns for each of the securities/assets will increase/decrease values and adjust the weights until/unless rebalancing occurs). This, at a high level, makes sense to me. But I'm honestly unsatisfied with it and am hoping for a more proof-like answer.
Thanks in advance.
Comment: For what it's worth, your colleague is right.  The method based on the covariance matrix uses a single set of weights; the method based on the portfolio returns uses the actual weights as they change over time; in general, the two won't give the same answer unless the weights are unchanging over time.  You can formalize all that, but it's going to be a fairly dull exercise in manipulating variances and covariances, unlikely to yield any further insight.
Comment: IMHO, it's "more accurate" as a description of what actually happened, but then again it could be misleading if the weights changed a lot.  The covariance method is a "more accurate" representation of a forward-looking view that assumes the weights are fixed at current values—but only to the extent the covariance matrix is itself fixed.
Comment: Thanks Laconic. It seems to me, then, that the covariance matrix method uses stronger assumptions through assigning one vector of weights. If that's the case, does that mean that the volatility derived from the portfolio returns is theoretically more accurate as it encompasses all the weights as they change through time?
Comment: Ahh, that actually makes a lot more sense to me. This was seriously confusing me for a while - thank you very much!
Answer: Denote the vector of returns at time $t$ by $X_t$. Assume the bivariate process is stationary with a covariance matrix of $
\Sigma$. Call the vector of portfolio weights at time $t$ $w_t$. Then your portfolio return is $r_t = w_t'X_t$, and its variance is
$$
\operatorname{Var}(r_t) = \operatorname{Var}(w_t'X_t) = w_t'\Sigma w_t .
$$
This process has a time-varying variance if you're changing your weights from period to period, even though (you're assuming) the raw returns don't. This is what your colleague is referring to. If it has a time-varying variance, it doesn't make a sense to calculate the sample variance of it.
Also note that $(1)$ is assuming your process has the same cross-sectional covariance matrix at every time point, which may or may not be acceptable in practice (it probably isn't).
Comment: Thanks Taylor. I think I'm missing something due to my own lack of understanding - can you outline what you mean by a time-varying mean? And how can there be a time-varying mean if the initial bivariate process is assumed to be stationary? Furthermore, can this be interpreted as an argument in favor of calculating the portfolio volatility by disregarding the covariance matrix and just using the portfolio returns?
Comment: @Ringleader I apologize--I meant "time-varying variance" instead of "time-varying mean." If you assume the mean is zero, the mean of the overall portfolio wouldn't change. But the variance would because you're changing the weights (say you go back and forth between riskier and less risky portfolios.) Regarding your last question: no, this is an argument against it.
Comment: I think it's starting to click. Why wouldn't it make sense to calculate the sample variance of a time-varying variance? And how does the covariance matrix bypass this weakness?
Comment: @Ringleader does it make sense to estimate one variance parameter when there are  more than one variance parameters? Regarding your second question, that is something that you should ask on another question. It has an answer that's too long to fit into the comment section.


HTTPS Links behind AWS Load Balancer

Question: I really tried many Solutions to get my site work on Amazon AWS. There is only one last thing i can not solve. Some of my links are still "http://"-links and i dont know how to get them right.
My Setup:

Drupal 8.6.10 running on an EC2-Instance managed via Beanstalk
the Load-Balancer is managing the access and has two open ports (http/80 and https/443) directing to the ec2-instance via http (port 80)
the ssl-certificate is configured
my domain has the correct dns-config directing to xxx.eu-central-1.elasticbeanstalk.com
settings.php includes the following lines

     $settings['reverse_proxy'] = TRUE;
     $settings['reverse_proxy_addresses'] = array($_SERVER['PROXY_ADDRESS']);
     $settings['reverse_proxy_proto_header'] = 'https';
     conf['https'] = TRUE;
     $_SERVER['HTTPS']='on';

$_SERVER['PROXY_ADDRESS'] is assgned to 'xxx.eu-central-1.elasticbeanstalk.com' by beanstalk
.htaccess includes the following lines:

     RewriteCond %{HTTPS} off
     RewriteCond %{HTTP:X-Forwarded-Proto} !https
     RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

The affected places are:

in a twig: url('')
in a php-file: fileentities->url();

All other links are nicely rendered as 'https'-links. I would be really thankfull for any advice you could give me.
Best Regards,
FirstSanny
Comment: Set the base URL in settings.php. htaccess should also be forcing every request to https.
Comment: Already tried that. But i removed it, because of various posts saying this is no longer supported. But it didn't work anyway.
Comment: You need to debug the request headers your Drupal install actually is receiving. It's unlikely it is receiving a header named `https`, this is normally the value. To reduce the affected places use relative urls whenever possible, in Twig path() instead of url().
Comment: @4k4 thanks for your answer i'll first change my code the way you mentioned it and then i'll try to debug the headers even though it don't know exactly how. But thats something i can find out on my own.
Will get back here afterwards.
Comment: You can dump `\Drupal::request()->server->all()`
Comment: You may need to set it at the level of virtual hosts according to this document https://aws.amazon.com/premiumsupport/knowledge-center/redirect-http-https-elb/
Comment: @4k4 Thanks for your additional advice. i tried dumping all that. But the entries
["HTTP_X_FORWARDED_FOR"]=>
  string(12) "xx.xx.xx.xxx"
  ["HTTP_X_FORWARDED_PORT"]=>
  string(3) "443"
  ["HTTP_X_FORWARDED_PROTO"]=>
  string(5) "https"
suggest a right config, i would say. But changing everything to relative paths solved my imediate problem. So thank you for that hint.
Comment: Then you need `$settings['reverse_proxy_proto_header'] = 'HTTP_X_FORWARDED_PROTO';`
Comment: @4k4 That's what i thought aswell. I updated my config to `$settings['reverse_proxy_header'] = 'HTTP_X_FORWARDED_FOR';
  $settings['reverse_proxy_proto_header'] = 'HTTP_X_FORWARDED_PROTO';
  $settings['reverse_proxy_port_header'] = 'HTTP_X_FORWARDED_PORT';`
But that didn't changed anything.
Answer: Try the folling setup:
In settings.php:
if (PHP_SAPI !== 'cli' && !empty($_SERVER['REVERSE_PROXY_ADDRESSES'])) {

  $settings['reverse_proxy'] = TRUE;
  $settings['reverse_proxy_addresses'] = array($_SERVER['REVERSE_PROXY_ADDRESSES']);

  $settings['reverse_proxy_header'] = 'X-Forwarded-For';
  $settings['reverse_proxy_proto_header'] = 'X_Forwarded-Proto';
  $settings['reverse_proxy_port_header'] = 'X-Forwarded-Port';

  // HTTPS behind reverse-proxy
  if (isset($_SERVER['HTTP_X_FORWARDED_PROTO']) && $_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https') {
    $_SERVER['HTTPS'] = 'on';
    // This is hardcoded because there is no header specifying the original port.
    $_SERVER['SERVER_PORT'] = 443;
  }
}

In your .htaccess:
RewriteCond %{HTTP:X-Forwarded-Proto} !https
#RewriteCond %{HTTP_HOST} !^(.*\.)*localhost(.*)$ [NC]
RewriteRule .* https://%{HTTP:Host}%{REQUEST_URI} [L,R=permanent]

(change $_SERVER['REVERSE_PROXY_ADDRESSES'] to $_SERVER['PROXY_ADDRESS'] for your setup)
This worked fine for me.
Comment: Thanks for your reply. Sadly that didn't changed anything on my side. The effects stayed the same. Everything is https expect file->url() in php and url() in twigs.

HTTPS Links behind AWS Load Balancer


How do I print the output of a field as a link on node.tpl.php?

Question: How do I print the output of a field as a link on node.tpl.php? I'm able to print the raw output of a field using this:
<?php $display = array('label' => 'hidden');
$view = field_view_field('node', $node, 'field_podcast_url', $display);
print drupal_render($view);?><?php $display = array('label' => 'hidden');?>

The problem occurs when I try to make it as a link using this method:
<?php $display = array('label' => 'hidden');
$view = field_view_field('node', $node, 'field_podcast_url', $display); ?>
<a href="<?php print drupal_render($view); ?>">text

The output looks like this:
http://www.podtrac.com/pts/redirect.mp3/feeds.soundcloud.com/stream/211653270-scott-johnson-27-tms-846.mp3
">text
and the link points to this:
http://controldaily.dev/<div class=

I've looked around at solving this issue and can't find a solution. Anything else just shows nothing.
Answer: This looks like it's happening because you are trying to input the entire rendered field (including styling) into the links href.
As per the description:

The field_view_value() function can be used to output a single formatted field  value, without label or wrapping field markup.

To use field_view_field, try something like:
if($node->type == 'article')
{
    $link_field = field_view_field('node', $node, 'field_href');
    print l(t($link_field['#items'][0]['safe_value']), $link_field['#items'][0]['safe_value']);
}

This example assumes a node of article type, and a field with a machine name field_href. running it through l() (despite using the safe value) will help ensure it is sanitised properly before being output.
You can also take a look at the Link field module:

The link module can be count to the top 50 modules in Drupal
  installations and provides a standard custom content field for links.
  With this module links can be added easily to any content types and
  profiles and include advanced validating and different ways of storing
  internal or external links and URLs. It also supports additional link
  text title, site wide tokens for titles and title attributes, target
  attributes, css class attribution, static repeating values, input
  conversion, and many more.
Comment: This is a great start but I need to pass this through a media player so how do I print the output like ?
Comment: the l() function produces it for you? do you mean you need a specific id or class? there is no functional difference between manually creating the tag with  and the l() except l() sanitises the output first


Common eigenfunctions of 0 eigenvalue for the angular momentum components

Question: I need to demonstrate (considering the natural representation in $L^2(S^2)$) that if two non zero vectors in $H^2(S^2)$ satisfy $J_a\phi=0$$ \forall a \in {1,2,3}$ (where $J_a$ are the components of the angular momentum operator https://en.wikipedia.org/wiki/Angular_momentum_operator) then they are equal less than normalization and phase.
Hope I posed my problem comprehensibly, otherwise I apologize. Thank you.
Answer: For each one of your vectors, you immediately see that they are in the kernel of the quadratic Casimir of SO(3), so $\sum_a J_a J_a$. That is, they are both SO(3) singlets, and do not vary with angle on a sphere. They are constants. Scalars. You don't even need a similarity transformation to connect them. They are merely proportional.


Is it possible to enable remote errors for a single aspx page?

Question: I have a page that is blowing up and I am getting this error:

An application error occurred on the server. The current custom error settings for this application prevent the details of the application error from being viewed remotely (for security reasons). It could, however, be viewed by browsers running on the local server machine.

In this particular case I am unable to remote into the server. I am however able to access the wwwroot folder where pages are being hosted. The page I am trying to upload works on my test server but when I put it into production it is throwing the error. I suspect due to a missing namespace.
Is there anyway to enable remote errors for this one page or otherwise capture the exception detail? It is being thrown before Page_Load()
Answer: If you have access to the Event Log of the server, this is the place where unhandled exception details are usually written by ASP.NET.
Another possibility is to enable temporarily exception details in web.config to be shown remotely by setting:
<customErrors mode="Off" />

This will display the exception stacktrace instead of a YSOD so that you know what the problem is.
And yet another possibility is to subscribe to the Appliaction_Error method in Global.asax and log yourself the exception being thrown. By the way that's something that you should do anyways.
You may also checkout ELMAH.
Of course all this will be futile if your application is not configured properly in IIS (like for example errors in your web.config, incorrect ASP.NET version in the virtual directory, .NET Framework not installed on the remote server, ...). In this case you probably need access to the server.
Answer: Enabling remote errors is a very bad idea.  Proper error handling is critical to prevent unintentional information disclosure, which gives attackers helpful information.  It's worth reading up on.
There's plenty of guidance here: http://msdn.microsoft.com/en-us/library/w16865z6.aspx
There are some errors, however, that you can't code around. Loading an ASP.NET 4.0 website to a server that does not have the .NET Framework 4.0 installed, for example.  Your error handling won't work because the code just won't run.  If you suspect this is the case, you may need to check with the administrator to ensure that the right version is set up and IIS is configured to run your application using the correct target framework.
Answer: If you temporarily want's to enable remote errors for one .aspx page just add "Debug=true" to the page like this <%@ Page Language="C#" Debug="true" .... %>


An interesting problem with binary repetend

Question: Proof. Let $b$ is an odd number, Then $1/b$ in the binary loop section, the number of $1$ does not exceed the number of $0$.
for example:
$1/3=0.010101……$
$1/5=0.001100110011……$
$1/7=0.001001001……$
$1/9=0.000111000111……$
We know that satisfy the congruence expression $2^d≡1\ (mod \ b)$ of the Least positive integer $d$ , it's $d$ that $1/b$ of the repetend longth.
In particular, when $b-d=1$, the number of $0$ is equal to the number of $1$.
Comment: Very interesting! Please explain what is $d$
Comment: $1/3=0.\overline{01}, d=2. \\ 1/5=0.\overline{0011}, d=4. \\ 1/7=0.\overline{001}, d=3.$
Comment: There are many cases where the group of $d$ digits is after a group of digits after the $0.$ that DO NOT repeat. How do you deal with those?
Answer: Unfortunately, this does not seem to be true.
In binary, we have that
$$
  \frac{1}{187} = 0.\overline{0000001010111100111010110111011100011010},
$$
and the repeating part contains $19$ zeroes, and $21$ ones.
Comment: wow!There are 13 counter examples within 1000.
Answer: As Dylan's answer shows, this is not true.  Let me give a bit of an explanation for why it's not surprising that counterexamples are rare.  Note that if the repetend has length $d$, then it is just the binary expansion of $\frac{2^d-1}{b}$, since $\frac{1}{b}=\frac{(2^d-1)/b}{2^d-1}$.  Roughly speaking, you might expect that $\frac{2^d-1}{b}$ is an "average" number of its size.  This means that typically, it will contain about $\frac{1}{2}\log_2\frac{2^d-1}{b}\approx\frac{d}{2}-\frac{\log_2 b}{2}$ digits which are $1$.  Because of the $-\frac{\log_2 b}{2}$ term, this is less than $\frac{d}{2}$, so the repetend needs to have an unusually large number of $1$s in order to have more $1$s than $0$s.  To put it another way, the digits of the repetend must start with a bunch of $0$s (namely, $\log_2 b$ of them), and so the rest of repetend must have an unusual excess of $1$s to make up this deficit.
Moreover, there is a class of cases (which is fairly common at least for small $b$) in which exactly half of the repetend is $1$s.  Namely, if there exists $n$ such that $2^n=-1$ mod $b$, then $d=2n$ and the repetend consists of a binary string of length $n$ followed by the same binary string but with all of its digits flipped.  Indeed, the $k$th digit of $1/b$ is determined by whether the remainder of $2^{k-1}$ mod $b$ is greater than or less than $b/2$, and so increasing $k$ by $n$ will always flip the digit.


The assumptions in the derivation of normal distribution using a dart board example?

Question: The derivation could be found here:
https://math.stackexchange.com/questions/384893/how-was-the-normal-distribution-derived
or here https://www.youtube.com/watch?v=cTyPuZ9-JZ0
They assume that (1) the pdf f(x), f(y) are independent from each other and (2) the pdf of a point g(x,y) only depends on the distance of the point to the bull's eye.
There are definitely examples of pdfs that do not satisfy both (1) and (2). For example, imagine a discrete example where the dart landing at (2,1) or (1,2) both have a probability of 0.4 and the dart landing at (2,2) has a probability of 0.2. This satisfy the assumption (2) that the probability depends only on the distance but not (1). What kind of pdfs can satisfy both (1) and (2)?
Comment: You seem to be denying the analysis presented in your first link, which gives the answer at the line "so if I set $\tilde f\ldots$".  Could you explain what might be unclear or missing in that post?
Comment: This is broken down really well in this 3b1b video at 13:00 https://www.youtube.com/watch?v=cy8r7WSuT1I&t=2s
Answer: Joint pdfs $f_{X,Y}(x,y)$ that have circular symmetry about the origin (property 2 in the OP's description) exist in profusion, and they all have the property that if we think of $X$ and $Y$ in terms of $R=\sqrt{X^2+Y^2}$ and $\Theta = \arctan\left(\frac YX\right)$, then $R$ and $\Theta$ are independent random variables, and $\Theta \sim U[0,2\pi)$.  But, properties 1 (independent $X$ and $Y$) and 2 (circular symmetry about the origin) are harder to satisfy.
It is shown in Sheldon Ross's A First Course in Probability, Prentice-Hall (see Example 2e in Chapter 6.2) that, under the slightly more onerous constraints that the marginal pdfs $f_X(x)$ and $f_Y(y)$ are differentiable functions of $x$ and $y$ respectively and that $f_{X,Y}(x,y)$ is differentiable with respect to $x$ and also with respect to $y$, the only pdfs $f_X$ and $f_Y$ that satisfy both properties 1 and 2 are zero-mean Gaussian pdfs with identical variances.
Comment: Wow, thank you very much!!! That's a great proof. Could you tell me what kind of approach would you take when solving such problems? Would you think more algebraically or geometrically? It was kind of hard to think of geometry that the functions must have to satisfy both properties.


Does Hölder's Inequality implies that $\int_{-\infty}^\infty |f(t) \,g(t)|\,\mathrm dt \leq \int_{-\infty}^\infty |f(t)|\,\mathrm dt \, \sup |g|$?

Question: Does Hölder's Inequality implies that $\int_{-\infty}^\infty |f(t) \,g(t)|\,\mathrm dt \leq \int_{-\infty}^\infty |f(t)|\,\mathrm dt \sup_t |g(t)|$ ?
I am thinking in the case $p=1$ and $q=\infty$ so $1=\frac{1}{p}+\frac{1}{q}$ holds, but I am not sure if the norms $||\cdot||_\infty$ and $||\cdot||_1$ are well interpreted from what it is said on Wikipedia here and here, and also if the inequality is valid on unbounded domains as $(-\infty;\,\infty)$.
Also please this related question I had left here. Thanks you very much.
Answer: Yes, it works ... and the proof is quite immediate: just notice that $|f(t)\,g(t)|≤ (\sup_s{|g(s)|})\ |f(t)|$, and now since $\sup_s{|g(s)|}$ is just a constant independent of $t$, you can put it out of the integral ...
Comment: I do not understand your problem but let me add details then ... $C = \sup |g|$ is real. So since $|f(t)g(t)|≤ |f(t)|\,|g(t)| ≤ C\, |f(t)|$, we deduce
$$
∫ |f(t) g(t)|\,\mathrm d t \leq ∫ C |f(t)|\,\mathrm d t = C ∫ |f(t)|\,\mathrm d t
$$
At which step do you have a difficulty here?
Comment: Is not clear for me since they are under a integral and $g(t)$ could be of the form $e^{-iwt}$ so $|e^{-iwt}|=1$, and then a sum of complex modulated values will have magnitude always less than the sum of the unmodulated values magnitudes, ¿is that right?


Specify Dynamic IP of SSH server on the client side

Question: I am trying to set up a ssh server at home connected to internet via wireless router. Since its IP is dynamic, how do you specify its address on the client side? Thanks!
Answer: DynDNS (or similar service), combined with port-forwarding and a static local IP address on the client is your friend.
Answer: You mean dynamic internally, set by DHCP? You would need to assign your server a static IP instead of a DHCP-assigned address.
If you mean your home address is dynamic from the outside network, you can either try a service like dynamic DNS or do what I do; my leases are rather long so I just go to www.whatismyip.com and write it down and that lease will keep for weeks (or months) at a time, but it varies by provider.
Another thing you can rig up is a way to script an automatic grab of your external IP and email it to you periodically.
Your only other "easy" option I know of is to check the provider's offerings and see if they'll offer a static IP to your location, but usually that's for business class lines. Plus they frown on the possibility that you're going to run a web server or mail server.
Answer: I recommend you to use some web dynamic IP services, like dyndns.
it offers a free web IP detection tool for use with dyndns services, and it's supported by many ADSL routers.
After you need to do, is to enable NAT at your firewall and configure it
to forward port 20000 (For example) to your local server inside the NAT at the port of your sshd.
Try to change the standard ssh port to another port (20000) to get less ssh brute force attack.
Comment: +1 for mentioning port forwarding (I think)


Rails 3 migrations: Adding reference column?

Question: If I create a new rails 3 migration with (for example)
rails g migration tester title:tester user:references

, everything works fine...however if I add a column with something along the lines of:
rails g migration add_user_to_tester user:references

the reference field is not recognised. In short, the question is: how do I add a referencing column to a rails migration from the command line?
Answer: If you are using the Rails 4.x you can now generate migrations with references, like this:
rails generate migration AddUserRefToProducts user:references

like you can see on rails guides
Comment: how do you specify a column name for the foreign key instead the auto generated name?
Comment: See section 2.1 from http://edgeguides.rubyonrails.org/active_record_migrations.html for example.
Comment: @jwill you can use polymorphic: user:references{polymorphic}.
Comment: @PauloFidalgo Can you explain a bit about how to do that? may be some guide of links? (talking about polymorphic)
Comment: @Anwar: here are the docs http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/TableDefinition.html#method-i-references
Answer: EDIT: This is an outdated answer and should not be applied for Rails 4.x+
You don't need to add references when you can use an integer id to your referenced class.
I'd say the advantage of using references instead of a plain integer is that the model will be predefined with belongs_to and since the model is already created and will not be affected when you migrate something existing, the purpose is kind of lost.
So I would do like this instead:
rails g migration add_user_id_to_tester user_id:integer

And then manually add belongs_to :user in the Tester model
Comment: No, afaik Rails never create foreign key restraints on the database unless you add plugins to do it for you.
Comment: remember to add the index with user:integer:index
Comment: But that won't create the appropriate foreign key constraints on databases that support it, right?
Comment: Answer is dated, see @Paulo's answer for modern rails.
Comment: just studying this post, pls how do i add the reference after all
Comment: The way I showed does add a reference. But instead of it being a one line generator, it also requires you to add the reference code into your existing model, which is also included in the answer
Comment: This answer is really good, but I'm not sure I understand it fully due to the confusing grammar (multiple 'and' and 'since'. not sure where sentences end and start.)

"the model will be predefined with belongs_to and since the model is already created and will not be affected when you migrate something existing, the purpose is kind of lost."
Comment: What should i do if the user model is like, `Subscription::User` so how will i write the migration for this?
Comment: @abahgat with Rails > 4.2 there is now an ```add_foreign_key``` method which facilitates adding a fk constraint at the db level
Answer: Please note that you will most likely need an index on that column too.
class AddUserReferenceToTester < ActiveRecord::Migration
  def change
    add_column :testers, :user_id, :integer
    add_index  :testers, :user_id
  end
end

Comment: Why? Is this true for most belongs_to relationships?
Comment: The `rails g migration ...` generated `add_reference :installs, :device, index: true` which also creates the index.
Comment: It is indeed for performance reasons and comes in handy if you have a has_many/has_one on the other side of that belongs_to relation. If you are absolutely sure that you will not go through `user.testers` you can omit the index.
Answer: With the two previous steps stated above, you're still missing the foreign key constraint. This should work:
  class AddUserReferenceToTester < ActiveRecord::Migration
      def change
          add_column :testers, :user_id, :integer, references: :users
      end
  end

Comment: This is the only actual answer here. The foreign key is the most critical part here
Comment: this should be marked as the correct answer since the questions asks for rails 3
Answer: You can use references in a change migration. This is valid Rails 3.2.13 code:
class AddUserToTester < ActiveRecord::Migration
  def change
    change_table :testers do |t|
      t.references :user, index: true
    end
  end
  def down
    change_table :testers do |t|
      t.remove :user_id
    end
  end
end

c.f.: http://apidock.com/rails/ActiveRecord/ConnectionAdapters/SchemaStatements/change_table
Comment: change and down methods? aren't up and down methods instead?
Comment: @MaicolBen yes, and you can also just leave off the down method.
Comment: @MaicolBen Without the `down` method, I got `ActiveRecord::IrreversibleMigration` when rolling back using Rails 3.2. I also had to change `change` to `up`.
Answer: Running rails g migration AddUserRefToSponsors user:references will generate the following migration:
def change
  add_reference :sponsors, :user, index: true
end

Comment: Which version of Rails is this for?
Answer: When adding a column you need to make that column an integer and if possible stick with rails conventions. So for your case I am assuming you already have a Tester and User models, and testers and users tables.
To add the foreign key you need to create an integer column with the name user_id (convention):
add_column :tester, :user_id, :integer

Then add a belongs_to to the tester model:
class Tester < ActiveRecord::Base
  belongs_to :user
end

And you might also want to add an index for the foreign key (this is something the references already does for you):
add_index :tester, :user_id

Answer: That will do the trick:
rails g migration add_user_to_tester user_id:integer:index

Comment: I like that this also adds the index that you will most likely want.
Answer: You can add references to your model through command line in the following manner:
rails g migration add_column_to_tester user_id:integer

This will generate a migration file like :
class AddColumnToTesters < ActiveRecord::Migration
  def change
    add_column :testers, :user_id, :integer
  end
end

This works fine every time i use it..
Answer: For Rails 4
The generator accepts column type as references (also available as belongs_to).
This migration will create a user_id column and appropriate index:
$ rails g migration AddUserRefToProducts user:references

generates:
class AddUserRefToProducts < ActiveRecord::Migration
  def change
    add_reference :products, :user, index: true
  end
end

http://guides.rubyonrails.org/active_record_migrations.html#creating-a-standalone-migration
For Rails 3
Helper is called references (also available as belongs_to).
This migration will create a category_id column of the appropriate type. Note that you pass the model name, not the column name. Active Record adds the _id for you.
change_table :products do |t|
  t.references :category
end

If you have polymorphic belongs_to associations then references will add both of the columns required:
change_table :products do |t|
  t.references :attachment, :polymorphic => {:default => 'Photo'}
end

Will add an attachment_id column and a string attachment_type column with a default value of Photo.
http://guides.rubyonrails.org/v3.2.21/migrations.html#creating-a-standalone-migration


Should I create DLLs for modularity?

Question: I'm working on creating an app that will parse MSDN articles for meta-information such as the article title and other articles that are in the same collection. This app also has a GUI front-end.
I'm interested in making this program more modular by separating the front-end from the back-end and the back-end into two parts -- one that handles the retrieval and general parsing of an HTML-document and one that does more specific parsing related to MSDN itself. The idea being that this will allow custom UIs to be slapped onto the back-end and allow for the app to be able to parse other sites (perhaps Google search results) just by plugging-in a different DLL.
My understanding is that you would normally create DLLs to share code across different applications. However, in this case, I'm only looking for modularity for this one particular application. Is it still appropriate to create DLLs in this case? Or should I be considering a different option, such as just glomming all of the classes together into a single assembly?
I should note that I'm relatively new to programming, so even if the answer is "no", then this will still be a good exercise for me to learn from. If that's the case, I'd like to know if this exercise should be modified in anyway to make it more appropriate.
Answer: I think that your idea of using DLLs in this situation is a good idea.  There is no appreciable extra cost in using DLLs (possibly a bit more startup cost at the most).  And even if you are currently only planning on using them in this one application, it doesn't hurt to still plan for future changes.  There is always a good chance that the DLLs can be used with little or no modification in another application if needed.
In addition, splitting it into separate DLLs for modularity might even help with the design and development process.  It makes "sharing" global data more difficult (and that is probably a good thing).  If everything is in one monolithic assembly, there may be some tendency to just grab some data from some other place.  If that data lives in another assembly, then that potentially bad practice is less likely to happen.  It may force the developer into rethinking how to solve issues.
Comment: +1 for the second paragraph. Modularity really does force you to use a more clean and well thought-out design for your application.
Comment: +1 for 2nd paragraph. Anything you can do that forces you to solve problems properly rather than just grope some unrelated module is good.
Answer: DLLs can help with maintenance of your application.  If you have future updates and/or bug fixes, you can update specific DLLs as opposed to the entire application.  This will also help reduce your testing surface area.
During development it will also be a bit easier to stub out certain classes/DLLs and swap in the actual DLLs later once developed.
I wouldn't go crazy and put every class in its own DLL.  From a high-level there should be a clear grouping of classes that should be together.
Answer: There is several advantage to make a DLL:

the DLL can be changed independently from the caller program (front-end)
the DLL can be distributed by itself, independently from the caller problem (front-end)
if several simultaneously running software use the same DLL the memory footprint will be smaller

It also comes with a cost:

you can't use simple inheritance scheme to extend DLL classes
sharing global data is still possible but more difficult (some may say this force better design, but it's still a constraint)
unit testing of code included in a DLL is more difficult
there are installation issues (the dll must be put in special places and registered, depending how you use it)

But the main point regarding modularity is that DLL is still a half measure. Even if we ignore costs, it is not much better than a lib or even just reusing existing classes. The main advantages are for resellers, not really for software developers, except if you expect third party contributors. And the called functions are still running in the same process that the main program.
If you want a real clean cut for modularity you could use a real multi-tiers architecture with separate processes for front-end and back-end and a communication layer between them (say TCP sockets). That is usually both more versatile and more robust, and not much more complex when done early enough in project life.

Should I create DLLs for modularity?


GraphQLError [Object]: Syntax Error: Cannot parse the unexpected character ";"

Question: While I was setting the server, connecting to the MongoDB database, and doing a mutation, I'm getting the following error after running node index in my terminal: GraphQLError [Object]: Syntax Error: Cannot parse the unexpected character ";".
Index.JS:
const { ApolloServer } = require('apollo-server');
const gql = require('graphql-tag');

const typeDefs = gql`;
type Query{
    SayHi: String!
    }
`;

const resolvers = {
    Query: {
        sayHi: () => 'Hello World!'

        }
    };
const server = new ApolloServer({
    typeDefs,
    resolvers
});

server.listen({ port: 5000 }).then((res) => {
    console.log('Server running at ${res.url}');
})

How can I resolve this issue?
Comment: `const typeDefs = gql`; ` on line 4?
Comment: For some reason, I still haven't gotten the host after changing the line you said. I get "TypeError: Cannot read property 'some' of undefined" when I type node index in the terminal.
Answer: I encountered the same issue, and the error message says exactly what is going wrong. There shouldn't contain ; in the gql block. I resolved the issue by removing the ; character.


iTextSharp eliminando documento se não tiver páginas

Question: Bom, eu tenho uma aplicação que em determinado momento gera um PDF com todos os JPG's, porém, ao invés de escolher um JPG eu escolher um arquivo Excel ele só move esse arquivo.
Certo, eu gostaria que se o documento que eu criei com o iTextSharp (Doc.Open() entre outras propriedades) caso não tenha nenhuma página fosse deletado, que não fosse criado, alguma ideia de como fazer isso?
Já sei que para saber a quantidade de páginas é Doc.PageNumber
iTextSharp.text.Document Doc = new iTextSharp.text.Document(PageSize.A1, 20, 20, 20, 20);
            string PDFOutput = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop), "Output.pdf");
        PdfWriter writer = PdfWriter.GetInstance(Doc, new FileStream("C:/AgendaComp/Encaminhamento/" + txtNomePasta.Text + SID.ToString() + ".pdf", FileMode.Create, FileAccess.Write, FileShare.Read));

        Doc.Open();

        string Folder = @"C:/AgendaComp/Encaminhamento/" + txtNomePasta.Text + " arquivos/";
        foreach (string F in System.IO.Directory.GetFiles(Folder, "*.jpg"))
        {
            if (System.IO.File.Exists(F))
            {
                Doc.NewPage();

                //Doc.PageSize.Height = 1275;
                Doc.Add(new iTextSharp.text.Jpeg(new Uri(new FileInfo(F).FullName)));
            }
        }
        int n = Doc.PageNumber;
        if (n == 0)
        {
            Doc.Close();

        }
        else
        {
            Doc.Close();
            File.Move("C:/AgendaComp/Encaminhamento/" + txtNomePasta.Text + SID.ToString() + ".pdf", "C:/AgendaComp/Encaminhamento/" + DdataAviso.Year + "-" + DdataAviso.Month + "-" + DdataAviso.Day + "/" + txtNomePasta.Text + SID.ToString() + ".pdf");
        }

Comment: poste o código que você está usando ate agora. Provavelmente é possível contar o numero de paginas durante a criação do documento.
Comment: Eu já contei o numero de páginas . Só não consigo fazer com que o documento seja excluido ,ou não criado . Tipo , se ele não tiver nenhuma página , eu não quero que ele exista
Comment: posta o trecho do código que você está utilizando para criar o PDF
Comment: Uma opção é criar o PDF em memória.
Comment: Pronto , adicionei o código , como eu crio ele em memoria ?
Answer: Para colocar em memória mude as seguintes linhas:
PdfWriter writer = PdfWriter.GetInstance(Doc, new FileStream("C:/AgendaComp/Encaminhamento/" + txtNomePasta.Text + SID.ToString() + ".pdf", FileMode.Create, FileAccess.Write, FileShare.Read));

para
PdfWriter writer = PdfWriter.GetInstance(Doc, Response.OutputStream);

e após fechar o documento
 HttpResponse resp = this.Response;
 resp.ContentType = "application/pdf";


Most wins by a second place team in NBA history

Question: What's the most wins an NBA team finishing 2nd in wins/loss in a conference has achieved?
This came to mind because the Dodgers recently set a record in the MLB for the most wins without winning a division title. And given that the Warriors and Suns are currently neck-and-neck in the West at 24-5, I was just curious what the record was for 2nd best record in a conference for the NBA.
Answer: Probably the Spurs in 2016, who went 67-15... finishing 6 games behind the record setting Warriors (73-9).  That is tied for 7th all-time by winning percentage for any team.


Two dice problem

Question: Two players are throwing each one die. The one that has higher value, receives a number of points equal to the difference of the values on both dice.
How do you estimate probability for a winner to collect more than 100 points in 100 throws?
Comment: Is this a homework problem? Sure sounds like it, so, it should have the self-study tag
Comment: Ivana - the point with the CLT is that you avoid the $n^{th}$ order convolution. Can you work out the probability that a given person (without knowing they won or lost overall) collects > 100 points in 100 throws? More basically, but more to the point, can you work out the (approximate) distribution of their total score? Failing that, can you work out the mean and variance of the distribution of the number of points they gain from a single throw?
Comment: @COOLSerdash If you treat the pair of scores as bivariate normal, could you work out $P(\text{both players score} < 100.5)$?
Comment: The answer--if I'm correct this time ;-)--will be around 0.713.
Comment: It's not too difficult to also work out the covariance of the per-round score of the two participants, and hence of the total score. From there, it's not difficult to compute the relevant probability - but there is probably an easier way to do it than that.
Comment: I know I'm supposed to avoid $n^{th}$ convolution. I also know how to work out that one person will win more than 100 points in 100 throws. I have trouble when it comes to two players and distribution of value differences.
Comment: With that in mind, the bivariate distribution of the winner's score against the losers [looks like this](http://i.imgur.com/9ndB8Ee.png), where probability of interest is the proportion of the distribution above the pink line. The complement - the proportion below it, relates to my previous comment (where you don't differentiate winner from loser). You may not be able to compute it algebraically, but many programs will compute the bivariate cdf for you.
Comment: I was hoping for help with method more than the final number...
Comment: I was wrong anyway. :-(.  How accurate an estimate do you need?  Are you familiar with the Central Limit Theorem?
Comment: @whuber I'm very unsure about this, but I'm getting an estimate of $0.3998819$. Is that realistic?
Comment: @COOL Yes, that's realistic and it's related to this problem, but it's not the answer.  You have accurately computed the chance that a *given* player will collect more than 100 points in 100 throws. The question asks for the chance that the *winner* will collect more than 100 points.
Comment: Yes, I am familiar with CLT. I assumed I would have to get convolution for the difference of values, but I don't know how since distribution for die is discrete.
Comment: @whuber I have the same answer as you.
Comment: I'm posting some simulation code to validate your answers.
Comment: @Glen_b Thanks for your comments. I have managed to calculate the probability that a given player will collect more than 100 point in 100 throws, the expectation and variance of the score of a single throw and the covariance of the per-round score of the two participants. I just don't know how to proceed from here.
Comment: In essence we're just exploiting the symmetry of the bivariate cdf. But as I said earlier, likely there's a simpler way. Indeed, I can see ways to exploit the symmetry further, but I expect there are ways to do it even more easily.
Answer: Begin with an array of the possible (equiprobable) outcomes of the two dice, with my die indexing the rows and your die indexing the columns.  In this table, the entries are the payoffs to each of us: the first is my payoff $X$ and the second is your payoff $Y$.
$$\begin{array}{c|cccccc}
  & 1 & 2 & 3 & 4 & 5 & 6 \\
\hline
 1 & \left(
\begin{array}{cc}
 0 & 0
\end{array}
\right) & \left(
\begin{array}{cc}
 0 & 1
\end{array}
\right) & \left(
\begin{array}{cc}
 0 & 2
\end{array}
\right) & \left(
\begin{array}{cc}
 0 & 3
\end{array}
\right) & \left(
\begin{array}{cc}
 0 & 4
\end{array}
\right) & \left(
\begin{array}{cc}
 0 & 5
\end{array}
\right) \\
 2 & \left(
\begin{array}{cc}
 1 & 0
\end{array}
\right) & \left(
\begin{array}{cc}
 0 & 0
\end{array}
\right) & \left(
\begin{array}{cc}
 0 & 1
\end{array}
\right) & \left(
\begin{array}{cc}
 0 & 2
\end{array}
\right) & \left(
\begin{array}{cc}
 0 & 3
\end{array}
\right) & \left(
\begin{array}{cc}
 0 & 4
\end{array}
\right) \\
 3 & \left(
\begin{array}{cc}
 2 & 0
\end{array}
\right) & \left(
\begin{array}{cc}
 1 & 0
\end{array}
\right) & \left(
\begin{array}{cc}
 0 & 0
\end{array}
\right) & \left(
\begin{array}{cc}
 0 & 1
\end{array}
\right) & \left(
\begin{array}{cc}
 0 & 2
\end{array}
\right) & \left(
\begin{array}{cc}
 0 & 3
\end{array}
\right) \\
 4 & \left(
\begin{array}{cc}
 3 & 0
\end{array}
\right) & \left(
\begin{array}{cc}
 2 & 0
\end{array}
\right) & \left(
\begin{array}{cc}
 1 & 0
\end{array}
\right) & \left(
\begin{array}{cc}
 0 & 0
\end{array}
\right) & \left(
\begin{array}{cc}
 0 & 1
\end{array}
\right) & \left(
\begin{array}{cc}
 0 & 2
\end{array}
\right) \\
 5 & \left(
\begin{array}{cc}
 4 & 0
\end{array}
\right) & \left(
\begin{array}{cc}
 3 & 0
\end{array}
\right) & \left(
\begin{array}{cc}
 2 & 0
\end{array}
\right) & \left(
\begin{array}{cc}
 1 & 0
\end{array}
\right) & \left(
\begin{array}{cc}
 0 & 0
\end{array}
\right) & \left(
\begin{array}{cc}
 0 & 1
\end{array}
\right) \\
 6 & \left(
\begin{array}{cc}
 5 & 0
\end{array}
\right) & \left(
\begin{array}{cc}
 4 & 0
\end{array}
\right) & \left(
\begin{array}{cc}
 3 & 0
\end{array}
\right) & \left(
\begin{array}{cc}
 2 & 0
\end{array}
\right) & \left(
\begin{array}{cc}
 1 & 0
\end{array}
\right) & \left(
\begin{array}{cc}
 0 & 0
\end{array}
\right)
\end{array}$$
There are only eleven combinations of payoffs $(X,Y)$.  By counting them up, we can  make a table of them and their probabilities.  To simplify the presentation, the next table expresses all probabilities as parts per $36$, so that for example $5$ represents a probability of $5/36$:
$$\begin{array}{l|llllll}
  & 0 & 1 & 2 & 3 & 4 & 5 \\
\hline
 0 & 6 & 5 & 4 & 3 & 2 & 1 \\
 1 & 5 & 0 & 0 & 0 & 0 & 0 \\
 2 & 4 & 0 & 0 & 0 & 0 & 0 \\
 3 & 3 & 0 & 0 & 0 & 0 & 0 \\
 4 & 2 & 0 & 0 & 0 & 0 & 0 \\
 5 & 1 & 0 & 0 & 0 & 0 & 0
\end{array}$$
Applying the (elementary) definitions of mean, variance, and correlation to this tabulated distribution of $(X,Y)$, we find that the common mean of $X$ and $Y$ is $35/36$, their common variance is $2555/1296$ (almost $2$), and their correlation is $-35/73$.  (The strong negative correlation makes this question a little trickier than one might expect.)
By virtue of the basic additive properties of means and variances of independent random variables, our payoffs after $100$ independent plays therefore have a joint distribution with means $3500/36$, variances $255500/1296$, and the same correlation $-35/73$.  Let's approximate this by a bivariate normal distribution having those parameters.  We can readily visualize it: it must be centered around $97$ (which is about $100$ times $35/36$) on both axes, have spreads around $\sqrt{255500/1296}\approx 14$, and have good negative correlation.

In this contour plot of the bivariate normal density, the region where $\text{round}(\max(X,Y)) \gt 100$ is shaded.  Its integral approximates the answer to the problem.  To the eye, the missing integral looks like about a quarter of the total, giving a rough preliminary estimate of $1 - 1/4 = 0.75$.
This much was completely elementary and can be done with mental arithmetic.  Numeric integration over the shaded region (from 100.5 out to infinity in both variables) yields $0.723$ as a possibly more accurate approximation; a computer helps with this calculation ;-).  The value must be a little high because the positive skewness of the distributions of $X$ and $Y$ will translate to positive skewness in the marginal distribution of both our scores after $100$ rounds, pulling a little of the probability out of the unshaded area of the figure into the shaded area.

For reference, here is an exact solution.  The $100$ outcomes in this game can be obtained sequentially as follows:

Determine the number of times we tied.  From the tables it is clear the chance of a tie is $6/36 = 1/6$.  This number therefore has a Binomial$(100, 1/6)$ distribution.
Conditional on the number of ties $k$, the number of times I got a positive payoff, $j$, has a Binomial$(100-k, 1/2)$ distribution.  The number of times you got a positive payoff equals $100-k-j$.
Conditional on $j$, the distribution of my payoffs is $1$ with $5/15$ chance, $2$ with $4/15$ chance, ..., and $5$ with $1/15$ chance.  Its probability mass generation function (pmf) is $p(x) = (5x + 4x^2 + 3x^3 + 2x^4 + x^5)/15$.

The probability that my payoff is $100$ or less therefore equals the polynomial of degree $100$,
$$p(x)^j \mod x^{101},$$
evaluated at $x=1$.  That's a reasonably fast calculation.  Replacing $j$ by $100-k-j$ gives the probability that your payoff also is $100$ or less.  The product of these two probabilities is the chance, conditional on $k$ and $j$, that we both had payoffs of $100$ or less.  Therefore, computing the double sum over $k$ and then $j$, we can obtain the chance that the maximum does not exceed $100$.  Subtracting that from $1$ gives the answer, equal approximately to
$$0.7131559594164915144059723.$$
We see the bivariate normal approximation was about $1.4$% too high.
Edit
For those who like details, these might help:

Define $t(j,m)$ to be the chance of a payoff of $m$ or less conditional on having received exactly $j$ positive payoffs. Its pmf $q(x;j,m)$ is the sum of all monomials of degrees up to and including $m$ in the expansion $p(x)^j$, the $j^\text{th}$ power of the fifth degree polynomial $p$.  Therefore $t(j,m) = q(1;j,m)$.  Using Mathematica I computed it as
(Expand[((5x + 4x^2 + 3x^3 + 2x^4 + x^5)/15)^j] + O[x]^(m+1) // Normal) /. {x -> 1}

Conditional on $k$, the chance that both of us score $m$ points or less when there is some positive score in $100-k$ throws is obtained by summing over all the possibilities:
$$r(k,m) = \sum_{j=0}^{100-k} \binom{100-k}{j}2^{k-100}\ t(j,m)\ t(100-k-j,m).$$
Summing these over all possibilities of $k$ gives the answer,
$$\sum_{k=0}^{100} \binom{100}{k}\left(\frac{1}{6}\right)^k\left(\frac{5}{6}\right)^{100-k} r(k,100).$$

Reality check
It's easy to make mistakes with calculations like these--I often do.  As a check, let's simulate the 100 throws a million times and track the winner's score.  This R code will do it in less than a half minute:
set.seed(17)
s <- replicate(10^6, {
  delta <- floor(runif(100, 0, 6)) - floor(runif(100, 0, 6));
  max(sum(delta[delta > 0]), -sum(delta[delta < 0]));
})
sum(s > 100) / length(s)

The first line within the replicate function simulates the payoffs--positive for me and negative for you--and then the second line calculates which payoff was greater in size.  The result is an array of a million winners' payoffs.  The final line reports what fraction of those exceeded $100$.
The result, $0.713436$, is only $0.62$ standard errors greater than the exact result previously obtained, providing support for the correctness of this answer.
Comment: Fantastic - a clear, detailed answer, covering all the bases - but also a very beautiful answer, wonderful tables and diagrams. A pleasure to read.
Comment: @whuber Thanks again for this wonderful answer. Forgive my ignorance, but I'm having trouble to understand the exact part. How exactly can we sum over $k$ and $j$? And what do we sum up, the polynomial at $x=1$? Does $p(x)^j$ mean that we take the whole polynomial to the $j$th power?
Comment: @COOL I added some equations for you.  I hope there aren't too many typos in them :-).
Comment: @whuber I'm really sorry to bug you further with this, but the expression for which you provided the *Mathematica* code seems to be 1 whenever $x=1$, irrespective of the values of $j$ and $m$. What am I missing here? Another thing I can't reproduce is the numerical integration: When I use the following Mathematica code, the result is 0.0924: `NIntegrate[Integrate[PDF[MultinormalDistribution[{3500/36,3500/36},{{(255500/1296),(255500/1296)*(-35/73)},{(255500/1296)*(-35/73),(255500/1296)}}],{x, y}],{x, 100.5,Infinity}],{y,100.5,Infinity}]`.
Comment: @COOL You need to give `m` and `j` actual values, rather than symbolic ones, in order to have that *Mathematica* expression properly evaluated.
Answer: If you want to check the solutions given above using a simulation study, here is some useful code for it:
B = 100000
counter = 0

for(i in 1:B){

    player1 = sample(1:6,100,replace=TRUE)
    player2 = sample(1:6,100,replace=TRUE)

    dice.diff = abs(player1 - player2)

    player1.wins = ifelse(player1>player2,TRUE,FALSE)
    player2.wins = ifelse(player2>player1,TRUE,FALSE)

    player1.score = sum(dice.diff[player1.wins])
    player2.score = sum(dice.diff[player2.wins])

    if(player1.score > 100 | player2.score > 100){
        counter = counter + 1
    }

}

prob = counter/B

After running the code we obtain the following probability:
> prob
[1] 0.71306

Comment: (1) For a 4X faster simulation use `s <- replicate(10^5, {
  delta <- floor(runif(100, 0, 6)) - floor(runif(100, 0, 6));
  max(sum(delta[delta > 0]), -sum(delta[delta < 0]))
}); sum(s > 100)/ length(s)`. (2) I believe the answer is 19024559499426570037611458829573021458074826188755762634838975523692331755102046755611268880392383888911950488196749907935871387917415599926170449751285047 / 26676576488251712549810930719923358271436796584655589343467442433444558096851233688692008708797318581050800966667258114434654077532349702649000974494990336, approximately equal to $0.713155959.$

Two dice problem


Proving that $\max(x_1, x_2, x_3) = x_1 + x_2 + x_3 - \min(x_1, x_2) - \min(x_1, x_3) - \min(x_2, x_3) + \min(x_1, x_2, x_3)$

Question: $$\max(x_1, x_2, x_3) = x_1 + x_2 + x_3 - \min(x_1, x_2) - \min(x_1, x_3) - \min(x_2, x_3) + \min(x_1, x_2, x_3)$$
Is there a more elegant proof to this than just trying out all the possibilities and showing that it works?  For example, assuming that the numbers are distinct, this statement is true if
$x_1 < x_2 < x_3$ since the equation would yield $x_1 + x_2 + x_3 - x_1 - x_1 - x_2 + x_1 = x_3$, etc.
It also seems to work if the numbers are not distinct, but then there are even more combinations to try.  Also, interestingly, it looks like inclusion-exclusion but that might be a red herring.  Also not sure how to tag this question...
Comment: WLOG, assume $x_1 \geq x_2 \geq x_3$
Answer: It is essentially inclusion-exclusion.  $x_1+x_2+x_3$ counts each number once.  $-\min(x_1, x_2)-\min(x_1,x_3)-\min(x_2,x_3)$ removes the smallest element twice and the second smallest number once.  This has now under-counted the smallest number (and removed the second smallest).  Then $+\min(x_1, x_2, x_3)$ puts back the under-counted smallest number.
Answer: It really is inclusion/exclusion.  For each number $x_i$, define $X_i$ to be the interval $[0,x_i]$.  Then for any set $I$ of indices, $|\cup_{i\in I} X_i|=\max_{i\in I} x_i$ and $|\cap_{i\in I}X_i|=\min_{i\in I}x_i$. So inclusion/exclusion translates directly to the formula in the problem.
Answer: For $x_1,\ldots,x_n$ distinct,
$$
\max\{x_1,\ldots,x_n\}=\sum\limits_{i=1}^{n}x_i\prod\limits_{j\ne i}(1-{\bf 1}_{x_j>x_i})
$$

For completeness, more generally,
$$
\max\{x_1,\ldots,x_n\}=\sum\limits_{i=1}^{n}x_i{\bf 1}_{x_i\geqslant x_{j\ne i}}-\sum\limits_{i<j}x_i{\bf 1}_{x_i\geqslant x_{k\ne i,j}, x_i=x_j}+\ldots +(-)^{n+1}x_1{\bf 1}_{x_1=\ldots=x_n}
$$
Answer: A formula that might be of use to you is the following:
$$
\max(a,b) = \frac{a + b + |b-a|}{2}
$$
If you don't want to do case by case, you can apply this formula repeatedly, although it is true that the algebra does get voluminous.


How well do Convolutional neural networks in other image domains?

Question: I was recently trying out caffe and learning about CNN. So far I have seen that the model used by Krizhevsky performs really well in natural images. However I wanted to know how these models or CNN based models would behave in domains such as biological images or medical images.
Answer: Yes, they do. See the following links:

http://people.idsia.ch/~juergen/nips2012.pdf
http://blog.kaggle.com/2014/04/18/winning-the-galaxy-challenge-with-convnets/


Delete em sql - Inner Join

Question: Como é a sintaxe de uma query usando Inner Join delete em sql, para comparar código de duas tabelas e excluir ser for igual?
Tabela itemVenda. na tabela itemvenda foi registrado uma venda cuja, o código é 4. Para executar o estorno, se for o caso, o usuário pode remover.

Tabela controleCaixa. Quando estornar, o registro da tabela itemvenda será removido e ao mesmo tempo será removido, o registro cujo codDoc = 4. Este código do documento é gerado quando a venda é finalizada.
Comment: utilize um sub-select
Comment: poste a estrutura das suas tabelas e a query que precisa para te ajudarmos
Comment: Segue a estrutura da tabela. Preciso de uma query usando Inner Join delete em sql, para comparar código de duas tabelas e excluir ser for igual.
Comment: @JosueSantos: o gerenciador de banco de dados é MySQL ou SQL Server? As duas tags estão definidas. // É para estornar somente um item da venda ou toda a venda? // Além das tabelas `itemVenda` e `controleCaixa`, há a tabela `venda`?
Comment: José, existe a tabela venda também. Neste caso consegui resolver estornando um item da venda e excluir, o registro do controle do caixa deixando, só o registro na tabela venda. Resolvi com a ajuda do Rovann abaixo. Obrigado!
Answer: Na sua situação, o recomendado seria uma FK com Cascade on delete, o banco irá se encarregar de fazer isso.
Mas ainda assim se não for utilizar os recursos do banco, no comando, imagino quer você irá executar algo do tipo:
<code>"Delete from Vendas where Codigo = 4;"
</code>
Sendo assim, deveria executar os seguintes comandos:
<code>Delete from controleCaixa where codDoc = 4;
Delete from itemVenda where venda =4;
Delete from venda where codigo = 4;
</code>
todos na mesma transação.
Comment: Rovann, neste caso consegui resolver estornando um item da venda e excluir, o registro do controle do caixa deixando, só o registro na tabela venda. Resolvi utilizando Delete from controleCaixa where codDoc = 4;  Obrigado!

Delete em sql - Inner Join


Pass variable to field template from theme_preprocess_field()

Question: I'm stuck trying to pass variables from my theme_preprocess_field function to my template file. I want to have a variable for each of the fields I am printing out to my template.
I'm using the Paragraphs module in Drupal 7, so that might be throwing me off, but I feel like I'm really close. Here's a simple example:
Preprocess function in my template.php:
function merge_preprocess_field(&$variables) {
    $element = $variables['element'];
  if (isset($element['#field_name'])) {
    if ($element['#field_name'] == 'field_title') {
      $variables['test_title'] = $variables['items'][0]['#markup'];
    }
  }
}

The name of the template I am using is paragraphs-item--full-text.tpl.php.
field_title is the name of a field inside a paragraphs bundle called full-text.
Everything that I have read so far indicates that I should be able to access the $variables['test_title'] variable in my template this way:
<?php print render($test_title); ?>

It doesn't print, even after clearing the cache. It's even showing up in the array when I krumo($variables);. Am I missing something?
Comment: Is your theme called "merge"? If not then your function is not named properly.
Comment: Yes the name of my theme is merge. Is it possible that it would only show up in the field--field-title.tpl.php template?
Answer: Ok, I figured this out. My preprocess function was for a field, but I was trying to access the variable in the wrong template. What I ended up doing was creating individual preprocess functions for each paragraphs-item, and preprocessing the variables in there.
First, I created a function that would allow me to create preprocess functions for entity types:
function merge_preprocess_entity(&$vars, $hook) {
  $function = 'merge_preprocess_' . $vars['entity_type'];
  if (function_exists($function)) {
    $function($vars, $hook);
  }
}

This allowed me to create a preprocess function for paragraphs items. Then I used this same idea to create preprocess functions for individual paragraphs items:
function merge_preprocess_paragraphs_item(&$vars, $hook) {
  $elements = $vars['elements'];

  $function = 'merge_preprocess_paragraphs_item__' . $elements['#bundle'];
  if (function_exists($function)) {
    $function($vars, $hook);
  }
}

Now I can set variables that are available in the paragraphs items templates, like so:
function merge_preprocess_paragraphs_item__smaller_full_width_text(&$vars, $hook) {

  if ($vars['content']['field_cta']) {

    $field_cta = !empty($vars['content']['field_cta'])? $vars['content']['field_cta'][0]['entity']['field_collection_item']: '';

    foreach ($field_cta as $field_cta) {

        // Make the CTA url available to the rest of the block
        $vars['content']['cta_url'] = $field_cta['field_link'][0]['#markup'];

        // Set character limit for CTA Text
        $vars['content']['cta_text'] = truncate_utf8($field_cta['field_link_text'][0]['#markup'], 24, TRUE, FALSE);

    }
  }
}


Como mudar o texto do elemento título?

Question: Segue código:

$(document).ready(function() {
  $('#1ano_card_title').text('R$20,00');
});

  R$10,00
  / 1 ano

Já tentei dessa forma:
$('#1ano_card_title').text('R$20,00');

Porem o elemento small some. Como faço para mudar somente o valor ? Mudar de 10,00 para 20,00.
Answer: Da forma como você está fazendo você está substituindo todo o conteúdo, segue abaixo um exemplo alterando só o o primeiro bloco de texto.

$(document).ready(function() {
  $('#1ano_card_title').contents().first()[0].textContent='R$20,00';
});

  R$10,00
   / 1 ano

Answer: Você pode separar o valor que deseja alterar colocando dentro de um  e alterar o  ao invés do
:

$('#valor').text('R$20,00');

    R$10,00
    / 1 ano

Answer: Você pode usar JavaScript puro para mudar o valor, veja:

document.getElementById('1ano_card_title').childNodes[0].nodeValue = "R$20,00";

  R$10,00
  / 1 ano

Tirei esta solução daqui.
Answer: Você pode adicionar um spam dentro do h2

$('.btn').on('click', function(){
  console.log('clique')
  $('span.valor').text('R$ 20,00');

});
<link href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" rel="stylesheet"/>

  R$ 10,00
  / 1 ano

<button class="btn btn-primary">Alterar</button>
Answer: Você pode armazerar o conteúdo atual do elemento  em uma variável e depois de alterar o conteúdo do
 adicionando a variável.

var ano = $("small")[0];
$('#1ano_card_title').text('R$20,00').append(ano);

    R$10,00
    / 1 ano

Answer: HTML

Texto antigo

< button id="mudar">Mudar Texto < /button>

jQuery
$(document).ready(function(){
  var msg = $("Texto Novo");
  var botao = $("#mudar");
  botao.on("click", function(event){
    event.preventDefault();
    $("#infoMsg").text(msg.val());
  });
});

A tag h1 à qual o título será mudado recebe a id="infoMsg" e o botão que executará o evento de mudança recebe o id="mudar".
No Jquery a variável "msg" recebe um determinado valor (texto novo que substituirá o antigo contido na h1 de id="infoMsg") a variável botao vai receber o id contido no button id="mudar".
Por ultimo, uma função que acontecerá no momento do clique no button onde eu informo que, a id $("#infoMsg") terá o seu texto mudado pelo valor contido na variável msg msg.val().

Como mudar o texto do elemento título?


PostgreSQL counting from a special date

Question: i have the following code,
SELECT
    years_month_count.day_date,
    years_month_count.year_date,
    years_month_count.month_date,
    years_month_count.no_of_customers_day,
    sum(years_month_count.no_of_customers_day) OVER (PARTITION BY year_date ORDER BY day_date) AS no_of_customers_ytd
FROM (
    SELECT
        DATE(date) as day_date,
        DATE_PART('year',date) as year_date,
        DATE_PART('month',date) as month_date,
        count(prepare_first_buyer.person_id) as no_of_customers_day
    FROM (
        SELECT
            DATE(bestelldatum),
            person_id,
            ROW_NUMBER() OVER (PARTITION BY person_id ORDER BY person_id)
        FROM ani.bestellung
    ) prepare_first_buyer
    WHERE row_number=1
    GROUP BY DATE(date), DATE_PART('year',date),DATE_PART('month',date)
    ORDER BY DATE(date), DATE_PART('year',date),DATE_PART('month',date)
) years_month_count

the Output looks like this:

day_date
year_date
month_date
no_of_customers_day
no_of_Customers_ytd

2017-04-04
2017
4
6
6

2017-04-05
2017
4
4
10

...
...
...
...
...

...
...
...
...
...

and so on.
The no_of_customers_ytd will be set to zero at the beginning of every new year (1.January).
But what i need is to set it to zero on a special date, lets say 1.June every year.
So i need a sum from everything between the 1.June till 30.March for every Year.
Thx for the help.
Comment: Please provide table descriptions (DDL) and sample data as text - **no images** ( or a [fiddle](https://dbfiddle.uk/)) and the expected output of that data.
Answer: step-by-step demo:db<>fiddle
SELECT
    *,
    SUM(value) OVER (PARTITION BY                             -- 4
        date_part('year',                                     -- 3
            the_date - interval '5 months'                    -- 2
        )
    )
FROM t
WHERE date_part('month', the_date)::int NOT BETWEEN 4 AND 5   -- 1

Filter all dates you are not requiring. In your example all dates with months 4 and 5
Shift your date range start to the beginning of the year. In your example you have to shift: year-06-01 to year-01-01, so you need to subtract 5 months. Because your date range never exceeds a year, all your relevant data now has the same year, which makes a great group criterion
Extract the year part to use it as group/partition criterion
Do your calculation on this criterion
Comment: Thx for the help and the explicit answer. But i dont understand 2 things. The 1. why do i not require those months? And the_date how is this defined?
Comment: 1. Your requirement was June until March. So you don't need April and May. Of course you can adjust the filter to your actual requirements 2. As you can see in the provided fiddle, the_date represents the date column
Comment: @hahakomisch thanks for accepting. Please upvote as well, I'd appreciate :)

PostgreSQL counting from a special date


Providing motivation for the importance of the concept of a 'basis'.

Question: In a few situations, I found myself being asked by younger students why the concepts of a basis was important.
First, in the concept of linear spaces, it's easy to explain that having a basis allows us to describe the spaces with a relatively small set and makes working with linear transformation a much easier task (there are many more reasons, but this was an introductory linear algebra course).
When I was asked the same question in the context of propositional logic (a basis here is a maximally independent set of formulas), I found it a bit more difficult to motivate the definition.
In general, when asked "What is the importance of a basis?", what do you reply?
Comment: The most basic question in linear algebra is "describe the general solution to $Ax=b$." Doing that amounts to writing $x=x_0+\sum_{i=1}^n c_i k_i$, where $c_i$ are arbitrary scalars and $k_i$ make a basis for the kernel of $A$. Each choice of the $c_i$ gives a different solution. So the meaningful information here is the particular solution $x_0$ and the basis $k_i$ for the kernel. They already know this fact before they get to vector spaces in the first place; it is just formal language for something they already understand.
Comment: I think putting everything in this framework will help them, especially if they cannot easily understand why a statement like "a linear transformation is determined by its action on a basis" is useful.
Answer: In every context, a basis is essentially a (not the) smallest set you need to understand, in order to understand the whole object in question.
For example, a linear transformation is determined by its action on  a basis. Similarly, a complete theory is determined by which elements of a (propositional) basis it contains.
The virtue of the word "smallest" in the first paragraph is that, roughly speaking, every behavior over a basis is possible. For example, if $\{b_i\}$ ($i\in I$) is a basis for $V$ and $w_i$ $(i\in I)$ are vectors in $W$, then there is a linear transformation $T$ sending $b_i$ to $w_i$ (and by the above paragraph, it's unique). Similarly, given a propositional basis $p_i$ $(i\in I$) and a map $t: I\rightarrow \{\top, \bot\}$, there is a complete theory $T$ such that $p_i\in T\iff t(i)=\top$ (and by the above, it's unique).
These two aspects of bases (determination and freedom) mean that they're invaluable for understanding the things that act (in a very loose sense) on them - linear transformations, or complete theories, or etc.


Bootstrap vs Monte Carlo, error estimation

Question: I'm reading the article Error propagation by the Monte Carlo method in geochemical calculations, Anderson (1976) and there's something I don't quite understand.
Consider some measured data $\{A\pm\sigma_A, B\pm\sigma_B, C\pm\sigma_C\}$ and a program that processes it and returns a given value. In the article, this program is used to first obtain the best value using the means of the data (ie: $\{A, B, C\}$).
The author then uses a Monte Carlo method to assign an uncertainty to this best value, by varying the input parameters within their uncertainty limits (given by a Gaussian distribution with means $\{A, B, C\}$ and standard deviations $\{\sigma_A, \sigma_B, \sigma_C\}$) before feeding them to the program. This is illustrated in the figure below:

(Copyright: ScienceDirect)
where the uncertainty can be obtained from the final $Z$ distribution.
What would happen if, instead of this Monte Carlo method, I applied a bootstrap method? Something like this:

This is: instead of varying the data within their uncertainties before feeding it to the program, I sample with replacement from them.
What are the differences between these two methods in this case? What caveats should I be aware of before applying any of them?

I'm aware of this question Bootstrap, Monte Carlo, but it doesn't quite solve my doubt since, in this case, the data contains assigned uncertainties.
Comment: Just to clarify: the "random change" in the MC method is randomly generated by the researcher? That is, noise/errors are being artificially added to the input data?
Comment: It is "randomly generated", based on the uncertainties of the measured data (ie: the $\sigma$s) and assuming a certain distribution for these errors (usually Gaussian). So no, errors are not artificially added. The input data has an associated error given by the measuring process.
Comment: I don't think I understand. That is artificial noise, but with a standard deviation estimated from the data
Comment: Then I probably don't understand what "artificial noise" is (and what would constitute "non-artificial noise"). Have you seen the article? It certainly explains things a lot better than me.
Comment: Natural noise: random variation in my data. Artificial noise: using a random number generator to draw numbers from a probability distribution, and adding those numbers to my data
Comment: Yes, you need a theory, what kind of data has what kind of noise associated with it. You can than see, how this noise (that you added via a random number generator) influences the result or the noise of the result. Whether this theory about data and noise stems from observations or a theory or is just hypothetical does not matter for this procedure.
Answer: As far as I understand your question, the difference between the "Monte Carlo" approach and the bootstrap approach is essentially the difference between parametric and non-parametric statistics.
In the parametric framework, one knows exactly how the data $x_1,\ldots,x_N$ is generated, that is, given the parameters of the model ($A$, $\sigma_A$, &tc. in your description), you can produce new realisations of such datasets, and from them new realisations of your statistical procedure (or "output"). It is thus possible to describe entirely and exactly the probability distribution of the output $Z$, either by mathematical derivations or by a Monte Carlo experiment returning a sample of arbitrary size from this distribution.
In the non-parametric framework, one does not wish to make such assumptions on the data and thus uses the data and only the data to estimate its distribution, $F$. The bootstrap is such an approach in that the unknown distribution is estimated by the empirical distribution $\hat F$ made by setting a probability weight of $1/n$ on each point of the sample (in the simplest case when the data is iid). Using this empirical distribution $\hat F$ as a replacement for the true distribution $F$, one can derive by Monte Carlo simulations the estimated distribution of the output $Z$.

Thus, the main difference between both approaches is whether or not
  one makes this parametric assumption about the distribution of the
  data.
Comment: Almost two years later, I know believe this to be the best answer because it explicitly mentions the difference between the parametric and non-parametric approaches (which I didn't know back then) Thus, I'm changing the accepted answer to this one.
Comment: but for the paramrtric approach one can also use parametric bootstrap right?
Comment: No, at least not in the "parametric" sense. Remember, under the parametric paradigm we believe we know exactly the parameters of the data AND (in this case) the data generating function behind the data. When we use monte carlo, we are claiming that we know BOTH the true parameters of the data AND that it is a certain distribution (e.g., data generating function). In bootstrapping, we claim that the parameter of the data is from a sample we collected (not the population!), subsequently, the said parameter does not represent the true population parameter and subsequently is non-parametric.
Answer: The Random Change in your Monte Carlo Model is represented by a bell curve and the computation probably assumes normally distributed "error" or "Change". At least, your computer needs some assumption about the distribution from which to draw the "change". Bootstrapping does not necessarily make such assumptions. It takes observations as observations and if their error is asymetrically distributed, then it goes into the modell that way.
Bootstrapping draws from the observation and thus needs a number of true observations. If you read in a book, that C averages at 5 with a standard deviation of 1, than you can set up a Monte Carlo Modell even if you don't have observations to draw from. If your observation is scarce (think: astronomy) you may set up a Monte Carlo Modell with 6 observations and some assumptions about their distribution but you will not bootstrap from 6 observations.
Mixed modells with some input drawn from observed data and some from simulated (say hypothetical) data are possible.
Edit:
In the following discussion in the comments, the original poster found the following helpfull:

The "original program" does not care, whether it gets a value, that
  you computed from a mean and a deviation or that is a true realisation
  of a mean and a deviation in a natural process.
Comment: Thank you for your answer Bernhard! A few questions that pop to my mind. 1. Am I understanding correctly that the only (main?) difference between these two methods is that MC needs to assume a distribution for the uncertainties while the bootstrap does not? 2. If I had a large enough dataset and I performed the iteration numerous times ($N\to\infty$), would these two methods converge then on the estimated uncertainty assigned to the *best value*? 3. Am I not *discarding* valuable data by not using the uncertainties assigned to the input data in the bootstrap method?
Comment: I am statistically/machine-learningly self-taught, so I will not claim that any of the differences I mentioned are the only ones. I am not even certain, whether Bootstrapping is considered a Monte Carlo method itself. Both of the algorithms simulate a large number of realistic scenarios. You can either draw the input from assumptions or from  observations. My field is medicine and assumptions are notoriously wrong in that field. Therefore I would try to go with observations whenever they are available in large enough numbers. It may well be, that in field closer to physics or chemistry,...
Comment: ...that in fields closer to physics or chemistry, assumptions are more reliable. As to point 2: If you go by large enough samples and iterations I assume, you would find that real data is never truely normally distributed and that your assumptions are always a little bit wrong, but I can't claim any knowledge. As to Point 3: I am not shure to have understood what you mean by discarding valuable data in the bootstrap method. "Assigning uncertainty" is man-made, Data comes from reality.  Again, this is my belief based upon my field. In reality, you will rarely have good theory and large data
Comment: By *discarding valuable data* I mean that the bootstrap method makes no use of the uncertainties assigned to the data (ie: $\sigma_A, \sigma_B, \sigma_C$) This is "information" that the MC method takes into account but the bootstrap discards.
Comment: Each observation is a measured value and thus already contains it's own measurement error and uncertainty. The "original program" does not care, whether it gets a value, that you computed from a mean and a deviation or that is a true realisation of a mean and a deviation in a natural process. But of course, all resampling techniques rely on a large data basis and you can compute arbitrary numbers or random numbers but usually not make arbitrary numbers of observations. So in cases where you have a large number of observations, I don't see, where data is discarded.
Comment: In reality, you will rarely have good theory/assumptions and abundant data to choose from or you theory/assumptions will be basend on that data, so the Information in it will be similar.
Comment: *The "original program" does not care, whether it gets a value, that you computed from a mean and a deviation or that is a true realisation of a mean and a deviation in a natural process.* That hadn't occurred to me. I assumed that, by not using the $\sigma$ somehow, I was tossing them away (and the information they contain about the measured data point).
Comment: @Gabriel Glad to help. I added that sentence to the answer as it seemed to be a relevant aspect of the Initial question.
Answer: If the function relating the output Z to the inputs is
reasonably linear (i.e. within the variation range of
the inputs), the variance of Z is a combination of the
variances and covariances of the inputs. The details
of the distribution do not matter too much... So, both
methods should return similar results.
See the  Supplement 1 to the GUM
Comment: What happens when the function is *not* reasonably linear? How will these two methods differ then?
Comment: In that case, you should refer to the answer above, by Bernhard. That is, for them to coincide, you should have a faithful description of the data pdf for Monte Carlo.
Answer: Bootstrap means letting the data speak for themselves.
With Monte Carlo method, you sample many random draws from the imposed CDF (normal; gamma; beta...) via uniform distribution and create an empirical PDF (provided that the CDF is continuous and derivable).
An interesting explanation of the whole Monte Carlo process is reported in: Briggs A, Schulper M, Claxton K. Decision modelling for health economic evaluation. Oxford: Oxford University Press, 2006: 93-95.


How to mock a module exporting a class with class variables using jest

Question: I have a module using react-native-sound in the following ways:
const Sound = require('react-native-sound');
...
const something = Sound.DOCUMENT;
const someOtherThing = new Sound();

How do I mock such a module?
Comment: For testing? Why are you mocking it?
Comment: Yes, I want to test a component which uses this module.
Comment: What is your testing framework? Jest?
Answer: I've mocked react-native-sound using a manual mock (in the __mocks__ folder) which looks like this:
const isFakeFilename = filename => /^blah.*/.test(filename);

const mockFunctions = {
  play: jest.fn(cb => {
    console.log('*** Playing sound! ***');
    cb && cb(isFakeFilename(this.filename) ? false : true);
  }),
  setCategory: jest.fn(),
  getDuration: jest.fn(() => 100),
  getNumberOfChannels: jest.fn(() => 8)
};

const Sound = function(filename, blah2, cb) {
  this.play = mockFunctions.play.bind(this);
  this.filename = filename;
  const savedFilename = filename;
  setTimeout(() => {
    if (isFakeFilename(savedFilename)) {
      cb && cb(new Error('File does not exist! (mocked condition)'));
    } else {
      cb && cb();
    }
  });
};

Sound.prototype.play = mockFunctions.play.bind(Sound.prototype);
Sound.prototype.getDuration = mockFunctions.getDuration;
Sound.prototype.getNumberOfChannels = mockFunctions.getNumberOfChannels;

Sound.setCategory = mockFunctions.setCategory;

export default Sound;
export { mockFunctions };

Note how methods on the Sound import are added directly (Sound.setCategory), and methods on instances of the class (play, getDuration etc.) are added using the prototype.
There's a little added complexity that you may not need using the mockFunctions export. I use that to check for calls to the mocked functions by importing it separately into the test file like
import { mockFunctions } from 'react-native-sound';
// ...
expect(mockFunctions.play).toHaveBeenCalled();

Comment: using the __mock__ folder it is impossible to have different mocked internals per test right?
Comment: You can change it on the fly using `mockImplementation` / `mockImplementationOnce`. https://jestjs.io/docs/en/mock-function-api#mockfnmockimplementationfn. You'll need to import the mocked module into your test file to do this; see the example ('SomeClass') in the link.


RAID performance with 10xSSD on an HP P420i

Question: I am planning to use 10x 8TB Samsung 870 QVO SATA III 2.5" SSD in an HP DL360 G8 server that uses P420i smart array controller.
Each SSD has a max of 530 MBs speed in sequential reads. Theoretically, a RAID-5 configuration gives 5300 MBs (10x530 MBs) read speed.
SSD's connection port is SATA. P420i has 6Gbps speed, so it is more than the SSD speed.
Is there any bottleneck in this configuration that may prevent reaching the theoretical read speed?
Answer: The QVO SSDs are absolutely improper for hardware RAID-5 or RAID-6 arrays.
Don't even try, I did, and it doesn't work at all. The reason is the way these drives work: they can't sustain the long writes necessary to initialize the array -- they're using a 4 GB SLC cache for performance, any write bigger than 4GB will be so slow that the RAID controller will think the SSD doesn't respond and fail it. So you'll set up your array, and after initializing the first 40GB of the array the initialisation/rebuild will fail.
Answer: If you want higher performance and throughput, make sure to use RAID10.By this will get doubled performance.
Also having Min.1GB of Cache is a good option on your controller.


Teaching a left-handed drummer on a right-handed kit?

Question: After several months, a drum student of mine still seems naturally to lead with the left, despite practising on a right handed set up kit. It was at my suggestion. I'm at the point where I'm considering changing the kit round. Anyone had similar experiences and solutions? I just feel for his future, it's better to put more effort in now, given that he's going to have far more opportunities to play on r.h. kits. Or go with the flow?
Comment: Can't say for drummers, but I've known a *lot* of left-hooker guitarists who can play rh guitar better than most rh guitarists.
Comment: @Tetsujin - yes, I've had this theory for decades that says guitars need the neck played by the better hand ! But with drums, it's rolling round the kit that's a problem, arms get crossed over.
Answer: Personally I consider myself a left handed drummer despite being right handed in everyday life. I write and play guitar right handed, but on drums it always felt natural to play with my left. Back when I learned in school I led with my left and when we had the opportunity to play on a real kit rather than our practice pads I played open handed. I also played timpani and preferred having the low drums on my left.
I think the bigger issue is footed-ness. It depends on the genres he intends to play, but a lot of genres have complex and heavy bass drum lines that require dexterity in that foot. Another issue is if he wants to play with double bass drum pedals; that is strictly speaking the only drum set equipment that is not reversible, and there are not quite as many choices for left hand DB pedals as right.
For switching sets around I've never had much of a problem. Cymbal stands, snare, and floor tom are easy to move around. The only tricky part is the rack toms since they will likely have to be readjusted height and angle wise to fit over the bass drum. This is more of a problem if the toms are mounted on the bass drum rather than on cymbal stands. Racks can also be really annoying, as would microphone set ups.
I'd say let him try it out and watch his bass drum foot to see if it can keep up. If he really feels comfortable playing that way it's going to be a lot easier for him to learn. Most gigs should give him enough time to switch a set around, unless there's not 5-10 minutes he could have. Bringing his own double bass drum pedal would be the only equipment he'd need to bring that's unique to a lefty setup.
Comment: Great insights, but I don't understand how a double pedal can't be used the other way around. unless it's not readily available as such. Will check my double pedal tomorrow. But for him that's way in the future, if ever, thanks.
Comment: A double pedal by design can only be used one way. The beaters are on the right pedal while the slave pedal extends to the left. As far as I know no one has yet invented a pedal that can be reversed. If they did they would surely have a money maker on their hands! Using two bass drums and two single pedals would of course not have this issue.
Comment: Looking at my Gibraltar pedal, it would appear that I could strip everything down and refit it in reverse., so the main pedal will be on the left. Not tried yet, but the theory seems o.k!
Comment: Several double pedals are reversible, the Mapex Falcon, and the co-incidentally named Tama Iron Cobra amongst others.
Comment: Wow, that's pretty cool. I've never owned a double pedal so I was just going by what I've seen in stores and online, since most manufacturers make specific left hand double pedals. My single pedal is an Iron Cobra so I may have to look into one when I can afford it. I suppose that means there's one less reason not to let him play left handed.
Comment: Oh, I forgot to mention this, but I recently changed over to right handed drumming. It wasn't too hard of a transition so even if he decides to switch at some point he shouldn't have a problem.
Comment: That's well worthy of a mention! My problem is that with a handed kit - either hand - the rolls are going to be lead differently, otherwise the hands are going to try to cross each other.
Comment: I always felt it was worth it to learn leading with either hand. Rolling from low to high toms is easier leading with left.
Comment: True, but generally I find myself rolling high to low more often. It's just that the student keeps on favouring the left - cymbal crashes, first hit of a roll, etc,. and it's becoming a battle. At home, I'm certain he doesn't think hard enough, so l.h. leading is being practised more and more.
Comment: Talking to a drummer the other day, he said that when he uses a double pedal, a lot of the time, he uses his left foot to play ordinary bass drum patterns. (He's right-handed and plays a right-handed kit.
Answer: Try switching the kit around for the guy, if you have not already - at least once to give him the chance of attempting to play the drums the way they were meant to be played when leading with the dominant hand... No harm ever came from trying.
Case in point, I played with a leftie on and off for a couple years (about 10 years ago.) He, however, played the drum-kit in a right-handed set-up. He did this because he "learned that way" and "always played drums that way" because of (according to him) the very reason you state: he had more opportunities to play on a right-handed kit over the years... After several instances of losing tempo (in either direction by speeding up or slowing down,) missing beats, and/or generally 'not playing well,' I insisted that he switch the kit around to accommodate his dominant hand... As a matter of fact, I did it for him (he was a stubborn artsy-type.) Saying that the difference was drastic is a complete understatement. In the couple minutes that it took to switch around the kit (including the rack toms and the double bass pedal,) this guy went from being 'just ok' to someone I might actually call a "drummer."
Now, this guy was not a drummer (at least in the same sense that I am not a drummer, hence the 'bassist' in my display name, even though I can play and write for drums fairly well.) We were doing another project at the time - he is a singer/guitarist. But, we liked to have fun while we worked and he could play drums. And after the switch, he could play them better than some others who I have heard call themselves "drummers." Seriously, it was like playing with 2 different people... I can't stress this enough. He went from 'bad' to 'good' almost instantaneously. It was embarrassing and we both laughed about it later.
Forgive me if I rant but, I am also getting tired of seeing some of these answers on this SE that one might pull out of motivational social media posts. "Do what works for you, man" or "as long as it feels good, it is good" and the like... I could start learning to play the clarinet by sticking it up my... nose. But, it does not matter how good I became at playing it that way. Someone that learned to play the instrument the traditional way would ALMOST ALWAYS have the creative advantage and higher level of skill - and, arguably, the more fulfilling musical experience. (Unless I am literally the one in a million through-the-nose-clarinet-player.) And, no music teacher would teach me to play that way or allow me to learn to play it that way no matter how much 'I like doing it like that.'
The instruments we use today are modern technology. They are manufactured and set-up the way that they are in order to be used as efficiently as possible. There may always be some novelty to seeing someone performing in some strange way; like playing piano while standing on your head. However, if anyone is playing an instrument in any way other than the way in which it was 'meant to be played,' then that person will LIKELY NEVER achieve the full potential of skill on said instrument (despite there always being exceptions to the rule.)
Comment: Have to admit - no. And his sister has the same 'affliction'! He has two kits, so I might admit defeat and swap one over, but still believe it's not so good for his future in the big bad world... And then, when he comes to my studio for lessons?
Comment: Played a gig last night with a drummer I've played with on and off for several years. Chatting after he admitted he is l.h. but learned to play r.h. - because most kits were/are set up that way (as the kit at the venue). Hasn't messed him up! I still believe that every (good) drummer needs to be ambidextrous, and should be able to play open or closed, but it's rolling round the kit that is a problem. I can only manage r.h., and really struggle l.h. Last night, 4 different drummers took the stand, so if the kit had to be rearranged, it would have been a real pain! ( I was on bass).
Comment: @Tim Yes, I see the predicament! And, I agree that really good drummers (Stewart Copeland) would not be vexed by the orientation, especially if playing open-handed... But, not all of us are that good ;) Did you ever try to switch around the kit for your student?