Reading shared memory data using Java that is written by C++

Question: I have an application in C/C++ that writes data in shared memory in Unix. Now I want to read that shared memory data through a program using Java.
The program should read the shared memory data. Can it be possible?
Need a small implementation if possible.
Comment: Reading shared memory does if ever only work through JNI - in Java you can't do that. My experience with JNI was that some JNI-plugins crashed the otherwise rock-solid Java VM. So maybe another solution is possible?
Comment: This answer http://stackoverflow.com/a/1492536/1741542 talks about memory mapped files and NIO FileChannel. Maybe this works for you too.
Comment: If you would benefit from persistence of the data (even for debugging purposes) you could consider memory mapped files.  Here an implementation which uses pure Java https://github.com/peter-lawrey/Java-Chronicle
Comment: @luukes
Are you sure that using a Java program we cannot read the shared memory data written by c++ program? And we have to have JNI in between to achieve this?
Answer: Take a look at this codeproject project (For Windows):

Using Memory Mapped Files and JNI to communicate between Java and C++
  programs

Or, this library:

CLIPC is an open-source Java library that gives developers
  interprocess communications (IPC) capabilities that may be absent or
  difficult to use in the Java distribution.
Comment: Can I create a method in C++ only to read the shared memory, and then call this method using JNI? Would that be effective and won't harm to performance also?
Comment: JNI methods should call from Java not C++. Yes you can create a method in C++ to read/write from/to the shared memory. And also you can do same thing in Java with JNI methods. Don't worry about performance if you do it right.
Comment: Can you please just tell me the suggestions how can I achieve the below functionality?


Logging in and scraping a site like ft.com with BeautifulSoup

Question: I have this url: https://www.ft.com/content/87d644fc-73a4-11e7-aca6-c6bd07df1a3c
It corresponds to an article that requires signing up. I signed up and can see the content in my browser. However when I use this code with the url above:
soup = BeautifulSoup(urllib2.urlopen(url), 'lxml')
with open('ctp_output.txt', 'w') as f:
    for tag in soup.find_all('p'):
        f.write(tag.text.encode('utf-8') + '\n')

Especially, it redirects me on the signup page. Is there any way to be logged in to have access to the article when scraping?
Answer: Here are the basics.
Go to the login page. If you use the Chrome browser you can position your mouse over the email input area and use the context menu (in Windows) and then its 'Inspect' entry to reveal the form element that will be used to submit your email address. It looks like this.
<form name="enter-email-form" action="/login/submitEmail" class="js-email-lookup-form" method="POST" data-test-id="enter-email-form" novalidate="true">
        <input type="hidden" name="location" value="https://www.ft.com/content/87d644fc-73a4-11e7-aca6-c6bd07df1a3c">
        <input type="hidden" name="continueUrl" value="">
        <input type="hidden" name="readerId" value="">
        <input type="hidden" name="loginUrl" value="/login?location=https%3A%2F%2Fwww.ft.com%2Fcontent%2F87d644fc-73a4-11e7-aca6-c6bd07df1a3c">

Sign in

            Email address
            <input type="email" id="email" class="o-forms-text js-email" name="email" maxlength="64" autocomplete="off" autofocus="" required="">
            <input type="password" id="password" name="password" style="display:none">

            <button class="o-buttons o-buttons--standout o-buttons--big" type="submit" name="Next">Next</button>

    </form>

You will need to gather the action attribute from the form element and all the name-value pairs from the input statements. You use these in a POST request with the requests library.
You do this once for your email address and once for your password. Then you should be able to issue the GET for the URL with requests.
I must warn you that I haven't actually tried this with that particular site.
Comment: If you do that then we could change the title of your question to, say, logging into ft.com so that others might benefit from your experience.
Answer: If you are to scrape a website using BeautifulSoup, I'd recommend the MechanicalSoup library. It is a very lightweight layer on top of BeautifulSoup (to parse HTML) and requests (to fetch pages), but it will deal for you with things like filling-in a form properly (i.e. what you need here), following relative links, ...
MechanicalSoup is also limited in the sense that it doesn't interpret JavaScript code, hence won't work on a website relying on JavaScript, but it reduces the manual effort compared to using BeautifulSoup and urllib or requests directly.
(Note: I'm one of the authors of MechanicalSoup)


Laço entre tabelas SQL Server

Question: Bom dia Pessoal, preciso de uma ajuda com SQL Server.
Tenho uma Tabela AAA, que me retornar itens e uma BBB que me retorna usuários, como faço para repetir todos os usuários para cada item?
Obs: essas tabelas não tem amarrações !
EX:
JOAO  - ITEM A
JOAO  - ITEM B
JOAO  - ITEM C
MARIA - ITEM A
MARIA - ITEM B
MARIA - ITEM C
JOSE  - ITEM A
.
.
.
Comment: Por favor, edite a pergunta para limitá-la a um problema específico com detalhes suficientes para identificar uma resposta adequada.
Answer: Para esse tipo de caso você pode utilizar um Cross Join. Ex:
select *
from AAA
cross join BBB

Answer: Seria uma query parecida com essa:
SELECT item.A, nome.B FROM AAA as A JOIN BBB as B ON user_id.A = id.B

Para mais ajuda, posta a estrutura das tabelas. ;)
Comment: Sua resposta pode ser melhorada adicionando informações fundamentais. Basta [edit] para adicionar mais detalhes, como citações ou documentação para que outras pessoas possam confirmar que sua resposta está correta. Você pode encontrar mais informações sobre como escrever boas respostas [na Central de Ajuda](/help/how-to-answer).
Comment: Obrigado @igoriglesias, era basicamente isso só mudei o Join para ON 1=1 !!

SELECT
 USUARIO,
 ITEM
FROM AAA
INNER JOIN BBB
1 = 1

Vlw mesmo !!

Laço entre tabelas SQL Server


ElasticSearch Nested aggregation with filters and query

Question: I want to calculate median on nested field. Nested field contains of list of objects that have some attributes. I want to filter out some of them before I calculate median.
For example, let's say I have 10 objects in nested field, but only 7 of 10 will be taken for calculating median.
query_median = {
    "query": {
        "bool": {
            "filter": [
                {
                    "term": {
                        "date": "2020-05-18"
                    }
                },
                {
                    "term": {
                        "group_name": "some_name"
                    }
                }
            ]
        }
    },
    "aggs": {
        "median_value": {
            "nested": {
                "path": "people"
            },
            "aggs": {
                "median": {
                    "percentiles": {
                        "field": "people.for_median_attr",
                        "percents": [50]
                    }
                }
            }
        }
    }
}

Above query works, but it has no filters. When I want to add additional filters in aggs, it gives me the same value as in case without any filter. Below what I tried:
query_median = {
    "query": {
        "bool": {
            "filter": [
                {
                    "term": {
                        "date": "2020-05-18"
                    }
                },
                {
                    "term": {
                        "group_name": "some_name"
                    }
                }
            ]
        }
    },
    "aggs": {
        "median_value": {
            "nested": {
                "path": "people"
            },
            "aggs": {
                "filter_out": {
                    "filter": {
                        "bool": {
                            "must": [
                                {
                                    "term": {
                                        "people.attr_not_wanted1": False
                                    },
                                    "term": {
                                        "people.attr_not_wanted2": False
                                    }
                                }
                            ]
                        }
                    },
                    "aggs": {
                        "median": {
                            "percentiles": {
                                "field": "people.for_median_attr",
                                "percents": [50]
                            }
                        }
                    }
                }
            }
        }
    }
}

Example document:
{
        "_index" : "some_index",
        "_type" : "_doc",
        "_id" : "some_id",
        "_score" : 1.0,
        "_source" : {
          "date" : "2020-05-10",
          "group_name" : "some_name",
          "org_code" : "some_code",
          "people" : [
            {
              "nickname" : "xxx",
              "review_count" : 20.0,
              "not_wanted_1" : false,
              "not_wanted_2" : false
            },
            {
              "nickname" : "yyy",
              "review_count" : 18.0,
              "not_wanted_1" : false,
              "not_wanted_2" : false
            },
            {
              "nickname" : "zzz",
              "value_for_median" : 11.0,
              "not_wanted_1" : true,
              "not_wanted_2" : true
            },
            ...
          ]
        }
      }
    ]
  }

In this case, median is calculated only from two numbers: 20 and 18.
Comment: Can you show a sample document with a nested array containing some element to keep and some to leave out?
Comment: **Val** I added it in initial thread - thanks!
Answer: You're almost there. You're just missing a few curly braces in the nested filter and you should pick true instead of false since you want to keep the nested documents to calculate the median on them.
Your query should look like this:
{
  "query": {
     ...
  },
  "aggs": {
    "median_value": {
      "nested": {
        "path": "people"
      },
      "aggs": {
        "filter_out": {
          "filter": {
            "bool": {
              "must": [
                {
                  "term": {
                    "people.not_wanted_1": true
                  }
                },
                {
                  "term": {
                    "people.not_wanted_2": true
                  }
                }
              ]
            }
          },
          "aggs": {
            "median": {
              "percentiles": {
                "field": "people.value_for_median",
                "percents": [
                  50
                ]
              }
            }
          }
        }
      }
    }
  }
}

Results:
  "aggregations" : {
    "median_value" : {
      "doc_count" : 3,
      "filter_out" : {
        "doc_count" : 1,
        "median" : {
          "values" : {
            "50.0" : 11.0
          }
        }
      }
    }
  }

Comment: Thank you for help. Acutally I want to exclude those with both values `true`. So in my case I take all objects that have `not_wanted_1/not_wanted_2 = False`. This query seems to show `'values': {'50.0': None}`
Comment: Ok then you can just change `true` to `false`. however, note that in those elements you don't have any `value_for_median` field.
Comment: Actually I re-arranged your answer to my needs and it's working! Great, Thank you!
Answer: From the documentation on https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filters-aggregation.html , you could try updating your 'filter_out' part of the query to this :
    "filter_out" : {
      "filters" : {
        "filters" : [
          { "term" : { "people.attr_not_wanted1" : false   }},
          { "term" : { "people.attr_not_wanted2" : false }}
        ]
      }
    }


redirect index.php?var1=aaa&var2=bbb

Question: I just moved a website to drupal 7 and I have problems with the redirects.
The old website had all urls like this:
www.example.com/index.php?var1=aaa&var2=bbb
(where aaa and bbb changed for the different pages)
In the new Drupal website I use alias, so for example the page:
www.example.com/index.php?var1=aaa&var2=bbb
now it is replaced by
www.example.com/content/something
I tried to redirect the old url to the new one with two different methods, but both failed. They redirect the users to the homepage.

I create an alias
I added a rewrite rule in the .htaccess file
RewriteRule /index.php?var1=aaa&var2=bbb /content/something

As I said, both of them redirect to the homepage.
Any idea?
thanks!
Comment: I am thinking it is because drupal already rewrites paths and any path that doesn't actually exist, eg, is a real file in your filesystem, gets sent to drupal's index.php for handling.  So.... you are probably going to have to rewrite them to drupal system paths, eg, rewrite index.php?var1=..... to index.php?q=node/# which is drupal's way of dealing with things.  alas, that will be pretty extensive, so you may want to consider a module that does this for you instead of hard coding in .htaccess.
Comment: Consider using http://example.com - that's the domain created to be used in documentation, questions, answers and all that whenever you don't want to give your real one. See [this meta question and it's accepted answer](http://meta.stackexchange.com/a/207155/225745) for rationale and clarifications :)
Comment: thank you boyh.
Jimajamma, ok, I could that with a module... can you give me some advice on how which functions do I have to use inside that?
Answer: Redirect  is the drupal 7 version of path redirect. Path redirect This module allows you to specify a redirect from one path to another path or an external URL, using any HTTP redirect status. and GlobalRedirect is a simple module which…
Checks the current URL for an alias and does a 301 redirect to it if it is not being used.
Checks the current URL for a trailing slash, removes it if present and repeats check 1 with the new request.
Checks if the current URL is the same as the site_frontpage and redirects to the frontpage if there is a match.
Checks if the Clean URLs feature is enabled and then checks the current URL is being accessed using the clean method rather than the 'unclean' method.
Checks access to the URL. If the user does not have access to the path, then no redirects are done. This helps avoid exposing private aliased node's.
Make sure the case of the URL being accessed is the same as the one set by the author/administrator. For example, if you set the alias "articles/cake-making" to node/123, then the user can access the alias with any combination of case.
Most of the above options are configurable in the settings page. In Drupal 5 you can access this after enabling the globalredirect_admin module. In Drupal 6, the settings page is bundled into the module.
Comment: path redirect doesn't exist for Drupal 7
Comment: See my edit on answer.

redirect index.php?var1=aaa&var2=bbb


Spacy Stopwords based on Frequency

Question: i'm currently searching for an easy solution to add custom stopwords to spacy. These stopwords shall be determined on basis of the absolute frequency of the words in the whole corpus. E.g., in my domain-specific texts, the term "patient" should be considered a stopword as it occurs in 70% of all documents.
My first idea was to implement this by the help of pandas apply, but this would require to write my own tokenizing function. Is there a possibility to customize Spacy?
Thank you for any advice
Answer: To add custom stopwords into Spacy you can follow the solution given here: Add/remove stop words with spacy
. Now in other to get a list recommended stopwords automatically, you can use NLTK package to calculate term frequency and document frequency (tf-idf), then define a trashold.
If you have any doubt, dont hesitate to comment.
Good luck!

Spacy Stopwords based on Frequency


Linear Algebra Word Problem

Question: I have been this for hours, yes it is a homework question, but did not turn it in last weeks when I was suppose to, but I have a test coming up, so gotta figure this out not sure how to formulate questions using a system of equations
The density of gold is $19.3g/cm^3$ and the density of silver is $10.5g/cm^3$ a certain crown is made entirely from silver and gold if the total volume of the grown is $220cm^3$, and the weight of the crown is $3823.6g$, what percentage of the total mass is gold?
I am not sure how to get around the units I mean I tried assigning x to be gold and y silver but this $g/cm^3$  how do you equate that to $3823.6g$ and have the unit match up.
I mean all I have on my paper is $19.3x + 10.5y = 3823.6$  but I know that is wrong and where do I get the second equation.  I am truly stuck and frustrated.  my book doesn't have a solutions manual so I wouldn't know if it is right or wrong.  Please help me set this up
Answer: Let $g$ be the volume of gold, in $ \text{cm}^3$ and $s$ be the volume of silver in the same units. The volume of the crown is $ 220 \space\text{cm}^3$, so
$$g +s = 220$$
Every $ \text{cm}^3$ of gold is $19.3\text{g}$ and the same volume of silver is  $10.5\text{g}$. Multiply the individual volume of gold & silver to get the total weight of the crown.
$$19.3g +10.5s = 3823.6$$
You now have 2 equations. Use simultaneous equations to solve the individual volume of gold & silver. Finally, find the percentage mass of gold by taking
$$\frac{19.3g}{3823.6g}$$
Update
You look like you're looking for the matrix way of solving. I'll re-write the equations for you in the form:
$$
\begin{align*}
g +s &= 220\\
19.3g +10.5s &= 3823.6
\end{align*}
$$
To put the equation in matrix form,
$$
\begin{pmatrix}
1 & 1\\
19.3 & 10.5
\end{pmatrix}
\begin{pmatrix} g \\ s\end{pmatrix} =
\begin{pmatrix} 220 \\ 3823.6\end{pmatrix}
$$
Find the inverse of the matrix
$$
\begin{pmatrix}
1 & 1\\
19.3 & 10.5
\end{pmatrix}
$$
which is
$$
\begin{pmatrix}
-1.193 & 0.114\\
2.193 & 0-.114
\end{pmatrix}
$$
And the equation you want to solve will be
$$
\begin{pmatrix}
-1.193 & 0.114\\
2.193 & 0-.114
\end{pmatrix}
\begin{pmatrix}
1 & 1\\
19.3 & 10.5
\end{pmatrix}
\begin{pmatrix} g \\ s\end{pmatrix} =\begin{pmatrix}
-1.193 & 0.114\\
2.193 & 0-.114
\end{pmatrix}
\begin{pmatrix} 220 \\ 3823.6\end{pmatrix}
$$
The left hand side will evaluate to be a $2 \times 2$ identity matrix, which you have identified as $I$. The right hand side will give you the values of $g$ and $s$ respectively.
Comment: So I start out with

g + s = 220

19.3g + 10.5s = 3823.6

then in matrix form:

Row I:  1        1       220,
Row II 19.3    10.5    3823.6,

Then I solve by doing -10.5*RI + RII which gives

so this is Row Echelon Form?  I believe I can't do Reduced Row Echelon Form (rref).  So I can now just solve for g using the second row and plug into the top to solve for s.  Why won't this equation ever reduce to rref?  Does it have to be in rref to to have a solution.  I appologize I feel like I am asking the lamest questions ever but I want to ace this class.
Comment: Hi rob, I have updated my answer. You might want to learn LaTeX to show your working.
Answer: let the crown be entirely of gold. Then for every  $cm^3 $you make of silver instead of gold you will lose 8.8 grams off the total crown. if it was all made of gold it would weigh $19.3 \frac{grams}{cm^3}*220 cm^3 = 4246$ grams so therefore the crown weighs $4246-3823= 423$ grams less than the pure gold.
Since or every $cm^3$ you substitute with silver you reduce the weight by 8.8 grams then if x is the number of $cm^3$ of silver then $x*8.8=423$ so $x=\frac{423}{8.8} $which is roughly 48.07
Looking at it in a system of equations (slightly different method) it would look like this, where $x_1$ and $x_2$ are the $cm^3$ of gold and silver respectively:
$x_1+x_2=220$ and $19.3x_1+10.5x_2=3823.6$ so now you just have to solve these two equalities.
Comment: how would that look like in a system of equations?

Linear Algebra Word Problem


2 stage least Squares as an Instrumental variable estimator

Question: $\mathbf{X}_{n \times K}= \begin{bmatrix}
      \mathbf{x}'_1 \\
       \cdots \\
       \mathbf{x}'_n
     \end{bmatrix}$
$\mathbf{Z}_{n \times L}= \begin{bmatrix}
      \mathbf{z}'_1 \\
       \cdots \\
       \mathbf{z}'_n
     \end{bmatrix}$
where $\{\mathbf{x}_i\}$ are instruments, and  $\{\mathbf{z}_i\}$ are the variables(some of which are endogenous).
If I regress each $z_{il}$ on $\mathbf{x}_i$ define the fitted value as $\hat{z}_{il}$, i'll get  $\hat{\mathbf{z}}_l=\mathbf{X}(\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{z}_l$.
So, $\hat{\mathbf{Z}}_{n \times L}= \begin{bmatrix}
      \hat{\mathbf{z}}'_1 \\   \cdots \\   \hat{ \mathbf{z}}'_n
     \end{bmatrix}=\begin{bmatrix}
      \mathbf{X}(\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{z}_1 &&   \cdots &&   \mathbf{X}(\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{z}_n
     \end{bmatrix}= \mathbf{X}(\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{Z}$
The IV estimator formula, when using these newly created instruments is
$(\hat{\mathbf{Z}}'\mathbf{Z})^{-1}\hat{\mathbf{Z}}'\mathbf{y}=((\mathbf{X}(\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{Z})'\mathbf{Z})^{-1}(\mathbf{X}(\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{Z})'\mathbf{y}$ which is the 2SLS estimator.
My question is how can we be sure that $\hat{\mathbf{z}}_{l}$ are instruments?
Edit: Here is the original text.
Any help would be appreciated
Comment: I'm a bit confused about your notation/set-up. Where are the coefficients to be estimated? Also it seems to me that you should refer to the $\widehat{\textbf{z}}$ as fitted values from the first stages rather than "new instruments". If so then I'm not quite sure anymore what the actual question is. Could you clarify this?
Comment: @Andy I've put up the original text. My doubt is «how can we prove that the new variables created by the regression of the data on instruments, create new instruments?» How do we prove that?
Answer: If by the question " How do we know that they are instruments", you mean "how do we know that they are orthogonal to the error term?", then:
We have assumed that $$E\big(\mathbf {X'} \mathbf u\big) = 0$$
For orthogonality of the fitted values we want
$$E\big(\mathbf {\hat Z'} \mathbf u\big) = E\left[\left(\mathbf{X}(\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf Z \right)'\mathbf u\right] = ? \;0$$
Applying the transpose operation we get
$$E\left[\mathbf {Z'} \mathbf{X}(\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}' \mathbf u \right] = ?\;0$$
As such it does not look like we are permitted to separate the matrix product and take separate expected values -after all, $\mathbf Z$ is not independent of $\mathbf u$, so even with the projection matrix of the original instruments "in the middle", how can we break the expected value?
The subtlety here is that we obtain the expected value as a limiting case of the sample size going to infinity (because it is then that we need orthogonality so that consistency is preserved). In other words, we first take probability limits and then "arrive" asymptotically at expected values. But probability limits distribute irrespective of independence. The IV estimator with the fitted values is
$$\hat \beta_{IV} = \beta + \left(\mathbf {\hat Z'}\mathbf {\hat Z}\right)^{-1}\mathbf {\hat Z'} \mathbf u = \beta + \left(\frac 1n\mathbf {\hat Z'}\mathbf {\hat Z}\right)^{-1}\left(\frac 1n\mathbf {\hat Z'} \mathbf u\right)$$
and
$${\rm plim}(\hat \beta_{IV} - \beta) = {\rm plim}\left(\frac 1n\mathbf {\hat Z'}\mathbf {\hat Z}\right)^{-1} {\rm plim}\left(\frac 1n\mathbf {\hat Z'} \mathbf u\right)$$
As usual a separate assumption is that the first ${\rm plim}$ converges to something non-zero and finite. So we are interested in the second ${\rm plim}$,
$${\rm plim}\left(\frac 1n\mathbf {\hat Z'} \mathbf u\right) = {\rm plim}\left(\frac 1n\mathbf {Z'} \mathbf{X}(\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}' \mathbf u \right)$$
Note that we already have one $1/n$ factor so we can write
$${\rm plim}\left(\frac 1n\mathbf {\hat Z'} \mathbf u\right) = {\rm plim}\left(\frac 1n\mathbf {Z'} \mathbf{X}\right)\left(\frac 1n\mathbf{X}'\mathbf{X}\right)^{-1}\left(\frac 1n\mathbf{X}' \mathbf u \right)$$
and now we can separate ${\rm plim}$'s again
$$... = {\rm plim}\left(\frac 1n\mathbf {Z'} \mathbf{X}\right){\rm plim}\left(\frac 1n\mathbf{X}'\mathbf{X}\right)^{-1}{\rm plim}\left(\frac 1n\mathbf{X}' \mathbf u \right)$$
Again the first two matrix products are assumed to converge to something non-zero and finite, so we focus on the last one, for which given the other assumptions of the model we have that
$$ {\rm plim}\left(\frac 1n\mathbf{X}' \mathbf u \right) = E\big(\mathbf {X'} \mathbf u\big) = 0$$
-zero by the orthogonality of the original instruments. So the fitted values are indeed valid instruments, if the original instruments are valid instruments.
Comment: I'm always learning. :) thanks

How would I go if I wanted to prove that the new instruments were correlated with the observed vectors. What condition would I impose on $Cov(\hat{\mathbf{Z}},\mathbf{Z})$?
Comment: $E(\hat{\mathbf{z}}_i \mathbf{z}'_i)$ being full column rank, right?
Comment: Could you please give me some insight on how to prove the generalization of the condition that the instruments must be correlated with the regressors? Maybe it's something trivial, that I'm missing.
Comment: @Anoldmaninthesea If they are not correlated then regressing the regressors on the instruments will give you nada, so this "indirect" IV method won't work. In the more basic case, where we start from the orthogonality condition $$E(\mathbf X'\mathbf u)=0$$ $$\implies ... (\mathbf X'\mathbf Z)^{-1}\mathbf X'\mathbf y=\beta$$ do you think the matrix $(\mathbf X'\mathbf Z)$ is invertible if $\mathbf X'$ and $\mathbf Z$ are orthogonal?
Comment: I'm getting a bit confused with the notation, due to my lack of knowledge, so I'll stick to Hayashi notation. In the the simple case of IV, we would still need to assume that $E(\mathbf{x}_i\mathbf{z}'_i)$ is full rank to reach the formula for beta. But how from $\mathbf{x}_i$ being orthogonal, can I prove that $E(\hat{\mathbf{z}}_i\mathbf{z}'_i$ is full rank? ($\hat{\mathbf{z}}_i$ are the new instruments)
Comment: I think I now understand your comment. What I thinkn that Hayashi was saying in this part of the book was that by constructing new variables by regressing known regressors $z_{il}$ on the known instruments $\mathbf{x}_i$, and then create a new variable $\hat{z}_{il}$ as the fitted value of this regression. He states that because these new variables are also intruments, then we can use the IV formula, and do a substitution in it to obtain the 2SLS estimator. In my answer below, I don't think we need to assume that the new variables(the fitted values)are correlated.
Comment: You write "I don't think  we need to assume that the new variables (the fitted values) are correlated". Correlated with what?
Comment: with the original regressors. We can deduce that from using the assumption that the original regressors are correlated with the 'old' instruments (x's). I think I do just that below...
Comment: So you are saying that $\hat Z$ is (or can be) uncorrelated with $Z$, do I understand correctly?
Comment: No. What I am saying is that they are correlated, but we cannot assume that. We have to prove it, and I tried to below.
Answer: The following is based on section 9.2 of Hamilton's Time Series, with suitable adaptations. This complements the answer by @AlecosPapadopoulos
I'll try to prove that $E(\hat{\mathbf{z}}_i\mathbf{z}'_i)$ is full column rank (the new instruments are correlated with the regressors.
To generate the new instruments, we first need to use OLS on the following equation:
$$ z_{il}=\mathbf{x}'_i\beta_l+\epsilon_{il} $$, where $z_{il}$ is element $i,l$ of matrix $\mathbf{Z}$.
By OLS, we know that $\hat{\beta}_l=(\sum^N_{i=1}\mathbf{x}_i\mathbf{x}'_i)^{-1}(\sum^N_{i=1}\mathbf{x}_iz_{il})$.
So, $\hat{\mathbf{z}}'_i=\mathbf{x}'_i\hat{\beta}$, where $\hat{\beta}=(\hat{\mathbf{\beta}}_1 \dots \hat{\mathbf{\beta}}_L)= (\sum \mathbf{x}_i\mathbf{x}'_i )^{-1}(\sum \mathbf{x}_i\mathbf{z}'_i)$
From $1/N \sum_i\hat{\mathbf{z}}_i\mathbf{z}'_i=\hat{\beta}'\frac{\sum \mathbf{x}_i\mathbf{z}'_i}{N}=(\sum \mathbf{z}_i\mathbf{x}'_i)(\sum \mathbf{x}_i\mathbf{x}'_i )^{-1}(\sum \mathbf{x}_i\mathbf{z}'_i)/N$
If we do the plim on both sides, we get $E(\hat{\mathbf{z}}_i\mathbf{z}'_i)=E(\mathbf{z}_i\mathbf{x}'_i)(E(\mathbf{x}_i\mathbf{x}'_i))^{-1}E(\mathbf{x}_i\mathbf{z}'_i)$
Now in the IV setting $K=L$ (numb.regressors=numb.instruments), so $E(\mathbf{x}_i\mathbf{z}'_i)$ is a square matrix and by assumption it's full column rank (the x's are also instruments). The $E(\mathbf{x}_i\mathbf{x}'_i)$ is invertible, since we're also assuming conditional homoskedasticity, and together with the assumption of $E(\mathbf{g}_i\mathbf{g}'_i)$ being non-singular , where $\mathbf{g}_i=\mathbf{x}_i\epsilon_i$ is a martingale sequence.Therefore, $E(\hat{\mathbf{z}}_i\mathbf{z}'_i)$ is invertible.
Comment: @AlecosPapadopoulos Do you agree with this answer?
Comment: That the fitted values of a dependent variable derived from a linear regression are correlated with the dependent variable itself (assuming correct specification of course), is a very basic fact of linear regression, and easily provable, and has nothing particular to do with Instrumental Variables. It doesn't even need asymptotics to be shown, just consider the covariance of $\hat Z$ and $Z$. Also it appears that you "equate" "full column rank" with "correlation". Why? _Extreme_ correlation (i.e. _exact_ linear dependence), would lead to deficient rank, if anything.
Comment: @AlecosPapadopoulos, reading page 200-201 from Hayashi, he states that the assumption of $E(x_iz'_i)$ being full column rank is a generalization of the case of one non-constant regressor/one non-constant instrument. How would we calculate the covariance of $\hat{Z}$ and $Z$ in this case?


perfect right-triangles that are not super-perfect

Question: I'm seeking some insight on the answer to this problem from Project Euler.

Consider the right angled triangle with sides $a=7, b=24$, and $c=25$.
The area of this triangle is $84$, which is divisible by the perfect numbers $6$ and $28$.
Moreover, it is a primitive right angled triangle as gcd$(a,b)=1$ and gcd$(b,c)=1$.
Also, c is a perfect square.
We will call a right angled triangle perfect if
1. it is a primitive right angled triangle
2. its hypotenuse is a perfect square

We will call a right angled triangle super-perfect if
1. it is a perfect right angled triangle and
2. its area is a multiple of the perfect numbers 6 and 28

How many perfect right-angled triangles with $c \le n$ exist that are not super-perfect?

The answer turns out to be 0.
This outcome is not immediately intuitive to me.
Is there some property of Pythagorean triples that leads to this conclusion?
For the given case, it is evident that the area is a multiple of the perfect numbers 6 and 28, so we see that this perfect right-angled triangle is super-perfect, but how can we generalize this property for all perfect right-angled triangles?
Any insight is much appreciated.
Thanks
Comment: What is the $n$ in the equation $c\le n$? There many perfects that are not super perfects and many that are superperfect. Here are the smallest in ten respective sets of triples that I looked at: $$7,24,25$$ $$33,56,65$$  $$95,168,193$$ $$63,16,65$$  $$207,224,305$$ $$231,160,281$$ $$273,136,305$$ $$615,728,953$$ $$425,168,457$$ $$399,40,401$$ If you could explain the $n$ in the inequality in your question, perhaps I could help more.
Answer: If the primitive right triangle has generators $m$ and $n$, with legs $2mn$ and $m^2-n^2$ and hypotenuse $m^2+n^2$, then $m^2+n^2 = k^2$, and $(m,n)=1$ since the original triangle is primitive. This defines a new right triangle, which must have generators $p, q$:
$$m = 2pq,\quad n = p^2-q^2,\quad k = p^2+q^2.$$
Then the area of the original triangle is
$$2mn(m^2-n^2) = 2pq(p^2-q^2)(4p^2q^2-(p^2-q^2)^2.$$
Since $(p,q)=1$, we have $p^2-q^2\equiv 0\mod{2}$, so the area is divisible by 4. Also, either one of $p$ and $q$ is divisible by $3$, so the area is divisible by $6$. Finally, if neither $p$ nor $q$ is divisible by 7, it's not hard to see that $4p^2q^2-(p^2-q^2)^2$ is, using the fact that $p^2, q^2\equiv 1, 2, 4\mod{7}$.
Answer: 33,56,65 and the rest are not perfect triangles. The hypotenuse is not a perfect square in any of them


Why can't I see the memory addresses of some objects?

Question: I have an object model, based on TCP Connections, which belong to Connections, which belong to Managers. It seems like two TCP Connections are different, which I can see, but when I try to see the same for the Connections they belong to, I get error message CS0211: Cannot take the address of the given expression. Do anybody know why and how I can handle this?
Watch window content for both TCP Connections:
&connectionManager.Connection.TcpConnection  0x00007fe80035f670
&Connection_For_Message.TcpConnection        0x00007fe70042bf08

=> both TCPConnections are in another memory address, so they are clearly different (although all their values, properties and attributes are equal).
When asking the same for the Connections, this does not work:
&connectionManager.Connection  error CS0211: Cannot take the address of the given expression
&Connection_For_Message        0x00007fe700375230

Why is that?
For your information: when I ask for connectionManager.Connection (not the address, just the data), everything is ok.
Thanks in advance
Comment: But `Object.ReferenceEquals(obj1, obj2)` would work
Comment: `ReferenceEquals` is the canocical way to determine if two references point to the same object. `Equals` is the canonical way to tell if two objects are "equal", which might mean they are the same object, or they have the same values (depending on the definition of `Equals`. IF you have a specific use case where `ReferenceEquals` does not work the way you want, then describe that in more detail, otherwise it just looks like you're trying to reinvent the wheel
Comment: If you can store the memory address to compare later, then you can store a reference (in an`object` variable) to compare later. Also I don't think you can assume that an object is in the same location in memory throughout it's life (but I may be wrong).
Comment: Do you really want to know the memory addresses or do you just want to evaluate if the two are the same object or not? In the latter case, the answers here might be of help: https://stackoverflow.com/questions/4251450/uniquely-identifying-reference-types-in-the-debugger
Comment: @KlausGütter: I would like to be sure they are equal and for that I thought comparing their memory addresses was a good idea (otherwise I might doubt about the quality of the equality operator).
Comment: @KlausGütter: sorry, but `Object.ReferenceEquals(obj1, obj2)` only works when both are available at the same time. I'm dealing with an object, and later on I'm again dealing with an object, which **should** be the same, but I'm quite sure it isn't. In C++ this is easy: just compare their memory address. How can this be done in C#?
Comment: I had a look at that other post, but I don't get it: I create an object (one in a list), I click on "Make object ID", and at the creation of the next object, I put `1#` in the watch-window. I get the error message "error CS2043: 'id#' syntax is no longer supported. Use '$id' instead.", and when I put `$id` I get `error CS0103: The name '$id' does not exist in the current context`. Do you know what to do?
Comment: in older versions of the debugger, the format was `id#`, where in this case, `id === 1`.  in the newer debugger, the format is `$id`, or, `$1`
Comment: What is the type of `connectionManager`? Could `Connection` be a property that _returns_ an object, and not necessarily an object itself?
Comment: @DStanley: the `Connection` is defined in the `ConnectionManager` as follows: `public TConnection Connection { get; set; }` where `TConnection` is the class of `Connection`. I don't worry about chaning values of properties of a`TcpConnection` object: I'm closing the socket inside a `TcpConnection` and later on that socket seems not to be closed. I would like to be sure that I'm dealing with the same `TcpConnection` (if ever those are different, then it's obvious that one socket is open while the other is closed. However, if it's one and the same object then something else is happening.
Comment: I believe I have a solution: I run my application and make a model ID on one object (this becomes `$1`). Later on I do the same on the other object (of which I want to verify if it's equal to that first one) and I also make a model ID (this becomes `$2`). In the immediate window, I type `? Object.ReferenceEquals($1, $2)`. If I get `true`, both are equal. If I get `false`, both are different. Can anybody confirm this is correct and in case yes, write this as an answer (preferably with a official reference)? I'll accept it and reward the bounty.
Comment: @Dominique That's not usually required. If the second instance is actually the same object, it already is `$1` and the debugger shows that. The debugger will always show the ID next to the value. Or the other way round, if two objects are marked as `$1` and `$2`, they are certainly _not_ equal.
Answer: Retrieving an object address in different parts of the code may be misleading because objects can be moved if GC runs in the middle. The simplest way to go is to check the object's Hash Code by calling GetHashCode() method. If you get different numbers, two objects are guaranteed to be different. But equal Hash Codes don't 100% mean that two references point to the same object. In that case, you can use a temporary hack and store the first object in public static variable, then, when the code reaches the second object, use Object.ReferenceEquals(obj1, obj2) as suggested by @KlausGütter in comments.
Comment: "If you get different numbers, two objects are guaranteed to be different." Except when they're not. `GetHashCode()` is a virtual method that can be overridden to produce any number you feel like. I've seen at least one implementation that returns a different code every time. It's *supposed* to produce consistent values but there's no enforcement of this.


How to remove empty div created by routing.yml file by specified routes?

Question: I have 2 routes
games.one:
  path: '/math/one'
  defaults:
    _controller: '\Drupal\games\Controller\GamesController::one'
    _title: 'My Title'
  requirements:
    _permission: 'access content'

games.two:
  path: '/math/two'
  defaults:
    _controller: '\Drupal\games\Controller\GamesController::two'
  requirements:
    _permission: 'access content'

Route games.one has _title and Route games.two does not have. Now the issue is When I access route game.two, It shows an empty div for title.
How to remove this empty div?
Comment: Better, you need show code class controller. What's code in controller **GamesController::one**
Comment: edit the page.html.twig
Comment: go to the block layout and change the visibility of the *Page Title* block to not show on `/math/two`
Answer: You can do this by overriding the page.html.twig for that page and stripping out the title.
To override the template for that specific page you should probably alter the hook suggestions:
/**
 * Implements hook_theme_suggestions_HOOK_alter().
 */
function THEME_NAME_theme_suggestions_page_alter(array &$suggestions, array $variables) {

  if ($node = \Drupal::routeMatch()->getRouteName() == 'games.two') {
    $suggestions[] = 'page__no_title';
  }
}

Now you can copy the page.html.twig template to  your theme folder, rename it to page--no-title.html.twig and strip out the title tag.
Hope this helps
Comment: I think have a lot of way do that. But if you want use preprocess better choose _preprocess_page_title() and set variable title to false.


$A\subseteq R$ is an upper bounded set that contains at least two items/numbers. If $x < \sup{A}$, then $\sup{\left(A\setminus\{x\}\right)}=\sup A$

Question: Prove: $A\subseteq \mathbb{R}$ is an upper bounded set that contains at least two items/numbers. If $x < \sup{A}$, then $\sup{\left(A\setminus\{x\}\right)}=\sup A$.
My attempt:
Since $A$ is upper bounded, and non-empty, $A$ has a supremum according to the upper bound property, which we'll denote with $\sup{A}=c$. Since $c$ is $\sup{A}$, for all $a\in A$, $a< c$.
Therefore $\sup{A\setminus\{x\}}=\sup{A}$.
My questions is:
Is my though process correct? I feel like I may not be understanding the question correctly, and this seemed to be too easy for an exam question.
Comment: Do you mean "upper bounded set" not "upper bounded group" because the notion of group isn't needed in this exercise.
Comment: Thank you for your answer. I don't speak Hebrew sorry, but in general, in a language you will find many synonyms for the same word. In my opinion, the word in Hebrew that means "set" should be the one that simply refers to a collection of things, while the word for "group" should be a synonym from which we infer there's a certain relation between the elements. Nevertheless, according to Wikipedia, there are two different words used for the mathematical words group and set in Hebrew: קבוצה for set and חבורה for group.
Comment: It appears the direct translation is "set"
Comment: I didn't understand but these will help you to see the difference: https://en.wikipedia.org/wiki/Set_(mathematics) https://en.wikipedia.org/wiki/Group_(mathematics)
Comment: The word used in my language for group (non-mathematical) and set (mathematical) is the word group. What I meant to say is that it should be set.
Comment: :o I see. Thank you for the clarification? This caught my curiosity: what's your language?
Comment: @Scientifica hebrew.
Answer: Firstly, you made a little mistake in

Since $c$ is $\sup{A}$, for all $a\in A,\,a<c$

It should be

Since $c$ is $\sup{A}$, for all $a\in A,\,a\le c$

To convince you, simply take the case where $\sup{A}\in A$ like in the case where $A=[1,2]$. Here $\sup{A}=2\in A$ and $\forall a\in A,\,a\le 2$.
Secondly, your conclusion

Therefore $\sup{A\setminus\{x\}}=\sup{A}$

seems quite unclear to me. You didn't explain what you did with $c'=\sup{A\setminus\{x\}}$.
Thirdly, when you want to prove the equality of the $\sup$ of two different sets (here $c$ for one set as you named it and $c'$ for the other one as I named it) , I recommend you to, in general, try to prove that $c\le c'$ and that $c'\le c$. $c'\le c$ should be easy since $\left(A\setminus\{x\}\right)\subset A$. To prove $c\le c'$, you must clearly show how you used the assumption $x<\sup{A}$.


Let $f(x) = x^n + a_{n-1}x^{n-1} + \cdots + a_0 \in \Bbb Z[x]$ and $r \in \Bbb Q$. If $x-r \mid f(x)$ and $f(x) \in \Bbb Z[x]$, then $r \in\mathbb Z$.

Question:
Let $f(x) = x^n + a_{n-1}x^{n-1} + \cdots + a_0 \in \Bbb Z[x]$ and $r \in \Bbb Q$. If $x-r \mid f(x)$ and $f(x) \in \Bbb Z[x]$, show that $r \in\mathbb Z$.

Since $x-r\mid f(x)$ and $f(x)$ is primitive, that implies that $f(x) = g(x)(x-r)$ where the product $g(x)(x-r)$ is primitive also.  I can't figure how to show that $r$ is an integer here besides saying that in order to take the $\gcd$ of a functions coefficients they must be integers.
Answer: Notice that if $x-r$ divides the polynomial $f(x)$, then in particular $f(r)=0$.
I will prove a more general property:

Proposition: If $r=\frac{p}{q}$ is a root of a polynomial
  $f(x)=a_nx^n+a_{n-1}x^{n-1}+\cdots+a_1x+a_0\in \mathbb{Z}[x]$ with
  $\gcd(p,q)=1$ (i.e., the fraction is already reduced), then $p$ divides
  $a_0$ and $q$ divides $a_n$.

Proof: Suppose $r$ is a root of $f(x)$, then we have
\begin{align*}
a_n\left(\frac{p}{q}\right)^n+a_{n-1}\left(\frac{p}{q}\right)^{n-1}+\cdots + a_1\left(\frac{p}{q}\right)+a_0&=0 &\\
a_n\left(\frac{p^n}{q^n}\right)+a_{n-1}\left(\frac{p^{n-1}}{q^{n-1}}\right)+\cdots +a_1\left(\frac{p}{q}\right)&=-a_0 &\\
a_np^n+a_{n-1}p^{n-1}q + a_{n-2}p^{n-2} q^2+\cdots + a_1p q^{n-1}&=-a_0\cdot q^n &\text{(multiplying by $q^n$ in both sides)}\\
p\left(a_np^{n-1}+a_{n-1}p^{n-2}q^2+\cdots+a_1q^{n-1}\right)&=-a_0q^n &\text{(taking $p$ as common factor in LHS)}
\end{align*}
That is, $p$ divides $-a_0q^n$, but since $\gcd(p,q)=1$, we necessarily have that $p$ divides $a_0$.
On the other hand, we also got
\begin{align*}
a_n p^n&=-a_{n-1}p^{n-1}q-a_{n-2}p^{n-2}q^2-\cdots - a_0q^n\\
&=q\cdot (-a_{n-1}p^{n-1}-a_{n-2}p^{n-2}q-\cdots - a_0q^{n-1})
\end{align*}
So, $q$ divides $a_n\cdot p^n$, but since $\gcd(p,q)=1$, we conclude $q$ divides $a_n$.

Corollary: If $r\in \mathbb{Q}$ is a root of a polynomial $f(x)=x^n+a_{n-1}x^{n-1}+\cdots + a_1x+a_0\in \mathbb{Z}[x]$, then $r\in \mathbb{Z}$.

Proof: Notice that for $f(x)=x^n+a_{n-1}x^{n-1}+\cdots + a_1x+a_0$, $a_n=1$. If $r=\frac{p}{q}$ is a root of the monic polynomial, then by the Proposition above we have that $q$ divides $1$, that is, $q=1$ or $q=-1$, and so $r=p$ or $r=-p$. In both cases, $r\in\mathbb{Z}$.


What is the formula for c_v coherence?

Question: I've recently been playing around with Gensim LDAModel. I use coherence to evaluate the results. Gensim offers a few coherence measures. This includes c_v and u_mass.
While there is a lot of materials describing u_mass on the web, I could not find anything interesting on c_v. There must be some significant difference sice c_v is always positive and u_mass is always negative.
I'd really like to understand the c_v measure, could you please provide me with the exact formula for calculating the c_v measure?
Comment: This article http://www.saf21.eu/wp-content/uploads/2017/09/5004a165.pdf, section ***B. Topic Coherence Measurement***, explains well the formula of the c_v coherence. Hope it helps !
Answer: I didn't succeed to find an exact formula, but here is at least more precise description:

CV is based on a sliding window, a one-set segmentation of the top
  words and an indirect confirmation measure that uses normalized
  pointwise mutual information (NPMI) and the cosinus similarity.
This coherence measure retrieves cooccurrence counts for the given
  words using a sliding window and the window size 110. The counts are
  used to calculated the NPMI of every top word to every other top word,
  thus, resulting in a set of vectors—one for every top word. The
  one-set segmentation of the top words leads to the calculation of the
  similarity between every top word vector and the sum of all top word
  vectors. As similarity measure the cosinus is used. The coherence is
  the arithmetic mean of these similarities. (Note that this was the
  best coherence measure in our evalution.)

Source: http://palmetto.aksw.org/palmetto-webapp/
Answer: In
This paper , c_v measure is explained and these three equations are used to describe it.

I hope this is helpful to you and anyone who comes back to this post
Comment: The notation of that paper is just horrible, unfortunately.
Answer: The c_v coherence measure was proposed and described in a systematic framework of coherence measures by Röder et al.

The best performing coherence measure [...] is a new combination found
by systematic study of the configuration space of coherence measures.
This measure (CV) combines the indirect cosine measure with the NPMI
and the boolean sliding window. This combination has been overlooked
so far in the literature.

I recommend to read the original paper to understand the different steps (segmentation, probability calculation, confirmation measure and aggregation) behind this measure and others, such as u_mass, c_uci or c_npmi, as well. The authors use a consistent notation to describe all coherence measures in a systematic framework. You will find the formulas in the paper.
⚠ However, there are a couple of issues with this coherence measure.
Therefore, the authors don't recommend to use it anymore.

It's not recommended to use CV because there are known issues associated with it.

Röder, M., Both, A., & Hinneburg, A. (2015). Exploring the space of topic coherence measures. WSDM 2015 - Proceedings of the 8th ACM International Conference on Web Search and Data Mining, 399–408. https://doi.org/10.1145/2684822.2685324
Comment: How did you paste in that yellow warning sign?
Comment: @kjetilbhalvorsen ⚠ is the [Warning emoji](https://emojipedia.org/warning/)
Comment: As B.Liu mentioned, I simply copied and pasted it from emojipedia
Comment: @J.Schneider, can you clarify whether the problems are with `c_v` itself, or just with the Palmetto implementation?
Comment: To be honest, I don't know the answer and [M. Röder neither:](https://github.com/dice-group/Palmetto/issues/13#issuecomment-371553052): "So finally I would suggest to use C_P, NPMI or UCI for evaluating topics."
Comment: It is worth noting that a fix back in 2022 seem to have fixed the said issue in the Palmetto implementation.
Comment: Reviewers:  Please read the linked issues, they have since been fix, which makes the statement in this answer false.  Edits are necessary, do not disapprove edit mindlessly...
Answer: The algorithm remained vague to me until I dove into the code. See my blogpost to walk through the mathematics of cᵥ step-by-step. In words, I would describe it as follows:
In cᵥ coherence, each topic word is compared with the set of all topics. A boolean sliding window of size 110 is used to assess whether two words co-occur. Then, the confirmation measure consists of a direct- and indirect confirmation. That is, for all N most probable words per topic, a 'word vector' of size N is created in which each cell contains the Normalized Pointwise Mutual Information (NPMI) between that word and word i, i in{1,2,…,N}. Then, all the word vectors in a topic are aggregated into one big topic vector. The average of all the cosine similarities between each topic word and its topic vector (this is the segmentation) is used to calculate the Cᵥ score.

What is the formula for c_v coherence?


What to do when the means of two samples are significantly different but the difference seems too small to matter

Question: I have two samples ($n \approx 70$ in both cases). The means differ by about twice the pooled std. dev. The resulting $T$ value is approximately 10. Whilst it's great to know that I have conclusively shown that the means are not the same, this seems to me to be driven by the large n. Looking at histograms of the data I certainly do not feel that such as small p-value is really representative of the data and, to be honest, don't really feel comfortable quoting it. I'm probably asking the wrong question. What I'm thinking is: ok, the means are different but does that really matter as the distributions share a significant overlap?
Is this where Bayesian testing is useful? If so, where is a good place to start, a bit of googling hasn't yielded anything useful but I may not by asking the right question. If this is the wrong thing does anyone have any suggestions? Or is this simply a point for discussion as opposed to quantitative analysis?
Comment: FWIW, if your means are 2 SD's apart, that seems like a pretty big difference to me. It will depend on your field, of course, but that's a difference people would easily notice w/ the naked eye (eg, the mean heights of US men & women age 20-29 differ by about 1.5 SD's.) IMO, if distributions don't overlap at all, you don't really need to do any data analyses; at a minimum, w/ $N$ as small as 6, $p$ will be <.05 if the distributions do not overlap.
Comment: I agree the difference is big although utterly irreverent as it turned out.
Comment: I just want to add to all the other answers that your first statement is wrong: **you have NOT conclusively shown that the means are different**. A t-test's p-value is telling you whether the probability of observing your data or more extreme values of it is likely/unlikely *given the null hypothesis* (which for the t-test is $\mu_A=\mu_B$, i.e., $H_0$: { "The means are equal" }), **which does not mean that the means are, in fact, different**.

Also, I assume that you also performed an F-test in order to test the equality of the variances before doing the pooled variance t-test, right?
Comment: Your question is very good since it brings up an important distinction and it shows that you're actually thinking about your data rather than looking for some stars on a statistical output and declaring yourself done. As several answers point out, *statistical significance* is not the same as *meaningful*. And when you think about it, they can't be: how would a statistical procedure know that a statistically significant mean difference of 0.01 means something in Field A, but is meaninglessly small in Field B?
Comment: Fair enough, the langauge wasn't spot on but when the p-value is like the ones i'm getting I tend not to be too fussy about the words. I did do an F-test (and a QQ-plot). Its close enough for jazz, as they say.
Answer: Let $\mu_1$ denote the mean of the first population and $\mu_2$ denote the mean of the second population. It seems that you've used a two-sample $t$-test to test whether $\mu_1=\mu_2$. The significant result implies that $\mu_1\neq\mu_2$, but the difference seems to be to small to matter for your application.
What've you encountered is the fact that statistically significant often can be something else than significant for the application. While the difference may be statistically significant it may still not be meaningful.
Bayesian testing won't solve that problem - you'll still just conclude that a difference exists.
There might however be a way out. For instance, for a one-sided hypothesis you could decide that if $\mu_1$ is $\Delta$ units greater than $\mu_2$ then that would be a meaningful difference that is large enough to matter for your application.
In that case you would test whether $\mu_1-\mu_2\leq \Delta$ instead of whether $\mu_1-\mu_2=0$. The $t$-statistic (assuming equal variances) would in that case be
$$ T=\frac{\bar{x}_1-\bar{x}_2-\Delta}{s_p\sqrt{1/n_1+1/n_2}}$$
where $s_p$ is the pooled standard deviation estimate. Under the null hypothesis, this statistic is $t$-distributed with $n_1+n_2-2$ degrees of freedom.
An easy way of carrying out this test is to subtract $\Delta$ from your observations from the first population and then carry out a regular one-sided two-sample $t$-test.
Answer: It is valid to compare several approaches, but not with the aim of choosing the one that favours our desires/believes.
My answer to your question is: It is possible that two distributions overlap while they have different means, which seems to be your case (but we would need to see your data and context in order to provide a more precise answer).
I am going illustrate this using a couple of approaches for comparing normal means.
1. $t$-test
Consider two simulated samples of size $70$ from a $N(10,1)$ and $N(12,1)$, then the $t$-value is approximately $10$ as in your case (See the R code below).
<code>rm(list=ls())
# Simulated data
dat1 = rnorm(70,10,1)
dat2 = rnorm(70,12,1)

set.seed(77)

# Smoothed densities
plot(density(dat1),ylim=c(0,0.5),xlim=c(6,16))
points(density(dat2),type="l",col="red")

# Normality tests
shapiro.test(dat1)
shapiro.test(dat2)

# t test
t.test(dat1,dat2)
</code>
However the densities show a considerable overlapping. But remember that you are testing a hypothesis about the means, which in this case are clearly different but, due to the value of $\sigma$, there is an overlap of the densities.

2. Profile likelihood of $\mu$
For a definition of the Profile likelihood and likelihood please see 1 and 2.
In this case, the profile likelihood of $\mu$ of a sample of size $n$ and sample mean $\bar{x}$ is simply $R_p(\mu)=\exp\left[-n(\bar{x}-\mu)^2\right]$.
For the simulated data, these can be calculated in R as follows
<code># Profile likelihood of mu
Rp1 = function(mu){
n = length(dat1)
md = mean(dat1)
return( exp(-n*(md-mu)^2) )
}

Rp2 = function(mu){
n = length(dat2)
md = mean(dat2)
return( exp(-n*(md-mu)^2) )
}

vec=seq(9.5,12.5,0.001)
rvec1 = lapply(vec,Rp1)
rvec2 = lapply(vec,Rp2)

# Plot of the profile likelihood of mu1 and mu2
plot(vec,rvec1,type="l")
points(vec,rvec2,type="l",col="red")
</code>
As you can see, the likelihood intervals of $\mu_1$ and $\mu_2$ do not overlap at any  reasonable level.
3. Posterior of $\mu$ using Jeffreys prior
Consider the Jeffreys prior of $(\mu,\sigma)$
$$\pi(\mu,\sigma)\propto \dfrac{1}{\sigma^2}$$
The posterior of $\mu$ for each data set can be calculated as follows
<code># Posterior of mu
library(mcmc)

lp1 = function(par){
n=length(dat1)
if(par[2]>0) return(sum(log(dnorm((dat1-par[1])/par[2])))- (n+2)*log(par[2]))
else return(-Inf)
}

lp2 = function(par){
n=length(dat2)
if(par[2]>0) return(sum(log(dnorm((dat2-par[1])/par[2])))- (n+2)*log(par[2]))
else return(-Inf)
}

NMH = 35000
mup1 = metrop(lp1, scale = 0.25, initial = c(10,1), nbatch = NMH)$batch[,1][seq(5000,NMH,25)]
mup2 = metrop(lp2, scale = 0.25, initial = c(12,1), nbatch = NMH)$batch[,1][seq(5000,NMH,25)]

# Smoothed posterior densities
plot(density(mup1),ylim=c(0,4),xlim=c(9,13))
points(density(mup2),type="l",col="red")
</code>
Again, the credibility intervals for the means do not overlap at any reasonable level.
In conclusion, you can see how all these approaches indicate a significant difference of means (which is the main interest), despite the overlapping of the distributions.
$\star$ A different comparison approach
Judging by your concerns about the overlapping of the densities, another quantity of interest might be ${\mathbb P}(X<Y)$, the probability that the first random variable is smaller than the second variable. This quantity can be estimated nonparametrically as in this answer. Note that there are no distributional assumptions here. For the simulated data, this estimator is $0.8823825$, showing some overlap in this sense, while the means are significantly different. Please, have a look to the R code shown below.
<code># Optimal bandwidth
h = function(x){
n = length(x)
return((4*sqrt(var(x))^5/(3*n))^(1/5))
}

# Kernel estimators of the density and the distribution
kg = function(x,data){
hb = h(data)
k = r = length(x)
for(i in 1:k) r[i] = mean(dnorm((x[i]-data)/hb))/hb
return(r )
}

KG = function(x,data){
hb = h(data)
k = r = length(x)
for(i in 1:k) r[i] = mean(pnorm((x[i]-data)/hb))
return(r )
}

# Baklizi and Eidous (2006) estimator
nonpest = function(dat1B,dat2B){
return( as.numeric(integrate(function(x) KG(x,dat1B)*kg(x,dat2B),-Inf,Inf)$value))
}

nonpest(dat1,dat2)
</code>
I hope this helps.
Comment: (+1) Thanks for a really useful reply on baysian methods. Also the P(X<Y) link answers another issue i've been wondering about in the same analysis.
Answer: Answering the Right Question

ok, the means are different but does that really matter as the distributions share a significant overlap?

Any test that asks whether group means are different will, when it works right, tell you whether means are different.  It will not tell you that distributions of the data itself are different, since that is a different question.   That question certainly depends on the whether the means are different but also on many other things that might be (incompletely) summarised as variance, skew, and kurtosis.
You correctly note that certainty about where the means are depends on the amount of data you have to estimate them, so having more data will allow you spot mean differences in more nearly overlapping distributions.  But you wonder whether

such as small p-value is really representative of the data

Indeed it is not, at least not directly.  And this is by design.  It is representative (approximately speaking) of the certainty you can have that a particular pair of sample statistics of the data (not the data itself) are different.
If you wanted to represent the data itself in a more formal way than simply showing the histograms and testing moments of it, then perhaps a pair of density plots might be helpful.  It rather depends really on the argument you are using the test to make.
A Bayesian Version
In all these respects, Bayesian difference 'tests' and T-tests will behave the same way because they are trying to do the same thing.  The only advantages I can think of for using a Bayesian approach are: a) that it will be easy to do the test allowing possibly different variances for each group, and b) that it will focus on estimating the probable size of the difference in means rather than finding a p-value for some test of difference. That said, these advantages are pretty minor: e.g. in b) you could always report a confidence interval for the difference.
The quote marks above over 'tests' are deliberate.  It is certainly possible to do Bayesian hypothesis testing, and people do.  However, I would suggest that the comparative advantage of the approach is in the focus on building a plausible model of the data and communicating its important aspects with appropriate levels of uncertainty.
Answer: First of all, this is not a problem to pin on frequentist testing.  The problem lies in the null hypothesis that the means are exactly equal.  Therefore if the populations differ in means by any small amount and the sample size is large enough the chance to reject this null hypothesis is very high.  Therefore the p-value for your test turned out to be very small.  The culprit is the choice of the null hypothesis.  Pick d>0 and take the null hypothesis to be that the means differ by less than d in absolute value.  You pick d so that the real difference has to be satisfactorily large to reject.  Your problem goes away.  Bayesian testing does not solve your problem if you insist on a null hypothesis of exact equality of means.
Comment: I was writing my answer at the same time as the other two.


7th chord notation confusion

Question: I simply cannot understand the notation used in this question:

I need to express this chord in 4-part, open or close position; I just can't seem to understand how the notation works. I know that this is a seventh chord; however, I don't think there's enough information to know what type of seventh chord it is.
Note: This is from my AP Music Theory homework; I'm not asking for an answer. I just want to understand the notation.
Answer: The notation builds up by intervals from the bass in close position (although you don't need to realise it in close position). For sevenths, you don't need all three intervals to specify: typically just the two most characteristic are used. In this case, you have an inversion of a minor seventh chord on ii that has a fifth and sixth from the bass in close position, that is, the first inversion where the third of the chord is in the bass. The usual convention is 7 for root position, 6/5 for first inversion, 4/3 for second inversion and 4/2 for third inversion.
Comment: Just to clarify for the OP, the 4/3 and 4/2 inversions have implied "6" in them: "6/4/3" and "6/4/2"
Comment: Thank you, @jjmusicnotes. For completeness (and to keep my OCD from getting out of hand... ;), the notation for root position implies 7/5/3, and, for first inversion, 6/5/3.
Comment: Not a problem; I always appreciate it when people point things out to me.
Answer: The symbol is Roman numeral analysis with figured bass which is more than enough information to build the specific chord. It is telling you that the harmony at that point is a minor 7th (from the lower case of the roman numeral & the figured bass) built on the second scale degree (from the value of the Roman numeral) of Gb major (the note  before the colon and the letter is capital so major) in first inversion (from the figured bass).
Brushing up on Roman numeral analysis and figured bass should be enough to help. MusicTheory.net has a very good lesson on the Roman numeral analysis of 7th chords and building chords based on that.  I suggest starting there and if there are general concept questions about this notation that you still don't  understand you can ask a follow up question here.
Comment: @wu2481632 yes the lower case means minor the actual Roman numeral the  means scale degree.
Comment: Does the "ii" mean that it is both minor and built on the second scale degree? I wasn't aware of that. Thank you.

7th chord notation confusion


Angular ng-block-ui is not working in angular 7?

Question: In my app.module.ts
import { BlockUIModule } from 'ng-block-ui';
imports: [
    BrowserModule,
    BlockUIModule.forRoot(),
]

In my dashboard component
import { BlockUI, NgBlockUI } from 'ng-block-ui';

export class DashboardComponent implements OnInit {

@BlockUI() blockUI : NgBlockUI;

 ngOnInit() {
    this.blockUI.start()
 }

}

Block Ui is not adding element in the Html
i don't know what is the issue the block ui is not starting
maybe the issue is related to angular version or not?
Angular version i am using is 7.
Comment: you should add it to your html too, made this stackblitz with angular 7.
https://stackblitz.com/edit/angular-7-ng-block-ui
Comment: is there any error in console?
Comment: no there is not any error in console
Comment: Can you show us your HTML template? Maybe you forgot to add the `` directive.
Answer: If ng-block-ui directive (*blockUI = "'your-variable'") isn't working in ANGULAR 7, 8 OR 9
Then here is the solution.
First of all, if you followed this mentioned docs of ng-block-ui. And even it isn't working,
Then you need to install :
npm install ng-block-ui@next

this dependency in your angular project.
And if you are getting this error :
ng-block-ui: TypeError: Cannot read property 'localName' of null
at ng-block-ui.js:878

Then you need to install :
npm install ng-block-ui@next

official bug link


Views and Taxonomy Filtering

Question: I have a question regarding filtering in Views.
My setup:
• I have a content-type, for example: Cars
• I have a few different taxonomy term fields attached to that node, something like: Car brand, Car color, Car type, with the following terms:

Vocabulary: Car brand

Term:Car Brand A
Term: Car Brand B
Term:Car Brand C

Vocabulary: Car color

Term: Blue
Term: Red
Term: Black

Vocabulary: Car type

Term: Two seater
Term: SUV
Term: Sedan

I have a View that displays all the available cars, and I used Better Exposed Filters to display a block that displays those taxonomy fields as filters. Now: this is were I get stuck.
Say for example, that 'Car Brand A' only only sells Blue and Black Sedans. What I would like is that the color option 'red' and car types 'SUV' and 'Two seater' are hidden as selectable option, and re-added if a user selects other brands. How do I do that?
I really hope you can help me with this! :)
Answer: If I understand your question correctly, you need the functionality of the Views Selective Filter module.
Although I haven't used this module myself, I've written a similar Views handler in a custom module. This filter handler removes all options from select lists that don't provide any results (due to other selected values). If the Views Selective Filter doesn't provide the desired functionality, I can create a sandbox project of my code.
Comment: Wooh!!! This module sound nice.. :)
Comment: Views Selective Filter did the trick. Documentation is not so good, but it works. Thanks!
Comment: Ok, good to hear your problem is solved :)
Answer: View's default expose filter didn't give such functionality. You may need the Search API module.
I have seen it in Commerce Kickstart package but didn't use it from fresh setup.

Views and Taxonomy Filtering


Can't access MySQL database in Docker container from another container

Question: I have MySQL 5.7 container pulled from here: https://hub.docker.com/_/mysql/
Here's how I run it:
docker run --name mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=pwd -d mysql:5.7

It works good, I'm able to connect to MySQL db from my host machine.
However, when I try to run another container with mysql container linked like this:
docker run --link mysql:mysql -p 8080:8080 -d app:dev

my container can't connect to mysql:
# 172.17.0.3 is mysql's ip taken from /etc/hosts of another container.
mysql -h 172.17.0.3 -u root -ppwd

ERROR 2003 (HY000): Can't connect to MySQL server on '172.17.0.3'

I tried to use docker networks but I'm getting the same error.
Here's nmap -p 3306 172.17.0.2 output:
Starting Nmap 7.01 ( https://nmap.org ) at 2018-06-03 08:34 UTC
Nmap scan report for e66874413058 (172.17.0.2)
Host is up (0.00012s latency).
PORT     STATE  SERVICE
3306/tcp closed mysql

Nmap done: 1 IP address (1 host up) scanned in 0.39 seconds

For unknown reason, the port is closed. If I run nmap command from my host, it's open.
How to connect to MySQL server from another docker container?
Comment: Apparently the problem was with my another container
Answer: I have to admit I don't see immediately where it's going wrong because also IP based communication should work but let me explain the recommended way to let containers communicate.
When you link your app container with the mysql container (like your doing) you can access the mysql just on it's container name without using ip's.
In the default bridge network:
docker run --name mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=pwd -d mysql:5.7

Now I start a random app and link it with mysql. curl and ping are installed in this container.
docker run -d -p 8080:8080 --link mysql:mysql randomapp

Now I go inside my randomapp container and try to ping the mysql container which works.
docker exec -it 7c4bc6f1ca7a bash
xxx@7c4bc6f1ca7a:/$ ping mysql
PING mysql (172.17.0.3) 56(84) bytes of data.
64 bytes from mysql (172.17.0.3): icmp_seq=1 ttl=64 time=0.076 ms
64 bytes from mysql (172.17.0.3): icmp_seq=2 ttl=64 time=0.049 ms

I can verify with an nmap container too
docker@default:~$ docker run --rm --link mysql:mysql uzyexe/nmap mysql 3306

Starting Nmap 7.60 ( https://nmap.org ) at 2018-06-06 05:54 GMT
setup_target: failed to determine route to 3306 (0.0.12.234)
Nmap scan report for mysql (172.17.0.3)
Host is up (0.000010s latency).
Not shown: 999 closed ports
PORT     STATE SERVICE
3306/tcp open  mysql
MAC Address: 02:42:AC:11:00:03 (Unknown)

Nmap done: 1 IP address (1 host up) scanned in 1.65 seconds
docker@default:~$

If you deploy your app and mysql in the same user defined bridge network you don't need to define the --link option and your containers can talk with each other by using their container name.
docker network create my-bridge
docker run --name mysql --net my-bridge -p 3306:3306 -e MYSQL_ROOT_PASSWORD=pwd -d mysql:5.7
docker run -d -p 8080:8080 --net my-bridge randomapp

It's recommended to use user defined networks and not the 'deprecated' --link feature in the default bridge network.
Comment: That's how I tried as well, but the port is still closed. What mysql image are you using?
Comment: Also calling `docker run --rm --network mynetwork uzyexe/nmap 172.17.0.2 3306` outputs `Host seems down`. `172.17.0.3` doesn't work neither. I've connected mysql to mynetwork this way: `docker network connect mynetwork mysql`
Comment: I'm using the official mysql image (latest tag).


Using Hugginface Transformers and Tokenizers with a fixed vocabulary?

Question: I have a special, non-language use case using a fixed vocabulary—i.e., a relatively small set of generated tokens that represent the entire vocabulary of our "language." I'd like to be able to use this with any of the different models and I'm wondering what would be the best approach? It's just a vocab.txt file of short strings, which I don't think will work with any of the BPE tokenizers. Am I correct in that assumption? Also, is there a way to "force" a vocabulary onto any of the tokenizers?

To clarify, our "language" uses prefixes to identify certain types of tokens, which have certain functions in the overall syntax. We want to be able to mask by type during inference, both on input and as part of the selection process, for example, by limiting top-k or top-p sampling to a give type. With a fixed/hand-tuned vocabulary we can be very specific about which ids, or how many ids we need; i.e., we know which tokens are used by each type, so we can mask/filter accordingly. However, with BPE tokenization a given type may be tokenized with any number of tokens, making this process much less straightforward.
The motivation is just to make life easier by fitting into the Huggingface universe a little better, so we can experiment with off-the-shelf models more fluently. We already have this working using the standard BertTokenizer with both GPT2 and RoBERTa, but it would be nice to be able to experiment with different Huggingface models "out of the box," so to speak (using Trainers, Pipelines, etc.). With the BertTokenizer we just load our vocab.txt and we're done, so I wondered whether there would be some way of doing this with the other tokenizers (really, the BPE ones are the only issue, at this point).
It seems to me that being able specify a vocab for any tokenizer would be more straightforward than getting our tokenizer working with other models. Though perhaps a better approach would be to look at streamlining that process? I suppose I could fork and modify AutoTokenizer... ??
Any help much appreciated.
Comment: @dennlinger; for me sounded like that jbm use case will not benefit from any pre-trained weights and that jbm wants to train a transformer from scratch for his own language. Depending on the structure of his language, it might be easier to use a custom tokenizer instead of one of the tokenizer algorithms provided by huggingface. But this is just a maybe until we know more about jbm's language.
Comment: How is one token separated from another in your language? Whitespace? You do not need a tokenizer in this case, just create a class or function that converts your langauge strings into a sequence of ids+attention_mask.
Comment: Without knowing the specifics of your vocab, it is hard to tell whether they will work with any of the BPE tokenizers; I'm not sure if I completely agree with the approach suggested by @cronoik, but could you elaborate a bit more why the use of existing model tokenizers is insufficient?
Comment: @cronoik: That's an interesting idea, thanks.
Comment: @dennlinger: I'll add more explanation to the question; perhaps there's a way to use the normal tokenizers?
Comment: I just realized that, since we're using our custom vocab with `BertTokenizer` for GPT2, then we are already dealing with what would no *normally* require a BPE tokenizer. Gulp... So maybe our current approach is good enough...?
Answer: As far as I understand the solution below might help you, as you can use this tokenizer, like you would the other pre-trained ones.
As I do not really understand all the inner workings of the tokenizer, I may very well be off with this solution, but hopefully it can help someone.
The main idea is to subclass the PreTrainedTokenizer. This way, you should only override some of the key methods like _tokenize, _convert_token_to_id, etc..., which are more straightforward than implementing a whole new tokenizer.
import json
from pathlib import Path
from typing import Optional, Tuple, Dict

from transformers import PreTrainedTokenizer

class FixedVocabTokenizer(PreTrainedTokenizer):
    def __init__(self, vocab: Dict[str, int], max_len: int = None):
        super().__init__(max_len=max_len)
        self.__token_ids = vocab
        self.__id_tokens: Dict[int, str] = {value: key for key, value in vocab.items()}

    def _tokenize(self, text: str, **kwargs):
        return text.split(' ')

    def _convert_token_to_id(self, token: str) -> int:
        return self.__token_ids[token] if token in self.__token_ids else self.unk_token_id

    def _convert_id_to_token(self, index: int) -> str:
        return self.__id_tokens[index] if index in self.__id_tokens else self.unk_token

    def get_vocab(self) -> Dict[str, int]:
        return self.__token_ids.copy()

    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
        if filename_prefix is None:
            filename_prefix = ''
        vocab_path = Path(save_directory, filename_prefix + 'vocab.json')
        json.dump(self.__token_ids, open(vocab_path, 'w'))
        return str(vocab_path),

    @property
    def vocab_size(self) -> int:
        return len(self.__token_ids)

if __name__ == '__main__':
    # your custom, fixed vocabulary
    custom_vocab = {
        '[UNK]': 0,
        'word0': 1,
        'word1': 2,
        'word2': 3,
        'word3': 4,
        'word4': 5,
        'word5': 6,
        '[CLS]': 7,
        '[SEP]': 8,
        '[PAD]': 9
    }
    model_max_len = 8
    tokenizer = FixedVocabTokenizer(custom_vocab, max_len=model_max_len)
    # tell your tokenizer about your special tokens
    tokenizer.add_special_tokens({
        'unk_token': '[UNK]',
        'pad_token': '[PAD]',
        'cls_token': '[CLS]',
        'sep_token': '[SEP]'
    })

    res = tokenizer(
        [
            'word1 word2 word word1 word3',
            'word2 word0 word0 word3 word5 word4 word2 word1 word0'
        ],
        padding=True,
        truncation=True
    )
    # the result should look like something like this
    # res -> BatchEncoding(
    #     data: {
    #         'input_ids': [[2, 3, 0, 2, 4, 9, 9, 9], [3, 1, 1, 4, 6, 5, 3, 2]],
    #         'attention_mask': [[1, 1, 1, 1, 1, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1]],
    #         ...
    #     },
    #     ...
    # )

This is the solution I could come up with, however I could not figure out if you could do something similar with PreTrainedTokenizerFast. So one more note being, that you can only use slow tokenizers using this method.


Dynamically define singleton method on class that evaluates properly with subclass

Question: I'm currently working on a Minitest extension that wraps Minitest::Runnable.run with before/after_suite callbacks. The way that Minitest works is that test classes inherit from Minitest::Runnable. Also, describe blocks dynamically create classes which inherit from the test class they're defined in. So
# Minitest::Spec inherits from Minitest::Runnable
class TestClass < Minitest::Spec
  describe 'a describe block' do
    it 'should say hi' do
      # test here
    end
  end
end

Generates a new class that inherits from TestClass whose name is "a describe block". This winds up going down the Minitest pipeline and calls Mintest::Runnable.run on each class (e.g. self in each case winds up being each of the classes including "a describe block"). For reference:
module Minitest::Runnable
  def self.run reporter, options = {}
    require 'rubygems'; require 'pry'; binding.pry
    filter = options[:filter] || '/./'
    filter = Regexp.new $1 if filter =~ /\/(.*)\//

    filtered_methods = self.runnable_methods.find_all { |m|
      filter === m || filter === "#{self}##{m}"
    }

    with_info_handler reporter do
      filtered_methods.each do |method_name|
        run_one_method self, method_name, reporter
      end
    end
  end
end

My extension is designed to work by allowing you to have a before_suite method which wraps Minitest::Runnable.run:
class TestClass < Minitest::Spec
  before_suite do
    # do something
  end

  describe 'a describe block' do
    it 'should say hi' do
      # test here
    end
  end
end

I've got it mostly working, however I'm running into an issue with inheritance. When I redefine the wrapper method, the receiver of my new singleton method winds up being TestClass even for subclasses (e.g. even for the descibe block). This causes my tests to not be found.
Below is my current implementation of before suite:
module Minitest::SuiteCallbacks
  def self.extended(base)
    base.class_eval do
      class << self
        def before_suite(&before_suite_proc)
          @before_suite_proc = before_suite_proc

          context = self
          original_singleton_run = method(:run)
          define_singleton_method :run do |*args, &block|
            # `self` here winds up being `TestClass` instead of the dynamic class
            # (e.g. self.name => "TestClass" instead of "a describe block")

            context.setup_before_suite
            original_singleton_run.call(*args, &block)
          end
        end
      end
    end
  end
end

This grabs the current implementation of run and wraps it. The issue is that when it's called from the context of the dynamic "a describe block" subclass that self inside of the method is TestClass instead of the dynamic class.
Any thoughts on what I could do to remedy the issue? My goal is to be able to dynamically wrap any method in an inheritance chain and have it work properly with all of the child classes.
Thanks in advance!
Answer: The bug was with the method binding. I either had to unbind the method or open up the singleton class and get reference to the UnboundMethod via instance_method:
def before_suite
  # ...
  original = instance_method(:run)
  class << self
    define_method :run do |*args, &block|
      # add decorated logic
      original.bind(self).call(*args, &block)
    end
  end
end

This was my initial guess, but the error was coming from further down the stack (I wrap the run method again with after_suite and forgot to change that implementation). When in doubt, comment it out!


javascripts added to page

Question: in my html page I do not have any external js file but when I load page in console lots of js scripts are loaded !

and this's html code of page :

<meta charset="utf-8">

    <title>سامانه مدیریت تلفن همراه</title>

    <link href="<?php echo  style_url ?>/css/stylesheets.css" rel="stylesheet" type="text/css" />

<img src="<?php echo  style_url ?>/img/loader.gif" />

<?php
$login=new login('default');
if ($login::check())
{
?>

<?php require "sidebar.view.php" ; } ?>

and this is sidebar page content which included in page :

مدیریت

همگام سازی

                    گزینه ها

UI Elements

Widgets

Buttons

Icons

Grid System

                    مالی

افزودن اعتبار

تراکنش ها

تعرفه ها

Statistic

                    Tables

Simple

Dynamic

                    Samples

FAQ

Invoice

Mailbox

Pricing Table

Timeline

Profile

Login

                    کاربری

تغییر گذر واژه

خروج

<form method="post" id="logout_form">
  <input type="hidden" name="function" value="<?php echo core::encrypt('logout',0) ?>" />
</form>

Thanks for any help
Comment: Not even Google Analytics? Anyway... as we can't see your HTML code, I can't tell you anything more.
Comment: Please take the [tour], have a look around, and read through the [help], in particular [*How do I ask a good question?*](/help/how-to-ask)
Comment: Do you have ads on your page?
Comment: No ads , No google Analytic codes! completely i mixed up
I will add my html code just a moment
Comment: From the domain name in the XHR request, I'd say your server's been compromised.
Comment: And what's in `sidebar.view.php`, since you're including it: `` ?
Comment: that's localhost in windows 7 with easyphp 5.4.6
Comment: @Farhad: Doesn't mean it's not compromised. But again: You haven't shown the content that will make up the page.
Comment: @T.J.Crowder I added sidebar content
Comment: @Farhad: The sidebar uses a `$` function you don't define anywhere. That usually means you're including jQuery somewhere (could be Prototype or MooTools or anything else, really). So where's *that* include?
Comment: @T.J.Crowder I deleted that sidebar but again I have the same result !
Comment: I've added a community wiki answer listing the possibilities. In the end, this isn't an appropriate question for SO, because it boils down to debugging things we can't see.
Answer: There are really only four options here:

You do have a script on your page that you're just missing. (There isn't any in your question except the code that seems to use jQuery even though jQuery isn't loaded — so that suggests what's shown is not the complete story.)
You have a browser extension/add-on (or compromise) which is adding these scripts to the page. (Whether you intentionally installed it, or it was installed by malware [and I include bundles like the Ask Toolbar bundled with Java installations and similar — e.g., installing something completely unrelated unless you untick a subtle tickbox if there even is one — in the term "malware").)
Your server has been compromised and the scripts are being injected by malware. (Yes, this is entirely possible even if it's your local computer. In fact, it's probably more likely.)
It's observation error and the scripts aren't really there.

We can't help you in most of those scenarios.
Comment: problem is related to my own browser I saw that codes in stack overflow consoles too :)
Comment: In both Firefox and chrome I had same result and I don not have extensions so i do not think number two could happen but i upload it to my C panel host to see the result , thanks for your help
Answer: these script are loading from a different source online...if there are other links such css file....ensure to check if there are any external links in the css file..
Solution
1.remove all css links,
2.run the page and inspect elements to view if the scripts are still loading
if the are not the external links are in your css files else the code of your page properly the javascript file link might be withing the body of the page...

javascripts added to page


rails after_save callback conditions

Question: I've a condition for a callback
before_save :method, :if => :email_changed? && :is_admin?

The method will execute even if only the second condition is true i.e is_admin? true.
While it works fine with lambda
before_save :method, :if => lambda { |u| u.email_changed? && u.is_admin? }

Can anyone explain?
Answer: You cannot use &&s, ||s or other similar operator in the bare :if definition because such code is statically executed when the class itself is loaded.
The particular statement :if => :email_changed? && :is_admin? is interpreted like { :if => :email_changed? && :is_admin? } (it's the last parameter of the before_save method which can be written as a hash without the curly braces). This eventually becomes { :if => :is_admin? }, see for yourself in the console:
{  :if => :email_changed? && :is_admin? }
# => { :if => :is_admin? }

The correct way to define this is indeed using a lambda so that it gets evaluated dynamically each time the callback is run.

rails after_save callback conditions


is uniform convergent sequence leads to bounded function?

Question: Suppose there is there is uniform convergent sequence $(f_n)$ on the set $A$, and each $f_n$ is bounded on $A$, i.e., there exist $M_n>0$ such that $|f_n(x)|\le M_n$ for all $x\in A$ Is it true that there exist a function $f$ that is bounded on set A? Can I prove it this way:
Since $f_n \to f$ uniformly on A, $\exists n \ge N$ such that $$|f_N(x) - f(x)| \le 1$$ and $$|f_n(x)-f_N(x)| \le 1$$
Therefore, $|f(x)| \le |f_N(x)|+1 \le M_N + 1$ and $|f_n(x)| \le |f_N(x)| + 1 \le M_N + 1$, $x \in A , n \ge N$
Let $M = \max\{M_1, M_2,\dots,M_N\} + 1$ : $|f_n(x)| \le M$, $\forall x\in A$
Therefore, $f$ is a bounded function on A.
Comment: What does that colon mean? Also, please don't abuse quantifier notation.
Answer: Your proof is almost correct, but it seems quite awkward at points and doesn't state the correct conditions on variables (you have incorrectly quantified $n$ and $N$, for example). What I think you mean is:

By uniform convergence, there exists an $N$ such that $|f_n(x) - f(x)| \le 1$ for all $n \ge N$. Then $$|f(x)| \le |f_N(x)| + 1 \le M_N + 1$$

Note that we can explicitly state the bound in terms of $M_N$ and don't need to take a maximum to bound $f$ itself.


Benefits of using QOpenGLWidget over normal QWidget

Question: Since Qt 5.4 version, the QOpenGLWidget was introduced to enable the OpenGL rendering capabilities.
Apart of calling OpenGL APIs, QOpenGLWidget can also be used as a normal QWidget, in which QPainter is used.
So I'm wondering, if I don't plan to directly call any OpenGL API to render my widget, but only QPainter APIs, is there still any (performance perhaps) benefits of using QOpenGLWidget instead of QWidget?
Comment: It may have benefits (I haven't seen yet high performance using a "casual" QPainter and used `QGLWidget` with OpenGL API; but who knows?). But when you mess with platform+hardware+driver specific things you should think twice. Because your widget may appear differently on each machine, it may be faster or slower, old machines with old drivers may reject new OpenGL features, OpenGL may require additional settings for network display, etc.
Comment: So there is actually no enough motivation to use the QOpenGLWidget, unless intensive OpenGL calls are required, right?
Comment: The purpose of `QOpenGLWidget` is to have a rectangular area inside your window to render OpenGL scene on. `QOpenGLWidget` brings only an overhead, because it needs to be a specific resource with a window ID, so the 3D driver and OS can use it directly.
Comment: @Wayee I don't see the motivation. Maybe someone does, don't know. But for me @Velkan is right, basically `QOpenGLWidget` is a window to use and display OpenGL scenes.
Answer: QOpenGLWidget, when directly painted on using QPainter, does all the painting using OpenGL - that's one of its two main purposes. Using QPainter on a QOpenGLWidget has Qt doing the legwork of translating the painter API into GL state setup and draw calls. If you have some OpenGL background and use a debug build of Qt, you can trace into the source and see how Qt translates your calls, so that you can issue your painter calls in a way that efficiently maps to OpenGL. State changes are expensive, so make sure you batch the operations that use the same pen/brush etc. The painting is done by the QOpenGL2PaintEngineEx.


How do FDR procedures estimate a False Discovery Rate without a model of base rates?

Question: Can someone explain how FDR procedures are able to estimate an FDR without a model / assumption of the base rate of true positives?
Answer: I think that's a really good question; too many people use the Benjamini-Hochberg procedure (abbreviated BH; possibly the most popular procedure to control the FDR) as a black box. Indeed there is an underlying assumption it makes on the statistics and it is nicely hidden in the definition of the p-values!
For a well-defined p-value $P$ it holds that $P$ is uniformly distributed ($P\sim U[0,1]$) under the null hypothesis. Sometimes it might even be that $\Pr[P\leq t] \leq t$, i.e. that $P$ is stochastically smaller than uniform, but this only makes the procedures more conservative (and therefore still valid). Thus, by calculating your p-values, using a t-test or really any test of your choice, you are providing the information about the distribution under the null hypothesis.
But notice here that I kept talking about the null hypothesis; so what you mentioned about knowledge of the base rate of true positives is not needed, you only need knowledge of the base rate of false positives! Why is this?
Let $R$ denote the number of all the rejected (positive) hypotheses and $V$ the false positives, then:
$$ \text{FDR} = \mathbb E\left[\frac{V}{\max(R,1)}\right] \approx \frac{\mathbb E[V]}{\mathbb E[R]}$$
So to estimate the FDR you need a way of estimating $\mathbb E[R]$, $\mathbb E[V]$. We will now look at decision rules which reject all p-values $\leq t$. To make this clear in the notation I will also write $FDR(t),R(t),V(t)$ for the corresponding quantities/random variables of such a procedure.
Since $\mathbb E[R(t)]$ is just the expectation of the total number of rejections, you can unbiasedly estimate it by the number of rejections you observe, so  $\mathbb E[R(t)] \approx R(t)$, i.e. simply by counting how many of your p-values are $\leq t$.
Now what about $\mathbb E[V]$? Well assume $m_0$ of your $m$ total hypotheses are null hypotheses, then by the uniformity (or sub-uniformity) of the p-values under the null you get:
$$\mathbb E[V(t)] = \sum_{i \text{ null}} \Pr[P_i \leq t] \leq m_0 t$$
But we still do not know $m_0$, but we know that $m_0 \leq m$, so a conservative upper bound would just be $\mathbb E[V(t)] \leq m t$. Therefore, since we just need an upper bound on the number of false positives, it is enough that we know their distribution! And this is exactly what the BH procedure does.
So, while Aarong Zeng's comment that "the BH procedure is a way to control the FDR at the given level q. It's not about estimating the FDR" is not false, it is also highly misleading! The BH procedure actually does estimate the FDR for each given threshold $t$. And then it chooses the biggest threshold, such that the estimated FDR is below $\alpha$. Indeed the "adjusted p-value" of hypothesis $i$ is essentially just an estimate of the FDR at the threshold $t=p_i$ (up to isotonization). I think the standard BH algorithm hides this fact a bit, but it is easy to show the equivalence of these two approaches (also called the "equivalence theorem" in the Multiple testing literature).
As a final remark, there do exist methods such as Storey's procedure which even estimate $m_0$ from the data; this can increase power by a tiny bit. Also in principle you are right, one could also model the distribution under the alternative (your true positive base rate) to get more powerful procedures; but so far the multiple testing research has mainly focused on maintaining control of type-I error rather than maximizing power. One difficulty would also be that in many cases each of your true alternatives will have a different alternative distribution (e.g. different power for different hypotheses), while under the null all p-values have the same distribution. This makes the modelling of the true positive rate even more difficult.
Comment: +1 Presumably "BH" refers to [Benjamini-Hochberg](https://en.wikipedia.org/wiki/False_discovery_rate#Benjamini.E2.80.93Hochberg_procedure).  (It's always a good idea to spell out acronyms, lest people misunderstand.)  Welcome to our site!
Comment: Thanks! Also yes you are right, I edited my post to reflect that.
Answer: As suggested by @air, the Benjamini-Hochberg (BH) procedure guarantees FDR control. It does not aim at estimating it. It thus requires a mere weak dependence assumption between test statistics. [1,2]
Methods that aim at estimating the FDR [e.g. 3,4,5] do require some assumptions on the generative process in order to estimate it.
They typically assume test statistics are independent.
They will also assume something on the null distribution of the test statistics. Departures from this null distribution, together with the independence assumption, can thus be attributed to effects, and the FDR may be estimated.
Note that these ideas reappear in the semi-supervised novelty detection literature. [6].
[1] Benjamini, Y., and Y. Hochberg. "Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing." JOURNAL-ROYAL STATISTICAL SOCIETY SERIES B 57 (1995): 289–289.
[2] Benjamini, Y., and D. Yekutieli. "The Control of the False Discovery Rate in Multiple Testing under Dependency." ANNALS OF STATISTICS 29, no. 4 (2001): 1165–88.
[3] Storey, J.D. "A Direct Approach to False Discovery Rates." Journal Of The Royal Statistical Society Series B 64, no. 3 (2002): 479–98. doi:10.1111/1467-9868.00346.
[4] Efron, B. "Microarrays, Empirical Bayes and the Two-Groups Model." Statistical Science 23, no. 1 (2008): 1–22.
[5] Jin, Jiashun, and T. Tony Cai. "Estimating the Null and the Proportion of Nonnull Effects in Large-Scale Multiple Comparisons." Journal of the American Statistical Association 102, no. 478 (June 1, 2007): 495–506. doi:10.1198/016214507000000167.
[6] Claesen, Marc, Jesse Davis, Frank De Smet, and Bart De Moor. "Assessing Binary Classifiers Using Only Positive and Unlabeled Data." arXiv:1504.06837 [cs, Stat], April 26, 2015. http://arxiv.org/abs/1504.06837.
Comment: You are right that following [3, Eq.2.5], one may see the BH procedure as using a conservative estimate of the FDR with $p_0=1$.
Comment: +1 though my main point from that paragraph was that the BH procedure actually *does* suggest a way of estimating the FDR (albeit a bit conservatively) and in fact *does* estimate it to arrive at the final rejection threshold. Its algorithmic definition as a step-up procedure in reference [1] obscures this, but at the end of the day estimation of FDR is exactly what the BH procedure does!! (Efron often makes that point, but also see Section 4. "A connection between the two approaches" in your reference [3].)
Answer: When the true underlying model is unknown, we cannot compute the FDR, but can estimate the FDR value by permutation test. Basically the permutation test procedure is just doing the hypothesis test multiple times by change the outcome variable vector with its permutations. It can also be done based on the permutations of the samples, but not as common as the former one.
The paper here reviews the standard permutation procedure for FDR estimation, and also proposed a new FDR estimator. It should be able to address your question.
Comment: The most common procedure like BH doesn't use a permutation test. What does it use? Also, permutation tests usually provide a distribution under the null, doesn't an FDR estimate require models of both the null and alternative as well as the underlying relative proportion of each?
Comment: First, the BH procedure is a way to control the FDR at the given level $q$. It's not about estimating the FDR. Second, the permutation tests are conducted under the null of all hypotheses. I am not sure what you mean by "require models of both the null and alternative as well as the underlying relative proportion of each". But when you set up your hypotheses, you already have your null and alternative pairs. Does this make sense?


ICA: plotting independent components

Question: How do I find the directions of the independent components if I have already found the mixing matrix?
Let's say that I have this mixing matrix:
$$\mathcal W = \begin{bmatrix} 2 & -2 \\ 2 & 4  \end{bmatrix}$$
so that $\boldsymbol{x}=W\boldsymbol{h}$, where $\boldsymbol{h}$ are the hidden sources: $p(\boldsymbol{h}) = \frac{1}{4}\prod_{i=1}^2exp(|-h_i|)$
I'd like to sketch the independent components and the contours of the distribution in $\boldsymbol{x}$-space.
Answer: I have an answer. A simple way to sketch the distribution is to project the x and y axis first so that it is easier to "see" what happens to the projected contours of the distribution. In this case for example projecting the x axis gives the direction of one of the two independent components, which is the line passing from $(0,0)$ and $(2,2)$:
$$ \begin{bmatrix} 2 & -2 \\ 2 & 4  \end{bmatrix} \begin{bmatrix} 1 \\ 0\end{bmatrix} = \begin{bmatrix} 2 \\ 2\end{bmatrix}$$
And similarly the other direction is found by projecting the y axis:
$$ \begin{bmatrix} 2 & -2 \\ 2 & 4  \end{bmatrix} \begin{bmatrix} 0 \\ 1\end{bmatrix} = \begin{bmatrix} -2 \\ 4\end{bmatrix}$$

ICA: plotting independent components