mod_proxy_wstunnel - Mac OS X 10.11.6, Apache 2.4.18

Question: I've been trying unsuccessfully for a few days to setup a reverse proxy to a localhost websocket url.
ProxyPass /chat/stream/ wss://localhost:8000/chat/stream/
ProxyPassReverse /chat/stream/ wss://localhost:8000/chat/stream/

I get an error in the apache error_log that reads:

No protocol handler was valid for the URL /chat/stream/. If you are
  using a DSO version of mod_proxy, make sure the proxy submodules are
  included in the configuration using LoadModule.

I have read countless pages via google of people using this method so I wonder if there is some issue in our setup/install of Apache that ships with Server.app 5.2?
I have all the standard modules loaded in httpd_server_app.conf
mod_proxy
mod_proxy_wstunnel
mod_proxy_http
...
Can anyone shed some light on this?
Thanks
Adam
Comment: What happens if you replace `wss://` by `http://` ?
Comment: Did you load mod_proxy_wstunnel with `sudo a2enmod mod_proxy_wstunnel` ?
Comment: If the module is correctly loaded you should see it in the output of `sudo apache2ctl -M`
Comment: Using http works however the web app needs to connect over wss (web sockets?) - the client receives a  200 response and the handshake fails. If I use the web app on the server directly via localhost:8000 the app works fine.
Comment: I'm using Mac OS X - from what I understand the mod_proxy_wstunnel module is loaded in the httpd_server_app.conf file - along with mod proxy
Comment: Yes it looks to be loaded along with the proxy module: proxy_wstunnel_module (shared)
Comment: LoadModule proxy_wstunnel_module libexec/apache2/mod_proxy_wstunnel.so
Comment: some more info from the error log:
Answer: In case someone finds themselves in a similar situation here is how I got WebSocket connections working via Apache in MacOS Server 5.2.
And the solution is simple.
The short version:
I use MacOS Server 5.2 (ships with Apache 2.4.23) to run a python Django application via the mod_wsgi module.
I had been trying to setup proxypass and wstunnel in MacOS 10.12 & Server 5.2 to handle websocket connections via an ASGI interface server called Daphne running on localhost on port 8001.
I wanted to reverse proxy any WebSocket connection to wss://myapp.local/chat/stream/ to ws://localhost:8001/chat/stream/
From what I had read on all the forums and mailing lists that I had scoured was to simply make some proxypass definitions in the appropriate virtual host and make sure that the mod_proxy and mod_proxy_wstunnel modules were loaded and it would work.
Long story short - from what I understand all of this trouble came down to MacOS Server 5 and one major change:

"A single instance of httpd runs as a reverse proxy, called the Service Proxy, and several additional instances of httpd run behind that proxy to support specific HTTP-based services, including an instance for the Websites service."

All I needed to do to proxy the websocket connection was the following:
in: /Library/Server/Web/Config/Proxy/apache_serviceproxy.conf
Add (around line 297 (in the section about user websites, webdav):
ProxyPass / http://localhost:8001/
ProxyPassReverse / http://localhost:8001/

RewriteEngine on
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
RewriteRule .* ws://localhost:8001%{REQUEST_URI} [P]

I then kicked over the service proxy:
sudo launchctl unload -w  /Applications/Server.app/Contents/ServerRoot/System/Library/LaunchDaemons/com.apple.serviceproxy.plist
sudo launchctl load -w /Applications/Server.app/Contents/ServerRoot/System/Library/LaunchDaemons/com.apple.serviceproxy.plist

And the web socket connections were instantly working!
The long version:
For many weeks I have been trying to get WebSocket connections functioning with Apache and Andrew Godwin's Django Channels project in an app I am developing.
Django Channels is

"a project to make Django able to handle more than just plain HTTP requests, including WebSockets and HTTP2, as well as the ability to run code after a response has been sent for things like thumbnailing or background calculation."

My interest in Django Channels came from my requirement of a chat system in my webapp. After watching a several of Andrew's demos on youtube and having read through the docs and eventually installing Andrew's demo django channels project I figured I would be able to get this working in production on our MacOS Server.
The current release of MacOS 10.12 and Server 5.2 ships with Apache 2.4.23. This comes with the necessary mod_proxy_wstunnel module to be able to proxy WebSocket connections (ws:// and secure wss://) in Apache and is already loaded in the server config file:
/Library/Server/Web/Config/apache2/httpd_server_app.conf

Daphne is Andrew's ASGI interface server that supports WebSockets & long-poll HTTP requests. WSGI does not.
With Daphne running on localhost on a port that MacOS isn't occupying (I went with 8001), the idea was to get Apache to reverse proxy certain requests to Daphne.
Daphne can be run on a specified port (8001 in this example) like so (-v2 for more verbosity):
daphne -p 8001 yourapp.asgi:channel_layer -v2

I wanted Daphne to handle only the web socket connections (as I use currently depend on some apache modules for serving media such as mod_xsendfile). In my case the websocket connection was via /chat/stream/ based on Andrew's demo project.
From what I had read in MacOS Server's implementation of Apache the idea is to declare these proxypass commands inside the virtual host files of your "sites" in: /Library/Server/Web/Config/apache2/sites/
In config files such as: 0000_127.0.0.1_34543_.conf
I did also read that any customisation for web apps running on MacOS Server should be made to the plist file for the required web app in: /Library/Server/Web/Config/apache2/webapps/
In a plist file such as: com.apple.webapp.wsgi.plist
Anyway...
I edited the 0000_127.0.0.1_34543_.conf file adding:
ProxyPass /chat/stream/ ws://localhost:8001/
ProxyPassReverse /chat/stream/ ws://localhost:8001/

Eager to test out my first web socket chat connection I refreshed the page only to see an error printed in the apache log:
No protocol handler was valid for the URL /chat/stream/. If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.

I had read of many people finding a solution at least with Apache on Ubuntu or a custom install on MacOS.
I even tried installing Apache using Brew and when that didn't work I almost proceeded to install nginx.
After countless hours/days of googling I reached to the Apache mailing list for some help with this error. Yann Ylavic was very generous with his time and offered me various ideas on how to get it going. After trying the following:
SetEnvIf Request_URI ^/chat/stream/ is_websocket
RequestHeader set Upgrade WebSocket env=is_websocket
ProxyPass /chat/stream/ ws://myserver.local:8001/chat/stream/

I noticed that the interface server on port 8001 Daphne was starting to receive ws connections!
However in the client browser it was logging:
"Error during WebSocket handshake: 'Upgrade' header is missing"

From what I could see mod_dumpio was logging that the "Connection: Upgrade" and "Upgrade: WebSocket" headers were being sent as part of the web socket handshake:
mod_dumpio:  dumpio_in (data-HEAP): HTTP/1.1 101 Switching Protocols\r\nServer: AutobahnPython/0.17.1\r\nUpgrade: WebSocket\r\nConnection: Upgrade\r\nSec-WebSocket-Accept: 17WYrMeMS8a4ImHpU0gS3/k0+Cg=\r\n\r\n
mod_dumpio.c(164): [client 127.0.0.1:63944] mod_dumpio: dumpio_out
mod_dumpio.c(58): [client 127.0.0.1:63944] mod_dumpio:  dumpio_out (data-TRANSIENT): 160 bytes
mod_dumpio.c(100): [client 127.0.0.1:63944] mod_dumpio:  dumpio_out (data-TRANSIENT): HTTP/1.1 101 Switching Protocols\r\nServer: AutobahnPython/0.17.1\r\nUpgrade: WebSocket\r\nConnection: Upgrade\r\nSec-WebSocket-Accept: 17WYrMeMS8a4ImHpU0gS3/k0+Cg=\r\n\r\n

However the client browser showed nothing in the response headers.
I was more stumped than ever.
I explored the client side jQuery framework as well as the  the Django channels & autobahn module to see if perhaps something was amiss, and then revised my own app and various combinations of suggestions about Apache and it's module. But nothing stood out to me.
Then I reread the ReadMe.txt inside the apache2 dir: /Library/Server/Web/Config/apache2/ReadMe.txt

"Special notes about the web proxy architecture in Server application
  5.0:
This version of Server application contains a revised architecture for all HTTP-based services. In previous versions there was a single instance of httpd acting as a reverse proxy for Wiki, Profile, and Calendar/Address services, and also acting as the Websites service. With this version, there is a major change: A single instance of httpd runs as a reverse proxy, called the Service Proxy, and several additional instances of httpd run behind that proxy to support specific HTTP-based services, including an instance for the Websites service.
Since the httpd instance for the Websites service is now behind a reverse proxy, or Service Proxy, note the following: ...  It is only the external Service Proxy httpd instance that listens on TCP ports 80 and 443; it proxies HTTP requests and responses to Websites and other HTTP-based services. ... "

I wondered if thus ServiceProxy had something to do with it. I had a look over: /Library/Server/Web/Config/Proxy/apache_serviceproxy.conf
and noticed a comment - "# The user websites, and webdav".
I figured it wouldn't hurt to try adding the proxypass definitions & rewrite rules that people had suggested on the forums as their solution.
ProxyPass / http://localhost:8001/
ProxyPassReverse / http://localhost:8001/

RewriteEngine on
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
RewriteRule .* ws://localhost:8001%{REQUEST_URI} [P]

Sure enough after restarting the ServiceProxy it all started to work!
Adam
Comment: Guy... you rock!
Answer: Take a look at the ReadMe.txt file in /Library/Server/Web/Config/apache2/
It writes about the new proxy service where requests first go to an http server running on port 80 and 443, which are then forwarded to internal ports 34580 and 34543.
For the wstunnel module there's a conflict with the mod proxy module, where mod proxy will strip out tunnel headers (https://lists.gt.net/apache/users/393509). I confirmed this by changing the LogFormat in apache_serviceproxy.conf by adding to LogFormat
c:\"%{Connection}i\" u:\"%{Upgrade}i\"

I did the same in httpd_server_app.conf and could see the Connection and Upgrade websocket headers were being removed before getting to my webapp.
The fix was merely to add a file in /Library/Server/Web/Config/Proxy for my application. Look at the last line in apache_serviceproxy.conf to see the expected naming format. In my case the file is called apache_serviceproxy_customsites_ws.conf
The contents:
ProxyPass /ws/ ws://localhost:34543/ws/
ProxyPassReverse /ws/ ws://localhost:34543/ws/

This will forward the ws request to the expected internal https port 34543 and retain the headers. You must forward it to 34543 or 34580. And also note the path is included so that it can be picked up in the next step.
Then, in my webapp_script (the include file for my webapp) I have:
ProxyPass "/ws/"  "ws://localhost:61614/"
ProxyPassReverse "/ws/"  "ws://localhost:61614/"

This forwards the request to my websocket server running on port 61614.
With that, it's now working as expected.
Answer: In addition to what @adamteale mentioned in his answer (the TL;DR) version, I also had to add
ProxyPreserveHost On

in my apache virtual host config. Without this, Daphne wasn't returning back any response. Perhaps it was, but it wasn't being sent back to the client.
The only other unresolved issue for my setup is that daphne isn't handling wss connection. Either Apache isn't terminating SSL for wss or something else is happening. It does terminate SSL for all other http requests. However, that question is for another thread. I will raise it as soon as I am done with more research on it.
Note: Couldn't add comments to Adam's answer due to MyCurrentRep < 50


Making menu items active on desired page

Question: I am creating a custom module for menus through which we can allow individual menu link to be active on desired path/paths. The module interface provides similar functionality as block module's visibility configuration.
I am using hook_translated_menu_link() to implement the visibility functionality. As I have referred menu item visibility module for this, but not getting the desired output. Do i need to implement any other hook for this or I am missing out something. As I have implemented hook_form_id_form_alter() for the interface and mymodule_submit() for form submit functionality and right now trying out hook_translated_menu_link() for visibility functionality.  I also have a database entry for my module.
Answer: Is there a way you could make the Context module a dependency for your module and hook into the Context module to do this?  Context already allows you to do this exact function.  You specify conditions and reactions when those conditions are met, per context.  In your case you would specify particular paths as conditions, and then your reaction would be to simply activate a particular menu item.  I would seriously consider checking it out rather than reinventing the wheel.
Answer: The documentation for hook_translated_menu_link_alter() says:

It is only invoked if $item['options']['alter'] has been set to a non-empty value (e.g., TRUE). This flag should be set using hook_menu_link_alter().

You need to also implement hook_menu_link_alter().
As said by Lester Peabody, the functionality you want should be already implemented by the Context module. If the functionality it's not exactly what you want, you should implement hook_translated_menu_link_alter(), and hook_menu_link_alter().


Anyway i can improve this multi class classification result?

Question: I am building a multi class classification model using SVM to predict the grade for essays. What can I do to improve the result especially for class 1 and class 3? Their precision and recall are really bad.
Things I have done:
1. Split train:test set to 70:30
2. Oversample the training set using smote
3. Scale the feature by using StandardScaler from Scikit-Learn
4. Perform parameter tuning by using GridSearchCv from Scikit-Learn to get the best parameter
Answer: First, I think your workflow makes sense but you have a problem: the number of observations is so different in all three classes. When you do your cross-validation completely random you naturally oversample the most prevalent class. Therefore you optimize the model to classify these observations correct. Intuitively, if you correctly classify a lot of the obs from class 2 your clf will achieve a decent performance. The 'low' number of observations from 1 and 3 are just a 'nice to have' if you classify them correctly.
One way out is sampling a similar number observations for each class or assigning weights to the classes. Lastly, in the multi-class case remember, that you are not really training a classifier for class 1, 2 or 3 with an SVM. It's either one-vs-all or all-vs-all, both options are also available in sklearn.
This is the first thing I would try and then move maybe also to other classifiers.


Existe alguma forma de comentar múltiplas linhas em Python?

Question: Para comentar uma linha, usamos o #.
Gostaria de de saber se é possível comentar múltiplas linhas em Python 3.
Se sim, como devo fazer?
Comment: Se estiver usando o Sublime Text e quiser comentar múltiplas linhas, basta selecioná-las e apertar `CTRL` + `/`
Answer: Assim como colocado na PEP 257 (That's PEP), strings que são a primeira instrução em um módulo, classe ou função serão consideradas como strings especiais: as docstrings.

A docstring is a string literal that occurs as the first statement in a module, function, class, or method definition.

As docstrings são utilizadas pelo próprio Python para documentação e ficam acessíveis através de seus objetos pelo campo __doc__.
Mais adiante afirma também que qualquer outra string em qualquer outro ponto do código será tratada como comentário e, portanto, ignorado pelo interpretador.

String literals occurring elsewhere in Python code may also act as documentation.

Então, muito cuidado! Enquanto os comentários iniciados por # serão sempre comentários, aqueles iniciados por aspas, seja simples ou dupla, podem não ser comentários.
""" Isto NÃO É um comentário.

String com múltiplas linhas que será docstring do módulo.
"""

# Isso É um comentário

def hello():
    'Isso NÃO É um comentário'
    print('hello')  # Isso É um comentário
    'Isso É um comentário'

Você pode confirmar isso utilizando o módulo dis:
  4           0 LOAD_CONST               0 (' Isto NÃO É um comentário.\n\nString com múltiplas linhas que será docstring do módulo.\n')
              2 STORE_NAME               0 (__doc__)

  8           4 LOAD_CONST               1 (<code object hello at 0x7f340fc28270, file "<dis>", line 8>)
              6 LOAD_CONST               2 ('hello')
              8 MAKE_FUNCTION            0
             10 STORE_NAME               1 (hello)
             12 LOAD_CONST               3 (None)
             14 RETURN_VALUE
None

Veja funcionando no Repl.it
Perceba que a primeira coisa feita é armazenar a string com múltiplas linhas em __doc__, pois, como é a primeira instrução do módulo, será a docstring deste. O comentário com # foi completamente ignorado ao gerar os opcodes pelo interpretador. Perceba que as strings dentro da função foram consideradas dentro de outra análise, exibida ali como "code object hello". Se analisar apenas o código interno a função, teremos:
  2           0 LOAD_CONST               0 ('Isso NÃO É um comentário')
              2 STORE_NAME               0 (__doc__)

  3           4 LOAD_NAME                1 (print)
              6 LOAD_CONST               1 ('hello')
              8 CALL_FUNCTION            1
             10 POP_TOP

  4          12 LOAD_CONST               2 (None)
             14 RETURN_VALUE
None

Veja funcionando no Repl.it
Onde, novamente a primeira string não será um comentário, pois será interpretada como docstring da função, enquanto as outras foram ignoradas.
Concluindo, você pode utilizar qualquer variação de string do Python como comentário, desde que esta não seja interpretada como uma docstring.
Answer: Segue abaixo um exemplo:
'''
Isso e um comentario multilinhas em Python.
'''

Comment: Só um detalhe para esclarecer aos leitores. Isso na verdade é uma string multilinha, que por acaso não vai ser usada para nada da forma que foi declarada, por não ter sido usada em função tampouco atribuida a alguma variável. Esta postagem ajuda a entender: https://pt.stackoverflow.com/questions/80551/70
Comment: Mais que isso, a primeira ocorrência no módulo é tratada como docstring. Mais detalhes em https://www.python.org/dev/peps/pep-0257/
Answer: No pycharm você pode configurar no settings>keymap, geralmente vem configurado como padrão Ctrl+shit+.
segue exemplo visual:
Answer: Utilize """ comentário """ para comentar em múltiplas linhas.
Comment: Vale o comentário que fiz na outra resposta


Maximum Likelihood estimation with a "dependent parameter"?

Question: Suppose we have a likelihood function
$$L(\theta | X) = f(X; \alpha, \beta, \gamma(\beta))$$
So that the third parameter, $\gamma$, depends on $\beta$ in such a way that they cannot be separated. For example, suppose
$$\gamma (\beta) = \int_0^1 g(s) e^{\beta s} ds$$
for some unknown function $g(s)$. Since we cannot reasonably estimate $g(s)$, we instead wish to replace the integral with a single parameter to estimate. Unfortunately, due to the integral's dependence on $\beta$, we cannot derive maximum likelihood estimates.
I have two questions:

Is there any well known procedure for over coming a situation like this?
What would happen if we just "ignore" $\gamma$'s dependence on $\beta$? So that we instead consider the function

$$f(X; \alpha, \beta, \gamma)$$
and derive maximum likelihood estimates from that? Would they still be consistent? Can anything "good" be said about this situation?
Comment: If you initially forget all the assumptions about $\gamma$ and simply estimate the three parameters, you will then have estimates $\hat\beta$ and $\hat\gamma$. If you wish you may choose any function $\hat g$ so that $\hat\gamma = \int_0^1\hat{g}(s)e^{{\hat\beta}s}ds$. Thus, you don't seem to have formulated anything any different than a standard probability model and a standard maximum likelihood estimate. This also demonstrates the impossibility of estimating $g$.
Comment: My point is that the function $g$ is superfluous: it adds nothing to the model and doesn't constrain it in any way.  If you make ML estimates of the parameters, they will of course be MLEs.
Comment: In this case the value of $\sigma$ does not constrain the allowable range of values for the mean $\kappa = \mu \sigma$, so there is no reason to treat $\kappa$ as being dependent on $\sigma$.
Comment: @whuber  I guess what I'm asking is, by ignoring the dependence between the two parameters, how would this affect the accuracy of these estimates? Presumably they are no longer MLE estimates, and so none of the nice properties MLE estimates posses are present. So I'm wondering if that route is even worth while
Comment: I apologise for being slow here, but just to make sure I understand, I'll bring it to a simpler example. Suppose we have an i.i.d sample from a $N(\mu \sigma, \sigma^2)$ distribution. Does this mean that if we let $\kappa = \mu \sigma$ and derived MLE estimates for $\kappa, \sigma$, while ignoring $\kappa$'s dependence on $\sigma$, the resulting estimates would still be MLE estimates and therefore consistent?
Answer: To deal with maximum likelihood estimation, you must formulate your problem in such a way that there is a clear range of allowable values for the unknown parameters.  In your problem you need to formulate an allowable range of values for the parameter $\gamma$ for each possible value of the dependent parameter $\beta$.  To do this you will need to consider what is the allowable range of functions $g$.
For example, suppose you have some independent function $g \in \mathcal{G}$ where the range of this function does not depend on the parameters in your problem.  You can then define $\mathcal{Y} (\beta) \equiv \{ \gamma(\beta) | g \in \mathcal{G} \}$ for each allowable value of $\beta$ and you have the partially-maximised likelihood:
$$\bar{L}(\alpha, \beta) \equiv \max_{g \in \mathcal{G}} L(\alpha, \beta, g) = \max_{\gamma \in \mathcal{Y}(\beta)} L(\alpha, \beta, \gamma).$$
This tells you that maximising the likelihood function over the set of allowable functions $g$ is the same as maximising it over the corresponding set of allowable parameters $\gamma$.  The latter requires you to specify the range of $\gamma$, and the partially-maximised likelihood then depends on $\beta$ through its direct appearance in the likelihood function, and also through its effect on the allowable range of $\gamma$.

In your problem, you have not been clear about the range of allowable functions $g$ (I am assuming this is some real function).  If you impose no constraint on this, you will have $\mathcal{Y} (\beta) = \mathbb{R}$ for all $\beta$, which gives $\bar{L}(\alpha, \beta) = \max_{\gamma \in \mathbb{R}} L(\alpha, \beta, \gamma)$.  In this case, there is no problem of parametric dependence and so this is an ordinary maximum-likelihood problem.
Answer: Part 1
Can you model $g(s)$ as a polynomial? If so, I believe you could turn $\gamma(\beta)$ into a closed form expression by analytically computing the integral. In general, if you can find some approximation of $g(s)$ that makes the integral analytically solvable, then you're back to ordinary maximum likelihood (without a $\gamma$ at all).
Let's say you don't know much about g(s). But you must know something! So put a Gaussian Process prior on it with mean function F(s). Use a tool like stan, sample the function during an MCMC draw and perform numerical integration within the loop. You can see me trying this here (see line 96). Now it's slow, but it is a way to estimate and incorporate uncertainty in $g(s)$.
If you have a complex nonlinear expression for $g(s)$ but don't want to go the bayesian route, you could incorporate something like the trapezoidal rule in your maximum likelihood routine, but it too will be slow!
Part 2
In order to even speak of dependence between parameters, you have to adopt a bayesian viewpoint, since the classical viewpoint has all parameters as fixed unknown quantities. If you take the classical interpretation, then I think you're okay just estimating $\gamma$ as a constant. In fact, why not use the estimates of $\beta$ and $\gamma$ to gain insight into $g(s)$?


How to use the MediaController

Question: I'm trying to integrate a MediaController on top of a MediaPlayer.
The controller is displayed all right, but its buttons are disabled, and the progress slider does not respond properly.
I found very little documentation about it and I still don't understand how to use it.
This is very important and urgent for me.
Can anyone who has dealt with it before explain how it works?
Thanks,
Eyal
Comment: can u put some code on how you are integrating them
Answer: Check out the Api demos under your SDK directory. There is an example in there.

How to use the MediaController


Fixed or Random Effects model? Data on murder rate and the death penalty

Question: I have a dateset that includes information on U.S. states in 1987, 1990 and 1993 and describes the relationship between murder rate and the death penalty. mrdrte is the number of murders per 100,000 population, exec is the number of total executions in past 3 years, and unem is an unemployment rate. We model:
$$mrdrte_{it} = \beta_0 + \beta_1 exec_{it} + \beta_2 unem_{it}+a_i+u_{it}$$
Below I have labelled my question in bold and big font. I do not know how to proceed with my analysis without answering them.
Descriptives
. xtsum mrdrte exec unem

Variable         |      Mean   Std. Dev.       Min        Max |    Observations
-----------------+--------------------------------------------+----------------
mrdrte   overall |  8.070588   9.192867         .8       78.5 |     N =     153
         between |             8.768838   1.333333   64.16667 |     n =      51
         within  |              2.93735  -19.89608   22.40392 |     T =       3
                 |                                            |
exec     overall |  1.228758   3.791432          0         34 |     N =     153
         between |             3.440601          0   22.33333 |     n =      51
         within  |             1.641031  -10.10458   12.89542 |     T =       3
                 |                                            |
unem     overall |  5.973203   1.680617        2.2         12 |     N =     153
         between |             1.316781   3.233333   9.966667 |     n =      51
         within  |             1.055167   3.573203   9.439869 |     T =       3

Question: max and min look strange, how can the be like that?
Is overall - between - within = 0 ? We check using R.
> 1.680617^2-1.316781^2-1.055167^2
[1] -0.0228161
> 3.791432^2-3.440601^2-1.641031^2
[1] -0.1557614
> 9.192867^2-8.768838^2-2.93735^2
[1] -1.011741

No. Why not?
Question: The sum is not 0. Is this a sign of an error?
Between is between different id's (i.e. we fix $t$ and move $i$). We have $n=51$ different id's, so $i$ go from 1 to 51. For example between different id's the Std.Dev. for $mrdrte$ (murder rate) is $8.768838$. Between different id's the exec Std.Dev. is $3.791432$.
Within is within the same id, over time (i.e. we fix $i$ and move $t$). We have $T=4$ time periods, so $t$ go from 1 to 4. For example within $mrdrte$ Std.Dev is $2.93735 < 8.768838$ so apparently between different id's. varies mor than within individuals over time.
We can conclude that $mrdrte$ varies more between different id's than within the same one.
So adding another $n$ gives, loosely speaking, more information than adding another time period $T$ and this is good for our analysis  since in this dataset $n>T$.
Question: How can we have zero variation in the table? What is the reason here?
By looking at these numbers, we can conclude there is a large portion of between variation in the data. This will be important later on when choosing FE or RE.
Should we use FE or RE?
In practice you could run them both and see if they differ, and if not - we need not worry about the choice. We run them both and report the output below. I set $i$ to be id and $t$ to be year.
. xtset id year

. xtreg mrdrte exec unem, fe

Fixed-effects (within) regression               Number of obs      =       153
Group variable: id                              Number of groups   =        51

R-sq:  within  = 0.0047                         Obs per group: min =         3
       between = 0.0007                                        avg =       3.0
       overall = 0.0002                                        max =         3

                                                F(2,100)           =      0.24
corr(u_i, Xb)  = -0.0635                        Prob > F           =    0.7909

------------------------------------------------------------------------------
      mrdrte |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        exec |  -.1140743   .1800836    -0.63   0.528    -.4713551    .2432065
        unem |    .095914   .2800721     0.34   0.733    -.4597411    .6515692
       _cons |   7.637844   1.684436     4.53   0.000     4.295971    10.97972
-------------+----------------------------------------------------------------
     sigma_u |   8.788124
     sigma_e |   3.612922
   rho |  .85542114   (fraction of variance due to u_i)
------------------------------------------------------------------------------
F test that all u_i=0:     F(50, 100) =    16.46             Prob > F = 0.0000

. xtreg mrdrte exec unem, re

Random-effects GLS regression                   Number of obs      =       153
Group variable: id                              Number of groups   =        51

R-sq:  within  = 0.0015                         Obs per group: min =         3
       between = 0.0732                                        avg =       3.0
       overall = 0.0433                                        max =         3

                                                Wald chi2(2)       =      0.90
corr(u_i, X)   = 0 (assumed)                    Prob > chi2        =    0.6369

------------------------------------------------------------------------------
      mrdrte |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        exec |  -.0351956   .1619968    -0.22   0.828    -.3527036    .2823124
        unem |   .2560543   .2708762     0.95   0.345    -.2748532    .7869619
       _cons |   6.584371   2.001338     3.29   0.001     2.661819    10.50692
-------------+----------------------------------------------------------------
     sigma_u |  8.1923983
     sigma_e |   3.612922
         rho |  .83717807   (fraction of variance due to u_i)
------------------------------------------------------------------------------

.

They are not similar. So which one can we take?
There are two kinds of variation in the data: within and between. FE uses only within. RE within and between and weighs them in a smart way, and the close $\hat \theta$ is to 1 the closer RE is to FE (when $\hat \theta = 1 \implies FE=RE.)$
In our case, $\hat \theta = .83717807$ (rho in stata output) and the closer it is to 1, the closer is RE to FE. %källa s34
0.83 is pretty high. So this is piece of evidence for using RE.
%qq is this really the $\theta$ or is the output telling me something else?
Remember from above that a large portion of between variation in the data. This is another piece of evidence for using RE.
Recall our model. In RE we assume $Cov( exec_{it}  ,  a_i ) = 0$ and $Cov( unem_{it}  , a_i ) = 0$. This is rarely the case, but do we violate this assumption? The correlation is weak - Stata reports that $corr = -0.0635$
Question: is this large or small? but more importantly - is this the correlation I'm really interested in checking?
It is close to zero so the assumptions is almost fulfilled.
Did this choice of RE matter? Yes it did. The output is different, of course, but more importantly we know in general that if the RE assumptions are fulfilled then (a) $se(\hat \beta_{RE}) < se(\hat \beta_{FE})$ because it uses between-variation as well and (b) we are able to use time dependent variables in our model if we wish (remember that FE removes this opportunity since we average over all values of $t$ in order to remove the fixed effect $a_i$).
Answer: This is a long comment...

The fixed effects and random effects estimates are not that different, look at the standard errors, not just the point estimates.
xtsum doesn't report what you think it does, the min and max report the means after group detrending. These values don't correspond directly to variance within or between.
Certain aspects of the data generation process will dictate whether random effects or fixed effects are appropriate. E.g. if there is a time invariant important omitted variable (one example in criminology is southern culture) you would prefer fixed effects.

For 2, here is demonstration of how to go from the listed standard deviations in the xtsum table to their corresponding sum of squares for a balanced panel dataset.
use http://www.stata-press.com/data/r13/gymdata, clear
*only selecting a few so it is balanced
keep if id == 1 | id == 2 | id == 4
xtset id month
anova wt id
xtsum wt

This produces the output:
. xtset id month
       panel variable:  id (strongly balanced)
        time variable:  month, 1 to 12
                delta:  1 unit

.
. anova wt id

                           Number of obs =      36     R-squared     =  0.9725
                           Root MSE      = 7.87064     Adj R-squared =  0.9708

                  Source |  Partial SS    df       MS           F     Prob > F
              -----------+----------------------------------------------------
                   Model |  72200.0556     2  36100.0278     582.76     0.0000
                         |
                      id |  72200.0556     2  36100.0278     582.76     0.0000
                         |
                Residual |     2044.25    33  61.9469697
              -----------+----------------------------------------------------
                   Total |  74244.3056    35  2121.26587

.
. xtsum wt

Variable         |      Mean   Std. Dev.       Min        Max |    Observations
-----------------+--------------------------------------------+----------------
wt       overall |  163.8611    46.0572        118        233 |     N =      36
         between |              54.8483   130.5833   227.1667 |     n =       3
         within  |             7.642457   150.0278   177.2778 |     T =      12

.

Subsequently the total sum of squares (in the ANOVA table) is equal to the overall variance multiplied by the number of observations minus one. The weight for the between sum of squares is the number of observations per panel multiplied by the number of groups minus one. The weight for the within (residual) sum of squares is the number of observations minus one.
*Total sum of squares ~ 74244.3056
display (46.0572^2)*(36-1)

*between sum of squares ~ 72200.0556
display (54.8483^2)*12*(3-1)

*residual sum of squares ~ 2044.25
display (7.642457^2)*(36-1)

Again these simplified calculations only work for balanced panels. There is not enough information in the summary xtsum table to aggregate the sum of squares if the groups have unequal numbers.
Comment: Exec does not  have 0 "variation", its within standard deviation is `1.641031`. In the `xtsum` output min and max refer to the group detrended **means**. So calculate the mean per id, and subtract that. Min and max then report the range on those values. So an id that has a mean of 10, if the minimum value in the series is 0 it will be -10 on the lower range.
Comment: **Between** variation is variation that occurs between indivudals, over time. So we have taken the data and on each variable $B_{it}$ we calculate $\frac1T \sum_{t=1}^T B_{it}.$
If a variable is not dependent on time, for example "id" in our dataset, it will have within variation of zero.

Likewise, **within** variation is variation that occurs between time, over indivudlas.

**Total variation** is calculated as if we have $N=nT$ individuals in one time period.

How come $$overall = 0 \text{ for } exec_{it}$$ since it has index it?
Comment: Furthermore, how can exec have negative within variation?
Comment: NEW: **Between** variation is variation that occurs between indivudals, over time. So we have taken the data and taken a standard deviation over time, just as the mean over time for a variable $B_{it}$ is $\frac1T \sum_{t=1}^T B_{it}.$
If a variable is not dependent on time, for example "id" in our dataset, it will have within variation of zero.

Likewise, **within** variation is variation that occurs between time, over indivudlas.

**Total variation** is calculated as if we have $N=nT$ individuals in one time period.

How come $$overall = 0 \text{ for } exec_{it}$$ since it has index it?
Comment: Thanks! I get rho = 0.8 which is large, so this is an argument for using FE-model.
Comment: I wouldn't use that to determine re or fe, but it is possible some others have recommended that. I think you should base it on the underlying model, not on any observed sample statistics.
Answer: Let's look at the data first. You say you have data for three time points. Did you throw those all together into the 153 observations? If so, why? (One would expect there to be a time dimension, a general national trend up or down.) Either way, why are there 153 observations? I can only surmise that this might be 3 observations for each of the 50 states plus the District of Columbia? Puerto Rico?, but other configurations are of course possible. Are there figures in there for individual cities?
According to data I found here, the minimum of 0.8 does not look out of line (look at New Hampshire and North Dakota). The maximum is a bit unusual and looks like an outlier (unless the aggregation is not truly per state). I would advise on doing some graphing before you even formulate the regression model.
Comment: This is panel data. n=51 as in 51 states. T=3 as in 3 time periods. 51*3=153. So in my model $mrdrte_{it}$ we have $i$ from 1 to 51 and $t$ from 1 to 3.
Comment: It is aggregated per state.
Answer: Inspired by @andy-w here is an answer: xtsum measures the standard deviation within and between, just as anova measures the sum of squares within and between.
R code below.
> library(foreign) #to read dta files
> #dataset is at https://www.dropbox.com/s/e7suemnmgxu3eib/MURDER.DTA?dl=1
> dfa<-read.dta("murder.dta")
> names(dfa)
 [1] "id"      "state"   "year"    "mrdrte"  "exec"    "unem"    "d90"
 [8] "d93"     "cmrdrte" "cexec"   "cunem"   "cexec_1" "cunem_1"

From xtsum (included in the code for the question) we expect the SS to be greatest for 1 mrtdrte 2 exec 3 unem
> anova(lm(id ~ mrdrte + exec + unem, dfa))
Analysis of Variance Table

Response: id
           Df  Sum Sq Mean Sq F value   Pr(>F)
mrdrte      1  1670.1 1670.07  8.0123 0.005288 **
exec        1   380.9  380.85  1.8272 0.178509
unem        1    41.9   41.86  0.2008 0.654715
Residuals 149 31057.2  208.44
---

This is correct
> anova(lm(year ~ mrdrte + exec + unem, dfa))
Analysis of Variance Table

Response: year
           Df Sum Sq Mean Sq F value Pr(>F)
mrdrte      1   5.17  5.1699  0.8451 0.3594
exec        1   0.98  0.9840  0.1609 0.6889
unem        1   0.37  0.3672  0.0600 0.8068
Residuals 149 911.48  6.1173

This is correct
> anova(lm(state ~ mrdrte + exec + unem, dfa))
Error in lm.fit(x, y, offset = offset, singular.ok = singular.ok, ...) :
  NA/NaN/Inf in 'y'
In addition: Warning message:
In model.response(mf, "numeric") : NAs introduced by coercion

Something went wrong.
Let's make mrdrte the dependent variable for fun,
> fit <- lm(mrdrte ~ year + id + exec + unem, dfa)
> anova(fit)
Analysis of Variance Table

Response: mrdrte
           Df  Sum Sq Mean Sq F value   Pr(>F)
year        1    72.3   72.34  0.9400 0.333856
id          1   647.1  647.14  8.4090 0.004303 **
exec        1   180.6  180.62  2.3471 0.127654
unem        1   555.5  555.50  7.2182 0.008042 **
Residuals 148 11389.7   76.96
---

I do not know how to interpret this.


Google Analytics setup not working

Question: I am trying to setup a Google Analytics property. I have done this before and have had no problems. However this time I am setting it up though the drupal module rather than manually inserting the code.
I have my tracking ID and everything filled out, and I inspect the page and I can see the tracking code is installed.
However my tracking code isnt being picked up by Google Analytics. This is what it says:
Status: Tracking Not Installed

I have enabled debugging for the module, and all seems to be sending correctly. Here is the log:
Initializing Google Analytics. analytics_debug.js:10
Running command: ga("create", "UA-49353541-2", "auto") analytics_debug.js:10
Creating new tracker: t0 analytics_debug.js:10
Auto cookieDomain found: "carbonfunding.co.uk" analytics_debug.js:10
Running command: ga("send", "pageview") analytics_debug.js:10

Sent beacon:
v=1&_v=j30d&a=1343148915&t=pageview&_s=1&dl=http%3A%2F%2Fwww.carbonfunding.co.uk%2F&ul=en-us&de=UTF-8&dt=Carbon%20Funding%20%7C%20Th…_u=MCCAAEQAI~&jid=&cid=2073925826.1413195127&tid=UA-49353541-2&z=327379316

 analytics_debug.js:10
_j1              (&jid)  analytics_debug.js:10
adSenseId        (&a)   1343148915 analytics_debug.js:10
apiVersion       (&v)   1 analytics_debug.js:10
clientId         (&cid) 2073925826.1413195127 analytics_debug.js:10
encoding         (&de)  UTF-8 analytics_debug.js:10
flashVersion     (&fl)  15.0 r0 analytics_debug.js:10
hitType          (&t)   pageview analytics_debug.js:10
javaEnabled      (&je)  1 analytics_debug.js:10
language         (&ul)  en-us analytics_debug.js:10
location         (&dl)  http://www.carbonfunding.co.uk/ analytics_debug.js:10
screenColors     (&sd)  24-bit analytics_debug.js:10
screenResolution (&sr)  1920x1080 analytics_debug.js:10
title            (&dt)  Carbon Funding | The ECO people analytics_debug.js:10
trackingId       (&tid) UA-49353541-2 analytics_debug.js:10
viewportSize     (&vp)  1903x955

I read that it can take up to 72 hours before Google will pick up the tracking code, but I also read that at least the real-time view should be picking myself up on the site. I have also disabled do-not-tack everywhere and no prevail.
Why is this the case?
Thanks,
Darryl
Answer: Adding Google Analytics using the module is not difficult. Please check this YouTube Video "Adding Google Analytics" it will walk you on the process of how to set-up your Google Analytics on your Drupal site. In addition it will to provide you the steps on how to set-up this video will give useful information about the different setting.
If you want to make sure that your website contain the Google Analytics you can check the output of your website (as an anonymous) and you should have something similar to this...
Usually takes few minutes to Google Analytics to recognize your website.

Comment: Hi, I have setup analytics to a few websites previously without issues. The data from yesterday is showing, but still Analytics says my code is not installed and the real-time functionality isn't working.

Google Analytics setup not working


Updating data in relational database, without affecting the already existing relation

Question: How do database systems make their changes in relational data that contains a partial history? For example, in a contract management system, I have a maintenance contract for a company to maintain some products. These products will be linked with the maintenance contract (within the database). Now assume that I have maintained these products for over 2 years within this contract, and suddenly the product changes in attributes, then I need to change the information about the product. But if I do that, because of the relation with contracts, the information would tell me that I maintained the product in it's changed attributes, for over 2 years, which is not true!
Now assume I have a simple Contracts and Products relationship (M:M relationsip, so Contracts_Products junction table is needed) and another Productattributes, in relationship with Products (M:1 relationship). Since I do not change anything about the Contract, neither about the Product, just the Productattributes, what are possible solutions to keep the relationship for the past the same, while only changing a Productattribute for the upcoming time?
Answer: As I understand your requirements, you need to separate properties of a product that change over time (e.g., supplier, operating temperature range, unit cost) from those that are invariant (e.g., description, color, weight).
Call the table that holds the invariant properties ProductFixedAttributes and the one that holds the changeable properties ProductVariableAttributes.  Then a row of your current Products table is 1-to-1 with ProductVariableAttributes and many-to-1 with ProductFixedAttributes, and you can drop the Productattributes table.
When a product's properties change, add a new row to Products that points to a new row in ProductVariableAttributes and an existing row in ProductFixedAttributes.  Use this new row for future associations with new Contracts; do not change the existing rows in Contracts_Products.


Position of white dots in guitar fretboard

Question: I was wondering why the white dots in the fretboard are usually placed in fret numbers 3, 5, 7, 9, 12, 15, 17, 19, and 21.
I know that their general purpose is for counting, but what I wonder is: Why are they placed on precisely those frets, and not different ones?
For example: Why are there two frets between the dots on 7 and 9, and three frets between the dots on 9 and 12?
Answer: The spacing is designed to offer useful milestones on the fretboard. Take the notes on the 6th string (in standard tuning), for example:

Open (0th fret) is E.
The F is only 1 fret away, why put a marker on the 1st fret? It's already marked by being the first fret
The G is on fret 3, so put a marker there.
The A is on fret 5, which is a perfect 4th from the open string, so it deserves a marker.
B follows on fret 7, which is a perfect 5th from the open string, so another marked fret.
Why there's a mark on fret 9, I'll never know, I wish it was on 10 for D instead of 9 for C#
But you need a marker on 12, for certain - it's the octave!
The 15, 17, and 19 are just 3, 5, and 7 + 12 (an octave), respectively.
Comment: "The A is on fret 5, so it deserves a marker" is p4th of the root. "B follows on fret 7, another marked fret" and is the p5th of the root. Quite important intervals.
Comment: Nice breakdown. With regards to the 9th fret, my suggestion would be that it marks the position of the relative minor to an open-string major scale. Having said that, however, I think the reason is more prosaic: the flanking two-fret intervals make the octave immediately stand out, even if the markers are bars or some such (i.e. no special double dot symbol or similar). The remaining seven frets are divided in the most efficient way for easy recognition: four marks, three spaces.
Comment: I own a very old russian made guitar with the marker on the 10th fret. And boy, is it hard to play because of that, even though I usually don't watch the markers...
Comment: I'd guess that the 9th fret has a marker and not the 10th because the 9th fret is a member of the major scale while the 10th is not.  But then, the same logic would put a marker on the second fret and not the third, so I don't know.
Comment: @ledfloyd - You're obviously much more of a theory guru than I am. Would you please edit this answer to include your insights? @mrbuxley - I wouldn't want to play that guitar now, either. I'm used to the 9th fret mark by now. I just wish it were the other way. However, ledfloyd's answer explains why it is where it is.
Comment: My teacher had a French acoustic guitar which had the dots on different positions as well. Very confusing.
Comment: The twelfth fret marks off 1/2 the length of the string.  The seventh fret marks off about 1/3 of the length of the string, and the fifth fret marks off about 1/4.  If one decides to put dots at those places, and wants every fret to either have a dot or be next to one, the 3/5/7/9/12/15/17/19/21/24 pattern would seem the most regular.
Comment: If you want a dot marker on the 10th fret, you can always switch to mandolin or tenor guitar. The dots usually go 3, 5, 7, 10, 12.
Comment: Hmm, if you start at the 12th fret and descend 3 frets, you hit a marker, descend another 2 frets, a marker, descend another 2 frets, a marker. When reversed from the nut or open position, you see the same order, same dot placement. Just a theory on the 9th marker.
Answer: Maybe lost to history, the dots are references for the Fibonacci series, which when harmonics are considered, give a pure major chord with octave and perfect fifth redundancies. The dot at the 9th fret marks 2/5ths of the string length. The harmonic there is of the 5th partial (the major third) -- this is the one that throws you off.
This is my observation, and it is so easy once seen, that I have little doubt that many people have stumbled across it. Nonetheless, Gibson and Martin guitars had nothing to say about it, and so far as I could see, nothing to see in google searches. THE DOTS MARK THE HARMONICS COMPRISING A PURE MAJOR CHORD.
Are the dots an atavism from pre-equal tempered tuning? In historical paintings of fretted instruments, or in museums,  when are the earliest dots seen in their modern location??
string
open = 1 = do
dots
12   = 2nd partial = do  (octave over root)
7    = 3rd partial = sol (perfect 5th + octave over root)
5    = 4th partial = do   (double octave over root)
9    = 5th partial = mi (double octave + major third over root)
3    = 6th partial = sol (double octave + perfect 5th over root)
Answer: The frets that are highlighted are the equal-tempered equivalents of the basic fractions, as seen in the table below. (The first column is the fraction, the second column is the corresponding number of half-tones):

 1      0      = 0
 6/5    3.156  ~ 3
 4/3    4.980  ~ 5
 3/2    7.020  ~ 7
 5/3    8.844  ~ 9
 2      12     = 12 (from now on, it's simply an octave up, i.e.,
 12/5   15.156 ~ 15     times 2 for the fractions and +12 for the frets)
 8/3    16.980 ~ 17
 3      19.020 ~ 19
 10/3   20.844 ~ 21

One can ask why 6/5 and not 7/5, 8/5 or 9/5. I don't know exactly, but the fractions used have the lowest possible sum of numer+denom in the interval [1,2]. (I'm not 100% precise here, we omit 5/4 which gives 3.863 ~ 4, but you wouldn't place two dots adjacent since it would only confuse people, would you?)
This as well explains why 9 got there and not 10: It corresponds to a very low-entry fraction. One can as well notice that you can try flajolets on bars 12, 7 and 5 easily, and with some adjustments on 3, 4 and 9 as well, but these don't sound quite strong. Flajolets on other positions are even more difficult.
Comment: Please let me know what exactly is wrong with my answer, so that I can do better next time. Thanks.
Answer: They are at important intervals: minor third, fourth, fifth, major sixth, octave.
Then they repeat for the second octave: minor third, fourth, fifth, major sixth.
Answer: I always thought the ninth fret marker was simply aesthetics. Being that the marker after the twelth is three frets away, the marker before the the twelth is the same distance away. It centers the twelve marker. Practically speaking, the twelve marker is also more noticeable.
Answer: The white dots are typically placed at harmonic points on the string. The first harmonic of the open string is the octave, halfway along the string at the 12th fret. The second harmonic is one-third the string length, and that falls at the seventh fret. On some models, you'll see that fret marked with a double-dot as well, as it's a very prominent harmonic:

The woman holding the guitar here is pointing at the octave marker, but you can see the seventh fret is also double-dotted.
You'll notice the above guitar also has fewer dots; there isn't one on the third fret, and the second octave doesn't match the first. The above layout is a more harmonic-oriented one; the average player, with a little practice, can coax out a harmonic at each of the marked frets. Now, because harmonics, by their nature, are Pythagorean "just temperament", while the guitar's fretboard is designed for "equal temperament", the actual location of the harmonic usually won't be over the fret marked by the dots, so this layout isn't an exact guide but it's useful at a glance.
Most fretboards have the frets marked as you describe, and this is a "compromise" layout of sorts that can be repeated on both octaves of the fretboard, so the guitarist can play in both octaves without losing his place. This layout is more a straight-up positioning guide, but it is still useful for identifying harmonic points.
Other harmonics of the open string besides the ones in the above picture are less prominent, though a skilled player with a well-made guitar can get them to speak. Jaco Pastorius was famous for coaxing third and even second-fret harmonics from his Jazz Bass: Jaco Pastorius - Loop Jam (incorporates part of his "Portrait of Tracy" solo).
Comment: Wondering why the 3rd fret doesn't have a dot then - the 3rd fret open natural harmonic is significant, and frequently used across many genres, from Jazz, as you mentioned, to Metal (see the Dimebag Squeal)
Comment: In my experience, and I'm not a professional guitarist so make of this what you will, the 3rd fret takes a little more coaxing on most guitars. IMO it has to do with the difference between Pythagorean and logarithmic temperaments; if you look at frequency charts for an octave in both systems and calculate percent error, the major and minor third are pretty far out (there are further, but compared to the major fourth and fifth they're really bad).
Comment: Now, the ninth fret harmonic (producing the major sixth) is even further out from Pythagorean intervals, but its placement closer to the center of the string makes it a little easier to pop this one out once you learn to touch slightly behind the ninth fret.
Answer: I think it's to signify intervals as much as anything else.  The 12th fret is the most obvious - it's the octave.  The 3rd fret is pretty obvious because it gives you some general orientation for most open chords.  The 5th and 7th frets are valuable because they signify the 4th and 5th interval of the open strings.
Answer: After 45 minutes or so reading people's answers to the ninth fret question, then 15 minutes of pondering, I came up with a crappy answer:
The NINTH FRET is a position of all flatted notes:
Db Ab Fb Cb Gb Db  --or-- Db Gb Cb Fb Ab Db
(The order depending upon whether You look at the notes from first string to sixth or vice-versa.)  But then, the fourth fret is also a position of all flatted notes.  I know we aren't supposed to call B a Cb or E an Fb, though there are exceptions.  Yeah, somebody, please come up with the answer to this ninth fret conundrum!  In any case I won't stop thinking about this.
Comment: Cb is actually B


double slash '//' with drupal_goto

Question: I am using drupal_goto and url methods to try and direct the user to a page.
url returns a string like "/whatever/example" and then I pass that into drupal_goto and they get sent to
localhost//whatever/example
I checked the value for $base_url in settings.php that has no trailing slash. what else could cause this behavior?
Comment: url should be `whatever/example` not `/whatever/example`
Comment: that's what the url() method returns for me when I give it node/
Answer: Whatever you pass to drupal_goto() will in turn get passed on to the url() function. If you're using url() on the path before hand, it'll get run through again and you'll get a double slash.
You can just pass the router path to drupal_goto() directly, and it will handle the alias for you:
drupal_goto('node/1234');

double slash '//' with drupal_goto


How to remove lock screen using android management api

Question: I am currently developing an android application that is supposed to run in kiosk mode. Everything else works well however, we have a lock screen issue, the application will e completely unmanned and therefore, the lockscreen prevents us from running upon restart.
I tried:
policy_name = enterprise_name + '/policies/policy1'

policy_json =
{
  "applications": [
    {
      "packageName": "com.micro.buscamera",
      "installType": "KIOSK",
      "defaultPermissionPolicy": "GRANT"

    }
  ],
  "debuggingFeaturesAllowed": true,
  "maximumTimeToLock": "0"
}

androidmanagement.enterprises().policies().patch(
    name=policy_name,
    body=json.loads(policy_json)
).execute()

However, the maximumTimeToLock did not seem to work. Any and all suggestions will be highly appreciated
Answer: "keyguardDisabled": true
This option disables the lock screen but does not keep the screen from turning off.
Alternatively you could try putting getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON); in your app.
Answer: To prevent your device from locking in right after you restart your device, I suggest that you remove your screen lock pin/password. With this, after the device restarted, it will continue to the home screen and will not be automatically locked.
After reboot you can continue using policies to prevent your device from locking out such as maximumTimeToLock or keyguardDisabled. If you still need to keep your device awake,your application can add the FLAG_KEEP_SCREEN_ON flag.


What is this called: Melting? Pivoting? Reshaping?

Question: This is a question about using pandas and ggplot in Python, but an R answer would also be very much appreciated.
I am trying to plot some timeseries data that look somewhat like what's shown below. X, Y, Z are well-plate ids (names of experiments), and 0,1,2 are different times. I want to be able to plot the time course as a line graph both in different panes/subplots/facets, and in the same facet but with different colors.
   X    Y    Z
0  0.1  0.2  0.3
1  1.1  1.2  1.3
2  2.1  2.2  2.3

I know that pandas combined with ggplot would let me say
from ggplot import *
ggplot(aes(x='T', y='value', color='well'), data = df) + geom_line()

or
from ggplot import *
ggplot(aes(x='T', y='value'), data = df) + geom_line() + facet_grid(x='well')

if df looked like this
well  T   value
X     0   0.1
X     1   1.1
X     2   2.1
Y     0   0.2
Y     1   1.2
Y     2   2.2
Z     0   0.3
Z     1   1.3
Z     2   2.3

, with rows not necessarily in that order.
My question is, how do I make the data frame above into the data frame below the code, and what is this operation called? Again, I'm mainly interested in pandas and ggplot in Python, but an answer as to how this could be done in R would also be very helpful.
I would also appreciate it if someone can recommend a good source to learn about doing such things with data.
Answer: It is called reshaping a a dataframe with methods such as pivot or melting and can include stack and unstack, pivot_table and various other methods.
To go from 'wide' to 'long'
print(df)

     X    Y    Z
0  0.1  0.2  0.3
1  1.1  1.2  1.3
2  2.1  2.2  2.3

You can reshape into a long dataframe with such methods as stack and reset_index:
df2 = df.stack().reset_index()
df2.columns = ['T','well','value']
print(df2)

   T well  value
0  0    X    0.1
1  0    Y    0.2
2  0    Z    0.3
3  1    X    1.1
4  1    Y    1.2
5  1    Z    1.3
6  2    X    2.1
7  2    Y    2.2
8  2    Z    2.3

Or using melt:
df.reset_index().rename(columns={'index':'T'}).melt(id_vars='T').sort_values(by='T')

   T well  value
0  0    X    0.1
3  0    Y    0.2
6  0    Z    0.3
1  1    X    1.1
4  1    Y    1.2
7  1    Z    1.3
2  2    X    2.1
5  2    Y    2.2
8  2    Z    2.3

To go from 'long' to 'wide'
print(df2)

T well  value
0  0    X    0.1
3  0    Y    0.2
6  0    Z    0.3
1  1    X    1.1
4  1    Y    1.2
7  1    Z    1.3
2  2    X    2.1
5  2    Y    2.2
8  2    Z    2.3

Using pivot:
df2.pivot(index='T',columns='well')

     value
well     X    Y    Z
T
0      0.1  0.2  0.3
1      1.1  1.2  1.3
2      2.1  2.2  2.3

Using set_index and unstack:
df2.set_index(['T','well']).unstack()

     value
well     X    Y    Z
T
0      0.1  0.2  0.3
1      1.1  1.2  1.3
2      2.1  2.2  2.3

Using pivot_table:
pd.pivot_table(df2,aggfunc='mean',index='T',columns='well')

     value
well     X    Y    Z
T
0      0.1  0.2  0.3
1      1.1  1.2  1.3
2      2.1  2.2  2.3

Comment: WOW. I think I just tread into some deep waters.

This is awesome. Thank you. Now I know where to start.
Comment: What is the advantage of stack vs melt?
And pivot_table vs unstack vs pivot?
Answer: Try this
df1 = df.T.stack().reset_index().rename(columns = {'level_0': 'well', 'level_1': 'T', 0: 'value'})

    well    T   value
0   X       0   0.1
1   X       1   1.1
2   X       2   2.1
3   Y       0   0.2
4   Y       1   1.2
5   Y       2   2.2
6   Z       0   0.3
7   Z       1   1.3
8   Z       2   2.3


How to display numerically number of displayed results based on specific input in R Shiny?

Question: library(shiny)

ui <- fluidPage(

titlePanel("BOILER MASTER DATA LIST"),
sidebarLayout(

sidebarPanel(
selectizeInput("Status", "Select a Status", choices = NULL)),

mainPanel(
tableOutput("BOILERDATA")),))

server <- function(input, output, session){(

updateSelectizeInput(session,"Status", choices = RawData$Status, server = TRUE)

output$BOILERDATA <- renderTable({statefilter <- subset(RawData, RawData$Status == input$Status, drop = TRUE)})}

shinyApp(ui = ui, server = server)

I am trying to add output in server part showing no. of results displayed based on input selected. Please help.
Comment: Where do you want to output number of results? How do *you* define number of results? What have you tried?
Comment: I want output to be shown below the input box. Result may be define by the no. Of rows filtered nased on inout data. I tried nrow, or count.fields. But all this is cusing errors in code.
Answer: You need an extra output, e.g. a textOutput, in you UI.  renderTable only handles tables and table-like structures. If you wanted something written beneath the table, you would literally have to append a row with the text. Ever done that in Excel? Then you know you'll also have to handle textoverflow, spanning cells, perhaps borders, etc. And it kinda breaks the idea that you have your tabular data and random non-tabular data, in a tabular scaffold.
Then you need to assign it some output. Ideally nrow(statefilter), but you can't because it is not defined outside of the scope of your renderTable. Instead of preparing your data in the table, and then somehow assigning it to a variable that other scopes can find, we break the task up in two distinct parts: 1) preparing the (subsetted) data, and 2) formatting it for presentation. Currently, you are doing both in renderTable. Which by itself is okay, it just makes it harder to build further on. So, to do part 1 outside of renderTable you have to do it in a reactive context, because it uses a reactive variable as input (input$Status).
filtered <- reactive({
  req(length(input$Status) == 0)
  subset(RawData, RawData$Status == input$Status, drop = TRUE)
})
output$BOILERDATA <- renderTable( filtered() )
output$rows <- renderText( nrow(filtered() )

Do yourself a favour and lookup what req does.


$\operatorname{Hom}(A,B)\bigotimes_BC\cong_C\operatorname{Hom}(A,C)$?

Question: Let $A$ be an abelian group, let $B$ be  a PID and $C$ be an unitary commutative ring which is also a $B$-module. Then all abelian group homomorphisms between $A$ and $B$ consist an abelian group $\text{Hom}(A,B)$ and it has a natural $B$-module structure. In the same way $\text{Hom}(A,C)$ is a $C$-module.
$\textbf{Question}$: Then do we have the $C$-module isomorphism?$\text{Hom}(A,B)\bigotimes_BC\cong_C\text{Hom}(A,C)$
By universal property, we have a obvious $C$-module homomorphism $f\bigotimes c\rightarrow (a\rightarrow f(a)\circ c)$, but I can't construct the inverse, and it seems to me the two $C$-modles can't be isomorphic.
Thanks!
Comment: If $C$ is only a $B$-module, what do you mean by $\operatorname{Hom}(A,C)$ being a $C$-module?
Comment: @lisyarus Sorry, I edit the question, $C$ is an unitary commutative ring and $c\circ f\rightarrow (a\rightarrow f(a)c)$
Answer: For $A = \mathbb{Z}_p, B = \mathbb{Z}$, and $C = \mathbb{Z}_p$, the module $\operatorname{Hom}(A, B)\otimes_B C = 0$, but $\operatorname{Hom}(A, C)$ is nonzero. This is similar to asking if every map $A\to C$ factors through $B$, which is clearly false in general.
Comment: Thanks, but I think you should write $\mathbb{Z}/p\mathbb{Z}$, not the $p$ adic integers $\mathbb{Z}_p$
Comment: @Sssss: Topologists' notation.


User registration page edit

Question: I want to add some div's and hrefs to user registration form. any ideas? I'm using ajax login/register module to get user login/register form in ajax windows. and i have no idea how to add some divs and hrefs to register form.
Answer: Assuming you're using Drupal 7.
Using hook_form_FORM_ID_alter and a custom module you can achieve it. Like below,
function mymodule_form_user_register_form_alter(&$form, &$form_state, $form_id) {
  $form['links'] = array(
    '#type' => 'item',
    '#markup' => l('some links','http://www.example.com'),
  );
  $form['additional_markup'] = array(
    '#type' => 'item',
    '#markup' => '

',
  );
}

Comment: thx. as i understand i should paste it in template.php?
Comment: yes, it would also work in template.php file.
Answer: In other hand you can use Display Suite. It provide you great solution to customize things.  With it you can create new registration fields  the way you want it and then you can customize de display...
Comment: Follow the screencasts of this guy about display suite and you will master it. The link http://m.youtube.com/watch?v=rSgd91tZSEc
Comment: Is it possible to manage user/register page form by Display Suite?
Comment: Yes, it's possible
Comment: Wow!! Thank you so much! I was looking for this ability so much time!

User registration page edit


Ao abrir solution, os folders com os projetos não aparecem

Question: Criei uma solution, baseado no video do Eduardo Pires. Salvei e fui fazer outra coisa. Bem, ao reabrir a solution, os folders com os projetos não aparecem, ficando apenas a Solution indicando (0 project). Como eu faço para mostrar os meus folders?
Lembrando que os folders estão lá, abrindo pelo windows explorer estão todos la´, os folders e os projetos.
Comment: Aconteceu isso comigo. Verifiquei que o sdk que defini no global.json não estava instalado. Instalei e ao criar os projetos dentro da mesma solution agora aparecem normalmente.
Answer: Já tentou ir na opção mostrar todos os arquivos na solution?

(Está destacado em vermelho o ícone)
E depois disso basta selecionar as pastas do projeto e dar botão direito e selecionar "Include in Project"
Comment: Então, o que estou achando estranho é que no meu Solution Explorer não tenho essa opção. Foi a primeira coisa que tentei fazer. Visual Studio 2015
Comment: Tenta então dar botão direito na `solution -> add -> Existing Project`  e seleciona o `.csproj `que deve estar dentro de alguma pasta da `solution`
Comment: Mas e os folders? Quando eu faço isso, só adiciono os class library, mas preciso adicionar todos os meus folders e não sei porque isso aconteceu. Gostaria de entender o que realmente aconteceu.
Comment: Quando você adicionou o .csproj apareceu outro item a baixo abaixo da solution certo?? Clicando no projeto, não aparece a opção de mostrar todos os arquivos?? Provavelmente seu .sln ou .csproj está corrompido.


The Curse of high Dimension And Distance

Question: For extracting features from video frames (2 sample/sec) I use keras framework in python and load VGG16 that input size is (150,150,3) and output size is (4,4,512). After the feature extraction step I want to cluster frame features with Hierarchical K-Means.
My problems are as follow:

I save each frame features in a vector which size is 8192. For a video that have 8000 frames if only reduce each frame size to (150,150) and extract features then I have a feature matrix with size (640,8192). As you can see feature matrix for even one video is very large ans besides "sparse". What is the best way to reduction its dimension?
What is the best metric for calculation distance between two pair of
frame features? The space is so sparse and even feature values are
so small, so Euclidean Distance is not a wise choise!!

CLARIFICATION
What is the frame feature:
As you already knew, videos are nothing but frames, and with the help of deep learning (VGG16 (without the last fully connected layer)) we can extract its features in the way we like. for more information kaggle.com/keras/vgg16 In this particular case, output features have the size of (4*4*512) that become 8192 number in a row vector.
Data:
My data as I mentioned above is a very sparse and large matrix (640,8192). Non-zero values are rarely up to 100.
IDEAS
For Dimension Reduction:
Two method are available for DR

Principal component analysis (PCA): A statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. (source: https://en.wikipedia.org/wiki/Principal_component_analysis)
Singular-Value Decomposition (SVD): A factorization of a real or complex matrix. It is the generalization of the eigen decomposition of a positive semi definite normal matrix (for example, a symmetric matrix with positive eigenvalues) to any m*n matrix via an extension of the polar decomposition. (source: https://en.wikipedia.org/wiki/Singular-value_decomposition)

Most important parameter of these two methods is "n_components" that is number of components to keep. This parameter have the value of min(n_samples, n_features). As you can guess, components that we kept with this module is depend of Sample Number and Feature Number. Suppose that we have two videos with feature matrices with size of (140, 8192) and (640, 8192). First element is number of frames and second element is number of features. The output of PCA for these two videos is (140, 140) & (640, 640). We have to have matrices with same axis to check distance and clustering. How to solve this problem? I know that this clarification is too long to read, but it's worth it!
Comment: It is uclear what your data look like. Their values. What is frame features, in what form were they extracted.
Comment: You did not understand my comment, probably. I said that a well-asked question would be a question which potentially could be understood and answered by an analyst not working with images, python or keras. Anyway.
Comment: @ttnphns as you probably know, videos are nothing but frames, and with the help of deep learning (VGG16 (without the last fully connected layer)) we can extract its features in the way we like. for more information
 https://www.kaggle.com/keras/vgg16 In this particular case, output features have the size of (4*4*512) that become 8192 number in a row vector. My data as I mentioned above is a very sparse and large matrix (640,8192). Non-zero values are rarely up to 100. I hope this clarification helps you to understand the problem. Feel free to ask for more :)
Comment: Very nice. It is what you ought to explain in your question. And show maybe a snippet of your data. So people not working with images or keras can quickly understand the situation.
Comment: If you read the question carefully, you can easily find the keywords out of it, keywords like "keras" or "video". You cannot say if a question belongs to python  or clustering category, I can solve it :)
Comment: Sorry for bad explanation of the problem @ttnphns
Comment: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/79126/discussion-between-shahroozevsky-andrea-and-ttnphns).
Answer: Dimension reduction is always a trade-off between resources and precision. You can use PCA (setting a high value of n, say 5000) and use the measure of exactly how much variation is addressed by each component to try and determine the number of principle components to keep. Maybe 80% is enough, but if you've only explained 30% of your data's variance after 5000 principle components, then maybe you need an alternative.
One way to plot this visually is to plot the number of components as x, and the explained variance as y. You're likely to get a curve that starts off shallow, and gets steeper at some point (or starts steep, and levels off to shallow, depending on which way your x-axis is ordered). Finding the "elbow" of this curve will deliver the point at which the cost of adding an additional component yields less and less explanatory value. It's not a perfect answer, but will give you an indication of roughly where to set your n.
sklearn.decomposition.PCA has an output that shows the explained variance after fitting - use this to help get a better feel for your data.
So the short version of this is, you can run PCA with any number n you want, but only pick the top k components where in combination, they do a good enough job (where "good enough" is defined by you) of explaining the variance in your data.
Alternately, look into building an auto-encoder using a convolutional neural net (Keras/Tensorflow), these tend to do very good job of compression, especially where more linear methods (like PCA) find the task more challenging.
[EDIT] Also, to address your final point - PCA wont ever reduce the feature space to a value less than the number of observations. That is, where you're feeding it with a small number of observations vs a large number of features, you can't force PCA to reduce the feature-size to a value lower than your observation count.
Either build up more data, or generate it using your initial dataset as a starting point, and duplicating it (sensibly, adding some random variation perhaps) so that you can reliably squirt in a large enough dataset.
This itself might pose you some problems, since now you'll have to deal with a larger memory footprint due to the input length, rather than the feature-width.
Brute-forcing it so that your PCA squeezes your feature-width down to only 50 components would limit your issue in the first place, and perhaps your resultant feature space will continue to provide you with enough explanation of variance to be useful. It's common to only use maybe 5 components for example, but with an associated loss of precision. The truth lies in your data.
Comment: nice and neat :)
Answer: For high dimensions, I read that Cosine Similarity works better than Euclidean Distance (https://en.wikipedia.org/wiki/Cosine_similarity). Maybe you can try this.
Comment: Hi  :)
Do you have any implementation of Cosine Similarity(in any programming language) ?
Comment: Yes, If you use Python just use scikit (http://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.cosine_similarity.html) otherwise here a post with several implementations (https://stackoverflow.com/questions/18424228/cosine-similarity-between-2-number-lists)
Comment: Thank you, I will try them and let you know about the reults :)
Comment: Note that the distance directly corresponding to cosine similarity, the chord distance, _is_ euclidean distance computed after both vectors are normalized to sum of squared values within a vector = 1.
Comment: so by this definition, cosine similarity and euclidean distance are the same? @ttnphns
Comment: $d_{chord}^2 = 2(1-s_{cos})$, and $d_{chord}$ is euclidean distance computed on the vectors which _were_ normalized to SS=1.
Comment: However you take cosine distance or similarity, it is very different than Euclidean distance as it only depends on the angle between two points.  So for any cosine measure, the distance between (0,1 ) and (1,0) is the same as the 'distance' between (0,1) and (1,000,000,000, 0 ) ! (for example)


Autenticação com segurança entre sistemas distintos

Question: Meu cenário é o seguinte: Um sistema em php autentica o usuário e deve chamar a minha aplicação(Java) informando que usuário é este.
A minha aplicação não autentica este usuário, apenas recebe um identificador dele e carrega as informações que precisa.
A dúvida é: Como fazer a comunicação entre sistemas com segurança ? Hoje tenho um Servlet no Java que recebe uma chamada http post informando o id do usuário, porém qualquer um que interceptar essa requisição pode se autenticar.
Comment: Olá, bem-vindo ao [pt.so]! Acho que você precisa ser mais preciso em sua pergunta. Esses sistemas ficam na mesma rede interna ou convivem em locais separados, sendo a comunicação via internet?
Comment: Desculpem pela falta de detalhes. Os dois sistemas são web, rodam em redes distintas sendo a comunicação via internet somente.
Answer: Conheço duas maneiras de fazer isso:

usando SSL na comunicação (mais seguro)
implementando medidas de segurança você mesmo (caso você não saiba
implementar SSL e quer economizar tempo)

Implementando medidas de segurança você mesmo
Você pode criar um token, que é uma string qualquer seguindo alguma lógica, ambas aplicações devem conhecer essa lógica para criptografar e descriptografar.
Feito isso, aplique o md5 nela para virar uma string aleatória, essa string será o seu token, que o PHP deve enviar para o Java, e este por sua vez irá descriptografar para ver se foi autenticado com sucesso, assim fica mais difícil de interceptar.
Mesmo assim, se interceptarem esta comunicação, vão precisar descobrir qual a lógica utilizada para poder burlar novas requisições.
Comment: Obrigado pelas sugestões vou pesquisar mais sobre o assunto. Pelo que consegui de informação até o momento a melhor maneira é usando uma criptografia mesmo por exemplo assimétrica.
Comment: Exato, eu usei md5 para criptografar mas existem outros meios, vai da sua preferência e situação :)
Answer: Acesso remoto
Se servidor de origem e destino ficam em redes diferentes e o acesso de um ao outro é remoto, via internet, o mais recomendável é usar autenticação via certificado.
No servidor, o certificado deve ser mapeado para um usuário especial que tem os acessos que o sistema remoto precisa.
Desse modo, a conexão é criptografada e a comunicação segura. Além disso você garante que somente o servidor remoto com aquele certificado vai ter o acesso.
Rede interna
Se os dois sistemas convivem dentro de uma rede interna, Intranet, você pode utilizar regras no proxy ou firewall da empresa para filtrar os acessos permitidos.
Um exemplo seria permitir o acesso à URL desse serviço somente a partir do servidor onde está o sistema PHP.
Considerações
Em ambos os casos, mencionei técnicas que delegam a autenticação e a privacidade dos dados para soluções de segurança reconhecidas e comuns.
Evite reinventar a roda (que geralmente não funciona direito), por exemplo criando o seu próprio mecanismo de criptografia.
Comment: Obrigado pelas sugestões e desculpe pela falta de informação. Achei interessante as sugestões @edson sobre criptografia assimetrica. Estou pesquisando mais sobre o assunto. Obrigado.
Comment: @hebertrfreitas Criptografia assimétrica é praticamente sinónimo de usar certificados.


Ionic File Encryption

Question: I'm trying to use Ionic Native File Encryption plugin which works with Cordova Safe plugin.
However, after I add the plugin and try to run (even without using it in my code) it using ionic cordova run android, I get this error:
BUILD FAILED in 5s
at ChildProcess.whenDone (C:\Users\User\Ionic\App\platforms\android\cordova\node_modules\cordova-common\src\superspawn.js:169:23)
at emitTwo (events.js:126:13)
at ChildProcess.emit (events.js:214:7)
at maybeClose (internal/child_process.js:925:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)
(node:70968) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)

What is the problem?
Comment: Which @ionic/app-scripts versions are you using?
Comment: @jeffchef 3.1.9
Comment: what about your cordova version?
Comment: @jeffchef "Cordova Platforms : android 7.0.0, Ionic Framework : ionic-angular 3.9.2"
Comment: Have you installed the correct Android SDK version which supports 7.0.0 and above?
Comment: @jeffchef Yes, as far as I know
Answer: I have tried to install CordovaSafe plugin as well but I got the same error as you did.
After trying for awhile, I found out that the issue can be solved if you downgrade your cordova to a previous version.
Try running

npm uninstall cordova

And then

npm install -g firstname.lastname@example.com

Tell me if it works for you after downgrading your cordova version and creating your project with CordovaSafe plugin again.

Ionic File Encryption


How do I get `gem` working again after upgrading to rubygems 1.8.x?

Question: Soooo, I upgraded to RubyGems 1.8.1 from 1.7.2 using gem upgrade --system, then immediately ran gem pristine --all --no-extensions as recommended on Eric Hodel's blog. After getting errors when running gem commands, I googled around a bit, found some suggestions to gem uninstall the gems with the deprecation warnings, but then realized I can't, as gem uninstall just errors out--as does any other command--including just plain ol' gem.
When I run any of those commands, I get a bunch of "deprecated" warnings still, then this backtrace:
/Users/sloveless/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:277:in `block in _resort!': undefined method `name' for nil:NilClass (NoMethodError)
from /Users/sloveless/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:276:in `sort!'
from /Users/sloveless/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:276:in `_resort!'
from /Users/sloveless/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:270:in `_all'
from /Users/sloveless/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:402:in `each'
from /Users/sloveless/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems.rb:477:in `map'
from /Users/sloveless/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems.rb:477:in `find_files'
from /Users/sloveless/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems.rb:1061:in `load_plugins'
from /Users/sloveless/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/gem_runner.rb:85:in `<top (required)>'
from /Users/sloveless/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
from /Users/sloveless/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
from /Users/sloveless/.rvm/rubies/ruby-1.9.2-p180/bin/gem:9:in `'

The same thing happened when I tried this again, but upgrading from RubyGems 1.6.2, using the same update command.  I tried a 3rd time, but instead used the suggestion below to do gem install rubygems-update && update_rubygems--that just resulted in a failed install, and kept me with a usable RubyGems 1.6.2.
I'd like to a) use RubyGems 1.8.x, and b) undo these changes as opposed to having to blow away my 1.9.2 install again.
Any ideas?  I'm running:

rvm 1.6.5
MRI 1.9.2

RubyGems Environment:
  - RUBYGEMS VERSION: 1.6.2
  - RUBY VERSION: 1.9.2 (2011-02-18 patchlevel 180) [x86_64-darwin10.7.0]
  - INSTALLATION DIRECTORY: /Users/sloveless/.rvm/gems/ruby-1.9.2-p180
  - RUBY EXECUTABLE: /Users/sloveless/.rvm/rubies/ruby-1.9.2-p180/bin/ruby
  - EXECUTABLE DIRECTORY: /Users/sloveless/.rvm/gems/ruby-1.9.2-p180/bin
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86_64-darwin-10
  - GEM PATHS:
     - /Users/sloveless/.rvm/gems/ruby-1.9.2-p180
     - /Users/sloveless/.rvm/gems/ruby-1.9.2-p180@global
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :benchmark => false
     - :backtrace => false
     - :bulk_threshold => 1000
     - :sources => ["http://rubygems.org"]
  - REMOTE SOURCES:
     - http://rubygems.org
FWIW, there was a similar bug logged in the RubyGems bug repo just today (May 10), and I added similar info to that, here
Comment: What does `which gem` tell you?
Comment: `/Users/sloveless/.rvm/rubies/ruby-1.9.2-p180/bin/gem`
Comment: Thank you so much for this - I had to re-install and ended up with a new version of rubygems, and everything was messed up. This helps!
Answer: Had this problem with gems 1.{7..8}.*.
Here's our production stack:

MRI 1.9.2 and JRuby 1.6.1
RVM 1.6.5
Gems 1.6.2

Stay away from upgrading gems itself, or shit may break.  I know how tempting it is to use bleeding edge, but it's more like profusely hemorrhaging all over the O.R. walls.
If you must:

Current user only (more common): gem install rubygems-update -v 1.6.2 && update_rubygems
System-wide (caution): sudo gem install rubygems-update -v 1.6.2 && sudo update_rubygems

If using RVM, be sure to rvm gemset global first.
Usual caveats apply such as backup your data, gemsets, apps and don't try this in production. :)
Comment: Thanks... I downgraded to 1.6.2, and life is better now. I used this command: "gem update --system 1.6.2"
Comment: Downgrading is good advice, at least for now. See this bug report for more details... and please leave a supportive comment if you want my patch to be accepted. https://rubyforge.org/tracker/?func=detail&atid=575&aid=29176&group_id=126
Comment: Great analogy there.  :)  FWIW, I gave upgrading a try using the `rubygems-update` method you suggested, and that fails too--the nice thing is, though, that I'm left with a usable `gem` command, unlike with the other method.  I guess I'm stuck on 1.6.2 for now...
Comment: One of my machines is running 1.3.7, and it works perfectly. Thanks a lot for figuring out which version works.
Comment: FWIW, things seem pretty stable up till 1.6.2.  I've had pretty good success with 1.7.2, but people at work haven't shared that success and are still holding at 1.6.2.
Comment: Dredging up an old thread: I think ruby gems has their act more together now.  Upgrade with caution in a test environment yadayada.
Answer: It's not tied to RVM, as I'm running the same version. It's not Ruby 1.9.2 either, again because I'm running the same version.
gem uninstall ... returning errors isn't normal. I've done the upgrade on several machines running RVM and 1.9.2 and gem itself continued running correctly. And, yes, I saw the deprecated warnings.
Is it possibly you ran gem update using sudo? That would probably break it since you're running under RVM, and sudo is a major no-no.
Comment: Interesting.  I figured it wasn't normal, as I didn't find anyone else saying anything similar.  I'm sure I didn't run using `sudo`--I've used RVM for so long that I'm totally out of that habit. :)
Answer: Well...  I encountered the same error as you after updating rubygems and doing gem pristine --all --no-extensions.  I, on the other hand, didn't understand the major no-no about doing my gem installs and updates using sudo, so I have a whole other mess to clean up.  However, this is how I got back to a working installation:
I updated rvm by doing a fresh install:  http://beginrescueend.com/rvm/install/
I encountered some other permissions issues with this step due to my no-no sudo'ing ways, but got past it.  I did have to open a new terminal session for the changes to work.
Then, $ rvm rubygems current as described here: http://beginrescueend.com/rubies/rubygems/
This installed version 1.6.2.  So far, I seem to have a working gem command again, and my deprecation warnings are gone.  Hope it helps.


How to identify the monitor from Java Swing application?

Question: I wrote a swing application for my clients and some of them are using multiple monitors.  Unfortunately that Java Swing application is confused in environment where multiple monitors present and pop up window appear in either first or second window.  How can I identify that client has multiple monitors and force application to show up consistently on one monitor?
Answer:
To detect number of screens, use GraphicsEnvironment.getScreenDevices().
To specify the monitor, either refer to this question, or the example in the GraphicsDevice API.


When should the dependency from a core module be explicit?

Question: The Standard profile declares the Node module as one of the modules it requires, and so does the Minimal profile. That is not true for the History module, which is required from the former but not the latter.
When should I declare my module as depending from a Drupal core module? Should I assume the installation profile / distribution always requires the module from which my module depends?
Answer: In all versions of Drupal, if your module depends on another module, you should always explicitly declare it as a dependency.
Comment: @user11153 The Node module doesn't contain `required: true` in its [node.info.yml](https://api.drupal.org/api/drupal/core!modules!node!node.info.yml/8), differently from the User module which contains that line in [user.info.yml](https://api.drupal.org/api/drupal/core!modules!user!user.info.yml/8).
Comment: Even Node module?
Comment: The Node module is optional in Drupal 8.


Why it is easier to draw on some grids?

Question:
I would like to know is there any way to prove that drawing in a grid on the left would be more precise than in a grid on the right. For me, that make sense since we have more points on the left. But, is there a mathematical way to prove something like this?
Comment: You first need to say what you mean by drawing (and by precision).
Comment: @MishaLavrov I meant intuitively, for example if a child wants to draw something it seems the figure on the left will be more similar to what child wanted to draw. We can also ask if we want to draw only by connecting points of intersection of vertical and horizontal lines, would we get a "better" shape. For example, circle or parabola? Is there a mathematical proof for such a kind of relation between grids and shapes we draw in them?
Answer: If you look at the middle two lines of your left grid, there is a point every $\frac 12$ unit while on the right there is only a point every unit.  Once you extend the grid enough beyond the boundary of an area, there will be about twice as many points inside the area with the grid on the left.  However, the blocks on the left have the same area of one square unit as the blocks on the right.
You have not defined what style of drawing you are doing or how you define precision.  Are you allowed to select line segments from each grid to make the drawing?  Do you fill in blocks to color an area?  Do you connect dots that are defined by the grid with lines?  In the third case, as the left grid has all the points of the right grid and more, you must be able to be more precise.  In the first and second, it probably depends on what picture you are trying to draw.  If it follows the grid lines on the right, the left will introduce jaggies for vertical lines.
Comment: If I can draw diagonals, there are more diagonals on the left because there are more points, so we can fit a curve more accurately on the left.  Is that what you are looking for?  Contrast that with a requirement to follow the grid lines, where if the desired line is vertical the right grid can reproduce it nicely while the left has to have a lot of stairsteps.  When you ask for a mathematical proof, you need to define what you want to prove.
Comment: I meant intuitively, for example if a child wants to draw something it seems the figure on the left will be more similar to what child wanted to draw. We can also ask if we want to draw only by connecting points of intersection of vertical and horizontal lines, would we get a "better" shape. For example, circle or parabola? Is there a mathematical proof for such a kind of relation between grids and shapes we draw in them?

Why it is easier to draw on some grids?


Is there any way to add an event listener in a method in a component A, and have the target in a separate component

Question: I want to add an event listener to the component Table so that every time I save a form in the component Form by the method saveForm() I call a method called showData() in the component Table.
Form Component

let persons = [];

if (JSON.parse(localStorage.getItem("personsForms")) !== null)
    persons = JSON.parse(localStorage.getItem("personsForms"));

saveForm(myForm) {
        persons.push(myForm);
        localStorage.setItem("personsForms", JSON.stringify(persons));

    }

Table Component

let persons;

let localStoragePersons = JSON.parse(localStorage.getItem("personsForms"));
persons = localStoragePersons !== null ? localStoragePersons : [];

    showData() {
        let table = document.getElementById('editableTable');
        let x = table.rows.length;
        while (--x) {
            table.deleteRow(x);
        }
        let i = 0;

        ...........
    }

Comment: What is the relation between `Table` and `Form`? Parent, Child, siblings? Who renders who?
Comment: Table Component and Form Component are both children of the App Component
Comment: Ok, they're siblings. Then that method should be declared inside the parent, component, which is `App`. So it can pass down as `props`. If you're not happy with that, you might need to change your component hierarchy. Would it make any sense  for `Form` to be a child of `Table`?
Comment: Unfortunately i am restricted not to do both ways.
Comment: Can `showData` be declared inside parent? If not, you might be able to get around it, but it would be an anti-pattern. This is usually because you're not building your component hierarchy in the best way.
Comment: I've updated my answer. I don't recommend doing the anti-pattern, though. See how ugly and unreadable it looks.
Answer: Assuming that Table component is the parent component of Form component.
Pass the method as a prop to Form Component and call it accessing to the props (this.props.parentMethod())
Comment: Table Component and Form Component are both children of the App Component
Comment: I think you want refresh the table with the new data. So I would create a flag In App Component State with a method to toggle it. Pass this method to the Form Component call it on finish saveForm(). Pass flag to Table component and use it at render to force refresh
Comment: Is not the best practice, but you cant do anything more with this structure
Answer: From React DOCS:

Remember: React is all about one-way data flow down the component hierarchy. It may not be immediately clear which component should own what state. This is often the most challenging part for newcomers to understand, so follow these steps to figure it out:

React's one-way data flow is meant to be top-down, parent-child. The way you're suggesting would be sibling-to-sibling, which would be side-to-side instead of top-down. This is an anti-pattern.
You can set up your component hierarchy like this:
See it that works for you.

function App() {

  function doSomethingApp(formValue) {
    console.log('Do something from App!');
    console.log('Form value is: ' + formValue);
  }

  return(
    <React.Fragment>

      <Form
        doSomethingApp={doSomethingApp}
      />
    </React.Fragment>
  );
}

function Table(props) {

  function doSomethingTable(formValue) {
    console.log('Do something from Table!');
    console.log('Form value is: ' + formValue);
  }

  return(

I am Table

  );
}

function Form(props) {
  const [value,setValue] = React.useState('');
  return(
    <React.Fragment>

I am Form

      <input type='text' value={value} onChange={()=>setValue(event.target.value)}/>

        <button onClick={()=>props.doSomethingApp(value)}>Do something from App</button>

    </React.Fragment>
  );
}

ReactDOM.render(<App/>,document.getElementById('root'));

SNIPPET: ATTENTION, THIS IS AN ANTI-PATTERN
Although I wouldn't recommend this at all. This is a working example with the anti-pattern you requested.
Sibling1 has a useEffect which updates a state on the common parent App with its method showData. Whenever showData changes, the effect is run again and keeps it updated. showData is a memoized callback. You can add a depedency array to update this memoized callback whenever some other variable changes.
Source: Hooks API Reference

function App() {

  const [sibling1Method,setSibling1Method] = React.useState({method: null});

  return(
    <React.Fragment>
      <Sibling1
        setSibling1Method={setSibling1Method}
      />
      <Sibling2
        sibling1Method={sibling1Method}
      />
    </React.Fragment>
  );
}

function Sibling1(props) {

  const showData = React.useCallback((formValue) => {
    console.log('A click from Sibling2 is triggering a Sibling1 Method');
  },[]);

  React.useEffect(()=>{
    props.setSibling1Method({method: showData});
  },[showData]);

  return(

Sibling 1

  );
}

function Sibling2(props) {
  return(
    <React.Fragment>
      Sibling 2
      <button onClick={()=>props.sibling1Method.method()}>Click</button>
    </React.Fragment>
  );
}

ReactDOM.render(<App/>,document.getElementById('root'));

Comment: I believe that as you said the best way to do this is to change the hierarchy of my components, now i change Form to be a child of Table. Hope this will have a good impact. Thanks cbdev420 !!
Comment: Glad it helped. If you wish, you can accept and upvote my answer too. Happy coding to you.