Drupal as a 3-tier system model

Question: We are planning to create an 'agency' site on drupal. The 'agency' site will serve as 3-tiered system model composed of the following levels:
1. Agents (users)
2. Agency (agent users will belong to a specific agency)
3. Super Administrator (controls everything, permissions etc)
The idea behind the site is to have the ability to register an 'agency' (only by the super admin), that will in-turn have the ability to create their own users (Agents).
If Agency1 logs in, he should be able to register their own sets of users. So goes with other agencies. New users created should belong exclusively to their respective agencies, automatically upon registration.
Are there any modules out there that can be configured to do this? I have 'Profile 2' Module but I don't think this fits the bill. I am looking at the Groups Module but not sure if this is really the right one.
Answer: You could use Organic groups.
In that case, every agency would be a group with a group administrator that could set the users that are part of that agency. Every agency could have its own content, which would be created from the users that are associated with the agency. A user of another agency would not be able to create content for another agency, if not in the case the user is member of both the agencies.
Answer: Organic groups is the only way to go.
Besides that, you should disable the option for the users to registers themselves (in User Management / User settings / Only site administrators can create new user accounts).
You can then create users with administrator privileges to create users (agency managers) and agencies.
With organic groups you can also customize if a user can / cannot see the content from another agency. Here are more modules which work with OG: link.
A Drupal distribution built on top of Drupal and OG is Drupal Commons: here's a live site with that functionallity (also in this case, you should disable user registration and group creation by normal users).
Install the OG module with others modules or the DC distribution and give it a test. Good luck!
Comment: Thanks for the feedbacks! I was playing around og. There seems to be a pending issue for og for when adding a new user to a group here: http://drupal.org/node/911256 which is still unresolved since december of last year. I have yet to try it out with drupal 6 but I trust that it would become a cumbersome installations of too many modules for my needs - a real estate (agency-based) website... any thoughts?
Comment: I am still building with D6! You should also stick with D6 if you want to customize things more... a lot of little modules for organic groups are not available for D7
Comment: Thanks for all your answers! I am now doing away with og. and simply use the built in user module and modify it to include 'grouping' fields and fortunately i am now about to finish this project just by using these modules in drupal 7: (core) user, node reference, and views. The views is the beast!

Drupal as a 3-tier system model


Pinterest-style image presentation page?

Question: Any Ideas how to create pinterest style image presentation page(view)?
Basically, it is like a page that have images in columns, images in each column have same width but different height.
Comment: I restored the question to the original one. I think the question could use some work however. Given how the question is phrased, I don't think neither my (now deleted) answer, or the edit was incorrect.
Comment: Please see [this meta discussion](http://meta.drupal.stackexchange.com/questions/763/is-how-do-i-copy-part-of-site-x-an-appropriate-type-of-question), to understand why I did the edits that I did.
Answer: For that, you need to use Masonry jQuery plugin. With Drupal you've this module but it's not available yet for Drupal 7.x, so you should either make your own template with the plugin or try to port the mentioned module into Drupal 7.
Answer: I've created a small module for making pinterest-style galleries with Drupal. It's in a sandbox at the moment here:
http://drupal.org/sandbox/MarkConroy/1903438
You can see a demonstration of it here: www.adesignforlife.net/pinterest-galleries - click on any gallery to see the images also in a Pinterest style.
It basically uses the masonry js library and adds some CSS to the images.


Envia dados via post

Question: Tenho o seguinte método
    namespace App\Http\Controllers;

    use Illuminate\Http\Request;
    use Validator;
    use App\Pessoa;
    class PessoaController extends Controller
   {

    public function lista( Request $request ){
            $nome     = "%".$request->input( 'nome' )."%";
            $telefone = "%".$request->input( 'telefone' )."%";
            $empresa  = $request->input( 'empresa' );
            $setor    = $request->input( 'setor' );
            $cargo    = $request->input( 'cargo' );
            $email    = "%".$request->input( 'email' )."%";
            $pessoa = Pessoa::with(['empresa', 'setor','cargo'])
                             ->where( [
                                         [ 'nm_pessoa', 'like', $nome],
                                         [ 'telefone', 'like', $telefone ],
                                         [ 'email', 'like', $email ],
                                         [ 'cd_empresa', 'like', $empresa ],
                                         [ 'cd_setor', 'like', $setor ],
                                         [ 'cd_cargo', 'like', $cargo ]
                                    ] )
                             ->get();
            return response()->json( $pessoa );

        }
}

Tenho as seguinte rota
Route::group(['prefix' => 'api'], function(){
    Route::group(['prefix' => 'pessoa'], function (){
         Route::get( '', 'PessoaController@lista' );
         Route::post( 'add', ['as' => 'add', 'uses' => 'PessoaController@add'] );
    });
});

Gostaria de enviar dados via post, mas só que dá essa mensagem aqui:

Symfony \ Component \ HttpKernel \ Exception \
MethodNotAllowedHttpException
No message

E via get funciona
No arquivo VerifyCsrfToken.php
Eu deixei assim
protected $except = [
        'api/pessoa/*'
    ];
}

E no Kernel.php
protected $middlewareGroups = [
        'web' => [
         //   \App\Http\Middleware\EncryptCookies::class,
         //   \Illuminate\Cookie\Middleware\AddQueuedCookiesToResponse::class,
          //  \Illuminate\Session\Middleware\StartSession::class,
            // \Illuminate\Session\Middleware\AuthenticateSession::class,
        //    \Illuminate\View\Middleware\ShareErrorsFromSession::class,
        //    \App\Http\Middleware\VerifyCsrfToken::class,
            \Illuminate\Routing\Middleware\SubstituteBindings::class,
        ],

Que é para não chegar o token porque não consegui enviar do cliente fora do laravel
Comment: O problema é a barra na frente do link, remova para testar e avisa. `http://localhost/controle/api/pessoa`
Comment: Que barra? Não entendi
Comment: Na chamada do POST, no Postman. Você colocou uma barra no final do link. Remove ela.
Comment: Isso mesmo. Deu certo. Muito obrigado
Answer: É necessário remover a barra nas chamadas POST.
Se houver uma barra no final, a requisição se converte magicamente para GET.
http://localhost/controle/api/pessoa
Confesso que não tenho certeza se a alteração disso é no cliente ou no servidor, mas acredito que seja do servidor pois já vi regras de .htaccess para tratar isso.
Este conteúdo pode ajudar com algumas coisas adicionais para manter sua API legal.
Comment: Cara muito obrigado mesmo

Envia dados via post


Factoring $({16+4t^2+\frac{t^4}{4}})^{1/4}$

Question: How do you go about factoring $({16+4t^2+\frac{t^4}{4}})^{1/4}$
Which factoring method would be the best?
Comment: I would suggest factoring the inside first.  Perhaps you could try writing $s=t^2$ and using the quadratic formula (or something similar).
Comment: Recognize that the parenthesized term is a square.
Comment: Do you want the answer factored or simplified?
Comment: factored then simplified.
Comment: Which factorisation methods do you know?
Answer: Notice that $t^4/4 = (t^2/2)^2$ so let $u = t^2/2$
Then $({16+4t^2+\frac{t^4}{4}})^{1/4} = (16 + 8u + u^2)^{1/4}$
$16 + 8u + u^2 = (4+u)^2$ and $(a^2)^{1/4} = a^{1/2}$
So $({16+4t^2+\frac{t^4}{4}})^{1/4} = (16 + 8u + u^2)^{1/4} = (4 +u)^{1/2} = (4+t^2/2)^{1/2}$
Is that simplified enough?
Comment: In the finish of answer you must replace $u=t^2/2$
Comment: Didn't I?  Oh, oops.  I put the wrong variable back.  I'll fix this.
Answer: $$\left({16+4t^2+\frac{t^4}{4}}\right)^{1/4}=\left({4^2+2\cdot4\cdot\frac{t^2}{2}+\left(\frac{t^2}{2}\right)^2}\right)^{1/4}=$$
$$=\left(\left(4+\frac{t^2}{2}\right)^2\right)^{1/4}=\left(4+\frac{t^2}{2}\right)^{2\cdot{1/4}}=\left(4+\frac{t^2}{2}\right)^{{1/2}}$$


Missing rates and multiple imputation

Question: Is there a limit which is the least acceptable when using multiple imputation (MI)?
For example can I use MI if the missing values in a variable are the 20% of the cases while and other variables have missing values but not to such a high level?
Comment: A lot will depend on how much you can assume your missings are missing completely at random. If there is a high percentage of missings and they're not missing at random, you may get biased estimates for the imputation. Because it has to be done on cases present in the data (by definition), where there is a systematic bias in the missing cases the present cases may not be very informative from an accuracy perspective.
Comment: I wouldn't consider 20% very high, so I'm sure the answer to your second question is yes, although I don't have a rigorous justification. My gut feeling is that the only limit is that which prevents the algorithm from working at all. I vaguely remember seeing a table in Rubin's book showing results for very high rates of missingness. Whether it's worthwhile (as opposed to valid) doing MI with very high rates of missingness is another question.
Comment: @mark999 - Thanks for the answer. When the rate of missingness for one variable can be considered as high? Regarding your last question do you have any answer?
Comment: @Michelle - Thank you. Fortunately the hypothesis MAR is quite plausible (and even the MCAR could be considered plausible)
Comment: @Nick: I don't know what should be considered a very high rate, and I don't think it's necessary to put a specific number on it. I don't have an answer to the last question.
Comment: @Michelle I think you've mixed MCAR, MAR and NMAR here, making the first two sentences of your comment false.  Imputation is potentially bias-removing for MAR (and MCAR) data.  Under MAR, biased estimates after imputation come from having an insufficiently rich imputation model and/or from the existence of NMAR elements. For purely NMAR elements, imputation need not help and does risk bias, obviously.
Comment: @ConjugatePrior my comment said "and they're not missing at random".
Comment: @Michelle, Ah ok, I assumed you were talking about MCARs of the first sentence, which are irrelevant to imputation bias.  Then I think the whole comment then makes good sense if 'completely' is removed in the first sentence.  Curse the useless terminology!
Comment: @ConjugatePrior yes, I wish I could delete just that one word.
Comment: @Michelle :-) I had a similar problem with 'non' on this very page, which Peter Flom just caught.  I don't normally expect karma to operate on such a short scale...
Answer: From the comments, you're confident that your in a MAR or MCAR situation.  Then multiple imputation is at least reasonable.  So how much missingness is tractable?  Think of it this way:
Basically, multiple imputation makes all your model parameter estimates less certain as a function of the accuracy with which the missing data can be predicted with your imputation model, which will depend, among other things, on the amount of missing that needs imputing, and the number of imputations you use.
How much is 'too much' missingness therefore depends on how much added variance/uncertainty you are willing to put up with.  A useful quantity for you might be the relative efficiency ($RE$) of an MI analysis.  This depends on the 'fraction of missing information' (not the simple rate of missingness), usually called $\lambda$, and the number of imputations, usually called $m$, as $RE \approx 1/(1+\lambda/m)$.
Rather than generate the definitions of missing information etc. here, you might simply read the MI FAQ which puts things very clearly.  From there you'll know whether you want to tackle the original sources: Rubin etc.
Practically speaking you should probably just try an imputation analysis and see how it works out.
Comment: the FAQ link is broken. Any chance you might have a current one? (it looks useful)
Comment: Added.  Not sure how official it is though.
Answer: You might find

Rubin, Donald B. and Nathaniel Schenker. 1986. "Multiple Imputation
  for Interval Estimation from Simple Random Samples with Ignorable
  Nonresponse." Journal of the American Statistical Association
  81(394):366–374.

helpful.
Comment: Could you please summarize in few words Rubin's recommendations so that this reply remains self-contained?
Comment: @Peter Flom: Thanks (I mistakenly interpreted your "much more recently" as "recently").
Comment: @Peter Flom. Typo on my part I'm afraid.  Rest of my comment makes sense if you put the missing 'non' in front!
Comment: I will have to get that.  I have also heard Joe Shaffer of Penn State say (much more recently) that MI is good unless the data are "REALLY nonignorable"
Comment: @Peter Flom: Shaffer's statement is interesting - do you know any details?
Comment: @mark999 It was made at a talk he gave, about 10 years ago, so I don't really recall details. I haven't searched for his recent papers on the topic, but it might be there.
Comment: @Peter Flom: presumably 'really ignorable' means data that is NMAR/NI and cannot be made reasonably MAR however much you try adding possibly informative covariates.
Comment: @ConjugatePrior "really nonignorable" not really ignorable. And he meant data sets where the missingness created a lot of bias.

Missing rates and multiple imputation


has_and_belongs_to_many association that acts as "through"

Question: MODELS
Patient:
Description:
Should have only 1 Referrer
Association:
has_and_belongs_to_many :referrer

Referrer:
Description:
Contains all the types of referrers such as, but is not limited to:

Partners
etc.

Association:
has_and_belongs_to_many :referrer

Referrer Type (a new model I think I should add):
Details of referrers are here, such as name of the website that lead the patient to go to this hospital, or name of the partner that suggested the hospital to the patient.

WORKFLOW & EXPLANATIONS
In a form, the hospital employee takes the details of the patient. One of the details is the referrer -- the entity that referred the patient to this hospital. The referrer is rendered as a dropdown listbox.
I made the association between Patient-Referrer as HABTM because of the following reasons:

The Referrer model will also be used by another model aside from Patient.
The Referrer model has its own scaffold where an admin can add/remove referrers.

REQUIREMENT (THE PROBLEM)
Every time the dropdown box's value is "Partner", there should be a field to input the name of that entity. That is where the 3rd model will come in. The 3rd model will contain the:

name of the partner
a reference to the Referrer model's record named "Partner"
a reference to the Patient record.

With the 3rd model now in play, how will I be able to insert new referrer details for each patient per referrer, and making it cascadable, in a sense that when I delete the Patient, the Referrer detail will be deleted, but the Referrer will still be kept untouched, the rails way.
Answer: has_and_belongs_to_many is not so great
has_and_belongs_to_many (HABTM) is one the most limited relations in ActiveRecord and really only fits where you need a very simple many-to-many relationship and:

You will never need a join model or to attach logic to the join table
You will never need to attach any extra data to the join table
You will never need to query the join table separately

There is no HABTM "through" because HABTM does not have a join model. Thats kind of the whole point. You cannot use polymorphism in a HABTM relationship - as that requires a join model.
Using HABTM is not the correct choice here.
In most cases you should be using has_many :though instead.
Here what you want is a one to many relation with a join model:
class Patient < ActiveRecord::Base
  has_one :referral, dependent: :destroy
  has_one :referrer, through: :referral
end

class Referral < ActiveRecord::Base
  belongs_to :patient
  belongs_to :partner
  validates_uniqueness_of :patient_id
end

class Partner < ActiveRecord::Base
  has_many :referrals
  has_many :referred_patients, through: :referrals
end

Polymorpism
It is possible to use polymorphism to let the same associations point to different tables:
class Referral < ActiveRecord::Base
  belongs_to :referred, polymorphic: :true
  belongs_to :referrer, polymorphic: :true
  validates_uniqueness_of :referred_id, scope: :referred_type
end

You would need to add referred_type:string and referrer_type:string columns to referrals.
# app/models/concerns/referable.rb
module Referable
  extend ActiveSupport::Concern
  included do
    has_one :referral, as: :referred, dependent: :destroy
    has_one :referrer, through: :referral
    has_many :referrals, as: :referrer
    has_many :referred_entities, through: :referrals
                        source: :referred
  end
end

class Patient < ActiveRecord::Base
  include Referable
end

class Doctor < ActiveRecord::Base
  include Referable
end

class Partner < ActiveRecord::Base
  include Referable
end

This example would let Patient, Doctor and Partner be either the referring party or have a referral. However one big drawback is the lack of joins and eager loading for polymorphic associations since the joined table cannot be known beforehand.


Copy with symblic link

Question: I have a directory which has many libraries (ex. A,B,C) and few with symbolic link to the libraries in the same directory (ex. A1, A2 and A3 are symbolic link to A). I am copying this directory from from orignal location to another location preserving the symbolic link information of these libraries.
So my command is something like this
cp -Rp
This works well in shell.
If i move this command to makefile, the -p option can't work the way it works from shell. In makefile, the symbolic links are not maintained and instead all the libraries are just copied.
Do you have a solution which retains sym link on copying from makefile.mk
Thank you
Answer: To preserve symbolic links, I think you need -d / --no-dereference, or --preserve=links
Or maybe just -a (= -dR )

cp -a

Copy with symblic link


"maximizing" logistic regression

Question: Suppose I have some simple data for when someone buys or not buys something at a certain price, looking something like this:
ind    buy price
1      1   11
2      1   14
3      0   20
4      1   13
5      1   19
6      0   16
.      .   .
.      .   .
.      .   .
10000  1   14

Where 1 would mean buying and 0 not buying at that certain price, the numbers are just made up. I would like to fit a logistic regression to this data, dont really care in what software/language but this example is in python with the result:
                        Logit Regression Results
==============================================================================
Dep. Variable:              dependent   No. Observations:                  306
Model:                          Logit   Df Residuals:                      304
Method:                           MLE   Df Model:                            1
Date:                Fri, 15 Apr 2016   Pseudo R-squ.:                 0.01063
Time:                        09:40:27   Log-Likelihood:                -200.91
converged:                       True   LL-Null:                       -203.07
                                        LLR p-value:                   0.03773
==============================================================================
                 coef    std err          z      P>|z|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
price         -0.2341      0.114     -2.058      0.040        -0.457    -0.011
intercept      2.4598      1.436      1.712      0.087        -0.356     5.275
==============================================================================

I'm not sure if a logistic regression would be the correct way to do this, but how would I maximize (not sure if im supposed to maximize anything though) the logistic function to determine what price would be the best price?
The optimal price would be defined where the total revenue is maximized, thus $E[R_i]=R_i⋅Pr(R_i)$ where $R_i$ would be the expected revenue of individ $i$ and $Pr(R_i)$ would be the probability of purchasing at that price. What would be the optimal price here?
Comment: @lrnzcig thanks for your comment, not exactly what I was after but still it is a good way to go and worth trying
Comment: This is not exactly what you want, but maybe closed enough. You could look at it as if you have run an A/B test with your customers, let's say for one of the groups you've kept the prices as they are, for others you've offered a discount. Then, you could use [uplift modeling](https://en.wikipedia.org/wiki/Uplift_modelling) to predict the effect on lowering the price. From the results, you could do a cost/benefit analysis. You can use logistic regression for fitting the uplift. It is not even closed to what you want, but anyway, in case it helps.
Answer: Let $p$ be the price of the item and let $Q$ be the corresponding indicator of a sale (your quantity variable).  Then your logistic regression can be specified as:
$$\mathbb{E}(Q) = \text{logistic}(\beta_0 + \beta_1 p).$$
Hence, the expected revenue at price $p$ is the unknown quantity:
$$\begin{equation} \begin{aligned}
\bar{R}(p)
\equiv \mathbb{E}(\text{Revenue})
&= p \cdot \mathbb{E}(Q) \\[6pt]
&= p \cdot \text{logistic}(\beta_0 + \beta_1 p) \\[6pt]
&= \frac{p}{1 + e^{-\beta_0 - \beta_1 p}}. \\[6pt]
\end{aligned} \end{equation}$$
Note that this function involves unknown parameters $\beta_0$ and $\beta_1$ and so you cannot maximise it.  Substituting the parameter estimates gives the predicted revenue:
$$\begin{equation} \begin{aligned}
\hat{R}(p)
\equiv \hat{\mathbb{E}}(\text{Revenue})
&= \frac{p}{1 + e^{-\hat{\beta}_0 - \hat{\beta}_1 p}}. \\[6pt]
\end{aligned} \end{equation}$$
This latter function is known and can therefore be maximised via standard calculus techniques.  In particular, it has first derivative:
$$\frac{d \hat{R}}{dp}(p) = \frac{1 + (1+\hat{\beta}_1 p) e^{-\hat{\beta}_0 - \hat{\beta}_1 p}}{(1 + e^{-\hat{\beta}_0 - \hat{\beta}_1 p})^2},$$
which leads to the critical point equation $1 + \hat{\beta}_1 \hat{p} = - e^{\hat{\beta}_0 + \hat{\beta}_1 \hat{p}}$.  As Ingolifs correctly points out in another answer, solving this critical point equation uses the Lambert W function.
Answer: So you have the following functions describing your data:

And you want the maximum of GLM result * price.
You have two approaches:
1. Solve analytically. You have an equation of the form $\frac{x}{1+e^{-(ax+b)}}$ where $a$ and $b$, according to your model, are -0.2341 and 2.45, respectively.
You can use calculus to find the optimum. It is a bit involved, however, as you will (I'm pretty sure, but could be wrong) need to use the Lambert W function.
The other alternative is to plot the values of $\frac{x}{1+e^{-(ax+b)}}$ from some range that contains the maximum, say, 1:100, and find the maximum value, and the price that gives the maximum value that way. It won't be exact, but you probably don't need exact.
Answer: Partially answered in comments:
This is not exactly what you want, but maybe closed enough. You could look at it as if you have run an A/B test with your customers, let's say for one of the groups you've kept the prices as they are, for others you've offered a discount. Then, you could use uplift modeling to predict the effect on lowering the price. From the results, you could do a cost/benefit analysis. You can use logistic regression for fitting the uplift. It is not even closed to what you want, but anyway, in case it helps.  – lrnzcig

"maximizing" logistic regression


Explain Numerically The Following

Question: This question is one of the unsolved problems in my Encryption and Cipher Systems subject:
For prime $p>2$ and $0<a<p\,;$
$
a^{(p-1)/2}(mod \,p)=
\begin{cases}
1;\ \ \ \ if \ \ \ \, a\in R_{2}\\
p-1;\ \ \ \ \text{otherwise}\\
\end{cases}
$
where $R_{2}$ denotes the set of quadratic residues modulo $p$ that is, there exists $x$ such that :
$x^{2} \equiv a \, mod \,p$
I really don't understand what it does mean by the term "Numerically " also i don't know how and where to start to write a proof of such problem, So i haven't tried anything. Any help will be appreciated and thanks a lot
Comment: You posted [the same question](https://math.stackexchange.com/q/3736922/409) a couple of hours ago. Re-posting is not the way to attract attention (nor to escape down- and close-votes). Rather, you could/should edit that question to perhaps address comments you have received to improve its quality.  (Voting to close as duplicate.)
Comment: You posted a similar question earlier.
Answer: Well, let's make an abstract algebric approach.
Let A the group quotient with prime $p$. And let $a$ be element of A/{0,1}
As U(p) is generator of A, we know:
$$a^{p-1}=1=e$$ where e is element neutre multiplicatif.
with $p-1$ is the smallest number which satistifies that equation.
Which also means: for all integer $a>1$ and $p$ prime we have:
$(1)$ $$a^{p-1}=1\left(mod\:p\right)$$
if   $a∈R_2$ (1) holds. For second case let's assume that there is $a$ integer $m<p$ such that:
$(2)$
$$a^{^{\frac{\left(p-1\right)}{2}}}=m\left(mod\:p\right)$$
Square both sides:
$$a^{\left(p-1\right)}=m^2\left(mod\:p\right)$$
By (1) we have
$m^2=1$ so $m=1$ or $m=-1$
By (1) $m$ cannot be $1$ because $p-1$ is smallest number which satisfies equation. If $m=1$ we
get (2) satisfies same equation with $\frac{\left(p-1\right)}{2}$.
Finally we get $m=-1$
Comment: i couldn't find on mathtype but 1 has a bar on it on first equation

Explain Numerically The Following


Passing props from react router to children on the server

Question: I'm building an Isomorphic app using React, react-router v3 and material-ui. One of the requirements of material-ui in server-side rendering is to pass to the theme the client's user agent, so MUI will be able to prefix their inline styles accordingly.
Originally the root component of the app was the router, i.e on the server side:
<RouterContext {...renderProps}/>

And on the client:
<Router children={routes} history={browserHistory} />

Now, since I didn't know how to pass the user agent to the RouterContext on the server, I came up with an (ugly?) solution: I've created a useless component named Root, to whom I passed the user agent, and Root had the router as his children, i.e. on the server side:
<Root userAgent={userAgent}>
  <RouterContext {...renderProps}/>
</Root>

and on the client:
<Root>
  <Router children={routes} history={browserHistory} />
</Root>

Now, everything works well but I don't really like creating that useless element if I don't have to, so my question is - is there a better way?
Can I somehow pass the user agent to RouterContext on the server, which will in turn pass it to the Index Route which will create the theme?
Here's the code of Root if someone's interested:
class Root extends React.Component {
  constructor(props){
    super();
    this.muiTheme = getMuiTheme(customTheme, { userAgent: props.userAgent });
  }

  render () {
    return (
    <MuiThemeProvider muiTheme={this.muiTheme}>
      {this.props.children}
    </MuiThemeProvider>
    );
  }
}

Answer: You can use createElement on RouterContext to achieve this.
<RouterContext
  {...renderProps}
  createElement={(Component, props) => <Component {...props} userAgent={data.userAgent} />}
/>

This will make userAgent available in the props of all route components.
Comment: Thanks, I eventually used a dummy component `Root` to which I pass the props, that will be the parent of `RouterContext`. Your solution is better ofcourse :)


Como reduzir o tempo de criação e gravação de uma imagem Linux para uso no RapsberryPi?

Question: Estou retomando um projeto no qual preciso do RapsberryPi, porém na última tentativa para criar uma imagem é gravá-la no SdFlash, utilizando o comando DD, o processo levou horas para concluir, acabei deixando e não contabilizei o tempo mas farei novamente este fim de semana e atualizo esta dúvida.
É importante salientar que estou usando o Mac, até onde tenho conhecimento os comandos do Shel, não tem nenhuma diferença, e este em especial não tem nenhum bug que possa levar a esta demora.
Outro dado importante, a imagem usada foi criada com 32GB.
O cartão é um Kingston SD HC 32GB
mac
Answer: Qual a marca do seu cartão? A primeira vez que eu tentei gravar a imagem do Raspberry Pi (também num Mac), eu usei um cartão Kingston no qual eu só conseguia gravar a 300 KB/s. Eu comprei um cartão ADATA Class 10, e agora eu gravo a 10 MB/s, que é uma velocidade bem mais razoável — a imagem do Raspberry Pi não ocupa os 16 GB, mas demoraria meia hora pra gravar a totalidade do cartão.
Aqui tem uma amostragem das marcas de cartões com as respectivas velocidades.


Did baroque composers expect you to "bring out" the voices in their pieces, the way today's critics seem to enjoy in players?

Question: So, I recently had a, sadly, way too short conversation with a pianist on the train.
According to the guy, in the 17th and 18th century polyphonic keyboard works were not played in such a way that the individual voices would be emphasized, but rather in a way so that they would "blend in".
With the music in front of him/her the listener would be able to admire the mastery of the composer in building the piece out of the individual voices.
These days, though, critics seem to appreciate a pianist's ability in "bringing out" the voices.
I hear that all the time whenever Gould enters the conversation.
So, is it true that, in fact, the most appreciated players of baroque music of the 20th and 21st centuries would not have met the tastes of the time?
Comment: Gould is both the most appreciated and most criticised performer of Baroque music! And he's translating the music to piano, which has a quite different set of expressive possibilities to the instruments the composers knew. Enjoy!
Answer: Well, the thing to remember is that the harpsichord and organ have no touch sensitivity like piano, and the piano wasn't invented yet.  So any kind of keyboard music was written to be played all at the same volume, and composers made the sound fuller or emptier by managing the voicing.
If you play a Bach fugue on a piano, you can add dynamics but it won't be what Bach intended.  And lots of people are, for better or worse, adamant purists about these things.
The question gets a bit trickier when you start talking about orchestration.
Comment: This is generally true (+1). That said, there is at least *some* ability to control individual voices on harpsichord and organ through articulation, by varying how connected or disjointed the notes are, and adding ornamentation. Also, many harpsichords and organs have multiple manuals which can be connected to different stops, allowing you to play with the registration in interesting ways.
Comment: True, @CalebHines, although switching registrations in midstream was not usually an easy task, not to the same extent as setting up the registrations first and switching manuals, hence seeing markings other than ***p*** and ***f*** is rare (and even seeing ***p*** and ***f*** is a bit rare - most pieces were written to default to a single manual). Still, in various times and places (Cabezón's Spain, Bach's Germany), the clavichord was a common, and even preferred, instrument for daily music-making...
Comment: This is great, but I don't feel it completely answers the question, which deals more with intention and aesthethics then and now than with limitation of the instruments (suppose we destroy all pianos and only harpsichords are left - would a time traveller from the 17th century approve of what he heard?).
Answer: If you take a look at music like the two-part inventions and three-part symphonies by Bach, they make a lot of musical sense, and while they are intended as practice pieces, the prevalent problem (and increase of diffuculty when going from inventions to symphonies) is not one of hitting particular combinations of notes at the same time but rather of following several simultaneous parts.  Even if we stipulate that listeners were expected to be sitting there score in hand (and at the price of good music copiers, I find that somewhat hard to believe: Bach had enough work writing down all the parts and delivered a lot just in time), the player would be expected to not distract in any manner from following the voices.
If you take a look at the first movement of Sonata III for violin solo (BWV 1006) in the original handwriting, it is strikingly obvious from the notation that even for a basically "monophonic" instrument like the violin, the player is expected in several passages to diligently distribute the voicing across two to three strings in a very particular manner even when crossing voices and echoing the same note in another voice.  This same prelude has been turned into an organ prelude by Bach: the voicing remains even though the execution may be entirely different.
The characteristic distribution of voices across several strings is also prevalent in the lute version of this same prelude (the last few movements of the sonata sound a bit lacklustre on lute since they are actually monophonic): so the instrument again is expected to be played in a manner matching the voicing of the original.
Even if we are talking about instruments with fixed dynamics like a harpsichord, you will often arpeggiate simultaneous notes and can represent voice crossings by reversal of order.  But larger harpsichords tend to have more than a single manual anyway, so properly distributing multiple voices is a lot easier than on a piano as long as the respective tessiture of ongoing voices don't force you to let a voice switch hands.
So if one sees how Bach lets the character of the "abstract" music surface on different instruments for the same piece, I don't find it easy to support the contentions that the voicing is just some theoretic thing in the score that the executioner is free not to bother with.
If Bach would have this view of his work, I'd have expected him to employ figured bass a lot more than he did since that pins down "just" the harmonic framework that results from playing several notes at a time.
But to put it in a nutshell: why would the common baroque cembalo go to the expense of providing two manuals if the keyboard player was not expected to bother with voicing?
Comment: So a big "no". Great answer. This is not Wikipedia, granted, but does this answer draw from any literature?


How to choose a golf club for beginners with at least cost?

Question: I'm just begin interested in golf, so i'm intend buy one golf club but I don't have the experience to choose. What are my options?
Comment: Are you sure you want to start that badly? Learning golf with just one club is impossible. if you don't want to spend too much money just buy a used set. And you should consider taking lessons. Golf isn't really learning by doing.
Comment: Does this question help? https://sports.stackexchange.com/questions/992/how-can-i-get-started-in-golf
Answer: You cannot play golf with just one club.  You need a fairway club and a putter. For a new player I would go with like an 8 iron and a putter.
Many golf shops have a used bin.
You can find whole golf sets in resale shops cheap.
Driving ranges often have loaner clubs.


Can $x'x$ be written as correlation matrix?

Question: $x'x=$ $$
        \begin{bmatrix}
         \sum_{i=1}^{n}(X_{1i}-\bar X_1)^2&\sum_{i=1}^{n}(X_{1i}-\bar X_1)(X_{2i}-\bar X_1)\cdots & \sum_{i=1}^{n}(X_{1i}-\bar X_1)(X_{ki}-\bar X_k) \\
       \sum_{i=1}^{n}(X_{2i}-\bar X_1)(X_{1i}-\bar X_1)&\sum_{i=1}^{n}(X_{2i}-\bar X_2)^2\cdots \cdots & \sum_{i=1}^{n}(X_{2i}-\bar X_2)(X_{ki}-\bar X_k)  \\
        \vdots &\vdots&\vdots \\
        \sum_{i=1}^{n}(X_{ki}-\bar X_k)(X_{1i}-\bar X_1)&\sum_{i=1}^{n}(X_{ki}-\bar X_k)(X_{2i}-\bar X_2)\cdots & \sum_{i=1}^{n}(X_{ki}-\bar X_k)^2 \\
        \end{bmatrix}
$$
how can i transform it to correlation matrix as :
$$x'x=
        \begin{bmatrix}
        1 & r_{12}\cdots & r_{1k}\\
        r_{21} & 1\cdots & r_{2k} \\
        \vdots &\vdots&\vdots \\
         r_{k1} &  r_{k2}\cdots &1\\
        \end{bmatrix}
$$?
I know the elements are  the numerator of correlation, i.e.,
$$r_{ij}=\frac{\sum_{u=1}^{n}(X_{iu}-\bar X_i)(X_{ju}-\bar X_j)}{\sqrt{\sum_{u=1}^{n}(X_{iu}-\bar X_i)^2\sum_{u=1}^{n}(X_{ju}-\bar X_j)^2}}$$
.But i can't transform it to correlation formula.
Comment: is this homework? Check the [self-stufy tag](http://stats.stackexchange.com/tags/self-study/info)
Answer: Let $p_i = [x'x]_{ii}$ be the $i$th diagonal element of $x'x$. Now define the $k\times k$ diagonal matrix $s$ so that $[s]_{ii} = 1/\sqrt{p_i}$ and all non-diagonal elements equal to zero Then I think the correlation matrix is
$$
r = s x'x s
$$
Comment: @harry That's a standard result from matrix algebra.
Comment: How did you find the way so quickly?


Zurb Foundation + Compass

Question: I've used Zurb Foundation Framework to build my website, now I'd want to use "Sticky footer" of Compass but I cannot include Compass
gem 'zurb-foundation', :git => "https://github.com/zurb/foundation.git"
gem "compass"

Application.scss
@import "bourbon";
@import "foundation_and_overrides";
@import "buttons";
@import "font-awesome";
@import "page";
@import "pagination";
@import "footer";
@import "app";

footer.css.sass
@import compass/reset.scss
@import compass/layout.scss

+sticky-footer(72px, "#layout", "#layout_footer", "#footer")

Answer: That looks like Rails4 -- compass is not yet ready for Rails4.
You could try your luck with version 2.0.alpha.0  on compass-rails -- it's been released for about 4 days now.
gem 'compass-rails', '2.0.alpha.0'

Comment: See also the discussion here, Foundation had a brief hiccup with Compass breaking on Bower update: http://foundation.zurb.com/forum/posts/18624-foundation-5-and-compass-10

Zurb Foundation + Compass


Displaying Line Item Fields in a View (SQL Error)

Question: I've built a conference registration system using Commerce and Custom Line Item types, and I'm having trouble figuring out how to display my Custom Line Item fields in a View. Here's a short summary of what I've done:

The registration fields (e.g. name, email, etc.) are fields in a
custom line item type.
My product view doubles as a registration form. It displays the fields of my custom line item type as part of the add to cart
form. It also references a custom product type I created for the
conference.
Some of the fields in the custom line type are booleans that trigger rules that add additional related products to the cart (e.g. hotel accommodations, meals, etc.).

I've built a View that summarizes the store's Orders, so that the conference organizer can see all of the Order info without having to know anything about Commerce or the Drupal admin pages. When I add the Line Items to this Order View, only the products show up (both the product referenced in my view, and those added by rules) - the other fields I've created for capturing the registration data (name, email, etc) don't show up.
I've also attempted to make a View that shows only Line Items (filtered to only my Custom Line Item Type), but nothing shows up and I get the following SQL error:

SQLSTATE[42S22]: Column not found: 1054 Unknown column 'commerce_line_item.uid' in 'field list'

Any help on getting my other Line Item fields to show up, or insight into this SQL error, would be MOST appreciated. Thanks!
(For some brief background, I have built other solutions using Commerce in conjunction with the Registrations module. This has caused a lot of problems, so I am attempting to recreate similar functionality without the Registration module)
Comment: whats the full SQL query being issued?
Comment: Hi, your question actually led me to a solution - I removed the Order ID and Line Item ID fields and it's displaying everything now with no error. I put the fields back in and it's still working. Not sure why it's working now but wasn't before...
Answer: I was running into the same problem too. You have to create the relationship "Commerce Line Item: Order ID" and display the field "Commerce Order: Order ID" to get around the mysql error.
You can get the view by importing the following code:
<code>    $view = new view();
    $view->name = 'commerce_line_items';
    $view->description = '';
    $view->tag = 'default';
    $view->base_table = 'commerce_line_item';
    $view->human_name = 'Commerce Line Items';
    $view->core = 7;
    $view->api_version = '3.0';
    $view->disabled = FALSE; /* Edit this to true to make a default view disabled initially */

    /* Display: Master */
    $handler = $view->new_display('default', 'Master', 'default');
    $handler->display->display_options['title'] = 'Commerce Line Items';
    $handler->display->display_options['use_more_always'] = FALSE;
    $handler->display->display_options['access']['type'] = 'none';
    $handler->display->display_options['cache']['type'] = 'none';
    $handler->display->display_options['query']['type'] = 'views_query';
    $handler->display->display_options['exposed_form']['type'] = 'basic';
    $handler->display->display_options['pager']['type'] = 'full';
    $handler->display->display_options['pager']['options']['items_per_page'] = '10';
    $handler->display->display_options['style_plugin'] = 'default';
    $handler->display->display_options['row_plugin'] = 'fields';
    /* Relationship: Commerce Line Item: Order ID */
    $handler->display->display_options['relationships']['order_id']['id'] = 'order_id';
    $handler->display->display_options['relationships']['order_id']['table'] = 'commerce_line_item';
    $handler->display->display_options['relationships']['order_id']['field'] = 'order_id';
    /* Field: Commerce Line Item: Line item ID */
    $handler->display->display_options['fields']['line_item_id']['id'] = 'line_item_id';
    $handler->display->display_options['fields']['line_item_id']['table'] = 'commerce_line_item';
    $handler->display->display_options['fields']['line_item_id']['field'] = 'line_item_id';
    /* Field: Commerce Order: Order ID */
    $handler->display->display_options['fields']['order_id_1']['id'] = 'order_id_1';
    $handler->display->display_options['fields']['order_id_1']['table'] = 'commerce_order';
    $handler->display->display_options['fields']['order_id_1']['field'] = 'order_id';
    $handler->display->display_options['fields']['order_id_1']['relationship'] = 'order_id';
    $handler->display->display_options['fields']['order_id_1']['link_to_order'] = 'customer';
</code>
Comment: You also need to remove the field "Commerce Line Item: Order Id" if it's been added.


Please help me integrate the following: $\int \frac{y^2 - x^2}{(x^2 + y^2)^2}dy$

Question: I'm self-studying a Cramster solution and I came across this integral and I don't know what they've done with it. Help would be appreciated.
$$\int \frac{y^2 - x^2}{(x^2 + y^2)^2} ~dy.$$
Comment: Is x constant wrt y?
Answer: In an integral $dy$, $x$ is a constant. Rewirite this as:
$$\int\frac{y^2-x^2}{\left(x^2+y^2\right)^2}dy=\int\frac{y^2+x^2-2x^2}{\left(x^2+y^2\right)^2}dy=\int\frac{dy}{x^2+y^2}-2x^2\int\frac{dy}{\left(x^2+y^2\right)^2}$$
Can you continue from here?
Comment: This is the same as the "partial fractions" method, correct? I think I can take it from here. Thanks.
Comment: Yes, this is a simple example of the partial fractions method.
Answer: If x is constant wrt y,
let $y=x. tan(z)$,  then $dy=xsec^2zdz $
Then $$\int \frac{y^2 - x^2}{(x^2 + y^2)^2} ~dy$$ becomes
$$\frac{-1}{x}\int cos(2z) ~dz$$
=$\frac{-sin(2z)}{2x} + C$, where C is an undetermined constant.
=$\frac{-tan(z)}{x(1+tan^2z)} +C$
=$\frac{-y}{x^2+y^2}+ C$


Url rewriting stops working after changing default port on iis7

Question: I have migrated the IIS6 webserver 2003 websites to IIS7 webserver 2008 using msdeploy tool. Application pool setting are changed with "Enable 32-bit Applications=true", "Managed_Pipeline_Mode=Classic","Identity=NetworkService" Framework=v1.1/2.0.
All the websites are working fine on default port along with url rewriting migrated from iis6. When I start the webserver on port other than default port by changing bindings, url rewriting stops workings and get 404 error in logs. I think I don't have to change the handler mapping cause I am running it in classic mode. How can I troubleshoot this?
Comment: You can start by showing us your rewrite rules
Comment: Thanks for the comment. Sorry for my late reply. My web development team said that url rewriting rules are written in the codes and so they are looking into the codes to add the port information for url rewriting. I am waiting for their revert on this. I'll be able to put more information once I get the clear idea hot url rewriting was done in iis6 by the team.
Answer:
Maybe silly, but I will ask anyway, did you restart IIS service?
When you switch your website to listening on a non standard port, the port number become part of your base URL like following
http://example.com:9000

Webapp or site config may have to be updated to match that.
Rewrite rules involving base url may have to be updated.


How to import a csv with copy function using php

Question: I need to import a csv file into my database.
I first wrote the function in SQL but now I need to do it in php, I don't want to change my entire code so I just figured I could execute the SQL query with php but I get an error.
<code>    $query="COPY bet_import FROM $1 DELIMITER ',' CSV HEADER";
    $result = pg_prepare($dbh, "", $query);
    $result = pg_execute($dbh, "",array('bet.csv'));
</code>
I get the error:

Error: Warning: pg_prepare(): Query failed: ERROR:  syntax error at or near "$1"
           LINE 1: COPY bet_import FROM $1 DELIMITER ',' CSV HEADER
Answer: You cannot bind identifiers (table name, field name) in the prepare statement. You need to add that into the query. If you are dynamically getting it you can escape it or sanitize it before you run the query. In your query the file path is an identifier.
<code>$filePath = 'bet.csv';
$query="COPY bet_import FROM '$filePath' DELIMITER ',' CSV HEADER";
$result = pg_prepare($dbh, "", $query);
$result = pg_execute($dbh, "",array());
</code>
You have to pass the filename in the query itself. This query will run on the postgresql so you need to specify the absolute path of the file.. If this file is placed in same code as your php code then also you need to specify the absolute path.


Como trabalhar com github quando A2F está habilitada?

Question: Ao habilitar a autenticação de dois fatores no Github sou obrigado a trabalhar com chave RSA para atualizar os repositórios?
Quando A2F (Authentication Two-Factor) está ativa o código único é solicitado, imagino que o Git em si não trabalha com isso, logo, acredito que, neste cenário sou obrigado a usar uma chave de autenticação, tipo RSA. Estou correto?
Answer: Não é obrigado.
Usando "A2F" (ou 2FA ou Two-factor Authentication), você pode trabalhar usando a chave RSA (chave SSH) ou via HTTPS.
Se quer usar HTTPS, você precisa criar um token de acesso pessoal que será usado como senha quando for autenticar via linha de comando usando uma URL https. Desta forma, o uso do token fica transparente mesmo que o Git em si não trabalhe com isto.
Comment: Bacana, essa do token eu não sabia. Obrigado!


How or can I, loop each track independently?

Question: There's plenty of awesome music where people on analog hardware, loop overlapping tracks with inequal sizes that have prime ratios of track lengths, so that they never quite repeat at the same point.
Since the point is that they don't repeat, it would be hard to do the usual fix of duplicating each short track till it lines up perfectly with a longer track, or duplicating the longer track until you find their common factor, that won't work, since these are nonrepeating decimals and whilst computers have finite precision so an effective point of overlap will occur eventually on a digital computer, it would be very large and inefficient.
So I was wondering if there's a way to loop each track independently, or a way to make that suggestion to official audacity devs so that a dev can add it at a future date.
Sorry if this is easily fixable.
If so how?
If not, what are the next steps for doing this?
Comment: So, is this a question about the usage of audacity? Because it wasn't very clear until I read the final reference and tag (the subject should be *clearly* specified in the title or, at least, in the beginning of the topic, tags are not intended as a "main description", but only as categorization). In any case, to me this question seems a bit off topic: it's clearly not about theory, and it doesn't really fit to music practice. And, I've to say, I don't really understand what is the actual question. Are you referring to the loop playback option?
Comment: In most DAWs you just align to a bar/beat, so it doesn't drift. Not sure if audacity even has that concept.
Answer: In audacity: select clip you want to loop, and then open menu Effects → Repeat. Then do the same for the each other clip on other tracks.
Comment: Effects->Repeat, is a way of making multiple copies. I was asking, about how to make each track loop independently rather than the silence that lasts for the difference between every other track and the longest track.
Comment: Effects Repeat, when I use it asks "repeat how many times" (paraphrasing), then you enter an integer between 1 => 1, and it makes that that many copies of that track appear.
Comment: Because for some ratios of track length: making them copied until they line up neatly, could be an actual calulation of pi. For example. That's a lot of manual repeat operations.
Comment: @user179283 Are you saying that you don't have the tracks actually cut to the length and you need some quantization tools? Then perhaps you need to use some proper DAW, rather than audacity.
Comment: Sure. I did recently get one. But that should be a feature in audacity too, shouldn't it? Audio generation like that would be wild, ya know?   Tracks that don't match should loop seperately on the same output. That should be an option atleast. But are you saying ir isn't? That's okay with me if that's that. Room for a feature suggestion I guess tho
Comment: @user179283 If you loop clips that are not the same length and their lengths ratios are not ratio of small integers, then they will not line up. It's not about audacity, it's about maths. Or I don't understand what exactly do you want to achieve. Perhaps some examples would be helpful.


Find Elementary Matrics E1 and E2 such that $E_2E_1$A = I

Question: I am studying Linear Algebra part-time and would like to know if anyone has advice on solving the following type of questions:
Considering the matrix:
$$A = \begin{bmatrix}1 & 0 & \\-5 & 2\end{bmatrix}$$
Find elementary Matrices $E_1$ and $E_2$ such that $E_2E_1A = I$
Firstly can this be re-written as?
$$E_2E_1 = IA^{-1}$$
and that is the same as?
$$E_2E_1 = A^{-1}$$
So I tried to find $E_1$ and $E_2$ such that $E_2E_1 = A^{-1}$:
My solution:
$$A^{-1} = \begin{bmatrix}1 & 0 & \\{\frac {5}{2}} & {\frac {1}{2}}\end{bmatrix}$$
$$E_2 = \begin{bmatrix}1 & 0 & \\0 & {\frac {5}{2}}\end{bmatrix}$$
$$E_1 = \begin{bmatrix}1 & 0 & \\1 & {\frac {1}{5}}\end{bmatrix}$$
This is the incorrect answer. Any help as to what I did wrong as well as suggestions on how to approach these questions would be aprpeciated.
Thanks
Comment: To get the matrices. Apply the elementary row operations on $A$. Each elementary row operation is equivalent to multiplying the corresponding elementary matrix. Two row operations get you $I$, so you get two elementary matrices corresponding to them.
Comment: Is your $E_1$ really an elementary matrix?
Answer: Just look at what needs to be done.
First, eliminate the $-5$ using $E_1 = \begin{bmatrix} 1 & 0 \\ 5 & 1 \end{bmatrix}$. This gives
$$ E_1 A = \begin{bmatrix} 1 & 0 \\ 0 & 2 \end{bmatrix}.$$
Can you figure out what $E_2$ must be?
Answer: You can reduce A to I in two elementary row transformation,

$R_2 + 5R_1 \rightarrow R_2$
$\frac{1}{2} R_2 \rightarrow R_2$

An elementary matrix is a matrix that differs from I in exactly one elementary transformation. It turns out that you just need matrix corresponding to each of the row transformation above to come up with your elementary matrices. For example, the elementary matrix corresponding to the first row transformation is,
$$\begin{bmatrix}1 & 0\\5&1\end{bmatrix}$$
Notice that when you multiply this matrix with A, it does exactly the first elementary row transformation. So, this is $E_1$. Similarly, figure out $E_2$ by getting the elementary matrix corresponding to the second transformation.
Answer: Thanks for the advice and hints TenaliRaman and copper.hat. I'll post the solution here for anyone else who needs to solve something similar.
So we have to find the elementary matrices $E_1$ and $E_2$ such that $E_2E_1A = I$:
Start by eliminating the -5 in matrix A.
So find $E_1$ so that
$$E_1A = \begin{bmatrix}1 & 0 & \\0 & 2\end{bmatrix}$$
the matrix $E_1$ is thus:
$$E_1 = \begin{bmatrix}1 & 0 & \\5 & 1\end{bmatrix}$$
If we rewrite the original equation like so:
$$E_2(E_1A) = I$$
Find $E_2$ (since we know what $E_1A$ is)
Solving this we get:
$$E_2 = \begin{bmatrix}1 & 0 & \\0 &  {\frac {1}{2}}\end{bmatrix}$$
Thanks everyone!


Violation of test set independence in classification?

Question: I perform multi-class classification using a Deep Neural Network. I split the data set into training, validation, and test set. To create a strong classificator, I train multiple models using various parameters (e.g. learning rate). Then I compare the results of each trained model on the test set, choosing the model that performed best.
Is this a valid approach? Or should choosing a model based on results on the test set be seen as an optimizing process, which might be biased towards the specific test set and thus not generalize well (overfitting)? Basically, am I violating the independence of the test set with this approach? If so, what would be the correct way to split the data with this approach?
Comment: You can use the test set to select your model, if you are indeed using the test set only to test the model.
Comment: If you use your validation set to select your model, you can keep your test set just for reporting on performance metrics
Answer: Using your test set for this purpose seems fundamentally wrong. Your algorithm choice is not conceptually different from tuning your hyper parameters. For example, a neural-net (NN) and a log-reg (LR) might be among your models to select; while a specific version of your neural-net (i.e. one neuron neural net with sigmoid activation func.) might correspond exactly to a log-reg in your hyper-parameter tuning process, where you advocate that there should be a validation set. Then, if you can use your test set for deciding between NN and LR; why can't you use them for hyper parameter tuning? This is why you should again have an independent test set.


NullPointerExecption when calling a PhoneStateListener in a service

Question: I've got a PhoneStateLister where I get a bunch of data and I send to a server.
If I call the listener in a fragment, it's all good, but when I'm calling it in a service, I get a NullPointerException.Here is my code:
PhoneStateListener:
public class CarrierStateListener extends PhoneStateListener {

private Context context;
private TelephonyManager telephonyManager;
private String carrierName;
private int mSignalStrength = 0;
private int mCellTowerID = 0;
private int mCellTowerAreCode = 0;
private int mcc = 0;
private int mnc = 0;

public CarrierStateListener(Context context) {
    this.context = context;
}

/**
 * Get the Signal strength from the provider, each time there is an update
 */
@Override
public void onSignalStrengthsChanged(final SignalStrength signalStrength) {
    super.onSignalStrengthsChanged(signalStrength);

    telephonyManager = (TelephonyManager) context.getSystemService(Context.TELEPHONY_SERVICE);
    GsmCellLocation loc = (GsmCellLocation) telephonyManager.getCellLocation();
    String networkOperator = telephonyManager.getNetworkOperator();

    /** Retrieving the carrier name **/
    carrierName = telephonyManager.getNetworkOperatorName();
    UserInformation.getSingleton(context).setNetwork(carrierName);

    /** Retrieving the RSSI value **/
    mSignalStrength = signalStrength.getGsmSignalStrength();
    mSignalStrength = (2 * mSignalStrength) - 113; // -> dBm
    UserInformation.getSingleton(context).setRssi(String.valueOf(mSignalStrength));

    /** Retrieve the cell tower id (CID) and the LAC (location area code) **/
    mCellTowerID = loc.getCid();
    mCellTowerAreCode = loc.getLac();
    UserInformation.getSingleton(context).setCellId(String.valueOf(loc.getCid()));

    /**tower mobile network code (mnc) and mobile country code (mcc) **/
    if (networkOperator != null) {
        mcc = Integer.parseInt(networkOperator.substring(0, 3));
        mnc = Integer.parseInt(networkOperator.substring(3));
        UserInformation.getSingleton(context).setMnc(String.valueOf(mnc));
        UserInformation.getSingleton(context).setMcc(String.valueOf(mcc));

    } else {
        Toast.makeText(context, "Couldn't retrieve mnc and mcc because network operator is "
                + networkOperator, Toast.LENGTH_SHORT).show();
    }

}

and My Service:
public class SendDataService extends Service {

private static final String TAG = "SendDataService";
private Thread t;
private TelephonyManager telephonyManager;
private CarrierStateListener pslistener;

@Override
public IBinder onBind(Intent intent) {
    return null;
}

@Override
public int onStartCommand(Intent intent, int flags, int startId) {

    PowerManager pm = (PowerManager) getSystemService(Context.POWER_SERVICE);
    PowerManager.WakeLock wl = pm.newWakeLock(PowerManager.PARTIAL_WAKE_LOCK, "SendDataService");
    wl.acquire();

    t = new Thread() {
        @Override
        public void run() {
            while (true) {
                try {

                    Log.e(TAG, "sending data...");

                    //post request to send data
                    Thread.sleep(6000);

                } catch (InterruptedException ie) {
                }
            }
        }
    };
    t.start();

    Log.e(TAG, "the SendDataService started");
    return START_STICKY;
}

and I start the Service in onResume() of a fragment:
getActivity().startService(new Intent(getActivity(), SendDataService.class));

when the fragment starts...this is the EXCEPTION I get:
09-14 16:11:05.034  21697-21697/? E/CallDataService﹕ the SendDataService started
09-14 16:11:05.036  21697-21729/? E/CallDataService﹕ sending data...
09-14 16:11:05.037  21697-21729/? E/AndroidRuntime﹕ FATAL EXCEPTION: Thread-2445
    Process: xxxxxxxx, PID: 21697
    java.lang.NullPointerException: Attempt to read from field

'android.os.MessageQueue android.os.Looper.mQueue' on a null object reference
            at android.os.Handler.<init>(Handler.java:229)
            at android.os.Handler.<init>(Handler.java:137)
            at android.telephony.PhoneStateListener$1.<init>(PhoneStateListener.java:270)
            at android.telephony.PhoneStateListener.<init>(PhoneStateListener.java:270)
            at android.telephony.PhoneStateListener.<init>(PhoneStateListener.java:240)
            at xxxxxxxxxxxxx.CarierStateListener.<init>(CarierStateListener.java:27)
            at xxxxxxxxx.SendDataService.listenForEvents(SendDataService.java:96)
            at xxxxxx.util.SendDataService.access$000(SendDataService.java:25)
            at xxxxxxxxxx.util.SendDataService$1.run(SendDataService.java:53)

Comment: show `SendDataService` class code
Comment: it's displayed under "My service"
Comment: Where is `listenForEvents` method? please show
Comment: `telephonyManager = (TelephonyManager) SendDataService.this.getSystemService(Context.TELEPHONY_SERVICE);
        pslistener = new CarrierStateListener(SendDataService.this);
        telephonyManager.listen(pslistener, PhoneStateListener.LISTEN_SIGNAL_STRENGTHS);`
Comment: As in log it's show problem is at line number 96 in `listenForEvents` method of `SendDataService` class
Comment: Did answer from Kvant help you? If so, please mark it, if not, could you provide your own answer to the problem?
Answer: This is a message queue problem you have to initialize the looper before calling telephonyManager.listen and loop afterwards.
 new Thread(new Runnable() {
                    @Override
                    public void run() {
                        quitLooper = false;
                        Looper.prepare();
                        tManager.listen(new PhoneStateListener() {
                            @Override
                            public void onSignalStrengthsChanged(SignalStrength signalStrength) {
                                super.onSignalStrengthsChanged(signalStrength);
                                lastSignalStrength = signalStrength;
                                if(quitLooper)
                                    Looper.myLooper().quit();

                            }
                        }, PhoneStateListener.LISTEN_SIGNAL_STRENGTHS);

                        Looper.loop();
                    }
                }).start();

Comment: Where do you set `quitLooper`? When doing it from "outside", perhaps it would be a reasonable approach to check for `Thread.currentThread().isInterrupted()` instead and stop the processing with `howYouCallTheThread.interrupt()`?
Comment: Does a new thread have to be created? Or, is it safe to just call `Looper.prepare()`?
Comment: @Anonsage If the listen method called from main thread there is no need for looper initialization at all.


A nice Combinatorial Identity

Question: I am trying to show that $\forall N\in\mathbb{N}$,
$$\sum\limits_{n=0}^{N}\sum\limits_{k=0}^{N}\frac{\left(-1\right)^{n+k}}{n+k+1}{N\choose n}{N\choose k}{N+n\choose n}{N+k\choose k}=\frac{1}{2N+1}$$
It's backed by numerical verifications, but I can't come up with a proof.
So far, I tried using the generating function of $\left(\frac{1}{2N+1}\right)_{N\in\mathbb{N}}$, which is $\frac{\arctan\left(\sqrt{x}\right)}{\sqrt{x}}$, by showing that the LHS has the same generating function, but this calculation doesn't seem to lead me anywhere...
Any suggestion ?
Edit: the comment of bof (below this question) actually leads to a very simple proof.
Indeed, from bof's comment we have that the LHS is equal to $$\int_{0}^{1}\left(\sum\limits_{k=0}^{N}(-1)^k{N\choose k}{N+k\choose k}x^k\right)^2dx$$
And we recognize here the shifted Legendre Polynomials $\widetilde{P_N}(x)=\displaystyle\sum\limits_{k=0}^{N}(-1)^k{N\choose k}{N+k\choose k}x^k$.
And we know that the shifted Legendre Polynomials form a family of orthogonal polynomials with respect to the inner product $\langle f|g\rangle=\displaystyle\int_{0}^{1}f(x)g(x)dx$, and that their squared norm with respect to this product is $\langle\widetilde{P_n}|\widetilde{P_n}\rangle=\frac{1}{2n+1}$;
so this basically provides the desired result immediately.
Comment: $$\sum_{n=0}^N\sum_{k=0}^N\frac{(-1)^{n+k}}{n+k+1}\binom Nn\binom Nk\binom{N+n}n\binom{N+k}k$$$$=\int_0^1\sum_{n=0}^N\sum_{k=0}^N(-1)^{n+k}\binom Nn\binom Nk\binom{N+n}n\binom{N+k}kx^{n+k}dx$$$$=\int_0^1\left(\sum_{n=0}^N(-1)^n\binom Nn\binom{N+n}nx^n\right)\left(\sum_{k=0}^N(-1)^k\binom Nk\binom{N+k}kx^k\right)dx$$$$=\int_9^1\left(\sum_{k=0}^N(-1)^k\binom Nk\binom{N+k}kx^k\right)^2dx$$ but I don't know why that integral evaluates to $\frac1{2N+1}$.
Comment: @bof : thank you very much, this actually sovled the problem (see my edit).
Comment: @HarmonicSun: Good observation. (+1)
Answer: We seek to verify that
$$\sum_{n=0}^N \sum_{k=0}^N
\frac{(-1)^{n+k}}{n+k+1}
{N\choose n} {N\choose k}
{N+n\choose n} {N+k\choose k}
= \frac{1}{2N+1}.$$
Now we have
$${N\choose k} {N+n\choose n}
= \frac{(N+n)!}{(N-k)! \times k! \times n!}
= {N+n\choose n+k} {n+k\choose k}.$$
We get for the LHS
$$\sum_{n=0}^N \sum_{k=0}^N
\frac{(-1)^{n+k}}{n+k+1} {N+n\choose n+k}
{N\choose n} {N+k\choose k}  {n+k\choose k}
\\ = \sum_{n=0}^N \frac{1}{N+n+1} \sum_{k=0}^N
(-1)^{n+k} {N+n+1\choose n+k+1}
{N\choose n} {N+k\choose k}  {n+k\choose k}
\\ = \sum_{n=0}^N \frac{1}{N+n+1} {N\choose n}
\sum_{k=0}^N
(-1)^{n+k} {N+n+1\choose N-k}
{N+k\choose k}  {n+k\choose k}
\\ = \sum_{n=0}^N \frac{1}{N+n+1}
[z^N] (1+z)^{N+n+1}
{N\choose n}
\sum_{k=0}^N
(-1)^{n+k} z^k
{N+k\choose N}  {n+k\choose n}.$$
Now the coefficient extractor controls the range and we continue with
$$\sum_{n=0}^N \frac{1}{N+n+1}
[z^N] (1+z)^{N+n+1}
{N\choose n}
\\ \times \sum_{k\ge 0}
(-1)^{n+k} z^k
{N+k\choose N}
\;\underset{w}{\mathrm{res}}\; \frac{1}{w^{n+1}} \frac{1}{(1-w)^{k+1}}
\\ = \sum_{n=0}^N \frac{1}{N+n+1}
[z^N] (1+z)^{N+n+1}
{N\choose n}
\;\underset{w}{\mathrm{res}}\; \frac{1}{w^{n+1}} \frac{1}{1-w}
\\ \times \sum_{k\ge 0}
(-1)^{n+k} z^k
{N+k\choose N} \frac{1}{(1-w)^{k}}
\\ = \sum_{n=0}^N \frac{(-1)^n}{N+n+1}
[z^N] (1+z)^{N+n+1}
{N\choose n}
\\ \times \;\underset{w}{\mathrm{res}}\; \frac{1}{w^{n+1}} \frac{1}{1-w}
\frac{1}{(1+z/(1-w))^{N+1}}
\\ = \sum_{n=0}^N \frac{(-1)^n}{N+n+1}
[z^N] (1+z)^{N+n+1}
{N\choose n}
\\ \times \;\underset{w}{\mathrm{res}}\; \frac{1}{w^{n+1}}
\frac{(1-w)^N}{(1-w+z)^{N+1}}
\\ = \sum_{n=0}^N \frac{(-1)^n}{N+n+1}
[z^N] (1+z)^{n}
{N\choose n}
\\ \times \;\underset{w}{\mathrm{res}}\; \frac{1}{w^{n+1}}
\frac{(1-w)^N}{(1-w/(1+z))^{N+1}}
\\ = \sum_{n=0}^N \frac{(-1)^n}{N+n+1}
[z^N] (1+z)^{n}
{N\choose n}
\\ \times \sum_{k=0}^n (-1)^k {N\choose k}
{n-k+N\choose N} \frac{1}{(1+z)^{n-k}}
\\ = \sum_{n=0}^N \frac{(-1)^n}{N+n+1}
{N\choose n}
\\ \times \sum_{k=0}^n (-1)^k {N\choose k}
{n-k+N\choose N} [z^N] (1+z)^k.$$
Now for the coefficient extractor to be non-zero we must have $k\ge N$
which happens just once, namely when $n=N$ and $k=N.$ We get
$$\frac{(-1)^N}{2N+1}
{N\choose N}
(-1)^N {N\choose N}
{N-N+N\choose N}.$$
This expression does indeed simplify to
$$\bbox[5px,border:2px solid #00A000]{
\frac{1}{2N+1}}$$
as claimed.
Comment: Thank you very much, nice solution !
I invite you to see my edit, based on a comment from bof I have found a much shorter proof though :)
Comment: Nice answer. Verified. (+1)
Comment: All the better for a varied page. BTW my proof is self-contained and follows from first principles. Thanks also go to @MarkusScheuer for the credit.

A nice Combinatorial Identity


Mean of maximum of exponential random variables (independent but not identical)

Question: I am looking for the the mean of the maximum of N independent but not identical exponential random variables. I found the CDF and the pdf but I couldn't compute the integral to find the mean of the maximum.
The CDF : $ \mathbf{F_{X_{max}}}(x)= \prod_{k=1}^{K} (1-exp(-\lambda_k x)) $ and
The pdf is  : $ \mathbf{f_{X_{max}}}(x)= \sum_{k=1}^{K}\lambda_k exp(-\lambda_k x) \prod_{q=1,q\neq k}^{K} (1-exp(-\lambda_q x)) $
so the mean that I am trying to find is :
 $ \mathbb{E} [X_{max}] = \int_{0}^{\infty} x \mathbf{f_{X_{max}}}(x) dx $
Comment: Could you write down the integral you have to evaluate?
Comment: Do you have some restrictions on the parameters of the exponential?
Comment: I added the expression of the integral.
Comment: I don't understand what you mean by restrictions on the exponentials parameters ?
Answer: Using this formula for expectation of positive random variables in terms of the survival function and expanding the product in your formula for the cdf,
\begin{align}
EX_\text{max}
  &=\int_0^\infty P(X_\text{max}>x)dx
\\&=\int_0^\infty1-\prod_{i=1}^n(1-e^{-\lambda_i x})dx
\\&=\sum_{S\subseteq\{1,2,\dots,n\}}(-1)^{|S|} \int_0^\infty e^{-x\sum_{j\in S}\lambda_j}dx
\\&=\sum_{S\subseteq\{1,2,\dots,n\}}(-1)^{|S|-1} \frac1{\sum_{j\in S}\lambda_j},
\end{align}
where the outer sum is over all non-empty subsets $S$ of $\{1,2,\dots,n\}$ and $|S|$ denotes the number of elements of $S$.
So for $n=1$, this simplifies to the usual formula
$$
E X_\text{max} = \frac1{\lambda_1},
$$
for $n=2$,
$$
E X_\text{max} = \frac1{\lambda_1} + \frac1{\lambda_2} - \frac1{\lambda_1+\lambda_2},
$$
for $n=3$,
$$
E X_\text{max} = \frac1{\lambda_1} + \frac1{\lambda_2} + \frac1{\lambda_3} - \frac1{\lambda_1+\lambda_2} - \frac1{\lambda_1+\lambda_3} - \frac1{\lambda_2+\lambda_3} + \frac1{\lambda_1+\lambda_2+\lambda_3},
$$
and so on.
Comment: It's just expanding the product:  $(1-a)(1-b)=1-a-b+ab$
Comment: Sorry this is not an answer, but a question to @Jarle for some clarification
I understand that there was a conversion between F(x) and S(x) between the two lines.
And the exponent changed to summation of lambda's
But can you please elaborate on the outer summation here, what rules were applied to get from line 1 to line 2 ? [![product to summation](https://i.stack.imgur.com/p8oCZ.png)](https://i.stack.imgur.com/p8oCZ.png)


Contradictory behavior in bash script versus interactive shell Mac OSx?

Question: I have the following in my .bashrc script:
alias sublime='open -a Sublime\ Text\ 2'

When the bash shell is initialized, it throws the following error:
-a: not found
Sublime Text 2' not found

However, if I run the following in the terminal, it executes as expected and opens up Sublime:
open -a Sublime\ Text\ 2

Finally, when I place the following in my .bashrc script it doesn't throw an error when the shell is initialized...
alias sublime='open\ -a\ Sublime\ Text\ 2'

...but it still throws an error when I run sublime in the shell:
'open: command not found

Why is this different behavior between shell, and bash script?  Thanks.
Comment: Did you copy and paste into the question, or retype it? I suspect that the ' symbol in your .bashrc script is not really a ', but rather a typographical mark ' which results from your editing your .bashrc file with a word processor.
Comment: what shell are you using? echo $SHELL
Comment: no copy paste - I wrote it.
Comment: That's odd. Try a function: `unalias sublime; sublime() { open -a "Sublime Text 2" "$@"; }`
Answer: You need to properly quote the command. Try this:
alias sublime='/usr/bin/open -a "Sublime Text 2"'

Comment: alias sublime='open -a Sublime\ Text\ 2'
Comment: Neither works.  @Unni, I tried that as shown in the question.  And Matt, while I see your point about the double quotes, it still doesn't like it:  `-a: not found`, and `"Sublime Text 2" not found` are the messages.
Comment: It sounds like `open` in this context is somehow referring to something else. Try using `/usr/bin/open` explicitly. I've updated my answer.
Comment: That works, although I don't understand why.  Perhaps there are two different `open` commands?


Using functional Api in tensorflow throws KeyError: 'input_1'

Question: I'm trying to build a simple linear model using TensorFlow functional API.
def create_model():
    input1 = tf.keras.Input(shape=(30,))
    hidden1 = tf.keras.layers.Dense(units = 12, activation='relu')(input1)
    hidden2 = tf.keras.layers.Dense(units = 6, activation='relu')(hidden1)
    output1 = tf.keras.layers.Dense(units = 2, activation='softmax')(hidden2)
    model = tf.keras.models.Model(inputs = input1, outputs = output1)
    model.compile(optimizer='adam',
                  loss='sparse_categorical_crossentropy',
                  metrics=['accuracy'])
    return model

This is my code to create the model.
I'm using a data pipeline to create the input dataset like this.
def make_dataset(dataframe, shuffle=True, batch_size=32):
    labels = dataframe.pop('target')
    ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), labels))
    if shuffle:
        ds = ds.shuffle(buffer_size=100000, seed = 121 ).repeat()
    return ds
pos_ds = make_dataset(train_data_pos)
neg_ds = make_dataset(train_data_neg)
train_ds = tf.data.experimental.sample_from_datasets([pos_ds, neg_ds], weights=[0.5, 0.5], seed = 45)
train_ds = train_ds.batch(BATCH_SIZE)
steps_per_epoch = np.ceil(2.0*count_neg/BATCH_SIZE)

Here the train_data_pos and train_data_neg are data frame containing positive and negative classes
history = model.fit(train_ds,
          validation_data=val_ds,
          epochs=100,
          verbose = 1,
          steps_per_epoch=steps_per_epoch)

This is my model.fit() cmd.
My error log is as follows:
Traceback (most recent call last):
  File "6.py", line 159, in <module>
    steps_per_epoch=steps_per_epoch)
  File "C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\keras\engine\training.py", line 66, in _method_wrapper
    return method(self, *args, **kwargs)
  File "C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\keras\engine\training.py", line 848, in fit
    tmp_logs = train_function(iterator)
  File "C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\eager\def_function.py", line 580, in __call__
    result = self._call(*args, **kwds)
  File "C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\eager\def_function.py", line 627, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\eager\def_function.py", line 506, in _initialize
    *args, **kwds))
  File "C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\eager\function.py", line 2446, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\eager\function.py", line 2777, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\eager\function.py", line 2667, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\framework\func_graph.py", line 981, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\eager\def_function.py", line 441, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\framework\func_graph.py", line 968, in wrapper
    raise e.ag_error_metadata.to_exception(e)
tensorflow.python.autograph.pyct.error_utils.KeyError: in user code:

    C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\keras\engine\training.py:571 train_function  *
        outputs = self.distribute_strategy.run(
    C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:951 run  **
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2290 call_for_each_replica
        return self._call_for_each_replica(fn, args, kwargs)
    C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2649 _call_for_each_replica
        return fn(*args, **kwargs)
    C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\keras\engine\training.py:531 train_step  **
        y_pred = self(x, training=True)
    C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\keras\engine\base_layer.py:927 __call__
        outputs = call_fn(cast_inputs, *args, **kwargs)
    C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\keras\engine\network.py:719 call
        convert_kwargs_to_constants=base_layer_utils.call_context().saving)
    C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\keras\engine\network.py:826 _run_internal_graph
        inputs = self._flatten_to_reference_inputs(inputs)
    C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\keras\engine\network.py:926 _flatten_to_reference_inputs
        return [tensors[inp._keras_history.layer.name] for inp in ref_inputs]
    C:\Users\Aniket\Documents\Aniket\learning-ML\ML_env\lib\site-packages\tensorflow\python\keras\engine\network.py:926 <listcomp>
        return [tensors[inp._keras_history.layer.name] for inp in ref_inputs]

    KeyError: 'input_1'

All of this works when I use sequential API to construct the model.
def create_model():
    model = tf.keras.Sequential([
    feature_layer,
    tf.keras.layers.Dense(units = 12, activation='relu', use_bias = True, kernel_initializer= 'glorot_uniform', bias_initializer = 'glorot_uniform', name = 'd1'),
    tf.keras.layers.Dense(units = 6, activation='relu', use_bias = True, kernel_initializer= 'glorot_uniform', bias_initializer = 'glorot_uniform', name = 'd2'),
    tf.keras.layers.Dense(units = 2, activation='softmax', name = 'out')
    ])
    model.compile(optimizer='adam',
                  loss='sparse_categorical_crossentropy',
                  metrics=['accuracy'])
    return model

Here feature_layer is tf.keras.layers.DenseFeatures
Here is the link to the entire code - LINK
Comment: It seems like there is a problem with the shape of your input data. What does `train_data_pos ` look like?
Comment: It is basically positive rows of [this](https://github.com/aniketbote/learning-ML/blob/master/Class/mytrain2.csv). My entire code is [here](https://github.com/aniketbote/learning-ML/blob/master/Class/6.py)
Answer: Since you have feature columns that are not directly compatible with tf.keras.Input, you have to use a workaround that assigns tf.keras.Input to each original feature column. The workaround via this Github issue is:
from __future__ import absolute_import, division, print_function

import numpy as np
import pandas as pd

import tensorflow as tf

from tensorflow import feature_column
from tensorflow import keras
from tensorflow.keras import layers
from sklearn.model_selection import train_test_split

URL = 'https://storage.googleapis.com/applied-dl/heart.csv'
dataframe = pd.read_csv(URL)
dataframe.head()

train, test = train_test_split(dataframe, test_size=0.2)
train, val = train_test_split(train, test_size=0.2)
print(len(train), 'train examples')
print(len(val), 'validation examples')
print(len(test), 'test examples')

# A utility method to create a tf.data dataset from a Pandas Dataframe
def df_to_dataset(dataframe, shuffle=True, batch_size=32):
  dataframe = dataframe.copy()
  labels = dataframe.pop('target')
  ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), labels))
  if shuffle:
    ds = ds.shuffle(buffer_size=len(dataframe))
  ds = ds.batch(batch_size)
  return ds

batch_size = 5 # A small batch sized is used for demonstration purposes
train_ds = df_to_dataset(train, batch_size=batch_size)
val_ds = df_to_dataset(val, shuffle=False, batch_size=batch_size)
test_ds = df_to_dataset(test, shuffle=False, batch_size=batch_size)

def create_inputs():
    age = feature_column.numeric_column("age")

    feature_columns = []
    feature_layer_inputs = {}

    # numeric cols
    for header in ['age', 'trestbps', 'chol', 'thalach', 'oldpeak', 'slope', 'ca']:
        feature_columns.append(feature_column.numeric_column(header))
        feature_layer_inputs[header] = tf.keras.Input(shape=(1,), name=header)

    # bucketized cols
    age_buckets = feature_column.bucketized_column(age, boundaries=[18, 25, 30, 35, 40, 45, 50, 55, 60, 65])
    feature_columns.append(age_buckets)

    # indicator cols
    thal = feature_column.categorical_column_with_vocabulary_list(
      'thal', ['fixed', 'normal', 'reversible'])
    thal_one_hot = feature_column.indicator_column(thal)
    feature_columns.append(thal_one_hot)
    feature_layer_inputs['thal'] = tf.keras.Input(shape=(1,), name='thal', dtype=tf.string)

    # embedding cols
    thal_embedding = feature_column.embedding_column(thal, dimension=8)
    feature_columns.append(thal_embedding)

    # crossed cols
    crossed_feature = feature_column.crossed_column([age_buckets, thal], hash_bucket_size=1000)
    crossed_feature = feature_column.indicator_column(crossed_feature)
    feature_columns.append(crossed_feature)

    return feature_columns, feature_layer_inputs

def create_model():
    feature_columns, feature_layer_inputs = create_inputs()
    dense_features = tf.keras.layers.DenseFeatures(feature_columns)(feature_layer_inputs)
    hidden1 = tf.keras.layers.Dense(units = 12, activation='relu')(dense_features)
    hidden2 = tf.keras.layers.Dense(units = 6, activation='relu')(hidden1)
    output1 = tf.keras.layers.Dense(units = 2, activation='softmax')(hidden2)
    model = tf.keras.models.Model(
        inputs = [v for v in feature_layer_inputs.values()],
        outputs = output1)
    model.compile(optimizer='adam',
        loss='sparse_categorical_crossentropy', metrics=['accuracy'])
    return model

model = create_model()

model.fit(train_ds)

Comment: I did the necessary changes. I'm getting this error. **tensorflow.python.framework.errors_impl.InternalError:  Blas GEMM launch failed : a.shape=(32, 27), b.shape=(27, 12), m=32, n=12, k=27
         [[node model/dense/MatMul (defined at 6.py:147) ]] [Op:__inference_train_function_1421]**. I have the update code [here](https://github.com/aniketbote/learning-ML/blob/master/Class/6.py)
Comment: Your updated code runs without issues for me. Do you have any other programs active?
Comment: I restarted my computer and it started to work. Thank you so much for your time.


Why is tomcat using more and more heap space during idling?

Question: I start tomcat using startup.bat and do nothing more with it. It is a plain vanilla installation (Version 7.0.47), as far as I know. When connecting with a JMX tool, like VisualVM I see the usage of Heap space constantly increasing. Until some garbage collection happens. And then it starts all over, again.
Why is that so? What is tomcat doing there?
Is my understanding right, that it constantly generates new Objects?

As suggested by @Thilo I took snapshots of the heap histogram. It seems to create mostly Arrays of Objects, chars and bytes. The picture below shows a diff between two snapshots taken 3 minutes apart.

As the "Per thread allocations" view indicates, it has something to do with RMI, because here the most increase can be seen.
Comment: Could this be the JMX probes itself (kind of a Heisenberg effect then)?
Comment: The easy non-answer is of course: because it continuously keep creating short-lived objects. Tomcat is never idle, even when it is not receiving requests. You'll likely see the same saw-tooth model in most Java-based servers when not requesting to it.
Comment: You could take a jmap heap histogram (that includes non-live objects, don't trigger a GC) a few times and compare what kind of object instances increase in numbers.
Comment: @Thilo could you expand on the jmap heap histogram. When doing a heap dump in VisualVM it triggers the GC. How do I take it without triggering GC? How could I rule out the "Heisenberg effect"?
Comment: The `jmap` command line has a command line option ("live") that forces a full GC. I do not know if it completely suppresses GC when you don't have that option. In the heap dump, you could maybe identify which objects are created by JMX.
Comment: Same question: http://stackoverflow.com/questions/17321609/why-does-heap-space-constantly-increase-when-monitoring-tomcat-with-java-visualv?rq=1
Comment: Which tomcat version are you talking about?
Answer: The short answer: Don't worry about it, this is perfectly normal. :-)
The long* answer, as @Gimby suggests, is that Tomcat (like most servers) has multiple threads running all the time, to do monitoring of resources and things like health check etc. These threads wake up at intervals and creates a lot of short-lived objects. The GC (garbage collector) will not collect these objects right away. A profiling application will see that the heap continually increases for a regular interval, or until a threshold value is reached and then suddenly drops. The drop is caused by the GC kicking in, collecting all the short lived objects in one sweep. This will cause the "saw tooth" pattern you see. Again, this is perfectly normal.
If you're interested, read more about Java garbage collection at Oracles Java tutorial pages.
The only thing to watch out for, is when the peaks of the saw tooth pattern becomes increasingly higher, or the duration between each peak becomes shorter. This might indicate a resource/reference leak in the application. In most cases though, it's just your application using more memory, because it actually does things.

*) I'm sure someone can provide a much longer and more in-depth answer than this, but that's better suited for a blog or technical article. ;-)
Comment: This a good start. And I probably will post a follow up question on how to measure certain things. But what I might like to see in an answer to this question, is more detail on the RMI thread. Why exactly these threads produce so much objects. Can it be disabled. What exactly is its purpose...


Considering two regions and a holomorphic function, show the following

Question:
I'm very lost on how to do this question. Do I use cauchy-reimann equations somehow? Thank you for your help!
Comment: Immediate by looking at the Kähler Laplacians.
Answer: We use $\triangle f=4\dfrac{\partial^2f}{\partial z\partial\bar{z}}$.
$$\begin{eqnarray}
\triangle (\psi\circ f)
&=& 4\dfrac{\partial^2}{\partial z\partial\bar{z}}(\psi\circ f)\\
&=& 4\dfrac{\partial}{\partial z}\left(\dfrac{\partial}{\partial\bar{z}}(\psi\circ f)\right)\\
&=& 4\dfrac{\partial}{\partial z}\left(
\dfrac{\partial}{\partial f}(\psi\circ f)\dfrac{\partial f}{\partial\bar{z}}+\dfrac{\partial}{\partial\bar{f}}(\psi\circ f)\dfrac{\partial\bar{f}}{\partial\bar{z}}
\right)\\
&=&4\dfrac{\partial}{\partial z}\left(
\dfrac{\partial}{\partial\bar{f}}(\psi\circ f)\dfrac{\partial\bar{f}}{\partial\bar{z}}
\right)\\
&=&4\dfrac{\partial^2}{\partial z \partial\bar f}
(\psi\circ f)\dfrac{\partial\bar{f}}{\partial\bar{z}} + 4\dfrac{\partial}{\partial\bar f}
(\psi\circ f)\dfrac{\partial\bar{f}}{\partial\bar{z}\partial z}
\\
&=&4\dfrac{\partial}{ \partial\bar f}\left( \dfrac{\partial}{ \partial f}(\psi\circ f)\dfrac{\partial\bar{f}}{\partial\bar{z}} \dfrac{\partial{f}}{\partial{z}} + \dfrac{\partial}{ \partial \bar f}(\psi\circ f)\dfrac{\partial\bar{f}}{\partial\bar{z}} \dfrac{\partial\bar{f}}{\partial{z}}\right)
 + 4\dfrac{\partial}{\partial\bar f}
(\psi\circ f)\overline{\left(\dfrac{\partial{f}}{\partial\bar{z}\partial z}\right)}
\\
&=&4\dfrac{\partial}{ \partial\bar f}\left( \dfrac{\partial}{ \partial f}(\psi\circ f)\dfrac{\partial\bar{f}}{\partial\bar{z}} \dfrac{\partial{f}}{\partial{z}}\right)
\\
&=&4\dfrac{\partial}{ \partial\bar f }\left( \dfrac{\partial}{ \partial f}(\psi\circ f)|f'|^2\right)
\\
&=&4\dfrac{\partial}{ \partial\bar f }\left( \dfrac{\partial}{ \partial f}(\psi\circ f)\right)|f'|^2
\\
&=&4\dfrac{\partial^2}{ \partial\bar f \partial f }(\psi\circ f))|f'|^2
\\
&=& \triangle \psi .|f'|^2
\end{eqnarray}
$$
Lines 4 and 7 we use the fact that $\dfrac{\partial f}{ \partial \bar z}=0$ because $f$ holomorphic
Comment: Where did delta(f) = 4*double derivative came from? thanks!
Comment: express $\dfrac{\partial f}{\partial z}$ in terme of partial dérivative of f according to
 x + i  partial dérivative of f according to
 y and do the same for $\dfrac{\partial f}{\partial \bar z}$


Value of $2^{2010} \frac{\int_0^1 x^{1004}(1-x)^{1004} dx}{\int_0^1 x^{1004}(1-x^{2010})^{1004} dx}$

Question: Find the value of the following Integral
$$2^{2010} \frac{\int_0^{1} x^{1004}(1-x)^{1004} dx}{\int_0^{1} x^{1004}(1-x^{2010})^{1004} dx}$$
Because it is in power of 1004 i am not able to find the correct substitution.
Comment: For the denominator, $x^{1005}=u$ and for the numerator $x=\sin t$
Answer: Hint
For the numerator, consider
$$I_n=\int x^n(1-x)^n\,dx$$ Let $x=\sin^2(t)$, $dx=2\sin(t)\cos(t)$ to make
$$I_n=\frac 1 {2^{2n}}\int \sin^{2n+1}(2t)\,dt$$ Have a look here.
Comment: This can be solved more easily by the application of the beta function.
Comment: I agree with you for sure. I did not know if the OP already knew it. Cheers :-)


Get the Venue ID from the Foursquare url

Question: Is it possible to get the Venue ID from an url?
If I have the Venue URL from the Foursquare (example: https://pt.foursquare.com/zahpee ), how do I know what is the venue ID of this location?
Thanks
Answer: I dont know any solution by using the 4sq url. But when you know the url scheme - you can access the venue-id.
zahpee = name of the venue
with that information you can perform a seach:
GET https://api.foursquare.com/v2/venues/search?query=zahpee&intent=global

example response, extract:
response: {
venues: [
{
id: "4fc123f4e4b08acecb574b62"
name: "Zahpee"
contact: {
phone: "+553125164812"
formattedPhone: "+55 31 2516-4812"
twitter: "zahpee"
}
location: {
address: "Avenida Álvares Cabral, 1315"
crossStreet: "Santa Catarina"
lat: -19.931101175274925
lng: -43.94602344328842
cc: "BR"
city: "Belo Horizonte"
state: "MG"
country: "Brazil"
formattedAddress: [
"Avenida Álvares Cabral, 1315 (Santa Catarina)"
"Belo Horizonte, MG"
"Brazil"
]
}

check it easily by using the API explorer
https://developer.foursquare.com/docs/explore
I hope this helps you.


Calculate intercept location and time

Question: I want to calculate the coordinates and time where object B2 can intercept object B1.
Asuming the following:
All coordinates are in pixels
B1: Moves from coordinates A1 to A2, it has a speed of 2 pixels per hour and starts moving at 15:00.
B2: Moves from A3 to B1 and has a max speed of 1,5 pixels per hour and starts moving when it can intercept B1
A1: has the coordinates 20,45 (X,Y)
A2: has the coordinates 60,86 (X,Y)
A3: has the coordinates 55,78 (X,Y)
Results of the calculations I would like to achieve:
The time (00:00 - 24:00) and coordinates (X,Y) B2 is able to intercept B1 while both moving on a straight line, keeping in mind that both objects have a different movement speed.
Comment: What does it mean to move "from A3 to B1" when B1 is constantly moving? Does B2 travel along a straight line, or a pursuit curve, or something else?
Comment: B1 wants to intercept B2, both objects want to travel along a straight line. To futher explain; all the A points are planets, B1 and B2 are spacecraft thus radius etc, can be left out when it comes to calculating when it can intercept it or not. From point to point calculations, center to center if you will.
Comment: The picture helps clarify the problem. If B2 can start moving at any time (including before B1) then you can have it intercept wherever you want. What are the constraints on when B2 can start moving?
Comment: B2 can only start moving when it can intercept B1, that is when B1 going from A1 to A2. There are no other constraints that apply to B2. Keep in mind that the movement speed on B2 can be dynamic, just as B1 can have a different movement speed. Hope this clarifies some parts
Comment: I think you are saying B2 cannot start moving before B1 does. If so, the earliest possible intercept point occurs at $P$ on line $A_1A_2$ such that $A_1P = \frac43 A_3P$. To intercept at any other point between that point $P$ and $A_2$, B2 just has to start later or move slower. If B1 is able to change its speed after B2 has already committed to a line of travel, you cannot be sure when the interception will occur; you cannot even guarantee that B1 will ever reach the interception point unless you specify a minimum speed.
Comment: Getting closer to the calculations iam looking for. Iam going to draw a new scematic that discribes my situation better, I will also add the other calculations ive already done that allow dynamic input. Also, the movement speed is a constant and cannot vary. To explain the use for this calculation: A1,A2,A3 are planets and B1 and B2 are spacecraft. B1 moves from planet A1 to A2; A3 picks up this travel and sends ship B2 to intercept. Both ships have different engines thus a different movement speed.
Answer: B2 can intercept B1 as long as A3-A2 distance <= 75% of (15,0) to A2 passing by A1
Let's call the point where B1 start A0 :
<code>B2 intercept B1 <=> dist(A3,A2) <= 75/100 (dist(A0,A1) + dist(A1,A2))
</code>
(Since 1.5 = 75% of 2)
Comment: I am sorry, maybe you misunderstood my question; The question is not if B2 is capable of intercepting B1, The question is how to calculate the the exact time and coordinates B2 is able to intercept B1.

Calculate intercept location and time


Drupal Lightbox2 fails W3C validation because of rel="lightbox[field_image][]" attribute

Question: I am using the latest Lightbox2 module for Drupal which works great but for my field images it adds a rel="lightbox[field_image][]" which fails W3C validation.
Is there a way to work around this so it passes validation? I have been reading somewhere to change the "rel" to "data-rel", but how would you apply this? Any ideas?
Answer: You can copy the lightbox2-insert-image.tpl.php file into your own theme and change it there - but this will break the functionality of the module as javascript uses rel attribute multiple times.
You could refactor the javascript, change all occurrences where it accesses rel and change it to data-rel and create patch and submit it to drupal.org to the author of the issue as a request (or just submit issue now requesting this and wait if/until the maintainer does it).
On a personal note - why do you need it to pass W3C validation? Is it a project MUST requirement? If not, I would not waste any time on this as valid HTML is not that important any more, not even for SEO and indexing.
Edit: http://google.com is not passing W3C validation, I think that sends a message. (http://validator.w3.org/check?uri=google.com&charset=%28detect+automatically%29&doctype=Inline&group=0)


Common for vocal octaves to be not exact with written sheet music?

Question: I have observed that some songs are sung in lower octaves than what's written on the official sheet music,
I only recall noticing this for male singers who are bass or baritones, not tenors, not that I recall,
for example I think I looked up some sheet music for Frank Sinatra, Michael Buble and etc,
I could be wrong but sometimes their vocals are octave(s) down from what's on the sheet, if the first note on the sheet vocal section is a F4, they would sing F3, and etc
I don't recall observing this kind of transposition with tenors or female singers, like Bruno Mars or Beyonce and etc, what's on the sheet is exactly what they sing
Maybe I'm mis-hearing and completely wrong, but I was wondering if it's even a thing that sometimes the vocals won't exactly match the sheet music (octave wise), and is it common?
(update)
Sorry forgot to provide an example,

If you listen to this song on youtube, for the very first note that Michael sings, "birds", I could be totally wrong but I don't think that is an E4, emphasis on the 4, I don't think it's E in fourth octave, I mean I listened to it and tried on piano keys and I don't think it's E4
Comment: Especially since you allow you might be mis-hearing, it would be helpful to have an example or two of score and performance excerpts that prompted the question.
Comment: Thank you for the input, I have updated my question with example
Answer: I've never cared about this. I think they edit all songs in treble clefs, maybe because pop singers are not all used to read bass or tenor clef.
(Most  pop songs for male voices  are in a higher range than Bariton while the girls mostly sing lower, say in the same range.)

I was wondering if it's even a thing that sometimes the vocals won't exactly match the sheet music (octave wise), and is it common?

Sheet music of pop songs isn't published for singers like classical musicians. It is meant for people who want to have the melody and lyrics and to play it on a keyboard. So most editions are made facilitated and reduced for easier reading and easy playing.
Answer: Your example song, "Feeling Good", was first performed in 1964 by Cy Grant in key of B minor. The score you show is in E minor. Michael Bublé sings it in Eb minor. Nina Simone sung it in G minor. The octave in which the melody is sung seems to be a minor detail here.
It is common for singers to transpose the song to match their vocal range.
As I presume, the score you show was written based on Bublé's performance and transposed half step up to make it easier for less experienced instrumentalists to play.
Probably the melody was moved up by an octave to make it easier to read. As @Albrecht Hügli writes, the violin key is the most commonly recognized by most musicians. I rarely see melodies of popular songs written in different keys.
Similarly to "Feeling Good", many songs sung by Frank Sinatra were or became hit songs. They were performed by multiple artists, in various keys, and the sheet music you will find for them can be in any key.
On the contrary, it seems to me songs performed by Bruno Mars or Beyoncé are mostly either original or written for them. They are contemporary popular performers, and their versions of their songs are the most popular (rather than covers), so the popular scores will likely match the performance in more detail.
Answer: Well, if a song is notated in G clef it is standard that a male sings one octave below, while a female sings the octave that is written.
So maybe that is what this is about?