Why there are too few black goalkeepers in high-level football?

Question: I'm watching a football match, and a black goalkeeper is guarding one of the goals. This seems to be very rare by my observation.
For example, in the 100m dash, all the recent Olympic and World champions have had Afro-Caribbean ancestry.
Why are so few goalkeepers at high level football black? Is there reputable scientific evidence that indicates a physiological difference between the different races which impacts their goalkeeping ability, or is the difference down to cultural factors?
Comment: Probably the same reason there are less white runningbacks in the NFL.   Society's prejudices come out with kids at a very very early age and kids are typecasted into positions on the sports they play.   Sure there are kids that break the mold.   But if you are white playing football and have speed you will quickly be put into a LB, safety, WR, QB position...  And I am sure black kids playing soccer will be put into a position that supposedly uses their "speed" more instead of goalie.    Chance has a much greater effect than anatomy.
Comment: Please guys, remember the story of the [Golden Tooth](http://hoaxes.org/archive/permalink/the_boy_with_the_golden_tooth): you need to observe a event before trying to explain it. Has it ever been observed that there is "*too few black goalkeepers*" in high level football?
Answer: Its just a choice they make when they first start playing. Most of black football players tend to play strikers, midfielders, or defenders. There's definitely a big number of black goalkeepers it's just that they might not be playing in big clubs at the moment (Although there are some that do).
Steve Mandanda - Marseille & France National Team
Carlos Kameni - Fenerbahçe SK (Turkey)
Francis Uzoho - Elche CF
Comment: I'm actually a Fenebahçe fan, so I personally know Kameni. But as I said, there is a huge difference and this cannot be explained with your way; it's my thought. Somehow there must be a physical or mental determinant.
Answer: I think there's also a reverse bias / selection reason. When a certain group is known for certain qualities, either due to fact or stereotypes, the market is more willing to hire people from that group for certain positions, and people from that group in turn are more likely to engage in career path that lead to that certain position.
Answer: Its just like quaterbacks , its a thinking rooted in racism that white people make better decisions.
Comment: @alamoot I don't think the poster is confused - they're making a comparison to American football.
Comment: That said, this answer would be better with more than a sentence; can you show this is true?
Comment: Comparing two completely different sports with different roots makes no sense. And goalkeeping has very little (nothing) in common with a QB's position.
Comment: It is notable that 1) both quarterbacks and goalkeepers have historically had a significantly lower proportion of non-white players, and 2) both quarterbacks and goalkeepers have been regarded as positions that require "brains" rather than just "brawn". I don't think it's a huge leap from those facts to the conclusion Star is drawing, although I entirely agree that this would be a much stronger answer if supported with evidence rather than just the raw assertion.
Comment: The question is asking about football aka soccer, not American Football!


Mapper.Map EnderecoViewModel.CidadeViewModel Endereco.Cidade

Question: Tenho a class Endereco
public class Endereco
{
 public virtual Cidade Cidade { get; set; }
 public int CidadeId { get; set; }
}

E tenho a ViewModel Dessa Class:
  public class EnderecoViewModel
{
public virtual CidadeViewModel CidadeViewModel { get; set; }
        public int CidadeId { get; set; }
}

Minha classe Endereço, tem uma Cidade.
Na minha controller, tenho uma Query assim:
 var end = Mapper.Map<IEnumerable<Endereco>, IEnumerable<EnderecoViewModel>>(_EnderecoServico.BuscarPorIdDoCliente(id));

Essa consulta me trás os endereços Do Cliente.
O problema está na Endereco.Cidade, que está vindo Null.
o Endereco.CidadeId está vindo corretamente.como eu faço esse mapeamento usando o Mapper? se eu estanciar a entidade, e atribuir seus valores: entidadeViewModel.CidadeViewModel.Nome = entidade.Cidade.Nome, funciona, mas como eu faço isso com o Mapper?
Essa é a consulta na Camada Data.Repositorio:
public class EnderecoRepositorio : RepositorioBase<Endereco>, IEnderecoRepositorio
    {
        public IEnumerable<Endereco> BuscarPorIdDoCliente(int? id)
        {
            return db.EnderecoDb.Where(e => e.ClienteId == id);
        }
    }

Class Configuração Mapper:
public class DomainToViewModelMappingProfile: Profile
    {
        public override string ProfileName
        {
            get { return "ViewModelToDomainMappingProfile"; }
        }

        protected override void Configure()
        {
            Mapper.CreateMap<ClienteViewModel, Cliente>();
            Mapper.CreateMap<CidadeViewModel, Cidade>();
            Mapper.CreateMap<ComplementoPessoaFisicaViewModel, ComplementoPessoaFisica>();
            Mapper.CreateMap<ContadorViewModel, Contador>();
            Mapper.CreateMap<ContatoViewModel, Contato>();
            Mapper.CreateMap<ControleDeCobrancaViewModel, ControleDeCobranca>();
            Mapper.CreateMap<ControleDeVencimentoViewModel, ControleDeVencimento>();
            Mapper.CreateMap<ConveniadosViewModel, Conveniados>();
            Mapper.CreateMap<EnderecoViewModel, Endereco>();
            Mapper.CreateMap<ParceiroComercialViewModel, ParceiroComercial>();
            Mapper.CreateMap<RamoDeAtividadeViewModel, RamoDeAtividade>();
            Mapper.CreateMap<RegistroDeBloqueiosViewModel, RegistroDeBloqueios>();
            Mapper.CreateMap<TipoDeSistemasViewModel, TipoDeSistemas>();
            Mapper.CreateMap<UsuarioViewModel, Usuario>();
        }
    }

Proxima
public class ViewModelToDomainMappingProfile : Profile
    {
        public override string ProfileName
        {
            get { return "DomainToViewModelMappingProfile"; }
        }

        protected override void Configure()
        {
            Mapper.CreateMap<Cliente, ClienteViewModel>();
            Mapper.CreateMap<Cidade, CidadeViewModel>();
            Mapper.CreateMap<ComplementoPessoaFisica, ComplementoPessoaFisicaViewModel>();
            Mapper.CreateMap<Contador, ContadorViewModel>();
            Mapper.CreateMap<Contato, ContatoViewModel>();
            Mapper.CreateMap<ControleDeCobranca, ControleDeCobrancaViewModel>();
            Mapper.CreateMap<ControleDeVencimento, ControleDeVencimentoViewModel>();
            Mapper.CreateMap<Conveniados, ConveniadosViewModel>();
            Mapper.CreateMap<Endereco, EnderecoViewModel>();
            Mapper.CreateMap<ParceiroComercial, ParceiroComercialViewModel>();
            Mapper.CreateMap<RamoDeAtividade, RamoDeAtividadeViewModel>();
            Mapper.CreateMap<RegistroDeBloqueios, RegistroDeBloqueiosViewModel>();
            Mapper.CreateMap<TipoDeSistemas, TipoDeSistemasViewModel>();
            Mapper.CreateMap<Usuario, UsuarioViewModel>();
        }
    }

Eu depurei o cód, e percebi, que na consulta, na camada repositório, ele esta trazendo a CIDADE, Só não está transformando esse Endereco.Cidade em EnderecoVireModel.CidadeViewModel... há algum mapeando que deva ser feito??
Comment: Você mapeou as classes Cidade e CidadeViewModel? `CreateMap().ReverseMap();`
Comment: Só fiz o mapeamento `var end = Mapper.Map, IEnumerable>(_EnderecoServico.BuscarPorIdDoCliente(id));` como que eu faço esse `CreateMap().ReverseMap();`, é na controller msm??
Comment: Você cria uma classe de configuração para registrar o AutoMapper e registra essa classe global. Seu projeto é ASP.NET MVC Core? Vou postar as duas maneiras de registar esta classe de configuração do AutoMapper.
Comment: Eu já tinha feito isso.. mas graças a vc, eu fui olhar novamente, e acho que encontrei o erro.... na minha class `DomainToViewModelMappingProfile`, o `ProfileName` ta assim: `get { return "ViewModelToDomainMappingProfile"; }` e na `ViewModelToDomainMappingProfile` ta do mesmo jeito --- > `get { return "ViewModelToDomainMappingToProfile"; }`
Comment: Na, vdd, continua do msm jeito, o `EnderecoViewModel.CidadeViewModel` continua vindo null
Comment: Você pode colocar a classe de configuração na pergunta? Está faltando alguma coisa...A classe Cidade tem propriedades iguais as da viewModel? Manda essas classes também, por favor.
Comment: Coloquei ai as class na pergunta
Answer: O AutoMapper não está conseguindo converter a Cidade por que na classe EnderecoViewModel a propriedade está como CidadeViewModel ao invés de Cidade.
  public class EnderecoViewModel
{
    public virtual CidadeViewModel Cidade { get; set; }
    public int CidadeId { get; set; }

}

 public class Endereco
{
    public int Id { get; set; }
    public virtual Cidade Cidade { get; set; }
    public int CidadeId { get; set; }
}

Comment: Por nada @RafaelPassos Depois que eu copiei as classes aqui, que fui ver que era o nome da propriedade. Valeu!
Comment: Perfeito, era isso mesmo... é a primeira vez que trabalho com AutoMapper.. obrigado pela força!


Expectation of a combination of random variables.

Question: Assume that $\xi_i, i=1, \dots, n$ are independent identically distributed  discrete random variables. What is an expectation of
$$\frac{\sum_{i=1}^{n} a_i \xi_i^2}{\left( \sum_{i=1}^n \xi_i \right)^2}$$
Note:
It is possible to show that $E\frac{\sum_i a_i \xi_i}{\sum_i \xi_i} = \frac{\sum_i a_i}{n}$.
Comment: @DilipSarwate Simply by symmetry (and no need to specify the distribution, only to assume that $\xi_i\gt0$ almost surely, to ensure the denominator is almost surely nonzero).
Comment: It would be helpful if you describe, or give a reference to, how the result in your last sentence was obtained.  Are the $\xi_i$ i.i.d. _Bernoulli_ random variables (and _not arbitrary_ discrete random variables) by any chance?
Answer: The expectation of $\eta_n^{-1}\sum\limits_{i=1}^na_i\xi_i$ is $\theta_1\sum\limits_{i=1}^na_i$, where $\theta_1=E[\eta_n^{-1}\xi_1]$ and $\eta_n=\sum\limits_{i=1}^n\xi_i$. Note that, by symmetry, $n\theta_1=E[\eta_n^{-1}\xi_1]+E[\eta_n^{-1}\xi_2]+\cdots+E[\eta_n^{-1}\xi_n]=1$ hence $\theta_1=n^{-1}$, irrespectively of the distribution of $\xi_1$. This yields the result recalled in the question.
Likewise, the answer here is $\theta_2\sum\limits_{i=1}^na_i$, where $\theta_2=E[\eta_n^{-2}\xi_1^2]$.
Note that $n^2\theta_2\geqslant1$ by Cauchy-Schwarz inequality and $n\theta_2\lt1$ for every $n\geqslant2$ since $n\theta_2+n(n-1)\varphi=1$, where $\varphi=E[\eta_n^{-2}\xi_1\xi_2]$. However, there is no reason to expect that $\theta_2$ is independent of the distribution of $\xi_1$, nor that it has a simple form in general.


What is the relationship between the Boolean Prime Ideal Theorem and the Countable Axiom of Choice?

Question: The base theory is $\mathsf{ZF}$.
The Boolean Prime Ideal Theorem ($\mathsf{BPI}$) states

Every Boolean algebra contains a prime ideal.

The Countable Axiom of Choice ($\mathsf{AC}^{\omega}$) states

Every countable collection of non-empty sets has a choice function.

The Infinite Ramsey Theorem ($\mathsf{RT}$) states

Let $A$ be an infinite set. For any $r,p\in\mathbb{N}$, and for any function $c:[A]^{p}\rightarrow [r]$, there exists an infinite subset $B\subseteq A$ such that $c|_{[B]^{p}}$ is constant.

Where $[X]^{p}$ is the set of $p$-subsets of $X$ and $[r]=\{1,\ldots,r\}$.
In Ramsey's theorem in the hierarchy of choice principles, by Andreas Blass, it is stated that

Neither $\mathsf{BPI}\not\Rightarrow\mathsf{RT}$ nor $\mathsf{RT}\not\Rightarrow\mathsf{BPI}$. Although, $\mathsf{BPI}\Rightarrow\mathsf{AC}_{\mathrm{fin}}$ (the axiom of choice for families of finite sets) and this implication is not reversible.
$\mathsf{AC}^{\omega}\Rightarrow \mathsf{RT}$, and this implication is not reversible. Although, $\mathsf{RT}\Rightarrow\mathsf{AC}^{\omega}_{\mathrm{fin}}$ (the countable axiom of choice for non-empty finite sets, which is $\mathsf{ZF}$-equivalent to Kőnig's Infinity Lemma) and this implication is not reversible.

Hence, $\mathsf{BPI}\not\Rightarrow\mathsf{AC}^{\omega}$, for otherwise $\mathsf{BPI}$ would imply $\mathrm{RT}$. Also, $\mathrm{AC}^{\omega}\not\Rightarrow\mathsf{BPI}$, for otherwise $\mathsf{AC}^{\omega}$ would imply $\mathsf{AC}_{\mathrm{fin}}$.
It appears then $\mathsf{BPI}$ and $\mathsf{AC}^{\omega}$ have no logical relation (in the sense described no one is weaker/stronger than the other). Is this correct? Are they related in any other way?
The reason why I wanted to find out how they relate to each other is that they both potentially involve the uncountable. On the one hand, for example, $\mathsf{BPI}$ is $\mathsf{ZF}$-equivalent to full propositional (or first-order) compactness, which may involve an uncountable set of formulas. On the other, the sets involved in the statement of $\mathsf{AC}^{\omega}$ may be uncountable. I guess, in a sense, this is a relationship between them.

Edit: This answer has related, and interesting results.
Answer: Yes. That is correct.
In the Cohen model the Boolean Prime Ideal theorem holds, but there is a Dedekind finite set, which is a contradiction to countable choice.
On the other hand, $L(\Bbb R)$ of the Cohen model satisfy Dependent Choice, which is stronger than countable choice, and there are no free ultrafilters on $\omega$, so the Boolean Prime Ideal theorem fails.
Comment: I see, right Ramsey's Theorem is false in the Cohen model. Then Dependent Choice and the Boolean Prime Ideal Theorem have no logical relationship either. A follow-up question: the statement that *every Dedekind finite set is finite* follows from countable of choice, does it not? Do you know if it implies implies Ramsey's Theorem (in ZF)?
Comment: I'm not sure that RT as you expressed it is false in the Cohen model, as it may very well follow from compactness. To the other things, DC and BPI are indeed independent, as I explained in my answer. Countable choice is enough to show that every infinite set is Dedekind-infinite, yes. To the final question, I'm not sure. Ramsey stuff was never my strong suit.
Comment: Thank you. It is from Blass' paper, which in turn references Jech's *The Axiom of Choice*. I checked and it is on p.164 problem 20: "The following version of Ramsey's Theorem is not provable without the Axiom of Choice: If $A$ is an infnite set and $[A]^2=W_1 \cup W_2$ is a partition of $[A]^2=\{\{a,b\}:a,b\in A, a\neq b\}$ then there is an infnite homogeneous subset $H\subseteq A$. [Hint: The second Fraenkel (or Cohen) model. Let $A=\cup_{n=0}^{\infty}P_{n}$, and let $\{a, b\}\in W_1$ just in case $a, b$ are in the same pair $P_{n}$. A homogeneous set would be an infinite choice function.]"
Comment: I do not think RT would follow from compactness alone without more choice, since BPI does not imply RT. Although, I am aware there is some way to use ultrafilters (hence compactness) to prove RT. But it uses more choice if I am not mistaken. Sorry, I don't know it from the top of my head. But after using a non-principal ultrafilter we have to pick an element somewhere.
Comment: But you asked about Cohen's model, and Cohen's model is not Fraenkel's second model.
Comment: My apologies, I rushed and misunderstood what Blass meant. The reference from Jech is correctly sated but has no bearing on what is proven in Blass' paper. There it is stated (p.288 - Theorem 1) "Ramsey's Theorem is false in the basic Cohen model" (Theorem 2 asserts that Ramsey's Theorem is true in the basic Fraenkel model). I hope that clears things up.


Colocar uma combobox numa coluna de um datagridview

Question: Usando o C#, com EntityFramework e codefirst, tenho as seguintes classes na minha camada de models:
public class Cliente
{
    public int ClienteId { get; set; }
    public string Nome { get; set; }
    public string Email { get; set; }
    public DateTime DataCadastro { get; set; }
    public bool Ativo { get; set; }

    public virtual ICollection<Endereco> Enderecos { get; set; }
    public virtual ICollection<Telefone> Telefones { get; set; }
}

    public class Endereco
    {
        public int EnderecoId { get; set; }
        public string Cep { get; set; }
        public string Rua { get; set; }
        public string Numero { get; set; }
        public string Complemento { get; set; }
        public string Bairro { get; set; }
        public string Cidade { get; set; }
        public string Estado { get; set; }
        public decimal TaxaDeEntrega { get; set; }

        public int ClienteId { get; set; }
        public virtual  Cliente Cliente { get; set; }
    }

    public class Telefone
    {
        public int TelefoneId { get; set; }
        public string Numero { get; set; }
        public int ClienteID { get; set; }

        public virtual Cliente Cliente { get; set; }
    }

Na minha camada de Forms, eu tenho um form com um datagridview com as colunas de nome do cliente, email, data de cadastro e uma combobox com as listas de telefones de cada um. Essas configurações estão montadas da seguinte forma:
private void FormClientes_Load(object sender, EventArgs e)
        {
            //Não exibe os ícones de Maximixação e minimização do form dentro de uma MDI Parent
            this.FormBorderStyle = FormBorderStyle.None;
            this.ControlBox = false;
            this.MaximizeBox = false;
            this.MinimizeBox = false;
            this.ShowIcon = false;
            this.Dock = DockStyle.Fill;

            dataGridViewClientes.AutoGenerateColumns = false;
            dataGridViewClientes.RowHeadersVisible = false;//Omite a primeira coluna

            DataGridViewTextBoxColumn colNome = new DataGridViewTextBoxColumn();
            colNome.HeaderText = "NOME";
            colNome.DataPropertyName = "Nome";
            dataGridViewClientes.Columns.Add(colNome);

            DataGridViewTextBoxColumn colEmail = new DataGridViewTextBoxColumn();
            colEmail.HeaderText = "EMAIL";
            colEmail.DataPropertyName = "Email";
            dataGridViewClientes.Columns.Add(colEmail);

            DataGridViewTextBoxColumn colDataCadastro = new DataGridViewTextBoxColumn();
            colDataCadastro.HeaderText = "DATA CADASTRO";
            colDataCadastro.DataPropertyName = "DataCadastro";
            dataGridViewClientes.Columns.Add(colDataCadastro);

            DataGridViewComboBoxColumn colTelefones = new DataGridViewComboBoxColumn();
            colTelefones.HeaderText = "TELEFONES";
            colTelefones.DataPropertyName = "Numero";
            dataGridViewClientes.Columns.Add(colTelefones);

            FillDataGrid();
        }

        private void FillDataGrid()
        {
            foreach (var c in _clienteApp.GetAll())
            {
                DataGridViewComboBoxCell cbc = new DataGridViewComboBoxCell();

                foreach (var t in c.Telefones)
                {
                    cbc.Items.Add(t.Numero);
                }

                dataGridViewClientes.Rows.Add(c.Nome, c.Email, c.DataCadastro, cbc);
            }
        }

Na função FillDataGrid()  eu pego os dados do BD e populo as colunas. O problema está quando eu vou adicionar a combobox de telefones ao final da linha. O programa compila mas ao carregar o form a combo não é montada e eu tenho a seguinte exception:

Se eu retirar o DataGridViewComboBoxCell as linhas são montadas mas a combobox fica sem os telefones. Também já tentei montar uma Combobox e adicionar ela ao final da linha dataGridViewClientes.Rows.Add(c.Nome, c.Email, c.DataCadastro, ... ); mas não obtive sucesso. Onde estou errando ?
Answer: Segui algumas dicas de um colega do Stackoverflow em inglês @JohnG e consegui fazer o que eu queria. How to put a combobox on datagridview programmatically iterating a Collection. Abaixo segue o código que fiz:
private void FillDataGrid()
        {
            ConfiguraPaginacao();

            dgvClientes.Rows.Clear();
            dgvClientes.Columns.Clear();

            var colNome = new DataGridViewTextBoxColumn();
            colNome.HeaderText = "NOME";
            colNome.DataPropertyName = "Nome";
            colNome.SortMode = DataGridViewColumnSortMode.Automatic;

            var colEmail = new DataGridViewTextBoxColumn();
            colEmail.HeaderText = "EMAIL";
            colEmail.DataPropertyName = "Email";
            colEmail.SortMode = DataGridViewColumnSortMode.Automatic;

            var colDataCadastro = new DataGridViewTextBoxColumn();
            colDataCadastro.HeaderText = "DATA CADASTRO";
            colDataCadastro.DataPropertyName = "DataCadastro";
            colDataCadastro.SortMode = DataGridViewColumnSortMode.Automatic;

            var colTelefones = new DataGridViewComboBoxColumn();
            colTelefones.HeaderText = "TELEFONES";
            colTelefones.DataPropertyName = "Numero";
            colTelefones.Name = "Telefones";
            colTelefones.SortMode = DataGridViewColumnSortMode.Automatic;

            dgvClientes.Columns.Add(colNome);
            dgvClientes.Columns.Add(colEmail);
            dgvClientes.Columns.Add(colDataCadastro);
            dgvClientes.Columns.Add(colTelefones);

            foreach (var c in _clienteApp.GetAllToPagination(paginaAtual, itensPorPagina).ToList())
            {
                int index = dgvClientes.Rows.Add(c.Nome, c.Email, c.DataCadastro);

                DataGridViewComboBoxCell cb = (DataGridViewComboBoxCell)dgvClientes.Rows[index].Cells["Telefones"];
                cb.DataSource = c.Telefones.ToList();
                cb.ValueMember = "Numero";
                cb.DisplayMember = "Numero";

                //Exibe o primeiro número de telefone na combobox
                if (c.Telefones.Count > 0)
                    cb.Value = c.Telefones.First().Numero.ToString();
            }

            dgvClientes.Sort(dgvClientes.Columns[2], ListSortDirection.Descending);
        }

Depois estudando melhor, notei que o DataGridViewTextBoxColumn já é adicionado automaticamente ao final da linha como se fosse já um Combobox. O que precisa ser feito depois é preencher a lista que o DataGridViewTextBoxColumn tem. Eu adiciono os campos texto na linha int index = dgvClientes.Rows.Add(c.Nome, c.Email, c.DataCadastro); menos a lista do combobox. Em index, eu pego qual linha estou. Em seguida eu pego o combobox dessa linha: DataGridViewComboBoxCell cb = (DataGridViewComboBoxCell)dgvClientes.Rows[index].Cells["Telefones"]; e preencho já com os objetos da lista.
cb.DataSource = c.Telefones.ToList();
                cb.ValueMember = "Numero";
                cb.DisplayMember = "Numero";

Outra dica legal é que para manter selecionado o primeiro item da DataGridViewComboBoxCell , eu só tenho a propriedade value.
//Exibe o primeiro número de telefone na combobox
if (c.Telefones.Count > 0){
   cb.Value = c.Telefones.First().Numero.ToString();
}


understanding lstm results

Question: I'm following an introduction to LSTM Neural Nets from this repo - https://github.com/jaungiers/LSTM-Neural-Network-for-Time-Series-Prediction
The author has a few ways to plot and test the predictions. The one that confuses me his method in LSTM.py called predict_sequences_multiple. The code reads like this:
<code>def predict_sequences_multiple(model, data, window_size, prediction_len):
    #Predict sequence of 50 steps before shifting prediction run forward by 50 steps
    prediction_seqs = []
    for i in range(int(len(data)/prediction_len)):
        import pdb
        curr_frame = data[i*prediction_len]
        predicted = []
        for j in range(prediction_len):
            predicted.append(model.predict(curr_frame[newaxis,:,:])[0,0])
            curr_frame = curr_frame[1:]
            curr_frame = np.insert(curr_frame, [window_size-1], predicted[-1], axis=0)
        prediction_seqs.append(predicted)
    return prediction_seqs
</code>
So essentially he looking at a sliding window of 50 ticks in the data, predicting next value in the sequence, sliding the window to the right by one and then including the predicted value as the newest item in the sliding window which he then again tests the model on. This way his predictions from the model are being fed into his test data. I'm pretty new to Data Sci and time series predictions but I don't understand why anyone would do this as you'd be feeding any error in your model back into it's self. Is there any specific reason one would analyse the results this way?
Answer: He is using recursive multi-step forecast to predict more than 1 step in the future. Your assumption about feeding the errors back into the model is correct, errors will accumulate and exponentially increase this way. But in case of the code you posted this is done to plot the trend lines and visualize the model predictions.
https://machinelearningmastery.com/multi-step-time-series-forecasting/

understanding lstm results


proving a matrix is symmetric

Question: Let $A$ be an $nxn $ symmetric matrix.
a) Show that $A^2 $ is symmetric.
b) Show that $2A^2 -3A + I$ is symmetric.
for part a), i have:
$A=A^T$
$A^2 = A\times A$
$A^2 = (A^T)\times(A^T)$
$A^2 = (A\times A)^T$
$A^2 = (A^2)^T$
since $A^2 $is transposable, it is symmetric.
part b) i can prove similar as part a).
Any correction?
Comment: For part (a), it is clearer to write $(A^2)^T = (AA)^T = A^TA^T = AA = A^2$.
Comment: why would you want to write it in that order? I'm proving A^2 not (A^2)^T.
Also, how can i write power function like you did? thanks.
Comment: A matrix $M$ is symmetric if $M^T=M$. So to prove that $A^2$ is symmetric, we show that $(A^2)^T = \cdots A^2$. (But I am not saying what you did was wrong.) As for typing A^T, just put dollar signs on the left and the right to get $A^T$.
Comment: It's a vice versa situation here with the $A^T$. Thanks Lee.
Comment: look at this. Prove: If $A^T * A$=$A$, then $A$ is symmetric and $A=A^2$.
i have,

$A=A^T * A = A*A = A^2$.
Answer: In part b) you will need the fact $(A+B)^T=A^T+B^T$ and $(3A)^T=3A^T$and the same idea solves the problem.
Comment: look at this. Prove: If $A^T * A$=$A$, then $A$ is symmetric and $A=A^2$.
i have,

$A=A^T * A = A*A = A^2$.
Comment: @mickey4691 if you have another question, you should make another post

proving a matrix is symmetric


How to latently cluster regressors based on regressors' relationship with the outcome?

Question: What is the best way/method to model patterns across coefficients and reduce number of coefficients based on these patterns?
We have hundreds of regressors on the same scale and try to reduce the number of coefficients with respect to the regressors' relationship with our outcome variable.
To some extent we try to invert the logic behind finite mixture regression: Instead of estimating different coefficients for different subgroups of our observations, we try to built subgroups of regressors which have a similar effect and are interested in the "overall coefficients" of these subgroups of regressors.
Our idea is to estimate both membership to the latent regressors and the final regression parameters for the latent regressors simultaneously and to allow for inference in the end. Our thought was as follows: While a finite mixture regression assumes a mixture distribution of the outcomes and estimates the membership of the observations to the distributions and the regression parameters simultaneously, we rather wanted to transfer this idea to the parameter vector. Instead of assuming a multivariate normal distribution of the regression parameters, our idea was to assume a multivariate mixture distribution at this point, so assign regressors to certain latent regressors and somehow estimate the overall effect for each latent regressor on our outcome.
One easy example would be the following:
Assume we have 500 regressors in total. 250 regressors have a positive coefficient around 100. 250 regressors have a negative coefficient around -100.
Assuming n "latent regressors", our model should (1) identify the regressors which have similar effects on the outcome (in the example of 2 latent regressors, the first 250 regressors should be identified as one latent regressor due to their positive effect, the next 250 regressors should be identified due to their negative effect), (2) should consider the original regressors as n latent regressors in the model and (3) should also give n coefficients and allow for inference for the latent regressors.
Is there any appropriate method? Since we do not come from econometrics in the first place, we unfortunately do not know whether our idea is realistic at all. PLSR that Peter mentioned may indeed be the best way and we have to rethink the method.
Comment: Point of information: in latent class regression, we assume the latent class is a multinomial dependent variable. We use observed independent variables (not necessarily the same as the latent class indicators!) to predict the latent class. I think what you describe is closer to finite mixture modeling. There, you have one $y$. You assume a regression relationship, e.g. $y = X\beta + \epsilon$. You assume that relationship varies over $k$ latent classes, and you ask your software to estimate the $\beta$s in each latent class.
Comment: Thanks for your comment! Of course, you are right! I adjusted my question accordingly.
Answer: First, I recommend using substantive knowledge to do some of this. Think about what all those variables are. Think about how reliable and valid they are. Think about whether they will be easy or hard to collect. Think about which ones will generate missing data. Etc.
Then I recommend partial least squares regression (PLSR).
PLSR is a date reduction method that considers the relationships among the IVs and between the IVs and the DVs. It is similar to principal components, but that (as you note in a comment) does not consider the DV (or outcome variable).
Comment: What is the difference between "predicting outcomes" and "the relationship between X and Y"? I guess you could mean having meaningful X's, but that's going to be a problem with any method of data reduction, I think.
Comment: Thank you very much for your answer! We have also been reading about PLSR, but have rejected this method because it has a strong focus on predicting outcomes. But we are much more interested in describing the relationship between X and Y. I have explained our basic idea in more detail in my question post now. But maybe our idea isn't even feasible and PLSR is actually the best way to do it.
Comment: Everything we read on the topic of PLS focuses on the explained variance or reducing prediction errors. We are not really interested in explaining or predicting y as good as possible but rather in identifying the effect of x on y as causal as possible.


How to render create view when model has reference to another model?

Question: I am trying to learn ASP.NET MVC. Currently I want to render a create page of a model, which has reference to another model, and make user select the other model  from drop down list. To be more specific, here is the code for Category model
public class Category
{
    public int Id { get; set; }
    public string Name { get; set; }
}

And here is the code of Food model
public class Food
{
    public int Id { get; set; }
    public string Name { get; set; }
    public Category Category { get; set; }
}

Consider I have an in memory list of categories called categories and I want to make sure that when I send create view to user with return View(); in HomeController's Create method he gets not only the name to choose but also the drop down of categories. Also consider that in near future I want to generate the tables of database according to these classes (Entity Frameworks code first approach) so adding properties to the classes doesn't seem like a good idea.
How can I do that?
Comment: What have you tried, and what is not working? Suggest you refer the code in [this Q/A](https://stackoverflow.com/questions/34366305/the-viewdata-item-that-has-the-key-xxx-is-of-type-system-int32-but-must-be-o) for example of how to generate a dropdownlist
Answer: You can make use of viewBag to pass more than one model from controller to view like this
Controller
Public ActionResult Index()
{
 viewbag.Childone=Childone();
 viewbag.Childtwo=Childtwo()
 return View(parentModel);
}

[HttpPost]
public ActionResult Index(ParentModel parentModel,Childone child_one ,Childtwo child_two)
 {
  //do something with models passed....
 }

View
 @model parentModel

 @{
    Childone  = viewbag.Childone as Childone;
    Childtwo  = viewbag.Childtwo as Childtwo;
  }

   //just use these models......like
 //this is Main parent model

@Model.propertyname

   //these are child

@Childone.propertyname

@Childtwo.propertyname

On Model Submit Button Click
var parentModel = [
  { id: 1, color: 'yellow' },
 ];
 var child_one = [
    { id: 1, color: 'yellow' },

];

 var child_two = [
    { id: 1, color: 'yellow' },

];

$.ajax({
    contentType: 'application/json; charset=utf-8',
    dataType: 'json',
    type: 'POST',
    url: '/Home/Index',
    data:JSON.stringify(parentModel:parentModel child_one:child_one,child_two:child_two),
    success: function () {

    },
    failure: function (response) {

    }
});

Comment: Isn't there a way to not use javascript and just send both classes to server without it.


DNS/Nameserver issue. Can't ping IP or domain

Question: I get this when I ping an IP:
21:31:50.136623 IP SITE_IP > 184.108.40.206: ICMP echo request, id 14941, seq 1, length 64
21:31:51.136138 IP SITE_IP > 188.8.131.52: ICMP echo request, id 14941, seq 2, length 64
21:31:52.136118 IP SITE_IP > 18.104.22.168: ICMP echo request, id 14941, seq 3, length 64
21:31:53.136129 IP SITE_IP > 220.127.116.11: ICMP echo request, id 14941, seq 4, length 64
21:31:54.136102 IP SITE_IP > 22.214.171.124: ICMP echo request, id 14941, seq 5, length 64
21:31:55.136153 IP SITE_IP > 188.8.131.52: ICMP echo request, id 14941, seq 6, length 64

and when I ping a domain:
21:29:33.631583 IP 22.214.171.124.52085 > SITE_IP.domain: 28952 A? google.com.MY_DOMAIN. (42)
21:29:38.626553 IP SITE_IP.42280 > 18.104.22.168.domain: 52435+ A? google.com.MY_DOMAIN. (42)
21:29:38.652675 IP 126.96.36.199.63658 > SITE_IP.domain: 36178 A? google.com.MY_DOMAIN. (42)
21:29:43.631626 IP SITE_IP.48205 > 188.8.131.52.domain: 52435+ A? google.com.MY_DOMAIN. (42)

The pinging of a domain is what worries me, because it looks like it is checking my DNS files for the resolution.
Here is etc/resolv.conf
nameserver 188.8.131.52
nameserver 22.214.171.124

/etc/hosts
127.0.0.1               localhost localhost.localdomain localhost4 localhost4.localdomain4
::1             localhost localhost.localdomain localhost6 localhost6.localdomain6
SITE_iP            server.MY_DOMAIN.com server

Will also add that I am seeing a number of 'SERVFAIL'..
I have no idea what could be causing this problem.  If there is any other information I need to provide, let me know.  I'm using CentOS.
route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
22.214.171.124     0.0.0.0         255.255.255.0   U     0      0        0 em1
0.0.0.0         188.8.131.52     0.0.0.0         UG    0      0        0 em1

ifconfig
em1       Link encap:Ethernet  HWaddr D0:*
      inet addr:server_ip  Bcast:18.104.22.168  Mask:255.255.255.0
      inet6 addr: inet6_addr Scope:Global
      inet6 addr: inet6_addr Scope:Link
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      RX packets:516053 errors:0 dropped:0 overruns:0 frame:0
      TX packets:557877 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000
      RX bytes:110304822 (105.1 MiB)  TX bytes:520208744 (496.1 MiB)
      Interrupt:16 Memory:c0000000-c0012800

Answer: When you ping a name, ping needs to resolve the host or domain name to an IP address first, thus a name lookup is expected. Check /etc/host.conf to see where the system looks first. order:hosts,bind would check /etc/hosts first, order:bind,hosts would check DNS first. Also, does the domain name you're trying to ping resolve at all?
Comment: If you can't even ping IP addresses, there's a problem with your network configuration. Please update your question with the output of `ifconfig` and `route -n`.
Comment: It resolves when I have nameserver 127.0.0.1 in resolv.conf, otherwise, nope.  The only thing in /etc/host.conf is 'multi on'.
Comment: And can you ping the name when you have `nameserver 127.0.0.1` in `/etc/resolv.conf`? If you can, then there's no address record for the name in public DNS. Adding the line `order:hosts,bind` to your `/etc/host.conf` should fix the issue.
Comment: can't ping anything at all, even IP addresses.. under any circumstance :(
Comment: Can you at least reach your gateway (`ping 22.214.171.124`)? According to your `RX` and `TX` counters, your host seems to be able to send and receive packets just fine.


Methods to combine ( e1071 svm ) models in R to generate a more complete, accurate model

Question:
I am using the function svm from the package e1071 in R to generate a support vector machine model.  I have a very large data set, and for the moment, while in an exploratory mode, want to simply read in small slices of the data that can be modeled on my single machine.

After obtaining the model results, I would like to read in more data and generate another SVM model, etc. until all of the data is modeled, which will generate about 50 different models.  I would then like to merge all of these models / results together in some fashion in order to get a grand model which would approximate what I would achieve if I could fit all the data in at once.

I know there are multiple ways to do this, theoretically.  But, sticking with SVM modeling, what are my options in R?  (I want to stick with the e1071 package because it has some things about it that are not in the others that I saw.)

If it matters, my data is not genomic, and it is highly weighted (meaning that I am modeling T/F, and usually find an F, but the feature set data is not sparse).

Below is a "pseudocode" snapshot of what I'm proposing / hoping, in case that helps clear things up:

require(e1071)
modelSet = NULL
for (i in c(1:50) ) { ## imagine there are 50 files
  dF <- read.csv(paste("file",i,"csv", sep=".") )
  modelSet[[i]] <- svm(myOutput ~ ., data= dF, probability= TRUE)
}
## Now I would like to find a way to merge all 50 of the "modelSet" models together
## to make 1 composite model derived from all 50 data files.

Thanks!
Mike
Comment: what about using stochastic gradient descent to estimate your svm instead? how large is your feature set
Comment: EnsembleSVM may be of interest to you. It is made for exactly such situations. Unfortunately there is no R interface yet. http://www.esat.kuleuven.be/sista/ensemblesvm/
Comment: Hi Sean, Marc.  Thanks for the advice!  Regarding feature set, it is about 85 features, but realistically about 20 that I care about.  Each feature could have 1000+ levels, though.  Marc, I will look through EnsembleSVM.  Might be nice later down the road, but for now, I wanna stick with R to speed up development.  Thanks!
Answer: First, make sure you shuffle your data before you separate it in to the 50 partitions.  If the original data is at all ordered, this can make a huge difference in the performance of your final model.  Also, make sure your 50 partitions are stratified, meaning they have roughly the same number of exemplars per class as the overall data set.  Then go ahead and train your 50 models (how you handle the hyperparameter selection is a bigger problem).  When you want to make a prediction, do one of the following, all variations on boosting:
1) Hard-decision boosting: Evaluate all 50 models and treat the output of each model as a vote.  Whichever class gets the most votes wins.  The problem with this approach is that each model gets an equal vote, regardless of the confidence of its prediction.
2) Soft-decision boosting: Evaluate all 50 models with the parameter decision.values = TRUE.  This tells e1071 to return the decision value in addition to the class label.  Average the decision values returned by each of the 50 models and make a prediction based on the average.  The problem with this approach is that the decision values will not necessarily be scaled equally across all models.  In other words, a decision value of 1.618 will not necessarily mean the same thing across all 50 models.
3) Soft-decision boosting with logistic meta-model: Train and evaluate the models with the parameter probability = TRUE.  This will fit a logistic regression on top of the SVM model that scales the output to be in the range [0, 1].  This is sort of a kludge but it may help with the scaling problem of option 2.
A final word of caution - you say there are only 20 features you care about but each feature can have 1000+ levels.  By your use of the word "levels", I assume these features are discrete.  Are they ordinal or categorical?  In other words, is there an order to the levels?  Is level 1 closer to level 2 than level 3?  If so, you're probably good to go.  If not, you might want to consider re-coding the categorical using one-hot encoding.
Answer: I would offer you to combine predictions, not models. E.g., after you have 50 models, you can apply all of them on the testing data. Then you will get 50 prediction sets. And all this prediction sets you could use as an input to train another classifier/model (could be SVM as well).
This is described in more details in the "Practical Machine Learning" course on Coursera from Johns Hopkins University: https://www.coursera.org/course/predmachlearn.
There is a "combining predictors" video, and slideset for it is available, for example, here: https://rpubs.com/pr9115/JLeek_025_CombiningPredictors
Answer: My solution is, If it is acceptable, better to compose all the observations into a single data set and go for stratified sampling, which might not be possible when data is separately in different files.
Then build SVM on it.


Same menu display only a couple of levels down when used normally but when used within the "Site map" module it would display all the levels?

Question: In order to display all the links using the "Site map" module I am adding everything we want in our site map to the primary links menu, going in as many levels deep as I need. Then I don't actually display all the levels in our menu but the site map will display all the links within the primary links menu.
Is there a way to have the same menu display only say a couple of levels down when used normally but when used within the "Site map" module it would display all the levels?
Comment: do you try with tow menu? i think you can create tow menu ?
Answer: You can install the Menu Block module to help with that.
You can define a define a block for a specific menu, and in the block configuration you can choose the maximum depth (see the active select list in the image below). Then you can add that block to your header/whatever region in place of the menu that's there now.

This won't affect the underlying menu itself though, just the display of it for that particular block. So in the sitemap, all levels of links will show.
Obviously you may need to make a few CSS tweaks to your menu as the wrapping classes will probably have changed, but I think this is by far the quickest method to get what you're looking for.


Setting up SSL in Apache2 with backend docker app

Question: I'm running Ubuntu server (18.04) on an Azure VM, with Apache2 (2.4.29). I have a backend app running in docker on the same host that is called by my site. This works easy enough without SSL. I added my SSL certificate, and a static page loads fine over https (so the certificate is valid). When I add the page that calls my app (listening on host port 8080) I get an error:

GET https://myhost.com:8080/api/fetchService net::ERR_SSL_PROTOCOL_ERROR

Since my app in the container isnt setup for SSL, how do I configure Apache2 to interact with the local container without SSL?
virtual host conf:
<IfModule mod_ssl.c>
 <VirtualHost *:443>
        ServerAdmin firstname.lastname@example.com
        ServerName myhost.com
        DocumentRoot /var/www/myhost.com/public_html

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

    LoadModule ssl_module modules/mod_ssl.so

    SSLProxyEngine on

    ProxyPreserveHost On

    ProxyPass *:8080/ http://myhost.com:8080/

    ProxyPassReverse *:8080/ http://myhost.com:8080/

    Header set Access-Control-Allow-Origin "*"

    SSLCertificateFile /etc/letsencrypt/live/myhost.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/myhost.com/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf
 </VirtualHost>
</IfModule>

Comment: You already did. But you are bypassing Apache by calling the container directly.
Comment: Ok, so is there anyway to serve up the web page via Apache that interacts with the backend docker service (which doesnt support ssl)?
Comment: You already did that. Just use `https://myhost.com/api/fetchService` instead of `https://myhost.com:8080/api/fetchService`.
Comment: So i test by going to my html page in the apache virtual host: `https://myhost.com/index.html`. That page calls my docker app at `https://myhost.com:8080/api/fetchService`
Answer: Posting for anyone else with the same problem. I ended up creating a 2nd virtual host for the reverse proxy.
so site1 -> for the apache webpage with ssl,
site2 -> reverse proxy to terminate ssl for container service.
Then copied orig ssl virtual host file and adjusted for new virtual host 8443:
comment out Document Root (e.g. # Document Root)
<Location />
    Order allow,deny
    Allow from all
    ProxyPass http://localhost:8080/ flushpackets=on
    ProxyPassReverse http://localhost:8080/
</Location>

Also added a 2nd listening port in ports.conf (e.g. 8443) and enabled site.
a2ensite reverse-ssl.conf
systemctl restart apache2
good to go!


How to drop excel rows using pandas based on an "OR" statement for 2 columns?

Question: I have a spreadsheet with about 4000 rows.  One of the columns is named "com" ( not my doing ) with values of com, gov or org.  I have these three statements, the first 2 work great when I only want to keep com or gov but the 3rd drops everything when I want to drop anything that is NOT com or gov.:

combinedMonthVulnDataFrame.drop(combinedMonthVulnDataFrame[combinedMonthVulnDataFrame['com'] != "com"].index, inplace = True)

combinedMonthVulnDataFrame.drop(combinedMonthVulnDataFrame[combinedMonthVulnDataFrame['com'] != "gov"].index, inplace = True)

combinedMonthVulnDataFrame.drop(combinedMonthVulnDataFrame[(combinedMonthVulnDataFrame['com'] != "com") | (combinedMonthVulnDataFrame['com'] != "gov")].index, inplace = True)

Unfortunately it's dropping all rows on the 3rd option.
Help?
Comment: To reduce visible noise, you might consider using `df` instead of `combinedMonthVulnDataFrame`.
Answer: You could pair a mask with a list of to filter your records. By using a mask you don't need to drop by using the indices of the records, you will be dropping records based on wether they meet the criteria "True/False", whereas
data =  [["com", "Namespace", "Deployment", "Component", "CVE", "Severity"],
 ["dev-stable", "default", "arango-agnt-zqdd3ekj-35ba4a", "ansi-regex", "CVE-2021-3807", "IMPORTANT"],
 ["demo-gov", "default", "arango-agnt-zqdd3ekj-35ba4a", "ansi-regex", "CVE-2021-3807", "IMPORTANT"],
 ["com", "rocketchat", "rocketchat-quantinuum", "coreutils", "CVE-2016-2781", "LOW"],
 ["utilities", "default", "deployments-webserver", "libonig", "CVE-2019-16163", "MODERATE"]]

df = pd.DataFrame(data[1:], columns=data[0])

mask = df['com'].isin(['com'])
df[~mask]

#       com         Namespace   Deployment                  Component
# 2     com         rocketchat  rocketchat-quantinuum       coreutils

Then you could also negate the mask if you want records not in your list, whereas
mask = df['com'].isin(['com'])
df[~mask]

#       com         Namespace   Deployment                  Component
# 0     dev-stable  default     arango-agnt-zqdd3ekj-35ba4a ansi-regex
# 1     demo-gov    default     arango-agnt-zqdd3ekj-35ba4a ansi-regex
# 3     utilities   default     deployments-webserver       libonig

This pattern is pretty flexible and easy to make changes to. It's one approach.
Comment: This looks great but i just tried it and it also drops everything, including some where the column is empty, as well as some where it's "demo-gov".
Comment: You are changing the values in the lists to make it match the logic you want?
Comment: Maybe post a sample of your dataset so we can see exactly what you're working with?
Comment: I updated the code above to this:

`mask = combinedMonthVulnDataFrame['com'].isin(['com','gov'])
combinedMonthVulnDataFrame.drop(df[mask].index, inplace = True)`
Comment: My dataset looks like this:

`com Namespace Deployment Component CVE Severity
dev-stable default arango-agnt-zqdd3ekj-35ba4a ansi-regex CVE-2021-3807 IMPORTANT
dev-stable default arango-agnt-zqdd3ekj-35ba4a ansi-regex CVE-2021-3807 IMPORTANT
com rocketchat rocketchat-quantinuum coreutils CVE-2016-2781 LOW
utilities default deployments-webserver libonig CVE-2019-16163 MODERATE`

( Sorry, I suck at formatting in comments )
Comment: Which of these are the column names?
Comment: The headers are:

`com Namespace Deployment Component CVE Severity`

The first of the data is:

`dev-stable default arango-agnt-zqdd3ekj-35ba4a ansi-regex CVE-2021-3807 IMPORTANT`
Comment: Updated answer to reflect changes given your dataset. Does this work for you?
Comment: The original answer works, it's the 'OR' version of if it's in "com" or "gov" that's failing.  I'm currently getting this:

`combinedMonthVulnDataFrame.drop(combinedMonthVulnDataFrame.drop[mask].index, inplace = True)`

and the code looks like this:

`mask = combinedMonthVulnDataFrame['com'].isin(['com','gov'])
combinedMonthVulnDataFrame.drop(combinedMonthVulnDataFrame.drop[mask].index, inplace = True)`


How to find $q,\beta$ such that $\nabla\cdot[\gamma\nabla u]=0\Leftrightarrow(-\Delta +q)v=0$ for some $v=\beta u $?

Question: Let $\Omega$ be an open subset of $\mathbb R^n$
. Let $\gamma\in C^1(\Omega)$ be bounded away from zero. Find $q,\beta\in C^1(\Omega)$ such that
\begin{equation*}
 \nabla\cdot[\gamma\nabla u]=0\Leftrightarrow(-\Delta +q)v=0 \text{ for some }v=\beta u
\end{equation*}

My attempt:
$$
\begin{aligned}
0&=~ (-\Delta+q)\beta u\\
&=~-(u\Delta \beta+\beta\Delta u+2\nabla u\cdot\nabla \beta)+q\beta u\\
&=~-2\nabla u\cdot\nabla \beta+u(-\Delta \beta+q\beta)-\beta\Delta u
\end{aligned}$$
Also we have
$$
\begin{aligned}
0&=~ \nabla\cdot(\gamma\nabla u)\\
&=~\nabla u\cdot \nabla \gamma+\gamma \Delta u
\end{aligned}
$$
I do not know how should I proceed now.
Answer: There's something a little bit funny about the regularity assumptions on $\beta$ and $q$ if you're working with classical solutions since you're asked to evaluate $\Delta(\beta u)$ but only assume $\beta \in C^1(\Omega)$.  I'm going to change the numerology to show the idea of the change of unknown while avoiding this issue.  Indeed, I'll assume $\gamma \in C^2(\Omega)$.
Set $\beta = \gamma^{1/2}$, which belongs to $C^2(\Omega)$ since $\gamma \in C^2(\Omega)$ and is bounded away from $0$.  Note that $\beta$ doesn't vanish, so we can define $q = \Delta \beta / \beta \in C^0(\Omega)$.
Then
$$
\nabla \beta = \frac{1}{2} \gamma^{-1/2} \nabla \gamma
$$
and
$$
-\Delta \beta + \beta q =0.
$$
Thus if the functions $u$ and $v$ are related by $v = \beta u$, then
$$
(-\Delta + q) v = -\beta \Delta u - 2 \nabla u\cdot \nabla \beta + u(q\beta -\Delta \beta) = -\gamma^{1/2} \Delta u -\gamma^{-1/2} \nabla \gamma \cdot \nabla u \\
= -\gamma^{-1/2} \left(\gamma \Delta u + \nabla \gamma \cdot \nabla u \right)  = -\gamma^{-1/2} \nabla \cdot (\gamma \nabla u).
$$
We immediately deduce from this that $v$ satisfies $-\Delta v + qv =0$ if and only if $u$ satisfies $\nabla\cdot(\gamma \nabla u) =0$.


C# Differences between two lists

Question: Good day all. I was looking through out the forums but have not found anything useful.
I have two custom lists and I need to display third list with differences.
list1[0].Id = 10;
list1[0].Name = "Peter";
list1[0].Age = 45;
list1[0].Group = "Group1";

list1[1].Id = 11;
list1[1].Name = "John";
list1[1].Age = 42;
list1[1].Group = "Group2";

list1[2].Id = 12;
list1[2].Name = "Mike";
list1[2].Age = 32;
list1[2].Group = "Group2";

list2[0].Id = 10;
list2[0].Name = "Peter";
list2[0].Age = 45;
list2[0].Group = "Group2";

list2[1].Id = 11;
list2[1].Name = "John";
list2[1].Age = 48;
list2[1].Group = "Group2";

list2[2].Id = 12;
list2[2].Name = "Mike";
list2[2].Age = 32;
list2[2].Group = "Group2";

I need to generate list 3 which will contain differences record with id 10 and id 11
Comment: First hit on search shows about a 100 results. One that will help you:
http://stackoverflow.com/questions/5636438/difference-between-two-lists
Comment: Define *Difference*. Example: Same ID but one or more different properties or what?
Comment: @user3067141 it should. If it didn't work, my bet is that it was **your** fault, not the answer's... ;-)
Comment: @bastijn just flag it as a duplicate.
Comment: This indeed seems to be a duplicate of the question pointed out by @bastijn.
Comment: I have seen this post but none of suggestions does not worked for me.
Comment: @user3067141 did you override the equals and gethashcode in your custom type? If so, can you share the implementation?
Answer: Before the answers get out of hand, let me post an answer specific to this question, but with method taken from:
https://stackoverflow.com/a/5636486/103139
Create the following two classes
public class Person{
    public int ID{ get; set;}
    public int Age{ get; set;}
    public string Name{ get; set;}
    public string Group{ get; set;}

    // constructors and other class specific
    // methods come here.
}

public class PersonComparer : IEqualityComparer<Person>
{
    public int GetHashCode(Person p)
    {
        if (p == null)
        {
            return 0;
        }

        // you can put any custom hashcode generation here.
        return p.Name.GetHashCode()
                    ^ p.Age.GetHashCode
                    ^ p.Group.GetHashCode();

    }

    public bool Equals(Person p1, Person p1) {
        if (object.ReferenceEquals(p1, p2)) {
            return true;
        }
        if (
            object.ReferenceEquals(p1, null) ||
            object.ReferenceEquals(p2, null)
        ) {
            return false;
        }
        return p1.Name == p2.Name &&
                p1.Age == p2.Age &&
                p1.Group == p2.Group;   // consider equal ordinal ignore case
    }
}

Than in the place you have your lists do something along the lines of:
var diff = list1.Except(list2, new PersonComparer()).ToList();

Given that your lists are of List. Note that this does not include ID in the check, as I took from you OP example.
Answer: var diff1 = list1.Where(p => list2.FirstOrDefault(q => (p.Age == q.Age && p.Group == q.Group && p.Id == q.Id && p.Name == q.Name)) == null).ToList();
        var diff2=list2.Where(p => list1.FirstOrDefault(q => (p.Age == q.Age && p.Group == q.Group && p.Id == q.Id && p.Name == q.Name)) == null).ToList();

        var diff = diff1.Concat(diff1).ToList();

Comment: This is not the proper way. Consider what happens when you have lists with a large number of items. This will become very expensive. You are running through the other list for each item in the first list, and vice versa (worst case).
Comment: This will work even if both the lists have different number of elements. In any case you have to iterate through each element.
Comment: The key here is to understand that you have to visit them "at least once", and not as many times as items in the list. That is why Except is better. The Except extension method uses a HashSet internally for the second sequence passed in - that allows it to look up elements in O(1) while iterating over the first sequence to filter out elements that are contained in the second sequence, hence the overall effort is O(n+m) where n and m are the length of the input sequences - this is the best you can hope to do since you have to look at each element at least once.
Answer: Use class with properties ID,age,name and group
eg : class example{
                int Id;
                int age;
                string name;
                string group;

              }

create list of class
List <example> list1=new List<example>();
list1.Add(new example{Id=1,age=22,name="n1",group="g1"});
list1.Add(new example{Id=2,age=30,name="n2",group="g2"});
3rd list to take difference
list1.Add(new example{Id=list1[0].Id-list[1].Id,age=list1[0].age-list[1].age});
now your
list1[2] contains diff between liss1[0],list[1]
Comment: Apart from being shady code at the very least, this is not what the OP asks for. You are adding a 3rd item to list1 that is a person with age is age1-age2, and id = id1-id2. It makes no sense.

C# Differences between two lists


Check if an argument from CMD has written or not using Python

Question: I'm new in python and I am trying to build a script that gets as an argument a URL and send a GET request to the website and then print the details from the response and the HTML body as well.
The code I wrote:
import sys
import urllib.request

url = 'https://%s' % (sys.argv[1])
res = urllib.request.urlopen(url)
print('Response from the server:'+'\n', res.info())
print('HTML web page:'+'\n', res.read())

Now, I want to check if an argument is passed through the CMD or not and if not to alert about it.
And also to print the Status Code and the Reason.
What is exactly Reason means?
This is what I got so far and I do not understand why I don't get any result.
My current code is:
import sys
import urllib.request, urllib.error

try:
    url = 'https://%s' % (sys.argv[1])
    res = urllib.request.urlopen(url)
except IndexError:
    if len(sys.argv) < 1:
        print(res.getcode())
        print(res.getreason())
    elif len(sys.argv) > 1:
        print('Response from the server:'+'\n', res.info())
        print('HTML web page:'+'\n', res.read())
        print('Status Code:', res.getcode())

Thank you.
Comment: you are printing the reponse in the except block. Move the print statements inside the else statements into the try block.I tried hardcoding the url and tested it. works fine
Comment: @DineshKumar , can you please attach me the code? I don't understand.
Answer: The except block will get executed only when you dont give any arguments in your command line.Basically only when an IndexError exception is raised.I believe just printing the error message should suffice in that usecase. The print statements should be handled inside the try block.
import sys
import urllib
from urllib2 import URLError

try:
    url = 'https://%s' % (sys.argv[1])
    #url = r'https://www.google.com'
    res = urllib.urlopen(url)
    print('Response from the server:' + '\n', res.info())
    print('HTML web page:' + '\n', res.read())
    print('Status Code:', res.getcode())
except IndexError as e:
    print e.message
except URLError as e1:
    print e1.reason

Note : I am using python 2.7. So I did not use urllib.request.urlopen() as this is not necessary in python 2.x
Comment: How can I print the Reason?
Comment: Do u mean reason for the Index Error?
Answer: You're only checking if len(sys.argv) < 1 or len(sys.argv) > 1 and never if len(sys.argv) == 1, which it will be if you run the script without any arguments, since sys.argv[0] is the name of the script.
In addition, if you use except IndexError, you are essentially handling (or supposed to handle) the exception, so python exits without an error without hitting any case in your if-elif clause.
You could do something like this:
import sys
import urllib

try:
    url = 'https://%s' % (sys.argv[1])
    res = urllib.request.urlopen(url)

except IndexError:
    if len(sys.argv) <= 1:
        print("Too few arguments")
        raise # <- or replace with sys.exit(1) if you don't want to propagate the exception

The actual urllib.request.urlopen(url) might throw other exceptions though. I would be careful with handling command line arguments in the same try-except block as the http request.


How to find parameters for ridge and lasso regularization when cost minimization does not converge?

Question: In the Stanford ML course, we were taught to find good values for the lambda parameters of ridge/lasso by iterating for various lambda values on several cross-validation sets and picking the values which correspond to the hypothesis with the minimum CV error.
The problem is, I am playing with a big data set (which might not even be well suited for logistic regression (??): Internet Ads set) and I can't use the method described above because, during optimization, the cost stays stable (changes only around the 8th decimal place between iterations) and doesn't seem to converge.
I need good values for the regularization in order to converge, but I can't converge without the aforementioned good values.
Any suggestions? Should I move on to using SVM, or is this data set solvable with logistic regression?
NB: I am doing this for learning purposes, so I'm much more interested in explanations why my approach is bad than I am in black-box libraries which will give me a solution.
EDIT: Some relevant code snippets (occasionally pseudocode-ish, for clarity). The usual notations apply.
The function used to compute the cost:
def computeCost(theta, X, y):
    global iter
    iter += 1
    if iter > 10:
        raise TooManyIterationsException(iter) # Because the cost doesn't converge, I force interruption in order to jump to another combination of lambda values
    m = y.size
    h = sigmoid(X.dot(theta.T))
    J = y.T.dot(log(h)) + (1.0 - y.T).dot(numpy.log(1.0 - h))
    J_reg2 = theta[1:]**2
    J_reg1 = theta[1:]
    cost = (-1.0 / m) * (J.sum()) + LAMBDA2 * J_reg2.sum() + LAMBDA1 * J_reg1.sum()
    print "Cost: ", cost
    return cost

Invoking scipy.optimize.fmin_bfgs:
initial_thetas = numpy.zeros((len(train_X[0]), 1))
myargs = (train_X, train_y)

for LAMBDA1 in [0.01, 0.02, 0.04, ..., 10]:
    for LAMBDA2 in my_range[0.01, 0.02, 0.04, ..., 10]:
        try:
            iter = 0
            theta = scipy.optimize.fmin_bfgs(computeCost, x0=initial_thetas, args=myargs)
        except TooManyIterationsException as e:
            print '\n'

A typical output looks like this:

EDITED AGAIN: Evolution of thetas!
Comment: How is it possible that your cost is not changing? The only thing I can think of is that your lambda values aren't actually different. Maybe posting code for what you have done will help to see if your situation can be re-created.
Comment: @idris I have tried manually various combinations of lambdas, but the cost is always the same. In the meantime, I have found a couple other secondary issues with my implementation; if after fixing them I'm still having problems, I'll post the relevant parts of my code here.
Comment: @idris Added source code, as well as an output example. Thanks!
Comment: @Denis I'm not sure what you meant by +/- 1-5%... shall I increase/reduce each of my thetas by those percents and use cost with them modified? By the way, updated my post -- there's an interesting trend in the thetas. The value in theta[0] is shifted in the next cost computation to theta[1], then to theta[2], and so on. I'm not sure whether this is relevant, since I don't know the details of fmin_bfgs implementation.
Comment: @Denis Aha, now I understand! Thank you, this kind of debugging suggestion is invaluable -- I'll check it out!
Comment: @Denis Thank you very much for all the pointers! I have some more ideas now, let's see where I get stuck next :-)
Answer: Is the question "why is cost() so flat near [0 0 0 0] ?",
or "why does fmin_xx not find a minimum on a flat surface ?"
A couple of suggestions anyway:
1) look at cost() near x0,
with look( f, x0, h ) -> f() at all corners of a cube of side 2h around x0,
or if that's too many at a random subset.
2) is [0 0 0 0] a reasonable start point for weights ?
3) what happened to the first 3, continuous, columns of X ?
4) start with fmin() a.k.a. Nelder-Mead
(at the best dim + 1 points from look())
before running fmin_xx -- more powerful but harder to drive.
5) scikit-learn SGDClassifier got to
97.3 % correct  SGDClassifier  uciml/ad*
    X (1572, 1558) Xtest (787, 1558)  -- 2/3, 1/3 split
    centre 3  -- scale all feature columns to [-1, 1]
    sgditer 100  loss log  penalty l2
11 sec
Confusion matrix: 97.3 % correct = 766 / 787
True classes down, estimated across  / true class sizes
0:  652    8  /  660  99 %
1:   13  114  /  127  90 %

Comment: Thank you for the extra suggestions and for all the effort you've put into my question! I feel smarter than I was a week ago :-)
Answer: I don't understand why you have two $\lambda$ values. In all the logistic regression I've encountered, there is only one $\lambda$ value, and you try different values of $\lambda$ to minimize the cost $J$. I also took the machine learning class you reference, and my Octave code looked like this.
J = 1/m * sum(-y' * log(sigmoid(X*theta)) - (1-y)'*log(1-sigmoid(X*theta)));
J += lambda/(2*m) * sum(theta(2:theta_len,:).^2);

Note that lambda should be a scalar value in this implementation.
This is the math equivalent of the above Octave statement.
$J(\theta) = \frac{1}{m}\sum\limits_{i=1}^m[y^{(i)}log(h_{\theta}(x^{(i)})) - (1-y^{(i)})log(1-h_{\theta}(x^{(i)}))] + \frac{\lambda}{2m}\sum\limits_{j=1}^n\theta^{2}_{j}$
Comment: What we did there was ridge regression. Based on previous suggestions in another post, I was trying to implement here [elastic net regularization](https://en.wikipedia.org/wiki/Elastic_net_regularization), which is a combination of ridge and lasso methods. Hence, I need lambda2 for ridge, and lambda1 for lasso.
Comment: Ok, this is out of my wheel house as I am not yet familiar with elastic net. I do notice your (\theta) look to be pretty much all zero. Might be something work looking at.
Comment: Yeah, I have a couple of directions I want to follow now, the thetas are only one of them. Thanks a lot for the idea-bouncing! :-)


How to find a 1-1 map from $B\to A$ where $B$ is set of all positive integers and $A$ is set of positive rationals - positive integers.

Question: I have been at this for a while, I need it for a small part of my proof, any methods for finding a map would be appreciated.
Comment: If "1-1 map" has the usual meaning of "injective", then how about $f(n)=n+\frac12$?
Answer: Asssuming that you actually want a bijection and not merely an "1-1 map" (which generally just means an injective function):
Call a number $n$ "spiffy" if $n=10^a+2\cdot 10^b$ where $a\ge 2$, $b\ge 2$ and $a,b$ are relatively prime.
Then consider
$$ f(n) = \begin{cases} a/b & \text{if }n=10^a+2\cdot 10^b\text{ is spiffy} \\
1/(k+1) & \text{otherwise, where }k = \#\{m \in\{1,2,\ldots,n\} \mid m\text{ is not spiffy} \} \end{cases} $$
Comment: i actually only need a 1-1 map.


Tracking Individual Processes in Windows Server 2008, Is It Possible?

Question: I'm hoping we can get a solution together on this.  I'm looking to try and narrow down what's happening on a windows server 2008 box that's causing us to have to reboot every 2 weeks.  Resource monitor would show that there's some sort of memory leak going on but we are having problems determining what application is causing the problem.
Does anyone know of a good solution for logging memory usage on a per-process level?  We're hoping to log the output over a section of like 2-4 weeks and see what happens.
Any help is appreciated!  Thank you.
Answer: There are many tools available to help you track down issues like this.
Start with the performance monitor which is provided with your OS. You can set it up to take periodic snapshot of pretty much any metric you can think of, including individual memory usage per process. In your case, that's probably the best tool for identifying rogue processes.
Also process explorer, which you'll have to download, is a very good way if looking into all processes of your system. It will allow you explore your system state in a very detailed way.
Finally, you can use process monitor, also a separate download, to watch a particular process activity in detail.
Comment: Got it.  Data Collector Set FTW I believe.  Thank you very much Stephane.
Answer: WSRM does this, if you enable the process accounting.
http://technet.microsoft.com/en-us/library/cc732553.aspx.
Comment: A better link for this approach might be [Audit process tracking](https://technet.microsoft.com/en-us/library/cc775520%28v=ws.10%29.aspx?f=255&MSPPError=-2147217396)


Forecasting data with multiple seasonality

Question: I'm attempting to forecast the number of taxi rides per hour that occur in NYC. I've turned the data into a time series using 24*7 as the frequency:
taxi_ts <- ts(taxi_train, frequency = 24*7)
I then use decompose() to split my ts into seasonal, trend, and random components.
parts<-decompose(train_ts)

side question if anyone knows, why does this include alot of NA in the data?

I'm assuming now that I would fit a ARIMA model to my trend component to forecast on that.
After I do this though how would I add the seasonal component back on to the predictions?
Answer: I don't think that your attempt to use a frequency of 168 will give you the results that are after as it may be too coarse i.e.  crude or unrefined .
https://stats.stackexchange.com/search?q=user%3A3382+hourly+data will give you some pointers as to how I think that you you should proceed with hourly data. Essentially daily habits can impact hourly responses/values.
I have have been routinely able to implement a two-pronged approach where hourly forecasts are developed based on good daily forecasts which are developed based upon :
what day of the week it is
what month your are in
what level changes have occurred
what trend changes have occurred
what days of the month exhibit statistically usual effect
arima structure
what week- of- the month you are in
holiday effects before, on and after
long weekend effects
month-end effects
and possibly/probably rainfall and weather conditions.
With good daily forecasts , my approach to your problem is to construct 24 hourly models using daily totals as an exogenous predictor and identify trends, level shifts , memory structure (arima) at the hourly level.
Answer: 1) I do not know for sure why you get many NAs in there, but most probably your window is too large. Decompose works by sliding window smoothing - it uses some radius $r$ (usually around 2 times smaller than the seasonality unless you changed it) to go through all values of time series, and sums up $r$ values to the left and right of each time series observation. The first and last $r$ values in the time series will be NA, since the window cannot be estimated ($r$ goes beyond the bounds). In your case you should have around $12 \times 7$ values as NA from each side of time series. However, you still have the necessary component in your $figure$ variable.
2) Take you initial time series, and simply subtract the "figure" from it repeatedly - simply move this window along the time series. This will give you a deseasonalized component (trend + cycle + error). However, if you have multiple seasonalities, you simply either repeat the procedure, but a better way is to use SARIMA - seasonal ARIMA. It will also difference your second season on demand by specifying the large $D$.
3) After you get your SARIMA forecast, simply add your seasonal figure (the one that you subtracted) onto it, step-by-step iteratively again. In case you had 2 seasonal removals and ARIMA, use ARIMA forecast, and first add second seasonal figure iteratively, then add the first seasonal figure iteratively.
However you should try modelling seasonality differently - such a large seasonal figure ($24 \times 7$) will be a bad estimate most probably. Try looking into deseasonalization with smaller period up to 20, or use Fourier series, which is very simple. Check on Hyndman's website: https://robjhyndman.com/publications/complex-seasonality


Download file field dependent on user role vs role field values

Question: I have a type downloadable_item which has a field field_roles_allowed that is based on the Role Field Module.
My goal is to have a field_download of file type that will show the file for download if the user in a role selected in the field_roles_allowed and a message that they need to request access otherwise.
I am attempting to add a formatter for the file field type using the Custom Formatters Module. I found this tutorial on how to use the Custom Formatters module, however, they use the "HTML + Tokens" option. I will need to use the PHP option to be able to output either the download button or a message depending on whether the user is in a role in the field_roles_allowed.
I cannot figure out how to get the roles that are allowed, download file information, and current user's roles for comparison.
My question is really "How do I do these things that I cannot figure out?", but if someone believes I am going about this totally in the wrong way... I am open to another way.

EDIT: I solved my own problem below, but not really in a preferable way. How could this be done without writing PHP logic in field?
Comment: I have a question. Do you have views and panels installed? If so you can do this in a round-about way using Panels and Views. I did this on a site I run where we sell digital downloads. The download links are only showed to users that have the appropriate role (that gets added in the checkout process).
Comment: Could you post your solution after "EDIT" section?
Comment: @milkovsky, I posted as an answer
Answer: After some digging... I found that I am able to get field information like...
$variables["#object"]->field_roles_allowed["und"]; //array containing id's of roles allowed
$variables["#object"]->field_download["und"]; //array containing information about the file

and that I can get the current user's roles like...
global $user;
$user->roles; //array containing current user's roles

Then I can compare the field roles allowed with the current user's roles and determine what to output from there.
Still not very happy with this, I would prefer something where I don't have to write PHP logic in a field.
Answer: As always with Drupal. There are often many solutions to your problem. Here are two ways of doing this.
Views and Panels
The power of views and panels can be leveraged to do this.
First step would be to build a view containing various panes built for the different displays you want per security role. You would want to use panes because they are only accessible to Panels and cannot be used by anything else on your Drupal website, this will be important for step two. In your case you would have a pane that did not show the download link then a pane that shows the download link. This can be done in one view with multiple panes.
Second step build a panel page. Panels has a unique capability of variations of a page "Called Variants in the Panel UI". The variants can be assigned selection rules to each variant. Here is where the security layer comes in and why you want to build your views as panel panes and not just view pages. Panes ensures that only Panels can access your views you have created with your protected content. The first variation will use a selection rule of User:role with the role you want to use to allow access to the protected content. The second variation will use no selection rules as this will be what the user sees if they do not have the role. Panel variations and selection rules work in the first one satisfied in the order of the variations. The first variation with the user selection rule will be shown before the last variation if the user does not have the role Panels goes to the next variation till it can find one that can be satisfied - if one exists. This has many other applications but I have used this feature to serve protected content to users based on their roles.
Themeing layer
Another way to approach this is to use Drupal's themeing layer. If you are using core node displays to show your content to the user this works very well. Using Drupal's node templates you can overide the display of the nodes containing the protected fields and use some simple PHP statements to hide or show the field with the download link. Example core pieces of the code in the template would be:
global $user;

foreach ($this_user->roles as $rid => $role) {
  switch ($role) {
    case 'has access':
      print render($content['field_download_link'])
      break;

  }
}

Comment: Thanks for the response. I've only started using Drupal about 2 weeks ago. In my case, I have a ContentType/View called `downloadable_item` with the download button. I have a ContentType/View on top of that called `downloadable_group` which uses the ViewsFieldView module and contextual filter to house multiple `downloadable_item`. The `downloadable_group` view pane is then placed on a page and uses contextual filters as well and may contain multiple downloadable_groups. I don't know if this changes anything or not and I will look into how these solutions will work in my project on Monday.
Comment: The reason I posted the above is that I am concerned about whether or the way I have things set up will conflict with the solutions. Also, in the first solution you mention "a selection rule of User:role with the role you want to use to allow access to the protected content", I am not at my workstation so I cannot look now, but I am not sure where or how I would create a rule that has the ability to look into the User's role and all of the fields contained in the UserRole field and compare the two. The UserRole field will have multiple roles that will be allowed.
Comment: Do you have a live dev site?
Comment: This is an internal company thing which I cannot share. Did you need any additional information from me?
Comment: I am just confused when you say ContentType/View. A View is a very specific thing in Drupal, it involves the Views module. So you build Views of your ContentTypes? Correct?
Comment: Regarding your second comment. I am explaining how to do this with Panels. Panels has a "Selection Rules" associated with each variant you create. This isn't something you have to create, it comes with the module.
Comment: I am pretty new to Drupal so there may be different ways of doing things. How I've been doing it is, I build a ContentType and then I build a View for that ContentType and use "Fields" in the View. So when I say ContentType/View I mean that I have both of them for the downloadable_item and the downloadable_group... and then using the Views Field View module I insert downloadable_items in the downloadable_groups. I was hoping that the downloadable_item view would be able to handle on it's own how it displays the downloadable button (based on the User role and the allowed_roles field).
Comment: So that is another way of doing it. You can restrict the Views you have created. The Views have a permission function. You will need to use contextual filters to change which view is displayed to switch between the "has rights" and the "not has rights". This is actually easier with Panels using the Variants and Selection Rules provided by Panels. Then you have one URL with no worry about contextual filters. Panels handles displaying the content for you based on the role the person has.
Comment: BTW, Panels was written by the guy who wrote Views. If that gives you some idea on how well they work together.
Comment: I don't remember there being a way to filter certain fields of a view out without using the Custom Formatters Module, but what I didn't like about that is that I then had to write PHP in the browser in a field to make it work.
Comment: There isn't. You will have to have two views with two different URLs. If you desire to do this from a single URL and display different content based on the user role you will need to either implement this in the theming layer overriding the core content view (not Views module - core Drupal theming of content types), use panels, or find another contrib module to give you this capability.
Comment: I'm not sure what you mean with the URL's as there will be multiple groups of multiple items where each group is a view and each downloadable item is a view within a group. On a single URL there might be a dozen or more downloadable items, some of which the user may see the "download" button (which is based on a file field) and others where they will see "you don't have access..."
Comment: It seems custom themes was the right answer for me, I followed [this tutorial](http://www.wdtutorials.com/2011/06/29/drupal-7-how-create-custom-template-view#.VMve6zHF98E) which provided more details on the process. I believe this is what you meant by your second solution.
Comment: Good to hear you found a solution!


A visual interpretation of probability?

Question: I was using the following website for a visual information theory : http://colah.github.io/posts/2015-09-Visual-Information/.
And I came across the following:

So it says that in the first image where the lines are straight the events are independent.  But in the second picture when the events are not independent then the lines are bloated. I am wondering why the reason is so. It says here that some of the areas are bloated and some are not, and this is how we know that one event is more probable than the other. But how does that factor into independence? Even in the first picture, one portion of the area is bigger than the other. So how does the area matter or give us an answer about independence?
Comment: Do you know what the definition of independence is?
Answer: Conditional probability works in a reduced sample space.
$$ \begin{array}{r}  \begin{array}{c|c|c}
  \style{font-family:inherit}{\text{clothing today}\bigg\\ \text{weather today}}  & \style{font-family:inherit}{\text{Raining}}  & \style{font-family:inherit}{\text{Sunny}}  \\\hline
  \style{font-family:inherit}{\text{T-shirt}}  & a  & b & 62\%
\\\hline
  \style{font-family:inherit}{\text{Coat}}  & c  & d & 38\%
  \\\hline&25\%&75\%&100\%\end{array}
 \end{array}\hskip-5.5pt$$
Coat and Raining are $\color{red}{\textbf{independent}}$ events iff $$\frac{c}{a+c}=\frac{P(\text{Coat}\cap \text{Raining})}{P(\text{Raining})}=P(\text{Coat}|\text{Raining})\color{red}{\textbf{ = }} P(\text{Coat})=\frac{c\quad+\quad d}{a+c\quad+\quad b+d}.$$ This means that Coat and Raining are independent events iff the Coat area (whether we choose box $c$ or $d$ or $c+d$) is proportional to its corresponding column area.
This is equivalent to the given illustration having dividing lines that are each either vertical straight or horizontal straight.
Answer: While I think that that visual is not particularly helpful, in no small part because the exterior is nonsensical, there is a way to interpret it reasonably. In the first case, you can pick any information from the clothing axis, and gain no more information about the weather. Similarly, you can pick any information from the weather axis and gain no more information about the clothing. This is to say that if we compare the whole graph to a subset obtained by focusing on part of one axis, the graph is largely unchanged.
On the other hand, this process fails in the second diagram. If I restrict to "coat" section, the graph is telling me that it is more likely to be raining - I have gained more knowledge about the weather variable by being more specific in my investigation of the clothing variable.
This is the crux of independence and dependence. Two variables are independent if adding information about one does not give you new information about the other.
Comment: That raises the question of what a better way would be to visualize the information introduced by dependence.
Comment: I guess it depends on the individual. Two examples that come to mind for me are bivariate Gaussians and possibly something to do with a filtration by time. More elementary examples in the style of the above could be done just by being more careful in sketching.
Comment: @Semiclassical I think they could have just had the unit box as in the upper sketch (no wavy boundary) and randomly sketched out four regions in the bottom box with the 4 colors used above to indicate the 4 outcomes of this experiment (and make sure that the volumes were different than what is given since for 2 Bernoulli RVs, those 4 volumes automatically imply independence). (So maybe just drag the second half of the upper line below to enlarge the upper right corner)

A visual interpretation of probability?


Are the standard errors in the coefficients of a linear regression corellated?

Question: Lowly physical scientist here so please excuse my ignorance.
I have a data set of thermal expansion coefficient, $\alpha$, against temperature, $T$ that I fit with a linear regression.
From this, I am trying to calculate another quantity, $T_{0}$, that depends on $\alpha$ as a function of temperature in quite a complicated way.
My question is in calculating the uncertainty in $T_{0}$ and how to interpret the uncertainty in $\alpha$ (assuming there is no uncertainty in $T$).
Are the standard errors in the coefficients (gradient and intercept) given by a linear regression of $\alpha$ vs. $T$ uncorrelated and hence would it be valid to propagate these errors as such as the error in $\alpha$? My intuition is that they are correlated as they stem from the same physical source (inaccuracy in the measuring device).
Answer: Your intuition is correct. A linear regression model provides a variance-covariance matrix for the coefficients that can be used with the formula for sums of correlated variables to estimate errors in predictions made from the model.*
This answer describes both the general multiple regression result and works through its application to your simple intercept/slope, single-predictor situation. In general, the variance-covariance matrix among linear regression coefficients depends on the estimated residual variance unexplained by the regression, $\hat\sigma^2$, and the design matrix representing the predictor values. The variance-covariance matrix for the estimates of the intercept and the slope in your single-predictor situation is:
$$\frac{\hat\sigma^2 }{n\sum x_i^2 - (\sum x_i)^2}
\left(
\begin{array}{cc}
\sum x_i^2 & -\sum x_i \\
-\sum x_i  & n
\end{array}
\right)$$
where the $x_i$ are the values of the independent variables and $n$ is the number of observations. The variances of the intercept and the slope are the diagonal elements of the matrix; the covariance between them is either of the diagonal terms.
So the errors in the estimates of the intercept and the slope are related to the $y$-value measurement errors (of the thermal expansion coefficients in your case) via $\hat\sigma^2$, the variance unexplained by the linear relationship. The covariance between the 2 coefficient estimates depends on the mean values of the independent variables. Note that the covariance between the coefficients is negative if the mean $x$ value, $\bar x$, is positive.
That relationship of the coefficient covariance to your particular choice of $x$ values might seem strange, but it comes down to a pretty simple result when you plug through the formula for the variance of a $y$ value predicted from the model. As shown on this page, the variance in a $y$ value estimated at any specified $x$ value, $x_d$, is:
$$
\hat\sigma^2\left(1+\frac{1}{n} + \frac{\left(x_d - \bar{x}\right)^2}{\sum (x_i - \bar{x})^2}\right)
$$
So the error in a value predicted from your linear regression depends on the measurement error via $\hat\sigma^2$, the number of observations $n$ that went into your model, and the distance of your specified $x$ value for the prediction, $x_d$, from the mean of your original predictor values, $\bar x$.
Those all make intuitive sense as contributions to the error in an estimate. Respectively, they are the estimated measurement error per se, the precision of your estimate of that measurement error as determined by the number of observations, and added extrapolation error as you move away from the mean of your original observations.
You shouldn't have to do all these calculations by yourself; any respectable statistical software suite should be able to provide the variance of a prediction from a linear regression model. You can then use that variance for your further error-propagation analyses.

*This is a somewhat more complicated issue in observational studies with multiple regression, in which several "independent" variables are often highly correlated among each other. Your single-predictor situation is fairly simple in practice,
Comment: Is there a difference between the errors in the coefficients being correlated and the actual variables themselves being correlated? My understanding was that the slope and intercept are not correlated with one another but their errors in effect stem from the same source? My intuition is that this should reduce the error in something dependent on the model (compared to propagating both error in gradient and slope as two independent, uncorrelated errors) where as treating them as correlated variables generally increases it?
Comment: @scomtott I've expanded the answer to show more details of where the covariances of the coefficients come from and how that plays out pretty simply in practice when there is only 1 independent variable in the model.


Global legend and title aside subplots

Question: I've started with matplot and managed some basic plots, but now I find it hard to discover how to do some stuff I need now :(
My actual question is how to place a global title and global legend on a figure with subplots.
I'm doing 2x3 subplots where I have a lot of different graphs in various colors (about 200). To distinguish (most) of them I wrote something like
def style(i, total):
    return dict(color=jet(i/total),
                linestyle=["-", "--", "-.", ":"][i%4],
                marker=["+", "*", "1", "2", "3", "4", "s"][i%7])

fig=plt.figure()
p0=fig.add_subplot(321)
for i, y in enumerate(data):
    p0.plot(x, trans0(y), "-", label=i, **style(i, total))
# and more subplots with other transN functions

(any thoughts on this? :)) Each subplot has the same style function.
Now I'm trying to get a global title for all subplots and also a global legend which explains all styles. Also I need to make the font tiny to fit all 200 styles on there (I don't need completely unique styles, but at least some attempt)
Can someone help me solve this task?
Comment: Global title: http://matplotlib.sourceforge.net/examples/pylab_examples/newscalarformatter_demo.html
Answer: Global title: In newer releases of matplotlib one can use Figure.suptitle() method of Figure:
import matplotlib.pyplot as plt
fig = plt.gcf()
fig.suptitle("Title centered above all subplots", fontsize=14)

Alternatively (based on @Steven C. Howell's comment below (thank you!)), use the matplotlib.pyplot.suptitle() function:
 import matplotlib.pyplot as plt
 # plot stuff
 # ...
 plt.suptitle("Title centered above all subplots", fontsize=14)

Comment: For those importing like this: `import matplotlib.pyplot as plt`, the command could be entered simply as `plt.figure(); plt.suptitle('Title centered above all subplots'); plt.subplot(231); plt.plot(data[:,0], data[:,1]);` etc...
Answer: In addition to the orbeckst answer one might also want to shift the subplots down. Here's an MWE in OOP style:
import matplotlib.pyplot as plt

fig = plt.figure()
st = fig.suptitle("suptitle", fontsize="x-large")

ax1 = fig.add_subplot(311)
ax1.plot([1,2,3])
ax1.set_title("ax1")

ax2 = fig.add_subplot(312)
ax2.plot([1,2,3])
ax2.set_title("ax2")

ax3 = fig.add_subplot(313)
ax3.plot([1,2,3])
ax3.set_title("ax3")

fig.tight_layout()

# shift subplots down:
st.set_y(0.95)
fig.subplots_adjust(top=0.85)

fig.savefig("test.png")

gives:
Answer: For legend labels can use something like below. Legendlabels are the plot lines saved. modFreq are where the name of the actual labels corresponding to the plot lines. Then the third parameter is the location of the legend. Lastly, you can pass in any arguments as I've down here but mainly need the first three. Also, you are supposed to if you set the labels correctly in the plot command. To just call legend with the location parameter and it finds the labels in each of the lines. I have had better luck making my own legend as below. Seems to work in all cases where have never seemed to get the other way going properly. If you don't understand let me know:
legendLabels = []
for i in range(modSize):
    legendLabels.append(ax.plot(x,hstack((array([0]),actSum[j,semi,i,semi])), color=plotColor[i%8], dashes=dashes[i%4])[0]) #linestyle=dashs[i%4]
legArgs = dict(title='AM Templates (Hz)',bbox_to_anchor=[.4,1.05],borderpad=0.1,labelspacing=0,handlelength=1.8,handletextpad=0.05,frameon=False,ncol=4, columnspacing=0.02) #ncol,numpoints,columnspacing,title,bbox_transform,prop
leg = ax.legend(tuple(legendLabels),tuple(modFreq),'upper center',**legArgs)
leg.get_title().set_fontsize(tick_size)

You can also use the leg to change fontsizes or nearly any parameter of the legend.
Global title as stated in the above comment can be done with adding text per the link provided:
http://matplotlib.sourceforge.net/examples/pylab_examples/newscalarformatter_demo.html
f.text(0.5,0.975,'The new formatter, default settings',horizontalalignment='center',
       verticalalignment='top')

Answer: suptitle seems the way to go, but for what it's worth, the figure has a transFigure property that you can use:
fig=figure(1)
text(0.5, 0.95, 'test', transform=fig.transFigure, horizontalalignment='center')

Global legend and title aside subplots


Existem alguma forma de importar um projeto no Android Studio solucionando as dependências automaticamente?

Question: Tenho tido dificuldades frequentes na importação de projetos no Android Studio. Sempre que faço uma nova importação, a IDE aponta várias dependências que não são satisfeitas e tenho que solucioná-las manualmente, tornando a importação sempre lenta demais.
Alguém tem uma dica de como configurar o Android Studio para solucionar pelo menos parte das dependências automaticamente.
Comment: Sues projetos utilizam o Gradle? Pois o gradle deveria cuidar das importações de dependências automaticamente
Answer: Como @LeoSantana sugeriu, utilize o Gradle, ele é um "automatizador de build".
Recomendo esse link: https://www.udacity.com/course/gradle-for-android-and-java--ud867, curso de gradle para Android, feito pelo Udacity.
[EDIT]
Para saber as dependências que podem ser acrescentadas, pesquise aqui: https://plugins.gradle.org
Comment: @Danilo obrigado, sobre a resposta é que não podia comentar até então.
Comment: Antes de mais nada bem vindo ao StackOverflow. Cuidado com as respostas, isso caberia como um comentário. Esses tipos de respostas deixa com votos negativos. Recomendo que faça um tour no StackOverflow http://pt.stackoverflow.com/tour
Comment: não posso ainda, mas já entendi sua sugestão. Valeu!


How to resize automatically when users upload images by CCK image field

Question: I would like to make the size of images smaller automatically when users upload them using CCK image field.
For example, I would like to let users upload big sized images like 5MB but resize them smaller automatically like 0.5MB before saved on the server.
The reason is that the capacity I can use on the shared server is very limited. Of course I can set "Maximum upload size per file" to a small size but then users who have big sized images but don't know how to resize them before upload will not be able to upload them. I have tried also "image cache" a bit but it doesn't seem to save the files with the name, attributes and in the directory specified by CCK image field.
I don't know if this kind of thing is possible to achieve or not.. Any Idea?
Answer: Imagecache is great, but since it keeps the original file around, it won't save space. In fact, if you have multiple presets, it will increase your disk usage as it creates its caches of images of those sizes.
What you may want to investigate is putting some logic into nodeapi() (or, if on Drupal7, its equivalent function) when saving your node that looks at the filepath of whatever image field you are concerned with and does some image manipulation on it if it is over a certain size and then saving it in place.
Comment: Thanks, however it seems to be over my head using  nodeapi() thing as I'm not a coder. So looks like this is not something easy to achieve? By the way, I'm on Drupal 6.
Comment: Well, you could also set maximum dimensions for your image upload, and then it would be resized within those maximums, which might help out a bit.
Comment: "maximum dimensions" is a kind of what I was looking for. Thanks!
Answer: The module you want is: Max Image Size

This module will resize uploaded images to be below the set
  dimensions. It is not an image style or other presentation layer
  module. If you want to maintain the original image you should use
  Image Resize Filter or an image style. The use case for this module is
  sites where users may upload very large images, but you do not
  want/need to keep the original.
This is different from the core image field size restrictions in that
  it will work on already uploaded files and imported content.
Comment: This module currently is for Drupal 7 only.
Answer: The imagefield cck field (which depends on filefield) provides this capability out of the box.  You add a filefield to your content type, set the display widget to "image", and then when you edit the settings for the field one of the settings is called "Maximum dimensions for images".  Here you can set this to something like 600x400, or whatever you like. If a larger image is uploaded, it will automatically be resized.
Answer: Not really what you're asking for but anyway: You already discovered the Drupal way of handling big images: image cache. If you can, switch to a hoster with a lot of storage (some offer unlimited space).
Comment: Yes, switching the host is probably the best solution. But my point is that I would like to let users upload big files but practically save them as small files if possible..