Rest API e Sessões, como funciona o Login?

Question: Eu sempre trabalhei com PHP/MySQL e Javascript/jQuery. Sempre fiz a conexão do PHP com o banco de dados direto, e uso sessões para fazer login.
Atualmente estou trabalhando em um projeto onde o uso AngularJS para o frontend e PHP/MySQL para o backend como uma espécie de API para alimentar a aplicação principal com dados do banco de dados. Só que como eu expliquei anteriormente, sempre fiz o uso de sessões para lidar com o login, a minha dúvida é: Qual a melhor opção para fazer esse login e manter autenticado? Devo enviar sempre os dados de usuário e senha para API? Para permanecer autenticado? Devo usar alguma espécie de token com tempo de expiração? Mas no caso de um browser (aplicação web), onde eu iria armazenar esse token? Então eu ainda sim faria o uso de sessões?
Enfim, não sei se eu disse alguma bobeira, se sim, por favor me corrijam.
Comment: Pelo que eu entendi, sua dúvida é sobre como gerenciar logins no backend de uma API, certo?
Comment: Isso. Mas também, como posso armazenar essa autenticação, já que não estarei usando sessões.
Comment: Tem algum motivo para não usar sessões?
Comment: @Gabe nada impede, porém não é um padrão de uso comum porque REST APIs também podem ser usadas por clientes que não possuem o conceito de sessão - por exemplo aplicações mobile para Android e iOS. Um dos padrões utilizados neste caso é o bearer token.
Comment: @lbotinelly Ah, faz sentido... Não considerei que a mesma API podia ser usada pra outros clientes. Isso que dá nunca ter trabalhado com apps `:D`
Comment: @Gabe *Live and learn*! Eu mesmo aprendi isso a pouco tempo atrás, questão de meses. Uma das vantagens do mecanismo de *bearer token* é que você pode simplesmente revogá-los em massa (como serviços tipo Google, Facebook e Twitter fazem quando você altera sua senha.)
Comment: Então... A minha ideia era justamente essa. Sou desenvolvedor Android/Java também. Estava pensando em trabalhar com Angular/JS + Phalcon Micro framework (frontend angular e backend como uma REST API), pois pensei que a API futuramente, além de alimentar minha aplicação principal, poderia alimentar um aplicativo mobile. Neste caso, eu precisaria implementar alguma forma de autenticação que funcione em ambos casos. Bearer token que você diz é tipo OAuth 2?
Comment: @ThiagoYoithi se a resposta ajudou, finaliza com a resposta. Abraços.
Answer: A ideia é construir uma RestAPI, ela é stateless(não mantem o estado). Cada usuário, solicita para RestApi um token, com esse token ele acessa os endpoints que precisa consultar.
Para isso você monta um cadastro de cada usuário, para as aplicações que usarão a sua API. Você entrega para o cliente um client_secrete client_id para cada aplicação.
Com o cliente_secret e client_id, o usuário pode chamar um endpoint que validará as informações e devolve o token para cada requisição.
Adaptando ao seu caso, imagina que cada chamada AJAX(ou $http, que usa AJAX de fundo) do Angular você passa o token junto com a chamada, internamente você valida o Token e permite o acesso.
Para cada token, você pode criar funções ou (roles), garantindo o acesso publico e parcial da sua API. Ou permitir que aplicações tenha um token por menor ou mais tempo.
Uma sugestão pessoal, considere deixar alguns endpoints da sua API público, as vezes as suas informações podem ajudar outros a criarem mais coisas em cima da sua API, o que pode acabar trazendo mais valor ao seu conteúdo. E dependendo do acesso você pode até cobrar por isso.
Essa é a forma resumida que você trabalha com uma RestAPI e oAuth, uma olhada atenta as especificações e também ao funcionamento de outras API, ajuda bastante na compreensão do assunto.
Segue algumas que eu particularmente gosto de olhar.
Github
Apontador
Twitter
Instagram
Comment: @WilliamOkano Ná pratica esse token é armazenado ao lado do cliente... local storage ou cookies. Ele é enviado em toda requisicao ao servidor. E devolvido em toda resposta. O servidor é sim, stateless. Quem guarda o estado é o front end. Existem implementacoes que guardam no banco de dados e enviam ao cliente, por exemplo para garantir acesso somente por um dispositivo
Comment: Só lembrando que apesar de não estar incorreta sua resposta, adicionar um token em toda requisição onde a API deve validar toda requisicação requer que o backend guarde um estado, logo não seria mais uma aplicação stateless.


partial trace is invariant

Question: Given an $n\times n$ hermitian matrix and m-dimensional subspace $V$ the partial trace is defined in tao's matrix book as $\sum_{i=1}^mv_i^*Av_i$, where the $v_i$ are a basis for $V$. He says it is independent of the basis chosen, and I want to show this using this hint:
"Given two orthonormal bases for V, write the elements of one basis as a linear combination of the other basis; the coefficients will then form a unitary matrix since both bases are orthonormal. Insert these expansions of the basis elements to write the partial trace in one basis in terms of the other basis, and use the definition of a unitary matrix to simplify."
Can someone please carry out the hint? I cannot.
Answer: Let $(v_1, \dots, v_m)$ and $(w_1, \dots, w_m)$ be two orthonormal bases of an $m$-dimensional space $V \subseteq \mathbb{C}^n$ (with respect to the standard inner product on $\mathbb{C}^n$) and write $w_i = \sum_{j=1}^m u_{ij} v_j$. The matrix $U = (u_{ij})_{i,j=1}^m$ is a unitary matrix and so $U^{*}U = UU^{*} = I_m$. Then
$$ \sum_{i=1}^m w_i^{*} A w_i = \sum_{i=1}^m \left( \sum_{j=1}^m u_{ij} v_j \right)^{*} A \left( \sum_{k=1}^m u_{ik} v_k \right) = \sum_{i,j,k} \overline{u_{ij}} u_{ik} v_j^{*} A v_k \\
= \sum_{jk} \left( \sum_i (U^{*})_{ji} U_{ik} \right) v_j^{*} A v_k = \sum_{jk} \delta_{jk} v_j^{*} A v_k = \sum_{j=1}^m v_j^{*} A v_j. $$

partial trace is invariant


Sum of reciprocals of primes for known primes.

Question: I was reading through some old analytic number theory notes earlier and found the interesting fact that even though $\sum\frac{1}{p}$ diverges:
$\sum_{\text{known primes}}\frac{1}{p} < 4$.
However these notes were written pre-$2003$. I was wondering if this is still the case. If not then how much bigger has this sum got?
Comment: Not much. $$\sum_{p \leqslant x} \frac{1}{p} = \log \log x + O(1),$$ and $e^{e^4} \approx 5.148435562634557\cdot 10^{23}$. Apart from a handful of huge primes, the "known" primes are all smaller than that.
Comment: The biggest complete list of "small" primes I've seen is up to $10^{10}$.  The contribution of the known larger primes to the sum of reciprocals is negligible.  So that puts us around $3.2$.
Comment: Also [T.R. Nicely](http://www.trnicely.net/twins/twins2.html) has gone up to $1.6 \times 10^{15}$ summing reciprocals of twin primes, and he's been at that for more than a decade.  So if he'd also been summing reciprocals of all primes, we'd be up to around $3.55$.
Comment: I once attended a talk where the speaker said that the sum of the reciprocals of the known primes was less than 4 --- and always would be. I think this was first suggested by Matiyasevich. (It's 10 years, almost to the day, since I made this observation on another internet math forum. http://mathforum.org/kb/message.jspa?messageID=581672)
Comment: Consider that the number of particles in the universe is about $10^{80}$, and $\log \log 10^{80} \approx 5.2$.
Comment: Well I kinda imagined we wouldn't have got past the 5 mark but wondered really if the fact was still accurate.
Comment: That is a really interesting fact. Thanks for the link.
Comment: If someone can turn one of their comments into an answer I will accept it!
Answer: Given the growth rate of $\exp \exp x$, I'd say we should be more careful about the contribution of the $O(1)$ term which is about $0.2615$.  That means we only need to know primes up to about $1.8 \times 10^{18}$ in order to reach $4$.  Numbers in this range are easily sievable to determine primality — you could essentially compute them as fast as you can record them.  So the main bottleneck would be storage: let's assume you require a record of all the primes found, rather than just the reciprocal sum (which would be much, much slimmer).
The total amount of storage needed to hold that many primes is about $2.6 \times 10^{18}$ bits, or roughly 320 petabytes.  The number of iPhones sold in the last year is about 160 million, so if each of them was allocated a different range of primes, each would only need to hold 2GB of primes, which I doubt would take more than an hour or two of computation.
So basically, depending on how generous you want to be with the definition of "known prime", Apple could break the $4$-barrier with very little effort in the next iOS system update, if they happened to be so inclined :).
Even if you are less generous and require the list of primes to be held in one location, it is physically possible with current high-capacity tape libraries.  Apparently an Oracle StorageTek SL8500 can hold 857 PB in a single large enclosure.  Just need to find someone who has a spare one lying around with the tapes to fill it...
Comment: Ah, this is actually interesting. So how high do you estimate we can go feasibly?
Comment: My question ignores whether it is feasible to actually compute the sum. Instead I was asking that given a powerful enough computing system should the sum be bigger than $4$ given todays knowledge?
Comment: @fretty Given that the sum **is** feasible to compute, doesn't this answer your latter question affirmatively?


Reverse order of view sort

Question: Ok so I've got this view, and it has a few dozen items. I want the last 3.
Sort ascending seems to work, however the events are in the opposite order than I want.
Sort descending pulls events back from June.
Is there any way to pull the events, but order them differently with views?
Comment: So basically you want pagination order the way it is, but reverse it inside one page?
Answer: It seems you need to have pages ordered ascending, but inside a page descending. To do that, you need to query database in ascending order. Then reverse order of one page in template that renders it.
<?php

/**
 * @file  views-view-unformatted--yourviewname.tpl.php
 * Default simple view template to display a list of rows.
 *
 * @ingroup views_templates
 */
?>
<?php if (!empty($title)): ?>

<?php print $title; ?>

<?php endif; ?>
<?php /* foreach ($rows as $id => $row): // That's how it looks by default */ ?>
<?php foreach (array_reverse($rows, TRUE) as $id => $row): /* That's what you need */ ?>
  <div<?php if ($classes_array[$id]) { print ' class="' . $classes_array[$id] .'"';  } ?>
    <?php print $row; ?>

<?php endforeach; ?>

Comment: This broke my view, which has custom styling.
Comment: @AndrewAlexander Well, here we show the way to do things. We are **not** going to make your work for you. If your styling wasn't quite ready for reversed order you asked for or if you just overrided previously overriden `.tpl` you haven't mentioned to us, it's either your concern or a matter for another question. Unless something went wrong in the exact area covered by a question - then I'll be willing to work some more on my answer.
Answer: To only show 3 items, change the settings in the "pager" section to "Display a specified number of items", and set the number of items to 3 in the settings (assuming you're using Drupal 7). Then you can use a filter of some sort to filter to the exact 3 you want using the post date or any other variable.

Reverse order of view sort


Error with typing certain problem.

Question: I need assistance in starting/solving this problem.
Any assistance would be greatly appreciated.
I know $x$ is a subset of $A$ or $B$ by the definition of a powerset.
So would this apply for this problem?
Comment: @PierreCarre may I ask you a personal question ? Is PierreCarre your real name, or just a pun on the formula for the area of a disc ?...
Comment: @TheSilverDoe, Not my real name! You got it right! In fact I even tried to change my login name to my real name but it seems that I have to wait for 20 days or so.
Comment: @PierreCarre Hahaha thanks :) I asked because I know someone whose name is really Pierre Carré ;)
Comment: I think you meant to say $P(A) + P(B) = P(A \cup B)$. Keeping in mind that $P(A \cup B) = P(A) + P(B) - P(A \cap B)$ you'll probably get to the correct answer. Also, keep in mind that you should provide some context for your posts and, more importantly, you should share you own attempts on solving the problem.
Comment: $P(A)$ denotes the set of all the subsets of $A$ ? Or a probability ?...
Comment: @TheSilverDoe its the power set of A.
Comment: @PierreCarre no this is not probability. Its dealing with sets/power sets/ subsets.
Comment: @k.rudin this is what i meant about context!
Comment: @PierreCarre I thought it was clear by saying sufficient and necessary conditions. Sorry.
Answer: Suppose first that $A \not\subset B$ and $B \not\subset A$. Then there exists $a \in A \setminus B$ and $b \in B\setminus A$. The subset $\lbrace a,b \rbrace$ of $A \cup B$ is not a subset of $A$, nor a subset of $B$. So you don't have equality.
Suppose now that $A \subset B$. Then $A \cup B = B$, so of course $P(B) \subset P(A) \cup P(B)$. Moreover, all subsets of $A$ are subset of $B$, so you get $P(A) \cup P(B) = P(A\cup B)$. It is the same if $B \subset A$.
So the condition is that $A \subset B$ or $B \subset A$.

Error with typing certain problem.


Exibir navbar apenas quando o usuário rolar para cima no iOS

Question: Estou usando a técnica de Marius Craciunoiu para exibir toda a minha navbar quando o usuário rolar a tela para cima.
Então, aqui está meu código JavaScript (usando jQuery):
var delta, didScroll, lastScrollTop, navbarHeight;

delta = 5;
lastScrollTop = 0;
navbarHeight = $('.navbar').outerHeight();

$(window).on('scroll touchmove', function() {
  didScroll = true;
});

setInterval(function() {
  if (didScroll) {
    hasScrolled();
    didScroll = false;
  }
}, 250);

function hasScrolled() {
  var scrollTop = $(this).scrollTop();

  if (Math.abs(lastScrollTop - scrollTop) <= delta) { return; }

  if (scrollTop > lastScrollTop && scrollTop > navbarHeight) {
    $('.navbar').addClass('scrolling');
  } else {
    if (scrollTop + $(window).height() < $(document).height()) {
      $('.navbar').removeClass('scrolling');
    }
  }

  lastScrollTop = scrollTop;
}

Para facilitar, eu coloquei o meu código no http://jsfiddle.net/caio/7HrK7/. Se você quiser testar no iOS, será necessário entrar por outro endereço http://fiddle.jshell.net/caio/7HrK7/show/light/.
VIDEO: http://hiperload.com/s/ua5k53n0x/d.mp4?inline
Como vocês podem ver, meu código funciona no desktop e em telefones Android, mas no iOS, é necessário inserir um evento touch para responder enquanto o evento de rolar acontece, ou então a ação só será disparada no final do movimento. Eu tentei adicionar o evento touchmove, mas sem sucesso. Vocês podem me ajudar?
Answer: Não é possível fazer isso no iOS. Há uma explicação aqui:
http://developer.apple.com/library/IOs/#documentation/AppleApplications/Reference/SafariWebContent/HandlingEvents/HandlingEvents.html

Basicamente, o vento só funciona quando inicia o toque na tela e quando para de tocar, quando você move não é realizado nenhum evento.

Para comprovar, fiz esse código abaixo em JavaScript para mostrar os segundos enquanto toca na tela, e só funciona quando inicia o toque e quando solta, se ficar segurando a tela e movendo não funciona:
var iOS = ( navigator.userAgent.match(/(iPad|iPhone|iPod)/g) ? true : false )
    if (iOS){
    document.addEventListener('touchmove', function() {
        $('#emtemporeal').html((new Date()).getSeconds());
    }, false);
}

Eu já havia passado por isso uma vez ao customizar um navbar e acabei desabilitando isso ao acessar pelo iPad, mas o motivo principal nem foi por causa desse detalhe, foi mais porque o navbar fixo ocupa no iPad um espaço considerável, visto que a tela é pequena comparada a um desktop.
Answer: Meu palpite é que você não deve usar o evento touchmove, já que o Safari só vai executar o código "depois" do ultimo step do scroll da viewport.
No video, fica bem evidente isso, então você pode tentar usar o evento touchend e touchcancel, pois eles serão executados logo após o gesto touch.


Php code-injection in telegram bot

Question: So I have this line of code that I wrote for a telegram bot, and I am wondering if I could inject some code by sending a specific type of variable to the bot.
**$this->tgMessage = htmlspecialchars($tgInput["message"]["text"], ENT_QUOTES, 'UTF-8');**

When i just do:
**$this->tgMessage = $tgInput["message"]["text"];**

I can inject everything I want, but now I don't seem to be able to.
Comment: What is your question?
Comment: I am wondering if I could inject some code by sending a specific variable to the bot
Comment: What do you mean with "injecting some code"?
Comment: before i used htmlspecialchars i could inject "); exit(1) and the script would stop. now it keeps running and just returns the input. So I am wondering if i could send a similar command with htmlspecialchars  changing the input
Comment: _"before i used htmlspecialchars i could inject "); exit(1) and the script would stop"_ - what script? Your own? Then you must be evaluating the input _data_ as _code_ somewhere.
Answer: If you just want to get rid of the html, try strip_tags() instead.
You are using htmlspecialchars(), which also escapes single and double quotes, which then leads to your problem, because intead of "); exit(); your are sending "); exit();
If you still want to use htmlspecialchars, you have to set the "ENT_NOQUOTES" flag, which is something different to ENT_QUOTES (which you have set currently).
ENT_QUOTES will convert single quotes to ' if it is set.

Php code-injection in telegram bot


Como desabilitar modo (permissão) do arquivo no GIT?

Question: Estou trabalhando com o GIT aqui na empresa onde trabalho, junto com mais dois programadores.
Toda vez que eu faço um git pull, cuja as alterações foram mandadas por esses dois programadores, sempre tenho problema com a permissão do arquivo ou da pasta que eles enviaram.
Percebi que esses dois programadores gostaram de executar os comandos do GIT depois de usar sudo su e logar com esse "Super usuário". Porém, eu não faço isso. Utilizo o meu usuário normalmente e, quando necessário, rodo sempre o comando sudo antes dos outros comandos.
E creio que isso seja o recomendado, pois alguns programas como git, composer, bower ou nodejs, geralmente recomendam a não utilização do sudo quando vamos rodá-los, justamente por conta das permissões de arquivos.
Parece-me que realmente é esse o problema. E toda vez, tenho que ficar executando o comando sudo chmod -R 777 . para resolver isso. E isso está se tornando cansativo!.
Suponhamos que eu não consiga convencer meus colegas de trabalho a parar de usar o sudo su, teria alguma maneira de desabilitar o  modo (os valores da permissão) do arquivo?
Ou seja, não quero que o git adicione mais as informações se um arquivo tem permissão de escrita ou não, e essas coisas.
Gostaria que todos os arquivos semprem fossem "reconhecidos" como 0777.
Como fazer isso no GIT? Como desabilitar esse change mode que sempre aparece quando dou um git pull?
Answer: Basta alterar a configuração core.filemode do repositório:
git config core.fileMode false

Se você quiser desabilitar isso para todos os repositórios do seu usuário, adicione a flag --global:
git config --global core.fileMode false

Comment: Muito bom, muito bom +1


Group data together and retain groupings in dataframe

Question: I have a dataframe:
ID   Deal  Party  Commodity       startdate  enddate  price    quantity  mtmvalue
---- ----- -----  ---------       ---------  -------  ------   --------  ---------
J1   Sell   J   (stock1, stock2)   01Jan23   01Feb23   10.00      10      100.00
J4   Buy    J   (stock1, stock2)   01Jan23   01Feb23   5.00       5       25.00
J2   Sell   J   (stock1, stock2)   01Jan23   01Feb23   10.00      10      100.00
J3   Buy    J   (stock1, stock2)   01Jan23   01Feb23   5.00       10      50.00

I need to group data together by Deal,Commodity, and startdate so that my dataframe looks like this:
ID   Deal  Party  Commodity       startdate  enddate  price    quantity  mtmvalue
---- ----- -----  ---------       ---------  -------  ------   --------  ---------
J1   Sell   J   (stock1, stock2)   01Jan23   01Feb23   10.00      10      100.00
J2   Sell   J   (stock1, stock2)   01Jan23   01Feb23   10.00      10      100.00
J3   Buy    J   (stock1, stock2)   01Jan23   01Feb23   5.00       10      50.00
J4   Buy    J   (stock1, stock2)   01Jan23   01Feb23   5.00       5       25.00

I am doing this which will create two groups, but I want it in one dataframe:
df.groupby(['Deal', 'Commodity', StartDate'])

How would I retain the groupings in the original dataframe?
Comment: Maybe you need just sorting instead of groupby. Do you want to do any calculations on the groupby object?
Comment: @sdawar yes I do have calculations on the groups. I calculate a result which is then merged onto the original dataframe. The only issue is that the original dataframe data is in random order and not grouped how it should be
Answer: Are you looking for sort_values?
>>> df.sort_values(['Deal', 'Commodity', 'startdate'], ascending=False)

   ID  Deal Party         Commodity startdate  enddate  price  quantity  mtmvalue
0  J1  Sell     J  (stock1, stock2)   01Jan23  01Feb23   10.0        10     100.0
2  J2  Sell     J  (stock1, stock2)   01Jan23  01Feb23   10.0        10     100.0
1  J4   Buy     J  (stock1, stock2)   01Jan23  01Feb23    5.0         5      25.0
3  J3   Buy     J  (stock1, stock2)   01Jan23  01Feb23    5.0        10      50.0

Note: There is not enough information here to correctly rank J3 ahead of J4.
Comment: For the note, those are just mock IDs. the IDs in my dataframe are long strings so the order does not matter. As long as the data is grouped  together by those columns then that will satisfy my case.


What do groups and rings "look like"?

Question: Taking undergraduate physics courses, I had to deal with Euclidean vectors often. In classes like Calc III, the concept was also there.
I'm not sure if this is why, but I've always had a more intuitive "picture" of what a vector space was than other algebraic structures. Even though in a linear algebra course, vector spaces are as arbitrary of a structure as any other, this association with a "space of scalable directed lines" stuck. It makes the concept of "dimension" of a vector space very intuitive, along with many other things.
For rings and groups, and other structures, I have no such intuition. I've heard groups compared to all sorts of things, involving symmetries, and Christmas tree ornaments. I don't see these things. I have completed graduate courses on group theory and am currently self-studying rings, but have little intuition on these things.
In other words, if I had to explain a vector space to someone with no knowledge in mathematics, I would probably go the route of explaining the three dimensional space with scalable directed lines, a very concrete example, and could do so in plain language comfortably and intuitively. If I had to explain a group, I would really have no choice but to say "a group is a set of objects endowed with a binary operation such that..."
What is your intuitive notion of these other algebraic structures? How do you "visualize" a group?
Comment: Someone once said that you might have $\Bbb{Z}/12\Bbb{Z}$ somewhere on your wall...
Comment: http://math.stackexchange.com/questions/13912/intuition-in-algebra?rq=1 might come useful.

Personally I don't think group can be visualised in 'picture' but to get a feel of how to work with it, it would help for you to see lots of examples and do exercises to see what you can do with them.
Comment: Is this a duplicate of http://math.stackexchange.com/questions/330531/algebra-best-mental-images ?
Comment: I'm sure you're going to get some answers to this, but it's my opinion that there's really very little connection to reality, and any attempt is contrived. It's a formalization of an algebraic operation which is just symbol manipulation. You should embrace that and accept that you know more math than can be intuitively explained to a layperson.
Comment: @MartinBrandenburg I don't think this is duplicate of the topic you have posted (although it does contain nice answers for it) as I don't think the level of question in your link is a lot higher whilst OP is looking for very basic intuition from a beginners point of view.
Comment: The hours on a clock face form an additive group of order 12 (normally that would be wiritten 0 - 11). Clearly that's real world everyday math. After that I've always let it get abstract in my mind, but there are plenty of "real" groups, if not rings.
Comment: Look [here](http://en.wikipedia.org/wiki/List_of_small_groups#List_of_small_abelian_groups). On the fifth column of the table you find pictures of groups up to order 30. Or [here](http://en.wikipedia.org/wiki/Cycle_graph_(algebra)) for more explanations.
Comment: Maybe relevant? https://math.stackexchange.com/questions/3932838/visualising-finite-groups-in-2d-3d
Answer: You shouldn't think of a group as a thing in the same way as a vector space is a thing.
Groups are not things, groups act on things.
If $V$ is a vector space, then the collection of invertible linear transformations $V\to V$ is a group.  If $X$ is a set, then the collection of all permutations of $X$ is a group.  If $A$ is any object, in any category, then the automorphisms of $A$?  They're a group.
Now, it turns out that many groups be visualized geometrically.  For example, the collection of rotations in $\mathbb{R}^2$ can be identified with the circle.  So you may from time to time be able to apply your understanding and intuition of geometry, vector spaces, etc., to the theory of groups, but at times you will be very surprised.
Note that every group is the symmetry group of something.  For example, any finite group $G$ is a group of permutations of some finite set, as well as a group of matrices of some finite dimensional vector space.  So it is reasonable to think of any finite group as a collection of symmetries that can be interpreted geometrically.
Comment: "Note that every group is the symmetry group of something." - in fact, $G$ is isomorphic to the group of automorphisms of the left $G$-set "$G$". This is a refinement of Cayley's Theorem (that $G$ embeds into a permutation group) and is a special case of the Yoneda Lemma. (It is said that everything is a special case of the Yoneda Lemma. ;-))
Answer: For Group theory, I can highly recommend "Visual Group Theory" by Nathan Carter. He does a great job delivering a good intuition not only of the concept of a group, but also of different group-related definitions, different groups...
(Like the title suggests; in a visual way)
http://web.bentley.edu/empl/c/ncarter/vgt/gallery.html
For Ring theory, what worked wonders for me is this thread:
https://mathoverflow.net/questions/2748/what-is-the-right-definition-of-a-ring
Answer: A very basic picture of an algebraic structure is that of just a set with "interconnections" between the elements. Imagine a simple set, an alphabet, say, of two letters, "A" and "B". Now suppose that you can combine the letters to get a bigger set by putting the letters together in any combination (I will talk about a monoid for simplicity). So, something like AAABBBAAB is in this set (the order counts). It's called a free monoid (there is also a so-called identity which is equivalent to an empty combination in our set-up). The "routes" between the words are governed by the rules of our algebraic structure. For instance, start with A, "go" to A to get AA or "go" to B to get B and so on and so forth. In a group, you also have the things like $\text{A}^{-1}$ and $\text{B}^{-1}$ with the rules $\text{A}\text{A}^{-1}=$"empty word" (the same for B). So, you can kinda "walk backwards".
In a ring, you have additional "routes" allowed by the second operation.
But I personally prefer to think of algebraic structures as being like algorithms or computer programs with data and methods. For instance, our monoid above is a program with data consisting of two values A and B along with an identity, say, E, and a method $\text{mult}$ which takes two words and simply concatenates them. This picture does not rely on sets so much, but it is not suitable for all structures of course.
Also, my personal recommendation is to try to think of categories. It gives a much better understanding than that based on set theory.

What do groups and rings "look like"?


Please Help: Utter confusion - First Variation of a function - Calculus of variations

Question: So i've been at this for most of the night. i was originally asked to find the first and second variation of the problem $$\int_{0}^{1} \sqrt{\dot{x}^{2}+\dot{y}^2}~dt$$
but at this point i'll settle with just understanding what the problem entails
so. Given a functional $J(x,u,u') = \int_{a}^{b} \Lambda(x,u,u') dx$ to find the first variation, is all we have to do simply to find the euler-legrange equation?
so would $\delta J = \int_{a}^{b} \left(\frac{\partial \Lambda}{\partial u} - \frac{d}{dx} \frac{\partial \Lambda}{\partial u'} \right) \delta u~dx$
would this be accepted as "the answer"?
then if we wish to find the second variation, would we simply have to set $$\delta J = \int_{a}^{b}\Phi(x,u,u') \delta u~dx$$ and find
$$\delta^2 J = \int_{a}^{b} \left[\frac{\partial \Phi}{\partial u} - \frac{d}{dt}\frac{\partial \Phi}{\partial u'} + \frac{d^2}{dt^2}\frac{\partial \Phi}{\partial u''} \right]\delta^{2}u~dt$$
where $\delta u$  and $\delta^2 u$ are arbitary functions (and thus do not have to be explicitly stated as $\delta u =~....$
is this all there is to it? i know there are a variety of different ways to do this, including the directional derivative $$\delta J = \left. \Lambda(u+\epsilon h) \right|_{\epsilon = 0}$$ and ive seen also $$\delta J = \Lambda(u+\epsilon h) - \Lambda(u)$$ etc...
finally if all of the above is correct... how does this work with multiple variables like the above parametric form? i would know how to derive the Euler-Legrange equations for both $\dot{x}$ and $\dot{y}$ would i just sum them in the integrand? or is there a specific format for the answer?
Thanks for the help...im kind of going nuts here.
Comment: Finding the first order variation means to keep the first order term. The result depends a bit on which class of functions you are varying. For example the Euler Lagrange formula that you quote arises when the endpoints are held fixed. Which implies that  your variations must be zero at the borders ($t=0$ and $t=1$)
Comment: Ok, so what you're saying is that SHOULD the end points be fixed at zero. then we would use the Euler-legrange equation (which makes sense), this would also explain the massive number of varying definitions and methods.

So; when asked explicitly to compute the first and second variations of say $$L(x,y) = \int_{0}^{1} \sqrt{\dot{x}(t)^{2}+\dot{y}^{2}}~dt$$ Since we've been working on the assumption that the end points vary then i should use the euler-legrange equation. IF on the other hand it was to the contrary. we would use the taylor expansion for $\epsilon$?
Answer: Let me give you a few pointers. Let's define $u=(x,y)$. You start with
$$
S(u,u') = \int_0^1 dt L(u,u')
$$
where $u'=du/dt$. To compute the first order variation we can discard quadratic terms in $\delta u $. Hence we get the following equation true up to first order
$$
\delta S = S(u+\delta u ) - S(u) = \int_0^1 dt \frac{\partial L}{\partial u} \delta u + \frac{\partial L}{\partial u'} \delta u'
$$
Now note: $\delta u' = d/dt \delta u $. So we can integrate by part the second term:
$$
\int_0^1 dt \frac{\partial L}{\partial u'} \frac{d}{dt} \delta u  = \left .   \frac{\partial L}{\partial u'}  \delta u \right |_0^1 - \int_0^1 dt \left ( \frac{d}{dt} \frac{\partial L}{\partial u'} \right ) \delta u
$$
If the path $u$ is fixed at the border (as is your case) we must have $\delta u(0) = \delta u(1)=0$ and the border term vanishes. Hence we get
$$
\delta S = \int_0^1 dt \left [ \frac{\partial L}{\partial u} - \left ( \frac{d}{dt} \frac{\partial L}{\partial u'} \right )  \right ] \delta u .
$$
This is the formula you quote (except that you called suddenly $t=x$). If you require $\delta S =0$ for all allowed variation $\delta u$ it can be shown that this implies
$$
\left [ \frac{\partial L}{\partial u} - \left ( \frac{d}{dt} \frac{\partial L}{\partial u'} \right )  \right ] =0
$$
that is, the Euler-Lagrange equations. This is sometimes called the fundamental lemma of calculus of variations.
Of course in your case $u=(x,y)$ is a vector and so you get
\begin{align}
\delta S &= \int_0^1 dt \left [ \frac{\partial L}{\partial x} - \left ( \frac{d}{dt} \frac{\partial L}{\partial x'} \right )  \right ] \delta x  \\
&+ \left [ \frac{\partial L}{\partial y} - \left ( \frac{d}{dt} \frac{\partial L}{\partial y'} \right )  \right ] \delta y
\end{align}
Again, specializing to your case
$$
\frac{\partial L}{\partial x}= \frac{\partial L}{\partial y}=0
$$
and so you obtain
$$
\delta S = - \int_0^1 dt \left [  \left ( \frac{d}{dt} \frac{\partial L}{\partial x'} \right )   \delta x + \left ( \frac{d}{dt} \frac{\partial L}{\partial y'} \right )   \delta y \right ]
$$
Since your action measures the length of the path $u$, it can be checked that the Euler-Lagrange equation gives back the equations for a geodesic in 2D, that is a straight line.
Answer: From my understanding of the problem now; and if i'm incorrect by all means someone feel free to correct me.
the parameterisation above of the arc length can be solved using the standard Euler-Legrange equation format (Summing the EL equations WRT
the explicit variables) the confusion stems over what we do with the "end points", Since in the current terms of the course the end points vanish, this essentially means that we're left with a general function in the direction of u, $$\delta u$$ before hand...now given variation at the end points the increment makes it so that we need to consider this and essentially shift the first variation resulting in (apologies for notation change)
$$\int_{x_0}^{x_1}\left[\Lambda_{u}-\frac{d}{dx} \Lambda_{u'} \right] h(x) dx +  \Lambda_{u'} \delta u \bigg\rvert^{x=x_1}_{x=x_0}+\left(\Lambda - \Lambda_u u'\right)\bigg\rvert^{x = x_1}_{x=x_0}$$
but this is apparently (as described by Gelfand and Fomin) to be the basic formula for the general variation of the functional if the end points are contrained to line on the straight line $x=x_0, x=x_1$ so this would further assume there are other formulations of the variation leading to the idea that the only sure fire way depending on the classification of problem is to do it the incremental way. ie
calculating $$\Delta \Lambda = \Lambda[u+h] - \Lambda[u]$$
hope this has helped anyone in a similair situation as i was. and thank you all for the comments


Computing pushforward measures explicitly

Question:
Between the usual measurable sets $\mathbb{R}$ and
   $S^1\subset\mathbb{R}^2$, there is a measurable mapping $$f:
 \mathbb{R}\to S^1$$ given by $f(t)=(\cos(t),\sin(t))$. The Lebesgue
   measure $\mu$ on $\mathbb{R}$ can therefore be pushforwarded to a
   mesure $\hat\mu$ on $S^1$, given by
   $$\hat\mu(A)=\mu(f^{-1}(A))=\int_{f^{-1}(A)} 1\ dt$$ for all $A\subset
 S^1$ measurable.

I was wondering if there was a possibility to express this measure as
$$\hat\mu(A)=\int_{A} ???\ d\underline x,$$
where this is the usual Lebesgue integral of $S^1\subset\mathbb{R}^2$.
If we were doing the pushforward between to measurable spaces in $\mathbb{R}^n$, this would be possible: the question marks would be a jacobian. But I don't recall that there is a change of variables formula between two different dimensions...
Comment: Does the reply below answers your question? If yes, would you accept it, if not - could you explain what is still unclear to you?
Answer: The Lebesgue measure of $S^1$ as a Borel subset of $\mathbb R^2$ is zero hence what you ask cannot be done: for every density $\varphi$, $\int\limits_{S^1}\varphi(x)\mathrm d\text{Leb}_2(x)=0$ because $\text{Leb}_2(S^1)=0$, hence one would get $\hat\mu(S^1)=0$ although you ask that $\hat\mu(S^1)=\mu(\mathbb R)=+\infty$.
Note that $\hat\mu$ as you define is such that $\hat\mu(A)$ is either $0$ or $+\infty$, for every measurable $A\subseteq S^1$:

If $\text{Leb}_1(f^{-1}(A)\cap[0,2\pi))\ne0$, then $\text{Leb}_1(f^{-1}(A))=\hat\mu(A)=+\infty$.
If $\text{Leb}_1(f^{-1}(A)\cap[0,2\pi))=0$, then $\text{Leb}_1(f^{-1}(A))=\hat\mu(A)=0$.
Comment: Did you try to compute $\hat\mu(S^1)$?
Comment: Oh, right... So there is no more explicit way to express this measure ?
Comment: The last sentence of my post characterizes (quite explicitly) the (not very interesting) measure $\hat\mu$.
Comment: Hm, I took this example from the first one of http://en.wikipedia.org/wiki/Pushforward_measure. Is this "arc-length" measure really trivial ?
Comment: It is trivial in your post because you forgot the sentence "Let λ also denote the restriction of Lebesgue measure to the interval [0, 2π)" from the WP page. Note that if you quote correctly the WP page, the first paragraph of my answer still holds, replacing the $+\infty$ which ends it by $2\pi$.
Comment: I have some trouble understanding your last paragraph (and why there is a difference with considering the measure on the whole thing or not). Why a nonempty intersection with $[0,2\pi]$ would imply an infinite Lebesgue measure ?


Helicopter shot in Cricket

Question: I wondering, who introduced the helicopter shot in cricket. MS Dhoni used to play that shot very often. If I remember correctly Sri Lankan opener Marvan Atapattu use to play similar kind of shots in his own style.
Does anybody knows who introduced that shot to the cricket world?
Answer: This Page says , Abdul Razzaq was the first player to try a helicopter shot.
But there is a similar quora question, it says Mohammad Azharuddin
 and Ajay Jadeja have tried helicopter shots back in 1996(youtube links are attached in that page, but I am unable to open it).
There might be a lot of players who have played similar kind of shots in the past.
But the name Helicopter was introduced in 2011 when Dhoni featured in a Pepsi advertisement.
I would say Dhoni was the only player who have used the shot to it's potential.

Helicopter shot in Cricket


When to use synthetic data and when to use regularization parameters to avoid the over fitting and which is better?

Question: Can anyone explain me when to consider generating the synthetic data or when to consider regularization parameters to reduce the error so the machine learning model will not overfit
Comment: Usually you use synthetic data only when you do not have real data. Usually you should be able to solve your problem without synthetic data. Using regularization parameters is one technique for some algorithms which tries to limit overfitting. One other situation where you use synthetic data (maybe this is what you are referring to) is when you have an unbalanced training set. If you only 5 positive examples and 10k negative, then your algorithm will not notice any interesting patterns. So you create synthetic positive ones in the hopes that then it will notice real distinguishing features.
Comment: If by "synthetic data" you mean SMOTE, I'd say "never".  I would however use synthetic data to induce invariances into the model (e.g. rotational invariance by rotating existing images), but regularisation is not an alternative implementation for that.
Comment: As far as I understand, we generate synthetic data for model evaluation.
Answer: If you have a problem with a time-consuming procedure for labeling, then you need to generate images synthetically for the training path. In the validation and test path, it is easier to label images since the number of inputs is few. Also, the model is robust if it validated by real images and works fine.
Comment: semi-spervised learning may be a better approach, where you give the model both labelled and unlabelled examples


Data exploration tools for Excel

Question: I have a sample of records, with string and numerical columns. My sample is currently hosted in an excel spreadsheet. I need a tool that wil produce discripte statistics for each colomn, such as max and min values, number of unique values, max string lenght. Are there any such tools available. Any tips? The ultimate goal is to derive a data dictionary.
Comment: Spreadsheets are not good models for reproducible research.
Comment: @FrankHarell Maybe so but other objectives than reproducibility might be just as important including getting results ASAP, especially if you are not going to analyze data every week. Even the fact that more people know it is a valid reason to choose it over something else (or for employers to demand it). How "reproducible" is something implemented with an R script if you are the only one knowing R and you leave your job? Bottom line is that the mere fact that Excel is so popular shows it's providing something to its users.
Comment: The funny thing is that we only ever think about making such comments when it's about Excel. Nobody goes around writing "R is too slow", "SAS is too expensive" or "Minitab does not run under Mac OS", let alone "Math is too complicated for many people" under every question on this site. It's all just as true or at the very least defensible but it's not relevant or helpful.
Comment: @FrankHarell I like Excel-bashing, R and literate programming as much as the next guy but spreadsheets help an awful lot of people actually getting stuff done. This should give pause to anybody wanting to push other tools. We don't even know if the OP is doing research at all (the data dictionary bit suggests that he might not be)…
Comment: You are taking many unnecessary risks when using Excel.  The R learning curve is worth every hour of study, if you are going to be analyzing data every week, in my humble opinion.
Comment: I have to agree with @GaëlLaurans on this; some of us aren't in a position to need or justify learning R, but we still have need of a stats tool.  Excel fills this need utilizing our existing skill set, software and data sets.
Answer: Excel has a Microsoft provided Descriptive Statistics Tool in the Data Analysis Add-in.  You'll need to activate it in your add-ins (varies based upon your version of Excel and you didn't mark which you use).
Here's a link loading the Analysis Toolpak for: Excel 2003, Excel 2007, or Excel 2010
In Excel 2010, once you've added it, you'll find Data Analysis in the Data Tab of the Ribbon.  Among the tools they provide is Descriptive Statistics, which for any given range you provide will give you: Mean,
Standard Error,
Median,
Mode,
Standard Deviation,
Sample Variance,
Kurtosis,
Skewness,
Range,
Minimum,
Maximum,
Sum,
Count
There's also quite a few other tools for you if you prefer Excel to a more specialized tool like R.
Answer: Basic descriptive stats can be computed in Excel, either with functions like AVERAGE, MAX, MIN or using a table (formerly known as a list, see http://office.microsoft.com/en-001/excel-help/overview-of-excel-tables-HA010048546.aspx).
Answer: AnswerMiner is a good option as well. You can explore your data fast and easy. There is a Relation map and table in the app that shows you the relationships (how strong they are) in your data. Automatic charts are a huge help when it comes to visualize your data.
The prediction tree is also a good feature that builds decision trees automatically in a minute and helps you make prediction based on your past data.
You can try for free here.

Data exploration tools for Excel


Guitar pickups Gibson 496r 500t difference

Question: I changed the pickups in my Gibson Flying V from the stock 496r in the neck and 500t from the bridge to signature Zakk Wylde. Now I want to change them back but I got them mixed and I dont have any manual to relate to.
How can I distinguish the 496r from 500t?
Answer: The simplest way is to count the magnets. The 500t has 3 ceramic magnets, the 496r has 2. Look inside the wrapping - there is a thick central magnet and one each side where you'd usually expect spacers.
Apparently (according to Seymour Duncan) you can also distinguish by resistance - should be around 15.5Kilohms for the 500t.
Comment: Where are the magnets located? On the bottom?
Answer: Arguably simpler yet less reliable (since it is not always the case) is to look at screws vs slugs on the pickups themselves.
The bridge pickup will have screws towards the bridge, whereas the neck pickup will have slugs towards the bridge and screws towards the neck.
Again, this isn't always the case - but with the 500T/496R it almost certainly is as Gibson pairs those as a set. Look at the Gibson logo on the back of the pickup. If the screws are on the bottom when the logo is correctly oriented - that's the bridge. Above, the neck.
There are some pickups with double screws, hex screws, etc. - but for those particular pickups this method should serve you well and saves you from mucking about with the wrapping.


$k$-regular $(k-2)$-edge connected graph with even number of vertices and no perfect matching for any even $k \gt 3$

Question: How can a graph be built such that it has an even number of vertices, and every vertex has degree $k$, but there are no perfect matchings (assuming $k$ is an even number greater than 3). It should be impossible to disconnect this graph by removing less than $k-2$ edges.
The complete graph on $k+1$ vertices works except for the fact that $K_{k+1}$ contains an even number of vertices. By taking the empty subset $S=\emptyset$, then number of odd components in $G-S$ is $1$, and $1>0=|S|$, so by Tutte's theorem, $G$ has no perfect matching.
Another thing that works when $k$ is odd is to use a variant of the 3-regular graph without perfect matchings, but this fails when $k$ is even (since then the components contain an even number of vertices).
How can a graph without perfect matchings that statifies all the conditions in the first paragraph be constructed?
Is it possible that this question was not well formulated and that it is in fact impossible to find such a graph that satisifies the condition of $|V|$ being even?
Comment: Is it even possible to construct such a graph? Or did the author of the question make a mistake in the statement? For k odd it's doable, and for k even when there is an odd number of vertices, but what about when k is even and there is an even number of vertices?
Answer: Here is an example of a $4$-regular, $2$-edge-connected graph with $30$ vertices and no perfect matching, taken from this paper:

The proof that there is no perfect matching is straightforward, Tutte's theorem style: each of the middle vertices can be matched to only one of the outer components, and the other two are left to fend for themselves with an odd number of vertices.
This picture generalizes similarly to how you generalized the bridgeless cubic construction to a $k$-regular construction for odd $k$. In fact, the above graph is slightly more complex than it has to be, because it's planar, which you didn't ask for. The generalization can just have $k-2$ vertices in the center and $k$ petals, each with a single edge to each of the vertices in the center $(k-2)$ edges total.
Then, of course, you need the petals to be $(k-2)$-edge-connected, nearly-$k$-regular graphs (with $k-2$ vertices of degree $k-1$, the remainder of degree $k$) on an odd number of vertices, but that shouldn't be hard. For example, let a petal be made up of three vertex sets $A$, $B$, $C$ of size $k-2$, $k-1$, $2$, respectively: $2k-1$ vertices in total. Each vertex in $A$ is adjacent to each vertex in $B$ - and the vertices in $A$ will also be connected to the center. Each vertex in $B$ is also adjacent to each vertex in $C$, bringing their degree up to $k$; finally, the two vertices in $C$ are adjacent to each other.
The entire graph has $k(2k-1)+(k-2) = 2k^2-2$ vertices, which is an even number.


How to execute the constructor of an Angular controller after registration (i.e. from console)?

Question: after trying the entire day here I am...
Angularjs newbie, though decent knowledge in js, I believe for some of you this question should be a no-brainer.
Suppose we have the following angular mini app:
<!DOCTYPE html>

{{firstname}}

I am able to register further controller afterwards (i.e. from the console), as follows:
// ought to overwrite the behaviour of the old one
app.controller('ctrl', function($scope) {
    $scope.firstname = "adam";
    $scope.toggle = function() {
        $scope.firstname= $scope.firstname == 'adam' ? 'eva' : 'adam';
    }
});

How on earth do I invoke programmatically the constructor of the latter controller in order to get new behaviour for the toggle function (preferable from the console)???
Edit
Accessing the scope of the controller and assign toogle to the new function does not really help in case the function refers to variables declared inside the constructor but outside the functions body.
Comment: This seems similar, inspect your dom then you should just be able to use a selector and run the toggle function http://stackoverflow.com/questions/22795628/calling-angular-controller-function-from-browser-console
Comment: Thanks for the hint. But I am not referring to the scope access of a controller. I am wondering how the constructor gets executed so that ```$scope.toggle``` is assigned to the new function. I've seen that post before btw.
Comment: I was playing around with this: `angular.element(document.querySelector('.my-class')).scope().toggle = function () { alert('ok'); }` where `my-class` would be a class for your `` tag and this overrides the toggle functionality, but I don't know if that's what you were looking for
Comment: Please check my edits. My goal actually is to execute the constructor of the snd controller
Answer: You can do something like this to assign the controller's constructor to a variable:
var ctrl = function($scope, t) {
    $scope.firstname = "adam";
    $scope.toggle = function() {
        $scope.firstname= $scope.firstname == 'adam' ? 'eva' : 'adam';
    }
    console.log("Controller was Instantiated!!!");
};

and then instantiate that using the injector as follows:
var injector = angular.element($0).injector();
injector.instantiate(ctrl, {
    $scope: injector.get("$rootScope"),
    t: injector.get("t")
});

(Assuming the service t has already been registered.)
Comment: Thats it! Thanks. Unfortunately, I cannot give you more than one upvote...
Comment: ```angular.element($0).inject()```

VM68:1 Uncaught TypeError: angular.element(...).inject is not a function     at :1:21
Comment: Try `angular.element($0).injector()` instead.
Comment: Sorry, my fault... However I still don't know how to inject methods that profits from the execution environment of a constructor with dependencies. Just think of something like:  ```app.controller('ctrl', ['$scope','d', function($scope,d) { $scope.toggle = function () {d.doSomething();} } ])```
Comment: You have to inject these manually yourself using the injector's `get()` method. I've updated my answer as an example.
Comment: Haha, no problem, pleased it helped anyway.


Remove id and rename action in urls in yii

Question: Is it possible to remove id from url and rename action name?
For example if I write this:
<code>'production' => 'products/index',
</code>
it works fine - in url I can see just "production", and is called  'products/index' page.
But if I write
<code>'audits' => '/customPages/1',
</code>
Then it gives me

The system is unable to find the requested action "1".

But as far as I understand this 1 is id , and its not an action, its a parameter in action.
How can I get '/customPages/1' page, but in url show something else?
Answer: So why do not you just try this
<code>'audits' => 'controller/action/id/1',
</code>
Suppose the controller name is SITE and action is Target so you can do this
<code>'audits' => 'site/target/id/1',
</code>
Comment: I tried, but it got `Your request is invalid`. I can acsess this page via _customPages/view/1_, but when I write rule in urlManager `'audits' => '/customPages/view/1'`, i got this incorrect request.


botocore.exceptions.ClientError: An error occurred (UnauthorizedOperation) when calling the DescribeInstances operation

Question: I am trying to filter out instances in AWS by matching particular tag added in EC2 instance. For this I have created new IAM role with following limited permissions :-
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:StopInstances"
            ],
            "Resource": [
                "arn:aws:ec2:<aws-region>:<aws-id>:instance/*"
            ],
            "Condition":{
                "StringLike":{
                    "aws:ResourceTag/Name" : ["sample_type_*"]
                }
            }
        }
    ]
}

I read about collections in boto3 and implemented following filter function to find the instances :-
ec2 = boto3.resource('ec2')
instance_name_prefix = "sample_type_"
filter_list = [
            {"Name": "tag:Name", "Values": [f"{instance_name_prefix}*"]},
            {"Name": "instance-state-name", "Values": ["running"]}
        ]
running_instance_list = ec2.instances.filter(Filters=filter_list)

But this throws me following error :-
botocore.exceptions.ClientError: An error occurred (UnauthorizedOperation) when calling the DescribeInstances operation: You are not authorized to perform this operation.

I tried to look for dependent actions for DescribeInstances in AWS docs where it is shown to be not dependent on other actions.
Can someone point out where it is wrong ? Would appreciate any hint on this.
Thanks
Comment: The docs only list `ec2:Region` as valid condition key: you cannot condition on the tag for `DescribeInstances`.
Answer: Most listing operations are either allowed or are not allowed, there is no middle ground and no Conditions that can be evaluated. Interacting with the resources that get listed is then obviously dependent on further IAM permissions, but not the listing itself.
That is the case here. ec2:DescribeInstances cannot be restricted with Conditions other than the ec2:Region.
Comment: This worked for me :-

{
            "Sid": "VisualEditor3",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances"
            ],
            "Resource": ['\*']
        },
        {
            "Action": [
                "ec2:StopInstances"
            ],
            "Resource": [
                "arn:aws:ec2:::instance/*"
            ],
            "Condition":{
                "StringEquals":{
                    "ec2:ResourceTag/" : ""
                },
            }
        },
Answer: You are not able to access your EC2 instances using Lambda function. To get the authorization you have to add one policy in your AWS Lambda function. Edit your Lambda function role and add AmazonEC2ReadOnlyAccess policy.


pyautogui in the same window

Question: I use the command pyautogui.press('enter') in a for loop that runs for many times and for a long time.
The problem is that if I run the code and I want to do something else, happens that enter is pressed in the window I'm working on and not only on the terminal as I would like.
Is there a way to run pyautogui.press('enter') only on the terminal and, in the meantime, work on other windows?
Answer: It's not possible, pyautogui manually moves the mouse so you can't be doing something else...

pyautogui in the same window


passing pointer ownership exception-safely

Question: I have a situation where passing ownership of a raw pointer may throw an exception. There are several such blocks, and it would be inelegant to wrap each in a try/catch just to delete pointers which weren't accepted. Instead I'm using a unique_ptr to manage my pointer with RAII:
unique_ptr a(new A());
obj.take(a.get()); // pass ownership to obj, might throw
a.release(); // release pointer since obj is now responsible

So the idea behind this is to keep ownership of the raw pointer in the current function until it has been successfully passed to somebody.take(). This seems like a very nice pattern to me, but is it really? Any pitfalls I'm overlooking?
Comment: Isn't it better to pass `unique_ptr` instead of raw pointer?
Comment: Well, from `obj` I want to access `a` through a raw pointer, so I do need to `release()` at some point. I know `unique_ptr` is fast, but this is really about the single cpu cycle I may be saving. Good observation though since doing the `release()` in `obj.take()` might simplify my code further ;-)
Comment: It seems that [libc++'s implemetation of `std::list` also uses this pattern](http://llvm.org/viewvc/llvm-project/libcxx/trunk/include/list?revision=245806&view=markup#l1510).
Comment: This is the [scope guard](https://en.wikibooks.org/wiki/More_C%2B%2B_Idioms/Scope_Guard) pattern, essentially.
Comment: Why not pass in the unique_ptr by rvalue reference and call release inside?
Comment: @Victor: It's extremely likely your premise is flawed -- have you actually checked the source code or profiled alternatives to verify that a raw pointer saves you *anything at all* as compared to a `unique_ptr` in regards to how quickly you can dereference it? Given that your class `delete`s the pointer upon destruction, I would expect your program to result in *identical* code whether you used a `unique_ptr` or a raw pointer. (assuming you don't compile with optimization disabled)
Comment: I haven't verified it myself, but according to [Dr. Dobb's](http://www.drdobbs.com/cpp/c11-uniqueptr/240002708), dereferencing a `unique_ptr` costs you one additional pointer dereference for the added level of indirection. Assuming by "compiled in Release mode" he means that optimization was enabled.
Answer: Alright, for anyone wondering, this is what I've done now to pass ownership of a pointer safely:
class C {
  A *mem_p = nullptr;

  /** take ownership exception-safely */
  void take(unique_ptr &&a) {
    dangerous_stuff(); // may throw exception
    this->mem_p = a.release();
  }

  ~C() { delete mem_p; }
}

void some_fun() {
  A *ptr = new A();
  C obj;
  obj.take(unique_ptr(a)); // either obj owns ptr or it is freed
}

What I like about this is that it makes the ownership semantics almost explicit, or at least it alerts any user of take() to pay attention to the  lifetime of the argument.


Calculating Wilks' $\lambda$ to test LDA result

Question: I am using DFA to analyze my data in R.
In SPSS it gives Wilks' $\lambda$ with the output, but in R I couldn't determine how to compute it. I have found code that is available to calculate the Wilks' $\lambda$ value for MANOVA.
I am new to R and it would be helpful if anyone can help me to compute $\lambda$ value for the LDA in R.
Comment: The man's name is Wilks, not Wilk. Wilks' lambda is the proportion of within-groups sum of squares to total sum of squares. Take these SS from ANOVA table for discriminant scores.
Answer: I found good information on the Wikipedia page on Linear Discriminant Analysis but the English page is not as clear as the French one where they give the formula for both Wilks' lambda and Fisher's F.
Wilks' lambda is the ratio of the determinants from the within-group covariance matrix to the total covariance matrix $\cfrac{|W|}{|V|}$, $W$ (resp. $V$) being the restriction of the within-group (resp. global) covariance matrix to the currently selected parameters.


How to use the same element name for different purposes ( in XML and DTD )?

Question: I Want to create a DTD schema for this xml document:

<root>

    <student>
        <name>
            <firstname>S1</firstname>
            <lastname>S2</lastname>
        </name>
    </student>

    <course>
        <name>CS101</name>
    </course>

</root>

as you can see , the element name in the course contains plain text ,but the element name in the student is complex type ( first-name, last-name ). The following is the DTD:
<!ELEMENT root (course|student)*>

<!ELEMENT student (name)>
<!ELEMENT name (lastname|firstname)>
<!ELEMENT firstname (#PCDATA)>
<!ELEMENT lastname (#PCDATA)>

<!ELEMENT course (name)>

When I want to validate it , I get an error because the course's name has different structure then the student's name .
My Question:

how can I make a work-around solution for this situation without changing the name of element name using DTD not xml schema .

Thanks.
Answer: Sorry! That's one of the major limitations of DTD: a given element name always has the same content model.
About all you can do is give up on full validation for that particular element, and allow all possible contents:
<!ELEMENT name (#PCDATA|lastname|firstname)*>

Comment: Thanks @bobince , I'll use what you wrote as a work-around ,it's easier then XML-schema .
Comment: When using PCDATA this is the best you can do, but if all three possibilities are elements you can get slightly better results.  Consider element ``.  Element `` may contain either element ``, or elements `` and ``.  We can specify this, it just won't be contextual (ie, dependant of ``'s parent).  ``.  PCDATA has to be used as mixed content or all by itself; it can't be an either/or situation, unfortunately.


How to make images fit the page in React/Bootstrap?

Question: I am using a function that maps every item in the props array to creating an image tag. I am trying to make every 3 images have a row div around them using bootstrap so they will fit the page correctly, but I cannot figure out how to do it. Any help would be appreciated. Here is the code:
import React, { Component } from 'react';
import "./Skills.css";

export default class Skills extends Component {
    static defaultProps = {
        images: [
            "https://upload.wikimedia.org/wikipedia/commons/thumb/6/61/HTML5_logo_and_wordmark.svg/1200px-HTML5_logo_and_wordmark.svg.png",
            "https://upload.wikimedia.org/wikipedia/commons/thumb/d/d5/CSS3_logo_and_wordmark.svg/1200px-CSS3_logo_and_wordmark.svg.png",
            "https://upload.wikimedia.org/wikipedia/commons/6/6a/JavaScript-logo.png",
            "https://p7.hiclipart.com/preview/306/37/167/node-js-javascript-web-application-express-js-computer-software-others.jpg",
            "https://bs-uploads.toptal.io/blackfish-uploads/skill_page/content/logo_file/logo/5982/express_js-161052138fa79136c0474521906b55e2.png",
            "https://webassets.mongodb.com/_com_assets/cms/mongodb_logo1-76twgcu2dm.png",
            "https://www.pngfind.com/pngs/m/430-4309307_react-js-transparent-logo-hd-png-download.png",
            "https://botw-pd.s3.amazonaws.com/styles/logo-thumbnail/s3/012015/bootstrap.png?itok=GTbtFeUj",
            "https://sass-lang.com/assets/img/styleguide/color-1c4aab2b.png"
            ]
        }
        photo() {
            return (

                    {this.props.images.map(image => (

                    ))}

            );

        }

    render() {
        return (

Skills I've Learned:

                            {this.photo()}

        )
    }
}

CodeSandbox
Answer: I think, I found the issue,

  <-----This div is the issue
    {this.props.images.map(image => (

    ))}

You have wrapped your col-md-4 with a div, and div has display: block style, so you are getting every image on separate row. Simply replace div with Fragments,
<>  <-----Make it Fragment
    {this.props.images.map(image => (

    ))}
</>


Sufficient and necessary conditions for matrix to be diagonalize or triangular

Question: There are many sufficient and necessary conditions that derive from the fact that a matrix can be diagonalize means it is similar to a diagonal matrix.
Is there a list of all of conditions?
Answer: No list of necessary and sufficient conditions for a linear operator (or equivalently, for a square matrix) to be diagonalizable will be complete, but a list of my favorites is below. In the result below, $E(\lambda, T)$ denotes the eigenspace of $T$ corresponding to $\lambda$, meaning that $E(\lambda, T)$ is the null space of $T - \lambda I$.
Theorem. Suppose $T$ is a linear map from a finite-dimensional complex vector space $V$ to $V$. Then the following are equivalent:

$T$ is diagonalizable.
There is a basis of $V$ consisting of eigenvectors of $T$.
The minimal polynomial of $T$ has no repeated roots.
There exist 1-dimensional subspaces $U_1, \dots, U_n$ of $V$, each
invariant under $T$, such that $V = U_1 \oplus \dots \oplus U_n$.
$V = E(\lambda_1, T) \oplus \cdots \oplus E(\lambda_m, T)$, where
$\lambda_1, \dots, \lambda_m$ are the distinct eigenvalues of $T$.
$\dim V = \dim E(\lambda_1, T) + \cdots + \dim E(\lambda_m, T)$,
where $\lambda_1, \dots, \lambda_m$ are the distinct eigenvalues of
$T$.
$V = \text{null } (T - \lambda I) \oplus \text{range } (T - \lambda I)$ for every $\lambda \in \mathbf{C}$.
$\text{null } (T - \lambda I)^2 = \text{null } (T - \lambda I)$ for every $\lambda \in \mathbf{C}$.
$\text{range } (T - \lambda I)^2 = \text{range } (T - \lambda I)$ for every $\lambda \in \mathbf{C}$.
Comment: Suppose the minimal polynomial of T has no repeated roots. Then every generalized eigenvector of T is an eigenvector of T. Because there is a basis of V consisting of generalized eigenvectors of T, this implies that there is a basis of V consisting of eigenvectors of T (and hence T is diagonalizable). For more on these results, see Chapter 8 of Linear Algebra Done Right, third edition, by Sheldon Axler.
Comment: where can I find a proof to 3?
Thanks


WooCommerce: Move price and subtotal under product name in cart

Question: I want to display the price and the subtotal of a cart item in the table cell of the product name. The reason is, that I need an other view on mobile phones. For desktop viewports the table cells for price and subtotal should stay. So I want to add an extra version for mobile and hide it on desktop.
I found the hook woocommerce_after_cart_item_name and I know how to add content below the product name.
Here's the code I tried: (not successful so far)
    add_filter('woocommerce_after_cart_item_name','copy_cart_product_price',10,3);
function copy_cart_product_price($cart_item, $cart_item_key){

    $cart_price_subtotal = sprintf(
        '
6,90€
',
        '
13,80€
'
    );

    $cart_item .= $cart_price_subtotal;
    return $cart_item;

}

My problem is, that I don't know how to get the price.
I guess I have to work with these two filters:
For the price:
echo apply_filters( 'woocommerce_cart_item_price', WC()->cart->get_product_price( $_product ), $cart_item, $cart_item_key ); // PHPCS: XSS ok.

For the subtotal:
echo apply_filters( 'woocommerce_cart_item_subtotal', WC()->cart->get_product_subtotal( $_product, $cart_item['quantity'] ), $cart_item, $cart_item_key ); // PHPCS: XSS ok.

But how can I show/use these filters with my snippet?
And is there a way to only show the subtotal if there are 2 or more of pieces one item in the cart?
Answer: I believe you mean this? explanation with comment added in the code
function action_woocommerce_after_cart_item_name( $cart_item, $cart_item_key ) {
    // Price
    $price = $cart_item['data']->get_price();

    // Line subtotal
    $subtotal = $cart_item['line_subtotal'];

    // Quantity
    $quantity = $cart_item['quantity'];

    echo '
' . wc_price( $price )  . '
';

    if ( $quantity >= 2 ) {
        echo '
' . wc_price( $subtotal ) . '
';
    }
}
add_action( 'woocommerce_after_cart_item_name', 'action_woocommerce_after_cart_item_name', 10, 2 );


Atomic Update in Cloud Storage APIs?

Question: I am working on an app where I would like to allow users to bring their own cloud storage account. That is, the app needs to work with the APIs for as many cloud storage services as possible.
One feature that seems important is atomic update. The app reads a version of a file from the cloud, makes some changes, then uploads a new version. However, it needs to be able to detect if a concurrent update was made to that file (for example, if the user was concurrently logged in from a different computer). The ifGenerationMatch parameter in the Google Cloud Storage API seems to make this kind of thing possible.
Does anyone know how to accomplish it with other APIs?
Answer: Sorry to answer my own question, but I think after months of intermittent digging I found at least a partial answer in the form of a couple HTTP headers:

If-Match
ETag

Atomic Update in Cloud Storage APIs?


WWW vs non WWW -- What's the best practice? I need to pick one for my new SSL Certificate

Question: Should my site be a www or non-www address? This is a pretty basic question. I wanted to see what people consider to be the best practice here. The reason I have to pick is I need to buy an SSL certificate for one or the other. Of course I can redirect to the one I pick on the back-end and the technicals are not a problem, but are there any opinions on which to pick and why?
Answer: Id answer both? Some sellers will offer both as standard. I believe Go Daddy offer the service help topic below.
http://help.godaddy.com/topic/234/article/850
Comment: Thanks for pointing this out. I think that's how all SSL certs should be. Unfortunately the brand I wanted to buy did not make this offer. This is definitely great advice for someone who has not yet chosen a specific vendor.
Answer: The host name is essentially a convenience for your users to remember the name. From this point of view, it's mainly a marketing decision. If you've promoted example.com use that, if you've promoted www.example.com use that.
Redirections on the back-end as you seem to suggest don't sound like a good idea. The point of the certificate is to let your user be able to verify the identity of your server. If you only have example.com and if they try www.example.com over HTTPS, they'll first get a warning about an invalid certificate (thus they shouldn't follow the redirection).
You may be able to get both. Some CAs issue certificates that have SANs for example.com and www.example.com (sometimes for the same price or a little bit more, I think).
Comment: Thanks for pointing out that some CAs offer both for the same price. I think that's something people looking for a vendor should look for.
Answer: My vote is for the shorter one. Comparing your question to writing for the web, and writing in general - strive for maximum salience in the fewest words possible. 4 saved bytes can come in handy with 3rd party services like Twitter.
Comment: I like this answer most because it answers my broader dilemma rather than focusing on the technical issues. The reasoning is sound and practical. Thank you.
Comment: @TomD: I think you need to do both. If you do the shorter one as suggested here then users who type `https :// www.site.com` will get the dreaded browser warning.
Comment: Hi GregS. Thank you for your comment. I disagree to an extent. This particular site just has a small secure area -- the rest is public and static. I think if I was running a bank site or a site that needs global SSL I would worry about that but honestly I don't want people who type in https: // www.example.com to put extra load on my server for no reason.
Comment: I think a wildcard SSL certificate will address Greg's concern.
Answer: SSL certs support multiple names and also support wildcard pattern matching. I would add 2 names to the cert; you don't need to choose just one.
Consider adding the two patterns site.com and *.site.com to cover you, if you need to use an additional hostname / subdomain in the future.
Please note that both are needed if you wish to access the server as site.com, as the name site.com does not match the pattern *.site.com
Comment: Right but all this costs money. If it were this simple, I wouldn't even waste time asking.
Comment: Unless you need Extended Validation services (EV) then SSL certificates are very cheap thesedays. If you are paying anything more than about $£€10-15, and price is they key factor for you, you may wish to consider switching supplier.
Comment: Where can I get a *.example.com (aka "wildcard") certificate for $£€10-15?
Comment: Sorry Tom, I was only meaning price per-name. Wildcards generally aren't cheap. But i've never really understood why, when most SSL issuers don't really do any validation thesedays.


Does COM allow closed generic types?

Question: I have created a .Net class assembly that eventually may see distribution to external customers that may want to use it from non-.Net languages, such as C++.
To this end I am trying to make it COM compatible. This being my first foray with COM I could be on completely the wrong track!
So I fired up tlbexp MyClassLibrary.dll but it produces a bunch of warnings about generic types:

TlbExp : warning TX8013117D : Type library exporter warning processing 'MyClassLibrary.IFoo.FooMethod(#0), MyClassLibrary'. Warning: Type library exporter encountered a generic type instance in a signature. Generic code may not be exported to COM.

Some are for nullable types (I have a plan to replace these with type-specific struct types maybe). The others are for occasions that I have used IList<>.
While looking for a good solution to this latter issue I stumbled on a comment by casperOne on this question: How do I suppress this COM Generics warning?, that seems to indicate that while open generic types are not allowable in COM, closed generic types should be.
All occasions I have exposed IList<> usage I have specified the type, ie they are closed generics, such as in the following example code:
public interface IFoo
{
    IList<string> FooMethod(IList<string> bar);
}

If this is correct why does my closed generic type usage cause these warnings? Have I misunterstood closed-ness? Is casperOne's comment incorrect in this regard?
Answer: So Hans Passant posted an excellent answer but then deleted it, I'm not sure why. In it he mentioned that of course the attributes applied to a generic type do not change simply because the type is specified by the user of that type. Obviously this means that closed or open, generic types are right out for COM.
The alternatives available are to create a concrete type that wraps the generic type by encapsulation, or to switch to using a non-generic collection type, such as ArrayList and be disciplined with the types I use with it.
The down-side with the latter option being that while I could have type discipline for data that is returned to the client, there is nothing to force the client to follow similar discipline in data that it passes as arguments.
The upside of the former option is that enforces the collection item type and is still as useful to both COM and non-COM clients.

Does COM allow closed generic types?


Best way to document network/servers/layouts for my replacement who isn't as tech savvy?

Question: So a quick bit of backstory...
I am 18 years old, in high school and am a sysadmin for said school. When I got here the networks were a mess and undocumented, crashing about once a day if not more. I have now brought the place up to snuff with Linux routers (Smoothwall), all Mac environment, two Mac mini's running Leopard Server and mostly new networking equipment (switches, routers, Airport Extremes etc. etc).
I am graduating at the end of this year. I have the challenge of documenting the entire network so it all doesn't come crashing down next year when I won't be around. I know the basics of using Visio and writing on wikis and such but I don't think it will be enough to get my replacement through a crisis. I have been trying to sit down with her and running through systems but there hasn't been much time or interest. She is competent with basics, upto a client level (i.e. updates, install software, etc) but doesn't understand much about the servers, the networking gear and what to do in a emergency.
So, the question is,
What is the BEST way to document a network and all critical services (network, servers, clients), well enough that my replacement can fully understand it and I won't be getting calls or emails next year.
(hoping this question won't be closed for subjectiveness because I don't know what else to do at this point :/)
Comment: Sounds like an employment opportunity.  :)
Comment: you mean employment for you or for me?
Answer: I think you may be trying to do the impossible.
The scope of what you seem to want to do, document everything to a point that someone who doesn't understand system administration can replace you, is not really possible.  Realistically you would have to basically re-write tons of books and documentation that already exists.
If what you want to do was possible, I am sure someone else would have done it a long time ago and they would no longer be paying people to be system administrators.
The fact is that networks can get very complex and the only way you can solve many complex problems is understanding the system as a whole.  Usually a person only gains this level of understanding by being interested enough in the topic to do lots of reading and research.
If the person you are training is not interested in putting in the effort to learn, then I don't think you will be able to help them they way you want to.  A simple cheat sheet and trouble shooting guide cannot replace learning and understanding how things work.
Personally, I think you need to help the school find a good person in the area they can go to when they need help with networking issues.  Look around for local network consultants.  If you are interested you can offer to work for them.

Update to address comments

The network is stable as-is. I just
  need to make sure she knows how to do
  very basics like troubleshooting and
  restarts and stuff like that.

Please do not reinvent the wheel, or in your case try to re-write the manuals for everything on your network.  Instead point at resource where someone can go learn.  Including links to sites like serverfault.com.

Your troubleshooting rules, approach to troubleshooting?
What's your checklist for when everything blows up?
What a beginner should know/learn for sysadmin job?
Cheat Sheets for System Administrators?
Etiquette of Troubleshooting Problems In The Workspaces Of Others

Good documentation should not repeat things that is in the manuals or standard reference material for a product.  For example, do not tell someone how to set an ACL on the filesystem.  Simply describe why the ACLs are set a specific way in your environment.

Our budget is already so tight that
  evening bringing up the idea of
  another person would cause outrage.

I understand that budgets in K12 are very tight, but it is not your responsibility to provide them support for free for your entire life.  It is also not your responsibility to completely train a person who does not have the interests or aptitude to fill your shoes.  Since budget is an issue, you may want to check out any local tech/linux user groups.  I know several members of the local LUG up here volunteer support time to charities/schools/non-profits.
Comment: Agreed. Your time would be much better spent helping the school find a proper replacement, instead of trying to train someone from the ground up. That's not the type of person that should be put in charge of an infrastructure.
Comment: Wow thanks for the answer so quick! A few points I hope you might be able to expand on...

1. The network is stable as-is. I just need to make sure she knows how to do very basics like troubleshooting and restarts and stuff like that.
2. They are interested but theres just no time at the moment and I am worried about running out of time in the school year
3. Our budget is already so tight that evening bringing up the idea of another person would cause outrage.
Answer: I disagree with this being an impossible task.
The first place I would start?  A Visio (or alternative) of your network.  Diagramming something as basic as what's connect to what, will go a long way to helping your replacement understand the fundamentals.  I would start here..because everything else that you might have time for, will build off of this.  If you don't have time left after the diagram, then you'll be handing her the most important ONE piece of information that you could be handing her.
Once the diagram is complete, move into creating a really basic desk manual.  Perhaps start with a page for each server and bullet-pointing things like IP address, name, function, warranty info, phone numbers, os version, etc. and do this for each server.  If you have time to complete the basics for each server, then move into network devices, etc.  Once the basics are covered for each 'appliance', then delve into more in-depth information for each appliance (why it's named what it is, plans for that server, etc.)
If time allows for anything after this, drive to your Secretary of State, create an LLC for yourself, get business cards, determine your hourly or per job rate, and staple your business card to the bottom of the visio diagram and desk manual.
Comment: If you have 1000+ pages of documentation already and you are still not confident in this person's abilities, you really need to get someone else. I'm not saying hire an Additional person, I'm saying hire a *different* person.
Comment: thanks for the advice! I already have a basic "desk manual" (1000+ pages) built but she never uses it. and, nice last paragraph there :D
Comment: Well then I may have to strike out my first sentence...you cannot help someone who doesn't want to help themselves.  Your best bet is really to form a consulting business and go back on a per hour charge.
Comment: I was tempted to downvote this initially, but I think you've come around to the same conclusion as everyone else. Documenting the network so another SA can figure it out it easy (time consuming, but easy). Documenting it so a disinterested user could do administration is borderline impossible.
Answer: General guideline for non-tech savvy replacement.  Lots of Pictures, lots of troubleshooting steps, don't bother with technical explanations because when they see them all they'll read is "call the old guy".
Keep it simple, and make sure they know how to back up someone's data and re-image/install their PC.
Comment: And don't forget to label everything. It should look just a little like the Batcave in the old Batman TV series, just on the off chance you've ever seen an episode.
Comment: Pictures work very well with less tech-savvy types.  Telling them to power cycle the switch is scary, telling them to find the blue box in the picture and shut it off for 30 seconds works great.
Answer: This is a case where training your replacement isn't really your job; you'll do plenty of that later in the real world.  But good for you for wanting to do a good job.  From now on, any problem that comes up, give it to your replacement to handle.  Help her as needed, explaining why you do such and such to fix or research the problem.  That's called on the job training.  Walk her through doing backups (you do backups, right?), complete system shutdown and startup, admin screens for the routers, etc.  You've only got two servers, it's not NASA.  It's learnable and more importantly for you, teachable.  Teaching is your role now.  Also, get a label maker and label every piece of networked hardware.
Answer: I would spend some time finding a competent contractor who could handle anything your replacement cannot, and ensure your replacement knows how to contact them.  Then spend a bit of time with the contractor to ensure they know how you've set things up.  Set up some documentation specifically for them (or another knowledgeable party)
Then I would proceed to document all the basic stuff, so that your replacement can handle the day-to-day issues, and knows when it's time to call in the "big-guns", and who they are.
Answer: How about put your replacement in the 'hot seat' from now, let faults come through and see how she goes about trying to fix the problem? You will still be around for any major major emergencies.
People often get into gear when thrown in the deep end. If your replacement fails without making any (learning) progress, it will tell them too that they should maybe not take on the role.
Your replacement can then take notes as they need to and ask you questions about what they need to know.


Probability curve for glmm

Question: I have some rather basic questions that I suspect I know the answer to but would like to confirm.
I have a simple experiment measuring the number of a certain insect growth stage in the proportional presence of a dietary treatment. I have modeled the data using a binomial glmm. I would now like to plot a probability curve of the fixed effect (Treatment) using the sjPlot package. Following this tutorial, I am using "fe.slope" as the plot type. However, in the tutorial the y-axis is numbered from 0 - 1, as befits a probability curve, whereas my y-axis has a proportional scale from 0% - 100%. I assume it has something to do with the proportional aspect of my glmm. Can I simply  change the y-axis scale manually to go from 0 - 1 without affecting the nature of the plot?
Further, my Treatments are named 'T1, T2, T3, T4 and T5', yet the x-axis is labelled 0 - 4. Can I also manually change these without affecting the plot's true meaning?
Finally, what do the points on this plot represent? Are they residuals?
Plot:

Code:
library(lme4)
library(sjPlot)

df<-as.data.frame(df)

df$OLRE <- seq_len(nrow(df)) #create observation level random effect

df[,2:3] %<>% lapply(function(x) as.integer(x)) #change to integers

m1<-glmer(Stage/Total~Treatment + (1|OLRE), data=df, family='binomial',weights=Total)

sjp.glmer(m1, type = "fe.slope",show.ci=TRUE)

Data:
dput(df)
structure(list(Treatment = c("0_diatoms", "0_diatoms", "0_diatoms",
"25_diatoms", "25_diatoms", "25_diatoms", "50_diatoms", "50_diatoms",
"50_diatoms", "75_diatoms", "75_diatoms", "75_diatoms", "100_diatoms",
"100_diatoms", "100_diatoms"), Stage = c(45L, 50L, 50L, 58L,
40L, 72L, 31L, 45L, 40L, 63L, 61L, 50L, 23L, 33L, 30L), Total = c(100,
100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100,
100), OLRE = 1:15), .Names = c("Treatment", "Stage", "Total",
"OLRE"), row.names = c(NA, -15L), class = "data.frame")

Comment: Why do you divide Stage by Total when deriving your outcome variable and also including Total in the weights= option of your model? In other words, what is it that you are trying to model? What does Stage represent, what does Total represent and what does OLRE represent?
Comment: Hi @IsabellaGhement, thanks for the comment. See [this post](https://stats.stackexchange.com/questions/189115/fitting-a-binomial-glmm-glmer-to-a-response-variable-that-is-a-proportion-or-f), which has the same design as mine. OLRE refers to observation level random effect.
Answer: The plot you produced doesn't make sense for your data, since Treatment is supposed to be a categorical variable (not an integer variable as per your plot).
I would re-fit the model by forcing Treatment to be a factor.  I think OLRE should be a factor too - usually, the random effect goes hand in hand with a grouping variable (i.e., a factor in R's parlance).
So maybe try something like this:
df$Outcome <- df$Stage/df$Total

df$Treatment <- factor(df$Treatment)

df$OLRE <- factor(df$OLRE)

m1 <- glmer(Outcome ~ Treatment + (1|OLRE), data=df,
            family='binomial',weights=Total)

set.seed(17)

p1 <- sjp.glmer(m1, type = "pred.fe", vars = c("Treatment"),
                show.ci = FALSE, show.scatter=FALSE)
p1

p2 <- sjp.glmer(m1, type = "pred", vars = c("Treatment"),
                show.ci = FALSE, show.scatter=FALSE)
p2

According to the help file for sjp.glmer, which is available at https://www.rdocumentation.org/packages/sjPlot/versions/2.4.0/topics/sjp.glmer, the options type = "pred.fe" and "type = "pred" will help plot the predicted values against the response, with the predicted values being derived only from the fixed effects (for type = "pred.fe") or conditional on random intercept (for "type = "pred").
The beauty with saving the two plots is that you can then inspect the output produced by R when typing the plot names and "steal" that output to create your own customized graph.  For example, the output for the plot p1 is as follows:
> p1
$data
x            y ci.low ci.high resp.y grp
1  1 0.4832236     NA      NA   0.45   1
2  1 0.4832236     NA      NA   0.50   1
3  1 0.4832236     NA      NA   0.50   1
4  3 0.5684214     NA      NA   0.58   1
5  3 0.5684214     NA      NA   0.40   1
6  3 0.5684214     NA      NA   0.72   1
7  4 0.3855458     NA      NA   0.31   1
8  4 0.3855458     NA      NA   0.45   1
9  4 0.3855458     NA      NA   0.40   1
10 5 0.5807438     NA      NA   0.63   1
11 5 0.5807438     NA      NA   0.61   1
12 5 0.5807438     NA      NA   0.50   1
13 2 0.2850919     NA      NA   0.23   1
14 2 0.2850919     NA      NA   0.33   1
15 2 0.2850919     NA      NA   0.30   1

$plot

attr(,"class")
[1] "sjPlot"        "sjpglm.ppresp"
Warning message:
In eval(family$initialize) : non-integer #successes in a binomial glm!

So the component p1$data of the plot gives you access to the data used to create the plot.
For example, for your first treatment,0_diatoms, you can compute the predicted probability by hand as follows and compare it against the one produced by sjp.glmer for type = "pred.fe":
intercept <- summary(m1)$coeff["(Intercept)","Estimate"]

exp(intercept)/(1 + exp(intercept))

The probability comes out to be 0.4832236 and this is listed under p1$data (in the column titled y) for each of the three observations corresponding to the first treatment group (i.e., 0_diatoms).  The observations themselves are listed under the column titled resp.y of p1$data.
For the second treatment, the predicted probability computed by hand would be:
intercept <- summary(m1)$coeff["(Intercept)","Estimate"]

slope25_diatoms <- summary(m1)$coeff["Treatment25_diatoms","Estimate"]

exp(intercept + slope25_diatoms)/(1 + exp(intercept + slope25_diatoms))

hence  0.5684214.  This is listed under the column y in p$data for each of the observations in your 25_diatoms treatment group.
The predicted probabilities for the first two treatment groups that are produced by sjp.glmer with the option type="pred" can be replicated by hand with the commands below:
intercept <- summary(m1)$coeff["(Intercept)","Estimate"]

ranef(m1)$OLRE[1, ]
ranef(m1)$OLRE[2, ]
ranef(m1)$OLRE[3, ]

exp(intercept + ranef(m1)$OLRE[1, ])/
  (1 + exp(intercept + ranef(m1)$OLRE[1, ]))

exp(intercept + ranef(m1)$OLRE[2, ])/
  (1 + exp(intercept + ranef(m1)$OLRE[2, ]))

exp(intercept + ranef(m1)$OLRE[3, ])/
  (1 + exp(intercept + ranef(m1)$OLRE[3, ]))

and
intercept <- summary(m1)$coeff["(Intercept)","Estimate"]

slope25_diatoms <- summary(m1)$coeff["Treatment25_diatoms","Estimate"]

exp(intercept + slope25_diatoms + ranef(m1)$OLRE[4, ])/
 (1 + exp(intercept + slope25_diatoms + ranef(m1)$OLRE[4, ]))

exp(intercept + slope25_diatoms + ranef(m1)$OLRE[5, ])/
 (1 + exp(intercept + slope25_diatoms + ranef(m1)$OLRE[5, ]))

exp(intercept + slope25_diatoms + ranef(m1)$OLRE[6, ])/
 (1 + exp(intercept + slope25_diatoms + ranef(m1)$OLRE[6, ]))

These probabilities come out to be 0.4646612,  0.492606 and 0.492606 for the first treatment (i.e., 0_diatoms) and 0.5748393, 0.4744047 and 0.6512543 for the second treatment (i.e., 25_diatoms).  You will recognize then in the first 6 rows of column y contained in the R output corresponding to p2$data:
> p2$data
x            y ci.low ci.high resp.y grp
1  1 0.4646612     NA      NA   0.45   1
2  1 0.4926060     NA      NA   0.50   1
3  1 0.4926060     NA      NA   0.50   1
4  3 0.5748393     NA      NA   0.58   1
5  3 0.4744047     NA      NA   0.40   1
6  3 0.6512543     NA      NA   0.72   1
7  4 0.3447117     NA      NA   0.31   1
8  4 0.4209869     NA      NA   0.45   1
9  4 0.3934527     NA      NA   0.40   1
10 5 0.6078542     NA      NA   0.63   1
11 5 0.5968777     NA      NA   0.61   1
12 5 0.5358618     NA      NA   0.50   1
13 2 0.2574995     NA      NA   0.23   1
14 2 0.3081921     NA      NA   0.33   1
15 2 0.2927040     NA      NA   0.30   1

The above should provide enough insight into what sjp.glmer is actually plotting to enable you to generate your own plots.  In particular, you can plot the predicted probabilities on the 0 to 1 scale (rather then converting them to percentage points, as sjp.glmer does).  You can also force sjp.glmer to return uncertainty intervals around the predicted probability by using the show.ci = TRUE option.
I am not sure why sjp.glmer jitters the response values in the column resp.y before plotting them when you include the option show.scatter=TRUE. I would have thought that using show.scatter=FALSE would have turned off the random jittering, but it actually removes the response observations altogether from the plot. So either use set.seed(17) (or some other random seed) before creating the plot and then invoke the option show.scatter=TRUE for sjp.glmer, or create your own plot from p1$data or p2$data (whichever you need), but superimpose the values stored in y.resp on your plot, which are the actual response values.  If you don't set the random seed, sjp.glmer will come up with a random configuration of jittered response values every time you invoke it for the same model.
Disclaimer: I do not know enough about your study, data and model to determine if your current modeling approach is sensible. Assuming it is, the code I provided here will make sense. Otherwise, it might not make sense.
Comment: You're welcome! I hope it gave you enough background info to enable you to get the plot you want.
Comment: That's great! There's nothing like understanding what it is that you are plotting!
Comment: Thank you so much for taking the time to write this very comprehensive answer.
Comment: Yes, definitely. I can now confirm if my plots are correct using the manual procedures you provided. I have now 'stolen' the info and made a graph with `ggplot()`!

Probability curve for glmm


Translate Matlab PLS Regression code to Python

Question:
Matlab
I have some PLSR regression in Matlab that I need to translate to Python. The Matlab code is as follows:
% PLSR with 15 latent variables
[~,~,~,~,~,MSEcv] = plsregress(X,y,15,"cv",5,"MCReps",100);
rRMSE = 100*sqrt(MSEcv(2,:))/max(y)-min(y));
% Find optimal number of latent variables
minl = 1; maxl = 15
nlv = find(rMSE(minl+1:maxl+1)==min(rRMSE(minl+1:maxl+1)))-1+minl;
nlv = nlv(1)
% Fit with optimal number of latent variables
[XL,yl,XS,YS,beta,PCTVAR;MSE;stats] = plsregress(X,y,nlv)

(PLS regression with a maximum of 15 latent variables, cross-validation with 1/5 of the values and 100 Monte-Carlo repetitions.)

Python
I tried to implement the code in Python as follows:
import numpy as np
from sklearn.cross_decomposition import PLSRegression
from sklearn.model_selection import cross_validate
from sklearn.model_selection import ShuffleSplit

# Maximum number of latent variables
mlv = 15
# Cross-validation
cv = 5
# Monte-Carlo repetitions
mcr = 100

# 1...mlv to fit models with various number of latent variables
try_latent_vars = np.arange(1, mlv)

##----------------------------------------------------------------------------|
# Define funtion to fit PLSmodel
def optimise_pls_cv(X_vals, y_vals, n_comp, crossval, mcreps):
    '''Fit PLS regression model using cross-validation.'''
    # Define PLS object
    pls = PLSRegression(n_components = n_comp)

    # Cross-validation fit
    cv_split = ShuffleSplit(n_splits = mcreps, test_size = 1/crossval,
                            random_state = 0)
    cvs = cross_validate(pls, X_vals, y_vals, cv = cv_split,
                         scoring = ["r2", "neg_mean_squared_error"])
    mean_r2_error = np.mean(cvs["test_r2"])
    test_mse = -np.mean(cvs["test_neg_mean_squared_error"])
    return pls, mean_r2_error, test_mse

##----------------------------------------------------------------------------|
# Fit PLS model

# Empty lists to store R^2 and mean squared error values
r2s = []
mses = []

for n_comp in try_latent_vars:
    model, r2, mse = optimise_pls_cv(X_vals = X.T,
                                     y_vals = y,
                                     n_comp = n_comp,
                                     crossval = cv,
                                     mcreps = mcr)
    r2s.append(r2)
    mses.append(mse)

index_max_r2s = np.argmax(r2s)
lv = try_latent_vars[index_max_r2s]

##----------------------------------------------------------------------------|
## Fit model with optimized number of components
model, r2, mse = optimise_pls_cv(X_vals = X.T,
                                 y_vals = y,
                                 n_comp = lv,
                                 crossval = cv,
                                 mcreps = mcr)
metrics = {"R2" : r2,
           "MSE" : mse}
metrics_str = "R2: %0.4f, MSE: %0.4f" % (r2, mse)

In both code snippets, X and y are a set of spectral information and some variable to predict from the spectra, respectively.

The problem
Unfortunately, I get vastly different results with those versions (R^2 values for the Matlab version are all 0.5 or higher while the Python version ends up with R^2 values close to zero or negative). How is that and how can I solve this issue?

Example data
Here is some example data as .txt files in Dropbox. Read as
import pandas as pd
X = pd.read_table("/path/to/saved/files/X.txt", header = None)
y = pd.read_table("/path/to/saved/files/y.txt", header = None)

Answer: I found that sklearn uses the NIPALS algorithm while Matlab's plsregress uses the SIMPLS algorithm to estimate PLS regression parameters. For some of my data, this does not make a huge difference, whereas I found the NIPALS version to produce output of highly variable quality for slight variation of the used sample spectra (e.g., outlier removal or continuum removal may have a huge impact for some variables).