What is the importance of practicing scales on piano, and the proper fingering of them?

Question: First of all I want to point out that I practice scales a lot and even enjoy it.
I just don't know why they are so important, as this site says for example:

"Proper fingering of the scales on the piano is very important to development of your skills and advancement as a pianist."

When you are playing a piece you are not doing anything remotely like playing a scale.
Maybe practicing scales is important to memorize them, and the proper fingering is important to improve dexterity. Is there a better answer?
Comment: in fact, I would say the *majority* of melodic fragments are either scale fragments or arpeggio fragments.
Comment: I beg to differ with "you are not doing something remotely like a scale."  Many songs have scales in them, and many things are similar.
Answer: Memorizing your scales accomplishes at least these four things:

Trains your fingers to play common patterns found in music. There are a lot of scales in music. They're just so satisfying, why not write them?
They can be a controlled environment for practicing other techniques, such as playing fast, playing in octaves, and playing fast in octaves.
It trains your ear to hear the qualities of a key and the different scale degrees within that key. When learning a new piece, practice scales for the keys within that piece. You'll catch your own mistakes more often since you'll be accustomed to tonal qualities of the keys in that piece.
It also trains you to play smoothly in time. Practice speed, but maintaining a steady tempo with a variety of articulations. You could also use them to experiment with rubato (credit to Matthew Read in the comments).

[EDIT]
Adding another good point from the comments:

In addition to @bearcdp's well written answer, I'd like to point out that almost every piano-teaching book I've read, strongly emphasizes the importance of playing a song/riff in all of the 12 keys!. This is annoying in the beginning, but it helps you to get equally familiar with all the different keys (and not just C major and A minor). –  @Saebekassebil
Comment: I use scales for practicing styles too. You haven't lived until you swing a F# major scale!
Comment: @bearcdp: great answer, I just don't get the end of point 1, "why not write them?" ?
Comment: @idober: The third movement of Mozart's flute concerto in G has scales everywhere. D and G are the ones I remember for sure. I think there's a harmonic minor in there too, but I can't say for sure.
Comment: @idober That sentence is a little weird. I mean "why not write them" to be from the composer's perspective. Scales feel satisfying to play in a piece (speaking for myself at least), so composers can take an opportunity to indulge the performer by throwing in a scale at appropriate moments. Oh and @Michael, +1 for F# major, gliding over black keys is so satisfying.
Comment: @bearcdp: Not to forget the two white keys, B and E# ;)
Comment: In addition to @bearcdp's well written answer, I'd like to point out that almost *every* piano-teaching book I've read, strongly emphasizes the importance of playing a song/riff *in all of the 12 keys!*. This is annoying in the beginning, but it helps you to get equally familiar with all the different keys (and not just C major and A minor).
Answer: There are a ton of scales in classical music.  They tend to be hidden, though, with only a few notes at a time.  Take, for example, Mozart's Rondo alla Turca: It's main theme has a snippet of a scale, and its middle section is almost entirely made up of scales.  The 3rd movement of Beethoven's Moonlight sonata has brief runs to build up tension.  Bach's inventions all have scales in the music.  Just because they just don't go on for octaves and octaves doesn't keep them from being scales.
As far as fingering, it just makes scales really easy to play.  The fingerings are optimized so that you can go really really fast without tripping over your fingers.  There is a small aspect of training dexterity--being able to perform cross-overs smoothly is extremely important--but it's secondary to the goal of allowing you to recognize patterns in music.
Of course, everything bearcdp said counts, especially hearing the qualities of the scale degrees in the key.  Knowing your scales and the sounds of each degree makes learning every style of western music infinitely easier.
Answer: Practicing things like scales allows you to augment your playing skill with muscle memory.
This is especially evident and useful when embellishing or improvising. I fool around with keyboards a bit, but I know My c and F blues scales well.  I can improvise pretty convincingly in related keys, and without thinking about the particular notes or which keys I am hitting.
Being practiced in scales, which is als being practiced in intervals, is like knowing how all the words you read are supposed to sound, rather than having to sound them out.
Comment: Good analogy. "Being practiced in scales ... is like knowing how all the words you read are supposed to sound, rather than having to sound them out."
Answer: I think the main advantage comes that when you have fingerings cold for whole scales, you are quite less likely to "paint yourself into a corner" with the fingerings when sightreading.  For example, if you play an upward scale part with the right hand and utilize the pinky, then you tend to have a problem if the melody goes on.  Practising scales will train you for fingering sequences that are not as "natural" all the time but where you don't run out of fingers.
It's like practising to eat everything with fork and knife instead of specific eating tools for each different food texture.  Though I actually prefer chopsticks for everything.
Answer: It also teaches the muscles. Muscle memory is an integral part of learning to play an instrument. The novel thing about your muscles is that you can teach them and they can learn but they have no intelligence and will learn bad things if taught badly. You have to approach teaching them like you would teaching a child that is both deaf and blind.
So the scales impart the knowledge in your hands so that when you want to play the music you are able to recall where the notes should be.
It is almost like a golfer practicing his swing. The way in which there muscles are taught is very much the same.


How to set access arguments visible to all

Question: i have a simple module which show user to set email frequency but it's content can be seen by only to logged admin users , i want to change it to all the users,in drupal how can i do that
function emailf_menu() {
  $items = array();
  $items['emailfrequency'] = array(
    'title' => 'Manage email Frequency',
    'page callback' => 'emailf_page',
    'access arguments' => array('view content')
  );
  return $items;
}

Answer: function emailf_menu() {
  $items = array();
  $items['emailfrequency'] = array(
    'title' => 'Manage email Frequency',
    'page callback' => 'emailf_page',
    'access arguments' => TRUE
  );
  return $items;
}

'acces arguments' => TRUE will allow access to all users, including anonymous
Comment: I want to restrict the page to only admin !! how to do it
Comment: put in a permission that only admin has, like 'access arguments' => array('administer content') or 'access arguments' => array('administer site configuration')
Answer: The 'access arguments' in your hook_menu is an array which has the permission 'view content'; which means that this menu would be accessible only to the users who could view published content. If you want your menu to be accessible to all the easiest way is to use the 'access callback' instead:
function emailf_menu() {
  $items = array();
  $items['emailfrequency'] = array(
    'title' => 'Manage email Frequency',
    'page callback' => 'emailf_page',
    'access callback' => TRUE,
  );
  return $items;
}


Prove $\tan(A+B)$ using $\cos(A-B)$ and $\sin(A-B)$

Question: Use $\cos(A-B)$ and $\sin(A-B)$ to prove $$\tan(A+B)=\frac{\tan{A}+\tan{B}}{1-\tan{A}\tan{B}}$$
It seems like we cannot simply change $A+B$ to $A+(-B)$ to prove it? Any ideas?
Answer: $$\begin{align*}\tan (A+B) = \frac{\sin (A+B)}{\cos (A+B)} &= \frac{\sin A\cos B + \sin B\cos A}{\cos A\cos B - \sin A\sin B} \div \frac{\cos A\cos B}{\cos A\cos B} \\ & = \frac{\frac{\sin A}{\cos A} + \frac{\sin B}{\cos B}}{1 - \frac{\sin A\sin B}{\cos A\cos B}} \\ & = \frac{\tan A + \tan B}{1 - \tan A \tan B}\end{align*}$$

Note that if you know $\sin (A-B)$ then use $B \to -B$ to get $\sin (A+B)$ with the usual odd sine and even cosine, same goes for $\cos(A-B)$.
Comment: @Z You may want to change that $\;\div\;$ sign by a $\;\cdot\;$ sign. It is the same, of course, but perhaps a little clearer. +1


Criptografar uma frase/palavra invertendo e trocando as vogais, pelas vogais seguintes

Question: Foi me passado um exercício no qual eu tenho que escrever um programa que leia uma palavra no intervalo [1-20] e fazer a criptografia nesta palavra utilizando os passos:

inverter a palavra
onde tem vogal, substituir pela vogal seguinte

E exiba a palavra criptografada.
Sei fazer a parte de inverter a palavra, mas substituir as vogais já tentei de várias maneiras e não consegui. Segue o que consegui fazer até então:
package ExerciciosPackage02;
import java.util.Scanner;

public class ExerciciosClass01h {
    static Scanner ler=new Scanner(System.in);
    static String palavra;
    public static void lerpalavra(int ini, int fim){
        System.out.println("Digite uma palavra:");
        do{
            System.out.print("-> ");
            palavra=ler.nextLine();
        }while ((palavra.length()<ini)||(palavra.length()>fim));
    }
    public static void inverter(){
        int ctcar;
        System.out.print("Criptografia: ");
        for(ctcar=palavra.length()-1;ctcar>=0;ctcar--){
            System.out.print(palavra.charAt(ctcar));
        }
    }
    public static void invertvogal(){
        //int ctcar;
        //System.out.println(palavra.replaceAll("a", "e"));
        //for(ctcar=palavra.length()-1;ctcar>=0;ctcar--){
        //}
    }
    public static void main(String[] args) {
        // TODO Auto-generated method stub
        lerpalavra(1,20);
        inverter();
        invertvogal();
    }

}

Comment: Este intervalo de 1 a 20 é o tamanho da palavra?
Comment: isso, por isso coloquei aquele do while
Comment: Vc tem algum exemplo de como deve ficar a saída final do programa? Exemplo, se eu alimentar com a palavra `abcei`, que tipo de saída ele deveria gerar?
Answer: Isso é apenas uma pseudo criptografia. O código tem alguns problemas e está um pouco confuso. Modifiquei para ficar mais organizado:
import java.util.Scanner;

class ExerciciosClass01h {
    public static String lerpalavra(int ini, int fim) {
        Scanner ler = new Scanner(System.in);
        System.out.println("Digite uma palavra:");
        String palavra;
        do {
            System.out.print("-> ");
            palavra = ler.nextLine();
        } while (palavra.length() < ini || palavra.length() > fim);
        return palavra;
    }
    public static String criptografar(String palavra) {
        String invertida = "";
        for (int ctcar = palavra.length() - 1; ctcar >= 0; ctcar--) {
             invertida += trocaVogal(palavra.charAt(ctcar));
        }
        return invertida;
    }
    public static char trocaVogal(char letra) {
        String vogais = "AaEeIiOoUu";
        int posicao = vogais.indexOf(letra);
        if (posicao == -1) {
            return letra;
        }
        return vogais.charAt(posicao + 2);
    }
    public static void main(String[] args) {
        System.out.println(criptografar(lerpalavra(1, 20)));
    }
}

Veja funcionando no ideone. E no repl.it. Também coloquei no GitHub para referência futura.
Note que fiz os métodos serem autocontidos, o que é o ideal a não ser quando realmente há necessidade diferente. Também separei lógica de interface com usuário (entra ou saída de dados) da lógica de criptografia. Dá pra fazer algumas melhorias cosméticas na interface.
Por se tratar de uma operação relativamente complexa e claramente diferente do resto separei a troca da vogal em um método próprio.
Tem algumas coisas que poderiam ser melhores, mas para um código simples não vale a pena. Um exemplo seria a concatenação dos caracteres serem feitos com uma StringBuiler por razões de performance. Mas não vamos complicar esse exemplo que não depende da melhor performance possível.
Outras lógicas poderiam ter sido feitas para trocar a vogal, talvez até melhores, mas essa foi a primeira que pensei.
Ela tenta achar se o caractere que está sendo analisado é uma vogal (maiúscula ou minúscula). Se não for, basta retorna o caractere. Se for ele pega a próxima vogal (soma 2 porque as maiúsculas e minúsculas estão intercaladas. Há uma exceção quando a vogal for U ou u, aí, imagino eu que, pega a primeira, foi o que eu fiz.
Note que não me preocupei com acentos e outras peculiaridades, acho que esta simplificação não vai trazer problema em um exercício. Provavelmente é desejável que seja simples mesmo.
De uma maneira geral o código está mais simples, sem redundâncias e mais legível, até pelo melhor uso do espaçamento. Eu poderia ter melhorado os nomes das variáveis, mas preferi não mexer nisso.
Comment: A parte do título que fala "criptografia" me faz lembrar de uma empresa que u trabalhava onde os sistemas de login de administração eram todos "criptografados" utilizando a seguinte lógica: `4dm1n1str4c40` , `suc3ss0` e por aí vai rs, será que não seria interessante mudar o título da pergunta para "Alterar um texto/frase..."
Comment: Esse título poderia gerar má interpretação pra futuros usuários rs
Comment: @MarceloBonifazio não sei se é necessário, era o que ele queria perguntar. Talvez outras pessoas procurem pelo o mesmo. As pessoas usam termos errados para as coisas, se tirar isso pode ficar mais difícil achar, ainda que seja estritamente errado falar em criptografia. Por isso faço a ressalva na minha resposta. Que ler verá que é algo errado. Esses comentários deixa ainda mais claro.


1)$A\subset X$ subset, then $\text{Int}(A)=\emptyset \iff \overline{A^{c}}=X$? 2)...

Question: $X$ is a metric space. Is it true that if $A\subset X$ subset, then $\text{Int}(A)=\emptyset \iff A^{c}$ is dense in $X$? And that $G$ is dense in $X$ $\iff$ $\forall\omega$ open nonempty in $X$ $\omega \cap G \neq \emptyset$?
For the first question I've tried to answer in this way (here I will denote $(T)'$ the set of all the limit points of $T$):
$(\implies)$ By contradiction: $x\in \text{Int}(A)$ $\implies$ $\exists r>0$ s.t. $B(x,r)\subset A$ $\implies B(x,r) \cap A^{c}=\emptyset \implies x \not\in (A^{c})^{'} \implies x\not \in A^{c}\cup(A^{c})^{'}=\overline{A^{c}}$ and then $A^{c}$ is not dense.
$(\impliedby)$ By definition we know $\text{Int}(A)=\{x\in A|\exists r>0 \text{ s.t. } B(X,r)\subset A\}$. If $A=\emptyset$ it's trivial ($A^{c}=X$). Otherwise: $x\in A$ then, because $\text{Int}(A)=\emptyset$, $\forall r>0$ $B(x,r)\cap A^{c}\neq\emptyset \implies x\in (A^{c})^{'} \implies A\subset (A^{c})'$. Then: $\overline{A^c}=A^{c}\cup (A^{c})'= A^{c}\cup (A\cup (A^{c})' \setminus A)=X$.
For the second question I tried in this other way:
$(\implies)$ Let us suppose $\exists$ $\omega$ open nonempty s.t. $\omega \cap G = \emptyset$. We know $\forall x \in \omega$ $\exists r>0$ s.t. $B(x,r)\subset \omega$ then $B(x,r)\cap G=\emptyset \implies x\not \in G' \implies x \not\in \bar{G}=G\cup G' \implies G$ is not dense.
$(\impliedby)$  We assume that $\forall$ nonempty open $\omega$ we have $\omega \cap G \neq \emptyset$. If $G^{c}=\emptyset$ it is trivial. Otherwise, let $x_{0}\in G^{c}$, $\forall r>0$ $B(x_{0},r)$ is open $\implies$ $B(x_{0},r)\cap G \neq \emptyset \implies x_{0}\in G^{'} \implies G^{c}\subset G^{'}$. Then, $\bar{G}=G\cup G'=G\cup (G^{c}\cup (G'\setminus G^{c}))=X$.
Are they correct? Could anyone help me? Thank you all in advance!
Comment: I think you might want to look at
https://math.stackexchange.com/questions/1604126/empty-interior-equivalent-definitions-from-munkres
Comment: Thank you for your comment!
Comment: I'd suggest shortening the proof along the following lines. For the first part: prove that $\operatorname{Int}(A)^c = \overline{A^c},$ for all $A \subset X.$ For the second part: take $A = G^c,$ getting $\big(\overline{G}\big)^c = \operatorname{Int}(G^c),$ for all $G \subset X.$
Answer: Let's start with the second: $G$ is dense in $X$ iff $G \cap O \neq \emptyset$ for all non-empty open subsets $O$ of $X$.
Left to right: Let $O \subseteq X$ be non-empty and open, so pick $x \in O$ and $r>0$ such that $B(x,r) \subseteq O$, using $O$ is open. We know $x \in \overline{G} = G \cup G'$. If $x \in G$ we are done as trivially $O \cap G \neq \emptyset$ is then witnessed by $x$ itself. Otherwise $x \in G'$ and the definition of a limit point gives that $B(x,r)$ intersects $G$ in some point $y \neq x$. But then $y \in O \cap G$ as well, and $O \cap G$ is again non-empty.
Right to left: We want to show $G$ is dense, so let $x \in X$. If $x \in G$ we are again done, as $x \in \overline{G}$ right away. So assume $x \notin G$ and let $r>0$ be arbitrary, then $O=B(x,r)$ is a non-empty open set, so by assumption it intersects $G$ and this must be in another point than $x$, as we assumed $x \notin G$. So then $x \in G' \subseteq \overline{G}$ and we're done too.
For the first we want to see that $\operatorname{int}(A) \neq \emptyset \iff A^\complement \text{ is dense}$.
I'll apply the characterisation of dense sets I just showed:
Left to right: Suppose that $O$ is a non-empty open set of $X$. So pick $x \in O$. Then $O \subseteq A$ cannot hold, as then $x \in \operatorname{int}(A)$ which we assumed to be empty. So not all points of $O$ lie in $A$, so some point of $O$ lies in $A^\complement$, or $A^\complement \cap O \neq \emptyset$. As $O$ was arbitary non-empty open, $A^\complement$ is dense.
Right to left:
Suppose that there is some $x \in \operatorname{int}(A)$.
Then $O= \operatorname{int}(A)$ is non-empty open and because $\operatorname{int}(A) \subseteq A$, we have that $O \cap A^\complement = \emptyset$. This contradicts the fact that $A^\complement$ is dense. Hence $\operatorname{int}(A) = \emptyset$.


How to generate from this distribution without inverse in R/Python?

Question: I am working with a distribution with the following density: $$f(x) = - \frac{(\alpha+1)^2 x^\alpha \log(\beta x)}{1-(\alpha + 1)\log(\beta)}$$ and CDF $$\mathbb{P} (X \leq x) = \int_0^x - \frac{(\alpha+1)^2 t^\alpha \log(\beta t)}{1-(\alpha + 1)\log(\beta)} \, dt = \frac{x^{\alpha+1}((\alpha+1)(\log(\beta x))-1)}{(\alpha+1)\log(\beta)-1}$$ with $x \in (0,1), \beta \in (0,1)$ and $\alpha >-1.$ How can I generate random samples from this distribution in Python/R? Which books can I use to learn about the simulation of random variables and random numbers? Any help is appreciated.
Comment: Voting to reopen: I disagree strongly that this question is anything other than "a statistical topic disguised as a coding question." Also this may be a good resource Devroye, L. (1986). *[Non-Uniform Random Variate Generation](http://www.nrbook.com/devroye/)*. Springer Verlag.
Comment: The parameter $\alpha$ can be "swallowed" by the change of variable $z=x^{\alpha+1}$, reducing to $\mathbb P(Z\le z)=z(\gamma\log z+1)$.
Comment: You could try the no u turn sampler if you confirm the likelihood is almost everywhere smooth.
Comment: For completeness, it might be worth clarifying what happens in the case where $\log(\beta) = \tfrac{1}{\alpha+1}$ (where you have denominators of zero).
Answer: A quick and dirty solution is to apply the inverse transform sampling in which the quantile function is computed via numerical inversion. Below is an R implementation along with an example.
# (your density)
myf <- function(x, a, b) {
 num = (a+1)**2 * x**a * log(b*x)
 den = 1 - (a+1)*log(b)
 return(-num/den)
}

# (your cdf)
myF <- function(x, a, b) {
  num = x**(a+1)* ((a+1)*log(b*x)-1)
  den = (a+1) * log(b) - 1
  return(num/den)
}

# quantile function computed numerically
myqf <- function(p, a, b) {
  uniroot(function(x) myF(x,a,b) -p,
          lower = .Machine$double.eps,
          upper = 1-.Machine$double.eps)$root
}

# get some uniform draws
u <- runif(1e+5)

# apply the inverse transform
x <- sapply(u, myqf, a = 5, b=0.99)

hist(x,probability = TRUE)
plot(function(x) myf(x,a=5, b=0.99), from=0, to=1,
     col=2, lwd=2, add= TRUE)

Surely there are less dirty solutions, i.e. more computationally efficient methods to handle this problem, one of these is suggested by Galen in the comments. Many other methods can be found in these three outstanding books:

Monte Carlo Statistical Methods by Robert and Casella, Springer;
Computational Statistics by Givens and Hoeting, Wiley;
Non-Uniform Random Variate Generation by Devroye.
Comment: Actually, I think that I can use this method.
Comment: @SextusEmpiricus, I might be wrong but I read it "the quantile function is not available" which is why I proposed to approximate it numerically.
Comment: @utobi you are right, I read it now as well as 'this distribution without inverse'. I read it before as a typo 'without using inverse transform sampling'.
Answer: Your pdf is the same as a transformation of a variable that follows a truncated gamma distribution.
The gamma distribution has shape $k=2$, rate $\alpha+1$ and is truncated at $-\log(\beta)$.
Transformation rule for probability density functions
If we apply the transformation
$$\begin{array}{}
Y &=& -\log(\beta X) \\
X &=& e^{-Y}/\beta
\end{array}$$
then the pdf for $Y$ is
$$g(y) = f\left({e^{-y}}/{\beta}\right) \cdot \left| \frac{dx}{dy}\right| = \frac{{\alpha^\prime}^2 \beta^{-\alpha^\prime} }{1-\alpha^\prime\log(\beta)} y e^{-\alpha^\prime y}$$
with $\alpha^\prime = \alpha +1$ and the domain of $y$ is $(-\log(\beta),\infty)$
Computational
A computational demonstration in R is:
alpha = 1
beta = 0.5

### sampling from truncated gamma distribution using inverse transform
p = runif(10^5, pgamma(-log(beta), shape = 2, rate = alpha+1), 1)
y = qgamma(p, shape = 2, rate = alpha+1)

### draw a histogram
hist(exp(-y)/beta, breaks = seq(0,1,1/40), freq =F)

### add lines with your density function
x = seq(0,1,0.001)
y = - ((alpha+1)^2 * x^alpha * log(beta * x))/(1-(alpha + 1)*log(beta))
lines(x,y)

Alternative transformation
Alternatively, one can also apply an additional shift
$$Z = Y+\log(\beta)$$
or effectively
$$Z = -\log(X)$$
which is a variable in the range $(0,\infty)$ and has a pdf like
$$h(z) = g(z-\log(\beta)) = \frac{{\alpha^\prime}^2 \beta^{-\alpha^\prime} \beta e^{\alpha} }{1-\alpha^\prime\log(\beta)} (z-\log(\beta)) e^{-\alpha^\prime z}$$
which is a mixture of two gamma distributed variables (or more specifically Erlang distributions).
So one should be able to perform the trick of Ben with the log transform of uniform variables (creating exponential/Erlang distributed variables), but without the need for rejections sampling.
The method will go something like

Draw to uniform variables $U_1$, $U_2$.
Depending on the value of $U_1$ being above or below some value $u_c$

have some sum like $z= a \log U_1 + b \log U_2 + c$
or use $z= d \log U_2$.

Compute $x = e^{-z}$

One will need to work out the values $a,b,c,d,u_c$ by writing out $h(z)$ more precisely as the mixture of $\alpha \exp(-\alpha z)$ and $\alpha^2 z \exp(-\alpha z)$. And one needs to take care that $U_1$ is used both as variable to decide which part of the mixture is used as well as generating the exponential distributed variable.
Comment: This [Wikipedia page](https://en.m.wikipedia.org/wiki/Probability_density_function#Function_of_random_variables_and_change_of_variables_in_the_probability_density_function) has more details about the transformation of the pdf when a variable is transformed.
Comment: Great answer (+1).  Your alternative transformation is not entirely clear to me, and I'm not convinced it would work without a rejection step, but perhaps seeing is believing.
Answer: It is possible to generate this random variable using a rejection-sampling algorithm using uniform random variables.  To do this, let $U_1,U_2 \sim \text{IID U}(0,1)$ and define the random variable:
$$Z = -\frac{\log(U_1) + \log(U_2)}{\alpha+1} \sim \text{Gamma}(2, \alpha+1).$$
Using the transformation shown in the excellent answer by Sextus Empiricus, it can be shown that:
$$\begin{align}
\mathbb{P}(X \leqslant x)
&= \mathbb{P} \bigg( \frac{e^{-Z}}{\beta} \leqslant x \bigg| Z \geqslant |\log(\beta)| \bigg) \\[6pt]
&= \mathbb{P}( Z \geqslant |\log(\beta x)| |Z \geqslant |\log(\beta)|). \\[6pt]
\end{align}$$
Hence, we can generate the random variable $X$ using a rejection-sampling method that generates the uniform values $(U_1,U_2)$ until we get a corresponding value $Z \geqslant |\log(\beta)|$, and then take the appropriate transformation to get $X$.  Below we code this method as the function rdist and we also code the CDF as pdist.$^\dagger$
rdist <- function(n, alpha, beta) {

  #Check input n
  if (!is.vector(n))        stop('Error: Input n should be a single numeric value')
  if (!is.numeric(n))       stop('Error: Input n should be a single numeric value')
  if (length(n) != 1)       stop('Error: Input n should be a single numeric value')
  if (as.integer(n) != n)   stop('Error: Input n should be an integer')
  if (min(n) <= 0)          stop('Error: Input n should be a non-negative integer')
  if (n == 0) { return(numeric(0)) }

  #Check input alpha
  if (!is.vector(alpha))    stop('Error: Input alpha should be a single numeric value')
  if (!is.numeric(alpha))   stop('Error: Input alpha should be a single numeric value')
  if (length(alpha) != 1)   stop('Error: Input alpha should be a single numeric value')
  if (min(alpha) <= -1)     stop('Error: Input alpha should be greater than minus-one')

  #Check input beta
  if (!is.vector(beta))     stop('Error: Input beta should be a single numeric value')
  if (!is.numeric(beta))    stop('Error: Input beta should be a single numeric value')
  if (length(beta) != 1)    stop('Error: Input beta should be a single numeric value')
  if (min(beta) <= 0)       stop('Error: Input beta should be greater than zero')
  if (min(beta) >= 1)       stop('Error: Input beta should be less than one')
  if ((alpha+1)*log(beta) == 1) stop('Error: Inadmissible parameters')

  #Generate pseudo-random values
  x <- rep(0, n)
  min.z <- -log(beta)
  for (i in 1:n) {
    z <- 0
    while (z <= min.z) { u <- runif(2); z <- -sum(log(u))/(alpha+1) }
    x[i] <- exp(-z)/beta }

  #Give output
  x }

#####################################################################

pdist <- function(x, alpha, beta, lower.tail = TRUE, log.p = FALSE) {

  #Check input x
  if (!is.vector(x))        stop('Error: Input x should be a numeric vector')
  if (!is.numeric(x))       stop('Error: Input x should be a numeric vector')
  if (length(x) == 0) { return(numeric(0)) }

  #Check input alpha
  if (!is.vector(alpha))    stop('Error: Input alpha should be a single numeric value')
  if (!is.numeric(alpha))   stop('Error: Input alpha should be a single numeric value')
  if (length(alpha) != 1)   stop('Error: Input alpha should be a single numeric value')
  if (min(alpha) <= -1)     stop('Error: Input alpha should be greater than minus-one')

  #Check input beta
  if (!is.vector(beta))     stop('Error: Input beta should be a single numeric value')
  if (!is.numeric(beta))    stop('Error: Input beta should be a single numeric value')
  if (length(beta) != 1)    stop('Error: Input beta should be a single numeric value')
  if (min(beta) <= 0)       stop('Error: Input beta should be greater than zero')
  if (min(beta) >= 1)       stop('Error: Input beta should be less than one')
  if ((alpha+1)*log(beta) == 1) stop('Error: Inadmissible parameters')

  #Check inputs lower.tail and log.p
  if (!is.vector(lower.tail))  stop('Error: Input lower.tail should be a single logical value')
  if (!is.logical(lower.tail)) stop('Error: Input lower.tail should be a single logical value')
  if (length(lower.tail) != 1) stop('Error: Input lower.tail should be a single logical value')
  if (!is.vector(log.p))       stop('Error: Input log.p should be a single logical value')
  if (!is.logical(log.p))      stop('Error: Input log.p should be a single logical value')
  if (length(log.p) != 1)      stop('Error: Input log.p should be a single logical value')

  #Generate probabilities
  a <- alpha+1
  n <- length(x)
  PROBS <- rep(0, n)
  for (i in 1:n) {
    xx <- x[i]
    if (xx >= 1) { PROBS[i] <- 1 }
    if ((xx > 0)&(xx < 1)) {
      PROBS[i] <- (xx^a)*(a*log(beta*xx)-1)/(a*log(beta)-1) } }
  if (!lower.tail) { PROBS <- 1-PROBS }

  #Give output
  if (log.p) { log(PROBS) } else { PROBS } }

We can now use this function to generate a large number of pseudo-random values.  Plotting the resulting ECDF for the values gives a close approximation to the true CDF.
#Set distribution parameters
alpha <- 2
beta  <- 0.6

#Generate values from the distribution
set.seed(1)
n <- 10^6
X <- rdist(n, alpha, beta)

#Plot empirical cumulative distribution function (ECDF) of values
plot(ecdf(X), col = 'blue', main = 'ECDF of pseudo-random values', ylab = 'Proportion')

$^\dagger$ Here is alternative code for rdist if you are want to avoid rejection sampling by using the pgamma and rgamma functions.
rdist <- function(n, alpha, beta) {

  #Check input n
  if (!is.vector(n))        stop('Error: Input n should be a single numeric value')
  if (!is.numeric(n))       stop('Error: Input n should be a single numeric value')
  if (length(n) != 1)       stop('Error: Input n should be a single numeric value')
  if (as.integer(n) != n)   stop('Error: Input n should be an integer')
  if (min(n) <= 0)          stop('Error: Input n should be a non-negative integer')
  if (n == 0) { return(numeric(0)) }

  #Check input alpha
  if (!is.vector(alpha))    stop('Error: Input alpha should be a single numeric value')
  if (!is.numeric(alpha))   stop('Error: Input alpha should be a single numeric value')
  if (length(alpha) != 1)   stop('Error: Input alpha should be a single numeric value')
  if (min(alpha) <= -1)     stop('Error: Input alpha should be greater than minus-one')

  #Check input beta
  if (!is.vector(beta))     stop('Error: Input beta should be a single numeric value')
  if (!is.numeric(beta))    stop('Error: Input beta should be a single numeric value')
  if (length(beta) != 1)    stop('Error: Input beta should be a single numeric value')
  if (min(beta) <= 0)       stop('Error: Input beta should be greater than zero')
  if (min(beta) >= 1)       stop('Error: Input beta should be less than one')
  if ((alpha+1)*log(beta) == 1) stop('Error: Inadmissible parameters')

  #Generate pseudo-random values
  min.p <- pgamma(-log(beta), shape = 2, rate = alpha+1)
  z <- qgamma(runif(n, min.p, 1),  shape = 2, rate = alpha+1)
  x <- exp(-z)/beta

  #Give output
  x }

Answer: There are already some great answers here.  Another way to do this would be to use MCMC a la the Metropolis Hastings Algorithm.  See my implementation below
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import beta

@np.vectorize
def target_distribution(x):

    a = 5
    b = .999

    top = -(a+1)**2 * x**a * np.log(b*x)
    bottom = 1 - (a+1)*np.log(b)

    return top/bottom

# Initialize the Metropolis-Hastings algorithm with an initial value of x = 0 and a number of iterations
x = 0.5
iterations = 100000

# Initialize an array to store the samples
samples = np.zeros(iterations)

# Run the Metropolis-Hastings algorithm
for i in range(iterations):
    # Sample from the proposal distribution
    x_proposal = beta(a=1, b = 1).rvs()

    # Calculate the acceptance ratio
    acceptance_ratio = target_distribution(x_proposal) / target_distribution(x)

    # Accept or reject the proposal
    if np.random.uniform() < acceptance_ratio:
        x = x_proposal

    # Store the sample
    samples[i] = x

# Plot the samples
plt.hist(samples, bins=50, density=True, alpha=0.5)
x_range = np.linspace(0, 1, 1000)
plt.plot(x_range, target_distribution(x_range))
plt.show()

Comment: Nice! My first thought was to use some Monte Carlo method, but I'm new to this area and still learning. I will read about this method, thank you!
Answer: Though you seem to be willing to avoid inversion, this can be done analytically.
After setting $z:=x^{\alpha+1}$, from
$$u(z):=\text{cdf}(z)=z(\gamma\log z+1)=\gamma e^{-1/\gamma}(e^{1/\gamma}z)\log(e^{1/\gamma}z)=\gamma e^{-1/\gamma} W^{-1}(e^{1/\gamma}z)$$
we draw
$$e^{1/\gamma}z(u)=e^{1/\gamma}\text{cdf}^{-1}(u)=W\left(\gamma^{-1} e^{1/\gamma}u\right)$$
where $W$ denotes Lambert's function.
Now it suffices to draw $u$ uniformly in $[0,1]$ and pass it to $\text{cdf}^{-1}(u)$.
Comment: @utobi: you probably mean numerically ?
Comment: God dangint, I LOVE the Lambert W Function, good on you +1
Comment: Nice, +1. Yet, W has to be computed analytically https://en.m.wikipedia.org/wiki/Lambert_W_function
Comment: @utobi The rnorm function in R, and probably derived from something else, also uses a numerical approximation with the use of a high order polynomial. The advantage is that you can use the same polynomial independent  of the scale and location. This seems to be the case here as well as the numerical approximation boils down to a single function $W$. Because you only need a single function, independent of the parameters, making a numerical approach should be achievable very efficiently.
Comment: @SextusEmpiricus: also, accuracy not being critical for RNG, a coarse precomputed tabulation can be used, I guess.
Comment: @YvesDaoust the programmers of the rnorm function think otherwise and sample two random numbers in order to generate more precision when applying the inverse transform sampling.
Comment: yea, I meant numerically.
Comment: @SextusEmpiricus: rnorm generates a normal distribution, not relevant here. And overkill.
Comment: @YvesDaoust I was mentioning rnorm as an analogy. One can choose between a transform that requires sampling two uniform variables (e.g. Box-Muller) or inverse transform sampling that requires sampling of only one uniform variable (or two for more precision). And while the inverse transform sampling requires a numerical solution of the function W it is efficiënt if it is the same function for different parameters as a single polynomial curve with high precision can be used to approximate the function and the coefficients need to be computed only once....
Comment: ... the relevance of the analogy is because the question might be interpreted more general about rng and not specific about this case.
Comment: @SextusEmpiricus: demanding precision for random sampling makes little sense.
Comment: For the curious: source code for R's `rnorm` is http://svn.r-project.org/R/trunk/src/nmath/snorm.c which gives a reference to Ahrens & Dieter. [Extensions of Forsythe's method for random sampling from the normal distribution](https://www.ams.org/journals/mcom/1973-27-124/S0025-5718-1973-0329190-8/S0025-5718-1973-0329190-8.pdf). *Math. Comput.* 27, 927-937. An interesting historic note on Forsythe's original method was [arxived by Richard Brent](https://arxiv.org/pdf/1005.0909.pdf). This method needs PDFs of the form $f(x)=\alpha \exp(-G(x))$ so not just for Gaussians but no help here sadly
Comment: @YvesDaoust With support on $z\in [0,1]$, I think that requires $-1\le \gamma \le 0$. It seems the inverse uses the lower branch of the Lambert W function.  The distribution for $Z$ seems to be a mixture of the $\gamma=0$ (uniform) case and the  $\gamma=-1$ (density $-\log z$) case.  Does the distribution with that density $-\log z$ on $(0,1]$ have a name?
Comment: @Henry: ask the OP, I know nothing about this distribution.
Comment: Digging further, [somebody asked about it a few years ago](https://stats.stackexchange.com/questions/379694/is-there-a-name-for-the-distribution-whose-pdf-is-lnx-on-its-support-0-1)
Answer: I think it is possible to simulate this simply using three independent standard uniform random variables, without any approximation or numerical interpolation or rejection sampling or use of the Lambert W function.
Yves Daoust's answer used $Z= X^{\alpha+1}$ and $\gamma= \frac{1}{{(\alpha+1)\log(\beta)-1}}$ to simplify the cdf for $Z$ to $z(\gamma\log z+1)$ on $(0,1]$ and so the density to $1+\gamma +\gamma \log(z)$.  You will have $-1 \le \gamma \le 0$.
This is a mixture distribution for $Z$, with probability $1+\gamma$ of being drawn from a uniform distribution on $(0,1]$ and probability $-\gamma$ of being drawn from a distribution with density $-\log(z)$ on the same support.  Somebody previously asked about the latter distribution with an answer suggesting it might be a negative log-gamma distribution and a comment suggesting it might be a product uniform distribution as the distribution of the product of two standard uniform random variables. That is the kind of thing we want to make random samples.   The R code is short and simple:
rthisdist <- function(n, alpha=0, beta=1){
   # require alpha > -1 and
   #         beta positive for logarithm and <= 1; 1E-99 will do
   gam <- 1 / ((alpha+1) * log(beta) - 1) # gamma without overloading
   uniformA <- runif(n)
   uniformB <- runif(n)
   uniformC <- runif(n)
   z <- ifelse(uniformA < -gam, uniformB, 1) * uniformC
   return(z ^ (1 / (alpha+1)))
   }

Testing this against your CDF gives a convincing match (simulations in blue, your CDF in red):
testalpha <- 2
testbeta  <- 0.6
plot.ecdf(rthisdist(10^4, testalpha, testbeta), col="blue")
curve((x^(testalpha+1) * ((testalpha+1) * (log(testbeta * x)) - 1)) /
      ((testalpha+1) * log(testbeta) - 1),
      from=0, to=1, add=TRUE, col="red")

Comment: It can be done with two uniform variables. After using the uniformA for the if/else condition, you can reuse it as the uniform variable in the further computation.
Comment: @SextusEmpiricus - yes indeed, though I think you would have to divide `uniformA` by $-\gamma$ to replace `uniformB` and that might make it slightly harder to see what was going on.  Another simplification could be to change the sign of $\gamma$ so it was $\frac{1}{1-(\alpha+1)\log(\beta)} \in [0,1]$ and directly represented a probability in the mixture distribution.


Problemas com a div do rodapé

Question: Bom dia galera,
Estou tendo problemas com a div do meu rodapé. Sempre que vou usando o atalho CTRL + (+), chega um momento que o rodapé fica preenchendo a parte final da página. Tem como eu resolver isso?
Segue abaixo o código código da página html:

            Congregação São João - RJ

            CNPJ: 27-001/11.280.416

            CEP: 250-25-515

            Rua: Santo Antônio, nº523

            Centro - São João de Mereti/RJ

            Congregação Pau Grande - Magé/RJ

            CEP: 25.933-145

            Rua: Lindsay Anderson, nº 15

            Pau Grande - Magé/RJ

            Congregação Santa Inês - Vila Velha/ES

            CEP: 29.108-041

            Rua: Avenida Rui Braga Ribeiro, nº961

            Santa Inês - Vila Velha/ES

E o css:
#footer {
    position: relative;
    margin-top: -150px; /* valor negativo da altura do rodapé */
    width: 100%;
    height: 150px;
    clear: both;
    background-repeat: no-repeat;
    background-position: center right;
    background-color: #7d0d0d;
}

#textorodape{
    margin-right: 100px;
    color: white;
    font-size: 16px;
    position: center right;
}

#coluna1 {
    float: left;
    width: 333px;
    color: white;
    font-size: 14px;
}

#coluna2 {
    float: left;
    width: 333px;
    color: white;
    font-size: 14px;
}

Comment: Julio, acho que você poderia enviar o html também, pois não entendi onde está sendo utilizado coluna1 e coluna2, imagino #footer seja o id de uma div, mas o textorodape também é id ou é class? é usada em uma div?
Comment: Obrigado @EduardoMendes, essa textoropade é um id, mas não estou utilizando. Vou enviar o html, só um minuto!
Comment: @EduardoMendes, pronto, editado!
Answer: Minha sugestão é utilizar o tamanho do height em porcentagem. Imagino que tenha outra div acima da div footer. No exemplo abaixo eu adicionei uma div chamada conteudo e coloquei seu tamanho em porcentagem. Observe também que coloquei as tags html e body com o width e height em 100%. Então a div conteudo com 85% e a div footer com 15%. Desta forma, o zoom não irá influenciar no tamanho das divs, somente no tamanho da fonte.
Veja o css:
html, body {
    width:100%;
    height:100%;
    padding: 0;
    margin: 0;
}

#conteudo {
    width:100%;
    height:85%;
    background:#FFED00;
}

#footer {
    position: relative;
    /*margin-top: -150px;*/ /* valor negativo da altura do rodapé */
    width: 100%;
    height: 15%;
    clear: both;
    background-repeat: no-repeat;
    background-position: center right;
    background-color: #7d0d0d;
}

#textorodape{
    margin-right: 100px;
    color: white;
    font-size: 16px;
    position: center right;
}

.coluna1 {
    float: left;
    width: 333px;
    color: white;
    font-size: 14px;
}

.coluna2 {
    float: left;
    width: 333px;
    color: white;
    font-size: 14px;
}

E aqui o html

            Congregação São João - RJ

            CNPJ: 27-001/11.280.416

            CEP: 250-25-515

            Rua: Santo Antônio, nº523

            Centro - São João de Mereti/RJ

            Congregação Pau Grande - Magé/RJ

            CEP: 25.933-145

            Rua: Lindsay Anderson, nº 15

            Pau Grande - Magé/RJ

            Congregação Santa Inês - Vila Velha/ES

            CEP: 29.108-041

            Rua: Avenida Rui Braga Ribeiro, nº961

            Santa Inês - Vila Velha/ES

Obs: Não é recomendável que tenha mais de um id no código html, quando isso acontece o ideal é substituir por class, no caso das divs coluna2.
Comment: Muito obrigado, funcionou corretamente!!
Comment: Beleza. Se você achar necessário, depois marque a resposta como útil e aceita lá como a melhor resposta. Abraços
Answer: Você já experimentou deixar o css da seguinte maneira ?
.footer {
    position:absolute;
Com isso, o seu rodapé não irá mais ser alterado pelo zoom do navegador.

Problemas com a div do rodapé


What is a "VPN Consolidator"

Question: A client recently mentioned using a VPN consolidator for remote access to their server. I've never heard this term used before. Is it some type of slang that I'm not familiar with?
I did a bit of searching and I came up completely empty.
Comment: "Concentrator," maybe?
Comment: Maybe they meant "VPN concentrator".
Comment: @Shane Madden: Curses! You beat me to the punch.
Comment: I'm not familiar with that either...What would that be? What is it used for?
Answer: Best guess: They used "VPN Consolidator" as a synonym for "VPN Concentrator". There is no hard'n'fast definition of a VPN concentrator either; it's generally just a device which is dedicated to being endpoint for (many) VPNs.
Say you had a good DMZ & dual firewall setup that is working well, and you don't want to change it. A new need for a large'ish number of simultaneous VPN connections for 'road warriors' arises. Your existing firewalls aren't built for high-speed cryptography, they don't have a processing power to handle many AES streams at once.
So you add a "VPN Concentrator" -- either a more modern firewall-style appliance with more CPU, or a full server -- and dedicate this box to handling the VPNs.
Comment: @Nate Bross: Yes, it is probably a dedicated VPN appliance. And most likely very similar to a regular firewall from the same vendor -- just with a faster embedded CPU and/or a dedicated AES co-processor for more encryption oomph. It could also just be an internal name given to a regular firewall dedicated to VPN handling; just to highlight its daily purpose in the architecture.
Comment: +1 usually these devices are intended to handle a large number of connections.  Cisco's 3000 is probably the most widely-known product with the 'concentrator' moniker.
Comment: So it is effectivly a dedicated VPN appliance (sold as such or maybe software to run on a server)?
Comment: As @Jesper mentioned, there's no hard & fast rule, but the 'concentrator' label is typically found on some VPN appliances.  I don't know of a software or distro out there that currently labels itself as a VPN concentrator, but that doesn't mean it doesn't exist.

What is a "VPN Consolidator"


Set a tuple of strings as a dict key

Question: I need to create a dictionary where the keys are a tuple of strings and the values are integers. I have done it using the CSV module but with pandas, I can better manage my data.
Basically I want this:
print(Supply)
{(1, 1): 400, (1, 2): 0, (2, 1): 1500, (2, 2): 0, (3, 1): 900, (3, 2): 0}

To become this:
print(Supply)
{('1', '1'): 400, ('1', '2'): 0, ('2', '1'): 1500, ('2', '2'): 0, ('3', '1'): 900, ('3', '2'): 0}

Comment: Why do you want the integers as strings? You'll just have to convert them to integers again if you want to use them for anything
Comment: Please update your question with the code you have tried.
Comment: @quamrana The code that I used is in Jethro`s answer, I just copied and pasted it. Let me know if you need anything else.
Comment: @mousetail I am building an optimization model using a variable with multiple indexes, since I am in an early stage I`m using some fake data. These indexes will be regions code as well. Idk if I have answered what you asked, let me know if you need anything else!
Answer: This does the transformation you want:
Supply = {tuple(str(i) for i in key): val for key, val in Supply.items()}

print(Supply)

# {('1', '1'): 400, ('1', '2'): 0, ('2', '1'): 1500, ('2', '2'): 0, ('3', '1'): 900, ('3', '2'): 0}

Comment: Oh, sorry I forgot! Done!
Comment: Glad to hear! Mind marking the answer as accepted then?

Set a tuple of strings as a dict key


Using compass in Aegir limited shell (user o1.ftp)

Question: I installed the latest BOA-stable and after it RVM with Compass and SASS. According to the changelogs of BOA-2.0.4, I should be able to use Compass commands:

Fixes:
Allow all standard compass/sass commands in limited shell.

As root, I am able to execute 'compass watch', but I am not with the limited shell (as user o1.ftp). It says:
<code>sh: 1: compass: not found
</code>
Answer: For me it didn't work in combination with RVM. I needed to install it the normal way:
<code>- apt-get install ruby
- apt-get install rubygems
- gem update --system
- gem install compass
</code>
I think that because of the isolated gem installs that RVM creates, the limited shell user has no access to the commands.
Answer: You should modify the .barracuda.conf file to include the installation of compass tools.
There is an option _XTRAS_LIST where you simply include CSS
Answer: Note that now BOA uses RVM for all gems by default. We have also fixed this for limited shell environment, so at least in the last few months it should work just fine.


UK bank sort code javascript regular expression

Question: I'm trying to create a regular expression in javascript for a UK bank sort code so that the user can input 6 digits, or 6 digits with a hyphen between pairs. For example "123456" or "12-34-56". Also not all of the digits can be 0.
So far I've got /(?!0{2}(-?0{2}){2})(\d{2}(-\d{2}){2})|(\d{6})/ and this jsFiddle to test.
This is my first regular expression so I'm not sure I'm doing it right. The test for 6 0-digits should fail and I thought the -? optional hyphen in the lookahead would cause it to treat it the same as 6 0-digits with hyphens, but it isn't.
I'd appreciate some help and any criticism if I'm doing it completely incorrectly!
Comment: Which digits can/can't be `0`?
Comment: Sorry, it doesn't matter - it just can't be all zeroes
Comment: You can ask user not to include hyphen `-` in the number like how online VISA/Master Card payment. Being extra strict is OK when dealing with money.
Comment: Yes I'd prefer to not have hyphens at all but unfortunately have to accommodate both
Answer: Just to answer your question, you can validate user input with:
/^(?!(?:0{6}|00-00-00))(?:\d{6}|\d\d-\d\d-\d\d)$/.test(inputString)

It will strictly match only input in the form XX-XX-XX or XXXXXX where X are digits, and will exclude 00-00-00, 000000 along with any other cases (e.g. XX-XXXX or XXXX-XX).
However, in my opinion, as stated in other comments, I think it is still better if you force user to either always enter the hyphen, or none at all. Being extra strict when dealing with anything related to money saves (unknown) troubles later.
Comment: @nhahtdh Ok, I just fixed the regex, adding the `{2}` to the `(\d{6}|(\d{2}-)\d{2})` to make it `(\d{6}|(\d{2}-){2}\d{2})`. @NickL this is the issue you are experiencing.
Comment: @ScottS: If there is something wrong, just comment if the post is new. I'm right here to edit it.
Comment: I think according to this http://jsfiddle.net/sd88a/2/ it's not working for 6 non-0 digits with hyphens
Comment: Cheers for that guys, and I agree with only allowing numbers but unfortunately dealing with some legacy applications, so have to be prepared
Comment: @NickL Edited my post to complete the use case.
Comment: Gravedigging a little here - this will match "௫௬-௭௮-௯౦" as a valid sort code (which it isn't) - should use [0-9] instead of \d in order to only match numeric digits relevant to the culture.
Comment: @PhonicUK: Nope. This is JavaScript. `\d` is effectively equivalent to `[0-9]` in JS.
Answer: Since any of the digits can be zero, but not all at once, you should treat the one case where they are all zero as a single, special case.
You are checking for two digits (\d{2}), then an optional hyphen (-?), then another two digits (\d{2}) and another optional hyphen (-?), before another two digits (\d{2}).
Putting this together gives \d{2}-?\d{2}-?\d{2}, but you can simplify this further:
(\d{2}-?){2}\d{2}

You then use the following pseudocode to match the format but not 000000 or 00-00-00:
if (string.match("/(\d{2}-?){2}\d{2}/") && !string.match("/(00-?){2}00/"))
     //then it's a valid code, you could also use (0{2}-?){2}0{2} to check zeros

You may wish to add the string anchors ^ (start) and $ (end) to check the entire string.
Comment: Your regex will match `45-5454`, not sure if this is allowed, since OP says that either no hyphen, or hyphen between pairs.
Comment: @nhahtdh But accepting that's less harmful than the contortions required to get everything into a single perfect RE. Better to normalize after acceptance by removing hyphens (at which point, `45-4545` won't matter).
Comment: @DonalFellows: It is possible to get everything into single regex, with what Javascript supports (negative look-ahead). I think online banking are doing right by enforcing a single way to enter the credit card numbers - no need for input normalization, only strict checking. The user won't be able to complain anything, since there is only one way to input.


Razor Pages ASP.NET Core 6 - How to add background image with blur

Question: I have successfully added a "background-image" to my Razor Pages application.
Now, when using "filter: blur(2px);" the header and footer blur, but not the image.
I've been at this for hours, saw tons of threads on how to add the image, but couldn't find a way to do this properly (the blur), or see someone else experiencing the same behavior.
My _Layout:

<!DOCTYPE html>

    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>@ViewData["Title"] - LifeCompanion</title>
    <link rel="stylesheet" href="~/lib/bootstrap/dist/css/bootstrap.min.css" />
    <link rel="stylesheet" href="~/css/site.css" asp-append-version="true" />
    <link rel="stylesheet" href="~/LifeCompanion.styles.css" asp-append-version="true" />

                        <button class="btn btn-secondary dropdown-toggle" type="button" id="dropdownMenu2" data-bs-toggle="dropdown" aria-expanded="false">
                            LifeCompanion
                        </button>

                                    Images!

                            Home

                            Privacy

            @RenderBody()

            © 2022 - LifeCompanion - Privacy

    @await RenderSectionAsync("Scripts", required: false)

My CSS - using site.css to apply changes globally - is:

html {
  font-size: 14px;
  width: 100%;
  height: 100%;
}

@media (min-width: 768px) {
  html {
    font-size: 16px;
  }
}

html {
  position: relative;
  min-height: 100%;
}

body {
    margin-bottom: 60px;
    background: url(../img/background.jpg) no-repeat center center fixed;
    background-size: cover;
    width: 100%;
    height: 100%;
    color: white;
}

So far so good.
Now when adding to my css:

body {
    margin-bottom: 60px;
    background: url(../img/background.jpg) no-repeat center center fixed;
    background-size: cover;
    width: 100%;
    height: 100%;
    color: white;
    filter: blur(2px);
}

The footer and header blurs, but not the background-image as anticipated.
Answer: Replace filter: blur(2px); with backdrop-filter: blur(2px);.
Answer: filter will affect the element itself, and backdrop-filter will affect the elements beneath the element within the bounding box of the element.So in your project, you need to use backdrop-filter. You can follow this link to learn more about backdrop-filter.


Renewal function with discrete interarrival times evaluated at non-integer points

Question:
Of this question, how do I do approach part d)? I do not recall any continuity results which I could use for this..
Answer: Since the interarrival time $X_i$ are positive integers, there will be jumps on integral time point only, and thus
$$ N_{t+\varepsilon} = N_t $$
To write this out more explicitly, note that the relationship between the counting process and interarrival time
$$ N_t \leq n \iff \sum_{i=1}^{n+1} X_i > t $$
Therefore
$$ N_{t+\varepsilon} \leq n \iff \sum_{i=1}^{n+1} X_i > t + \epsilon $$
Moreover,
$$ \Pr\left\{\sum_{i=1}^{n+1} X_i > t\right\}
= \Pr\left\{t < \sum_{i=1}^{n+1} X_i \leq t + \varepsilon \right\} +
\Pr\left\{\sum_{i=1}^{n+1} X_i > t + \varepsilon \right\}$$
As $X_i$ are integer, $\displaystyle \sum_{i=1}^{n+1} X_i$ is also an integer, So for $t \in \mathbb{N}$,
$$ \Pr\left\{t < \sum_{i=1}^{n+1} X_i \leq t + \varepsilon \right\} = 0$$
As a result, for $t \in \mathbb{N}$,
$$ \Pr\left\{\sum_{i=1}^{n+1} X_i > t\right\} = \Pr\left\{\sum_{i=1}^{n+1} X_i > t + \epsilon\right\} $$
$$ \Pr\{N_t \leq n\} = \Pr\{N_{t+\epsilon} \leq n\}$$
$$ E[N_t] = E[N_{t+\epsilon}] $$


How to match this expression via regex?

Question: How can I match lines like the following:
 SomeText SomeText SomeText SomeText SomeText
SomeText SomeText SomeText SomeText SomeText SomeText SomeText SomeText SomeText SomeText.
SomeText SomeText SomeText SomeText SomeText SomeText SomeText SomeText SomeText SomeText.
SomeText SomeText :
SomeText SomeText SomeText SomeText SomeText SomeText  (BlockID: 12345,TestCaseID: 12345, 1)

What I want from this is the whole part before (BlockID: 12345,TestCaseID: 12345, 1)
Another part I want is the 12345 of BlockID, and another one for TestCaseID
I have tried the following to mark all lines but I don't know further:
(?:[^\n]*(\n*))+.*

Comment: Something like [`(?s)(.*?)\s*\(BlockID:\s*(\d+),TestCaseID:\s*(\d+)`](https://regex101.com/r/xI3iJ2/1)?
Comment: Or [`(?s)(.*?)\s*\(BlockID:\s*(\d+),TestCaseID:\s*((?:,?\s*\d+)+)`](https://regex101.com/r/xI3iJ2/2)?
Answer: Everything until BlockID:
([\S\s]*)(?=\(BlockID)

12345 from BlockID:
(?:BlockID: )([0-9]{5})

12345, 1 from TestCaseID:
(?:TestCaseID: )([0-9]{5}, [0-9])

Answer: Depending on the types of actual input you have I can recommend something like
(.+)\\(BlockID: (\\d+),TestCaseID: (\\d+), 1\\)

which means:

Capture a sequence of characters (including new lines since we'll add the DOTAL flag) of length 1 or more,
until you reach BlockID:,
at which point capture a sequence of 1 or more digits,
followed by ,TestCaseID:,
then again capture a sequence of 1 or more digits,
followed by , 1).

Here it is in work:
public class Regex {

    public static void main(String[] args) {

        String text = " SomeText SomeText SomeText SomeText SomeText \r\n" +
        "SomeText SomeText SomeText SomeText SomeText SomeText SomeText SomeText SomeText SomeText.\r\n" +
        "SomeText SomeText SomeText SomeText SomeText SomeText SomeText SomeText SomeText SomeText.\r\n" +
        "SomeText SomeText :\r\n" +
        "SomeText SomeText SomeText SomeText SomeText SomeText  (BlockID: 12345,TestCaseID: 12345, 1)";

        Pattern p = Pattern.compile("(.+)\\(BlockID: (\\d+),TestCaseID: (\\d+), 1\\)", Pattern.DOTALL);
        Matcher m = p.matcher(text);
        while(m.find()) {
            System.out.println(m.group(1));
            System.out.println(m.group(2));
            System.out.println(m.group(3));
        }
    }
}

Output:

SomeText SomeText SomeText SomeText SomeText
  SomeText SomeText SomeText SomeText SomeText SomeText SomeText SomeText SomeText  SomeText.
  SomeText SomeText SomeText SomeText SomeText SomeText SomeText SomeText SomeText SomeText.
  SomeText SomeText :
  SomeText SomeText SomeText SomeText SomeText SomeText
  12345
  12345

How to match this expression via regex?


Goodness of fit for multivariate gaussian

Question: What are good tests to check if the data is generated by multivariate jointly gaussian distribution.
I know $\chi^2$ test is commonly used as a goodness of fit metric but I did not understand how to use it for multidimensional random vector case.
Most of the examples I saw were for univariate cases.
An example would help a lot.
Note: the covariance matrix isn't diagonal. No assumption of independent variables.
Answer: You can use the Mardia's Test.
The test has R package : https://www.rdocumentation.org/packages/MVN/versions/4.0/topics/mardiaTest
(https://www.youtube.com/watch?v=pBJNXhl4cxQ&feature=youtu.be)


create 2d array then using user input print the total number of odd no. and the sum of the odd numbers in that array

Question: This what I have done so far. I can't get to print the total number of odd numbers in the array and the sum of that odd numbers.
import java.util.Scanner;

public class arrayssumofodd
{
    public static void main(String args[])
    {
        Scanner scan = new Scanner(System.in);

        int rows=scan.nextInt();
        int columns=scan.nextInt();
        int array[][]=new int [rows][columns];

        int odd=0;
        int sum=0;
        int num=0;

        for(int i=0;i<rows;i++)
        {
            for(int j=0; j<columns;j++)
            {
                array[i][j]=scan.nextInt();
                num=array[i][j];

            }

            for(int k=0;k<rows;k++)
            {
                for(int m=0;m<k;m++)
                {
                    if(num%2!=0)
                    {
                        odd++;
                        sum+=odd;
                    }
                }
            }
        }
        System.out.println("Odd number count "+" = " +odd);
        System.out.println("Sum of odd numbers "+" = " +sum);
    }
}

I've tried everything to get the odd numbers to print but I haven't been able to. It keeps outputting zero when I run the program.
Answer: Your for loops are mixed with each other.
for(int i=0;i<rows;i++)
{
    for(int j=0; j<columns;j++)
    {
        array[i][j]=scan.nextInt();
        num=array[i][j];
        // you can do operation with odd count and sum here
}

Or you can use the same two for loops to go through the numbers in the 2D array and just count the odd number and find the sum.
Comment: so i only need two for loops
Comment: It depends. You can use two for loops to get numbers from the user. And then again two for loops to go through those numbers(in total 4). Also you can check the numbers while getting them from the users in that case only two loops.
Comment: thanks for that but when i type in the user inputs i have to type it twice for some reason i dont know y
Comment: when i did the count the odd numbers its not printing out the correct number of odd numbers
Comment: @basilroy you should update your code in the question based on your changes.
Comment: i did update when i type in 1 2 3 4 5 its gives me odd count =2 and sum=3


How do I perform operation on Chinese characters (UTF-8) in C?

Question: If the input is something like 世界+你好 how can I perform these UTF-8 unicode operations in C?:

Split the string at the + character and place the two sections of Chinese characters into two separate arrays: str1 = 世界 and str2 = 你好.
Compare two arrays to see if the Chinese characters are the same.

The Chinese characters will be inputted from terminal.
Comment: Read the characters into a byte array. Begin by figuring out where a character begins and where it ends. How to do this is described in the utf-8 specification. PS. FWIW, '+' can not be a part of a utf-8 encoding of a chinese character anyway :)
Comment: (... and then compare the bytes, if comparison is your goal)
Comment: There's usually no need to split the byte stream into characters.
Answer: One of the nice things about UTF-8 is that if you find a byte that is a valid ASCII character (in particular, <128), that byte is representing that ASCII character. Therefore, you can just split at the + character as if you had a single-byte encoding string.
Comparison in your case is also simply byte-wise. It gets much harder when you have to worry about canonical forms or case-sensitivity, but as far as I know, neither of those applies to Chinese. (Of course, you might have different characters you want to treat as identical, such as 気 and 氣. If so, normalize the strings first with a standard search-and-replace.)
Answer: I work with Chinese characters for many years, and I do not remember ever "performing operations in UTF-8". Let me explain, UTF-8 is encoding, it is not supposed to be used in-memory to do operations. When UTF-8 was invented, the idea was that english is the important language, and we need the rest some-how, so in UTF-8 English characters are first-class citizens, unlike Chinese.
As the word "encoding" implies, you must DECODE the data before you can use it. It is like "performing operations on characters in ZIP encoding". Of course, you can do something with characters if you load the file into a buffer, but you will be decoding any way, either decode the whole buffer and then perform the operations, or decode on fly, character by character and doing some operations at the same time.
What exactly I mean by "decoding"? Normally you will use C-type unsigned short or wchar_t, or sometimes int to hold each character. So you load your UTF-8 text into a char utf8buffer[] buffer, then decode it to another buffer wchar_t utf16buffer[]. Then you do whatever you need to do, then you encode back to UTF-8 and save to disc.
As you can see UTF16 is enough to deal with Chinese:
 L'一' == 0x4e00; // first Chinese character "yi" - "one"
 L'龥' == 0x9fa9; // the last Chinese char that I know of.
                  // From 0xa000 the Korean alphabet (Hangul) starts.

But this only applies to common Chinese, there are rare characters that are only used in ancient literature by scholars that will not fit into 0xFFFF range. Actually Chinese "alphabet" is not fixed, you can combine any Chinese "radicals" and characters into a new character. This is not fixed! There is even a UNICODE utility for this called Ideographic Description Sequences "IDS" in short. But hopefully you do not need that at all.
Comment: You have to use *some* encoding to store any kind of strings in memory. If you store them in `wchar_t`, you'll most likely use UTF-16 encoding (which for some reason Microsoft calls "Unicode" – as if UTF-8 were not Unicode).


SHAP algorithm for feature selecion

Question: I want to perform feature selection on my data. I have too many features, about 50 - 60, for not so much samples.
Until today I was using the importance function of the xgboost package, but lately I was introduced to SHAP, and although it is more of an interpretation tool than anything, I was told it is a powerful and robust tool for feature selection, since it uses some advanced ideas from the game theory and stuff like that.
I wanted to make sure I'm on the right track. my code works fine, if it helps in any way, here is the code:
 compute_and_filter_features = function(k_top, filter_features=TRUE, model = initial_model, train.x = train_x, test.x = test_x){
    # Compute SHAP values
    shap_values = SHAPforxgboost::shap.values(xgb_model = initial_model, X_train = data.matrix(train_x))
    shap_values$shap_score = as.data.frame(shap_values$shap_score)

    feature_importance = colSums(abs(shap_values$shap_score))

    sorted_features = sort(feature_importance, decreasing = TRUE, index.return = TRUE)
    top_features = names(feature_importance)[sorted_features$ix[1:min(length(sorted_features$ix), k_top)]]

    if (filter_features == TRUE) {
      train_x_filtered = train_x[, top_features]
      test_x_filtered = test_x[, top_features]
    } else {
      train_x_filtered = train_x
      test_x_filtered = test_x
    }

    View(train_x_filtered)
    View(test_x_filtered)

    return(list(train_x_filtered=train_x_filtered, test_x_filtered=test_x_filtered, top_features = top_features, feature_importance = feature_importance))
  }

Comment: What is your statistics question?
Comment: This is for a classifier model (xgboost). My goal is keeping only the informative features for predicting, that's all. @Dave
Comment: So what question do you have? Cross Validated is not about code verification or debugging.
Comment: code verification or debugging is not my question. My question : is SHAP a robust way to preform feature selection? more specially, the `SHAPforxgboost` package in R. I can't find a straight answer for this anywhere ..
Answer: SHAP probably is not as useful as you would like.
In a keynote address to "Why R?", Frank Harrell discusses how feature selection is a mirage. While his simulations do not address SHAP in particular, they demonstrate considerable instability in selecting features. Later comments of his in the presentation, in response to a question about LIME and SHAP, lead to one of my favorite quotes.

Whatever method you're using, if you're afraid to calculate confidence intervals for it, you shouldn't use it.

In the presentation, he mentions bootstrap confidence intervals as a possible approach, particularly bootstrapping the ranks of feature importance.
If you do this for your models and find that you reliably identify the important features and screen-out the unimportant features, that seems like a positive sign (maybe not definitive, but it's at least a positive sign). In a situation like you seem to be in where you have a fairly small number of observations relative to the number of features, you probably do not have enough data to make such reliable claims. That is, I suspect the feature selection based on SHAP importance to be unstable, and with the selected features bouncing all over the place as you make changes to the data (which will be the case when you go predict on new data), there is justifiable doubt that the variables selected based on the training data will be the right variables for making predictions on new data (almost the verbatim phrasing I used in the context of stepwise selection in March).
Finally, I have heard of bizarre attempts of feature selection where a hugely nonlinear model like yours is used to select the features for a (generalized) linear model (such as a logistic regression). Given that a model like XGBoost will model nonlinear and interaction terms that are not part of a (generalized) linear model unless you explicitly code them to be, even stable feature selection for an XGBoost model might lead to silly features being selected for a (generalized) linear model. The XGBoost might find that a feature is important because of a quadratic relationship with the outcome, but if your (generalized) linear model only uses the linear relationship and not the squared term, such a feature might be quite worthless (think about fitting a line to a parabola). You have not said that this is part of your plan, but I include it as a warning to other readers as well as to you, in case your goal is to run a (generalized) linear model on the selected features due to a lack of data.
Comment: Cool, thank you!
Comment: I see. so using SHAP could yield unstable and potentially misleading results. Knowing that feature selection can often be unstable, is there any approach that you recommend that might be better than SHAP and `xgb.importance`  (a non-linear approach cause obviously I'm using non-linear models).
Comment: My suggestion is to quantify the uncertainty in whatever feature selection technique you use. If you're afraid to calculate a confidence interval because it might expose considerable uncertainty, then you should not be using that technique. On the other hand, if such a confidence interval shows the technique to be stable (which I doubt for the amount of data to which you've alluded, but it's is largely an empirical question), you can have more faith in such an approach.
Comment: Thanks! one final question please .. does lightGBM even need feature selection? it should be doing it's own feature selection with pruning right ?
Comment: Your follow-up questions warrant their own posted questions. Remember that Cross Validated is strictly Q&A, not a discussion forum.

SHAP algorithm for feature selecion


Cluster analysis or regression?

Question: I have a table with each row representing single printer model, its features, and price. I want to know how price is formed based on these features. What should I start with? Multiple regression, so I could cut off insignificant features? Cluster analysis to define small clusters with equal price? What are the ways to do the task?
Comment: I think this depends on what you really hope to achieve with the analysis. In your case (given how you describe your data), both methods will be descriptive. Regression will help you answer a question such as _which features have the strongest impact on price?_, whereas clustering (such a MCA) will help you answer questions like _which features are shared by printers of different price range?_.
Comment: @AntoineVernet, that's really an answer -- and a good one. If you post it as an answer, you can earn reputation points. I will upvote.
Comment: @Placidia Thanks. I will make it an answer. However, I think PeterFlom answers the question better, I am merely stating that methods always depend on the question you are asking and that the first questions we ask from a dataset are usually too vague.
Answer: Welcome to the site.
I don't see how cluster analysis helps you with what you want to do.  Regression is much more appropriate.  That is, you have a dependent variable (price) and a bunch of independent variables (features) = a classic regression problem.
Of course, problems may arise. This would depend on how many different printer models there are, how many features there are, how many levels each feature has, and so on.
Comment: +1, in addition, I might note that "cut[ing] off insignificant features" is unlikely to be a good idea.
Comment: Sounds reasonable. I thought clusterization could say for example that there are two segments, 1)If a printer has wifi it will cost $300 no matter what other features it has 2)All other printers where each feature is significant.
Comment: @Ivan Those factors can be very important and ordinary regression cannot reveal them; but lots of plotting of the data can do so. After that, you may wish to fit splines or hockey stick models.
Answer: I think this depends on what you really hope to achieve with the analysis. In your case (given how you describe your data), both methods will be descriptive. Regression will help you answer a question such as which features have the strongest impact on price?, whereas clustering (such a MCA) will help you answer questions like which features are shared by printers of different price range?

Cluster analysis or regression?


Resampling with replacement permutation analysis

Question: My question might be easy for most of you to answer. I am starting to learn statistics and coding in R so my questions are on the basic level.
I have a dataset of (two groups, replicates). My data is split based on the groups so I have 24 samples in group 1 and 20 samples in group 2. My data has replicates. So each set has 4 replicates, hence I have 6 sets in group 1 and 5 sets in group 2. Hence I have assigned indices to them to make it easier during permutation (indices from 1-11). What I want to do now is a routine permutation analysis to obtain the test statistic. I am using non paramteric method with resampling with replacement.
My doubt\problem in R coding is that I have to pool the data together and then resample the groups. When I try to do this, I have to make sure I maintain the sample size for respective groups (that is after resampling the group lables, my new dataset should still contain 6 sets (24 samples) in group 1 and 5 sets (20 samples) in group 2. I am unable to achieve the latter.
I tried something like this in my code:
n_resamples=100

permdat=function(dat){
for (i in 6:nc)
{
 for (j in 1:n_resamples)
   {
        inds=unique(dat$ind)
     cc=dat$grp[dat$ind==inds]
    pcc=sample(cc,replace=T) ##for resampling with replacement
    #m<-list()
    pdat=dat

for(i in 1:length(inds))
{
pdat$grp[pdat$ind==inds[i]]=pcc[i]
#browser()
}

This does not produce the desired result. Kindly help with R codes.
Thanking you all for your help. :)
Regards
Ap
Comment: Could you describe--either in English or mathematics, but please not in non-working `R` code!--what set of permutations you are attempting to generate?
Comment: @whuber: Sorry. I was trying to show my logic because my code is not running properly yet. I am trying to permute the group labels. My null hypothesis is that there is no difference between the mean values between both the groups.
Comment: Are you trying to keep each "set" together, so that at each iteration six of the 11 sets will be randomly chosen to be in group 1 and the remaining five will be in group 2?  (Note that in this case there are just 462 possible combinations without replacement and not a large number with replacement either, so you would be better off generating them systematically rather than randomly.)
Comment: "at each iteration six of the 11 sets will be randomly chosen to be in group 1 and the remaining five will be in group 2?" Yes that is what I should do. Ok I will check further on generating them systematically. Thanks a lot.
Comment: There are 8008 possible selections for group 1 and 3003 for group 2 (which are made independently of group 1).  It might be possible to avoid computing all 3003*8008 combinations, depending on exactly what the test statistic is. However, the coding will likely be easier for random sampling rather than the full distribution.
Answer: With such small numbers of "sets" within each group, it is possible to generate the full permutation distribution.  The following account focuses on the statistical ideas and techniques, using R to illustrate with simulated data like those described in the question.

The number of samples of size $k$ (such as $k=5$ or $k=6$) with replacement from a set of $n$ items (such as $n=11$) equals $\binom{n+k-1}{k}$.  It is useful to know why that is, because that knowledge helps with the coding.  It is based on a one-to-one correspondence between $k$-subsets of $n+k-1$ things and $k$-multisubsets of $n$ things.  (A "multisubset" can have repeated elements.)  By indexing these things $1, 2, \ldots, n+k-1$, we can represent a subset by an ordered list of $k$ distinct indexes
$$1 \le i_1 \lt i_2 \lt \cdots \lt i_k \le n+k-1.$$
Now subtract $0$ from the first index, $1$ from the second, etc., producing the sequence
$$1 \le j_1 = i_1-0 \le j_2 = i_2-1 \le \cdots \le j_k = i_k - (k-1) \le n.$$
Because the $i_s$ can be recovered as $i_s = j_s + s - 1$, this is a one-to-one correspondence.  But the second set of indexes, which range only from $1$ to $n$, allows replicates and therefore enumerates all samples of $n$ things with replacement, QED.
From this we can compute that there are $\binom{11+6-1}{6} = 8008$ ways to simulate the first group and $\binom{11+5-1}{5} = 3003$ ways to simulate the second group.
Depending on the test statistic, there are two ways to proceed.  The most efficient is where the statistic depends separately on the values in the first group and the values in the second.  A simple example would be the difference between the group means.  In such cases we can obtain the full permutation (actually, bootstrap) distributions $F_1$ and $F_2$ of means for groups $1$ and $2$, respectively, by doing just $N=3003+8008$ calculations.  By computing the convolution of $F_1(x)$ and $F_2(-x)$--which can be done in just $O(N)$ operations--we can obtain directly the permutation distribution of the difference of group means.  So, for a computational cost equivalent to a few tens of thousands of random samples, we can obtain the distribution exactly with no sampling error at all.
Less efficient is when the statistic depends intimately on the data in both groups. Then we might have to perform $3003 \times 8008$ calculations.  However, because these can be very fast, the prospect is still not daunting.  The following R code illustrates this method.
To illustrate, let's begin by simulating data like those in the question.  To make it interesting, I will draw medians in each "set" from a standard normal distribution, so that in reality there is no intrinsic difference between the two groups in this regard.  However, I will make the four replicates within each set lognormally distributed and will make the variability greater in the first group.  (This actually makes the expected values in the first group greater than those in the second.)  These conditions of highly skewed distributions (within sets) and heteroscedasticity call into question the validity of conventional parametric analyses, such as ANOVA: that's why a permutation test can be useful.
# Simulation #
n1 <- 6; n2 <- 5; n <- n1+n2 # Group sizes
m <- 4                       # Replicates per set
col1 = rgb(.2, .2, .8, .2)   # Color coding for group 1
col2 = rgb(.8, .2, .2, .2)   # Color coding for group 2

set.seed(17)                                    # Use reproducible numbers
means <- c(rnorm(n))                            # Groups have common means
sds <- c(rep(1,n1), rep(1/4,n2))                # Group 2 has smaller SDs
x <- matrix(exp(rnorm(m*n, means, sds)), nrow=n)# Sets go into rows of `m` values each
par(mfrow=c(1,1))
boxplot(log(t(x)), col=c(rep(col1, n1), rep(col2, n2)),
        main="Simulated data", xlab="Set", ylab="Log value")

The data x are arranged by rows, one row per set.  The first n1 rows contain data for group 1 and the next n2 rows contain data for group 2.

Note that these boxplots are on a logarithmic scale.  The groups are distinguished by color: blue for group 1, red for group 2.  The heteroscedasticity is evident in the consistently shorter red boxes.  The red boxes tend to be higher than the blue ones, but this is an accident due to chance alone.
To proceed, we need to choose a test statistic.  A simple one that should work well is a robust non-parametric analog of group grand means: obtain the median within each set, then return the median of those medians to characterize the general tendency of a group.  To compare the two groups, we will subtract the group 2 median from the group 1 median.  If that is close to $0$, the groups look similar.
statistic <- function(x, i, ...) {
  median(apply(x[i,], 1, median, ...))
}
group1.stat <- statistic(x, 1:n1)
group2.stat <- statistic(x, -(1:n1))
test.stat <- group1.stat - group2.stat

(Note that this is inefficient: in this particular example, we would be better off computing all $11$ set medians once and for all and applying a permutation test to them.  In order to illustrate the general procedure, I don't do that.)
For these data, the test statistic turns out to be $-1.76$, reflecting the evident lowering of the blue boxes compared to the red ones.  We will subject this to a significance test.  Here is the code:
permute <- function(x, stat, n1) {
  # `x` is an array of rows.
  # `stat` is a statistic that can be computed for any group of the rows of `x`.
  # Returns the full resampling (with replacement) distribution of `stat` for
  # `n1` rows at a time.
  multisubsets <- function(k, n) combn(n+k-1, k) - 0:(k-1)
  apply(multisubsets(n1, dim(x)[1]), 2, function(i) stat(x, i))
}

This could be written more compactly without multisubsets, but it might be useful to see the code for obtaining all $k$-multisubsets of $n$ things, so I set it apart.  As previously explained, it obtains all $k$-subsets of $n+k-1$ things (via combn) and then adjusts the indexes.  The result is an array whose columns contain the indexes of rows of x.  The apply line then iterates over those columns, using each in turn to extract a sample of the rows of x and compute its test statistic (median of medians).
Because apply and the sub-indexing that go on are not inherently fast, this is a fairly slow calculation: the next two lines take four seconds on my machine to simulate the full permutation distributions (having $8008$ and $3003$ elements, respectively).
system.time(group1.sim <- permute(x, statistic, n1))
system.time(group2.sim <- permute(x, statistic, n2))

Before we proceed, let's look at these results separately.  I use histograms to show the two distributions.  Remember, these are exact: they are computed without error.
par(mfrow=c(1,2))
hist(group1.sim, probability=TRUE, breaks=25, col=col1)
abline(v = group1.stat, lwd=2)
hist(group2.sim, probability=TRUE, breaks=25, col=col2)
abline(v = group2.stat, lwd=2)

The vertical black lines mark the actual (observed) statistics: that of group 1 is fairly low and that of group 2 is fairly high.  We proceed to compute the exact distribution of the bootstrap statistic--the difference between the group medians--using the brute force approach rather than the efficient $O(N)$ method.  This is very efficiently carried out using a generalized outer product:
par(mfrow=c(1,1))
system.time(simulation.all <- outer(group1.sim, group2.sim, "-"))
hist(simulation.all, probability=TRUE)
abline(v = test.stat, lwd=2, col="Red")

This goes quickly, taking only a quarter second.

The vertical red line shows the observed test statistic of $-1.76$ previously noted.  It looks a little low.  Let's compute its p-value by averaging the upper and lower p-values (this accounts for ties in the data):
p = sum(simulation.all <= test.stat) / length(simulation.all)
q = sum(simulation.all >= test.stat) / length(simulation.all)
(p + 1 - q)/2

The value is $0.0452$: mild evidence of a "significant" difference between these groups.  (It should be doubled, though, for a two-sided alternative hypothesis.)  Because we simulated these data, we know there is no underlying difference in medians.


Intuition for uniform continuity of a function on $\mathbb R$

Question: I know the definitions and examples and differences theoretically about continuity and uniform continuity. Can I tell if a function is uniformly continuous intuitively just by looking at the graph? For example how could I tell that $1/x$ is not uniformly continuous but $\frac{1}{1+x^2}$ is?
Comment: What's your domain, some interval $X$ such that $0\notin X$ but $0\in\overline X$?
Answer: Uniform continuity is somewhat related to bounded derivative. It's not quite the same, of course. For one thing, a uniformly continuous function doesn't even have to be differentiable. And also, there are functions, like the cube root, which has unbounded derivative (and even undefined derivative at the origin), yet they are uniformly continuous.
Here is (what I can consciously conjure right now from) my personal intuition for looking at a function and making an informed guess as to whether they are uniformly continuous:

A differentiable function with bounded derivative is always uniformly continuous (if $M$ is that bound, picking $\delta = \varepsilon/M$ will work, by the mean value theorem)
A differentiable function with unbounded derivative (and possibly isolated points without a derivative), but which doesn't use that "infinite" derivative to actually change much in value, is still likely uniformly continuous (see the cube root function, compared to counterexamples like $\frac1x$, or $\sin(1/x)$, or $x^2$, or $\sin(x^2)$)
A non-differentiable continuous function that's "close enough" to one of the above categories that you can make it into one of them by just "smoothing out" a little roughness is also likely uniformly continuous (the absolute value function is probably the most immediate example, but there are many more)
A function that converges towards a limit (or even towards any straight line) as $x\to \infty$ and as $x\to-\infty$ will be "eventually uniformly continuous", as in if there are any obstructions to uniform continuity, they must be your usual (non-uniform) discontinuities, or holes in the domain, at specific points (consider $x^2$, which doesn't have a limit, where the obstruction to uniform continuity is towards $\pm \infty$, and on the other hand $\sin(1/x)$, which does have a limit towards $\pm \infty$, where the obstruction happens at the origin, as it must by this point)
Comment: Thank you for the clarification, as you pointed out I made an error I meant the function 1/(1+x^2) so I edited it.
Comment: What do you mean by "infinite" derivative to actually change much in value?
Comment: @MojoTop Look at the examples I list. They all have unbounded derivatives (and the cube root even has infinite derivative, if you allow something like that). But apart from the cube root, all the other examples use this huge derivative to actually change a lot in value (in the case of the sine functions, change faster and faster back and forth between $-1$ and $1$, but still). On the other hand, the cube root function doesn't do that. It moves once across the $x$ axis with infinite slope, and then "calms down" again.
Comment: I see now. I think this is a very good and interesting intuitive approach as I will analyze it more thorough with time. Thank you for the beautiful answer!
Answer: In my real analysis course, I came across the following exercise.

Exercise
(a) Let $f: A \to \mathbb R$ be uniformly continuous and suppose $(x_n)$ is a Cauchy sequence in $A$. Prove that $(f(x_n))$ is a Cauchy sequence.
(b) Let $g: (a, b) \to \mathbb R$ be uniformly continuous. Prove that there is a continuous function $\tilde g: [a, b] \to \mathbb R$ such that $\tilde g(x) = g(x)$ for all $x \in (a, b)$, i.e., prove that it is possible to extend $g$ to a continuous function on $[a, b]$ by defining values of $g$ at $a$ and $b$.
(c) Explain why a continuously extendable function $g: (a,b) \to \mathbb R$ must be uniformly continuous.

Remark: This exercise illustrates the essence of uniform continuity: uniformly continuous functions are exactly the ones that can be extended to continuous functions on closed sets. Wikipedia has a nice discussion on this topic.
I encourage you to complete this exercise to build your own understanding.

For your examples, $f(x) = \frac{1}{x}$ is defined on $\mathbb R\setminus \{0\}$ which is not a closed set and it cannot be extended continuously to its closure $\mathbb R$ either. However, $g(x) = \frac{1}{1 + x^2}$ is continuous on $\mathbb R\setminus \{0\}$ and can be continuously extended to its closure $\mathbb R$ by setting $g(0) = 1$.
Comment: I don't see how this can be "easily extended" to unbounded domains in a way that makes $x^2$ fail yet $x$ succeed to be uniformly continuous. But maybe I am just not seeing some clever trick.
Comment: This is only about functions that are uniformly continuous (or not) on a bounded set. It fails to capture that, for instance, $x^2$ is _not_ uniformly continuous on the real numbers.
Comment: @Arthur Sure, but this is an exercise to build intuition which is why it considers a simplified case. One can easily extend the argument to work for unbounded domains.
Comment: @Arthur I think I understand what you mean now. You are correct that this does not explain why $x^2$ is not uniformly continuous. I guess $x^2$ is not uniformly continuous on any dense subset is one explanation, but you would still require an explanation of why that is the case which this answer does not give. However, I think this still provides intuition for why we care about uniform continuity.
Comment: To explain why $x^2$ is not uniformly continuous, I would draw a picture of the definition. For every vertical radius $\epsilon > 0$, we require a constant horizontal radius $\delta > 0$ such that if a box of height $2\epsilon$ and width $2\delta$ is centred at a point on the graph, the graph will not penetrate the top or bottom of the box. The picture would be akin to the animations and visualisations found on [Wikipedia](https://en.wikipedia.org/wiki/Uniform_continuity). As you mention, the derivative of $x^2$ is unbounded so such a $\delta$ does not exist.


Eigenvectors belonging to different eigenvalues are linearly indepenent. Problem with proof.

Question: I am reading up on some functional analysis to prepare for an upcoming subject. I came across the following theorem and proof.
Let $X$ be a linear space and $T:X\to X$ be a linear operator. Then the eigenvectors corresponding to different eigenvalues are linearly independent. Then the proof proceeds as:
Assume that $x_1,\ldots,x_n$ are eigenvectors corresponding to $\lambda_1,\ldots,\lambda_n$ distinct eigenvalues. Then suppose that $x_m$ is the first eigenvector such that
$$x_m = \mu_1x_1+\ldots +\mu_{m-1}x_{m-1}.$$
Then a contradiction follows and I have no problems with this part of the proof. My problem/confusion is with the first part. What if we have a subset of eigenvectors that are not linearly independent that has cardinality greater than $\mathbb R$. Then how are we supposed to order those and talk about a 'first' eigenvector.
Maybe I am missing something trivial, I feel like my understanding of these concepts is shaky which does not help. Maybe someone can explain what I am missing or confirm my confusion in that this is indeed a problem. Perhaps there is an alternative proof that deals with this situation. Thanks!
Answer: A family of vectors is linearly dependent iff there exists a nontrivial linear combination of them equal to zero. Note that a linear combination is an expression of the from
$$ \sum_{i\in I}\mu_i x_i$$
such that all but finitely many of the $\mu_i$ are zero!

So to recap, let $I$ be an index set, for each $i$ let $x_i\in X$ be a (nonzero!) eigenvector of eigenvalue $\lambda_i$ and assume that $i\ne j$ implies $\lambda_i\ne \lambda_j$.
If we asssume that the family of vectors $\{x_i\}_{i\in I}$ is linearly dependent, then there exists a function $c\colon I\to \mathbb R$, $i\mapsto c_i$ such that $\operatorname{supp}(c):=\{\,i\in I\mid c_i\ne 0\,\}$ is finite and nonempty and $\sum_{i\in I}c_ix_i=0$.
Since we assume that $\operatorname{supp}(c)$ is finite, there exists among all such $c$ one that minimizes the natural number $|\operatorname{supp}(c)|$. Clearly, we have $|\operatorname{supp}(c)|\ge 2$ as all $x_i$ are nonzero.
Then for such a minimal $c$, one picks $i_0\in I$ and notes that also $T(0)-\lambda_{i_0}\cdot 0=\sum_{i\in I}\underbrace{(\lambda_i-\lambda_{i_0}) c_i}_{d_i}x_i=0$, and that $\operatorname{supp}(d)=\operatorname{supp}(c)\setminus\{i_0\}$. If $i_0\in\operatorname{supp}(c)$, this contradicts the minimality of $c$. Hence $\{x_i\}_{i\in I}$ is linearly independent.
Comment: Thanks for your answer, I get what you are saying but I am afraid I am not sure how it helps. The contradiction in the proof comes from the fact that $x_m$ is the first eigenvector that can be written as a linear combination of the first $m-1$ eigenvectors. But how can we talk about such ordering if we have an index set of cardinality greater than $\mathbb R$? In other words, how can we possibly say this is the first eigenvector such that...
Comment: @Slugger the point that Hagen is making is that we can always reduce the indices under consideration to a finite set by throwing out the $x_i$ with zero coefficients.
Comment: @Slugger Good point. No order is needed as can be seen from my additional remarks.
Comment: Ah perfect I see now! Thanks for clearing up the confusion. I did not realize that linear combinations should satisfy the finite support property you describe.


Como criar expressão regular para buscar números em parênteses?

Question: Estou tentando criar uma Regex para pegar números que estão em parênteses. Assim:
1) Pergunta 1
2) Pergunta 2
3) Pergunta 3
4) Pergunta 4
.
.
10) Pergunta 10

Até agora consegui chegar nisso: (^[0-9].* )
Mas não pegou o 10).
Necessariamente eu preciso pegar o número, o parênteses e o espaço depois.
Answer: Uma opção é usar:
^(\d+\) )

Os parênteses possuem significado especial em regex, então para que ela capture o próprio caractere ), você deve escrevê-lo como \). E repare que há um espaço entre o \) e o último ), para que a regex pegue o espaço depois do parênteses.
Eu também usei o quantificador + em vez de *, pois o * significa "zero ou mais ocorrências" (ou seja, se não tiver nenhum dígito, também serve), enquanto o + significa "uma ou mais ocorrências" - assim eu garanto que deve ter pelo menos um dígito. (se eu usar *, a regex pode pegar linhas que não tem dígitos no início, como ) etc... - veja).
Eu coloco o trecho que quero entre parênteses (os números, \) e espaço), pois assim eles formam um grupo de captura que contém todo este trecho, assim ele pode ser facilmente recuperado depois. Veja no regex101.com que os grupos ficam destacados em verde (e veja que, graças ao + em vez de *, ela não pega mais as linhas que não tem dígitos).
E o marcador ^ garante que este trecho só será verificado no início da string.

Você também pode trocar \d por [0-9]. Dependendo da linguagem/engine não fará diferença, mas em algumas o \d pode aceitar quaisquer caracteres da categoria Unicode "Number, Decimal Digit", o que inclui caracteres como ٠١٢٣٤٥٦٧٨٩, entre outros (veja esta resposta para mais detalhes). Se você sabe que seus textos não possuem tais caracteres, tanto faz usar um ou outro. Mas se quiser limitar a somente os dígitos de 0 a 9, use [0-9] em vez de \d.
A outra resposta sugeriu usar \s em vez do espaço (que também funciona). Mas este atalho também corresponde a outros caracteres, como quebras de linha (\n, \r), TAB, entre outros (também varia conforme a linguagem/engine). Se quiser que a regex pegue somente o espaço em branco, use o que eu sugeri acima.
Comment: Muito boa sua explicação. Obrigado. Vai ajudar bastante gente.
Answer: Você precisa escapar o parênteses com uma barra para a regex não considerá-la parte da expressão, e a expressão "[0-9]" pode ser trocada por "\d":
^(\d+\)\s)


Interpreting effects of linear model

Question: In Alan Agresti's book, Foundations of Linear and Generalized Linear Models (section 1.2.3), letting $y_i$ is a college student's math test score, $x_{i1}$ the number of years of math education, $\mu_i = E(y_i) = \beta_0+\beta_1x_{i1}$, he writes "...we might say 'if this model holds, a one-year increase in math education corresponds to change of $\beta_1$ in the expected math achievement test score.' However, this may suggest the inappropriate casual conclusion that if a student attains another year of math education, her or his math achievement test score is expected to change by $\beta_1$."
He writes that this conclusion is not valid because "a higher mean test score at a higher math education level (if $\beta_1 >0$) could at least partly reflect the correlation of several other variables with both test score and math education level....".
I'm not sure why the first statement is wrong. If the model holds, then my understanding is that we know that $E(y_i) = \beta_0+\beta_1x_{i1}$ holds, hence it doesn't matter what the correlation is at other levels because the equation suggests that the first statement holds.
He write that a more appropriate interpretation is "If the model holds, when we compare the subpopulation of students having a certain number of years of math education with the subpopulation having one fewer year of math education, the difference in the means o their math achievement test scores is $\beta_1$."
Why is this interpretation more appropriate? More specifically, why is he using subpopulation, and why is he switching to a difference in means, rather than sticking to one mean?
Answer: This is best explained by considering what object you're really estimating and viewing this problem in a causal framework.
Let's first consider what you're really modeling.
Let me write your model on an individual level as $$y_i = \beta_0 + \beta_1 x_i + \epsilon_i \,,$$where I introduce an error term $\epsilon_i$, which is supposed to capture everything that is unexplained by the model.
To give the model any meaning, as in your question, we have to impose restrictions on the errors. Most commonly we assume $\mathbb{E}[\epsilon_i \mid x_i=x] = 0$. Then the we get the same model for the conditional mean as in your question$$\mu(x) = \mathbb{E}[y_i \mid x_i=x] = \beta_0 + \beta_1x\,.$$
If you now consider $\mu(x')$ at some point $x'$, then this object corresponds to the mean of the outcomes $y_i$ of the subpopulation where $x_i = x'$. Hence, we may write $\beta_1 = \mu(x + 1) - \mu(x)$, but this does not mean that the individual outcomes increase (or decrease) by $\beta_1$ but only the conditional means, i.e. the means of the subpopulations are different by $\beta_1$.
Secondly, without getting technical, assume that there is another variable influencing $y_i$ and $x_i$. In your setting this could be mathematical intelligence. Individuals who have a higher mathematical intelligence score better on math tests (on average) but are also more likely to have a longer math education (on average). If you then compare individuals who have a one-year longer education, it is not clear if their better performance on the math test is due to the increase in education alone, or also due to their higher affinity towards math which led to the longer education. Hence, you cannot simply interpret $\beta_1$ as the effect of one-year more education on the individual alone, but only as a description of the subpopulations, i.e. the conditional mean, as seen above. For reference see omitted-variable bias and endogeneity.
Lastly, it is possible to get causal interpretations like the ones you aim for; however, you have to make causal assumptions for that. One way to do this is to describe how each observed and unobserved variable is related in your model. I suggest reading up on causal graphs.

Interpreting effects of linear model