Computing $(A\otimes I + I \otimes A)^{-1} \text{vec}B$

Question: Suppose I have two positive semi-definite $n$-by-$n$ matrices $A$, $B$ and an $n$-by-$n$ identity matrix $I$, and I'm looking for a way to compute, approximate or bound the following quantity:
$$(A\otimes I + I \otimes A)^{-1} \text{vec}B$$
Concretely I'm dealing with matrices with $n$ ranging from $100$ to $4000$, so $A$ is easy to invert, while $A \otimes I$ is too large, so need a way to compute this using operations on $n$-by-$n$ matrices
Additionally, I found the following to give a decent approximation when $A$, $B$ can be Kronecker-factored, wondering if there's a reason for that.
$$0.5 A^{-0.5} B  A^{-0.5}$$
Any tips or literature pointers are appreciated!
Comment: Is your issue connected to the vector space of matrices commuting with the given matrix $A$ like in https://www.hindawi.com/journals/mpe/2009/650970/ ?
Comment: You might want to try using the [integral formula](https://en.wikipedia.org/wiki/Lyapunov_equation) for the solution, with some kind of quadrature method.
Comment: The integral formula mentionned by @Omnomnomnom might be a way to reach your approximation $0.5 A^{-0.5} B  A^{-0.5}$.
Comment: Why don't you answer my little question asked 2 days ago ?
Comment: The motivation was to compute Equation 3 in https://arxiv.org/pdf/1610.03774.pdf , I don't know if matrices commuting helps here
Answer: Your equation is equivalent to solving the following equation for $P$
$$
A\,P + P\,A^\top = B. \tag{1}
$$
Such equation also solves for the infinite time (controllability) Gramian, which can also be calculated with
$$
P = -\int_0^\infty e^{A\,t}\,B\,e^{A^\top t}\,dt. \tag{2}
$$
This integral should also be of size $n \times n$. As time goes to infinity $e^{A\,t}$ is not well defined when $A$ is positive semi-definite. It can also be noted that in order to be able to solve for $P$ the spectra of $A$ and $-A$ need to be disjoint. In order words $A$ can't have an eigenvalue of zero, which would imply that $A$ should be positive definite instead of positive semi-definite. When multiplying both sides of $(1)$ by minus ones keeps it equivalent to the original problem. This is also equivalent to multiplying both $A$ and $B$ by minus one, which transforms $(2)$ into
$$
P = \int_0^\infty e^{-A\,t}\,B\,e^{-A^\top t}\,dt. \tag{3}
$$
Now the term $e^{-A\,t}$ does stay bounded and converges to zero as time goes to infinity. The rate of how fast this integral converges should be roughly inversely proportional to the smallest eigenvalue of $A$, however any finite time numerical approximation of this integral can be used as a lower bound for $P$.
Comment: Kwin, the equation is $AP+PA^T=B$ and in the formula for $P$: i)  if the condition  on $A$ is that its eigenvalues have $>0$ real part, then change $A,A^T$ into $-A,-A^T$. ii)   if the condition  on $A$ is that its eigenvalues have $<0$ real part, then change $B$ into $-B$.
Comment: Thanks! Looks like it's implemented in scipy already as https://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.linalg.solve_sylvester.html
Comment: @RodrigodeAzevedo you can just multiply both $A$ and $B$ with minus one in the first equation, which results in the following integral$$P = -\int_0^\infty e^{-A\,t}\,B\,e^{-A^\top t}\,dt.$$
Comment: @RodrigodeAzevedo I have updated my answer.
Comment: @loupblanc I indeed mixed up a minus a sign in $(1)$ (and the equation after that), it should be fixed now.
Answer: $
\def\LR#1{\left(#1\right)}
\def\op#1{\operatorname{#1}}
\def\vcc#1{\op{vec}\LR{#1}}
\def\trace#1{\op{Tr}\LR{#1}}
\def\qiq{\quad\implies\quad}
\def\p{\partial} \def\grad#1#2{\frac{\p #1}{\p #2}}
\def\c#1{\color{red}{#1}}
\def\fracLR#1#2{\LR{\frac{#1}{#2}}}
$The SANE algorithm by Raydan & Martinez is basically Barzilai-Borwein applied to a system of nonlinear equations
rather than a scalar cost function. It satisfies a least-squares form of the Secant Equation, so it can also be viewed as a Quasi-Newton method.
Anyway, their algorithm can be easily adapted to matrix-valued functions
$$\eqalign{
F(X) &\equiv AX+XA^T-B = 0 \qquad&\{ {\rm matrix\;NLE\;function} \} \\
\\
X_0 &= I &\{ {\rm initialize} \} \\
F_0 &= F(X_0) \\
\\
k &= 1  &\{ {\rm first\;step} \} \\
dX_1 &= -F_0\cdot 10^{-5} \\
X_1 &= X_0 + dX_1 \\
F_1 &= F(X_1) \\
dF_1 &= F_1 - F_0 \\
\\
{\rm while\;} \big\|F_k\big\|_F^2 \:&> 10^{-12} \\
dX_{k+1} &= -\fracLR{dX_k:dF_k}{dF_k:dF_k}F_k
 \qquad&\{ {\rm Barzilia\,Borwein\;step} \} \\
X_{k+1} &= X_k + dX_{k+1} \\
dF_{k+1} &= A\:dX_{k+1} + dX_{k+1}\,A^T
 &\equiv\; F_{k+1} - F_k \\
F_{k+1} &= F_k + dF_{k+1} \\
k &= k+1 \\
}$$
where a colon has been used to denote the matrix inner product, i.e.
$$\eqalign{
A:B &= \sum_{i=1}^n\sum_{j=1}^n A_{ij}B_{ij} \\
A:A &= \|A\|^2_F \\
}$$
The nice thing about this algorithm is that it deals with
$n\times n$ matrices rather than $n^2\times n^2$ Kronecker matrices.
Furthermore, it does not require any matrix inversions or the evaluation of transcendental matrix functions. Each iteration requires exactly one evaluation of the objective function. All of these features make each iteration relatively quick.
Answer: Given symmetric matrices $\mathrm A \succ \mathrm O_n$ and $\mathrm B \succeq \mathrm O_n$, let
$$\mathrm X (t) := \int_0^t e^{-\tau \mathrm A} \mathrm B e^{-\tau \mathrm A} \,\mathrm d \tau$$
Using integration by parts,
$$\mathrm A \mathrm X (t) = \cdots = -\dot{\mathrm X} (t) + \mathrm B - \mathrm X (t) \mathrm A$$
and, thus, we obtain the following matrix differential equation
$$\boxed{\dot{\mathrm X} = - \mathrm A \mathrm X - \mathrm X \mathrm A + \mathrm B}$$
with initial condition $\mathrm X (0) = \mathrm O_n$. Assuming $\rm X$ converges to a steady-state, $\dot{\mathrm X} = \mathrm O_n$, we obtain the following linear matrix equation
$$\mathrm A \mathrm X + \mathrm X \mathrm A = \mathrm B$$
whose solution we denote by $\bar{\rm X}$. This is the desired solution.
Using a numerical ODE solver, we then integrate the matrix differential equation above and find an approximate value of matrix $\bar{\rm X}$. Since matrix $\rm X (t)$ is symmetric for all $t \geq 0$, we can half-vectorize both sides of the matrix differential equation in order to obtain a $\binom{n+1}{2}$-dimensional state vector (rather than a $n^2$-dimensional state vector, which would be wasteful).

Appendix
Note that the solution of the matrix differential equation is given by
$$\mathrm X (t) = \bar{\mathrm X} - e^{-t \mathrm A} \bar{\mathrm X} e^{-t \mathrm A}$$
Since $\mathrm A \succ \mathrm O_n$,
$$\lim_{t \to \infty} \mathrm X (t) = \bar{\mathrm X}$$
If matrix $\rm A$ were allowed to have an eigenvalue at zero, it would be unpleasant.


Rewriting output of a field in node.tpl

Question: One way is to use $node->field_myfield[LANG][0][value]="value" that needs another line for getting field's language using field_language().
Is there an easier and cleaner way?
Answer: You should use the field_view_field() function. Have a look at Rendering Drupal 7 fields (the right way).

Rewriting output of a field in node.tpl


Download Web Page with JSON in order?

Question: I am trying to get all of the currency rates from this website and processing it in my app.
This is the website: https://raw.github.com/currencybot/open-exchange-rates/master/latest.json
I am using this class to get all of the currency rates, but the problem is that it is getting it in a random order because of Iterator keys = ratesObj.key();
How do I fix this so that it gets all of the currency rates in order?
This is the code:
private class DownloadWebPageTask extends AsyncTask<String, Void, ArrayList<Double>>
{
    protected ArrayList<Double> doInBackground(String... urls)
    {
        ArrayList<Double> response = new ArrayList<Double>();
        for (String url : urls)
        {
            DefaultHttpClient client = new DefaultHttpClient();
            HttpGet httpGet = new HttpGet(url);
            try
            {
                HttpResponse execute = client.execute(httpGet);
                InputStream content = execute.getEntity().getContent();
                InputStreamReader isr = new InputStreamReader(content);
                StringBuilder sb = new StringBuilder();
                char[] buffer = new char[1024];
                while (isr.read(buffer) != -1)
                {
                    sb.append(buffer);
                }
                JSONObject jobj = new JSONObject(sb.toString());
                JSONObject ratesObj = jobj.getJSONObject("rates");
                Iterator<String> keys = ratesObj.keys();
                while (keys.hasNext())
                {
                    response.add(Double.parseDouble(ratesObj.getString(keys.next())));
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
        return response;
    }
    @Override
    protected void onPostExecute(ArrayList<Double> result)
    {
        currency_values = result;
    }
}

Thank you all for your help! I really appreciate it.
UPDATE: This is the code now.
try
            {
                HttpResponse execute = client.execute(httpGet);
                InputStream content = execute.getEntity().getContent();
                String s = EntityUtils.toString(execute.getEntity());
                JSONObject jobj = new JSONObject(s);
                JSONObject ratesObj = jobj.getJSONObject("rates");
                Iterator<String> keys = ratesObj.keys();
                while (keys.hasNext())
                {
                    response.add(Double.parseDouble(ratesObj.getString(keys.next())));
                }
            } catch (Exception e) {
                e.printStackTrace();
            }

Comment: Don't use `InputStreamReader` and `StringBuilder` when there's `EntityUtils.toString(execute.getEntity())` which obeys the content encoding set by the server, is much less code and has proper error handling.
Comment: Then this would change the loop below it. How would I fix the loop then?
Comment: If you mean the `while (isr.read(buffer) != -1)` loop, it would not be changed -- you would remove it completely, as it's no longer necessary if you use `EntityUtils.toString()`.
Comment: What do you intend to do with just the doubles that relies on the correct ordering? Why not use a `Map` mapping currency code to exchange rate and not care about the ordering at all?
Comment: Philipp, I understand what you mean. I did it separately, but your solution seems to make more sense. However, I tried your method, but now I'm getting IndexOutOfBoundsException. This is what the code looks like:
Comment: You should remove the line that reads `InputStream content = execute.getEntity().getContent();`, too.
Comment: I did, however it still does not work. It runs, it just outputs the wrong answer.
Comment: My advice to use `EntityUtils` had nothing to do with preserving the order of exchange rates, just cleaner code. For that, see the 4th comment.
Comment: Would you happen to know how to use Map? I understand why its more efficient and easier to implement, but it would be greatly appreciated if there was an example I could look at. Thanks!
Comment: Check my answer below where I tried to put everything from my comments together :)
Answer: Here's one way to implement what I suggested in the comments, using a SortedMap<String, Double> to map the three-letter currency codes to exchange rates while keeping it alphabetically ordered by currency code:
  protected SortedMap<String, Double> doInBackground(String... urls) {
    SortedMap<String, Double> result = new TreeMap<String, Double>();

    HttpClient client = new DefaultHttpClient();
    for (String url : urls) {
      try {
        HttpGet request = new HttpGet(url);
        HttpResponse response = client.execute(request);
        String jsonString = EntityUtils.toString(response.getEntity());

        JSONObject json = new JSONObject(jsonString);
        JSONObject rates = json.getJSONObject("rates");
        Iterator<String> currencyCodes = rates.keys();
        while (currencyCodes.hasNext()) {
          String currencyCode = currencyCodes.next();
          double rate = rates.getDouble(currencyCode);
          result.put(currencyCode, rate);
        }
      } catch (Exception ex) {
        Log.e(TAG, "Failed to download exchange rates from " + url, ex);
      }
    }
    return result;
  }

Comment: Yes, it does work. Thank you so much for helping a young programmer like me. I appreciate it!

Download Web Page with JSON in order?


Ordenar duas listas com base na ordem da primeira

Question: Vou dar um exemplo pois fica mais claro para explicar:
x = [7,5,9,4,2,3,8,1,10]
y = ['sete', 'cinco', 'nove', 'quatro', 'dois', 'tres', 'oito', 'um', 'seis', 'dez']

Preciso ordenar a lista x, de modo que x[i] = y[i], ou seja, as listas devem ficar assim:
new_x = [1,2,3,4,5,6,7,8,9,10]
new_y = ['um', 'dois', 'tres', 'quatro', 'cinco', 'seis','sete', 'oito', 'nove', 'dez']

Exemplo:
x[5] = 3
y[5] = 'tres'

Após ordenar:
new_x[5] = 6
new_y[5] = 'seis'

Basicamente precisaria ordenar a segunda lista ao mesmo tempo que ordeno a primeira. De modo que os elementos sigam pareados.
Como poderia fazer isso?
Comment: Na sua lista `x` está faltando o número 6, mas pela descrição do problema, creio que foi erro de digitação...
Answer: Uma alternativa é criar uma lista de índices, e ordenar estes índices de acordo com o respectivo valor em x.
Depois você usa estes índices para ordenar as listas:
x = [7, 5, 9, 4, 2, 3, 8, 1, 6, 10]
y = ['sete', 'cinco', 'nove', 'quatro', 'dois', 'tres', 'oito', 'um', 'seis', 'dez']

# ordene os índices em vez dos elementos em si
indices = list(range(len(x)))
indices.sort(key=lambda i: x[i]) # ordene os índices com relação ao seu respectivo valor em x

# crie as listas baseado na ordem dos índices
new_x = [x[i] for i in indices]
new_y = [y[i] for i in indices]

print(new_x) # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
print(new_y) # ['um', 'dois', 'tres', 'quatro', 'cinco', 'seis', 'sete', 'oito', 'nove', 'dez']

list(range(len(x))) cria a lista com os índices (no caso, os números de zero até o tamanho de x).
A seguir, ordenamos os índices, usando como critério o respectivo valor na lista x. Após o sort, a lista indices será [7, 4, 5, 3, 1, 8, 0, 6, 2, 9], o que quer dizer que x[7] é o primeiro elemento de x (quando este for ordenado), x[4] é o segundo, etc, e o mesmo vale para y.
Sendo assim, basta colocar as listas x e y na ordem indicada pela lista indices.

Outra forma de criar a lista de índices já ordenada seria:
indices = sorted(range(len(x)), key=lambda i: x[i])

Comment: @LucasVasques
Se a resposta resolveu o seu problema, vc pode aceitá-la, [**veja aqui**](https://pt.meta.stackoverflow.com/q/1078/112052) como e porque fazê-lo. Não é obrigatório, mas é uma boa prática do site, para indicar a futuros visitantes que aquilo resolveu o problema.
Comment: Muito obrigado!! Ajudou muito!
Answer: Tem a caminho da programação funcional utilizando as funções builtin zip() para juntar as duas lista, sorted() para ordenar essa junção e map() para aplicar uma função separadora da listas ordenadas:
x = [7, 5, 9, 4, 2, 3, 8, 1, 6, 10]
y = ['sete', 'cinco', 'nove', 'quatro', 'dois', 'tres', 'oito', 'um', 'seis', 'dez']

xy = zip(x, y)                               #Junta as duas listas numa lista d tuplas: [(7, 'sete'), (5, 'cinco'), (9, 'nove'), (4, 'quatro'), (2, 'dois'), (3, 'tres') (8, 'oito'), (1, 'um'), (6, 'seis'), (10, 'dez')] .

sorted_xy = sorted(xy, key= lambda t: t[0])  #Orderna xy com base no primeiro elemento de cada tupla.

new_x= list(map(lambda t: t[0], sorted_xy))  #Extrai o resultado da ordenação da lista x.
new_y= list(map(lambda t: t[1], sorted_xy))  #Extrai o resultado da ordenação da lista y.

print(new_x)
print(new_y)

Resultando:
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
['um', 'dois', 'tres', 'quatro', 'cinco', 'seis', 'sete', 'oito', 'nove', 'dez']


Como criar árvore binária com algoritmo iterativo?

Question: Estou fazendo um programa que aloca um dicionário na memória para verificar palavras com erros ortográficos em arquivos de texto. A função abaixo converte as palavras do dicionário em inteiros e os armazena em estruturas árvore.

Minha dúvida é como escrever corretamente e iterativamente a parte do
código que vai armazenando os números na árvore(a partir da linha 24).

Vi muitos exemplos, mas todos recursivos e não entendi muito bem, fique à vontade em dar qualquer tipo de exemplo para outros usuários que possam ter a mesma dúvida.
typedef struct arvore
{
    unsigned int n;
    struct arvore *esq, *dir;
}arvore;

arvore *raiz;

unsigned int
BPHash(const char* str, unsigned int len)
{
    unsigned int hash;
    for (unsigned int i = 0; i < len; str++, i++)
        hash = hash << 7 ^ (*str);
    return hash;
}

bool
load(const char *dict)
{
    FILE *fp = fopen(dict, "r"); // dict = arquivo que serve como dicionário
    if (fp == NULL)
        return false;

    char str[LENGTH+1]; // LENGTH = tamanho máximo da string (45)
    unsigned int strtam, hash; // Tamanho da string e string convertida em inteiro com função hash
    struct arvore dicio = {0, NULL, NULL};
    raiz = &dicio; // Ponteiro global

    while (fgets(str, LENGTH, fp) != NULL)
    {
        strtam = strlen(str);
        hash = BPHash(str, strtam); // BPHash = função hash

        if (raiz->n == 0) // Minha dúvida se refere as linhas abaixo
            raiz->n = hash;

        else if (raiz->n < hash)
        {
            raiz->esq = (arvore*)malloc(sizeof(arvore));
            raiz->dir = NULL;
            raiz->n = hash;
        }

        else
        {
            raiz->dir = (arvore*)malloc(sizeof(arvore));
            raiz->esq = NULL;
            raiz->n = hash;
        }
    }

    return true;
}

Comment: Poderia colocar a sua função `BPHash` também?
Answer: Seu primeiro problema é que a raiz da árvore (dicio) está declarada dentro da própria função load, alocada estaticamente na pilha de execução (variáveis locais são alocadas na pilha de execução). Isso significa, que quando a função terminar, a parte correspondente da pilha de execução vai ser limpa e com ela a raiz da árvore. A raiz ainda poderia ser recuperada pela variável global raiz, mas isso vai ser um ponteiro que vai apontar para uma parte da pilha que não existe mais e que em breve será sobreescrita, tornando-se então em lixo.
A melhor solução para isso seria retornar a raiz da árvore ou NULL se ela não puder ser criada. Evite usar variáveis globais, pois fazer isso é uma péssima prática de programação. Mas, como você está sendo obrigado a trabalhar dessa forma, então infelizmente não há outra saída.
Outro problema é que a sua árvore tem no máximo três nós: raiz, raiz->dir e raiz->esq. Aliás, só terá dois, pois quando você cria o nó esquerdo, você apaga o direito e vice-versa. Além disso, você apaga esses nós sem desalocá-los, ocasionando vazamento de memória. Você nunca está criando ou acessando nós em níveis mais profundos.
Seu código revisado é esse:
// LENGTH = tamanho máximo da string
#define LENGTH 45

// função hash
unsigned int BPHash(const char *str, int len) {
    unsigned int hash = 0;
    for (unsigned int i = 0; i < len; str++, i++) {
        hash = hash << 7 ^ (*str);
    }
    return hash;
}

typedef struct arvore {
    unsigned int n;
    struct arvore *esq, *dir;
} arvore;

arvore *raiz = NULL;

bool load(const char *dict) {

    // Primeiro, destroi qualquer árvore que já houvesse antes, para evitar vazamento de memória.
    destruir_arvore(raiz);
    raiz = NULL;

    FILE *fp = fopen(dict, "r"); // dict = arquivo que serve como dicionário
    if (fp == NULL) return false;

    char str[LENGTH + 1];

    while (fgets(str, LENGTH, fp) != NULL) {
        arvore *novo = (arvore *) malloc(sizeof(arvore));
        novo->n = BPHash(str, strlen(str));
        novo->esq = NULL;
        novo->dir = NULL;

        arvore *no = raiz;

        while (1) {
            if (no == NULL) {
               raiz = novo;
               break;
            } else if (no->n < hash) {
                if (no->esq == NULL) {
                    no->esq = novo;
                    break;
                }
                no = no->esq;
            } else {
                if (no->dir == NULL) {
                    no->dir = novo;
                    break;
                }
                no = no->dir;
            }
        }
    }

    return true;
}

Observe que temos dois whiles aqui. O while externo percorre o arquivo lendo as strings dele, e a cada iteração, um nó novo é criado. O while interno usa um ponteiro no, que começa na raiz e vai descendo na árvore (no = no->esq; e no = no->dir;) até encontrar um lugar vazio onde o nó novo possa ser inserido. Quando o nó novo é inserido na árvore, dá-se um break; no while interno. O if (no == NULL) é para criar a raiz da árvore. Essa abordagem ainda tem a vantagem de que o zero não é especial e a função funciona mesmo se BPHash retornar zero. Não é utilizada recursão aqui.
Uma função para destruir a(s) árvore(s) criada(s) também é necessária a fim de evitar-se vazamentos de memória (dessa vez com recursão, já que a versão sem recursão é bem mais complicada):
void destruir_arvore(arvore *no) {
    if (no == NULL) return;
    destruir_arvore(no->esq);
    destruir_arvore(no->dir);
    free(no);
}

É claro que isso tudo depende de a função BPHash ter sido implementada adequadamente (faltou inicializar o hash com zero nela). Não sei se a implementação é ou não adequada. Também não sei se essa árvore será de fato útil ao que você quer, mas se for, é do jeito que coloquei acima que você irá construí-la.
Comment: @Matheus [continuação] É de se crer que `sizeof(arvore) == sizeof(unsigned int) + sizeof(arvore *) + sizeof(arvore *)`, mas é permitido ao compilador acrescentar uns bytes a mais de *padding* para se aproveitar de alinhamentos de dados na memória. Também é de se esperar que todos os ponteiros tenham o mesmo tamanho, seja ele 4 ou 8 bytes (em arquiteturas de 16 bits isso não acontecia, havia dois tipos de ponteiros, `near` e `far` de tamanhos diferentes). [continua]
Comment: @Matheus [continuação] De acordo com [**este teste**](https://ideone.com/zHKfY2), o `unsigned int` tem 4 bytes, cada ponteiro tem 8 bytes, os componentes da estrutura somam 20 bytes, mas o tamanho da estrutura é de 24 bytes, havendo portanto 4 bytes de sobra (*padding*). Mas novamente, isso daí é algo que o compilador tem poder de decidir e depende da plataforma, do compilador em questão, de otimizações aplicadas, do ambiente de execução, etc.
Comment: @Matheus https://ideone.com/ft87Kj
Comment: Adicionei a função `BPHash` e mais alguns comentários.
_"Seu primeiro problema é que a `raiz` da árvore está declarada dentro da própria função `load`"_
O ponteiro `raiz` é uma variável global, está declarado fora de qualquer função.
_"A melhor solução para isso é retornar a raiz da árvore ou NULL se ela não puder ser criada"_
Neste exercício não posso mudar a declaração de `load`, tem que retornar um `bool`, por isso criei o ponteiro global raiz, para acessar a `arvore dicio` de qualquer função.
Comment: _"Aliás, só terá dois, pois quando você cria o nó esquerdo, você apaga o direito e vice-versa"_
Essa é minha maior dificuldade, criar novos nós iterativamente, e não sobrescrevê-los.
_"Além disso, você apaga esses nós sem desalocá-los, ocasionando vazamento de memória. Você nunca está criando ou acessando nós em níveis mais profundos."_
Estou ciente disso, não me preocupei em desalocar porque ainda nem consegui criar _rsrs_.
Comment: @Matheus Resposta editada. Quanto ao primeiro item, eu havia essquecido de dizer que me referia a variável `dicio`.
Comment: Consegui subir mais um degrau na escada da estrutura de dados, obrigado! Só mais uma dúvida: o ponteiro do tipo arvore `raiz` possui quatro bytes (int) que armazenam endereço, e os bytes seguintes dados da struct arvore(n, *esq, *dir)? Não entendi se `raiz` armazena só um ponteiro pra memória ou toda struct.
Comment: @Matheus O tamanho do ponteiro depende da plataforma. Se não me engano, em programas compilados com arquiteturas de 32 bits ele tem 4 e ccom arquiteturas de 64 bits ele tem 8 bytes, mas posso estar enganado. O ponteiro aponta para algum lugar no heap. Nesse lugar está alocado a estrutura. O tamanho da estrutura pode ser computado com `sizeof(arvore)`, tanto que é essa quantidade de memória que você aloca com o `malloc` exatamente por essa razão. [continua]


Linux Multiple Users Permission Read / Write

Question: I host my websites on my Linux server each out of their own accounts (e.g. /home/siteA-com/http, /home/siteB-com/http etc.). I chown www-data on the inner files for the http folders so that the server can make changes where needed (namely for Wordpress). How can I also allow these users to SFTP into these directories and be able to change files?
Answer: You need to make more groups (and set the appropriate group permissions), or use ACLs.
Comment: Ok so if I make a group that both the www-data and SiteA-com user belong to and chgroup on that directory to be that group, then both users will be able to change files?
Comment: As long as the group read/write/execute bits are set correctly, yeah.
Answer: Even though this probably belongs on ServerFault, I'll go ahead and answer it.
I presume the users are in the same group, if not, you will want to do that through your access control system (ACL, LDAP, etc). If they're all in the same group, you can give group permissions by doing
chmod g+rXw /path/to/file
See the appropriate man pages (chmod, usermod, etc) for more options.
Answer: It's mentioned in the first reply, but the answer really is Unix Access Control Lists, it extends the standard unix file permissions into something a lot more flexiable and specific.
These two commands are your friend:

setfacl - set ACLs
getfacl - see current ACLs

They really are awesome, they can be set recursively and also set the default permissions of all new files and folders (so you don't have to keep settings perms over and over again when you make new files).
Example:
Apache & FTP can read/write httpdocs, but the user joebrown can only read httpdocs.
setfacl -R -m u:apache:rwx ./httpdocs
setfacl -R -m u:ftpuser:rwx ./httpdocs
setfacl -R -m u:joebrown:r-x ./httpdocs

You can run the same commands with the -d flag to set it as default policy for new files too.
Answer: What I do is put my users under "dev" and set the public_html directory as follows.
chgrp dev public_html
chmod 775 public_html


Colocar line-height do mesmo tamanho do font-size

Question: Quando aplico o tamanho de uma fonte através do font-size, sempre fico em dúvida se devo utilizar line-height também. Para a fonte e linha ter o mesmo tamanho.
Pois, convenhamos, que fica chato acertar margin e padding se o line-height não está de acordo.
Eu me refiro mais a fontes especificas, aquelas que não são padrões do Windows. Onde utilizo o font-face.
Vocês fazem algo parecido?
Answer: Por mais que os browsers tendam a ter um comportamento considerado padrão, a especificação do line-height diz que o parâmetro normal recomendado é de 1.0 a 1.2, ou seja, pode variar de browser para browser significativamente.
Em outras palavras, é sempre bom determinar o line-height para que a apresentação das suas páginas seja consistente.
O que você pode levar em consideração é que, quando você especifica a fonte usando a sintaxe composta font, pode simplificar utilizando apenas uma barra, não precisando de uma diretiva line-height explícita em separado:
h1 { font: 26px/30px sans-serif }
h2 { font: 20px/20px sans-serif }
p  { font: 12px/1.5 sans-serif }

Neste caso, especificamos que a fonte do
 tem 12px de altura, e o line-height equivale a um espaçamento 1.5, ou seja, vai resultar em 18px. Já o
 tem 26px de altura, e 30px de line-height, assim como o
 terá 20px tanto de altura quanto de distância entre uma linha e outra. Assim, não há necessidade da diretiva line-height separada em nenhum dos casos.
Interessante notar que o line-height é uma das propriedades que aceita um valor numérico sem unidade, assim como fizemos com o font acima. Para especificar um espaçamento 2, basta usar desta forma:
p  { line-height: 2 }

Assim, a linha terá o dobro da altura da fonte.
Leitura complementar:
Por que é recomendado utilizar a unidade "em" ao invés de "px" para fontes?


Why do I keep getting the error s is not defined?

Question: I'm working on a homework problem and it keeps returning the same error. Here is the problem
Write a function named "json_average" that takes a JSON formatted string as a parameter in the format of an array of objects where each object has keys "mass", "density", "temperature", and "velocity" and each key maps to a floating point number. This function should return the average "velocity" of all the objects in the array as a JSON string in the format {"velocity": }
I tried switching things around like using let s = 0 and var = 0, either way it's still not working. Here is the code I tried.
function json_average(x) {
    let data = JSON.parse(x);
    var s = 0 ;
    var n = 0;
    var a;
    for (let i of data) {
        a = i["velocity"];
        s = s + a ;
        n = n + 1 ;
    }
    let d = {"velocity" : (s/n)};
    return(JSON.stringify(d));
}

When I submit the code this is what it returns.
`error on input ['[{"mass": 3.55, "density": 380.72, "velocity": 33.11, "temperature": 46.8}, {"mass": 91.37, "density": 572.04, "velocity": 64.43, "temperature": -0.13}, {"mass": 40.4, "density": 124.52, "velocity": 52.8, "temperature": 38.81}, {"mass": 68.92, "density": 326.77, "velocity": 31.64, "temperature": 43.71}, {"mass": 3.22, "density": 419.85, "velocity": 70.64, "temperature": 23.58}]']:`
ReferenceError: s is not defined

Comment: `let s = s + 1;` introduces a new uninitialized variable `s` which you try to add 1 to. Read up on what `var` and `let` actually do.
Comment: You already declared s and n variable before loop using var. Inside for loop block you declared again using let, which is accessible only within for loop block. You need to understand scopes in Javascript.
Answer: You are trying to use s when it is not initialized. Updated the function, hope this helps.

function json_average(x) {
    let data = JSON.parse(x);
    var s = 0 ;
    var n = 0;
    for (let i of data) {
        let a = i["velocity"];
        s = s + a ;
        n = n + 1 ;
    }
    let d = {"velocity" : (s/n)};
    return(JSON.stringify(d));
}
Answer: You are redefining already declared variable inside a loop to have block scope (in loop). Variables declared with let inside a block is no accessible outside the block scope. Also if you want to give s and these other variables block scope in te loop you will not be able to get its value outside the loop for calculating velocity. See js fiddle.

function json_average(x) {
    let data = JSON.parse(x);
    var s = 0 ;
    var n = 0;
    for (i in data) {
        a = data[i]["velocity"];
        s = s + a ;
        n = n + 1 ;
    }

    let d = {"velocity" : (s/n)};
    return(JSON.stringify(d));
}

var result = json_average('[{"mass": 3.55, "density": 380.72, "velocity": 33.11, "temperature": 46.8}, {"mass": 91.37, "density": 572.04, "velocity": 64.43, "temperature": -0.13}, {"mass": 40.4, "density": 124.52, "velocity": 52.8, "temperature": 38.81}, {"mass": 68.92, "density": 326.77, "velocity": 31.64, "temperature": 43.71}, {"mass": 3.22, "density": 419.85, "velocity": 70.64, "temperature": 23.58}]');

console.log(result);


How do I add attibution links to 'teaser thumbnails' in Drupal blog posts?

Question: I'm running a professional blog on, I believe, Drupal 6.17. It allows me to include a 'teaser thumbnail' of an image with each blog post. However, many times I want to use free stock photos from sites such as flickr and freedigitalphotos.net as that teaser. However, these sites require you to include a link back to the photographer's page along with an attritbution on the same page as the photo. I'm not sure how to do that with Drupal's 'teaser thumbnail' option.
Answer: For D6, use the ImageField Extended module. It provides a very simple admin interface to give you a way of adding textfields, textareas, WYSIWYG textareas and checkboxes.


Filtering a View that is output as a Block

Question: I have a view which has a page display and a block display. The block display has Use Ajax enabled so the filters show up. When I output the block display to my template using:
<code><?php print views_embed_view('list_disciplines', 'block_1'); ?>
</code>
It outputs correctly, and shows the filters. The problem is when I submit the filters form it redirects me to the path I have set up for the page display, and doesn't use ajax at all.
Any ideas?
Answer: Ok finally dug up my answer.
http://drupal.org/node/386388
In summary, the scripts for ajax views cannot be added if you embed your view in a page template. You either have to create the view in preprocess_page or in a node template file.


How to remove main from .htaccess

Question: Currently this code works:
RewriteRule ^main/([[0-9a-zA-Z-]+)(/?) /index.php?menu=$1 [L]

but I don't want example.com/main/start
I want it to be example.com/start
I tried to remove only main but I get error then, can someone help me?
Answer: Try changing your rules to:
RewriteCond %{THE_REQUEST} \ /+main/([0-9a-zA-Z-]+)
RewriteRule ^ /%1 [L,R]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^([0-9a-zA-Z-]+)$ /index.php?menu=$1 [L]

Comment: Thanks. It worked mate but it will collapse since I have a code that transfers people to main page incase they enter wrong url.. is it impossible to do this in the single line?
Comment: I'm sorry I just had to copy the last line and it worked. Thanks alot for the help mate :)

How to remove main from .htaccess


Mount a SSHFS volume into a Docker instance

Question: I use SSHFS to mount a remote filesystem on my host and I want to be able to access it from inside a Docker container.
I mount the remote filesystem
sshfs -o idmap=user,uid=$(id -u),gid=$(id -g) user@remote:directory /path/to/sshfs
And, using Docker, I get the following errors depending on me using --mount:
docker run  -it -v /path/to/sshfs:/target myimage bash
docker: Error response from daemon: error while creating mount source path '/path/to/sshfs': mkdir /path/to/sshfs: file exists.

or -v:
docker run -it  --mount src=/path/to/sshfs,target=/target,type=bind  myimage bash
docker: Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /path/to/sshfs.
See 'docker run --help'

Is it possible to mount a sshfs mountpoint into a container?
Answer: Requires the following steps:

uncomment user_allow_other in /etc/fuse.conf
unmount the FUSE filesystem
remount the FUSE filesystem with sshfs -o allow_other user@.... (making sure to include the -o allow_other option)
try starting the container again
Comment: Thanks! I did as you said and then mounted with `sshfs -o allow_other` and it worked!
Comment: Great, it works fine for me, even without the `-o allow_other` option added
Comment: If you cannot use `-o allow_other` because `user_allow_other` isn't set and you're not allowed to change that, you might find my answer below helpful: Instead of mounting your directory, mount its parent.
Answer: Detailed step by step with external references:

Create and copy an SSH key to the remote server

Only need to do this the first time we make a SSHFS mount. Here's an example on how to do this in ubuntu.

Mount the Remote File System(s) Over SSHFS

Follow the steps from this tutorial.

Mount a SSHFS volume into the Docker instance

docker run  -it -v /path/to/sshfs:/target myimage bash
Answer: I had the same issue but couldn't use "allow_other", as I have no root permission. But what I found out is, that you can mount the mount-point's parent.
So your example would work for me like this:
docker run  -it -v /path/to:/target myimage bash

Of course it's then not avalable as /target but only as /target/sshfs.


add multiple columns with liquibase using yml

Question: I want to add multiple columns to my database using liquibase and a yml-file.
I know that I can do this with one step using xml like this:
<changeSet author="liquibase-docs" id="addColumn-example">
    <addColumn catalogName="cat"
            schemaName="public"
            tableName="person">
        <column name="job" type="varchar(255)"/>
       <column name="designation" type="varchar(255)"/>
    </addColumn>
</changeSet>

Can I do the same in a yml-file?
- changeSet:

    id: myId
    author: malt

    changes:

      - addColumn:

        tableName: myTable

        columns:

          - column:

            name: name01

            type: decimal(21,2)

          - column:

            name: name02

            type: decimal(21,2)

This does not seem to work...
I just don't want to repeat the -addColumn-Tag again and again.
Thanks for your help!
Matthias
Answer: According to addColumn liquibase documentation you can do this in yaml as such (just their example):
databaseChangeLog:
   - changeSet:
       id: addColumn-example
       author: liquibase-docs
     changes:
         - addColumn:
              tableName: person
              columns:
                 name: middlename
                 type: varchar(50)

Answer: You almost got it right, just need an extra indent after column:
- changeSet:

    id: myId
    author: malt

    changes:

      - addColumn:

        tableName: myTable

        columns:

          - column:
 # note the indent below here!
              name: name01

              type: decimal(21,2)

          - column:

              name: name02

              type: decimal(21,2)


Prove $(X \backslash Y) \cap Z = (X \cap Z) \backslash (Y \cap Z)$

Question: I know I'll have to prove that the LHS is a subset of RHS and vice versa. But I'm just not too sure where to start.
Comment: Just write it out: what does it mean for an element $a$ to be in $(X \backslash Y) \cap Z$?
Comment: Given two sets $A,B$, the standard way of proving that $A = B$ is to prove that $A\subseteq B$ and $B\subseteq A.$  To prove that $A\subseteq B$, you assume that element $e$ is in $A$, and then prove that element $e$ must therefore also be in $B$.
Answer: $(X \cap Z) \backslash (Y \cap Z) = \{x: x\in X \text{ and } x \in Z \text{ and } x\notin Y \cap Z\}$
$= \{x: x\in X \text{ and } x \in Z \text{ and } (x\notin Y \text{ or }  x\notin Z)\}$
$= \{x: (x\in X \text{ and } x \in Z \text{ and }  x\notin Y) \text{ or }  (x\in X \text{ and } x \in Z \text{ and } x\notin Z)\}$
$= \{x: (x\in X \text{ and } x \in Z \text{ and }  x\notin Y\} = (X \backslash Y) \cap Z$
I used these facts in this order:
(1) DeMorgans Laws: not (A and B) = not A or not B
(2) (A and B) or C = (A and C) or (B and C),
(3) (A) or (a false statement) = A
Comment: Oh!! That makes sense now thank you.
Comment: @zinvolle your welcome :3 could u accept the answer btw haha


SSMS 18.4 installation will not run

Question: I am having trouble installing SSMS on a fresh installation of Windows 10 - on a new Dell XPS 15.
I am installing the latest version (18.4) from here: https://learn.microsoft.com/en-gb/sql/ssms/download-sql-server-management-studio-ssms
When I run SSMS-Setup-ENU.exe I get the User Access Control dialogue, I say Yes and nothing happens.

I cannot see any processes starting in Task Manager.
Logs indicate that the installationg cannot start. From googling this error suggests that I'm missing a version of dotnet framework.

I have installed Visual Studio 2019 + all the dotnet frameworks (everything from 4.6 to 4.8) + Dotnet Core.
Does anybody have any ideas - what am I missing or doing wrong? :/
Thanks for your help!
Error (ommitting all of the noise)
[2E90:33E8][2019-12-09T10:55:43]i000: Loading managed bootstrapper application.
[2E90:33E8][2019-12-09T10:55:43]e000: Error 0x80131902: Failed to create the managed bootstrapper application.
[2E90:33E8][2019-12-09T10:55:43]e000: Error 0x80131902: Failed to create UX.
[2E90:33E8][2019-12-09T10:55:43]e000: Error 0x80131902: Failed to load UX.
[2E90:33E8][2019-12-09T10:55:43]e000: Error 0x80131902: Failed while running
[2E90:33E8][2019-12-09T10:55:43]e000: Error 0x80131902: Failed to run per-user mode.
[2E90:33E8][2019-12-09T10:55:43]i007: Exit code: 0x80131902, restarting: No

Installed dotnet framework
PS C:\> wmic /namespace:\\root\cimv2 path win32_product where "name like '%%.NET%%'" get version
Version
4.5.50932
4.8.03928
4.6.01055
4.7.02558
4.5.50710
24.64.28315
22.214.171.124
4.8.03761
3.0.1.014347
13.0.19213.0
4.7.03081
4.7.02053
12.16.14727
24.0.28113
4.7.02558
184.108.40.206
4.7.03062
12.2.18292.0
3.1.0.014727
12.0.30731
24.64.28315
2.1.14.014727
1.0.20910.0
4.6.01590
4.7.02053
24.4.28216
4.8.03928
220.127.116.11
24.64.28315
4.8.03761
4.8.03928
3.0.0127.060
126.96.36.199
16.120.28209
4.6.00081
24.64.28315
3.1.100.014727
220.127.116.11
24.64.28315
24.4.28216
4.0.30319
18.104.22.168
4.5.51651
4.7.02558
4.6.01055
4.7.02558
4.6.01590
24.64.28315
24.64.28315
24.64.28315
22.214.171.124
24.0.28113
24.4.28216
16.4.457.38025
4.6.01590
4.7.03062
24.64.28315
3.0.1.014727
24.4.28216
4.7.02053
24.4.28216
4.6.01055
24.64.28315
4.8.03761

Answer: Argh I have worked out my problem - it's pretty dumb.
I forgot that I updated the machine.config with some settings but there was a mistake and missing a closing ". This meant that the configuration was invalid and therefore the installer wouldn't run.
Answer: WE had similar problem:
Issue 1:
The management studio update checker package did not load correctly.
The problem may have been caused by a configuration change or by the installation of another extension.
Restarting Visual Studio could help resolve this issue.
Issue 2:
The configuration section 'system.web' cannot be read because it is a missing a section declaration.
\?D:\wwwroot\testsite\web.config
While troubleshooting, we found that .NET Framework directory for 32 bit 4.7.1 was missing required config directory which contains necessary default machine.config file. These files are necessary for all applications to run. We copied the files from a working server on to the server which had the problem. We were then able to use current SSMS (v18.4) as well as upgrade it to v18.6. Also, IIS level config errors were gone.
Directory path: C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config

SSMS 18.4 installation will not run


How to find the closed form of $\int_{0}^{\frac{\pi}{2}} \frac{d x}{\left(p \cos ^{2} x+q \sin ^{2} x+r\right)^{n}}$, where $n\in N$?

Question: In my post, I found the integral
$$\int_{0}^{\frac{\pi}{2}} \frac{d x}{\left(a^{2} \cos ^{2} x+b^{2} \sin ^{2} x\right)^{2}}=\frac{\pi\left(a^{2}+b^{2}\right)}{4 a^{3} b^{3}}$$
Then  I want to find the generalized integral using similar technique by using the integral
$$
I(p, q, r)=\int_{0}^{\frac{\pi}{2}} \frac{d x}{p \cos ^{2} x+q \sin ^{2} x+r} \quad \text{ where }p+r> 0 \textrm{ and }  q+r>0.
$$
Letting $t=\tan x$ yields
$$
\begin{aligned}
I(p, q, r)&=\int_{0}^{\infty} \frac{d t}{p+qt^{2}+r\left(1+t^{2}\right)}\\
&=\int_{0}^{\infty} \frac{d t}{(q+r) t^{2}+p+r}\\&=\frac{1}{\sqrt{(p+r)(q+r)}} \left.\tan ^{-1}\left(\frac{t\sqrt{q+r} }{\sqrt{p+r}}\right)\right]_{0}^{\infty}\\
&=\frac{\pi}{2 \sqrt{(p+r)(q+r)}}
\end{aligned}
$$
Differentiating it w.r.t. $r$ by $n$ times yields
$$
\begin{aligned}\int_{0}^{\frac{\pi}{2}} \frac{(-1)^{n}n! d x}{\left(p \cos ^{2} x+q \sin ^{2} x+r\right)^{n+1}}=\frac{\pi}{2 }\cdot \frac{\partial^{n}}{\partial r^{n}}\left(\frac{1}{\sqrt{(p+r)(q+r)}}\right) \\
\boxed{\int_{0}^{\frac{\pi}{2}} \frac{d x}{\left(p \cos ^{2} x+q\sin ^{2} x+r\right)^{n+1}}=\frac{(-1)^{n} \pi}{2n !} \frac{\partial^{n}}{\partial r^{n}}\left(\frac{1}{\sqrt{(p+r)(q+r)}}\right)}
\end{aligned}
$$
Using the formula, we can evaluate
$$
\int_{0}^{\frac{\pi}{2}} \frac{d x}{\left(p \cos ^{2} x+q \sin ^{2} x+r\right)^{2}}=\frac{-\pi}{2} \frac{\partial}{\partial r}\left(\frac{1}{\sqrt{(p+r)(q+r)}}\right)= \frac{\pi(p+q+2r)}{4\left[(p+r)( q+r)\right]^{\frac{3}{2}}}
$$
For higher derivative, applying Leibniz's Rule gives
$$
\begin{aligned}\frac{d^{n}}{d r^{n}}\left[(p+r)^{-\frac{1}{2}}(q+r)^{-\frac{1}{2}}\right]&= (-1)^n \sum_{k=0}^{n}\left(\begin{array}{l}
n \\
k
\end{array}\right)\left(\frac{1}{2} \right)_k(p+r)^{-\frac{1}{2}-k}\left(\frac{1}{2} \right)_{n-k}(q+r)^{-\frac{1}{2}-n+k} \\&=\frac{(-1)^n}{(q+r)^{n} \sqrt{(p+r)(q+r)}} \sum_{k=0}^{n} \binom{n}{k} \left(\frac{1}{2}\right)_{k}\left(\frac{1}{2}\right)_{n-k}\left(\frac{q+r}{p+r}\right)^{k}\end{aligned}
$$
where $\displaystyle (\alpha)_n=\frac{\Gamma(\alpha+n)}{\Gamma(n)} $.
Now we can conclude that for any natural number $n\geq 2$,
$$
\int_{0}^{\frac{\pi}{2}} \frac{d x}{\left(p\cos ^{2} x+q\sin ^{2} x+r\right)^{n+1}}\\=\boxed{\frac{\pi}{2n!(q+r)^{n} \sqrt{(p+r)(q+r)}} \sum_{k=0}^{n} \binom{n}{k}\left(\frac{1}{2}\right)_{k}\left(\frac{1}{2}\right)_{n-k}\left(\frac{q+r}{p+r}\right)^{k}}
$$
Question:
Is there any other solution?
Comment: Your questions, solutions and answers are impressive.
Answer: With $a=p+r$ and $b=q+r$
\begin{aligned}I_n=\int_{0}^{\frac{\pi}{2}} \frac{1}{\left(p \cos ^{2} x+q \sin ^{2} x+r\right)^{n}}dx
=\int_{0}^{\frac{\pi}{2}} \frac{1}{\left(a \cos ^{2} x+b \sin ^{2} x\right)^{n}}dx
\end{aligned}
Integrate, instead
\begin{align}
 \sum_{n=1}^\infty I_n t^n=&
\int_0^{\frac{\pi}{2}} \sum_{n=1}^\infty \left(\frac{t}{a \cos^2x+ b \sin^2x}\right)^n dx\\
=&\int_0^{\frac{\pi}{2}}\frac{t}{a\cos^2x+ b\sin^2x-t}dx
=\frac{\pi}{2}\frac{t}{\sqrt{(a-t )(b-t )}}\\
\end{align}
and apply the Taylor series $ \frac{1}{\sqrt{1-x}}=\sum_{i=0}^\infty\frac1{2^{2i}}{2i\choose i}x^i$ to express the result in power of $t^n$
\begin{align}
 \sum_{n=1}^\infty I_n t^n
=\>
 \sum_{i=0}^\infty\sum_{j=0}^\infty
\frac{\frac{\pi}{2}{2i\choose i} {2j\choose j} t^{i+j+1}}{2^{2i+2j}a^{i+1/2} b^{j+1/2}}
=
\sum_{n=1}^{\infty} \frac{\pi}{2^{2n-1}}\sum_{i+j+1=n} \frac{{2i\choose i}{2j\choose j}}{a^{i+1/2} b^{j+1/2}} t^n
\end{align}
Thus
$$I_n= \frac{\pi}{2^{2n-1}}\sum_{i+j+1=n} \frac{{2i\choose i}{2j\choose j}}{(p+r)^{i+1/2} (q+r)^{j+1/2}}
$$
Comment: I had just learn your magical use of of the comparison of two series.  Thank you very much.
Answer: Let $a=p+r$, $b=q+r$ and, for sure, as you did $x=\tan^{-1}(t)$
$$I_n=\int_{0}^{\frac{\pi}{2}} \frac{d x}{\left(p \cos ^{2} (x)+q \sin ^{2} (x)+r\right)^{n}}=\int_0^\infty \frac{(1+ t^2)^{n-1} }{(a+b\,t^2)^n }\,dt$$There is an antiderivative
$$\int\frac{(1+ t^2)^{n-1} }{(a+b\,t^2)^n }\,dt=\frac t {a^{n}}\, F_1\left(\frac{1}{2};1-n,n;\frac{3}{2};-t^2,-\frac{b }{a}t^2\right)$$ where appears  the Appell hypergeometric function of two variables.
Using the bounds and simplifying, this gives
$$I_n=(-1)^n  \frac{\pi ^{3/2}}{2 a^n}\Bigg[\frac{\, _2F_1\left(\frac{1}{2},n;n+\frac{1}{2};\frac{b}{a}\right)}{\Gamma (1-n)
   \Gamma \left(n+\frac{1}{2}\right)}-\left(\frac{a}{b}\right)^{n-\frac{1}{2}}\frac{\, _2F_1\left(\frac{1}{2},1-n;\frac{3}{2}-n;\frac{b}{a}\right)}{\Gamma
   \left(\frac{3}{2}-n\right) \Gamma (n)} \Bigg]$$ where appear two Gaussian    hypergeometric functions.
Edit
Let $b=k\,a$ to get
$$I_n=\frac{\pi}{a^n \,k^{n-\frac{1}{2}} }\,\frac {P_n(k)}{c_k}$$  The $c_k$'s form sequence $A101926$ in $OEIS$ and the first polynomials are
$$\left(
\begin{array}{cc}
n & P_n(k) \\
 1 & 1 \\
 2 & k+1 \\
 3 & 3 k^2+2 k+3 \\
 4 & 5 k^3+3 k^2+3 k+5 \\
 5 & 35 k^4+20 k^3+18 k^2+20 k+35 \\
 6 & 63 k^5+35 k^4+30 k^3+30 k^2+35 k+63 \\
 7 & 231 k^6+126 k^5+105 k^4+100 k^3+105 k^2+126 k+231 \\
 8 & 429 k^7+231 k^6+189 k^5+175 k^4+175 k^3+189 k^2+231 k+429
\end{array}
\right)$$ which are interesting (it would be worth to analyse the patterns using $OEIS$).
Comment: Mr Claude Leibovici, your solution is Interesting though I don't understand.  Thank you very much again.


polar transformation

Question: i need some help.
i have given
$F(t)=\int\limits_{0}^\infty e^{\,-tx^2}cos(x^2)\,dx$ and $G(t)=\int\limits_{0}^\infty e^{\,-tx^2}sin(x^2)\,dx$ with t>0
I have to show with polar transformation that
$F^2(t)-G^2(t) = \frac{\pi \cdot t}{4(1+t^2)}$ and  $2F(t)G(t) = \frac{\pi}{4(1+t^2)}$
the hint was to start with $(F(t)+i\,G(t))^2$
$(F(t)+i\,G(t))^2 = \int\limits_{0}^\infty e^{-x^2(t-i)\
}\,dx= \int\limits_{0}^{2\pi} \int\limits_{0}^\infty r e^{-r^2(t-i)\
}\,dr\,d\varphi$
so i did so far but i don't get the right solution.
can someone give me the right solution please ?
Comment: See [Fresnel Integral](https://en.wikipedia.org/wiki/Fresnel_integral).
Comment: @FelixMarin See how I've implemented your suggestion below.
Comment: @CyeWaldman I just read it. Quite nice.
Answer: Picking up on the comment of @FelixMarin above, I demonstrate how to evaluate the integral directly in terms of the error function. Given
$$F(t)=\int\limits_{0}^\infty e^{\,-tx^2}cos(x^2)\,dx \quad\text{ and }\quad G(t)=\int\limits_{0}^\infty e^{\,-tx^2}sin(x^2)\,dx$$
we can write
$$F(t)+iG(t)=\int_0^{\infty}e^{x^2(i-t)}dx$$
Now, let $i\frac{\pi}{2}u^2=(i-t)x^2$, and $dx=\sqrt{\frac{i\pi}{2(i-t)}}du$, then
$$
\begin{align}
\int_0^{\infty}e^{x^2(i-t)}dx
&=\sqrt{\frac{i\pi}{2(i-t)}}\int_0^{\infty}e^{i\frac{\pi}{2}u^2}du\\
&=\sqrt{\frac{i\pi}{2(i-t)}}\left(\frac{1+i}{2}\right)\text{erf}\left( \frac{1-i}{2}\sqrt{\pi}u\right)\big|_{u=\infty}\\
&=\sqrt{\frac{i\pi}{2(i-t)}}\left(\frac{1+i}{2}\right),\quad [\text{erf}(\infty)=1]\\
&=\sqrt{\frac{\pi}{4(t-i)}}
\end{align}
$$
The polar integral is quite straightforward, so we can readily show that
$$[F(t)+i\,G(t)]^2 = \left[\int\limits_{0}^\infty e^{-x^2(t-i)\
}\,dx\right]^2=\frac{1}{4} \int\limits_{0}^{2\pi} \int\limits_{0}^\infty r e^{-r^2(t-i)\
}\,dr\,d\theta=\frac{\pi}{4(t-i)}$$
in agreement with the solution of Jack D'Aurizio.
Comment: +1. Fresnel, in this case, is a straightforward approach.
Answer: $F^2-G^2$ and $2FG$ are respectively the real and imaginary part of $(F+iG)^2$. Since
$$ F+iG = \int_{0}^{+\infty}\exp\left((i-t) x^2\right)\,dx = \frac{1}{2}\int_{\mathbb{R}}\exp\left((i-t)x^2\right)\,dx\tag{1}$$
we have:
$$ (F+iG)^2 = \frac{1}{4}\iint_{\mathbb{R}^2}\exp\left((i-t)(x^2+y^2)\right)\,dx\,dy =\frac{1}{4}\int_{0}^{2\pi}\int_{0}^{+\infty}\rho e^{(i-t)\rho^2}\,d\rho\tag{2}$$
and:
$$ (F+iG)^2 = \frac{\pi}{4(t-i)} = \frac{\pi(t+i)}{4(t^2+1)}\tag{3}$$
The claim easily follows.
Answer: note if you are not familiar with solving complex integrals: to solve the integral in $(2)$ just compute it like you would solve an integral in $\mathbb{R}$. You can apply the fundamental theorem of analysis for the inner integral. the resulting does not depend on $φ$, so you can simply multiply the result by $2\pi$ to solve the outer integral.
Comment: thank you for the hint but im familiar with complex integrals :)
Answer: I'm posting this in the role of a devil's advocate. My problem is that it is obvious that
$$z=e^{-x^2(t-i)}=re^{i\theta}$$
is akin to a logarithmic spiral. Just plot it for $t<<1$ (e.g., $t=.05$) to see it clearly.
So my point is, how do you get to $r\to\infty$ if $\theta\to 2\pi$?
Comment: you have this one $\frac{1}{4}\int\limits_{\mathbb{R^2}}e^{-(x^2+y^2)(t-i)}dx\,dy $
and you want to do the polar transformation and you want to integrate over $\mathbb{R^2}$ so you have to use $\varphi \in [0,2\pi]$ to integrate over a circle and to reach all of $\mathbb{R^2}$ you must take $r\in[0,\infty]$.
Comment: @Otto.t In your original post, did you mean that $$(F(t)+i\,G(t))^2 = \int\limits_{0}^\infty e^{-x^2(t-i)\
}\,dx=\frac{1}{4} \int\limits_{0}^{2\pi} \int\limits_{0}^\infty r e^{-r^2(t-i)\
}\,dr\,d\varphi?$$ Because that is what JackD'Aurizio has shown. Notice the factor of 1/4 in the polar integral.

polar transformation


Modify code to use batch api?

Question: I have a custom module written to import data from a Salesforce API and update organic group info by adding/updating/removing members from groups. I originally wrote it as a drush command for ease of development, and now I've moved it to a set of functions initiated from hook_cron() using a cron queue. An additional requirement is that the import process have the ability to be run from a form submission. The process is long running enough that it times out when running from a simple form submission, so I need to use the batch api. Looking at all the examples, it looks like I need to have one function handle each item so that the batch process can call that function for each individual item. Here's what's working so far:
<code>/**
 * Main function to get data from Salesforce and put it into an array.
 * Called both from a form and drush command.
 */
function naa_committee_get_salesforce_data() {
  $base_url = 'https://api/url';
  // The web service expects the body to be in JSON format, so we create it as an
  // array and convert it to JSON.
  $url_data = array(
    "Name" => "GetCommittees",
    "AuthenticationKey" => "my_secret_key"
  );
  $url_data = json_encode($url_data);
  // Set the Content-Type header.
  $headers = array('Content-Type' => 'application/json');
  // Query the URL and get the returned data.
  $data = drupal_http_request($base_url, array('headers' => $headers, 'method' => "POST", 'data' => $url_data));

  $records = drupal_json_decode($data->data);

  $sf_members = array();
  foreach ($records['Records'] as $record) {
    $user_id = $record['NU__Account__r']['Id'];
    $user_name = $record['NU__Account__r']['Name'];
    $email = $record['NU__Account__r']['PersonEmail'];

    // If array doesn't yet exist for this user, create it.
    if (!$sf_members[$user_id]) {
      $sf_members[$user_id]['user_name'] = $user_name;
      $sf_members[$user_id]['email'] = $email;
    }
    // Add committee to committees element.
    $sf_members[$user_id]['committees'][] = array(
      'committee_name' => $record['NU__Committee__r']['Name'],
      'committee_id' => $record['NU__Committee__r']['Id'],
      'committee_code' =>  $record['NU__Committee__r']['NU__CommitteeShortName__c']
    );
  }

  // Pass data to parsing function.
  if (naa_committee_parse_records($sf_members)) {
    return TRUE;
  }
  else {
    return FALSE;
  }
}

/**
 * Parse returned records
 *
 * @param records
 *   An array of committee member records returned from Salesforce.
 *
 */
function naa_committee_parse_records($sf_members) {
  // Loop through returned records.
  foreach ($sf_members as $user_id => $member) {
    // Do an EntityFieldQuery to see if user already exists.
    $query = new EntityFieldQuery();
    $query->entityCondition('entity_type', 'user')
          ->propertyCondition('name', $user_id);
    $result = $query->execute();

    if ($result) {
      // Since user_id is distinct, we're just taking the first
      // one from $result['node'];
      $uid = current(array_keys($result['user']));
      $account = entity_load('user', array($uid));
      $user = $account[$uid];
      $groups = og_get_entity_groups('user', $user);

      // Need to filter groups to make sure they are committees.
      if ($groups) {
        foreach ($groups['node'] as $key => $group_nid) {
          $group_node = entity_load('node', array($group_nid));
          if ($group_node[$group_nid]->type != 'committee') {
            unset($groups['node'][$key]);
          }
        }

        // Check to see if user should be removed from any memberships.
        naa_committee_check_removed($user, $groups['node'], $member['committees']);
      }

      foreach ($member['committees'] as $committee) {
        if (!$member['email']) {
          $name_array = explode(' ', $member['user_name']);
          $member['email'] = strtolower($name_array[0] . '.' . $name_array[1] . '@naahq.org');
        }
        naa_committee_update_membership($user, $committee, $groups);
      }
    }
    else {
      // Doesn't exist, so create user.
      naa_committee_create_user($user_id, $member);
    }
  }

  return TRUE;
}
</code>
So what I have now is one function that loops through all of the records, and for each one determines if it exists, and then hands the record off to the appropriate function. In order to do the batch api, do I need to do something like move the code that loops through the results array to naa_committee_get_salesforce_data() and then pass each individual record to naa_committee_parse_records()? Or can I use it with what I have now?
Thanks.
Answer: I assume that you have solved your issue by now but I was just doing some batch API work and found your question.  Hopefully, this answer helps.
I added code to create the batch in the first function.  The second function will be called by the batch API while processing the batch.
<code>/**
 * Main function to get data from Salesforce and put it into an array.
 * Called both from a form and drush command.
 */
function naa_committee_get_salesforce_data() {
  $base_url = 'https://api/url';
  // The web service expects the body to be in JSON format, so we create it as an
  // array and convert it to JSON.
  $url_data = array(
    "Name" => "GetCommittees",
    "AuthenticationKey" => "my_secret_key"
  );
  $url_data = json_encode($url_data);
  // Set the Content-Type header.
  $headers = array('Content-Type' => 'application/json');
  // Query the URL and get the returned data.
  $data = drupal_http_request($base_url, array('headers' => $headers, 'method' => "POST", 'data' => $url_data));

  $records = drupal_json_decode($data->data);

  $sf_members = array();
  foreach ($records['Records'] as $record) {
    $user_id = $record['NU__Account__r']['Id'];
    $user_name = $record['NU__Account__r']['Name'];
    $email = $record['NU__Account__r']['PersonEmail'];

    // If array doesn't yet exist for this user, create it.
    if (!$sf_members[$user_id]) {
      $sf_members[$user_id]['user_name'] = $user_name;
      $sf_members[$user_id]['email'] = $email;
    }
    // Add committee to committees element.
    $sf_members[$user_id]['committees'][] = array(
      'committee_name' => $record['NU__Committee__r']['Name'],
      'committee_id' => $record['NU__Committee__r']['Id'],
      'committee_code' =>  $record['NU__Committee__r']['NU__CommitteeShortName__c']
    );
  }

  // If there is data to process then setup a batch.
  if (count($sf_members)) {
    $operations = array();
    // Add an operation for each user to be processed.
    foreach ($sf_members as $user_id => $member) {
      $operations[] = array(
        'naa_committee_process_record',
        array(
          $user_id,
          $member,
        )
      );
    }

    $batch = array(
      'operations' => $operations,
      // If desired set the function to be executed when the batch finishes processing.
      'finished' => 'naa_committee_batch_finished',  // remove this line if not desired.
    );

    batch_set($batch);
    return TRUE;
  }
  else {
    return FALSE;
  }
}

/**
 * This function is called by the batch API
 */
function naa_committee_process_record($user_id, $member, &$context) {

  // Use native drupal function user_load_by_name rather than EntityFieldQuery
  // to load user or return false.
  $user = user_load_by_name($user_id);
  if ($user) {
    $groups = og_get_entity_groups('user', $user);

    // Need to filter groups to make sure they are committees.
    if ($groups) {
      foreach ($groups['node'] as $key => $group_nid) {
        $group_node = entity_load('node', array($group_nid));
        if ($group_node[$group_nid]->type != 'committee') {
          unset($groups['node'][$key]);
        }
      }
      // Check to see if user should be removed from any memberships.
      naa_committee_check_removed($user, $groups['node'], $member['committees']);
    }

    foreach ($member['committees'] as $committee) {
      if (!$member['email']) {
        $name_array = explode(' ', $member['user_name']);
        $member['email'] = strtolower($name_array[0] . '.' . $name_array[1] . '@naahq.org');
      }
      naa_committee_update_membership($user, $committee, $groups);
    }

    // Add information to the $context for use by the naa_committee_batch_finished function
    // if desired
    $context['results'][$user->uid] = 'User record updated.';
  }
  else {
    // Doesn't exist, so create user.
    naa_committee_create_user($user_id, $member);

    // Add information to the $context for use by the naa_committee_batch_finished function
    // if desired
    $context['results'][$user->uid] = 'User record created.';
  }
}
</code>
Hopefully this helps.
Additional information about passing data to the finished function.

Modify code to use batch api?


R: loop and operation on file name in a directory

Question: I have a directory in which I have several files for several sites and years, with exactly the same internal structure. I want to merge (rbind) each files that are from the same sites.
Here is the code that is working for one site (define in "pattern"), but instead of doing this manualy for each site name, I want to create a loop that is doing this for all the site.
ls1 <- list.files(path = dossier, pattern = "CH", all.files = FALSE,
              full.names = FALSE, recursive = FALSE,
              ignore.case = FALSE, include.dirs = FALSE, no.. = FALSE)
ld1 <- lapply(ls1, function(x) read.csv(file=file.path(dossier,x), sep=",", header=T, stringsAsFactors=FALSE))
tot1 <- do.call("rbind", ld1)
write.table(tot1, file=file.path(dossier,"CH-Oe1_2002-2006.csv"),col.names=TRUE, row.names=FALSE, sep=";")

ls2 <- list.files(path = dossier, pattern = "FR-Fon", all.files = FALSE,
              full.names = FALSE, recursive = FALSE,
              ignore.case = FALSE, include.dirs = FALSE, no.. = FALSE)
ld2 <- lapply(ls2, function(x) read.csv(file=file.path(dossier,x), sep=",", header=T, stringsAsFactors=FALSE))
tot2 <- do.call("rbind", ld2)
write.table(tot2, file=file.path(dossier,"FR-Fon_2002-2006.csv"),col.names=TRUE, row.names=FALSE, sep=";")

ls3 <- list.files(path = dossier, pattern = "FR-Gri", all.files = FALSE,
              full.names = FALSE, recursive = FALSE,
              ignore.case = FALSE, include.dirs = FALSE, no.. = FALSE)
ld3 <- lapply(ls3, function(x) read.csv(file=file.path(dossier,x), sep=",", header=T, stringsAsFactors=FALSE))
tot3 <- do.call("rbind", ld3)
write.table(tot3, file=file.path(dossier,"FR-Gri_2002-2006.csv"),col.names=TRUE, row.names=FALSE, sep=";")

ls4 <- list.files(path = dossier, pattern = "FR-Lq2", all.files = FALSE,
              full.names = FALSE, recursive = FALSE,
              ignore.case = FALSE, include.dirs = FALSE, no.. = FALSE)
ld4 <- lapply(ls4, function(x) read.csv(file=file.path(dossier,x), sep=",", header=T, stringsAsFactors=FALSE))
tot4 <- do.call("rbind", ld4)
write.table(tot4, file=file.path(dossier,"FR-Lq2_2002-2006.csv"),col.names=TRUE, row.names=FALSE, sep=";")

So, what is changing here is : the name in "pattern" and the name to save in write.table.
Any idea to simplify this?
Many thanks in advance
Comment: Just put your pattern names in a vector and loop over them using `lapply`, for example.
Answer: Something like this:
for(runPat in c("CH","FR-Fon","FR-Gri","FR-Lq2"))
  ls <- list.files(path = dossier, pattern = runPat, all.files = FALSE, full.names = FALSE, recursive = FALSE, ignore.case = FALSE, include.dirs = FALSE, no.. = FALSE)
  ld <- lapply(ls, function(x) read.csv(file=file.path(dossier,x), sep=",", header=T, stringsAsFactors=FALSE))
  tot <- do.call("rbind", ld)
  write.table(tot2, file=file.path(dossier,sprintf("%s_2002-2006.csv",runPat)),col.names=TRUE, row.names=FALSE, sep=";")
}

Update: (to 2end comment) you mean something like this?
allCont  <- c("CH", "FR")
allSites <- c("", "Fon", "Gri", "Lq2")

for(runCont in allCont){
  for(runSize in allSites){
    if(length(runSite)>0){
      runPat <- sprintf("%s-%s",runCont,runSite)
    }else{
      runPat <- runCont
    }
    ls <- list.files(path = dossier, pattern = runPat, all.files = FALSE, full.names = FALSE, recursive = FALSE, ignore.case = FALSE, include.dirs = FALSE, no.. = FALSE)
    ld <- lapply(ls, function(x) read.csv(file=file.path(dossier,x), sep=",", header=T, stringsAsFactors=FALSE))
    tot <- do.call("rbind", ld)
    write.table(tot2, file=file.path(dossier,sprintf("%s_2002-2006.csv",runPat)),col.names=TRUE, row.names=FALSE, sep=";")
  }

}

how about replacing the two Loops above by
allCNT <- dir()
for (runCnt in allCNT){
  allSites <- dir(runCnt)
  for (runSite in allSites){

  }
}

?
Comment: I removed my answer as it was nearly identical. One note, doing this in a for loop is perfectly fine in this case as no objects are built up with each iteration.
Comment: This is really nice answer in the case I have like this 4 sites. It works, but I want to do this for hundred of site names in the same directory, so it would take time to put the name into runPat iteration. The standard name of one file is : CC-Sss.yyyy.synth.hourly.allvars.csv. CC stand for country (as in the example CH, FR...in capital letters) and sss stand for site (as in the example: Fon, Lq2), sss can be capital or not and numbers. yyyy stands for years. Can we improve this based on the previous answer?
Comment: you can use the `dir` function to get all the files and subfolders in a directory and afterwards use e.g. `regexp` function to get the patterns you are looking for
Comment: @ HolgerBarlt: In answer to your update, this looks good too, but I want to avoid creating a vector with all the names c("CH", "FR"...) because actually, I have hundred of them..so it would be too long. You are right, I should use something with dir and regexp...I'm trying, but not very familiar !
Comment: See latest answer update? If you do not provide more Details on the structure of your filesystem a more detailed answer is difficult...
Comment: Sorry. To summarize: I have a folder with files that have exactly the same structure. Each file have the name : CC-Sss.yyyy.synth.hourly.allvars.csv , for example: FR-Lq2.2002.synth.hourly.allvars.csv FR-Lq2.2003.synth.hourly.allvars.csv CH-Oe1.2005.synth.hourly.allvars.csv CH-Oe1.2006.synth.hourly.allvars.csv etc....and I want to merge file of the same site (all CH-Oe1 together for exemple). But I have hundred of CC-Sss.yyyy.synth.hourly.allvars.csv in the folder. So it would be too long to manually write "CH...FR....". I should use a general way to read CC-Sss and merge them.
Comment: Finally the simplest way to explain is that I want to merge (rbind) the files that have the same starting name "CC-Sss" in the folder...


Convolutional neural network with non-image input data

Question: Can CNNs be used with input data which is not an image? The reason I'm asking is because the original image is often clipped in size because of border effects when doing the convolution.
But if the input is not an image, and I really need to use all the input data, is it possible to overcome the problem?
Comment: To what "problem" are you referring?
Comment: typical problem of classification in ML. not necessarily applied to images or video.
Comment: CNNs exploit topological structure in the input variables.  Do your input variables have topological structure?
Answer: Convolutional networks work so well because they exploit an assumption about with weight sharing. This is why they only work with data where that assumption hold.
The assumption is a spatial one. It is best explained with a picture, where you do not care where exactly something is, which is sometimes called translational invariance.
As long as that assumption holds on your data, you can apply it. Other modalities are e.g. audio or (to some extent) text.
Comment: I don't say that. For convnets, you need to bring all inputs to the same spatial dimension, though.
Comment: I'm not sure if I understand. In all tutorials I've seen, the input is always cropped at the boundaries. Are you saying that is not the case?
Answer: Nothing in the CNN method requires clipping of the input data - that is simply a design choice by the modeller.
I am not completely sure what you are asking for, but it seems that you are interested in the problem associated with the edges of the input matrix. Each pixel at the edge of the image is only captured in a single neuron in the first convolutional layer, making the information less used for classification (which reduces accuracy). Zero-padding is often used to overcome this problem, and could easily be used for other data types as well.


Calculate Overtime based on a sum of columns

Question: I would like to calculate hours worked that are over 40 per week.
table = shift_log_data
database = shift_logs

Field           Type            Null    Key     Default     Extra
id              int(100)        NO      PRI     NULL    auto_increment
admin_hrs       decimal(20,2)   NO              0.00
shop_hrs        decimal(20,2)   NO              0.00
fab_hrs         decimal(20,2)   NO              0.00
paint_hrs       decimal(20,2)   NO              0.00
vacation_hrs    decimal(50,2)   NO              0.00
control_hrs     decimal(20,2)   NO              0.00
test_hrs        decimal(20,2)   NO              0.00
engine_hrs      decimal(20,2)   NO              0.00
hydraulic_hrs   decimal(20,2)   NO              0.00
699_paint       varchar(100)    NO              0.00
699_shop        varchar(100)    NO              0.00
maint_hrs       varchar(100)    NO              0.00
date            date            NO              NULL
name            varchar(50)     NO              NULL
job             varchar(50)     NO              NULL

Need the columns admin_hrs through maint_hrs total per week using the date field.
If hours are greater than 40 hours per week, make an alias called overtime showing the rest of the hours greater than forty.
I have tried.
SELECT SUM(overtime)
  FROM
  (
    SELECT name,
           IF(SUM(test_hrs+fab_hrs+control_hrs+shop_hrs+paint_hrs+engine_hrs+hydraulic_hrs)>40,
             SUM(test_hrs+fab_hrs+control_hrs+shop_hrs+paint_hrs+engine_hrs+hydraulic_hrs))-40,
              0) AS overtime
      FROM shift_log_data
     GROUP BY name
  ) TOTAL_OVERTIME

Comment: Some critiques of table design - `name` is not a unique identifier for a person, you'll want some sort of unique person id fk to a `person` table.  You probably also want a `job` table for roughly similar reasons.  This table should probably be normalized so there's a single `hours_worked` field (or maybe `minutes_worked`, as an hour doesn't divide cleanly into base 10 fractions), with an `hour_type` field.  Why are **vacation** hours being counted in the total?  That sounds exploitable.  Alternatively, have a table with a clockin/out time (and type).  And which is it, MySQL or SQL Server?
Comment: _"...`admin_hrs` through `maint_hrs`..."_ -- `maint_hrs` is a `varchar`, not a number. So are `699_paint` and `699_shop`.  This does not make sense.
Comment: What is the actual problem with your solution? Is there a syntax error? Is the computed result wrong?
Comment: The computed results are not correct. It is not grouping per week or calculation overtime per week.
Answer: Try this:
SELECT s.Name, YEAR(s.Date), WEEKOFYEAR(s.date),
      IF(SUM(test_hrs+fab_hrs+control_hrs+shop_hrs+paint_hrs+engine_hrs+hydraulic_hrs)>40,
          SUM(test_hrs+fab_hrs+control_hrs+shop_hrs+paint_hrs+engine_hrs+hydraulic_hrs))-40,
          0
         ) AS overtime
FROM shift_log_data s
GROUP BY s.Name, YEAR(s.Date), WEEKOFYEAR(s.date);


$P(\lvert X - \mu\rvert \geq \sigma)$ as a measure of tailedness

Question: I know that one of the standard measures for the "tailedness" of a distribution is kurtosis, i.e. fourth standardized central moment $\frac{\mu_4}{\sigma^4}$. This measure is sort of intuitive to me: as the 4th power "punishes" the outliers much more than 2nd power of the variance does, we would expect a distribution with a heavy tail to have a large kurtosis.
Recently, I thought of using another measure of tailedness--computing the probability content "far away" from the mean, let's call this measure $T$ (for "tail"). Then
$$T(X) = \mathbb{P}(\lvert X - \mu\rvert\geq\sigma)$$
where $X$ is a random variable whose tailedness we are measuring, with $\mu = \operatorname{E}X$, and $\sigma^2 = \operatorname{Var}(X)$
We could also perhaps use different multiples of $\sigma$ in the definition (which would, however, also bound the value of $T$ to less than $1$ because of Chebyshev's inequality).
Has this measure been used for judging the tailedness of a distribution? What are the properties causing it not to be in (widespread) use? It seems to me like it would be a good way to capture the "Amazon business model phenomenon," where they can exploit the high overall probability content among niche titles:
Comment: Because $\Pr(|X-\mu|\gt \sigma)$ is given by the *survival function* of the standardized variable $|(X-\mu)/\sigma|,$ you can find related definitions of "heavy" or "long" or "fat" tails based on the asymptotic behavior of the survival function.  See [this search](https://stats.stackexchange.com/search?q=heavy+tail*+survival).
Comment: You wrote "and $\sigma=\text{Var}(X)$". Did you mean to write either "and $\sigma^{2}=\text{Var}(X)$" or "and $\sigma=\sqrt{\text{Var}(X)}$"?
Answer: One attribute of the statistic $P(|X - \mu| \ge \sigma)$ is that it is only giving a single snapshot of "tailedness" that may capture what we want when comparing very specific distributions, but is very misleading for others. As an example, consider a Bernoulli with $p = 0.5$. By this metric, it has a maximally heavy tail, yet most would not consider this a heavy tailed distribution.
As you noted, this statistic can be adjusted by changing $P(|X - \mu| \ge \sigma)$ to $P(|X - \mu| \ge \lambda \sigma)$. Note that if we make $\lambda = 1 + \epsilon$ instead of $\lambda = 1$, our Bernoulli goes from the heaviest possible tails (by this metric) to the lightest possible tails. This suggests we would need to consider this metric over a range of $\lambda$'s to have meaningful insight into our distribution rather than just a particular value of $\lambda$.
With that said, evaluating $P(|X - \mu| \ge \lambda \sigma)$ for the values $\lambda = 1, 2, 3,...$ may help provide some interpretable measures of how the tails of a distribution decay, especially given that most analysts are aware that this metric is approximately 0.32, 0.05 and 0.003 for the normal distribution with $\lambda = 1,2,3$.
Comment: @whuber: I tried to clarify things a bit. And after a little thought, I do agree that it can provide helpful insight over a range of values. I was thinking of trying to combine those grid of values down to a univariate metric, which would probably loose a lot of interpretability.
Comment: Why not use kurtosis as that univariate metric? Let $\kappa$ denote (non-excess) kurtosis. One connection between kurtosis and Cliff's suggested combination of the grid of values into a univariate metric is $\kappa \ge \sup_\lambda \lambda^4 P(|X-\mu|) > \lambda \sigma).$  Another connection, even more directly related to Cliff's suggestion, is $\sum_{\lambda=1}^\infty [\lambda^4 - (\lambda-1)^4] P(|X-\mu|) \ge \lambda \sigma) \le \kappa \le 1 +  \sum_{\lambda=1}^\infty [(\lambda+1)^4 - \lambda^4] P(|X-\mu|) \ge \lambda \sigma).$
Comment: It's unclear what "this statistic" refers to: initially I thought you were talking about kurtosis, but a closer analysis of the rest of this post suggests you are referring the the tail probability in the question.  Since a range of $\lambda$ is contemplated in the question, it's unclear what your middle paragraph is addressing.  As far as "bit hard to easily comprehend" goes, the resulting function is just the survival function of the size of the standardized variable, a rather familiar object and one routinely used to study distributional tails.
Answer: It is intuitive as you say. But, I don't think it is more informative than the kurtosis because probabilities of such inequalities don't give much hint about the distribution after the boundaries. To be more specific, let's call $Z=\frac{|X-\mu|}{\sigma}$, and calculate $P(Z\geq k)$ as $0.01$. Density of $Z$ after $k$ could be either focused near $k$, or far away from it. In either case, you'd have the same probability. However, $E[Z^4]$ would immensely differ.
Answer: Heavier-tailed distributions do not necessarily have "more probability in the extreme tails," as is sometimes thought.  See here for a counterexample.https://math.stackexchange.com/a/2510884/472987
It is the rare, extreme values that characterize heavy tails. The Bernoulli(p) distribution provides a good example:  The smaller the p, the heavier the tail, in the sense that the rare event is more extreme, as measured by number of standard eviations from the mean.
Kurtosis is a good measure of tail heaviness (quantile-based for infinite moments, but for actual data, the ordinary Pearson kurtosis works).
Comment: The meaning of your first paragraph is unclear.  You seem to interpret "heavy tailed" in the sense of large kurtosis rather than in the more established senses described elsewhere [here on CV](https://stats.stackexchange.com/search?q=%22Heavy+tail*%22) and on [Wikipedia](https://en.wikipedia.org/wiki/Heavy-tailed_distribution), for which your statement would be incorrect.  Could you clarify what you mean by "heavy tailed"?
Comment: I would say that "heavy tailed" means "capable of producing rare, extreme data values" in the case of a pdf, and "exhibiting rare, extreme data values" in the case of data. "Extreme" can be interpreted as "extreme relative to what a normal distribution produces." In that sense, the Wikipedia definition of heavy-tailedness is unnecessarily restrictive to characterize models for real processes that produce rare, extreme values. Kurtosis types of measures are more general as measures of heavy-tailedness.
Comment: Okay, thank you for the clarification.  But I must say it sounds an awful lot like you are defining "heavy tailed" as some version of having "more probability in the extreme tails" (relative to a Normal distribution)!
Comment: Yes, definitely, "relative to the extreme tails normal distribution." But that is different from considering a sequence of distirbutions where heaviness of tails increases, and then saying that $P(|X-\mu| > k\sigma)$ must also increase, as the OP seems to suggest.
Comment: I'm unable to read the question as suggesting that, because it's a mathematical impossibility (assuming by "increase" you mean "is an increasing function of $k$") and does not correspond to the graphic in the question.  But the reason I was asking is you seem now to have flatly contradicted the opening statement in your post.  This renders it less than clear.
Comment: No, $k$ is fixed. What is increasing is the heavy tailedness of the sequence of distributions, as in my counterexample. That sequence of distributions has increasing tailedness, but $P(|X−\mu|>k\sigma)$ converges to zero for every fixed $k > 1.2$.


Finding alpha in pmf $P(X=i) = \frac{\alpha}{2^i}$

Question: I have to solve the following problem:
Let X be distributed over N (natural numbers) with probability mass function $P(X=i) = \frac{\alpha}{2^i}$ for some fixed alpha (which is an element of R, the real numbers). Find alpha.
I know that $\sum_{i=1}^{\infty} P(x=i) = 1$. Therefore $\sum_{i=1}^{\infty} \frac{\alpha}{2^i} =  \frac{\alpha}{\sum_{i=1}^{\infty}2^i}= 1$ and $\alpha = \sum_{i=1}^{\infty}2^i$. But this cannot be right as $\alpha$ would be infinite and we are looking for a fixed alpha.
Thanks for any help!
Answer: Hint:
$$
\sum_{i=1}^{\infty} \frac{\alpha}{2^i} \neq  \frac{\alpha}{\sum_{i=1}^{\infty}2^i}.
$$
Just as $1/2 + 1/3 \neq 1/5$. Also recall that $\sum_{i=0}^{\infty} a^i = (1-a)^{-1}$ as long as $|a| < 1$.
Comment: Thank's for the hint. But I don't succeed in rearranging my equation so that I can apply it. What would be $a$ and what $k$?
Comment: @motormiquel sorry I made a small typo. See the edit.
Answer: Ok here is my try, please correct me if I'm wrong.
$s_n = \sum_{i=1}^{\infty} P(X=i) = \sum_{i=1}^{\infty}\frac{\alpha}{2^i}
= \frac{\alpha}{2}+\frac{\alpha}{4}+\frac{\alpha}{8}+...+\frac{\alpha}{2^n}$
Then, $2s_n  = \frac{2\alpha}{2}+\frac{2\alpha}{4}+\frac{2\alpha}{8}+...+\frac{2\alpha}{2^n}$
$= \alpha+\frac{\alpha}{2}+\frac{\alpha}{4}+\frac{\alpha}{8}+...+\frac{\alpha}{2^{n-1}}$
$= \alpha + s_n -\frac{\alpha}{2^n}$
and $s_n = \alpha-\frac{\alpha}{2^n}$.
If now $n\rightarrow \infty, s_n \rightarrow \alpha$. And as $s_n$, the sum over all probabilities, must equal $1$, $\alpha = 1$.
Comment: $\sum_{i=1}^{\infty}\frac{\alpha}{2^i}
= \frac{\alpha}{2}+\frac{\alpha}{4}+\frac{\alpha}{8}+...+\frac{\alpha}{2^n}$ you have an extra $\alpha$
Comment: Right, thank's. I've made the corrections above, please verify!
Comment: nice! Looks good


Is it possible to select multiple columns in Virtual Treeview?

Question: I need to add functionality to copy a rectangular selection of nodes and columns, but I can't find any way to actually select multiple columns in a Virtual Treeview (beside toFullRowSelect).
Am I just missing something? And if not, is there a descendant out there with grid-like multicolumn select support?
Answer: So after some testing I came up with the following, thanks DiGi for the extra push. DrawSelection won't work with this solution so it needs to be disabled. Since I don't think I'll need to do this again soon I didn't write a descendant.
Set toDisableDrawSelection, toExtendedFocus and toMultiSelect to True.
Declare the following variables/properties somewhere suitable:
StartSelectedColumn: integer;
FirstSelectedColumn: integer;
LastSelectedColumn: integer;
Selecting: boolean;

Update the following events:
OnKeyDown
if (not Selecting) and (Key = VK_SHIFT) then
begin
  StartSelectedColumn := vtMain.FocusedColumn;
  FirstSelectedColumn := StartSelectedColumn;
  LastSelectedColumn := StartSelectedColumn;
  Selecting := true;
end;

OnKeyUp
if Key = VK_SHIFT then
  Selecting := false;

OnFocusChanged
if Selecting then
begin
  if column < StartSelectedColumn then
  begin
    FirstSelectedColumn := column;
    LastSelectedColumn := StartSelectedColumn;
  end
  else if column > StartSelectedColumn then
  begin
    FirstSelectedColumn := StartSelectedColumn;
    LastSelectedColumn := column
  end
  else
  begin
    FirstSelectedColumn := column;
    LastSelectedColumn := column;
  end;
end
else
begin
  StartSelectedColumn := column;
  FirstSelectedColumn := column;
  LastSelectedColumn := column;
end;

OnBeforeCellPaint
if vtMain.Selected[node] and InRange(column, FirstSelectedColumn, LastSelectedColumn) then
begin
  if vtMain.Focused then
    TargetCanvas.Brush.Color := vtMain.Colors.FocusedSelectionColor
  else
    TargetCanvas.Brush.Color := vtMain.Colors.UnfocusedSelectionColor;
  TargetCanvas.Brush.Style := bsSolid;
  TargetCanvas.FillRect(CellRect);
end;

OnPaintText
if vtMain.Selected[node] and InRange(column, FirstSelectedColumn, LastSelectedColumn) then
begin
  if vtMain.Focused then
    TargetCanvas.Font.Color := clHighlightText
  else
    TargetCanvas.Font.Color := vtMain.Font.Color;
end;

Answer: One more tip - look at OnStateChange event, maybe you can use
procedure TSomeForm.VTreeStateChange(Sender: TBaseVirtualTree; Enter,Leave: TVirtualTreeStates);
begin
  if tsDrawSelecting in Enter then
  begin
    // Save position
  end;
end;

Comment: Interesting, I'll look at it later. :) Thanks!
Answer: You can try enable/add toGridExtensions in TreeOptions.MiscOptions. It enables free moving in columns by cursor keys, but VT still deselect column on leaving. But I'm sure that is possible to "fix" it by custom draw and remembering starting node and column.
Comment: I've tried this and it doesn't work very well when you want to use a mouse. It does however "solve" the problem so upvote. :)


Does the tension of a tennis racket really affect the game play?

Question: I'm an amateur who has played for a couple of years. In the beginning I bought just a random racket - "for beginners", they say. Then I saw some Big Helmet tournaments and got curious about players who punch their rackets and change the behavior.
How exactly does the tension affect the game?  And if the tension really matters, then how will I know if I need to restring?
Does it make sense to try to play with a different racket - will I "feel" the difference?
Comment: Try to stick with asking one question at a time (read [about] that in the [help]).
Comment: If there was no tension, there surely would be a bad effect on play.
Answer: I'll try my best to answer your question - I don't quite understand what you're asking when you say

punches their racquet's strings and changes the racquet after that

If you are a beginning tennis player, the rule of thumb is that you should restring your racquet as many times in a year as you play in a week. So if you play twice a week, you should get your racquet restrung every 6 months. Some people play with the same strings for years or until the strings break.
The reason some players get the strings replaced before they break is because of tension loss. It depends on the string material used - but all strings lose tension after being used for a while.
For professional players, they get their racquets restrung constantly because any amount of tension loss can potentially affect their shots, so they try to minimize how much that can happen. A single player could get anywhere from 2-10 racquets restrung for a tournament. The very top players (like Federer, Nadal, Djokovic, Murray, etc.) can get about 8 racquets restrung for them before every match they play. They pay a professional racquet stringing company to travel with them to various tournaments to restring their racquets for them.
Unless you are a fairly advanced player, you are not likely to notice a little difference in the amount of tension in strings - like 3-5 kg difference. But for advanced players, they will notice because they have a better feel for the tension.
Tension typically affects the amount of power (or control) you get from the strings. The lower the tension - the more power you get. The higher the tension, the less power (thus control) you get. It's just like a trampoline - a loose trampoline will bounce you higher in the air than a tight one will.
Answer: Not sure where anyone got the information that higher/lower tension changes the size of a sweet spot. It's the racquet head size and the number of strings. (Definition of a sweet spot is the most efficient power generated with the least amount of vibration)
That said, I found an interesting article published by a Dr. at the University of Sheffield (http://www.tennisindustrymag.com/articles/2005/01/does_higher_string_tension_giv.html) which shows the correlation between low tension - 40lbs and high tension - 70lbs and their effects on spin. (Seeing as we know that more power will be generated with lower tension)
If you don't want to read - it summarises that with higher tension you are more likely to execute a shot correctly as the ball does not travel along the stringbed as much as it would at a lower tension.
The conclusion: "Changing racquet tension does not affect spin, but it does affect string movement, dwell time, and ball contact distance. These latter parameters all can affect the ball trajectory as well as the player's feel of the impact.
The main advice is that high string tensions make your shot more consistent and make it easier to hit topspin shots." - Dr. Simon Goodwill
Answer: The tension of a racket really matters !!

high racket tension allows for more precise hitting, but requires more experience
low racket tension will give you more control. but with less accuracy

All has to do with the so called sweet spot. The higher the tension, the smaller the sweet spot, the less likely for you to hit optimally.
After a racket has been restrung, tension usually drops a bit (0.5 - 1.0 kg), maintaining a steady tension afterwards. Restringing a racket (for me at least) only happens when one of the strings snap.
Only experienced players or (semi)-professional players will notice a decline in racket tension. You shouldn't be worrying about that.
Comment: low tension does NOT give you more control - it gives you more power, which, unless you are an experienced player who knows how to properly control that power - will lead to the ball flying further on your shots than you want it to. I don't see how you can say "more control" and "less accuracy". Control = accuracy.
Comment: @jamauss It gives more power due to the trampoline effect. It gives more control, as the sweet spot is larger, thus increasing the likelihood of hitting in that zone (for an inexperienced player that means more control). Because of the trampoline effect, it will give less accurate ball flights, due to an increased contact area with the ball (more diffuse interaction).
Answer: Yes, the tension on your racket matters.  A loosely strung racket will act like a trampoline.  When you hit the ball, your strings will generate more speed on your shots.  But with a higher tension, this rebound effect is reduced, giving you a smaller sweet spot but more control.


Are all non-linear classifiers using a kernel function?

Question: This is obviously a very general question but most classifiers would use a kernel to map the input into another space where the data could be linearly separated. I also wonder why this is done - I understand that linear separability possibly makes the calculation easier. However if I understand for instance Support Vector Machines correctly, the hyperplane in the kernel space does have a non-linear equivalent in the input space - so why do we not use this non-linear equivalent in the input space from the start? Is the only reason the easier calculation of the kernel function in the kernel space?
Answer: Of course not. Random forests are known for being capable to separate even very complex classes with nonlinear decision boundary, for which one will think about SVM with some kernel ( http://scikit-learn.org/stable/modules/ensemble.html )
Also ( deep ) autoencoders are not always based on kernel computations but they are nonlinear.
Typically, it's very difficult to find this non-linear equivalent in the input space from the beginning, although, of course, when you know some hidden magic behind you data you can build much simpler classifiers ( and in general you are even obliged to do some investigation of your data prior to solving classification task )
However, to my experience, when you build very strong classifiers for image processing tasks, all really good results are thanks to deep architectures, which are kernel-based under the hood ( and these kernels are learnable! )


Adding "Unblock the user" option in Views Bulk Operations field

Question: I have created a user based view and I used the Views Bulk Operations module, but I don't see Unblock user or something like that. I also want to show tree-based roles in the bulk operation options list.
How can I achieve this?
Answer: In drupal 7-
We need a Rules module, after enable Rules module and Rules UI go to
1- Configuration -> Workflow-> Rules-> Component-> Add Component
In Component plugin dropdown-> select Action Set
then add next page
Fill the mandatory field Name as (Ex: Unblock user)
2- Select Data type from list User ---Label Unblock user ---machine name unblock_user -- usage parameter
3- In setting option you can set Permission roles wise.
then click on continue button
then click on Add Action Link
4- then Select the action to add User ->Unblock User
5- Select Data Selector -> unblock-user and Save
6- Now go to- Views Page add field Bulk operations there will be present Unblock user option
for reference - Add Unblock user option by rules in views and set permission
Answer: Below is the code to add 'Unblock the user' option in Views Bulk Operations field.
<code>/**
* Implements hook_action_info().
*/
function mymodule_action_info() {
return array(
'mymodule_unblock_user_action' => array(
'label' => t('Unblock the user'),
'type' => 'user',
'configurable' => FALSE,
'triggers' => array('any'),
 ),
 );
}

/**
 * Unblocks a user, defaulting to the current user.
  *
 * @ingroup actions
 */
 function mymodule_unblock_user_action($entity, $context = array()) {
 // First priority: If there is a $entity->uid, unblock that user.
 // This is most likely a user object or the author if a node or comment.
 if (isset($entity->uid)) {
 $uid = $entity->uid;
    }
   // Otherwise get user ID from the context.
 elseif (isset($context['uid'])) {
  $uid = $context['uid'];
  }
 $account = user_load($uid);
  $account = user_save($account, array('status' => 1));
 watchdog('action', 'Unblocked user %name.', array('%name' => $account->name));
    }
</code>
Taken answer from This Link.

Adding “Unblock the user” option in Views Bulk Operations field


MCMC algorithm to generate samples

Question: I read that MCMC algorithm is used to draw samples from a distribution.
The example mentioned in the text book is about a 6x6 matrix which after 1000 iterations will converge to a steady state 1x6 matrix as shown below.

Okay I understand how this reaches the state state part,
<code>[ 0.1, 0.2, 0.2, 0.2, 0.2, 0.1 ]
</code>
How will I generate samples from this matrix ? This is where I am confused.
Answer: The matrix $P$ is, presumably, a transition matrix for a Markov chain. For some Markov chains, a stationary distribution exists. For your example, you get
\begin{equation}
P^{1000}=\begin{pmatrix}
0.1&0.2&0.2&0.2&0.2&0.1\\
0.1&0.2&0.2&0.2&0.2&0.1\\
0.1&0.2&0.2&0.2&0.2&0.1\\
0.1&0.2&0.2&0.2&0.2&0.1\\
0.1&0.2&0.2&0.2&0.2&0.1\\
0.1&0.2&0.2&0.2&0.2&0.1\\
\end{pmatrix}.
\end{equation}
So you can see, informally, that a stationary distribution for the Markov chain with transition matrix $P$ exists.
The basic idea for MCMC methods is that the stationary distribution is chosen to be the distribution you want to sample from. An MCMC algorithm then constructs a Markov chain that has the appropriate stationary distribution. The most commonly used MCMC algorithm is the Metropolis-Hastings algorithm, with which the Gibbs sampler is a special case you can use when the full conditional densities of the target distribution are known. http://en.wikipedia.org/wiki/Metropolis-Hastings_algorithm gives a good explanation of how it works and how to use it.

MCMC algorithm to generate samples


Reading from file in R when I have a part of its name

Question: How can I read from a file in R when I have only a part of its name including starting or ending characters?
Thanks
Comment: How about using `list.files()` to get a list of all the files in the working directory, then see which one matches your criteria, then read it. You've given so few details it's hard to be more specific.
Answer: You can use list.files, which has a pattern argument, to try to match as close as you can.
writeLines(c('hello', 'world'), '~/tmp/example_file_abc')
filename <- list.files(path = '~/tmp', pattern = 'file_abc$', full.names = TRUE)[1]
readLines(filename)
# [1] "hello" "world"

Answer: There is also Sys.glob which will expand patterns using stars and question marks according to glob syntax.
Here it is wrapped in a function to match filenames of the form "first*last" where "*" is anything. If you really have stars or other special chars in your filename... then you need to do a bit more.. Anyway:
> match_first_last = function(first="", last="", dir=".")
   {Sys.glob(
      file.path(dir,paste(first,"*",last,sep=""))
      )
    }

# matches "*" and so everything:
> match_first_last()
[1] "./bar.X" "./foo.c" "./foo.R"

# match things starting `foo`
> match_first_last("foo")
[1] "./foo.c" "./foo.R"

# match things ending `o.c`
> match_first_last(last="o.c")
[1] "./foo.c"

# match start with f, end in R
> match_first_last("f","R")
[1] "./foo.R"


Install sw RAID 1 on production server from single disk?

Question: Is it possible to install software RAID 1 on a single disk Debian server without erasing/format the data on the original disk?
What approach would you recommend?
Comment: WHY would you do that? On a single disc there is no advantage at all to use RAID 1 - only disadvantages (writes REALLY get slower).
Comment: It can be done but why would you ever have a need for this?!...
Comment: The reason is redundancy, I don't need the extra speed with RAID 0, just the security if one disk fails.
Comment: How will you have redundancy with only one disk ?
Comment: I don't think he means to use a single disk all along. He has a single disk right now and wants to plug in another and convert to a RAID1. But even with a single disk you would get some amount of redundancy against bad blocks - ZFS for example has [*ditto blocks*](http://www.c0t0d0s0.org/archives/3107-Ditto-data-blocks-in-ZFS.html) which is quite similar when set on a storage pool with a single physical disk.
Answer: You can. I don't understand why you want only one disk, but you can. And this won't slow your disk accesses.
You need to shrink the filesystem to make sure there is 128K free at the end of the device, because Linux software RAID need it:
<code>umount /
 resize2fs /dev/xxx your_size
 mdadm --create /dev/mdx --raid-devices=2 --level=raid1 /dev/xxx missing
 mount /dev/md0 /xxxxx
</code>
I strongly suggest you to test it with a dummy computer or a virtual machine.
Comment: Ok, it sounds like it's nothing you recommend? Would it be better to set up a new server with configured RAID and migrate all data to it?
Comment: Seems you are not aclimated with software RAID and/or Linux, so I suggest you to test before playing with datas. The procedure itself is simple and relatively safe, but Murphy's watching you.


ETL as a transaction

Question: For all the ETLs I have written so far, I have never made them transactions - i.e. if table 4 fails, roll everything back.
What is the best practice in this regard?
To "BeginTran + Commit" or not to "BeginTran + Commit"
EDIT: I have one master package calling 4 other packages - is it possible to roll them all up into one transaction?
Answer: In SSIS, I always Begin Trans + Commit. I want to make sure that I can re-run the package without issue (or having to find what rows actually got inserted) if it fails.
It just makes recovery and cleanup so much easier.
Answer: begin+commit in manageable batch sizes. You don't want to wrap a 6 hours import into a single transaction every night. Keep your batches at a size that can finish in 2-3 minutes at most. That you will hit data purity issues that will fail an ETL is a given, so at least reduce the impact to something manageable (ie. don't trigger a rollback that will last another 6 hours to complete).
Answer: You are often moving too much data in ETL to use a SQL transaction (the log has to store ALL the data to roll back, remember). I prefer to design packages such that they can be re-run nondestructively. Ideally they should be set up so that if they die in mid-stream, you can just start them and they'll continue somewhere approximately where they left off. Sometimes there's a performance penalty for that, but I think it's worth it.
Technically you can roll packages up into a single transaction; practically, maybe not.

ETL as a transaction