oversampling in nested cross validation

Question: Introduction
I have a small mixed dataset consisting of continuous and categorical independent variables with a dichotomous dependant variable. I'm running various algorithms (neural networks, random forests and logistic regression) to find the best model to generalise on unseen data. To do this i have performed a nested cross validation so that I can train and optimise hyper parameter and feature selection (on the inner loop) without being too biased on the model performance. I have conducted a 10 fold cross validation in the inner loop and outer loop.
My Question
My dataset has a class imbalance. To overcome this problem I have oversampled with replacement the minority class so that the majority is now at 80% and the minority at 20%. I know this still has imbalance, however it is now more appropriate to what can be found in the field and better than the original imbalance. My question is will this cause bias in the model performance because I used a oversampling technique with replacement?
Comment: Why do you think the class imbalance is a problem, and what problem is the oversampling addressing?
Comment: Hi Matthew, Class imbalance is a problem because the algorithms im working with tend to favour the majority class and disregard the minority. Oversampling will hopefully prevent this bias in the model
Comment: Can you say more about the algorithms you are working with?
Comment: Hi matthew, im working with the algorithms i mentioned in the original question
Answer: As far as I know, in spite of how widespread this idea is, there is no convincing argument that artificial oversampling makes sense or leads to a better model. See for instance this question, or this question, both about whether class imbalance is a real problem, both highly voted/often viewed, neither with any clear indication that this practice is a good idea.
Oversampling will clearly introduce bias in your model because $P(class|data) = \frac{P(data|class)P(class)}{P(data)},$ and you're artificially messing with $P(class)$.

oversampling in nested cross validation


Ferramenta para desenho de gráficos em java

Question: Gostaria de saber se tem alguma ferramenta com IReport que eu possa utilizar para fazer um gráfico parecido com essa imagem. Já utilizei o JFreechart, mas não consegui deixar parecido! Os valores das barras tem que ficar em baixo e os rótulos em cima conforme a imagem.
Answer: Acho que talvez você tenha que desenhar os gráficos na mão. Fiz um programa que faz isso.
Aqui está o código:
import java.awt.Color;
import java.awt.FontMetrics;
import java.awt.Graphics2D;
import java.awt.image.BufferedImage;
import java.util.ArrayList;
import java.util.List;

/**
 * @author Victor
 */
public class GraficoDeBarras {

    // Formato é AARRGGBB, aonde A é o alpha, R é o vermelho, G é o verde e B é o azul.
    private static final int AZUL = 0xFF99D8EA;
    private static final int VERMELHO = 0xFFEA9999;

    private final int larguraRetangulo;
    private final int alturaRetangulo;
    private final int alturaTotal;
    private final int yBaseBarra;
    private final int xBaseBarra;
    private final int larguraBarra;
    private final int profundidade;
    private final int yRetangulo;
    private final int yRotulo;
    private final int yValor;

    public GraficoDeBarras(
            int larguraRetangulo,
            int alturaRetangulo,
            int alturaTotal,
            int yBase,
            int xBase,
            int larguraBarra,
            int profundidade,
            int yRetangulo,
            int yRotulo,
            int yValor)
    {
        this.larguraRetangulo = larguraRetangulo;
        this.alturaRetangulo = alturaRetangulo;
        this.alturaTotal = alturaTotal;
        this.yBaseBarra = yBase;
        this.xBaseBarra = xBase;
        this.larguraBarra = larguraBarra;
        this.profundidade = profundidade;
        this.yRetangulo = yRetangulo;
        this.yRotulo = yRotulo;
        this.yValor = yValor;
    }

    private static class Barra {
        private final int alturaBarra;
        private final int cor;
        private final String valor;
        private final String rotulo;

        public Barra(int alturaBarra, int cor, String valor, String rotulo) {
            this.alturaBarra = alturaBarra;
            this.cor = (cor & 0x00FFFFFF) | 0xFF000000; // Elimina a transparência.
            this.valor = valor;
            this.rotulo = rotulo;
        }

        public void desenhar(GraficoDeBarras grafico, int x, int y, Graphics2D g) {
            g.setColor(Color.BLACK);
            Color preenchimento = new Color(cor);

            // Desenha as bordas externas.
            int x1 = x;
            int x2 = x + grafico.larguraRetangulo - 1;
            int y1 = y + grafico.yRetangulo;
            int y2 = y + grafico.yRetangulo + grafico.alturaRetangulo - 1;
            g.drawLine(x1, y1, x2, y1);
            g.drawLine(x1, y1, x1, y2);
            g.drawLine(x2, y1, x2, y2);
            g.drawLine(x1, y2, x2, y2);

            // Prepara o desenho do paralelepípedo.
            int xBarra1 = x1 + grafico.xBaseBarra;
            int xBarra2 = x1 + grafico.xBaseBarra + grafico.larguraBarra;
            int yBarra1 = y1 + grafico.yBaseBarra;
            int yBarra2 = y1 + grafico.yBaseBarra - alturaBarra;

            // Se a altura for negativa, precisará inverter o yBarra2 e o yBarra1.
            if (yBarra2 > yBarra1) {
                int t = yBarra2;
                yBarra2 = yBarra1;
                yBarra1 = t;
            }

            // Desenha as bordas frontais do paralelepípedo.
            int[] xFrente = {xBarra1, xBarra1, xBarra2, xBarra2};
            int[] yFrente = {yBarra1, yBarra2, yBarra2, yBarra1};
            g.setColor(preenchimento);
            g.fillPolygon(xFrente, yFrente, 4);
            g.setColor(Color.BLACK);
            g.drawPolygon(xFrente, yFrente, 4);

            // Desenha as bordas laterais do paralelepípedo.
            int[] xLateral = {xBarra2, xBarra2, xBarra2 + grafico.profundidade, xBarra2 + grafico.profundidade};
            int[] yLateral = {yBarra1, yBarra2, yBarra2 - grafico.profundidade, yBarra1 - grafico.profundidade};
            g.setColor(preenchimento);
            g.fillPolygon(xLateral, yLateral, 4);
            g.setColor(Color.BLACK);
            g.drawPolygon(xLateral, yLateral, 4);

            // Desenha as bordas superiores do paralelepípedo.
            int[] xSuperior = {xBarra1, xBarra1 + grafico.profundidade, xBarra2 + grafico.profundidade, xBarra2};
            int[] ySuperior = {yBarra2, yBarra2 - grafico.profundidade, yBarra2 - grafico.profundidade, yBarra2};
            g.setColor(preenchimento);
            g.fillPolygon(xSuperior, ySuperior, 4);
            g.setColor(Color.BLACK);
            g.drawPolygon(xSuperior, ySuperior, 4);

            // Escreve o valor e o rótulo.
            FontMetrics fm = g.getFontMetrics();
            int larguraValor = fm.charsWidth(valor.toCharArray(), 0, valor.length());
            int larguraRotulo = fm.charsWidth(rotulo.toCharArray(), 0, rotulo.length());
            g.drawString(valor, x + (grafico.larguraRetangulo - larguraValor) / 2, y + grafico.yValor);
            g.drawString(rotulo, x + (grafico.larguraRetangulo - larguraRotulo) / 2, y + grafico.yRotulo);
        }
    }

    private BufferedImage desenharBarras(List<Barra> barras) {
        BufferedImage bi = new BufferedImage((larguraRetangulo - 1) * barras.size() + 1, alturaTotal, BufferedImage.TYPE_INT_ARGB);
        Graphics2D g = bi.createGraphics();
        int x = 0;
        for (Barra b : barras) {
            b.desenhar(this, x, 0, g);
            x += larguraRetangulo - 1;
        }
        return bi;
    }

    public static final class Dado {
        private final int valor;
        private final String rotulo;

        public Dado(int valor, String rotulo) {
            this.valor = valor;
            this.rotulo = rotulo;
        }

        public int getValor() {
            return valor;
        }

        public String getRotulo() {
            return rotulo;
        }
    }

    public static BufferedImage desenhar(int alturaTotalDasBarras, List<Dado> dados) {
        if (dados == null || dados.isEmpty()) {
            throw new IllegalArgumentException("Apenas uma lista de dados não-vazia pode ter um gráfico gerado.");
        }
        if (alturaTotalDasBarras < 2) {
            throw new IllegalArgumentException("A altura total das barras deve ser de no mínimo 2 pixels.");
        }

        // Define detalhes fixos das dimensões do gráfico.
        int larguraBarra = 40;
        int profundidade = 10;
        int margem = 10;
        int yRetangulo = 20;

        // Encontra o maior e o menor valor e o tamanho da faixa de valores. Sempre considerando que 0 está dentro desta faixa.
        int max = 0;
        int min = 0;
        for (Dado d : dados) {
            int valor = d.getValor();
            if (valor > max) max = valor;
            if (valor < min) min = valor;
        }
        int variacao = Math.max(1, max - min); // min nunca será positivo. Se todos os números forem zeros, não queremos dividir por zero.

        // Agora que "variacao" foi calculada, percorre a lista novamente para calcular a altura de cada barra em pixels.
        List<Barra> lista = new ArrayList<Barra>(dados.size());
        for (Dado d : dados) {
            int valor = d.getValor();
            int altura = (int) (((double) valor) * alturaTotalDasBarras / variacao);
            Barra barra = new Barra(altura, valor >= 0 ? AZUL : VERMELHO, String.valueOf(d.getValor()), d.getRotulo());
            lista.add(barra);
        }

        // Recalcula os tamanhos das barras em pixels ao invés dos valores brutos.
        int maxPixels = (int) (((double) max) * alturaTotalDasBarras / variacao);
        int minPixels = (int) (((double) min) * alturaTotalDasBarras / variacao);
        int variacaoPixels = Math.max(1, maxPixels - minPixels); // min nunca será positivo.

        // Calcula os demais detalhes das dimensões do gráfico.
        int larguraRetangulo = larguraBarra + profundidade + 2 * margem;
        int alturaRetangulo = variacaoPixels + profundidade + 2 * margem;
        int alturaTotal = alturaRetangulo + 40;
        int yBase = alturaRetangulo + minPixels - margem;
        int yRotulo = yRetangulo - 4;
        int yValor = alturaTotal - 4;

        // Cria e desenha o gráfico.
        GraficoDeBarras g = new GraficoDeBarras(larguraRetangulo, alturaRetangulo, alturaTotal, yBase, margem, larguraBarra, profundidade, yRetangulo, yRotulo, yValor);
        return g.desenharBarras(lista);
    }

    public static BufferedImage desenharTamanhoFixo(int larguraTotal, int alturaTotal, List<Dado> dados) {
        if (dados == null || dados.isEmpty()) {
            throw new IllegalArgumentException("Apenas uma lista de dados não-vazia pode ter um gráfico gerado.");
        }

        // Define detalhes fixos das dimensões do gráfico.
        int profundidade = 10;
        int margem = 10;
        int yRetangulo = 20;

        // Calcula detalhes sobre a altura e largura.
        int alturaRetangulo = alturaTotal - 40;
        int alturaTotalDasBarras = alturaRetangulo - profundidade - 2 * margem;
        if (alturaTotalDasBarras <= 1) throw new IllegalArgumentException("A altura da imagem é insuficiente.");
        int larguraRetangulo = 1 + ((larguraTotal - 1) / dados.size());
        int larguraBarra = larguraRetangulo - profundidade - 2 * margem;
        if (larguraBarra <= 1) throw new IllegalArgumentException("A largura da imagem é insuficiente.");

        // Encontra o maior e o menor valor e o tamanho da faixa de valores. Sempre considerando que 0 está dentro desta faixa.
        int max = 0;
        int min = 0;
        for (Dado d : dados) {
            int valor = d.getValor();
            if (valor > max) max = valor;
            if (valor < min) min = valor;
        }
        int variacao = Math.max(1, max - min); // min nunca será positivo. Se todos os números forem zeros, não queremos dividir por zero.

        // Agora que "variacao" foi calculada, percorre a lista novamente para calcular a altura de cada barra em pixels.
        List<Barra> lista = new ArrayList<Barra>(dados.size());
        for (Dado d : dados) {
            int valor = d.getValor();
            int altura = (int) (((double) valor) * alturaTotalDasBarras / variacao);
            Barra barra = new Barra(altura, valor >= 0 ? AZUL : VERMELHO, String.valueOf(d.getValor()), d.getRotulo());
            lista.add(barra);
        }

        // Recalcula os tamanhos das barras em pixels ao invés dos valores brutos.
        int minPixels = (int) (((double) min) * alturaTotalDasBarras / variacao);

        // Calcula os demais detalhes das dimensões do gráfico.
        int yBase = alturaRetangulo + minPixels - margem;
        int yRotulo = yRetangulo - 4;
        int yValor = alturaTotal - 4;

        // Cria e desenha o gráfico.
        GraficoDeBarras g = new GraficoDeBarras(larguraRetangulo, alturaRetangulo, alturaTotal, yBase, margem, larguraBarra, profundidade, yRetangulo, yRotulo, yValor);
        return g.desenharBarras(lista);
    }
}

Para testar eu fiz esta classe:
import java.awt.BorderLayout;
import java.awt.Dimension;
import java.awt.EventQueue;
import java.awt.Graphics;
import java.awt.Toolkit;
import java.awt.image.BufferedImage;
import java.util.Arrays;
import java.util.List;
import javax.swing.JFrame;
import javax.swing.JPanel;

/**
 * @author Victor
 */
public class FazGrafico {

    public static void main(String[] args) {
        List<GraficoDeBarras.Dado> dados = Arrays.asList(
                new GraficoDeBarras.Dado(5, "Janeiro"),
                new GraficoDeBarras.Dado(8, "Fevereiro"),
                new GraficoDeBarras.Dado(3, "Março"),
                new GraficoDeBarras.Dado(0, "Abril"),
                new GraficoDeBarras.Dado(14, "Maio"),
                new GraficoDeBarras.Dado(-5, "Junho"));
        BufferedImage imagem = GraficoDeBarras.desenharTamanhoFixo(500, 180, dados);
        //BufferedImage imagem = GraficoDeBarras.desenhar(100, dados);
        // Java 8: EventQueue.invokeLater(() -> mostraImagem(imagem));
        EventQueue.invokeLater(new Runnable() {
            @Override
            public void run() {
                mostraImagem(imagem);
            }
        });
    }

    private static void mostraImagem(BufferedImage imagem) {
        JFrame jf = new JFrame("Teste do gráfico");
        JPanel jp = new JPanel() {
            @Override
            public void paintComponent(Graphics g) {
                g.drawImage(imagem, 0, 0, null);
            }
        };
        jf.add(jp, BorderLayout.CENTER);
        jp.setPreferredSize(new Dimension(imagem.getWidth(), imagem.getHeight()));
        jf.setResizable(false);
        jf.pack();
        jf.setDefaultCloseOperation(JFrame.DISPOSE_ON_CLOSE);
        Dimension screen = Toolkit.getDefaultToolkit().getScreenSize();
        jf.setLocation((screen.width - jf.getWidth()) / 2, (screen.height - jf.getHeight()) / 2);
        jf.setVisible(true);
    }
}

E aqui está alguns dos gráficos gerados:
Este é utilizando GraficoDeBarras.desenhar(100, dados) com os valores {5, 8, 3, 0, 14}:

Este é utilizando GraficoDeBarras.desenhar(100, dados) com os valores {5, 8, 3, 0, 14, -5}:

Este é utilizando GraficoDeBarras.desenharTamanhoFixo(500, 180, dados) com os valores {5, 8, 3, 0, 14, -5}:

No método desenhar, o primeiro parâmetro é a altura máxima que a face frontal de uma barra pode ter. Os demais valores necessários para calcular o tamanho da imagem, bem como o tamanho e posição dos seus elementos são baseados nisso.
No método desenharTamanhoFixo, o primeiro parâmetro é a largura total da imagem e o segundo é a altura total. Os demais valores necessários para calcular o tamanho e posição dos seus elementos são baseados nisso.
Em ambos os métodos, as barras tem 10 pixels de profundidade e uma margem de 10 pixels para as bordas do retângulo na direita, esquerda, abaixo e acima. Também há uma margem de 20 pixels acima e abaixo dos retângulos aonde serão colocados os valores e os rótulos, que tem cada um a sua baseline 4 pixels acima da borda inferior da respectiva faixa. No caso do método desenhar, a face frontal da barra sempre tem 40 pixels de largura. As cores usadas para os positivos são o azul 0xFF99D8EA (RGB = (153, 216, 234)) e para os negativos o vermelho 0xFFEA9999 (RGB = (234, 153, 153)). Mudar isso não deverá ser algo difícil ou trabalhoso.
Além disso, o código é bastante flexível para você colocar as variações que quiser, tal como evidenciado no construtor da classe GraficoDeBarras. Você não deverá ter grandes dificuldades para alterar a largura e a profundidade das barras, o tamanho das margens entre as barras e as bordas do retângulo que as contém, a largura e altura destes retângulos, as cores, as posições do texto, e obviamente os valores e os rótulos, entre outras possibilidades.
Comment: @RenatoRomão Alterei o código para ser compatível com Java 6. Além disso coloquei uns comentários para explicar o 0xFF99D8EA e o 0xFFEA9999.
Comment: Olá Victor agradeço sua ajuda. Gostaria de saber qual versão do java você esta usando, porque esta dando erro nessa parte que tem esse valores: 0xFF_99_D8_EA : 0xFF_EA_99_99
Comment: Victor tem como eu fazer espaçamento nos rotulos?
Comment: @RenatoRomão Estou usando o Java 8. A partir do Java 7 é permitido usar `_` em literais numéricos. Exemplo: `1_234_567` ao invés de `1234567`. Para o Java 6 ou inferior você terá que remover os `_` (fica `0xFF99D8EA : 0xFFEA9999`). Além disso aonde está `ArrayList<>` só funciona do Java 7 para cima, no java 6 deverá ser `ArrayList`. Por fim, no programa de exemplo, estou usando `EventQueue.invokeLater(() -> mostraImagem(imagem));`, que no Java 7 e inferior terá que usar a classe anônima: `EventQueue.invokeLater(new Runnable() { @Override public void run() { mostraImagem(imagem); } });`.
Comment: @RenatoRomão O que você quer dizer exatamente por "espaçamento nos rótulos?"
Comment: Deixar os rotulo centralizado junto com as barras. Tipo ficar na mesma posição das barras.
Comment: Victor é deixar os rotulo centralizado para ficar na mesma posição das barras
Comment: @RenatoRomão Implementado. Centralizei os rótulos e atualizei as figuras. :)
Comment: Vitor obrigado, já estou conseguindo fazer que eu precisava, me ajudou bastante esse código. Uma duvia eu consigo deixar um tamanho fixo na imagem tipo Largura e altura é porque eu vou usar isso no ireport passando uma imagem.
Comment: @RenatoRomão Calma aí que estou tentando dar um jeito nisso...
Comment: Vitor eu já conseguir fazer, faço ele desenhar as bordas externas, mas não desenhar as barras. Era bem isso mesmo que eu precisava. Agora eu consigo fazer 15 colunas mesmo sem aparecer as barrinhas.
Comment: @RenatoRomão Consegui mesmo assim, veja o método `desenharTamanhoFixo`. Também aproveitei a oportunidade para melhorar um pouco a organização do código e acrescentar mais comentários.
Comment: Victor agradeço pela sua atenção. Esse código me ajudou muito...
Comment: @RenatoRomão Valeu, foi até divertido desenvolvê-lo. :)


What is the top speed of swimmers when they leave the blocks?

Question: At the beginning of a race, swimmers dive off the starting blocks into the water. I would like to know how fast they are moving when they enter the air, before they hit the water.
Thus far my search attempts have resulted only in discussions of response time, training to increase this speed, or speed of the athletes as they move through the water.
I am looking for a concrete empirical measurement of their horizontal movement, such as in meters per second.
Answer: I just tried roughly calculating it with a video of a championship and I estimate the speed at 3 m/s but the video is a bit shaky.
Comment: Evidence would need to be a bit more scientific than that, I'm afraid.


Irreducibility Prime Numbers Field Theory

Question: I am struggling with the best approach for this problem:
Let $p$ be a prime integer and consider the polynomials $f(x) = x^p$ and $g(x) = x$ over $\Bbb Z_p$. Prove that $f(c) = g(c)$ for all $c$ in $\Bbb Z_p$.
Any help would be hugely appreciated!
Thanks!
Comment: The multiplicative group of $\mathbb Z_p$ is of order $p-1$
Answer: Have a look at $\Bbb Z_p^\times$, the multiplicative group of the field $\Bbb Z_p$; it is clearly of order $p - 1$:  $\vert \Bbb Z_p^\times \vert = p - 1$; thus, any $c \in \Bbb Z_p^\times$ satisfies
$c^{p - 1} = 1; \tag 1$
thus it also obeys
$c^p = c^{p - 1} c = 1 c = c, \tag 2$
which also obviously binds when $c = 0$.  Thus every $c \in \Bbb F_p = \Bbb F_p^\times \cup \{ 0 \}$ satisfies (2), which is precisely the same as affirming that
$f(x) = x^p \tag 2$
and
$g(x) = x \tag 3$
agree on every element of $\Bbb Z_p$.
Answer: A simple proof can be found here: Fermat's Little Theorem


Custom file uploads seem impossible from a POST request?

Question: Why are $_Files and/or HTTPFoundation\FileBag invalid in custom module Controller?
I have noticed a very strange issue with accessing an individual file from the  $_FILES object or the Symfony derived \Symfony\Component\HTTPFoundation\FileBag from within a custom module controller.
The exact location of the issue seems to be in Symfony\Component\HttpKernel\HttpKernel -> handleRaw
Specifically, the file exists prior dispatching the KernelEvent::REQUEST but it does not after.
// request
$event = new GetResponseEvent($this, $request, $type);

// THE FILE EXISTS HERE
// file_exists( $request->files->get(0)->getRealPath()) == TRUE

$this->dispatcher->dispatch(KernelEvents::REQUEST, $event);

// THE FILE DOES NOT EXISTS HERE
// file_exists( $request->files->get(0)->getRealPath()) == FALSE

Here is the entire method for reference also including some code to move the file, which works as expected as long as it is before the event dispatch.
  /**
 * Handles a request to convert it to a response.
 *
 * Exceptions are not caught.
 *
 * @param Request $request A Request instance
 * @param int     $type    The type of the request (one of HttpKernelInterface::MASTER_REQUEST or HttpKernelInterface::SUB_REQUEST)
 *
 * @return Response A Response instance
 *
 * @throws \LogicException       If one of the listener does not behave as expected
 * @throws NotFoundHttpException When controller cannot be found
 */
private function handleRaw(Request $request, $type = self::MASTER_REQUEST)
{
    $this->requestStack->push($request);

    $directory = \Drupal::service('file_system')->realpath(file_default_scheme() . "://");

    /* @var \Symfony\Component\HttpFoundation\File\UploadedFile $uploadedFile*/
    $uploadedFile = $request->files->get(0);

    if (is_uploaded_file($uploadedFile->getRealPath())){
      $success = $uploadedFile->move($directory);
    }

  // request
    $event = new GetResponseEvent($this, $request, $type);
    $this->dispatcher->dispatch(KernelEvents::REQUEST, $event);

    if ($event->hasResponse()) {
        return $this->filterResponse($event->getResponse(), $request, $type);
    }

    // load controller
    if (false === $controller = $this->resolver->getController($request)) {
        throw new NotFoundHttpException(sprintf('Unable to find the controller for path "%s". The route is wrongly configured.', $request->getPathInfo()));
    }

    $event = new FilterControllerEvent($this, $controller, $request, $type);
    $this->dispatcher->dispatch(KernelEvents::CONTROLLER, $event);
    $controller = $event->getController();

    // controller arguments
    $arguments = $this->argumentResolver->getArguments($request, $controller);

    $event = new FilterControllerArgumentsEvent($this, $controller, $arguments, $request, $type);
    $this->dispatcher->dispatch(KernelEvents::CONTROLLER_ARGUMENTS, $event);
    $controller = $event->getController();
    $arguments = $event->getArguments();

    // call controller
    $response = call_user_func_array($controller, $arguments);

    // view
    if (!$response instanceof Response) {
        $event = new GetResponseForControllerResultEvent($this, $request, $type, $response);
        $this->dispatcher->dispatch(KernelEvents::VIEW, $event);

        if ($event->hasResponse()) {
            $response = $event->getResponse();
        }

        if (!$response instanceof Response) {
            $msg = sprintf('The controller must return a response (%s given).', $this->varToString($response));

            // the user may have forgotten to return something
            if (null === $response) {
                $msg .= ' Did you forget to add a return statement somewhere in your controller?';
            }
            throw new \LogicException($msg);
        }
    }

    return $this->filterResponse($response, $request, $type);
}

I am trying to write a custom GraphQL module mutation plugin that handles file uploads and unless I can get access to the file object from the POST request, I don't think this will be possible.
Any thoughts are greatly appreciated.
I believe this may be a core issue or an upstream issue with Symfony or at the very least a side effect of the assumptions of how new requests are created.
https://www.drupal.org/project/drupal/issues/2934486#comment-12406531
Answer: Update: this has been addressed upstream here
https://github.com/symfony/psr-http-message-bridge/issues/39#issuecomment-558306949
There is an issue when a Symfony\Bridge\PsrHttpMessage\Factory\HttpFoundationFactory request is created from anywhere. If there are any POST $_FILES (File object) on that request, they are renamed and moved to the tmp directory with a new name. This new name is not propagated on the original request.
During this process of creating the new psr7 request, any temporary files are moved and renamed in the temporary folder.
The tmp_name values starts as something like: /private/var/tmp/phpEHH19a and then on line 102 of the HTTPFoundationFactory, the file is moved and renamed to something like:
symfony5a4e551619af73.65717941BYwCkd
This renaming flow happens like this
HttpFoundationFactory->createRequest() -> getFiles --> createUploadFiles and this line 102 of HTTPFoundationFactory
$psrUploadedFile->moveTo($temporaryPath);
causes the file to be renamed.
This type of request can happen in a number of places, but definitely in a decoupled app, especially during authentication. For example with the simple_oauth module when updating the tokens. This new request is never passed along and the original request still retains the original tmp_name file paths.
Once you finally get access to the request object in a module controller via
\Symfony\Component\HttpFoundation\Request
and check all the files via: $request->files->all();
The Symfony\Component\HttpFoundation\File\UploadedFile Object contains the original file path.
This is a major problem that needs to be overcome in order to handle multipart/form-data with decoupled javascript file uploads. I noticed the issue when working on a custom graphql module to handle file uploads.
Here is the path you can use to trace through the issue:
The SimpleOauthAuthenticationProvider calls validateAuthenticatedRequest($request) which creates a new psr7 request (see above). During this process, the files are moved and the new request does not include the updated request files object.
I think this is a bug upstream with Symfony or it's a bug with how files are handled with sub requests or maybe it's a bug with how simple_oauth handles getting the new tokens. I don't think it's a simple_oauth bug though because it's just making a request and there is a side effect of the temporary files being moved.
See the screenshot below. The $request variable is created and the tokens are reset. As a result of the new request, the temporary files are moved and renamed. The screenshot shows the new spl filepaths but this request dies in this method and the files do not get updated on the original request object. If you look at the watched variables on the right you can see that the file at the original tmp_name path no longer exists.


Why isn't cocoapods working on Apple Silicon?

Question: On the new m1 mba, I downloaded cocoapods but none of the commands are working, they just crash. I've seen others with the same issue and some said that ticking the 'open with Rosetta' box worked for them but I tried that and it does the exact same thing. Has anyone found a solution for this?
Answer: I'm a newer developer, but this worked for me on Mac mini M1.

As you say, Right click the terminal app -> Get info -> "tick open with Rosetta"
Terminal: sudo gem install cocoapods
sudo gem install ffi

pod binary should now work fine.
Comment: shouldn't that be the other way around? install ffi in the rosetta terminal and then go back to normal terminal and install cocoapods? thats what worked for me.

also a good thing to note is you need the xcode command line tools to do the above


HTTP POST + ANDROID not giving response

Question: I am beginner for Android. I want to send the data to my PHP page. but here i m trying to toast that post values. but there is no response. Plz help me.
My Code is:
public class send_msgActivity extends Activity{

//static final String KEY_NAME = "name";
@Override
public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.send_msg);

    Button btn_ok = (Button) findViewById(R.id.btn_ok);

    btn_ok.setOnClickListener(new View.OnClickListener() {
        @Override
        public void onClick(View arg0) {
            EditText editText1 = (EditText)findViewById(R.id.sender);
            String S_name = editText1.getText().toString();
            EditText editText2 = (EditText)findViewById(R.id.reciever);
            String S_email = editText2.getText().toString();
        postData(S_name,S_email);
        }
    });
};

public void postData(String name,String email) {
    // Create a new HttpClient and Post Header
    HttpClient httpclient = new DefaultHttpClient();
    HttpPost httppost = new HttpPost("http://www.URL.com/yourpage.php");

    try {
        // Add your data
        List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(2);
        nameValuePairs.add(new BasicNameValuePair("Fname", name));
        nameValuePairs.add(new BasicNameValuePair("Femail", email));
        httppost.setEntity(new UrlEncodedFormEntity(nameValuePairs));

        // Execute HTTP Post Request
        //HttpResponse response = httpclient.execute(httppost, new BasicResponseHandler());
        HttpResponse response = httpclient.execute(httppost);
        String reverseString = response.toString();
        Toast.makeText(this, "response" + reverseString, Toast.LENGTH_LONG).show();

    } catch (ClientProtocolException e) {
        // TODO Auto-generated catch block
    } catch (IOException e) {
        // TODO Auto-generated catch block
    }
}
}

Comment: You never called your method. And you should only call it in an AsyncTask.
Comment: VM is right. The only thing you're doing right now is setting the texts of the edittext fields to a string. You still have to use the postData method. Until Android 2.3 you were able to just add this method in your on click listener but this is bad practise. Use AsyncTask instead. There are plenty of tutorials online for this.
Comment: Sry for my mistake. but i hv called that methods,and forget to written here. now my actual code is this.
Answer: Use this example how to code HTTPpost method
package com.example.login;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;

import org.apache.http.HttpResponse;a
import org.apache.http.NameValuePair;
import org.apache.http.client.HttpClient;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.params.BasicHttpParams;
import org.apache.http.params.HttpConnectionParams;
import org.apache.http.params.HttpParams;
import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;

import android.os.AsyncTask;
import android.os.Bundle;
import android.app.Activity;
import android.app.AlertDialog;
import android.app.ProgressDialog;
import android.content.Context;
import android.content.DialogInterface;
import android.content.Intent;
import android.util.Log;
import android.view.Menu;
import android.view.MenuInflater;
import android.view.MenuItem;
import android.view.View;
import android.widget.AdapterView;
import android.widget.AdapterView.OnItemClickListener;
import android.widget.ArrayAdapter;
import android.widget.Button;
import android.widget.EditText;
import android.widget.ImageView;
import android.widget.ListView;
import android.widget.Spinner;
import android.widget.TextView;
import android.widget.Toast;

public class pdf extends Activity
{
    public boolean connect=false,logged=false;
    public String db_select;
    ListView l1;
    String mPwd,UName1="Success",UName,ret;
    public Iterator<String> itr;
    private String SERVICE_URL = "http://184.108.40.206:8080/Agero/person/pdf";
    private String SERVICE_URL1 = "http://126.96.36.199:8080/Agero/person/url";
    //private final String SERVICE_URL = "http://10.1.1.138:8080/Agero/person/pdf";
    //private final String SERVICE_URL1 = "http://10.1.1.138:8080/Agero/person/url";
    private final String TAG = "Course";
    ArrayList<String> todoItems;
    Boolean isInternetPresent = false;
    ConnectionDetector cd;
    ArrayAdapter<String> aa;
    public List<String> list1=new ArrayList<String>();
    @Override
    protected void onCreate(Bundle savedInstanceState)
    {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.pdf);
         l1 = (ListView)findViewById(R.id.list);
         todoItems = new ArrayList<String>();
            aa = new ArrayAdapter<String>(this,R.layout.list_row,R.id.title,todoItems);
            l1.setAdapter(aa);
            todoItems.clear();
            cd = new ConnectionDetector(getApplicationContext());
            isInternetPresent = cd.isConnectingToInternet();
            if(isInternetPresent)
            {
            try
            {
                validat_user();
                //display("hi");

            }
            catch(Exception e)
            {
                display("Network error.\nPlease check with your network settings.");
            }
            }
            else
            {
                display("No Internet Connection..");
            }

            l1.setOnItemClickListener(new OnItemClickListener() {
                public void onItemClick(AdapterView<?> parent, View view,
                    int position, long id) {
                 // TODO Auto-generated method stub
                 String name=(String)parent.getItemAtPosition(position);

                 /*Toast.makeText(getBaseContext(), name, Toast.LENGTH_LONG).show();
                  Intent i = new Intent(getBaseContext(),Webview.class);
                  i.putExtra("USERNAME", name);
                  startActivity(i);*/
                 cd = new ConnectionDetector(getApplicationContext());
                    isInternetPresent = cd.isConnectingToInternet();
                 if(isInternetPresent)
                    {
                 try
                    {
                        validat_user1(name);

                    }
                    catch(Exception e)
                    {
                        display("Network error.\nPlease check with your network settings.");

                    }

                    }
                 else
                    {
                        display("No Internet Connection..");
                    }
                }
            });

    }
    public void display(String msg)
    {
        Toast.makeText(pdf.this, msg, Toast.LENGTH_LONG).show();
    }
    private void validat_user()
    {

        WebServiceTask wst = new WebServiceTask(WebServiceTask.POST_TASK, this, "");

       // wst.addNameValuePair("State", stg1);
       // wst.addNameValuePair("Emp_PWD", stg2);
       // db_select=stg1;
        //display("I am");
        wst.execute(new String[] { SERVICE_URL });
        //display(SERVICE_URL);

    }
    private void validat_user1(String stg1)
    {
        db_select=stg1;
        WebServiceTask wst = new WebServiceTask(WebServiceTask.POST_TASK, this, "Loading...");

        wst.addNameValuePair1("PDF_NAME", stg1);

        wst.execute(new String[] { SERVICE_URL1 });

    }
    @SuppressWarnings("deprecation")
    public void no_net()
    {
        display( "No Network Connection");
        final AlertDialog alertDialog = new AlertDialog.Builder(pdf.this).create();
        alertDialog.setTitle("No Internet Connection");
        alertDialog.setMessage("You don't have internet connection.\nElse please check the Internet Connection Settings.");
        //alertDialog.setIcon(R.drawable.error_info);
        alertDialog.setCancelable(false);
        alertDialog.setButton("Close", new DialogInterface.OnClickListener()
        {
            public void onClick(DialogInterface dialog, int which)
            {
                alertDialog.cancel();
                pdf.this.finish();
                System.exit(0);
            }
        });
        alertDialog.setButton2("Use Local DataBase", new DialogInterface.OnClickListener()
        {
            public void onClick(DialogInterface dialog, int which)
            {
                display( "Accessing local DataBase.....");
                alertDialog.cancel();
            }
        });
        alertDialog.show();
    }

    private class WebServiceTask extends AsyncTask<String, Integer, String> {

        public static final int POST_TASK = 1;

        private static final String TAG = "WebServiceTask";

        // connection timeout, in milliseconds (waiting to connect)
        private static final int CONN_TIMEOUT = 3000;

        // socket timeout, in milliseconds (waiting for data)
        private static final int SOCKET_TIMEOUT = 5000;

        private int taskType = POST_TASK;
        private Context mContext = null;
        private String processMessage = "Processing...";

        private ArrayList<NameValuePair> params = new ArrayList<NameValuePair>();

        private ProgressDialog pDlg = null;

        public WebServiceTask(int taskType, Context mContext, String processMessage) {

            this.taskType = taskType;
            this.mContext = mContext;
            this.processMessage = processMessage;
        }

        public void addNameValuePair1(String name, String value) {

            params.add(new BasicNameValuePair(name, value));
        }
        @SuppressWarnings("deprecation")
        private void showProgressDialog() {

            pDlg = new ProgressDialog(mContext);
            pDlg.setMessage(processMessage);
            pDlg.setProgressDrawable(mContext.getWallpaper());
            pDlg.setProgressStyle(ProgressDialog.STYLE_SPINNER);
            pDlg.setCancelable(false);
            pDlg.show();

        }

        @Override
        protected void onPreExecute() {

            showProgressDialog();

        }

        protected String doInBackground(String... urls) {

            String url = urls[0];
            String result = "";

            HttpResponse response = doResponse(url);

            if (response == null) {
                return result;
            } else {

                try {

                    result = inputStreamToString(response.getEntity().getContent());

                } catch (IllegalStateException e) {
                    Log.e(TAG, e.getLocalizedMessage(), e);

                } catch (IOException e) {
                    Log.e(TAG, e.getLocalizedMessage(), e);
                }

            }

            return result;
        }

        @Override
        protected void onPostExecute(String response) {

            handleResponse(response);
            pDlg.dismiss();

        }

        // Establish connection and socket (data retrieval) timeouts
        private HttpParams getHttpParams() {

            HttpParams htpp = new BasicHttpParams();

            HttpConnectionParams.setConnectionTimeout(htpp, CONN_TIMEOUT);
            HttpConnectionParams.setSoTimeout(htpp, SOCKET_TIMEOUT);

            return htpp;
        }

        private HttpResponse doResponse(String url) {

            // Use our connection and data timeouts as parameters for our
            // DefaultHttpClient
            HttpClient httpclient = new DefaultHttpClient(getHttpParams());

            HttpResponse response = null;

            try {
                switch (taskType) {

                case POST_TASK:
                    HttpPost httppost = new HttpPost(url);
                    // Add parameters
                    httppost.setEntity(new UrlEncodedFormEntity(params));

                    response = httpclient.execute(httppost);
                    break;
                }
            } catch (Exception e) {
                display("Remote DataBase can not be connected.\nPlease check network connection.");

                Log.e(TAG, e.getLocalizedMessage(), e);
                return null;

            }

            return response;
        }

        private String inputStreamToString(InputStream is) {

            String line = "";
            StringBuilder total = new StringBuilder();

            // Wrap a BufferedReader around the InputStream
            BufferedReader rd = new BufferedReader(new InputStreamReader(is));

            try {
                // Read response until the end
                while ((line = rd.readLine()) != null) {
                    total.append(line);
                }
            } catch (IOException e) {
                Log.e(TAG, e.getLocalizedMessage(), e);
            }

            // Return full string
            return total.toString();
        }

    }
    public void handleResponse(String response)
    {    //display("JSON responce is : "+response);
        if(!response.equals(""))
        {
       try {

            JSONObject jso = new JSONObject(response);

                  int UName = jso.getInt("status1");

                  if(UName==1)
                  {
                      String status = jso.getString("status");
                      ret=status.substring(13,status.length()-2);
                      todoItems.add(0, status);
                    aa.notifyDataSetChanged();
                  }
                  else if(UName==-1)
                  {
                      String status = jso.getString("status");
                      //display(status);
                        Intent intObj=new Intent(pdf.this,Webview.class);
                         intObj.putExtra("USERNAME", status);
                        startActivity(intObj);
                  }
                  else
                  {
//                     int count=Integer.parseInt(UName);
//                     display("Number of Projects have been handling in AFL right now: "+count);
                    list1=new ArrayList<String>();

                    JSONArray array=jso.getJSONArray("reps1");
                    for(int i=0;i<array.length();i++)
                    {
                        list1.add(array.getJSONObject(i).getString("pdfName"));

                    }
                    itr=list1.iterator();
                    while(itr.hasNext())
                    {
                         //str1=itr.next()+"\n";
                        todoItems.add(0, itr.next().toString());
                        aa.notifyDataSetChanged();
                    }

                    //tv1.setText(str1);

                  }
        } catch (Exception e) {
            Log.e(TAG, e.getLocalizedMessage(), e);
            return;
        }
        }
        else
        {
            display("unable to reach the server");
        }

    }

    @Override
    public boolean onCreateOptionsMenu(Menu menu) {
        MenuInflater inflater = getMenuInflater();
        inflater.inflate(R.menu.activity_main, menu);
        return true;
    }

    @Override
    public boolean onOptionsItemSelected(MenuItem item) {
        switch (item.getItemId()) {
        case R.id.home:
            Intent intObj=new Intent(pdf.this, MainActivity.class);
            intObj.putExtra("finish", true); // if you are checking for this in your other Activities
            intObj.setFlags(Intent.FLAG_ACTIVITY_CLEAR_TOP |
                            Intent.FLAG_ACTIVITY_CLEAR_TASK |
                            Intent.FLAG_ACTIVITY_NEW_TASK);
            startActivity(intObj);
            //pdf.this.finish();
            finish();
            return (true);

        }
        return super.onOptionsItemSelected(item);
    }

}

HTTP POST + ANDROID not giving response


How to validate XQuery source file for XSD schema

Question: I mean without input XML file.
I'm using Saxon-EE 9.2.
Comment: What do you mean "validate XQuery file for XSD"? XML file can be validated with XSD ... Be more specific, pls.
Answer: if you mean to validate the Xquery source file then, the only route I know of is to first convert it to XqueryX using xq2xqx.xsl and then use a xsd schema based on that
Answer: An XQuery source file isn't an XML document, so can't be validated with an XML schema. If you really need to, you can use the xq2xqx library to convert XQuery source files into XQueryX documents:
http://monet.nag.co.uk/xq2xml/
The code there needs some tidying up, the XQuery parser linked at
http://www.w3.org/2005/qt-applets/xgrammar.zip
and the Saxon jar - the free one here should work:
http://saxon.sourceforge.net/#F9.4HE
You should end up with a command line something like:
java -cp "saxon9.jar;xquery.jar;trans2.jar" net.sf.saxon.Transform -it:main -o:"xq2xqx.log" -xsl:"xq2xqx.xsl" dump="no$2" xq=test.xquery
which will generate test.xqueryx, and you can then validate the document against the official w3.org schema:
http://www.w3.org/2005/XQueryX/xqueryx.xsd


What are bifunctors?

Question: I'm relatively new to Haskell and having trouble understanding the utility of bifunctors. I think I understand them in theory: say for example if I wanted to map across a type that abstracts multiple concrete types, such as Either or Maybe, I'd need to encapsulate them in a bifunctor. But on the one hand, those examples seems particularly contrived, and on the other it does seem like you could achieve that same functionality simply through composition.
As an example, I came across this code in The Essence of the Iterator Pattern by Jeremy Gibbons and Bruno C. d. S. Oliveira:
import Data.Bifunctor

data Fix s a = In {out::s a (Fix s a) }

map' :: Bifunctor s => (a -> b) -> Fix s a -> Fix s b
map' f = In . bimap f (map' f) . out

fold' :: Bifunctor s => (s a b -> b) -> Fix s a -> b
fold' f = f . bimap id (fold' f) . out

unfold' :: Bifunctor s => (b -> s a b) -> b -> Fix s a
unfold' f = In . bimap id (unfold' f) . f

I understand the point is to compose mapping and folding functions to create an iterator pattern and this is achieved by defining a data constructor that requires two parameters. But in practice I don't understand how this is any different then using a regular functor and composing the functions with fmap instead of bimap. I think I clearly must be missing something, both with this example and, likely, in general.
Answer: I think you're overthinking it slightly. A bifunctor is just like a two-parameter functor. Gibbons and Oliveira's idea is just one application of bifunctors, just like the standard zoo of recursion schemes is just one application of functors.
class Bifunctor f where
    bimap :: (a -> c) -> (b -> d) -> f a b -> f c d

Bifunctors have a kind of * -> * -> * and both parameters can be covariantly mapped over. Compare this to regular Functors, which have only one parameter (f :: * -> *) which can be covariantly mapped over.
For example, think about Either's usual Functor instance. It only allows you to fmap over the second type parameter - Right values get mapped, Left values stay as they are.
instance Functor (Either a) where
    fmap f (Left x) = Left x
    fmap f (Right y) = Right (f y)

However, its Bifunctor instance allows you to map both halves of the sum.
instance Bifunctor Either where
    bimap f g (Left x) = Left (f x)
    bimap f g (Right y) = Right (g y)

Likewise for tuples: (,)'s Functor instance allows you to map only the second component, but Bifunctor allows you to map both parts.
instance Functor ((,) a) where
    fmap f (x, y) = (x, f y)

instance Bifunctor (,) where
    bimap f g (x, y) = (f x, g y)

Note that Maybe, which you mentioned, doesn't fit into the framework of bifunctors because it has only one parameter.

On the question of Fix, the fixed point of a bifunctor allows you to characterise recursive types which have a functorial type parameter, such as most container-like structures. Let's use lists as an example.
newtype Fix f = Fix { unFix :: f (Fix f) }

data ListF a r = Nil_ | Cons_ a r deriving Functor
type List a = Fix (ListF a)

Using the standard functorial Fix, as I have above, there's no generic derivation of an instance of Functor for List, because Fix doesn't know anything about List's a parameter. That is, I can't write anything like instance Something f => Functor (Fix f) because Fix has the wrong kind. I have to hand-crank a map for lists, perhaps using cata:
map :: (a -> b) -> List a -> List b
map f = cata phi
    where phi Nil_ = Fix Nil_
          phi Cons_ x r = Fix $ Cons_ (f x) r

The bifunctorial version of Fix does permit an instance of Functor. Fix uses one of the bifunctor's parameters to plug in the recursive occurrence of Fix f a and the other one to stand in for the resultant datatype's functorial parameter.
newtype Fix f a = Fix { unFix :: f a (Fix f a) }

instance Bifunctor f => Functor (Fix f) where
    fmap f = Fix . bimap f (fmap f) . unFix

So we can write:
deriveBifunctor ''ListF

type List = Fix ListF

and get the Functor instance for free:
map :: (a -> b) -> List a -> List b
map = fmap

Of course, if you want to work generically with recursive structures with more than one parameter then you need to generalise to tri-functors, quad-functors, etc... This is clearly not sustainable, and plenty of work (in more advanced programming languages) has been put into designing more flexible systems for characterising types.
Comment: @dfeuer, one example of such work (the only I'm aware of) is [Generic Programming with Indexed Functors](http://dreixel.net/research/pdf/gpif.pdf). If you sacrifice first-orderness they have in the paper, you'll get [something](https://github.com/effectfully/random-stuff/blob/master/Desc/InternalMuSet.agda) quite powerful, but it's not ideal still as you can't e.g. make an index depend on a parameter or define several data types mutually such that they have distinct parameter telescopes.
Comment: May I ask what sort of work in what sorts of programming languages? Can you point to something accessible? I've been thinking it might be possible to define something like `class Covariant (n :: Nat) p` expressing that `p` is covariant in its `nth` parameter, but I'm not quite sure what that would look like.
Comment: @dfeuer PS, thinking aloud here, but I think a "Covariant-in-parameter-n" class wouldn't be the right way to go. Instead make a more flexible `Functor` which goes between arbitrary categories: `class (Category c1, Category c2) => Functor c1 c2 f | f -> c1, f -> c2 where fmap :: c1 a b -> c2 (f a) (f b)`. Then a bifunctor (after uncurrying) is just a functor from the product category.
Comment: Thanks for the clear explanation! I was thinking of bifunctors as superfluous syntactic sugar, but your example of overloading map to take two parameters demonstrates how they're actually much simpler semantically as well. I'm also intrigued by what you were referring to in your comment at the end, though. Doesn't OCaml have polymorphic functors?
Comment: @dfeuer Oh, I was just making reference to the constellation of "universe"-style datatype descriptions that have been developed in DT languages.

What are bifunctors?


Lilypond \easyHeadsOn: How to increase font size and make them bold inside a note head?

Question: I am trying to annotate each note using tonic pitch names, so that it will be easier to sing these notes.
Lilypond has this very nice feature \easyHeadsOn:
https://lilypond.org/doc/v2.21/Documentation/notation/note-heads
I increased the staff size using
<code>\relative c' {
  \easyHeadsOn  % show numbered music notation (Galin-Paris-Chevé system) in the note heads
  \magnifyStaff #(magstep 1.2)
  \key g \major
  \numericTimeSignature
  \time 4/4
  ...
}
</code>
Currently the rendering looks like this:

I'm happy with the size of the note head, but the numbers therein are not easy to read. I'd like the increase the font size and make the font bold inside the note head. But do not want to increase the global font size, since I want to keep all other fingering symbols in the sheet look normal, not too big.
Any help will be highly appreciated!
Answer: As the docs say this is only really usable with really large staff sizes. This font size of the text is linked to the font size of the note head, so increasing the size of the heads also increases the size of the numbers. Also the numbers are in fact already boldface.
It would not be very hard to rewrite this function to use bigger letters, but the problem kind of lies in the design. You will probably not be very happy trying to get this into a readable form. Rather I'd suggest to do things differently like this one:
<code>$(define notenames
   #("C" "D" "E" "F" "G" "A" "B"))

#(define (notename_annotation_engraver context)
   (make-engraver
    (acknowledgers
     ((note-head-interface engraver grob source-engraver)
      (let* ((cause (ly:grob-property grob 'cause))
             (pitch (ly:event-property cause 'pitch))
             (nn (ly:pitch-notename pitch))
             (mup (vector-ref notenames nn))
             (tgrob (ly:engraver-make-grob engraver 'TextScript cause)))
        (ly:grob-set-property! tgrob 'text mup)
        (ly:grob-set-parent! tgrob Y grob)
        (ly:grob-set-property! tgrob 'whiteout 1)
        (ly:grob-set-property! tgrob 'Y-offset 0.7)
        (ly:grob-set-property! tgrob 'parent-alignment-X -0.4)
        (ly:grob-set-property! tgrob 'self-alignment-X CENTER)
        (ly:grob-set-property! tgrob 'parent-alignment-Y CENTER)
        (ly:grob-set-property! tgrob 'self-alignment-X CENTER)
        (ly:grob-set-property! tgrob 'font-size -2)
        (ly:grob-set-property! tgrob 'outside-staff-priority '()))))))

\layout {
  \context {
    \Voice
    \consists #notename_annotation_engraver
  }
}

\relative c' {
  \key g \major
  \numericTimeSignature
  \time 4/4
  c d e fis g a b c
}
</code>
UPDATE: Similarly you can have the names go up or down like this
<code>$(define notenames
   #("C" "D" "E" "F" "G" "A" "B"))

#(define (notename_annotation_engraver context)
   (make-engraver
    (acknowledgers
     ((note-head-interface engraver grob source-engraver)
      (let* ((cause (ly:grob-property grob 'cause))
             (pitch (ly:event-property cause 'pitch))
             (nn (ly:pitch-notename pitch))
             (mup (vector-ref notenames nn))
             (tgrob (ly:engraver-make-grob engraver 'TextScript cause)))
        (ly:grob-set-property! tgrob 'text mup)
        (ly:grob-set-property! tgrob 'direction UP)
        (ly:grob-set-property! tgrob 'font-size -2)
        (ly:grob-set-property! tgrob 'staff-padding 3)
        (ly:grob-set-property! tgrob 'outside-staff-priority 10)
        )))))

\layout {
  \context {
    \Voice
    \consists #notename_annotation_engraver
  }
}

\relative c' {
  \key g \major
  \numericTimeSignature
  \time 4/4
  \tempo "a tempo"
  b4-1--\mp^\markup\box"*---" d8.-3( e16-1) g8.-3( a16-4) e16-2( g-4 d8-1) |
}
</code>
or maybe you might want to use something like this:
<code>$(define notenames
   #("C" "D" "E" "F" "G" "A" "B"))

#(define (note-head::note-name-stencil grob)
   (let* ((cause (ly:grob-property grob 'cause))
          (pitch (ly:event-property cause 'pitch))
          (nn (ly:pitch-notename pitch))
          (durlog (ly:duration-log (ly:event-property cause 'duration)))
          (mupt (markup #:whiteout #:fontsize -3 #:bold
                       #:sans (vector-ref notenames nn)))
          (mup (if (< durlog 0)
                   (markup #:box mupt)
                   (if (>= durlog 2)
                       mupt
                       (markup #:circle mupt))))
          (stc (grob-interpret-markup grob mup)))
     (ly:stencil-aligned-to stc Y CENTER)))

\layout {
  \context {
    \Voice
    \override NoteHead.stencil = #note-head::note-name-stencil
  }
}

\relative c' {
  \key g \major
  \numericTimeSignature
  \time 4/4
  \tempo "a tempo"
  b4-1--\mp^\markup\box"*---" d8.-3( e16-1) g8.-3( a16-4) e16-2( g-4 d8-1) |
  c16 d e fis g8 a b c d4 | e2 f | g1 | a\breve
}
</code>
Comment: Now this is hardcore Lisp....my respect! I tested it in my code. However, the note I am engraving will have a lot of fingering symbols around the note heads, your solution is a good idea, but I cannot really apply it.
Comment: @NicoleNaumann Nothing hardcore, Lisp is just hard to parse visually. I've added a few more suggestions. Note that the last one does not yet handle note values longer than breve (longa, maxima) properly.


Views group different number of divs of a view by wrapping them

Question: I have a views block that is displayed in the content section of a node. What I need is wrapping the rows of the views block by divs. I have a list of 10 divs like so:

Some Text here

Some Text here

Some Text here

Some Text here

Some Text here

Some Text here

Some Text here

Some Text here

Some Text here

Some Text here

Now I want to wrap the first 3 divs, div 4 to 8 and div 9 and 10 in a separate wrapper div like so.

Some Text here

Some Text here

Some Text here

Some Text here

Some Text here

Some Text here

Some Text here

Some Text here

Some Text here

Some Text here

How would I edit my views.tpl.php? Thanks!
Comment: Stack Exchange sites are not forums: Answers are just for answering the question, not to ask further information to who answered your question. For that, there are comments, which you can use for answers given to your questions.
Comment: This thread got too rich in comments and misunderstandings, mostly due to beginner's level of the asker, also in terms of php itself. I still think the final answer might help someone using Views. My vote would be to delete all the ramblings, then I paste the final answer that worked for criscom and then criscom accepts it.
Comment: Ok. See bottom for what worked for me.
Answer: Here is what worked for me. Mind that I only got it to work in an unformatted list format in views.
<?php
  $currIx = 1; // This will be our incremental index variable. Let's make it 1-based to maintin consistency with your example.
  $currWrapperIx = 1; // Wrapper counter
  $wasOpen = FALSE; // This variable will be used later as a flag to watch for properly closing wrapper divs.
?>

<?php if (!empty($title)): ?>
  <p class="<?php print $title ?>"><?php print $title; ?>

<?php endif; ?>
<?php foreach ($rows as $id => $row): ?>
  <?php switch ($currIx){
    // List the row indices that are supposed to begin a new wrapper.
    case 1:
    case 27:
      if($wasOpen == TRUE){ // Close current wrapper div before opening a new one.
        print "
\n";
        $wasOpen = FALSE;
        $currWrapperIx ++; // Increment the index for next wrapper's class.
      }
      // Open a new wrapper div
      print "
\n";
      $wasOpen = TRUE;
      break;

    default:
      // For other row indices do nothing.
    }
  ?>

  <div class="<?php print $id ?>"><?php print $row; ?>

  <?php $currIx ++; ?>
<?php endforeach; ?>

<?php
    // Close the last wrapper.
    if($wasOpen == TRUE){
      print "
\n";
      $wasOpen = FALSE;
    }
?>

Kudos go out to @Artur. Great support. Many thanks.
Comment: paste that into your views-view-unformatted--default.tpl.php and put it into your template folder in your theme.
Answer: Sure thing criscom. Below is a very rudimentary piece of code.
<?php
  $rows = array(
    1 => "row1",
    2 => "row2",
    3 => "row3",
    4 => "row4",
    5 => "row5",
    6 => "row6",
    7 => "row7",
    8 => "row8",
    9 => "row9",
    10 => "row10",
  );
?>

<?php
  $currIx = 1; // This will be our incremental index variable. Let's make it 1-based to maintin consistency with your example.
  $currWrapperIx = 1; // Wrapper counter
  $wasOpen = FALSE; // This variable will be used later as a flag to watch for properly closing wrapper divs.
?>

<?php foreach ($rows as $id => $row): ?>
  <?php switch ($currIx){
    // List the row indices that are supposed to begin a new wrapper.
    case 1:
    case 4:
    case 9:
      if($wasOpen == TRUE){ // Close current wrapper div before opening a new one.
        print "
\n";
        $wasOpen = FALSE;
        $currWrapperIx ++; // Increment the index for next wrapper's class.
      }
      // Open a new wrapper div
      print "
\n";
      $wasOpen = TRUE;
      break;

    default:
      // For other row indices do nothing.
    }
  ?>

<?php print $row; ?>

  <?php $currIx ++; ?>
<?php endforeach; ?>

<?php
    // Close the last wrapper.
    if($wasOpen == TRUE){
      print "
\n";
      $wasOpen = FALSE;
    }
?>

Sorry, I did not bother to make it pretty-print, but the source of the generated page has the div structure you asked for. The code is abstract and has dummy rows array, so you can test it by simply pasting it in an empty PHP file and trigger it outside Drupal.
Anyway you should get the idea.
Update
You used the PHP code from view-view.tpl.php. This is a template file with a broader picture on a view's output structure. In this file you don't have any control over the traversal (the foreach loop). It just says print $rows and that's it, meaning "print out the loop, whatever it looks like." What you need is to specify how the loop is generated - that is: open your view, open the Theming information and pick one of the tpl in the Style group (where tpl's contain the word unformatted). There you'll be able to mess with the loop. Remove my code from the former file and move it to that file. Rescan templates.
Comment: Thanks a lot for that code, Artur. I have implemented above code into my view-view-list--block-1.tpl.php.
I suspect that the tpl.php file I am using is the wrong one since the divs are inserted too low down in the html. I get something like this:

Some text...
But thanks a lot. This gives me something to play with. I just have to get in a little higher.
Comment: Quite a chestnut to get it to work in my views, though it works great when i run it outside drupal.
Comment: when I add the code to views-view--block-1.tpl.php I get an error message: Warning: Invalid argument supplied for foreach() in include() (line 64 of views-view--block-1.tpl.php
line 64 is:  $row): ?>
Comment: I'm sorry this is causing you so much trouble, it was meant to be of some help. I'm not sure why it wouldn't work. The code is based on the generic Style view tpl (/views/theme/views-view-unformatted.tpl.php). I haven't tested it within Drupal, but if you comment out the initial $rows array, views should use the real rows. Feel free to PM me or paste the entire tpl code you're using.
Comment: Hey nothing to be sorry about. Your code taught me a lot to say the least. Very much appreciate your help on this.
Comment: how could I send you a PM?
Comment: Oops, must have confused the website functionality with drupal.org, but anyway I thought it'd be more acceptable to continue the thread here so that other users can follow and learn as well.
Comment: With jquery no problem to do this but, you will agree, not very elegant. here is what I want to achieve: total 49 rows first 21 wrapped by div, then rows 22 - 37 wrapped by div, then row 38 - 49 wrapped by div. http://www.evernote.com/shard/s4/sh/917a0718-af94-4a90-85af-4ff4094ce9ba/424c34cf1c87465d6de4b711e16f2950


Matrix $C \in M_n(\mathbb{R})$ $\lambda_1=\lambda_2=0$ rank

Question: If a have a matrix $C \in M_n(\mathbb{R})$ with $C^2=O_n$ and $n=2k+1$ that has at least to eigenvalues equal to $0$. Can I say from this that the $rank C\leq \frac{n-1}{2}?$
Comment: You need only assume $C^2 = O$, then in fact all eigenvalues are $0$, since $C$ is [nilpotent](https://en.wikipedia.org/wiki/Nilpotent_matrix#Characterization).
Comment: Since $C^2=0$, then $\operatorname{Im}(C)\subseteq\operatorname{Ker}(C)$. Then $n=\operatorname{dim}\operatorname{Ker}(C)+\operatorname{dim}\operatorname{Im}(C)\geq 2\operatorname{dim}\operatorname{Im}(C)$. Since $n$ is odd and $\operatorname{dim}\operatorname{Im}(C)$ a natural number, then $\frac{n-1}{2}\geq \operatorname{dim}\operatorname{Im}(C)$.
Answer: $\newcommand{\rank}{\operatorname{rank}}\newcommand{\nullity}{\operatorname{nullity}}\newcommand{\c}{\mathbf{c}}\newcommand{\0}{\mathbf{0}}$Yes. Hints: You want to show that $\rank(C)\le k$. Let the columns of $C$ be $\c_1,\ldots,\c_n$. Then since $C^2 = O$, we have $C\c_j = \0$ for all $j=1,\ldots, n$. Now suppose by way of contradiction that $\rank(C) > k$. Then since (by the rank-nullity theorem) $$\nullity(C) = 2k+1-\rank(C),$$ we have $$\nullity(C) < 2k+1-k = k+1 \Rightarrow \nullity(C) \le k.$$ But the columns of $C$ are all from the null space of $C$ and $\rank(C) > k$. In view of the above, can you explain why this is impossible?


Using CLT for hypothesis testing

Question: I have two not normal distributions (~1k samples in each), looks like exponential:

So, I need to check its means, that's why I have following questions:

The easiest way to do it - is to use Mann-Whitney criteria?
How to check, is it really exponential distribution? Kolmogorov-Smirnov? But it suits only for standard distr?
And is it possible to use CLT for comparison of means: to generate a lot of means using bootstrap from two distr. Accordingly to CLT, means' distributions have to be normal and it becomes possible to use Students criterias. (Presuming, that it is a rather silly question...)
Comment: the CLT is the easiest way to do it, the assumption is that the mean of 1000 samples is approximately normally distributed. then you calculate the standard error etc, just using the variance of the individual samples. ( so you only need to bootstrap if you want to double check that over 1000 samples, the distribution is normal, not to calculate the standard error)
Answer: Are you sure you want to compare the means?

Mann-Whitney is the non-parametric version of the t-test when the assumptions of the t-test do not hold, you should check the assumptions before you needlessly underpower your test with the MW test.
KS test can be used to compare similarity of two distributions, however be advised that the test is very sensitive with big sample sizes.
You're not going to use the CLT, but the CLT allows you to compare the two means using confidence intervals, so testing for differnce.

Using CLT for hypothesis testing


how to create admin notification mail for newsletter subscription in magento

Question: I am trying to create a module for magento 1.7 which will send a notification mail to admin whenever a new user subscribes to newsletter. I have succeeded so far in getting the mail sent. however, my code is not taking the getEmail and getId values to be displayed in the mail sent to admin. if anyone can throw some light on where i am going wrong, it would be great. here is the code:
app/code/local/Notify/Biju/etc/config.xml
    <?xml version="1.0" encoding="utf-8"?>
<config>
    <modules>
        <Notify_Biju>
            <version>0.1.0</version>
        </Notify_Biju>
    </modules>
    <global>
        <models>
            <Notify_Biju>
                <class>Notify_Biju_Model</class>
            </Notify_Biju>
        </models>
        <template>
            <email>
                <newsletter_alert_template translate="label" module="n">
                    Newsletter Alert to Admin
                    <file>newsletter_subscription_notify.html</file>
                    <type>html</type>
                </newsletter_alert_template>
            </email>
        </template>
    </global>
    <frontend>
        <events>
            <newsletter_subscriber_save_after>
                <observers>
                    <Notify_Biju_Model_Observer>
                        <type>singleton</type>
                        <class>Notify_Biju_Model_Observer</class>
                        <method>newsletteralert</method>
                    </Notify_Biju_Model_Observer>
                </observers>
            </newsletter_subscriber_save_after>
        </events>
    </frontend>
</config>

app/code/local/Notify/Biju/Model/Observer.php
<?php

class Notify_Biju_Model_Observer {

    const XML_PATH_EMAIL_TEMPLATE = 'newsletter_alert_template';

    public function newsletteralert($observer){

            $eventname=$observer->getEvent()->getName();
            $subscriber=$observer->getEvent()->getSubscriber();
            $email=$subscriber->getEmail();
            $id=$subscriber->getId();

            $emailtemplate=Mage::getModel('core/email_template')->loadDefault(self::XML_PATH_EMAIL_TEMPLATE);
            $sender=array();
            $sender['name']="admin";
            $sender['email']="email@example.com";
            try{
            $emailtemplate->sendTransactional(
                    self::XML_PATH_EMAIL_TEMPLATE,
                    $sender,
                    'firstname.lastname@example.com, email@example.com', // email id of website/store admin
                    'admin',
                    array('subscirber'=>$subscriber)
                    );

            }
            catch(Mage_Core_Exception $e){
                // echo $e->getMessage();
                Mage::log($e->getMessage(),null,'newsletter.log');

            }

        }

}

app/etc/modules/Notify_Biju.xml
<?xml version="1.0" encoding="UTF-8"?>
<config>
    <modules>
        <Notify_Biju>
            <active>true</active>
            <codePool>local</codePool>
        </Notify_Biju>
    </modules>
</config>

app/locale/en_US/template/email/newsletter_subscription_notify.html

Dear Admin

 Congratulations! A new subscriber has registered for Newsletter. Please login to the admin back-end to manage subscriptions.

Subscriber Email: {{var subscriber.getEmail()}}

Subscriber ID: {{var subscriber.getId()}}

Answer: You have a typo:
Search for array('subscirber'=>$subscriber).
You have "subscriber" misspelled as "subscirber"
Answer: To get the subscriber email value & id in observer,
$subscriber=$observer->getEvent()->getSubscriber() ;
       $email=$subscriber->getSubscriberEmail();
       $id=$subscriber->getSubscriberId();

$emailtemplate->sendTransactional(
                    XML_PATH_EMAIL_TEMPLATE,
                     Mage::getStoreConfig(XML_PATH_CONFIRM_EMAIL_IDENTITY),
                    'firstname.lastname@example.com', // email id of website/store admin
                    'admin',
                    array(
                    'subscriber_email'  => $email,
                    'subscriber_id' => $id
            )

Then in mail - add the variables like this instead of your lines

Subscriber Email: {{var subscriber_email}}

Subscriber ID: {{var subscriber_id}}


Reducing the time to locate four XYZ coordinates from a large pandas dataframe

Question: Sample CSV file: sample csv file with force in Y direction
We are trying to four points on a 3D grid stored in a dataframe. Each point has x,y,z values and a corresponding force in the y and z direction. The first point we want is at a given radius and at angle 0, the next point is 180 degrees away from that point, the third point is at the same y,z coordinates as the second, but with the other x value (there are only two possible x values), the last point is at the same y,z point as the first point, but has the other x. The dataframe looks like this:

Node Number
X Location
Y Location
Z Location
Y Force
Z Force

1
82.916
-182.25
-200.32
6.95E-02
7.11E-03

2
81.542
50.54
...
...
...

3
...
...

4
...

...
...

497500
...

The dataframe contains 5 columns; X Location (mm), Y Location (mm), and Z Location (mm), Force_y_direction, Force_z_direction. These coordinates are read from a .csv file and stored in a pandas dataframe as such;
def link_csv_files():
    # Read xyz coordinates from csv file
    df_x = pd.read_csv('x-data.csv', engine='python')

    create_nodes(df_x, df_y, df_z)

def create_nodes(df_x, df_y, df_z):
    # Copy the common columns into the df_combined dataframe (Node Number, X Location, Y Location, Z Location)
    df_combined['Node Number'] = df_x['Node Number']
    df_combined['X Location (mm)'] = df_x['X Location (mm)']
    df_combined['Y Location (mm)'] = df_x['Y Location (mm)']
    df_combined['Z Location (mm)'] = df_x['Z Location (mm)']

    # Iterate over the columns for the y-directional force
    for column in df_y.columns[4:]:
        # Copy the column values to the combined DataFrame
        df_combined['Y Elastic Strain (mm/mm)'] = df_y[column]

    # Iterate over the columns of the z-directional force
    for column in df_z.columns[4:]:
        # Copy the column values to the combined DataFrame
        df_combined['Z Elastic Strain (mm/mm)'] = df_z[column]

There are three .csv files, they all share the same Node Number, and xys locations. The only difference is the rightmost column; one file has the x force, one file has the y force, last file has the z force
sample csv image
There are 500,000 points
The points fall on different radius' on a circle, and they are located at every 11.25 degrees around the radius. y is the length, z is the height, and x is the depth. 3D point, y-z view
We want to find two sets of points that will be used in an equation: the first set of points have the same depth, but are located 180 degrees apart. The second set of points are at the same y,z locations but on the other side (there are 2 depth values).
The problem is at the moment it takes 11 minutes to find the four points for one radius. We want to reduce this time. We can use any libraries.
This is my current implementation:
# Function to find the combination of 4 points
# df_points: The dataframe containing all node locations
# points_arr: An array of every points along 0-180 degrees
def find_point_combinations(df_points, points_arr):

    # Array to store the two sets of points.
    voltage = []

    for n in range(0, len(points_arr)):
        point1 = points_arr[n]
        point3 = point1

        #Finding point3
        for pt in range(0, len(df_points)):
            if df_points['Y Location (mm)'].loc[pt] == -1 * point1[2] and df_points['Z Location (mm)'].loc[pt] == -1 * point1[3]:
                if df_points['X Location (mm)'].loc[pt] == point1[1]:
                    point3 = df_points.loc[pt]
                    break

        #Finding point4, which is on the other side
        for pt in range(0, len(df_points)):
            if df_points['Y Location (mm)'].loc[pt] == point3[2] and df_points['Z Location (mm)'].loc[pt] == point3[3]:
                if df_points['X Location (mm)'].loc[pt] != point3[1]:
                    point4 = df_points.loc[pt]
                    break

        #Finding point2
        for pt in range(0, len(df_points)):
            if df_points['Y Location (mm)'].loc[pt] == point1[2] and df_points['Z Location (mm)'].loc[pt] == point1[3]:
                if df_points['X Location (mm)'].loc[pt] != point1[1]:
                    point2 = df_points.loc[pt]
                    break

       voltage.append(point1, point2, point3, point4)
       return voltage

It uses four for-loops which causes a lot of overhead. I believe this is one reason why the processing time is so high. One set of points results in this output:
Node Number                 2.708330e+05
X Location (mm)             1.168100e+02
Y Location (mm)             2.060000e-04
Z Location (mm)             2.958000e+02
X Elastic Strain (mm/mm)             NaN
Y Elastic Strain (mm/mm)    2.420000e-06
Z Elastic Strain (mm/mm)   -5.970000e-07
Error                       4.200000e+00
Angle (deg)                 1.800000e+02
Radius                      2.958000e+02

Name: 15, dtype: float64
Node Number                 90690.000000
X Location (mm)               124.010000
Y Location (mm)               -58.527000
Z Location (mm)               294.240000
X Elastic Strain (mm/mm)             NaN
Y Elastic Strain (mm/mm)        0.000005
Z Elastic Strain (mm/mm)        0.000001
Error                         597.115221
Angle (deg)                   180.000000
Radius                        295.800000

Name: 367794, dtype: float64
Node Number                 2.708330e+05
X Location (mm)             1.168100e+02
Y Location (mm)            -2.060000e-04
Z Location (mm)            -2.958000e+02
X Elastic Strain (mm/mm)             NaN
Y Elastic Strain (mm/mm)    2.420000e-06
Z Elastic Strain (mm/mm)   -5.970000e-07
Error                       4.200000e+00
Angle (deg)                 1.800000e+02
Radius                      2.958000e+02

Name: 15, dtype: float64
Node Number                 90509.000000
X Location (mm)               124.010000
Y Location (mm)                58.527000
Z Location (mm)              -294.240000
X Elastic Strain (mm/mm)             NaN
Y Elastic Strain (mm/mm)        0.000003
Z Elastic Strain (mm/mm)       -0.000001
Error                          58.809755
Angle (deg)                   180.000000
Radius                        295.800000

Name: 3266, dtype: float64

Comment: Throw your data into an [RDBMS](https://docs.python.org/3/library/sqlite3.html), e.g. using [df.to_sql()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_sql.html). Index the Y and Z columns. Now your WHERE clause can efficiently find nearby coordinates, without even considering most of those half-million points you keep looping over. Or do a [.merge()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.merge.html) if you're allergic to tables.
Comment: Can you also provide sample CSVs? Also how fast do you want it compared to your current implementation?
Comment: @ken Would like to have it go from 11.2 minutes to 5-8 minutes. I have updated the question with a sample CSV at the top
Comment: - or less than 5 mins
Comment: Thank you. However, what I requested was not an image, but actual (or parts of) CSV files that could be used for testing and benchmarking. But since your goal was not very high, maybe the way I just posted will be enough for you.
Answer: There are many solutions to meet your needs, but you should first learn the basics of using pandas.
Here is a reimplementation of your function in the pandas style:
def find_point_combinations(df_points, points_arr):
    # Array to store the two sets of points.
    voltage = []

    for n in range(0, len(points_arr)):
        point1 = points_arr[n]

        # Finding point3
        x_matched = df_points["X Location (mm)"] == point1[1]
        y_matched = df_points["Y Location (mm)"] == -point1[2]
        z_matched = df_points["Z Location (mm)"] == -point1[3]
        xyz_matched_points = df_points[x_matched & y_matched & z_matched]
        # If you want to check how many points matched, check the length here.
        assert len(xyz_matched_points) == 1
        point3 = xyz_matched_points.iloc[0]  # Extract the first matched point.

        # Finding point4, which is on the other side
        x_matched = df_points["X Location (mm)"] != point3[1]
        y_matched = df_points["Y Location (mm)"] == point3[2]
        z_matched = df_points["Z Location (mm)"] == point3[3]
        xyz_matched_points = df_points[x_matched & y_matched & z_matched]
        assert len(xyz_matched_points) == 1
        point4 = xyz_matched_points.iloc[0]

        # Finding point2
        x_matched = df_points["X Location (mm)"] != point1[1]
        y_matched = df_points["Y Location (mm)"] == point1[2]
        z_matched = df_points["Z Location (mm)"] == point1[3]
        xyz_matched_points = df_points[x_matched & y_matched & z_matched]
        assert len(xyz_matched_points) == 1
        point2 = xyz_matched_points.iloc[0]

        # voltage.append(point1, point2, point3, point4)  # This can't be right.
        voltage.append([point1, point2, point3, point4])
    return voltage

Note that since you did not provide us with any data to test, this is just a reimplementation of your function, and I have not checked if it meets all your requirements (except for the execution time).
Also note that if this is not fast enough for you, I recommend using KDTree which can handle point clouds more efficiently.
Comment: Thank you very much for your help! I think this should be good enough to meet my needs, and I will learn about KDTree.


Why everybody say that percussions like drums or whispering does not have notes?

Question: Nevertheless I think each time there is one sound, there is one fundamental note and harmonics ?
Comment: Just to know how to word a potential answer, how familiar are you with the concept of Fourier transform?
Comment: Semantics: Since this question wants to get specific and analytical, it's more useful to talk about "pitches" than "notes," since the word "note" comes with other meanings (duration, confusion over octaves, etc.) Also, thought experiment: does white noise have pitches? (Sure, all sounds are "made of" frequencies, but it doesn't mean there has to be a fundamental.)
Comment: Not everyone says this. It's common knowledge that drums can be pitched, and some percussion like the xylophone family is _clearly_ pitched.
Comment: I just know that in Fourier transform there is fundamental and harmonics that are at multiples frequencies of fundamentals
Comment: Not everybody says that: http://www.youtube.com/watch?v=dNlu6d3Xjbc&t=1m28s (for those who don't want to click the link, it's Terry Bozzio explaining his pentatonic drum kit)
Answer: Each time there is a periodic sound/signal, there is a fundamental and harmonics.  There are proper periodic sounds (like a regularly interrupted air stream) and there are almost proper periodic sounds where higher modes of vibration are approximately harmonic, like piano strings where thicker strings exhibit some disharmonicity with higher partials being somewhat sharp of being proper harmonics.  Those are pitched sounds.
But something like the crash of a cymbal or a hissing sound just does not have enough dominant frequencies to lend itself to a periodic or almost-periodic representation.
Drums tend to be unspecific enough in frequency (also depending on just how you strike them) to count as unpitched while timpani are constructed and tuned specific enough to count as pitched.  That's sort of the borderline that is drawn musically.
Comment: Good answer, periodicity is the key!


Curl function to select options from a select box and auto submit

Question: i am a newbie and try different things everyday and always come here when i am stuck with something.
I want to write a script using curl and php that goes to this link :http://tools.cisco.com/WWChannels/LOCATR/openBasicSearch.do and then goes through each page for each country capturing a list of every partner in every country and saving it to database.
i have no ideas how script will select countries one by one from select box and redirect page to country page...which is the very first thing to do, once we are on the page pattern matching comes in play for storing name and address in database which i can manage.
Problem is before we select any country url is::http://tools.cisco.com/WWChannels/LOCATR/BasicSearch.do
and after we select country say 'india' url is:http://tools.cisco.com/WWChannels/LOCATR/performBasicSearch.do , there is no reference to any country selected.
The Idea that i had was to traverse the HTML page, and enter all countries in an array and then make a recursive function to call a page with specific country but for that we need something different in URL for each country in recursive function right?
Please help
Answer: Your url is messed up, so I can't see the exact page you are talking about, however what is most likely happening is that when you change the country the website is making a POST request to the same page
 with a variable like country (although it may be something else) with the value of the country name/country id that you selected. If you View Source on the page you will be able to see the input field's name that is being passed on. Once you do that, while making your cURL request you can set the cuRL option of CURLOPT_POSTFIELDS, which reads like so:

The full data to post in a HTTP "POST" operation. To post a file, prepend a filename with @ and use the full path. This can either be passed as a urlencoded string like 'para1=val1&para2=val2&...' or as an array with the field name as key and field data as value.

So, keeping that in mind you would do something like this:
$ch = curl_init('http://tools.cisco.com/WWChannels/LO...BasicSearch.do');
$ch = curl_setopt($ch, CURLOPT_POSTFIELDS, array('country' => 'India'));
$ch = curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$contents = curl_exec($ch);
curl_close($ch);

As I said, though, the country => India part of it is an educated guess as to what the field might be passing. You have to inspect the HTML to find out for yourself.
Answer: For automation/scraping, I would recommend that you use a virtual browser, such as SimpleBrowser. It's part of SimpleTest, but you can use it on its own.


Unable to insert form data to the sql database

Question: I have simple form which reads users name,email,and city. I have created a database and a table with fields id,name,email and city. When I run the code I don't encounter any errors or exceptions,but unable to see the inserted data into table. Following is my code
SqlConnection con =
    new SqlConnection("Data Source=(LocalDB)\\MSSQLLocalDB;AttachDbFilename=F:\\Database1.mdf;Integrated Security=True");
con.Open();
SqlCommand cmd =
    new SqlCommand("insert into tbl_insert  values ( " + Convert.ToInt32(txtId.Text) + "," + "'" + txtName.Text + "'" + "," + "'" + txtEmail.Text + "'" + "," + "'" + txtboxCity.Text + "'" + ")", con);
cmd.ExecuteNonQuery();
con.Close();

I have attached my database screenshots.
    Database design:

This is output I get after running my code.
Output with null value stored:

Comment: Are you getting any error when executing this query?
Comment: I'm not too sure on the problem you are having, but just wanted to give you a heads up on SqlParameter's https://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlparameter(v=vs.110).aspx The code you have written is highly susceptible to SQL injection, so I would definitely recommend adding SqlParameter's if this project is to be more than just practice / fun
Comment: @un-lucky iam not getting errors while running this program
Comment: are you sending ID as well ??? :O:O
Comment: @AnkitkumarBhatt yeah i am sending ID as well
Comment: @SameerBetageri what error come when you try to insert?
Answer: You should always use parameterized queries to avoid SQL Injection and get rids from quote problems:
SqlCommand cmd = new SqlCommand("insert into tbl_insert  values (@id,@name,@Email,@City)");
cmd.Parameters.AddWithValue("@id", Convert.ToInt32(txtId.Text));
cmd.Parameters.AddWithValue("@name", txtName.Text);
cmd.Parameters.AddWithValue("@Email", txtEmail.Text);
cmd.Parameters.AddWithValue("@City", txtboxCity.Text);

Although specify the type directly and use the Value property is more better than AddWithValue:
cmd.Parameters.Add("@name", SqlDbType.VarChar).Value = txtName.Text;

Can we stop using AddWithValue() already?
Answer: Try this
con.Open();
SqlCommand cmd = new SqlCommand(@"insert into tbl_insert values(@name,@email,@add)", con);
cmd.Parameters.AddWithValue("@name", txtname.Text);
cmd.Parameters.AddWithValue("@email", txtemail.Text);
cmd.Parameters.AddWithValue("@add", txtadd.Text);
cmd.ExecuteNonQuery();
con.Close();

i suspect the column id is in Auto Number
if not just add another parameter to the query and cmd.Parameters
Comment: If the column `id` is Identity, he should gives error but he already mentioned: *i don't encounter any errors or exceptions*.
Comment: same problem continued...no error debugging showed data is added correctly but it is not appearing in table


Vue.js modal component example not closing

Question: I'm trying to use Modal Component Example from Vue.js website examples in my project. When I use the modal component in my-project component, it works fine, but when I add a close button in the header slot, this button doesn't work, it doesn't close the modal while the default button in the footer still works fine.
The modal component :
<template>
    <transition name="modal">

                <slot name="header">
                default header
                </slot>

                <slot name="body">
                default body
                </slot>

                <slot name="footer">
                default footer
                <button class="modal-default-button" @click="$emit('close')">
                    OK
                </button>
                </slot>

    </transition>
</template>

How I'm calling it :
<template>
    <button id="show-modal" @click="showModal = true">Show Modal</button>

    <modal v-if="showModal" @close="showModal = false">

            <button class="modal-default-button" @click="$emit('close')">Close</button>

    </modal>
</template>

What's wrong in my implementation of this code ?
Answer: Since this is at the parent:
<template>
    <button id="show-modal" @click="showModal = true">Show Modal</button>

    <modal v-if="showModal" @close="showModal = false">

            <button class="modal-default-button" @click="$emit('close')">Close</button>

    </modal>
</template>

This code:
<button class="modal-default-button" @click="$emit('close')">Close</button>

Actually makes the parent emit the 'close', not the modal child.
So, since the property that controls the modal is showModal, just set it to false instead:
<button class="modal-default-button" @click="showModal = false">Close</button>

Demo JSFiddle: https://jsfiddle.net/acdcjunior/mwLbw11k/4208/


What Am I doing Wrong In Regressing and Graphing Exponential Plot?

Question: So I am new to R and have just been learning it for a month, and have encountered this frustrating problem.  I have some basic data and am attempting to fit an exponential curve to it and then afterwards graph it.  This is my code so far:
y <- c(1594,343,57,20,9)
time <- 1:5
fit <- lm(log(y) ~ log(time))

plot(time,y)
abline(fit)

After doing all this, I simply get this weird line on the bottom of my graph.

What exactly am I doing wrong?
Comment: This question is only seemingly about R. It is actually a mistake in the model, ,as @Hiromi 's answer shows.
Answer: Your linear model is fitted on $log(y)$ and $log(time)$, so when you call abline(fit), it produces the fitted line for that model. You could do this instead:
y <- c(1594,343,57,20,9)
time <- 1:5
fit <- lm(log(y) ~ log(time))
plot(log(time),log(y), pch=16)
abline(fit)

But if you would like to visualise on your original scale, you would need to exponentiate your $y$ to revert back the scale. Actually, I'm not sure why you need to log your time variable, it will make it difficult to interpret your coefficient. Rather, I would do it this way:
fit2 <- lm(log(y) ~ time)
fitted_y2=exp(fit2$fitted.values)
plot(time,y,ylim=c(0,max(fitted_y2,y)), pch=16)
lines(time,fitted_y2)


JQuery Drop Down issue

Question:
**Script for Drop Down and click event **
        OUTPUT : I Randomly select multiple time with different values When I view the console I do get multiple on change events and

they all are execute. I need to obtain the recent drop down value to
  be executed.

Comment: ` $('input[type="checkbox"]').click` inside `$('#dr4').change` means that as many time you make the change in dropdown selection those many times `.click` event listener will get attached to those checkboxes. move that outside of `.change`
Answer: You can directly get the selected value from dropdown i providing you a sample of your code please check it may it help you to solve your issue

    <select id="dr4">
        <option value="1">1</option>
        <option value="2">2</option>
        <option value="3">3</option>
    </select>
    <input type="checkbox">checkbox

Comment: Thanks ajay its my pleasure
Comment: Hi Niraj can you please help me out with one more question

JQuery Drop Down issue


Mystery spacing on svg web graphic

Question: This is my code: http://jsfiddle.net/C8d34/17/
This is my code for the SVG:
<svg viewBox="0 0 100 100"> /* this here */
  <circle cx="50" cy="50" r="40" stroke="green" stroke-width="4" fill="yellow" />
</svg>

I've tried a few different things, but it seems that although in theory the circle should hit the very top and bottom of the blue box, it instead leaves a space. I've tried this:
* {
    padding:0px;
    margin:0px;
}

All to no avail. Can anyone please tell me why the circle doesn't fill up this area completely.
Answer: Basic geometry.
Your numbers must be such that (r * 2) + strokeWidth = height
In this case, if you want to keep stroke-width="4", then you want
r = (height - strokeWidth) / 2 = 96 / 2 = 48
So:
<circle cx="50" cy="50" r="48" stroke="green" stroke-width="4" fill="yellow" />

Mystery spacing on svg web graphic


How do I use and show a vocabulary with multiple fields?

Question: When I create a new vocabulary, I can create it in such a way that there are multiple fields. However, when I then create a content type with a field a term reference to that vocabulary, I only can see the name field from the vocabulary and not the other fields.
This regardless of the method I use, whether be radio buttons, check list, select list, autocomplete, autocomplete deluxe. I want to show those other fields so I can discriminate between two very similar, but with distinct values in the vocabulary.
For example, if it is a vocabulary of names, to add a date of birth which gets at least shown (and preferably is editable when adding a new term using autocomplete deluxe) when I fill in the form when adding an item of that content type.
Answer: What you're looking for is Entity Reference.  This will let you create a taxonomy term field on your content type and give you the option to render the entire term using the display of your choice (teaser, full etc).
Setting it up would be very similar to what you've already tried with term reference:

Create an entity reference field in your content type
Target type = taxonomy term
Set target bundle if preferred, then save
Under manage display, set the field format to rendered entity
Set the view mode to your preferred option (you would configure these options under your vocabulary manage display screen)
Answer: Install the Views and Entity Reference modules, create a new View and Add an Entity Reference Display:

Then add in the fields the Term Name and the Term ID (this is just an example, you can add more fields), click in the Term ID and check Exclude from display,

Save and click in the Term Name and go to rewrite the output of the title as <code>[name] - ([tid])</code>

Go to edit the settings of the format and check the Term Name, this will allow you to search by the Name of the Term.

Save the view.
Go to edit your Entity Reference field and select in the Mode Views: .... (like the following image) and select your View (in this case the name is: etiquetas - Entity Reference) and save the settings:

Then go to see the result when you go to add a new node:
Comment: This is very thorough, but misses the first point of having an entity reference field. Therefore I accepted the other answer, sorry.
Comment: @Ernst the field is an Entity Reference field, but to a Taxonomy Term, this is not what you need?
Comment: @Ernst in the last Picture, the field Articulo, is an Entity Reference field to a Taxonomy Term


Remap one of the keyboards

Question: I have a notebook which sometimes I use with an external keyboard. Its built-in keyboard has some keys which I'd like to remap, however, I don't want to make any changes to the external keyboard.
I am aware of this question from 2012, but I'd like to have a different solution (a more user-friend one) if possible.
I use Win 10 on an Acer Aspire E 13.
Answer: You can use SharpKeys to remap any key you want. I also use Win 10 and it work verry nice. Don't forget to restart your computer after Write to Registry.
Comment: Nice way to test the result: https://css-tricks.com/snippets/javascript/javascript-keycodes/

Remap one of the keyboards


Erro ao adicionar dependências no maven

Question: Quando adiciono os imports solicitados para as classes
 KeycloakAuthenticationToken keycloakAuthenticationToken = (KeycloakAuthenticationToken) principals;
         AccessToken accessToken = keycloakAuthenticationToken.getAccount().getKeycloakSecurityContext().getToken();

Aparece o seguinte erro no Package
The type org.springframework.security.authentication.AbstractAuthenticationToken cannot be resolved. It is indirectly referenced from required .class files

Ja tentei solucionar realizando maven update, apagando o diretorio dentro de .m2 e recriando com um update, ja realizei um project -> clean, tudo sem sucesso.
E assim esta o meu .m2
Comment: Você poderia postar seu pom.xml para visualizarmos. Para isso, edite a pergunta.
Answer: Certifique que esteja adicionado ao pom esta dependência.
<dependency>
    <groupId>org.springframework.security</groupId>
    <artifactId>spring-security-core</artifactId>
    <version>5.5.2</version>
 </dependency>

Se já estiver, tente buscar uma versão mais nova em
Maven Repository Spring Security Core
Se estiver utilizando spring-boot, veja se está adicionado:
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-security</artifactId>
</dependency>

Erro ao adicionar dependências no maven