tensorflow pb file inferencing takes more than 3seconds for an image

Question: I am generating a .pb file from the piece of code given below
import tensorflow as tf
with tf.Session() as sess:
    gom = tf.train.import_meta_graph('C:\\chhaya\\CLITP\\Tvs_graphs\\job.ckpt-20.meta')
    gom.restore(sess,tf.train.latest_checkpoint('C:\\chhaya\\CLITP\\Tvs_graphs'))
    graph = tf.get_default_graph()
    input_graph = graph.as_graph_def()
    output_node_name = "predictions"
    output_graph = tf.graph_util.convert_variables_to_constants(sess,input_graph,output_node_name.split(','))
    res_file = 'C:\\chhaya\\CLITP\\Tvs_graphs\\Savedmodel.pb'
    with tf.gfile.GFile(res_file,'wb') as f:
        f.write(output_graph.SerializeToString())

But while inferencing from the pb file it takes 5seconds for the first image and 3 seconds for others after that. The code for inferencing is given below.
import tensorflow as tf
import os
from tensorflow.python.platform import gfile
from PIL import Image
import numpy as np
import scipy
from scipy import misc
import matplotlib.pyplot as plt
import cv2
import time
from aug_tool import data_aug
frozen_graph = 'C:\\chhaya\\CLITP\\Tvs_graphs\\Savedmodel1.pb'
with tf.gfile.GFile(frozen_graph,'rb') as f:
    reco = tf.GraphDef()
    reco.ParseFromString(f.read())

with tf.Graph().as_default() as gre:
    tf.import_graph_def(reco,input_map=None,return_elements=None,name='')

    l_input = gre.get_tensor_by_name('input_image:0') # Input Tensor
    l_output = gre.get_tensor_by_name('predictions:0')
    files=os.listdir('C:\\chhaya\CLITP\\Tvs Mysore\\NQCOVERFRONTLR\\')
    imageslst=np.zeros((len(files),224,224,3))
    i=0
    for file in files:
        image = scipy.misc.imread('C:\\chhaya\CLITP\\Tvs Mysore\\NQCOVERFRONTLR\\'+file)
        image = image.astype(np.uint8)
        Input_image_shape=(224,224,3)
        resized_img=cv2.resize(image,(224,224))
        channels = image.shape[2]
        #print(channels)
        if channels == 4:
            resized_img = cv2.cvtColor(resized_img,cv2.COLOR_RGBA2RGB)
 #image = np.expand_dims(resized_img,axis=2)
    #print()
    image=np.expand_dims(resized_img,axis=0)
    height,width,channels = Input_image_shape
    imageslst[i,:,:,:]=image
    i=i+1

    init = tf.global_variables_initializer()
    with tf.Session(graph=gre) as sess:
        sess.run(init)
        for i  in range(4):
            t1=time.time()
            Session_out = sess.run((tf.nn.sigmoid(l_output)) , feed_dict = {l_input : imageslst[:1]} )
            print(Session_out.shape)
            t2=time.time()
            print('time:'+str(t2-t1))

I am using tensorflow-gpu 1.12 with windows 7 using anaconda python 3.5
I also tried assigning gpu and cpu to predictions something like this
with tf.device('/gpu:0'):
    for i  in range(4):
        t1=time.time()
        Session_out = sess.run((tf.nn.sigmoid(l_output)) , feed_dict = {l_input : imageslst[:1]} )
        print(Session_out.shape)
        t2=time.time()
        print('time:'+str(t2-t1))

The thing I noted here is no matter what I assign cpu or gpu the time taken is always the same.The model is a transfer learning vgg16 model. Am i doing something wrong in the code? My gpu is Quadro 8GB
Answer: After a lot of trial and error what I noticed was if I remove the tf.nn.sigmoid from while inferencing then the time taken was less than 10 milliseconds exactly how the model should have performed.
Before we were using the below line to get predictions which resulted in high inference time:-
Session_out = sess.run((tf.nn.sigmoid(l_output)) , feed_dict = {l_input : imageslst[:1]} )

So instead of using the above line now we use this:-
Session_out = sess.run(l_output , feed_dict = {l_input : imageslst[:1]} )

This proves to give us the accurate results. Although I am pretty unsure as to why that happens.


Filter custom post type based on post meta value

Question: I want to create a filter in Wordpress Admin for a custom post type using the value of post_meta. I found that it's possible to filter based on a taxonomy from the hook 'restrict_manage_posts', and I managed to get this working using the following code. I'd like to modify it to filter based on the value of the post_meta. The reason I want this to be meta instead of taxonomy is so that it doesn't show on the front end. It should be a filter for the user's admin only. How can I use the filter with meta instead of taxonomy?
<code>add_action( 'restrict_manage_posts', 'add_admin_filters', 10, 1 );

public function add_admin_filters( $post_type ){
    if( 'my_post_type' !== $post_type ){
        return;
    }
    $taxonomies_slugs = array(
        'my_taxonomy',
        'my_other_taxonomy'
    );
    // loop through the taxonomy filters array
    foreach( $taxonomies_slugs as $slug ){
        $taxonomy = get_taxonomy( $slug );
        $selected = '';
        // if the current page is already filtered, get the selected term slug
        $selected = isset( $_REQUEST[ $slug ] ) ? $_REQUEST[ $slug ] : '';
        // render a dropdown for this taxonomy's terms
        wp_dropdown_categories( array(
            'show_option_all' =>  $taxonomy->labels->all_items,
            'taxonomy'        =>  $slug,
            'name'            =>  $slug,
            'orderby'         =>  'name',
            'value_field'     =>  'slug',
            'selected'        =>  $selected,
            'hierarchical'    =>  true,
        ) );
    }
}
</code>
Answer: The solution to this is to use a private taxonomy instead of post_meta, and then you can create the filter. That way it doesn't show on the front end.
<code> /**
 * Register a private 'Genre' taxonomy for post type 'book'.
 *
 * @see register_post_type() for registering post types.
 */
function wpdocs_register_private_taxonomy() {
    $args = array(
        'label'        => __( 'Genre', 'textdomain' ),
        'public'       => false,
        'rewrite'      => false,
        'hierarchical' => true
    );

    register_taxonomy( 'genre', 'book', $args );
}
add_action( 'init', 'wpdocs_register_private_taxonomy', 0 );
</code>


Call stored procedure Using C#

Question: This is my stored procedure
CREATE DEFINER=`root`@`localhost` PROCEDURE `sp_CPC`(IN _B VARCHAR(100),IN _G      VARCHAR(2), IN _R VARCHAR(30), IN _D VARCHAR(30), OUT _C FLOAT, OUT _P FLOAT)

BEGIN
      //Something Hear
 END$$
DELIMITER ;
I Call this stored procedure by C# flowing Code
   DataSet tmpDataSet = new DataSet();

    mCommand.CommandText = "sp_CPC";
         mCommand.CommandType = CommandType.StoredProcedure;
        // mCommand.CommandText = "sp_select_all_employees";
         mCommand.Parameters.AddWithValue("@_B", "bty-23");
         mCommand.Parameters.AddWithValue("@_G", "3");
         mCommand.Parameters.AddWithValue("@_R", "9000");
         mCommand.Parameters.AddWithValue("@_D", "92");

         mCommand.Parameters.Add("@_C",SqlDbType.Float);
        mCommand.Parameters["@_C"].Direction = ParameterDirection.Output;
        mCommand.Parameters.Add("@_P", SqlDbType.Float);
         mCommand.Parameters["@_P"].Direction = ParameterDirection.Output;

         try
         {
             mConnection.Open();
             mCommand.ExecuteNonQuery();
             mAdapter.Fill(tmpDataSet);

         }
         catch (Exception ex)
         {
             strErrorInfo = ex.ToString();
         }
         finally
         {
             mConnection.Close();
         }
         return tmpDataSet;

     }

It show following Error
Input string was not in a correct format.
How can i solve it
Answer: I think you did error in the following section
mCommand.Parameters.Add("@_C",SqlDbType.Float);
    mCommand.Parameters["@_C"].Direction = ParameterDirection.Output;
    mCommand.Parameters.Add("@_P", SqlDbType.Float);
     mCommand.Parameters["@_P"].Direction = ParameterDirection.Output;

change SqlDbType into MySqlDbType
  mCommand.Parameters.Add("@_C",MySqlDbType.Float);
    mCommand.Parameters["@_C"].Direction = ParameterDirection.Output;
    mCommand.Parameters.Add("@_P", MySqlDbType.Float);
     mCommand.Parameters["@_P"].Direction = ParameterDirection.Output;

Comment: Thank you it work now . But Now i face new problem
After first time call it work when second time again call it show Parameter '@_C' has already been defined. How can i solve it.help me

Call stored procedure Using C#


eclipse jave project with two source directories - how to set the visibility of the source folders

Question: I have a Java project in eclipse - within the project I want to configure two source directories:

src-api
src-core

This is simple.
The problem is that I want src-core to be able to access src-api, but I don't want src-api to access src-core, I want eclipse to not let me compile and be able to find classes configured in src-core only from the src-api.
Is it possible to do so?
I would very much appreciate your help.
Answer: No, it is not possible.
Just create two different project api & core. Then make core project dependent to api project.
Answer: I would use two different projects, one for src-api and one for .
The src-core should have src-api as a dependency, but not vice versa.


Data da máscara altera após dar reload na página

Question: Fiz uma máscara pelo PHP pra mudar o formato que a data aparece (para o padrão dd/mm/yyyy), aonde ao adicionar um novo registro, ele mostra a data corretamente. Porém, ao atualizar a página ele volta pra um formato fixo (01/01/1970).
PHP das variáveis das datas
<?php
  session_start();

  include('../_comum/config.php');

  if(isset($_POST['submit'])){
  $idContrato = $_POST['idContrato'];
  $numContrato = $_POST['numContrato'];
  $dataInicio = $_POST['dataInicio'];
  $dataFinal = $_POST['dataFinal'];
  $responsavel = $_POST['responsavel'];
  $nomeCliFor = $_POST['nomeCliFor'];
  $ativo = isset($_POST['ativo']) ? 1 : 0;

  $result = mysqli_query($conexao, "INSERT INTO Contrato(idContrato,numContrato,dataInicio,dataFinal,responsavel,ativo) VALUES('$idContrato', '$numContrato', '$dataInicio', '$dataFinal', '$responsavel','$ativo')");

  }
  header('contrato.php');

Lista dos Resgistros:

              Data Inicio
              Data Final

        <?php

          while($user_data = mysqli_fetch_assoc($result))
          {
            echo "
";
            echo "".date('d/m/Y', strtotime($dataInicio))."	";
            echo "".date('d/m/Y', strtotime($dataFinal))."	";
            echo "
";
          }
        ?>

Formulário:

      <form action="contrato.php" id='cadastroRegistro' method = "POST">

Numero do Contrato:

              <input type='text' id='' data-ls-module="charCounter" maxlength="8" name = 'numContrato'/>

Data Inicio:

                <input type="date" id="birthday" name="dataInicio">

Data Término:

                <input type="date" id="birthday" name="dataFinal">

Responsável:

                <input type="text" id='' data-ls-module="charCounter" maxlength="30" name = 'responsavel'/>

              <input class="form-check-input" type="checkbox" role="switch" id="flexSwitchCheckDefault" name= 'ativo' value="on">
              Ativo:

            <button class="btnSalvar" type = "submit" name="submit">Salvar</button>
            <button class="btnSalvar" type = "button" formnovalidate>Voltar</button>

</form>

Comment: você já verificou se essas informações estão sendo ( persistidas ) salvas no banco? Geralmente essa data padrão do mysql `01/01/1970` é exibida quando é feita a tentativa de inserção de uma data inválida
Comment: é muito provável que você esteja passando a data com um formato inválido no banco. Já tentou formatar no padrão `YYYY-DD-MM` ao salvar dentro do banco?
Comment: edita a pergunta e coloca seu código por completo ( tanto a parte de formulário html , quando o resto do código em php ). Apenas com esse trecho parece que você ao dar o `refresh` na página estar salvando as datas como nulas, que pode estar vinculado com o seu problema.
Comment: @William no banco esta salvando tudo certinho, até quando eu edito ele altera de forma correta
Comment: @William coloquei o form certinho, junto com o código do insert do php
Answer: Bom achei a solução. O erro estava propriamente na máscara do PHP. Sem a variavél $user_data, ao dar o reload a informação não era carregada.
echo "".date('d/m/Y', strtotime($user_data['dataInicio']))."	";
echo "".date('d/m/Y', strtotime($user_data['dataFinal']))."	";

Comment: Sua resposta pode ser melhorada adicionando informações fundamentais. Basta [edit] para adicionar mais detalhes, como citações ou documentação para que outras pessoas possam confirmar que sua resposta está correta. Você pode encontrar mais informações sobre como escrever boas respostas [na Central de Ajuda](/help/how-to-answer).


how to create a frequency histogram with predefined non-uniform intervals?

Question: How can I make a histogram in RStudio with the frequency on the y-axis and on the x-axis pre-defined non-uniform intervals, for example: 0-50, 50-150, 150-500, 500-2000.
I have hypothetical data, as I would not like to share my original data.
data <- c(1,1.2,40,1000,36.66,400.55,100,99,2,1500,333.45,25,125.66,141,5,87,123.2,61,93,85,40,205,208.9)

Answer: Set the breaks you want with the breaks argument in the hist function.
hist(data, breaks=c(0, 50, 150, 500, 2000), freq=TRUE)

Update
If you want even bar widths you need to put your data into a categorical format rather than continuous. You could achieve this by doing the following.
# Cut your data into categories using your breaks
data <- cut(data,
            breaks=c(0, 50, 150, 500, 2000),
            labels=c('0-50', '50-150', '150-500', '500-2000'))
# Make a data table (i.e. a frequency count)
data <- table(data)
# Plot with `barplot`
barplot(data)

Comment: This is because your data is continuous and not categorical. I'll update my answer to show what to do if you want even bar sizes.
Comment: but why did the bar vary in size?


tangent vector of a right action

Question: Given a G right action on a principal bundle P,  $\triangleleft: P \times G \rightarrow P$, if we have a curve $\gamma(t) = \delta(t) \triangleleft g(t)$ for $\gamma ,\delta \in P, g \in G$, I would guess the corresponding tangent vector to $\gamma$ is:
$$
\gamma'(t) = (\delta \triangleleft g)'(t) = (\triangleleft g)_*\delta'(t)
$$
Based on the definition of the pushforward, and where the slight abuse of notation $\triangleleft g : P \rightarrow P$ indicates the right action, and $\triangleleft g_*$ is its corresponding pushforward.
However, in Theorem 5.8.2 of Hamilton's "Mathematical Gauge Theory", it is stated that (in the context of horizontal lifts) the tangent vector to $\gamma$ is:
$$
\gamma'(t) = (\triangleleft g)_*\delta'(t) + \tilde\Xi(g'(t))
$$
Where $\Xi$ is the Maurer-Cartan form and the tilde denotes the fundamental vector field, ie for a Lie algebra element X, the corresponding fundamental vector field is $\tilde X_p = (p\triangleleft \exp(tX))'(t)$
I'm not sure why this second term is necessary, and why my original equation is wrong. The theorem in "Mathematical Gauge Theory" refers to another theorem 3.5.4, which seems to manipulate the pushforward of the map $\triangleleft: P \times G \rightarrow P$, but once again I'm not sure how this works and why it is necessary.
Comment: This may be of help. https://math.stackexchange.com/questions/4600296/checking-killing-fields-are-g-invariant
Comment: For the most part I understand what the fundamental vector field is, but I'm not sure how that applies here
Comment: So I reviewed theorem 3.5.4 as mentioned in the original post, and I think my main source of confusion is that its talking about a pushforward of the group action $\triangleleft$, and therefore about the space $T_pP \bigoplus T_gG$, whereas the tangent vector $\gamma'(t)$ is just $T_pP$. There seems to be some kind of local trivialization property that is used, and @ZackFox's post mentions this, but once again I'm not sure why its necessary
Answer: It seems what I was missing was that g was also a curve, and therefore the Leibniz formula should have been applied. Following proposition 1.4 of Kobayashi & Nomizu's "Foundations of Differential Geometry Vol 1",  If we have two manifolds $X,Y$ with $x(t)\in X, y(t)\in Y$, then the cartesian product of these two manifolds will have $z(t):= (x(t), y(t))\in X\times Y$. My mistake was assuming the corresponding tangent vector at $t = t_0$ was $z'(t_0) = (x'(t_0), y'(t_0))$, however actually apply $z(t_0)$ on an arbitrary function $f:X\times Y \rightarrow \mathbb{R}$, then
$$\begin{aligned}
z'(t_0)f &= \frac{d}{dt}\bigg\rvert_{t=t_0}f(z(t)) = \frac{d}{dt}\bigg\rvert_{t=t_0}f(x(t), y(t))\\
&=\lim_{\Delta t \rightarrow 0} \frac{f(x(t_0+\Delta t),y(t_0+\Delta t)) - f(x(t_0),y(t_0))}{\Delta t}\\
&=\lim_{\Delta t \rightarrow 0} \frac{f(x(t_0+\Delta t),y(t_0+\Delta t)) - f(x(t_0),y(t_0+\Delta t)) + f(x(t_0),y(t_0+\Delta t)) - f(x(t_0),y(t_0))}{\Delta t}\\
&=\frac{d}{dt}\bigg\rvert_{t=t_0}f(x(t), y(t_0)) + \frac{d}{dt}\bigg\rvert_{t=t_0}f(x(t_0), y(t))\\
&= \bar{x'(t_0)}f + \bar{y'(t_0)}f
\end{aligned}$$
where $\bar{x'(t_0)} = (x(t),y(t_0))'$ and similarly for $\bar{y'(t)}$. The second term involving the Maurer Cartan form therefore comes from this rule.

tangent vector of a right action


Toggle class between fixed-bottom and position-sticky

Question: I want the footer to have class position-sticky but toggled to fixed-bottom when it reached the bottom of the page.
Here is what i have so far.

$(window).on("scroll touchmove", function() {
        var scrollHeight = $(document).height();
        var scrollPosition = $(window).height() + $(window).scrollTop();
        if ((scrollHeight - scrollPosition) / scrollHeight === 0) {
            $('#bottom-fix').addClass('position-sticky').removeClass('fixed-bottom');
        }
        else {
            $('#bottom-fix').removeClass('position-sticky').addClass('fixed-bottom');
        }
    });
<link href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" rel="stylesheet"/>

Welcome, folks… welcome to Kitty Bingo, the online bingo site where treats are on the loose! Claim your share of the freebies, the Welcome Bonus, by registering via any devices. 300% bingo bonus and 100 FREE Spins are up for grabs on your first deposit. Seize the occasion whenever you want, furry friends!

How can you claim the offer? It's simple. You need to deposit at least £10 to be eligible. Select the offer in the cashier and you're done! The fresh bucks in your account seep in along with 300% the deposit amount as a bonus. Does that mean Kitty Bingo is tripling your credits to play? Exactly! You can enjoy up to a maximum bonus of £150!

The funds are credited instantly. And you may browse through the list of online slots or bingo rooms to put those bonuses to play! Visit the many 75-ball bingo, 90-ball bingo and 5-line bingo rooms. They feature varying ticket prices, gameplays and significantly, Progressive Jackpots. Each time a wager is set, it contributes to the pot that grows in size!

And what about those 100 Free Spins? Get them in batches of 20 Free Spins for 5 consecutive days. A way to let the fun last longer! Free Spins are playable on a range of selected games from the Exclusive Games Collection. Find popular titles including Jazz Cat Slot, Gladiator Gold, Blackbeard Booty Slot and The Lost Slot Of Riches.

At Kitty Bingo, the fun knows no bounds! Drop by the Promotions Page whenever you want to refill your bonus balance. The Welcome Package issues second deposit bonuses as well. If you're in the mood to double your deposit with 100% Bingo Bonus or claim 50% Slots Bonus, do not hesitate to deposit a second time. A third deposit leads to a 150% Bingo Bonus or 75% Slot Bonus at Kitty Bingo!

At Kitty Bingo, the fun knows no bounds! Drop by the Promotions Page whenever you want to refill your bonus balance. The Welcome Package issues second deposit bonuses as well. If you're in the mood to double your deposit with 100% Bingo Bonus or claim 50% Slots Bonus, do not hesitate to deposit a second time. A third deposit leads to a 150% Bingo Bonus or 75% Slot Bonus at Kitty Bingo!

At Kitty Bingo, the fun knows no bounds! Drop by the Promotions Page whenever you want to refill your bonus balance. The Welcome Package issues second deposit bonuses as well. If you're in the mood to double your deposit with 100% Bingo Bonus or claim 50% Slots Bonus, do not hesitate to deposit a second time. A third deposit leads to a 150% Bingo Bonus or 75% Slot Bonus at Kitty Bingo!

At Kitty Bingo, the fun knows no bounds! Drop by the Promotions Page whenever you want to refill your bonus balance. The Welcome Package issues second deposit bonuses as well. If you're in the mood to double your deposit with 100% Bingo Bonus or claim 50% Slots Bonus, do not hesitate to deposit a second time. A third deposit leads to a 150% Bingo Bonus or 75% Slot Bonus at Kitty Bingo!

At Kitty Bingo, the fun knows no bounds! Drop by the Promotions Page whenever you want to refill your bonus balance. The Welcome Package issues second deposit bonuses as well. If you're in the mood to double your deposit with 100% Bingo Bonus or claim 50% Slots Bonus, do not hesitate to deposit a second time. A third deposit leads to a 150% Bingo Bonus or 75% Slot Bonus at Kitty Bingo!

At Kitty Bingo, the fun knows no bounds! Drop by the Promotions Page whenever you want to refill your bonus balance. The Welcome Package issues second deposit bonuses as well. If you're in the mood to double your deposit with 100% Bingo Bonus or claim 50% Slots Bonus, do not hesitate to deposit a second time. A third deposit leads to a 150% Bingo Bonus or 75% Slot Bonus at Kitty Bingo!

At Kitty Bingo, the fun knows no bounds! Drop by the Promotions Page whenever you want to refill your bonus balance. The Welcome Package issues second deposit bonuses as well. If you're in the mood to double your deposit with 100% Bingo Bonus or claim 50% Slots Bonus, do not hesitate to deposit a second time. A third deposit leads to a 150% Bingo Bonus or 75% Slot Bonus at Kitty Bingo!

At Kitty Bingo, the fun knows no bounds! Drop by the Promotions Page whenever you want to refill your bonus balance. The Welcome Package issues second deposit bonuses as well. If you're in the mood to double your deposit with 100% Bingo Bonus or claim 50% Slots Bonus, do not hesitate to deposit a second time. A third deposit leads to a 150% Bingo Bonus or 75% Slot Bonus at Kitty Bingo!

At Kitty Bingo, the fun knows no bounds! Drop by the Promotions Page whenever you want to refill your bonus balance. The Welcome Package issues second deposit bonuses as well. If you're in the mood to double your deposit with 100% Bingo Bonus or claim 50% Slots Bonus, do not hesitate to deposit a second time. A third deposit leads to a 150% Bingo Bonus or 75% Slot Bonus at Kitty Bingo!

What's left to do? Click on the Join Now and be part of the cat-loving bingo site, Kitty Bingo!

        JOIN NOW

There is some kind of flickering when reached the bottom.
Any help ?
Comment: why can't it just be fixed all the time? I believe position sticky and fixed are essentially the same thing... the flicker is caused by the toggle between the classes.
Comment: You many not need  either of them. Could you explain what you want to achieve?
Comment: @TJWeems if it's fixed all the time, the button will cover part of the text on the footer
Comment: @mahan I want the join now button to stick to the bottom when it reached there. In Bootstrap 4 the class used should allow use to see the footer text clearly
Answer: change the if condition with this:
if($(window).scrollTop() + $(window).height() == $(document).height()) {
       alert("bottom!");
   }

Comment: Not working for me. Can you please make a codepen with Bootstrap 4


How can I reduce the dimension of the netCDF file and change data?

Question: dimensions:
    i1 = 3 ;
    x = 11 ;
    s1 = 1 ;
    mid1 = 8 ;
    mid2 = 8 ;
variables:
    double Height(i1,x) ;
    double Temp(s1, x) ;
    short Soil(s1, x) ;
    double Liq(mid1, x) ;

I have a netCDF file on which I want to reduce the size of one of the dimensions mid1 and replace values:
icond <- ncdf4::nc_open('dat.nc)

#New dimensions for new file
idim <- icond$dim[['i1']]
xdim <- icond$dim[['x']]
s1dim <- icond$dim[['s1']]
mid1dim <- ncdim_def("mid1", "", 1:3) #3 layers
mid2dim <- icond$dim[['mid2']]

mv <- -9999

#Get variable data
Liqxdat <- ncvar_get(icond, 'Liq')[,1:3] #3 Layers

#Define new variable
Liqx = ncvar_def( "Liq", "units", list(mid1dim, i1), mv, prec="double")

#Create netCDF file
nc =  nc_create("test.nc", list(Height, Temp, Soil, Liqx)

#Write data to the NetCDF file
ncvar_put(nc, Liqx, Liqxdat)

But this is not giving me any data in the output file.
Answer: Unfortunately I cannot run your code. So I just can guess why it's not working.

There is a closing bracket missing on the line nc = nc_create(..).
Height, Temp, and Soil are not defined.
With nc_create you create an new file, but you do not add data to an existing file, is that what you want? If you want to add a new variable you have to use ncvar_add()


Gaming the ROUGE metric for text summarization

Question: ROUGE seems to be the standard way of evaluating the quality of machine generated summaries of text documents by comparing them with reference summaries (human generated).
$$ROUGE_{n}= \frac {\sum_{s\in \textrm{Ref Summaries} } \sum_{gram_{n}\in s}{Count_{match}(gram_{n})}}{\sum_{s\in \textrm{Ref Summaries} } \sum_{gram_{n}\in s}{Count(gram_{n})}}$$
Based on the formula above, ROUGE checks only for recall so I could just generate a summary which is the concatenation of all reference summaries and get a perfect ROUGE score.
Is it always the case that ROUGE has to be considered in the light of some other metric which is related to Precision (either BLEU or some cap on length of summary)?
Comment: I just read some papers on the subject of text summarization, all of them used ROUGE-1, ROUGE-2 and ROUGE-L as their measure of performance. I also stumbled on a paper that kind of deals with gaming evaluation metrics (http://arxiv.org/abs/1603.08023) - haven't read it yet, but it was mentioned in another paper in the context of gaming evaluation metrics.
Answer:
Based on the formula above, ROUGE checks only for recall so I could just generate a summary which is the concatenation of all reference summaries and get a perfect ROUGE score.

Yes, but  typically your summarization algorithm will not have access to the reference summaries at test time. You could get a perfect ROUGE recall by simply outputting the entire text as your summary though.

Is it always the case that ROUGE has to be considered in the light of some other metric which is related to Precision (either BLEU or some cap on length of summary)?

You can compute both the  ROUGE recall and  ROUGE precision, or the combination ROUGE F1. Or set a maximum length for your summary.


The set of compatible vector fields

Question: Let us fix some smooth vector field $F(x)$ in some open submanifold $M\subseteq \mathbb R^n$, $n\ge3$. A vector field $G(x)$ is said to be compatible with $F(x)$ if the Lie bracket $[F,G]$ is a linear combination of $F(x)$ and $G(x)$, i.e.
$$
\forall x\in M \quad [F(x),G(x)]= \alpha(x) F(x)+\beta(x) G(x),
$$
where $\alpha(x)$ and $\beta(x)$ are some smooth scalar functions.
Denote by $C_F$ the set of all smooth vector fields compatible with a given vector field $F(x)$. What is the structure of the set $C_F$? Does it always contain any vector fields other than fields of the form $\gamma(x) F(x)$,
where $\gamma(x)$ is a smooth scalar function? And if so, how can one get such vector fields?
Comment: The condition at a generic point (where $F$ and $G$ are linearly independent) is equivalent to $$F \wedge G \wedge [F, G] = 0,$$ which is a system of ${\dim M} \choose 3$ first-order, quadratic differential equations in the components of $G$. At each point this equation determines a quadratic condition on the space of $1$-jets of the component functions of $g$. Locally, there is an infinite-dimensional family of solutions.
Answer: The vector fields $F(x)$ and $G(x)$ determine an integrable distribution. So you would look for $2$-dimensional foliations of $M$ whose leaves contain the trajectories of $F$. That can be done in many ways...
Note: if $G(x)$ is such a vector field, and $\phi$ is a smooth function then $\phi \cdot G$ is again.
I am not sure that it is a vector space ( closed under addition). Have you checked that?
Comment: And yes, I know the Frobenius theorem. I don't know how to "turn it inside out" :)
Comment: @AVK: such vector fields $F$, $G$ ( assuming they are independent on some small open subset) are tangent to the leaves of a foliation ( family of submanifolds of dimension $2$. The leaves have to contain the trajectories of $F$. You can look at the intergral curves of $F$ an put them together to get those leaves.  Say every trajectory starts from a certain point of $S$ ( a codim $1$ section of $M$ transversal to $F$) Now, partition $S$ into curves, and those will be the starting points for the leaves.
Comment: @AVK: the point is this $F(x)$, $G(x)$ at each point generate a two dimensional vector space (assume they are not collinear) This  family $2$-dim subspaces has to be an intgrable distribution. So if we change from $G(x)$ to $\phi G(x) + \psi F(x)$ it's still OK. And they generate the same distribution.
Comment: Yes, you are right, I confused with the case $[F,G]=0$. I'll rephrase the question.
Comment: @AVK: No worries :-) And it is a neat question, gets into foliations
Comment: "You can look at the intergral curves of F an put them together to get those leaves." - I don't see how this can be done in the general case. In terms of dynamical systems, this is the problem of finding the first integral (or its level surfaces). There are no general ways to find it, but the main thing is that its existence is guaranteed only locally in some neighborhood of a non-singular point and there are no guarantees that the first integral/foliation exists globally on M.
Comment: To make sure that this is not easy, it is enough to take any chaotic system. It is extremely doubtful that for them there exists such a global family of submanifolds of dimension 2. And if they do exist, it is completely incomprehensible how to combine the integral curves into 2-dimensional manifolds
Comment: @AVK: Indeed, there could be complications. Even $F$ can have complicated singularities. Perhaps the easier cases would shed some light.

The set of compatible vector fields


Difference of means of Binomial variables

Question: If the formula for a hypothesis test is
$$
z = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{\frac{\sigma^2_{x_1}}{n_{x_1}} + \frac{\sigma^2_{x_2}}{n_{x_2}}}}
$$
And the variance for a binomial variable is given by the equation
$$
\sigma^2 = np(1-p)
$$
Then why is the sum of the variances given by
$$
\sigma_{x_1 - x_2} = \sqrt{\frac{p_1(1-p_1)}{n_{x_1}} + \frac{p_2(1-p_2)}{n_{x_2}}}
$$
Rather than
$$
\sigma_{x_1 - x_2} = \sqrt{p_1(1-p_1)+ p_2(1-p_2)}
$$
Comment: When you divide a random variable by $n$ you divide its variance by $n^2$
Answer: The correct formulas are
$$
\text{st.dev}\left(\frac{x_1}{n_{x_1}}+\frac{x_2}{n_{x_2}}\right)=\sqrt{\frac{p_1(1-p_1)}{n_{x_1}}+\frac{p_2(1-p_2)}{n_{x_2}}}
$$
and
$$
\text{st.dev}\left(x_1+x_2\right)=\sqrt{p_1(1-p_1)+p_2(1-p_2)}.
$$
Let me derive the first formula:
$$
\text{st.dev}\left(\frac{x_1}{n_{x_1}}+\frac{x_2}{n_{x_2}}\right)=\sqrt{\text{var}\left(\frac{x_1}{n_{x_1}}\right)+\text{var}\left(\frac{x_2}{n_{x_2}}\right)}
$$
and because
$$
\text{var}\left(\frac{x}{n_{x}}\right)=\frac{1}{n^2}\text{var}\left(x\right)=\frac{p(1-p)}{n}
$$
you get the first formula.
Comment: I don't follow what you did. For a binomial random variable the Var(x) = np(1-p). Where does var(x1/nx1) come from?
Comment: @hot_whiskey I am not sure what you are asking; perhaps you didn't see the step var(X/n-Y/n)=var(X/n)+var(Y/n)? Or perhaps you did not recognize that your 'correct' formula(your 3rd) is wrong if sigma_(x1-x2) is notation for the standard deviation of x1-x2
Comment: Sorry, I don't mean to be obtuse. I don't understand the st.dev() notation, and I don't understand why the first formula you give doesn't have an n in front of p1 and p2, where everywhere I have read gives the variance equation as σ2=np(1−p). https://onlinecourses.science.psu.edu/stat414/node/71. Then, in the third equation what are you taking the variance of?


What is the signal chain for a rackmount guitar interface, audio interface, and a power conditioner?

Question: I have been thinking about upgrading my audio equipment to a rack system.
It is going to be nothing crazy, just a

Audio Interface
Guitar Interface (multi-FX processor)
Power Conditioner

Here are ones I've been looking at:
Audio Interface
Focusrite Clarett 18i20
Guitar Interface
Line 6 Helix Rack Mount
Power Conditioner
Furman PL-PLUS DMC
I guess I just have a question about how to connect these pieces and what the chain of the signal would be. How would I hook up the Helix to my audio interface so that I could have everything hooked up to my monitors at once?
Comment: The power conditioner does not pass signal at all, it's just for power. So it's not part of the signal chain at all.
Comment: I'm not sure what I was thinking there. I guess, my question is how do I hook up the audio interface and the Helix both to my monitors? Would I chain the Helix through line inputs on the audio interface? I know the Helix can be plugged in via USB.
Answer: All the powered devices in your rack will be powered by the Power Conditioner - it's basically a rack mounted power-strip.
The Guitar Interface takes a guitar input and modifies it with guitar effects. It then supplies the output to any of it's output ports - quarter inch, USBMidi, etc.
The Audio Interface takes a guitar, mic or Midi Keyboard input and presents it at a USBMidi output.
There are two things missing from your setup; A Computer, to process any USBMidi information being supplied by either interface (this is optional), and an amplifier (unless you have powered monitors).
If you only play guitar, there is some duplication here. I'd bin the Audio Interface, and just use the Guitar interface. If you play keys as well, or sing, then that's where the Audio Interface comes into its own; it will supply that information to a computer for further processing.
If you do want to use both devices, then (personally) I wouldn't chain them; I'd get a rack mounted mixer, and feed four channels (stereo from the guitar rack, and stereo from your computer) into your mixer, and run outputs to an amplifier (or just run left/right from the mixer into powered monitors).
Comment: Thanks a ton. Doing more research I realized that it would not even be worth getting the rack version of the Helix as there is a native version instead that would suit my needs. I'm not gigging or anything, so no need to have a physical piece of hardware. I did however find your routing them both to a mixer idea very helpful in my understanding of all this equipment. Thanks!
Answer: The power conditioner supplies power to all your audio gear.  It isn't in the audio signal chain though.
The Helix, apart from it's other functions, seems to BE an 8-in 8-out audio interface.   So you maybe don't need another one.


Making a solidity transfer automatically without metamask

Question: I am very new to solidity. I am making a free token giver on my front end website where the user inputs his or her address and gets a free token.
The contract is linked to my web3js and everything is working well. But I can't automatically transfer tokens without having to approve the transaction on metamask. I really want to automate this process so I could implement a lot of new functions to my website.
When I interact with my contract in remix it works well but in java it has to do metamask things.
Here is my function code in javascript:
<code>function Coingiver()
    {
        const web3 = new Web3(window.ethereum);
        var slotContract = new web3.eth.Contract(abi, address);

        const addressreceive = document.getElementById("address").value;
        const addressend = "0x582077e61D52bB1Ae6FB9EcEDa1eb39153E9A269";

        window.ethereum.enable();

        slotContract.methods.transferFrom(addressend, addressreceive, 1).send({from : addressend},function(err, res)
        {
            if(err)
            {
                console.log("error", err)
                return
            }
            console.log("Hash of the transaction: ", res)
        });

        checkBalance();
    }
</code>
How do I automate this process so I don't have to approve the transaction every time?
Answer: In order for the user to send a transaction to get your coins, they will have to pay for the gas for the transactions. One reason metamask always asks the user first before sending the transacting through is to prevent sites from draining all a users ETH by repeatedly spending their ETH as gas.
If you really want to do this at no cost for the user, you could use node.js or python on the server, and have it sign and send the coins when a user fills out a form.
Note that there is a huge amount of fraud around collecting airdrops, and if you just send to any address, scammers will generate endless addresses to get coins to, and you will also pay the gas costs.
Answer: To automate any transaction from your JavaScript code, you need to sign transactions with a local private key that has ETH balance for gas fees. Then, your application manages the private key, not MetaMask.
Usually such private key is distributed through environment, secrets file, etc. and never in the source code directly - to avoid leaking important keys.
A good solution to manage this is Truffle's hdwallet-provider package.


$\operatorname{Cov}(x,Y)$ in linear regression model

Question: $\newcommand{\Cov}{\operatorname{Cov}}$Let $Y_i=a+bx_i+\varepsilon$ the simple regression model. The expression of the pearson coefficien is given by
$$\rho_{xY}=\frac{\Cov(x,Y)}{\sigma_x\sigma_Y}.$$
My question is about the interpretion of $\Cov(x,Y)$ and $\sigma_x$, since $x$ is not random.
I think that
$$\Cov(x,Y)=\Cov(x,a+bx+\varepsilon)=\Cov(x,a)+\Cov(x,bx)+\Cov(x,\varepsilon)=b\Cov(x,x)$$
$$\Cov(x,Y)=b\cdot\operatorname{Var}(x).$$
Is this correct? What is the interpretation of this result?
Comment: If $X$ is random and $\operatorname E(Y\mid X) = a+bX$ and $\operatorname{var}(Y\mid X) = \sigma^2,$ then the value of $R^2$ arising from an i.i.d. sample $(X_i,Y_i),\,\,i=1,\ldots,n$ is in general _not_ equal to the square of the correlation between $X$ and $Y,$ although it approaches that as $n\to\infty. \qquad $
Comment: If you're wondering about the interpretation of the expression $\rho_{xY} = \dfrac{\operatorname{Cov}(x,Y)}{\sigma_x\sigma_Y}, \vphantom{\frac.{\displaystyle\sum}}$ that seems to suggest that someone other than yourself used that expression. In what context did it come up? Sometimes $X$ and $Y$ are both random but $X$ is treated as non-random because what is of interest is only the _conditional_ distribution of $Y$ _given_ $X.$ In other contexts, an experimenter can judiciously choose the $x$ values and then nature provides the $Y$ values. $\qquad$
Comment: Is there any reference where I can check the proof of this asymptotic result?
Comment: The context is that $Y|x\sim N(a+bx,\sigma^2)$ and I'm trying to prove that $\rho_{xY}^2=R^2$ (coefficient of determination).
Comment: On the other hand, if you have fixed, not random, values of $x_i,$ then you can examine the distribution of $R^2,$ (which is a random variable if $Y_1,\ldots,Y_n$ are random), but I don't know what $\text{"}\rho^2\text{''}$ would mean in that case. $\qquad$
Comment: $\ldots\,$but now that you mention it, I think there may be only one sensible way to define $\rho^2$ in that last case, but then the same remark about $R^2\to\rho^2$ as $n\to\infty$ would apply. $\qquad$
Comment: My first thought is Serfling's book _Limit Theorems of Mathematical Statistics_. $\qquad$
Comment: In the case of $X$ being random, it is correct that $\operatorname{Cov}(X,Y)=b\cdot\operatorname{Var}(X). \qquad$
Comment: $\ldots\,$and BTW, there is no need to assume anything is normally distributed to get that equality. Just $\operatorname E(Y\mid X) = a+bX$ and $\operatorname{var}(Y\mid X) = \sigma^2<+\infty$ and $\operatorname{var}(X)<+\infty$ is enough. And here the import of the statement $\operatorname{var}(Y\mid X) = \sigma^2<+\infty$ is not just that the conditional variance is finite but also that that it does not depend on $X. \qquad$
Comment: I like this way of expressing that equality: $$ \frac{\operatorname E(Y\mid X) - \operatorname E(Y)}{\operatorname{sd}(Y)} = \rho\cdot\frac{X - \operatorname E(X)}{\operatorname{sd}(X)}. $$ I.e. multiply the $z$-score for $X$ by $\rho$ to get the expected (conditional) $z$-score for $Y. \qquad$
Answer: $Cov(X,Y)$ is only defined for two random variables $X$ and $Y$. The notation $x$, as in $Cov(x,Y)$,  implies that the random variable is degenerated, that is, $Prob(X=x)=1$. In such a case, $Cov(x,Y)=0$. On the other hand, it is not unheard of that people use (misuse) the notation of $Cov(x,y)$ to stand for a computational formula. I suspect that this may be the case here.


Implementing Taxonomy hierarchy in View and Search

Question: I am working on a biology website that needs to make use of Drupal 7's Taxonomy system, for a biological taxonomy.
My goal is to allow users to tag all content with related species information, and provide the following functionality:

Display all nodes that are associated with a child term of the
specified taxonomy term (so if they access the "Birds" term,
"sparrow" would be included)
Include taxonomy terms in search
Allow for species nicknames, (synonyms), abbreviations and variations in spelling or singular/plural usage

I have installed the Hierarchical Select module, but am not sure it is working correctly. Or, I am not using it correctly. When I insert new terms deep in the hierarchy, they sometimes end up at the top level. There have also been intermittent errors showing in a red box. I'll add that to this post if it shows again.
Example taxonomy
<code>Animalia
- Chordata
-- Aves
--- Passeriformes
---- Passeridae
----- Passer
------ Passer domesticus (primary species term)
------- Common sparrow   (synonym)
------- English sparrow  (synonym)
------- House sparrow    (synonym)
------- ENSP             (synonym/abbreviation)
------- House sparrows   (synonym/plural)
</code>
In the above example, I need users to see English Sparrow content when they view the Taxonomy page for Aves, as well as the Taxonomy page for "ENSP" if the node has been tagged with Passer domesticus or any of its child terms.
Ideally, a basic search for any of the child terms for Passer domesticus would also return a list of nodes tagged with any of the term's siblings.
I am open to configuring a View for this, if Search is not possible.  All advice is appreciated.
Similar questions from others:
http://drupal.org/node/1792256
How to have parent taxonomy terms include their children terms' nodes?
How to show the parent and child taxonomy term?
https://stackoverflow.com/questions/5910071/views-show-child-taxonomy-terms
Comment: Have you consider using the https://www.drupal.org/project/taxonomy_tree_select ? I
Answer: I would look into using Search API for this and build up facets using the Facets API.
Once you have Search API installed you need to create a search index, choose which fields to index and what facets to create.
With the search index in place you can from Views create a Search view that uses your index as base.
Place the facets where you want them and tell them to show up at your search views url.
There is a guide that explains how to setup Search API here. In the guide they use Apache Solr as backend for the search index. That is not necessary, you can use the database as backend but I recommend using Apache Solr as backend as I've stumbled on a few bugs in the database backend which work with Solr.
You would be able to do searches such as this:

Good luck
Comment: This answer mentions some modules but doesn't address hierarchy specifically.
Comment: @DanielFeder Facets API provides facets for hierarchial taxonomy vocabularies.


Adyen Web Drop In OnChange fires more than once

Question: I have the Adyen Web Drop In integrated, which works fine. Now I want to add tracking so I can see when a customer changes the payment option or payment info. I use the OnSelect and OnChange event for this.
Problem:

OnChange fires immediately after OnSelect only when it's the first time the payment option is selected. When the page and thus the Drop In is loaded it fires in order: OnSelect, OnReady, OnChange. So I can't check if the component is done loading.
By default Ideal is selected. When you then select "Card" it fires OnSelect again and then OnChange 3 times. This also only happens the first time you select the "Card" payment option.

Is there a solid way to either:

only track user changes?
preload the options underneath a payment option, so that the changes don't get fired when the form is loaded?
wait on form load when changing payment option selection

=============== CODE ===============================

const configuration = {
  environment: this.model.environment || 'test',
  clientKey: this.model.clientKey,
  locale: this.model.locale,
  showPayButton: true,
  session: {
    id: this.model.sessionId,
    sessionData: this.model.sessionData,
  },
  onError: this.onFailed,
  paymentMethodsConfiguration: {
    ideal: {
      showImage: true
    },
    card: {
      hasHolderName: true,
      holderNameRequired: true,
    },
  },
  // https://github.com/Adyen/adyen-web/blob/master/packages/lib/src/language/locales/nl-NL.json
  translations: {
    'nl-NL': {
      payButton: this.$t('Veilig betalen').toString(),
      "idealIssuer.selectField.placeholder": this.$t('Kies uw bank').toString(),
      continue: this.$t('Veilig betalen')
    },
  },
  onChange: (state, component) => {
    console.log("ONCHANGE");
    if (this.tracking) {
      this.tracking.ecommerce.payment_method = state.data.paymentMethod.type;
      AnalyticsManager.getInstance().pushTracking(JSON.parse(JSON.stringify(this.tracking)));
    }
  },
  onPaymentCompleted: (result, component) => {
    this.handleCompleted(result, component);
  },
};

this.checkoutApi
  .create('dropin', {
    onReady: () => {
      console.log("OnReady");
    },
    onSelect: () => {
      console.log("ONSelect");
    }
  })
Comment: Hey Sebastiaan, can you give me some more info about your stack, and do you happen to have a working example I can have a look at ?
Comment: Hey @jlengrand thanks for your reply. I don't really have a working example that is publicly available unfortunately. I added my config object for the dropin and my create method to the question above.

In those I added logs to see which event fires when. When I load the page I see in order: OnSelect, OnReady, OnChange;

In this case the OnChange event shouldn't be tracked (because the user didn't do anything.

When I change from Ideal to Card I see in order: OnSelect, OnChange, OnChange, OnChange.

In this case the 3 OnChange events shouldn't be tracked.
Comment: Ok, I've been diving into it and I can reproduce the behaviour you're mentioning @sebastiaan. Let me come back to you about that.
Answer: I also asked this in the GitHub for the Adyen DropIn.
https://github.com/Adyen/adyen-web/issues/2011#issuecomment-1460319246
In the OnChange event you can check if the form for that payment method is valid. I now check that and only track changes if the form is valid. This way no changes are tracked on page load.
                    onChange: (state, component) => {
                    if (this.tracking, state.isValid) {
                       //only do something when form valid
                    }
                },

Besides that I use the OnReady event to set a boolean. In the OnSelect method I only track changes if the boolean is true, so that I don't track changes on page load there.
                this.checkoutApi
                .create('dropin', {
                    onSelect: (state) => {
                        if (!this.isReady)
                            return;

                        //do something
                    },
                    onReady: () => {
                        this.isReady = true;
                    }
                })

Comment: Hey! oh my, apologies, I thought I had answered your question already. That's on me. Glad the folks on GitHub were more useful. The combination sounds indeed like what you wanna be doing. nice one!
Answer: We're investigating the onChange behavior internally. In the meantime, you can use the onSelect optional configuration event handler.
const checkout = await createAdyenCheckout(checkoutSessionResponse)
checkout.create(type, {
      onSelect: obj => console.log('Payment method Selected:', obj.props.name)
}).mount(document.getElementById(type));

This will do exactly what you need, by raising an event every time a customer selects a different enabled payment method.
Comment: I got an answer in the github that I used combined with the OnReady method. See my answer below. Thanks very much for your help!
Comment: Thanks for the update! I already use OnSelect which works fine (see code above). It fires once at pageload, but it does that before the OnReady event. This way you can choose wether you do something with it or not. So the only thing I think would be nice is if:
- OnChange also fires before OnReady or not at load.
- OnChange doens't fire after OnSelect so only user changes in the form can be tracked


Is it valid to select a model based upon AUC?

Question: I have plot ROC for several models.  These models were used to classify my samples into 2 classes.
Using these commands, I can obtain sensitivity vs. specificity plots for each model:
perf <- performance(pred, "sens", "spec")
plot(perf)

Should I rely on the area under the curve (AUC) for each model to conclude which model is better?  Other than AUC, should we consider other results so as to conclude which model is better?
If yes, how to get AUC with R? Am I right in assuming that "the smaller it is the better is the classification power of the model?"
Comment: Right, should have entered this as a comment.  The AUC is a useful summary measure once you've finished fitting the model.  It is not the ideal objective criterion for selecting the model (likelihood should be used for that).  It does not provide a rational basis for a decision threshold as it assumes that utilities are data driven instead of subject driven.
Comment: @Frank I see, maybe we have a term problem. 1. For me as ML, model=anything which predicts the response and modelselection=choose the model whose predictive power is the best (as long as the complexity is not too high etc.) 2. I do not get the point regarding the decision threshold: If one calculates a weighted ROC on basis of a cost-benefit matrix one can determine the optimal decision threshold (bound to the model, of course)... for example for the task of direct marketing (mailing).
Comment: @evdstat You actually hold two unregistered accounts: the present one, and this [older one](http://stats.stackexchange.com/users/5137/evdstat). Please, choose one, register, and we'll merge them afterwards.
Comment: I don't see what ROC curves, sensitivity, or specificity have to do with the problem at hand.
Comment: @Frank 1. this is a comment to the question, not an answer. 2. AUC, the area under a ROC-Curve is an acceptable way of measuring the predictive power of a classification-model and hence a measure which can be used in the process of model - selection. The ROC itself visualizes the AUC and allows the calculation of a decision threshold.
Comment: It seems only utilizing likelihood for model selection limits the modeling toolbox (when we are talking predictive models in the data mining realm - the OP does not mention model type, purpose etc.). For example - how to calculate -2LL with gradient boosted decision trees?
Comment: The likelihood provides various optimality properties.  Decision thresholds cannot come from data.  They must come from the larger setting and loss/utility function.  Use of ROC to develop a classifier is at odds with decision theory.  For direct marketing you can compute the predicted risk for each person and use a lift curve to find the cutoff that gives the best bang for the buck.  No ROC needed or helpful.
Answer: AUROC is one of many ways of evaluating the model -- in fact it judges how good ranking (or "sureness" measure) your method may produce. The question whether to use it rather than precision recall or simple accuracy or F-measure is only depending on a particular application.
Model selection is a problematic issue on its own -- generally you should also use the score you believe fits application best, and take care that your selection is significant (usually it is not and some other factors may be important, like even computational time).
About AUC in R -- I see you use ROCR, which makes nice plots but it is also terribly bloated, thus slow and difficult in integration. Try colAUC from caTools package -- it is rocket fast and trivial to use. Oh, and bigger AUC is better.
Answer: As, mbq wrote, the answer to whether you should use AUC depends on what you are trying to do.  Two points that are worth considering:
AUROC is insensitive to changes in class distribution. It places even emphasis on the different classes, which means it can poorly reflect an algorithm's performance if there is a big imbalance in the distribution of classes.  On the other hand, if you are more interested in identifying characteristics of the classes rather than their prevalence, this is a strength.
AUROC does not capture the different costs of different outcomes and it is seldom the case that you care equally about false positives and false negatives.
I find AUROC sensible.  The curves easy to read: they are like an intuitive version of a confusion matrix.  But it is important to know what we're reading and what's left off.
See also: Evaluating and combining methods based on ROC and PR curves
Answer: As you are using ROCR, you can get the point of the ROC curve that maximizes the area and use this to determine the corresponding threshold:
my_prediction <- predict.gbm(object = gbm_mod, newdata = X, 100)
pred <- prediction(my_prediction, Y)
perf <- performance(pred, 'tpr', 'fpr')

r <- rev((as.data.frame(firstname.lastname@example.com)*(1-as.data.frame(firstname.lastname@example.com)))[,1])
threshold <- as.data.frame(firstname.lastname@example.com)[which(r==max(r)),1][1]

You can think of this optimization simply as the point that makes the largest possible rectangle under the ROC curve.
Comment: Although this is correct and helpful in general, frankly, I do not see how this answers any of the OP's questions. Aside, it seems to me that the call to "rev" should be removed.
Comment: Oh, I forgot: Welcome to the site :).
Comment: Hmm, after re-reading the question, I thought he was asking how to get the optimal threshold for classification, when he was just asking how to use ROC/AUC.  Oh well, but you need the rev() because y*(1-x) "reverses" the x axis.
Comment: As far as I see, the OP asked how to select a model based on AUC (and whether this is ok). When determining a threshold solely based on the curve, one can also stick to the AUC. I applied your function on `data(ROCR.simple)` and found that with and without rev the result is correct, but without rev I get a threshold closer to the optimal point in roc-curve (0,1).
Comment: Please see earlier comments.  AUC and ROC should play no role.  The likelihood function plays a role for a reason, and there are generalized $R^2$ measures based on log likelihood.
Comment: @FrankHarrell when you say likehood function, do you mean the likelihood of the parameters given the data? doesn't the use of the likelihood function in this case helps to determine which model fit the data best?
Comment: Since the likelihood is a great optimality criterion it is also the best basis for judging information content (along with its variances such as generalized $R^2$ and AIC).


A smartphone was purposefully dropped from a height of 10m to test its durability from external physical impact.

Question: A smartphone was purposefully dropped from a height of 10m to test its durability from external physical impact. (The smartphone 100% damaged during a drop test). It has been shown that 80% of the damages occur in the glass screen, while the other 20% occur in the battery. A number of smartphones were tested and the tests were independent. Find the probability that the first battery damage happens on the third trial or later.
I believe we're supposed to use some type of random variable formula for this, but I don't know if it's Bernoulli or geometric.
Answer:
You can solve this without knowing the name of the random variable.
Consider the complement event: the first battery damage happens on the first or second trial. If you find the probability of this event, you can answer the original question.
What is the probability the first battery damage happens on the first trial? (You just need the first trial to have battery damage.)
What is the probability the first battery damage happens on the second trial? (You just need the first trial to have screen damage, and then the second trial to have battery damage.)
Answer: Probability of screen damage is 80% or ${4\over 5}$
Probability of battery damage is 20% or ${1\over 5}$
Probability of battery damage in $i^{th}$ trial is $ P_i = {({4\over 5})}^{i-1}\times {1\over 5} = {4^{i-1}\over 5^i} $
Hence to find probability of battery damage on 3rd trial of later we have to add $P_3+P_4+P_4+{\cdots}+P_{\infty}$
$$P={4^{2}\over 5^3}+{4^{3}\over 5^4}+{4^{4}\over 5^5}+ {\cdots}$$
$$P={4^{2}\over 5^3}\times {1\over {1-{4\over 5}}} $$
$$P={4^{2}\over 5^2}$$
$$P={16\over 25}$$


Trying to make an Exit Button

Question: I have some code that I'm not sure about, I thought it worked on an old project but not sure where I've messed it up at.
Private Sub Form1_FormClosing(sender As Object, e As FormClosingEventArgs) Handles MyBase.FormClosing
    If MessageBox.Show("Are you sure?", MessageBox.Show(vbYesNo)) = Windows.Forms.DialogResult.No Then e.Cancel = True
End Sub

This is my giant error:
Error   1   Overload resolution failed because no accessible 'Show' can be called without a narrowing conversion:
    'Public Shared Function Show(owner As System.Windows.Forms.IWin32Window, text As String) As System.Windows.Forms.DialogResult': Argument matching parameter 'owner' narrows from 'String' to 'System.Windows.Forms.IWin32Window'.
    'Public Shared Function Show(owner As System.Windows.Forms.IWin32Window, text As String) As System.Windows.Forms.DialogResult': Argument matching parameter 'text' narrows from 'System.Windows.Forms.DialogResult' to 'String'.
    'Public Shared Function Show(text As String, caption As String) As System.Windows.Forms.DialogResult': Argument matching parameter 'caption' narrows from 'System.Windows.Forms.DialogResult' to 'String'

Comment: Is this [vb.net] or [vba]?
Comment: It is just a simple calculator I'm attempting to make in vba
Answer: why is there MessageBox.Show(vbYesNo) inside instead of just vbYesNo?
edit:
to be less snide: there is no overload of Show which takes a string and a DialogResult as parameters, nor one which takes a single MsgBoxStyle argument.
there is also no overload which takes a string and an MsgBoxStyle constant.
you probably mean
MsgBox("Are you sure?", vbYesNo)

or
MessageBox.Show("Are you sure?", "Caption", MessageBoxButtons.YesNo)

Comment: my attempt at getting it to work, I've tried it without and Ill keep trying but it was just last attempt at something I thought I've seen before.

Trying to make an Exit Button


Comparing outliers in two distributions

Question: I apologize in advance as I am not well-versed in statistics, but I hope that this question makes sense.
I have 2 populations which are normally distributed and have a near-identical mean. I would like to know if the upper quantiles (say >0.99) differ - ie, does one population extend further than the other, and how can I see if the difference is significantly different?
One approach I have thought to take is to subset my dataset on just the upper quantiles, then plot the ECDF, but I don't know that this is a correct approach.
Comment: It depends on whether the two populations have the same distribution. Difficult to know what 'nearly identical' might mean. Then the answer depends on sample sizes; as normal samples become larger, there is more chance to see values far into the right tail. The distribution of the max may be hard to handle, so you should not ask for 99th percentile unless you have a very large sample. Maybe 95th of 97th percentiles would be safer. // Once you find the difference in 95th percentiles for two samples, you could simulate to see if they are unusually far apart.
Comment: @BruceET Thanks for the response - I agree that as the sample size becomes larger, there is more chance to see values that extend further into the right tail. I've randomly downsample my dataset so that each population has the same number of events (Approx 400,000 events).

I am not sure how to test that the distributions are non-identical. They are both normal distributions. I have limited knowledge of how one tests if two distributions are identical, so I apologize for this vague statement.
Comment: @BruceET Regarding your comment 'once you find the difference in 95th percentiles for two samples, you could simulate to see if they are unusually far apart': how might one go about this? Can I subset on the 95th percentile for each population, then simply visualize the results as a boxplot or ECDF?
Comment: Not exactly sure I understand what you're proposing. Maybe my answer will persuade you that simulation is not too difficult. Or the note will encourage you to see if your quantiles are nearly normal. Maybe so with samples of size 400,000 and you use 95th percentiles Otherwise @whuber has suggested nonparametric tests..
Comment: Normal probability plot of 95th percentiles from 100 normal samples of size 400,000 has points very near a straight line. So 95th percentiles of sample of 400,000 may be close to normally distributed. // Even 99th percentiles not bad.
Answer: Two samples of size 1000 from the same normal population.
set.seed(2022)
x1 = rnorm(1000, 50, 7)
x2 = rnorm(1000, 50, 7)
q1 = quantile(x1,.95); q1
     95%
61.17923
q2 = quantile(x2,.95); q2
     95%
61.98871
dq = abs(diff(c(q1,q2)));  dq
      95%
0.8094812

Is this an unusually large discrepancy?
set.seed(106)
m = 10^5;  dq.95 = numeric(m)
for (i in 1:m) {
 x1 = rnorm(1000, 50, 7)
 x2 = rnorm(1000, 50, 7)
 q1 =quantile(x1,.95)
 q2 =quantile(x2,.95)
 dq.95[i] = abs( diff(c(q1,q2) ))
 }
mean(dq.95 >= dq)
[1] 0.22008

No. Not unusually large; about 22% of such comparisons
have 95th percentiles farther apart.
hist(dq.95, prob=T, col = "skyblue2")
abline(v = dq, col="red")

Note: As you might guess from the histogram, the
distribution of the 95th percentile of a sufficiently
large sample is approximately normal. The variance gets
larger for percentiles in the far tails. This CLT for
quantiles (except the min and max) a fundamental
result in the theory of order statistics. Depending
on the circumstances of your project, it might be
worth your while to see if your samples are large
enough to use this asymptotic result.
Comment: Relying on the Normality assumption is contraindicated here, because it's so unlikely this assumption would be reliable out in the tails.  Use a nonparametric test.

Comparing outliers in two distributions


Ethereum truffle MetaCoin fails with 'intrinsic gas too low'

Question: Pretty much as stated in the title, after running truffle test I get this:
<code> TestMetacoin
    1) "before all" hook: prepare suite

  Contract: MetaCoin
    ✓ should put 10000 MetaCoin in the first account
    2) should call a function that depends on a linked library
    > No events were emitted
    ✓ should send coin correctly (6078ms)

  2 passing (6s)
  2 failing

  1) TestMetacoin "before all" hook: prepare suite:
     Error: Intrinsic gas too low
      at Object.InvalidResponse (/usr/lib/node_modules/truffle/node_modules/truffle-contract/node_modules/web3/lib/web3/errors.js:35:16)
      at /usr/lib/node_modules/truffle/node_modules/truffle-contract/node_modules/web3/lib/web3/requestmanager.js:86:36
      at XMLHttpRequest.request.onreadystatechange (/usr/lib/node_modules/truffle/node_modules/web3/lib/web3/httpprovider.js:119:13)
      at XMLHttpRequestEventTarget.dispatchEvent (/usr/lib/node_modules/truffle/node_modules/xhr2/lib/xhr2.js:64:18)
      at XMLHttpRequest._setReadyState (/usr/lib/node_modules/truffle/node_modules/xhr2/lib/xhr2.js:354:12)
      at XMLHttpRequest._onHttpResponseEnd (/usr/lib/node_modules/truffle/node_modules/xhr2/lib/xhr2.js:509:12)
      at IncomingMessage.<anonymous> (/usr/lib/node_modules/truffle/node_modules/xhr2/lib/xhr2.js:469:24)
      at endReadableNT (_stream_readable.js:974:12)
      at _combinedTickCallback (internal/process/next_tick.js:74:11)
      at process._tickDomainCallback (internal/process/next_tick.js:122:9)

  2) Contract: MetaCoin should call a function that depends on a linked library:
     AssertionError: Library function returned unexpected function, linkage may be broken: expected 0 to equal 20000
      at test/metacoin.js:25:14
      at process._tickDomainCallback (internal/process/next_tick.js:129:7)
</code>
My genesis block is as follows:
<code>{
    "nonce": "0x0000000000000042",
    "difficulty": "0x4000",
    "alloc": {
            "0xfd0ceb7965a0e4c3146c1f901fbab9668653af6b": {
            "balance": "325406000000000"
        }
    },
    "mixhash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "coinbase": "0xfd0ceb7965a0e4c3146c1f901fbab9668653af6b",
    "etherbase" : "0xfd0ceb7965a0e4c3146c1f901fbab9668653af6b",
    "timestamp": "0x00",
    "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "extraData": "0x",
    "gasLimit": "0x3d0900"
}
</code>
And my truffle.js is like this:
<code>module.exports = {
  networks: {
    development: {
      host: "localhost",
      port: 8545,
      network_id: "*", // Match any network id
      gas: 1000000
    }
  }
};
</code>
I fail to understand what is wrong with gas at these settings. Even if noone has the exact answer, some additional explanation of the gas limits (not the gas itself) would be helpful.
Answer: I ended up changing genesis block gasLimit to
<code>"gasLimit": "0x2FEFD800",
</code>
config for homestead block to 10 (apparently otherwise there are issues when working with geth):
<code>"config": {
            "homesteadBlock": 10
    },
</code>
and truffle.js gas setting to 5000000.
Tests pass.


Shortening of formulas (from using IF, MAX)

Question: Is there a way to shorten any of these these formulas? Processing them seems to take a very long time (up to 15 minutes) given the time needed to recalculate the data.
This  link provided is for the file. The headings in the file are for the respective formulas below.
<code>latest 24 mth data =IF(C4>=EDATE(MAX($C$4:$C$1048576),-23),"Latest Months","Consol. Years")

Past 7 Years =IF(D4>=MAX($D$4:$D$1048576)-7,"Past 7 years","")

Latest Season =IF(AND(MONTH(MAX($C$4:$C$1048576))>=4,MONTH(MAX($C$4:$C$1048576))<=10),IF(AND(D4=MAX($D$4:$D$1048576)-5,E9="W"),"Latest Seasons",IF(D4>=MAX($D$4:$D$1048576)-4,"Latest Seasons","")),IF(D4>=MAX($D$4:$D$1048576)-4,"Latest Seasons",""))

Latest Year =IF(D4>=MAX($D$4:$D$1048576)-4,"Latest yrs","")
</code>
Comment: Well, for one thing, you could make a reference cell somewhere to contain the calculation for `MAX($D$4:$D$1048576)` and `MAX($C$4:$C$1048576)`. Those would thus be calculated only once and shorten all your formulae by quite some characters.
Comment: As we can't really audit a formula without the data for us, I suggest two things: provide a file with the above formulas and similar dummy data, or tell us exactly what the above are for. I'm seeing too many references to the bottom row of Excel 2010 (1048576). I will say outright that this is not necessary unless you have data until that row.
Comment: Pardon me, but how would I upload a file?
Comment: Sorry about that. I meant, upload a file somewhere and provide the link here. :)
Comment: Thanks alot! I have uploaded the link to the file and made a few adjustments to the formulas. Please take a look!
Answer: The main issue I see here is that your range is way too large. Your formulas are evaluating 900,000 more cells than they need to determine the max date. You can set the range as $C$$:$C$150000, which would cut down on the processing time.
If your range keeps expanding, consider making a named range that expands by going to the name manager and typing in something like: =OFFSET(Sheet1!$C4,0,0,COUNTA(Sheet1!$C$4:$C$150000),1)
Another option is to get the max date in one cell, then just keep referring to the value of that cell in the rest of your formulas.


PHP - Converting PDO to normal code

Question: I was wondering if someone could help me.
Im trying to integrate some code into my application, the code that i need to integrate is written with PDO statements and i have no idea how it goes.
I was wondering if someone could help me convert it.
The code is as follows
$sql = "insert into message2 (mid, seq, created_on_ip, created_by, body) values (?, ?, ?, ?, ?)";
$args = array($mid, $seq, '188.8.131.52', $currentUser, $body);
$stmt = $PDO->prepare($sql);
$stmt->execute($args);
if (empty($mid)) {
    $mid = $PDO->lastInsertId();
}
$insertSql = "insert into message2_recips values ";
$holders = array();
$params = array();
foreach ($rows as $row) {
    $holders[] = "(?, ?, ?, ?)";
    $params[] = $mid;
    $params[] = $seq;
    $params[] = $row['uid'];
    $params[] = $row['uid'] == $currentUser ? 'A' : 'N';
}
$insertSql .= implode(',', $holders);
$stmt = $PDO->prepare($insertSql);
$stmt->execute($params);

Comment: How is the PDO module not "normal code"? What exactly do you want to convert it to?
Comment: **Use the PDO and prepared statements**.
Comment: You should learn how to use PDO. Don't go back to the stone ages of the mysql extension. At least use mysqli with prepared statements, but if you do so, PDO really isn't that far off. If you really want to rewrite this using the mysql extension, there are two SQL statements in this code that you need to rewrite. Shouldn't be too hard with a little trying.
Comment: Presumably from PDO to MySQL/MySQLi
Comment: Oh yes, sorry about that ... im used to mysql statements
Comment: Codeigniter inbuilt doesn't have PDO but it can be easily implemented. Just need to write a few extra lines on db library.

The 1st link google search "codeigniter PDO" explains in detail how to implement.
Answer: You shoudl use PDO unles for some technical reason you cant. If you dont know it, learn it. Maybe this will get you started:
/*
This the actual SQL query the "?" will be replaced with the values, and escaped accordingly
- ie. you dont need to use the equiv of mysql_real_escape_string - its going to do it
autmatically
*/
$sql = "insert into message2 (mid, seq, created_on_ip, created_by, body) values (?, ?, ?, ?, ?)";

    // these are the values that will replace the ?
    $args = array($mid, $seq, '126.96.36.199', $currentUser, $body);

    // create a prepared statement object
    $stmt = $PDO->prepare($sql);

    // execute the statement with $args passed in to be used in place of the ?
    // so the final query looks something like:
    // insert into message2 (mid, seq, created_on_ip, created_by, body) values ($mid, $seq, 126.96.36.199, $currentUser, $body)
    $stmt->execute($args);

    if (empty($mid)) {
        // $mid id is the value of the primary key for the last insert
        $mid = $PDO->lastInsertId();
    }

    // create the first part of another query
    $insertSql = "insert into message2_recips values ";

    // an array for placeholders - ie. ? in the unprepared sql string
    $holders = array();

    // array for the params we will pass in as values to be substituted for the ?
    $params = array();

    // im not sure what the $rows are, but it looks like what we will do is loop
    // over a recordset of related rows and do additional inserts based upon them
    foreach ($rows as $row) {
        // add a place holder string for this row
        $holders[] = "(?, ?, ?, ?)";

        // assign params
        $params[] = $mid;
        $params[] = $seq;
        $params[] = $row['uid'];
        $params[] = $row['uid'] == $currentUser ? 'A' : 'N';
    }
    // modify the query string to have additional place holders
    // so if we have 3 rows the query will look like this:
    // insert into message2_recips values (?, ?, ?, ?),(?, ?, ?, ?),(?, ?, ?, ?)
    $insertSql .= implode(',', $holders);

    // create a prepared statment
    $stmt = $PDO->prepare($insertSql);

    // execute the statement with the params
    $stmt->execute($params);

PDO really is better. It has the same functionality as MySQLi but with a consistent interface across DB drivers (ie. as long as your SQL is compliant with a different database you can theoretically use the exact same php code with mysql, sqlite, postresql, etc.) AND much better parameter binding for prepared statements. Since you shouldnt be using the mysql extension any way, and MySQLi is more cumbersome to work with than PDO its really a no-brainer unless you specifically have to support an older version of PHP.
Comment: Hey, thanks for that .... yeh the reason i cant use PDO statements for this project is that im using a framework that doesnt use PDO
Comment: Sounds like you shouldnt be using that framework - which one is it?
Comment: Umm CodeIgniter does indeed have a PDO db driver.
Comment: Ohhh, hahaha well, ill have to look into that ... thanks for your hlep

PHP - Converting PDO to normal code


Error: getaddrinfo ENOTFOUND error when making an HTTPS/HTTP request

Question: Here's the code of my AWS Lambda function:
console.log('Loading function');

const AWS = require('aws-sdk');

const https = require('https');

const data = JSON.stringify({
secretKey:"someSecretKey",
userType:"someCoolUser",
})

const options = {
 hostname: "backend-staging.programmingpathshala.com:8080/rest/admin",
 path: '/sendEmailToUsers',
 method: 'POST',
 headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length
 }
}

 exports.handler = function(event, context, callback) {
    var dataString = '';
    const req = https.request(options, function(res) {
      res.on('data', chunk => {
        dataString += chunk;
      });
      res.on('end', () => {
                callback(null,dataString);
      });
    });
    req.write(data)
    req.end();
    req.on('error', (e) => {
      console.error(e);
    });
   }

When I test my API using postman it works fine. But when it is called from lambda function I get the following error:

Also, When I run the same API using ngrok and use its link in my lambda function it works then too.
Comment: It worked. Thanks. Please write that as the answer. I'll accept.
Comment: Are you sure your api works with `https`?
Comment: using ngrok it does. but it's hosted with HTTP. Also, I tried both HTTP and HTTPS getting the same error.
Comment: I see. Then is your `hostname` valid?. Usually, it would be just `backend-staging.programmingpathshala.com`, then you would have `port` option in `options` and `path` would be `/rest/admin/sendEmailToUsers`.
Answer: Based on the comments, the options should be:
const options = {
 hostname: "backend-staging.programmingpathshala.com",
 port: 8080,
 path: '/rest/admin/sendEmailToUsers',
 method: 'POST',
 headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length
 }
}


Is there any way to find average of song's amplitude?

Question: The question is a bit confusing, I know. Let me explain it. As you know song volume is represented by amplitude. When a singer sings loudly, the amplitude is high. For slow music, the amplitude is low.
I need to find the average amplitude of a song and a value for the average amplitude. According to these values I will try to decide what kind of music it is.
Does any one have any advice for me?
Comment: Volume is not represented by wavelength. Wavelength is exactly what is sounds like, the length of a sound wave, which is influenced by frequency, or pitch. If you are looking for a measure of the average volume, you are talking about amplitude; however, I doubt that an analysis consisting purely of amplitude would be able to tell you with any certainty what style of music it is.
Comment: Thanks for your answer @tpburch Do you know any way to do the same thing for amplitude of songs??
Comment: Without looking it up, I'd guess that there would have to be software to find something like the root-mean-square of the waveform.  I don't think this tells you anything useful about the style, though, particularly since your analysis would be almost entirely dependent on how the recording was made, not on the style of the music.  You would have to have complete control over the recordings, not just the analysis.  Even then, I'm not sure how you plan to use the information to determine "what kind of music it is."  Could you clarify?
Answer: To measure the average sound level over time, you would use an integrating-averaging sound meter. This will give you a decibel reading representing the average amplitude over the allotted period of time. Unfortunately, these can be quite expensive.
Comment: There's plenty of free software levelling plugins about; many are capable of doing long-time average RMS measurements.
Answer: In order to decide what kind of music, you will have to do an awful lot more than take an average of either amplitude or frequency.
Have a look at this article on how Shazam works - this identifies specific songs, but a subset of the analysis used in Shazam would be able to identify genres with some accuracy.


regression with circular response variable

Question: I am analyzing some data where the variables of interests are circular (angles). I use R and the circular package.
In my dataset, every observation consists in a 2D Euclidean vector representing a movement on a plane (the length is constant, only the angle varies), together with a measure of a response to that movement, expressed as displacement in cartesian coordinates (i.e. the displacement along the x and y axis). I would like to fit a model where the initial movement (either the x-y cartesian representation or only the angle, since the length is constant) is the response variable. Ultimately, I would like to use the fitted model to estimate the initial movement vector in a related dataset where I only know the response to the movement.
My measurement of the response to the movement are affected by noise, independent over x and y dimensions, with possibly different variances and different additive biases.
To represent graphically the problem, what I want to do is estimate the direction of the original vectors (gray thick arrows in the figure below) starting from the noisy measurements (the thin black arrows).

I have tried fitting different models:

one model with both predictor and dependent variable being circular
(the angle of the initial movement vector, and the angle computed
from the (x,y) displacement)
one multivariate, linear model, with the cartesian (x,y) components of the initial movement as dependent variables, and the
measured x,y displacement as linear predictors;
one model with circular dependent variable, and linear predictor (this last one with no success)

First, I wasn't able to fit the 3rd model for some reason that I don't understand. I report here a reproducible example
require(circular)
theta <- circular(runif(500,0,2*pi),units="radians",type="angles",zero=0,rotation="counter")
rho <- rep(8,500)

# add measurement noise (different for x and y)
mX <- as.numeric(rnorm(500,0,2) + rho * cos(theta))
mY <- as.numeric(rnorm(500,-1,1) + rho * sin(theta))
linearPred <- cbind(mX,mY)

# fit model
mcl <- lm.circular(y=theta, x=linearPred,init=c(1,1), type="c-l",verbose=T)

Here is the output:
> mcl <- lm.circular(y=theta, x=linearPred,init=c(1,1), type="c-l",verbose=T)
Iteration  1 :    Log-Likelihood =  2.392981
Iteration  2 :    Log-Likelihood =  1.082084
Iteration  3 :    Log-Likelihood =  0.8013503
Iteration  4 :    Log-Likelihood =  NA
Error in while (diff > tol) { :
  valore mancante dove è richiesto TRUE/FALSE

(the last line says: missing value where a TRUE/FALSE was required).
Can anyone shed light on this? I don't understand where this error comes from.
Second question, which model would you suggest to use among the first two? Here is the code that I used for the two models
# fit sin(theta) and cos(theta) with a multivariate linear model
mmv <- lm(cbind(sin(theta),cos(theta)) ~ mX+mY)

# "circular-circular" model
angularPred <- circular(atan2(mY,mX),units="radians",type="angles",zero=0,rotation="counter")
mcc <- lm.circular(y=theta, x=angularPred, type="c-c")

I can compute the angle from the multivariate fitted values as atan2(mmv$fitted[,1],mmv$fitted[,2]). Both seems to perform similarly in terms of mean angular error. To compare the two I computed the correlation between the predicted angles and the initial angle theta (Jammalamadaka - Sarma correlation coefficient), and the multivariate model seems to perform slightly better:
fitted.mmv <- circular(atan2(mmv$fitted[,1],mmv$fitted[,2]),units="radians",type="angles",zero=0,rotation="counter")
> cor.circular(fitted.mmv,theta) # multivariate
[1] 0.9851422
> cor.circular(mcc$fitted,theta) # "circular-circular"
[1] 0.7262862

However, the distribution of residuals of the multivariate model shows a strange pattern (figure below).

Is this pattern in the residuals a problem? Can anyone gives some advice on this? Which model would you use in this case? Is there any other possible approach that you would recommend?
Any advice is appreciated, thanks!
Answer: The pattern in the residuals is not necessarily a problem. One way to check this is to simulate a set of responses from the model that you just fitted (that is, under the assumption that the model is correct), fit a new model to the results, and plot its residuals. This gives you a measure of how weird you would expect the plot to look even if nothing were wrong.
If you can reliably pick out the original model from several plots produced in this way, then you should start worrying about the residual pattern.
Comment: Thanks for your answer. I did as you suggested, and the new plot of residuals (of the model fitted to the simulated responses) does not show the weird pattern. The difference is due to the fact that the simulated set of responses have a distribution that is more uniform and not clipped between -1 and 1. Instead, the original distribution of the responses is clipped, with values near -1 and 1 being more frequent (the pdf of the sine of a uniform random angle is "U-shaped", see [here](http://atif-razzaq.blogspot.fr/2011/02/probability-density-function-pdf-of.html)).
Comment: So ultimately the problem is that the linear model would predict also impossible values of sine and cosine, above 1 and below -1. With regard to my original question, I don't see this as a big problem for the multivariate model.


python3 script fails when run as sudo

Question: When I run the python script below on a raspberry pi using the pi user, it works perfectly.
<code>#!/usr/bin/python3

from prometheus_client import start_http_server, Summary, Gauge
import random
import time
import subprocess
import json

# Create a metric to track time spent and requests made.
REQUEST_TIME = Summary('request_processing_seconds', 'Time spent processing request')

# use Gauge to record the metrics
labels = ['server_name', 'server_id']
latencyGauge = Gauge('ping_latency', 'Ping Latency in ms', labels)
downloadGauge = Gauge('download_speed', 'Download speed in bytes', labels)
uploadGauge = Gauge('upload_speed', 'Upload speed in bytes', labels)

# launch a subprocess running the speedtest command
# Decorate function with metric.
@REQUEST_TIME.time()
def process_request(id, t):
    comp = subprocess.run(['speedtest', '-s', id, '-f', 'json'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
#    print(comp)
    output_str = comp.stdout
#    print(output_str)
    res = json.loads(output_str)
    ping = res["ping"]["latency"]
    upload = res["upload"]["bandwidth"]
    download = res["download"]["bandwidth"]
    server_name = res["server"]["name"]
    print("Result for %s (%s): %d, %d, %d" % (id, server_name, ping, download, upload))
    latencyGauge.labels(server_name, id).set(ping)
    uploadGauge.labels(server_name, id).set(upload)
    downloadGauge.labels(server_name, id).set(download)

    time.sleep(t)

if __name__ == '__main__':
    # Start up the server to expose the metrics.
    start_http_server(8000)
    # Generate some requests.
    while True:
        process_request("1686", 120)
</code>
However, if I run it with sudo it fails with the following error.
<code>pi@raspberrypi:~/speedtest $ sudo ./speedtest.py
Traceback (most recent call last):
  File "./speedtest.py", line 46, in <module>
    process_request("1686", 120)
  File "<decorator-gen-1>", line 2, in process_request
  File "/usr/local/lib/python3.7/dist-packages/prometheus_client/context_managers.py", line 66, in wrapped
    return func(*args, **kwargs)
  File "./speedtest.py", line 27, in process_request
    res = json.loads(output_str)
  File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
</code>
The reason for running the script in sudo is that I want it to start up automatically by editing the /etc/rc.local
What could be the cause?
Answer: Resolved.
This root cause of the problem was that upon the very first use of the speedtest software, the user is required to accept by typing YES (or decline) twice before being able to proceed.
Since the script was expecting a JSON reply, the text presented upon the initial run was breaking the JSON parsing.
Hence the resolution is to run the speedtest without any options (default) at least once using sudo as shown below.

python3 script fails when run as sudo


Find all the polynomials f which satisfies the conditions

Question: Find all the polynomials f such that f(2t) can be written as a polynomial in f(t), i.e. for which there exists a polynomial h such that f(2t) = h(f(2t))
This question is from the book Polynomials. I have thought about this problem but could not find any proper approach to it so I saw the solution but still couldn't quite grasp it. So it would be great if anybody could further elaborate the proof or give any other proof.

Could somebody also tell me how or why is that (2^n)ant^n....a0 equal to the next step i.e. uant^n+.....(ua0+v)
Comment: OH Yeah. Damn, I didn't think in that manner. Thanks a lot.
Comment: If $f(t) = a_n t^n + \ldots + a_1 t + a_0$, then $f(2t) = 2^n a_n t^n + \ldots + 2 a_1 t + a_0$ and $u f(t) + v = u a_n t^n + \ldots + u a_1 t + u a_0 + v$.
Answer: That step is simply $f(2t) = h(f(t))$ written out with an assumed form $f(t) = a_nt^n +\dots+ a_0$ and $h(t) = ut+v$. Just write it all out and it will be clear.
Answer: If $f$ is not constant, he proved that $h$ is linear, that is $h:t\mapsto ut+v$. On the other hand, $f:t\mapsto a_nt^n+\cdots+a_1t+a_0$. So the equality $f(2t)=h(f(t))$ writes
\begin{align*}
f(2t)&=a_n(2t)^n+\cdots+a_2(2t)+a_0\\
&=2^na_nt^n+\cdots+2a_1t+a_0\\
&=h(f(t))\\
&=uf(t)+v\\
&=u(a_nt^n+\cdots+a_1t+a_0)+v\\
&=ua_nt^n+\cdots+ua_1t+ua_0+v.
\end{align*}


What do you think of attempting to tune an Acoustic 12-string one full step up?

Question: Many like to tune a 12 string 1/2 to 1 full step down. I understand it reduces tension, reduces breaking the high G string and/or allows you to run thicker gauges for a fuller sound.
However I normally run the capo on the 2nd fret. It works but I always have problems getting the capo fully set on the 12 string, and I don't like the sound as much as playing the guitar normally. So I'd like to tune it 1 full step up and then not have to use the capo.
So I've ordered an extra light set of strings that start at .008. I hear this will work but that I may have to go with two standard G strings rather than the high G (which would actually be playing high A)
But to experiment I also ordered a few individual .007 strings, hoping I can get to that octave high A, and also hoping to reduce chances of breaking the high E (one step up to F) strings too.
I understand the sound will be weaker. This is a slight concern for me finger picking but when I strum with standard strings the my guitar is actually too loud for my voice, so I think I probably won't mind a quieter guitar overall.
And I'm very much a thumb strummer. I don't use a pick and I don't bend the strings or move them up the fretboard too far. I hope to get more versatile at some point but will worry about that later.
So I wonder your thoughts as I go into this experiment? Meanwhile I've tried a few capos and never been happy, but I have ordered a new 12-string specific capo as well, as I may be going back to one if the experiment fails, and I still like to use a capo to go even higher if need be.
I'd appreciate your thoughts?
Comment: Risky, I'd go for the capo option instead of risking your strings.
Answer: I recommend against doing this. It's been my experience that the octave G string is the most likely to break, and tuning it up would make that even more of a problem. I agree that transposing (as Tim has suggested) or simply using a capo would be best. If tuning the guitar up is the only option, I highly recommend getting a different gauge of strings and having the truss rod adjusted. The tension on a 12-string neck is enormous and changing it could cause problems with the guitar down the road.
I've had to try several capos until I found one that works on my 12-string. In my case, the Paige 12-E capo did the trick, but several musicians I spoke to swear by the G7 12-string capo. Keep looking until you find one that works.
If the guitar is too loud for you, consider getting a soundhole blocker, or using a lighter pick; but learning to play more quietly is your best long-term solution.
Also: If a 12-string at standard tuning isn't working for you, perhaps you need a different instrument altogether. Maybe check out something like a tenor guitar, or an 8-string tenor ukulele. There are also travel guitars that have shorter necks and will be quieter. It's also possible that this particular guitar you have is simply very loud.
Comment: I love the look of the other capos, but I ended up getting this [one](http://www.shubb.com/store/capos/3.html) from Shubb.   I did get the strings too but as others also mentioned the quality Capo seems to do the trick. I think my issues with the sound primarily came from only using lesser capos that didn't fully lock down the strings. I may try the lighter strings and tuning one day just to hear the difference for myself, but this works for now.  tks.
Comment: Thanks. I'm forget exactly what I ordered but it is a 12 string specific capo that seems to be well rated. If it doesn't work though I'll give the others a go.
Answer: A set of .008s will do the job and a .007 for the octave G will be o.k. What I don't understand is why it needs to be tuned non-standard, or up a tone. Changing the key might solve a vocal range problem, or is it that you prefer particular (open) chord shapes? What I mean is instead of playing a song in C, play it in D; instead of D, play it in E, etc. You could even tune down a semitone, and instead of playing in A, play in C.
Comment: Yeah I play most things in D for my limited vocal range. Actually it's not necessarily D, but the chords I use are the standard ones; D G Am Em C G A E and all the basics. If I try to do songs one key up, say hitting Bm instead of Em, it means I'm making a bar chord and when I'm picking I never have enough strength to pull off all the notes, so essentially my index finger is the worst capo!    Just many of my compositions sound best to me one full step up.
It could be just that I don't have the right capo, but no capo I've ever seen quite seems to have the same warm sound as the full guitar.
Answer: You can do this, but I would refrain from doubling the g- (i.e. actually a-) string an octave up then. I have tried playing a .007 tuned up to a' for a while, but it was short-lived and really pretty painful to play. (And that even was even on the shorter Les Paul scale!)
Just use two identical .016 strings (unwound) and tune them both to a. If you otherwise keep the total tension low enough, and I think .008 should be ok for that, then I see no reason why tuning up a whole step shouldn't be fine.


How to test keep-alive is working on client end

Question: What are some different ways/tools to verify that keep-alive is working on the server from the client's end?
Answer: As Ron Garrity says, you can use Curl like this:
curl -Iv http://www.aptivate.org 2>&1 | grep -i 'connection #0'

And it outputs these two lines if keep-alive is working:
* Connection #0 to host www.aptivate.org left intact
* Closing connection #0

And if keep-alive is not working, then it just outputs this line:
* Closing connection #0

The output Connection ... left intact proves that the server did not close the connection, and it is available for the client to reuse. It's up to the client to decide whether it actually wants to reuse the connection or not. You can demonstrate it with Curl by listing the same URL twice on the command line
curl -Iv http://www.aptivate.org --next http://www.aptivate.org 2>&1 | grep -i '#0'

in which case it will give output something like:
Re-using existing connection! (#0) with host ...

Answer: You could try
ab -n 500 -c 5 -k http://www.domain.com/

and look via top, if 5 workers are constantly serving the requests keepalive should work (-k) switch.
Do the same without -k and see the difference.
Cheers Izac
Comment: that's what i wound up doing, though i read that curl can tell you this too.  if it says "Connection #0 to host www.example.com left intact", it means keep-alive is on.


SQL Server backup with C#

Question: I have a console app for backup my SQL Server database.
I used this code:
static void CreateBackup(string connectionString, string databaseName, string backupFilePath)
{
    backupFilePath = backupFilePath + "\\" + databaseName + ".bak";
    backupFilePath = @"" + backupFilePath;

    var backupCommand = "BACKUP DATABASE @databaseName TO DISK = @backupFilePath";

    using (var conn = new SqlConnection(connectionString))
    using (var cmd = new SqlCommand(backupCommand, conn))
    {
        conn.Open();

        cmd.Parameters.AddWithValue("@databaseName", databaseName);
        cmd.Parameters.AddWithValue("@backupFilePath", backupFilePath);

        cmd.ExecuteNonQuery();
    }
}

When I start the app, I always get the same error:

System.Data.SqlClient.SqlException (0x80131904): The parameterized query '(@databaseName nvarchar(4000),@backupFilePath nvarchar(5))BACKUP' expects the parameter '@databaseName', which was not supplied.

How can I fix this?
Comment: is it possible the `databaseName` variable being passed into your function is `null`?
Answer: The database name in BACKUP DATABASE is SYS Name, and cannot be a string variable, you can build the backupCommand to be:
var backupCommand = "DECLARE @cmd varchar(max) = 'BACKUP DATABASE ' + @databaseName + ' TO  DISK = ''' + @backupFilePath + ''' '
                     EXEC (@cmd)"

SQL Server backup with C#