Evaluate prediction model - K Fold Cross Validation

Question: I have a dataset of 240 samples data, with 12 variables (independent variables). From these 12 variables, i would like to identify significant variables for prediction. I perform Gamma analysis, since the data is highly positive skewed.
Here's what I am doing currently:
1. Split the data set into 70% (training data) and 30% (test data)
2. I used 70% data (168 data) to build prediction model using Gamma analysis. I run the analysis few times by excluding each variable at one time to get the best final model.
3. Then validate the final model using the remaining 20% data.
My problem: When should I use K-fold cross validation? Is it when building the prediction model using 70% data, or after i get the final model and use k-fold CV on the remaining 20% data?
Answer: Traditionally, you do three steps of "tuning", so you need to split your sample into three parts: a training set, a cross-validation set and a test set.
Training (~60%)
In training, you simply estimate your model, but you don't make any changes to the model based on the results (accuracy, goodness of fit) from the training data to avoid overfitting the training set.
Cross validation (~20%)
After training your model, you can tune it - vary hyperparameters, remove features, or even select between different models - based on its performance on the cross validation set.
As an example, let's say you want to test which variables to include and which to leave: You specify three different variable combinations (three different models). You train all of them using your training set. Then you evaluate all of them using the cross validation set and select the one that performs best on the CV set.
K-fold CV
If you are interested in doing k-fold validation, you repeat exactly what's written above, with one major difference: instead of hard-selecting the 60% and 20% for your training and CV sets, you run the training and validation procedures K-times, each time selecting a different random subsample for training and cross validation. Then you get a set of K results (accuracy, goodness of fit) that you can average to get a more robust estimate of your model's performance.
E.g., if you do 10-fold CV, you'd run it 10 times, and each time you'd randomly sample 10% of your data to be a cross-validation set, with the rest being a training set.
Test set (~20%)
After tuning the model and/or selecting the best one, you can test it using the test set. This is data that the model has not seen yet and you shouldn't make any changes to the model based on the test set. This is the very last stage of building the model, only used to evaluate your final model, not to tune it any more (you don't want to overfit your test set).
If doing k-fold CV, you still have to leave out a test set that is separate from your training/CV set you are sampling from.
Putting it all together
So in your case, you have $N=240$ and the number of variables is $12$. So the first split of the data would be training/CV (70-80%) and test (20-30%). Which in your case would be $168-192$ for training/CV and $48-72$ for test. Then, in selecting the variables to include, for each model (combination of variables), do K-fold CV as follows:

Split your training/CV set into K equal (random) subsets.
Estimate your model K times, each time leaving out one of the K subsets.
Cross-validate each estimate with the subset that was left out.
Pool your cross-validation results across all the K estimates.

Then pick the model that performs best in CV (on average). Evaluate it on the test set. Don't change it any more.
Comment: Hi @metjush. For Gamma analysis, do you know what K results should I refer to? You did mention (accuracy, goodness of fit) --> is it the deviance value?
Comment: If by gamma analysis you mean gamma regression (or GLM with family Gamma), then yes, looking at deviance residuals could be one way of estimating fit. Or the dispersion parameter. Or log-likelihood. I am not very familiar with gamma regressions, but that's where I would start.

Which software/package are you using?
Comment: Hi @metjush. For the last step, how to evaluate the final model on the test set?.
Comment: @Shahida after you have your final model done and estimated, have it make predictions on the test set that you left out. compare prediction accuracy (F1, AUC, RMSE, depending on your specific problem...) with the ground truth. report this accuracy as your model's accuracy.
Comment: I would be dubious of a single hold-out sample like this procedure to estimate "model performance".  I would prefer nested cross-validation (K-folds probably) to capture the fitting process and sensititvity to subsample hold-out selection.
Answer: Why can we use bootstrapping instead of split so that we can use the whole date for internal validation esp in view of effective small sample size given the number of predictors
Comment: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Comment: This does not really answer the question. If you have a different question, you can ask it by clicking [Ask Question](https://stats.stackexchange.com/questions/ask). To get notified when this question gets new answers, you can [follow this question](https://meta.stackexchange.com/q/345661). Once you have enough [reputation](https://stats.stackexchange.com/help/whats-reputation), you can also [add a bounty](https://stats.stackexchange.com/help/privileges/set-bounties) to draw more attention to this question. - [From Review](/review/late-answers/333443)


How does Flink watermark exactly determines an event is late?

Question: I am reading https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/event_timestamp_extractors.html#fixed-amount-of-lateness, looks it is saying that if t_eventime < t_watermark(less than),then the event is determined as late.
How about when the event time is equal to the watermark? If t_eventime = t_waterwark, then this event is not late?
I have always thought before that if t_eventime <= t_watermark, then the event determined as late.
Could you please show me the code where the determination happens, thanks.
Comment: could someone take look? Thanks!
Answer: Indeed, it seems the code to determine if an event is late in that case is using a <= comparison, so an event is considered late if its timestamp + allowed lateness is before or equal to the watermark, i.e. if its lateness is >= 0:
    protected boolean isElementLate(StreamRecord<IN> element) {
        return (windowAssigner.isEventTime())
                && (element.getTimestamp() + allowedLateness
                        <= internalTimerService.currentWatermark());
    }

Now for completeness sake, note that the value of the watermark itself for the strategy you are referring to, is computed as the latest timestamp ever seen, minus the "out of orderness", minus 1.
    public void onPeriodicEmit(WatermarkOutput output) {
        output.emitWatermark(new Watermark(maxTimestamp - outOfOrdernessMillis - 1));
    }

That means the actual lateness that is used for the comparison in the first snippet above is 1 millisecond more then what intuition might tell us, s.t.  lateness > 0 might actually be what us humans need to read for understanding what's going on.
Now those parameters are meant as estimates of how out of order we believe our data might be in the real world, maybe due to network race conditions or non aligned clocks or so, and we are typically much less precise than 1 ms when making that estimation. So it all should not matter all that much: hopefully such occurrences are rare in our data, though how many there are is typically a bit random due to the data itself.
Comment: thanks @svend for the helpful answer. `isElementLate` and `onPeriodicEmit1` both treat event time `<=` watermark as late. The key point is that in our intuition, watermark = max event time seen - lateness, but it is not true for the `onPeriodicEmit` method.
Comment: there is another similar question there: https://stackoverflow.com/questions/65527824/cant-understand-the-result-for-the-event-time-group-by-window. @svend, could you please help take a look? looks it has treated `event time = watermark` as not late


Why does shifting right on a two's complement binary number divide it by 2?

Question: I understand how logical right shift works. Given an unsigned binary number $n = a_{n-1}a_{n-2}...a_0$, digit $a_k$ contributes $a_k\times 2^k$ to the value of the $n$. after applying a logical right shift, this digit will contribute $a^k\times 2^{k-1}$ and the result will be $\lfloor\dfrac{n}{2}\rfloor$ (rounding towards zero because we lose $2^0$).
I also understand how two's complement works, but have yet to understand  how arithmetic right shift divides a negative number by $2$ and  rounds down towards negative infinity (as mentioned in Wikipedia).
Comment: Since you mention the Wikipedia article, what part don't you get? Specifically? Have you tried examples yourself?
Comment: @Somos I don't get the logic behind it. Why is adding a $1$ to the right when shifting a negative number results in dividing the number by $2$?
Answer: Let $\,W\,$ be the word length of a
bit field, $\,N=2^W\,$ be the number of different words.The two's complement choice to represent a negative integer
$\,x\,$ is $\,N+x \,$ (essentially modulo $\,N\,$). A logical right shift gives the result
$\,\lfloor \frac{N+x}2\rfloor.\,$ To convert this to an
arithmetic right shift we have to add a sign bit
 $\,\frac{N}2.\,$ Thus the result becomes
$$ \frac{N}2 + \left\lfloor\frac{N+x}2\right\rfloor =
 \left(\frac{N}2+\frac{N}2\right) +
\left\lfloor\frac{x}2\right\rfloor  = N +
\left\lfloor\frac{x}2\right\rfloor. $$ However, this is the two's complment representation
of $\,\left\lfloor\frac{x}2\right\rfloor.\,$


Relation Between Wasserstein Distance and KL-Divergence (Relative Entropy)

Question: Consider the Wasserstein metric of order one $W_1$ (a.k.a. the Earth Movers Distance). I would like to know whether it is possible to link $W_1$ and Kullback–Leibler divergence (a.k.a. relative entropy) and what this would mean intuitively. I can't find it anymore, but if I am not mistaken the following holds true for some constant $C$
$$
W_1(\mu, \nu)\le \sqrt{C\cdot \text{KL}(\nu ||\mu)},
$$
where $\text{KL}$ is the KL-divergence. My first question would be: Is the above-mentioned inequality true? Secondly, how should one interpret this estimation?
Comment: I am looking for inequalities relating W_p to KL (ideally in the direction opposite to yours), so if you find references please answer your own question and include them here. Thanks
Comment: You might want to have a look at this paper >>> https://arxiv.org/abs/1709.10219 Didn't read more than the abstract but seems relevant.
Comment: For an explicit example where the inequality fails see the example in my answer here: https://stats.stackexchange.com/a/295729/150025 in that Gaussian example, taking $w$ to be arbitrarily close to $0$ but still positive will violate the inequality stated.
Answer: This post gives inequalities for a bunch of distances, including total variation
$$\frac{1}{2}d_{TV}(\nu,\mu)<\sqrt{KL(\nu,\mu)}$$
and this says the Wasserstein distance is bounded by the total variation distance
$$2W_1(\nu,\mu)\leq Cd_{TV}(\nu,\mu)$$
if the metric is bounded by $C$.
There isn't a simple bound in the other direction, since you can make the KL divergence infinite by moving the probability off an arbitrarily small spot onto the neighbouring area, and this can be done with arbitrarily small $W_1$ distance.  For example, take two standard Normals. For one of them, set the density to zero on $[0,\epsilon]$ and to twice the existing value on $[-\epsilon,0]$. Do the opposite for the other one.  The Wasserstein distance is proportional to $\epsilon$, but the KL-divergence is infinite.
Answer: As was pointed out in the previous post this inequality is not true in general. However, there has been a lot of research on those measures $\mu$ for which it is true for all measures $\nu$. Most prominently this holds for the standard normal distribution.
Those kinds of estimates go under 'Talagrand inequality'.
Take a look at the paper of Otto and Villani and refernces therein: http://cedricvillani.org/sites/dev/files/old_images//2012/08/014.OV-Talagrand.pdf


What to do if my linear model doesn't have neither homoscedasticity or residuals normal distribution?

Question: lme.resultsA <- lme(root~factor(week)*factor(fertilizer), random=~week|factor(planti), data=results, control=lmeControl(maxIter=1000, msMaxIter=1000, niterEM=1000, opt='optim'))
Details of the Model:

They are Repeated measurements.
Length root (in cm) registered each 2, 4, 6 and 8 weeks.
root: Root Length in cm.
factor(week): 2, 4, 6, 8.
factor(fertilizer): control and added
N°planti: 1, 2, 3, ... 12

According to AIC, it is the best model, but statistical tests (and graphics?) show that it hasn't got neither homoscedasticity or residuals normal distribution. Furthermore, errors are autocorrelated.
What do you suggest me to do next?
In case the best option would be to delete such five numbers to fit residuals normal distribution (Q-Q plot). How is it suggested to find and delete them?.
By the way, please, let me know if I am doing these two Levene tests correctly:
leveneTest(residuals(lme.resultsA) ~ results$fertilizer)
leveneTest(residuals(lme.resultsA) ~ results$planti)
This is just my model LMM.
Than you in advance for your help
Answer: The model appears to be quite good.
From the plots on the question, I do not see much evidence that the residuals are not normally distributed, or of heteroskedasticity. It is pointless to look for a perfectly normal distribution.
You don't include a acf/pacf plot so I can't comment on autocorrelation, but if present then use the corAR1 or one of the other correlation structure functions available in lme.
Comment: Does this answer your question ? If so, please consider marking it as accepted. Thanks !


KVM guest broken after taking snapshot

Question: I needed to do some work on a KVM guest. To be able to go back to it's previous state, I took a snapshot:
virsh snapshot-create vph-share

After this, I tried to SSH into the VM:
ssh vph-share.hexvm

But I got

ssh: connect to host vph-share.hexvm port 22: No route to host

This worked fine before I took the snapshot.
I took a look at the snapshot-list, it looked normal.
I then tried to log in again, and got

ssh: connect to host vph-share.hexvm port 22: Connection timed out

I also tried a virsh shutdown, but although virsh reports that the VM is being shutdown in reality it isn't. A following virsh start says that the VM is already active.
I tried virsh reboot and got:
error: Failed to reboot domain vph-share
error: Requested operation is not valid: Reboot is not supported without the JSON monitor

I tried the same actions on another VM on my server, without problems.
The log (/var/log/libvirt/qemu/vph-share.log) only has a note about the shutdown attempt:

2015-03-19 09:14:43.445+0000: shutting down

How can I get this VM back to work again?
The OS of the host machine is Ubuntu Server 12.04.
The OS of the KVM guest is Ubuntu 12.04.
Answer: I noticed that, unlike the other VM's, this VM's image was owned by root:root.
I tried to chown it to libvirt-qemu:kvm like the other VM's, but the ownership changed back to root:root automatically.
However, this did cause the VM to be in "shut off" state.
Starting it with sudo virsh start vph-share brought it back to life.

KVM guest broken after taking snapshot


My jquery load script is just not working, why?

Question: I am simply trying to load a PHP file and replace the content of a div, but I cannot get the load to work.
$(document).ready(
    $('img.img1').click(function(){
        $('Close-div-1').load('gcard_save.php');
        document.getElementById("demo").innerHTML= "";
    })
);

I have ran out of things to try, does anyone know how to solve this?.
Comment: Thanks to everyone for helping. @wouter I will continue to read the docs, but I just could not figure this out.  And why did my question get voted down? Its a valid question, its very specific and a real problem( for me at least).
Comment: it's down voted because the last 2 things you said: It's very specific (to specific) and it is a real problem for you. It just isn't helpful for others, which means it does not really belong to a Q/A site like SO.  Could you please accept the correct answer?
Answer: There are no Close-div-1 elements in HTML, I think that you are looking for an id:
$('#Close-div-1')

or a class:
$('.Close-div-1')

Answer: Every event needs a callback function in jQuery. Please take a look at the documentation and see why this code is completely wrong. Or take a look at the console (Ctrl + J or F12 in IE) and see what error you get.
$(document).ready(function () {
    $('img.img1').click(function (){
        $('Close-div-1').load('gcard_save.php');

        document.getElementById("demo").innerHTML= "";
    });
});

EDIT: I had to reformat your code to see this mistake. It really helps you to make your code readable (that means indent consistently, use some whitespace, ect)


Material design conventions: toolbar text content

Question: Preface: while this may seem like a question based on personal preference, I am fairly sure there are set guidelines for material layouts, providing rules to adhere to. (I just cant seem to find them)
In order to prepare for an upcoming exam, I've made a simple quiz app for me and my classmates that goes through questions that will likely be asked on said exam. The layout of the main question screen is displayed below.

As you can see, rather than using the toolbar to display a title ("quiz screen") I have used it to display the entire question, and question number. In addition, I've disabled the back button. Now for the question: is this layout, and use of the toolbar specifically, in concordance with the material design guidelines? I have looked through https://www.google.com/design/spec/material-design/introduction.html, where one would expect to find such information, rather extensively, but have found no answer to this question.
Thanks in advance.
Answer: As far as I know the toolbar is used to provide hint of what the data in the current screen is about. It should not contain more than more two lines of text.(For two lines of text, use a collapsing toolbar and for a single line use the ordinary toolbar). Coming to your app, I'd suggest you put the question number in the toolbar i.e "Question 3" and the question should be an ordinary textview with about 24sp in text size. Moreover, remove the FAB from the screen and use a  flat button instead. The FAB button must not be abused
Comment: Maybe try some bold colors for the question and toolbar. Try to use amber with HEX color value of #FFC107 for the color of your toolbar and then add the question number as the text like I told you before "Question 3" but make it's textcolor #747474 and the typeface roboLight. The question's text color should also be #747474 but not in the toolbar. It's text should be a lot larger than the answers, maybe 32sp or 36sp. Lastly, make the text color for the answers #bebebe (this is a beautiful grey color).  [Read this] ( https://www.google.com/design/spec/style/color.html#color-color-palette)
Comment: You are welcome!! If you need some UI design inspiration check out [dribbble](https://dribbble.com/)  you're find lots of designs there. All the best with your exams
Comment: Thanks a lot for the flat button suggestion, will definitely change that.
Comment: The problem I have with just making the question a plain textview is that it makes the app look extremely... linear and boring. I have tried it previously,  and even gave the layout that contained the question TextView my primary light color as background, but it didn't look amazing. Whereas right now, because the background of the toolbar is the primary app color, the question really jumps to the foreground.

If the guidelines say this is not how a toolbar should be used, however, I will comply :)
Comment: Thanks a lot for your help and references. As you can probably tell, im fairly new to android/material design. This will most definitely come in handy!


Verificar se o numero digitado ja esta presente na struc

Question: Boa tarde, estou enfrentando o seguinte problema. Preciso fazer um algorítimo para o cadastro de contas bancarias de 15 pessoas com as seguintes informações: número da conta,nome e saldo. Porém é preciso verifcar se o número da conta ja foi digitado,se sim será impressa uma mensagem de erro se não ira solicitar o resto dos dados.
Só que não estou conseguindo fazer essa verificação, sempre esta dando a mensagem que o número ja existe mesmo não existindo. Segue meu codigo:
<code>#include <stdio.h>
#include <stdlib.h>
#define TAM 2

typedef struct {
    int num_conta;
    char nome[40];
    float saldo;
} Dados;

Dados Clientes[TAM];
int posConta = 0;

//Função para imprimir menu de opções
void Imprimir_Menu();

//Função para cadastrar clientes
void Cadastro(Dados *Clientes, int *pos);

int main(int argc, char** argv) {
    int opcao;

    do {
        Imprimir_Menu();

        printf("Digite a opção desejada: ");
        scanf("%d", &opcao);

        switch (opcao) {
            case 1:
                Cadastro(Clientes, &posConta);
                break;
        }
    } while (posConta != TAM);

    return (EXIT_SUCCESS);
}

//-----------------------------------------------------------------------------//

void Imprimir_Menu() {
    printf("MENU \n");
    printf("\t 1. Cadastrar contas; \n");
    printf("\t 2. Visualizar todas as contas de determinado cliente; \n");
    printf("\t 3. Excluir a conta com menor saldo; \n");
    printf("\t 4. Sair \n\n");
}

//-----------------------------------------------------------------------------//

void Cadastro(Dados *Clientes, int *pos) {
    int Num_temp, i, achou = 0;

    printf("\n--------------------CADASTRO DE CLIENTES--------------------\n");

    printf("%dº Cliente \n", *pos + 1);

    /* O programa iria prosseguir somente quando o usuario digitar um número de conta
     que ainda não existe */

    do {
        printf("\tDgite o número da conta: ");
        scanf("%d", &Num_temp);

        for (i = 0; i < *pos + 1; i++) {
            if (Clientes[i].num_conta == Num_temp) {
                printf("achou = %d \n", achou);
                achou = 1;
            }
        }
        if (achou = 1) {
            printf("Conta ja existente. Favor digitar um número diferente. \n");
        }
    } while (achou != 0);

    Clientes[*pos].num_conta = Num_temp;

    printf("\tDgite o nome do cliente: ");
    setbuf(stdin, NULL);
    gets(Clientes[*pos].nome);

    printf("\tDigite seu saldo: ");
    scanf("%f", &Clientes[*pos].saldo);

    *pos = *pos + 1;

    printf("---------------------------------------------------------------------\n");
}

//-----------------------------------------------------------------------------//
</code>
Segue a saída do programa:
Comment: `if (achou = 1)`. O compilador faz milagres. Sempre compile o código e preste extrema atenção aos avisos, pois 99.99% deles são erros. Naturalmente o código tem outros problemas, mas estou me a focar apenas no que foi citado
Comment: O mais importante do comentário do @Isac é que o código tem vários problemas.
Comment: Ja imaginava, to aprendendo estes conteúdos por conta própria durantes as minhas ferias da faculdade, para ter uma boa base no 2º Período. Mesma assim orbigados a todos que me ajudaram a encontrar o erro.
Answer: Bom amigo pelo que eu vi na condição If(achou = 1), você faz achou receber 1, e não achou==1
Comment: Nem acredito q era so isso, tava um tempão presso nisso kk brigadão Victor!
Comment: Você tá em qual semestre da faculdade? Por que pelo que estou vendo tem bastante erros no seu código.
Comment: to indo pro segundo
Comment: Bom, estou no quinto semestre... Já ganhei duas vezes em primeiro lugar a maratona de programação da minha faculdade... Se precisar de ajuda pode me enviar e-mail ou mensagem.
Comment: Não dispenso não Victor, mt obrigado. Como você tem mais experiência que eu, o que você acha que preciso melhoar em meus codigos? Qual erro mais encontrado no meu codigo acima?


Reset the Google Compute engine VM Could not SSH into the instance

Question: I m using Past 3 years Google Compute Engine VM instance.Using Putty I generated a private key and public key.Public key added in instance SSH metadata.
So using the private key I can ssh using gcloud SDK installed on my local machine.also, I can access GCE VM instance via PHPSTROM & SFTP FileZilla.
After RESET VM instance recently I can't connect via SSH.
My Local Gcloud SDK Shell Logs:

C:\Users\admin>gcloud init

<code> Welcome! This command will take you through the configuration of gcloud.

    Settings from your current configuration [default] are:
    compute:
    region: asia-east1
    zone: asia-east1-a
    core:
    account: firstname.lastname@example.com
    disable_usage_reporting: 'True'
    project: instance-1

    Pick configuration to use:
    [1] Re-initialize this configuration [default] with new settings
    [2] Create a new configuration
    Please enter your numeric choice:  1

    Your current configuration has been set to: [default]

    You can skip diagnostics next time by using the following flag:
    gcloud init --skip-diagnostics

    Network diagnostic detects and fixes local network connection issues.
    Checking network connection...done.
    Reachability Check passed.
    Network diagnostic (1/1 checks) passed.

    Choose the account you would like to use to perform operations for
    this configuration:
    [1] firstname.lastname@example.com
    [2] Log in with a new account
    Please enter your numeric choice:  1

    You are logged in as: [email@example.com].

    Pick cloud project to use:
    [1] instance-2
    [2] instance-1
    [3] Create a new project
    Please enter numeric choice or text value (must exactly match list
    item):  2

    Your current project has been set to: [instance-1].

    Your project default Compute Engine zone has been set to [asia-east1-a].
    You can change it by running [gcloud config set compute/zone NAME].

    Your project default Compute Engine region has been set to [asia-east1].
    You can change it by running [gcloud config set compute/region NAME].

    Your Google Cloud SDK is configured and ready to use!

    * Commands that require authentication will use firstname.lastname@example.com by default
    * Commands will reference project `instance-1` by default
    * Compute Engine commands will use region `asia-east1` by default
    * Compute Engine commands will use zone `asia-east1-a` by default

    Run `gcloud help config` to learn how to change individual settings

    This gcloud configuration is called [default]. You can create additional configurations if you work with multiple accounts and/or projects.
    Run `gcloud topic configurations` to learn more.

    Some things to try next:

    * Run `gcloud --help` to see the Cloud Platform services you can interact with. And run `gcloud help COMMAND` to get help on any gcloud command.
    * Run `gcloud topic -h` to learn about advanced features of the SDK like arg files and output formatting
</code>

C:\Users\admin>ssh-keygen

<code> Generating public/private rsa key pair.
    Enter file in which to save the key (/c/Users/admin/.ssh/id_rsa): 1
    Enter passphrase (empty for no passphrase):
    Enter same passphrase again:
    Your identification has been saved in 1.
    Your public key has been saved in 1.pub.
    The key fingerprint is:
    SHA256:weTBQM1j92Lx7iCtubzAZ2+4a5PHron3FDRIyOmb1iY admin@admin-PC
    The key's randomart image is:
    #ERROR!
    |     ooB+        |
    |      +=*oo      |
    |     . .=oo+     |
    |      .  oo.o    |
    |       +So.o     |
    |     .E + o..    |
    |     .oooB.o     |
    |       *O++ .    |
    |      .oO%+      |
    #ERROR!
</code>

C:\Users\admin>gcloud compute ssh lamp-bafs

<code> WARNING: The PuTTY PPK SSH key file for gcloud does not exist.
    WARNING: The public SSH key file for gcloud does not exist.
    WARNING: The private SSH key file for gcloud does not exist.
    WARNING: You do not have an SSH key for gcloud.
    WARNING: SSH keygen will be executed to generate a key.
    This tool needs to create the directory [C:\Users\admin\.ssh] before
    being able to generate SSH keys.

    Do you want to continue (Y/n)?  Y

    Updating project ssh metadata.../Updated [https://www.googleapis.com/compute/v1/projects/instance-1].
    Updating project ssh metadata...done.
    Waiting for SSH key to propagate.
    FATAL ERROR: Network error: Connection timed out
    ERROR: (gcloud.compute.ssh) Could not SSH into the instance.  It is possible that your SSH key has not propagated to the instance yet. Try running this command again.  If you still cannot connect, verify that the firewall and instance are set to accept ssh traffic.
</code>
I got the above issue. So I followed as per the below image steps

I ran the below command in Gcloud SDK Shell

C:\Users\admin>gcloud compute ssh lamp-bafs

I got network : connection timed out issue in PuTTy

I could not find solution SSH via Gcloud SDK Shell, Using Public key to connect VM instance via PhpStrom. I don't know What firewall blocking or any other new process I missed.
Please help to solve this
Answer: "Connection timed out" means that the problem is not that you are using the wrong key.  Instead it means either:

The OS itself has crashed, e.g. boot failure
The SSH daemon (sshd) is not running on the machine
There is a firewall (either GCP or iptables) blocking your access to the instance.

If you look at the serial console for your instance (you should find a link to it from the Cloud Console's page for your VM) the log should give you a clue about which of these is at fault.
If you cannot find/fix the cause this way, I advise you to make a snapshot of the instance's disk, create a new disk from that snapshot, create a new instance from a fresh debian image with that disk as a secondary disk. You can then use that fresh instance to fix the disk or recover your data.
Comment: Thanks, David, I tried with different ISP. I can SSH to VM.I think ISP blocking port 22.


What should be desired output for ANN

Question: The Back-Propagation Algorithm as described in this guide/introduction is confusing, the final formula generated for del(w)(change in weights) is

del(wji) - Change in weights of synapse from j to i;
eta (n) - learning factor;
Oj - Output of previous layer j;
dj - Desired output from previous layer j;
xi - Input

I have few doubts associated with the formula.
What is the desired output for hidden layer and Input layer??
What is the j and i notation?
What should be the learning factor how can I decide(or generate) the value of learning factor i.e eta(n)?

Note: I want it to work like this (there would be differences in some factors but formulations should be similar..I cannot see any similarity in the formulations).
Answer: What is the desired output for hidden layer and input layer?
The hidden layers and input layers don't have a desired output.
Each neuron has an error, which for the output neurons is calculated as target activation - current activation.
Hidden neurons and output calculate their error as follows (psuedocode):
foreach(otherNeuron in myProjectedConnections):
  myError += otherNeuron.error * connection.weight

myProjectedConnections is a list containing all the neurons to which a certain neuron passes on its activation value.
What is the `i` and `j` notation?
Your question contains the answer: del(wji) - Change in weights of synapse from j to i;
A synapse is a fancy word for a connection. So j is a neuron and i is a neuron, with j projecting a connection to  i.
What should the learning factor be?
The learning factor should be a factor anywhere in the range of 0-1. If you set your learning factor to 1, a network will learn the output to one test case immediately. If you want to teach it patterns however, choose a value anywhere in the range of 0-0.3, depending on your situation ofcourse.
Comment: Do you know anything about Rprop(Resilient Back propagation), Rprop uses (changes) learning factor to calculate new delw. What if I use Rprop instead of BackProp, which is better?

What should be desired output for ANN


How to query SSMS via SAS EG ODBC connection while using data from WORK?

Question: Is there a way to send a SAS dataset through a proc sql odbc query in SAS EG so that it can be taken in and used by SQL server?
ex)
SAS Data WORK.A contains 3 columns ID, col1, col2.
I have a table in Sql Management Studio environment with the same ID column.
I would like to (somehow) do as shown in Figure A below:
Figure A)
proc sql;
Connect to odbc("driver=SQL Server; database=SSMSDatabase; Server=SSMSServer");
create table WORK.B as
select * from connection to odbc
(
Select t1.*, t2.*
from SSMSTable1 t1
INNER JOIN WORK.A t2 ON t1.ID = t2.ID
);
disconnect from odbc;
quit;

This throws an obvious error in SAS as SSMS doesn't understand what WORK.A is... It's expecting pure SSMS code to be executed.
I have passed macro variables created in SAS through to SQL passthrough to be used in the WHERE statement like in figure B, but that has it's limitations (especially with macro character length) and is not as elegant as passing a whole table.
Figure B)
proc sql;
select cat("'",trim(ID),"'")
into :list1 separated by ","
from WORK.A;
quit;

    %macro ODBCRun();
proc sql;
    Connect to odbc("driver=SQL Server; database=SSMSDatabase; Server=SSMSServer");
    create table WORK.B as
    select * from connection to odbc
    (
    Select *
    from SSMSTable1
    WHERE ID IN (&list1.)
    );
    disconnect from odbc;
    quit;
%mend;

%ODBCRun();

Any ideas would be helpful.
Answer: Create a temporary table in SQL Server, against which you can perform your later pass through query.
Example:
libname SS odbc
  dbmstemp=yes
  noprompt="driver=SQL Server; database=SSMSDatabase; Server=SSMSServer"

;

* upload SAS data into SQL Server;

proc delete data=ss.'#idlist'n;
run;
data ss.'#idlist'n;
  set WORK.A;
run;

* Use uploaded data in pass through query;

proc sql;
  connect using SS as REMOTE; * reuses connection libref made;

  create table WORK.B as
    select * from connection to REMOTE
    (
      select t1.*, t2.*
      from
        SSMSTable1 t1
      INNER JOIN
        #idlist t2
      ON
        t1.ID = t2.ID
    );

  disconnect from REMOTE;
quit;

Comment: The code to create a temp table in the remote server seemed to work. no errors. I tried querying Select * from #idlist in SQL Management Studio and it resulted in an Invalid Object Name. In SAS the remote part of the query threw the error  ERROR: CLI describe error: [Microsoft][ODBC SQL Server Driver][SQL Server]Invalid object name '#idlist'.: [Microsoft][ODBC SQL Server Driver][SQL Server]Statement(s) could not be prepared. Any ideas? I think this is very close as a solution
Comment: in SSMS I ran select * from tempdb.sys.objects WHERE name LIKE '%list%'. and the result was ###idlist with 3 hashtags. Select from ###idlist works. Any idea why 3 hashtags are generated from the SAS code?
Comment: I don't have SQL Server handy for testing. In SQL Server local temporary tables start with `#` and global temporary tables `##`.  So maybe the SAS name literal `'#idlist'n` got garbled into a global temporary table (`##`) named `#idlist`.  More research or trial and error would give more insight.
Comment: Its just a quirk. The main thing is that is works great. the triple hash ### makes the query special as you dont usually see that in code. Great solution!!


ARIMA: extract date/time information from ARIMA model

Question: I create an ARIMA model for my ts-object. My data is available in seconds or even miliseconds. I didn't find a way to specify the time information for the start- and end-parameters when creating the ts object?
I need the exact time, because I want to extract the time information when I do the forecast based on the ARIMA model to return the exact times for my forecasted values. It would be enough to store the end-time information somewhere in my ARIMA model, so that I can use it later when I do the forecast.
How is this done usually with ARIMA models?
Thanks!
Answer: Did you see the help file for the ts function? At the bottom there are examples for quarterly and monthly time series. Perhaps you could directly extend this to second-wise series? You only need to specify the start date, not the end date.
For example, instead of using start = c(1954, 7), frequency = 12 for monthly data starting in July 1954 you could use start = c(d, s), frequency = 60*24 for second-wise data starting on day d, second s. The time attribute of your data would be given in days.
set.seed(1)
x=rnorm(10^4) # generate random data
d=1
s=1
x=ts(x,start=c(d,s),frequency=60*24)
plot(x)
print(time(x)) # extract the 'time' attribute of the 'ts' object

There might be a better solution, but perhaps this one is sufficient?
Comment: I thought about that, but then I actually have hourly based data and the frequency is 24. When I define the start time as a miliseconds or seconds based "timestamp", I have to change the frequency parameter as well to milliseconds or seconds, which would be wrong for my arima model, because obviously arima uses the frequency information from the ts object.
Comment: Depending on what you will do with the model, the frequency may only be used for the seasonal part of a seasonal ARIMA model, so it should not do harm if a non-seasonal ARIMA model is considered.
Comment: For now I just added the end-timestamp and the interval as additional attributes to my model. When I use the same model for forecasting I can extract that information from the model to calculate the weekdays. I don't know if there is a better solution.


What evaluation metric to use for high class imbalance where i want to capture most of the positive (ones) in the dataset

Question: I have a dataset that contains 99.95% 0's and 0.05% 1's as the target. The dataset contains million rows.
I want to build a binary classification model that predicts almost all the 1's correctly while keeping the false positives at minimum.
I have read it somewhere that AUC-PRC is a better metric for the above scenario compared to AUC-ROC. Is it correct?
Answer: Neither seems appropriate. Rather, assign whatever penalty scores you want to the two kinds of errors (mistaking a 0 for a 1, and mistaking a 1 for a 0) and sum the errors. This allows you to precisely control the tradeoff.
Comment: I think you are over-simplifying things. While imperfect (a "perfect metric" is very subjective) AUCPR is far from totally inappropriate. See some references on the subject. (eg. see @Marc's answer [here](http://stats.stackexchange.com/questions/90779) and papers like [here](http://machinelearning.wustl.edu/mlpapers/paper_files/icml2006_DavisG06.pdf) and [here](http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0118432). It is better for people to use AUCPR as they are unlikely to misspecify it than have them put arbitrary weights and penalties ending up with a "salad metric".
Comment: Thank you for your answer. I am new to machine learning and don't know how to implement your answer in a code in R. Could you please elaborate a little? How can i assign penalty score in  a model? I plan to use some boosting method to model the data and later modify things based on the results i get.
Comment: I believe Kodo is telling about weighed accuracy
Comment: @Aman That sounds like a programming question, in which case Stack Overflow, not here, is the right site to ask.
Comment: @usεr11852 Notice that the OP wants to build a classifier, whereas the two AUC measures attempt to summarize the performance of a signal over a number of classifiers that could be defined with it. Only the classifier actually used matters.
Comment: @Kodiologist:The OP describes an extremely common scenario ("*build a binary classification model that predicts almost all the 1's correctly while keeping the false positives at minimum*"), I cannot see how this makes the question any different than dozens of other imbalanced classification tasks. A PR curve will let the OP make a much more informed decision about the threshold to use than a simply defining a (potentially arbitrary) cost. Your suggestion would just move the question of picking a threshold to picking relative weights but without the direct diagnostics provided by a PR curve.
Comment: @usεr11852 I'm saying that AUC isn't useful here, not that the entire precision–recall graph isn't useful.
Comment: @Kodiologist: That much is obvious. :) I am saying though that this assertion is over-simplifying here. We are presented with a *standard* imbalanced learning problem, you advocate a very particular approach (cost-sensitive learning) and dismiss another one (AUCPR) with no obvious reasons.
Comment: @usεr11852 Because of what I just said, that the AUC summarizes performance over all thresholds rather than the performance of a particular classifier, which is what's of interest.
Comment: @Kodiologist: But PR curves are of interest! I agree that what you propose has some merit (after all I have not downvoted your answer exactly for this reason). On the other hand plotting the PR curves and using the AUCPR should not be dismissed. You are effectively suggesting using an 0.50 threshold and hope that the weighting will be adequate. I think this is an over-simplifying idea about the use of PR curves as well as the difficulty of finding a weighting scheme.
Answer: You can look at the Precision,Recall and the F1 score which is nothing but the harmonic mean of the Precision and Recall.
Answer: Your reading is correct in the sense that AUC-PRC is a better metric for imbalanced classification compared to AUC-ROC. I disagree with Kodi in sense that AUC could be useful in these scenarios. Like Santanu said you could look for precision, recall and F1. I would want to add Sensitivity and Kappa.
However, choice of a metric is not only the way to handle imbalanced classification. You could look for sampling techniques such as SMOTE, converting it to a probability estimation problem with biased threshold and others discussed here and elsewhere.
Comment: Why is AUC-PRC better than AUC-ROC for imbalanced classes. I could not find a good explaination
Comment: Have you had a look at this: http://stats.stackexchange.com/a/90783/73547 ?


REST - GET returns different result than POST/PUT

Question: In our project, a book can be added by sending the book structure (in XML, JSON, ..) via a POST or PUT request. For example, in XML, the book structure looks like this (simplified):
<book>
    <title>My Book</title>
    <author>John Q.</author>
</book>

When this book is inserted in our backend database, some auto-generated properties are automatically added, such as the creation date, the user id who submitted the book, an identifier, ...
When the book is retrieved through a GET, these additional properties are included in the book definition:
<book>
    <title>My Book</title>
    <author>John Q.</author>
    <info>
        <creation_date>2011...</creation_data>
        <user_id>48</user_id>
        <identifier>my_book_john_q</identifier>
    </info>
</book>

This basically means that the XML scheme of a new/edited book (= from client to server) is different than a retrieved book (= from server to client). This makes things confusing.
A possiblity is to make these additional properties available in a different URI, for example:
http://server/books/:id/             -> returns the short version
http://server/books/:id/information/ -> returns the generated properties

A downside of this approach is that two separate requests are needed to have all data.
How would you solve this inconsistency?
Answer: This is perfectly normal.  There is no problem having the server augment the representation with some additional information.  A good example of this is when the server adds links to the representation.  There is no requirement for the client to send "copies" of those links to the server when doing a PUT.  The resource representations that you GET and PUT should be conceptually the same, not necessarily byte for byte identical.
Comment: Alright. Indeed, I wasn't sure whether GET and PUT must be identical conceptually or content-wise (byte by byte).
Comment: I agree: patterns serve us; we do not serve them. tl;dr: emerson

BTW: Do you know the origin or proponents of this pattern?
Answer: You're not using mimetypes correctly. I bet you're using an application/xml generic mimetype and your clients know what to expect based on the endpoint, right?
The proper way to deal with your problem would be having different representations for the same resource, with different mimetypes. For instance, you can have a application/vnd.yourcompany.book.short+xml for the short representation, and application/vnd.yourcompany.book+xml for the complete representation. Clients can use the Content-Type header to say which one they are sending, and the Accept header to say which one they want.
This doesn't mean the clients must send the short representation in the POST or PUT. You can document some fields as optional and it's perfectly fine for the clients to omit them.


response grid view and adaptive themes

Question: I have:

a view which displays three pictures of 300 * 200 each
it should be responsive at breaks of 300 and 600
adaptive themes

1) I installed responsive grid view. In here I need to add a wrapper – but I don't know which one or what to create. Is there already a wrapper in the adaptive theme for this?
2) Also I generated a css file at gridpack.com. This css file I added to the theme and I replaced:

.col with the views-responsive-grid label (css is in the page)
.col with .views-column (the css is in the page)
.row with .views-row (I can't see this back in the page)

It still doesn't work. Still on the big screen all images are display vertically. Is this the way it should work or there any other suggestions?
Answer: I don't know if you mean adaptive theme as the theme on drupals repository or adaptive theme as a general behavior. Either way, you don't need a module to handle it. You just need the proper media queries, regardless if they're served by AdaptiveThemes or you insert them directly into your CSS files, somewhere at the end.
Once again, displaying three elements as a grid doesnt require a grid regulation. Is just a view, could be unformatted or html list or whatever. You need to declare responsive widths in your CSS for each row of the view, like so:

Consider using .gridimg as your row class in unformatted format for
  example

HTML

CSS
            .gridimg {
                width: 33.333333%;
                float: left;
            }

Then the media query where needed
            @media all and (max-width: 600px) {
                gridimg {
                    width: 100%;
                    float: none;
                    margin-bottom: 10px;
                }
            }

Do not display elements for the sake of your media queries, display elements for the sake of your content. Replace 600px with whatever resolution you want your rows to adjust.
Your classes could be named differently. Also the wrapper of your three generated rows should be set to 100%. Rows within should be position: relative;. They're like this by default, unless specified in a different manner.
Maybe your image element should have some css applied to them, for responsive purposes, if they're not set already:
            .gridimg img {
                width: 100%;
                height: auto;
                display: block;
            }

Comment: Thxs, i'm trying to get this working. Should this be done with a normal grid or a responsive grid in views or doesn't it matter?
Comment: GRID is a fabricated term and now got to be used as cliche. There is no such thing as grid in HTML. You should use div's or li's, whatever suites you best. In Views, thats either unformatted list or html list.

I'd go for unformatted, adding a class to each row, as explained above. You only have 3 pictures so it should be easy! Give your row class a percentual width and float it with the CSS provided above. You only have one field - the image field. Add media queries at the end of your CSS style and that's it. If you're using a responsive theme, img should be { max-width: 100%; height: auto; }
Comment: I implemented this. The current result can be found on http://whayme.thedrupalchecklist.com/ . It is the first block under the main images. If I add the statement float:left into the css the layout is good (it is not finalized but oke) but... it is not clickable any more.... how to deal with that? Is there another approach needed for this?
Comment: Well, you didn't listen. You have chosen Grid instead of a few lines of code and markup got a bit heavy. But, because of the float: left, now you need to clear the wrapper for the three, otherwise things frm beneath will cover your divs (and links). So adding an overflow: hidden to your wrapper will do it. Problem is, under these circumstances, using grid columns instead of classic unformatted view, your rows bottom padding is 100px, a bit too much. You didn't need the grid.

response grid view and adaptive themes


How do you use `if` when applying a `raw_input`?

Question: I am somewhat new to Python and coding in general, and I require some help working with raw_input and an if statement. My code is as follows;
    age = raw_input ("How old are you? ")
    if int(raw_input) < 14:
    print "oh yuck"
    if int(raw_input) > 14:
    print "Good, you comprehend things, lets proceed"

Answer: Issues
There are three issues with your code:

Python uses indentation for creating blocks.
You've assigned the input to the variable age, so use age.
In Python 3, you have to use print(...) instead of print ...

Correct solution
age = raw_input("How old are you? ")

if int(age) < 14:
    print("oh yuck")
else:
    print("Good, you comprehend things, lets proceed")

Note that this is not equivalent to your code. Your code skips the case age == 14. If you want this behaviour, I suggest:
age = int(raw_input("How old are you? "))

if age < 14:
    print("oh yuck")
elif age > 14:
    print("Good, you comprehend things, lets proceed")

Learning Python

http://www.codecademy.com/tracks/python - A tutorial that introduces you to programming / Python step by step
Manual - Looking things up and discovering new stuff, e. g. raw_input() and int()
Answer: if int(raw_input) < 14:

Should be int(age), and same for the other if. raw_input is the function you called, but you stored the result from it in the variable age. You can't turn a function into an integer.
Rather than converting the age to an integer repeatedly, you could just do it once, when you do the input:
age = int(raw_input("How old are you? "))

Then you can just do if age > 14 and so on, since it's already an integer.
I assume the indentation problems (the line following each if should be indented at least one space, and preferably four) are just a formatting issue.


Variáveis são automaticamente executadas quando declaradas?

Question: Por exemplo, se guardarmos um setInterval() em uma variável, ele já executa automaticamente mesmo sem aquela variável ter sido chamada em lugar nenhum?
Exemplo:

let i = 0;
let myInterval = setInterval(function(){
  console.log(i++)
  if(i > 10){
    clearInterval(myInterval)
  }
},1000)

Eu nem chamei o myInterval em lugar nenhum e ele ja está executando...
Variáveis já são automaticamente executadas quando declaradas?
A dúvida mesmo é só porque na função temos que chamar a mesma em algum lugar, entendem?
Isso vale só pra JS ou pra todas as linguagens?
Comment: Para todas, pois é o comportamento da função. O que você atribuiu à variável não foi a função, foi o **retorno** da função; e para ter o retorno, a função será executada.
Comment: Assim como indicado na [documentação](https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/setInterval) o retorno de `setInterval` é um valor númerico que identifica o *timer* criado para que possa mais tarde o desligar através da função `clearInterval`. Se não vai querer desligar o timer então também não precisa de capturar o retorno.
Answer: Variáveis são estados e não ações, então não faz sentido em falar em execução de variáveis.
O que você fez foi chamar o setInterval(), isto basta. Como argumento passou uma ação, como ele espera. Sem a variável funciona da mesma forma, ela é completamente inútil neste código:

let i = 0;
setInterval(function(){
  console.log(i++);
},1000)

Coloquei no GitHub para referência futura.
Comment: @LucasCarvalho, respondendo sua pergunta, quando o setInterval é chamado, ele já executa tal função inciando o timer. A função retorna um handler para cancelá-lo posteriormente. É a mesma coisa que uma função assim: `function teste() { console.log("EXECUTEI"); return "valor-qualquer"; }`. Quando você atribuir a EXECUÇÃO dessa função a uma variável: `var a = teste();`, aparecerá no console `"EXECUTEI"` e sua variável tera o valor `"valor-qualquer"`.
Comment: @LucasCarvalho Se você fizer `x = fatorial(5)`, você precisará *chamar* a função `x()` para obter o valor 120 ou o valor de `x` já será 120?
Comment: Justamente porque a função `setInterval` é chamada para produzir o resultado que será atribuído à variável. Quando a função é executada, seu efeito é que a função passada por parâmetro seja executada no tempo especificado. Ter, ou não, a variável não mudará o comportamento da função - foi isso que o Maniero disse, não que ela fosse completamente inútil (cc @Allan).
Comment: Talvez lhe falte conhecimento anterior para entender a diferença entre cada coisa, ou seja está fazendo algo muito mais avançado do que é capaz neste momento. Não é questão de entender JS, é entender o que cada coisa é em programação, um erro que vejo as pessoas cometerem. Querem ver resultado e não se preocupam em aprender de fato antes de fazer. Eu repeti diversas vezes que variável serva para guardar valores, por isso eu tirei a variável para mostrar que ela não serve para nada, afinal você não está guardando valor ali.
Comment: @LucasCarvalho primeiro, editar a pergunta depois de ter sido respondida não é legal. Se continuar fazendo isso desestimula as pessoas responderem suas perguntas porque pode invalidar a resposta dada. O que mudou importa zero, não vou mudar minhas resposta, mas eu poderia mudar isso nela e continuar sem, a variável. Eu não sei se consigo explicar de outra forma, pra mim é óbvio demais para conseguir tentar explicar melhor. Como uma amigo meu que dá aula na Unicamp diz "eu não sei explicar para o aluno o que é uma mesa"
Comment: No caso eu coloquei a variável para um possível clearInterval.
Comment: Mas, entendi mais ou menos ainda, mesmo ela sendo executada, o valor era pra ficar na variavel e não ser executado; Desculpa a ignorância!
Comment: mas eu não sei disso, só posso responder em cima do que você postou. Você não entendeu o que o `setInterval()` faz então. Provavelmente quer fazer outra coisa, mas isso também não está na pergunta.
Comment: Perdão Maniero, foi so pra ilustrar mesmo que eu estava guardando a mesma. Sei que é inútil. Só ainda não bate o porque ela executa e não "guarda". De qualquer forma, obrigado pelas respostas!
Comment: Acho que você não leu a resposta. A variável é inútil, ela não tem que guardar nada. Ele executa nada, como toda variável. Quem executa é a função. Se não é o que deseja faça outra coisa, o que está aí bate 100%.
Comment: Eu editei a pergunta como mencionei acima, de qualquer forma, eu ainda não entendi; Não foi pela sua má explicação (Que foi muito boa), foi pela minha talvez falta de percepção e maturidade em JS ainda. Sei que ela executa, etc.. Mas pra mim, variáveis servem para guardar valores... Talvez eu tenha me expressado mal.
Comment: A variável não é inútil @Maniero, ela é o handler do seu interval. Que no caso, posteriormente você pode cancelá-lo:  `clearInterval(myInterval)`.

https://www.w3schools.com/jsref/met_win_clearinterval.asp
Comment: Allan, antes estava sem o clearInterval. Mas é pra isso mesmo que eu usaria, mas não é culpa do Maniero, eu não coloquei toda a pergunta, desculpem!
Comment: Anderson, agora entendi melhor! O que me intriga é do porque ele mostrar o resultado no console, sendo que não estou usando a variável, apenas declarei a mesma.
Comment: Obrigado a todos! Com as explicações eu entendi agora! Value @Allan
Answer: Isso é porque você não declarou a variável com a função, apenas o retorno desta.
O seu código deveria ser:
let i = 0;
let myInterval = function(){
    setInterval(function(){
      console.log(i++)
      if(i > 10){
        clearInterval(myInterval)
      }
    },1000)
}

Sendo assim, apenas ao chamar explicitamente:
myInterval()

O setInterval será executado.
Comment: Neste caso `myInterval` não será o retorno de `setInterval`, então não faria sentido coloca-lo no `clearInterval`; a menos que faltou um *return* do retorno do `setInterval`
Answer: Você está salvando na variável o retorno da função setInterval, você pode armazenar a função dessa forma:
let interval = setInterval

E depois você executa chamando o nome da função mais os parênteses
interval()

let interval = setInterval
let c = null
let i = 0

let funToExecute = function() {
    console.log(i)
    i++
    if (i > 10) {clearInterval(c)}
}

c = interval(funToExecute, 1000)


Negative Beat / Silence

Question: Some music these days contain tiny pauses after/instead of each beat.
Hard to describe but one example I just heard is
Petite Bisquit - Sunset Lover
What is this effect or style called?
Thank you!
Comment: Possibly sidechain compression?
This question/answer might help: https://music.stackexchange.com/questions/7265/depression-sensation-after-the-beat-in-electronic-music/7270#7270
Answer: Listen to the track from about 40 seconds in, where the percussion starts.  You'll see that the beat wasn't quite where you thought it was!  Starting the song with a syncopated figure against an un-stated beat gives a pleasantly ambiguous effect.  It's very much a technique of music created in a sequencer rather than conceived for live performance.  Similar to the trick where the music starts with a 'beat', layers are added and then, half way through the track there's a 'break down' to just one element.
Answer: There is one main thing happening in this song : sidechain.
Sidechains enable composers to control the output volume of some tracks (or instruments) according to another instruments volume output. Here every single track is sidechained to the kick drum. This results in every instruments going very quiet each time the kick drum hits, creating this "tiny pause" effect. And it really is just an effect and don't really translate into musical theory terms.
Combining this effect with syncopated harmonic or melodic rhythm emphasize said syncopation.
Comment: FTR: _sidechain_ per se is not an effect. The effect you're talking about is a _ducking compressor_, whose detection (sidechain-input) happens to be controlled by an entirely different source.

Negative Beat / Silence


Firestore import/export per API

Question: I'm searching for a way to call the firestore import/export functionality programmatically from java code.
What i found so far is that the nice firestore client library does not yet support the import/export calls. But the more low level rest/grpc api already supports them. Using the java library i tried the following:
Firestore firestoreApi = new Firestore
    .Builder(UrlFetchTransport.getDefaultInstance(), new GsonFactory(), null)
    .setApplicationName(SystemProperty.applicationId.get())
    .build();

GoogleFirestoreAdminV1beta2ImportDocumentsRequest importRequest = new GoogleFirestoreAdminV1beta2ImportDocumentsRequest();
importRequest.setInputUriPrefix(String.format("gs://{}/{}/", BUCKET, image));

GoogleLongrunningOperation operation = firestoreApi
    .projects()
    .databases()
    .importDocuments("projects/" + SystemProperty.applicationId.get() + "/databases/(default)", importRequest)
    .execute();

Which sadly ends with missing permissions when run in app engine:
com.google.api.client.googleapis.json.GoogleJsonResponseException: 401
{
  "code": 401,
  "errors": [
    {
      "domain": "global",
      "location": "Authorization",
      "locationType": "header",
      "message": "Login Required.",
      "reason": "required"
    }
  ],
  "message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
  "status": "UNAUTHENTICATED"

I cannot get the official way to login to work, because the firestore builder does not have a method to accept a instance of AppEngineCredentials.
I already checked the python client library which also seems not support these methods (yet).
Does anyone have a idea how i can either login with the old rest api or get a client library which supports these methods (some language which runs on app engine please :) )
Thanks for reading!
Carsten
Comment: Here's an example in Node:
https://firebase.google.com/docs/firestore/solutions/schedule-export. For Java, you can adapt this [Cloud Datastore example](https://github.com/GoogleCloudPlatform/java-docs-samples/tree/master/appengine-java8/datastore-schedule-export).
Comment: @JRLtechwriting Thank you very much! Adopting the datastore example worked! I think the app engine account needs 'Cloud Datastore Import Export Admin' and 'Firebase Admin SDK Administrator Service Agent' But those are minor details.
btw: If you add your comment as answer i will accept it :)
Answer: You can adapt this Cloud Datastore example for Cloud Firestore. See how they get an access token here:
import com.google.appengine.api.appidentity.AppIdentityService;
import com.google.appengine.api.appidentity.AppIdentityServiceFactory;

// Get an access token to authorize export request
      ArrayList<String> scopes = new ArrayList<String>();
      scopes.add("https://www.googleapis.com/auth/datastore");
      final AppIdentityService appIdentity = AppIdentityServiceFactory.getAppIdentityService();
      final AppIdentityService.GetAccessTokenResult accessToken =
          AppIdentityServiceFactory.getAppIdentityService().getAccessToken(scopes);
      connection.addRequestProperty("Authorization", "Bearer " + accessToken.getAccessToken());

Firestore import/export per API


Enforce Registration of Derived Types

Question: I have an implementation of a registrar for classes that are derived from a certain base class. The intent is to have each derived class register itself and in the process provide some information about itself, which in the example below is provided through a string.
What I'm missing is a way to force classes that derive from Base to register themselves through initializing the static member reg. In other words, is it possible to have the compiler somehow produce an error if a derived class doesn't define/initiatlize the static member?
struct Registrar {
  Registrar(string type) {
    registry().push_back(type);
  }
  static vector<string> & registry() {
    static vector<string> * derivedTypes = new vector<string>;
    return *derivedTypes;
  }
};

//CRTP
template <typename Derived>
class Base
{
  static Registrar reg;
};

class Derived1 : public Base<Derived1> {/*Class definition*/};
class Derived2 : public Base<Derived2> {/*Class definition*/};
class Derived3 : public Base<Derived3> {/*Class definition*/};
//...

//Initialize the static members of each derived type
//Commenting out any of the following 3 lines doesn't produce an error.
//Ideally, I want it to produce a compile error.
template<> Registrar Base<Derived1>::reg("Derived1");
template<> Registrar Base<Derived2>::reg("Derived2");
template<> Registrar Base<Derived3>::reg("Derived3");

int main() {

  cout << "Registered Types:" << endl;
  for(vector<string>::const_iterator it = Registrar::registry().begin();
      it != Registrar::registry().end(); ++it) {
    cout << *it << endl;
  }

  return 0;
}

Comment: best I can think of is a linker error (not compile error), and the only way to do that would be to somehow access `reg` in the `Base` class (say in the constructor) - this should trigger a linker error (with the caveat that you have indeed used the derived type in question somewhere...)
Comment: @Nim: I was thinking exactly the same ;-)
Answer: You can cause a linker error in g++ at least by adding a dummy function to the registrar: void blah() { } and then calling it from a dummy constructor in the CRTP base class:
public:
    Base() { reg.blah(); }

I can't see a more elegant way to solve this problem.

Enforce Registration of Derived Types


XGBoost heavily overfitting when containing the minimum/maximum of a variable?

Question: I've been building an XGBoost Regressor model with some good success. Currently, the training accuracy is 68% and testing 66% - indicating some, but not too much, overfitting.
However, I've noticed today a strange behaviour:
Let's say the model had 2 independent features, I'll call them X and Z, and then it's trying to predict the likelihood of event Y (either 1 or 0). We'll say X and Z are normally distributed between 0 and 1. Let's also assume that X is highly +correlated to Y, and Z is less so but still +correlated. We have also added monotonic constraints to X and Z, so that if all else is equal then increasing them will only increase the model prediction of Y.
What I'm seeing is that in a "normal" prediction if X is say, 0.1 and Z is 0.1 then the model predicted probability of Y is 10%
If you increase Z to 0.5, then the model predicted probability of Y goes to 15%..
If you increase Z to 0.95, then the model predicted probability of Y goes to 20%..
But when Z is 1 then the model predicted probability of Y is suddenly 90% likely! The reason, I think, is because in the training dataset the maximum value of Z is 1, and when Z was this value, Y was 1. So it's like the model has hyper overfit to this one row of data? It's very strange, as like I say the model in general doesn't seem overfit - but it has this very weird interaction with the row that contains the maximum value?
Is there a specific type of parameter or technique that can help defend against this type of interaction?
Reshuffling the train/test dataset isn't possible as it's timebased and so they are split by a date (before date = train, after date = test).
A Partial Dependency Plot (from the actual model), where X=feature[0] and Z=feature[20]. It shows what I expect to see from the model - however the prediction still jumps (from 16% to 73%!) after Z (feature[20]) goes higher than the training set maximum of 2.16.
Comment: Could you show us partial dependence plots for the two variables? I think it would help to see where the prediction function jumps.
Comment: @MichaelM, just added one - weird thing is it looks as expected..
However, when feature X = 0.27 and Z = 2.16 the prediction is 16% and
when I put it Z = 2.17 it suddenly shoots up to 73%. It's only at this exact point that it switches. Every other number below 2.17 sees normal behaviour.
Comment: Hmm. Thx! Do you see similar behavior without mononucleosis constraints?
Answer: You are probably right in diagnosing what went wrong, and yes there are techniques to mitigate this. Xgboost offers regularization options to make sure nodes have "enough" data. The "gamma" parameter does this by requiring a minimum improvement in the training objective - you should be able to increase gamma to a point where this value isn't being singled out.
There is also a "min_child_weight" parameter - if you are using the sum of squared errors objective this is just a minimum number of rows per node. Increasing this should also be able to prevent the problem.
But even more simply, if you know that you only have one row with this value, and you know you never want your decision trees to use this feature to distinguish that record, you could always just cap the variable at some lower threshold before training the model.
Comment: Thanks, I've experimented with the two parameters above and although they do reduce model overfitting - this strange behaviour persists!

The problem with simply capping the datapoint is that I have 30+ features and I believe it may be a problem with them all so manual intervention doesn't seem wise. Thanks for your answer anyway!
Answer: You could do a principal component analysis as a data preparation step and use the principal components as predictors instead of the original variables.  The highly correlated variables should get collapsed into a smaller number of principal components.  What would this accomplish?  Hard for me to answer.  Perhaps make it more robust to noise in the predictors?  Make the partial dependency plots more valid?
A few things to keep in mind

Partial dependency plots are kind of problematic in that they purport to show the isolated effect of one variable, but due to multicollinearity in the training data the way the tree regression works, this is not exactly what you are seeing
Tree regression models are unable to extrapolate, so if you have two variables highly correlated in your training data, and you predict on a new data set with a different relationship between the variables, it may give poor predictions


How do I make entity reference breadcrumbs?

Question: I need to make entity reference breadcrumbs, with the first level being a node title, and the second level the title of a referred node.
As you can see in the image below, the nodes are different content types.

How do I make entity reference breadcrumbs?
Comment: @No Sssweat You can answer this question now.
Answer: 0) download and enable the Entity Reference module
1) Go to eLesson content type Manage Fields.
2) Add a new field, call it "File Under" select  Entity Reference as your type and use auto complete as your widget.

3) Select Node as your Target Type

Target Bundles: checkmark Course only

4) Go to your Forex Advantage node and edit it.

For File Under type Beg, and select BEGINNERS COURSE.
Save changes.

5) If you have any breadcrumbs module, disable it and uninstall it.

Install Crumbs, the Breadcrumbs suite Module
Enable Crumbs only (no need to enable Crumbs Example nor Crumbs Lab)

6) Since there is no configuration link, the only way to access the config is by going to it directly, so to go www.yourwebsite.com/admin/structure/crumbs
7) Under Disabled by default, drag up the entityreference.* to Enabled.

Save Changes

8) Go to your Forex Advantage node and now you should see
BEGGINERS COURSE >> Forex Advantage
Comment: "Entity Reference" does not exist in FILE TYPE
Comment: oh, sorry missed a step, you need to download the [Entity Reference](https://www.drupal.org/project/entityreference) module
Comment: 4) Go to your Forex Advantage node and edit it.
   For File Under put BEGINNERS COURSE
   Save changes.---->I don't understand what it means.
Comment: #1 Go to your Forex Advantage page. #2 click edit button #3 For File Under put BEGINNERS COURSE #4 Click save button.
Comment: by the way, in Drupal: **node = page**
Comment: I did't it ,but it's not work!!
Comment: Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/31341/discussion-between-wuchuan-and-no-sssweat).
Comment: you must of done something wrong. Did you save changes in step 7?
Comment: yes.I saved it,I give you a screenshot
Comment: perhaps, you need to Clear the cache `/admin/config/development/performance` click on clear all caches
Comment: ok ok I see where the error is. In File Under, type `Beg` and you should see it shows you an option for BEGINNERS COURSE, click on it, now you should see something like this BEGINNER COURSE (8)  <--- note your number is probably not 8, but that's ok, don't worry about that. Then click save.
Comment: Go To eLesson content type Manage Fields. Maybe it's better if you change the File Under widget from autocomplete to select list.
Comment: Now only show:Home » BEGINNERS COURSE;
Comment: ok ok if I have a problem, I will ask you for advice.:)
Comment: Just saying, instead of entityreference.* you maybe only want to enable entityreference.field_file_under.*. This way you avoid that other entityreference fields have any effect on the breadcrumb.


php session destroyed in android application

Question: I am building a login application in android in which i am hitting a url(with username and password) upto that part it works fine but after that whenever I am hitting a url(once the user is authenticated) , it return nothing(i.e. a error message like please login first).
However it works fine in very similar iphone app and on browser.
I could not understand that what is the actual problem. Is there any problem on server side(i.e. in php coding ) or there is something wrong with android app. We cant actually blame the server side coding because it works fine for same iphone app and on browser.
Any idea or help will be highly appreciated.
Thanks.
Answer: Your Php script uses a session token to identify the visitor when you browse the site via a browser. Ususally its stored in a PHPSESSID variable.
If you want your Android application to stay authenticated on the server side you need to fetch that id after the first connection and then send it in the headers of all your subsequent requests.
**EDIT : ** This is what you want to do : How do I manage cookies with HttpClient in Android and/or Java?
Comment: Because on the browser and probably on your iphone(browser i guess) the browser actually handles the passing of the php session for you via the usage of cookies which it sends automatically to the server in the headers of the requests.
Comment: you mean i have to sent that unique id with the url for further request.
Comment: Exactly, that way your server will know what session it needs to fetch to see if the user is authenticated. It's really a cookie usually that you need to resend with every request. Take a look here : http://stackoverflow.com/questions/3587254/how-do-i-manage-cookies-with-httpclient-in-android-and-or-java
Comment: But one thing, the same thing is working on the browser and in iphone . So , is this issue only for android and why. Can you give some idea ?
Comment: Thanks , I got your point but can you tell me that how that can be handled in android .
Comment: I believe the link that i provided in my question edit should put you on the right track. Basically : Get the session id from the header after your authentification and then send it back with every request as a cookie like the link explains how to. Good luck
Comment: Thanks yahel. I will try this and let you know if the problem continues. Thanks a lot.
Comment: @Yahel i have something similar problem can you check it?  http://stackoverflow.com/questions/26840060/how-to-pass-id-with-url-while-requesting
Answer: You can use an active DefaultHttpClient connection and make validations on each mobile-request, at server side, through $_SESSION. Here is a small sample code (is not for production)
First there is a class:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.ArrayList;

import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.BasicNameValuePair;

public class NetClient {

    private HttpClient client = null;

    public NetClient() {
        client = new DefaultHttpClient();
    }

    public String request(String url) throws ClientProtocolException, IOException {
        HttpPost post = new HttpPost(url);
        HttpResponse res = client.execute(post);

        BufferedReader br = new BufferedReader(new InputStreamReader(res.getEntity().getContent()));
        String data = "";
        String line = "";
        while ((line = br.readLine()) != null) {
             data = data + line;
        }
        return data;
    }

    public String login(String url) throws ClientProtocolException, IOException {
        HttpPost post = new HttpPost(url);

        ArrayList pa = new ArrayList();
        pa.add( new BasicNameValuePair( "username", "admin"));
        pa.add( new BasicNameValuePair( "password", "admin"));
        post.setEntity( new UrlEncodedFormEntity(pa, "UTF-8"));

        HttpResponse res = client.execute(post);

        BufferedReader br = new BufferedReader(new InputStreamReader(res.getEntity().getContent()));
        String data = "";
        String line = "";
        while ((line = br.readLine()) != null) {
             data = data + line;
        }
        return data;
    }
}

Then at mainactivity.java
StrictMode.ThreadPolicy policy = new StrictMode.ThreadPolicy.Builder().permitAll().build();
StrictMode.setThreadPolicy(policy);

try {
    NetClient n = new NetClient();

    String k = n.login("http://x.com/testAREA/securetrans/login.php");
    Log.w("login result", k);

    String l = n.request("http://x.com/testAREA/securetrans/content.php");
    Log.w("ask if logged in", l);

} catch (ClientProtocolException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
} catch (IOException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
}

Where login.php
<?php
session_start();

if(isset($_POST['username']))
{
    $username = $_POST['username'];
    $password = $_POST['password'];

    if($username == 'admin' && $password == 'admin')
    {
        $_SESSION['username'] = $username;

        echo "admin setted";

        exit;
    }
    else
    {
        echo "-1";
    }
}
?>

and content.php as :
<?php
session_start();

if(isset($_SESSION['username']))
{
    echo "login ok";
}
else
{
    echo "not login";
}
?>

When we run this sample code will end with
echo "login ok";

More information at: http://www.pipiscrew.com/2014/11/phpandroid-secure-connection-with-session-variable/
Answer: I have encountered this very recently, the PHP session was not getting created when we hit the API from Android.
After spending some time, I figured it out.

For creating sessions you need to enable cookies for the API call.
  you have to set the handler that can accept cookies from incoming HTTP responses and provides cookies to outgoing HTTP requests.

For enabling cookies you have to add cookieJar in your OkHttpClient.
import okhttp3.JavaNetCookieJar
import java.net.CookieManager

    val okHttpClient = OkHttpClient.Builder()
                .cookieJar(JavaNetCookieJar(CookieManager()))
                .build()


Show why the estimate of variance component using REML is unbiased

Question: I'm trying to use a very simple example to illustrate how REML makes the estimate of variance component unbiased:
Consider $X_1,\dots,X_n\overset{i.i.d.}{\sim}\mathcal{N}(\mu,\sigma^2)$, we denote one realization as $x_1,\dots,x_n$. We are interested in estimating $\sigma^2$. If we choose ML approach, we will have $\hat{\mu}_{\text{ML}}=\bar{x}=\sum_{i=1}^nx_i,\hat{\sigma}^2_{\text{ML}}=\frac{1}{n}\sum_{i=1}^n(x_i-\bar{x})^2$. Let $X^\top=[X_1,\dots,X_n]^\top$, so $X\sim\mathcal{N}(\mu\mathbf{1}_n,\sigma^2\mathbf{I}_n)$. We can transform $X$ into $N_{\mathbf{1}_n}X$, where $N_{\mathbf{1}_n}=\mathbf{I_n}-\frac{\mathbf{1}_n\mathbf{1}_n^\top}{n}$, now $N_{\mathbf{1}_n}X\sim\mathcal{N}(\mathbf{0}_n,\sigma^2N_{\mathbf{1}_n})$. However, we cannot write the p.d.f. of $N_{\mathbf{1}_n}X$ since $N_{\mathbf{1}_n}$ is not invertible.
Let
$$
E_{n-1,n}=
\begin{bmatrix}
1 & 0 & \cdots & 0 & 0\\
0 & 1 & \cdots & 0 & 0\\
\vdots & \vdots & \ddots & \vdots & \vdots\\
0 & 0 & \cdots & 1 & 0
\end{bmatrix}_{n-1\times n}
$$
Then $E_{n-1,n}N_{\mathbf{1}_n}X\sim\mathcal{N}(\mathbf{0}_{n-1},\sigma^2E_{n-1,n}N_{\mathbf{1}_n}E_{n-1,n}^\top)$, with
$$
(E_{n-1,n}N_{\mathbf{1}_n}E_{n-1,n}^\top)^{-1}=
\begin{bmatrix}
2 & 1 & \cdots & 1 \\
1 & 2 & \cdots & 1 \\
\vdots & \vdots & \ddots & \vdots \\
1 & 1 & \cdots & 2
\end{bmatrix}_{n-1\times n-1}
$$
Hence we can write down the likelihood for $E_{n-1,n}N_{\mathbf{1}_n}X$:
$$
\frac{1}{(2\pi)^{\frac{n-1}{2}}(\sigma^2)^{\frac{n-1}{2}}[\det(E_{n-1,n}N_{\mathbf{1}_n}E_{n-1,n}^\top)]^{\frac{1}{2}}}\exp(-\frac{1}{2 \sigma^2}X^\top N_{\mathbf{1}_n}E_{n-1,n}^\top (E_{n-1,n}N_{\mathbf{1}_n}E_{n-1,n}^\top)^{-1}E_{n-1,n}N_{\mathbf{1}_n}X)
$$
Take log:
$$
\text{const}-\frac{n-1}{2}\log\sigma^2-\frac{1}{2\sigma^2}\color{red}{X^\top N_{\mathbf{1}_n}E_{n-1,n}^\top (E_{n-1,n}N_{\mathbf{1}_n}E_{n-1,n}^\top)^{-1}E_{n-1,n}N_{\mathbf{1}_n}X}
$$
I'm stuck with showing $\color{red}{X^\top N_{\mathbf{1}_n}E_{n-1,n}^\top (E_{n-1,n}N_{\mathbf{1}_n}E_{n-1,n}^\top)^{-1}E_{n-1,n}N_{\mathbf{1}_n}X}=SSE=\sum_{i=1}^n(x_i-\bar{x})^2$, which doesn't seem very obvious to me.
Or could someone suggest me a more elegant way to show $\hat{\sigma}^2_{\text{REML}}=\frac{1}{n-1}\sum_{i=1}^n(x_i-\bar{x})^2$ in this example?
Any help appreciated!
Comment: I read it on _Extending the Linear Model with R. Generalized Linear, Mixed Effects and Nonparametric Regression Models(Faraway)_. Supposing $y\sim\mathcal{N}(X\beta,\sigma^2K)$, from the book, it says "The idea (of REML) is to find all independent linear combinations of the response, $k$, such that $k^\top X = 0$. Form matrix $K$ with columns $k$, so that: $K^\top y\sim \mathcal{N}(0, K^\top VK)$ "
Comment: So I guess my choosing $K^\top$ to be $E_{n-1,n}N_{\mathbf{1}_n}$ is the right thing to do here?
Answer: I found this REML estimation of variance
components lecture note extremly helpful!
I'm inspired to do eigendecomposition with $N_{\mathbf{1}_n}$ so I get
$$
N_{\mathbf{1}_n}=B
\begin{bmatrix}
1 & 0 & \cdots & 0 & 0\\
0 & 1 & \cdots & 0 & 0\\
\vdots & \vdots & \ddots & \vdots & \vdots \\
0 & 0 & \cdots & 1 & 0\\
0 & 0 & \cdots & 0 & 0
\end{bmatrix}
B^\top,
$$
where $B^\top B=\mathbf{I}_n$. If we take first $n-1$ rows of $B^\top$ and denote them as $C^\top$, then we have
$$
C^\top C=\mathbf{I}_{n-1},CC^\top=N_{\mathbf{1}_n}
$$
Note $C^\top X\sim\mathcal{N}(\mu C^\top\mathbf{1}_n,\sigma^2C^\top C)$. Since $\mathbf{0}_n=N_{\mathbf{1}_n}\mathbf{1}_n=CC^\top \mathbf{1}_n$, we can left multiply both sides by $\mathbf{1}_n^\top$ to get $\mathbf{1}_n^\top\mathbf{0}_n=0=\lVert C^\top\mathbf{1}_n\rVert$, which implies $C^\top\mathbf{1}_n=\mathbf{0}_{n-1}$!
So we have $C^\top X\sim\mathcal{N}(\mathbf{0}_{n-1},\sigma^2\mathbf{I}_{n-1})$. Writing down the log-likelihood:
$$
\text{const}-\frac{n-1}{2}\log\sigma^2-\frac{1}{2\sigma^2}X^\top N_{\mathbf{1}_n}X
$$
I suppose this is a so much better way than my first attempt :)


How to compute a lot of digits of $\sqrt{2}$ manually and quickly?

Question: After having read the answers to calculating $\pi$ manually, I realised that the two fast methods (Ramanujan and Gauss–Legendre) used $\sqrt{2}$. So, I wondered how to calculate $\sqrt{2}$ manually in an accurate fashion (i.e., how to approximate its value easily).
Comment: Check [this page](http://en.wikipedia.org/wiki/Square_root_of_2#Computation_algorithms) [Wikipedia, Square root of 2 -- Computation algorithms].
Comment: More generally, see [this wikipedia page](http://en.wikipedia.org/wiki/Methods_of_computing_square_roots) for how to approximate any square root. You can use [Newton's method](http://en.wikipedia.org/wiki/Newton%27s_method) for how to approximate roots of real-valued function (so in this case you can use it for the roots of $x^2-2$).
Comment: Do you know by any chance how many digits of root 2 that we need for accuracy in order to get the 707 digits of pi using Ramanujan's formula?
Answer: One really easy way of approximating square roots surprisingly accurately was actually developed by the Babylonians.
First they made a guess at the square root of a number $N$--let this guess be denoted by $r_1$. Noting that
$$
r_1\cdot\left(\frac{N}{r_1}\right)=N,
$$
they concluded that the actual square root must be somewhere between $r_1$ and $N/r_1$. Thus, their next guess for the square root, $r_2$, was the average of these two numbers:
$$
r_2 = \frac{1}{2}\left(r_1+\frac{N}{r_1}\right).
$$
Continuing in this way, in general, once we have reached the $n$th approximation to the square root of $N$, we find the $(n+1)$st using
$$
r_{n+1}=\frac{1}{2}\left(r_n+\frac{N}{r_n}\right).
$$
All that you really need to do is make a moderately decent guess of the square root of a number and then apply this method two or three times and you should have quite a good approximation.
For $\sqrt{2}$, simply using a guess of $1$ and applying this method three times (the algebra involved is remarkably simple) yields an approximation of
$$
\frac{577}{408}\approx \color{red}{1.41421}\color{blue}{568627},
$$
whereas
$$
\sqrt{2}\approx \color{red}{1.41421}\color{green}{356237}.
$$
That's quite a good approximation using an easy and quick manual method.
Comment: This is surprisingly the exact recursion that results from applying Newton's method.  So any method that is better than this would have to be better than Newton's method.
Answer: you can use the formula
$$\frac{a_{n+1}}{b_{n+1}}=\frac{a_n^2+2b_n^2}{2a_nb_n}$$
if we take an initial value of $\sqrt{2}$ as $\frac{3}{2}$
now the new value will become
$$\frac{a}{b}=\frac{3^2+2*2^2}{2*2*3}=\frac{17}{12}$$
the new value of $a=17$ and $b=12$
and then continue
Comment: The identity you start with is not true. For example, taking $a=b=1$ it claims that $\frac11 = \frac32$ which is blatantly false.
Comment: @HenningMakholm $$\frac{1^2+2*1^2}{2*1*1}=\frac{3}{2}$$ and the next $$\frac{3^2+2*2^2}{2*3*2}=\frac{17}{12}$$ and then $$\frac{17^2+2*12^2}{2*17*12}=\frac{577}{408}$$ if you need more accuracy, continue
Comment: Yes, that evaluates to $\frac32$ but $\frac32$ is not the same as $\frac11$, so your claim $\frac ab=\frac{a^2+2b^2}{2ab}$ is false. Do you have _any_ values of $a$ and $b$ that it holds for?
Comment: ok, I understood what do you mean.The formula should be written $$\frac{a_{n+1}}{b_{n+1}}=\frac{a_n^2+2b_n^2}{2a_nb_n}$$
Comment: @HenningMakholm now ,Is it ok?
Comment: x @Hard: Yes, and the same iteration as in crash's answer.
Answer: I think crash's post's method is the best, but if you don't want to do a lot of long divisions, then here is an alternative method for the lazy.
Suppose you want to compute $\sqrt{2}$ to $k$ decimal places.  That is, you want to find $x$ in:
$$x \cdot 10^{-k} \approx \sqrt{2}$$
$$x^2 \approx 2 \cdot 10^{2k}$$
This allows you to find $x$ using a binary search:  doing approximately $\log(k)$ multiplications of a $k$ digit number (and computing the average of the upper and lower bound, just adding and dividing by $2$ is easy by hand).
And the accuracy of this result guaranteed by construction.  Suppose you want to calculate $\sqrt{2}$ to $8$ decimal places:
$$x^2 = 2 \cdot 10^{16}$$
$$\begin{array} {c|ccc} \text{Step} & \text{LowerBound} & \text{UpperBound} &\text{MidPoint} \\
1 & 100000000 & 1000000000 & 550000000 \\
2 & 100000000 & 549999999 & 324999999 \\
3 & 100000000 & 324999998 & 212499999 \\
4 & 100000000 & 212499998 & 156249999 \\
5 & 100000000 & 156249998 & 128124999 \\
6 & 128125000 & 156249998 & 142187499 \\
7 & 128125000 & 142187498 & 135156249 \\
8 & 135156250 & 142187498 & 138671874 \\
9 & 138671875 & 142187498 & 140429686 \\
10 & 140429687 & 142187498 & 141308592 \\
11 & 141308593 & 142187498 & 141748045 \\
12 & 141308593 & 141748044 & 141528318 \\
13 & 141308593 & 141528317 & 141418455 \\
14 & 141418456 & 141528317 & 141473386 \\
15 & 141418456 & 141473385 & 141445920 \\
16 & 141418456 & 141445919 & 141432187 \\
17 & 141418456 & 141432186 & 141425321 \\
18 & 141418456 & 141425320 & 141421888 \\
19 & 141418456 & 141421887 & 141420171 \\
20 & 141420172 & 141421887 & 141421029 \\
21 & 141421030 & 141421887 & 141421458 \\
22 & 141421030 & 141421457 & 141421243 \\
23 & 141421244 & 141421457 & 141421350 \\
24 & 141421351 & 141421457 & 141421404 \\
25 & 141421351 & 141421403 & 141421377 \\
26 & 141421351 & 141421376 & 141421363 \\
27 & 141421351 & 141421362 & 141421356 \\
28 & 141421357 & 141421362 & 141421359 \\
29 & 141421357 & 141421358 & 141421357 \\
\end{array}$$
$29$ $8$-digit multiplications for $8$ decimal places of accuracy (and a lot of even that was redundant).
Comment: Hm, can you explain how you did it?
Comment: @Mageek Do you know what a binary search is?  Like if I say "I'm thinking of a number between 1 and 100" and you guess 50 and I say "too low" so now you know between 50 and 100 so you guess 75, keep going, that is a binary search.
Comment: @Mageek This example is much less efficient than crash's, it is just an alternative that is easy to understand.
Answer: Another technique might be to use the Taylor series
$$(1+x)^{1/2} = 1+ \frac 12 x - \frac 18 x^2 + \frac{1}{16} x^3 - \frac{5}{128} x^4 +\cdots.$$
The coefficients of this series are
$\frac{(-1)^k }{k!} \left(\frac12\right) \left(-\frac12\right)\left(-\frac32\right)\cdots\left(\frac32 - k\right)$.
You can plug in $x=1$ so that the series evaluates to $\sqrt2$,
but the series converges faster if you start with a rational approximation
of $\sqrt2$ and use the Taylor series to compute a correction factor,
for example $\sqrt 2 = 1.4 \cdot \left(1 + \frac{1}{49}\right)^{1/2}.$


ASP.NET WebService deny remote access

Question: I've created an ASP.NET WebService that is to be consumed using ASP.NET Ajax. The WebService is located on the same box and same web application that it is to be used by, so I do not want to allow remote access to this webservice, but have it only respond to requests from localhost.
The Web.Config DOES NOT have a  configuration section and therfore does not have httpPost and httpGet turned on. This is fine. However, if I navigate directly to the WebService URL from a remote machine, it still loads and shows me a list of methods. Clicking on the method does give me a message stating that the testing form is not available to remote machines (as intended), but it does list information on how to issue a Soap Request and handle a Soap Response.
Additionally, I believe I'm being scraped by a bot of some sort of just a curious user, because I'm now getting error message in my log such as this...
System.InvalidOperationException: Request format is unrecognized for URL
unexpectedly ending in '/ValidateUsername'.

This happens if you try to issue a GET request (by manipulating the query string) against the service remotely. I'm glad that it's not handling the request as I don't want remote users access to this service, but I would prefer it not throw an error.
How can I lock down the webservice so that it is not available to remote machine, but still available to the local machine as a ScriptService consumably by ASP.NET Ajax?
UPDATE:
Okay, here is workable example of what is happening.
WebSite: http://so.weirdwes.dyndns.org/default.aspx
WebService: http://so.weirdwes.dyndns.org/services/services.asmx
Web.Config:
<webServices>
  <protocols>
    <remove name="HttpGet"/>
    <remove name="HttpPost"/>
  </protocols>
</webServices>

The website is consuming the WebService using a ScriptManager tag and ScriptReference. You'll note if you click the button, the web service is called and everything works, even though Post and Get have been removed. This is fine as this is how we want it to work. The issue is this....
http://so.weirdwes.dyndns.org/services/services.asmx/GetRemoteAddr
Server Error in '/' Application.
--------------------------------------------------------------------------------

Request format is unrecognized for URL unexpectedly ending in '/GetRemoteAddr'.

A bot or something is scraping this URL and it's generating errors that we're get notified of. I want to supress this error or block it entirely. If I alter the Web.Config and add the Get and Post protocols back in, this error goes away - but then it allows access to the web service remotely using Get which we don't want.
Answer: Test against  the ip address
Request.ServerVariables ["REMOTE_ADDR"]
Comment: I've thought about that, but I'm not even sure it gets to that a point that I can do any testing before it throws the exception I mentioned above. I can give it a shot though.
Answer: IIS lets you white/black list IP groups in its configuration.  Use that to lock it down to localhost.  You can also configure your firewall to prevent anyone from hitting that port from outside.
Answer: I am kinda confused here.
Is this going to be called by the browser? if it will be, then you should allow remote access.
Comment: Hmm, that's a good question. I guess I'm not sure how that works. It's being called via the Web Application and the Web Application itself is publically available. Basically I have a  tag in my page and that has a  tag pointing at the .asmx file.

Then the webservice is being called in various places using JavaScript from the same web application. So, as I understand it, those requests are sceen as coming from localhost - which I want to allow. But if someone were to access the URL directly, the request would an external IP and then I'd want to deny access.
Comment: I am pretty sure it is getting called directly by the browser. One way to make sure is log the ip in your service and access your page from a different machine.
Comment: check this out..it might be what you are looking for..

http://weblogs.asp.net/rashid/archive/2007/09/20/asp-net-ajax-web-service-security.aspx
Comment: or this..
http://www.google.com/#hl=en&ei=-QQpS9r8CMmztgerr9jDCw&sa=X&oi=spell&resnum=0&ct=result&cd=1&ved=0CAoQBSgA&q=securing+ajax+web+services&spell=1&fp=cbc2f75bf9d43a8f
Comment: Thanks PS, I think the weblog.asp.net thing might be the way to go. It seems, as you've said, that POST and GET really should be enabled for a public facing Web Service like this - and then just use a security measure to block access. It appears if I remove the POST and GET, then it's going to generate errors when people access it and there really isn't a way around it - so I think I'll go your route.

Thanks!

ASP.NET WebService deny remote access


Javascript passing parameter with arrow function

Question: I have a class for creating a jquery datatable, it works but I'm having some issue with the following piece of code:
 let table = $("#" + this.name).DataTable();
 table.on( 'click', 'tr td.details-control', function() {
        let detailRows = [];
        let tr = $(this).closest('tr');
        let row = table.row( tr );
        let idx = $.inArray( tr.attr('id'), detailRows );

        if ( row.child.isShown() ) {
            tr.removeClass( 'details' );
            row.child.hide();

            detailRows.splice( idx, 1 );
        }
        else {
            tr.addClass( 'details' );
            let rowData = table.row(tr).data();
            row.child(this.formatChildRows( rowData ) ).show();

            if ( idx === -1 ) {
                detailRows.push( tr.attr('id') );
            }
        }
    } );

Because of the way "this" works in javascript, the function formatChildRows is no longer found as this no longer refers to the class but to the table row, I solved this by changing the code like so:
table.on( 'click', 'tr td.details-control', (event) => {
    ...
} );

The function now runs but the "table" variable is no longer accessible as it is undefined in the arrow function.
I tried plenty of things like:
evt => function(table) {}

instead of:
(event) =>

I thought this might work but the code inside the function is no longer being executed for some reason.
Any ideas?
Comment: Unless you supersede it with something like `(table) =>` or `var table` within the callback, or change its value after the fact `table` should continue to refer to the original `table` value. Can you show us a complete example of some code where this isn't working?
Comment: `The function now runs but the "table" variable is no longer accessible as it is undefined in the arrow function.` -> NO?!
Answer: This should work:
const self = this;
table.on( 'click', 'tr td.details-control', function (event){
    row.child(self.formatChildRows( rowData ) ).show();
    ...
} );

But if it is possible you should avoid this keyword, for example you should refer directly to object objectName.formatChildRows().