Does this approach to simulation for survival analysis, of breaking the analysis into deaths versus survivors, appear reasonable?

Question: I've spent last several weeks learning about survival analysis, see one of the last posts at How to simulate variability (errors) in fitting a gamma model to survival data by using a generalized minimum extreme value distribution in R?
Now I am primarily concerned with simulating death rates and secondarily deriving survival curves for the deceased. Ultimately, this is leading towards simulating deaths/survival using extreme value distributions with heavy right tails (even if not best-fitting) for simulating conservative, very-bad-case scenarios especially when dealing with a paucity of data. The code below is a first step in that direction.
Does the approach I describe and per the code below appear reasonable? If so, are there easier or better approaches?

I use the lung dataset from the survival package as my example.
I use bootstrap sampling (bootSample() in code below) to derive death rates (deathRate <- ...) and to extract the lung data for only the deaths from the same bootstrapped samples where "status" == 2 (bootDeaths[[i]] <<- ...).
Using AIC, lognormal provided the best fit for bootstrap sampled death rates. Code not shown for this goodness-of-fit testing.
I draw lognormal random samples for each of the bootstrap samples and derive a histogram of death rates per the image below on the left.
I then take the deaths from the same bootstrapped samples and fitting the deaths with the survreg() function and applying the lognormal distribution, plot their survival curves (plot_survival_curves(...)) as shown in the image below on the right.

Code:
library(MASS)
library(survival)

nbr <- 100
timeLine <- seq(0, max(lung$time))
bootDeaths <- list()

# Use bootstrapping for both average death rates and for plotting survival curves for deaths
bootSample <- sapply(
  1:100,
  function(i) {
    sampleData <- lung[sample(nrow(lung), replace = TRUE), ]
    bootDeaths[[i]] <<- sampleData[sampleData$status == 2, ] # used in plotting death survival curves later
    deathRate <- with(sampleData, mean(status == 2))
    return(deathRate)
  }
)

### Generate random samples for the lognormal distribution, calculate and plot death rates ###
fit <- MASS::fitdistr(bootSample,"lognormal")
params <- fit$estimate
sampLognorm <- rlnorm(1000, params[1], params[2])
hist(sampLognorm, breaks = "FD", col = "steelblue",
     xlab = "Death rate", ylab = "Frequency", main = "Histogram of Lognormal Samples")
sampDeathRate <- mean(bootSample)
abline(v = sampDeathRate, col = "black", lty = 1, lwd = 3)
popDeathRate <- with(lung, mean(status == 2))
abline(v = popDeathRate, col = "red", lty = 1, lwd = 3)
legend("topright", legend = c(paste("Sample Average:", round(sampDeathRate, 4)),
                              paste("Population Average:", round(popDeathRate, 4))),
       lty = c(1,1), lwd = c(3,3), col = c("black", "red"), bty = "n")

### Lognormal survival curves for patients who die ###
plot(timeLine, type = "n", xlab = "Time", ylab = "Survival Probability", main = "Lung Data Survival Plot", ylim = c(0, 1), xlim = c(0,max(lung$time)))

# Fit lognormal distribution and plot survival curves for each deceased sample
plot_survival_curves <-  sapply(
  1:nbr,
  function(i){
    sampleDat <- data.frame(bootDeaths[[i]])
    fit <- survreg(Surv(time, status == 2) ~ 1, data = sampleDat, dist = "lognormal")
    meanlog <- fit$coef
    sdlog <- fit$scale
    surv_prob <- 1 - plnorm(timeLine, meanlog = meanlog, sdlog = sdlog)
    lines(seq(0,length(surv_prob)-1), surv_prob, col = "lightblue", lty = "solid", lwd = 0.25)
  }
)

Comment: This appears to be a very roundabout way to go about it.  And you are assuming that the bootstrap distribution mimics the sampling distribution which is often not the case.
Answer: One big problem comes to mind: even without censored observations, there is no single "death rate" (e.g., deaths per person at risk per year) unless there's an exponential survival curve. The hazard function is the continuous death rate over time, and it's far from constant over time for a lognormal distribution; see the NIST page for example plots of lognormal hazard functions.
Without an exponential survival curve, any "death rate" based on dividing a number of deaths by a number at risk will depend on the particular time window involved. What exactly do you mean by a "death rate" in this context? How would you apply it in practice?
In your situation this problem is exacerbated by omitting right-censored observations from the calculations. That necessarily introduces bias into the estimate, in a way that depends heavily on the censoring pattern in the underlying data. It's hard to think of a scenario in which omitting censored event times leads to anything other than trouble.
Survival analysis done properly allows for other than exponential survival curves while handling censored event times. It allows for predictions of survival at specific times of interest, so that if you are particularly concerned about, say, early events you can focus attention on them.
In terms of the sampling variability you want to capture (variability in survival-curve estimates arising from random sampling of the population) the covariance matrix of the coefficient estimates should contain the information you need.
If the data set is large enough and the form of the model is adequate, then the asymptotic multivariate normality of the estimates can be used directly without resampling. I showed that at the end of my answer to the question you linked, where resampling from the survival distribution gave essentially the same coefficient covariance as the original model's coefficient covariance matrix.
If the form of the model isn't adequate, then modeling on resampled cases might give an estimate of a type of variability: the variability in coefficient estimates in a model that doesn't properly fit the data. What's the point? You are better served by developing an adequate model.
If the data set isn't large enough, you have to consider whether to trust the model at all. Your statement about "paucity of data" is thus troubling. It might be more useful to present the actual scenario that you are interested in, as I fear that the "paucity of data" will limit what can be accomplished.
Comment: @Village.Idyot unless you have complete follow up on all patients, those calculations of "death rates" are erroneous. The denominator continues to include people lost to follow up for reasons other than death. If you do have complete follow up that might be OK, but that's not typical of survival studies and certainly is not the case in the `lung` data set. Survival analysis takes that problem into account, provided that censoring isn't informative about death status. If your real data have complete follow up, why not ask a question directly about analyzing your real data?
Comment: @Village.Idyot for example, if you believe the lognormal model, the expected fraction of individuals who will have died by `time=500` is 69.2%, and by `time=1022` it's 87.6%. Those are both substantially higher than your estimates for those times based on observed deaths and the original numbers of patients. You are correct that more events leads to better precision in fitting a model, but the real problem in predicting beyond the last observation time is whether you have fit the proper parametric form. Perfectly identifying coefficients of the wrong model accomplishes nothing for prediction.
Comment: By paucity I mean working with data when part way through a study. In the `lung` example the death rate over the complete study time window (max time 1022) is 72.4% calculated as the number of status = 2 out of 228 patients. If we were sitting in the 500th month of the study, 60.5% of the patients would have died. Based on the data through month 500, how can we start molding our expectation of future death rate probabilistically? Going back to earlier queries like https://stats.stackexchange.com/questions/614198/how-to-generate-multiple-forecast-simulation-paths-for-survival-analysis
Comment: I note your answer in https://stats.stackexchange.com/questions/608652/which-are-the-statistical-methodologies-to-consider-when-examining-study-group-d where I could use binomial for live/die, and then estimate the time curves for deaths through survival or other. However, I prefer survival because in addition I'll need to introduce multi-state variables such as stage 1, 2, 3, 4, as patients get sicker which I understand the `survival` package accommodates.


Why all subgroups of order 2 of Alternating group A5 are conjugate?

Question: By Sylow Theorem, all Sylow p-subgroups of $A_5$ are conjugate, for any $p\in \{ 2,3,5\}$.
But why  all subgroups of Alternating group A5 of order 2 are conjugate?
Answer: Hint: The elements of order $2$ in $S_n$ are the products of disjoint transpositions. What are the elements of order $2$ in $A_5$?
Comment: I don't know exactly.
Comment: Could you explain a bit more?
Comment: @M.Ramana, here are two: $(12)(34)$, $(24)(35)$. Can you prove that they are conjugate?


Facet blocks in a dialog don't retain applied facets

Question: Cross posting an issue I posted from Drupal.org in the hope that it gets a bit more visibility here.
I'm currently loading facet blocks via AJAX into an off canvas dialog provided by the core Dialog API. To achieve this, I have a controller set up that accepts a facet ID as a parameter, and then loads the block (I've attached the code at the end of this question as an example).
This works perfectly on the face of it (all of my facets are listed, the dialog works and shows the facet widgets for each facet). However, what I've noticed is that if I have a facet enabled, this isn't taken into account when generating facet links. For example; if I have a facet applied for language, and then look at the link generated for my collection facets, the facets link to my search page with the collection facet applied, but without the existing language facet.
The search view itself is not an AJAX view, it's just the loading of the facet blocks. Unfortunately, this is a requirement as we have a large amount of facets that kill the performance if loaded as part of the main request.
Here are some things I've tried:

Adding the main page parameters to my dialog links
Using these query parameters to configure, load, and execute my search view during the AJAX request before the facet block is generated (see following example)

These facet blocks are only ever rendered on the same page as the facet source View, however as it's not the same request it may as well be on a different page.
I guess my question is one of multiple parts:

Am I missing something obvious in my approach
Is it possible to do what I'm trying to achieve
If it is and I'm going the wrong way about it, what's the right way to go about it

As an aside, I'm rendering the blocks isn't a requirement if it's easier to render the facets themselves - it just felt like it would be easier.
As promised, here's my controller code. As mentioned, the facet blocks display, but already-applied facets aren't taken into account during link generation.
MY_MODULE.routing.yml
my_facets.ajax_facet_block:
  path: '/ajax/my-facets/{facet_id}'
  defaults:
    _title: 'Filter by'
    _controller: '\Drupal\my_facets\Controller\AjaxFacetBlockController::renderFacetBlockAction'
  requirements:
    _permission: 'access content'

Controller
<?php

namespace Drupal\my_facets\Controller;

use Drupal\Core\Block\BlockManagerInterface;
use Drupal\Core\Controller\ControllerBase;
use Drupal\Core\Plugin\PluginBase;
use Drupal\Core\Render\RendererInterface;
use Drupal\views\Views;
use Symfony\Component\DependencyInjection\ContainerInterface;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpKernel\Exception\NotFoundHttpException;

/**
 * Returns responses for Facets routes.
 */
class AjaxFacetBlockController extends ControllerBase {

  public function __construct(
    protected BlockManagerInterface $blockManager,
    protected RendererInterface $renderer,
    protected Request $request
  ) {
  }

  public static function create(ContainerInterface $container): static {
    return new static(
      $container->get('plugin.manager.block'),
      $container->get('renderer'),
      $container->get('request_stack')?->getCurrentRequest()
    );
  }

  /**
   * Builds the response.
   */
  public function renderFacetBlockAction(string $facet_id) {
    // We need to execute the search view to correctly generate links.
    // @ToDo: Once this is working inject the entity type manager rather than calling `Views::getView`
    $search_view = Views::getView('search');

    if (!$search_view) {
      throw new NotFoundHttpException();
    }

    $search_view->setDisplay('search_page');

    $search_view->setExposedInput(array_filter(
      $this->request->query->all(),
      static fn ($k) => !str_starts_with('_', $k),
      ARRAY_FILTER_USE_KEY
    ));

    $non_facet_params = array_filter(
      $this->request->query->all(),
      static fn ($key) => $key !== 'f' && !str_starts_with($key, '_'),
      ARRAY_FILTER_USE_KEY
    );

    $search_view->exposed_raw_input = array_merge(
      $search_view->exposed_raw_input,
      $non_facet_params
    );

    $search_view->exposed_data = array_merge(
      $search_view->exposed_data,
      $non_facet_params
    );

    $search_view->preExecute();
    $search_view->execute();

    try {
      /** @var \Drupal\facets\Plugin\Block\FacetBlock $block */
      $block = $this->blockManager->createInstance(
        'facet_block' . PluginBase::DERIVATIVE_SEPARATOR . $facet_id, []
      );
    }
    catch (\Exception) {
      throw new NotFoundHttpException();
    }

    if (!$block || !$block->access($this->currentUser())) {
      throw new NotFoundHttpException();
    }

    return $block->build();
  }

}

Comment: What are you doing about caching of your controller? Sounds like the response is being cached
Comment: Is the Ajax response supposed to vary by `$facet_id`? If yes, you haven't set caching metadata on the block.
Comment: Thanks both - but the caching wasn't the issue, it was a Facets internal thing.
Answer: This ended up being due to the QueryString processor provided by the Faces module performing a check for XmlHttpRequests as it's very first step (I assume to cover ajax based Views). If it finds that a request is an XmlHttpRequest, then it'll act on a subrequest (which in my case doesn't exist).
My rather rudimentary fix was to strip the X-Requested-With header from the request with an event listener on the KernelEvents::REQUEST event. This makes the underlying Symfony request respond as a standard request (read: non-XMLHttpRequest), and allowed the query string processor to act on the correct response instead of the sub-response.


Como remover a borda do  sem remover minha borda feita no css?

Question:

#botoes {
  width: 270px;
  margin-left: 20px;
}

#botao {
  float: right;
  padding: 5px;
  15px;
  font: bold 12px sans-serif;
  border-radius: 20px;
  box-shadow: 0px 1px 0px white;
  border: 1px solid #9eb9c3;
  background: #edf6f9;
  background: -webkit-linear-gradient(#edf6f9 0%, #cde5ee 100%);
  background: -moz-linear-gradient(#edf6f9 0%, #cde5ee 100%);
  background: -o-linear-gradient(#edf6f9 0%, #cde5ee 100%);
  background: linear-gradient(#edf6f9 0%, #cde5ee 100%);
  filter: progid:DXImageTransform.Microsoft.gradient( startColorstr='#edf6f9', endColorstr='#cde5ee', GradientType=0);
  color: #527988;
  box-shadow: 0px 0px 10pc #c9c9c9
}

#botao:hover {
  background: #cde5ee;
  background: -moz-linear-gradient(top, #cde5ee 0%, #edf6f9 100%);
  background: -webkit-linear-gradient(top, #cde5ee 0%, #edf6f9 100%);
  background: linear-gradient(to bottom, #cde5ee 0%, #edf6f9 100%);
  filter: progid:DXImageTransform.Microsoft.gradient( startColorstr='#cde5ee', endColorstr='#edf6f9', GradientType=0);
  cursor: pointer;
}

    <input type="submit" name="enviar" value="Enviar">

    <input type="reset" name="limpar" value="Reset">

    <input type="checkbox" />Lembrar senha

Answer: Basta na primeira linha do seus CSS vc colocar assim
[type="reset"] {
  all: unset;
}

Para o IE vc pode fazer dessa forma já que ele não aceita o unset (fonte)
[type="reset"] {
  border: none;
  background: none;
  padding: 0;
  margin: 0;
}

O importante é que esses estilos venham logo no início do seu CSS, antes dos estilos personalizados que vc colocou.
Dessa forma vc reseta todos os valores default desse tipo de input, depois vc coloca os estilos que vc deseja. OBS: Repare que eu não limpei os estilos do btn type="submit" para vc ver a diferença entre um e outro.
Veja o Exemplo aplicado com o seu código.

/* reseta estilos desse input */
[type="reset"] {
  all: unset;
}

  #botoes{
width:270px;
margin-left:20px;

}
#botao{
float:right;
padding:5px 15px;
font: bold 12px sans-serif;
border-radius:20px;
box-shadow:0px 1px 0px white;
border:1px solid #9eb9c3;

background: #edf6f9;
background: -webkit-linear-gradient(#edf6f9 0%, #cde5ee 100%);
background: -moz-linear-gradient(#edf6f9 0%, #cde5ee 100%);
background: -o-linear-gradient(#edf6f9 0%, #cde5ee 100%);
background: linear-gradient(#edf6f9 0%, #cde5ee 100%);
filter: progid:DXImageTransform.Microsoft.gradient( startColorstr='#edf6f9', endColorstr='#cde5ee',GradientType=0 );
color:#527988;
box-shadow:0px 0px 10pc #c9c9c9
}

#botao:hover{
background: #cde5ee;
background: -moz-linear-gradient(top, #cde5ee 0%, #edf6f9 100%);
background: -webkit-linear-gradient(top, #cde5ee 0%,#edf6f9 100%);
background: linear-gradient(to bottom, #cde5ee 0%,#edf6f9 100%);
filter: progid:DXImageTransform.Microsoft.gradient( startColorstr='#cde5ee', endColorstr='#edf6f9',GradientType=0 );
cursor:pointer;
}

        <input type="submit" name="enviar" value="Enviar">

        <input type="reset" name="limpar" value="Reset">

      <input type="checkbox"/>Lembrar senha

Existem outras formas de corrigir isso, mas essa eu acho a mais prítica, pois ele limpa os estilos do user-agent e vc pode começar "do zero" os estilos do seu elemento.
Answer: O ideal seria estilizar o próprio elemento ao invés de criar um wrapper para o reset, isso só tende a complicar mais como você está aplicando a estilização nos elementos. A princípio, se tratando somente da borda que mencionou na pergunta, basta remover a borda (do elemento filho) e herdar os valores das propriedades background e cursor do elemento pai:
input[type='reset']{
  border: none;
  background: inherit;
  cursor: inherit
}

Você pode ler mais sobre inherit nessa resposta.

#botoes {
  width: 270px;
  margin-left: 20px;
}

#botao {
  float: right;
  padding: 5px;
  15px;
  font: bold 12px sans-serif;
  border-radius: 20px;
  box-shadow: 0px 1px 0px white;
  border: 1px solid #9eb9c3;
  background: #edf6f9;
  background: -webkit-linear-gradient(#edf6f9 0%, #cde5ee 100%);
  background: -moz-linear-gradient(#edf6f9 0%, #cde5ee 100%);
  background: -o-linear-gradient(#edf6f9 0%, #cde5ee 100%);
  background: linear-gradient(#edf6f9 0%, #cde5ee 100%);
  filter: progid:DXImageTransform.Microsoft.gradient( startColorstr='#edf6f9', endColorstr='#cde5ee', GradientType=0);
  color: #527988;
  box-shadow: 0px 0px 10pc #c9c9c9
}

#botao:hover {
  background: #cde5ee;
  background: -moz-linear-gradient(top, #cde5ee 0%, #edf6f9 100%);
  background: -webkit-linear-gradient(top, #cde5ee 0%, #edf6f9 100%);
  background: linear-gradient(to bottom, #cde5ee 0%, #edf6f9 100%);
  filter: progid:DXImageTransform.Microsoft.gradient( startColorstr='#cde5ee', endColorstr='#edf6f9', GradientType=0);
  cursor: pointer;
}

input[type='reset']{
  border: none;
  background: inherit;
  cursor: inherit
}

    <input type="submit" name="enviar" value="Enviar">

    <input type="reset" name="limpar" value="Reset">

    <input type="checkbox" />Lembrar senha

Comment: Só fiz um comentário para vc testa ai jovem, da uma testadinha no que falei, como a cor do gradiente dele é suave fica difícil notar, mas se o gradiente for mais acentuado entre a primeira e a segunda cor vc vai ver que não fica legal. Faz o teste ai rapidão usa esse gradiente `linear-gradient(to bottom, red 0%, blue 100%)` vc vai ver que herdando o gradiente não fica com as cores alinhadas. O padding sugeri só pra ficar mais fácil de vc ver. Mas vc pode só testar com outra cor. De boa foi só um comentário pra vc testar e avaliar.
Comment: Também falei de boa. =)
Comment: Legal que vc levou na boa, é pq aqui tem gente que tem o ego nas alturas e não aceita nem comentário, ou nem para pra pensar 2x antes de sai negativando. Só dei a dica pq quando eu abri no meu monitor mesmo a cor sendo suave eu vi que deu uma quebra no gradiente, ai te sugeri fazer o teste. Se outra pessoa aplicar o exemplo com outras cores pode resultar em um efeito indesejado. Tmj
Comment: Jovem... só uma coisinha pra vc testar ai. Tenta colocar um padding de 50px nessa div, ou então muda a cor do gradiente do red para blue. Vc vai ver que o gradient erdado pelo input fica bugado, pois ele não tem a mesma altura do pai. O ideal seria mesmo remover a div envolta do input e estilizar o input, com o `inherit` o gradiente do filho não "casa" com o do pai
Comment: É esse o cenário que o AP colocou na pergunta dele?


java programm thats initialise the values of an 2 diemnsonal array

Question: I have to write a programm by using a 2d imensional array were you able to type in how many students are in a classroomm . Next step would be that every student has a different number of grades so the out print should look like this example :

Number of Students : 3
1.Student  -> First Grade : 5; Second Grade :3; Third Grade : 2;
  2.Student First Grade : 1;
  3.Student  First Grade : 6; Second Grade: 2;

Question: My Programm stops at the point were every student should get a different number of notes . I don`t get it why zeile[i] can´t take the values of i  so zeile[0] would get the first value zeile[1] get the second and so on . The same problem at spalte[n].
Here is what I have tried so far:
<code>import java.util.*;

public class NotenZeugnis {

    /**
     * @param args
     */
    public static void main(String[] args) {
        // TODO Auto-generated method stub

        int zeile,spalte;
        int Noten [] [];
        int anzahlS ;
        int anzahl[];

        Scanner in = new Scanner(System.in);

        System.out.println("Bitte geben sie die Anzahl der Schüler ein : ");
        anzahlS = in.nextInt();//EINGABE DER SCHÜLER BSP .3 DANN 3 ZEILEN {{},{},{}};

        for (int i = 0; i <= anzahlS - 1; i++) {
            System.out.println("Wieviele Noten hat der " + (i+1) +".Schüler ?");
            zeile[i] = in.nextInt();//EINGABE DER ANZAHL DER NOTEN
            System.out.println(zeile);}
            for (int s = 0; s < Noten[zeile].length; s++) { //I LASSEN WIR HIERBEI SO OFT LAUFEN WIE DIE ANZAHL DER NOTEN BSP 3 DANN GEHEN
                //WIR DIESE SCHLEIFE 4 MAL DRUCH
                System.out.println("Bitte geben sie die Noten für die ");
                spalte[s] = in.nextInt();} //DABEI ERHÄLT SPALTE 0 EINEN WERT SPALTE 1 EINEN WERT SPALTE 2
                Noten = new int [zeile][spalte];
            }
    }

// WAS ICH IMPLEMENTIEREN MÖCHTE
//MAN SOLL EINE ANZAHL VON SCHÜLERN ANGEBEN
//FÜR DIESE ANZAHL SOLL ER MICH JEWEILS FRAGEN WIE VIELE NOTEN DIESE SCHÜLER HABEN------BIS HIER HIN GESCHAFFT
//UND DABEI IMMER DIESE WERTE DEN SCHÜLERN ZUORDNEN DABEI SOLLEN DIESE WERTE DAS ERSTE FELD ÜBERNEHMEN
//SODASS BSP. 3 SCHÜLER 1.SCHÜLER HAT 3 NOTEN 2.SCHÜLER 2 NOTEN 3.SCHÜLER HAT 5 NOTEN
//BEI AUSGABE VON FELD ZEILE SOLLTE DER COMPUTER DAS JETZT SO WISSEN
//zeile[0]=3;
//zeile[1]=2;
//zeile[2]=5;
</code>
Question: My Programm stops at the point were every student should get a different number of notes . I don`t get it why zeile[i] can´t take the values of i  so zeile[0] would get the first value zeile[1] get the second and so on . The same problem at spalte[n]. Thanks for the help:)
Thanks for the help:)
Comment: You may want to reformat your question, so it is readable.
Comment: You didn't give us information about zeile[i] and spalte
Comment: Question : Is it possible to determine the value of the array by typing and not at the beginning . Additionally the students have a different number of marks thats why zeile has to be different
Comment: zeile [i] should respresent the number of grades spalte[n] should represent the grades example number of grades 3 so {{},{},{}}  marks 3,5,1 ->{{3},{5},{1}}
Answer: Are you looking for something like this?
<code>    int studentNumber = in.nextInt();

    int[][] students = new int[studentNumber][];

    for (int i = 0; i < students.length; i++) {
        int gradeCount = in.nextInt();
        students[i] = new int[gradeCount];
        for (int j = 1; j <= gradeCount; j++) {
            int grade = in.nextInt();
            students[i][j] = grade;
        }
    }
</code>


PAM authentication failure with HTTP Basic auth and mod_auth_pam on Apache

Question: I'm trying to set up HTTP Basic auth with PAM on Apache (running on Ubuntu 10.04).
I have a VirtualHost setup with SSL and the options below for HTTP Basic authentication:
AuthBasicAuthoritative off
AuthPAM_Enabled On
AuthType Basic
AuthName "PAM"
require valid-user

However, PAM refuses to log me in, despite having the right credentials.
Here's what I have in auth.log:
unix_chkpwd[25522]: password check failed for user (test)
apache2: pam_unix(apache2:auth): authentication failure; logname= uid=33 euid=33 tty= ruser= rhost=XX.XX.XX.XX user=test

Could this be related to HTTPS sending the encrypted password to Apache, and Apache failing to decrypt it somehow, instead sending it directly to PAM?
Help is greatly appreciated.
Answer: Most likely you are using file-based name services and apache cannot read /etc/shadow.
Comment: yes indeed, but wouldn't the system be designed to work without giving Apache read permissions to `/etc/shadow`, which is highly insecure?
Comment: Yes, it is insecure to allow apache to read `/etc/shadow`. That's why mod_auth_pam isn't used; rather, mod_auth_external is.
Comment: Thanks, your suggestion to use mod_auth_external turned out fine, although the documentation is pretty confusing. Still, working as expected. Thank you


Doing nested SSH but getting Too many authentication failures?

Question: This is my script that I have
ssh server1 << EOF
  ssh server2 << EOB
EOB
EOF

I am able to enter the password for server1 successfully, but then it just freezes for a couple seconds and outputs an error saying "Too many authentication failures". Can anyone help? I want to have nested SSH'es to automate some stuff I'm doing.
Comment: What version of debian is that, I just tried stretch, buster and sid. They all support it.
Comment: The second `ssh` doesn't have access to your tty, so it's trying to read from the same `< EOF ... EOF` heredoc that the command itself is in. Can you use the proxy jump option (`ssh -J server1 server2`, or [via the `ProxyJump` config directive](https://serverfault.com/questions/954281/how-to-ssh-directly-to-target-via-jumphost-using-ssh-config-without-additional-s)) instead of nested heredocs?
Comment: @GordonDavisson I am on debian linux and that -J option doesn't exist
Comment: What version of OpenSSH do you have? If it's at least 7.1, you should be able to use `ProxyJump` (either in your config file or with the `-o` option), and for older versions you can use `ProxyCommand` to get the same effect (though it's more complicated to use). See [OpenSSH: Proxies and Jump Hosts (at WikiBooks)](https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Jump_Hosts#Old_Methods_of_Passing_Through_Jump_Hosts).
Answer: In the comments you mention that you have a old version of openssh that doesn't support -J yet.
(Think about upgrading, that is already a really old openssh)
As a replacement you can use ssh -t server1 ssh server2(Do not forget the -t or your terminal input won't work)


Unable to upload CSV file to database

Question: I have an uploads page where a user can upload CSV files only. However, when i hit the upload button, nothing is uploaded to my database and no error messages are displayed. Only the go back button shows which means that my code must not be executing my code properly. How would i resolve this?
    CREATE TABLE `Jobs` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `date` date NOT NULL,
  `order_ref` varchar(250) NOT NULL,
  `first_name` varchar(250) NOT NULL,
  `last_name` varchar(250) NOT NULL,
  `postcode` varchar(250) NOT NULL,
  `country` varchar(250) NOT NULL,
  `quantity` varchar(250) NOT NULL,
  `scott_packing` varchar(250) NOT NULL,
  `packing_price` varchar(250) NOT NULL,
  `courier_price` varchar(250) NOT NULL,
  `dispatch_type` varchar(250) NOT NULL DEFAULT '',
  `tracking_number` varchar(250) NOT NULL,
  `job_status` varchar(250) NOT NULL,
  `created` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`)
)

HTML:
    <?
session_start();
if(!session_is_registered(myusername))
{
    header("location:../index.php");
}
?>
<?
include("../template/header.php");
?>
<!DOCTYPE html>

    <title></title>
    <meta http-equiv="content-type" content="text/html; charset=UTF-8">

Upload New File:

    <form id="upload" action="upload_file.php" method="post" enctype="multipart/form-data">
        <input type="file" name="uploaded_file">

        <input type="submit" value="Upload file">
    </form>

upload_file.php:
    <?php

if(isset($_FILES['uploaded_file'])) {
    if($_FILES['uploaded_file']['error'] == 0) {

        $database = "spineless";

        $dblink = mysql_connect("localhost", "root", "vario007")
           or die("Could not connect");

        $db = mysql_select_db($database)
        or die("Could not select database");

        if(mysql_errno()) {
            die("MySQL connection failed: ". mysql_error());
        }

        $file = $_FILES  ['uploaded_file']['tmp_name'];
        $handle = fopen($file, "r");
        $row = 1;
        while (($data = fgetcsv($handle, 0, ",","'")) !== FALSE)
        {
            if($row == 1)
            {
                // skip the first row
            }
            else
            {
                //csv format data like this
                //$row[0] = date
                //$row[1] = order_ref
                //$row[2] = postcode
                //$row[3] = country
                //$row[4] = quantity
                //$row[5] = packing_price
                //$row[6] = dispatch_type

                $query = "
                INSERT INTO `Jobs` (
                `date`, `order_ref`, `postcode`, `country`, `quantity`, `packing_price`, `dispatch_type`, `created`
                )
                VALUES (
                '".$data[0]."', '".$data[1]."', '".$data[2]."', '".$data[3]."', '".$data[4]."', '".$data[5]."', '".$data[6]."', NOW()
                )";
                $result = mysql_query($query);

                 // Check if it was successfull
                if(!$result) {
                    echo 'Success! Your file was successfully added!';
                }
                else {
                    echo 'Error! Failed to insert the file';

                 }
            }
            $row++;
        }

    }
    else {
        echo 'An error accured while the file was being uploaded. '
           . 'Error code: '. intval($_FILES['uploaded_file']['error']);
    }

    // Close the mysql connection
    mysql_close();
}
else {
    echo 'Error! A file was not sent!';
}

// Echo a link back to the main page
echo '
Click here to go back
';
?>

This is how the CSV looks like:
http://i754.photobucket.com/albums/xx182/rache_R/Screenshot2014-04-24at151348_zps807cedb0.png
Comment: I'd be much more concerned that the uploaded file doesn't follow the format you are expecting.
Comment: @TheBlueDog please can you elaborate?
Comment: Well, you are not checking the uploaded file format at all. What happens if you have text where you expect a number? What happens if there are 8 columns instead of 7? All of these things (and more!) can prevent the data being inserted into the DB. You are opening yourself up to a whole world of hurt here...
Comment: @TheBlueDog i understand what you are saying however, aside from setting the file format restrictions, my main concern at the moment is that i can't upload anything to my database
Comment: Ok, that's fine that you understand that :) I deal with CSV files daily, it's easy when they are generated by another application (until the vendor changes the format), but trying to insert data that came from a user is another matter.
Comment: Echo your query, and then paste the query in your databse.. See if thats works..
Comment: @user3519721: First things first, please post your DB structure, and the first few lines of your CSV file...
Comment: @TheBlueDog please see the above amendments
Comment: I don't see anything obvious, try commenting out `$result = mysql_query($query);` and adding `echo $query` - does it show anything?
Comment: @TheBlueDog nothing seems to be happening. My code just isn't executing any of the code within the if-statement. The only thing that is displayed is the final echo statement that says "click here to go back"
Comment: @user3519721: That would imply that `if(isset($_FILES['uploaded_file'])) {` is false... What does you post HTML look like?
Comment: @TheBlueDog right ok. I have added my post HTML above
Comment: @user3519721: I dunno fella, that code works absolutely fine for me... It uploads a file then fails on the db part as I don't have one set up - I do at least get `could not connect`.
Answer: So i have managed to solve my own question. After looking through my code i noticed that my code was fine up until the while loop. PHP was not properly recognizing the line endings when reading the file so enabling the auto_detect_line_endings run-time configuration helped solve my issue. NOTE that auto_detect_line_endings should be set before fopen and not after.
    <?php

if(isset($_FILES['uploaded_file'])) {

    if($_FILES['uploaded_file']['error'] == 0) {

        // Connect to database
        $database = "spineless";
        $link = mysql_connect("localhost", "root", "vario007") or die("Could not connect");
        $db = mysql_select_db($database,$link) or die("Could not select database");

        // Upload file
        $file = $_FILES['uploaded_file']['tmp_name'];
        ini_set("auto_detect_line_endings", true); // auto_detect_line_endings added here
        $handle = fopen($file, "r");
        $row = 1;
        while (($data = fgetcsv($handle, 0, ",","'")) !== FALSE)
        {
            if($row == 1)
            {
                // skip the first row
            }
            else
            {
                //csv format data like this
                //$row[0] = date
                //$row[1] = order_ref
                //$row[2] = postcode
                //$row[3] = country
                //$row[4] = quantity
                //$row[5] = packing_price
                //$row[6] = dispatch_type

                $query = "
                INSERT INTO `Jobs` (
                `date`, `order_ref`, `postcode`, `country`, `quantity`, `packing_price`, `dispatch_type`, `created`
                )
                VALUES (
                '".$data[1]."', '".$data[2]."', '".$data[5]."', '".$data[6]."', '".$data[7]."', '".$data[9]."', '".$data[11]."', NOW()
                )";

                $result = mysql_query($query);

                 // Check if it was successfull
                if($result) {
                    echo 'Success! Your file was successfully added!';
                }
                else {
                    echo 'Error! Failed to insert the file';

                 }
            }
            $row++;
        }

    }
    else {
        echo 'An error accured while the file was being uploaded. ';
    }

    // Close the mysql connection
    mysql_close();
}
else {
    echo 'Error! A file was not sent!';
}

// Echo a link back to the main page
echo '
Click here to go back
';
?>

Answer:     <?
session_start();

You cannot start a session after output has been sent to the browser, in this case, a few blankspaces.
Adjust this and better not use any sort of short_open_tags:
<?php
session_start();

Comment: I uploaded my new code above which already no longer uses the short opening tags
Answer: Ok, I set up a DB as per your info and created a CSV file using the data you supplied. The good news is that your code works absolutely fine (for me) as is, but for two small issues.
Your success message is the wrong way around, remove the !:
if($result){
echo 'Success! Your file was successfully added!';
}else{
echo 'Error! Failed to insert the file';
}

You are also trying to insert a date string 03/02/2014 into a DATE datatype. This won't work and you'll need to format this string accordingly. As a temporary measure I just changed the datatype in the DB to a VARCHAR(20).
With those two small things fixed:
Comment: i didn't realise i had the success message the wrong way round. Thanks for noticing that. Also, i have no idea why it doesn't work for me
Comment: @user3519721: The only other thing I just noticed is that in your HTML doc you are using the short PHP tags `<?`, I don't know whether this is causing your file upload problem or not. Just try `<?php` out of interest...
Comment: :( still doesn't work. It worked before when i first wrote the code but then it just stopped working which is very strange
Comment: @user3519721: I'll say! Sorry I can't offer any more help, but I just can't replicate your problem... What happens if you comment out `include("../template/header.php");`? Maybe something in there is somehow causing an issue.
Comment: i'll try fiddling with the code, thank you for your help though
Comment: @user3519721: You're welcome. Post an update when you find the problem!

Unable to upload CSV file to database


How do you process invalid HTML in PHP?

Question: I've seen this question, which is very nice and informative. However, it doesn't deal with a rather common scenario.
Say I need to scrape a multitude of websites (or even pages in the same domain), but the author of that website didn't care enough for his code, and has some seriously malformed code "that kinda works". I need to take information from that website.
How do I do it in this case? Ideally without going í͞ń̡͢͡s̶̢̛á̢̕͘ń̵͢҉e̶̸̢̛.
Is it possible? Do I have to revert to RegExp?
Comment: Is the problem that you would like to use PHP DOM manipulation tools on the parsed HTML, but with the HTML being malformed, you are having difficulty doing this?
Comment: Personally, I have always wondered how the browser makers do it and live to tell the tale.
Comment: Perhaps try `strip_tags` then use regex to find the remaining tags?
Answer: You need a DOM Parser. Php has one. And then there are some alternatives (and more... just google for them). You can even run the "garbled HTML" trhu HTML Purifier if you want.
Comment: @Truth So? "Broken" (malformed, invalid, whatever...) HTML is just as parseable... Maybe the DOM tree isn't as the author intended but you can perfectly fine access all the required nodes/attributes you need.
Comment: You don't say! But it's broken HTML we're talking about. This isn't valid HTML.
Comment: RobIII is right. Have a look at the `loadHTML` method http://www.php.net/manual/en/domdocument.loadhtml.php
Answer: I don't know how your are scraping the site, but working with RegExp will allow you to add many conditions to the scrap code. This may take time, depending on the number of footprints and your RegExp skills.
You may also use Tidy on the site HTML, but this will lead to strange results as well IMO.
Comment: Have a look at [my all-time favourite StackOverflow answer](http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454) on using Regexes to "parse" HTML. And then read [Parsing Html The Cthulhu Way](http://www.codinghorror.com/blog/2009/11/parsing-html-the-cthulhu-way.html) for some nuance. For me, the biggest reason **NOT** to use a regex is because a regex in itself [quickly becomes "unreadable"](http://ex-parrot.com/~pdw/Mail-RFC822-Address.html).
Comment: Well, I never said you could parse HTML with RegExp, this is a nonsense. Obviously, scraping (ie extracting data) can be done very well with RegExp.
Thanks for the reading though !
Comment: To extract data (and do it *correctly*) you would have to parse the document and use the DOM instead of relying of a big mess of a string. Having said that; yes, indeed, you *could* use a regex. That's why I added Jeff Atwoods article ;-)
Comment: I'll get into DOM parsing someday maybe, true it's useful and easy, and you don't have to mess with strings (until you have to parse the text nodes). But it's not like I'm parsing every morning, and I never failed to extract a data with RegExp. More over, I really enjoy training myself with RegExp, this is like a swiss-army-knife for many tasks.
Answer: Does it have to be PHP? Python has a wonderful library called Beautiful Soup ("You didn't write that awful page. You're just trying to get some data out of it"). From my experience I'd recommend it so much that I'd say if you have the option, write a quick Python script to parse your nodes into a clean file that your PHP can pick up.
(Know PHP is in the title & this doesn't directly answer your question. Apologies if you don't have the option of (or dislike) Python, just wanted to present a good alternative.)

How do you process invalid HTML in PHP?


SSRS link with PDF failing

Question: In a SSRS2008R2 report with an action linked to a field value; this code is working (from here):
<code>="FILE:\\E:\SharedServer\"+Fields!PDFFileName.Value
</code>
Then I wrapped it with JavaScript to open in a new window and it is failing.
<code>="javascript:void(window.open('FILE:\\E:\SharedServer\'+Fields!PDFFileName.Value','_blank'))"
</code>
The mouse hovers shows a hyperlink but when I click on it, absolutely nothing happens. I had posted before but this is a new issue. I've tried several minor variations with the quotation marks but have not hit upon the correct syntax.
Answer: You are missing some double quotes and a <code>+</code> sign:
<code>="javascript:void(window.open('FILE:\\E:\SharedServer\" + Fields!PDFFileName.Value + "','_blank'))"
</code>
But I'm not even sure if that will work; some other posts (like Opening a file in local file system in javascript) indicate that javascript can't do this.
Comment: The code is working (when I get Chrome to show the report). And, thanks for the link to the IE issue.

SSRS link with PDF failing


Numerical Example of Independent Component Analysis

Question: Can somebody explain ICA(Independently Component Analysis) with a small practical example over here.
I have seen lot of programs and libraries written and you can just apply that to your data to find ICA components. One such library is famous python FastICA?
There is a whole  Book on ICA, some Tutorial on ICA , some nicely explained PPT on ICA and some Practical Use of ICA to remove ECG artifacts But none of those references gave some practical small example to explain those mathematical concepts behind the ICA stepwise.
It would be I am sure very useful for beginners like me to understand the step wise mathematics of the ICA as just applying library is not enough for deep understanding if ICA.
Note:
I really appreciate the effort if somebody could do that using famous fastICA algorithm and explaining step by the step the mathematics involved in that with evidence of equivalent result using tools like python fastICA library.
Comment: You already mentioned some very good resources (e.g. the Hyvarinen et al. book and tutorial paper). These contain exact descriptions of both the principles and math involved, and also example cases. Perhaps you could say more about what the remaining confusion is, or why these were unsatisfactory?
Comment: It could help to elaborate or specify a bit more on that since it might still be unclear what the difference is with the already mentioned resources.
Comment: what I am asking here is a small workable example that somebody expert in ICA could solve for all of us. I want this resource to be added online so that every beginner could utilize that.
Comment: Are you sure? I believe the cocktail party problem would nicely qualify such an example, no? If not, then a more mathematical approach could be to compare it with PCA. The simplified hand-wavy version is that you assume that the signals are uncorrelated and that you can "clean" all of the signals in by finding the linear combination of the signals that gives least correlation.
Comment: Some small workable example using any ICA algorithm..but I think fastICA is so far the famous algorithm for ICA. So if you could do that using fastICA then thats enough.
Answer: I found working out an ICA problem (a simple one about unmixing audios) for the Stanford cs229 course very helpful in understanding its inner working. The basics of ICA isn't that complicated.
Check these out:

Andrew Ng's note on ICA: http://cs229.stanford.edu/notes/cs229-notes11.pdf
The simple problem on unmixing audio: it's the 4th question.
My implementation in Jupyter notebook format, not many lines of code, and you could hear the audio before and after applying ICA.
Comment: It is a numerically stable sigmoid implementation and calculates positive and negative exponents separately. I've updated its docstring to make it more explicit. And W was initialized as an identity matrix first, see the line with `W = np.eye(N)`
Comment: @NaseerAhmed, not sure what you mean by normalization transformation. The `W_expected` is just the result I got after running ICA, I put it there as a test for the future if the code is to be modified, to ensure I still get the same `W`. I don't the raw separate audio, that downloaded from the course is already a mixed one, but you could listen to the unmixed one to see the difference.
Comment: Please explain sigmoid part of your code it is a bit cryptic, normalized function and  where did you get expected W matrix?
Comment: @zyuxe please also explain two things 1) what type of normalization transformation are you using? and 2) How you got expected W matrix of weights?
Comment: @zyuxe It would have been much intuitive and easy to understand if you could first show us unmixed signals.Then you would mix those signals with some random mixing matrix. After that you applied ICA to compute W matrix and finally you unmix the mixed signals using the W matrix you obtained using ICA?
Comment: By normalization I mean how you used your normalize function. What is the reference behind this normalize function that you used you are multiplying each value by a factor of 0.99 and dividing it by the max absolute value. How did you get that function any reference?
Comment: The `normalize` function is provided by the assignment already, see [this line](https://github.com/zyxue/stanford-cs229/blob/master/Problem-set-4/4-Independent-components-analysis/bellsej.py#L18). Looking at its definition, I think it just brings all values to be between (-1, 1). I am not super familiar with audio data, but I tried listening to the mixed sound both with and without normalization, they sound the same to me, so I suppose only the relative intensity in the audio matters. I don't think the normalization is part of the ICA algorithm.


Python subprocess package does not report errors from subprocess?

Question: On some machine, for the following piece of code
p = subprocess.Popen(cmd, stdout = subprocess.PIPE, stderr = subprocess.STDOUT);
out, err = p.communicate()

This script would just hang at p.communicate() and does not return.
After I manually run the command, I finally see the error messages.
Why is this, and how should I solve it?
Thanks.
Comment: What are the error messages?  Do you have the same problem if you try a different command?  What is `cmd`?
Comment: your process seems not to terminate, and communicate() method just waits for it to happen, as it is documented. and not that the error code is undefined until subprocess terminates (returning this code to the OS and eventually to parent process)
Comment: so please define exactly what is the desired interaction scheme you are trying to achieve. Do you need to feed process with input data and capture output?... For instance, if you need only wait for termination and capture error code, the `subprocess.call()` will suffice.
Comment: @mgilson That is a script written by myself, reporting some basic numpy errors, like shapes don't align.
Comment: @spacediver But why wouldn't it terminate? It throws an unhandled exception already. I don't need to feed it with input, but I need to process its output.
Comment: @CodeNoob -- you haven't answered my question.  What happens if you try a different command (say `cmd='ls'`).  One thing that you may need to do is split command `cmd = shlex.split(cmd)`, but it's really hard to know without knowing what `cmd` actually is.
Comment: yep, if cmd contains your python script, then it does matter how you invoke the python interpreter to run it
Comment: I see that Popen does recieve not the string `cmd`, but the list of argument substrings, so you might need to pass `cmd.split()` instead of just `cmd`
Comment: @mgilson if I run any other command (like ls), it is fine. my cmd variable is really just a properly split python list, like ['./a.py', '-C', 'blah]. where a.py was written by myself.
Comment: @spacediver I invoke it like ['./a.py', '-C', 'blah]. Why does how I invoke the python interpreter matter?
Comment: try to invoke interpreter in most explicit manner possible, like this: `['/usr/bin/python', '/a.py', ...]` Thus you would isolate your invocation from shell/shebang settings.
Answer: try stderr = subprocess.PIPE not stderr = subprocess.STDOUT
Answer: I guess your program never ends?
when you call communicate() it does different things under different os. But it always waits for the launched process to exit on its own eg. calls p.wait().
p.wait() terminates only if the rocess terminates.
Solutions:

You could copy the source code of subprocess.Popen._communicate and alter it so it does not use wait() but time.sleep and some timeout
You write your own code that reads stdout and stderr and stops if the program outputs too much stderr
You change the main() function of the file that never ands this way

way
def main():
    ## here is you program code

if __name__ == '__main__':
    import thread, sys, time
    _id = thread.start_new(main, ())
    time.sleep(1)
    t = time.time() + 100 # execute maximum 100 + 1 seconds
    while t > time.time() and _id in sys._current_frames():
        time.sleep(0.001)

If you want to look at the source of subprocess you can find it at subprocess.__file__.


How to get ip address?

Question: I used the following code:
services:
  foo:
    class: Drupal\MYMODULE\Foo
    arguments: ['@request']

In MYMODULE/src/Foo.php
namespace Drupal\MYMODULE;

use Symfony\Component\HttpFoundation\Request;

class Foo {
  /**
   * @var \Symfony\Component\HttpFoundation\Request
   */
  protected $request;

  /**
   * Constructs a Foo object.
   *
   * @param Symfony\Component\HttpFoundation\Request $request
   *   The request object.
   */
  public function __construct(Request $request) {
    $this->request = $request;
  }

  public function bar() {
    if ($ip == $this->request->getClientIp()) {
      drupal_set_message('ip', t('You may not ban your own IP address.'));
    }
  }

}

but this gives me error:

Symfony\Component\DependencyInjection\Exception\ServiceNotFoundException'
  with message 'The service "foo" has a dependency on a non-existent
  service "request".
Comment: There is no service called `@request`, you need to use the service called `@request_stack`, and call `getCurrentRequest()` on the injected RequestStack object.
Answer: According to Get the user IP address and Symfony HttpFoundation you can get visitor IP by
first add use Symfony\Component\HttpFoundation; then
 \Drupal::request()->getClientIp()

Comment: With DI in the class, you would get the request from the RequestStack, which is available as a service.
Comment: It's better to use service injection instead of relying on the global `\Drupal` object.  See more details in my answer below.
Comment: *Note: The use of this wrapper in particular is especially discouraged* https://api.drupal.org/api/drupal/core!lib!Drupal.php/function/Drupal%3A%3Arequest/8.2.x
Answer: The correct way to do this is via service injection.  This way your class is unit testable.
In your mymodule.services.yml file:
services:
  foo:
    class: Drupal\MYMODULE\Foo
    arguments: ['@request_stack']

In your class:
namespace Drupal\MYMODULE;

use Symfony\Component\HttpFoundation\RequestStack;

class Foo {
  /**
   * The current request
   *
   * @var \Symfony\Component\HttpFoundation\Request
   */
  protected $request;

  /**
   * Constructs a Foo object.
   *
   * @param Symfony\Component\HttpFoundation\RequestStack $request_stack
   *   The request stack object.
   */
  public function __construct(RequestStack $request_stack) {
    $this->request = $request_stack->getCurrentRequest();
  }

  public function bar() {
    $ip = $this->request->getClientIp();

    // Do something with $ip, just return it back as an example.
    return $ip;
  }

}

In your test file:
namespace Drupal\Tests\MYMODULE\Unit;

use Drupal\MYMODULE\Foo;
use Drupal\Tests\UnitTestCase;
use Prophecy\Argument;

class FooTest extends UnitTestCase {

  public function testBar() {
    $request = $this->prophesize('\Symfony\Component\HttpFoundation\Request');
    $request->getClientIp()->willReturn('126.96.36.199');
    $request = $request->reveal();

    $request_stack = $this->prophesize('\Symfony\Component\HttpFoundation\RequestStack');
    $request_stack->getCurrentRequest()->willReturn($request);
    $request_stack = $request_stack->reveal();

    $foo = new Foo($request_stack);

    $this->assertEquals($foo->bar(), '22.214.171.124');
  }
}

Comment: Just for completion, Does the above code create a drupal service, a reference to which can be obtained using `$ip_service = Drupal::service('foo');` and the IP address can be obtained by `$ip_service->bar()` ?
Answer: What does the error you received tell you? It tells you that the service request does not exist. You can look/discover requests loaded by core reading the core.services.yml file, and discover something similar, a service called request_stack.
The request_stack service provides Symfony\Component\HttpFoundation\RequestStack with which you can get the current request. The static \Drupal::request is an example implementation of getting the current request.
  public function __construct(RequestStack $requestStack) {
    $this->request = $requestStack->getCurrentRequest();
  }

  public function bar() {
    if ($ip == $this->request->getClientIp()) {
      drupal_set_message('ip', t('You may not ban your own IP address.'));
    }
  }

Comment: Almost correct, but by assinging the request in the constructor, you're defeating the whole point of why request_stack exists :) The current request can change (sub requests and stuff like that). Always assing the request stack and get the request when you actually need it.
Comment: I have used exactly same code now and page gives "The drupal-8 page isn't working

drupal-8 is currently unable to handle this request.
HTTP ERROR 500"   without any error in log messages. Whenever i call this bar function inside any file.
Answer: Following code worked for me:
use Symfony\Component\HttpFoundation;
$ip_address = Drupal::request()->getClientIp();

Comment: **Note: The use of this wrapper in particular is especially discouraged.** https://api.drupal.org/api/drupal/core!lib!Drupal.php/function/Drupal%3A%3Arequest/8.2.x
Comment: Why do you need "use Symfony\Component\HttpFoundation;"? HttpFoundation is not referenced.
Answer: If you want to show the ip in twig:
Suppose you want to get the ip in the page.html.twig file.
in your
YOURTHEME.theme
you can try
use Symfony\Component\HttpFoundation;

function YOURTHEME_preprocess_page(&$variables) {
  $variables['page']['ip'] = \Drupal::request()->getClientIp();

}

in your template
page.html.twig
you can try:
{{ page.ip }}

Comment: This won't work, as the code in preprocess functions is going to be cached until you flush the cache the next time.
Answer: I make the following module for get ip address of visitors
create following route custom_location/custom_location.info.yml and following info
name: Custom Location
description: Custom Location.
package: Custom
type: module
version: 1.0
core: 8.x

custom_location/custom_location.links.menu.yml
location.admin:
  title: 'Location module settings'
  description: 'location'
  parent: system.admin_config_development
  route_name: location.content
  weight: 100

custom_location/custom_location.routing.yml
location.content:
  path: '/location'
  defaults:
    _controller: '\Drupal\custom_location\Controller\CustomLocation::content'
    _title: 'Ip Table'
  requirements:
    _permission: 'access content'

in the route custom_location/src/Controller/CustomLocation.php
<?php
/**
 * @file
 * Contains \Drupal\custom_location\Controller\CustomLocation.
 */
namespace Drupal\custom_location\Controller;
use Symfony\Component\HttpFoundation;
use Symfony\Component\HttpFoundation\RequestStack;

class CustomLocation {
  public function content() {
    $ip_address = \Drupal::request()->getClientIp();
    $html = $ip_address;
    return array(
      '#type' => 'markup',
      '#markup' => $html,
    );
  }
}

and visit in your path your location in /location and ready appear the ip
Comment: There are already answers showing how to get the IP from a controller, so this answer doesn't add anything to what the other answers already say. `\Drupal` should never be used in a controller class, though. It is also probable the output is going to be wrongly cached, without using cache tags or cache contexts.
Comment: oh yes you try the module, without \Drupal no working
Comment: See [this answer](https://drupal.stackexchange.com/a/243888/199), which doesn't use `\Drupal` but correctly uses dependency injection.

How to get ip address?


Is variance homogeneity check necessary before t-test?

Question: According to statistics textbooks, t-tests require the dependent variable to be normally distributed and the variance to be homogenous across conditions.
What's the authorized standard in statistics? Is it OK or even necessary to check homogeneity before running a t-test?
Comment: @Mark Indeed --  a number of questions on site give several references that indicate that you're generally better off just using a Welch test than testing equality of variance in order to choose between an equal variance test and an unequal variance test.
Comment: @WhiteGirl There's no "authorized standard"... which is largely a good thing, not a bad thing. Better to learn how to figure out how to choose between tests (e.g. by the simulations Mark mentions) and explain the reasoning (which can be situational) than to bow to some arbitrary standard which cannot hope to be suitable for every circumstance. (Considering the frequent bad practice I see caused by slavish adherence to decades-old rules of thumb, I'd really prefer not to encourage even more of it.)
Comment: Note that a hypothesis test of variance answers the wrong question. It looks at whether the variances differ more than would be accounted for by random variation. The question we want to answer is not that one --  we would rarely believe that the population variances are exactly equal, for one thing, so in large samples it will tell us what we already know and in small samples it will fail to tell us even what we already know. What matters is the impact of the heteroskedasticity that we have on our inference about the means.... ctd
Comment: ctd..  *How* different they might be is more important than whether we can tell that they're different (in small samples failure to reject would be no consolation, in large sample a rejection may barely matter -- because we may pick up even trivial differences). The test also can't tell us that if the sample sizes are equal, it hardly matters whether we assume equal variance.
Comment: Someone can point me to a relevant citation if I am mistaken, but I have never heard of any controversy surrounding whether or not to check for homogeneity of variance before running a t-test (and the commenter over at SO does not provide any references). It seems like it would be a good idea to always check that assumptions are met when running statistical tests...? You can get around this altogether by always running a Welch's t-test, which does not assume homogeneity of variance. This is the default option in R, using `t.test`
Comment: If the variance is  homogeneity,and you didn't set `var.equal=TRUE`,still run Welch test will give you a wrong result.That's why I ask [another question](https://stats.stackexchange.com/questions/289331/whats-the-recognized-standard-of-test-selecting)
Comment: Where did you read that? I will do a quick simulation real quick, but I do not think Welch's test will give you a biased answer if variance is actually equal.
Comment: You can find the difference at this [link](http://statistics.berkeley.edu/computing/r-t-tests) from berkeley.It's a big problem if we have no a recognized standard.
Comment: Interesting. I got fine power when doing Welch's when variances were the same: https://paste.ofcode.org/d8FXJVHSQKPkRX7L434Bsw
Comment: And if you change it to be two conditions with means of 100, you will get a flat distribution for ps. I would just do the Welch's test always.
Comment: @Glen_b thanks! Always good to know my simulations aren't fooling me
Comment: @Glen_b,You can see the selected answer,which overthrow my knowledge from textbooks.I guess even some statistics professors don't read that paper.That's why I ask for a recognized standard in [another question](https://stats.stackexchange.com/questions/289331/whats-the-recognized-standard-of-test-selecting)
Answer: No, it is not necessary. Given that there is a test that accounts for heterogeneous variances (Welch's t-test), you can simply conduct it. For one, the tests for homogeneity of variance (HOV) are problematic in a number of ways. Some lack power, they - like other statistical tests - are too powerful with large sample sizes, effect sizes are missing for these tests, some are faulty under non-normality, ...
The typical approach for most applied researchers is to conduct Levene's test, then decide whether to conduct Student's t-test or Welch's t-test based on the result of Levene's test. However, Zimmerman (2004) showed through simulation that conditioning the test on the result of Levene's test distorts the p-value of the test i.e. your p-value from Student's or Welch's is not reliable when you choose which one to do based on Levene's test. Furthermore, given that Welch's test is almost as powerful as Student's test under HOV, and it is much more powerful when HOV is absent, it is advisable to "just do Welch's test".
Zimmerman, D. W. (2004). A note on preliminary tests of equality of variances. British Journal of Mathematical and Statistical Psychology, 57(1), 173–181. https://doi.org/10.1348/000711004849222
Here is another paper that gives the same basic advice:
Delacre, M., Lakens, D., & Leys, C. (2017). Why Psychologists Should by Default Use Welch's t-test Instead of Student's t-test. International Review of Social Psychology, 30(1), 92–101. https://doi.org/10.5334/irsp.82
Comment: Note that R uses Welch's t-test by default.
Comment: Yeah, but most applied researchers do not use R. Many use SPSS and are taught to decide based on Levene's test.
Comment: Oh, thanks! That's something else I've now learned, check user tags. I used to teach the same thing in SPSS, until I read the second article up there. Then I read the other article which one can find on the Wikipedia Welch t page. Then I ran some simulations, and I now teach "just do Welch". With small sample sizes though, it gets nuanced, but doesn't everything?
Comment: Agreed.  When I learned on SAS, I believe we looked at a type of F-test on the output to decide which t-test output to use.... But the original poster tagged R.
Answer:
According to statistics textbooks, t-tests require the dependent
  variable to be normally distributed and the variance to be homogenous
  across conditions

This is misleading. Generally, introductory statistics textbooks teach 2 (maybe 3 if you count paired stuff) two sample t-tests. Both tests assume that each of the two random samples are iid normal random samples, and are independent between each other. However they are different in that

one assumes further that the two groups have equal variance,
one makes no additional assumptions, but the sampling distribution of your test statistic is only approximately t-distributed.

The assumption that both groups have the same variance is unverifiable. This is because this is an assumption about unobservable variance parameters. However, 1) there do exist tests that can test equality of variances between the two groups, and 2) you can sometimes reassure yourself looking at, perhaps, histograms of the two sets of data, checking to make sure they have the same variance, roughly.
Regarding the first technique: like any hypothesis tests, there are the associated type 1 and type 2 error events. If you decide to formally test equality of variances before you test the means, since you are running two tests, you need to realize that there is some type 1 and type 2 error for your overall strategy.
Comment: Then, do we need  variance homogeneity check before t.test?
Comment: I  mentioned two options; I would do atleast one of them. Comparing these two techniques would require more information about your specific problem.
Comment: The assumption that both groups have the same variance is unverifiable.??`var.test` and `bartlett.test` in R can do that!
Comment: @WhiteGirl we are not sharing the definition of unverifiable. Please read the rest of my answer.
Answer: Not only is it not necessary, see user162986's answer, it can also imperil the interpretability of your test.


How to test rake tasks, cron jobs, and

Question: I know in Rails application, I can write tests for controllers and models by using Rspec.
But:

How about to test some rake task? What is the good way to test some rake task?
How about to test a cron job which run certain rake task every day at a fixed time?

Can Rspec also be used for above two scenarios in Rails app development or are there some other ways to implement those tests?
In addition:
I have a rake task which is used to update the database of the Rails app by fetching data from another database and insert to the app database (clean the app database first of course)
I would like to test these, how to do it?
Answer: Instead of having code in your rake tasks, do something like this:
<code>desc "Charge Customers Daily"
task :charge_customers => :environment do
  CustomerCharges.create
end
</code>
That way, you can write rspec tests in the customer_charges_spec.rb file as you normally would.
Answer: Maybe this helps :
http://www.philsergi.com/2009/02/testing-rake-tasks-with-rspec.html
To test a cron job, you can just redirect to a log file on cron itself, like :

command > /tmp/log.txt 2>&1

Generally, if you intend on doing db insertions and stuff in a rake task, i would reconsider and write that as a separate ruby module. I think it's much more flexible this way.

How to test rake tasks, cron jobs, and


Congruence class $[a]$ modulo $m$, $\gcd(x, m) = \gcd(a, m)$

Question: I'm currently stumped on this question:
Let $a$ and $m$ be integers such that $m\ge1$. Consider the congruence class of $a$, i.e., $[a]$ modulo $m$. Prove that: For all $x\in[a]$, $\gcd(x,m)=\gcd(a,m)$.

From this, I know that the congruence class of $[a]$ modulo $m$ is the set of integers

$[a]={x\in Z|x\equiv a\pmod m}$}. And by the Linear Congruence Theorem, $x\equiv a\pmod m$ if and only if $\gcd(x,m)|a$.
I'm not sure where I can go from knowing this. Any ideas?
Answer: The congruence class of $a$ consists of all numbers of the form $a+qm$, where $q$ ranges over the integers. So we want to show that for all $q$, we have $\gcd(a,m)=\gcd(a+qm,m)$.
We can attack this directly. (i) Show that if $k$ divides $a$ and $m$, then $k$ divides $a+qm$ and $m$; (ii) Show that if $k$ divides $a+qm$ and $m$, then $k$ divides $a$ and $m$.
Answer: Another way to write your equivalence class: $x\equiv a \mod m \iff a=km+x$ for an integer $k$, which you can easily prove since $a\equiv b\mod m\iff m|(a-b)$.
Now I suggest thinking about the Euclidean algorithm, and you might find yourself being done fairly soon.
Edit: I've just looked at your criterion $x\equiv a\mod m\iff GCD(x,m)|a$ again, and it seems wrong to me. Take $x=1,m=3$, for instance. $GCD(3,1)=1|a$ for any integer $a$, but obviously $1\equiv a\mod 3 \quad\forall a\in\mathbb{Z}$ is not true.
Comment: Ah thank you. Makes perfect sense now. Much appreciated!


Using ALL in SQL query

Question: I have a table that has an identifier and an attribute, something like:

Number
Status

5555001
Open

5555001
Closed

5555002
Open

5555003
Open

5555003
Open

5555004
Closed

I want to write a SQL query that would pull back ID numbers that ONLY have every status as Open.  So the output should bring back 5555002 and 5555003.  It would not count 5555001 because one of the entries has a status of Closed.
The query I wrote was:
  FROM table t
  where t.status NOT IN (select t1.number
               from table t1
               where t1.status != 'Open')

While this brings back a count, I don't know that it's the right count.  Is there a better way to do this query using "ALL"?
Answer: You can use aggregation:
select number
from t
group by number
having min(status) = max(status) and
       min(status) = 'Open';

If you have a separate table with one row per number, you could also use:
select n.*
from numbers n
where not exists (select 1
                  from t
                  where t.number = n.number and
                        t.status <> 'Open'
                 );

However, I can't think a natural way to write this using ALL.  I mean, you could write:
select n.*
from n
where 'Open' = all (select t.status from t where t.number = n.number);

That is actually an interesting approach to the problem.  It is just unusual to check that all values in a set (returned by the subquery) are all the same.

Using ALL in SQL query


Hyper-V and 12 VMs

Question:
Possible Duplicate:
Can you help me with my software licensing question?

From what i understand from microsoft's website is that with Enterprise Edition Server 2008, you are allowed to run 4 guest VMs. what if i want to run 12 guest VMs ? is data center edition my only options or i can just get a different licence (cheaper ?) ?
thanks
Answer: I'm not sure what you're asking. The 4 VMs from Enterprise means that the single license that you purchase allows for 1 physical installation plus installation of that license in 4VMs. 4 is not the limit of VMs that you can run in Hyper-V, it is just the limit that you can run without having to buy additional licenses.
Comment: about the license, is it the CAL or do i need to buy another 8 windows server 2008 OS ?
Comment: Client Access Licenses are what Microsoft uses to say that client PCs are allowed to access resources on a server. If you purchase them on a per-seat basis, you don't need any extras assuming that you're legal with your licensing now.
Answer: You can run as many VMs as your resources (memory, CPU and disk space) on the host allows. The 4 VM count with Enterprise Edition means you get 4 windows licenses for free with the purchase of Enterprise Edition.  You can buy additional licenses if you go over 4
Answer: If you're planning on running 12 vm's on one physical host , with no previous licensing, Datacenter Edition will probably be a better deal money-wise in the long run.  Especially if those are going to be Enterprise versions that you want to run (Datacenter allows you to install any lower version of Server).  It also allows you expansion in the future if you decide to upgrade hardware capacity.  Hyper-V itself can run as many VM's as you have license for and that your hardware can support.  You can also use existing Win client/server licenses if you're migrating those physical machines into the virtual space.
Comment: any cheaper solutions other than datacenter edition ? because the setup is only for 12 vms
Comment: There is a virtualization pricing calculator available on the MS website.
Comment: Well, you have 3 options.  Use existing licenses if you have them.  Which obviously would make it cheaper because you're leveraging existing resources, or you could purchase individual Windows Server licenses for the 12 hosts (expensive), or you could purchase 1 copy of Windows Server Datacenter edition (still expensive, but cheaper than buying 12 individual licenses).
Answer: Unless you have some good raid / multiple hard drives, running 12 may be a IO problem!
That being said, there are no limitations to the number of VM's that you can run in Hyper-v
I believe that each licence of Enterprise gives you the ability to run 4 Enterprise guests. You may either buy 3 licences for what you want, or as you said, buy the data centre edition.
Comment: You can't just buy 3 enterprise licenses to run 12 VM's on a single machine - that's outside of the licensing terms.  The 4 guest licenses with each Enterprise license are only if you run them under THAT SPECIFIC LICENSE on a server.
Comment: That is not true. I have been to events at Microsoft and they stated that EACH copy of Windows 2008 gives you x additional licences for virtualisation. This includes if you use ESX or any other hypervisor - or multiple licences for one server
Answer: I know this has been answered, but want to record this...
Here's the URL you're looking for:  Microsoft Hyper-V Server 2008 . Note the "Guest Virtualization Rights Included in Host Server License" row, where it states:

Each additional Windows guest VM requires a license.

So Enterprise means you get a license for the host, plus 4 additional VMs.  If you need a fifth, you buy a liceneses for Windows Server.
I'm not sure if you can "stack" enterprise licensing - buying two enterprise licenses doesn't get you 8 VM licenses, I believe it gets you 1 host, 4 VMs plus one more Enterprise.

Hyper-V and 12 VMs


Link a Page to another Page using Views and taxonomy

Question: I have a Page created using a View using a Content Type called Sectors, which displays rows of images with a title underneath the image. The Sector content type has the fields - Title, Image, Sector Category (uses a Taxonomy). A Sector can have many Projects.... so.....
When the user clicks on the Sector Image or Title, i want my site to go to another Page which is created using a View and display all the Projects belonging to that Sector using the Sector Category field (which is a field on both Sector and Projects content type, passed to it from the previous Sectors page), to select all the Projects that are held in the Projects Content Typle - which has fields Title, Image, Sector Category.
How can I link the 2 views?
Answer: You first need to create your other page, using Views, with a path pattern similar to the following:
<code>projects_by_sector/%tid
</code>
This page can then be configured to display only content of type Projects and the fields you need (Title, Image, Sector Category, etc). Most importantly, you need to configure a contextual filter with the following configuration:

For the "Content: Sector Category" field
"Provide default value"
"Raw value from URL"
"Path component" => 2 (this follows from the example URL above)

Now back in your existing Views page for the Sectors list, you need to do some field rewriting. This can be done in a number of ways, the easiest would be to add the Sectors Taxonomy reference again as a field and do the rewriting on that field. You can do these using the following steps:

Add the "Content: Sector Category" field
Save with the default configuration
Rearrange your fields and move the new field below/after the "Content: Title" field. This is important so that the replacements token [title] will be available for the rewriting.
Hide the "Content: Title" field by selecting "Exclude from Display" on its configuration
Configure the new "Content: Sector Category" field
Select "Plain text" as "Formatter"
Expand "Rewrite Results"
Select "Rewrite the output of this field"
Enter the following in the "Text" textarea to use the title of your sector
[title]
Select "Output this field as a link"
Enter the following as the "Link path"
projects_by_sector/[field_sector_category-tid]

This is a rough idea, you need to tune your views page accordingly, but the end-result should be what you asked for. Hope this helps!
Comment: In my Sector List View, I have 3 fields selected  - Sector Image, Title, and now I have Sector Category. I have set Sector Category so that it Excludes from Display ticked, Output this Field as a Link ticked, Link Path set as projects-by-sector/[field_sector_category-tid] . I have also changed the settings for the Sector Image and Title so they do nothing (i.e. don't link to Content). On my site, now when I hover over a Sector  image to click and select one, it does nothing.
Comment: I read your question again, it seems that you want to use the Sector's Title or the Sector's Image as the link to the Sector's projects list. I suggested an additional field for that. Is that an option for you? Otherwise, I will update my answer accordingly and specifically for using the Sector's Image as a link.
Comment: Yes, I want to use the Sectors Title or Image to link to the Projects View using the Sector Category . thanks
Comment: I updated my answer.
Comment: when Im configuring the Projects by Sector View, I have added a Contextual Filter for Sector Project Category as specified above, Default Value ticked , Raw value from URL ticked, Path Component = 2. This results in nothing being displayed in the Preview at the bottom of the view. If I select  "Display all results for the specified field " then all the projects appear.  Is it because Sector Project Category is a taxonomy term and needs different configuration?
Comment: I found a solution to my contextual filter with taxonomy problem here http://drupal.stackexchange.com/questions/9332/drupal-7-views-contextual-filters-taxonomy-name    I have the Projects Views working now, and will now look at the Sectors View
Comment: The preview should work as long as you type in a valid taxonomy term id (for Sector Category terms) in the "Preview with contextual filters" text box.
Comment: Fantastic! Good luck with your project!


Views Contextual filter: php code for getting the value from url

Question: Im trying to set up a Contextual filter in drupal views based on the URL.
The contextual filter is operating with a text field that sets the node alias.
I have it working with "RAW value from URL" based on the position (parentpage/childpage/grandchild-page), but the position could change (only parents and child for example), so do i need to use the "PHP code", or is there a better way?
What i want to happen:
"If last section of page url, (grandchild-page), matches field set in contextual filter, then true (show fields).
Comment: Maybe this will help: [Get last parameter of url in php](http://stackoverflow.com/questions/11257994/get-last-parameter-of-url-in-php)
Comment: I think you can just give it the arg number. Eg: mysite.com/foo/bar/baz has 3 args: 0=foo, 1=bar, 2=baz.
Answer: Though the solution you have implemented worked but my suggestion is to use the request_uri() instead of the $_SERVER['REQUEST_URI'] because of the following reasons:

$_SERVER['REQUEST_URI'] is only available on Apache, whereas request_uri() generate an equivalent using other environment variables.
Apache does nothing by default to sanitize the $_SERVER variables so directly using it without passing through the check_plain should be considered unsafe.

So its better to get the URL like this:

$current_url = check_plain(request_uri());

Example: if the url is http://www.example.com/param1/param2/param3 than the $current_url will contain /param1/param2/param3.
Rest you can build the logic to get the last params, there are numerous ways of doing that.
Comment: Thank you! I thought there might be a better drupal way of getting the url.
Comment: @petergus Always welcome. If it worked for you than pls. accept the answer so that it may others looking for the same.
Answer: You do not have to use PHP code - there is a module for that:
https://www.drupal.org/project/views_contextual_filter_query
OR you could use Context: https://www.drupal.org/project/context_query_param
Answer: Thank you J. Reynolds for pointing me in the right direction!
I have used the following in "When the filter value is NOT available > provide a default argument > PHP Code"
$url = $_SERVER["REQUEST_URI"];
$url_path = parse_url($url, PHP_URL_PATH);
$basename = pathinfo($url_path, PATHINFO_BASENAME);
return $basename;

($basename is matching the last segment of the URL which is set via pathauto from a custom pseudo title field on my content type, and this field is also used the operational contextual filter in Views.)

EDIT: I also have a Relationship "Content referencing Content from field_related_products" that is selected in the contextual filter.
Please refer to the answer from Ankit Agrawal for a better way to retrieve the URL


Regression on small values

Question: I am working on a problem to predict a stock's returns with a certain set of features.
The problem which I am facing is that when I try to predict the stock price itself, the models capture the trend very well and the R-Squared scores are closer to 1.
But, when I try to predict the daily returns of the stock (a pct_change on the Stock Price), my values are not able to capture the trend, and the R-Squared scores go negative (sometimes even beyond -1).
I know that a workaround for this would be simply to predict the stock prices and then do a pct_change() on the predicted price (which captures the trends in daily returns accurately). However, I want to know as to what I could do to improve my current models with the dependent variable being the returns themselves as opposed to the Stock Price.
My hunch is that this could be because the daily returns themselves have a lower values (the average returns for my stock turn out to be about ~0.0002) whereas the stock prices themselves have somewhat high values (average stock price over 90 days ~ $30).
But, I believe that the models should still work well enough. My independent variables also have a range between -1 and 1, and they have a positive correlation with the stock's returns and the stock's price.
What could I do to overcome this issue?
Thanks!
Comment: Sorry to say, but this will not ever work the way you want it. If you could predict the stock with such a simple model, we would all be rich by now :-)
Comment: @Repmat I understand this. I am working on this as an educational exercise because I have encountered this problem before where my models predict the stock price well enough, however, if I try to compute the returns of the price, the models fail to capture the trend.
Answer: Lot's more data and trading capital.
If I understand you right, you want to predict prices at, say P(t+1) based on today's price at P(t).  The problem is that any time series like stock prices there is a natural correlation from today today's price and tomorrow's: much like weather, the best estimate for tomorrow's price is today's price likely with a but of drift.
The problem with this is that it gives a false sense of security.  In cross-sectional data a high R-squared gives you a sense of what you have accomplished.  Usually one looks at both the "adjusted R-squared" (which is, loosely, the R-squared once that serial correlation is removed or accounted for.  Hence, we try to predict the amount by which tomorrow's price can be estimated today, and that ideally captures the "real" R-squared - not one boosted by serial correlation.  One would also look at the t-statistic on the "growth per day" to make sure it is statistically sound.
One thing that should not matter is the size of the typical variables.  you could laser-measure movements in the Hoover dam measured in thousandths in a millimeter based on the weight of the water in the lake as measured in quadrillion pounds.  All of the scaling should be irrelevant, it just gives you more accurate or intuitive results.
Comment: Ah, yes. I agree on the part about the false sense of security. I am using some lagged variables as well in my predictive model for the returns (not the stock price) and based on the coefficients, I do get how that could be misleading. That being said, I want to know as to why the same model, with the same indep variables, and the same degree of correlation on both stock price and stock returns offers such a different trend for each of them. Is there a way by which I could try to at least predict the trends well enough (not the magnitude of the returns, just the trend)
Comment: I'm not sure I follow, but think of it this way.  Suppose prices keep going up in general The easiest case to think about is a .01% constant daily increase.  Now, this very simple return process is easily predictable - it's +.01% daily.  But the price itself - which is pretty much a straight lining going up - has a lot of volatility, since the price keeps changing from day-to-day.  The best estimate for tomorrow's price is today's price.  That's why returns are looked at.
Comment: Ah, got it. Apologies for not getting back earlier - I have been thinking about what you have said and this makes sense to me now. Thanks a ton!

Regression on small values


How to handle password related tasks in FireFox WebExtensions?

Question: There have been many really great extensions for FireFox which has been nullified by the WebExtensions change, like "Master Password Timeout", "Password Maker", or "Unhide Passwords".
I am not an ext developer but some of these are really almost one-liners (like Master PW Timeout), so I thought it'd be trivial to port ot over. However I wasn't able to find any info about the required API calls, methods, or really anything related to anything passwordy.
Is it possible to convert these somehow that they actually work under latest FireFox (like nightly and future releases)?
Answer: It seems that Firefox developers are not quite clear about what they want; some insists to drop the internal password support altogether and use external wallet providers, others are pretty busy creating the API required for modern secure data storage.
The API is not there yet, but you can follow its progress in various Bugzilla tickets:

Master password API for WebExtensions
Land logins API
Security review of logins API
Implement login storage API

I should update this answer as things progress.


Fetch the external json file through Angular JS

Question: I have try to fetch the json data by using Angular JS. I have successfully fetch through static json data. But now I have try to fetch through external JSON file. Below is my code & there output :
index.html
<!doctype html>

 {{item.name }}

 {{item.city}}

 {{item.state}}

data.json - js/data.json
[
        {
            "name"       : "myname1",
            "city": "mycity1",
            "state" :  "mystate2"
        },
        {
            "name"       : "myname2",
            "city": "mycity2",
            "state" :  "mystate2"
        },
        {
            "name"       : "myname3",
            "city": "mycity3",
            "state" :  "mystate3"
        }
]

controllers.js : js/controllers.js
 var nameSpace = angular.module("MyInfo", []);

nameSpace.controller("MyInfoFunction", function MyInfoFunction($scope, testService)  {

    function Init() {
      $scope.data = {};
      testService.getData().then(function(data) {
          console.log(data)
      });
   }
   Init();
});

nameSpace.service('testService', function ($http) {
    this.getData = function () {
        return $http.get('js/data.json');
    }
});

Output :

{{item.name }}
{{item.city}}
{{item.state}}

I have try to fetch but unable to get the data. What am I doing wrong please help.
Answer: your controller is messy and incorrect, extra } & )
nameSpace.controller("MyInfoFunction", function MyInfoFunction($scope, testService)  {

    function Init() {
      $scope.data = {};
      testService.getData().then(function(data) {
          $scope.myinfoVariable = data.data;
         // console.log(data)
      });
   }
   Init();
});

your .json should be like
[
 {"name":"a","city":"b","state":"c"},
 {"name":"x","city":"y","state":"z"}
]

if your data.json file is in inside of the js folder then the getting json should be like
 this.getData = function () {
    return $http.get('js/data.json');
}

Comment: that means your angular part is working check whether data is  coming ?
Comment: no data is not coming @KalhanoToressPamuditha. Same blank screen showing only.
Comment: Please help to find out I am unable to fetch my json data.
Comment: page loads only, unable to see that demo.
Comment: oh there is a problem with plunker. it will work after sometime. i update the answer please check.
Comment: I have already found this & update in my code. Now blank screen is showing.
Comment: `$scope.myinfoVariable = data.data;` line added please check the answer again
Comment: Again white screen. No output.
Comment: One thing I want to inform you that. I have a folder named AJExample which is situated in my system desktop not in host. Inside them my index.html  & my js folder. Inside Js folder my json file & controllers.js file are there.
Comment: I got the output. Thanks. Actually my issue is that I have put the folder in desktop & I just put the folder to my htdocs & Run, I got output.
Comment: omg yeah. you need to put in the server, if not the $HTTP get is not working , $HTTP will handle by the sever :)
Comment: I got output in the form of list. How can I see in the format of table where Name, City & State are the column name. Please help me to solve this.
Answer: You should try to get js/data.json instead of data.json.
nameSpace.service('testService', function ($http) {
  this.getData = function () {
      return $http.get('js/data.json');
  }
});

You need to provide the url relative to your index.html.
If you are hosting your application on http://myapp.com, and the json file you need to get is  hosted on http://myapp.com/js/data.json then you can either do $http.get("js/data.json") either $http.get("http://myapp.com/js/data.json").
However, if your json file is hosted on another website, let's say http://mysecondapp.com, then you need to do $http.get("http://mysecondapp.com/data.json")
Comment: thanks @GeoffreyB. What do mean by "need to provide the url relative to index.html."
Comment: You need to provide the url from where it can be accessed if you were to open it on your browser.
Comment: you mean that I have to put whole path or copy the whole path from browser.
Comment: Thanks but my data is not in host. Data is in my local machine (i.e. Desktop)
Comment: I have a folder named AJExample. Inside them my index.html is situated & my js folder. Inside Js folder my json file & controllers.js file are there.
Comment: I got the output. Thanks. Actually my issue is that I have put the folder in desktop & I just put the folder to my htdocs & Run, I got output.


Construction of a line segment with ruler only

Question: We have piece of paper with a point P and two line segments on it. The lines fit on the line segments intersect at a point off of the paper (call it Q). How do I construct, with a (long enough) ruler only, the segment of the PQ line that is on the piece of sheet?
Comment: What kind of constructions can you carry out with your ruler?
Comment: @Aretino you can make sections longer (e.g. reach the edges of the paper), the usual kind of things but you can't measure length
Answer: Pappus' theorem will do the trick. Referring to this image from the Wikipedia page:

let $X$ the given point $P$ on the paper, $Z$ be the point of intersection $Q$ that's off the paper, and $BZ$, $bZ$ be the given lines. Choose a point $A$ on the paper so that the line $AX$ intersects $bZ$, then choose another point $a$ such that $aX$ intersects $BZ$. It's often convenient to choose these points so that they're on opposite sides of the two given lines. Now, construct in turn the lines $AB$, $ab$, $Ac$ and $aC$. Let $Y$ be the point of intersection of $Ac$ and $aC$. The points $X$ (=$P$), $Y$ and $Z$ (=$Q$) are colinear.  You might have to play around with the locations of $A$ and $a$ a bit to get everything to fit on the paper.


What are the numbers preceeding a DNS packets flags for?

Question: What does the Flags: 0x0500 section of this DNS query packet mean?
Domain Name System (query)
    Transaction ID: 0x4242
    Flags: 0x0500 Standard query
        0... .... .... .... = Response: Message is a query
        .000 0... .... .... = Opcode: Standard query (0)
        .... ..0. .... .... = Truncated: Message is not truncated
        .... ...1 .... .... = Recursion desired: Do query recursively
        .... .... .0.. .... = Z: reserved (0)
        .... .... ...0 .... = Non-authenticated data: Unacceptable
    Questions: 1
    Answer RRs: 0
    Authority RRs: 0
    Additional RRs: 0
    Queries
        secure.net: type A, class IN

This packet seems abnormal as this section would normally be Flags: 0x0100.  What is the relevance of this section?
Answer: Those flags are described in RFC 1035 section 4.1.1. THe bit you have set, to get 0x0500 is this

AA            Authoritative Answer - this bit is valid in responses,
and specifies that the responding name server is an
authority for the domain name in question section.
Answer: For any reason, Wireshark is not showing the AA (Authoritative Answer) flag, which is enabled in your example.
On my Wireshark that flag is showed:
Flags: 0x8180 Standard query response, No error
    1... .... .... .... = Response: Message is a response
    .000 0... .... .... = Opcode: Standard query (0)
    .... .0.. .... .... = Authoritative: Server is not an authority for domain
    .... ..0. .... .... = Truncated: Message is not truncated
    .... ...1 .... .... = Recursion desired: Do query recursively
    .... .... 1... .... = Recursion available: Server can do recursive queries
    .... .... .0.. .... = Z: reserved (0)
    .... .... ..0. .... = Answer authenticated: Answer/authority portion was not authenticated by the server
    .... .... ...0 .... = Non-authenticated data: Unacceptable
    .... .... .... 0000 = Reply code: No error (0)