How to write diff() mathematically?

Question: I am using R/Python diff() operation. e.g., https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.diff.html
I would like to know if there is a mathematical function that denotes what diff does?
In other words, assume I have a list of N numbers. How can I denote this using a methematical function?
mylist = [x1, x2, x2, ..., xN]
I am happy to provide more details if needed.
Answer: Assume you want to apply diff to a vector $(x_1, \dots, x_n)$ of length $n$. The result will be the vector $(d_1, \dots, d_{n-1})$ of length $n-1$ with entries
$$ d_i = x_{i+1}-x_i. $$

Some people will use the notation $x_{[i]}$ or similar to indicate the vector $x$ with the $i$th component left out, i.e., $x_{[i]}=(x_1, \dots, x_{i-1}, x_{i+1}, \dots, x_n)$. With this, you can write more concisely in vector form
$$ d = x_{[1]} - x_{[n]}.$$
Comment: That sounds useful enough. You can always write down the formulas for the mean and SD in terms of the differences $d_i$. And if you really want, you can substitute in the $x_i$, but it will get progressively less enlightening.
Comment: Thanks a lot for the great answer. Just wondering how to write this function `sd(diff(x))/abs(mean(diff(x)))` mathematically. Currently, I am having it as `σ(diff(x))/|µ(diff(x))|`. Please let me know your thoughts on this. I look forward to hearing from you. Thank you very much :)
Comment: Interesting notation for leaving out $i$; haven't seen that one before. I have often seen $x_{-i}$ for that
Comment: @StephanKolassa I did not get what you  said. It would be really great if you could show me what you said symbolically. I look forward to hearing from you. Thank you once again :)

How to write diff() mathematically?


Which evaluation statistical criteria are proper for DBSCAN algorithm output?

Question: I want to ask advice about the DBSCAN clustering algorithm. I am using it on latitude & longitude matrix data from a seismic catalogue. My question is which evaluation criteria are appropriate to find the correct number of clusters produced by DBSCAN?
I am working on Matlab, and I am using the GAP ('elbow') evaluation criterion with k-means, but I read that it may not be appropriate, since k-means does not work well with density based clustering.
Also, the Matlab implementation of DBSCAN has two outputs, the type & class. Could someone tell me what is the class output? I think it is assigning data points to respective clusters but I am not sure. Any help would be appreciated, thank you,
Dennis
Comment: Please add a [mcve] so we can take a look and try to help you.
Comment: This looks like a question for academic work, for such general questions it's better to ask your supervisor/coworkers who've used this algorithm before for help and information.
Comment: To be honest I do not know if I can do that. My code seems to be working but for the time being is a mess and rather long. However, as soon as I finish the background studying I will ask more focused questions. Thank you.
Answer: Most validation methods do not work with noise (i.e. DBSCAN).
You should try

Moulavi, D., Jaskowiak, P. A., Campello, R. J. G. B., Zimek, A., & Sander, J. (2014). Density-based clustering validation. In Proceedings of the 14th SIAM International Conference on Data Mining (SDM), Philadelphia, PA.

which is the only approach that I am aware of that is designed for density-based clusters. I have not yet tried it though, I prefer manual evaluation.
Instead of DBSCAN, also try OPTICS, and HDBSCAN*.
Comment: Thank you sir. I am aware of that paper and Dr. Moulavi has sent me the source code. I was just trying to figure out if I could do seismic clusters evaluation using the GAP built in evaluation criterion of Matlab, along with K-Means algorithm or the Gaussian Mixture Distribution. I am using a modified DBSCAN version, that instead of Minimum Points sums up Seismic Events Magitudes up to a certain threshold. Then the cluster is being expanded in the same manner, as the classic DBSCAN.


Passing arguments to a node with an alias url

Question: There is a similar issue in here:
Passing an argument to a node causes 404
But there is no real viable solution.
I tried using the sub-pathauto module but the restiction of 8 arguments is too little, since the urls will look something like this:
www.site.com/alias/key/value-1/key/value-2/key/value-3/key/value-4
and it can go on for a while.
It works when instead of alias i have node/nid.
Is it possible to have it work with any amount of dynamic arguments?
Answer: Iv'e not tested with more than 8 args, but I enabled path_alias_xt when passing args to a node Alias with /mynode/term-name/tid and the 404s magically went away.
https://www.drupal.org/project/path_alias_xt
Further more using
print_r(arg(2));
in the node template prints "term-name"


Extensive revision of a question, or ask a new question?

Question: This question was put on hold (for good reasons I think) but it prompted me to think a bit about how the underlying question could be addressed from a theoretical perspective. I mentioned a bit of this in my comment to that question. The OP added enough in the comments that I think the question could be re-worded and reworked to make for an interesting "thought experiment" question.
It would entail some extensive re-writing, though. Would it be better to simply post a new question, perhaps with a link back to the "motivation" (even if it is closed)?
As an aside, I think that "very open" questions are often potential fodder for theory or empirical applications. They may need to be fleshed out a bit and one may need to ask "under what sort of model is the question likely to occur." It may be the case that the necessary model is simply intractable, or has no good theory developed, or no data exists which could be used to estimate or calibrate the models -- thus the question is (at least currently) unanswerable in a rigorous way.
Aside 2: this process -- turning a "too open" question into a potentially answer-able research question -- is one aspect of the "grad cafe" that I'd love to see develop here on Econ S.E.
Answer: I think the appropriate response here is to edit the existing question and then flag it for a moderator's attention for potential re-opening.
The objective should be to have a single great example of each possible question, rather than crowding the site with multiple iterations on the same question.


Condicional bugando em C

Question: Quando rodo o programa e digita a opção ele não escaneia o valor dado pelo user. Não sei o que pode estar bugando a variável escolha dessa vez. O que poderia ser?
#include <stdio.h>
#include <stdlib.h>

float medias[4][5];
int sair = 0;
int escolha = 0;

int main(){

    int i;
    int j;

    do{
    for (i = 0; i <4; i++){
            for (j = 0; j < 5; j++){

                    medias[i][j] = 0;

}}

    void menu(void);
    void entrada(void);
    float calculo(float nota1,float nota2);
    void varrer(void);
    void mediaturma(void);
    char aprovado(float nota);

    menu();
    printf("\n");

    if (escolha = 1){
            entrada();
            printf("\n");
            printf(" entrada de dados cadastrada com sucesso! \n");
        }
     else if (escolha = 2){
             varrer();
             printf("\n");
             printf(" medias calculadas com sucesso! \n");
        }
     else if (escolha = 3){
            mediaturma();
            printf("\n");
        }
     else if (escolha = 4){

            printf("\n");
            printf("--------------------------------------------------------------------------------");
            printf("                            RESULTADO FINAL");
            printf("\n");

            for (i=0; i < 5; i++){

                  printf(" RA: %f ' nota 1: %f' nota 2: %f' media: %f \n",
                         medias[1][i], medias[2][i], medias[3][i], medias[4][i]);
            }

            printf("\n");
            printf("--------------------------------------------------------------------------------");
        }
     else if (escolha = 5){

            printf("\n");
            printf(" Saiu com Sucesso! \n!");
            sair = 1;
        }
     else{

            printf(" Opcao Invalida!");
     }

        escolha = 0;

    }while (sair != 1);

    printf(" ");
    system("pause");
    return 0;
}

void menu(void){

    printf("\n");
    printf("                                      MENU \n");
    printf(" 1 - Entrada de dados \n");
    printf(" 2 - Calcular Medias \n");
    printf(" 3 - Calcular Media da turma \n");
    printf(" 4 - Imprimir Matriz \n");
    printf(" 5 - Sair \n");
    printf("\n");
    printf(" Escolha = ");
    scanf("%i", &escolha);
}

char aprovado(float nota){

    if (nota > 6){

        return "Aprovado";
    }
    else{

        return "Reprovado";
    }
}

void entrada(void){

    int i;
    int j;

    for (i = 0; i <4 ; i++);{
        for (j = 0; j <5 ; j++){

                if (i = 1){

                        printf(" Digite o RA do aluno %i: ",j+1);
                        scanf("%f", &medias[i][j]);

                }

                if (i = 2){

                        printf(" Digite a nota 1 do aluno %i: ",j+1);
                        scanf("%f", &medias[i][j]);

                }

                if (i = 3){

                        printf(" Digite a nota 2 do aluno %i: ",j+1);
                        scanf("%f", &medias[i][j]);

                }

    }}
}

float calculo(float nota1,float nota2){

     float calc = 0;
     calc = ((nota1 + nota2) / 2);
     return calc;
}

void varrer(void){

     int j = 0;
     for (j = 0; j < 5; j++){

            medias[4][j] = calculo(medias[2][j],medias[3][j]);
     }
}

void mediaturma(void){

  float total = 0;
  float mediat = 0;
  int i = 0;

  for (i=0; i < 5; i++){

    total = (total + medias[4][i]);
  }

  mediat = total / 5;

  printf(" A media da turma eh de %f ", mediat);
}

Answer: O valor da sua escolha sempre é 1 por causa da primeira condição if. Pois você não faz a comparação, e sim uma atribuição.
if (escolha = 1) // atribui 1 em escolha

Assim ele sempre entra nessa condição porque escolha será diferente de 0. O correto é:
if (escolha == 1) // assim ele compara escolha com 1 sem fazer atribuição de valor.

Isso vale para todas as outras condições que você fez, todas devem ser corrigidas.

Obs: O ideal é fazer a declaração dos métodos acima do main, não dentro dele.
Obs: O método aprovado, deveria ser do tipo const char por que o retorno é um texto constante.
Obs: As observações não são obrigatórias, é só uma questão de semântica.
Comment: Obrigado Cara... Ajudou muito

Condicional bugando em C


How to calculate the average age of observations in forecasting models of various types?

Question: Suppose you have a time series $\ Z_t$ that is used as a forecasting model for $\tau$ steps ahead from origin $T$. $\ Z_t$ is defined as:
$\ Z_t = 0.05 Z_T + 0.10 Z_{T-1} + 0.15 Z_{T-2} + 0.20 Z_{T-3} + 0.15 Z_{T-4} + 0.10 Z_{T-5} + 0.05 Z_{T-6 }$
How to calculate the $\ average\ age$ of this series observations in this forecasting model?

Is it right to assume the average age does not depend on the weights in each $\ Z_{T-i}$? In this case, the age of observation $\ Z_T$ is zero, since it's the observation at the origin $\ T$. The age of observation $\ Z_{T-1}$ is 1 since it happened one time-step ago. And so, ages will be respectively, from $\ T$ to $\ T-6$: 0, 1, 2, 3, 4, 5, and 6. Is this right?

And once the$\ average\ age$ is found, how to calculate a smoothing coefficent $\alpha$ that will yield a exponential smoothing model with the $\ average\ age$  equal to model $\ Z_t$?

If $\ Z_t$ were a exponential smoothing model, I'd know the value of $\alpha $; hence, I'd be able to calculate the average age of observations using equation: ($ \  1$ - $\alpha $)/$\alpha $. However, it seems to me that $\ Z_t$ is a simple $\ AR$ model (moving average).
Answer: The average age of observations in a model is calculating by attributing age 0 for the current observation at time $t$, age 1 for the first past observation $t-1$, age 2 to the second past observation or the observation $z_{t-2}$, and so on.
In the end, if the model goes back to observation $z_{t-m}$ and it has one observation for all the intermediate periods between $z_t$ and $z_{t-m}$, the average age $AA$ will be:
$\large AA = \large \frac{\frac{(r_0 + r_m).m}{2} }{m+1}$
Knowing that, by the one of the criteria above, $AA(r_0)$ = 0, one can write:
$\large AA = \large \frac{\frac{(r_m).m}{2} }{m+1}$
where $r_m$ is the age of the last observation, and $m$ is the index of the last observation. So, if the last observation is $z_6$, $r_m$ = 6, and $m$ = 6.


Derivation of confidence and prediction intervals of predictions for probit and logit (and GLMs in general)

Question: The derivation of the prediction interval for the linear model is quite simple: Obtaining a formula for prediction limits in a linear model .
How to derive the confidence and prediction intervals for the fitted values  of the logit and probit regressions (and GLMs in general)?
Comment: @statslearner I don't understand. Do you want a finite interval, which covers the future value of $y$, $100\%$ of times? Here it is: $I=[0,1]$. It doesn't even depend on $x$, what more you could ask for? Seriously, I don't think it makes sense to ask for a prediction interval with a Bernoulli output. On the other hand, if you were using logistic regression for a Binomial variable, then a prediction interval would make sense...
Comment: Are you defining "predictions" for a binary outcome to be the sampling average, like a proportion or percentage?
Comment: @AdamO I think he/she is defining predictions of logistic regression as the predicted conditional probability. I.e., the prediction is $\hat{p}(x)$, an estimator for $P(Y|X=x)$, not $\hat{Y}(x)$.
Comment: @AdamO good question,  I want an interval that makes my predictions cover the future values of $y$ 95% of the time. That could be done with an interval for $\hat{p}$ that translates into an interval of predictions (which would trivially be either just 0, just 1, or 0 and 1 I guess).
Comment: @statslearner hmmm wait, maybe we're talking about the same thing. Are you fixing $x$, sampling $y|x$ $n$ times, and looking for a prediction interval for the number of successes? The point estimate is obviously $n\hat{p}(x)$, and a trivial $100\%$ PI is $[0,n]$, but you would like a better (shorter) interval. If so, look here: https://stats.stackexchange.com/questions/255570/prediction-interval-for-binomial-random-variable/256706#256706
Answer: In GLM, prediction is a non-linear function $f$ of the product of covariates $X$ with estimated coefficient vector $\hat{\beta}$:
$$\hat{y} = f(X\hat{\beta})$$
Finite-sample distribution of $\hat{\beta}$ is generally unknown, but as long as  $\hat{\beta}$ is a maximum likelihood estimate, it has asymptotic normal distribution $\mathcal{N}(\beta, -H^{-1})$, where $H$ is the Hessian matrix of the likelihood function in its maximum. The p-values of $\beta$ that are shown as an output of a regression are nearly always based on this asymptotics. But if you feel your sample is too small for asymptotics, use numerical distribution (e.g. bootstrapping).
When you use asymptotic normal distribution of $\hat{\beta}$ (and therefore $X\hat{\beta}$), distribution of $\hat{y}$ is still non-normal due to non-linear $f$. You can ignore it - get normal confidence bounds $(z_{lower}, z_{upper})$ for  $X\beta$, and plug them into $f$, getting bounds for $y$ as $(y_{lower}, y_{upper}) = (f(z_{lower}), f(z_{upper}))$.
Another strategy (called delta method) is to take a Taylor expansion of $f$ around $X\hat{\beta}$ - it will be linear in $\hat{\beta}$. Therefore, you can approximate distribution of $f(X\hat{\beta})$ as
$$f(X\hat{\beta}) \sim \mathcal{N}\left(f(X\beta), -(f^{'}(X\beta))^2 X H^{-1} X^T \right)$$
Then the asymptotic 95% confidence interval for $f(X\beta)$ would look like
$$ f(X\hat{\beta}) \pm 1.96 \sqrt{(f^{'}(X\hat{\beta}))^2 X H(\hat{\beta})^{-1} X^T}$$
Now you need only to find expression for Hessian matrices for particular models, like logistic regression in this question. And this question presents practical comparison of bootstrap, transformed normal bounds, and delta method for logistic regression.
Comment: But does that give a confidence interval or a prediction interval of $\hat{y}$. It looks like a confidence interval only, doesn't it? It feels there should be an extra noise considered for a prediction interval.
Comment: That is only a confidence interval indeed.
But for binary responce models (like logit and probit), prediction is already probabilistic - real output is 1 or 0 with some probabilty. That is, "prediction interval" is always either $[0, 1]$, or $[0, 0]$ (if predicted probability is very small), or $[1,1]$ (if predicted probability is very large). But this predicted probability may vary, and confidence interval reflects this.
Answer: When all else fails, you can always construct bootstrapped CIs for any statistic. Here's a simple algorithm:

Draw $N$ samples with replacement from $X$ (where $N$ is the number of rows in $X$). You'll find that about 2/3rds of your observations will appear in such a sample.
Use these samples to fit a model
Use this model to generate predictions for the observations in $X$ that weren't used in training.
Repeat this process 100 or so times (the more the merrier) to accumulate a collection of predictions for each observation. This collection is an approximation to the distribution of your predictions. Call these your "bootstrapped predictions".
Construct confidence intervals by taking quantiles on the predictions. E.g. for a particular observation, calculate the .025 and .975 quantiles for a 95% confidence interval.
Comment: This is interesting, but under what circumstances can you guarantee me this will cover the true y 95% of the time?


Webpack maximum chunk size

Question: My current configuration is 1 bundle.js file with the main stuff the application needs (angular etc..) and all the others are lazy loaded via require.ensure in the application code.
After minify the bundle.js is above 100KB with just the basics for the application.
Is it possible to tell webpack to split the main bundle to max 20KB or so chunks so the browser can download them concurrently?
Answer: I haven't come across a good way to split chunks based on the bundle result itself.  You can set a min size for output chunks so that they are merged when there aren't many bytes in your chunk.
<code>    // This plugin prevents Webpack from creating chunks
    // that would be too small to be worth loading separately
    new webpack.optimize.MinChunkSizePlugin({
        minChunkSize: 51200, // ~50kb
    }),
</code>
But, I know that's not what you're looking for. Unfortunately, it looks like nobody has written anything like that yet.  Sounds like an interesting plugin to write!
Also, Take a look at this blog, which walks through some of the splitting basics.
http://blog.madewithlove.be/post/webpack-your-bags/
Answer: It is possible with "AggressiveSplittingPlugin". You can find it here:
https://github.com/webpack/webpack/tree/master/examples/http2-aggressive-splitting
Please note that this plugin is available only if you use webpack 2.
Comment: If you are using webpack 1, you can use some other plugins, that split your chunks by path - e.g. all  the files from chunk 1 that are from "lib" directory will be in a new chunk. Here are 2 plugins that can do it for you: "https://www.npmjs.com/package/webpack-split-chunks and "https://www.npmjs.com/package/webpack-split-by-path
Comment: Whilst this may theoretically answer the question, [it would be preferable](//meta.stackoverflow.com/q/8259) to include the essential parts of the answer here, and provide the link for reference.

Webpack maximum chunk size


How to assign roles to grant permissions to update particular pages?

Question: I have a menu like Publications, About us , Services and i want to assign a roles to the users like one user can update the content of about us and other can services means need to give permission for specific pages.
Is this is possible?
Some more details about what I'm trying to do: I do not want to give the permission to whole content type (that is not required). I want a user wise permissions like I have a role named ouroffice-user. Then under this I have a 10 users and I want to allocate different permissions to these 10 users like no. 1 have a permission to update the welcome content and no. 2 have a permission to update the about us page all users belong to single role that ouroffice-user.
Comment: Please check my edit of your question, to integrate your comment and improve the title. Feel free to further refine or improve it. If you don't like my edit just perform a rollback, ok?
Answer: The Group module allows for creating arbitrary collections of your content and users on your site, and grant access control permissions on those collections.
Group creates groups as entities, making them fully fieldable, extensible and exportable. Every group can have users, roles and permissions attached to it. Groups can also act as a parent of any type of entity. Group provides an extensive API to attach entities to groups.
For way more details about this module, such as how to configure it or available integrations with other modules, refer to my answer to "What are the alternatives for the Organic groups module?".
For this specific question you'd only need 1 Group Type, with 1 Group (of that Group Type) for each set (= collection) of "Specific Pages" (as in your question). BTW, available for either D7 or D8.
Have a look at the Video tutorial "How to use the Group module in Drupal 8 to manage editorial access control to different areas of a web site". Even though it is about D8 (already), the key concepts of it are the same as in the D7 version. Maybe also review the answers to various questions around here tagged with the group tag.
Some more details about this part of your question:

Some more details about what I'm trying to do: I do not want to give the permission to whole content type (that is not required). I want a user wise permissions like I have a role named ouroffice-user. Then under this I have a 10 users and I want to allocate different permissions to these 10 users like no. 1 have a permission to update the welcome content and no. 2 have a permission to update the about us page all users belong to single role that ouroffice-user.

Be aware that "roles" in the Group module are different from the typical (standard) roles in Drupal. For more details about the various roles and permissions that can be configured, refer to my answer to "What are the various roles supported by the Group module?".
Assume you create a custom Group role like "content manager". And you allow members with this group role to edit nodes of content type ABC.
From those 10 users (in your comment), those that correspond to your "no 1" should become members of some group (say titled Group no 1), in which they receive access to that Group role "content manager". And assume you create a similar group titled Group no 2, for those that correspond to your "no 2" users.
Using the gnode submodule you assign the nodes such as Welcome to the Group no 1 group and About us is assigned to the Group no 2 group. Note that it doesn't matter what the content types are (if they are the same or not), which is part of the beauty of the Group module.

Done ... really! Give it a try to get the idea ... those who try, love it ... after they get it ...
Comment: @its give the permission to whole content type but that is not required.

i want a user wise permissions like i have a role name ouroffice-user then under this i have a 10 users and i want to allocate different different permissions to these 10 users like no. 1 have a permission to update the welcome content and no. 2 have a permission to update the about us page all users belong to single role that ouroffice-user
Comment: @SumitAggarwal : any further feedback about my answer? Also, I noticed that you posted 3 new questions ater this one, 2 of them on hold already (be carefull you don't get question banned!). Correct me if I'm wrong, but I have the impression these questions are all a variation of the same. Moreover, if you'd actually fully reaize the power of the Group module, part of those questions might be solve-able ... using the Group module.
Comment: Actually i don't understand the group module. Basically i want a page where i can assign a permission rather than admin, another user which is created by admin have permission to assign the permissions to another users. The group module is quite confusing, i want a simple theme where a two dropdown is there and one can select  the user and other can select the permission as simple as that
Answer: Yes its possible Look into Content Access module.

This module allows you to manage permissions for content types by role
  and author. It allows you to specifiy custom view, edit and delete
  permissions for each content type. Optionally you can enable per
  content access settings, so you can customize the access for each
  content node.

One can install the ACL module and set per user access control settings per content node.

The ACL module, short for Access Control Lists, is an API for other
  modules to create lists of users and give them access to nodes. It has
  no UI of its own and will not do anything by itself; install this
  module only if some other module tells you to.
Comment: This module is asking for rebuild permissions? what to do ?
Comment: Click on rebuild.
Comment: i want a user wise permissions like i have a role name ouroffice-user then under this i have a 10 users and i want to allocate different different permissions to these 10 users like no. 1 have a permission to update the welcome content and no. 2 have a permission to update the about us page all users belong to single role that ouroffice-user
Comment: Check the module documentation and configuration. You can get more details.


Why can't we generalize straight line equations for all curves?

Question: Apologies, but I'm confused.
Let $y = f(x)$ be curve such that at a point $(a,b)$ lying on it, the slope of the tangent kissing that point is $f'(x)$
Now, the equation of the tangent passing through $(a,b)$ is found using the point-slope equation:
$$y - b = f'(x)(x-a)$$
But this would be the same equation I would get if I were asked to substitute the point and slope to find the equation of the curve.
Now, I know that the point slope form is for straight lines but I think this should be extendable to any curve because all curves just have variable slopes.
Maybe this is stupid and may not work. Could someone tell me why this is so?
Here's a demonstration of the point slope not yielding back the curve:
$$y=2(x−5)^2 +5 \stackrel{\frac{\mathrm d}{\mathrm dx}\text{ing}}{\implies} y' = 4(x-5)$$
It can clearly be seen that the point $(5,5)$ satisfies the given curve. So, using point-slope:
$$ y - 5 = y'(x - 5) \implies y = 4(x−5)^2 +5$$
From this example it is observed that using that the straight line equation does not yield back the curve. My only conclusion is that this is the equation of the tangent passing through $(5,5)$
Please correct where my thinking has gone wrong.
Comment: If we have an equation $$y-b=s(x)\cdot (x-a)$$ for a curve, then $s(x) = \frac{y-b}{x-a}$ is the _average_ slope of the curve between $a$ and $x$. If the slope is constant (the curve is a straight line), the average slope is of course always the pointwise slope. If the curve is not a straight line, then the average slope generally is neither the slope at $a$ nor the slope at $x$. It is, however, the slope at a point between $a$ and $x$: mean value theorem. So for a differentiable function $f$ you have an equation $f(x)-b=f'(\xi(x))\cdot(x-a)$, with a function $\xi$ that usually is complicated.
Comment: @DanielFischer: Your emphasis on the word "average" leads me to think that I have committed some blunder in equating an instantaneous measure to an average measure. Also, could you give some idea as to where $\xi(x)$ popped out of?
Comment: By the mean value theorem, there is a $c$ between $a$ and $x$ such that $\frac{f(x)-f(a)}{x-a} = f'(c)$. For each $x$, choose one of the possible values $c$ and you have your function $\xi$. That function is generally not uniquely determined, but there are such functions. It's not very helpful, however.
Answer: I'm not sure if you are still looking for clarification on this issue, but here's a go.
For simplicity, let's assume $y=f(x)$ is differentiable for all $x$, and $(x_0, y_0)$ is a point on the curve defined by this equation.  (I'll use $x_0$ and $y_0$ instead of $a$ and $b$, for convenience in an explanation below.)  The derivative is defined so that $f'(x_0)$ is tangent slope to the curve $y=f(x)$ at the point $(x_0, y_0)$.  Then, indeed point-slope form gives us the equation
$$ y - y_0 = f'(x_0)(x-x_0). $$
Note what you wrote was not quite right: $y-b = f'(x)(x-a)$ is not true for all $x$, only in general when $x=a$ (so $y=b$ and both sides will be $0$).  This was probably the source of your confusion.  I believe what you were thinking about is trying to reconstruct the curve $y=f(x)$ from knowing the derivative $f'(x)$ and a specific point $(a,b)$.  This is an important topic actually, and is the basic premise behind antiderivatives and differential equations.
Put another way, the equation for the tangent line has 3 components that depend on the specific point: $x_0$, $y_0$ and the slope $f'(x_0)$ at that point, so you can't just replace the $f'(x_0)$ with an arbitrary $f'(x)$ to get the curve, but keep $x_0$ and $y_0$ fixed.  (By the way, the curve you got by doing this faulty procedure is not the tangent line either because it's not linear--I don't know that that curve has much meaning.)

Here's a better way to think about trying to reconstruct the curve from the derivative and a specific point.  I'll try to explain this as briefly as I can, but feel free to ask for clarification.
Let's work with the simple example of $y=f(x) = x^2$, and pick the starting point $(x_0, y_0) = (0, 0)$.  Here $f'(x)= 2x$,
so at the starting point the slope is $m_0 = f'(x_0) = 0$.  This means the curve is close to the tangent line $y-y_0 = m_0(x-x_0)$, i.e., $y=0$ nearby the point $(0,0)$.  That is, we can approximate the curve near $(0,0)$ geometrically by drawing a short horizontal tangent through $(0,0)$.
Now we can do this again.  For simplicity, let's say we approximate our curve with the horizontal tangent to $(x_1, y_1) = (1, 0)$ (this is not a point on the curve, but on the tangent line).  Now we can use the derivative at 1, $f'(x_1) = f'(1) = 2$, to draw another line segment, say to $(x_2,y_2) = (2,4)$, which is an approximation of the tangent line to $y=f(x)$ at $x_1$.
This is much easier to understand with pictures--take a look at the pictures here for what's going on.  I'm just saying we can successively use derivatives (tangent slopes) to approximate the curve.  Of course my approximations in my example were very bad, but they get better and better by taking smaller steps (e.g., $x_1 = 0.1$ or $0.00001$, and $x_2 = 2x_1$).
To actually recover the curve $y=f(x)$ exactly, you need to take infinitely small steps, or more precisely, take the limit of doing this process with smaller and smaller steps.  This is precisely what the antiderivative is.
Comment: "*trying to reconstruct the curve y = f(x) from knowing the derivative f′(x) and a specific point (a,b)*" was exactly what I was attempting. I was an idiot to not see that $y = f'(a)(x-a) + b$. If I had done the math correctly, the result of my example would have been $y = 5$ instead of $y= 4(x-5)^2 + 5$ and this would have been right since I've plugged in the point $(5,5)$. This answer means a lot to me and it has possibly re-established my foundation in straight lines and derivatives. Thank you very much :D


How to limit logon time access for groups

Question: I have to limit logon time access for some users. I have googled and found it is possible with pam modules and time.conf.
My question is: is it possible to do the same but with groups so, instead of dealing with users, just gather these users and set limited logon time to this group?
I am using an ldap server with centos7.
Answer: Yes this can be accomplished with some work with a combination of pam_listfile and pam_time. You will have to fill in the details but here is the skeleton in your pam.conf
auth [success=1 default=ok] pam_listfile.so item=group sense=deny file=/path/to/restricted/groups onerr=fail
account required pam_time.so ...

The success=1 means, if this module returns success, then skip the next 1 module. I.e. if the user is not in any of the denied groups, don't run pam_time.so, otherwise do run it.


Coefficients of a linear combination

Question: Suppose I have a vector space $V$, with $\{\vec e_1, \ldots, \vec e_n\}$ forming a basis of $V$. Also, suppose that I have a vector $\vec x \in V$, which I want to express as the unique linear combination of $\vec e_1, \ldots, \vec e_n$, e.g.
$$\vec x=
  a_1 \vec e_1 + \cdots + a_n \vec e_n $$
Is there any MATLAB code, so I can get the coefficients $a_1, \ldots, a_n$?
Comment: You can set it up as a linear system $Ea=x$ where $E$ is the coefficient matrix of the basis elements (here, the $e_i$ are column vectors), $x$ is your vector, and $a$ is the vector of coefficients you are solving for.  Then any of the linear system solution routines in Matlab would work.
Comment: Multiply by the inverse :)
Comment: @Paul, Ok, I'll check it out! Thank you!
Answer: (sorry, I didn't mean to restate the top comment as an answer, I just typed way too slowly )
Finding the coefficients $c_1 ... c_n$ for a vector $\vec{x}$ given basis $\{ \vec{b_1} ... \vec{b_n} \}$ amounts to solving
$
\left[
\begin{array}{ccc}
\vec{b_1} & ... & \vec{b_n}
\end{array}
\right]
 $
$
\left[
\begin{array}{c}
c_1 \\
... \\
c_n
\end{array}
\right]
= $$
\left[
\begin{array}{c}
x_1 \\
... \\
x_n
\end{array}
\right]
$
So you should be able to solve this in MATLAB with the standard solve operator:
c = B\X

Where B is the matrix composed of your basis vectors and X is your $\vec{x}$ (all as column vectors).

Coefficients of a linear combination


Melting a spherical ball with density

Question: Six spherical copper balls of radius 6 centimeters fit snugly, one on top of the other, inside a cylindrical can. The copper balls are melted and poured into the can. How high, in centimeters, will the molten copper rise? Assume that the density of solid copper is 8.96 g/mL and the density of liquid copper is 8.02 g/mL.
NOTE: 1 cu. cm. = 1 mL.
I got the volume of the copper balls and the volume of the cylinder, I don't know the next step to find the height.
Comment: The volume of the cylindre doesn't matter.
Answer: You say you know the volume of the cylinder. I assume that you computed it using some height (probably 72cm) for the cylinder. However, if you can do that computation you can probably also compute the volume of a cylinder with any given different height. And if you can do that, I have no doubt that you can even compute the volume of a cylinder with an 'abstract' height $h$.
Once you can do that, the question 'for which $h$ is the volume of a cylinder with height $h$ (and radius equal to that of the original cylinder) equal to the volume of the molten balls?' is not that hard.
And the answer to that question is the answer to the question of how high the copper will rise.
Answer: The mass of copper is
$$m=6d_s\frac{4\pi r^3}{3}.$$
When melted, the volume is
$$v=\frac m{d_m}=6\frac{d_s}{d_m}\frac{4\pi r^3}{3},$$
hence the height
$$h=\frac v{\pi r^2}=8\frac{d_s}{d_m}r.$$

Melting a spherical ball with density


Restoring database to a point in time not working

Question: I have a database that is backed up nightly via a maintenance plan I created using the wizard in management studio.  The recovery model is set to full.
On the 26th something happened and SQL Agent was shut down (or maybe the computer was reset and it didn't start back up properly).  I didn't realize it until about 5 days ago at which time I turned it back on (and am now in the process of making sure I get notified when SQL Agent is off in the future).
I need to restore the database to a few points in time between the 26th and now, but when I try to do this Management Studio acts like it is doing it but ends up just restoring to the 26th no matter what time I actually specify (unless the time I specify is after the point when I turned Sql Agent back on).
Even though it didn't get backed up for a little over a week, I would expect the log to still have kept all the transactions so that I could restore to any point in time.  Shouldn't I still be able to restore to any point in time or am I missing something here?
Answer: You can use a third party tool to specify exactly what transactions you want to be rolled back while recovering to a specific point in time, using various filters and even be able to review them before the recovery process. Check out the Reverting your SQL Server database back to a specific point in time online article for detailed information
Disclaimer: I work as a Product Support Engineer at ApexSQL


Auto set height of table view based on UICollectionView inside it

Question: I have a UITableView with cells that have inside a UICollectionView.
Now all works great except of UICollectionViewCell scrolling i want that the height of table view cell follow the Collection view cell content.
In simple word i want scrolling in table view but not inside single cell i need to display all contents of UIcollectionViewCell without scrolling inside it.
Thanks
Answer: You have to calculate the collectionviews' heights and set a height constraint on each of them. then add top and bottom constraints from the collectionviews to their superviews (the tableviewcells / their contentviews) and let autolayout do its work.
don't forget
tableView.estimatedRowHeight = 44 // or something similar
tableView.rowHeight = UITableViewAutomaticDimension

in the controller's viewDidLoad to make the automatic row height calculation work.


TypeError: bad argument type for cv2.imread() from Raspberry Pi Camera on Jetson Nano

Question: I am running a raspberry pi camera on the Jetson Nano with Python 3.6.9, OpenCV version 4.1.0, and the latest version of Ubuntu. I am running the code:
import cv2
print(cv2.__version__)
dispW=320
dispH=240
flip=2
cam_set='nvarguscamerasrc !  video/x-raw(memory:NVMM), width=3264, height=2464, format=NV12, framerate=21/1 ! nvvidconv flip-method='+str(flip)+' ! video/x-raw, width='+str(dispW)+', height='+str(dispH)+', format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink'
cam = cv2.VideoCapture(cam_set)
while True:
    ret, frame = cv2.imread(cam)
    cv2.imshow('Camera',frame)
    if cv2.waitKey(1)==ord('q'):
        break
cam.release()
cv2.destroyAllWindows()

When I run it, it throws an error with a traceback to line 11, saying that "ret, frame = cv2.imread(cam)" has a TypeError: bad argument type for built-in operation. I can use opencv to load images, but running a video always seems to throw this error. "ls -l /dev/video0" in the command line works fine, so I know I am connected to the camera. Thanks!
Answer: As per documentation:
Python: cv2.imread(filename[, flags]) → retval
imread expects a path while you need this snippet:
import numpy as np
import cv2

cap = cv2.VideoCapture(0)

while(True):
    # Capture frame-by-frame
    ret, frame = cap.read()

    # Our operations on the frame come here
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    # Display the resulting frame
    cv2.imshow('frame',gray)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# When everything done, release the capture
cap.release()
cv2.destroyAllWindows()


Is there a way to get get all mentions of a certain pattern and following values in a string in python

Question: Im trying to get all values in a string that's looks like the following:
example = 'jnkfksieufsieufsejfuies Title="Something" daowijdaowdnobngfoinbijfgh Title="something else"
is there a way to get the values following the Title variable in the string.
I thought to use some sort method to get all Title mentions in the string, then be able to get the following values but im not sure how that is possible in python.
Im guessing i could use some pattern regognizion from the "re" dependency
Comment: Yes, module "re" is the usual approach. You can use https://regex101.com to play around with regular expressions (set flavor to "Python").
Answer: Use re.findall with a pattern having a group to match what's after the Title
example = 'jnkfksieufsieufsejfuies Title="Something" daowijdaowdnobngfoinbijfgh Title="something else"'
titles = re.findall(f'Title="([^"]+)"', example)
print(titles)  # ['Something', 'something else']

Comment: always search for documentation of the package you're using: https://docs.python.org/3/library/re.html
Comment: @pvpb0t learn about regexes, I won't re-copy the websites that does it very well like regex101.com or many others
Comment: Is there a way to read up on what all the symbols used in the re.findall do? Like for example "[^"]+"


Beethoven piano concerto 3 mvt 3

Question: How should I consider this part ?

Is there something different that we should hear, or that should appear when I play apart from the regular scale ?
Why aren't the notes organized regularly 4-4-4 or 8-8 ?
Answer: The notes would be grouped 8 - 8, except that the notation in the first bar shows (by the stem directions and the rests) that the left hand plays the first 7 notes (written with stems down) and the right hand the remainder (with stems up).
Beethoven wrote this concerto to perform himself, so the notation that shows how the notes are split between the hands is probably by Beethoven himself, not a later editorial addition.
Notation conventions say that you shouldn't write dotted rests, so it makes sense to beam the first 7 notes in groups that match up with the three rests.
In the second bar, the beaming of 4 - 5 is easier to read than a single beam over 9 notes.
This has nothing to do with "Mixolydian mode". It's just a scale that happens to start on G in the key of Ab major, over an Eb7 chord. The scale doesn't start on Eb for the rather prosaic reason that when Beethoven wrote it, the lowest note on his piano was F - the compass had not yet increased beyond 5 octaves F to F.
The beaming in the third bar may be following a notation convention that was common in Haydn and Mozart, where the length of the beams tended to correspond to what would now be marked by short slurs. If Beethoven was following that convention, beaming the triplets in 4 groups of 3 would have implied something different from 2 groups of 6.
Contemporary evidence suggests that Beethoven was somewhat "mathematically challenged", so perhaps one shouldn't read too much into the notation of triplets rather than sextuplets!
Comment: Would it be fine then if my right hand take over at the first Eb in the second bar, conventionally ?
Comment: In that second bar, it has to be beamed 4 and 5, as the 5 make a quintruplet, so couldn't be joined with the 4. I like the last bit, but the'd only get played in one way regardless.
Answer: Guessing the time sig. is 4/8, it is written regularly. the 1st bar shown has two lots of four semidemis (7+rest) probably played with l.h., then eight beamed together, played r.h. No problems there.
Next bar is 8 semidemis beamed together, then four, but the last 5 are played in the time of 4, so no other way to write it.
Bar 3 - they're all triplets, admittedly not written perfectly clearly, but you'll play 6 in the time of 4, and again to finish the bar off.So, the notes are structured in 8+8. They couldn't be in just 4+4+4 without another +4.
The scale seems to be an Eb Mixolydian mode run.Regular Eb scale, but with a flattened 7.
Hope that all clears up the confusion!
Comment: Not really Mixolydian - the cue notes at the beginning suggest that the scale is a prolongation of an E♭7, i.e., V7 of IV. In fact, the subsequent solo passage is a prolongation of this, ending with a statement of IV in root position. The time signature is 2/4, which your explanation fits well.
Comment: @Patrx2 - Isn't Eb Mixolydian made up with the exact same notes as Eb7? Yes, of course, the dominant of Ab (IV) using notes from Eb, with the flattened 7. All leading up, as Ludwig often did, to the IV. Same thing, in reality.
Comment: Not really - the cue notes show only a small part of the orchestral accompaniment, which sets the scale with a held dominant seventh on E♭ (with the seventh D♭ held by the violas, 'cellos and contrabasses). The dominant seventh sonority isn't stable enough in Beethoven's language to act as a tonic, not even as a modal final - the implication is fairly clearly A♭ major.
Comment: The thing to remember sometimes is that `mode != scale` - a mode is articulated by a lot more than just a scale.
Comment: @Patrx2 - fair enough, but I only had that snippet to go on, and if those notes only are played, it's not a scale, but I felt it would be recognised as the notes from Eb Mixolydian. Of course it's in Ab; that's where it's all going, but doesn't Eb Mixolydian use the Ab notes? Is that wrong?
Comment: The mode does use the same notes, but context and setup determine which mode is in play. In this particular case, there is no change of mode, but there *is* a temporary change of key - this single phrase is in A♭ major, which passes back to the home key in the following phrase. It's about 2/3 or so through the piece, which is where references to the subdominant tonality are common in High Classical music (Beethoven included).
Comment: In the score, you'd notice that the G that opens the scalar run is below D♭, which becomes the bass as the piano scale rises. Both are chord tones, but that's a heck of an unstable sonority for a tonic chord. It does make sense as starting a scale run with a leading tone, though.

Beethoven piano concerto 3 mvt 3


Using MSE to determine prediction intervals?

Question: I have a model that I've used MSE as the accuracy metric. After reading this article on calculating prediction intervals using MSE I looked around for other resources.
I then found this PDF from Wharton that suggests y_hat +/- 2 * RMSE forms a prediction interval for y. However it doesn't tell me what percent prediction interval this is. Is this the 95% prediction interval for y_hat?
As a result I'm a little confused on how to derive the right formula for a N% prediction interval given the RMSE or MSE of the model. Could anyone point me to the right resources, or show me how to derive the formula for this, so I can actually understand how they've arrived at what seems like two different formulas?
Comment: That reference isn't quite right.  It is making some (unstated) approximations, presumably to address an audience that might be confused by a full and accurate account of a prediction interval.  Correct formulas are provided in several threads here, such as http://stats.stackexchange.com/questions/9131.  They are needed in any situation where the fitted curve has appreciable uncertainty.
Comment: @whuber would you be willing to outline those unstated approximations in a comment or answer for completeness sake?
Comment: I already have: the approximation completely ignores uncertainty in the fitted curve.
Answer: To your first question:  Wharton does give the ~95% confidence interval.  You know this because the formula requires a quantile to be multiplied by a standard error.  In this case the quantile represented is 2, which corresponds roughly to ( 1.96 ) a 95% C.I.
In order to calculate N% prediction intervals using this formula you need to adjust the quantile variable accordingly.  For example to calcuate a 90% confidence interval change the 2 in the Wharton equation to 1.645.
Comment: @grldsndrs You mean "if the d.f. is *infinite* then it is the same as the normal distribution", not 1.
Comment: Ah I see, this assumes a normal distribution correct? If I wanted a t distribution I would have to adjust this I assume.
Comment: @ChrisHaug  Yes you are quite right.  I should have written infinity.
Comment: Yes according to the number of degrees of freedom, but if the d.f. is infinite then it is the same as the normal distribution.
Comment: What is your sample size? The degrees of freedom is not infinite but rather is determined by your sample size. If the sample size is less than 20 the normal may not provide a good approximation. But if it is greater than 100 it is probably good enough.
Comment: Shouldnt the default general interval be a t-interval since we are using an estimator for the standard deviation?


How does changing the log base affect the intepretation of information entropy

Question: Entropy of random variable is defined as:
$$H(X)= \sum_{i=1}^n p_i \log_2(p_i)$$
Which as far as I understand can be interpreted as how many yes/no questions one would have to ask on average, to find out the value of the random variable $X$.
But what if the log base is changed to for example e? What would the interpretation be then? Is there an intuitive explanation?
Comment: it will multiply $H(X)$ by $\ln 2$..
Comment: To those calculating the new value but not explaining what it "means" conceptually, you seem to be missing the point of the question. The OP wants an _interpretation_ of the concept of information when the log base isn't 2.
Comment: Perhaps this is a little disingenuous, but I see no reason why there should be a nice interpretation. One could argue that the reason we use base two to define entropy is precisely because one can then couch it in the intuitive and pleasing terms of 'number of yes/no questions' (although this itself has issues - what are $1.5$ yes/no questions?)
Comment: @stochasticboy321 well you could take the ensemble average over all recursive trees whose average branching is 1.5 -- in any case there is a clear interpretation for integer bases
Comment: @William right, so we've gone from the simple yes/no question to ensemble averages over recursive trees, which is already a sharp decrease in simplicity/niceness of interpretation. Far as I see, that supports my point. (That said, I do agree with you, interpretations for integer bases seem well within reach).
Comment: @stochasticboy321 I agree, of course the simplicity of the interpretation has decreased. Defining logarithms for non-integer bases and values is already fairly non-trivial itself (see problems 6 and 7 of chapter 1 of Baby Rudin) so we shouldn't expect any simple interpretation for those cases. The natural logarithm is actually a fairly nasty/difficult function to define accurately/precisely from the rational numbers -- it just leads to neater formulas. (There's a reason why base 2 is more commonly used in computer science, or base 10 in science when reporting measurements of data.)
Comment: I get the feeling that we're converging to a similar place - Clearly, it's possible to construct interpretations for integer bases, and of these base $2$ is the cleanest in some sense. What this doesn't naturally lead to is an interpretation of the entropy in an arbitrary base, and I posit that a satisfying one is unlikely.
Answer: Obviously this interpretation breaks down (at least somewhat) for non-integer bases, but for any logarithm base $b$, not just base $b=2$, we can interpret the information with respect to that base, $$H_b(X) = \sum_{i=1}^n p_i \log_b(p_i)$$ to be the average number of $b-$ary questions one would have to ask on average in order to find out the value of the random variable $X$.
What do I mean by a $b-$ary question? One that has at most $b$ possible answers. Any yes/no question is a $2-$ary question, and if one can reword it cleverly enough, any $2-$ary question can be stated as a yes/no question (since yes and no are both exhaustive and mutually exclusive).
More precisely, a $b$-ary question is one that has $b$ possible answers, which together exhaust all possibilities and each of which is mutually exclusive. In general, for $b \not=2$, it might be difficult to think of truly $b-$ary questions which don't involve artificially restricting the possible answers.
Consider, for example, trying to determine the value of a dice roll. If you are limited to $2-$ary questions, (e.g. "did it roll a $1$?") then on average it will take $\log_2 6$ questions to ascertain what value it rolled. However, if you are allowed to ask the $6-$ary question "did it roll a 1, 2, 3, 4, 5, or 6"? Then it will only take you $\log_6 6 =1$ question on average to ascertain the value.
If you consider the question tree consisting of all possible questions to derive the answer, then $b$ is just the number of branches emanating from each node (since each node is a question). If you increase $b$, you will decrease the number of questions necessary on average to ascertain the answer because each node will have more branches emanating from it, and thus you can traverse a larger portion of the set of possible answers by hitting fewer nodes.
This is in fact very similar to the idea of recursion equations and recursion trees in computer science; in particular, I encourage you to look at Section 4.4 of Cormen et al, Introduction to Algorithms, for an explanation of how the logarithm enters naturally into these types of situations.
We can think of asking a $b-$ary question as dividing the problem of finding the value of the random variable $X$ into $b$ subproblems of equal size -- then the analogy with recursion should become more clear (hopefully).
Comment: I think we can extend this to say that base $2.5$ entropy counts the total number of 2-ary and 3-ary questions required on average (given that we ask the same number of each type of question on average).
Comment: @CurtisBechtel That sounds about right, although it might be something like the geometric mean of $2$ and $3$, $\sqrt{6}$, and not the arithmetic mean $2.5$, if both $2$-ary and $3$-ary questions are asked on average. (Since of course as you know logarithms translate multiplicative features into additive ones.) To be honest though I never really bothered to put in enough effort into thinking about it.
Answer: One has
$$\log_a(x)=\frac{\log b}{\log a}\:\log_b(x)$$
thus
$$H(X)= \sum_{i=1}^n p_i \log_a(p_i)=\frac{\log b}{\log a}\sum_{i=1}^n p_i \log_b(p_i)$$ or
$$
H_a(X)=\frac{\log b}{\log a}H_b(X).
$$


Generating survival times for a piecewise constant hazard model with two change points

Question: When there are two change points in  a piecewise constant hazard model then the density function becomes some triangle exponential distribution. In this situation I can't generate the survival time from the CDF using probability integral transformation. Can any one help me to generate the survival time from this model?
Answer: There are two basic approaches to generating data with piecewise constant hazard: inversion of the cumulative hazard and the composition method.

Inversion of the cumulative hazard - essentially the inverse CDF method. Since $F(t) = 1-\exp(-H(t))$. If $U \sim Unif(0,1)$, then $F(X) = U$ is equivalent to $1-\exp(-H(X)) = U$, so $X=H^{-1}(-\log(1-U))$. You can also note that $-\log(1-U) \sim Exp(1)$, so you can apply the inverse cumulative hazard to an exponential random variable.
The cumulative hazard is piecewise linear for your case, and should be easy to invert.

Edit (more detail): with two change-points, the hazard is:
$$h(t) = \left\{ \begin{matrix} f_1 , & 0\leq t\leq t_1\\
                                f_2, & t_1 < t \leq t_2\\
                                f_3, & t > t_2 \end{matrix}\right.$$
The cumulative hazard is:
$$H(t) = \left\{ \begin{matrix} f_1 t , & 0\leq t\leq t_1\\
                                f_1 t_1 + f_2(t-t_1), & t_1 < t \leq t_2\\
                                f_1t_1 + f_2(t_2-t_1) + f_3(t-t_2), & t > t_2 \end{matrix}\right.$$
The inverse of the cumulative hazard is:
$$H^{-1}(x) =  \left\{ \begin{matrix} x/f_1 , & 0\leq x\leq f_1t_1\\
                                t_1 + (x-f_1t_1)/f_2, & f_1t_1 < x \leq f_1t_1 + f_2(t_2-t_1)\\
                                t_2 + (x-f_1t_1-f_2(t_2-t_1))/f_3, & x > f_1t_1 + f_2(t_2-t_1)\end{matrix}\right.$$
Now generate an exponentially distributed random variable, and plug it in into $H^{-1}$.
End edit

The Composition method uses the fact that if $X_1$ has hazard $h_1$, and $X_2$ has hazard $h_2$, then $X=\min(X_1,X_2)$ has hazard  $h=h_1+h_2$. You can represent your piecewise constant hazard as a sum of hazards that are constant on an interval and 0 outside. Generate a value $X_i$ for each interval (it could be $\infty$, since the resulting distributions are not necessarily proper), and take their minimum.

Edit (more detail): with the above notation, the composition hazards are
$$h_1(t) = \left\{ \begin{matrix} f_1 , & 0\leq t\leq t_1\\
                                0, & t > t_1 \end{matrix}\right.$$
$$h_2(t) = \left\{ \begin{matrix} f_2 , & t_1 < t\leq t_2\\
                                0, & \text{otherwise} \end{matrix}\right.$$
$$h_3(t) = \left\{ \begin{matrix} f_3 , & t_2< t\\
                                0, & \text{otherwise} \end{matrix}\right.$$
You can easily calculate the CDF or the cumulative hazard for each of these hazards.
One resource with R-Code
Comment: Thanks for your answer. However, I'm talking about the density function which takes value f1 in the range 0 to t1, value f2 for the range t1 to t2, and f3 for more than t2. In this situation, can I use the 2nd approach? I know the first approach is more applicable, but I can't derive the CDF for this complicated function.
Comment: Thank you so much Aniko for your such detail suggestion. Great Support!!
Comment: @Aniko how would the Composition method be applied to a truncated piecewise exponential?
Comment: @user43290 A simple method for generating any truncated distribution is to generate the untruncated version, then throw away all values outside the range of truncation. If the truncation rate is high, then this is not efficient, and one could work out the component hazards more precisely. They would depend on the probability of truncation.


wordpress images staggered

Question: Wordpress noob alert! In my home page's HTML editor, I've added a div that is the footer. Loaded in this  is a row of logo images that are supposed to sitting a straight line - however, they display staggered vertically. The top left logo (Ocean Basket) sits correctly, the next one has too much padding / margin, the next even more and then next logo even more and so on. I've set the padding and margin to 0 in the css, floated the images left - I cannot understand why this is happening! Any ideas anybody?
test site at http://inexshopfitting.co.za/
Answer: try adding display: block; to the images and get rid of the
 in between.
(for example by adding #home_logostrip_box br{ display:none; } to your themes css file)
That should do it.
Comment: I am very nervous about hacking into the core, let me be honest :D I wouldn't have suggested that, especially since i'm such a noob at wordpress but didn't know how else to get rid of the line breaks!
Comment: Hi Max. Thanks for your suggestions! I never inserted any line breaks after the images but after your suggestion I went and viewed source and in there I see that there are 's after the images. They are not there in the HTML in the WP editor - what to do? How do I get them out? Thank you!!!
Comment: Found the answer yay! On line 248 of wp-includes/formatting.php,
change "\n" to "" - to remove any pesky unwanted line breaks. Hope it doesn't break anything!!!
Comment: i edited to answer sugesting another solution (which does not involve hacking into the core ;)
Answer: I had 5 images floated next to each other (20% each) and it worked fine in HTML, but not when site was converted to Wordpress; the images where floated but they 'stairstepped/staggered'. Like mentioned before, there are bracketed 'br' elements between each image---inserted by Wordpress. I fixed by adding to my images' container class, .classname br { display: none }. It worked perfectly.

wordpress images staggered


What should be the signature value in the URL for HTTP-Redirect Binding SAML Request

Question: This query is in addition to the question asked in post - HTTP-Redirect Binding SAML Request.
I am trying to implement the dotnet SSO solution by sending the SAML Authentication Request via HTTP-Redirect Binding.
After generating the SAML AuthnRequest and computing the signature using the private certificate of SP I have tried two ways to add the signature in the url.

Generate SAML AuthnRequest XML.
<?xml version="1.0" encoding="utf-8"?>
<AuthnRequest xmlns:xsd="http://www.w3.org/2001/XMLSchema"
              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              ID="_d726aced-4c62-4562-8237-062f7faf0750"
              Version="2.0" IssueInstant="2015-04-03T19:59:58.6709058Z"
              Destination="https://idp.client.com/adfs/ls/"
              ProtocolBinding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST"
              AssertionConsumerServiceURL="https://serviceprovider.clientportal.com/samlsso.aspx"
              xmlns="urn:oasis:names:tc:SAML:2.0:protocol"
>
        <Issuer xmlns="urn:oasis:names:tc:SAML:2.0:assertion">https://serviceprovider.clientportal.com</Issuer>
        <NameIDPolicy Format="urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified" />
</AuthnRequest>

Compute Signature using the SAML AuthnRequest XML which generates an XML Signature tag as shown below
<Signature xmlns="http://www.w3.org/2000/09/xmldsig#">
    <SignedInfo>
        <CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
        <SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1" />
        <Reference URI="#_d726aced-4c62-4562-8237-062f7faf0750">
            <Transforms>
                <Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature" />
                <Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
            </Transforms>
            <DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />
            <DigestValue>rxoGtdmI...rgY=</DigestValue>
        </Reference>
    </SignedInfo>
<SignatureValue>XRLdJHIGWClJxp....fTPUfIghl5kTfV4=</SignatureValue>

Method-1 to add signature in the query string
Encode the AuthnRequest XML (without signature embedded) using Base64 encoding and URL encoding
Encode the above xml signature string using Base64 encoding and URL encoding
Construct the URL in the following way
https://IDP.com/adfs/ls/?SAMLRequest=UrlEncodedBase64AuthnRequest&RelayState=value&SigAlg=value&Signature=UrlEncodedBase64EncodedXMLSignature

Method-2 to add signature in the query string
Encode the AuthnRequest XML (without signature embedded) using Base64 encoding and URL encoding
Extract the value of tag from the above xml signature string to be sent as URL encoded value of "Signature" query string parameter.
Construct the URL in the following way
https://IDP.com/adfs/ls/?SAMLRequest=UrlEncodedBase64AuthnRequest&RelayState=value&SigAlg=value&Signature=UrlEncodedValueExtractedFromSignatureValueTag

In both the cases I am getting signature validation failure at the IDP. Any idea on how to generate the signature to be sent in the URL?
Any help on this would be of great help!
Thanks,
Piush
Comment: I am also facing similar issue. Can you please share the solution.
Answer: See "Section 220.127.116.11 DEFLATE Encoding" in the SAML 2.0 Bindings document for specific instructions on how you are supposed to include the DSig when using the REDIRECT Binding.
Comment: I am also facing similar issue. Can you please share the complete solution.
Comment: How did you actually go about creating the Signature value in code?
Answer: The signature used in the HTTP-Redirect binding is not an XML signature. I suggest taking another look at the SAML v2.0 Bindings Specification, section 3.4.
Alternatively, take a look at our ComponentSpace SAML v2.0 component. This is a commercial product for .NET/ASP.NET which fully supports the SAML specification.
Comment: Thanks ComponentSpace.


Very fast virtual machine rollback

Question: Summary: Is there a VM that's optimized for fast in-memory rollback to a previous state?
I've tried in vain on StackOverflow and SuperUser — maybe this is the right place to ask this question. :)
I'm looking for the fastest way to rollback a virtual machine to a previously known state. I'll be doing a lot of rollbacks to the same state. Taking a snapshot can be slow, but the rollback needs to be fast.
Ideally, I'd like the performance of the rollback to be as close to the performance of a single memcpy() as possible.
I'm currently testing things on my portable machine (Mac OS X), but will later deploy this to a beefy Linux server. Portability would be nice, but only running on Linux is fine. Commercial solutions are fine, if they get the job done.
For reference's sake, here's how long a memcpy() takes on my machine: (Code available here.)

     Size      Time
 ========= =========
    64 MB   0.030 s
   128 MB   0.085 s
   256 MB   0.140 s
   512 MB   0.290 s
  1024 MB   0.600 s

So far I've tried VirtualBox and VMWare Fusion on my machine. Performance was comparable for both.
Ideally, I'd also like to rollback the disk state. VirtualBox seems to have support for "immutable" disks, where all disk I/O is logged to a differential file and thrown away on rollbacks. That seems like a natural fit.
For VirtualBox rolling back a VM with 2.5 GB of assigned RAM with the state saved on disk (a SSD) took ~35 seconds.
Since disk I/O is expected to be slow, I next created a RAM disk (formatted as HFS+) and moved the snapshots there. The rollback of the otherwise unchanged VM to the same state then takes ~16.5 seconds.
The snapshot data takes up 1.11 GB. So VirtualBox takes ~15s / 1 GB of data when rolling back from a RAM disk. That's 6.72x the time of a memcpy(). (Does going through Mac OS X' I/O system cause that much overhead?)
Since I know of no way to rollback a VirtualBox VM without first shutting it down, and later starting it up again, it could be that some of this is caused by other factors.
Ideally, the hypervisor would keep all snapshot state in main memory, and use memcpy() for a rollback, letting us rollback in ~5 seconds. (An even smarter hypervisor might implement copy on write on pages and only rollback memory pages that actually changed.)
Are there any hypervisors which are optimized for rollback-frequently workloads?
If not: What might be the most viable way of implementing such a thing myself?
(I've also done a bit of research into process-level rollback and there seems to be some interesting solutions for that for Linux; but having rollbacks work for arbitrary applications and being able to rollback I/O are big pluses.)
(What I really want to do is rollback to a checkpoint in arbitrary Java applications, ideally together with their I/O. I've only found dead research projects for rolling back a JVM, but maybe somebody is aware of more current work? It'd however need to do with pretty much arbitrary Java applications.)
Comment: tbh situations where 'rollback-frequently workloads' are truely appropriate are exceptional and chances are in most cases would indicate that you are doing it wrong. Not saying you are, but it explains why there is little support for it.
Comment: Do they have it where it is required to do it every 30 seconds though?  Time seems to be a considerable concern with this instance.
Comment: When a snapshot is made, it starts a new differential "disk" on the drive. At least in Virtualbox, if you stop the server and delete the changes, it's made instantly.
Comment: @JamesRyan Not necessarily. Especially automated build machines, unit tests etc. may require machines in a known state (install/uninstall testing). Start, run test, roll back to known state for next test. Lab setups have this requirement at times.
Comment: @TomTom: Absolutely. What I'm actually doing here is to automatically come up with tests for an application. It makes sense to be able to start from a sane state for that, and throwing a differential disk file together with memory changes would perfectly address my use case.
Answer: I do not know what your actual needs are. However is it possible to run multiple clones and do some kind of round robin loadbalancing? Then you could rollback those machines in the background and you have more time at hand to do so.
Comment: Absolutely. However cheaper rollbacks would mean I require less clones; so minimizing the rollback time is still interesting to me.
Answer: Not really.  The purpose of a snapshot is to take a "shot" of the current system state so if something goes wrong in the immediate future (software update or install), it can quickly be returned to the state previous to the installation.  That way you don't have to do a full recovery from back up.  The snapshots are meant to be retained for only a short period of time and keep those clean out per VMWare.
Something similar to what you are looking for is VMWare Site Recovery Manager which allows for you to have a completely replicated VM infrastructure and will only partially work for what you need.  I do not believe that it exactly fits your purpose because it is more for a mirrored copy of your VM, not a "roll back" state of your VM.
The other option I know of with VMWare and possibly VirtulBox is to create a scheduled tasks that automated the snapshot and reversion with command lines (PowerCLI).  Then it will create the snapshot and revert back on a schedule.
Also, if you are having to snapshot and roll back that frequently then may something is wrong in your process.  Maybe explaining exactly why you are doing it would garner better responses.
Edit
I think the better approach to this may be to simply have multiple VM running, all with the same baseline.  Basically clone the first one to 3 or 4 additional VMs and use them for your testing.  What this will do is as soon as the original set of testing is done on VM1, start the testing on VM2.  Roll back VM1 with the snapshot while this is going on.  Then once VM2 is done move to VM3 while rolling back VM2.  Then go back to VM1 when VM3 is done and roll back VM3.  If by the time you finish with VM3 that VM1 has not completed its roll back, add an additional VM.  That way when you reach the end of the "line" so to speak, VM1 will be done with the roll back.  Since it is automated you can use PowerCLI or some sort of command line to revert the snapshot upon completion of the test.
Comment: See above. This is absolutely an unusual use case. (I plan to use this for returning a system into a sane state so I can automatically create good system tests for it.) It could be that rollback needs to be 20-30s expensive, but if so, I'd like to understand why. (I only see throwing away differential I/O plus returning parts of the memory to a known state. Shouldn't that be much cheaper, if we can avoid touching the disk as much as possible?)
Comment: I don't think what you are asking for is possible.  The restoration process of the snapshot can depend on too many factors.  I will edit my answer to explain an another alternative.

Very fast virtual machine rollback


How to generate stats or reports for multi-sites?

Question: We've a large deployment of multi-sites, each site with its own database, and I am looking at generating reports from these sites and making those available online in some format.
For e.g., our sites implement many redirects. I want to be able to pull a report on what redirects are in use by which site. I can readily do this via drush and shell scripts, but it would be nice to have a "stats site" to do the generating of these kinds of reports for all sites in a multi-site setup.
Answer: Here's how I do this:
I created a "meta" site and database in the system where I added a new table called "databases".  In that table I store the access info of all the DBs in the system.
Then loading an array from a query to that table, query the other DBs individually using the methods outlined here:
http://drupal.org/node/18429
Answer: The Forena module seems to me like a pretty perfect fit to address this requirement. No PHP coding involved, just add your custom SQL queries to fit your requirements for getting the data you need, and then use your XHTML / CSS skills to beef up your output (report). Eventually adding some charts to it also, which even support drill-down links.
Here is the intro near the top of that linked page (use the link for more details, and its child pages):

Forena Reports is built of the idea of using SQL to get data out of a
  database and use XHTML and CSS to format it into web reports.
It is designed to leverage existing knowledge of HTML, CSS, SQL and
  JavaScript to help you create rich interactive web reports.

Its Supported database connections you'd want to take advantage of in this case would be either "Drupal" or "PDO" ... done! Don't believe me (yet), just try it in your own environment).
Be aware: I'm a co-maintainer of forena, and created most of the community docu for it.