When using :=, why is with=TRUE the default?

Question: In data.table, by default with = TRUE and j is evaluated within the frame of x. It helps then to use the column names as variables. And when with = FALSE, j is a vector of names or positions to select.
I managed to find some examples of with = FALSE.
set.seed(1234)
DT <- data.table(x=rep(c(1,2,3),each=4), y=c("A","B"), v=sample(1:100,12))

## The askers's solution
#first step is to create cumsum columns
colNames <- c("x","v"); newColNames <- paste0("SUM.",colNames)
DT[, newColNames := lapply(.SD,cumsum) ,by=y, .SDcols = colNames, with=FALSE];
test <- DT[, newColNames:=lapply(.SD,cumsum) ,by=y, .SDcols=colNames, with=TRUE];

We can check that DT is:
> DT                       # setting `with=FALSE` - what I require
    x y  v SUM.x SUM.v
 1: 1 A 12     1    12
 2: 1 B 62     1    62
 3: 1 A 60     2    72
 4: 1 B 61     2   123
 5: 2 A 83     4   155
 6: 2 B 97     4   220
 7: 2 A  1     6   156
 8: 2 B 22     6   242
 9: 3 A 99     9   255
10: 3 B 47     9   289
11: 3 A 63    12   318
12: 3 B 49    12   338

and test is:
> test                     # this is when setting " with = TRUE"
    x y  v newColNames
 1: 1 A 12           1
 2: 1 B 62           1
 3: 1 A 60           2
 4: 1 B 61           2
 5: 2 A 83           4
 6: 2 B 97           4
 7: 2 A  1           6
 8: 2 B 22           6
 9: 3 A 99           9
10: 3 B 47           9
11: 3 A 63          12
12: 3 B 49          12

I don't understand why the result is this when setting with = TRUE. So my question basically is when is with = TRUE useful?
I don't get the point why default setting is with = TRUE, though there must be a good reason for it.
Many thanks!
Comment: Bigchao, you ask nice questions, but you don't put much effort in formatting or writing sentences clearly (with proper grammar). It's off-putting if it happens every time. Please pay attention to your question. Take your time and re-read before posting. Learn the formatting options S.O. has in store and use them.
Comment: `with=TRUE` is nice so you don't have to quote names.  The code: `DT[, x.sum:=cumsum(x)]` would not be possible otherwise.
Comment: @Arun Sure, I'll pay more attention for the formatting. Thanks for your suggestions.!
Answer: I see your point. We've moved away from using with=TRUE|FALSE in combination with :=.  Since it isn't implicitly clear whether with=TRUE refers to the left had side or the right hand side of :=.  Instead, wrapping the LHS of := with brackets is now preferred.
DT[, x.sum:=cumsum(x)]     # assign cumsum(x) to the column called "x.sum"
DT[, (target):=cumsum(x)]  # assign to the name contained in target's value

As Justin alluded to, most of the time we assign to a new or existing column that we know up front. In other words, most commonly, the column being assigned to isn't held in a variable. We do that a lot so that needs to be convenient. That said, data.table is flexible and allows you to define the target column name programatically, too.
I suppose a case could be made that it should be :
DT[, "x.sum":=cumsum(x)]   # assign cumsum(x) to the column called "x.sum"
DT[, x.sum:=cumsum(x)]     # assign to the name contained in x.sum's contents.

However, since := is an assignment operator and j is evaluated within the scope of DT, to me, it would be confusing if DT[, x.sum:=cumsum(x)] didn't assign to the x.sum column.
Explicit brackets, i.e. (target):=, implies some sort of evaluation, so that syntax is clearer. In my mind anyway.  Of course, you can call paste0 etc directly in the left hand side of := too without needing with=FALSE; e.g.,
DT[, paste0("SUM.",colNames) := lapply(.SD, ...), by=...]

In short, I never use with when I'm using :=.


object of NSOperation doesn't removed from NSOperationQueue after executing

Question: In my subclass of NSOperation I set 4 flags, and when an operation finishes  its execution it is not removed to NSOperation queue, where it was added at the beginning, this thing cause a lot of issues in my app.
I suppose that the way I set these flags is not correct, could you please help with it. cause I really spend a lot of time on identifying this issue.
<code>@property(assign, nonatomic) BOOL isCancelled;
@property(nonatomic, getter=isExecuting) BOOL executing;
@property(nonatomic, getter=isFinished) BOOL finished;
@property(readonly, getter=isAsynchronous) BOOL asynchronous;

//in initialisation
- (id)initWithURL:(NSURL*)url andRaw:(NSInteger)row
{
    if (![super init])
        return nil;
    [self setTargetURL:url];

    return self;
}
//the way I override KVO
- (BOOL)isExecuting
{
     NSLog(@"Exec");
    return (self.defaultSession != nil);//it doesn't work
}

- (BOOL)isFinished
{
    NSLog(@"Finished");
    return (self.defaultSession == nil); //it doesn't work, so I explicitly set the value
}

- (BOOL)isAsynchronous
{
    return YES;
}

- (void)cancel
{
    [super cancel];
    [self willChangeValueForKey:@"isExecuting"];
    [self willChangeValueForKey:@"isFinished"];
    self.isExecuting = NO;
    self.isFinished  = YES;
    [self didChangeValueForKey:@"isFinished"];
    [self didChangeValueForKey:@"isExecuting"];

    if(self.downloadTask.state == NSURLSessionTaskStateRunning)
        [self.downloadTask cancel];
    [self finish];
 }
- (void)finish
{
    [self willChangeValueForKey:@"isExecuting"];
    [self willChangeValueForKey:@"isFinished"];
    self.defaultSession = nil; //NSURLSession
    self.isFinished  = YES;
    [self didChangeValueForKey:@"isFinished"];
    [self didChangeValueForKey:@"isExecuting"];
 }
</code>
Thank you in advance
EDIT:
finally I found the issue - it was NSURLSession inside the queue. It kept strong reference to the queue and didn't allow it to be deallocated and removed from NSOperationQueue.
Answer: I have done the exact same thing, albiet in Swift.
There are a couple of aspects that I have implemented differently and are listed below:

I have noticed that we do not have to override cancel() method in the
Asynchronous operation. The default behavior of NSOperation's cancel
method is to set the self.cancelled boolean to true.
self.executing should be set to true only inside the overriden start() method of operation, but not in the init ( Not sure if this wil cause any issues). Also before setting the self.executing = true in the start() method, I ensure that the boolean self.cancelled is false. If self.cancelled = true, we should set self.finished = true.
Also , I have created a "didSet" property observer for the isExecuting and isFinished properties where I call the willChangeValueForKey and didChangeValueForKey. I am not 100% sure how to replicate the didSet behavior in Obj C. ( This says to overrride setter)

Please refer to the Ray Wenderlich video tutorials about "Concurrency" where Sam DAvies explains the creation of NSoperation subclass for Asynchronous operation. Please note, that it is only for subscribers and it is explained in Swift. I believe if you fix points 1 and 2, you should see your issues being fixed.
Comment: Thank you, I will try to follow your instructions, and unfortunately I am not a subscriber of Ray Wenderlich(.
Comment: I would highly recommended to become one, if you can. They have some amazing tutorials. https://www.raywenderlich.com/video-tutorials


Usernames in git commit messages on Drupal.org

Question: I recently changed my username on drupal.org, and as a result is now different from my git username.
When you attribute a patch someone else using the commit author, you use the anonymous email address, which is based off of the git username (which also shows up in that user's profile page.
When you can commit directly to a project, there is a standard format for commit messages:

Issue #[issue number] by [comma-separated usernames]: [Short summary of the change].

Are these supposed to be the git username or the drupal.org username?
Answer: The usernames are the usernames for the Drupal.org accounts, not the git usernames.
The reason is simple: Everybody can provide a patch, even the users who don't have a git account. As far as I recall, everybody can have read access user to a Drupal.org git repository. You can create a patch that solves an issue; once the issue has been resolved, or the feature request implemented, the git commit will report the usernames of those who contributed with a patch.
The other reason for using the Drupal.org usernames is that not every users see somebody else's git username, and it is more practical to use what is visible in the issue where the patch is being discussed.


Who implemented the first infield shift?

Question: Today many MLB teams utilize numerous types of defensive alignments depending on the batter in order to gain a defensive advantage.
Which MLB manager was the first to implement an infield shift for his defense in order to gain said advantage?
Comment: @waxeagle I am sorry it has been a long day for you. However, I disagree with your assessment of my questions being "simple looking and low effort" As for deserving a second look perhaps you should aid in the process of editing the question if you feel my questions aren't appropriate rather than downvoting and later admitting to being "grumpy". Not exactly a pleasant welcome to sports.stackexchange.com.
Comment: Oh yeah, I forgot. Welcome to Sports.SE! I try really hard to encourage *good* trivia questions here, a lot of what's been posted so far is really bad, honestly if a single link to wikipedia answers it, its not worth our time. I don't blame you, I blame the examples that you looked at. Sorry for being unpleasant to you. While I could edit your questions into something good, I'd much rather help users figure out what good questions look like. Teach a man to fish and all that.
Comment: @waxeagle As noted by my comments, the answer to this question has yet to be provided. Even the inclusion of JacobG's supplemental info I do not see the **name** of a specific manager who implemented the **infield shift** Perhaps you may be able to help identify this manager.
Comment: That does make it significantly better.
Comment: @waxeagle Your highest rated question (How is offside determined in Soccer?) can be answered directly from wikipedia
Comment: Perhaps, an edit to the question would be appropriate? rather than a quick downvote prior to reading all the comments?
Comment: Long day and I'm a bit grumpy, you posted 3 simple looking low effort questions in a row. This is the only one that's even worth a second look.
Comment: @waxeagle do you have an answer to the question?
Comment: @waxeagle Ironically when reviewing questions previously posted on this site, I read a number of your questions including your highest rated question (How is offside determined in Soccer?) Based on this reading I felt this question was appropriate. As for this question neither the wiki nor additional post nor your comments have answered it as of yet. I am not trying to be a pain or just gain points. I love watching sports with my family and my daughter. I am unsure why you continue with negativity. If I can improve the questions with your aid I would be glad to.
Comment: Not trying to be negative, sorry if I'm coming off that way. My intent is to encourage good content. I've done some preliminary research on this question and do intend to answer if I find something new. Definitely interesting.
Answer: Wikipedia says that this first happened in the 1920's, but apparently became most famous when employed against Ted Williams by Eddie Dyer.
Minor supplemental info
From here:

"The only year that [Cy] Williams didn't bat .300 over that span was 1923,
  when he batted .293 and blasted 41 round-trippers to take over the NL
  career home-run record from Roger Connor. National League managers
  recognized that the best defense was to play him extremely deep and
  around toward right field"

It seems like "infield shift" would be one of those things logged as an item of interest in a sports article, but I can't find anything definitive.
Comment: Thanks for the link. Unfortunately, I am already aware of Eddie Dyer the St. Louis Manager and Lou Boudreau the Cleveland Manager (for whom the shift is named after) This being said the same article discussed the first implementation of the shift came against Cy Williams in the 1920's. What manager was it that implemented this 1920's infield shift against Cy Williams?
Comment: You mean Cy Williams, right? It's a little unclear that the quote discusses Cy and not Ted. ;)
Comment: Edited to clarify... I was so immersed in reading, I forgot to note the distinction.
Comment: This looks to be about the best answer available on the web. Unfortunately record keeping in the 20s just wasn't very good. It'd be awesome if there was some old Radio archives available to see if we could find it, but that might be the only way to know. It seems pretty certain that there was a shift for Cy, and that several teams used it, but the details are very thin from what I've been able to find.
Comment: wow, more research, even radio would probably be poor as radio broadcasts from that era were recreations from wire reports, which likely wouldn't have positioning information. looks like first hand accounts will be the only way. Now to find some.


Python split function shows some wrong result

Question: currently I am developing a program which helps user to adjust the significant figures. Hereby, I used following codes to split decimal numbers by [integer, ., decimal] format using split('.')
len_number1 = (str(number1)).split('.')

when I enter the value of 1.3333 it successfully separate numbers into ['1', '.', '3333'] but when I enter the value of 0.000043 it shows the result of ['4', '3e-05'] whereby I need to get the result of ['0', '.', '000043']
Is there any ways to solve such problem?
Comment: What's the `number1`?if `number1` is `1.3333`,it will be `['1', '3333']
` instead of `['1', '.', '3333']` when you use `.split(".")`
Comment: My mistake, Sorry
Answer: First, when you do str(0.43).split('.') you will get ['0','43'] not ['0','.', '43']. The problem is the casting function str(), because the string format of these decimal numbers, as @MarkMeyer said, is in a scientific notation, that means str(0.000043) will return you '4.3e-05'.
Even print(0.000043) will give you something like 4.3e-05 too. You can try something like this to replace that function:
import decimal

dec = decimal.Context()

dec.prec = 20  #this gives us a precition of 20 decimals

def num_to_str(n):

    dec1 = dec.create_decimal(repr(n))
    return format(dec1, 'n')

print(num_to_str(0.0000043).split('.'))
>>>['0','0000043']

If you want to get ['0','.', '0000043'], you could use insert(), since you will only add one '.' at index 1. You could try
num = num_to_str(0.0000043).split('.')
num.insert(1,'.')
print(num)
>>>['0','.', '0000043']

Checkout the documentation of decimal, repr() and split() for more info.
Comment: is it impossible to make 0.000043 into ['4', '000043'] using split?
Comment: The problem isn't `split()` @JungJaeWon. It's that you need to turn the number into a string before calling split and `str(0.000043)` is `'4.3e-05'`
Comment: Yeah that´s right. So here is a trick to do this `str()` casting function for get the result you are looking for.
Comment: I thought making number1 as str(number1) will make number1 into string format. it isn't work in such way is it?
Comment: @JungJaeWon see the answer [here](https://stackoverflow.com/questions/38847690/convert-float-to-string-in-positional-format-without-scientific-notation-and-fa).
Comment: Yeah that's right, `str(n)` will convert `n` into a string. The thing is that the string format of these decimal numbers, as @MarkMeyer said, is in a scientific notation, that means `str(0.000043)` will return you `'4.3e-05'`. You can check this and another different  string casting functions in these [link](https://stackoverflow.com/questions/38847690/convert-float-to-string-in-positional-format-without-scientific-notation-and-fa).


Como usar JProgressBar em classes de comparação de ArrayList?

Question: Tenho uma classe que compara dois ArrayList, um com informações adicionadas pelo usuário e outro com consulta do banco de dados.
Depois que a consulta é feita, mando este objeto para outra classe que realiza a escrita desses dados em um arquivo. Gostaria que, enquanto estas verificações fossem feitas, mostrasse um JProgressBar pra o usuário não ter a impressão de que o sistema está travado.
public ArrayList<ModeloCadastroTerceiro> verificaCodigoTerceiro(ArrayList<ModeloCadastroTerceiro> modelo) {

    ConectaBanco conecta = new ConectaBanco();

  conecta.conexao();
  StringBuilder str_busca = new StringBuilder();

    ArrayList<ModeloCadastroTerceiro> modeloRetorno = new ArrayList<>();

    ModeloCadastroTerceiro mod;

    str_busca.append('(');
    for (ModeloCadastroTerceiro modelo1 : modelo) {
        str_busca.append("'");
        str_busca.append(modelo1.getCpfTerceiro());
        str_busca.append("'");
        str_busca.append(',');
    }

    str_busca.deleteCharAt((str_busca.toString().length() - 1));

    str_busca.append(')');

    String busca = String.valueOf(str_busca);

    conecta.executaSQL("select * from cadastro_terceiros where cpf_cnpj_terceiro in" + busca);

    try {
        conecta.rs.first();

        do {
            mod = new ModeloCadastroTerceiro();
            mod.setCodTerceiro(conecta.rs.getInt("id_prosoft_terceiro"));
            mod.setCpfTerceiro(conecta.rs.getString("cpf_cnpj_terceiro"));
            modeloRetorno.add(mod);
        } while (conecta.rs.next());

    } catch (SQLException ex) {
        if (ex.getErrorCode() == 0) {

        } else {
            Logger.getLogger(RecuperaInformacaoBanco.class.getName()).log(Level.SEVERE, null, ex);
        }

    }

   ArrayList<ModeloCadastroTerceiro> fim = new ArrayList<>();
    ModeloCadastroTerceiro modFim;

    for (ModeloCadastroTerceiro mod1 : modelo) {

        boolean flag = false;//usada para marcar terceiro não encontrado

        for (ModeloCadastroTerceiro te1 : modeloRetorno) {

            if (mod1.getCpfTerceiro().equals(te1.getCpfTerceiro())) {
                flag = true;
            }
        }

        if (flag == false) {
            modFim = new ModeloCadastroTerceiro();
            modFim.setCodTerceiro(mod1.getCodTerceiro());
            modFim.setNomeTerceiro(mod1.getNomeTerceiro());
            modFim.setCpfTerceiro(mod1.getCpfTerceiro());
            fim.add(modFim);
        }else{
            System.out.println("TERCEIRO JA CADASTRADO... NÃO SERA IMPORTADO");
        }

    }
    return fim;
}

Depois disso eu chamo a classe que escreve no arquivo a partir de um ArrayList
public void escreverNoArquivo(File arquivo, ArrayList<String> texto) {

    FileWriter fw;
    BufferedWriter bw;

    try {

        fw = new FileWriter(arquivo);
        bw = new BufferedWriter(fw);

        // Recupera cada linha do texto e escreve no arquivo
        for (int i = 0; i < texto.size(); i++) {

            // Escreve o texto no arquivo
            bw.write(texto.get(i));
            // Quebra de linha
            bw.newLine();

        }

        bw.close();

    } catch (IOException e) {
        e.printStackTrace();
    }

}

Como fazer um JProgressBar que mostre o andamento disso?
Comment: A progress vai aparecer quando?
Comment: A ideia é que aparece quando estiver escrevendo no arquivo... Realizando o processo pra gera-lo do inicio ao fim.
Comment: Então a progressbar deve seguir o andamento do método `escreverNoArquivo`?
Comment: Sim isso mesmo.
Answer: Você pode fazer uso da classe SwingWorker, colocando nela a execução do método em paralelo a execução da tela.
public void escreverNoArquivo(File arquivo, ArrayList<String> texto) {

    SwingWorker worker = new SwingWorker<Void, Integer>() {

        //executará a ação em uma thread paralela a EDT
        @Override
        protected Void doInBackground() throws Exception {
            FileWriter fw;
            BufferedWriter bw;
            fw = new FileWriter(arquivo);
            bw = new BufferedWriter(fw);

            // Recupera cada linha do texto e escreve no arquivo
            for (int i = 0; i < texto.size(); i++) {
                // Escreve o texto no arquivo
                bw.write(texto.get(i));
                //publica a atualização para a progressbar
                // através do método process(List<type> chunks)
                publish(i);
                // Quebra de linha
                bw.newLine();
            }
            bw.close();
            return null;
        }

        //carrega a progressbar a medida que o publish
        //envia o andamento do doInBackground
        @Override
        protected void process(List<Integer> chunks) {
            int progress = chunks.get(chunks.size() - 1);
            suaProgressBar.setValue(progress);
        }

        @Override
        protected void done() {
            try {
                //captura o retorno do doInBackground
                //mas neste caso, apenas retorna se houver alguma
                //exceção lançado na execução do doInBackground
                get();
                //reseta o progresso, já que a execução terminou
                progressBar.setValue(0);
                JOptionPane.showMessageDialog(progressBar.getParent(),"Terminou!");
            } catch (InterruptedException | ExecutionException ex) {
                ex.printStackTrace();
            }
        }
    };
    //executa a ação anterior.
    worker.execute();
}

Obs.: Como se trata de uma execução em classe anônima, suaProgressBar deve ser uma variável da sua classe de tela, ou caso seja local, deverá ser final, senão não será possível acessá-la dentro do swingWorker, da forma como foi apresentado.

Caso se interesse em saber mais sobre o funcionamento e uso do SwingWorker, eis alguns links que podem ajudar:

Worker Threads and SwingWorker(Oracle)
Passar a execução de um determinado método para o SwingWorker como argumento
Métodos que exigem retorno mesmo tendo Void na "assinatura"
Capturar exceções na execução do swingworker
Passagem de argumentos opcionais em Java
Comment: Cara 2 vezes no mesmo dia? rsrs Obrigado mesmo ajudou novamente e muito!
Comment: Cara sei que já faz  muito tempo, mas só consegui testar agora. Estou com uma dúvida no método protected void process(List chunks) {
            int progress = chunks.get(chunks.size() - 1);
            suaProgressBar.setValue(progress);
        }, não é chamado momento algum e esta dando erro de "type List does not take parameters". sabe me dizer o que pode ser?
Comment: @Valdecir Quem chama o método process é o método publish(), veja que inseri ele dentro do laço. Agora quanto ao erro, veja se você não importou erroneamente a classe `java.awt.List` ao invés de `java.util.List`.


Show Magento custom (yes/no) Attribute value in success.phtml

Question: In a Magento webstore I am creating people can order PDF products which they will receive by email. An external system will handle these emails, this isn't part of the Magento store. On the success.phtml page I'd like to add a message for the customer if he/she has ordered a PDF product saying something like "thanks for ordering, you should get the product in your inbox soon".
I'm using the following code to get info on the order:
$order_id = Mage::getSingleton('checkout/session')->getLastRealOrderId();
$order = Mage::getModel('sales/order')->loadByIncrementId($order_id);

$items = $order->getAllVisibleItems();
$itemcount=count($items);

foreach ($items as $itemId => $item)
{
    echo 'itemcount: '.$itemcount.'
';
    echo 'itemname: '.$item->getName().'
';
    echo 'itemprice: '.$item->getPrice().'
';
    echo 'itemsku: '.$item->getSku().'
';
    echo 'id: '.$item->getProductId().'
';
    echo 'qty: '.$item->getQtyToInvoice().'
';
    echo 'pdf: '.$item->getData('pdf_ticket');
}

Everything gets put on the success page, except for my custom attribute pdf_ticket. I've also tried getPdfTicket(). The attribute is a Yes/No attribute. After this works I should be able to do something like if($pdf == 'Yes'){ echo 'thanks!...' }
Answer: I've found an answer that worked for me by Thomas Nielsen. Question >
$custom = Mage::getModel('catalog/product')->load($item->getProductId());
echo $custom->getAttributeText('pdf_ticket');

I'm not really sure WHY this works. So if anyone could clarify this for me, it would be great!


Identifying a (psalm?) tune from its notes

Question: I recently discovered four lines of numbers painted on the inside wall of a church bell tower in Wales, and I'm trying to interpret them.  From the style of the writing, degree of decay and location, I believe the writing is most likely eighteenth or nineteenth century in origin.
I've thoroughly investigated the possibility that they may be related to (largely English) art of change ringing, and have rejected this possibility for several reasons.  My leading theory is that they represent a melody that was played on the bells.  Until recently the church had five bells in a major scale tuned to G.  From the other writing on the wall, I know the numbers referred to notes as follows: 5=G, 4=A, 3=B, 2=C, 1=D.  Translated to notes, the four lines read as follow:

GACBDACG / GACBDC / GACBDACG / DCBACG

I don't recognise this tune.  However, lines of 8, 6, 8, 6 notes suggest common metre, and the fact that most lines start and finish on the tonic seems to support its interpretation as a melody.  I think the title above the numbers says 'The IV PSALM' (the 'P' is totally illegible, and several other letters are difficult to read, but the 'IV' is clear and not space for there to be a missing letter before the 'IV').  Psalm 4 is not one I know well, but its words (at least in the King James Version) don't easily fit a common metre.  There is no indication of the rhythm in the numbers on the wall.
Obviously it would be fantastic if someone here could identify the tune, though I'm not necessarily expecting that.  Failing that, is there a good strategy for looking up a song or hymn from its notes?  Alternatively, is there a way of finding settings of Psalm 4, especially those in common metre?
Comment: Did you try a web search for melody or music lookup?
Comment: Yes.  I found a few sites, but searching them either found nothing or found lots of tunes that bore no resemblance that I could see to the phrase I was searching for.  Quite possibly I wasn't using them right.
Comment: Another possibility is that the melody has never been cataloged.
Comment: That's very possible.  I can well believe an obscure Victorian setting of Psalm 4 that's no longer commonly used today might well not have been catalogued.
Comment: "lines of 8, 6, 8, 6 notes suggest common metre": but the fourth line has seven notes.
Comment: @phoog Thanks for spotting that.  Typo corrected.
Comment: Played around with this a bit. It's nothing resembling an answer, but I'll leave the result in the [Auditorium](https://chat.stackexchange.com/rooms/126233/the-auditorium) if you want to see (not hear unless you tell me how to upload a sound file).
Answer: A friend who is a cathedral lay clerk has recognised this tune as a version of the old Scottish tune 'York' (also called 'Stilt'), slightly adapted only to need five notes.  It uses common metre.  I'm not quite sure how old that tune is, but it's certainly old enough that it could written on this church wall.
I've also found a seventeenth century book, A New Version of the Psalms of David, which includes a version of Psalm 4 in common metre.  There's no indication of the intended tune, however 'York' is supplied as a tune for Psalm 23 ('The Lord's My Shepherd', though these words do not appear in its version, adapted to use common metre).
I'm not sure whether using 'York' for Psalm 4 was a local peculiarity to this particular Welsh village, or whether it was a wider trend, but either way, I think the mystery is solved.  Thanks for everyone's input!


Markov chain for a board game with dice rolls

Question: I was hoping for some help with this question:
Using a fair 6 sided dice
If you are given a board which goes in circles, it has 20 steps and you have a starting position. Each roll of the die moves you the amount of positions shown on the die forward, but as it is a circle if you near the end and you roll past the 20th position then you would just be back at the start, but you overshoot based on your roll.
So for example you are on the 18th step and you roll a 4, you would be on the 2nd square again.
How would you calculate the exact probability of landing on the say 15th square after you have already done a four full laps of the entire board? Not too sure how to use markov chains in this context especially since if you roll a number near the end of the board you may not land exactly on go and you may overshoot it.
Comment: **Hint:** each square of the board is your state. Can you write down the state transition matrix $A$ and consider $A^4$ (4 moves)?
Comment: What exactly do you mean by "after you have already done a four full laps of the entire board"? Are you asking for the probability that you first circle the board in one direction 4 times, and then land on the 15th square?
Comment: I would unroll the 4 circles and sum the probabilities of landing on steps 15, 35, 55 and 75.
Answer: Straighten the board out into a long straight track, so we are asking what is the probability of eventually landing on square $95$.  This can only happen if we land on square $94$ and then roll a $1$ or we land on square $93$ and then roll a $2$, and so on.  Define
$$p_n=\begin{cases}0,&n<0\\1,&n=0\\\frac16\sum_{k=1}^6p_{n-k},&n>0
\end{cases}$$
Then $p_n$ gives the probability of landing on square $n$ for $n>0$.
I wrote a python script to evaluate this and got $$p_{95}=0.2857142857142553$$
I've been playing with this for a couple of days, and after an embarrassingly long time, I've solved the general problem.
A point moves on the real line according to the rolls of a fair $s$-sided die.  It starts at the origin at time $t=0$.  Thereafter, if it is at $x=k$ at time $t$, it moves to $x=k+r$ with probability $1/s$ for $r=1,2,\dots,s$.  Let $p_n$ be the probability that the point lands at $x=n$ at some point in time.
I claim that $$\lim_{n\to\infty}p_n=\frac2{s+1}.\tag1$$  It's clear that $p_n$ is the solution to the linear difference equation $$p_n=\frac1s\sum_{k=1}^sp_{n-k}\tag2$$ with initial conditions $$p_n=0 \text{ for } n<0,\text{ and }p_0=1.$$
The characteristic function of $(2)$ is $$f(x)=x^s-\frac{x^{s-1}}s-\cdots-\frac xs-\frac1s$$  Let the roots of $f$ be $\theta_1=1,\theta_2,\dots,\theta_s$.  I will show that $|\theta_k|<1$ for $1<k\leq s$, and that the solution to $(2)$ is $$p_n=\frac2{s+1}+\frac1{s+1}\sum_{k=2}^s\theta_k^n,\ n\geq0.\tag3$$  Of course, $(1)$ follows immediately.
Define $\pi_n=\sum_{k=1}^s\theta^k,\ n\geq0$, so that $(4)$ becomes $$p_n=\frac1{s+1}+\frac{\pi_n}{s+1},\ \ n\geq0\tag4$$
For $k=1,\dots,s$ let $e_k$ be the $k$th elementary symmetric function of the $\theta_k$.  Vieta's formulas give
$$e_k=\frac{(-1)^{k+1}}s$$ and Newton's identities give $$
\begin{align}
\pi_n&=(-1)^{n-1}ne_n+\sum_{k=1}^{n-1}e_{n-k}\pi_k\\
&=\frac ns+\frac1s\sum_{k=1}^{n-1}\pi_k,\ 1\leq n\leq s\\\tag5
\end{align}$$
and
$$\begin{align}
\pi_n&=\sum_{k=n-s}^{n-1}(-1)^{n-1+k}e_{n-k}\pi_k\\
&=\frac1s\sum_{k=1}^s\pi_{n-k},\ n\geq s\\\tag6
\end{align}$$
Now it's easy to show that the $p_n$ defined in $(4)$ satisfy the difference equation.  When $1\leq n\leq s$, $(5)$ gives
$$\begin{align}
p_n&=\frac1{s+1}+\frac1{s+1}\pi_n\\
&=\frac1{s+1}+\frac1{s+1}\left(\frac ns+\frac1s\sum_{k=1}^{n-1}\pi_k\right)\\
&=\frac1{s+1}+\frac n{s+1}+\frac1{s(s+1)}\sum_{k=1}^{n-1}((s+1)p_k-1)\\
&=\frac1{s+1}+\frac n{s+1}-\frac{n-1}{s+1}+\frac1s\sum_{k=1}^{n-1}p_k\\
&=\frac1s+\frac1s\sum_{k=1}^{n-1}p_k
=\frac1s\sum_{k=0}^{n-1}p_k=\frac1s\sum_{k=1}^sp_{n-k},
\end{align}$$
When $n>s$, $(6)$ gives
$$\begin{align}
p_n&=\frac1{s+1}(1+\pi_n)\\
&=\frac1{s+1}+\frac1{s+1}\left(\frac1s\sum_{k=1}^s\pi_{n-k}\right)\\
&=\frac1{s+1}+\frac1{s(s+1)}\left(\sum_{k=1}^s((s+1)p_{n-k}-1)\right)\\
&=\frac1{s+1}-\frac s{s(s+1)}+\frac1s\sum_{k=1}^sp_{n-k}=\frac1s\sum_{k=1}^sp_{n-k}
\end{align}$$
It remains to show $|\theta_k|<$ when $k\neq1$. We have $$
\begin{align}
sf(x) &=sx^s-x^{s-1}-\cdots-x-1\\&=(x-1)\sum_{k=0}^{s-1}(k+1)x^k\\&:=(x-1)g(x)\end{align}$$ and
$$\begin{align}
g(x)&=\frac d{dx}(x^s+x^{s-1}+\cdots+x+1)\\&=\frac d{dx}\frac{x^s-1}{x-1}\\&:=\frac d{dx}G(x)
\end{align}$$
The roots of $G$ are $s-1$ of the $s$th roots of unity, and by the Gauss-Lucas theorem the roots of $g$ lie in their convex hull.  Note that a root of $g$ can only have modulus $1$ if it coincides with a zero of $G$, and we know that $G$ has only simple roots.  $\square$
Answer: Let's say that we want the probability of doing exactly 4 laps and then landing on the 15th square after $n$ rolls of the die.
Note that the number of steps you have moved forward is simply the total of all $n$ die rolls. In other words, if $X_i$ denotes the result of the $i$th roll of the die, then we wan the probability that
$$
X_1 + X_2 + \cdots + X_n = 4 \cdot 20 + 14 = 94.
$$
Each $X_i$ is an i.i.d. uniform variable over $\{1,2,\dots,6\}$.  If we write $Y = X_1 + X_2 + \cdots + X_n$, where looking for the probability that $Y = y = 94$. If you're interested in an exact formula for this probability, see this paper, for instance.
For large $n$, however, this can be nicely approximated using the central limit theorem. In particular, each $X_i$ has mean $7/2$ and variance $\frac{6^2 - 1}{12} = \frac{35}{12}$.  With that established, the probability that $Y = y$ is approximately equal to
$$
P \approx \Pr \left(\frac{y - \frac 12 - \frac{7n}{2}}{\sqrt{\frac{35n}{12}}} \leq Z \leq \frac{y - \frac 12 - \frac{7n}{2}}{\sqrt{\frac{35n}{12}}}\right),
$$
where $Z$ is normally distributed with mean $0$ and standard deviation $1$.


Unable to delete a folder from server 2012 R2

Question: I had a replication enable a folder which was replicating data to a server 2012 r2 but i have disbale the replication but i'm unable to delete this folder or re-enable the replication.
We had a probelm the dfs replication on a folder which we disble it to remove any data from other site but now when I re-enable to dfs i get error "server(folder): Security cannot be set on the replicated folder. access is denined"
When I try to delete the folder from server 2012 R2 i get "You required permission from admin to mkae changes to this folder"
Please find the images
https://1drv.ms/f/s!At0UGnArT7e5nDVt3MPjYJvv3dZ1
UPDATE:
After delete the membership i'm now able to delete the data but i got new problem now. I cant delete the connection. When I delete the connection i get following error "Server to mainserver: The dfs replication connection object cannot be deleted. the user has insuffuient access reights."
Comment: "It doesn't work" is not a helpful description of a problem. More details, please. Your question is unanswerable because you don't say what happens when you try to delete it or re-enable replication.
Comment: Explain this replication. Replicating how? With what?
Answer: You don't have Permissions into the folder.
Make sure in the security tab that your user has permissions.

And also check the Shared tab permissions and make sure that your user is listed.
Comment: I do have full admin rights and also I cant make changes to any security.
Comment: Take ownership of the folder. and then delete it
https://technet.microsoft.com/en-us/library/cc753659%28v=ws.11%29.aspx?f=255&MSPPError=-2147217396
Comment: I have try it and still no luck
Comment: i have upload new image to show that I cant apply new permission or make changes to current permission. https://1drv.ms/i/s!At0UGnArT7e5nDooKig2kpn5wKon
Comment: as you can see i have full control. https://1drv.ms/i/s!At0UGnArT7e5nDtyMAPpXSsBkXRy
Comment: I see, in this case, you are getting the permissions from a parent folder (permissions are being taken using the inherence) So check the permissions on the parent folder, and also check is inherence is enable or disable.
Comment: the image is for parent folder. I'm trring to delete the parent folder which i cant because it saying i dont have permission.
Comment: this is only happening on server 2012 r2 server but 2008 r2 its all ok.
Comment: Where is located the "parent folder" E:\Folder\x86 what I mean is look for the permissions in "folder" and if it's not there you need to look in the Drive E: permissions from My Pc.
Comment: please see my update
Comment: Check permissions in the delegates:

https://social.technet.microsoft.com/Forums/windowsserver/en-US/893419c6-a9b5-4b1d-a7b4-377607426b1b/unable-to-delete-dfs-replication-group?forum=winserverfiles (Use MedicalS CSG) answer a bit lower from the middle of the page
Answer: Maybe is still in use by someone.
Try to disconnect all clients (reboot machine etc..) and set owner and permissions as described in other answers.
Comment: restart the server and also disable to dfs service
Comment: You deleted all data from other side and now you cannot start service because it can't set permission on other side. Try to restore data, enable service, set permissions on both and then try to update folders
Comment: the problme i cant delete the data, I cant do anything with replicated folder on server 2012 r2
Answer: After deleteing the membership from dfs management i was able to delete the files from server 2012 r2
Comment: Wait...so were you trying to delete the DFSRPrivate folder prior to removing the server from the DFS rep group membership?


Diagnolization of a non-invertible matrix?

Question: Let $A$ be a $n \times n$ matrix. Let $\mathcal{B}$ be a basis for the subspace formed by the columns of $A$. Can there exist a diagonal matrix $C$ such that:
$$Cx_{\mathcal{B}} = (Ax)_{\mathcal{B}}$$
if $\text{span}(A) $ doesn't equal $\mathbb{R^n}$ and
a) $\mathcal{B}$ consists only of the columns of $A$
b) $\mathcal{B}$ doesn't consist only of the columns of $A$
This isn't homework, just curiosity. Thanks!
Comment: What do the symbols $Cx_{\mathcal{B}}$ and $(Ax)_{\mathcal{B}}$ mean?
Comment: I guess $(\cdot)_{\mathcal{B}}$ means the coordinates of $\cdot$ w.r.t. the basis $\mathcal{B}$
Answer: It is not possible to diagonalize a non-zero , non-invertible matrix,  that is not already in a diagonal form with $0$'s on the diagonal ,since the determinant of a diagonal matrix is the product of its diagonal entries $a_{ii}$, where the $a_
{ii}$ are eigenvalues, and $0$ is not considered to be an eigenvalue. The diagonal entries are the eigenvalues.
EDIT: I may be wrong, or this may be a technical issue. I thought eigenvalues were supposed to be non-zero, but after a search, it seems many sources do include the possibility of $0$ as an eigenvalue. IF $0$ is accepted as an eigenvalue, then a non-invertible matrix may be diagonalizable, as the example in the comments below shows.
EDIT2: Diagonalizability ultimately comes down to having enough linearly-independent eigenvectors to form a basis for the eigenspace; I think this is what is called the geometric multiplicity.
Comment: The zero matrix is diagonal.
Comment: Right, let me edit.
Comment: This is still false. Consider $\begin{pmatrix} 2 & -1 \\ 2 & -1\end{pmatrix}$, which diagonalizes to $\begin{pmatrix} 1 & 0 \\ 0& 0\end{pmatrix}$
Comment: Please see the edit I just did.
Comment: I never saw zeros being expelled from the party of acceptable candidates to join the ranks of eigenvalues.
Comment: Maybe my class notes where from some eccentric prof. It may be better to delete the answer.
Comment: But, at the end of the day, it comes down to having enough L.I eigenvectors to form a basis for the eigenspace, so you're most likely right that $0$ is/should be acceptable eigenvalue.
Comment: $\lambda$ is an eigenvalue of $A$ if there's a nonzero $x$ such that $Ax=\lambda x$. This classical definition of eigenvalues implies that if a $\lambda$ is an eigenvalue then its associated eigenspace is at least one-dimensional (span of that $x$). I'd bet there's a typo in your (or your prof's?) definition of the eigen-guys and which of them must be nonzero.
Comment: Pavel: I just got too much of a head-start on senility: I realized what I was really thinking is that the zero _vector_ is not consider an eigenvector, but instead, I stupidly confused it with the zero _eigenvalue_. I thought I had at least some $40$ years before becoming senile, but maybe not.
Comment: Anyone else here also believe I should delete my answer?


Matplotlib 2D polygon in 3D axes

Question: I'm using matplotlib to plot some data as 2D curves in a 3D axes. I use the ax = axes3d.Axes3D() to define my axes and ax.plot(x, y, zs='z') to plot my 2D curves in the 3D space.
Now I want to add something that looks like what is on this page http://matplotlib.sourceforge.net/examples/mplot3d/polys3d_demo.html, but I want to enter directly the coordinates of the vertex.  In this example they have a fixed z array and they plot polygons with vertices (x,y), whereas what I want is z(x) and (x,y) vertices. Does anyone know how to do something like this?
And actually I don't need the collection thing, I just want to plot one big square in an arbitrary direction.
I tried to simply enter the coordinates and use ax.plot():
verts = np.array([[0,0,0], [0,1,0], [1,1,1], [1,0,1], [0,0,0]])
ax.plot(verts[:,0], verts[:,1], verts[:,2], zdir='y')

and it indeed plots a square, but I can't (or don't know how to) fill the face.
Answer: I hope this is what you wanted!

My rectangle will go from 0 to 10. So MAX=10.
What I did at first is try to define an axis between z and x.
z=MAX-x was my choice.
And then I preceded to draw this axis for every small increment of x, thus creating the rectangle.
This is the code:
anArray = []
for i in range(0, 101, 1):
    z = float(i)/10
    #print "x: ", x
    y = 10 - z
    #print "y: ", y
    for a in range(0, 101, 1):
        anArray.append([float(a)/10, y, z])
    #endfor
#endfor
verts = np.array(anArray)
ax.plot(verts[:,0],verts[:,1],verts[:,2], zdir='y')

Matplotlib 2D polygon in 3D axes


Persisting Spark Streaming output

Question: I'm collecting the data from a messaging app, I'm currently using Flume, it sends approx 50 Million records per day
I wish to use Kafka,
consume from Kafka using Spark Streaming
and persist it to hadoop and query with impala
I'm having issues with each approach I've tried..
Approach 1 - Save RDD as parquet, point an external hive parquet table to the parquet directory
// scala
val ssc =  new StreamingContext(sparkConf, Seconds(bucketsize.toInt))
val lines = KafkaUtils.createStream(ssc, zkQuorum, group, topicMap).map(_._2)
lines.foreachRDD(rdd => {

    // 1 - Create a SchemaRDD object from the rdd and specify the schema
    val SchemaRDD1 = sqlContext.jsonRDD(rdd, schema)

    // 2 - register it as a spark sql table
    SchemaRDD1.registerTempTable("sparktable")

    // 3 - qry sparktable to produce another SchemaRDD object of the data needed 'finalParquet'. and persist this as parquet files
    val finalParquet = sqlContext.sql(sql)
    finalParquet.saveAsParquetFile(dir)

The problem is that finalParquet.saveAsParquetFile outputs a huge number of files, the Dstream received from Kafka outputs over 200 files for a 1 minute batch size.
The reason that it outputs many files is because the computation is distributed as explained in another post-  how to make saveAsTextFile NOT split output into multiple file?
However, the propsed solutions don't seem optimal for me , for e.g. as one user states - Having a single output file is only a good idea if you have very little data.
Approach 2 - Use HiveContext. insert RDD data directly to a hive table
# python
sqlContext = HiveContext(sc)
ssc = StreamingContext(sc, int(batch_interval))
kvs = KafkaUtils.createStream(ssc, zkQuorum, group, {topics: 1})
lines = kvs.map(lambda x: x[1]).persist(StorageLevel.MEMORY_AND_DISK_SER)
lines.foreachRDD(sendRecord)

def sendRecord(rdd):

  sql = "INSERT INTO TABLE table select * from beacon_sparktable"

  # 1 - Apply the schema to the RDD creating a data frame 'beaconDF'
  beaconDF = sqlContext.jsonRDD(rdd,schema)

  # 2- Register the DataFrame as a spark sql table.
  beaconDF.registerTempTable("beacon_sparktable")

  # 3 - insert to hive directly from a qry on the spark sql table
  sqlContext.sql(sql);

This works fine , it inserts directly to a parquet table but there are scheduling delays for the batches as processing time exceeds the batch interval time.
The consumer cant keep up with whats being produced and the batches to process begin to queue up.
it seems writing to hive is slow. I've tried adjusting batch interval size, running more consumer instances.
In summary
What is the best way to persist Big data from Spark Streaming given that there are issues with multiple files and potential latency with writing to hive?
What are other people doing?
A similar question has been asked here, but he has an issue with directories as apposed to too many files
How to make Spark Streaming write its output so that Impala can read it?
Many Thanks for any help
Comment: this to me seems like a very common use-case, I'm surprised that no-one answered it. I guess I would suggest using a database, as Spark on its own can't really replace it. Try Cassandra or HBase (very steep learning curve for HBase).
Comment: You can set a different window for the output stream. `val lines = KafkaUtils.createStream(ssc, zkQuorum, group, topicMap).map(_._2).window(Minutes(15)).foreachRDD(rdd =>`
Answer: In solution #2, the number of files created can be controlled via the number of partitions of each RDD.
See this example:
// create a Hive table (assume it's already existing)
sqlContext.sql("CREATE TABLE test (id int, txt string) STORED AS PARQUET")

// create a RDD with 2 records and only 1 partition
val rdd = sc.parallelize(List( List(1, "hello"), List(2, "world") ), 1)

// create a DataFrame from the RDD
val schema = StructType(Seq(
 StructField("id", IntegerType, nullable = false),
 StructField("txt", StringType, nullable = false)
))
val df = sqlContext.createDataFrame(rdd.map( Row(_:_*) ), schema)

// this creates a single file, because the RDD has 1 partition
df.write.mode("append").saveAsTable("test")

Now, I guess you can play with the frequency at which you pull data from Kafka, and the number of partitions of each RDD (default, the partitions of your Kafka topic, that you can possibly reduce by repartitioning).
I'm using Spark 1.5 from CDH 5.5.1, and I get the same result using either df.write.mode("append").saveAsTable("test") or your SQL string.
Comment: it's difficult to control, as mentioned, if the data is large, I think it shouldn't simply repartition it into one file.
Answer: I think the small file problem could be resolved somewhat. You may be getting large number of files based on kafka partitions. For me, I have 12 partition Kafka topic and I write using spark.write.mode("append").parquet("/location/on/hdfs").
Now depending on your requirements, you can either add coalesce(1) or more to reduce number of files. Also another option is to increase the micro batch duration. For example, if you can accept 5 minutes delay in writing day, you can have micro batch of 300 seconds.
For the second issues, the batches queue up only because you don't have back pressure enabled. First you should verify what is the max you can process in a single batch. Once you can get around that number, you can set spark.streaming.kafka.maxRatePerPartition value and spark.streaming.backpressure.enabled=true  to enable limited number of records per micro batch. If you still cannot meet the demand, then the only options are to either increase partitions on topic or to increase resources on spark application.

Persisting Spark Streaming output


Isolated execution context in JavaScript

Question: I'm trying to execute a piece of code within an empty isolated execution context in JavaScript. In the below sample, I'm trying isolate isolated execution scope. What I want to do is to execute a function in context where no global variables are in.
(function() {
  'use strict';

  var scope = Object.create(null);
  var isolated = function() {
    'use strict';
    console.log(document); // Trying to get undefined
                           // but traces `document`.
  };

  isolated.call(scope);
})();

I thought it was simple to nullify global variables but there are too many!
var isolated = function(window, document, location /* etc */) {
  // ...
};

isolated.call(scope, undefined, undefined, undefined /* etc */);

Is there a better way to do this?
Comment: How about using web workers? https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers
Comment: I found a way in EcmaScript 6 to adjust `with(context) { ... }`, so that any new variables we assign will go into the context object, not the global / window object. More info here: https://stackoverflow.com/a/62350258/218294
Answer: There is no good way to do this within javascript itself (but see Gareth Hayes answer for another option).
There are a couple of bad ways.
(function() {
  var scope = Object.create(null);
  var obscurer = {};
  for (var key in this) {
     obscurer[key] = undefined;
  }

  with (obscurer) {
    var isolated = function() {
      'use strict';
      console.log(document);
    };
  }

  isolated.call(scope);
})();

Note that you'll actually get an error because console is not defined rather than document, although you can fix this by not blocking 'console' in the obscurer object.  You'll probably find that you need a whole bunch more globals than you realised.
You're also only blocking the enumerable properties of window.  If you become aware of nonenumerable properties that you want to block too, you'll have to add those to obscurer.
Of course, using with means you can't use strict mode any more as well, and everyone will look down their noses at you..
There are more interesting options available if you are working within node rather than the browser.
Comment: It works on Chrome as well actually 32.0.1700.72 m. But yeah this is my point you miss one thing and it's a full bypass whereas if you use a parser and whitelist it's much stronger against attack.
Comment: Window.prototype.alert.call(window,1) bypasses this sandbox.
Comment: I get "TypeError: window is undefined", but yes, I wouldn't really consider this method to be *safe*.  As far as I know, you can't make a safe sandbox without parsing the input code.  And even then, there are plenty of mistakes that could be made (e.g. people often forget that the Function constructor lets them eval code).
Comment: It's a Firefox specific vector.
Comment: I ran it in the console on firefox nightly, so I don't know if it's the version of firefox or the console that made it work for me.  I suppose it highlights the point that if you're really looking for safety there are a lot of edge cases to think about.  It'd be interesting to know exactly why Florent wants to do this...
Comment: Just to emphasise that this code isn't safe - I noticed that in non-strict mode you can delete top level values, if you are in a non strict function, you could, for example `delete window;` and that would delete the window in obscurer, which would let you access the global.... In es5, you could make the undefined non-deletable, but at some point it's like pushing down bubbles in a screen protector film.
Comment: what options might we have in Node.js rather than browser?
Comment: I created this gist, and I can't get this to work with a local variable instead of a global variable, any help appreciated: https://gist.github.com/ORESoftware/6f616ea88d34dba83e1cfdc43b7aff67
Answer: Use my MentalJS parser to isolate the environment. You can then choose which objects/variables it has access to by customizing the code.
http://businessinfo.co.uk/labs/MentalJS/MentalJS.html
http://code.google.com/p/mentaljs/
By default it allows access to document but you can prevent this, customize the environment here http://code.google.com/p/mentaljs/source/browse/trunk/MentalJS/javascript/Mental.js#260 you can then choose if they have access to Math etc.
Answer: It can be done without ECMA6 by using an IIFE that contains your trusted-needs-protection code, into which you inject your untrusted-needs-isolation-code (see example).
(function(injectedFunction) {
    /* Trusted code, that needs protection from untrusted code access */
    var hostingFuncPrivatePrimitive = "Hello there";
    var hostingFuncPrivateObject = {
        this_is_mine: true
    };

    var sharedPrimitive = 'This is shared';
    var sharedObject = {};

    // Running the untrusted code:
    injectedFunction(sharedPrimitive, sharedObject);

    console.log("sharedObject is: " + JSON.stringify(sharedObject));
    console.log("hostingFuncPrivateObject is: " +
        JSON.stringify(hostingFuncPrivateObject));
})(

(function(primitiveArg, objArg) {
    /* Untrusted code that needs isolation */

    // 1. using primitive (legal)
    console.log('primitiveArg is: ' + primitiveArg);

    // 2. Writing value to given objArg (legal):
    objArg.mumu = 'mimi';

    // 3. Trying to access host function variables (illegal)
    try {
        console.log('hostingFuncPrivatePrimitive is:' +
            hostingFuncPrivatePrimitive);
        hostingFuncPrivateObject.this_is_mine = false;
    } catch (e) {
        console.log(e);
    }
})

);

If you place the above in Chrome's console, you will get:
    primitiveArg is: This is shared
    VM117:29 ReferenceError: hostingFuncPrivatePrimitive is not defined
            at <anonymous>:26:17
            at <anonymous>:11:5
            at <anonymous>:16:3
    VM117:12 sharedObject is: {"mumu":"mimi"}
    VM117:13 hostingFuncPrivateObject is: {"this_is_mine":true}

P.S.: I know I'm late to the party, but maybe this helps anybody.


Gamma function, showing identity of terms

Question: How can I show this identity?
$$t^{a+h-1}e^{-t}=t^{a-1}e^{-t}\sum_{k = 0}^{\infty} \frac{ \log(t)^k}{k!}h^k $$
Anyone can give me hint?
Answer: By definition,
$$e^x = \sum_{n = 0}^\infty \frac{x^n}{n!}.$$
Now $t^h = \exp(\log(t^h)) = \exp(h\log t) \ldots$


How to prove that if $F$ belongs to $C^k$ then $F^{-1}$ also belongs to $C^k$?

Question: Suppose I have a function $F : \mathbb{R}^n \rightarrow \mathbb{R}^n$ and that $F$ is $k$ times continuously differntiable.
Let $V$ be an open subset of $\mathbb{R}^n$ for which $F^{-1}$ is well defined.
I am interested in learning how to show
$F^{-1}: V \rightarrow \mathbb{R}^n$ is $k$ times continuously differntiable as well. I would greatly appreciate any comments. Thank you very much.
PS I forgot to add that I am also assuming the Jacobian of $F$ is non-singular on all of $F^{-1}(V)$.
Comment: This is false. ${}$
Comment: @user296602 This is true locally when the derivative matrix is nonsingular. Perhaps that is what OP meant.
Answer: As pointed in the comments, the result is false as stated, but true if you assume that ${\sf D}F(x)$ is non-singular at each point. In this case, the result follows from the Inverse Function Theorem. Here's a counter-example in the general case: $F(x) = x^3$ is a $C^\infty$ bijection, but $F^{-1}(x) = \sqrt[3]{x}$ is not differentiable at $x=0$ (note that the Inverse Function Theorem says that $F^{-1}$ is $C^\infty$ near enough each point $x \neq 0$).
Comment: I am interested in seeing how to prove this without appealing to Inverse function theorem (because I want to see it how it works.) thank you
Comment: You don't, because it is false in general. I already gave a counter-example.
Comment: I fixed the problem, I meant to add that $DF(x)$ is non-singular at each point...
Answer: You need to do part of the proof of the inverse function theorem, namely to prove that $F^{-1}$ is differentiable, with derivative $$DF^{-1}(x) = \big(DF(F^{-1}(x))\big)^{-1}.$$
Since $F$ is $C^1$, $DF^{-1}$ is the composition of continuous functions and hence is continuous.
You then can bootstrap: If $F$ is $C^2$, this formula tells us that $DF^{-1}$ is the composition of two $C^1$ functions, hence $C^1$. So $F^{-1}$ is $C^2$ as well. Continue by induction.
(For this point you can see the beginning of my lecture. For the actual proof of the bulk of the inverse function theorem, see the preceding lecture.)
Comment: We need $DF(z)$ nonsingular at every $z$ so that we can take the inverse matrix.
Comment: How come we need that $DF$ is not zero for this?


Most home runs in first n games of the season

Question: Adrian Gonzalez of the Los Angeles Dodgers has now hit 5 home runs in the first 3 games of the 2015 season.
Is this a major league record for most home runs in the first 3 games of a season?
In general, is there somewhere I can find records for most home runs in the first n games of a season, for various values of n?
Answer: To answer the first question, according to Fox Sports this is a record (since 1970).
Update: Listening to Mike and Mike (ESPN Radio) this morning they quoted Elias Sports Bureau in saying that this is the most home runs through three games in the history of baseball.
The previous record through three games was a tie between three players with four home runs:

Vernon Wells, 2010 with Toronto
Chris Shelton, 2006 with Detroit
Barry Bonds, 2002 with San Francisco

Fox Sports Link
As for the second part, I have not seen a publicly searchable database with that fine of granularity.
Comment: Is there any reason for the 1970 cutoff, or is that just the time period that Fox Sports were able to check?
Answer: According to this ESPN article, the Elias Sports Bureau says Adrian Gonzalez set a new record by hitting 5 HRs in the first three games of the season:

Adrian Gonzalez set a major league record by clubbing five home runs in the Los Angeles Dodgers' first three games, according to the Elias Sports Bureau.

As for finding the most home runs in the first n games of a season, I don't think there is anything.  I checked baseball-reference.com's Batting Game Finder, and it doesn't appear to have that capability.


How to start multiple FlutterActivities?

Question: I have an app made in Flutter that runs only in Android and it has one screen that is made entirely in Native code.
Now a new request from the Project Manager is to include a button on that native screen that opens a new page. I want to make that page (activity) entirely in Flutter but I can't make it work in any ways I tried.
This is how the app looks:
Flutter page(MainActivity) -> Native page (another activity) -> A new flutter page (a third activity)
THE APP IS NOT AN EMBEDDING APP, IT IS A FULL FLUTTER APP WITH JUST ONE NATIVE PAGE
Does anyone know how to do that?
Doctor summary (to see all details, run flutter doctor -v):
[√] Flutter (Channel stable, v1.2.1, on Microsoft Windows [versão 10.0.17763.379], locale pt-BR)
[√] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
[√] Android Studio (version 3.3)
[√] IntelliJ IDEA Community Edition (version 2019.1)
[√] VS Code (version 1.30.1)
[√] Connected device (1 available)

• No issues found!

Comment: Can't you swap out the views?
Comment: What you mean? can you show me a sample?
Comment: Kinda like this: https://stackoverflow.com/a/36775658/2413303
Comment: Wont work, because the native page is a full activity, not just a view.
Comment: Are you 100% sure of this? Looks like [just a View to me](https://github.com/flutter/engine/blob/66fdeb163ef6fed6e9bac652bb36c7495aa17fb8/shell/platform/android/io/flutter/app/FlutterActivityDelegate.java#L163). Of course, it'd be easier to animate it if it weren't the ContentView, but if it existed inside a FrameLayout, if able. I wonder if `FlutterFragmentDelegate` exists. If not, it doesn't seem to hard to write it, though.
Comment: Couldn't you just "fake" it? finishing the native view, and returning to the flutter activity, but change the state of the flutter app to appear as if it's been launched by the button press? Otherwise, take a look at the [guide to add flutter to an existing app](https://github.com/flutter/flutter/wiki/Add-Flutter-to-existing-apps). `View flutterView = Flutter.createView(
      MainActivity.this,
      getLifecycle(),
      "route1"
    );`
Comment: @herbert I tried creating another FlutterActivity, but for some reason the plugins stoped working, the SharedPreferences for example, it returns empty values everytime I try to get something from it.
Also, for some reason my "Flutter" java class has no methods called "createView".
Answer: The key was the GeneratedPluginRegistrant that was causing the issues. Now I could make it work like this:
class ANewFluterActivity : FlutterActivity() {

    override fun createFlutterView(context: Context): FlutterView {
        val matchParent = WindowManager.LayoutParams(-1, -1)
        val nativeView = this.createFlutterNativeView()
        val flutterView = FlutterView(this, null as AttributeSet?, nativeView)
        flutterView.setInitialRoute("YOUR_NEW_ROUTE")
        flutterView.layoutParams = matchParent

        this.setContentView(flutterView)
        return flutterView
    }

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        GeneratedPluginRegistrant.registerWith(this) //THIS IS THE SECRET TO MAKE PLUGINS WORK PROPERLY
    }

}

Comment: Keep in mind that whis will start a new Engine, so, there is a loading time for that.


Issues with wrapping PrismForms NavigationService

Question: In PrismForms we got the problem, that the NavigationStack is empty after navigating to a new page. That means after using the hardware back-button on the SecondPage, the app is closed. Although the back-arrow in the header on Android isnt shown. If looking closely you can see the back-arrow for a short moment after the page is switched. I guess thats before the NavigationStack gets cleared.
To the first page we navigate with the following command in OnInitialized() in our App.xaml.cs which derives from PrismApplication.
<code>    NavigationService.NavigateAsync("NavigationPage/StartPage");
</code>
(If only Navigating to "StartPage" here, the Stack doesnt get cleared.)
That has do to with PageNavigationService.ProcessNavigationForNavigationPage(...) calling
        bool clearNavStack = GetClearNavigationPageNavigationStack(currentPage); and PageNavigationService.ProcessNavigationForContentPage(...) not.
From the StartPage to the next we navigate with NavigateAsync("SecondPage")". Here the described behaviour appears.
For navigation we use a class which wraps the Prism NavigationService. We hold him as a property and get him via Unity in our constructor:
<code>    this.PrismNavigation = prismNavigation ?? throw new ArgumentNullException(nameof(prismNavigation));
</code>
The methods "NavigateAsync" and "GoBackAsync", etc. we just pass through.
This way we want to seperate our ViewModel-Project from references to XamarinForms to later be able to use the same ViewModels for for example a WPF-GUI.
Why is the stack beeing cleared by our own NavigationService? If we register the original Prism NavigationService in App.xaml.cs instead, navigating back works as expected again. We found the point in the framework and could avoid the clearing with a drity hack, but that's against the navigation-logic implemented in PrismForms, but we don't understand how to do it the correct way.
Every help appreciated!
Comment: It sounds like you have an exception being thrown when either the View or ViewModel is being resolved. You might try resolving your ViewModels and Page's within a try/catch before your navigate, and log the exception.
Comment: We will try that and report back later. Thanks for the first ideas Dan.
Comment: That was not the case. Everything could be resolved well.
Answer: We edited a few things to get it working after finding some interesting information by Brian Lagunas in the forlast-post here:
https://github.com/PrismLibrary/Prism/issues/591
Although the topic was about something else, it led to improvements for overwriting the Navigation Service.
Remember that in your viewModels the Navigation Service must be named "navigationService" by convention. Also we switched from just holding the Prism Navigation Service as a parameter to deriving from it as suggested in the link above.
<code>    public class MyNavigationService : UnityPageNavigationService
</code>


Why is it preferable to test the effect of a predictor using a likelihood ratio test?

Question: In their fantastic book Applied Longitudinal Data Analysis: Modeling Change and event Occurrence Singer and Willett advocate a iterative model comparison technique for testing the effect of predictors: starting with a baseline model and then adding predictors, one at a time, to a series of subsequent models, using a likelihood ratio test to determine whether the addition of the new predictor to the model significantly reduces the -2LL deviance. These are nested models so each LR test in effect tests the effect of each new predictor. If the -2LL deviance is significantly reduced the new model retained, if not the previous model is retained.
They say they recommend this technique over simply including all the predictors one intends to test in a single regression model and reporting the confidence intervals and p-values.
I have accepted their recommendation however I don't really understand why (i.e. on what grounds) they consider their iterative approach superior to the 'all-in' approach.
The reason I ask is that I have conducted analyses for several papers now where my co-authors have been very confused by the iterative approach. It could be I am not explaining it well enough but I would like some more solid justification for using it than 'because Singer and Willett said so'.
A plain-language explanation would be very much appreciated.
Comment: This sounds like [stepwise regression, which _can_ be useful](https://stats.stackexchange.com/q/594106/247274) but [requires more delicate handling](https://www.stata.com/support/faqs/statistics/stepwise-regression-problems/) than your description of the text suggests is present in the book. (This is to say that my impression of that book, based on what is written here (I haven't read it), is that it is not correct.) $//$ Do you mean to ask about a likelihood ratio test as opposed to, say, a Wald or score test?
Comment: Thanks for responding Dave, My understanding is that this is not the same thing stepwise regression, it is testing difference in -2LL deviance of nested models. Have added a few more details to make this more clear.
Answer:
These are nested models so each LR test in effect tests the effect of each new predictor.

If you want to define stepwise regression as using information criteria instead of p-values to decide which variables should be kept, fine, but this is following the spirit of such stepwise regression and carries the same issues listed by Frank Harrell.

It yields R-squared values that are badly biased to be high.
The F and chi-squared tests quoted next to each variable on the printout do not have the claimed distribution.
The method yields confidence intervals for effects and predicted values that are falsely narrow; see Altman and Andersen (1989).
It yields p-values that do not have the proper meaning, and the proper correction for them is a difficult problem.
It gives biased regression coefficients that need shrinkage (the coefficients for remaining variables are too large; see Tibshirani [1996]).
It has severe problems in the presence of collinearity.
It is based on methods (e.g., F tests for nested models) that were intended to be used to test prespecified hypotheses.
Increasing the sample size does not help very much; see Derksen and Keselman (1992).
It allows us to not think about the problem.
It uses a lot of paper.

Issues 1, 2, 3, 4, and 7 are the most problematic in my mind. The issues center on the fact that stepwise-style regression is cherry-picking the variables that happen to work well in the one given sample and then act like you knew from the beginning that those were the right variables instead of accounting for the uncertainty in the stepwise selection process.
Therefore, I disagree with your reference that this is the statistically sound way to go.
However…
…I understand the appeal. Look at the issues. Performance metrics are biased high. Confidence intervals are falsely narrow (so p-values are falsely low). If you can pull a fast one on the reviewers and use this approach to make your results look better than they are, that makes it easier to publish your work and make your case for tenure/promotion/bonus/etc. This is basically the same as how it is easier to look at the smart kid's exam answers than to learn the material by studying, though.
With the caveat that I only know about the source material from what is described in the question, this sounds like cheating, and I would discourage this practice.
(Stepwise-style regression can be useful if some care is taken to account for the uncertainty in the variable selection. However, this approach does not seem to do so.)
Comment: Like I said, I don't think it is the same thing as stepwise regression. We're not talking about throwing 20 variables at the outcome and seeing what sticks. Singer and Willett certainly do not advocate that. They advocate a careful iterative approach. From what I understand their book is one of the pre-eminent texts for performing longitudinal data analysis (https://stats.oarc.ucla.edu/other/examples/alda/)
Comment: Your description of their idea is exactly a stepwise-style approach that distorts all inferences downstream of the feature-selection steps, perhaps only mildly but perhaps severely. How do you see their approach as accounting for the variable selection? @llewmills
Comment: Theory guides the variable selection, as it always should. This is not feature selection as we understand it now, a fishing expedition. It is a careful measured approach. This discussion has gone into the weeds a bit I may have to reframe the question in a separate post.
Comment: If this process does not select features, then what does it do? $//$ Retaining the smaller model if the likelihood does not improve enough sure sounds like a way to screen out variables.
Comment: The process tests the effect of predictors. And it differs from what I think of as stepwise selection in that it is not automated.
Comment: So stick it in a loop with some conditional logic related to the log-likelihood or p-value, and you have an automated process. The important part about what goes wrong with naïve stepwise regression is not that the process it automated but that the inferences downstream of selection steps should account for the fact that there has been a data-driven variable selection that might not always select the same variables, leading to the issues mentioned in my answer. @llewmills


Why does solving a derivative of sec(x) via power rule gets me a wrong value?

Question: I'm trying to get a derivative of $\sec(x)$ with respect to x.
The correct derivative is $\dfrac{\sin(x)}{\cos(x)^2}$, though this is not match the value that I was trying to solve.
Since $\sec(x) =\dfrac{1}{ \cos(x)}, \sec(x)' =\bigg(\dfrac{1}{\cos(x)}\bigg)' =  \dfrac{-1}{\cos(x)^2}$
But when I applied a random value, such as $\frac{\pi}{3}$, to the equation, the first (correct) derivative gets me 3.46 while the second one gets me -4.
I wonder what I'm making a mistake on. I think the problem is I used power rule instead of quotient rule, but am not sure why power rule here gives me the wrong value. In fact, I tried to solve $x^{-1}$ using both quotient rule and power rule, both got me $\dfrac{-1}{x^2}$.
Answer: You have to apply the chain rule: $(f(g(x))^{'}=f^{'}(g(x))\cdot g^{'}(x)$ In your case it is
$\left(\sec(x)\right)^{'} = \left(\cos^{-1}(x)\right)^{'}=-1\cdot \cos^{-2}(x)\cdot (-\sin(x))=\tan(x)\cdot \sec(x)$
Comment: Thanks! I seemed to have a mistake in not seeing `cos(x)^(-1)` as a composite function of `cos(x)` and `x^(-1)`.
Answer: Alternatively, you can use the reciprocal rule: if $f$ is differentiable at $x$, and $f(x)\neq0$, then $1/f$ is differentiable at $x$, and
$$
\left(\frac1f\right)'\left(x\right)=-\frac{f'(x)}{f(x)^2} \, .
$$
Applying this theorem in the case $f=\cos$, we see that
$$
\sec'(x)=\left(\frac1\cos\right)'\left(x\right)=-\frac{\cos'(x)}{\cos^2(x)}=\frac{\sin(x)}{\cos^2(x)}=\frac{1}{\cos(x)}\cdot\frac{\sin(x)}{\cos(x)}=\sec(x)\tan(x) \, ,
$$
at all points where $\cos(x)\neq0$, i.e. whenever $x$ is not of the form $\pi/2+k\pi,k\in\mathbb Z$.
The reciprocal rule is a special case of the quotient rule, which states that under the appropriate conditions,
$$
\left(\frac{f}{g}\right)'=\frac{f'g-fg'}{g^2} \, .
$$


Criando uma lista de elementos percorrendo um array associativo em EcmaScript

Question: Eu estou tentando criar uma parte do menu superior com EcmaScript em vez de html puro, porém não estou entendendo o que falta para acrescentar o elemento.
Código HTML

Código EcmaScript:
var listElement = document.querySelector("#list ul")
var listsElement = ['Home',
                    'Cursos',
                    'Promoções',
                    'Eventos',
                    'Blog',
                    'Quem-somos',
                    'Soluções Corporativas',
                    'Parceiros'
                    ]
var x = listsElement.map(function(item){

    return item;

})

listElement.createElement('li')
listElement.appendChild(x)

Sinto que está faltando alguma coisa para acrescentar ou está errado... Alguém poderia me explicar?
Answer: Aluns erros de lógica:

tens de usar o createElement uma vez para cada `li``
o createElement é um método de document
para adicionar conteudo podes usar o textContent
usa .forEach o .map não é possível inserir em bruto

Exemplo:

var listElement = document.querySelector("#list ul")
var listsElement = ['Home',
  'Cursos',
  'Promoções',
  'Eventos',
  'Blog',
  'Quem-somos',
  'Soluções Corporativas',
  'Parceiros'
]
listsElement.forEach(function(tipo) {
  const li = document.createElement('li')
  li.textContent = tipo;
  listElement.appendChild(li)
});

Comment: @RodrigoPires mais ou menos... semânticamente são diferentes, o `.map` é para criar uma nova array. Mas na prática ambas percorrem a array.
Comment: Opa @Sergio obrigado pela resposta, eu pensava que `.map` tinha o mesmo efeito de um `foreach.`


Google crawler doesn't index Angular page properly

Question: we have a website running on Angular 16 hosted on AWS Cloudfront. Everything works fine so far but sometimes the Google crawler doesn't index it properly.
Then the crawled page in the google search console just show this source code:

  ...

    <app>App is loading...</app>

This of course leads to a bad SEO ranking because nothing could be indexed.
Is there a reason why the gogole crawlers sometimes are able to load the javascript part without any problem and sometimes not?
Comment: If you want a good SEO do server side rendering and in this way all web search engines will work (https://angular.io/guide/universal)
Comment: Would take a lots of effort to translate everything towards SSR. Think Angular from Google should also work fine with at least the Google crawler in 2023
Answer: You'll want to try server side rendering. When the google crawler navs to the page, it's typically pretty quick - it won't wait too long for the loading to complete.
If you do SSR, the page should be rendered without any loading state.


count of Balance installments

Question: i got a requirement to write a MS-SQL stored procedure, the requirement is
Table 1
It consists total student fee to be paid in installments
EX:
SID   SNAME     INST.Date   Amount

123     XYZ     01-01-2013  3500
123     XYZ     01-05-2013  3500
123     XYZ     01-10-2013  3500
123     XYZ     01-04-2014  3500
123     XYZ     01-06-2014  3500

`
Table 2
It consists Payment recd details
SID     SNAME    Paydate    amoount
123     XYZ     01-01-2013  1167
123     XYZ     01-02-2013  1167
123     XYZ     01-03-2013  1167
123     XYZ     01-05-2013  1750
123     XYZ     01-05-2013  1750
123     XYZ     01-10-2013  1167
123     XYZ     01-10-2013  1167

now the requirement is as on jan2014,how many installment to paid,what is the Recd amount, no of balance installments to be paid as on jan-14.
Expected output
SID     Sname   Total Inst.Amt as on01.01.2014  Paid.amount     Balance amt     Balance.Installments

123     XYZ     10500                           9335            1165            1

For reference I gave one student data.
The stored procedure should run for 36000 students.
Comment: What does the column `Balance.Installments` represent? In your example it has `1`. How did you arrive at this value for the sample student?
Comment: please post what you have tried
Comment: Nice homework assignment! Why don't you share with us your attempts so far?
Comment: Thanks for all valuable outputs
Comment: Thanks for the reply,Actually i am looking out for balance installments, i will try to make better understanding, small ex: Total Installment amount = 100,single installment = 50 ,paid 40, balance 60 ,balance installments = 2.because single installment is 50 but balance is 60,it exceeding one installments so we need to count balance installments as 2,hope this will help you :)
Answer: Here's a query that will give you what you want, expect Balance.Installments I don't know what it means :
SELECT distinct t1.SID
, t1.SNAME
, t1.Total_Inst
, t2.Paid_amount AS 'Paid.Amount'
, t1.Total_Inst - t2.Paid_amount as 'Balance amt'
, CEILING( (t1.Total_Inst - t2.Paid_amount) / t.Amount ) as 'Balance.Installments'
FROM
    table_1 as t
    JOIN
    ( SELECT SID, SNAME, SUM(Amount) as Total_Inst
      FROM table_1
      WHERE Date < STR_TO_DATE('01/01/2014', '%m/%d/%Y') group by SID, SNAME
    )
    ON ( t1.SID = t.SID AND t1.SNAME = t.SNAME )
    AS t1
    LEFT JOIN
    ( SELECT SID, SNAME, SUM(Amount) as Paid_amount
      FROM table_2
      WHERE Paydate < STR_TO_DATE('01/01/2014', '%m/%d/%Y') group by SID, SNAME
    ) AS t2
    ON ( t1.SID = t2.SID AND t1.SNAME = t2.SNAME )

Comment: Your `JOIN` should be a `LEFT JOIN`, so that for students that haven't made any payment, you can still show the installments and balance due (which would be the full amount).
Comment: Thanks for the reply,Actually i am looking out for balance installments, i will try to make better understanding, small ex: Total Installment amount = 100,single installment = 50 ,paid 40, balance 60 ,balance installments = 2.because single installment is 50 but  balance is 60,it exceeding one installments so we need to count balance installments as 2,hope this will help you
Comment: Now if Total Installment amount = 100, single installment = 20 ,paid 10, balance 90 so balance installments = __5__ ? Is it right ?
Comment: Yes Sir,Total installment,paid,balance amount should be as on 01-01-2014.
Comment: thanks you rida,and all guys who gave valuable answers
Answer: why you will need to run for 36000  record at one time.Also what is actially installment.There is only one instalment for any student no several.So i think  database is lil wrong.
There should only one instalment amount and time interval(due date) per student in table.
Check the latest change.It is only for one student .
   Declare @student table(SID int,SNAME varchar(50),INST Date ,Amount int)
insert into @student
select 123,'XYZ','01-01-2013',3500 union all
select 123,  'XYZ',  '01-05-2013',  3500 union all
select 123,  'XYZ',  '01-10-2013',  3500 union all
select 123,  'XYZ',  '01-04-2014',  3500 union all
select 123,  'XYZ',  '01-06-2014',  3500

Declare @instalmentamount float
select @instalmentamount=amount from @student where sid=123
--select @instalmentamount
Declare @Table2 table(SID int,Paydate date,amoount int)
insert into @Table2
select 123,    '01-01-2013',  1167 union all
select 123,    '01-02-2013',  1167 union all
select 123,    '01-03-2013',  1167 union all
select 123,    '01-05-2013',  1750 union all
select 123,    '01-05-2013',  1750 union all
select 123,    '01-10-2013',  1167 union all
select 123,  '01-10-2013',  1167

declare @input date='01-01-2014'

;With CTE as
(select s.SID, sum(s.amount) as [Total Inst.Amt ] from @student s
where s.INST<=@input
group by s.SID),
cte1 as
(
select t.SID, sum(t.amoount) as [Paid.amount] from @Table2 t
where t.Paydate<=@input
group by t.SID
)

select c.sid,
(select top 1 s.SNAME from @student s where s.SID=c.SID) [Name],
c.[Total Inst.Amt ],c1.[Paid.amount],
c.[Total Inst.Amt ]-c1.[Paid.amount] [BalanceAmount],cast(((c.[Total Inst.Amt ]-c1.[Paid.amount])/@instalmentamount) as int)+1 [Balance.Installments]
from CTE c inner join cte1 c1 on c.SID=c1.SID

Comment: Thanks for the reply,Actually i am looking out for balance installments, i will try to make better understanding, small ex: Total Installment amount = 100,single installment = 50 ,paid 40, balance 60 ,balance installments = 2.because single installment is 50 but  balance is 60,it exceeding one installments so we need to count balance installments as 2,hope this will help you
Answer: TRY THIS...
SELECT  t.ID,
        T.SNAME,
        T.AMT  'Total Inst.Amt as on01.01.2014',
         ISNULL(SUM(amoount),0) AS 'PAID AMT',
        T.AMT - ISNULL(SUM(amoount),0) AS 'BALANCE AMT' ,
        CEILING((T.AMT-  ISNULL(SUM(amoount),0) )/( T.InstAmt*1.)) 'Balance.Installments'
FROM
    (
     SELECT ID,SNAME,SUM(AMT)AMT,MAX(AMT) AS InstAmt FROM @Table1 WHERE DT_ < '01-31-2014' GROUP BY  ID,SNAME
    ) T
     LEFT JOIN @Table2 P ON P.ID = T.id AND P.Paydate < '01-31-2014'
 GROUP BY t.ID,SNAME,t.InstAmt,T.AMT

Comment: Thanks for the reply,Actually i am looking out for balance installments, i will try to make better understanding, small ex: Total Installment amount = 100,single installment = 50 ,paid 40, balance 60 ,balance installments = 2.because single installment is 50 but  balance is 60,it exceeding one installments so we need to count balance installments as 2,hope this will help you

count of Balance installments


How do I ensure that two modules use the same string cache?

Question: I'd like to have polars dataframes in multiple modules use the same string cache, but I'm finding it difficult to manage. Imagine I have a package with modules A and B:
# A.py
import polars as pl

df_A = pl.DataFrame({
    'A': pl.Series(['a', 'b', 'c'], dtype=pl.Category)
})

# B.py
import polars as pl

df_B = pl.DataFrame({
    'A': pl.Series(['a', 'b', 'c'], dtype=pl.Category),
    'B': pl.Series(['d', 'e', 'f'], dtype=pl.Category)
})

# __init__.py

from . import A, B

pl.toggle_string_cache(True)

df_C = A.df_A.join(B.df_B, on='A')

exceptions.ComputeError: joins/or comparisons on categoricals can only happen if they were created under the same global string cache

This works if I move the import after the call to pl.toggle_string_cache(True), but this is bad code design: imports should always come first. I can think of moving A.df_A and B.df_B into global variables with generator functions, but I was hoping for a more elegant solution.
Comment: The other solution i can think about is to cast the columns to Utf8 and then cast them back to Categorical in the main program, after activating the String cache. That will make sure they have the same string cache. The negative aspect of it is the additional processing time for the casting
Comment: Thanks Luca. I think I've settled on simply making sure to call `pl.toggle_string_cache(True)` at the beginning of each module, that seems to work.
Comment: Good you found a solution. I am curious if it still works if you don't set it in the init.py, I will check that
Answer: I believe the simplest answer is to simply ensure that the call to pl.toggle_string_cache(True) is made in each module that creates categorical data. In other words, the solution to the original post would be:
# A.py
import polars as pl

pl.toggle_string_cache(True)

df_A = pl.DataFrame({
    'A': pl.Series(['a', 'b', 'c'], dtype=pl.Category)
})

# B.py
import polars as pl

pl.toggle_string_cache(True)

df_B = pl.DataFrame({
    'A': pl.Series(['a', 'b', 'c'], dtype=pl.Category),
    'B': pl.Series(['d', 'e', 'f'], dtype=pl.Category)
})

# __init__.py
from . import A, B

pl.toggle_string_cache(True)

# now this works properly
df_C = A.df_A.join(B.df_B, on='A')


Logistic regression with only one variable?

Question: I mistyped in stata "logistic a" where a is a binary variable and to my surprise it actually ran.
What is a logistic regression doing when it's running without any independent variables?
Comment: It might be just fitting the null model (intercept only, log-odds of being in the class of interest). To verify, please edit your question to show the class breakdown of your binary variable a and the output from the command you ran.
Comment: Edited my answer, does this suffice?
Comment: Incidentally, you can do this with regress, ologit, etc. In all cases you get an intercept only model: a heavy-handed way to characterize the mean/proportion, etc. of the dependent variable. Contrast `mean a` with `regress a` with Odds/(Odds+1) from your output above.
Answer: It's an intercept-only model. Stata's notation <code>_cons</code> is the intercept term. With no other regressors, the odds is $p/(1-p)$ where $p$ is the proportion with positive response.
Comment: I understand what the intercept and the constant is, but I'm still not sure if this actually tells me anything about my variable?
Comment: @Paze I have already shown you how the value of "Odds" relates to the frequency of your variable `ShuntEffekt`.


Singular Value decomposition positive components

Question: I am using Singular Value Decomposition (SVD) applied to Singular Spectrum Analysis (SSA) of a timeseries.
<code>% original time series
x1= rand(1,10000);
N = length(x1);

% windows for trajectory matrix
L = 600;
K=N-L+1;

% trajectory matrix/matrix of lagged vectors
X = buffer(x1, L, L-1, 'nodelay');

% Covariance matrix
A = X * X' / K;

% SVD
[U, S_temp, ~] = svd(A);

% The eigenvalues of A are the squared eigenvalues of X
S = sqrt(S_temp);
d = diag(S);
% Principal components
V = X' * U;
for i = 1 : L
    V(:, i) = V(:, i) / d(i);
end
</code>
EDIT 1:
If I try to reconstruct the original matrix X adding the i-matrices given by d(i)*U(:,i)*V(:,i)' I get exactly the original matrix.
<code>X_rec = zeros(size(X));
for index_col = 1 : L
    disp(index_col)
    X_i = d(index_col)*U(:,index_col)*V(:,index_col)';
    X_rec=X_rec+X_i;
end
</code>
I wanted to know if there is a way to have the singular components V always positive with the compromise of having an error in the recostructed matrix.
In this case the reconstructed matrix would be just an approximation of the original.
X is always > 0 in my case (and also the Covariance matrix A)
Comment: What exactly bothers you? That `S` sometimes has zero values or that `U` is partly negative?
Comment: According to [Frobenius-Perron](http://en.wikipedia.org/wiki/Perron%E2%80%93Frobenius_theorem) theorem, if `X` is positive the fisrt vector in `V` and in `U` are complitely positive too. Or, more strictly, either completely positive or completely negative - depending on your svd function realization.
Comment: @ttnphns Yes, this is right! indeed I get the first singular vector always positive. How can I get all the singular vectors positive? I would like that SVD (or something else) decomposes my original matrix in additive matrices of positive values..
Comment: I'm not an expert in specifically math side of it, but I think it is impossible to get what you want with svd: some values except in the 1st vector will always be negative. Perhaps you want some other orthogonalization procedure, not svd?
Comment: @ttnphns this is fine with me, but I couldn't find any so I thought that modifying SVD with the compromise of losing the exact recontruction of the original matrix.
Comment: @ttnphns I edited the question to explain this point..
Comment: @gung: I don't really know which one to close. The OP has posted this question at least five times on SO and stats.SE, I just wanted to link the question.
Comment: The singular values are in $S$, and they are always non-negative.
Comment: That some values of V are negative...I edited my question to be more specific
Comment: possible duplicate of [SVD (singular value decomposition) and SSA (singular spectrum analysis) adding constrains on components](http://stats.stackexchange.com/questions/143385/svd-singular-value-decomposition-and-ssa-singular-spectrum-analysis-adding-c)
Answer: Yes, you can add a non-negativity constraint to the right-singular vectors, $V$.
However, you must specify how "far" from 0 the elements can be to be an acceptable solution.
Let your optimisation problem be
$$
    \text{maximise}\; \frac12\|Xv\|_2^2,
$$
$$
    \text{subject to}\; \|v\|_2\leq1 \;\text{and}\; v_i > \varepsilon, \forall i.
$$
This is a convex optimisation problem with convex constraints. The projection onto the non-negative orthant is
$$
    \rho_+(x) = \begin{cases}
                  x_i & \text{if}\; x_i > \varepsilon, \forall i\\
                  \varepsilon   & \text{otherwise}.
              \end{cases}
$$
and the projection onto the $\ell_2$ norm ball is
$$
    \rho_{\ell_2}(x) = \begin{cases}
                           x & \textit{if}\; \|x\|_2<1 \\
                           \frac{x}{\|x\|_2} & \text{otherwise}.
                       \end{cases}
$$
I am not 100 % sure, but I think that the projection onto the intersection of these constraints is:
$$
    \rho(x) = \rho_{\ell_2}(\rho_+(x))
$$
A trivial algorithm would then be:
$$
    v^{k+1} = \rho(v^k - t\nabla \frac12\|Xv^k\|_2^2) = \rho(v^k - tX^TXv^k),
$$
where $t$ is a step size. This method is called projected gradient descent.
This solution will be the closest approximation (in a squared $\ell_2$ sense) to the SVD, but with a non-negativity constraint on the right-singular vectors.
Comment: @TommyL: [Wiki states:](http://en.wikipedia.org/wiki/Singular_value_decomposition#Singular_values.2C_singular_vectors.2C_and_their_relation_to_the_SVD) *Non-degenerate singular values always have unique left- and right-singular vectors, up to multiplication by a unit-phase factor*. So you won't get lucky most of the time.
Comment: @gabboshow Yes, $v$ would be a right-singular vector. In order to obtain more than one you would have to either add an orthogonality constraint, or deflate the component from $X$. The simplest is deflation (but it has its own problems). After the first component, let $X := X - Xvv^T$, and run the algorithm again to find the second component.
Comment: @TommyL What you are suggesting is to iteratively run SVD? run the first time, reconstruct X taking out and saving the first component and running again and again SVD?
Comment: @knedlsepp I'm not sure what you mean by being lucky. This would not be the singular vectors, but would be the ones maximising the variance, under the constraint of non-negativeness.
Comment: @gabboshow Not really. I suggest that you extract one singular vector at the time, and constraint them all to non-negativeness.
Comment: The OP asks for a way to enforce non-negativity of every component of the singular vectors, while still maintaining a singular value decomposition. Changing the matrix `V` of a SVD is only possible for degenerate singular values and not even then guaranteed to be able to yield all non-negative values.
Comment: Thanks to both of you. What I would like to have is a decomposition of the initial matrix in additive matrices. I can do this by using SVD, but I get negative singular vectors (that I do not want). A good point of using SVD is of course that I can organize the singular vectors according to the values of the eigenvalues and then group them together, but this is not my priority...
Comment: @knedlsepp Of course, what I propose is not the SVD, but it is the closest non-negative approximation to the SVD. What I propose is a decomposition that maximises the variance of the data matrix $X$ (just like SVD does) but which is constrainted to non-negative right-singular vectors.
Comment: @TommyL: I see. Reading your answer it doesn't come across really well, that the thing you are generating is not a SVD anymore. (However the OP doesn't really know what he wants anyway and has posted this question for the 5th time at least.)
Comment: @knedlsepp Ok, I added a clarification to my answer. I thought it should be obvious that this is not really the SVD (just as ridge regression is not MLR). OP cannot use the SVD, as he points out himself, and so his only option is to constrain the solution like this.
Comment: @TommyL thanks for your answer! Would you be so kind to provide with the matlab implementation of your answer? I am struggling with that..
Comment: Hi Thanks for your answer. I am not familiar with optimization problems... could you explain a bit more? is v the same of V?


Is there a proper ideal of $B(H)$ that contains a proper projection

Question: Let $H$ be a infinite-dimensional separable Hilbert space and $\mathcal{I}$ be a proper closed two-sided ideal of $B(H)$. Can $\mathcal{I}$ contain a projection for a infinite dimensioal proper closed subspace? If $H$ is not separable (or $\mathcal{I}$ not closed) while all other conditions remain the same, will the conclusion change? What if $\mathcal{I}$ is only one-sided (other conditions remain the same)?
(Added) This question is inspired by Corollary 5.11 in Banach Algebra Technique in Operator Theory written by Ronald G. Douglas. This corollary claims that in an infinite dimensional separable Hilbert Space the ideal of compact operators $\mathcal{K}$ is the only proper closed two-sided ideal in $B(H)$. The proof first assumes that $T \in \mathcal{I}$ and $T$ is NOT compact. Hence by the converse of Lemma 5.8 in the range of $T$ there is a closed infinite dimensional subspace say $M$. Define $  S_0: M \rightarrow H, S_0(Tv) = v\,\implies\,S_0 \in B(M)$ according to closed mapping theorem. Define $S\,\vert_M = S_0, S\,\vert_{M^{\perp}} = 0\,\implies\,TS = P_M \in \mathcal{I}$. At last the proof claims that hence $\mathcal{I}$ contains the identity mapping. I am not sure if the last statement is correct.
Lemma 5.8: $H$ is a $\infty - dim$ Hilbert space. $T$ a compact operator iff $ran(T)$ contains NO closed $\infty - dim$ subspace.
In the original text there is only $\implies$ direction but the converse is also true. Let $H_{\leq 1}$ be the closed unit ball of $H$ and hence $\overline{TH_{\leq 1}}$ is a finite dimensional closed and bounded subspace and hence compact (with respect to the original topology).
Comment: The application of Lemma 5.8 is wrong as written; what is being used is the converse of Lemma 5.8.
Comment: Thank you for pointing it out. I should state it more precisely there.
Answer: Suppose $\mathcal{I}$ contains a projection $P$ onto an infinite-dimensional closed subspace $M$.  In particular, $\dim M=\dim H=\aleph_0$ since $H$ is separable, so we can pick a Hilbert space isomorphism $U:M\to H$.  Extend $U$ to an operator $T:H\to H$ which vanishes on $M^\perp$, and let $S:H\to H$ be the composition of $U^{-1}:H\to M$ with the inclusion $M\to H$.  Then $TPS=1$, so $1\in\mathcal{I}$ and the ideal is not proper.
This argument does not use the assumption that $\mathcal{I}$ is closed, but it does use the assumption that $H$ is separable and $\mathcal{I}$ is two-sided.  If $H$ has uncountable dimension, then the set of operators whose range is separable forms a proper closed two-sided ideal and contains projections onto infinite-dimensional subspaces.  If $\mathcal{I}$ is only a one-sided ideal, then it could be the ideal of all operators that vanish on $M^\perp$ (for a left ideal) or operators whose range is contained in $M$ (for a right ideal).
Comment: Thank you for your answer. I should have tried applying the same idea and in fact it is not too complicated. Also your examples are very helpful :)


Equivalence between statements on linear independence

Question: Consider the integers $N,T,J$. In what follows, $I_N$ is the $N\times N$ identity matrix. $0_{N\times N}$ is the $N\times N$ zero matrix.
Let $A$ be an $NT\times (N+J)$ matrix structured as follows
$$
A\equiv \begin{pmatrix}
I_N & A_1\\
I_N & A_2\\
\vdots & \vdots\\
I_N & A_T
\end{pmatrix}
$$
where each of $A_1,...,A_T$ are $N\times J$ matrices.
Could you help to show that (or give me an informal intuition on why)
(1) The columns of $A$ are linearly independent if and only if the columns of
$$
B\equiv \begin{pmatrix}
I_N & A_1\\
0_{N\times N} & A_2-A_1\\
\vdots & \vdots\\
0_{N\times N} & A_T-A_1
\end{pmatrix}
$$
are linearly independent
(2) The columns of $A$ are linearly independent if and only if the columns of
$$
C\equiv \begin{pmatrix}
 A_2-A_1\\
\vdots \\
A_T-A_1
\end{pmatrix}
$$
are linearly independent
Answer: You are simply doing row operations (in blocks), subtracting $[I_{N} \ A_{1}]$ from each block below it. Row operations don't effect the linear dependence/independence relations of the columns.
Once you do this, you can clearly see that there is a pivot position in each of the first $N$ columns. Any pivot position in the columns below that would correspond to a pivot position in
$$\begin{bmatrix} A_{2} - A_{1} \\ \vdots \\ A_{T} - A_{1}\end{bmatrix}.$$


Java Execute Jar-File NullPointerException when updateLayoutState

Question: i exported my java programm to an executable JAR-File.
When i now run this JAR-File:
java -jar myFile.jar

this Exception appears:
Exception in thread "main" java.lang.NullPointerException
    at javax.swing.JList$4.getSize(Unknown Source)
    at javax.swing.plaf.basic.BasicListUI.updateLayoutState(Unknown Source)
    at javax.swing.plaf.basic.BasicListUI.maybeUpdateLayoutState(Unknown Source)
    at javax.swing.plaf.basic.BasicListUI.getPreferredSize(Unknown Source)
    ....

Never saw this before and havent found a solution yet.
What is actually happening here and how can i solve this problem?
Answer: It doesn't seems to be a packaging related issue. It looks more like there is a JList not properly initialized - e.g. like if you passed null as the array in the constructor. So check your code...
Comment: Yes..That might be the problem ..http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4920074
Answer: Procedure:

Expand the jar in exactly the same location as the jar as a directory, say, ExpDir.
Run the main class using -cp ExpDir and main class.  Use the same current directory, JVM, and everything else.
If the program now works (1% probability, in my experience), comment on this on the question.  Also, confirm that you are not inadvertently changing something else.
If it does not (99% probability), correct the programming error and/or ask a different question appropriate to the error.