Linux set min_rto on IPv6 device

Question: I came across https://unix.stackexchange.com/questions/210367/changing-the-tcp-rto-value-in-linux which pretty much does exactly what I want, but I need to do it with an IPv6 tunnel.
$ sudo ip -6 route
aaaa::/64 dev tun0  proto kernel  metric 256
fe80::/64 dev eth0  proto kernel  metric 256
fe80::/64 dev wlan0  proto kernel  metric 256
fe80::/64 dev tun0  proto kernel  metric 256

Then, if I look at my active connections we see:
$ ss -i
tcp   ESTAB      0      0                                                                                aaaa::1:51692

         cubic rto:360 rtt:157.424/6.438 ato:40 mss:64 cwnd:10 send 32.5Kbps pacing_rate 65.0Kbps rcv_space:28800

It currently has an rto of 360. This device only deals withs super low bandwidth, high latency, lossy networks (i.e. 6LoWPaN) so I want to increase the min_rto to at least several seconds.
I have disabled frto:
$ sudo cat /proc/sys/net/ipv4/tcp_frto
0

Now I want to set min_rto, as shown in the linked question.
$ sudo ip -6 route change aaaa:: dev tun0 rto_min 2000
RTNETLINK answers: No such file or directory

Why do I get this error and how can I fix it?
It's worth noting that if I do 2000ms (or 5ms, as in the linked answer), it tells me it's an invalid value, hence why I dropped off the ms part. In case it matters, my Linux info is below:
Linux 4.4.13-v7+ #894 SMP Mon Jun 13 13:13:27 BST 2016 armv7l GNU/Linux

Comment: @030 "Why do I get this error and how can I fix it?" (hint: it's bolded)
Answer: The tcp_* settings listed under IPv4 also apply to IPv6, according to the documentation.

IPv6 has no global variables such as tcp_*.  tcp_* settings under ipv4/ also
apply to IPv6 [XXX?].

Given the question in the documentation there, it would be wise to verify this applies to your specific (older) kernel before proceeding.
Comment: I believe they do, changing `ipv6/tcp_frto` has a noticeable impact on my ipv6 rto values

Linux set min_rto on IPv6 device


property of a subgroup generated by two subgroups

Question: Let $A, B, C$ be subgroups of a finite group $G$. Is it true that the following holds:
$$\langle A,B\rangle\cap C=\langle A\cap C, B\cap C\rangle$$
Answer: No. Consider $A=\Bbb Z/n\Bbb Z \oplus 0, B = 0 \oplus \Bbb Z/n \Bbb Z$ and $C = \langle (1,1) \rangle$ as subgroups of $(\Bbb Z/n\Bbb Z)^2$. We have
$$C = (\Bbb Z/n\Bbb Z)^2 \cap C = \langle A, B\rangle \cap C$$
but
$$ 0 = \langle 0,0\rangle = \langle A \cap C, B\cap C\rangle.$$
Answer: The answer is no. We can take, for example, $G=S_5$, $A=A_5$, $B=\langle(12)\rangle$ and $C=\langle (13)\rangle$. Then $\langle A,B\rangle=S_5$ and $\langle A,B\rangle\cap C=C$. On the other hand, $A\cap C=B\cap C=1$ and so $\langle A\cap C, B\cap C\rangle=1\neq C=\langle A,B\rangle\cap C$.


Binding Spring Form Tags to A Map

Question: I have a requirement to have my command object for the spring form tag binding to contain a Map to map a keyword to a "Y" or "N".
In the command object, the map is defined as such:
private Map<String, String> keywordResponseMap = new LinkedHashMap<String, String>();

In the JSP, I use radio buttons to generate the Y/N response:
    <c:forEach items="${form.eligibilityQuestions}" var="question">
        <c:set var="keyword" value="${question.keyword}" />

            ${question.text}
            <form:radiobutton path="keywordResponseMap['${keyword}']" value="Y"/>
            <form:radiobutton path="keywordResponseMap['${keyword}']" value="N"/>

    </c:forEach>

The radio button control is NOT binding the Y or N into the map on form submit.
The Chrome dev tools show me that the submitted request parameter looks like this (yes, the space is intentional, the joys of fixed width data):
keywordResponseMap['RECORD4 ']:Y

Can anyone help?
Jason
Comment: What does your form handler method look like? What mechanism you use to determine the binding failed?
Answer: I solved this by creating a postprocessing method in the backing controller:
public void postprocessQuestions(RequestContext requestContext) {

    log.debug("Entered postprocessQuestions()");

    TrainingForm form = (TrainingForm) requestContext.getFlowScope().get("trainingForm");

    // manually get the reservation responses from the external request
    // context parameters
    Map<String, Object> requestParamMap = requestContext.getExternalContext().getRequestParameterMap().asMap();
    for (String key: requestParamMap.keySet()) {
        if (key.startsWith("keywordResponseMap")) {
            String keyword = StringUtils.substringBefore("['", "']");
            String value = (String) requestParamMap.get(keyword);
            form.getKeywordResponseMap().put(keyword, value);
        }
    }

}

Then I had to turn off the automatic validation on the transition in order to run the postprocessign method before validation:
    <view-state id="renoExtendedEligibility" view="renoExtendedEligibility" model="renegotiationForm">
        <on-render>
            <evaluate expression="flowController.enterQuestions(flowRequestContext)" />
        </on-render>
        <transition on="prev" to="QuestionsTransition" validate="false" />
        <transition on="*" to="QuestionsTransition" validate="false">
            <evaluate expression="renoFlowController.postprocessQuestions(flowRequestContext)" />
            <evaluate expression="renegotiationFormValidator.validateQuestions(renegotiationForm, validationContext)" />
        </transition>
    </view-state>

Thanks for the help!
Jason

Binding Spring Form Tags to A Map


Rijndael support in Java

Question: We have a requirement to do some Rijndael development in Java.
Any recommendations for articles, libraries etc. that would help us?
Any pointers to keystore maintenance and how store the keys securely?
Edit:
It would need to be open source. Essentially, it's just standard encrypt / decrypt of data using Rijndael.
Answer: Java includes AES out of the box. Rijndael is AES. You don't need any external libraries. You just need something like this:
<code>byte[] sessionKey = null; //Where you get this from is beyond the scope of this post
byte[] iv = null ; //Ditto
byte[] plaintext = null; //Whatever you want to encrypt/decrypt
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
//You can use ENCRYPT_MODE or DECRYPT_MODE
cipher.init(Cipher.ENCRYPT_MODE, new SecretKeySpec(sessionKey, "AES"), new IvParameterSpec(iv));
byte[] ciphertext = cipher.doFinal(plaintext);
</code>
And that's it, for encryption/decryption. If you are processing large amounts of data then you're better off reading chunks that are multiples of 16 bytes and calling update instead of doFinal (you just call doFinal on the last block).
Comment: This is exactly the reverse, AES is a subset of Rijndael
Comment: Rijndael isn't equal to AES, but instead is AES with some restrictions - fixed block size of 128 bits, and some crypto modes not supported. Any full AES implementation should be usable as Rijndael.
Comment: @BrunoRohée is correct. AES is a subset of Rijndael, check this http://stackoverflow.com/questions/748622/differences-between-rijndael-and-aes
Comment: I don't want to edit my answer because these comments were made after it was selected as the right one, and I think 3 comments stating that AES is a subset of Rijndael should be enough for anyone to notice I got it backwards.
Comment: what are the two values that you supply in the sessionKey & iv? from where you get them?
Comment: both should be random
Answer: For a great free library, I highly recommend BouncyCastle. It is actively maintained, high quality, and has a nice array of code examples. For reference documentation, you'll have to rely more on the general JCE docs.
I can't say what library we use to meet FIPS certification requirements. But there are alternatives to CryptoJ that are much, much cheaper.
In general, I'd recommend generating a new key for each message you encrypt with a symmetric cipher like Rijndael, and then encrypting that key with an asymmetric algorithm like RSA. These private keys can be stored in a password-protected, software-based key store like PKCS #12 or Java's "JKS", or, for better security, on "smart card" hardware token or other crypto hardware module.
Answer: As my company recently found out, AES is not quite Rijndael.  AES has the restriction that keys MUST be 128, 192, or 256 bit - however, Rijndael allows for keys that are 160 and 224 as well.
As indicated by erickson above, BouncyCastle provides a Rijndael object that DOES support the additional key lengths: 128/160/192/224/256 bits.  Specifically, take a look at the lightweight API.
Gnu-crypto is another open source library - however, it also does NOT provide support for 160 and 224 bit keys.
So, if you are specifically looking for full Rijndael support, then BouncyCastle is the only one I've found so far.
Comment: If you need to use the java for android developement (although the question does not state that, it is worth noting that Bounty Castle has a brother now, that gives more flexibility on android known as Spongy Castle, which is a derivative.. and quite good actually)
Answer: javax.crypto has AES support: http://java.sun.com/developer/technicalArticles/Security/AES/AES_v1.html
As for secure key storage, the usual method is to derive an encryption key from user input (a passphrase) using a cryptographic hash function, and use the derived key to encrypt the keychain. Or, if you only need one key, you can use the derived key itself.
Always keep in mind that the security of the system is directly related to the strength of the hash function used. Use a cryptographically secure hash function, along with a salt if possible, and hash more than once (hundreds of times, for example).
That being said, the question is very vague.
Comment: it is also worth noting that `javax.crypto` uses a very optimised version of AES and is quite fast.

Rijndael support in Java


OrientDB ETL with CSV, no headers and multiple join fields

Question: I am trying to load some CSV files into OrientDB.  They have been extracted from a MySQL database holding the Unified Medical Language System (NIH UMLS) data. The two files contain vertices:
<code>"C0484850"  "A18164418" "Troponin T.cardiac [Mass/volume] in Venous blood"  "Y" "Clinical Attribute"
"C0484850"  "A18241423" "Troponin T.cardiac:MCnc:Pt:BldV:Qn"    "Y" "Clinical Attribute"
"C0484850"  "A18861342" "Troponin T.cardiac:Mass Concentration:Point in time:Blood venous:Quantitative" "Y" "Clinical Attribute"
"C0484851"  "A18280127" "Troponin T.cardiac [Mass/volume] in Serum or Plasma"   "Y" "Clinical Attribute"
"C0484851"  "A18357585" "Troponin T.cardiac:MCnc:Pt:Ser/Plas:Qn"    "Y" "Clinical Attribute"
"C0484851"  "A18816754" "Troponin T.cardiac:Mass Concentration:Point in time:Serum/Plasma:Quantitative" "Y" "Clinical Attribute"
</code>
and relationships:
<code>"C0484850"  "A18164418" "has_common_name"   "C0484850"  "A18241423"
"C0484850"  "A18241423" "class_of"  "C0201682"  "A18205079"
"C0484850"  "A18241423" "component_of"  "C3538889"  "A18284809"
"C0484850"  "A18241423" "property_of"   "C0560150"  "A18367132"
"C0484850"  "A18241423" "scale_of"  "C1442116"  "A18405933"
"C0484850"  "A18241423" "system_of" "C1442207"  "A18136032"
"C0484850"  "A18241423" "time_aspect_of"    "C1442880"  "A18406936"
"C0484850"  "A18241423" "fragments_for_synonyms_of" "C2603360"  "A18401194"
</code>
I'm finding the OrientDB documentation for extractors and for CSV rather lacking.

For the "row" extractor, there is only one example with no full documentation.  I do not have row headers, so how to I use the "row" extractor to name the fields in the vertices (cui, aui, description, pref, syn) ?  I'm guessing there is a syntax like id:row2 but I can't find it.
The edges are joined using via the 2 and 5th fields of the vertices, which are unnamed.  Also, the edge property is unnamed.

For silly reason, I can't pull directly from MySQL right now, but if there are better examples than the official site I would be interested in seeing them.
Answer: use the csv extractor (see : http://orientdb.com/docs/2.2.x/Extractor.html )
set "columnsOnFirstLine" to false
set "columns" to be the explicit list of columns, in the order they exist in the csv file


How to tell search engines like Google to display its tabs/url?

Question: When we search in google engine it displays top web site tabs or links too. Like when we search "bing" or "net beans".
Q: How it displays those links. Do we have to tell it to display these links.
Q: Does it something have to do with sitemap.xml/robots.txt or it displays the links present in index.php of that website?

Robots.txt: allow/disallow bots to crawl which page.
sitemap.xml: tells the map/loc of your website pages and also tells the frequency.

Q: How does it display description of a website?
I have searched about description it has to do with meta tag name description. But i open the source file of net beans
<META NAME="description" CONTENT="Welcome to NetBeans">

But the description google showing is
Fully-featured Java IDE written completely in Java, with many modules available, such as: debugger, form editor, object browser, CVS, emacs integration, ...

Answer: For your first question I should say that those links which you've mentioned are automatically genereted in top most visited websites and portals. If you'd set the sitemap.xml and robots.txt correctly in your website root folder, After a while if your website has a lot of visitors traffic, google detect your top most visited links which users most redirect to and show them in its result as you wish.
For the second question meta tags are not the only criteria the search engines show them in their results, Rather they catch the page content and extract the context from the text content of the page and show the description based on your entered keyword. However your meta description will be shown when keyword is the website name or its domain.
Take a look at Open Graph Protocol to extend your information about meta tags and your requirements for seo.
Regards


How do I access an objects methods through a different class?

Question: So basically I have a main (DrawDriver) where I create 2 objects at the top.
 public class DrawDriver {

  public static void main(String[] args) {

    final GraphicPanel pannel1 = new GraphicPanel();
   final Frame2UserInput pannel2 = new Frame2UserInput();

    TitledBorder border = new TitledBorder("Input");
    border.setTitleColor(Color.BLACK);
   pannel2.setBorder(border);

    JFrame frame = new JFrame("DrawGoGo!");
   frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
    frame.setPreferredSize(new Dimension(500, 500));
   frame.setLayout(new GridLayout(2, 0));
    frame.pack();
   frame.setVisible(true);
   frame.setResizable(false);

            // Creating the MENU BAR

            JMenuBar menubar = new JMenuBar();
           frame.setJMenuBar(menubar);
            JMenu file = new JMenu("File");
        menubar.add(file);
        JMenuItem New = new
           JMenuItem("New"); // Creates the jmenu item
           file.add(New); // adds
           JMenuItem Load = new JMenuItem("Load");
            file.add(Load);
           JMenuItem Save = new JMenuItem("Save");
            file.add(Save);
           JMenuItem Exit = new JMenuItem("Exit");
            file.add(Exit);
            JMenu help = new JMenu("Help");
            menubar.add(help);

           JMenuItem about = new JMenuItem("About");
           help.add(about);

            frame.getContentPane().add(pannel1);
           frame.getContentPane().add(pannel2);

 }

In Frame2UserInput I want to access the GraphicPanel instance through the DrawDriver (main) class. This will be a canvas where I can type commands in & click the button & then the canvas will draw accordingly. You can see I've attempted to access it already in my ActionListener but can't because it makes a new instance of the class & not changes anything in the program (gui) I run.
So how would I access the object I created for GraphicPanel in the main but through a different class ? Any help would be appreciated thank you!
    public class Frame2UserInput extends JPanel {

    private static JTextArea input;
       private static JButton draw;

        Frame2UserInput() {

                   input = new JTextArea(2,35);
                  add(input);
                   draw = new JButton("Draw!");
                  draw.addActionListener(new DrawListener());
                  add(draw);
   }

   private static class DrawListener implements ActionListener {

            private String userInput;
            GraphicPanel pannel1 = new GraphicPanel();

            public void actionPerformed(ActionEvent event)  {

           if (event.getSource()==draw) {

            userInput = input.getText();
            System.out.println(userInput);

            if (userInput.equalsIgnoreCase("penup")) {

            }

            if (userInput.equalsIgnoreCase("pendown")) {

            }

            if (userInput.equalsIgnoreCase("turnleft")) {

            }

            if (userInput.equalsIgnoreCase("turnright")) {

            }
            if (userInput.equalsIgnoreCase("black")) {

            }

            if (userInput.equalsIgnoreCase("green")) {

            }

            if (userInput.equalsIgnoreCase("red")) {
                pannel1.drawLine(Color.RED,0,0,100,100);
            }

            if (userInput.equalsIgnoreCase("reset")) {
                pannel1.clear();
            }

                        }
                }   }    }

Comment: 1. Swing isn't thread safe, all of this code needs to be executed on the EDT.  2. Normally you'd supply a reference of the class you want to call methods on, try adding a parameter of the class to the ctor of `Frame2UserInput`.
Comment: This kind of stuff is OOP 101 that should be sorted out long before attempting GUIs.
Comment: @WreckTangle You can use Observer pattern. For example: [updating change to another class](http://stackoverflow.com/questions/35876982/updating-change-to-another-class/35877451#35877451)
Answer: Take it as a parameter.
final GraphicPanel pannel1 = new GraphicPanel();
final Frame2UserInput pannel2 = new Frame2UserInput(pannel1 );

New Frame2UserInput constructor:
Frame2UserInput(GraphicPanel pannel1) {
   .
   .
   draw.addActionListener(new DrawListener(pannel1));
}

And new DrawListener constructor
DrawListener(GraphicPanel pannel1) {
    .
    .
    //Don't need this just use pannel1
    //GraphicPanel pannel1 = new GraphicPanel();
}

Comment: While this works (and conceptually I don't have an issue), it does expose `GraphicPanel` to unwanted change (what stops `Frame2UserInput` from adding/removing content or changing other properties) as well as tightly couples the code. These are just things that the OP needs to know when balancing their choices ;)
Comment: Agree, this is not the best design. I prefer Observer pattern as you mentioned in your answer. Here I just find a workaround with minimum change.
Answer: Basically, you want to start by setting an Observer Pattern, where DrawDriver listens to events from Frame2UserInput and makes changes to the instance of GraphicPanel.
This is also a conceptual example of the Model-View-Controller paragdigm.
The idea here is, neither GraphicPanel or Frame2UserInput have any idea of each other (and shouldn't really know about DrawDriver either).  DrawDriver is the bridge between the two, passing information between them as needed.
This will decouple your code and make it easier to modify and expand in the future, as changing one part shouldn't change how the other parts work.  To this end, you should make heavy use of interfaces to define the core contractual logic between these classes.
Also, don't rely on static, it's not helping you and you should learn to work without it
Comment: 1+ for design pattern and warning on static members :)
Answer: The dot operator
object.objectMethod();

the method must be public, or protected if there is a level of inheritance & polymorphism going on
For your case, few pointers:

firstly your spelling of panel is wrong.
MEANINGFUL NAMES for your variable names is so very important

panel1 or panel2 are awful names for variables.
Next, you'd need to have an object in DrawDriver class to call an objects method from
Answer: You can

make it as a parameter of your class
make it as a static class member(Not recommend). But it save your time in some degree if you operate on the same canvas.


Event analysis after treatment

Question: Hey I am doing an analysis,
I have 12 securities with weekly observations regarding spread and volume.
There was a certain law which I want to test if it significantly influenced liquidity once it got implemented.(liquidity independent variable)
Hence, I am testing whether the spread increased or the volume declined after the implementation compared to the period before.
Consequently, I am doing an event window period for the time before the law and the time after.
My regression looks like this:
Liqi,t = a + b1 *spreadi,t +b2 * volumei,t + b3 * Dt * spreadi,t * b4 * Dt * volumei,t
i = security
t = time
b1 would be interpreted like if the spread goes up by 1 than liquidity should also go up by b1 (same for volume right?)
Dt is a time dummy which is 1 for everything from law and later, 0 for everything for the period before.
So how can I interpret b3 and b4 regarding liqudity?
Also if the signs of the coefficents b3 and b4 are different than from the signs of b1 and b2 what does that mean?
I have observations for the period before and for the period after. The number of firms is constant for all observations and there is no change in firms for the whole data set.
For my data set I only have firms that were subject to the law. That is why I created the time dummy variable to measure if the event lead to a significant change in liquidity. Since I don't have observations regarding firms not subject to the law the difference-in-differences estimate can not be applied.
All in all:
I try to write a regression that gives me the answer to the question whether liquidity changed after law by observing spreads and volume
It got pretty long. Thank you for your help!

If it helps you I attached a sample of my Data set.
Liquidty is the dependent variable NOC = volume Pread = spread DummyNOC 0 or 1 * NOC for i on time t (same for spread)
is it wrong that I already included in the excel the result (ie. 1 * 1.62)? Or do ihave to keep the dummies only 1 and 0 and tell stata to multiply it?
Answer: You could create groups from the data, one group for before the change and one for after the change.
Then you could use an "F test about the variance" to test whether the variance between the two groups are significantly different.
Likewise, with the same two groups, you could use ANOVA to test whether the mean of the "volume" variable has changed significantly.
Comment: I added the dataset so you can have a better understanding of my problem
Comment: thank you for your help!

using the F-Test would not give me information whether liquidity has gone up or down right?

since I only have two variables is an independent-samples t-test better?

and what about my suggested regression?
are the dummies bad or is the approach wrong?
Comment: The F test alone would not, but if the test did show a significant difference you can assume the group with the greater mean is greater. Or you could perform a tukey post hoc if you really wanted to. Though with only two variables I don't see why this would be necessary. I think the dummy vars Dt would work (it's hard for me to thing through it without data), but those binary variables are essentially creating a factorial ANOVA, which I use more and am more comfortable with.

Event analysis after treatment


What is the following derivative: $\frac{d^k}{dx^k} x^n$

Question: I believe it will be of the form $n(n-1)(n-2)\cdots$ (with a string of $k$ terms) multiplied by $x^{n-k}$ but I can't understand how to find the length of this string.
Comment: Do it for small values of $k$, then find a pattern and attempt induction. For example, how many terms in the string when $k=1$? How many when $k=2$?
Comment: The third in the string should be $n-2$.  You can express the result using factorials.
Answer: If you claim that the number of factors is $k$ (as it is, since you derive $x^n$ $k$ times) then the coefficient is simply $n(n-1)\cdots(n-k+1)$. In this way
$$
\frac{d^k}{dx^k}(x^n)=n(n-1)\cdots(n-k+1)x^{n-k},
$$
for $n\in\mathbb{N}$ and $0\leq k\leq n$.
Answer: Hint
$${d^k\over dx^k}x^n={n!\over (n-k)!}x^{n-k}=k!\binom{n}{k}x^{n-k}$$


Quais as diferenças entre os estados dos arquivos do Git (untracked, unmodified, modified, staged)?

Question: Gostaria de saber, quais as características e diferenças entre os estados dos arquivos durante seu ciclo de vida no Git.
Estados:

untracked
unmodified
modified
staged
Answer:
Acho que podemos entender melhor estes estados criando um projeto do zero. Primeiro vou criar uma pasta chamada "projeto" e entrar nela:
$ mkdir projeto
$ cd projeto/

Por enquanto ela está vazia. Vou usar git init para inicializá-la como um repositório Git:
$ git init
Initialized empty Git repository in C:/Users/usuario/projeto/.git/

Agora vou começar meu projeto adicionando um arquivo (com o conteúdo "hello world" dentro dele), e em seguida verificar o status deste arquivo com git status:
$ echo "hello world" > hello.txt

$ git status
On branch master

Initial commit

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        hello.txt

nothing added to commit but untracked files present (use "git add" to track)

Veja que o arquivo hello.txt está listado como "Untracked files". Ou seja, o Git não está "trackeando" este arquivo. Podemos dizer que o Git não está controlando o ciclo de vida deste arquivo (e confesso que não sei se "ciclo de vida" é o termo tecnicamente correto).
É como se o Git dissesse: "Vi que que tem um arquivo aqui, mas eu não sou responsável por ele".
Para que o Git passe a ser responsável por este o arquivo, use git add:
$ git add hello.txt

$ git status
On branch master

Initial commit

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)

        new file:   hello.txt

Agora o arquivo está listado como "Changes to be committed", ou seja, ele fará parte do próximo commit. Diz-se então que o arquivo está "staged" (ou está na staging area, ou no index, ou no cache, não sei pra que tantos nomes para a mesma coisa).
De qualquer forma, repare que há inclusive uma instrução para retirar o arquivo do staging area (use "git rm --cached <file>..." to unstage - nas versões mais novas do Git a mensagem é "git restore --staged <file>... to unstage").
Em vez de unstagear o arquivo, vamos commitá-lo (adoro esses estrangeirimos...)
$ git commit -m"Primeiro commit"
[master (root-commit) ccc2a05] Primeiro commit
 1 file changed, 1 insertion(+)
 create mode 100644 hello.txt

$ git status
On branch master
nothing to commit, working tree clean

Após o commit, o git status diz que não há o que mostrar ("nothing to commit, working tree clean"), ou seja, todos os arquivos estão atualizados: o seu conteúdo bate com o último commit, não houve nenhuma alteração). Pode-se dizer então que o arquivo hello.txt está unmodified (não há modificações, se comparado com o último commit).
Então vamos alterar o arquivo, adicionando algum texto ao seu final:
$ echo "Novo texto" >> hello.txt

$ cat hello.txt
hello world
Novo texto

Agora o arquivo tem 2 linhas, vamos ver o que o git status mostra:
$ git status
On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   hello.txt

no changes added to commit (use "git add" and/or "git commit -a")

Ele mostra que o arquivo está modified, ou seja, foi modificado, se comparado ao último commit. Mas repare que esta modificação não estará no próximo commit, pois o arquivo está listado como "Changes not staged for commit" e no final ainda aparece a mensagem "no changes added to commit".
Eu posso ver as modificações usando git diff:
$ git diff
diff --git a/hello.txt b/hello.txt
index 3b18e51..7d34649 100644
--- a/hello.txt
+++ b/hello.txt
@@ -1 +1,2 @@
 hello world
+Novo texto

O trecho +Novo texto indica que esta linha foi adicionada (+). Para a alteração ficar disponível no próximo commit, basta usar git add:
$ git add hello.txt

$ git status
On branch master
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        modified:   hello.txt

Repare que o arquivo continua com status "modified" (afinal, ele está diferente do último commit), mas agora estará no próximo commit ("Changes to be committed").
Ou seja, ele está ao mesmo tempo modified (diferente se comparado ao último commit) e staged (pois está na staging area).
O fato de estar na staging area faz com que git diff não mostre mais as diferenças:
$ git diff
(não mostra nada)

Para saber o que mudou, deve-se usar a opção --cached, que compara o staging area (também chamado de cache) com o último commit:
$ git diff --cached
diff --git a/hello.txt b/hello.txt
index 3b18e51..7d34649 100644
--- a/hello.txt
+++ b/hello.txt
@@ -1 +1,2 @@
 hello world
+Novo texto

A propósito, --cached é equivalente a --staged - ou seja, compara os arquivos que estão staged com o último commit.

Resumindo:

untracked: o git não sabe nada a respeito do arquivo. Quer dizer, até sabe que ele existe, mas não vai controlar seu ciclo de vida
staged: os que estão no staging area/index/cache. São os que estarão no próximo commit
modified: os que foram alterados, se comparados ao último commit
unmodified: os que não foram alterados

Por fim, vale lembrar que a documentação diz que na verdade só existem dois estados: tracked e untracked. Ou seja, ou o Git está controlando o ciclo de vida do seu arquivo (tracked) ou não está (untracked). Sendo que um arquivo no estado tracked pode estar staged, modified ou unmodified.
Comment: Muito bom! Excelente!
Comment: @rbz Obrigado! Esqueci de comentar que usei git 2.11.0 para windows, então pode ser que algumas mensagens sejam diferentes em outras versões (não tenho certeza). Mas a ideia é a mesma, independente disso.
Answer:
Untracked é o estado onde o arquivo ainda não foi 'rastreado'.
Unmodified é o estado onde o arquivo não sofreu alterações em relação a sua referência.
Modified é o estado onde o arquivo sofreu alterações em relação a sua referência.
Staged é o estado onde o arquivo já foi endereçado e aguarda ser transferido ao repositório.

Observe neste link que comandos são utilizados em cada etapa do processo até que o arquivo seja adicionado ao repositório. Acredito que ajudará no entendimento.
No mais, seguem links de boas referências: Git - Book | Git - Guia Prático
Comment: Caio, a `staged`, o que seria o "endereçado"? Seria tipo "marcado para envio"?
Comment: Seria algo semelhante a isso. O Stage pode ser visto como uma área temporária na qual o arquivo permanece até que não seja enviado ao repositório. Irei adicionar mais um link a resposta, ok?


Golang asks for GOROOT in liteIDE

Question: I use Go on Windows 10 and am having problem.
I've tried running a basic 'Hello, world' Go program using LiteIDE but when run it returns next error:
<code>C:/Program Files (x86)/Go/bin/go.exe build [C:/Users/MyUser/Desktop/Go_projects]
go: cannot find GOROOT directory: c:\go
Error: process exited with code 2.
</code>
A lot of people recommend not setting up GOROOT path variable, meaning Go should work fine without it, yet LiteIDE doesn't.
Just in case, I had set up GOPATH to C:\Program Files (x86)\Go\bin
Comment: @nipuna: `go env` will always show a `GOROOT` value, because that is the root of the Go installation.
Comment: The behavior described is what happens when you have GOROOT set incorrectly in your environment. Aside from checking your user env variables, check the ide setting to see how it is calling the Go binary. Did you install Go using the official packages?
Comment: run `go env` command in your terminal and see whether GOROOT is there or not. and please share the output. If its not in there, update your bash_profile file with `export PATH=:$PATH`. Installation directory is normally `/usr/local/go/bin`
Comment: @JimB looks like I do have problem with GOROOT since I didn't set it but in this article https://dave.cheney.net/2013/06/14/you-dont-need-to-set-goroot-really it says not to set a GOROOT, which is leaving me confused. And yes, I installed Go using official packages.
Comment: If the go installation is in `C:/Program Files (x86)/Go`, then GOROOT is set _somewhere_, but we unfortunately cannot find that for you. My best guess would be in the ide settings. Verify that you can at use Go normally  from the cli, and `GOPATH` should most definitely _not_ be set to `C:\Program Files (x86)\Go\bin`. Sometimes when after missing up the installation when starting out it's easier to just clean out everything and start again.
Answer: I had removed GOROOT from win32.env in liteIDE settings and everything seemed to just work.
View > Options > LiteEnv > double-click win32.env > delete GOROOT
I don't know the longlasting consequences of this action yet, but for now this solution seems to work.

Golang asks for GOROOT in liteIDE


Número máximo de carateres em uma String e StringBuffer em Java

Question: Qual o limite máximo de caráteres que os tipos String e StringBuffer suportam em Java?
Answer: O limite teórico é o valor máximo de um int - 2^31-1 - já que as strings são representadas internamente por um array de char, e a forma de indexar esse array é através de um int. Entretanto, o que vai limitar na prática o tamanho das strings é o espaço físico (memória RAM), que certamente será bem menor que esse limite teórico.
Não sei como o StringBuffer é representado internamente, mas ainda que ele possua representação capaz de suportar mais do que o limite mencionado, a forma de se referir a um caractere em particular (charAt) ainda utiliza um int, de modo que pode-se considerar o mesmo limite na prática.
Comment: @Cold isto pode lhe ajudar: http://pt.stackoverflow.com/questions/15510/o-que-imut%C3%A1vel-realmente-significa/15512#15512
Comment: @wryel Quando disse que não sabia como `StringBuffer` funcionava internamente, me referia à possibilidade dele usar arrays de strings por exemplo - o que implicaria numa capacidade total teórica maior que o tamanho de um `int`. Mas você está certo, é apenas um array de caracteres mesmo, com "espaço sobrando".
Comment: Uma `String` é imutável, de modo que você pode compartilhar referências da mesma livremente. Um `StringBuffer` e `StringBuilder` são mutáveis, você edita a sequência de caracteres à vontade para - só no final - transformar em uma `String` (*se* precisar). O `StringBuffer` é *thread-safe*, e o `StringBuilder` não é. Mais informações sobre seu funcionamento [nessa pergunta relacionada](http://pt.stackoverflow.com/q/7671/215)
Comment: para documentação: StringBuffer é implementado usando arrays, e de acordo com a especificação, no que diz sobre arrays: `The variables contained in an array have no names; instead they are referenced by array access expressions that use non-negative integer index values`.
Comment: @Cold Nunca refleti profundamente sobre o assunto, simplesmente tomei que "não existem métodos na API de `String` para modificá-la" e fiquei satisfeito com isso... Agora, na prática, pode ser que existam otimizações feitas pela JVM ou coisas do tipo. Se tiver interesse, sugiro abrir uma pergunta separada, assim quem tiver mais conhecimento no assunto poderá responder (eu não tenho).
Comment: Assim qual seria a principal diferença entre ambos?
Comment: Bom. Agora o assunto pra mim é perceber a baixo nível o que significa um `String` ser imutável.
Comment: Questionando o verdadeiro significado de `String` ser imutavel @mgibsonbr?
Answer: O limite maximo que ambos suportam é 2^31- 1 (ou aproximadamente 2 bilhões)
O StringBuffer te da eficacia na concatenação de String
Comment: http://www.oficinadanet.com.br/post/2124-java_diferencas_entre_as_classes_string_stringbuffer_e_stringbuilder
Comment: E qual a diferença entre ambos se possuem a mesma capacidade?
Comment: http://www.devmedia.com.br/diferencas-entre-string-stringbuilder-e-stringbuffer-em-java/29865


Sample Autocovariance

Question: Suppose that we have a wide sense stationary process $X_t$.
Without loss of generalization assuming that mean is 0, lag one covariance is $\mathbb{E}[X_t X_{t-1}]$.
In order to estimate the covariance, I use the sample version:
$$\frac{1}{N}\sum_1^N X_t X_{t-1}$$
What's bothering me here is the consecutive products i.e. $X_t X_{t-1}$ and $X_{t-1} X_{t-2}$ are not independent due to the shared $X_{t-1}$ component.
It looks like this is a mean of non-IID samples.
Question: Is this something to worry about?
What are some issues to be careful about (especially for the small sample case)?
Answer: We need the stochastic process $\{Z_t\} = \{X_tX_{t-1}\}$ to have a constant mean  and be "ergodic for the mean".
If these properties hold then the Birkhoff–Khinchin Ergodic Theorem states that
$$\frac 1T \sum Z_t \to_p \mu_z = \text{Cov}(X_t,X_{t-1})$$
Let $\gamma_k \equiv E(X_tX_{t-k})$ . If
$$\sum_{k=0}^{\infty} |\gamma_k|< +\infty$$
then the process $\{X_t\}$ is ergodic for the mean.
Assume that it is. Then continuous transformations of $\{X_t\}$ are also ergodic for the mean. Then (to be a little pedantic), $W_t \equiv LX_t = X_{t-1}$ is ergodic for the mean because the transformation induced by the lag operator is continuous. Also, multiplication and addition are continuous transformations. Then
$$Z_t = X_t \cdot W_t = X_tX_{t-1}$$
is ergodic for the mean.
Moreover, $\{Z_t\}$ has a constant mean since
$$E(Z_t) = E(X_tX_{t-1}) = \text{Cov}(X_t,X_{t-1}) = \gamma_1$$
which is constant and does not depend on the index $t$, due to $\{X_t\}$ being covariance-stationary.
So for asymptotic consistency of the sample covariance, the essential condition is the absolute summability condition on the autocovariance series of $\{X_t\}$.
I note that the fact that autocovariances of $\{X_t\}$ "do not depend on the index only on the distance", does not make them absolutely summable. We can conceive of a situation (although perhaps counter-intuitive both from a space-perspective and from a time-perspective), in which as the distance increases, the autocovariance increases also.
Comment: @CagdasOzgenc Informally speaking, existence of dependence may be good for prediction - but it is bad for estimation, since it reduces the information contained in the sample. That said, we "automatically" take it into account, because the variability of our $Z$ sample is partly "self-induced/explained".
Comment: Thanks. Interesting take on my somewhat vague question. When we look at the autocorrelation structure of $Z_t$ for some process $X_t$, it is very likely to find significant terms due to the dependency. Basically what I am trying to ask is why don't we have an estimator that takes this into account instead of simple mean estimator?

Sample Autocovariance


Get form parameters from multipart request without getting the files

Question: I'm looking for a way to get the form parameters of a HTTP multi-part request in a Servlet-filter without uploading files (yet).
request.getParameterMap() returns empty. I understand this is because of the request being multi-part.
I've looked at commons.HttpFileUpload but this seems to be overkill for my situation. In this filter I'm only interested in the normal parameters, and don't want to handle the file-upload yet.
Edit: the main problem is that I need to have an intact HttpRequestObject further down the filter stack. The HttpFileUpload seems to consume part of the request data (probably by using the data stream object and closing it again.)
Comment: Quote: "i need to have an intact HttpRequestObject further down the filter stack": why and what for?
Comment: because i am working on a fairly large existing project which relies on having the HttpRequest object, which i cant easily change.
Comment: That's too little information. Even without commons fileupload you'd have the same problem. Please elaborate.
Answer: It's certainly not overkill, it's the right way and always better than writing the parser yourself. The Apache Commons FileUpload is developed and maintained for years and has proven its robustness in handling multipart/form-data requests. You don't want to reinvent the wheel. If you really want to do it (I don't recommend it), then read on the multipart/form-data specification and start with reading the HttpServletRequest#getInputStream() (warning: this is a mix of binary and character data!).
You can if necessary also write a Filter which makes use of Apache Commons FileUpload under the hood and checks every request if it is multipart/form-data and if so, then put the parameters back in the request parameter map with help of Commons FileUpload and put the uploaded files (or exceptions) as request attributes, so that it's finally a bit more transparently in your servlet code. You can find here a basic example to get the idea.
Hope this helps.
Answer: Just to add to the answers already provided - I had a very similar problem in that I was trying to add some CSRF validation to our existing web app. We decided to include a special token in each form using some JS and add a servlet filter to check that the token existed (therefore a generic, isolated solution).
The servlet would check if the token was present but broke for every form that provided a file upload option. Hence I landed at this page frequently while doing some googling.
The work around we used (while attempting to avoid any dealings with the uploaded files) was to get some JavaScript to add the token as a GET parameter, i.e. We modified the form's action URL to include the token and therefore could use the HttpServletRequest.getParameter() method for the token (and only the token).
I have tested this in IE, FF and Chrome and all seem to be happy.
Hope this helps anyone who also finds themselves in a similar situation.
Comment: problem with this approach is that the token stays in the URL so https can not protect it http://docs.spring.io/spring-security/site/docs/current/reference/html/csrf.html#csrf-include-csrf-token-in-action
Comment: just ran into this exact scenario. Adding csrf token as a get param did the trick
Answer: The Oreilly Servlets website has some sample code which you can download customise and use. This includes MultipartRequest which sounds like it does what you require, it un-boxes a multipart request and allows access to the parameters and the files separately.
Comment: The c.o.s.MultipartRequest is a well known class indeed.
Answer: Commons FileUpload provides a mechanism to read request params from a multipart form upload.
There's a really great example of how to grab the request parameters here:
How to upload files to server using JSP/Servlet?


How to check and disable one of the checkboxes in a form?

Question: I found this snippet in the code for a form ($roles is an array of roles).
$form['config_options']['roles'] = [
  '#type' => 'checkboxes',
  '#options' => $roles,
];
// Check and disable the 'authenticated' role.
$form['config_options']['roles'][RoleInterface::AUTHENTICATED_ID] = [
  '#default_value' => TRUE,
  '#disabled' => TRUE,
];

What this does is to check the checkbox for the 'authenticated' role, and then disables it (so it can't be unchecked). This works.
I have a very similar requirement.  In a form I have checkboxes for a bunch of fields, where $fields is an array of them. The fields are from user profile. I want to check and disable the 'email' field.
Based upon the code snippet above, I think the solution should be something like this:
$form['config_fields']['fields'] = [
  '#type' => 'checkboxes',
  '#options' => $fields,
];
// Check and disable the 'email' field.
$form['config_fields']['fields'][ -- What to put here? -- ] = [
  '#default_value' => TRUE,
  '#disabled' => TRUE,
];

However, I have no idea what to put in the selector where I've written " -- What to put here? --".
Based upon a comment from Clive I've tried to examine $fields. This is what the devel Default variables-dumper gives me (using short array syntax):
$fields = [
  …
  mail => stdClass Object (
    [__CLASS__] => Drupal\Core\StringTranslation\TranslatableMarkup
    [translatedMarkup:protected] =>
    [options:protected] => []
    [stringTranslation:protected] =>
    [string:protected] => Email
    [arguments:protected] => []
  )
  …
]

So it is a bit more complex than the example given in Clive's comment (e.g. it is an object, not an array (as implied by Clive).
So far I've tried 'Email'.
$form['config_fields']['fields'] = [
  '#type' => 'checkboxes',
  '#options' => $fields,
];
// Check and disable the 'email' field.
$form['config_fields']['fields']['Email'] = [
  '#default_value' => TRUE,
  '#disabled' => TRUE,
];

This does not work.
Help will be appreciated.
Also, if I am barking up the wrong tree, and there exists another way to do this, I shall equally appreciate an alternative solution.
(I am using Drupal 9).
Comment: It would be the relevant array key from `$fields`. e.g. if `$fields = ['foo' => 'Foo', 'bar' => 'Bar'];`, and you want to check `Bar`, the key is `bar`
Comment: @Clive Thanks for the suggestion. I've expanded `fields` and updated the question with the expanded version.  It looks like an object, while your suggestion implies that it is an array. It is still not clear to me what to use as key. Following your suggestion, I think it should be 'Email', but 'Email' does not work for me.
Answer: OK, it turns out that the key is simply the machine name of the field you want to check and disable.  This works:
$form['config_fields']['fields'] = [
  '#type' => 'checkboxes',
  '#options' => $fields,
];
// Check and disable the 'mail' field.
$form['config_fields']['fields']['mail'] = [
  '#default_value' => TRUE,
  '#disabled' => TRUE,
];


How to find time differences between two time in c#

Question: I have a form that gives me time_in and time_out hours of all staff members. Time_in and time_out also  sql tables. Basically form returns the database table values. What I would like to do is to display the work hour of them. Work hour can be get the differences from time_out to time_in. I have two text boxes that display time in and Time_out. The third one should display work hour.
Here is what I have for time_out value for Wednesday:
//Selected TimeOutWednesday
        SqlCommand TimeOutWednesdayMain = cs.CreateCommand();
        TimeOutWednesdayMain.CommandText = "SELECT CONVERT(VARCHAR(8), time_out, 108) AS time_out  FROM job_punch_card  WHERE emp_key='" + listBoxNames.SelectedValue.ToString() + "'and punch_day= DATEADD(week, DATEDIFF(day, 0, getdate())/7, 2)";
        Object TimeOutWednesdayMainTemp = TimeOutWednesdayMain.ExecuteScalar();
        txtTimeInWed.Text = TimeOutWednesdayMainTemp.ToString();

This code gives me time_out for wednesday for the selected user from my listbox. I have the same code for time_in as well. What I couldn't do is to figure out how find the work hour? How can I display their work hour in a label or text box like I have above?
Comment: Your code is open to SQL injection
Comment: Why not use a query that pulls the time in, time out, and the difference in hours?  Also I assume `time_in` and `time_out` are `DateTime`, is that correct?
Comment: Correct. How to get the difference in sql ?
Comment: In SQL, you can get the hour by doing `DATEPART(HOUR, time_out)`
Answer: The DateTime types in OS are simples Integer values, you should make a arithmetical operation and get the Hours or TotalHours:

Hours get the integer portion of difference (dont lose parse your textboxes):

DateTime time_in;
DateTime.TryParse(time_in_TextBox.Text, out time_in);

DateTime time_out;
DateTime.TryParse(time_out_TextBox.Text, out time_out);

int hours = (time_in - time_out).Hours;

TotalHours get a double value with the exact difference in DateTimes:

DateTime time_in;
DateTime.TryParse(time_in_TextBox.Text, out time_in);

DateTime time_out;
DateTime.TryParse(time_out_TextBox.Text, out time_out);

double totalHours = (time_in - time_out).TotalHours;

Comment: That's exactly what I am looking for. All the answers work but this one seems easy to me. Thanks guys!
Answer: I don't know if you're set on doing the time difference in SQL, but here's how you can do it in C# - though you would need to convert to a datetime object, not just a time (at least in this example).
See https://dotnetfiddle.net/DSuHmH :
using System;

public class Program
{
    public static void Main()
    {
        DateTime startTime = new DateTime(2014, 1, 1, 1, 5, 1);
        DateTime endTime = new DateTime(2014, 1, 1, 5, 10, 55);

        TimeSpan ts = endTime - startTime;

        Console.WriteLine(ts); // returns "04:05:54"
    }
}

to specifically get the hours use:
ts.Hours;

To accomplish in sql you can do something like the following - again you'll need to convert your time to a datetime and then you can utilize the datediff function.
See: http://sqlfiddle.com/#!6/b7a7c/7
select convert(datetime, '2014-01-01 ' + startTime, 101),
       convert(datetime, '2014-01-01 ' + endTime, 101),
       datediff(
           hh,
           convert(datetime, '2014-01-01 ' + startTime, 101),
           convert(datetime, '2014-01-01 ' + endTime, 101)
       )
from test

Note that in the examples above i'm using an arbitrary date to accomplish creating a valid datetime
Comment: Note that `TimeSpan.Hours` get the hours part of the `TimeSpan`.  The OP might want to use `TimeSpan.TotalHours` instead.
Comment: @Karatay you can still do that in this case it was just meant to be used as an example
Comment: Yeah. I have to retrieve the start and end time from sql server.
Answer: You can get the time difference, in minutes, using this:
DATEDIFF(MINUTE, @punch_day, @time_out)

This will return the total minutes between the two dates. If you need hours, divide it by 60. If you need days, then divide it by (60*24). Should you need weeks, divide it by (60*24*7). If you need years, divide it by (60*24*365). You get the idea, I hope!
Check out this example:
DECLARE @StartDate DATETIME
DECLARE @EndDate DATETIME

SELECT @StartDate = '12/9/2014 11:04am'
SELECT @EndDate = '12/10/2014 1:38pm'
SELECT DATEDIFF(MINUTE, @StartDate, @EndDate) AS TotalMinutes,
       DATEDIFF(MINUTE, @StartDate, @EndDate) / 60 AS TotalHours,
       DATEDIFF(MINUTE, @StartDate, @EndDate) / (60*24) AS TotalDays

Comment: why not just use `datediff(hh....)` in this instance rather than calculating hours based on minutes?  is it partial hours you're looking at?
Comment: @Kritner -- He could, but I am assuming he'll want it in minutes so that he can calculate left over minutes using the modulus operator `%`. Plus this way, he's only doing one hit to the database server, vs one hit for hours, one hit for minutes, etc.
Answer: You should be able to do this in one query.  This also shows the best practices for dealing with ADO.Net including putting your sql objects in using statements and passing values into your query as a parameter.  Additionally you'll have to use ExecuteReader to get more than one value back and you can add checking for retrieving more or less than 1 row.
using (var cs = new SqlConnection("your connection string"))
{
    cs.Open();
    using (var command = cs.CreateCommand())
    {
        command.CommandText =
            @"SELECT
                  time_in,
                  time_out,
                  DATEDIFF(minute, time_in, time_out) As minutesWorked
              FROM job_punch_card
              WHERE emp_key=@EMPKEY
                    and punch_day= DATEADD(week, DATEDIFF(day, 0, getdate())/7, 2)";
        command.Parameters.AddWithValue(
            "@EMPKEY",
            listBoxNames.SelectedValue.ToString());

        using (var reader = command.ExecuteReader())
        {
            if (!reader.HasRows)
            {
                // There was no match for your key
            }

            reader.Read();

            DateTime timeIn = reader.GetDateTime(0);
            DateTime timeOut = reader.GetDateTime(1);
            int minutesWorked = reader.GetInt32(2);

            if (reader.Read())
            {
                // There was more than one match on key
            }
        }
    }
}


how to RewriteRule https in php mvc

Question: i am setting up https to my website but my mvc codes contradiction with RewriteRule https how to i merge them?
https :
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

mvc:
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-l
RewriteRule ^(.+)$ index.php?url=$1 [QSA,L]

Answer: try this
 RewriteCond %{HTTPS} on
 RewriteCond %{SERVER_PORT} 443
 RewriteCond %{SERVER_NAME} !^www. [NC]
 RewriteRule .* https://www.%{SERVER_NAME}%{REQUEST_URI} [R=301,L]

how to RewriteRule https in php mvc


Can someone explain the inclusion exclusion principle for me?

Question:
This is the inclusion exclusion principle. Can someone explain this to me? Why does it end up being a summation and where do the -1 and the l term come from?
Comment: The best way to understand the inclusion-exclusion is by checking a Venn diagram for $n = 3$ and then the generalization should follow. This [this](https://en.wikipedia.org/wiki/Inclusion%E2%80%93exclusion_principle) wikipedia article.
Comment: Or to use indicator functions. But the lack of context...
Answer: Here is the Venn diagram (for n=3) as suggested by a comment:

If one adds up $P(A)$, $P(B)$ and $P(C)$ then the grey and brown regions  will be considered twice, or, more precisely, the grey areas will be added twice and the brown region will be added three times.
To take care for the double addition, we may think the formula below to be true, but is it?
$$P(A\cup B\cup C)=^{\color{red}{???}}P(A)+P(B)+P(C)-(P(A\cap B)+P(A\cap C)+P(B\cup C)).$$
Now, we subtracted the brown region three times. That is , we did not take care of the brown region in the total. So we have to add it back again. Finally we get
$$P(A\cup B \cup C)=$$
$$P(A)+P(B)+P(C)-(P(A\cap B)+P(A\cap C)+P(B\cup C))+P(A\cap B\cap C).$$
This thread of thinking may be extended to 4 sets and any further.


Untracable SSH connection

Question: This question is for the security Gurus.
Where i come from, we have a free online space, however some neighboring countries and friends and bloggers are getting arrested frequently for blogging .
I am trying to help some activists and friends in different ways to either bypass censorship and to protect their identities online
I am doing research on how to make an untracable ssh connection - we are thinking an ssh tunnel/proxy
I would appreciate any advice, tutorial, links you can share.
I am also writing some more general articles for bloggers - you suggestions and ideas are welcomed
here was my first entry couple of days ago http://migh.info/2010/09/outsmarting-censorship-1/
I really appreciate the help
edit: the plan so far is to have the following set up :

establish three ssh bridging reverse tunnels: a->b (normal)  b->c(tor enabled)  c ->d (normal)
all hosts a,b,c,d need to be publicly crowded hosts and some banana countries
on host e run a perl script that interacts with ssh connecting to d, the perl script input/output would be directed to http
on f run an app that interacts with the perl script, so by now, your connection
a->b (normal)  b->c(tor enabled)  c ->d (normal) d->e (perl connection) e->f (https connection, tor and proxy enabled)

so now, between a and f, there is different protocols, different connectivity and traversing thru so many points, that the forensics guys will be searching for a needle in the haystack. to be really silly, we could continue to use other protocols and ip package mangling in between. that would put the forensics experience to the testing.
Comment: @James: Warning Hypocrisy detected!! You're telling someone in another country what to do?! Is that not meddling with meddling with another counties politics?
Comment: I am from Lebanon - we have 0 censorship - like really zero - but friends/bloggers in arab countries get arrested and beaten for stuff as simple as posting an image of the king house or making a comment.- i don't consider it as mingling in other countries business, more like trying to help non-techies get protected either via simple easy tips or by setting up more advanced stuff for them to use
Comment: Sounds like China...
Comment: or North Korea or Iran.  I'm guessing Iran from the name.
Comment: There is also the issue of whether you should be meddling in another countries' politics at all whether you agree with them or not.  In most free countries we elect and entrust people to make decisions based on the full facts and consequences rather than uninformed individuals going off on one based on some moral crusade.
Comment: nice try but I didn't tell him what to do, merely what his own country tells him what to do :)
Answer: You are starting from several false assumptions - that risk can be eliminated (it can only be mitigated), that censorship can be 'outsmarted' (it can merely be evaded) and that you are better informed than the people implementing the censorship.
You are requesting information on how to implement functionality to protect anonymity but you expect to get all the answers you need here? And once you have an answer you like, you're going to start telling people that you can protect their identities when they are doing something which would be considered illegal in their home countries?
I'd hate to have it on my conscience that I made some suggestion to protect people and it either encouraged them to put themselves at greater risk or was not applicable to their circumstances. Particularly given the potential consequences.
While your efforts seem to be well motivated, you need to be more realistic about your objectives, stop underestimating the opposition and accept that there is no quick route to knowledge.
Comment: +1, not to mention that since it's posted on the net, the people doing the censorship can read-up on how you're getting around it just as easily as you can.
Answer: Besides the moral part mentioned above (I totally agree), just a creation of an ssh tunnel is not a solution. You cannot see the content, but you can still be aware of a lot of other things (such as destination, traffic etc).
A solution could be something like TOR system (http://www.torproject.org/) ...
But don't forget that it's "You VS Government" ... so be careful :>
Comment: @Michael: The reality is that in nearly every country it is otherwise. In some countries more, in others less, but governments are always more powerful and usually more dangerous than the citizens.
Comment: @Sirex: You're such an optimist! Reminds me of the people that say the USA is a democracy, and stare blankly when I say "representative republic". But as we say in ServerFault, this is an HR issue, not a technology problem :-)
Comment: People should not fear their government, their government should fear the people.
Comment: Sven is right.  "Government should fear the People", where "People" is capitalized, and the People are united as one against the Government.
Comment: "of the people, by the people, for the people".... i thought that *was* government ;-/
Comment: first time i've ever been called that, things are looking up !
Comment: +! for the "HR issue not a tech problem" comment.
Answer: The only solution that seems to obfuscate connections that I've heard of is TOR. Even then I remember reading grumblings that there were ways to trace certain kinds of data.
All connections, given enough persistence, can be traced. They have to be if you expect to make a connection that expects return of data.
If you don't own the medium transmitting the data, you have no idea who is tapping what data in between you and the destination. The "land of the free" in the good ol' USA has projects that tap data for certain agencies. Books have been published discussing it.
Anyway, TOR basically bounces your data among all the different TOR nodes, obfuscating where the packets are originating from and heading to then bounces the return data through the TOR peer to peer network back again for a response. Slow as molasses but hides information that you send through it (as a proxy). You might want to invest some time in spreading word about TOR and encouraging others to create TOR nodes on their systems to help advance the cause.
It's also possible, depending on the situation, that if this is an advocacy/human rights thing that packages of data could be encrypted and sent to drop-boxes or special websites outside the country if allowed. But that just protects direct snooping. If you're talking about protecting messages it's possible to use steganography to hide text and data in graphics that are uploaded to places like flickr, for example, to share information.
No matter where you go there's going to be risk that you'll be traced, depending on records being kept by the network providers, law enforcement, and the servers on which the data comes to rest on.
Encrypting via SSH just encrypts data in transit. It does nothing to hide origin point and destination, and might not even hide how much data was being sent. There was an old joke (anecdote? True story? Don't know) where reporters knew when the US was about to launch a war in the mid-east because of the spike in pizza deliveries in the middle of the night to the Pentagon. The network owner/law enforcement may not know WHAT you're saying, but they know when you made the connection, and can approximate how much data you're transmitting, and that alone can raise red flags.
In conclusion there's no single answer to your question. It's a mix of techniques to obfuscate, but never completely hide, what is being done. If someone is getting specifically targeted they will be traceable; it's just a matter of dedicating enough resources to find them and sift through logs and get cooperation of the companies involved. And it's always going to be risky.
Comment: +1 "All connections, given enough persistence, can be traced"
Answer: A thing to throw out there as well is keep in mind how secure something needs to be.
My teacher once explained with a example:
Imagine your transporting critical data, and that data can't fall into the right hands.
then you need to keep in mind, how long will this data need to be secure ?
first of all, there's no such thing as indefinite, it doesn't exist. Sooner or later someone will find out a loophole or alike and the data will be compromised.
So as such, keep in mind how long it needs to stay a secret for. If your data is only valuable for a week then there is no point in the fact that it takes the opposition 2 weeks to access the data, your mission'll be achieved and afterwards it won't / shouldn't matter.
On the flipside of the coin, if your data needs to stay sensitive and safe for 5 years, then you need to make sure it won't be compromised in 1 month time.
Why do I say this ? Not to give you bright ideas, not to aid you, or stop you in your endeavour, but to give warning that you for one shouldn't over do it, but also, don't underestimate it.
If your data needs to be vital for a few months then make sure its safe enough to stand the test of time ....
Comment: If it can't fall into the right hands, whose hands does it? :-)
Comment: yhea I messed up with that one I see now :)

Untracable SSH connection


Try to conclude how much you have positive, negative and zero eigenvalue of $B^TAB$

Question: Let $A\in M_3$ some arbitrary positive definite matrices and

B=$\begin{bmatrix}
    1& 1& 0& 2\\
    2& -1& 1& 2\\
    1 & 0& 0& 1
   \end{bmatrix}$.

Conclude connection between eigenvalue of matrices $B^{T}AB$, how much it has zero eigenvalues, how much it has positive and negative eigenvalues.
I find that $rankB=3$ then if I use some eigenvector $x\in \mathbb R^4$ such that $x\not=0$ then exist some eigenvalue $\lambda$ that $B^{T}ABx=\lambda x$, if I multiply both side with $x^T$ I get this $x^TB^{T}ABx=\lambda x^Tx$
then if $Bx=y$ and if I write $x^TB^{T}ABx=\lambda x^Tx$ like this $y^TAy=\lambda x^Tx$ then since $y^TAy>0$, $x^Tx>0$ so then $\lambda>0$ but If I pick some eigenvector from $kerB$ then $\lambda=0$ so it can be one zero and three positive eigenvalue, what you think?
Comment: Whan you write that $y^TAy>0$, that's true only if $y\ne0$.
Comment: i know that $A=Q\Lambda Q^T$
Comment: yes with positive eigenvalues
Comment: Yes, but I have no idea how to prove,
Comment: Do we have any self-adjoint assumption on A?
Comment: Is $\Lambda$ a diagonal matrix?
Comment: At least $\boldsymbol B ^{\mathsf T}\boldsymbol {AB}$ has one $0$ eigenvalue, since its rank is at most $3<4$.
Comment: Yes I write that
Comment: Oh, sorry, I missed it.
Answer: From $A = Q \Lambda Q^\top$ with $\Lambda$ a diagonal matrix we know that $A$ is self-adjoint. Thus, we know that all eigenvalues are real. Moreover, $B^\top A B$ is self-adjoint:
$$
(B^\top A B)^\top = B^\top (B^\top A)^\top = B^\top A^\top B = B^\top A B
$$
Thus, the eigenvalues of $B^\top A B$ are also real. Moreover, since $A$ is positive definite, $B^\top A B$ is positive semidefinite and we know that the eigenvalues are all nonnegative, as you showed. This would be the formal argument.
Your remaining analysis is correct up to typos.


How to create dicts dynamically from iterating over a dictionary itself?

Question: I'm using Wikipedia's Pageviews API in Python and I'm trying to get time and views for a list of pages. My idea was to create a dictionary with the wikipedia page (eg. Rome) as key and all the attributes as a dict of values. Then I want to create a dict with the wikipedia page name as the name of the dict and for the key the time and for the value the views. I can't create one dynamically, however.
My code is:
dfs = []

import pageviewapi
emptylist = []

dictrome = {}
dictparis = {}
dct0 = {}

list = ['Paris', 'Rome']
for x in list:
    dct0[x] = pageviewapi.per_article('it.wikipedia', x, '20150101', '20210101',
                        access='all-access', agent='all-agents', granularity='monthly')
    if 'Rome' in dct0:
        for i in dct0.values():
            for p in range(0,len(i['items'])):
                dictrome[(i['items'][p]['timestamp'])] = (i['items'][p]['views'])
    else:
        for i in dct0.values():
            for p in range(0,len(i['items'])):
                dictparis[(i['items'][p]['timestamp'])] = (i['items'][p]['views'])

And the output I want, which is the one I have, is:
dictrome: {'2015070100': 890, '2015080100': 879, '2015090100': 971, '2015100100': 1097, '2015110100': 2259}
dictparis : {'2015070100': 482, '2015080100': 467, '2015090100': 371, '2015100100': 425, '2015110100': 408}

I just want to automate that "if" condition, because the pages in my list will be hundreds.
Answer: Let me first address the variable name list. Thing is, list is a build-in function to create lists. You are reassigning it, which should not be done in any circumstance.
So now to your problem. I don't know, why you wouldn't just use a result dictionary with the city as the key, like:
result[<city>]=result_dict
Anyways, you can accomplish what you want by using the globals() function, which returns a global symboltable as a dict. I don't recommend this but here is an example:
for i in range(0,number):
    globals()[f"var_{i}"]=i # or dict() in your case.

Comment: I could also use a result dictionary but how do I insert the variable city name?
Comment: You just generate the results as you did so far, but instead of storing it in `dictrome` for example you can do 'result[x] = result_dict' since x is the name of the city in the list you iterate over. Just define the `result_dict` instead of `dictrome`, `dictparis`, etc. Hope this gives you an idea.
Comment: I would overwrite result_dict every time right?
Comment: like what exactly is result[x]? should I instantiate it how?


Limit of sequence of improper integrals

Question: I have no idea. How to solve that
$$\lim_{n \to \infty } \int_0^{\infty} \! x^{\frac{n}{n+1}} \cdot e^{-x^2} \, dx. $$
Comment: What is the limit of $\dfrac n{n+1}$ as $n\to\infty$ ? As for the integral, let $t=x^2$.
Answer: Note that $f_n(x) = x^{n/(n+1)}e^{-x^2}$ is bounded above by $g(x) = (1+x)e^{-x^2}$ and $\int_{\mathbb{R}} g(x) dx$ exists as a real number. Hence, by dominated convergence theorem, we have
$$\lim_{n \to \infty} \int_{0}^{\infty} f_n(x)dx = \int_{0}^{\infty} \lim_{n \to \infty} f_n(x)dx = \int_0^{\infty} xe^{-x^2}dx = -\left.\dfrac{e^{-x^2}}2 \right \vert_0^{\infty} = \dfrac12$$
Comment: I think, i am  not allowed to use  teh dominated convergence theorem because we havent learned it yet.
Is there another option
Comment: @TED There is generally speaking not really an easier way to prove that we can interchange limit and integral. For some particular cases you may find your way out by explicitly calculating the sequence of integrals. In case you can not, the dominated/monotone convergence theorem is the way to go.
Answer: $f_n(x) = x^{\frac{n}{n+1}}\cdot e^{-x^2} \leq (x + 1)\cdot e^{-x^2}$ on $[0,\infty)$,
and $g(x) = (x + 1)\cdot e^{-x^2}$ is integrable on $[0, \infty)$ as well. So: $\displaystyle \lim_{x \to \infty} x^{\frac{n}{n+1}}\cdot e^{-x^2} = x\cdot e^{-x^2}$. So apply the dominated convergence theorem: $\displaystyle \lim_{n \to \infty} \int_{0}^\infty x^{\frac{n}{n+1}}\cdot e^{-x^2} dx = \int_{0}^\infty x\cdot e^{-x^2} dx = \dfrac{1}{2}$
Comment: I think, i am  not allowed to use  the dominated convergence theorem because we havent learned it yet.
Is there another option
Answer: In $\newcommand\eps\epsilon\let\geq\geqslant\let\leq\leqslant$case you want to avoid the Dominated Convergence Theorem, we can directly prove that
$$\lim_{n\to\infty}\int_0^\infty(x^{\frac n{n+1}}-x)e^{-x^2}=0.$$
Let $\eps>0$.
The idea is to show that $|x^{\frac n{n+1}}-x|e^{-x^2/2}<\eps$ for all $x>0$ and for sufficiently large $n$. The desired limit will follow from
$$\left|\int_0^\infty(x^{\frac n{n+1}}-x)e^{-x^2}\right|\leq\eps\int_0^\infty e^{-x^2/2}=\eps',$$
since $\int_0^\infty e^{-x^2/2}<\infty$.
Because $\lim_{x\to\infty}2xe^{-x^2/2}=0$ we can find $x_0>1$ (independent from $n$) such that
$$|x^{\frac n{n+1}}-x|e^{-x^2/2}\leq(|x^{\frac n{n+1}}|+|x|)e^{-x^2/2}\leq2xe^{-x^2/2}<\eps$$
for all $x\geq x_0$ and $n\in\mathbb N$.
For $x<x_0$ we can use $e^{-x^2/2}\leq e^0=1$. It remains to show that we can find $N$ such that for all $n>N$ and all $x<x_0$, $|x^{\frac n{n+1}}-x|<\eps$. We split this into a few cases.

$x\leq\eps^2$:
$$|x^{\frac n{n+1}}-x|=x^{\frac n{n+1}}(1-x^{1/n})\leq\eps^{\frac{2n}{n+1}}\cdot1\leq\eps\qquad\text{for all }n.$$
$\eps^2\leq x\leq1$:
$$|x^{\frac n{n+1}}-x|=x^{\frac n{n+1}}(1-x^{1/n})\leq1\cdot(1-\eps^{2/n})\leq\eps\qquad\text{for }n\text{ large enough},$$
since $\lim_{n\to\infty}\eps^{2/n}=1$.
$1\leq x\leq x_0$:
$$|x^{\frac n{n+1}}-x|=x-x^{\frac n{n+1}}\leq x_0-x_0^{\frac n{n+1}}\leq\eps\qquad\text{for }n\text{ large enough}$$
because $x-x^{\frac n{n+1}}$ is monotonically increasing for $x\geq1$ with derivative $1-\frac n{n+1}x^{-1/(n+1)}\geq0$ and because $\lim_{n\to\infty}x_0^{\frac n{n+1}}=x_0$.

Thus in either case we have the desired bound for $n$ large enough.
I believe this method can be generalized to prove the continuity of $\Gamma(t)=\int_0^\infty x^{2t-1}e^{-x^2}dx$ for $t>0$, without using the DCT.

Limit of sequence of improper integrals


Detecting unreaded mails in Maildir

Question: I like to detect unreaded mails in Maildir mailboxes. The subfolder "new" only content mails that are not already pulled from an MUA. But I want the unreaded mails within INBOX (no virtual subdirs). Any suggestions how this can be implemented?
PS: I do not enter the raw mail!
Using Debian Squeeze with Postfix and Dovecot (POP, IMAP).
Comment: What is the purpose of this? A script to report something? The way you asked the question, I could just answer: Use Thunderbird, it will display all unread mails in the Inbox.
Comment: The purpose is to show logged on control panel users how many unread messages they have. And I have not the option to connect via imap... only via Maildir.
Comment: I am quite certain that the IMAP library of the language you use for your control panel will have support for this kind of task, but I consider that a topic for StackOverflow.
Comment: don't force all down to script... when the server is not listen on 127.0.0.1 and firewall blocks soliloquy I can not connect via imap even if perl has an imap interface. I have to find a way with Maildir. Many programmers before me have tried expect Maildir.
Answer: You use Dovecot, so each file in the maildir folder that has :2,S near the end of the file name means it has been flagged as "Seen" or "Read".
More useful info about Maildir formats:
http://wiki.dovecot.org/MailboxFormat/Maildir
http://cr.yp.to/proto/maildir.html
An example file name from my Maildir:
1324304849.M312689P14620.subdomain.domain,S=24816,W=25415:2,Sa
Comment: Haha `find . -type f | grep -E ',[^,]*S[^,]*$'` finds all readed and `find . -type f | grep -vE ',[^,]*S[^,]*$'` find all unreaded. I use shell commands...
Comment: Obviously the above sets you on the right path, you may want to check with the stackoverflow guys for syntax examples for various languages. Perl, Python or Bash would probably be appropriate, just my opinion.
Comment: Awesome, glad it is working out for you :)

Detecting unreaded mails in Maildir


test for increasing correlation over time

Question: I have some data that tracks 2 variables over time. Let's call them A and B. So I have data on A and B for 15 years. I know how to calculate the correlation coefficient,  but my goal is to see if the correlation between the two is increasing over time. If anyone knows of a way to do this I would really a precise your input!
Comment: So, what is the problem? Calculate correlation across the whole time frame (saving in some structure both correlation coefficient and date/time stamp), which essentially will produce a time series. Then plot it and/or analyze analytically, depending on your needs.
Comment: You could estimate & plot a conditional correlation, see https://stats.stackexchange.com/questions/235442/investigate-correlation-conditional-on-a-threshold/299958#299958    and  https://stats.stackexchange.com/questions/203494/can-i-analyze-or-model-a-conditional-correlation/368228#368228,    https://stats.stackexchange.com/questions/239645/what-is-the-physical-significance-of-cumulative-correlation-coefficient/371412#371412
Comment: You could determine the correlation coefficient for subsets of time and then compare. 1-10 years
1-11 years
1-12 years
...and so on.
Answer: You can estimate a time-varying coefficient in a linear regression model. To do so, you could for example fit an interaction between one variable and
a set of splines as proposed here. As a complement to the example given in the previous link, here I show three examples for some simulated data. Three cases are considered: constant correlation and time-varying correlation (linear and quadratic).

The plot can be reproduced with the following R code.
<code>require(splines)
par(mfrow = c(1,3), mar = c(2, 2, 2, 2), las = 1, cex = 1.1,
                    oma = c(1,1,1,1))

# example 1: the true coefficient is constant
set.seed(123)
n <- 200
x1 <- rnorm(n)
x2 <- -0.7 * x1 + rnorm(n, sd = 0.5)
Bspline <- bs(seq(0, 1, len = n), df=2)
fit <- lm(x2 ~ Bspline * x1)
summary(fit)
plot(seq_along(x1), coef(fit)[5] + Bspline %*% coef(fit)[6:8],
     ylim = c(-1, 1), xlab = "index", ylab = "", type = "l",
     col = "red")
lines(rep(-0.7, n), col = "blue")
legend("topleft", col = c("blue", "red"),
       legend = c("true coefficient", "estimated coefficient"),
       lty = c(1,1), bty = "n")
mtext(text = "constant correlation", side = 3, adj = 0, font = 2)

# example 2: the true coefficient follows a linear slope
set.seed(125)
n <- 200
x1 <- rnorm(n)
x2 <- seq(-0.5, 0.8, len=n) * x1 + rnorm(n, sd = 0.5)
Bspline <- bs(seq(0, 1, len = n), df=2)
fit <- lm(x2 ~ Bspline * x1)
summary(fit)
plot(seq_along(x1), coef(fit)[5] + Bspline %*% coef(fit)[6:8],
     ylim = c(-1, 1), xlab = "index", ylab = "", type = "l",
     col = "red")
lines(seq(-0.5, 0.8, len=n), col = "blue")
legend("topleft", col = c("blue", "red"),
       legend = c("true coefficient", "estimated coefficient"),
       lty = c(1,1), bty = "n")
mtext(text = "time-varying correlation (linear)", side = 3, adj = 0,
      font = 2)

# example 3: the true coefficient follows a quadratic form

set.seed(127)
n <- 200
x1 <- rnorm(n)
x2 <- (seq(-0.5, 0.8, len=n)^2) * x1 + rnorm(n, sd = 0.5)
Bspline <- bs(seq(0, 1, len = n), df=5)
fit <- lm(x2 ~ Bspline * x1)
summary(fit)
plot(seq_along(x1), coef(fit)[7] + Bspline %*% coef(fit)[8:12],
     ylim = c(-1, 1), xlab = "index", ylab = "", type = "l",
     col = "red")
lines(seq(-0.5, 0.8, len=n)^2, col = "blue")
legend("topleft", col = c("blue", "red"),
       legend = c("true coefficient", "estimated coefficient"),
       lty = c(1,1), bty = "n")
mtext(text = "time-varying correlation (quadratic)", side = 3,
      adj = 0, font = 2)
</code>
Answer: A simple method would be to run moderated regression with time as the moderator. Center time and A then multiply them to create a product term. Using hier reg, enter A and time, then the product term. R2 change at the latter step shows mod effect of time on AB r.
Comment: Following previous suggestion, you need to plot the slopes to confirm the direction of the moderation (strengthening or weakening over time).


Drupal 7 Commerce - How do you style the Please wait loading message/icon when you select an attribute?

Question: I'm trying to style the please wait loader icon when you select an attribute for a product in drupal 7 commerce. I'm using chrome and i can't figure out a way to right click inspect element fast enough as it loads to fast. Or is there another way to expose this? I just need the HTML and classes associated with it.
Comment: Aha! i got it to show up by unplugging my internet connection so it throws an error. Here it is: http://prntscr.com/2131ny .. what we're looking for was: .ajax-progress .ajax-progress-throbber
Comment: Note that this is simply the Drupal core AJAX throbber, not specific to Commerce itself.
Comment: oh yes thats true.. its everywhere on the site. its annoying how it breaks the styling when it shows up.
Answer: For example:
<code>/* Custom ajax throbber style */
.ajax-progress .throbber {
  /* New icon */
  background: transparent url(../images/ajax-loader.gif) no-repeat 0px 0px;
  /* icon is 15px square */
  height: 15px;
  width: 15px;
  /* Alignment, may tweak to suit */
  float: left; /* LTR */
  clear:both;
  margin: 2px 2px 8px 2px;
}
.ajax-progress-throbber .message {
  /* Hide the "please wait" msg */
  display: none;
}
</code>


shell script execution successful but output has errors, how to determine error and exit main script?

Question: I have a main script. Inside it I call other three shell scripts, A,B and C. All were successful. Exit codes are all equal to zero. However, when I looked into the output file of the first script which is A, it contains an error message. Now I want to exit the main script and not to continue running the other scripts after the script that has output error. Can anyone help me on this? Thanks!
Answer: Even if some command in your first bash script results in an error, the script as a whole may complete with exit code 0.
You can check the exit code of any individual command in your script by using the $? variable. This variable stores the exit code of the previous command. This will allow you to check for errors within the script.
Comment: My first script still has an exit code of 0 though it produces an error. Here is what my script does. I have a shell script that writes an sql script and dumps its result to a csv file. Though the output has errors the exit code of the script is still 0 since it ran. Is there a way I could capture error on the output file?
Comment: @smflores07 I don't know exactly what you mean by "capture the error", but whatever you choose to do with the error you can detect it by checking `$?` within your script after any error prone commands. If the previous line of code produced an error then `$?` will be something other than zero. This is how I've been taught to catch errors in bash. I don't understand why this wouldn't be sufficient.
Comment: Oh I understand the confusion! You would check `$?` **within your first script** not in the other ones. You have to detect the error immediately after the failed line of code or else there will be no easily accessible record of it.
Comment: Could you please look my subscript below?                                             #!/bin/bash
FILE=/data/temp/Jollibee/TableauExtract/Infile/MainJobs/LastExtractionDate.csv
#echo "Start time:"
#date;
while IFS=',' read lastdate_suc
do
echo "\vdelim |
\trim
SELECT * from factsales \g" > /data/temp/Jollibee/TableauExtract/Infile/MainJobs/test/ExtractStoreInc.sql
/opt/Actian/Vectorwise/ingres/bin/sql -S jfcdw_Jollibee_is < /data/temp/Jollibee/TableauExtract/Infile/MainJobs/test/ExtractStoreInc.sql > /data/temp/Jollibee/TableauExtract/Infile/MainJobs/test/StoreKey_All.csv
Comment: I want to know if this has error, if it has the main script will exit but if not process will continue to execute the next subscript.
Answer: The easiest way is to append || exit 1 to the statement which is throwing the error. That will cause the script to exit if the exit code of the command is 1 (i.e. an error).
So assuming you had a command sqlscript and you wanted the entire script to exit if sqlscript exited with a non-zero exit code you would do
sqlscript || exit 1

As a point of trivia, the 1 in exit 1 is not needed. A plain exit command would also exit with the exit status of the last executed command.
Which would be false (code=1) if the sqlscript command fails.  If the sqlscript command succeeds, the exit code is the exit code of sqlscript. In that case, the || does not trigger and the exit command is not executed.
Answer:
I have a main script. Inside it I call other three shell scripts, A,B
  and C. All were successful. Exit codes are all equal to zero. However,
  when I looked into the output file of the first script which is A, it
  contains an error message. Now I want to exit the main script and not
  to continue running the other scripts after the script that has output
  error.

Since script A doesn't return an error exit code, you have to inspect its output. This is quite easy with grep provided that you have a search string which clearly identifies an error message, e. g.:
# this echo command simulates script A - it outputs "error" and exits with 0:
echo "contains an error message" >StoreKey_All.csv  # assumed this output file
grep error StoreKey_All.csv && exit 1               # exit if output has error
# continue with scripts B and C
echo B


Benefit of nested CV for algorithm comparison?

Question: Let's consider some small classification dataset D, and 2 classifiers A and B:

a classifier A, with hyper-parameter $\alpha$, and a set of candidate values for $\alpha$: (1, 10)
a classifier B, with hyper-parameter $\beta$, and some set of candidate values for $\beta$: (5, 500)

The goal is to identify the best performing classifier on D: A or B? Which hyper-parameter value? Which parameter value?
I can think of 2 approaches to answer this question:

Without nested CV: estimate the k-fold CV error for each value of $\alpha$ and each value of $\beta$. For example, we could get: ($\alpha$=1, err=0.7), ($\alpha$=10, err=0.1), ($\beta$=5, err=0.6), ($\beta$=500, err=0.9). The lowest error is achieved by (A, $\alpha$=10). We retrain A with $\alpha$=10 on the full dataset, and that's our final model.
With nested CV: estimate the test errors of A and B using nested CV. For example, we could get (A, err=0.3), (B, err=0.5). We now know that A is better, but we don't know which value to pick for $\alpha$. So we do a regular k-fold CV for model selection. For example, we could get ($\alpha$=1, err=0.7), ($\alpha$=10, err=0.1). We retrain A with $\alpha$=10 on the full dataset, and that's our final model.

Are both approaches correct?
Answer: Use nested CV in the following way:

in the inner loop, you optimize both algorithms and their hyperparameters, i.e. A (incl. α) vs. B (incl β)
Basically, what your "without nesting" approach does.
This will give you a readily trained model, i.e. A with some optimal α or B with optimal β

the outer loop gets you two important internal validation (or rather, verification) results:

is the obtained optimum stable (i.e., is one algorithm consistently preferred, and with reasonably stable hyperparameter)?
an estimate of generalization performance which both your approaches lack

Keep in mind, though:

it is not necessarily a sensible assumption that one algorithm should outperform another. E.g., if LDA works well, there should be no surprise in finding that logistic regression does so, too.

Given that your dataset is small, the corresponding uncertainty on your performance estimate may be the limiting factor for the optimization. Make sure you keep an eye on that (and do use proper scoring rules for optimization)


How is q value defined?

Question:
From Applied Statistical Genetics with R by Foulkes (pFDR = positive false discovery rate):

The q-value is defined similarly as the greatest lower bound of the $pFDR$ that can result from rejecting a test statistic $T$ and a
  sample $x$ with $T(x)=t$ based on a rejection region $Γ$ in the set
  of nested rejection regions. Formally, the q-value is written $$q(t)
    =\inf_{Γ :t∈Γ } [pFDR(Γ )]$$

Since $pFDR$ is involved here, there must be several null
hypotheses, and different null may have different testing rule. But
here does it assume that there is one common testing rule $(T, Γ)$
for all the nulls?
From Wikipedia

The q-value is defined to be the FDR analogue of the p-value. The q-value of an individual hypothesis test is the maximum FDR at which the test may be called significant. One approach is to
  directly estimate q-values rather than fixing a level at which to
  control the FDR.

Why are we talking about the q-value of an individual hypothesis test? Isn't the q-value for multiple nulls and their test
rules?
Should "maximum FDR"  be "minimum FDR" instead?
What does "One approach is to directly estimate q-values rather than fixing a level at which to control the FDR" try to mean? What is "one approach" trying to solve?

Am I right that pFDR depends on and changes with the sample's distribution, so does q-value?

Thanks and regards!
Answer:
Indeed. It is assumed that all test statistics that fall in $\Gamma$ are rejected and others are not.
Also note that in a pure (i.e., subjective) Bayesian framework, there need not be multiple hypotheses to define the pFDR. This is, however, now the view Storey has been promoting.
This Wikipedia entry actually discusses a different meaning/definition of the q-value, which preceded Storey's definition and makes life rather confusing (see also here).
The "individual" hypothesis discussed is the marginal hypothesis rejected.
It is true that hypothesis will typically not be rejected alone. That is why the marginal rejection is discussed.
It should indeed be the minimal FDR.
The two competing approaches to multiplicity control are: (1) Set an error level and find a procedure that controls it. (2) Estimate the error level and reject when the estimated error level reaches a desired value.
The estimated $pFDR(\Gamma)$ depends on the sample. It's value does not depend on the sampling distribution of each test statistic (it is simple arithmetics), but it's sampling properties will naturally depend on the sampling distribution. Particularly, the test statistics need to be independent for Storey's estimator to be unbiased to the true $pFDR(\Gamma)$.
Comment: Thanks! pFDR is defined in terms of expectation wrt the true sample distribution, but not directly depend on the sample?
Comment: @Tim: I distinguish between the estimated pFDR and the true pFDR. The true pFDR of any rejection region will surely depend on the sampling distribution.
Comment: In the first definition, is the q-value defined for each marginal null, or for all the marginal nulls in the multiple comparision?
Comment: Also in the first definition, is $T$ the test statistic of just one of the multiple comparisons? If yes, does the definition of q-value not require other comparisons's test statistics to be rejected?
Comment: Also- have a look at this manuscript of mine for a quick intro to error rates:
http://arxiv.org/abs/1304.4920
Comment: I am confused. In both definitions, is the q value defined for a single comparison in the multiple comparisons? In both, is the rejection region of that comparison the only one that changes, while the rejection regions for the other comparisons are fixed?

How is q value defined?


Send a variable containing "&" or "%" to a PHP page using Javascript

Question: I've got a Javascript script that sends 4 variables to a php page.
The fact is that, if one (or more) of these variables contain a character such as "&" or "%", it just
won't work, it only sends what there was before "&" or "%". For instance, if the content of a variable is
"Bolliger&Mabillard", the result will be "Bolliger". Obviously, this happens because the script separate each
variable with an "&", how can I solve this?
Here's the script I'm talking about

Comment: http://stackoverflow.com/questions/3541711/javascript-url-encode
Comment: Unlike forum sites, we don't use "Thanks", or "Any help appreciated", or signatures on [so]. See "[Should 'Hi', 'thanks,' taglines, and salutations be removed from posts?](http://meta.stackexchange.com/questions/2950/should-hi-thanks-taglines-and-salutations-be-removed-from-posts). BTW, it's "Thanks in advance", not "Thanks in advantage". That doesn't even make _sense_.
Answer: You need to URL-encode the values using encodeURIComponent().


Is it possible to use a condition set component as a parameter for another component?

Question: Some of my Rules (say they are part of a collection S) execute in their actions Rules components of the following form: If conditions A and conditions B, then do actions. Now, conditions A and actions are always the same; only conditions B changes according to the particular Rule in S.
It is frustrating to add the same conditions and actions to the Rules components every time. Therefore, my idea is to enclose every conditions B in a Rules conditions component, and then use those components as a variable in  1 "abstract" Rules component that captures the above pattern. Is that possible?
Answer: This answer provides a way to capture the following common pattern for Rules: "If conditions A and conditions B, then do actions" for fixed conditions A and actions. Instead of creating a separate condition set component for every conditions B (and then use those components inside some kind of "abstract" Rules component as a variable), the following approach is way easier:

Create a condition set component for conditions A, say My component for conditions A.
Create an action set component for actions, say My component for actions.
For every conditions B, create a new Rule.
For every new Rule, add My component for conditions A to the conditions, and My component for actions to the actions. Then, complement the conditions with conditions B.

Result: we fixed the never-changing conditions A and actions in 2 Rules components. That saves us a lot of redundant repetitive work. If we want to implement 10 cases of the general pattern, then we end up with 10 Rules + 2 Rules components = 12 Rules in total.
Let's compare this approach with the initial idea. We would have to create 10 Rules components for the conditions B + 1 "abstract" Rules component + 10 Rules that contain this "abstract" Rules component in their actions = 21 Rules in total.
The advantage of the proposed alternative approach is less Rules in total. The disadvantage is that every Rule needs to include the same action set and condition set components. But to be honest, adding those 2 components to the Rules takes much less clicks and time than creating new Rules... I think this problem nicely illustrates the power of action sets and condition sets in the Rules module (a feature that might seem somewhat obscure for beginners).


Property 'apiUrl' does not exist on type '{}' error on my Angular App

Question: in my Angular Frontend I have environment.ts like this
~~~~~~`
export const environment = {

    production: false,
    apiUrl: 'http://127.0.0.1:8000'
};

and my data.service.ts like this
export class DataService {

  constructor(private http:HttpClient) {}

  registerUser(data: any) {
    return this.http.post(environment.apiUrl+'/api/register/', data);
  }
}

but when I try to run the project it is encountrd following error msg
Error: src/app/service/data.service.ts:17:39 - error TS2339: Property 'apiUrl' does not exist on type '{}'.

17     return this.http.post(environment.apiUrl+'/api/register/' data);

Answer: Here is a working example. https://stackblitz-starters-9l8rvp.stackblitz.io
You need to import your object in the class you want to use it in.
import { environment } from './environment';
Comment: I have import in data.service.ts as `import { environment } from '../../environments/environment';`
Answer: It doesn't recognize the apiUrl property on the environment object. This issue could arise if TypeScript is not able to infer the type of the environment object. To resolve this, you can explicitly specify the type of environment to avoid type-related errors.
Approach 1
export interface Environment {
  production: boolean;
  apiUrl: string;
}

export const environment: Environment = {
  production: false,
  apiUrl: 'http://127.0.0.1:8000',
};

OR
Approach 2
import { environment as env } from 'path-to-your-environment-file';

export class DataService {
    constructor(private http: HttpClient) {}

    registerUser(data: any) {
        return this.http.post(env.apiUrl + '/api/register/', data);
    }
}

Comment: same error here
Answer: One of the reasons can be that all the environment do not have the same property names.
Another workaround for this issue, is by assigning any type. Like shown below
return this.http.post((environment as any).apiUrl+'/api/register/' data);

Comment: it is working fine


Show that $\{1,x,x^2\}$ is $\mathbb{Q}-$base of $\mathbb{Q}(x)$

Question: I want to show that $\{1,x,x^2\}$ is $\mathbb{Q}-$base of $\mathbb{Q}(x)$.
We're looking at the field extension $\mathbb{C}/\mathbb{Q}$ and $x\in \mathbb{C}$.
$\mathbb{Q}(x)$ is the smallest subfield of $\mathbb{C}/\mathbb{Q}$ that contains $x$. Also, $x$ is a complex root of $X^3-2X+2\in \mathbb{Q}[X]$.

First I showed that $m = X^3-2X+2$ is the minimal polynomial of $x$. Since it's already given that $x$ is a root of $m$, I only need to show that it's irreducible.
Let $p=2$ and $m = X^3-2X+2 = aX^3+bX+c$.
Then $p$ is a prime and $p\mid b \wedge p\mid c$ but $p^2\nmid c \wedge p\nmid a$. By Eisenstein criterion this polynomial is irreducible and therefore the minimal polynomial of $x$.
That means that $[\mathbb{Q}(x) : \mathbb{Q}] = 3$, right?
So we only need to find $3$ linearly independent vectors. Two of them are obvious to me, namely $\{1,x\}$. But I don't really get why I need $x^2$ and how I can prove that it's independent from $x$ (or $1$).
My guess is that if it was linear dependent, we then have by linear combination:
$$\lambda_1 \cdot 1 + \lambda_2 \cdot x = \lambda_3 \cdot x^2 \Rightarrow  -\lambda_3\cdot x^2 + \lambda_1 \cdot 1 + \lambda_2 \cdot x =0$$
Since that polynomial has degree $2$ and $x$ is a root, it's a contradiction because the minimal polynomial of $x$ has degree $3$.
Any help is appreciated.
Comment: Using $x$ for a complex number is confusing when you also talk about polynomials.
Comment: Of course $1,x,x^2$ are linearly independent over $\Bbb Q$, right? Compare this with the standard example of $f(X)=X^3-2$, where the basis also is $\{1,\sqrt[3]{2},\sqrt[3]{2}^2\}$.
Comment: @DietrichBurde agreed. I didn't have a good idea of $x$ since it could have had any value in $\mathbb{C}$.
Answer: If $1,x,x^2$ are not linearly independent then $\lambda_0,\lambda_1,\lambda_2\in\mathbb C$ exists such that: $$\lambda_0+\lambda_1x+\lambda_2x^2=0\text{ and }(\lambda_0,\lambda_1,\lambda_2)\neq(0,0,0)$$
However then $x$ is the root of a non-zero polynomial with degree less than $3$ contradicting the fact that its minimum polynomial has degree $3$.