How to test whether variables are more frequently associated than expected by chance?

Question: I have this presence/absence matrix (S2 is missing on purpose).
<code>dat <- read.table(text="S1  S3  S4  S5  S6  S7  S8  S9  S10
0   0   1   1   0   0   1   0   0
0   0   1   1   0   0   1   0   0
0   0   1   0   0   0   0   0   0
0   0   0   1   0   0   0   0   1
0   0   1   1   0   0   1   0   0
0   0   1   1   0   0   0   0   0
0   0   1   1   0   1   1   0   0
0   0   0   0   0   0   0   1   0
0   0   1   1   0   0   1   0   0
0   0   1   0   0   0   0   0   1
1   0   0   0   1   0   0   0   0
0   0   1   0   0   0   0   0   0
0   1   0   0   0   1   0   0   0
0   0   0   1   0   0   1   1   0
0   0   0   0   0   0   0   1   0
0   0   1   1   0   0   1   0   0
1   1   0   0   0   1   0   1   1
1   1   1   0   0   0   0   1   1
0   0   0   1   0   0   1   0   0
0   0   0   1   0   0   1   0   0
0   0   0   1   0   0   0   0   0
0   0   1   1   0   0   1   0   0
0   0   0   0   0   0   1   0   0
1   0   0   0   0   0   0   0   0
0   0   1   0   0   0   0   0   0
1   0   0   0   0   0   0   0   0
0   0   0   1   0   0   1   0   0
1   0   0   1   0   0   0   0   0
0   0   0   0   0   0   1   0   0
1   0   0   0   0   0   0   0   0
0   0   0   0   0   0   1   0   0
0   0   0   0   0   0   1   0   0
0   0   1   1   0   0   1   0   0
0   0   1   1   0   0   0   0   0
0   0   0   1   0   0   0   0   0
0   0   0   0   0   0   1   0   0
0   0   0   0   0   0   1   0   0
0   0   1   1   0   0   0   0   0
0   0   0   1   0   0   0   0   0
0   0   0   1   0   0   1   0   0
1   0   1   1   0   0   0   0   0
0   0   0   1   0   0   0   0   0
0   0   1   1   0   0   0   0   0
0   0   1   0   0   0   0   0   0
0   0   0   1   0   0   0   0   0
0   0   1   1   0   0   0   0   0
0   0   1   0   0   0   0   0   0
0   0   0   0   0   0   1   1   0
0   0   1   0   0   0   0   0   0
1   1   0   0   0   0   0   1   1
0   0   1   1   0   0   0   0   0
0   0   1   0   0   0   1   0   0
0   0   1   1   0   0   0   0   0
0   0   0   0   0   0   1   0   0
0   0   1   1   0   1   1   1   0", header=T)
</code>
On looking at the number of times a particular variable is found with another, it seems to me that S4, S5 and S8 are frequently found together. Is there a way of testing this statistically either for pairs of variables or for triplets?
Cheers
Answer: Would a $\chi^2$-test of independence achieve what you want to do? You basically crosstabulate the variables of interest, and determine expected frequencies by assuming independence of features (if feature A occurs with frequency $f_A$ and feature B with frequency $f_B$ for $N$ observations, you expect frequency ($f_Af_B/N$) for $A \land B$. This can be done for any number of variables. Given the number of possible pairs (36) and triplets (84) in this case (9 variables), you need to be very careful to interpret the outcome of any one test, especially if you do not have any conceptual hypothesis beforehand. Using your intuition as the criterion for choosing the pair or triplet to look at does not solve the proble, as you used actual data to eliminate tests from consideration.


Euclidean Geometry Area Problem

Question: Let $\Gamma $ be the circumcircle of  triangle $ABC$. Let $A_0$ be the center of the circle lying outside of $\triangle ABC$ and which is tangent to the segment $BC$ and to rays $\overrightarrow{AB}$ and $\overrightarrow{AC}$. Points $B_0$ and $C_0$ are similarly defined. Let $A_1$ be the second intersection point of line $AA_0$ with $\Gamma$. Points $B_1$ and $C_1$ are similarly defined.
Prove that the area of $\triangle A_0B_0C_0$ is equal to twice the area of hexagon $AB_1CA_1BC_1$.
Working through this I found the alternate segment theorem very useful as I was trying to establish congruence between triangles that are in the hexagon but outside the triangle. I couldn't quite get there. Any help would be greatly appreciated.
Answer:
Let $I$ be the point common to the three angle bisectors. (Note that $I$ is the center of the triangle's incircle (not shown); it is not the center of the circumcircle.)

Claim: $A_1$ is the midpoint of $\overline{IA_0}$.

Given the claim, we must have $|\triangle ICA_0| = 2 |\triangle ICA_1|$, because the "base" $\overline{IA_0}$ of the first triangle has twice the length of the "base" $\overline{IA_1}$ of the second, whereas their "heights" to $C$ match. Since $\triangle ICA_0$ and its counterparts throughout the figure cover $\triangle A_0B_0C_0$, while $\triangle ICA_1$ and its counterparts cover hexagon $AB_1CA_1BC_1$, the triangle's area must be twice the hexagon's.
Proof of claim:
Since the vertical angles $\angle AIC_1$ and $CIA_1$ subtend circumcircle arcs $\stackrel{\frown}{AC_1}$ and $\stackrel{\frown}{CA_1}$ that are respectively subtended by inscribed angles $\angle ACC_1$ and $\angle CAA_1$, their measure is equal to the sum of the measures of those inscribed angles:

Since $\angle A_1AB$ and $\angle A_1CB$ subtend arc $\stackrel{\frown}{A_1B}$, they are congruent:

Consequently, $\triangle A_1IC$ has congruent angles, $\angle I \cong \angle C$, and thus also congruent sides, $\overline{A_1I}\cong\overline{A_1C}$.

Note that, just as incenter $I$ lies on $\triangle ABC$'s three (interior) angle bisectors, excenter $A_0$ lies on two exterior angle bisectors; likewise for $B_0$ (and $C_0$), so that $\overline{A_0B_0}$ must contain $C$ and be perpendicular to $\overline{CC_0}$. We find, then, that $\angle IA_0C$ and $\angle A_1CA_0$ are complements of congruent angles and must be congruent to each other, whence $\overline{A_0A_1}\cong\overline{A_1C}$ ($\cong\overline{A_1I}$, from above). Point $A_1$ is therefore the midpoint of $\overline{IA_0}$ as claimed, completing the proof. $\square$
Answer: Hint: The circumcenter and excenter lies on the respective angle bisectors of the triangle. Let $O$ be the circumcenter of $ABC$, and $I$ be the incenter of $ABC$.
Hint: Consider right triangle $ICA_0$. Show that triangles $IA_1 C$ and $CA_1 A_0$ are isosceles, which gives that $IA_1 = A_1 C = A_1A_0$.
Hint: Decompose the hexagon into 3 parts, and the triangle into 3 parts, to show that the area is twice.
Comment: How do we know that those lines are the angle bisectors of the triangle???
Comment: @JohnMarty: I believe Calvin may have mis-typed in his first hint. The (centers of the ) *in*-circle and excircles lie on angle bisectors; note that the centers of excircles also lie on bisectors of *external* angles, so that, for instance $A_0$, $B$, and $C_0$ are collinear, and their line makes a right angle with the (internal) angle bisector at $B$.
Comment: Thanks: I thought their was something wrong
Comment: @JohnMarty: Also, I believe that Calvin's second hint is meant to show $IA_1 = CA_1 = A_0A_1$. Specifically, if you can show that $A_1$ is the midpoint of $IA_0$, then you're done (because then $2|\triangle ICA_1| = |\triangle ICA_0|$, and likewise around the figure).
Comment: @JohnMarty Sorry, edits made.
Comment: [+1] A small remark : "executer" hasn't its place in the first line...

Euclidean Geometry Area Problem


Let $F$ be a free centralizer in $G$. Then $F=P\times A$, where $P$ is a Sylow $p$-subgroup of $F$ and $A$ is abelian.

Question: This question is from the Proposition A.23.2 in page 520 of the book ``Berkovich, Yakov, and Zvonimir Janko. Berkovich, Yakov; Janko, Zvonimir: Groups of Prime Power Order. Vol. 2. Walter de Gruyter, 2008.''
Suppose $G$ is a finite nontrivial group.
$F$ is a centralizer in $G$ is means that
$F$ is a centralizer of an element.
$F$ is free means that $F\neq G$ and
if a centralizer $C_G(a)\leq F$,
then either $C_G(a)={1}$ or $C_G(a)=F$ and
if a centralizer $C_G(b)\neq G$, $F\leq C_G(b)$,
then $C_G(b)=F$.
The author first prove that $F=C_G(x)$, where $x\in G$
and the order $|x|$ of $x$ is a power of a prime $p$.
And he prove that if $y\in F$ and $|y|=q^r$,
for some positive integer $r$ and prime $q\neq p$,
then $y\in Z(F)$, the center of $F$.
I already know the reason of these two results,
but the author then says that $F=P\times A$,
where $P$ is a Sylow $p$-subgroup of $F$ and
$A$ is abelian
(Use Burnside's normal $p$-complement theorem),
I don't know how to use the Burnside's normal $p$-complement theorem
in here.
Could anyone give me a favor, thanks very much.
Comment: $C_G(a)=1$ iff $G=1$. Perhaps that part is miscopied?
Answer: The statements you do believe imply that every element of order prime to $p$ in $F$ is in $Z(F).$ Actually, I think I agree with you. The version of Burnside's normal $p$-complement theore that is usually stated is that if $P \in {\rm Syl}_{p}(G)$ and $P \leq Z(N_{G}(P)),$ then $G$ has a normal $p$-complement, but this requires that $P$ should be Abelian, and I don't think that is necessarily true in this situation. There are ways around it: you can use Frobenius's normal $p$-complement theorem,, since $N_{F}(Q)/C_{F}(Q)$ is certainly a $p$-group for each $p$-subgroup $Q$ of $F.$ Then the normal $p$-complement theorem of Frobenius says that $F$ has a normal $p$-complement $K$ and $F = PK$ for $P \in {\rm Syl}_{p}(F).$ But we already have seen that $K \leq Z(F)$ so $K$ ia Abelian and $F = P \times K.$ You can also use the Schur-Zassenhaus theorem, since $P \lhd F,$ as every element of order prime to $p$ in $F$ is central. Hence $P$ is complemented in $F$ and that complement is contained in $Z(F)$ as before.
Later edit: I suppose that there are ways to do this via Burnside's normal $p$-complement theorem. Another way without Burnside's theorem is to take $A \lhd P$ be be a maximal Abelian normal subgroup of $P.$ Then $C_{P}(A) = A,$ and $C_{P}(A)$ is a Sylow $p$-subgroup of $C_{F}(A).$  Hence $C_{F}(A)$ is Abelian, and if $K$ is the direct product of its Sylow $q$-subgroups for all primes $q \neq p,$ then $K$ char $C_{F}(A) \lhd F,$ so $K \lhd F$ is a normal $p$-complement for $F.$ But one way to use Burnside's theorem is to say that for each prime $q \neq p,$ a Sylow $q$-subgroup $Q$ of $F$ is in $Z(N_{F}(Q)),$ so $F$ has a normal $q$-complement, and then proceed by induction on the number of prime divisors of $|F|$.
Comment: We know that $P \lhd F$ for the following reason: note that $P \leq N_{F}(P)$ clearly. For every prime $q \neq p,$ and $Q \in {\rm Syl}_{q}(F),$ we have $Q \leq Z(F) \leq N_{F}(P).$ Hence $N_{F}(P)$ contains  a Sylow $r$-subgroup of $F$ for each prime divisor $r$ of $|F|$, so that $|N_{F}(P)|$ is divisible by $|F|$ and we must have $N_{F}(P) = F.$
Comment: Dear Robinson,
thank you for your answer,
you give me a new direction,
but I have a question.

How do we know that $P\lhd F$?
We need to show that for any $g\in F$,
$gPg^{-1}\subseteq P$.
If $\gcd{(|g|, p)}=1$, then $g\in Z(F)$ and $gPg^{-1}=P$.
But if $\gcd{(|g|, p)}=p$, does it still must be $gPg^{-1}\subseteq P$?
Comment: How excellent the answer is, how could you know such method. Thank you so much.


Fetch data to React Native Victory charts

Question: I managed to create an object that holds both key(service name) and value(counter of each service)
What i dont get is how i fetch the data properly to the graph, meaning that the X axis will hold the key(Service name) and Y axis will hold the value(counter)
Code of creating the object:
const servicesCounter = services.reduce((counterObj, service) => { //Reduce => reduce the array to a single value, then count it.
    if (counterObj.hasOwnProperty(service)) {
      counterObj[service] += 1;
      return counterObj;
    }

    return {
      ...counterObj,
      [service]: 1
    };
  }, {});

Console Log:
Service Counter =>  Object {
  "clean": 4,
  "haircut": 3,
  "wash": 1,
}

Victory Pie:
  data={[
    { x: "One" /*Here i need to present the key(service name) */ , y: 2 /*Here i need to present the value(counter)*/},
    { x: "Two", y: 4 },
    { x: "Three", y: 3 }
  ]}

Firestore query to get the service names:
  useEffect(() => { //Query to get the services from database and count them to later use.
    firebase
    .firestore()
    .collection("users")
    .doc(uid)
    .collection("confirmed-appointments").get().then((querySnapshot) => {
      let arrayofServices = [];
      querySnapshot.forEach((doc) => {
        arrayofServices.push(doc.data().serviceType)
      })
      //console.log("Array of services ", arrayofServices);
      setServices(arrayofServices);
    })

  }, [])

Any Suggestions that will make my life easier please??
Answer: You can use object.entries and iterate through the properties and create an array for the chart and use it in the chart. you can run the snippet below.

const data = {
  "clean": 4,
  "haircut": 3,
  "wash": 1,
};

const arr = new Array();

for (const [key, value] of Object.entries(data)) {
  arr.push({x:key,y:value});
}

console.log(arr)
Comment: by creating const data im assuming that i know the keys and values. actually im getting the data from Firestore, so if the user add a service , i will need to update const data manually.
Im editing the post with the firestore query
Comment: no it can be any object i just gave a sample, you can use any object which has the same format of key and value and convert it to the way you want
Comment: i will try to match your code to mine and will update here my results
Comment: ok so you have an array 'arrayofServices' which you want to convert to that form right ? may i know whats not working ?
Comment: I tried your way and do as u suggested, but when i pass the data to the Victory Pie like this : data={pieData}
i got this red warning: Warning: Failed prop type: Invalid prop `data` of type `object` supplied to `VictoryPie`, expected `array`.
Comment: you should pass the arr not data
Comment: My bad, as usual. cant focus on this hour haha sorry man
Comment: Hope you got it sorted :)


Samba server - files created as user root instead of normal user

Question: I have a Debian Wheezy with a Samba (3.6) server installed with a /share folder, and I don't know why but files get created as user root instead of user (uid=1000), here is my conf:
[global]
        server string = %h server
        map to guest = Bad User
        obey pam restrictions = Yes
        pam password change = Yes
        passwd program = /usr/bin/passwd %u
        passwd chat = *Enter\snew\s*\spassword:* %n\n *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* .
        syslog = 0
        log file = /var/log/samba/log.%m
        max log size = 1000
        dns proxy = No
        panic action = /usr/share/samba/panic-action %d
        idmap config * : backend = tdb

[share]
        comment = my share
        path = /share
        valid users = user
        admin users = user
        write list = user
        force user = user
        force group = user
        read only = No
        create mask = 0660
        directory mask = 0660

I tried force user = user and force group = user but it doesn't look like it worked...
On my windows machine, when I create a file I get:
$ la /share
-rw-rw---- 1 root user       0 Jun  2 10:52 testfile.txt

Any insight appreciated :-)
Comment: Just to confirm, you are logging in as `user` and the directory `/share` is writable by `user`?
Comment: Yes, the rights on the directory are `drwxr-xr-x   5 user nogroup  4096 Jun  2 10:52 share`
Comment: Do you get the same result without any kind of 'force' option?
Comment: Yes, same result.
Answer: It is a mix of two items
admin users = user
force user = user

The first causes files created by user to be owned by root, the second causes all users to act as if there were user.
Remove admin users = user and then make sure that /share has read/write for the user user.


Pushing news to my iOS app

Question: What would be the easiest way to set up a way in which my iPhone app can read news data online from my website, for example. I want to be able to push news or updates to the bottom of the app such as "Check out our new app, blah blah" or something along those lines. I thought of one way, but I didn't know if there's better ways.

Keep some html file on my server (my website), and have the iPhone read that html data

Any other ideas? Anyone have experience in this field?
Answer: If you already have a blog or are familiar with some popular blogging software, you can likely take advantage of RSS feeds that are automatically generated for you.
For example, you can have Wordpress automatically generate RSS for a post type, a tag, a category, or pretty much anything else that returns a set of posts. Your application could then consume that data without you needing to manually update anything code-laden, just add to the blog and it's available in the feed.
Comment: Do those news apps like Bloomberg, ABCNews, CBSnews, apps use the same technique?

Pushing news to my iOS app


Feature selection - correlation ranking for tree-based methods

Question: I had a discussion with my colleague some time ago and he stated that a correlation between the predictor variable and the outcome variable is not a good measure to perform feature selection, especially when it comes to tree-based classification/regression methods. His justification for that opinion was that the certain predictor $A$ might not be correlated to the outcome $Y$ at all, but after a split by some other predictor $B$ (remember we are talking about tree-based methods), predictor $A$ may reveal a great predictive power. Just as a note - let's say it does not necessarily to be correlation, what he actually meant is any pre-training (filter approach) measure that scores one predictor variable at a time.
For that time, I agreed with him because it seems quite intuitive, but when I think about this now, I can't really create an example to prove this statement. When I think about it or try to create an example, I get an impression that if $B$ is able to split the $A$ in a way that it becomes greatly correlated with $Y$ (within the subsets defined by $B$), then $B$ itself is in fact very highly correlated with $Y$, so there's probably no need to use $A$ anyway. On the other hand, the statement must be somehow true, because it is basically how the tree-based methods work. So maybe there is a possibility in correlation improvement, but only in a small amount and both of the variables have to be meaningfully correlated to $Y$ anyway (resulting in a fact that feature elimination due to low correlation would still make sense)?
So my main question is: is the statement true, and if it is, could you provide an example?
I have one side question as well: is it possible to combine two predictor variables, that are very weakly correlated to the outcome variable, in a way that the engineered third predictor (e.g. $C = A*B$ as a silly example) will be very strongly correlated to the outcome variable?
Answer: Here's an example that should be in everyone's toolbox regarding regression/decision trees, as it succinctly makes two very important points.

The vertical axis shows a response $y$.  There are two predictor variables, $x_1$ ranges from $-1$ to $1$ continuously.  The relationship between $y$ and $x_1$ is constructed so that no matter how you partition the $x_1$ axis, the response always averages out to zero.  In particular:
<code>> cor(df$x_2, df$y)
[1] -0.001792121
</code>
Or, for all intents and purposes, zero.
The other variable $x_2$ allows you to distinguish the "arm" of the $X$ shaped data.
This data has two very interesting features:

The true or correct decision tree for this data splits on $x_2$ first.  This allows it to distinguish the arms of the $X$ shape, and immediately breaks the zero correlation structure between $x_1$ and $y$ for all sub-splits.
A greedy algorithm will generally not find the optimal fist split (you could get very lucky, but probably it will find some noisy split on $x_1$)!  The first split on $x_2$ does not lead to a reduction in squared error, it is only important because it lets later splits focus on the association between $x_1$ and $y$.

Here is the R code I used to make this plot, so you can experiment yourself
<code>df <- data.frame(
  x_1 = c(seq(-1, 1, length.out=250), seq(1, -1, length.out=250)),
  x_2 = rep(c(1, -1), each=250)
)
df$y <- df$x_1*df$x_2 + rnorm(500, 0, .05)

library(ggplot2)
ggplot(data=df) + geom_point(aes(x=x_1, y=y, color=factor(x_2)))
</code>
Comment: Good answer, just a couple of small additions: This is also closely related to the so-called XOR problem. A range of solutions has been suggested for this, including explicitly adding interaction variables in the tree, looking two steps ahead (instead of just one) in the greedy search, or trying to grow globally optimal trees (e.g., through evolutionary algorithms). However, these do not come "for free" as these require either more a-priori knowledge or computational effort.
Comment: Our R package `evtree` provides evolutionary learning of globally optimal trees (http://CRAN.R-project.org/package=evtree). The corresponding JSS paper (http://dx.doi.org/10.18637/jss.v061.i01) also has some references to related work.
Comment: Well, this basically answers my question. It seems the statement was right after all. Thanks for the great example.
Comment: @AchimZeileis This XOR problem is actually very interesting. Would you be able to provide some names/papers about algorithms trying to tackle this problem? Especially regarding decision trees?


I am trying to implement expandablelistview in my test app but the app crashes everytime

Question: This is the part of Logcat :
It says that its caused by
MainActivity.java:41)
I checked the ids and they are done the same way as in the tutorial.
.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:779)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:595)
        at dalvik.system.NativeStart.main(Native Method)
 Caused by: java.lang.NullPointerException
        at com.appt.shreyabisht.test.MainActivity.onCreate(MainActivity.java:41)
        at android.app.Activity.performCreate(Activity.java:5231)
        at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1087)
        at android.app.ActivityThread

This is main activity.java
    package com.appt.shreyabisht.test;
import android.app.Activity;
import android.content.Intent;
import android.os.Bundle;
import android.view.Menu;
import android.view.MenuItem;
import android.view.View;
import android.widget.Button;
import android.widget.ExpandableListView;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;

public class MainActivity extends Activity implements View.OnClickListener {
HashMap <String, List<String>> Services_category;
List<String> Services_list;
ExpandableListView Exp_list;
services_adapter adapter;

Button btn_sign;
Button btn_sign_up;

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);
    btn_sign = (Button) findViewById(R.id.btn_sign);
    btn_sign.setOnClickListener(this);
    btn_sign_up = (Button)findViewById(R.id.btn_sign_up);
    btn_sign_up.setOnClickListener(this);
    Exp_list=(ExpandableListView) findViewById(R.id.exp_list);
    Services_category = Dataprovider.getinfo();
    Services_list = new ArrayList<String>(Services_category.keySet());
    adapter = new services_adapter(this, Services_category, Services_list);
    Exp_list.setAdapter(adapter);

}

@Override
public boolean onCreateOptionsMenu(Menu menu) {
    // Inflate the menu; this adds items to the action bar if it is present.
    getMenuInflater().inflate(R.menu.menu_main, menu);
    return true;

}

@Override
public boolean onOptionsItemSelected(MenuItem item) {
    // Handle action bar item clicks here. The action bar will
    // automatically handle clicks on the Home/Up button, so long
    // as you specify a parent activity in AndroidManifest.xml.
    int id = item.getItemId();

    //noinspection SimplifiableIfStatement
    if (id == R.id.action_settings) {
        return true;
    }

    return super.onOptionsItemSelected(item);
}

private void btclick() {
    startActivity(new Intent("com.appt.shreyabisht.second"));
}

private void btsignclick() {
    startActivity(new Intent("com.appt.shreyabisht.third"));
}

@Override
public void onClick(View view) {
    switch (view.getId()) {
        case R.id.btn_sign:
            btclick();
            break;
        case R.id.btn_sign_up:
            btsignclick();
            break;
    }

}

}
Manifest.xml:
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="com.appt.shreyabisht.test" >

<application
    android:allowBackup="true"
    android:icon="@drawable/ic_launcher"
    android:label="@string/app_name"
    android:theme="@style/AppTheme" >
    <activity
        android:name="com.appt.shreyabisht.test.MainActivity"
        android:label="@string/app_name" >
        <intent-filter>
            <action android:name="android.intent.action.MAIN" />

            <category android:name="android.intent.category.LAUNCHER" />
        </intent-filter>
    </activity>
    <activity
        android:name="com.appt.shreyabisht.test.second"
        android:label="@string/app_name" >
        <intent-filter>
            <action android:name="com.appt.shreyabisht.second" />

            <category android:name="android.intent.category.DEFAULT" />
        </intent-filter>
    </activity>
    <activity
        android:name="com.appt.shreyabisht.test.third"
        android:label="@string/app_name" >
        <intent-filter>
            <action android:name="com.appt.shreyabisht.third" />

            <category android:name="android.intent.category.DEFAULT" />
        </intent-filter>
    </activity>
    <activity
        android:name="com.appt.shreyabisht.test.after_signin"
        android:label="@string/title_activity_after_signin" >
        <intent-filter>
            <action android:name="com.appt.shreyabisht.test.after_signin" />

            <category android:name="android.intent.category.DEFAULT" />
        </intent-filter>
    </activity>
</application>

after_signup.xml ( This page has expandable list)
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools" android:layout_width="match_parent"
android:layout_height="match_parent" android:paddingLeft="@dimen/activity_horizontal_margin"
android:paddingRight="@dimen/activity_horizontal_margin"
android:paddingTop="@dimen/activity_vertical_margin"
android:paddingBottom="@dimen/activity_vertical_margin"
tools:context="com.appt.shreyabisht.test.after_signin"
android:background="@drawable/download">

<TextView android:text="@string/welcome_header" android:layout_width="wrap_content"
    android:textColor="#FFFFDD76"
    android:textStyle="bold"
    android:textSize="20dp"
    android:layout_height="wrap_content"
    android:layout_alignParentTop="true"
    android:layout_centerHorizontal="true"
    android:id="@+id/textView5" />

<ExpandableListView
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:id="@+id/exp_list"
    android:indicatorLeft="?android:attr/expandableListPreferredItemIndicatorLeft"
    android:layout_alignParentBottom="true"
    android:divider="#A4C739"
    android:dividerHeight="0.5dp"
    android:layout_alignParentRight="true"
    android:layout_alignParentEnd="true"
    android:layout_below="@+id/textView5" />

parent file (expand_1.xml):
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:layout_width="match_parent" android:layout_height="match_parent">

<TextView
    android:layout_width="fill_parent"
    android:layout_height="match_parent"
    android:paddingTop="10dp"
    android:paddingBottom="10dp"
    android:paddingLeft="?android:attr/expandableListPreferredItemPaddingLeft"
    android:id="@+id/parent_id"
    android:textColor="#A4C739"/>

child file (child.xml):
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:layout_width="match_parent" android:layout_height="wrap_content"
android:orientation="vertical">

<TextView
    android:id="@+id/child"
    android:layout_width="match_parent"
    android:paddingTop="10dp"
    android:paddingBottom="10dp"
    android:layout_height="match_parent" />

services_adapter.java
package com.appt.shreyabisht.test;

import android.content.Context;
import android.database.DataSetObserver;
import android.graphics.Typeface;
import android.view.LayoutInflater;
import android.view.View;
import android.view.ViewGroup;
import android.widget.BaseExpandableListAdapter;
import android.widget.ExpandableListAdapter;
import android.widget.TextView;

import org.w3c.dom.Text;

import java.util.HashMap;
import java.util.List;

/**
 * Created by SHREYA BISHT on 11/19/2014.
 */
public class services_adapter extends BaseExpandableListAdapter {
private Context ctx;
private HashMap<String, List<String>> Services_category;
private List<String> Services_list;

public services_adapter(Context ctx, HashMap<String, List<String>> Services_category, List<String> Services_list)
{
    this.ctx = ctx;
    this.Services_category = Services_category;
    this.Services_list = Services_list;

}
@Override
public void registerDataSetObserver(DataSetObserver dataSetObserver) {

}

@Override
public void unregisterDataSetObserver(DataSetObserver dataSetObserver) {

}

@Override
public int getGroupCount() {
    return Services_list.size();
}

@Override
public int getChildrenCount(int arg0) {
    return Services_category.get(Services_list.get(arg0)).size();
}

@Override
public Object getGroup(int i) {
    return Services_list.get(i);
}

@Override
public Object getChild(int parent, int child) {
    return Services_category.get(Services_list.get(parent)).get(child);
}

@Override
public long getGroupId(int i) {
    return i;
}

@Override
public long getChildId(int parent, int child) {
    return child;
}

@Override
public boolean hasStableIds() {
    return false;
}

@Override
public View getGroupView(int parent , boolean isExpanded, View convertview, ViewGroup parentview) {
        String group_title = (String) getGroup(parent);
    if(convertview == null)
    {
        LayoutInflater inflater = (LayoutInflater) ctx.getSystemService(Context.LAYOUT_INFLATER_SERVICE);
        convertview = inflater.inflate(R.layout.expand_1, parentview, false);

    }
    TextView parent_textview = (TextView) convertview.findViewById(R.id.parent_id);
    parent_textview.setTypeface(null, Typeface.BOLD);
    parent_textview.setText(group_title);
    return convertview;
}

@Override
public View getChildView(int parent, int child, boolean lastchild, View convertview, ViewGroup parentview) {
    String child_title = (String) getChild(parent,child);
    if(convertview == null)
    {
        LayoutInflater inflater = (LayoutInflater) ctx.getSystemService(Context.LAYOUT_INFLATER_SERVICE);
        convertview = inflater.inflate(R.layout.expand_child, parentview, false);
    }
    TextView child_textview = (TextView) convertview.findViewById(R.id.child);
        child_textview.setText(child_title);

    return convertview;
}

@Override
public boolean isChildSelectable(int i, int i2) {
    return false;
}

@Override
public boolean areAllItemsEnabled() {
    return false;
}

@Override
public boolean isEmpty() {
    return false;
}

@Override
public void onGroupExpanded(int i) {

}

@Override
public void onGroupCollapsed(int i) {

}

@Override
public long getCombinedChildId(long l, long l2) {
    return 0;
}

@Override
public long getCombinedGroupId(long l) {
    return 0;
}

}
and the strings file
strings.xml:

<string name="app_name">test</string>
<string name="hello_world">Hello world!</string>
<string name="action_settings">Settings</string>
<string name="enter_name">Username</string>
<string name="enter_username">Enter</string>
<string name="calling_activity_name">The username and password entered was</string>
<string name="sign_in">Sign In</string>
<string name="user_password">Password</string>
<string name="App_name">StayMax</string>
<string name="user_name">Username:</string>
<string name="details_header">Please provide login details</string>
<string name="hello_blank_fragment">Hello blank fragment</string>
<string name="title_activity_second">second</string>
<string name="title_activity_third">third</string>
<string name="confirm_password">Confirm Password</string>
<string name="enter_location">Enter Location</string>
<string name="sign_up">Sign Up</string>
<string name="sign_up_btn">Sign Up</string>
<string name="title_activity_after_signin">after_signin</string>
<string name="button_login">Log In</string>
<string name="welcome_header">Welcome to Ayaya</string>

logcat after making the suggested changes:
Process: com.appt.shreyabisht.test, PID: 942
java.lang.RuntimeException: Unable to start activity        ComponentInfo{com.appt.shreyabisht.test/com.appt.shreyabisht.test.MainActivity}:     java.lang.NullPointerException
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2195)
        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2245)
        at android.app.ActivityThread.access$800(ActivityThread.java:135)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1196)
        at android.os.Handler.dispatchMessage(Handler.java:102)
        at android.os.Looper.loop(Looper.java:136)
        at android.app.ActivityThread.main(ActivityThread.java:5017)
        at java.lang.reflect.Method.invokeNative(Native Method)
        at java.lang.reflect.Method.invoke(Method.java:515)
        at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:779)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:595)
        at dalvik.system.NativeStart.main(Native Method)
 Caused by: java.lang.NullPointerException
        at com.appt.shreyabisht.test.MainActivity.onCreate(MainActivity.java:34)
        at android.app.Activity.performCreate(Activity.java:5231)
        at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1087)
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2159)
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2245)
at android.app.ActivityThread.access$800(ActivityThread.java:135)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1196)
at android.os.Handler.dispatchMessage(Handler.java:102)
at android.os.Looper.loop(Looper.java:136)
at android.app.ActivityThread.main(ActivityThread.java:5017)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:515)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:779)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:595)
at dalvik.system.NativeStart.main(Native Method)
11-21 06:07:32.739      942-942/com.appt.shreyabisht.test I/Process﹕ Sending signal. PID: 942 SIG: 9

When this is set:
 setContentView(R.layout.activity_main);
The error i get is :
Caused by: java.lang.NullPointerException
        at com.appt.shreyabisht.test.MainActivity.onCreate(MainActivity.java:41)
        at android.app.Activity.performCreate(Activity.java:5231)

mainactivity.java:41 is :         Exp_list.setAdapter(adapter);
When this is set: setContentView(R.layout.activity_after_signin);
The error I get is :
             Caused by: java.lang.NullPointerException
        at com.appt.shreyabisht.test.MainActivity.onCreate(MainActivity.java:34)
        at android.app.Activity.performCreate(Activity.java:5231)

MainActivity.java:34 is :      btn_sign.setOnClickListener(this);
which is for a button on another activity.
Comment: Please provide your logcat.
Comment: `following a tutorial`. You must have very good reasons not to provide a link.
Comment: @greenapps this is the tutorial
Comment: https://www.youtube.com/watch?v=_h94Kqyc-Ag&src_vid=BkazaAeeW1Q&feature=iv&annotation_id=annotation_1141555157
Comment: You should post your code again ofcourse. And mark line 41.
Comment: @greenapps deleted the code by mistake. I have uploaded all the files again. Please help me figure out the error. Thanks
Comment: I also have the dataprovider file in place and ive checked it. I believe that the error has got to do with exp_list but i cant figure out what to correct.
Comment: `MainActivity.java:41`. You still did not tell which line that is. Please tell to resolve the null pointer problem.
Comment: @greenapps I have updated the question with the answer.
Comment: Finally! `mainactivity.java:41 is : Exp_list.setAdapter(adapter);`. (Please edit your post and tell that at the third line).  This tells Exp_list==null because findViewById could not find it as you used the wrong .xml file. So you took the right .xml file and now you have a different pointer which is null. Now  btn_sign is null as it is not defined in that .xml file. You are clearly mixing up buttons/views from two .xml files.
Comment: @greenapps indeed yes! btn_sign is on a diff xml and exp_list on a different one. btn_sign is for a button that lets me sign in. after that i reach to a page which has expandable view. How do i make it work?
Comment: @greenapps Suppose I set it to :   setContentView(R.layout.activity_after_signin) Now the content is set to the xml file that has the expandanble view. But now since the contentview is set to that xml file, it wont recognize btn_sign because it is on another xml? Am i correct? If yes, how do i solve this?
Comment: Ypu have two pages/activites apparently. So use the one for the one and the other for the other.
Answer: You are using activity_main to set content view
setContentView(R.layout.activity_main);

Where is your activity_main.xml? Are you using right xml file?
I think you are refering to wrong xml file and your views are in another xml. please check.
Comment: Indeed. That will be it. `setContentView(R.layout.activity_main);` should be `setContentView(R.layout.after_signup);`.
Comment: @greenapps  I made the suggested changes but it still doesnt work. I will add the logcat to the question
Comment: @kumarranjan, i made it :setContentView(R.layout.activity_after_signin);
Comment: do i need to declare child and parent activity in the manifest file?
Comment: Also, i would really appreciate it if someone could explain why i had to make the above changes. i have a main activity and then second activity which leads me to after_signin. which xml file should i put in this command and why?   thanks
Comment: you are getting this exception because your layout file is not containing the view you are refering ie. btn_sign. Its simple to understand, you can access any view in your activity only if it exists in your activity. To set a content ie. layout or views to activity you use setContentView. I guess it will help you to understand the concept.
Comment: @KumarRanjan thank you for your input. So how is it done? I have my content view set to after_signup.xml and i have a find view by id for a button thats in another xml. can i set contentview to both these files? :/
Comment: No, you can't...at a time you can set content view to only one layout.
Comment: i dont understand. To make this work, Ill have to ditch the btn_sign = (Button) findViewById(R.id.btn_sign but then my button wont work. Is there another way to solve this?
Comment: Whats the design of your application? Do you have a screen with two buttons and an another screen with a expandable list view?
Comment: yes. imagine a first page which asks user to either sign in or sign up. Suppose the user wants to sign in, he will press the sign in button and will be redirected to a page where he can enter username and password and press a button "sign in" which directs him to the page with the expandable view.
Comment: For this case you need three different activity. Lets say activities A,B,C. Activty A will show two buttons hence it will use layout with two button and activity A will handle the click of these two buttons. Lets say user click button 1 ie. sign in button, it will start another activity B, which will use another layout with username, password and login button. On click of login button it will start activity C, which will contain your expandable view in content view. Hope now you get your answer.
Comment: Thats how ive made it so far.
Comment: Then why you want to refer the sign up button in activity with expandable list view? They should be on different activities, right? and in that case you can set content for both activities separately.
Comment: Thats how ive made it so far. page_first and its xml. another page for sign in and one for signup. I have now 2 activities and a main  activity . These two activities basically have textview, editview and a button in them. till now, I had the contentview set for activity_main. then i made another expandable page added that to the manifest file, edited mainactivity but it wont work because i have to change the contentview. I hope im making the situation clear.
Comment: yes. how do i set content for both activities separately, thats my question now.  In mainactivity i set it to the page with the expandable view and the other?
Comment: The code that you have shared have only one activity ie. MainActivity. Dude, I hope you understand the difference between activity class and layout xml..Please get your basic good first.
Comment: I do understand the difference between the layout xml and activity  but you are right, i dont know many things that i should know before building a test. I kind of grasp things as I go. I will try to ask better questions next time :)
Comment: btw you can accept it as answer if it solved your basic problem :)


Pytorch: Why does altering the scale of the loss functions improve the convergence in some models?

Question: I have a question surrounding a pretty complex loss function I have.
This is a variational autoencoder loss function and it is fairly complex.  It is made of two reconstruction losses, KL divergence and a discriminator as a regularizer. All of those losses are on the same scale, but I have found out that increasing one of the reconstruction losses by a factor of 20 (while leaving the rest on the previous scale) heavily increases the performance of my model.
Since I am still fairly novice on DL, I dont completely understand why this happens, or how I could identify this sort of thing on successive models.
Any advice/explanation is greatly appreciated.
Answer: To summarize your setting first:
loss = alpha1 * loss1 + alpha2 * loss2

When computing the gradients for backpropagation, we compute back through this formular. By backpropagating through our error function we get the gradient:
dError/dLoss

To continue our propagation downwards, we now want to compute dError/dLoss1 and dError/dLoss2.
dError/dLoss1 can be expanded to dError/dLoss * dLoss/dLoss1 via the cain rule (https://en.wikipedia.org/wiki/Chain_rule).
We already computed dError/dLoss so we only need to compute dLoss derived with respect to dLoss1, which is
dLoss/dLoss1 = alpha1
The backpropagation now continues until we reach our weights (dLoss1/dWeight). The gradient our weight receives is:
dError/dWeight = dError/dLoss * dLoss/dLoss1 * dLoss1/dWeight = dError/dLoss * alpha1 * dLoss1/dWeight

As you can see, the gradient used to update our weight does now depend on alpha1, the factor we use to scale Loss1.
If we increase alpha1 while not changing alpha2 the gradients depending on Loss1 will have higher different impact than the gradients of Loss2 and therefor changing the optimization of our model.
Comment: So because I am changing the gradients this way, I am probably avoiding some local minima that might cause my NN to get stuck right? But what I don't quite understand is that, if I remove alpha 1, loss 1 converges anyway (albeit slower), while loss 2 stays in a local minima. Also, is there anyway to be able to detect this kind of behavior in loss functions?
Comment: With "remove alpha1" do you mean chaning the formula to `loss = loss1 + alpha2 * loss2`? This would be equivalent to setting `alpha1` to 1. In that case you would still optimize towards loss1. If removing alpha1 means you change the formula to `loss = alpha2 * loss2` you should not optimize loss1. Thus optimizing loss2 may shape the network to give lesser loss1 a decrease of loss1 could of course happen by chance.
Comment: In the end the gradient of a weight is `dError/dWeight = dError/dLoss * (alpha1 * dLoss1/dWeight + alpha2 * dLoss2/dWeight)`. By changing alpha1 and alpha2 you can control which loss function "has a higher share" on the gradient. Choosing optimal alpha1 and alpha2s is called hyperparameter tuning. This may be worth a read ;) https://towardsdatascience.com/what-are-hyperparameters-and-how-to-tune-the-hyperparameters-in-a-deep-neural-network-d0604917584a
Comment: Those are fair comments, it is just that, in my specific case, when you set the alpha1 to 1, the model gets stuck for one of the tasks (which is one of the reconstructions) while the other reconstruction improves. In a way, loss 1 gets stuck in a local minima while loss two doesnt. Thanks for the link, I will read it in depth :)


Intervention With Differencing

Question: When conducting an intervention analysis with time series data (aka Interrupted Time series) as discussed here for example one requirement I have is to estimate the total gain (or loss) due to the intervention - i.e. number of units gained or lost (the Y variable).
Not entirely understanding how to estimate the intervention function using a filter function within R, I went about it in a brute force manner, hoping this is general enough to work in any situation.
Lets say that given the data
 cds<- structure(c(2580L, 2263L, 3679L, 3461L, 3645L, 3716L, 3955L,
    3362L, 2637L, 2524L, 2084L, 2031L, 2256L, 2401L, 3253L, 2881L,
    2555L, 2585L, 3015L, 2608L, 3676L, 5763L, 4626L, 3848L, 4523L,
    4186L, 4070L, 4000L, 3498L), .Dim = c(29L, 1L), .Dimnames = list(
        NULL, "CD"), .Tsp = c(2012, 2014.33333333333, 12), class = "ts")

we decide that the best fitting model is as follows, with the intervention function as
$m_t= \frac{\omega_0}{(1-\delta B)}X_t$ where $X_t$ is a pulse at October 2013.
fit4 <- arimax(log(cds), order = c(1,1,0),include.mean=FALSE,
               xtransf = data.frame(Oct13 = 1*(seq_along(cds)==22)),
               transfer = list(c(1,0))
               ,xreg=1*(seq_along(cds)==3))
fit4

#    ARIMA(1,1,0)

#    Coefficients:
#              ar1    xreg  Oct13-AR1  Oct13-MA0
#          -0.0184  0.2718     0.4295     0.4392
#    s.e.   0.2124  0.1072     0.3589     0.1485

#    sigma^2 estimated as 0.02176:  log likelihood=13.85
#    AIC=-19.71   AICc=-16.98   BIC=-13.05

I have two questions:
1) Even though we have differenced the ARIMA errors , to assess the intervention function which was then technically fit using the differenced series $\bigtriangledown X_t $ is there anything we need to do in order to "change back" the estimate of $\omega_0$ or $\delta$ from using $\bigtriangledown X_t $ to $ X_t $?
2) Is this correct: In order to determine the gain of the intervention, I constructed the intervention $m_t$ from the parameters. Once I have $m_t$ then I compare the fitted values from the model fit4 (exp() to reverse the log) to exp( fitted values minus $m_t$ ) and determine that over the observed period, the intervention resulted in  3342.37 extra units.
Is this process the correct one to determine the gain generally from an intervention analysis?
    int_vect1<-1*(seq_along(cds)==22)
    wo<- 0.4392
    delta<-0.4295

    mt<-rep(0,length(int_vect1))

    for (i in 1:length(int_vect1))
    {

      if (i>1)
      {
        mt[i]<-wo*int_vect1[i]+delta*mt[i-1]
      }

    }

    mt

sum(exp(fitted(fit4)) - (exp(fitted(fit4) - mt)))

Comment: Wondering if anyone can shed light on the proper way to estimate the impact of an intervention - generally, if the procedure I demonstrated is in fact correct?
Comment: This is a great question. I suspect that the choice of indicator function might not be the best choice depending on the nature of the intervention. Perhaps an exponential decay function, $m_t = \exp(\alpha(t - i)) \text{if} i \ge t \text{else} 0$ for some scaling parameter $\alpha$.
Answer: Assuming this is toy example:
To answer your first question:
1) Even though we have differenced the ARIMA errors , to assess the intervention function which was then technically fit using the differenced series ▽Xt is there anything we need to do in order to "change back" the estimate of ω0 or δ from using ▽Xt to Xt?
When you difference the data, you should difference the response/intervention variables. When you back difference (transform) after you model then this would  automatically take care  of differencing** I know this is very easy when you use SAS Proc ARIMA. I dont know how to do this R.
Second Question:
2) Is this correct: In order to determine the gain of the intervention, I constructed the intervention mt from the parameters. Once I have mt then I compare the fitted values from the model fit4 (exp() to reverse the log) to exp( fitted values minus mt ) and determine that over the observed period, the intervention resulted in 3342.37 extra units.
To determine, gain in intervention, you need to take exponent and then subtract -1, this would give the proportion or incremental effect. To demonstrate this in your case, see below. For the first month, the impact was 55% of original sales and rapidly decays. Cumulativelt you have 4580 units of incremental effect (Oct 13  thru Feb 2014. (I referred to Forecasting Principle and Applications by Delurgio P: 518. There is an excellent voluminous chapter on intervention analysis).
Someone please correct if this methodology is correct ?
Pulse intervention + decay is clearly not sufficient in this case, I would do a pulse + permanent level shift as shown in the diagram (e) below which is from the classic paper by Box and Tiao.
Comment: Hi @forecaster. How did you get 3170 as the effect? Here is what I did, I looked at the fitted values of the model, which was 8.64245833 (still on the log scale). Then, exp(8.64245833) = 5667.244674. Then, I took 8.64245833 - 0.4392 = 8.20325833. Since exp(8.64245833) - exp(8.20325833) = 2014.411599 this is the effect. exp(8.64245833)/ exp(8.20325833) = 1.55 which looked to me like support for this.
Comment: You used the actuals and the modeled effect I guess, versus my approach which used the model for both. I used the idea of what does the model say with and without the effect. Which is right?
Comment: Hi @B_miner, with logarithmic transformation scale we need to look at rate of change. The approach I outlined is direct approach per the textbook, I quoted. However, your approach is reasonable as well. I'll screenshot the textbook pages in the near future.
Comment: The rate of change was 0.55 which is also the rate of change in the model approach I took. I wonder which approach is more right? I lean towards mine since the approach is based on the model (fitted veruss actuals). If the model is very close to actual, the two approaches will be the sample. I would like to see the pages. I see the book looks out of print?
Comment: yes, the book is out of print. The book example is a permanent change vs. pulse intervention in your example. I think your approach is straight forward and accurate.
Comment: Has your reading lead you to believe that regardless of how you difference the model (including seasonal), the coefficients from arimax for the intervention are on the scale of the original data and hence you can use them directly in the transfer function formula for the intervention to get the effect? I cant find anywhere that states this but it seems to be the case from examples I found.
Comment: @B_Miner, differencing simply removes spurious corelation, and will not impact the coefficients.
Comment: It seems like that is true for intervention variables....from a couple examples (including from the Pankratz book you once recommended - which is great!) I have seen. But then there are other accounts, including a recent question I posted that IrishStat responded to, that seems the interpretation should be different with differencing. Confusing.
Answer: @forecaster  After allowing AUTOBOX to identify 3 outliers using 29 values (not inappropriate in y experience) a useful model was found  and here  . The residual acf plot does not suggest an under-specified model . The Actual/Fit/Forecast plot is  with Fit/Forecast here  . Forecaster had (correctly) previously mentioned how a pulse variable can morph into a level/step variable when a denominator coefficient of nearly 1.0 is introduced. In finding two level shifts (the most recent one starting in 9/2013) and a pulse at 10/2013 , the model presents a clearer picture. In terms of the impact of the pulse at 10/13 it is simply the value of the coefficient. HTH
Comment: Which of the two questions were you answering?
Comment: @Irsihstat, the real question is **NOT** about differencing and transforming, it is about **HOW** to handle the aforementioned in the context of transfer function models. The example provided in the OP is simply a toy example to help formulate the question. There are many instances where I had to take both differencing and tranformation in transfer function models.
Comment: The first question premised a model that assumed a log transform which I believe is not warranted. The pulse at 10/2013 = 1710 which is the estimate of the effect at 10/2013
Comment: @B_Miner you could say the level shift at 9/2013 raised things by 1480 thus the net lift at 10/2013 would be 1710+1480=3190
Comment: This post seems to be more of an extended comment on the question than it is an answer to any part of the question. Perhaps it could be augmented to address the questions directly?
Comment: The false premise of the first question is the crux of my answer : taking logs and incorporating unnecessary differencing is questionable/incorrect in my opinion ... thus my "answer" is  in part to correct the premise and to suggest the effect of the impact at 10/2013 is simply the sum of a temporary and permanent change. Even though the OP has accepted another answer I haven't.

Intervention With Differencing


Sequence defined by max$\left\{a_n, b_n \right\}$. Proving convergence

Question: Assume $(a_n)$ and $(b_n)$ are two real sequences, and define $$ c_n = \text{max}\left\{a_n, b_n\right\} $$ for $n \in \mathbb{N}$. Suppose $(a_n)$ and $(b_n)$ are two convergent sequences. Prove then that $(c_n)$ is also a convergent sequence, and that $$ \lim_{n \to \infty} c_n = \text{max} \left\{ \lim_{n \to \infty} a_n, \lim_{n \to \infty} b_n \right\}. $$
Attempt: $(a_n)$ and $(b_n)$ are both convergent. Hence there exists an $n_0 \in \mathbb{N}$ such that $\forall n \geq n_0: | a_n - L | < \epsilon$. Furthermore, there exists an $n_1 \in \mathbb{N}$ such that $\forall n \geq n_1: | b_n - K | < \epsilon $. Here $L$ and $K$ are the limits of resp. $(a_n)$ and $(b_n)$. Now let $n_2 = \text{max} \left\{n_0, n_1 \right\}$. Let $n \geq n_2$ be arbitrary. Then we have that $L - \epsilon < a_n < L + \epsilon$ and $K - \epsilon < b_n < K + \epsilon$. Since $c_n = \text{max}\left\{a_n, b_n\right\}$, we have that $c_n \geq a_n$ and $c_n \geq b_n$. So $ L - \epsilon < a_n \leq c_n$ and thus $ L - \epsilon < c_n$.
But now I'm stuck. Help would be appreciated!
Comment: I would probably split into three cases: $L K$, and tackle each separately.
Answer: Remember that $\max\{x,y\}=\frac{x+y+|x-y|}{2}$.
Thus $c_n=\frac{a_n+b_n+|a_n-b_n|}{2}$. So, if $\lim a_n=a$ and $\lim b_n=b$:
$$\lim c_n=\frac{1}{2}(\lim a_n+\lim b_n+|\lim a_n-\lim b_n|)=\frac{1}{2}(a+b+|a-b|)=\max\{a,b\}$$
Comment: I see, thanks. But what about proving convergence? Should I split that into three cases, depending on the two limits of $(a_n)$ and $(b_n)$?
Comment: No!. In general, if $\{x_n\}$ is a convergent sequence and $f:\mathbb{R}\to\mathbb{R}$ is continuos, then $\{f(a_n)\}$ is convergent and $\lim f(a_n)=f(\lim a_n)$. My answer shows that $\{c_n\}$ is convergent and find its limit
Comment: @sinbadh You wrote max$(x,y)= \frac{x+y + |x-y|}{2}$. I didn't get the logic behind this formulation. Can you provide me with any source or reasoning behind it?
Comment: @Damn1o1 If $x \geq y$, then $x-y$ is positive (or zero) and hence $|x-y|=x-y$. Then the formula simplifies to $\frac{x+y+x-y}{2}=x$.
Comment: @JannikPitt What about min$(x,y)$ will it be convergent?
Comment: @Damn1o1 Yes, because $\min(x,y)=-\max(-x,-y)$.
Comment: @JannikPitt The way I am interpreting this is that limit of min$(x,y)$ and limit of max$(x,y)$ exists thus making it convergent. Is it a good way to think in my head about the convergence of both cases ??
Answer: Let $M= \max (L,K).$ When you reach $$n>n_2\implies (\;(L-e<a_n<L+e)\land (K-e<b_n<K+e)\;)$$ you have, for any $n>n_2,$   $$M-e=\max (L,K)-e =\max (L-e,K-e)<\max (a_n,b_n)<\max (L+e,K+e)=$$ $$=\max (L,K)+e=M+e.$$
Comment: Thank you! The kind of proof I was looking for.


twig form_widget value variable doesn't work

Question: <code>{{ form_widget(form.category,{value:"3"}) }} //works!!
{{ form_widget(form.category,{value:'3'}) }} //works!!
{{ form_widget(form.category,{value:3}) }} // doesn't work !!

{{lastCatId}} // echos 3 !!!

{{ form_widget(form.category,{value:"lastCatId"}) }} //doesn't work ???
{{ form_widget(form.category,{value:'lastCatId'}) }} //doesn't work ???
</code>
I know after symfony 2.3. Value has to be quoted or double quoted. But I don't know why the variable doesnt work
Comment: What is the error? Doesnt work does not really help... I could assaume that value has to be an integer? Otherwise no idea if you do not copy the error-message!
Comment: I am sorry its not an error, I am trying to set "selected" on the select form. with 3 it works and defaults to number 3 but with the variable it doesn't default
Answer: never mind. I found the problem
{{ form_widget(form.category,{'value' : lastCatId|number_format }) }}
I had to cast it to int.


$GLOBALS['user']->uid returns same uid for different logged in users?

Question: I'm implementing hook_menu in my custom module. Implementation works fine except, it turns out that $GLOBALS['user']->uid returns always same value no matter which user is currently logged in. Sounds absurd! What could be the problem?
Below is a full hook_menu:
function my_tab_menu(){

    $current_user_id = $GLOBALS['user']->uid;  // $current_user_id=202 for each

    $query = new EntityFieldQuery();
        $result = $query
                ->entityCondition('entity_type', 'node')
                ->entityCondition('bundle', 'profile')
                ->propertyCondition('status', 1)
                ->propertyCondition('uid', $current_user_id)
                ->propertyOrderBy('created', 'DESC')
                ->range(0,1)
                ->execute();

    if (isset($result['node'])) {
        return array(
        'user/%/alumni-spotlight' => array(
                'title' => 'Alumni spotlight',
                'page callback' => 'user_tab_page',
                'page arguments' => array(1,$result['node'],$current_user_id),
                'access callback' => 'user_tab_access',
                'access arguments' => array(1),
                'type' => MENU_LOCAL_TASK,
            ),
           );
    }
}

Comment: hook_menu is cached, looks like the wrong place to be adding that code, especially the db_query. Use the %user load argument in your page callback instead for user pages
Comment: Also maybe include your full hook_menu() so we can see what you are trying to achieve and can give you a better answer for your specific problem.
Comment: @David Tomas, could you please explain further how I can use %user load argument? I tried this: `$uid = arg(1);` but I'm not getting user id form url? Why?
Comment: @rooby I edited the post and included the full hook_menu().
Answer: Hook menu is not for non-menu functionality.
Using %user in the menu item will call user_load() with whatever is in that part of the url (in this case the uid) and then when that part of the url is used in callback or access arguments it will pass in the loaded user object instead of the user id.
See wildcard loader arguments for more info on that.
To control who sees the tab, do your database check in the access control callback and deny access if there is no result. This way the tab will only show if there is a node.
Callbacks like the access control call back and the title callback run every time, not just when the cache is refreshed, so it is ok to use them in this way.
Try
/**
 * Implements hook_menu().
 */
function MYMODULE_menu() {
  $items = array();

  $items['user/%user/alumni-spotlight'] = array(
    'title' => t('Alumni spotlight'),
    'page callback' => 'MYMODULE_user_tab_page',
    'page arguments' => array(1),
    'access callback' => 'MYMODULE_user_tab_access',
    'access arguments' => array(1),
    'type' => MENU_LOCAL_TASK,
  );

  return $items;
}

/**
 * Access callback for user alumni spotlight page.
 */
function MYMODULE_user_tab_access($user) {
  // Do something here to check access & return TRUE or FALSE to grant access or not.
  $query = new EntityFieldQuery();
  $result = $query
    ->entityCondition('entity_type', 'node')
    ->entityCondition('bundle', 'profile')
    ->propertyCondition('status', 1)
    ->propertyCondition('uid', $user->id)
    ->propertyOrderBy('created', 'DESC')
    ->range(0, 1)
    ->execute();

  // If there is a node, allow the user to access the page.
  if (isset($result['node'])) {
    return TRUE;
  }

  // If there is no node, deny access.
  return FALSE;
}

/**
 * Page callback for user alumni spotlight page.
 */
function MYMODULE_user_tab_page($user) {
  // Return the content of the page.
}

Comment: In that case, do your entityFieldQuery in the MYMODULE_user_tab_access($user) function and return true if there is a node and false if there is not one. - I'll tweak the answer
Comment: Thanks robby! I think this code can't work in my scenario. Note that I need to add tab **only** if EntityFieldQuery() returns a result. If not, tab should not be showed at all. Makes sense?
Comment: Wow that's awesome Roby! That is the solution I was hoping for. For some reason I couldn't realize that access callback is the key here. :) THX!
Answer: As I can see from your hook_menu implementation, you're trying to add a tab on the user page if the user published at least one content of type 'profile'. And inside this tab you want to show the latest 'profile' the user has published.
Your implementation is wrong. First, for the reason that the other answers also mention, hook_menu will be cached. Second, you shouldn't do the logic in hook_menu, but in the page or access callback functions.
However, there is also another way to get what you want to achieve, using Views.
Using hook_menu:
function my_tab_menu() {
  return array(
    'user/%/alumni-spotlight' => array(
      'title' => 'Alumni spotlight',
      'page callback' => 'user_tab_page',
      'page arguments' => array(1),
      'access callback' => 'user_tab_access',
      'access arguments' => array(1),
      'type' => MENU_LOCAL_TASK,
    ),
  );
}

This way you register a menu item, that can be cached. In the page callback you can do the query and show the content. Note that, even if you don't want to show the menu item if there is no result, you should register it with hook_menu. With 'access callback' function you can decide where to show or not.
Using Views

Create a View page and give it a path user/%/alumni-spotlight, show fields
Items to display: 1, no pager.
Set Menu: Menu Tab
Add relationship: 'Content: Author'
Add filter: 'User: Current', select relationship 'author', is the logged in user: 'yes'
Comment: your guess is pretty close. Everything you wrote is correct except I'm not showing a profile in the tab. Instead, I'm showing a link to specific node (profile) that I'm getting from query result. I look forward to getting you suggestions. thx a bunch!
Comment: So the scenario: I log into your site, and if **I** ever had published a node (profile), I'll see an extra tab (Alumni Spotlight) on **EVERY** user's page, which includes a link to **MY** latest node. Right?
Comment: That is correct! Note that I had tried  `return FALSE;` in  page callback funtion but it's not working!
Comment: Updated my answer and deleted the option with Panels, since according to your correction on my assumptions it wouldn't make many sense to do it with Panels. But the Views option does it well.
Comment: Thanks again for your help! I was considering that choice (view implementation) when I was at the beginig but later realized it would be more efficient way to get it done without view, using just a db_query.
Answer: Can you try using
global $user;
$uid = $user->uid;

Regards`
Comment: It still won't work because he has the code in hook_menu(), which is cached.
Comment: I had tried that before. It is basically the same thing as $GLOBALS['user']->uid; Won't work!
Comment: Hello. Code-only answers are hardly any better than link-only ones. They may answer how to make things work, but does not answer why, what was wrong, what mechanisms take place. Could you elaborate a bit more why you was thinking it might help?
Answer: Try using cache_clear_all() above the global $user. This will clear the cache and the present uid can be obtained.
Note : I have not tried this hook before.
Comment: Tried but no effect :(. Clearing cache manually doesn't work either.


Unable to use Qt + DXFLib

Question: I'm going to use dxflib in my Qt application. Build process is OK but when I'm going to run it, the application stops working at the beginning with this error :
Unhandled exception at 0x776415de in DXFOpen.exe: 0xC00000FD: Stack overflow.

Here is my simple app :
class CreatorAdaptor : public DL_CreationAdapter
{
    QGraphicsScene *_scene;
public:
    explicit CreatorAdaptor(QGraphicsScene *scene);
    virtual void addLine(const DL_LineData &d);
    virtual void addLayer(const DL_LayerData& data);
    virtual void addPoint(const DL_PointData& data);
    virtual void addArc(const DL_ArcData& data);
    virtual void addCircle(const DL_CircleData& data);
    virtual void addPolyline(const DL_PolylineData& data);
    virtual void addVertex(const DL_VertexData& data);
    virtual void add3dFace(const DL_3dFaceData& data);

};

And the definition of the class :
CreatorAdaptor::CreatorAdaptor(QGraphicsScene *scene) :_scene(scene)
{
}

void CreatorAdaptor::addLine(const DL_LineData &d)
{
    qDebug()<<"add line";
    _scene->addLine(d.x1, d.y1, d.x2, d.y2);
}
...

Here is my main :
int main(int argc, char *argv[])
{
    QApplication a(argc, argv);
    QGraphicsView  *view = new QGraphicsView;
    QGraphicsScene  * scene = new QGraphicsScene;
    view->setScene(scene);
    CreatorAdaptor *dxfscene = new CreatorAdaptor(scene);
    DL_Dxf dxf;
     if(dxf.in("demo.dxf", dxfscene))
         QMessageBox::warning(0,"error", "error");
    view->show();
    return a.exec();
}

Call Stack :
    ntdll.dll!77af15de()
    [Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]
    ntdll.dll!77af15de()
    ntdll.dll!77ae014e()
    kernel32.dll!755e14dd()
>   msvcr100d.dll!_free_base(void * pBlock)  Line 50 + 0x13 bytes   C
    0041f648()
    QtCored4.dll!qFree(void * ptr)  Line 60 + 0xa bytes C++
    QtCored4.dll!QVectorData::free(QVectorData * x, int alignment)  Line 82 + 0x9 bytes C++
    QtCored4.dll!QVectorTypedData<char *>::free(QVectorTypedData<char *> * x, int alignment)  Line 99 + 0x10 bytes  C++
    QtCored4.dll!QVector<char *>::free(QVectorTypedData<char *> * x)  Line 468 + 0x12 bytes C++
    QtCored4.dll!QVector<char *>::~QVector<char *>()  Line 124 + 0x30 bytes C++
    QtCored4.dll!qWinMain(HINSTANCE__ * instance, HINSTANCE__ * prevInstance, char * cmdParam, int cmdShow, int & argc, QVector<char *> & argv)  Line 193   C++
    0041f7e0()
    DXFOpen.exe!__security_init_cookie()  Line 143  C
    DXFOpen.exe!WinMainCRTStartup()  Line 371   C
    kernel32.dll!755e339a()
    ntdll.dll!77b09ef2()
    ntdll.dll!77b09ec5()

Comment: I meant edit your question and post it there, not post it in the comments.
Comment: Do you know how to use a debugger? If not, see [this tutorial](http://qt-project.org/doc/qtcreator-2.5/creator-debugging-example.html) for debugging with Qt Creator.
Comment: @sashoalm ~> Are you joking ?! Application doesn't start at all ! Even debugging stops!
Comment: I am not joking. Did you run your program under a debugger before posting this question?
Comment: Sure ! I'm a programmer like you! :D
Comment: OK, on which line did you get the unhandled exception?
Comment: The DXF part of code works perfectly outside of a Qt application.
Comment: @sashoalm~> wait a moment !
Comment: in **qWinMain** function at line `argv = qWinCmdLine(cmdParam, int(strlen(cmdParam)), argc);appCmdShow = cmdShow;` . As I said it's not a regular expection!
Comment: What is the entire call stack, all the functions that were called.
Comment: "too long by 262 characters" Stackoverflow said ! :D . May I send the project to your mail pls ?! It doesn't depend on any external lib! Then post the solution here and I'll accept it :D .
Comment: @sashoalm ~> :| OK. got it!
Comment: Btw, how come your project doesn't depend on any external lib, I thought you use dfxlib in it?
Comment: @sashoalm : I added its source files and headers to my project.
Comment: First I checked it with linking the dxflib.lib to my project and then I tried to compile the source files with my project. None of them worked !
Answer: I think you have to write all of the virtual functions as you did with below:
void CreatorAdaptor::addLine(const DL_LineData &d){
     // something something
}

You may left empty them but you must define them because they are defined as virtual functions in the dxf library. Of course, the other option is to delete all of virtual function definitions in the CreatorAdaptor class.

Unable to use Qt + DXFLib


Can I add Vertical Tab on a Node Edit Page in admin

Question: I am hoping to add a custom Vertical Tab in node edit page. Is it possible?
As an example, someone did it like this:-

Thanks
Answer: You can achieve this with hook_form_alter. Similar example can be found in the Examples for Developers module. Take a look at this after you are done with your form tabs, add a custom submit handler like $form[#submit][] = 'my_custom_submit'; and then save the data in submit handler using variable_set() method like this:
variable_set('MYMODULE_' . $form_state['values']['your_value'] . '_special_option', $values);

Note: put it in template.php or in a custom module.
Comment: Thanks Shabir ... should I put it in template.php?
Comment: Yes. or in custom module
Comment: Please check my edits I just applied, to ensure I didn't introduce anything you didn't want. Maybe you also want to reword the "**Note**" part I added somehow (eg: remove the "note"-word itself?). If anything else fails: just rollback
Comment: Thanks, it makes sense to everyone now. I am sorry for my poor english ;)
Answer: You can use this module to add groups and 1 of the options is vertical tabs:
https://www.drupal.org/project/field_group


What is "class" in mutual information based feature selection?

Question: I'm having a little hard time understanding this specific feature selection algorithm.
Specifically, I am looking into maximum-relevance-minimum-redundancy method for feature selection.
If I have a feature matrix $\ X = \{x_1, x_2, ... , x_n\} ,  x_i \in R^D $
then I can compute mutual information between two specific individual features, but when talking about
relevance of that specific feature to a "class", in terms of mRMR algorithm,
what is exactly the definition of "class" ?
Comment: If you look at Tom Cover's Information Theory book the result will be clear.
Answer: With no knowledge about this particular algorithm, I would guess that the word "class" is being used in the usual way in machine learning, which means "level of a discrete dependent variable". For example, if you're trying to build a classifier that distinguishes men and women, then there are two classes, men and women.
Comment: But the mRMR algorithm requires that I find the mutual information of a feature to that "class", and I'm not so sure how I could calculate that. but, as Michael said, I will take a look at Tom Cover's book about information theory


Is the p-value equivalent to the false alarm value in the Bayesian rule?

Question: Is a p-value from a traditional significance test the same as the false alarm value in the Bayesian rule?  And/or is it "close enough" to give correct results when used that way?
The definitions of the two terms seem to be talking about the same things, but I know it's easy to be tripped up by subtleties.  Wikipedia says that a p-value is "the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true."
This page says the false error rate P(D|H') in the Bayesian rule is the probability of observing D if H' is true.
In both cases, you are talking about the probability of seeing data that says H when the reality is H'.
However, I see two possible problems with assuming that the two are equivalent: D in the definition of P(D|H') seems to refer to a single datum, while the definition of a p-value seems to refer to a range of values ("at least as extreme as"); and I'm not quite smart enough to figure out whether H' is equivalent to the null hypothesis.  In all the simple Bayesian examples I've worked through it certainly seems to be, but I haven't yet found a definitive statement.
I also haven't found a definitive statement of how p-value and the false alarm value are related if they aren't the same, given that they're both saying at least loosely analogous things about data and hypotheses.
Comment: The concept of probability can be different (in most cases they are). Also, the p-value has asymptotic properties that the Bayesian counter part doesn't necessary have. So, be carefull.
Answer: They're related in that selecting the critical value for p amounts to selecting a false alarm rate, under the assumption that the null hypothesis is true. The difference between the two is what is observed and known prior to the test.
It's conventional to denote the probability of making a type I error, i.e. a false positive, as $\alpha$; in the example you linked above, $\alpha$ is known from observations of previous tests. In other words, under the (null) hypothesis that a patient doesn't have cancer, $\alpha = 0.2$. In that example, the prior probability of a positive condition is also known, and is what ultimately allows for computing the odds of cancer.
In a traditional significance test, the idea is that the experimenter sets a critical value for p at $\alpha$, such that they only report false findings one in $\alpha^{-1}$ times. Because experimenters wish to avoid reporting false positives, they set their own false alarm rate under the assumption that their test hypothesis is incorrect. You can consider a test result of $p < \alpha$ equivalent to the positive test result D in the example.


Nginx serving C++ cgi script: response is binary format

Question: I am trying to run C++ CGI script on nginx. I am using FCGIWrap with script from nginx website. The program code is like this:
#include <iostream>
using namespace std;

int main ()
{

   cout << "Content-type:text/html\n\n";
   cout << "\n";
   cout << "\n";
   cout << "<title>Hello World - First CGI Program</title>\n";
   cout << "\n";
   cout << "\n";
   cout << "
Hello World! This is my first CGI program
\n";
   cout << "\n";
   cout << "\n";

   return 0;
}

And I compiled with g++ -o start.cgi start.cpp. So when I run ./start.cgi, I got the correct output. But when I use curl localhost/cgi-bin/start.cgi, I got binary output(and actually see cout responses and information like GCC ... So I suspect it is the compiled executable file)
My nginx.conf:
# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user root;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        root         /var/www;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location /cgi-bin/*\.cgi {
          gzip off;
          fastcgi_pass unix:/var/run/fcgiwrap.sock;
          include /etc/nginx/fastcgi_params;
          fastcgi_param SCRIPT_FILENAME /var/www/cgi-bin$fastcgi_script_name;
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }
}

Answer: Long story short, you need to set your cgi location to use regex matching.
location ~ /cgi-bin/.*\.cgi

Comment: But we like long stories here!


Problem with showing properties of linear transformation

Question: Let $F:K^3\rightarrow K^3$ be a linear transformation, such that

$(F-\lambda I)(F-\mu I)\neq0,$
$(F-\mu I)^2\neq 0,$
$(F-\lambda I)(F-\mu I)^2=0.$

With this knowledge I want to show that

$\lambda$, $\mu$ are eigenvalues of $F$,
$Ker (F-\mu I)\subseteq Im(F-\mu I)$,
$F$ has in a certain basis matrix
\begin{equation*}
\begin{pmatrix}
\lambda & 0 & 0  \\
0 & \mu & 1  \\
0  & 0 &\mu
\end{pmatrix}.
\end{equation*}

My attempt:

It seems to be a consequence of the third assumption, but I am not sure.
We know that $Ker(F-\mu I)=\{v\in K^3:F(v)-\mu v=0\}$ and $Im(F-\mu I)=\{(F-\mu I)(v):v\in K^3\}=\{(F(v)-\mu v):v\in K^3\}$. Then the inclusion seems to be trivial, but I am also not sure.
Unfortunately I have no idea how to begin.
Comment: Why do you use the non-classical name "transition" instead of "transformation" ?
Comment: I got used to this name. Anyway, I'll edit it.
Comment: What about the transformation $v\mapsto\lambda v$ for all $v\in K^3$ with $\lambda\neq\mu$? It satisfies all the conditions, but its only eigenvalue is $\lambda$.
Comment: @Vercassibelaunos unfortunately I don't understand your advice.
Comment: @Novice He is telling you that the statements that you want to show are incorrect. The transformation he describes is a counterexample.
Comment: I have done some typos. Maybe now situation is clearer
Answer: I will assume $\lambda\neq \mu$.
Note that the factors $F-k I$ commute for all $k$.

Let $v$ be a vector $v\in K^3$ such that $(F-\lambda I)(F-\mu I)\,v=u\neq 0$. Then:
$$
(F-\mu I)\,u=(F-\lambda I)(F-\mu I)^2\,v=0
$$
so $u$ is an eigenvector for $\mu$.

Analogously, let $w$ be a vector $w\in K^3$ such that $(F-\mu I)^2\,w=z\neq 0$. Then:
$$
(F-\lambda I)\,z=(F-\lambda I)(F-\mu I)^2\,w=0
$$
so $z$ is an eigenvector for $\lambda$.

In part 3. we can see that the dimension of $Ker(F-\mu I)$ is 1, otherwise $F$ would be diagonalizable. So $u\in Ker(F-\mu I)$ from previous point is a basis of $Ker(F-\mu I)$. But $u=(F-\mu I)v'$ where $v'=(F-\lambda I)v$ thus $u\in Im(F-\mu I)$.

NOTE: I do not like this proof because it uses part 3. but right now I cannot think of something easier.

The polynomial $(F-\lambda I)(F-\mu I)^2$ from statement 3 is the characteristic polynomial, because it has degree three, and it does not vanish when removing any of its factors. Thus $\mu$ has multiplicity 2 and the diagonal of the Jordan form of $F$ is $(\lambda,\mu,\mu)^\top$. But if $F$ were diagonalizable, then $(F-\lambda I)(F-\mu I)=0$, which is in contradiction with statement 1.
Comment: I really appreciate your solution. Can I ask why _$dimKer(F−μI)$ is 1, otherwise $F$ would be diagonalizable_?  $F$ would be diagonalizable if $dimKer(F−μI)=2$ in our case?
Comment: If there is a basis of eigenvectors, then $F$ is diagonal when expressed in that basis. To be an eigenvector corresponding to $\mu$ is equivalent to belonging to $Ker(F-\mu I) $.


Show that all the zeros of $g$

Question: Let us consider the function:
$$g(α,β)=\sum_{n=1}^{\infty}(-1)^{n-1}((n^{2α-1}-1)/n^{α}) n^{iβ}$$
My question is: Show that all the zeros of $g$ in $0<α<1$ have the form $(1/2,β)$ where $β$ is real.
Comment: Is this not equivalent to the Riemann Hypothesis?
Comment: Not to be too harsh about it, but... do you have any reason to believe that this phrasing of the question will bring some insight to the problem that a hundred years of concerted mathematical effort have not?
Comment: @robjohn: No, this assertion would imply the Riemann hypothesis. See http://vixra.org/author/zeraoulia_elhadj.
Answer: As robjohn suggests in his comment, this assertion would imply the Riemann hypothesis and proving it would win you a million dollars.
To be precise, your equation is equivalent to $$\eta(s)=\eta(1-\bar s),$$ where $s=\alpha-i\beta$ and $$\eta(s)=\sum_{n=0}^\infty \frac{(-1)^{n-1}}{n^s}=(1-2^{1-s})\zeta(s)$$ denotes the Dirichlet Eta function.
If the Riemann hypothesis were false, there would be an $s$ with $0<\Re{ s}<1$, $\Re s\neq 1/2$ such that $\zeta(s)=0$ and (by the functional equation of the zeta function) $\zeta(1-\bar s)=\overline{\zeta(1-s)}=0$, hence you would obtain a solution to $\eta(s)=0=\eta(1-\bar s)$ with $\alpha\neq 1/2$.

Show that all the zeros of $g$


Understanding matrix representation of a linear map

Question: Let $V$ be a vector space over a field $K$ with ordered basis $B$. If $F:V\rightarrow V$ is a linear mapping then $M_B^B(F)$ is called the matrix representation of $F$ with respect to $B$.
Does this definition mean $F(v)=M_B^B(F)v$ for all $v\in V$?
Comment: Yes, the definition means that. That $B$ in the subscript corresponds to the basis in the domain, whereas the other corresponds to the basis in the codomain
Answer: The matrix $M_B^B(F)$ is chosen precisely so that, if you write $v$ in the basis $B$, then the coordinates of $Fv$ in the basis $B$ are $M_B^B(F)\,v_B$.
Answer: Strictly speaking, the expression $M_B^B(F) v$ is not defined in general, as $M_B^B(F)$ is a matrix in $M(n, K)$ and $V$ is an abstract vector space (here, we assume that $n := \dim V$ is finite).
The appropriate statement is this: The basis $B$ determines vector representations $[w]_B \in K^n$ of vectors $w \in V$, in which case we have the matrix equation
$$[F(v)]_B = M_B^B(F) [v]_B$$
(for all $v \in V$). In fact, you can think of our conventions as being true exactly so that this is true (or rather the generalization of this to general linear maps $V \to W$ between f.d. vector spaces).
In this notation, we often write the matrix representation of $F$ w.r.t. $B$ using the 'functorial' notation $[F]_B$, giving the pleasing equation
$$[F(v)]_B = [F]_B [v]_B .$$


PG::Error: ERROR: relation "mailboxer_notifications" does not exist

Question: I just updated to rails 4.1 and mailboxer doesn't seem happy for the change. Below is what happens when I try to get inbox from my console.
<code> [18] snifme »  Master.last.mailbox.inbox
  Master Load (0.6ms)  SELECT  "masters".* FROM "masters"   ORDER BY "masters"."id" DESC LIMIT 1
PG::Error: ERROR:  relation "mailboxer_notifications" does not exist
LINE 5:                WHERE a.attrelid = '"mailboxer_notifications"...
                                          ^
:               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod
                FROM pg_attribute a LEFT JOIN pg_attrdef d
                  ON a.attrelid = d.adrelid AND a.attnum = d.adnum
               WHERE a.attrelid = '"mailboxer_notifications"'::regclass
                 AND a.attnum > 0 AND NOT a.attisdropped
               ORDER BY a.attnum

(pry) output error: #<ActiveRecord::StatementInvalid: PG::Error: ERROR:  relation "mailboxer_notifications" does not exist
LINE 5:                WHERE a.attrelid = '"mailboxer_notifications"...
                                          ^
:               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
                     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod
                FROM pg_attribute a LEFT JOIN pg_attrdef d
                  ON a.attrelid = d.adrelid AND a.attnum = d.adnum
               WHERE a.attrelid = '"mailboxer_notifications"'::regclass
                 AND a.attnum > 0 AND NOT a.attisdropped
               ORDER BY a.attnum
</code>
Does anything stand out as fixable? I'm in development environment. thanks
Comment: why the downvote on this one? this is a gem issue, not an obvious postgresql trivial issue he/she missed
Answer: I found that I needed to run <code>$ rails generate mailboxer:namespacing_compatibility</code>for rails 4.1 upgrade
Answer: I was also getting the same issue and above command will not working for me.
You need to mention the 'mailboxer' gem version to '0.11.0' in Gemfile.If you do not mention any version it could use higher version (currently '0.12.0' is higher) and your page can break.
It happens on production environment (heroku etc) mostly.

gem 'mailboxer', '0.11.0'

Thanks.
Comment: README mentions to run the command from the other answer when upgrading to 0.12.0. More details here: https://github.com/mailboxer/mailboxer#upgrading


Heavy Metals in Leaves of Rice Plant in Organic and Conventional Rice Field

Question: what statistical tool should I  use to compare heavy  metal concentration between organic and conventional rice field? the heavy  metals are Cadmium,Cobalt, Iron, Lead, and Chromium. What should I use if the variety of rice is ignored, and if the variety of rice will be considered
Answer: Statistics is a tool that is useful only if one is able to translate the problem at hand, at least to some degree, into mathematical formalism. In your case this means you should probably start your analysis with the following questions:
1) What is my response variable of interest $Y$ and on what scale is it defined. I guess that heavy metal concentration is somehow positive continuous but it might by bounded from above $Y\in[0,1]$? Depending on your answer to this question you can think of a probabilitstic model that allows to describe the random behaviour of such a variable.
2) What are my covariates of interest and how are they coded and related to your response? In your case this might be related to the variety of rice. Depending on your research setting you should think clearly whether to include or exclude explaining covariates.
3) What kind statistical hypothesis you want to test? This typically starts with defining a Null Hypothesis as for example in your case: $"H_0":$ The concentration of heavy metal in an organic rice field is smaller, equal to the concentration of heavy metal in a conventional rice field. Or formalized if $Y_o$ ($Y_c$) is the concentration in an organic (conventional) rice field,  $H_0:Y_o\leq Y_c$. Then your research design, based on your probability model from above allows you to reject $H_0$ given some measure of certainty. In your case this could be for example a comparison of means if you ignore covariates or a regression-based test if you want to control for covariates. This however, also depends on the data at hand. I.e. whether you have cross-sectional information or time-series and/or panel data.


Windows Azure Website with a Startup task

Question: I know that Azure Websites are made to be easy, not have role configs and such...but I was still wondering, is there ANY way to create a Startup Task when using a Website?
I'm asking because I would like to ease the deployment of an existing website through FTP or GIT (not recompiling the source and such), and then use Startup Task to deploy an additional IIS provider to GAC?
Kind regards,
Vladan
Answer: In case of "Shared Instance" deployment model, I don't think it's possible because your application is sharing the VM with many other tenants (applications). Thus you will not have permissions to access GAC. Not sure how this would work in case of "Reserved Instance" deployment model where the whole VM is at your disposal.
Comment: I don't think it is feasible in the Reserved Instances either. Although the VM is "at your disposal", it is still maintained by MSFT, you are not deploying the worker role code.
Answer: Actuall it's possible. Have a look at: Windows Azure Website with a Startup task using Webmatrix 3 or similar (without using Visual Studio)
However, you can't run it with elevated privileges.


the degree of the discriminant of a cubic irreducibble polynomial over the rational adjoint a root

Question: Let $f(x) \in \mathbb Q[x]$ and $$f(x)= x^3 + px +q=(x-r_1)(x-r_2)(x-r_3),$$  and $r_1, r_2,r_3\in K$ for the splitting field $K$ of $f(x)$.
Let the $D = (r_3 - r_1)^2(r_2- r_1)^2(r_3 - r_2)^2$, which is the discriminant of $f(x)$.
The square root of the discriminant $D$ is clearly in the splitting field because $\sqrt D= |(r_3 - r_1)(r_2- r_1)(r_3 - r_2)|$.
How to find
$[\mathbb Q(r_1)(\sqrt{D}):\mathbb Q]$, i.e. the degree of the extension field of $\mathbb Q$ adjoining $\sqrt D$ and $r_1$?
I tried to find the degree of $\sqrt D$ over $\mathbb Q(r_1)$ and did not find the minimal polynomial of it. The degree is surely at most $2$. It is just not clear to me whether it is $1$ or $2$, or it depends on $f(x)$.
related: Splitting field of a degree 3 polynomial is generated by a root and the square root of the discriminant
Since proposition points out this extension field is simply the splitting field of $f(x)$ over $\Bbb Q$.
In short, my question:

what is the degree of $\sqrt D$ (over $\Bbb Q(r_1)$ and $\Bbb Q$) ? What are all possibilities and what condition does these depend on?
Equivalently, what is the degree of the splitting field (which is equal to $\Bbb Q(r_1)(\sqrt D)$ according to the linked question) of $f(x)$ given its discriminant? (Is there even relations between them?)
Comment: I came up with the following: $[\Bbb Q(r)(\sqrt D) : \Bbb Q(r)]$ is either $1$ or $2$. When  $D$ is not a square, so $\sqrt D \not\in \Bbb Q$, I claim $$[\Bbb Q(r)(\sqrt D) : \Bbb Q(r)] =2$$. Because if it is equal to $1$, then $\sqrt D \in \Bbb Q(r)$ meaning $Q(\sqrt D)$ is a subfield of $\Bbb Q(r)$, hence $3 = [\Bbb Q(r) : \Bbb Q] = [\Bbb Q(r): \Bbb Q(\sqrt D)][\Bbb Q(\sqrt D) : \Bbb Q]$. This would imply $2\mid 3$ a contradiction, from where I conclude $[\Bbb Q (r)(\sqrt D):B(r)] = 6$ if $\sqrt D \not\in \Bbb Q$.
Comment: OK. You ask, "what condition does these [sic] depend on?" The condition it depends on is whether $-27q^2-4p^3$ is a square.
Comment: If Galois Theory is in your toolbox, then $D$ is rational because it is invariant under all elements of the Galois group of the splitting field. If you don't have Galois Theory, it's a little hard to explain, but, very loosely, the idea is that any bijection $\sigma$ from the splitting field to itself that preserves the field operations must act as a permutation on the roots of $f$, from which it follows that $\sigma(D)=D$; but the only elements of the splitting field that are fixed by all the operation-preserving bijections of the splitting field are the rationals.
Comment: Are you familiar with the expression for $D$ in terms of $p$ and $q$? When that expression is a perfect square, then of course $\sqrt D$ is an integer, so it's of degree one over the rationals. Otherwise, degree two. The degree one case is called a *cyclic cubic*.
Comment: Yes. I know that $D=-27q^2 -4p^3$.
Comment: I was actually a bit confused on this point.   Now I get it. "If this expression is a perfect square, then of course $\sqrt D$ is an integer". I wondered for a while if it would be valid to say "D is a square in $\Bbb Q$. "without pointing out that $D$ is expressed using coefficients in $\Bbb Q$. Perhaps this is because I need to know that $D$ is in $\Bbb Q$ in the first place, before I can assume that it is or is not a square in $\Bbb Q$...
Answer: The degree of $\sqrt D$ over the rationals is one if $D=-27q^2-4p^3$ is a square, two otherwise. The degree of the splitting field is three if $D$ is a square, six otherwise.
The full story is best explained via Galois Theory, which is a bit of a long story. Copying from my comment on the question,
"very loosely, the idea is that any bijection $\sigma$ from the splitting field to itself that preserves the field operations must act as a permutation on the roots of $f$, from which it follows that $\sigma(D)=D$; but the only elements of the splitting field that are fixed by all the operation-preserving bijections of the splitting field are the rationals."


How to extract subset of data in groups before and after a string

Question: I have a text file. In text file based on specific word it should make the data into two groups like everything before specific word as 1 group and every thing after specific word as another group
text file some thing like this
hello every one
Is any space here?

CHAIN

every thing of the

file lies here

Based on CHAIN we separate text file into two groups
group 1
hello every one
Is any space here?

group 2
every thing of the

file lies here

Comment: you mean use `split`?
Comment: you can split string by keyword, just load your file to some variable, for example `contents`, and call split method with your keyword like
`contents.split("CHAIN")`
Answer: You mentioned you have a text file say test.txt.
You code:
with open("test.txt", "r") as f:
    data = f.readlines()

part1, part2 = ("".join(data).split("CHAIN"))
print(part1)
print(part2)

Gives me:
hello every one
Is any space here?

every thing of the

file lies here

Otherwise other solution is also good.
Answer: you can try a solution with split ans access each string using index as given below.
a = """
hello every one
Is any space here?

CHAIN

every thing of the

file lies here
"""

print(a.split("CHAIN")[0])
print(a.split("CHAIN")[1])

Answer: just for completeness (other answers work as well):
if you have a text file
file = open('file.txt', 'r').read()

print(file.split('CHAIN'))

# if you want to remove the new spaces (\n)

print([text.strip() for text in file.split('CHAIN')])


wildly ramified extensions and $p$-power roots of unity

Question: Let $K$ be a finite extension of $\mathbf{Q}_p$. I have this vague intuition that $K$ having a lot of wild ramification is closely related to $K$ being close to containing high degree $p$-power roots of unity.
I was wondering if this intuition is true and if maybe the following result (which would formalise it) is true :
Let $e_K$ be the ramification degree of $K$ over $\mathbf{Q}_p$ then $p^n$ divides $e_K$ if and only if $K(\zeta_p)$ contains $p^n$ roots of unity where $\zeta_p$ is a $p$-root of unity.
If this is false is there a way to salvage the result/my intuition ?
Answer: The correct statement (for $p$ odd) is as follows. If $K/\mathbf{Q}_p$ is abelian, then $p^n | e_K$ if and only if $L(\zeta_p)$ contains $\zeta_{p^{n+1}}$ for some unramified extension $L/K$. Similarly, for $p$ odd, $p^{n-1}(p-1) | e_K$ if and only if $K$ contains $\zeta_{p^n}$ for some unramified extension $L$. It is false without the requirement that one can pass to an unramified extension, as can be seen by "crossing" the cyclotomic extension with an unramified one, as in the answer of nguyen (although there are $p+1$ cyclic extensions of $\mathbf{Q}_p$ of degree $p$ rather than $3$).
It also fails without the abelian hypothesis; there are extensions $K$ with arbitrarily large ramification such that $L \cap \mathbf{Q}_p(\zeta_{p^{\infty}}) = \mathbf{Q}_p$ for all unramified extensions $L/K$. For example, let $E/\mathbf{Q}_p$ be an unramified quadratic extension, and consider $\mathbf{Z}/p^n \mathbf{Z}$ extensions of $E$ on which $\mathrm{Gal}(E/\mathbf{Q}_p)$ acts by minus one.
Comment: Great ! Thank you very much :)) Do you have a reference for the first statement ?
Comment: It's an easy consequence of (local) class field theory, in particular the fact that the *maximal* abelian extension of $\mathbf{Q}_p$ is given by adjoining all roots of unity (or equivalently, by taking the maximal [abelian] unramified extension of $\mathbf{Q}_p$ and then adjoining all $p$-power roots of unity).
Answer: First, I think that your guess should be "$K(\zeta_p)$ contains $\zeta_{p^{n+1}}$", not just $\zeta_{p^n}$. Second, I think there is a counterexample, with $K$ being a cyclic extension of $Q_p$ of degree $p$, totally ramified ($p$ odd). Actually, by local CFT, $Q_p$ admits exactly 3 cyclic extensions $K_i$ of degree $p$, because $Q^*_p/(Q^*_p)^p$ considered (in additive notation) as a vector space over the prime field $F_p$ has dimension 2. The first extension, say $K_0$, is the unramified one. The 2 others, say $K_1$ and $K_2$, are necessarily totally ramified, and their compositum contains $K_0$. Add $\zeta_p$ to get fields $L_i=K_i(\zeta_p)$. If your guess were founded, we would have $L_i=Q_p(\zeta_{p^2})$ for $i=1, 2$, hence also for $L_0$ : impossible, since $L_0$ is unramified over $Q_p(\zeta_p)$.
Comment: This is because the cyclotomic p-extension of degree p^n over Q_p(zeta_p) is obtained by adding zeta_p^n+1
Comment: Hi thank you very much for your answer. But why should my guess be $K(\zeta_p)$ contains $\zeta_{p^{n+1}}$ ? Do you think it's still false with $\zeta_{p^n}$ ?