Can a neural network learn a functional, and its functional derivative?

Question: I understand that neural networks (NNs) can be considered universal approximators to both functions and their derivatives, under certain assumptions (on both the network and the function to approximate). In fact, I have done a number of tests on simple, yet non-trivial functions (e.g., polynomials), and it seems that I can indeed approximate them and their first derivatives well (an example is shown below).
What is not clear to me, however, is whether the theorems that lead to the above extend (or perhaps could be extended) to functionals and their functional derivatives. Consider, for example, the functional:
\begin{equation}
    F[f(x)] = \int_a^b dx ~ f(x) g(x)
\end{equation}
with the functional derivative:
\begin{equation}
    \frac{\delta F[f(x)]}{\delta f(x)} = g(x)
\end{equation}
where $f(x)$ depends entirely, and non-trivially, on $g(x)$. Can a NN learn the above mapping and its functional derivative? More specifically, if one discretizes the domain $x$ over $[a,b]$ and provides $f(x)$ (at the discretized points) as input and $F[f(x)]$ as output, can a NN learn this mapping correctly (at least theoretically)? If so, can it also learn the mapping's functional derivative?
I have done a number of tests, and it seems that a NN may indeed learn the mapping $F[f(x)]$, to some extent. However, while the accuracy of this mapping is OK, it is not great; and troubling is that the computed functional derivative is complete garbage (though both of these could be related to issues with training, etc.). An example is shown below.
If a NN is not suitable for learning a functional and its functional derivative, is there another machine learning method that is?
Examples:
(1) The following is an example of approximating a function and its derivative: A NN was trained to learn the function $f(x) = x^3 + x + 0.5$ over the range [-3,2]:

from which a reasonable approximation to $df(x)/dx$ is obtained:

Note that, as expected, the NN approximation to $f(x)$ and its first derivative improve with the number of training points, NN architecture, as better minima are found during training, etc.
(2) The following is an example of approximating a functional and its functional derivative: A NN was trained to learn the functional $F[f(x)] = \int_1^2 dx ~ f(x)^2$. Training data was obtained using functions of the form $f(x) = a x^b$, where $a$ and $b$ were randomly generated. The following plot illustrates that the NN is indeed able to approximate $F[f(x)]$ quite well:

Calculated functional derivatives, however, are complete garbage; an example (for a specific $f(x)$) is shown below:

As an interesting note, the NN approximation to $F[f(x)]$ seems to improve with the number of training points, etc. (as in example (1)), yet the functional derivative does not.
Comment: Interesting question. How are you representing the input f of the functional F? I assume f is being quantized to some vector of f-values (say a vector of 1000 samples). If so, what does the x-axis of your third plot mean? It seems to be different than the x-axis of your 4th plot. Is the network being trained to learn F[f] and dF/df, or are you computing dF/df once the network is trained?
Answer: This is a good question. I think it involve theoretical mathematical proof. I have been working with Deep Learning (basically neural network) for a while (about a year), and based on my knowledge from all the papers I read, I have not seen proof about this yet. However, in term of experimental proof, I think I can provide a feedback.
Let's consider this example below:

In this example, I believe via multi-layer neural network, it should be able to learn both f(x) and also F[f(x)] via back-propagation. However, whether this apply to more complicated functions, or all functions in the universe, it require more proofs. However, when we consider the example of Imagenet competition --- to classify 1000 objects, a very deep neural network are often used; the best model can achieve an incredible error rate to ~5%. Such deep NN contains more than 10 non-linear layers and this is an experimental proof that complicated relationship can be represented through deep network [based on the fact that we know a NN with 1 hidden layer can separate data non-linearly].
But whether ALL derivatives can be learned required more research.
I am not sure if there any machine learning methods that can learn the function and its derivative completely. Sorry about that.
Comment: Thank you for your answer. I was actually a bit surprised at first that a neural network could approximate a functional at all. Accepting the fact that it could though, it then does then intuitively seem that information about its functional derivative should be contained in the solution (as is the case with functions), especially for simple functions and functionals (as in your example) In practice, however, this is not the case. In light of your example, I added some examples to my original post.
Comment: Cool, what is the setting for your neural network? Such as number of layers, hidden units, activation functions, etc.
Comment: I have tried various settings: 1-3 hidden layers, 5 to 100 hidden units (per layer), various numbers of input (while the functional is defined as the limit that this goes to infinity, I have tried as few as four points), sigmoid and tanh (normal, as well as that recommended by LeCun) activation functions, and various training methods (backpropagation, QRPROP, particle swarm optimization, and others). I have tried both in-house and some well-known software. While I can get improvement in approximating the functional as I change things, I can't in the functional derivative.
Comment: Cool. What software did you use? Have you done cross-validation to optimize your network setting? Here are some of my thoughts: (1) I would expect 3 or more hidden layers maybe required because the problem is highly non-linear, (2) try to use undercomplete setting for hidden units, i.e., input-100-50-20-output, instead of input-20-50-100-output, (3) use ReLU instead of sigmoid or tanh; a research publish few papers in 2010s and proved that ReLU can lead to better result, (4) parameters such as weight decay, learning rate are important, make sure you tune them appropriately, (5) caffe as a tool
Comment: Besides in-house software, I have used stats++, Encog, and NeuroSolutions (the latter was only a free trial, and I don't use it anymore). I have not yet tried cross validation to optimize things, but I will; I will also try your other suggestions. Thank you for your thoughts.
Comment: You're welcome. Also, look for caffe---it is a really great tool implemented by UC Berkeley folks, and it is published last years and got 250 citations already.
Answer: Neural nets can approximate continuous mappings between Euclidean vector spaces $f : \mathbb{R}^M \to \mathbb{R}^N$ when the hidden layer becomes infinite in size. That said, it's more efficient to add depth than width. A functional is simply a map where the range is $\mathbb{R}$ i.e. $N=1$. So yes, neural nets can learn functionals as long as the input is a finite dimensional vector space and the derivative is easily found by reverse-mode differentiation aka backpropagation. Also, quantising the input is indeed a good way to extend the network to continous function inputs.
Answer: If the functional is in the form
$$F[f(x)]=\int\limits_a^bf(x)g(x)dx$$ then $g(x)$ can be learned with a linear regression given enough training functions $f_i(x), ~i=0,\dots,M$ and target values $F[f_i(x)]$. This is done approximating the integral by a trapezoidal rule:
$$F[f(x)]= \Delta x\left[\frac{f_0g_0}{2}+f_1g_1+...+f_{N-1}g_{N-1}+\frac{f_Ng_N}{2}\right]$$
that is
$$\frac{F[f(x)]}{\Delta x}=y= \frac{f_0g_0}{2}+f_1g_1+...+f_{N-1}g_{N-1}+\frac{f_Ng_N}{2}$$
where $$f_0=a,~f_1=f(x_1),~...,~f_{N-1}=f(x_{N-1}),~f_N=b,$$
$$a<x_1<...<x_{N-1}<b,~~\Delta x=x_{j+1}-x_j$$
Suppose we have $M$ training functions $f_i(x),~i=1,\dots,M$. For each $i$ we have
$$\frac{F[f_i(x)]}{\Delta x}=y_i= \frac{f_{i0}g_0}{2}+f_{i1}g_1+...+f_{i,N-1}g_{N-1}+\frac{f_{iN}g_N}{2}$$
The values $g_0,\dots, g_N$ are then found as a solution of a linear regression problem with a matrix of explanatory variables
$$X=\begin{bmatrix}
    f_{00}/2  & f_{01} & \dots & f_{0,N-1}  & f_{0N}/2 \\
    f_{10}/2  & f_{11} & \dots & f_{1,N-1}  & f_{1N}/2 \\
    \dots & \dots & \dots & \dots & \dots\\
    f_{M0}/2  & f_{M1} & \dots & f_{M,N-1}  & f_{MN}/2
\end{bmatrix}$$
and the target vector $y=[y_0,\dots,y_M]$.
Let's test it for a simple example. Suppose, $g(x)$ is a Gaussian.
import numpy as np

def Gaussian(x, mu, sigma):
    return np.exp(-0.5*((x - mu)/sigma)**2)

Discretize the domain $x \in [a,b]$
x = np.arange(-1.0, 1.01, 0.01)
dx = x[1] - x[0]
g = Gaussian(x, 0.25, 0.25)

Let's take sines and cosines with different frequencies as our training functions. Calculating the target vector:
from math import cos, sin, exp
from scipy.integrate import quad

freq = np.arange(0.25, 15.25, 0.25)

y = []
for k in freq:
    y.append(quad(lambda x: cos(k*x)*exp(-0.5*((x-0.25)/0.25)**2), -1, 1)[0])
    y.append(quad(lambda x: sin(k*x)*exp(-0.5*((x-0.25)/0.25)**2), -1, 1)[0])
y = np.array(y)/dx

Now, the regressor matrix:
X = np.zeros((y.shape[0], x.shape[0]), dtype=float)
print('X',X.shape)
for i in range(len(freq)):
    X[2*i,:] = np.cos(freq[i]*x)
    X[2*i+1,:] = np.sin(freq[i]*x)

X[:,0] = X[:,0]/2
X[:,-1] = X[:,-1]/2

Linear regression:
from sklearn.linear_model import LinearRegression
reg = LinearRegression().fit(X, y)
ghat = reg.coef_

import matplotlib.pyplot as plt

plt.scatter(x, g, s=1, marker="s", label='original g(x)')
plt.scatter(x, ghat, s=1, marker="s", label='learned $\hat{g}$(x)')
plt.legend()
plt.grid()
plt.show()

The Gaussian function is successfully learned although the data are spread somewhat around the true function. The spread is larger where $g(x)$ is close to zero. This spread can be smoothed with a Savitzky-Golay filter
from scipy.signal import savgol_filter
ghat_sg = savgol_filter(ghat, 31, 3) # window size, polynomial order

plt.scatter(x, g, s=1, marker="s", label='original g(x)')
plt.scatter(x, ghat, s=1, marker="s", label='learned $\hat{g}$(x)')
plt.plot(x, ghat_sg, color="red", label='Savitzky-Golay $\hat{g}$(x)')
plt.legend()
plt.grid()
plt.show()

In general, $F[f(x)]$ does not depend linearly on $f(x)$, that is
$$F[f(x)]=\int\limits_a^b\mathcal{L}\left(f(x)\right)dx$$
It is still can be written as a function of $f_0, f_1\dots,f_N$ after discretizing $x$ which is also true for the functionals of the form
$$F[f(x)]=\int\limits_a^b\mathcal{L}\left(f(x),f'(x)\right)dx$$
because $f'$ can be approximated by a finite differences of $f_0, f_1\dots,f_N$. As $\mathcal{L}$ is a non-linear function of $f_0, f_1\dots,f_N$, one may attempt to learn it with a non-linear method, e.g. neural networks or SVM, although it will probably not be so easy as in the linear case.


If you delete widget from list it reset state of all the widgets below it in flutter?

Question: Using floating action button you can add questions, but when you delete particular question it reset state of all the widget below it in the list, everything above it maintains its state. TextField didn't retain data though I'm using TextFieldController. I'm unable to figure out why this is happening.
import 'package:flutter/cupertino.dart';
import 'package:flutter/material.dart';
import 'package:kaloory/constants.dart';

class AddQuestionsForTest extends StatefulWidget {
  AddQuestionsForTest({Key key}): super(key: key);

  @override
  _AddQuestionsForTestState createState() => _AddQuestionsForTestState();
}

class _AddQuestionsForTestState extends State<AddQuestionsForTest> {
  int count = 1;
  List<InputWidget> children;

  void removeInputWidget(index){
    setState(() {
      children.remove(index);
    });
  }
  @override
  void initState(){
    super.initState();
    setState(() {
      children = List.generate(count, (int i) => new InputWidget(removeInputWidget, key: ValueKey(i),));
    });
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(

        appBar: titleAppbar(title: "Create Test"),

        body: Container(
          color: backgroundColor,
          height: MediaQuery.of(context).size.height,
          child: ListView.builder(
            itemCount: children.length,
            itemBuilder: (context, index) {
              final child = children[index];
              return child;
            },
          ),
        ),
        floatingActionButton: new FloatingActionButton(
          backgroundColor: primaryColor,
          splashColor: primaryColor.withOpacity(0.5),
          child: new Icon(Icons.add),
          onPressed: () {
            setState(() {
              children.add(InputWidget(removeInputWidget,key: ValueKey(count),));
              count = count + 1;

            });
          },
        ));
  }

}

class InputWidget extends StatefulWidget {
  final Function(InputWidget) removeInputWidget;

  InputWidget(this.removeInputWidget,{Key key}): super(key: key);

  @override
  _InputWidgetState createState() => _InputWidgetState();
}

class _InputWidgetState extends State<InputWidget> {
  @override
  Widget build(BuildContext context) {
    return Container(
        margin: EdgeInsets.symmetric(horizontal: 30, vertical: 5),
        padding: EdgeInsets.symmetric(horizontal: 20, vertical: 10),
        decoration: BoxDecoration(
          borderRadius: BorderRadius.circular(10),
          color: Colors.white,
        ),
        child: Column(
          crossAxisAlignment: CrossAxisAlignment.end,
          children: <Widget>[
            BuildDropDown(),
            Divider(),
            IconButton(icon: Icon(Icons.delete_outline), onPressed: (){
                widget.removeInputWidget(this.widget);
            })
          ],
        ));
  }
}
class BuildDropDown extends StatefulWidget {

  @override
  _BuildDropDownState createState() => _BuildDropDownState();
}

class _BuildDropDownState extends State<BuildDropDown> {
  TextEditingController ques = TextEditingController();
  String newValue;
  @override
  Widget build( BuildContext context) {
    return Column(
      crossAxisAlignment: CrossAxisAlignment.start,
      children: [
        new Padding(
          padding: const EdgeInsets.symmetric(vertical: 10),
          child: new Column(
            mainAxisAlignment: MainAxisAlignment.start,
            children: <Widget>[
              new ListTile(
                title: Container(
                  padding: EdgeInsets.all(8),
                  height: 40,
                  decoration: BoxDecoration(
                      borderRadius: BorderRadius.circular(10),
                      border: Border.all()),
                  child: DropdownButtonHideUnderline(
                    child: new DropdownButton<String>(
                        hint: Text('Choose'),
                        onChanged: (String changedValue) {
                          newValue = changedValue;
                          setState(() {
                            print(newValue);
                          });
                        },
                        value: newValue,
                        items: <String>[
                          'Multiple Choice',
                          'Short Answer Type',
                          'Long Answer Type'
                        ].map((String value) {
                          return new DropdownMenuItem<String>(
                            value: value,
                            child: new Text(value),
                          );
                        }).toList()),
                  ),
                ),
              ),
            ],
          ),
        ),
        Container(
          height: 40,
          margin: EdgeInsets.only(top: 10),
          padding: EdgeInsets.symmetric(horizontal: 20),
          decoration: BoxDecoration(
            borderRadius: BorderRadius.circular(10),
            border: Border.all(color: Colors.black.withOpacity(0.5)),
          ),
          child: TextField(
            controller: ques,
            decoration: InputDecoration(
              border: InputBorder.none,
              focusedBorder: InputBorder.none,
              enabledBorder: InputBorder.none,
              errorBorder: InputBorder.none,
              disabledBorder: InputBorder.none,
              hintText: "Untitled Question",
            ),
          ),
        ),

      ],
    );
  }
}

Answer: This is because TextField has it's own internal state, and it's not being saved even though you have assigned a key to the parent.
I think the solution is to save the text value somewhere in the widget state, and in build() check if the contents of the controller.text != _currentText, if they don't match, then inject the last known _currentText.
Comment: Not particularly `TextField` but the whole widget doesn't maintain its state. @shawnblais
Comment: That's odd. As I understand it, same type, same key, it should get the same state, as long as the widget was never totally removed from the tree. Is your key valid?
Comment: That's what I'm doing, removing the whole widget from the List of the widgets.


Android Studio adds both @androidx.annotation and @android.support.annotation problem

Question: I updated Android Studio to version 3.2.0.
When I accept any change from list of suggestion or alt + enter.

Problem is @androidx.annotation auto created.
Thus suggestions method have two annotation @androidx.annotation.Nullable and android.support.annotation.Nullable.
I don't want to manually remove this unwanted suggestion, so what can I do to have only android.support.annotation?
Also androidx.annotation package is not added in my app. and I don't want to add androidx.annotation.

See example.
import android.support.annotation.Nullable;
import android.support.v7.app.AppCompatActivity;

public class LoginActivity extends AppCompatActivity {
    @Override
    public void onCreate(@androidx.annotation.Nullable @Nullable Bundle savedInstanceState, @androidx.annotation.Nullable @Nullable PersistableBundle persistentState) {
        super.onCreate(savedInstanceState, persistentState);
    }
}

Is there some IDE setting to remove auto generation of androidx.annotation?
Update
I could get rid of this problem with migrating to androidx (from official doc), but I am not able to migrate all projects to androidx currently. So need a solution.
Comment: @JeelVankhede You can see document - `The flag is false by default if it is not specified.`, so this is unnecessary lines. Any way, I tried this also.
Comment: @Redman Enjoy the party :D
Comment: @Mohsen Thanks for help but I don't know how this is related to my question.
Comment: @Mohsen 1. I did not migrate to androidx, I just updated AS. 2. I tried this setting as well, there is no change. 3. My project does not have dependency for androidx annotations.
Comment: @Mohsen If you want see this issue, just update your Android Studio, and create new project, then type oncreate() and accept suggestion.
Comment: @Mohsen I just tried with fresh project again. Its happens with me , yes it is bug of IDE. I will report this issue. I wanted know if there is a quick workaround for this issue.
Comment: try adding `android.useAndroidX=false` & `android.enableJetifier=false` in **gradle.properties** file in your project.
Comment: Its not about each file bro, I am getting this just when I am coding now. when I am accepting changes. Like `onCreate()` and any suggestion. You can try this too in your updated Android Studio.
Comment: @Khemraj sorry I misunderstood , but I am not getting `@androidx.annotation.Nullable` in my android studio its 3.2.0 too
Comment: I suggested that because of : `Is there some IDE setting to remove auto generation of androidx.annotation?` But the point is, if `@androidx.annotation` automatically creates, then this is because you're actually already migrated to `AndroidX` and that's why this generates automatically. And in your codes, you actually have **two different annotations** from **support annotations** and **androidx**. Please clarify me If i'm not getting your point.
Comment: Just did what you said. It did not created another `@androidx.annotation` And there was just `android.support.annotation`. You probably should report the issue. That's a bug. However, I'm alreading using v3.2 and it's a little bit buggy so far.
Comment: @Mohsen I'd like to hear your explanation how you think braces and wrapping are related to annotation imports
Comment: @NickCardoso Didn't get the OP question at first. Anyways Remove that part. Thanks
Answer: AndroidX is the new extension libraries for backward compatibility support. In future new feature backward compatibility support will be addressed in AnddroidX. As stated in this blog https://android-developers.googleblog.com/2018/05/hello-world-androidx.html

The stable release of 28.0.0 will be the final feature release packaged as >android.support. All subsequent feature releases will only be made available as >androidx-packaged artifacts.

https://developer.android.com/topic/libraries/support-library/revisions

Revision 28.0.0 Production
(September 21, 2018)
This is the stable release of Support Library 28.0.0 and is suitable for use >in production. This will be the last feature release under the >android.support packaging, and developers are encouraged to migrate to >AndroidX.

Moving your app from android.support to androidx-packaged dependencies
Refer to this link, https://developer.android.com/jetpack/androidx/migrate
If you depend on a library that references the older Support Library, Android Studio will update that library to reference androidx instead via dependency translation. Dependency translation is automatically applied by the Android Gradle Plugin 3.2.0-alpha14, which rewrites bytecode and resources of JAR and AAR dependencies (and transitive dependencies) to reference the new androidx-packaged classes and artifacts. We will also provide a standalone translation tool as a JAR.
So, In Step 1(dependency translation): In your gradle.properties file set the android.useAndroidX flag to true and the android.enableJetifier flag to true.
android.useAndroidX=true
android.enableJetifier=true

in Step 2(source refactoring): With Android Studio 3.2 and higher, you can quickly migrate an existing project to use AndroidX by selecting Refactor > Migrate to AndroidX from the menu bar.
Comment: Why do I get an error: **Cannot find a version of 'androidx.annotation:annotation' that satisfies the version constraints**
Comment: [Migrating to AndroidX](https://developer.android.com/jetpack/androidx/migrate) in this page under **Artifact mappings** search for **Old build artifact** for example in your case, **'com.android.support:support-annotations'**. then replace with respecting **AndroidX build artifact** here it is, androidx.annotation:annotation:1.0.0 in your app gradle dependencies **implementation 'androidx.annotation:annotation:1.0.0'**
Comment: How do it without Android studio?
Answer: Just add annotationProcessor 'androidx.annotation:annotation:1.1.0' as a dependency in your build.gradle(Module: app) file.
Comment: @red-devil In your dependencies. Just like as an 'implementation' use 'annotationProcessor'
Comment: add where in app/build.gradle?
Answer: Android Studio has a preference option "Exclude from import and completion" that you could use to suppress the suggestion of androidx packages:

In this screenshot, for example, I've added the java.time package to this list because I want my autocomplete to suggest org.threeten.bp.LocalDate but never java.time.LocalDate.
You should be able to add androidx.annotation to this list to solve your problem.
Comment: It is still same bro. I excluded `androidx.annotation` and when I type some method, and override it from suggestion, it adds both `androidx.annotation` and `android.support.annotation`.
Answer: Android Studio can only import files which are included in your classpath. This includes Androidx libraries - thankfully, if the classes are not present, Studio won't suggest them (which is why the rest of us don't see this issue in our projects).
That being said, the implication here is that you have updated your gradle file to include the package containing androidx.annotation.NonNull
com.android.support:support-annotations became androidx.annotation:annotation:1.0.0 according to the migration guide, so you can look for this group in your module Gradle file.
The simplest fix is to remove that package from Gradle, then Clean and Rebuild.
Note: If you want to stick with Support annotations then the latest and greatest version is 28 as mentioned here
Answer: Using Android Studio Arctic Fox (equivalent to version 4.3). I did not migrate to AndroidX for my project. What worked for me is adding
implementation 'androidx.annotation:annotation:1.1.0'

to the app-level build.gradle.
Answer: Using Android Studio 3.3.1, I was able to replicate the problem. My project had just been created, with no AndroidX dependency. When I created the first fragment and overrode onCreateView(), the 2 annotations @androidx.annotation.Nullable and @Nullable were automatically added by Android Studio to the method (no prompt given with a possible choice of import). The only annotation import automatically added was import android.support.annotation.Nullable;
If you do not wish to migrate to AndroidX, here is a fix that worked for me:

Manually delete all the @androidx.annotation.Nullable annotation(s), leaving only the original @Nullable annotation(s).
Do File > Invalidate Caches / Restart

When a new Fragment will be created in the app, only the original Support Library's @Nullable annotation will be added.
Answer: As stated on the support library release note.

This is the stable release of Support Library 28.0.0 and is suitable
  for use in production. This will be the last feature release under the
  android.support packaging, and developers are encouraged to migrate to
  AndroidX.

You could still use support's class path. (remember to clean and build)
But highly recommend you to migrate to AndroidX as 28.0.0 of support is the last release.
(remember, NOT BOTH)
Comment: Where to set the classpath?
Answer: I will start by saying I am a very novice slapstick hobbyist, but this same issue got me stuck and its been resolved for some time and I could not find a solution.
Everyone talks about adding dependencies to your build.griddle files, and if that works great, for me it never took until I changed the actual import listings (API 28) and changing them from...
import android.support.annotation.Nullable;
-to-
import androidx.annotation.Nullable;
I had to do a lot of improvisation to find the new directories because they didn't seem to be listed anywhere I could easily find.
This was also the case for Fragment, FragmentManager, DialogueFragment, and many others. I hope this helps someone suffering like I was!
Answer: Since this problem has emerged for me after updating to Android studio 4.2.1 (or maybe 4.2?), I am adding the way to fix it here, for anyone who stumbles upon this question, as I did.
I started getting double nullability annotations, as described in this question, recently.
To fix it, do the following:

Go to Android studio preferences
Go to Editor > Inspections
Go to Java > Probable bugs > Nullability problems > @NotNull/@Nullable problems
Press the "Configure Annotations" button
Select "androidx.annotation.Nullable" in the top box, and press the checkmark button below the box, titled "Select annotation used for code generation"
Do the same for "androidx.annotation.NonNull" in the bottom box.
Press OK

Android studio should now only use the androidx annotations when generating code.


If $X_1,\ldots,X_n$ are continuous and IID with sample median $m$, are $1(X_1\geq m),\ldots,1(X_n\geq m)$ IID?

Question: Suppose $X_1,\ldots,X_n$ are continuous random variables and IID with sample median $m$ for some $n\geq 2$. Is it then true that $1(X_1\geq m),\ldots,1(X_n\geq m)$ are IID?
Here the definition of $1(X_i\geq m)$ is that it is 1 if $X_i\geq m$ and 0 otherwise.
I ask because I often see researchers create these dummy variables (i.e. indicator variables) in regression analyses and I am worried that there is dependence between them by design (i.e. by construction). Mathematically I feel that since the dummy variables are functions of independent random variables, they ought to be independent. On the other hand, by the law of large numbers, for large $n$ we intuitively have that $(1/n)\sum_i1(X_i\geq m)$ is approximately equal to the expectation of $1(X_i\geq m)$ with $m$ being close to the population median, i.e. 0.5, with probability 1, suggesting that there is a linear dependence between the dummy variables for large $n$. (EDIT: Since $m$ is the sample median, we also know that $(1/n)\sum_i1(X_i\geq m)$ is at least 0.5.)   I am unsure what I should conclude from this simple line of reasoning and unable to proceed in a more mathematically rigorous manner.
EDIT: I guess the answer to my question is simple. The indicator variables are not independent because all of them cannot be equal to 1 by the definition of the sample median.
Comment: The ambiguity of the question is whether or not the OP means the population median (as opposed to the sample median).
Comment: @Xi'an Thanks for emphasizing this. Here I meant $m$ to be equal to the sample median of $(X_1,\ldots,X_n)$. My question has been updated.
Comment: @Xi'an Excellent point.  The question is a little more interesting when the sample median is intended, for then these indicators are *exchangeable* but not independent.
Comment: @Dave I do not think this answers my question since the functions are not functions of independent random variables because $m$ is the sample median.
Comment: [This](https://math.stackexchange.com/q/8742/672549) seems to take care of the "independent" part.
Comment: I don't see how the law of large numbers "... imply[s] that there is a linear dependence between the dummy variables...".  This seems to me to be related to the classical fallacy that, e.g., a run of heads in a coin-flip experiment must make tails more likely so that the fraction of heads can get closer to 0.5.   Am I misunderstanding something?
Comment: Apply the duplicate to the function $x\to1(x\ge m).$
Answer: A fairly rigorous argument is that
$$\mathbb P(X_1\ge m(X_1,\ldots,X_n),\ldots,X_n\ge m(X_1,\ldots,X_n))=0\ne \mathbb P(X_1\ge m(X_1,\ldots,X_n))^n$$
Comment: What is interesting about the question is that although it specifies a continuous distribution, I believe the result is true for *all* distributions (except for the obvious exceptions $n=1$ and when the $X_i$ are constant a.s.)  But for discrete distributions your final inequality does not necessarily hold.
Comment: Right, all $X_i$'s could be identical.
Answer: Another argument is that:

If $n$ is even, then $\sum_{i=1}^n1(x_i \geq m) = n/2$, therefore the $1(x_i \geq m)$ cannot be independent,

If $n$ is odd, then $\sum_{i=1}^n1(x_i \geq m) = (n+1)/2$, therefore the $1(x_i \geq m)$ cannot be independent.
Comment: The two "therefores" still need justification.  Admittedly it is easy for continuous variables--but it still needs to be there.


draytek vigor 2862 how to setup a remote access VPN

Question: We have the simplest network possible, and are trying to setup the simplest VPN possible to allow remote workers access to the LAN and to administer the draytek firewall.
We have a ZTE router supplied by the ISP which has a fixed external ip of x.x.x.x
This routers LAN address is 192.168.10.1, and provides DHCL to LAN hosts from 192.168.10.10 onwards.
The drayteks fixed WAN address is 192.168.10.2 (and its gateway is obviously the ZTE on 192.168.10.1)
The drayteks LAN DHCP range is 192.168.1.x
Everything works in terms of internet access. I can connect to either the ZTE LAN (with an address of 192.168.10.x "infront" of the FW) or the draytek LAN (192.168.1.x i.e. "behind" the FW)
This is the article we have followed to the letter:
https://www.draytek.com/support/knowledge-base/5322
But it cant establish a connection.
We dont know how to debug this, or what configuration is missing.
Apart from setting a fixed IP, we have made no changes to the default state of the vigor.
I am guessing I dont have to do anything with the firewall, as the article does not mention this.
In my local mac VPN settings, I am putting 192.168.10.2 as the "server address", and connecting my latop to the LAN port of the ZTE (So it can see the WAN port of the draytek directly with NAT/port forwarding as a test).  In the "authentication" settings, I put the user password and "shared secret" I setup in the Vigor.
When i try to connect, I get "The L2TP-VPN server did not respond. Try reconnecting. If the problem continues, verify your settings and contact your Administrator"
Has anyone come across a setup guide for VPN including what has to be setup on the client, using whatever protocol is the easiest?
Answer: I assume you have fixed this but
The Server Address should be the External Fixed Ip address not the Local Address.


Trouble with checking/unchecking checkboxes via JQuery

Question: Here is a JSFiddle of my problem.
Note I'm using JQuery 2.0.2 for this.
I'm using a JQuery checkbox tree plug-in and when I manually click every element inside Subgroup B2, Subgroup B2 will also become checked. Conversely, if I manually uncheck one of it's children, Subgroup B2 will go to a "partial selection" state and unchecking all of the children will completely uncheck Subgroup B2. This works great and as intended.
However, notice the buttons at the top.
I'm trying to trigger these click events through code.
The first button, fires the click event on the label (which is what gets clicked when you manually click the boxes/text). And as you'll notice, it checks all the boxes, however Subgroup B2 remains only partially selected.
Clicking the button again should toggle the checkboxes off again, but it doesn't. That's what button two is for. For some reason to uncheck the buttons, you need to click a different element than was required to check the checkbox. This makes no sense to me.
Finally, if you mix interactions (actual clicks and the coded clicks) things start falling apart even more.
I've been working on this for hours now, and ever time there's a problem with the interactions, it's come from the checkboxes.
How can I get the checkboxes to respect my coded clicks as much as my manual clicks?
Comment: It seems that the relationship between the checkbox and the label clicking is not working as you may expect. Are you quite sure you know where your clicks are firing?
Answer: I believe the issue with the buttons comes from the triggered event bubbling up the DOM, you can use triggerHandler() instead of trigger() to avoid that, just note that this method only affects the first matched element so you need to use it in a loop:
$("#trigChildren").on('click', function () {
    $('#one, #two, #three').each(function () {
        $(this).parent().triggerHandler('click');
    });
});

Updated fiddle
Comment: I had my suspicions about this -- I didn't realize that `trigger()` was bubbling up. I also found an alternate solution which involved commenting out two lines in the `labelClick` method -- seems to work the same. Thanks for this answer though, works great!


Script removal of Insider Hub app-Win10

Question: I'd like to script the removal of the Windows 10 Insider Hub app. I've had no success with Powershell-admin: get-appxpackage -allusers -name *insider* | remove-appxpackage
I can do it manually in the GUI, but its not automated. There is a 3rd-party EXE that'll remove it, but I'd prefer not to use that (on top of the fact I tested it in a VM & the process froze)
Be aware: I'm using the free Win10 Pro upgrade, so I can't go Enterprise LTSB (which I understand can solve alot of this) & consequently can't DISM the WIM file.
Bonus points: how-to remove 'Contact Support' & 'Windows Feedback' script`omatically
Comment: Removal of all the windows store app a issue?
Comment: No? I used the script below to remove most apps & it works wonders, but it didn't get rid of the "Insider Hub" app, I can uninstall manually, but would like to do it via a powershell script: https://blogs.technet.microsoft.com/mniehaus/2015/11/11/removing-windows-10-in-box-apps-during-a-task-sequence/
Comment: I asked as I was writting an answer, but it was hitting the store, thus I asked before :)
Answer: Tested the below on a PC successfully:
dism.exe /online /Remove-Package /PackageName:Microsoft-Windows-InsiderHub-Package~31bf3856ad364e35~amd64~~10.0.10586.0

I got the packagename from the DISM cmd switch get-packages:
dism /online /get-packages > getpackages.txt

Script removal of Insider Hub app-Win10


Event for when UI is shown to user?

Question: I would like my main activity to load a loading message. Once this is displayed i would like to start my tab activity which has to load a fairly heavy RSS feed. Problem is i have tried overriding onStart and onWindowFocusChanged if hasFocus but neither draw the view before starting the other activity.
Anyone know how to do this? All the splash screen examples use a timer which is not what i want to do, i want it to stay while the data is loading and clear itself when the onPause is fired.
Answer: You cannot start another activity while you're still showing another one. You have to change you control flow.
Load you RSS feed with an AsyncTask while you're still in your main activity showing the loading message. When you finished loading all the data you need, store it somewhere, then start you tab activity. So you tab activity don't need to load the data itself but can fetch it from the place you stored it.

Event for when UI is shown to user?


How can I express this function in terms of s and t?

Question: I suppose that this equality can be satisfied:
$$
\frac{t}{x^s}=\frac{1}{x^{Y(s,t)}}
$$
Where $ Y(s,t) $ is a function of s and t.
Is there a function sloution for $Y(s,t)$?
Answer: hint
$${x^{Y(s,t)}}=e^{lnx Y(s,t)}$$


How would I get a cron job to run every 30 minutes?

Question: I'm looking to add a crontab entry to execute a script every 30 minutes, on the hour and 30 minutes past the hour or something close. I have the following, but it doesn't seem to run on 0.
*/30 * * * *

What string do I need to use?
The cron is running on OSX.
Comment: ... Couldn't you just change 0,30 to 0,5 ? If 0,5 works, why wouldn't 0,30?
Comment: I'm waiting to see if one of the answers below works, just have to wait till the next 30 minutes is up...in 3 minutes.
Comment: You never told us if it worked on the hour!
Comment: Have you succeeded in getting your crontab to work?  Your problem may not be the crontab, but getting it to be used.
Comment: It's running 2 other scripts, one every minute.
Comment: @Gortok, I guess you're right, but it's always good to test in the real situation as well :)
Answer: Do:
0,30 * * * * your_command

Comment: Works great! In CentOS crontab, I had to do something like `0,30 * * * * root your_command` where root would be the user I'm using.
Comment: @prograhammer - Was this line part of a root's crontab or some other user's crontab?
Comment: @MuhamedHuseinbašić good question, since there are 2 different crontabs (the global one and then a user's). I'll have to check and see...
Answer: crontab does not understand "intervals", it only understands "schedule"
valid hours: 0-23 -- valid minutes: 0-59
example #1
30 * * * * your_command
this means "run when the minute of each hour is 30" (would run at: 1:30, 2:30, 3:30, etc)
example #2
*/30 * * * * your_command
this means "run when the minute of each hour is evenly divisible by 30" (would run at: 1:30, 2:00, 2:30, 3:00, etc)
example #3
0,30 * * * * your_command
this means "run when the minute of each hour is 0 or 30" (would run at: 1:30, 2:00, 2:30, 3:00, etc)
it's another way to accomplish the same results as example #2
example #4
19 * * * * your_command
this means "run when the minute of each hour is 19" (would run at: 1:19, 2:19, 3:19, etc)
example #5
*/19 * * * * your_command
this means "run when the minute of each hour is zero or evenly divisible by 19" (would run at: 1:19, 1:38, 1:57, 2:00, 2:19, 2:38, 2:57, 3:00 etc)

note: several refinements have been made to this post by various users including the author
Answer: Try this:
0,30 * * * * your command goes here

According to the official Mac OS X crontab(5) manpage, the / syntax is supported.  Thus, to figure out why it wasn't working for you, you'll need to look at the logs for cron.  In those logs, you should find a clear failure message.
Note:  Mac OS X appears to use Vixie Cron, the same as Linux and the BSDs.
Comment: @Adam Hawes: I've also used */xx with success in the past, but on Linux and other UNIXes.  I've not personally tried Mac OSX.  The original question didn't say which OS.  Some older UNIXes don't support */xx
Comment: I've always used */30 with success but I don't recall an application of mine where it was too critical if it only ran once an hour so I don't think I've ever checked.
Answer: If your cron job is running on Mac OS X only, you may want to use launchd instead.
From Scheduling Timed Jobs (official Apple docs):

Note: Although it is still supported, cron is not a recommended solution. It has been deprecated in favor of launchd.

You can find additional information (such as the launchd Wikipedia page) with a simple web search.
Answer: You can use both of ',' OR divide '/' symbols. But, '/' is better. Suppose the case of 'every 5 minutes'. If you use ',', you have to write the cron job as following:
0,5,10,15,20,25,30,35,....    *      *     *   * your_command

It means run your_command in every hour in all of defined minutes: 0,5,10,...
However, if you use '/', you can write the following simple and short job:
*/5  *  *  *  *  your_command

It means run your_command in the  minutes that are dividable by 5 or in the simpler words, '0,5,10,...'
So, dividable symbol '/' is the best choice always;
Answer: You mention you are using OS X- I have used cronnix in the past. It's not as geeky as editing it yourself, but it helped me learn what the columns are in a jiffy. Just a thought.
Comment: This could be helpful otherwise as well.


Spectrum of the infinite biregular tree (with an example)

Question: This question concerns the spectrum of the universal covers of graphs. In particular, some results in:

Angel, Omer; Friedman, Joel; Hoory, Shlomo The non-backtracking spectrum of the universal cover of a graph. Trans. Amer. Math. Soc. 367 (2015), no. 6, and
Nagnibeda, Tatiana Random walks, spectral radii, and Ramanujan graphs. Random walks and geometry, 487–500, Walter de Gruyter, Berlin, 2004 .

It's well known that in the case of infinite $k$-regular trees the spectrum is:
$$
[-2 \sqrt{k-1}, 2 \sqrt{k-1}] .
$$
Moreover, in the case of an infinite $(k,l)$-biregular tree the spectrum is:
$$
\{ \lambda \in \mathbb{R}\ \big| \  |\sqrt{k - 1} - \sqrt{ l - 1} |\leq |\lambda| \leq |\sqrt{k - 1} + \sqrt{ l - 1} |  \} \cup \{ 0\}\ ,
$$
as one can see, for example, in

Mohar, Bojan; Woess, Wolfgang A survey on spectra of infinite graphs. Bull. London Math. Soc. 21 (1989), and
Li, Wen-Ch'ing Winnie; Solé, Patrick Spectra of regular graphs and hypergraphs and orthogonal polynomials. European J. Combin. 17 (1996) .

However, very little is known about the spectrum of the universal cover of irregular graphs, in general. (1) and (2) are the only sources I found that give an expression or algorithm to compute values in that spectrum. Are there other sources that you may be aware of?
EDIT: this second question doesn't really make sense, because $0$ is in the spectrum of the infinite $(3,2)$-regular tree.
A second question is that, while trying to apply the algorithm described in (1) to the $(3,2)$-complete bipartite graph, I wasn't able to see why $0$ is not in the spectrum (of the adjacency operator) of the infinite $(3,2)$-biregular tree. So,

What should be the ratio system $\{ r_e\}$, so that the corresponding Perron eigenvalue $\alpha(r) < 1$?
Given that $r_e = \frac{G(u,v)}{G(u,u)}$, with $e = (u,v)$, and the Green function is invariant under automorphisms of the base graph, can we assume that $r_e$ is also invariant under the same automorphisms?
Is the Green function associated to the adjacency operator unique? And does that say that the, if it exists, the ratio systems are unique?

I appreaciate any insight from people, because computing this simple example has got me very confused. Thank you in advance!
Comment: My second question was incorrect, because $0$ is, in fact, in the spectrum of infinite $(3,2)$-biregular tree, so there should exist any ratio system $\{ r_e \}$ such that $\alpha(r) < 1$. Moreover, it is true that when such ratio system exists, there is also one ratio system (perhaps the same) which is invariant under automorphisms.
I would still like to see some literature suggestion, if anyone has relevant suggestions. Thanks!
Answer: I recommend the paper "Combinatorics of Free Product Graphs" by Quenell, which gives a very friendly and readable account of the determination of the spectrum of free product graphs.  Since many non-regular infinite trees are (basically) free product graphs, you might get some use out of it.
Quenell, Gregory, Combinatorics of free product graphs, Brooks, Robert (ed.) et al., Geometry of the spectrum. 1993 joint summer research conference on spectral geometry, July 17-23, 1993, Univ. of Washington, Seattle, USA. Providence, RI: American Mathematical Society. Contemp. Math. 173, 257-281 (1994). ZBL0808.05066.


Adjust alarm time to match clock time in python

Question: I'm just learning Python, and as one of the basic challenges that were suggested to me by friends I've been working on an alarm clock. I successfully made an alarm clock that played a .wav sound at a predetermined time. Now I've been using Pygame for a GUI, and it all worked great, until I had to set the buttons to adjust the alarm time. See when I compare the alarm time to the clock time, the clock time is in string form, so the alarm time has to be as well. But the buttons are unable to + or - from a string, so I'm kind of stuck. I tried ways of turning it into a string, but everything has been fairly unsuccessful so far. Wondering if anyone here had a suggestion.
Here's the code:
#!/usr/bin/python
import os.path, sys, datetime, time
import os, sys, math
import pygame, random
from pygame.locals import *

main_dir = os.path.split(os.path.abspath(__file__))[0]
data_dir = os.path.join(main_dir, 'data')
currenttime = datetime.datetime.now()
clocktime = currenttime.strftime("%H:%M")
alarmtime = "13:23"
pygame.init()

#Screen and background
width, height = 600, 600
screen = pygame.display.set_mode((width, height))
pygame.display.set_caption("Alarm Clock")
background = pygame.image.load(os.path.join(data_dir, 'diamondplate.jpg'))
background = pygame.transform.scale(background, (width, height))

#Current time
font = pygame.font.Font(None, 250)
text = font.render("%s" % clocktime, True, (255,140,0), (0,0,0))
textRect = text.get_rect()
textRect.centerx = screen.get_rect().centerx
textRect.centery = screen.get_rect().centery - 200

#Alarm time
text2 = font.render("%s" % '00:00', True, (255,140,0), (0,0,0))
text2Rect = text2.get_rect()
text2Rect.centerx = screen.get_rect().centerx
text2Rect.centery = screen.get_rect().centery + 200

#Alarm noise
def alarmsound(file_path=os.path.join(main_dir, 'data', 'boom.wav')):
    pygame.mixer.init(11025)
    sound = pygame.mixer.Sound(file_path)
    channel = sound.play()
    pygame.time.wait(1000)

#Image load function
def load_image(file):
    file = os.path.join(data_dir, file)
    surface = pygame.image.load(file)
    return surface.convert_alpha()

#Hour arrow up
class Hourup(pygame.sprite.Sprite):

    def __init__(self):
        pygame.sprite.Sprite.__init__(self,self.groups)
        image = load_image('arrowup.png')
        image = pygame.transform.scale(image, (85,85))
        self.image = image
        self.rect = self.image.get_rect()
        surface = pygame.display.get_surface()
        self.area = surface.get_rect()
        self.rect.bottomleft = text2Rect.topleft

    def click_check(self,eventpos):
        if self.rect.collidepoint(eventpos):
            pass

    def update(self):
        pass

#Hour arrow down
class Hourdown(pygame.sprite.Sprite):

    def __init__(self):
        pygame.sprite.Sprite.__init__(self,self.groups)
        image = load_image('arrowdown.png')
        image = pygame.transform.scale(image, (85,85))
        self.image = image
        self.rect = self.image.get_rect()
        surface = pygame.display.get_surface()
        self.area = surface.get_rect()
        self.rect.bottom = text2Rect.top
        self.rect.left = 159

    def click_check(self,eventpos):
        if self.rect.collidepoint(eventpos):
            pass

    def update(self):
        pass

#Minute arrow up
class Minuteup(pygame.sprite.Sprite):

    def __init__(self):
        pygame.sprite.Sprite.__init__(self,self.groups)
        image = load_image('arrowup.png')
        image = pygame.transform.scale(image, (85,85))
        self.image = image
        self.rect = self.image.get_rect()
        surface = pygame.display.get_surface()
        self.area = surface.get_rect()
        self.rect.bottomright = (442,414)

    def click_check(self,eventpos):
        if self.rect.collidepoint(eventpos):
            pass

    def update(self):
        pass

#Minute arrow down
class Minutedown(pygame.sprite.Sprite):

    def __init__(self):
        pygame.sprite.Sprite.__init__(self,self.groups)
        image = load_image('arrowdown.png')
        image = pygame.transform.scale(image, (85,85))
        self.image = image
        self.rect = self.image.get_rect()
        surface = pygame.display.get_surface()
        self.area = surface.get_rect()
        self.rect.bottomright = text2Rect.topright

    def click_check(self,eventpos):
        if self.rect.collidepoint(eventpos):
            pass

    def update(self):
        pass

#Groups
allsprites = pygame.sprite.Group()
Hourup.groups = allsprites
Hourdown.groups = allsprites
Minutedown.groups = allsprites
Minuteup.groups = allsprites
hourup = Hourup()
hourdown = Hourdown()
minutedown = Minutedown()
minuteup = Minuteup()
clickableobjects = [hourup, hourdown, minutedown, minuteup]

def main():
    while 1:
        currenttime = datetime.datetime.now()
        clocktime = currenttime.strftime("%H:%M")
        screen.blit(background,(0,0))
        text = font.render("%s" % clocktime, True, (255,140,0), (0,0,0))
        text2 = font.render("%s" % alarmtime, True, (255,140,0), (0,0,0))
        screen.blit(text,textRect)
        screen.blit(text2,text2Rect)
        for event in pygame.event.get():
            if event.type == pygame.QUIT or (event.type == KEYDOWN and event.key == K_ESCAPE):
                sys.exit()
            if event.type == MOUSEBUTTONDOWN:
                if event.button == 1:
                    for object in clickableobjects:
                        object.click_check(event.pos)

        if clocktime == alarmtime and soundcheck = False:
            alarmsound()
            soundcheck = True
        allsprites.draw(screen)
        allsprites.update()
        pygame.display.update()
        pygame.display.flip

if __name__ == '__main__':
    main()

Answer: You are looking for strptime() which will convert a string to a datetime instance.
see here for how to properly use it.
Comparing two datetime instances will give you a timedelta instance which you can read about here. Essentially it will give you the difference between the two times to the nearest milisecond.
Learn everything you can about the datetime, time, and calendar modules.  Once you learn those dealing with times and dates in python becomes really easy.


WHY does the difference between the fractions get smaller?

Question: $\frac{51}{101} - \frac{50}{100} > \frac{52}{102} - \frac{51}{101}$
I get that by adding 1 to both the numerator and the denominator of the same fraction that has a numerator smaller than the denominator, the fraction gets bigger since proportionally the numerator grows quicker than the denominator.
But why does the difference between the fractions get smaller? I underline the term "why", since the focus here is talking about the logical principle, not about examples (even though those can be useful as a supplement to explain the principle).
Comment: It helps to think about extreme examples. If you start from 2/3 and keep adding $1$ to both sides the result will become close to .5.  e.g., 1,000,002 / 1,000,003 is very close to 1/2, as is 1,000,0003/1,000,004, so their difference is very small.
Comment: @JairTaylor don't you mean the result is close to 1?
Comment: @qwr Yes, duh, thank you.
Answer: $\frac{50}{100},\frac{51}{101},\frac{52}{102}$
The numerator grows bigger in proportion to itself compared to the denominator. Meaning the numerator was multiplied by a bigger number than the denominator. Hence the fractions grow in size as we add more terms to the sequence.
However, in proportion of multiplication the factor reduces, as for example from $1.0099$ to $1.0096$ since adding one to bigger numbers becomes less significant.
Bonus: The sequence is approaching one. I can't put up the plot here but for this specific sequence plot $f(x)=\dfrac{x}{x+50}$ and see what $\displaystyle \lim_{x \to \infty} f(x)$ is.
In addition, if you check the slope of the graph either by calculus or looking, it becomes less steep for bigger values of $x$ which also kinda answers your question.
Comment: Yes man. That's the whole point.
Comment: So, what you're saying is that as the numerator gets bigger, that 1 becomes less in proportion to the numerator, so the fractions grows more slowly?
Answer: Well, it can be proved from the point that you made.
$ \frac{m+1}{n+1} + \frac {m-1}{n-1}$
$= \frac {2(mn-1)}{n^2-1}< \frac{2mn}{n^2}$
$ \Rightarrow \frac {m+1}{n+1} -\frac{m}{n} < \frac {m}{n} - \frac {m-1}{n-1}$
Answer: Hint: Observe a sequence $$a_n =  {50+n\over 100+n}- {49+n\over 99+n}$$
$$ a_n = {545\over (100+n)(99+n)}$$
It is now obviously that this sequence is decreasing.


How to express "dreaming" with other than the whole tone scale?

Question: I am working on a little theater play accompanying a poetic clown on piano on the stage. Most of time I improvise to emphasize his emotional moods, since there are no words at all.
When the character falls asleep I want to play something that expresses that which follows is (or may be) a dream, but being subtle and not so cliché-like as a several octaves whole tone scale ascension.
I like NReilingh's approach:

Chords don't give us feelings, we give chords feelings.

So I am not looking for a magical chord or scale to do this. I understand that it is my job to get into the character feelings and to find how to express them through my hands using dynamics, tempo... and even corporal expression. But it would be helpful to know other ways, suggestions, examples... to improve my improvisation background.
Comment: Suspended chords, and Lydian tonalities (though admittedly similar to whole tone)
Comment: The ballet "Somewhere" from WSS is a dream sequence and most certainly not whole-tone.
Comment: @CarlWitthoft Impressive! I was already looking for the film before your comment to get ideas for another emotional mood. It seems to be a good reference ;)
Comment: I think this question is too subjective to keep. However in my head I am hearing a repeating rising pattern of 6 notes of equal length.
Comment: Quartal harmony (chords built in fourths) can do it too, but this one tends to sound a bit jazzy or avant garde
Answer: One of the main reasons that a whole-tone scale works so well to indicate dreaming and rootlessness is that it's a symmetrical structure that divides the octave into equal parts. Multiple notes can therefore work equally well as a "tonic" which kind of means that they're also equally bad at being tonic. A symmetrical structure makes it much easier to avoid accidentally rooting the improvisation too much around a single note.
So you might want to explore other symmetric pitch collections. Augmented triads, diminished 7ths, tritones, French augmented sixths, hexatonic scales, octatonic scales, etc.
Comment: @Rober Good! Just a note however, the German augmented sixth isn't symmetrical and is actually just an enharmonic respelling of a dominant seventh chord. This makes it extremely useful for modulating to distant keys, but it actually can have a strongly directional sound. Still, if it works for your purposes, then it's all for the good!
Comment: Different symmetric scales and french (and german) augmented sixths satisfy my needs : )
Answer: The style of the impressionist can be very dreamy. Listen to Le Plus Que Lent by Debussy, or perhaps Ondine. A number of his preludes create this dreamy ambiance too. As far as Ravel goes, La Valse does a good job of presenting a foggy, distorted version of the main theme. If I were to summarize these techniques briefly, parallelism and extended tertian harmony (7 9 11 13 etc chords).
As mentioned in the comments, WSS's ballet is a good one, so is Oklahoma's.
Answer: High pitched rolled chords played softly (usually with some pedal) are sometimes used for dream-like stuff. I've used things like grouping 5-note arpeggios in 16th notes (so the pattern migrates through time) with a simple bass melody to evoke dreams. Or perhaps I'm really evoking running water.


Comparing the active cell value to the value of the cell before it (VBA)

Question: First off, thank you for clicking into my question
I want the code to go through a list of values in a sheet called "Tests" and copy those values into a sheet called "List". The values in "Tests" are sometimes repeated though and I only want "List" to have unrepeated values.
Example:
"Tests" column A: dog, dog, cat, rabbit, rabbit, rabbit, turtle, turtle
"List" column A: dog, cat, rabbit, turtle
Can someone please point me in the right direction/ tell me where I went wrong? This is my code:
Option Explicit

Sub Main()

'create new sheet
' ActiveWorkbook.Sheets.Add

'adds a sheet and names it "List"
Sheets.Add.Name = "List"

'Select sheet1 (Tests)
Sheets("Tests").Activate

'Activate cell A2 in Tests -- not really necessary but ill leave it for now
Range("A2").Activate

'Create for loop for cells a2 through a1019
Dim cell As Integer
Dim listcell As Integer
listcell = 1

    For cell = 2 To 1019

    'create if statement for if Tests cell does match previous cell
    If (StrComp(ActiveCell.Value, ActiveCell.Offset(-1, 0).Value, vbTextCompare) = 0) Then

            ActiveWorkbook.Sheets("Tests").Range("A" & (cell + 1)).Select
            Sheets("List").Range("A" & listcell).Value = Sheets("Tests").Range("A" & cell).Value
            listcell = listcell + 1

    'else if doesn't match then write down and move to next cell in sheet
    ElseIf (StrComp(ActiveCell.Value, ActiveCell.Offset(-1, 0).Value, vbTextCompare) = 1) Then

            Application.Goto ActiveWorkbook.Sheets("List").Range("A" & cell)
            Sheets("List").Range("A" & listcell).Value = Sheets("Tests").Range("A" & cell).Value
            listcell = listcell + 1

    ElseIf (StrComp(ActiveCell.Value, ActiveCell.Offset(-1, 0).Value, vbTextCompare) = -1) Then

            Application.Goto ActiveWorkbook.Sheets("List").Range("A" & cell)
            Sheets("List").Range("A" & listcell).Value = Sheets("Tests").Range("A" & cell).Value
            listcell = listcell + 1

End If
Next
End Sub

Comment: No VBA necessary - you could use the `UNIQUE` worksheet function
Comment: ... or use Power Query to return a unique list.  ... or use conditional formatting to flag duplicates and use the format to determine which rows to delete.   ...or output the data to a table formatted pivot table.   There are a lot of easier methods than walking your entire dataset and doing row by row comparisons.
Answer: If you need to use VBA as part of a bigger process that uses VBA or just don't like using functions that spill, then you can still use unique
Dim lArray As Variant
lArray = Application.WorksheetFunction.Unique(Thisworkbook.worksheets("Tests").range("A1:A1290"))

Then once you have it in a vba array, it is nearly trivial to put it somewhere else in your workbook.  You might have to trap for the case where only 1 value is unique, since that looks like it puts it in a 1D array instead of a 2D array.
Comment: Also note that Empty cells are treated differently than cells with empty strings if you use unique.  Your string comparison method wouldn't make that distinction, so if that's an undesired feature, you'd have to trap for that.


How to disable CSS in Python Selenium using ChromeDriver using ChromeOptions

Question: I try to show the page without CSS to make loading faster, and I have managed to disable images and javascript using the following code:
option = webdriver.ChromeOptions()
prefs = {'profile.default_content_setting_values': {'images': 2, 'javascript': 2}}
option.add_experimental_option('prefs', prefs)
driver = webdriver.Chrome(chrome_options = option)

So, I changed:
{'profile.default_content_setting_values': {'images': 2, 'javascript': 2}}

to
{'profile.default_content_setting_values': {'css': 2}

and thought it would disable CSS, but it didn't work.
I have seen many answers for Firefox, now I want to do this in Chrome.
Comment: Are you trying to show the page without CSS or prevent downloading of CSS files?
Comment: I want to show the page without CSS to make loading faster
Answer: To show the page without CSS, to make page loading faster you can disable the Preferences used for storing the default values for the individual content settings using the following solution:

Code Block:
from selenium import webdriver

options = webdriver.ChromeOptions()
prefs = {'profile.default_content_setting_values': {'cookies': 2, 'images': 2, 'javascript': 2,
                            'plugins': 2, 'popups': 2, 'geolocation': 2,
                            'notifications': 2, 'auto_select_certificate': 2, 'fullscreen': 2,
                            'mouselock': 2, 'mixed_script': 2, 'media_stream': 2,
                            'media_stream_mic': 2, 'media_stream_camera': 2, 'protocol_handlers': 2,
                            'ppapi_broker': 2, 'automatic_downloads': 2, 'midi_sysex': 2,
                            'push_messaging': 2, 'ssl_cert_decisions': 2, 'metro_switch_to_desktop': 2,
                            'protected_media_identifier': 2, 'app_banner': 2, 'site_engagement': 2,
                            'durable_storage': 2}}
options.add_experimental_option('prefs', prefs)
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get('https://play.google.com/store')

Browser Snapshot:
Comment: I know this is old but I am looking for the answer too - the screenshot shows images are disabled, but CSS is not, otherwise the layout would not be working, right? And the code block shown does not mention stylesheets ... does it mean it is not possible to disable CSS?
Comment: How would we do this with Edge driver?
Answer: Unfortunately, it is not possible in Chrome.
ChromeDriver disable rendering and CSS
Chrome/chromedriver cannot run headless without some virtual screen like Xserver, and it can't not-render the html and css.
Exhaustive list of all options that are allowed:
profile.default_content_setting_values:
cookies,
images,
javascript,
plugins,
popups,
geolocation,
notifications,
auto_select_certificate,
fullscreen,
mouselock,
mixed_script,
media_stream,
media_stream_mic,
media_stream_camera,
protocol_handlers,
ppapi_broker,
automatic_downloads,
midi_sysex,
push_messaging,
ssl_cert_decisions,
metro_switch_to_desktop,
protected_media_identifier,
app_banner,
site_engagement,
durable_storage

Answer: I've been wondering the same thing, and came across this (credit to https://www.scrapehero.com/how-to-increase-web-scraping-speed-using-puppeteer/)
It's not selenium, but in Pyppeteer you can intercept and modify network requests made by the page using setRequestInterception. This would allow you to intercept and abort css network requests, which would avoid downloading css files and their subsequent rendering.
Relevant Pyppeteer Docs


Evaluating $\int\frac{x^b}{1+x^a}~dx$ for $a,b\in\Bbb N$

Question: In this previous answer, MV showed that for $n\in\Bbb N$,
$$\int\frac1{1+x^n}~dx=C-\frac1n\sum_{k=1}^n\left(\frac12 x_{kr}\log(x^2-2x_{kr}x+1)-x_{ki}\arctan\left(\frac{x-x_{kr}}{x_{ki}}\right)\right)$$
where
$$x_{kr}=\cos \left(\frac{(2k-1)\pi}{n}\right)$$
$$x_{ki}=\sin \left(\frac{(2k-1)\pi}{n}\right)$$
I am now interested in the case of $n=\frac ab\in\Bbb Q^+$. By substituting $x\mapsto x^b$, we get
$$\int\frac{bx^{b-1}}{1+x^a}~dx$$
Thus, the given integral in question is really
$$\int\frac{x^b}{1+x^a}~dx$$
By expanding with the geometric series and termwise integration, one can see that
$$\int_0^p\frac{x^b}{1+x^a}~dx=\sum_{k=0}^\infty\frac{(-1)^kp^{ak+b+1}}{ak+b+1}=\frac{p^{b+1}}a\Phi\left(-p^a,1,\frac{b+1}a\right)$$
where $\Phi$ is the Lerch transcendent.
A few particular cases that arise may be found:
\begin{align}\int\frac1{1+x^{1/n}}~dx&=C+(-1)^{n+1}n\left[\ln(1+x^{1/n})+\sum_{k=1}^{n-1}\frac{(-x^{1/n})^k}k\right],&a=1\\\int\frac1{1+x^{2/n}}~dx&=C+(-1)^nn\left[\arctan(x^{1/n})+\frac1{x^{1/n}}\sum_{k=1}^{(n-1)/2}\frac{(-x^{2/n})^k}{2k-1}\right],&a=2,n\ne2b\end{align}
Or, more generally, with $x=t^{an+1}$,
$$\int\frac1{1+x^{a/(an+1)}}~dx=(-1)^{n+a}(an+1)\left[\int\frac1{1+t^a}~dt+\frac1{x^{(a-1)/(an+1)}}\sum_{k=1}^{(n-1)/a}\frac{(-x^{a/(an+1)})^k}{a(k-1)+1}\right]$$
which reduces down to the previously solved problem.

But what of the cases when $n=a/b$ with $(b\bmod a)\ne0,1$?

For example,
$$\int\frac1{1+x^{3/2}}~dx=C+\frac16\left[\log(1-x^{1/2}+x)-2\log(1+x^{1/2})+2\sqrt3\arctan\left(\frac{2x^{1/2}-1}{\sqrt3}\right)\right]$$
Comment: From [wolfram](https://www.wolframalpha.com/input/?i=integral+x%5Eb%2F(1%2Bx%5Ea)) and the previous answer, the only approach I could think of would be seeing if the hypergeometric function form reduces in the case where a/b is rational.
Comment: @Tyberius :-P In terms of complexity, I place Lerch transcendents below hypergeometric functions, but yes, it can also be written in terms of hypergeometric functions.
Answer: Let $\gamma=\exp(2\pi i/a)$.
The polynomial $Q(x)=x^a+1$ has $a$ simple roots $z_k=\gamma^{k+1/2}$,
$0\leq k<a$.  Since $z_k^a=-1$, we have $Q'(z_k)=a z_k^{a-1}=-a z_k^{-1}$, so
  $$ \frac{1}{Q(x)} =
  \sum_{k=0}^{a-1} \frac{1}{Q'(z_k)} \frac{1}{x-z_k} =
  -\frac{1}{a} \sum_{k=0}^{a-1}  \frac{z_k}{x-z_k} $$
For the numerator we first make a reduction in the degree (when $b\geq a$).
Let $p= b \mod a \in \{0,1,...,a-1\}$ and $m=(b-p)/a$. Then
$$ \frac{x^b - (-1)^m x^p}{x^a + 1} =
\sum_{j=1}^m (-1)^j x^{b-ja}
$$
We deduce that
$$ \frac{x^b}{x^a+1} -
\sum_{j=1}^m (-1)^j x^{b-ja}  = \frac{(-1)^mx^p}{x^a+1} =
  - \frac{(-1)^m}{a} \sum_{k=0}^{a-1} z_k \frac{x^p}{x-z_k}=
  - \frac{(-1)^m}{a} \sum_{k=0}^{a-1}  \frac{z_k^{p+1}}{x-z_k} $$
The last equality follows from the fact that the difference is a polynomial
which must vanish since $p<a$. So a part from the trivial part on the LHS
(which I leave aside),
the problem is reduced to integrating
the RHS. We have
$$
   - \int \frac{(-1)^m}{a} \sum_{k=0}^{a-1}  \frac{z_k^{p+1}}{x-z_k}dx
   =
   -  \frac{(-1)^m}{a} \sum_{k=0}^{a-1}  z_k^{p+1}
    \ln (x-z_k) $$
To avoid complex log and too long formulas,
let us write
$$ u_{k,p} =
  \cos \left(  \frac{2\pi (k+1/2)(p+1)}{a}\right) , \; \;
  v_{k,p} =
  \sin \left(  \frac{2\pi (k+1/2)(p+1)}{a}\right) , \; \;
$$
Using
$\overline{z_{a-1-k}} = z_{k} = u_{k,0}+i v_{k,0}$ we obtain for $a$ even:
   $$
   -  \frac{(-1)^m}{2a} \sum_{k=0}^{\lfloor a/2 \rfloor}
    \left[ u_{k,p} \ln \left(x^2- 2  u_{k,0}
    x+1\right) +
      v_{k,p} \arctan
      \frac{x-u_{k,0}}{v_{k,0}} \right]
   $$
For $a$ odd you should add to this expression
the "middle term" (which has no arctan part)
$$
     \frac{(-1)^{m+p}}{a}
     \ln(x+1) $$
No guarantee for the above being free of errors ...
Answer: One can use this answer for evaluating $\dfrac1{1+x^a}$ in the following form
$$\frac1{1+x^a}=\sum_{k=1}^aa_k(x-x_k)^{-1} \tag {2}$$
where $a_k=\frac{-x_k}{n}$ and $x_k=e^{i(2k-1)\pi/n}$, $k=1, \cdots,n$
After that this integral can be evaluate
$$\int\frac{x^b}{1+x^a}~dx=\int\sum_{k=1}^aa_k(x-x_k)^{-1} (x-x_k+x_k)^b=
\int\sum_{k=1}^aa_k(x-x_k)^{-1} \sum_{l=0}^b C_b^l(x-x_k)^l x_k^{b-l} $$
$$
=-\sum_{k=1}^a\Big(\sum_{l=1}^b\frac{C_b^l}{ln}(x-x_k)^l x_k^{b-l+1}+x_k^{b+1}\log(x-x_k)\Big)
$$
Comment: Would prefer the solution to only involve real numbers, but will see where this leads. You also appear to have a typo on the last line.


count the number of calls of a clause

Question: I have  a clause like following:
lock_open:-
        conditional_combination(X),
        equal(X,[8,6,5,3,6,9]),!,
        print(X).

this clause succeed. But I want to know how many times conditional_combination() is called before equal(X,[8,6,5,3,6,9]) is become true. the program is to generate a permutation by following some rules. And I need to how many permutation is need to generate to get a particular value like 865369.
Answer: What you actually want is something slightly different: You want to count the number of answers (so far) of a goal.
The following predicate call_nth(Goal_0, Nth) succeeds like call(Goal_0) but has an additional argument which indicates that the answer found is the n-th answer. This definition is highly specific to SWI or YAP. Do not use things like nb_setarg/3 in your general programs, but use them for well encapsulated cases as this one. Even within
those two systems, the precise meaning of these constructs is not well defined for the general case. Here is a definition for SICStus. Update: use unsigned_64 in newer versions instead of unsigned_32.

call_nth(Goal_0, Nth) :-
   nonvar(Nth),
   !,
   Nth \== 0,
   \+arg(Nth,+ 1,2), % produces all expected errors
   State = count(0,_), % note the extra argument which remains a variable
   Goal_0,
   arg(1, State, C1),
   C2 is C1+1,
   (  Nth == C2
   -> !
   ;  nb_setarg(1, State, C2),
      fail
   ).
call_nth(Goal_0, Nth) :-
   State = count(0,_), % note the extra argument which remains a variable
   Goal_0,
   arg(1, State, C1),
   C2 is C1+1,
   nb_setarg(1, State, C2),
   Nth = C2.

A more robust abstraction is provided by Eclipse:
call_nth(Goal_0, Nth) :-
   shelf_create(counter(0), CounterRef),
   call(Goal_0),
   shelf_inc(CounterRef, 1),
   shelf_get(CounterRef, 1, Nth).

?- call_nth(between(1,5,I),Nth).
   I = Nth, Nth = 1
;  I = Nth, Nth = 2
;  I = Nth, Nth = 3
;  I = Nth, Nth = 4
;  I = Nth, Nth = 5.

So simply wrap it around:

lock_open :-
   call_nth(conditional_combination(X), Nth),
   X = [8,6,5,3,6,9],
   !,
   ....
Comment: I see a way to do an **aggregator** O(N) in time **and** space using such primitive. Thanks!
Comment: Have to rethink the limit/2 and offset/2 implementation, maybe the more primitive and universal predicate would be call_nth/2.
Comment: I did not realize that a goal can be called like that (the third line of the `call_nth/2` listing). I thought one always needs `call(Goal)`, but obviously, just `Goal` is enough!
Answer: If you are using SWI prolog you can use nb_getval/2 and nb_setval/2 to achieve what you want:
lock_open:-
  nb_setval(ctr, 0),  % Initialize counter
  conditional_combination(X),
  nb_inc(ctr),  % Increment Counter
  equal(X,[8,6,5,3,6,9]),
  % Here you can access counter value with nb_getval(ctr, Value)
  !,
  print(X).

nb_inc(Key):-
  nb_getval(Key, Old),
  succ(Old, New),
  nb_setval(Key, New).

Other prologs have other means to do the same, look for global variables in your prolog implementation. In this snippet I used the term ctr to hold the current goal counter. You can use any term there that is not used in your program.
Comment: The point was that you cannot guarantee that without inspecting your entire program.
Comment: A call to `nb_setval/2` within `conditional_combination/1` would affect the result. A name like `ctr` might be used for some other counting...
Comment: @false: true, OP should use any term not used in his program.
Comment: You are right, but if you take that position then you would never use assert/call/retract or in fact anything with global effect. In fact just, like in any language, if you use anything global you have to be cautious...
Comment: I disagree: Updating a knowledge base needs something like assert/retract.  And there are ways to prevent that multiple uses of a global are possible. Take as an example `findall/3` using asserta/retract.  You cannot nest such calls, nor can you have two calls at the same time.
Answer: While working on a module "micro", I recently invented pivots. They are inspired by the thread / pipe pattern to pass around data. A pivot is a bounded queue of maximum length one, the pivot_put/1 does a copy of the given term as well. But for performance reasons they don't use a synchronized and are non-blocking.
In as far they are very similar to nb_setarg/3, except that they don't destruct a Prolog term, but instead they update a Java data structure. As a result they are little bit safer than the non-logical term operations. Also they don't need some call_cleanup/3, since they are Java garbage collected.
In as far they are more similar than nb_setarg/3, than using some explicit allocate and dealloccate of structures. So for example a solution for SICStus Prolog could be:
call_nth(Goal_0, Nth) :-
   new(unsigned_32, Counter),
   call_cleanup(call_nth1(Goal_0, Counter, Nth),
           dispose(Counter)).

call_nth1(Goal_0, Counter, Nth) :-
   call(Goal_0),
   get_contents(Counter, contents, Count0),
   Count1 is Count0+1,
   put_contents(Counter, contents, Count1),
   Nth = Count1.

With pivots, there is even no 32-bit limitation, and we can directly do:
call_nth(G, C) :-
   pivot_new(P),
   pivot_put(P, 0),
   call(G),
   pivot_take(P, M),
   N is M+1,
   pivot_put(P, N),
   C = N.

count the number of calls of a clause


Counting Biased Coins

Question: Edit Note:
While this question is very interesting and relevant in its own right, I have come to a realisation that I have to make it a bit more complicated in order for it to be applicable to my actual research question. In order to avoid confusion, I have asked it as a follow-up question.
Problem:
There is a device that produces coins. Most of the time it makes unbiased coins, but sometimes it makes biased coins. Let $q$ be the probability of producing a biased coin. Biased coins can have different bias. There is no prior knowledge about the distribution of these biases. However, in practice we frequently see strongly-biased coins.
We use the device to produce $N=100$ coins. Then we toss each coin $M=200$ times and record the results. We would like to test if $q > 0$, with the null hypothesis that $q=0$. We would like to estimate $\hat q$ and get a confidence interval for it.
Clearly, the problem is unsolvable if we allow for coins of arbitrarily small but non-zero bias. In order to circumvent this problem, we will only be counting the strongly biased coins.
Given the bias of the coin $b$ we will define a coin as strongly-biased if $|b-0.5|\geq 0.05$. So the goal is to estimate the fraction of strongly biased coins $\hat q'$ that the device produces, as well as its confidence interval.
Attempt: I have attempted a non-parametric approach. It seems to work in practice, but I'm not sure if the mathematics behind it is solid, or if it is the most powerful test I can do.

For each coin, perform a binomial test against the null hypothesis, obtain a p-value
Threshold each of the obtained p-values to $T = 5\%$, obtain a list of 1s and 0s
Perform a binomial test on that list, with null hypothesis $P[1]=T$ and alternative hypothesis $P[1] > T$.

The logic behind it is that by testing multiple coins, some of them will test positive by chance. In order to answer the first question, we test if the number of coins we have classified as biased significantly exceeds that chance. My worry about this solution is that the threshold is somewhat arbitrary, that I don't know how it relates to the above definition of strongly-biased coins, and that I don't have a confidence interval.
It would be cool to alternatively try a model-based approach, addressing the hierarchical nature of this random process directly. However, I know nothing at all about the parametric methods used to tackle such problems. Any suggestions, names, links to literature are appreciated.
Note: This is a minimal example of a problem I have encountered in experimental design in neuroscience. I have judged that further details are unnecessary to make progress on this problem.
Answer: There are certainly many ways you could construct a test statistic giving the evidentiary ordering for this hypothesis test.  Your method is coherent, but it is unlikely to be powerful.  Since there is no a priori information on the degree of bias in a biased coin, a natural thing to do here would be to compute the maximum deviation from the expected value under fairness over all of the coins (or some simple scaled version of this).  To do this, let $X_{i,j}$ denote the outcome of coin $i$ on toss $j$ and construct the test statistic:
$$R \equiv R(\mathbf{X}) \equiv 2 \times \max_{0 \leqslant i \leqslant N} \Bigg| \frac{1}{M} \sum_{j=1}^M X_{i,j} - \frac{1}{2}  \Bigg|.$$
This test statistic is a scaled version of the maximum deviation from the expected value under fairness, where the scaling is used so that it takes on values in the range $0 \leqslant R \leqslant 1$.  Larger values of the test statistic are more conducive to the alternative hypothesis that there is some non-zero probability of bias, so this gives a simple evidentiary ordering for the hypothesis test.  Use of this evidentiary ordering gives the p-value function:
$$\begin{align}
p(\mathbf{x})
&\equiv \mathbb{P}(R(\mathbf{X}) \geqslant R(\mathbf{x}) | H_0) \\[16pt]
&= 1 - \mathbb{P}(R(\mathbf{X}) < R(\mathbf{x}) | H_0) \\[12pt]
&= 1 - \prod_{i=1}^N \mathbb{P} \Bigg( M-R(\mathbf{x}) < \sum_{j=1}^M X_{i,j} < R(\mathbf{x}) \Bigg| H_0 \Bigg) \\[6pt]
&= 1 - \prod_{i=1}^N \mathbb{P} \Bigg( M-R(\mathbf{x})+1 \leqslant \sum_{j=1}^M X_{i,j} \leqslant R(\mathbf{x})-1 \Bigg| H_0 \Bigg) \\[6pt]
&= 1 - \Bigg( \sum_{s=M-R(\mathbf{x})+1}^{R(\mathbf{x})-1} \text{Bin}(s|M, \tfrac{1}{2}) \Bigg)^N. \\[6pt]
\end{align}$$
(Note that the summation here is non-empty if and only if $R(\mathbf{x}) \geqslant \frac{2}{M}$, so if $R(\mathbf{x}) < \frac{2}{M}$ then we have a p-value of one.)  Note that this test assumes that all coins are tossed the same number of times; if you were to remove this assumption then things would become more complicated and a generalised test statistic (accounting for the number of trials for each coin) would be desirable.

Programming the test: We can program this as a custom hypothesis test in R as follows.  The input for the function is a binary matrix with rows corresponding to different coins (test objects) and columns corresponding to different trials.
binary.bias.test <- function(x) {

#Check validity of inputs
if (!is.matrix(x))             {
  stop("Error: Input x should be a binary matrix") }
if (sum(x == 0) + sum(x == 1) != length(x)) {
  stop("Error: Input x should be a binary matrix") }

#Set description of test and data
N           <- nrow(x)
M           <- ncol(x)
method      <- "Binary bias test"
data.name   <- paste0("data matrix ", deparse(substitute(x)), " with outcomes of ", N,
                      " test objects used for ", M, " trials each")
null.value  <- NULL
alternative <- "non-zero probability that test objects are \"unfair\""

#Calculate test statistics and p-value
RX          <- 2*max(abs(rowSums(x)/M - 0.5))
estimate    <- RX/2
statistic   <- RX
attr(estimate,  "names") <- "estimated maximum deviation from fairness"
attr(statistic, "names") <- "R"
if (statistic < 2/M) {
  p.value <- 1    } else {
  p.value <- 1 - (pbinom(RX-1, size = M, prob = 0.5) -
                  pbinom(M-RX, size = M, prob = 0.5))^N }

TEST        <- list(method = method, data.name = data.name,
                    null.value = null.value, alternative = alternative,
                    estimate = estimate, statistic = statistic, p.value = p.value)
class(TEST) <- "htest";
TEST }

Now let us examine this test using an example where we take $q = 0.03$ and we take $p \sim \text{U}(0,1)$ for a biased coin.  As can be seen from the example, the test identifies the non-zero probability of bias in the coin, with a p-value of $p < 2.2 \times 10^{-16}$ for the test.
#Generate data
set.seed(1)
N <- 100
M <- 200
q <- 0.03
DATA <- matrix(0, nrow = N, ncol = M)
for (i in 1:N) {
  BIASED <- (runif(1) <= q)
  if (!BIASED) { DATA[i,] <- sample(c(0, 1), size = M, replace = TRUE) }
  if (BIASED)  { PROB     <- runif(1)
                 DATA[i,] <- sample(c(0, 1), size = M, replace = TRUE,
                                    prob = c(1-PROB, PROB)) } }

#Apply the test
binary.bias.test(DATA)

        Binary bias test

data:  data matrix DATA with outcomes of 100 test objects used for 200 trials each
R = 0.77, p-value < 2.2e-16
alternative hypothesis: non-zero probability that test objects are "unfair"
sample estimates:
estimated maximum deviation from fairness
                                    0.385

Comment: I find this maximum a smart way to tackle the problem. The distribution of the maximum absolute deviation will shift to larger values in both cases that the size of the bias increases or that the frequency of the biased coins increases. But, I wonder whether possibly a different order statistic might be more powerful. For instance, if the frequency is high but the bias is small then possibly using the second largest deviation might be more sensitive. Possibly some scheme like the Holm Bonferroni method could be more powerful.
Comment: This nested maximum is a bit difficult to decipher. I imagine that this might be more easy to write down.
Comment: Possibly --- I'm open to you proposing an edit (which I might reject!) if you have a preferable way of writing it.
Comment: I would write it by expressing it like the absolute value of the deviation (instead of the maximum of the negative/positive deviation) $$R \equiv R(\mathbf{X}) \equiv  \max_{0 \leqslant i \leqslant N} \Bigg|\frac{1}{M} \sum_{j=1}^M (2 X_{i,j} -1)\Bigg|$$ I was also thinking about abbreviating the sum as a mean  $$R \equiv R(\mathbf{X}) \equiv  \max_{0 \leqslant i \leqslant N} \Bigg| (2 \bar{X}_{i} -1)\Bigg| \quad \text{with} \quad \bar{X}_{i} = \frac{1}{M} \sum_{j=1}^M X_{i,j}$$ but this notation for a mean with two indexes might not be so clear (are there alternatives?).
Comment: @SextusEmpiricus: Yes, that looks nicer --- edited.
Comment: Dear Ben, thank you for your effort. I completely agree with you - this procedure looks to be very powerful in testing the null hypothesis (that all coins are fair). + it has the nice R code, which I am still learning and will thoroughly digest in the coming days. It would still be great to also properly address the 2nd part of the question, namely, the estimation of the number of the biased coins. I'm already happy, but if you get more ideas about the 2nd part I would appreciate if you post them
Comment: @SextusEmpiricus: Without some assumptions about the magnitude of the bias, I doubt that there is any unifromly most powerful test here.  I suspect that use of other order statistics would be more powerful in some cases and less in others.  Use of the maximum is likely to be best when the deviation of $q$ from zero is small, since this gives many cases where only one coin is biased (best detected by the maximum).  In my view the maximum is the most natural orderr statistic to use because it is simplest, and it is the best at detecting small probabilities of bias.
Answer: There are a few solutions from a Fequentist perspective, but I'd like to offer a Bayesian perspective.
The Model
It would be useful to model the data generating process as a mixture of binomials; one with 0 bias and the other with some non-zero bias left to be estimated.  You can then partially pool the data from all the coins to help estimation.  Here is the model I'm using in math...
$$ \mu \sim \operatorname{Beta}(1,1)$$
$$ \kappa \sim \operatorname{Half-Cauchy}(0,1) $$
$$ p \sim \operatorname{Beta}(1,1) $$
$$ b_i \sim \operatorname{Beta}(\mu \times \kappa, (1-\mu) \times \kappa) $$
$$ y_i \sim p \operatorname{Binom}(n_i, b_i) + (1-p)\operatorname{Binom}(n_i, 0.5)$$
Here, $p$ is the probability of producing a biased coin, $b_i$ is the probability of heads for the biased coins, which is assumed to come from a beta distrbution with parameters $\alpha = \mu \times \kappa$ and $\beta= (1-\mu) \times \kappa$.
If I were to write this in Stan, it might look like
data{
    int n;
    int heads[n];
    int coin[n];
  }
  parameters{
    real<lower=0, upper=1> prob_of_bias;
    vector<lower = 0, upper = 1>[n] b;

    real<lower=0, upper=1> mu;
    real<lower=0> kappa;
  }
  model{
    prob_of_bias ~ beta(1,1);
    mu ~ beta(1,1);
    kappa ~ cauchy(0, 1);
    // the prior below takes a parameterization in terms of mu and kappa
    // and turns it into alpha beta parameterization
    b ~ beta_proportion(mu,kappa);
    for (i in 1:n){
    target+=log_mix(prob_of_bias,
                     binomial_lpmf(heads[i] | 200, b[i]),
                     binomial_lpmf(heads[i] | 200,  0.5));
  }

Here is a simulation example.  The probability of generating a biased coin is 0.77.  Here is the simulation code
n = 100 #Number coins
is_biased = rbinom(n, 1 , 0.77)
b = 0.5 + is_biased*(rbeta(n, 90, 10) - 0.5)
x = rbinom(n, 100, p) #100 flips
d = tibble(coin = factor(1:n), heads = x)

Fitting this model, we see the estimated posterior means of each parameter recover their true values to within 1 decimal place (which is fine because the credible intervals capture the truth).
Your Questions
The question about the fraction of strongly biased coins has now been turned into a question of estimating $p$ from the model -- which is the probability of drawing a biased coin. We can obtain a variety of credible intervals but perhaps the easiest would be to just compute quantiles of the posterior for $p$.
Additionally, we can examine the probability that any given coin is biased by computing the posterior probability directly.  Appending the following generated quantities block to the Stan model
 generated quantities{
  matrix[n,2] ps;
  for (i in 1:n) {
    vector[2] pn;
    // log-probability that there is bias
    pn[1] = log(prob_of_bias) + binomial_lpmf(heads[i] | 200, b[i]);
    // log-probability that there is no bias
    pn[2] = log1m(prob_of_bias) + binomial_lpmf(heads[i] | 200,  0.5);
    // posterior probabilities for bias and no bias
    ps[i,] = to_row_vector(softmax(pn));
  }

Will produce a matrix in which the rows correspond to coins and the columns are probabilities of being biased and not being biased respectively.
As an example, the first few coins from my simulation are
52 83 90 85 59

The middle 3 are biased coins.  The first 5 rows of the matrix ps rounded to 4 decimal places are
      P(bias) P(No bias)
    [1,]       0          1
    [2,]       1          0
    [3,]       1          0
    [4,]       1          0
    [5,]       0          1

In this case, the probabilities are quite large, but when the bias is smaller these probabilities might be more reasonable (e.g. 30% and 70%).  In such a case, you have a probabilistic model for bias and can leverage techniques from that literature (or from Bayesian decision theory, setting some sort of utility function) to decide if a coin is biased or not.
I don't expect you to actually use this approach, and I'm  sure similar approaches can be made with Expectation Maximization.  Just thought I would throw my in two cents in case someone else found it interesting. Additionally, if this is something you are interested in, I can add details as needed.
Comment: Thanks a lot, this is very interesting. I will need a bit of time to digest what is going on. I see you use some kind of hierarchical model, where parameters themselves are random variables. That sounds highly relevant, but I don't know at all how to work with such constructions. Could you hint me to the name of this construction (is it simply "hierarchical bayesian model"?), and perhaps lecture notes or any place where I can see a basic example solved with as much detail as possible? Thanks again for your effort
Comment: Yes,  "hierarchical bayesian model" is the name of the construction.  Chapter 5 of Bayesian Data Analysis of Gelman et. al is a good resource.
Comment: Ok, I understand now that if I follow this route, I will eventually want to arrive at the marginal posterior $P[p|Data]$, where $p$ is the fraction of biased coins. This will involve computing some complicated integrals over the joint distribution. I don't know Stan at all. To what extent can it simplify my life for this task. I could write all expressions explicitly in python and use some Gibbs sampler to do the integrals numerically. Is this procedure automatized in Stan? How hard is the installation / how steep is the learning curve?
Comment: @AleksejsFomins You've understood correctly.  I'm not sure how much you could simplify the task.  Gibbs samplers take a long time to converge to the posterior, so learning Stan is your best bet.  Alternatively, pymc3 is a library in python which does the same thing.  Installation is easy (see cmdstanpy library), but the learning curve is steep for the first little bit.
Comment: @AleksejsFomins I can try to help if you'd like
Comment: Thanks, you are very kind. However, I don't think I immediately have time to learn Stan. I expect that it would take me like a week until I am able to produce desired results with it. I can come back to learning it when I am under less pressure to publish. I'm a bit surprised when you say that Gibbs sampler is slow. Sure, I have heard that VB was designed exactly because Gibbs was slow for large-scale problems. But this problem is a 2D integral of a well-behaved function of ~200 summands. I don't believe it would take more than a few seconds to integrate
Comment: @AleksejsFomins I didn't say Gibbs was slow, I said it was slow to converge *to the target distirbution*.  You'd need far more samples as compared to using Hamiltonian Monte Carlo.  In any case, the time constraint is understandable.  I didn't even intend on this being taken seriously; I thought it was just an interesting approach.
Comment: Sorry, I did not mean to sound too negative. I have learned a lot from your suggestions. One day this will open doors for me
Answer: I am not sure just what your question is asking you to do,
Here are some relevant computations from R:
Test $H_0:p=.5$ vs $H_a: p < .5$ at level 5% (or lower).
If $p = .5,$ then $P(X \le 41) = 0.0443,$ where $X$ is the number of Heads in 100 tosses. So you can have a one-sided test at
level 4.4%. Changing the critical value to 42 would give
a test above the 5% level.
pbinom(41, 100, .5)
[1] 0.04431304
pbinom(42, 100, .5)
[1] 0.06660531

If $p = .45$ then $P(X \le 41)= 0.2415.$
So the power of the test at level 4.4% is only about 24%
Thus, only about 24% of coins biased with $p = 0.45$ will
be detected as biased (using critical value 41).
pbinom(41, 100, .45)
[1] 0.2414906

By contrast, if the critical value 41 is used, a coin biased with $p=0.3$ will be detected a little
more than 99% of the time.
pbinom(41, 100, .3)
[1] 0.9928264

Thus, for a test at level just below 5%, 100 tosses is enough to do a pretty good job of
detecting coins with bias $p = .45,$ but not so good
at detecting biased with $p = .45.%
Note: If you use normal approximations for the
binomial probabilities, you might delude yourself into
thinking you have a test at exactly the 5% level,
but that is an artifact of the continuous normal
approximation:
$\mu = np = 50,\;  \sigma = \sqrt{np(1-p)} = 5.$
qnorm(.05, 50, 5)
[1] 41.77573

For the discrete binomial distribution, "critical value" 41.78 amounts to integer 41.

Counting Biased Coins


pushViewController in Xcode

Question: I'm trying to push a view from a table cell to another view, I know that there's a lot of tutorials out there, I've tried a lot of them, been trying for 2 days and still can't get it working, here's my code..
AppDelegate.h
#import <UIKit/UIKit.h>

@class RootViewController;
@interface AppDelegate : UIResponder <UIApplicationDelegate>

@property (strong, nonatomic) UIWindow *window;

@property (strong, nonatomic) RootViewController *rootViewController;
@end

AppDelegate.m
#import "AppDelegate.h"

#import "RootViewController.h"
@implementation AppDelegate

@synthesize window = _window;
@synthesize rootViewController;
- (void)dealloc
{
    [_window release];
    [super dealloc];
}

- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions
{
    self.window = [[[UIWindow alloc] initWithFrame:[[UIScreen mainScreen] bounds]] autorelease];
    // Override point for customization after application launch.
    self.rootViewController = [[[RootViewController alloc] initWithNibName:@"RootViewController" bundle:nil] autorelease];
    self.window.rootViewController = self.rootViewController;
    [self.window makeKeyAndVisible];
    return YES;
}

@end

RootViewController.h
#import <UIKit/UIKit.h>

@interface RootViewController : UITableViewController{

    NSMutableArray* array;

}
@property(nonatomic, retain)NSMutableArray* array;
@end

RootViewController.m
#import "RootViewController.h"
#import "SubLevelViewController.h"
@implementation RootViewController
@synthesize array;
- (void)viewDidLoad {
    [super viewDidLoad];

    array = [[NSMutableArray alloc] init];

    [array addObject:@"One"];
    [array addObject:@"Two"];
    [array addObject:@"Three"];

    // Uncomment the following line to display an Edit button in the navigation bar for this view controller.
    self.navigationItem.rightBarButtonItem = self.editButtonItem;
}

- (void)didReceiveMemoryWarning {
    // Releases the view if it doesn't have a superview.
    [super didReceiveMemoryWarning];

    // Release any cached data, images, etc that aren't in use.
}

- (void)viewDidUnload {
    // Release anything that can be recreated in viewDidLoad or on demand.
    // e.g. self.myOutlet = nil;
}

#pragma mark Table view methods

- (NSInteger)numberOfSectionsInTableView:(UITableView *)tableView {
    return 1;
}

// Customize the number of rows in the table view.
- (NSInteger)tableView:(UITableView *)tableView numberOfRowsInSection:(NSInteger)section {
    return [array count];
}

// Customize the appearance of table view cells.
- (UITableViewCell *)tableView:(UITableView *)tableView cellForRowAtIndexPath:(NSIndexPath *)indexPath {

    static NSString *CellIdentifier = @"Cell";

    UITableViewCell *cell = [tableView dequeueReusableCellWithIdentifier:CellIdentifier];
    if (cell == nil) {
        cell = [[[UITableViewCell alloc] initWithStyle:UITableViewCellStyleDefault reuseIdentifier:CellIdentifier] autorelease];
    }

    NSString *cellValue = [array objectAtIndex:indexPath.row];

    cell.textLabel.text = cellValue;

    // Configure the cell.

    return cell;
}

- (void)tableView:(UITableView *)tableView didSelectRowAtIndexPath:(NSIndexPath *)indexPath {

    SubLevelViewController *sub = [[SubLevelViewController alloc] initWithNibName:@"SubLevelViewController" bundle:nil];

    sub.title = @"My First View";

    [self.navigationController pushViewController:sub animated:YES];

}

- (void)dealloc {
    [array release];
    [super dealloc];
}

@end

and main.m
#import <UIKit/UIKit.h>

#import "AppDelegate.h"

int main(int argc, char *argv[])
{
    @autoreleasepool {
        return UIApplicationMain(argc, argv, nil, NSStringFromClass([AppDelegate class]));
    }
}

Comment: You'll need to give us more detail on what your problem is. What's happening? Are you getting an error?
Comment: It just doesn't be able to go to another view when I clicked on the table cells.
Answer: Actually, your problem is simple, you reference self.navigationController, yet the view controller has no navigation controller set up in the AppDelegate!  You're sending a message to nil, which produces nil (ergo, nothing happens).  Try this in the appDelegate:
- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions
{
    self.window = [[[UIWindow alloc] initWithFrame:[[UIScreen mainScreen] bounds]] autorelease];
    // Override point for customization after application launch.
    self.rootViewController = [[[RootViewController alloc] initWithNibName:@"RootViewController" bundle:nil] autorelease];
    UINavigationController * controller = [[[UINavigationController alloc]initWithRootViewController:self.rootViewController]autorelease];
    self.window.rootViewController = controller;
    [self.window makeKeyAndVisible];
    return YES:
}

Comment: I would be much obliged if someone could format this for me, seeing as I typed this out on an iPhone.
Answer: Agree with CodaFi, you need to create a UINavigationController and remember it to push and pop any UIViewController if you want to keep the push/pop stack. Do not use self.navigationController.
Comment: why do u mean by not use self.navigationController?

pushViewController in Xcode